How crazy can I get? part 2

5.8.2018 03:41 | Přečteno: 3126× | | poslední úprava: 5.8.2018 03:51

Let's get the insanity to continue...

OK these were funny early tests, but with a new engine I've had to solve the real limitations of shell. Let's go a little back first ... :-P

Input system

If you want to write any complex shell script with the user input you need to get the input data inside script variables. This actually seems to be impossible. You cannot make multithreaded scripts even in the bash and the interprocess communication is limited to the signal trapping or reading the data through pipes.

A possible solution would be to have one reader and multiple writers, where every writer can block-read the device (keys, touchscreen, ...). The problem is even with this "multiprocess" input reads it is probably impossible to parse stdin in a shell. Maybe in the bash but not in the busybox shell used in the phone.

The bash "read" function supports settings for the reading a specified number of characters and the timeout when they are not delivered. You need read a mupltiple bytes from the stdin for arrows, function keys, ctrl+something, etc... ). The busybox shell (an implementation of a posix Almquist shell) doesn't reliably support this (and in the past it only supported an integer number of the timeout in seconds). Funny thing: if a stdin read ends with a newline, the read function returns immediately.

As the parsing of a touchscreen events gone wrong too (otherwise you could have parsed an evtest utility or just a raw evdev input data) I've decided to write a C backend. The libc read() does block and it returns any number of characters (depends on the terminal setting). Same is the parsing of the touchscreen events where we can just use the kernel uapi headers for the evdev. And finally multiple threads in a C are a trivial thing.

This backend just formats all inputs (there is a timer event too) and write them all to the stdout where the shell script is listening with its own (dumb) read.

Almquist shell

Yeah the busybox doesn't even use bash. It is /bin/sh equivalent with some expansions which are compatible with bash (many features are configurable during build). The most obvious disadvantage is the lack of bash arrays, but it's fine, the language is still turing-universal even without them . I've actually never used arrays in a bash so yeah screw them. Well you don't need them until you want to write a GUI in a shell script.

So, back to the history...

ANSI escape codes craziness:

Wow what a nicely moving ζ version demo... (and what ugly colors). How this ugly beauty can be done in the shell?

BTW it's a rose on the background...

Do you know how unicode UTF-8 works for parsing? The ansi escape codes are a level of magnitude worse. They are basically a terminal control sequences (colors, goto, clear, bell or switching the fonts). That's why when you run cat /dev/urandom in a framebuffer terminal it can make all texts unreadable. The parsing of the ansi escape codes works this way: it doesn't. You would need to make a state machine in shell without arrays and it would be too much. Best thing is to design and to limit ourselves in a way where every string fits on its place. This is the first limitation of the shell GUI: you cannot have partially visible line, where the rest of it is outside the screen.

Problems begin if you actually want to write something new over something old, without a parsing hydra obviously. A simple solution is to just write the lowest layer (background), the middle layer (icons menu) and the top layer (flying window). With the layers the strings can be partially visible, but they still need to be rendered under a higher layer. The leaking colors from lower layers can bleed to a higher layer, but luckily there is an escape code "\x1b[0m" which resets all attributes (and it even works in the most cases). To fix the color leaks we just need to end every line of the layer with this code.

OK let's escape all strings, so there is no attribute leakage... oh and change the background:

But now we need to draw even invisible strings, so what is the solution to speed things now? Let's implement a z-buffer in the shell! ... Did I mention no arrays?

These versions had about 2-3 layers and every graphical object consisted of line elements. So how do we make a z-buffer? We can put every line of the every layer into as an item of an array. And as we don't have arrays, let's make them artificially by using the eval command.

In that way, a C equivalent of:

struct line {
    char *data;
};

struct line layer[40];
...
layer[idx].data="hello world";

became this:

eval MENU_LINE_${idx}=\"hello world\"

... yes, this makes a new variable for every line. No, there is no safety checking and if you do this:

badvar=" echo hello"
eval somevar="${badvar}"

The eval won't store the string in "somevar" variable, instead it will execute the echo command. This is because the spaces in a string assigned to the badvar variable will be split in the second stage of the eval. We would need to escape the quotation marks for the second stage of the eval like this:

badvar=" echo hello"
eval somevar=\"${badvar}\"

The problem is that this code is still wrong, let's demonstrate it by running this code:

badvar=" echo hello\" \"format c:"
eval somevar=\"${badvar}\"
...
bash: format c:: command not found

The evil \" in the string will get evaluated in the second stage so the command will effectively do this:

somevar=" echo hello" format c:

which is a normal command with a set environment variable. Now imagine escaping a string with spaces, unicode characters and ansi escape codes mixed with variables in an eval expression. Yum!

There are two ways to fix that: either use apostrophes everywhere or make the variable to evaluate in the second eval stage by escaping its dollar sign (too bad there is no third eval stage ... I'm really using double evals in the current code of the engine).

So what about doing it at the same time?

eval somevar=\'\${badvar}\'

Nope it gets into the second stage as '${badvar}' and things inside the apostrophes doesn't evaluate at all. We can chose only one of the two solutions. Are as we are making an artificial arrays, we need to evaluate the index in the first stage:

eval oneline=\"\${array_line_${idx}}\"

So only way to be at least a little bit save is by escaping the final variable name and hope the index variable has a safe content.

BTW we can make multidimensional sparse arrays this way too.

BTW the first idea of the array of the lines was to put it in one really long string and to do an offset strcpy:

output="${var:position:offset}"

It turned out that any string operation in the shell is really slow (as we will see in the future).

Z-buffer

We have individual lines of different layers, so let's make them ordered. For θ version it was easy:

there were hardcoded levels for the background (tux got colors!), scrollbar, frame focus and menu. When there was a change on the specified line the engine would just redraw it with a specified layer. All the modification events would be stored in the lisp-like string. For example an event buffer "1 A 2 B 3 C ":

while [ "$ZBUFFER_LINE" != "" ] ; do
    T_LINE="${ZBUFFER_LINE%% *}"
    ZBUFFER_LINE="${ZBUFFER_LINE#* }"
    T_LAYER="${ZBUFFER_LINE%% *}"
    ZBUFFER_LINE="${ZBUFFER_LINE#* }"
    ...

We can use a string pattern match and delete everything after the first space. We get T_LINE=1 that way, and then delete everything before and including with the first space and repeat the procedure for getting T_LAYER="A" ... and repeating the whole procedure until the buffer (ehm string) is empty.

Let's see how the command looked like for the line reading from arrays for a focus frame (yeah it had a hardcoded size):

elif  [ "$T_LAYER" == "C" ] ; then  #focus
    eval TEMP_FOCUS_LINE="\$FOCUS_LINE_$(($T_LINE-($FOCUS_LINE_Y_BEG-$LCD_LINE_CURR+$LCD_LINE_FIRST)))"

It is just a trivial computation for the position of the menu below and it is using an equation evaluation as a part of the "array" name. I actually found a bug in the busybox almquist shell with this code (can somebody tell me why nobody found it ahead of me? Such an everyday-use code ).

BTW during the development I've found an ansi escape function "\x1b[nM" and "\x1b[nL" which written to the stdout can scroll a region of the terminal by "n" lines up or down. This is a very useful function as you have to only redraw the new added lines on the bottom or on the top. With this I've got so much computation power I've could even use touchscreen to drag the menu up and down in nearly a realtime speed .

The final part of a flush is to dump the whole buffer to /dev/tty device.

echo -e -n "\x1b[11m${FB_CMD_MENU_SCROLL}${T_FB_BUFFER}${FB_FOCUS}${FB_POPUP}${FB_HOVER}${FB_SCROLL}" > $SERVICE_TTY

The code is inefficient as it is called every loop in a flush() function. If we run strace on the script we will find the redirection does an open-write-close sequence every loop, which is slow on PXA272 SoC (the context switches!). We need to have the stdout file descriptor constantly opened outside the main loop.

The horror of the main loop

As written before, the shell scripts are singlethreaded. Every input read for the whole system must be done in a single loop as it depends on an active pipe from the input backend and the shell "read" function. It is the only place where the code can block with the waiting for an event. This makes things like a "choice" dialog above a big challenge (one big state machine challenge).

In the current version the main loop takes just a few lines, but there were times it took about 20% lines of the whole project. But let's look on an actual version rather (with some spoilers):

exec 4>${c_service_tty}
rm ${c_event_fifo}
mkfifo ${c_event_fifo}
exec 3<>${c_event_fifo}

./backend 2>/dev/null >&3 <${c_service_tty} &

while read -s time type params ; do
    case ${type} in
    ms1|drag1|unms1)
        lib_engine_first_method_at_xy ${type} ${params}
        if [ ${ret_wid} -ge 0 ] ; then
            lib_engine_queue ${ret_wid} ${type} ${time} ${params}
        else
            continue
        fi
        ;;
    dump)
        dump
        continue
        ;;
    quit)
        break
        ;;
    menu)
        lib_engine_queue ${fs_start} menu ${time} ${params}
        ;;
    timer)
        lib_engine_timer_trig ${time}
        ;;
    *)
        #event on focused WID
        lib_engine_top_method_of_focus_tree ${type}
        if [ ${ret_wid} -ge 0 ] ; then
            lib_engine_queue ${ret_wid} ${type} ${time} ${params}
        else
            continue
        fi
        ;;
    esac

    lib_engine_flush

done <&3 >&4

One problem with this solution (except it is in a shell) is the use of an explicitly created named pipe (mkfifo). There is a hack to steal an anonymous pipe "|" from a subprocess, but it has timing problems.

The stdin/stdout descriptors on the end of the "while" loop makes sure there is no open/close kernel calls.

How to load an external "applet"?

Executing menu items (scripts with devmem/modprobe/gpio toggling) everytime from the filesystem would be slow. Why not preload all the applet code into a single shell function. Guess which built-in command we use? .... yup eval again.

So how we do that?

T_APPLET="applet_code() { ${LF} $(cat "${FILENAME}") ${LF} }"
eval "$T_APPLET"

And with multiple objects the situation gets better:

fcn="wid_${new_wid}_exec() { $(cat "${filename}") ; }"
eval "$fcn"

As the function name is stored as a string, we can dynamically construct names!

And it gets even better:

fcn="wid_${new_wid}_exec() { param1=${value1}; $(cat "${filename}") ; }"
eval "$fcn"

Why not add some constant code inside a general function?

I could continue with dynamic function renaming by using set, grep, sed and eval, but the behavior is not enough consistent between the shell implementations and it is an ugly code, so let's draw a line here. I left out the feature in a new rewrite of the engine actually.

But first let take a look what can be achieved with calls of a dynamically loaded script:

... a very fast (for a shell script) virtual touchscreen keyboard. The fields (icons) of all applets (= scripts = buttons) are changed by the top level layout of the keyboard. Written text can be run in a real linux terminal. The return button is just a directory change to "../" so the engine can load other applets (only one directory was loaded at the same time - variable names are still highly hardcoded).

One last change before the complete rewrite .. of course the background .

You can do it with any libcaca application by setting the environment variables, for example mplayer:

CACA_GEOMETRY=40x30 CACA_DRIVER=ncurses mplayer -vo caca ./tux2.png -really-quiet > /tmp/tux.nc

And what happens when you try to save the ansi escape code file in a variable:

In the next part we will have a fun with evaluation and finally some source code.

If you have found this idea funny or you want me to rewrite the engine in a more sane language, consider a donate to my BTC address: 1CqzWKDLdcbjbzvzkrcP5bHsScjm5CwcjH or to my ETH address: 0x1440d31eD70fc6b084c71f03590F6e1d121C919E

       

Komentáře

[Odin]

Tolik textu, jiste mnoho straveneho casu a efekt nijaky. Toto uz dnes neni cool a ani uzitecne.

[pc2005]

Tak to bych tedy nečetl další části tam to teprve nabere ty grády .

[_]

Česky prosím, nevidim přidanou hodnotu v louskání czenglish...

[Nechce se mi přihlasovat]

Číst to nemusíš. Ber to jako jazykové cvičení, jež se pisatel z nějakého důvodu rozhodl dát na web.

[pc2005]

Jsem to psal v angličtině s tím, že bych na konci dal odkazy na reddit, ale bystroushaak to už udělal za mě. Asi bych to mohl dát i do češtiny kdybych měl čas, klidně mě motivuj, viz adresy na konci :-P.

[Bystroushaak]

Na konci je tam určitě hoď všechny, ale jak jsem psal dole, asi bych dal lepší titulek o čem to je.

[Bystroushaak]

Good job. Kinda creepy, but nicely done at the same time. I've reposted your blogs to /r/programming to get some love for them:

• How crazy can I get? part 1
• How crazy can I get? part 2

[Bystroushaak]

Yeah. I should have used more descriptive titles for this subreddit.

[mech]

Dobrá práce jako vždy /s

[Bystroushaak]

¯\_(ツ)_/¯