Friday, November 6, 2015

2 issues: bash Shell Expansion, Command substitution and Quoting | ffmpeg (or mplayer) and loops in a shell

So yesterday I needed to execute a loop, written in a bash shell, to convert a bunch of files using ffmpeg, and as always, I tried to handle spaces in file names. quoting variable names is obvious, and it should get you going, but the issue was that in one instance It just failed ... and that simple one-liner had a problem ... or two.

I had a variable holding the file name passed to "dirname" ; to get the path of containing directory of that file. but since I was using it this lovely one-liner and i was using, obviously, command substitution, that beutiful $() ; or "`" enclosing that would capture the output of a command and put it wherever you use this syntax.

So, the problem:

variable $filename holds a File name that has a space:

"./files/camera 1/file 1.mov"

that should be easy to handle, so dirname "$filename".

now we wrap it with the command substitution magic : $(dirname "filename"), and then when we are using the whole things in a line such as cp /$somevariable/$(diname "filename") ... well, I need to quote that again, right, sure thing, and inside it I'll escape the internal quotes, simple...

Escaping, using single quotes jumping in the air did not help. after substantial amount of time, i use a workaround, I run it separately to populate a variable and then quote them all in one... but there must be a workaround!

turns out its fairly easy, yet not really simple when it comes to bash syntax, it would be simply

"$(dirname "$filename")" or further quoting if needed such as " /$somevariable/$(diname "$filename")" ... but why does that work ? why is it not interpreted as "/somevariable/$(dirname "+$filename+")" ? as you would expect from any similar quoting ... turns out that command substitution is "special" ... and would handle that case! I was in pain ... I was glad I figured it out, its easy and clean, but really broke my heart. and to me it seems that this is the only case where something similar happens, otherwise, you'll need to wrestle with quoting and escaping.

I'm ashamed as I should have known this - and frankly I probably did, back when I was doing everything using bash - when usually moaning while telling how adorable bash is; I decided to write this down, hope it'll help someone out.

Now, another issue was there, the script always processed the first line it was fed and then stopped, I was puzzled, echoing the commands instead of executing them demonstrated the desired behaviour, but not when executing them. luckily I had a vague memory of a similar behaviour in mplayer; so what happened is that i was feeding the lines to be processed to a while loop reading from stdin; and ffmpeg and mplayer both keep reading from stdin probably expecting control instructions from the keyboard as it assumes its being run in an interactive shell. so the workaround was simple, but I'm not sure why not present in ffmpeg/avconv man pages, but you can read about it here as well. its to add the option "-nostdin" to the ffmpeg command.

On the Eastern shores of the Mediterranean lies a few "states" that are openly racist and promote segregation and spreads hate; ISIS, JSIL and SA ... and probably Iran


persistent ssh tunnel, for reverse tunneling and similar

Ok, so this is a very useful function when you need access to your machine which is behind a nat or so, the theory is simple, let your machine connect to remote host and create a tunnel. you will be then be able to connect to that remote machine (from a third node) and use that tunnel to connect back to your machine.

So how to make sure that this connection is always working?
1 - Make sure your machine and connect to the remote machine passwordlessly (use ssh-copy-id user@remotehost to accomplish that)
2- Start something to connect automatically on boot (using rc.local is quick enough)
3- Make sure that this thing detects disconnections and attempts to reconnect (autossh sounds like a good choice)

So, down to details, I will not go into details of step #1, but mind that you want to local (and remote) user you want to create the connection with when you do the ssh-copy-id to be the same ones you want to create the persistent connection with, generally speaking, unless there is a reason such as low port numbers to listen to when doing the tunneling, root privileges are not required.

So lets write something up that would make sure that the connection will stay up, using autossh:

#/bin/bash

#while true
#    do
        autossh -t -t -R 1234:localhost:22 $1
#done

the commented out parts are to re-run autossh in case it fails (i'm not sure this is needed, but well, test it and if needed just uncomment the loop; the $1 is a positional argument, which is the user@ip of the remote host, that needs to be passed to this script, save the content of the script somewhere like /usr/local/bin/autossh-wrapper.sh or something

Now we put this in the rc.local to start on boot time, append this to the /ect/rc.local

su localuser -c "/usr/local/bin/autossh-wrapper.sh user@ip" &

change localuser, user@ip to values of your local user and remote user@remote_host_ip_or_name