Bash Bash Bash!
by Douglas Berdeaux (Douglas@WeakNetLabs.com)
I recently read that there is a struggle in the U.S. lately with computer science majors and passion. Getting students excited enough to fuel their imaginations into producing innovative ideas, devices, and code seems to be a hard task. Being inspired isn't something that can be thrust upon students by just anyone. Being an inspirational teacher means that you are capable of showing your own passion for the subject along with sturdy knowledge to back it up.
Hackers, many of whom never even went to college or have a degree, come up with brilliant ideas every day. Is there then something to be said about our teachers today, if this struggle really does exist? I was recently asked if, in the last month, anything on the web really caught my attention or seemed innovative. My answer was, "No." A lot of things have simply repeated themselves, in different colors, shapes, or sizes. I was just hoping that my answer, plus the articles I had read about the struggle, were purely coincidental.
Sadly, those fond of mathematics have no real consideration of coincidence. Let me attempt to help, by speaking of something for which I have passion: Bash.
I love Bash.
In fact, in a recent job interview, thanks to my ADD, I was pondering what really fuels my passion for IT and realized that it was bash. Bash was coded in 1987 by Brian Fox, and is the most beautiful thing in the software side of computer science, in my humble opinion.
So powerful and lightweight, it makes CMD.EXE look like a game. In fact, CMD.EXE was actually a hidden game I added to versions 1 and 2 of WeakNet Linux Assistant.
When I see command line manuals and Linux magazines that talk about "shell commands," I laugh to myself. If you open those magazines, you will most likely find commands that do not come with the shell, like apt-get or awk.
Sure, those commands can be invoked by the shell, but they aren't really part of the shell itself. In fact, the shell only comes with a few commands built into it that you can call "shell commands."
These are called directly from the current shell, making them super fast. All other commands are spawned as new processes, spawned with a new instance of the shell, or loaded by the shell when called.
The Wikipedia entry for "Shell Builtin" states, "Usually used for simple, almost trivial, functions, such as text output." I guess real system administrators don't have time to edit Wikipedia pages.
Here's a small list:
There are tons of great Bash references online.
The best reference of all, I'd say, would have to be the O'Reilly books1 on Bash.
Bash is a language, interface, interpreter, input, and output for errors and non-error IO data as "terminals." It's flexible, powerful, resilient, and found almost everywhere you find Linux.
Just the other day, I realized that my system opened a new instance of VLC every time I double-clicked a media file in Nautilus. What a pain in the ass!
I clicked around for a few minutes in VLC settings and Nautilus settings and couldn't find any solution. Well, Bash to the rescue!
In Linux/UNIX, all applications can be run from the command line.
If you install something extra, it usually goes into /usr/local/bin or, if it's an administrative application, /usr/local/sbin.
Sometimes you will see extra applications in /opt. Any applications that come pre-installed with your OS, or installed by the OS developer's pre-compiled repositories, will usually end up in /bin or /sbin.
If you type which <command name>, you can see where the command is located. This is useful for debugging purposes if, say, you forget to uninstall an application before recompiling it and installing it from source.
Anyway, I typed which vlc and saw: /usr/bin/vlc
I then moved the command to /usr/bin/vlc_start and used Vim to make a new VLC file (vim /usr/bin/vlc). I added the lines:
I then made the command an executable, by issuing chmod +x /usr/bin/vlc, and bash! The problem was solved.
Let's review the code.
The first line is the "she-bang!" interpreter line. Bash knows that if it sees a file with this line in it, it uses the command to run the rest of the lines in the file that do not begin with a pound (#) symbol. A few interpreters include #!/usr/bin/perl, #!/usr/bin/ruby, etc. So in our case, it runs each line through a new instance of Bash.
The first line it runs through the new bash instance is the line beginning with MEDIA.
This assigns the first argument to the throw-away environment variable $MEDIA, after running the command in the back ticks. If you remember back to your algebra days, you might recall an acronym: PEMDAS. Parentheses, exponents, multiplication, division, addition, and subtraction all happen in that order, no other.
It's part of math's "syntax," so to speak. Everything in back ticks will happen first, as if they were in parentheses. The $1 is the built-in Bash variable that represents the first argument given to the command. You can have up to $9, but then you have to add more syntax. You can also use $* as a glob for all arguments, but I'll show you that later.
The next line forcefully kills all instances of vlc_start which, if you remember, is the actual binary for the VLC media player. The last line starts VLC again, with the new file in quotes. Problem solved.
If you are a programmer, Bash is a playground for you and your OS.
There are the same looping and logical constructs found in most languages available to you right in Bash. Once, at work, I was asked to fix a bad fstab file on an old Solaris 5 machine. This machine dropped me to a single-user mode shell without mounting /usr, so I had no access to any commands, besides those built into the shell.
This wasn't a Bash shell, but this can certainly be done in a Bash shell. There was a backup of the old fstab file in /etc, but I couldn't use cat or cp to replace the broken fstab file.
Thanks to help from an IRC friend, I ended up typing a one line, simple shell program like so:
This code opened my eyes and mind to the the possibilities available with just the shell alone.
What this code does is start a while loop and make a variable $foo for each line in the input file fstab.backup using the input pipe <.
It then redirects the output to the broken fstab file, overwriting anything inside, using the > output redirection pipe.
This was my first introduction to shell built-ins. This fixed the boot problem and made me wonder what else could be done with shell built-ins.
Another cool example I use in system administration practices, which I use almost on a daily basis, is creating functions. Just like in a programming language, you can make functions or groups of code and pass data to the code.
In this example I will keep it simple and create a l33t translator. You can start typing this out on the command line, as it will not finish the command until you add the ending } followed by a newline character:
This will change all of your text into cool l33t text!
To pass data to it, simply call it with an application:
$ l33t () { > $* 2>/dev/stdout | sed -e 's/E/3/gi' -e 's/L/1/gi' -e 's/A/4/gi' -e 's/o/0/gi' -e 's/G/9/gi' > } $ l33t ls /sbin 42disc0nf 42dism0d 42dissit3 423nc0nf 423nm0d 423nsit3 42qu3ry 44-r3m0v3-unkn0wn 44-st4tus 44-t34rd0wn 4cc3ssdb 4dd9nup9h0m3 4dd9r0up 4dd-sh311 4ddus3r 493tty 4irb4s3-n9 4ir3p14y-n9 4irm0n-n9 4ir0dump-n9 4ir0dump-n9-0ui-upd4t3 4irs3rv-n9 4irtun-n9 4irv3ntri10quist-n9 4p4ch32 4p4ch32ct1 4p4ch3ct1 4pp4rm0r_p4rs3r 4pp4rm0r_st4tus 4pp1y9nup9d3f4u1ts 4rp 4rpd 4rp-fin93rprint 4rpin9 4rp-sc4n 4rpsp00f 4rpt4b13s 4rpt4b13s-nft 4rpt4b13s-nft-r3st0r3 4rpt4b13s-nft-s4v3 4rpt4b13s-r3st0r3 4rpt4b13s-s4v3 4sp311-4ut0bui1dh4sh 4str4c3r0ut3 4tftpd 4tm31_fw1 4ut0m0unt 4v4hi-d43m0n 4vcst4t b4db10cks ...Some downfalls are that tab auto-completion breaks and some captive applications seem to go slower, but this simple exercise shows you how to group applications to simply change the output. Think of the possibilities for applications that do even more!
One last example I would like to show is one that benefited me in a time of need.
I purchased music from LegalSounds.com and was given a text file detailing where the MP3s were on their servers.
I ran GNU Wget on each file in a row after creating a shell script that simply added wget to the beginning of each line in the text file.
Then I used chmod and ran the executable.
I dumped the songs onto my Android phone and left for the day. When I tried to play one of the songs, I realized that Android was not detecting the files on my SD card!
When I browsed the directory tree, I saw them and the problem. Somehow an extra %3D was added to the end of each file extension! I then thought the best way to handle all of these was to loop through them and rename them, right? Beautiful Bash can do this.
Here is how I did it on my Android phone using the terminal:
See how this does more than just output text?
Think of the powerful possibilities! This solved my problem rather quickly!
Let's wrap this article up by covering a few Bash favorites of mine.
Tab auto-completion is number one. A systems administrator isn't lazy, but has too much on his or her mind to be using ls or find to run applications or pass files as arguments.
Recently, the matched strings from grep filters have been colorized by default2.
This is awesome for anyone who is new to regular expression syntax and wants to see exactly what he or she matched.
History, found in ~/.bash_history is also an amazing feature. You can use the up and down arrows to access your recent command history. Sure, this is available in DOS, but does DOS have a Ctrl+R command history shortcut that allows you to type strings to match patterns of old commands right from the command line? I don't think so.
The terminal emulators that display Bash and other shells have also come far since I started using them. Now you have Compiz, and graphics drivers that allow you to have full, true transparency while coding! Who knew 20 years ago, that people would be using UNIX shells in X Windows, let alone with beautiful transparent windows and fonts?!
DOS can't even maximize properly and it's the year 2010. Environment variables can be made, changed and removed. If you export a variable, it goes away once you exit the shell. These are immensely useful when used in the right places.
sed, AWK, and grep also need to be mentioned.
These don't come with Bash, but exploit the beauty of the Bash pipeline. Bash can pipe IO into or from other commands or files using the |, >, <, and >> operators. If you add a 1 or 2 in front of the pipes, you can send STDOUT and STDERR into files and other commands as well!
Here is a small example of AWK/sed/grep and pipelines with STDERR:
$ cat file.txt 2>/dev/null | awk '{print $1}' | sed 's/e/3/gi' | grep -v 'LOL HI'This dumps the contents of file.txt to the screen (STDOUT), but is interrupted by the 2>/dev/null, which sends all errors3 (binary file matches, no file found, etc.) to /dev/null (the UNIX garbage can).
It gets interrupted once more by the pipe |, which sends the output to AWK, which prints only the first word in each line, delimited by any whitespace character.
The output still doesn't quite make it to the screen, as it is once again interrupted by a pipe and sent to sed, which substitutes all e for 3.
Then one last interrupt sends the parsed data to grep and grep discards all lines that have LOL HI somewhere in them but prints all the lines that don't to the screen in real time.
Everything in UNIX is a file. Files have words, and strings and such, which make these utilities powerful and beautiful.
There is so much more to Bash that I couldn't cover in this article, and if I had, may have interrupted the spark that makes someone interested enough to find out more for him or herself. The spark of passion.
References
- Learning the Bash Shell, Bash Cookbook, and Classic Shell Scripting.
- Try changing the environment variable GREP_COLOR!
- This is left up to the author of the application used. Some simple applications will print errors to STDOUT by default.