Sub Directories Tree Script: Sed

When working on the shell sooner or later one realizes that "sed" is a basic tool that although it seems to be a bit confusing at first glance, if you learn how to use it, on the long run will help you a lot.

The best way to learn how to use sed for me is by examples. So here is a script that shows all sub directories in a path with a tree-like look.

It's based on a script by Dem Pilafian I found somewhere on the Internet.

The script shows the sub directories tree on the current path if no argument is provided or the sub directories tree of the provided path.

Usage: ./lstree.sh [<path>]

Script code:

     1  #!/bin/bash
     2  
     3  # if path is provided then list directories in path
     4  if [ ! -z "$1" ] && [ ! -d "$1" ]; then
     5    echo "$1: directory not found"
     6    exit 1
     7  elif [ ! -z "$1" ]; then
     8    cd "$1"
     9  fi
    10  
    11  # show actions
    12  pwd
    13  echo '   |'
    14  ls -R | grep ':$' | sed -r 's/:$//' | sed -r 's/[^/]*\//---/g' | sed -r 's/(-*)-/   |\1>/' | 
                                                                                    sed '/\./d'
    15  
    16  # check if no folders
    17  if [ `ls -F -1 | grep "/" | wc -l` = 0 ]
    18    then echo "-> no sub-directories"
    19  fi
    20  
    21  exit 0

Sample output:

Line 14 is the line that does all the "magic":

• sed #1: removes the trailing colons
• sed #2: replaces sub directories with "---"
• sed #3: replaces the first and the last "-" with "|" and ">" respectively
• sed #4: removes the "." line (current directory)

Sometimes it is hard for me to figure out what is the regular expression defined in a sed command actually matching. 

Lets take the 2nd sed and pipe a fake path to it:

All regular expression matches have been replaced because of the "g" at the end of the sed command. Now, replacing the "g" with numbers starting from 1 to N, one can see what are the single substitutions that have been made:

That's a simple way to check what the regular expression defined in the sed command is actually matching.

As I said at the beginning of this post, spend some time learning sed!

Painless Backup with Bash Script

This is a bash script I made to make my files backup process a little bit easier. The script has four basic options:

1. Show usage
2. Create backup
3. Save the backup on my backup drive on remote machine
4. Retrieve backup from remote machine

The script is written to backup the files and directories I want in my computer but it should be pretty easy to configure it to suit your needs.

I don't want this post to be too long so I'm not posting here the whole text. If you want to download the whole script, here is the link.

Run the script with the "b" option to see a brief description of usage, options and actions.
Here are just a few comments about how it works and how to modify it to make you own backups.

Configuration

Lines 23-52 determine what files and directories to backup, what programs to use, etc. Modify these lines to suit your needs.

    23  DATE=$(date --rfc-3339=date)
    24  BASE="$HOME"
    25  BASE_SYS="/etc"
    26  HOME_CONF_DIR=".config"
    27  BACKUP_DIR="backup"
    28  SYNC_DIR="sync"
    29  DOCS_DIR="documents"
    30  DOWN_DIR="downloads"
    31  PACK_DIR="packages"
    32  PICS_DIR="pictures"
    33  SCRIPTS_DIR="scripts"
    34  VIDS_DIR="videos"
    35  LOCAL_HOST="$(uname -n)"
    36  BACKUP_HOST="linuxmachine"
    37  BACKUP_HOST_BASE_DIR="/mnt/backup"
    38  TAR_OPTS="cf"
    39  SYNC_OPTS="-avz --human-readable --progress"
    40  BZIP_OPTS="-9 -v -f"
    41  MIN_PARM_NUM=1
    42  MAX_PARM_NUM=2
    43  PROGS_LIST="bzip2 rsync"
    44  
    45  # options
    46  VALID_MODES=(b s r h)
    47  VALID_ACTIONS=(home documents downloads packages pictures scripts videos conffileshome 
                      conffilessys conffilesall full)
    48  
    49  # What will be backed up. Files and directories
    50  CONF_DIRS_HOME=$(echo "$HOME_CONF_DIR/"{openbox,tint2,conky})
    51  CONF_FILES_HOME=".Xresources .xinitrc"
    52  CONF_FILES_SYS=$(echo "$BASE_SYS/"{mkinitcpio.conf,inittab,fstab,vimrc,rc.conf,bash.bashrc,
                       modprobe.d/modprobe.conf})

Backing up

Making a "full" backup which includes home directories, user's configuration files and system wide configuration files:

Sync with remote host

 Retrieving backup from remote host

Main part description

Basically the script works as follows:

1. parameter check
2. options check
3. check programs
4. perform requested action

These steps correspond to the last lines on the script:

   254  #####################################################################
   255  ########################## parameter checks #########################
   256  #####################################################################
   257  
   258  # check parameters and programs
   259  check_parm_num $# $MIN_PARM_NUM $MAX_PARM_NUM
   260  [[ $? != 0 ]] && usage
   261  check_valid_parm $1 $2
   262  [[ $? != 0 ]] && usage
   263  check_programs "$PROGS_LIST"
   264  
   265  #####################################################################
   266  ############################# main part #############################
   267  #####################################################################
   268  
   269  
   270  # builds the list of actions to be done (only needed when backing up)
   271  [[ "$1" == "b" ]] && build_action_list "$2"
   272  
   273  # perform requested actions
   274  case "$1" in
   275    "b")
   276      # create backup directory for current date and localhost
   277      [[ -d "$BASE/$BACKUP_DIR/$LOCAL_HOST/$DATE" ]] || mkdir -p 
   "$BASE/$BACKUP_DIR/$LOCAL_HOST/$DATE"
   278      cd "$BASE/$BACKUP_DIR/$LOCAL_HOST/$DATE"
   279      for action in $ACTION_LIST; do backup "$action"; done
   280    ;;
   281    "s")
   282      sync "$BACKUP_HOST" "$BACKUP_HOST_BASE_DIR" "$LOCAL_HOST"
   283    ;;
   284    "r")
   285      retrieve "$BACKUP_HOST" "$BACKUP_HOST_BASE_DIR" "$BACKUP_DIR" "$LOCAL_HOST"
   286    ;;
   287    "h")
   288      usage
   289    ;;
   290  esac

I tried to keep the main part as small and clear as possible by using functions. This functions are commented including parameters that they need and what they do.
For instance here is the "build_action_list":

    59  #builds a list of "actions" (what is going to be backed up)
    60  # $1: action
    61  function build_action_list () {
    62    case "$1" in
    63      "full")
    64        ACTION_LIST="documents downloads packages pictures scripts videos conffileshome 
                          conffilessys" 
    65      ;;
    66      "home")
    67        ACTION_LIST="documents downloads packages pictures scripts videos"
    68      ;;
    69      "conffilesall")
    70        ACTION_LIST="conffileshome conffilessys"
    71      ;;
    72      *)
    73        ACTION_LIST="$1"
    74      ;;
    75    esac
    76    
    77    return 0
    78  }

Any comments or questions?
Hope this will make your backing up a little bit easier.

track gmail unread messages with conky in two steps

Step 1:

Here is how you can make conky show how many unread messages you have in your gmail accounts. In my case I wanted to track two gmail accounts so I added the following lines to my conkyrc configuration file.

GMAIL 

<user_1>@gmail.com $alignr ${execi 30 ~/scripts/gmail/checkgmail.sh '<user_1>' '<password_1>'}

<account_2>@gmail.com $alignr ${execi 30 ~/scripts/gmail/checkgmail.sh '<user_name_2>' '<password_2>'}

         The "execi 30" tells conky to run the "checkgmail.sh" script every 30 seconds.

         Change <user_N> and <password_N> with your gmail username and password.

Step 2:

In addition to the above mentioned lines added to the conky configuration file I slightly modified this script I found on the crunchbang linux forums.It seems there are some kind of problem in Germany with "gmail" not beeing allowed to use this name, so this is the modified version of the script that works for me in Germany:

#!/bin/bash

wget -q -O - https://mail.google.com/mail/feed/atom --http-user="$1"@gmail.com --http-password="$2" -
-no-check-certificate | grep fullcount | sed 's/[^0-9]//g'

exit

It takes the gmail account name and the password as parameters of the conkyrc file. Don't forget to make the script executable.
Et violà, the lines with gmail accounts appear at the bottom of conky showing the number of unread messages for each gmail account:

Openbox Window Manager: Keyboard Shortcuts

Openbox uses keyboard shurtcuts defined in the "rc.xml" file usually located in your /home/USER/.config/openbox/ directory. Following shortcut examples include raise/lower volume with amixer, open a terminal with terminator, take a screenshot with scrot, open thunar file manager and exit openbox.

    <!-- My keybindings -->
    <keybind key="W-t">
      <action name="execute">
        <execute>terminator -m</execute>
      </action>
    </keybind>
    <keybind key="W-KP_Add">
      <action name="execute">
        <execute>amixer sset Master 5+</execute>
      </action>
    </keybind>
    <keybind key="W-KP_Subtract">
      <action name="execute">
        <execute>amixer sset Master 5-</execute>
      </action>
    </keybind>
    <keybind key="C-A-BackSpace">
      <action name="execute">
        <execute>openbox --exit</execute>
      </action>
    </keybind>
    <keybind key="W-f">
      <action name="execute">
        <execute>thunar</execute>
      </action>
    </keybind>
    <keybind key="Print">
      <action name="execute">
        <execute>/usr/bin/scrot</execute>
      </action>
    </keybind>

After modifying the "rc.xml" file remember to run "openbox --reconfigure" on the shell to make changes take effect.

sed basics

If you often have to modify some configuration files like fstab, menu.lst, bashrc, etc by removing lines, change some device or comment/uncomment one or more lines, "sed" will be the simplest and fastest way to do so without needing to open the files with an editor.

But first you will have to know a few thing about the "sed" command.

One of the most common uses o f "sed" is to replace one string by another, maybe a device name or the path for something.

From the sed man page:

s/regexp/replacement/
              Attempt to match regexp against the pattern space. If successful, replace that portion matched with replacement.

Here is an example changing /dev/sda with /dev/sdb in /etc/fstab:

Now lets comment out my samba share line:

Note you will have to escape the "/" characters with a leading "\".

Deleting empty lines is just as easy as follows:

Note the "^" character stands for "Line start" and "$" for "End of line" so "^$" means "blank line". Those special character are oftern used in regular expression matching. Take a look at my grep post to see more examples of using regular expessions.

You can also make sed to work on a specific line or range. I created a file with binary numbers 0-16 to work on it.
I will use "nl" to show line numbers from now on to make it easier to see what lines where affected by sed.

Deleting line 8 (0111):

Deleting range 4-13:

If there is a string that repeats many times and you want to change just one specific apparition on a specific line:

Lets change "FF" whith "--"on range 2-3 3rd apparition and range 5-9 6th apparition:

Note you can concatenate sed commands with "-e".


Sed is a very powerfull tool and can do much more complex tasks than the ones exposed in this post. It will take some time to learn his usage but on the long run using sed will speed up many repetitive tasks on the command line.

Playing Radio Stream with Bash Script

Small bash script I wrote to play radio streams trough mplayer. It shows a simple radio list where one can choose the radio stream to be played.
I wrote this a long time ago and it has still some thing written in spanish but it is cery simple and you will find it pretty straight forward.

#!/bin/bash

#Emisoras

SER="http://194.169.201.177:8085/liveser.mp3"
AWESOME80="http://uplink.duplexfx.com:8000"
LITE80="http://uplink.duplexfx.com:8040"
HAIRBAND="http://uplink.duplexfx.com:8014"
THEHEART="http://uplink.duplexfx.com:8006"
BEATLES="http://uplink.duplexfx.com:8062"
GOODTIMEOLDIES="http://uplink.duplexfx.com:8046"
CLASSICALGUITAR="http://uplink.duplexfx.com:8020"

#Comprobar mplayer instalado
PLAYER=$(which mplayer 2> /dev/null)
if [ -z "$PLAYER" ]; then
    printf "Instala mplayer!\n"
    exit 1
fi

DIALOG=$(which dialog 2> /dev/null)
if [ -z "$DIALOG" ]; then
    printf "Instala dialog!\n"
    exit 1
fi

#Terminar con mplayer
killall 2> /dev/null "$PLAYER"

#Seleccionar emisora
EMISORA=$($DIALOG --stdout --title "Radio" --radiolist "Emisoras" 0 0 0 "Cadena SER" "" ON "Awesome 
80's" "" OFF "Lite 80's" "" OFF "80's Hairband" "" OFF "The Heart" "" OFF "Beatles" "" OFF "Good 
Time Oldies" "" OFF "Classical Guitar" "" OFF)

clear

case $EMISORA in
    "Cadena SER")
        printf "Playing Cadena SER\n"
        "$PLAYER" 2>/dev/null "$SER" ;;
    "Awesome 80's")
        printf "Playing Awesone 80's\n"
        "$PLAYER" 2>/dev/null "$AWESOME80" ;;
    "Lite 80's")
        printf "Playing Lite 80's\n"
        "$PLAYER" 2>/dev/null "$LITE80" ;;
    "80's Hairband")
        printf "Playing 80's Hairband\n"
        "$PLAYER" 2>/dev/null "$HAIRBAND" ;;
    "The Heart")
        printf "Playing The Heart\n"
        "$PLAYER" 2>/dev/null "$THEHEART" ;;
    "Beatles")
        printf "Playing Beatles\n"
        "$PLAYER" 2>/dev/null "$BEATLES" ;;
    "Good Time Oldies")
        printf "Playing Good Time Oldies\n"
        "$PLAYER" 2>/dev/null "$GOODTIMEOLDIES" ;;
    "Classical Guitar")
        printf "Playing Classical Guitar\n"
        "$PLAYER" 2>/dev/null "$CLASSICALGUITAR" ;;
    *)
        printf "No se ha encontrado la emisora.\n" ;;
esac

exit 0

Running the script:

grep with regular expressions

If you are famliar with the "grep" command for searhching for line containig a particular character sequence in a file or filtering a command output like "ls" or "ps" you may found out that sometimes just passing to grep a plain string is insufficient for you.

There is a more advanced way for searching patterns by using regular expresions.

Here is a simple script i used to learn the basics abaout how regular expressions are used along with grep:

#!/bin/bash

echo -e {0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,
D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}"\n" | sed /^$/d | sed s/^\ // > hex

# lines containig AA somewhere
echo "lines containig AAA somewhere"
echo "beginning with no 1,3,5,7,9,C,D or E and ending with no 0-5"
grep --color=auto -n -E '^[^13579CDE]?AAA[^012345]?$' hex

# lines starting with AAA and contain no digit
echo -e "\nlines starting with AAA and contain no digit at the end"
grep --color=auto -n -E '^AAA[^[:digit:]]' hex

# lines starting with 3-5 and ending with BBB
echo -e "\nlines ending with BBB"
grep --color=auto -n -E '[3-5]BBB$' hex

# lines beginning with no less than two 1 and no more than three 1
# followed by at least 1 D or more
echo -e "\nlines beginning with no less than two 1 and no more than three 1\nfollowed by at least 1 
D or more"
grep --color=auto -n -E '^1{2,3}D+' hex

# lines starting either with 3BB or 3CF ending with 6 to 9
echo -e "\nlines starting either with 3BB or 3CF"
grep --color=auto -n -E '^3(BB|CF)[6-9]' hex

# lines with 4 or 6 only
echo -e "\nlines with 4 or 6 only"
grep --color=auto -n -E '[4]{4}|[6]{4}' hex

exit 0 

The script creates a file with one 16-bit hexadecimal number per line from 0000 to FFFF and performs several grep commands on it.
Options i used for grep were "-n" for showing the line numbers (green on screenshot) and "-E" to use extended regular expressions.
Grep with "-E" option should behave the same way as "egrep" does.

Running the script throws following output to the terminal:

There are a few more lines that didn't fit on the screenshot.