How to deal with operations and parameters in bash script like a master

This is a simple post, based in a previous post in which I learned how to work with parameters in bash, to have option (e.g. -h, –help, etc.), combined options (e.g. -cf that means the same than –cool –function), etc. That post consisted in pre-processing the commandline to get the proper flags and dealing with them.

Now I want to implement bash scripts that work in the form of

$ ./my_script operation --flag-op

And this is why I extended the previous post to learn

How to deal with operations and parameters in bash script like a master.

In this case I want to have scripts that accept the following syntax:

$ ./my_script operation -p <parameter> -ob --file conf.file

And even sintax like the next one:

$ ./my_script --global-option operation -p <parameter> -ob --file conf.file

Using the preprocessing introduced in the previous post, this is very straightforward to implement, as we only need to intercept the name of the operations and continue with the options.

As a plus, in my solution I will delegate each operation to deal with its specific options. My solution is the next one:

function get() {
 while (( $# > 0 )); do
  case "$1" in
   --all|-a) ALL=1;;
   *) usage && exit 1;;
  esac
  shift
 done
 # implement_the_operation
}

n=0
while (( $# > 0 )); do
 if [ "${1:0:1}" == "-" -a "${1:1:1}" != "-" ]; then
  for f in $(echo "${1:1}" | sed 's/\(.\)/-\1 /g' ); do
   ARR[$n]="$f"
   n=$(($n+1))
  done
 else
  ARR[$n]="$1"
  n=$(($n+1))
 fi
 shift
done

n=0
COMMAND=
while [ $n -lt ${#ARR[@]} -a "$COMMAND" == "" ]; do
 PARAM="${ARR[$n]}"
 case "$PARAM" in
  get) COMMAND="$PARAM";;
  --help | -h) usage && exit 0;;
  *) usage && exit 1;;
 esac
 n=$(($n+1))
done

if [ "$COMMAND" != "" ]; then
 $COMMAND "${ARR[@]:$n}"
else
 echo "no command issued" && usage && exit 1
fi

In this script I only accept the operation (COMMAND) “get”. And it is self-serviced using a function with the same name. In order to implement more operations, it is as easy as including it in the detection and creating a function with the same name.

How to compact a QCOW2 or a VMDK file

When you create a Virtual Machine (VM), you usually have the option of use a format that reserves the whole size of the disk (e.g. RAW), or to use a format that grows according to the used space in the disk (e.g. QCOW or VMDK).

The problem is that the space actually used in the disk grows as the disk files are written, but it is not decreased as they are deleted. But if you writed a lot of files and you deleted after they were needed, you’d probably have a lot of space reserved for the VMDK file, while that space is not actually used. I wanted to reclaim that space, to move the VMs using less space, and so this time…

I learned how to compact a VMDK file (the same method applies to QCOW2)

The method is, in fact, very easy… you simply have to re-encode the file using the same output format. If you have your original-disk.vmdk file, you simply have to issue a command like this one:

$ qemu-img convert -O vmdk original-disk.vmdk compressed-disk.vmdk

And that will make the magic (easy, isn’t it?).

But if you want to compact it more, you can claim more space from the disk before re-enconding the disk. First, I’d go to the solution and then I’ll explain it:

If the VM is a linux-based, you can boot it and create a zero-file, and once the file has exhausted the disk, delete it:

$ dd if=/dev/zero of=/tmp/zerofile.raw...$ rm /tmp/zerofile.raw

If the VM is a Windows-based, you can get the command sdelete from Microsoft website decompress it and execute the following commandline:

c:\> sdelete -z c:

Now you can power off the VM and issue the qemu-img command. You’ll get a file that correspond to only the used space in the disk:

$ qemu-img convert -O vmdk original-disk.vmdk compressed-disk.vmdk

Explanation

(Disclaimer: Please take into account that this is a simple and conceptual explanation)

If you knew about how the disks are managed, you’d probably know that when a file is deleted, it is not actually deleted from the disk. Instead, the space that it was using is marked as “ready to be used in case that it is needed”. So if a new file is created in the disk, it is possible that it uses that physical space (or not).

That is the trick from which the file recovery applications work: trying to find those “ready to be used” sectors. And that is why the “low-level format” exists: in order to “zero” the disk and to avoid that files are recovered.

When you created the /tmp/zerofile.raw file, you started to write zeros in the disk. When the physical empty space was exhausted, the disk controller started to use the “ready to be used” sectors, and the zerofile wrote zeros on them, and the zeros were written in the VMDK file.

The good thing here is that when a VMDK file is created (from any format… in our case, it is VMDK), the qemu-img application does not write those zeros in the file that contains the disk, and that is how the storage space is reclaimed.

How to deal with parameters in bash scripts like a pro

I use to develop bash scripts, and I usually have a problem with flags and parameters. I like to allow parameters like a pro: using the long flags (e.g. –flag), the reduced flags (e.g. -f), but I want to allow combinations of several flags (e.g. -fc). And so this time…

I learned how to deal with parameters in bash scripts like a pro

This time I have started to use bash arrays, that are like C arrays or python arrays, but in bash 😉 I could explain little by little my script, but I’m including here the final script (this is an extract from one of my developments: ec4docker):

CREATE=
TERMINATE=
CONFIG_FILE=
n=0
while [ $# -gt 0 ]; do
    if [ "${1:0:1}" == "-" -a "${1:1:1}" != "-" ]; then
        for f in $(echo "${1:1}" | sed 's/\(.\)/-\1 /g' ); do
            ARR[$n]="$f"
            n=$(($n+1))
        done
    else
        ARR[$n]="$1"
        n=$(($n+1))
    fi
    shift
done
n=0
while [ $n -lt ${#ARR[@]} ]; do
    case "${ARR[$n]}" in
        --create | -c)          CREATE=True;;
        --terminate | -t)       TERMINATE=True;;
        --yes | -y)             ASSUME_YES=True;;
        --config-file | -f)     n=$(($n+1))
                                [ $n -ge ${#ARR[@]} ] && usage && exit 1
                                CONFIG_FILE="${ARR[$n]}";;
        --help | -h)            usage && exit 0;;
        *)                      usage && exit 1;;
    esac
    n=$(($n+1))
done

In this way, you allow to issue commands like

$ ./myapp -ctyf config.conf

But also mix parameter styles

$ ./myapp --create -ty --config-file myapp.conf

Technical details

I like the solution, but I also like the technical details (because I am a code-freak). So I share some technical issues here:

  • The first “while” simply parses the commandline to expand the combined parameters. In fact, if searches for expressions like ‘-fct’ and splits them into a set of expressions ‘-f’, ‘-c’, ‘-t’. So if you do not want to split parameters in this way, you can substitute the first “while” by
ARR=( "$@" )
  • The second “while” is needed because we want to allow parameters that need more than one flag (e.g. -f <config file>). Any time that it is expected to have a parameter for a flag, we need to check if we have enough parameters and if not, raise an error. If you do not need any parameter with extra values, you could substitute the second while by:
for ARRVAL in "${ARR[@]}"; do
  case "$ARRVAL" in