Notes

Grymoire Tutorial

These are notes from the tutorial on POSIX shell. There is also a splash from rwxrob in here too.

I did not cover every topic covered nor are these notes as thorough as the resources used.

Shell Basics

The shell operates in a predictable and linear manner 1. Meta-characters are handled 1. The name of the executable is found 1. Arguments are passed to the program 1. File redirection is setup 1. The program is executed

File Name Expansion AKA Globbing

Meta-Characters and Filename Expansion

Pattern	Matches
*	Every file in the current directory
?	Files consisting of one character
??	Files consisting of two characters
??*	Files consisting of two or more characters
[abcdefg]	Files consisting of a single letter from a to g
[a-g]	Same as above
[a-cd-g]	Same as above
[a-zA-Z0-9]	Files that consist of a single letter or number
[!a-zA-Z0-9]	Files that consist of a single character not a letter or number
[a-zA-Z]*	Files that start with a letter
?[a-zA-Z]*	Files whose second character matches a letter
*[0-9]	Files that end with a number
?[0-9]	Two character filename that end with a number
*.[0-9]	Files that end with a dot and a number

File Name Expansion with Directories

Pattern	Matches
*	All non-hidden files
abc/*	All non-hidden files in the abc directory
abc/.*	All hidden files in the abc directory
*/*	All non-hidden files in all subdirectories
*/.*	All hidden files in all subdirectories

Understanding file name expansion is important for security. When you use * at the command line it will expand the file names and add it to the command. Why would this be important? Let's say we want to run rm * and the contents of the director are 0.md 1.md. We would expect that the command will delete both files without any imput from the user. However, compare that to a directory with -i 0.md 1.md and you will have entirely different behavior.

The first command expands to rm 0.md 1.md while the second expands to rm -i 0.md 1.md wand the -i is what causes the change in how rm operates.

Arguments Vs Options

The shell does not see either different from each other. It is the program or executable that makes the difference and treats them differently. Let's consider rm -i 01.md 02.md. In this example, "-i, 01.md, and 02.md" are all arguments passed to the command rm. By convention programs use a dash "-" to differntiate a program option from a program argument.

Space as a meta character

One meta character that tends to be forgotten about is the space. The shell treats a space as an indicator of the end of a word. Taking the previous example, rm -i 01.md 02.md all four components are words. The first word "rm" is the command and the last three are arguments to be passed into that command.

Quotes

Using Quotes and the Escape Character

Quotes can be used to group text together and can, in the case of the single quote, escape special characters. Assume you want to delete the file with the name: "delete me.md". As discussed above, the space is a meta-charactger that deliniates words. If you were to try to delete it by using rm delete me.md the shell will first look for the file "delete" and error out. Instead you need to "escape" the space with either the single quote (''), double quote (""), or the backslash (\). Here are some examples that will work:

rm "file1 file2" rm file1 file2 rm file1" "file2 rm 'file1 file2' rm file1' 'file2 rm f'ile1 file'2

Nested Quotes and Escape Character

It can become very complicated when trying to nest quotes within quotes. Nesting double quotes

echo '"'
echo "\""

Nesting single quotes

echo "'"

Nested Escape Character

echo '\'
echo "\\"

Strong vs Weak Quoting

The Single quote is seen as a strong quote where the double quote is seen as a weak quote. Strong quoting prevents characters from having any special meaning while weak quoting allows meta-characters to maintain their special meanings. Example: echo '$HOME' will display $HOME where echo "$HOME" will display the home path of the current user.

Command Substitution (Backquotes)

The backquotes' special use is for command substitution. Anything between backquotes is executed by the shell and the results replace the backquoted portion of the command. For example if you want to echo the current working directory you can use:

echo `pwd`

This will repalce the pwd with the current working directory the user is in.

Another format of command substitution is using $(pwd)

Variables

There is a simple syntax for variables =. Notice that there is no space between the variable name and the value of the variable. The space character will terminate the variable. You can set more than one variable on a single line. Exmaples of setting a variable:

    A=1
    A=1 B=2 C=3
    a="This is a value"

You can view the values of all variables with the set command.

Inherited Variables

To get a variable's value accessible to a subshell, you must export that variable. This gives the subshell a variable of the same name and value which can be changed in that subshell but the subshell cannot change the value of the parent shell.

var="My Variable"
export var

IFS Variable

This variable lists the characters used to terminate a word. Attackers can modify the IFS variable to gain privileged access to a system. If a program runs /bin/ps without using quotes, it is possible that an attacker changes the IFS to include "/" which will have the program run bin ps. Now the attacker needs only to put a program called bin in the PATH and will gain privileged access.

PS1 Variable

The PS1 variable specifies the prompt you see on the terminal before each command entered. It is common to see the "$" for non-root users and "#" for root users.

PS2 Variable

The PS2 variable specifies the prompt you see on secondary prompts. For example, a multiline command.

Substitution Operations

These are good for when you want to do something based on if a variable has a value or not.

Operator	Description	Example
${varName:-value}	Shows the value if a variable is not defined.	$COST:-'not defined'
${varName:=value}	Assigns the value to the variable if the variable does not exist.	${NAME:='Arron'}
${varName:value}	Assigns a value if the parameter has no value or it does not exist.	${LNAME:'PATTON'}
${varName:?value}	Shows the error message of the value if the variable does not exist or is empty.	${NAME:?'is not defined'}
${varName:+value}	Shows the message that is defined if the variable has a value.	${NAME:+"has the value of $NAME"}
${varName%value}	Remove the value from the end of the variable.	${NAME%'Arron'}
${varName%%*value}	Remove the value from the end of the variable (greedy).	${NAME%%'Arron'}
${varName#value}	Remove the value from the beginning of the variable.	${NAME#'Arron'}
${varName##*value}	Remove the value from the beginning of the variable (greedy).	${NAME##'Arron'}

Positional Parameters $1...$9

Shell scripts can recieve arguments and those arguments are stored in special variables based on the argument position. For example script arg1 arg2. The variable $1 would be arg1 and $2 would be arg2. This continues up to and including $9. However, it is possible to pass more than 9 arguments to your script. It will just take a little more work to access those arguments. Some shells such as bash will let you access them using ${10} POSIX does not.

Shift

The shift command allows you to shift the arguments down. We will use this as an example script arg1 arg2 arg3 arg4 arg5.

#!/bin/sh
a1=$1; shift
a2=$1; shift
a3=$1; shift
a4=$1; shift
a5=$1; shift

echo "$a1 $a2 $a3 $a4 $a5"

#!/bin/sh
a1=$1;
a2=$2;
a3=$3;
a4=$4;
a5=$5; shift 5

echo "$a1 $a2 $a3 $a4 $a5"

Other Special Variables

$0

This variable contains the script name. This variable is not affected by the shift command.

$*

This variable contains all of the arguments. When attempting to loop through "$*", it will just return all of the arguments in the first iteration of the loop.

$@

This variable is much like the $* in that it contains all of the arguments; however, $@ retains spaces in arguments where $* does not. When looping through "$@", you will get each individual parameter as it was entered.

$#

This variable is equal to the number of arguments passed to the script.

$$

This variable gets the current process id.

Debugging

If you are having issues understanding how a script is functioning you can use the x flag of sh to get more information about how the script runs. You can set this in a few different ways. First, you can set it when executing the script /bin/sh -x script arg1. Second, you can add it to the shebang #!/bin/sh -x. Third, you can turn it on and off as you need to in the script. This is helpful when the script is long and you are only debugging a specific section. To turn it on use set -x and to turn it off again use set +x.

Check for Syntax Errors

You can use the -n argument to have the shell read the script but not execute anything.

Shell exit Flag

By default a script will exit on errors because the shell exit flag is set. To ignore errors you can set set +e and to turn it back on again set -e.

Flow Control

Simplest Form of Flow Control using &&, ||, and ;

The simplest way to use flow control is to use && and ||. command1 && command2 Both commands will run if and only if the first is successful. This can be read "command1 and command2" command1 || command2 The second command will only run if the first command is NOT successful. This can be read "command1 or command2"

true && echo "This line IS printed."
false || echo "This line IS printed."
true || echo "This line is NOT printed."
false && echo "This line is NOT printed."

To control what happens in succession you can use the ;. command1; command2; command3. This will execute command1 then command2 then command3.

Precedence

There is a precedence in operations when it comes to using &, &&, |, ||, and ;. && and || have a higher precendence than ; and & but lower than |.

a|b && c; d||e|f; will be evaluated to (a|b(b&&c)); ((d||e)|f)

Changing Precedence

You can use "()" or "{}" to change the precedence. However, be careful when choosing which to use as there are some major differences. Using "()" causes the shell to execute a new process while "{}" does not.

a=OLD
(a=NEW); echo $a
{ a=NEW; }; echo $a

Note the space after the "{" and before the "}" as well as the ";" inside and after the "{}".

Ways Commands Can be Grouped

Simple, pipeline, and lists are the three ways commands can be grouped together. A simple command is a collection of words separated by spaces. A pipeline is a group of simple commands separated by a "|". A list is a series of pipelines, sepatrated by &, ;, &&, or || and terminated by a ;, &, or a new line character.

If, While, Until

If

The structure of an if statement is: if list then list fi. This means after each "list" a newline or semicolon is required.

if true
then
    echo "This was true"
fi

or

if true; then
    echo "This was true"
fi

or

if true; then echo "This was true"; fi

This is also true if using elif and else

if true
then
    echo "This is true"
elif false
then
    echo "this is flase"
else
    echo "this should never happen"
fi

if true; then echo "This is true"; elif false; then echo "This is false"; else echo "This should never happen"; fi

While

Basic struture of while is while list do list done

while true
do
    echo "this will run for ever"
done

while true; do echo "this will run for ever"; done

Until

Basic structure of until is until list do list done

until testa
do
echo "Not yet"
done

until testa; do echo "Not yet"; done

For and Case

For

For iterates through one or more words executing a list. for name do list done for name in word do list done

for i in 1 2 3 4 5 6 7 8 9
do
    echo $i
done

for i in 1 2 3 4 5 6 7 8 9; do echo $i; done

This example will iterate through both variables of multiple words. It will first iterate through num1 then num2.

num1='1 2 3 4 5'
num2=`5 4 3 2 1`
for i in num1 num2
do
    echo $i
done

You can use command expansion to create the list to iterate through

for i in $(ls )
do
    echo $i
done

You can combine variables, constants, and command expansions together to make them one iteration.

for "$a $b $c"
do
    echo $i
done

Case

Case functions like a complex if statement with multiple clauses. case word in pattern | pattern )list;; esac

echo answer yes or no
read word
case $word in 
    yes | YES )
        echo "You answered yes"
        ;;
    no | NO )
        echo "You answered no"
        ;;
esac

You can use pattern matching as well

echo answer yes or no
read word
case $word in
    [Yy]* )
        echo "You answered yes"
        ;;
    [Nn]* )
        echo "You answered no"
    ;;
esac

There can only be one word after case. So if you are ever checing more than one word, you need to use a variable.

arg='a b c'
case $arg in 
    [aAbBcC] )
        echo "This never matches
        ;;
    "a b c" )
        echo "This will match"
        ;;
esac

Break and Continue

If you want to jump out of a for or while loop you can use the break command and if you want to end the current iteration at a specific point you can use the continue.

arg='1 2 3 4 5 6 7 8 9'
for i in $arg
do
    case "$arg" in
        2|4|6|8 )
            continue
            ;;
        0 )
            break
            ;;
    esac
    echo $arg

Functions

To create a function use the format of funcName() { list }

This will attempt to increment the argument given to it by one

inc() {
   echo $(echo $1 + 1 | bc)
}
echo $A

Parsing Arguments

Using flags that have arguments

#!/bin/sh

usage () {
    echo "$0 Error: $*" 1>&2
    echo "usage: $0 [-i IP] [-p PORT]" 1>&2
    exit 1
}

i='' p=''

while :
do
    case "$1" in
        -i ) shift; i=$1 ;;
        -p ) shift; p=$1 ;;
        -* ) usage "Bad argument $1";;
        * ) break ;;
    esac
    shift
done

echo "You entered the IP: $i and PORT: $p"