grammar
variables | variable expansion | brace, tilde, command, and pathname expansion | special variables
arithmetic and conditional expressions
quoting
arrays
functions | command resolution
execution control
redirection | echo and read | files and directories
process and job control
arguments and options | startup files
history commands and expansion | readline and tab completion | prompt customization

Grammar

All shell grammars consist of lists of pipelines of simple commands.

Simple commands usually resolve to external commands, but they may also resolve to built-in functions or user defined functions.

Pipelines are a sequence of simple commands in which the standard output of each command is redirected to the standard input of its successor.

The semantics of a list of pipelines depends on which list separator is used.

simple command

In its simplest form a line in a shell script is a command followed by its arguments. The shell script will attempt to resolve the command to an external command in the search path. This property and the shell's concise syntax for specifying the standard input and standard output of the commands are what make the shell so effective at running and combining other executables.

When resolving a command user defined functions take precedence over built-in functions which take precedence over external commands. csh and tcsh lack user defined functions but built-ins still take precedence over external commands.

simple command with environment variable

tcsh:

How to isolate an environment variable to a single command:

( setenv COLUMNS 20 ; ls )

pipeline

command list separators

command list terminators

group command

subshell

Variables

global variables

How to set a global variable; how to get the value of a global variable; how to list all the global variables; how to unset a global variable; how to edit a variable.

Variables are global by default.

In tcsh if var is undefined then encountering $var throws an error. The other shells will treat $var as an empty string.

If there is a variable named foo, then

unset foo

will unset the variable. However, if there is no such variable but there is a function named foo, then the function will be unset. unset -v will only unset a variable.

read-only variables

How to mark a variable as read-only; how to simultaneously set and mark a variable as read-only; how to list the read-only variables.

An error results if an attempt is made to modify a read-only variable.

exported variables

How to export a variable; how to set and export a variable; how to list the exported variables.

Exported variables are passed to child processes forked by the shell. This can be prevented by launching the subprocess with env -i. Subshells created with parens ( ) have access non-exported variables.

The tcsh example for exporting a variable without setting it isn't the same as the corresponding examples from the other shells because in tcsh an error will result if the variable isn't already set.

arrays

associative arrays

options

Variable Expansion

substring

zsh:

Substring expansion appears to be partly one based when operating on a positional parameter:

$ set -- 0123456789

$ echo $1 
0123456789

$ echo ${1:0:4}
0123

$ echo ${1:1:4}
0123

$ echo ${1:2:4}
1234

$ foo=0123456789

$ echo ${foo:0:4}
0123

$ echo ${foo:1:4}
1234

Brace, Tilde, Command, and Pathname Expansion

Special Variables

Arithmetic and Conditional Expressions

Expressions are implemented as either command expressions which return an integer status like a command, or variable expressions which evaluate to a string. Command expressions return a status of 0 for true and a nonzero status for false. Only commands and command expressions can be used as the conditional in if, while, and until statements.

Expressions which support arithmetic only support integer arithmetic.

test command

true command

A no-op command with an exit status of 0. One application is to create an infinite loop:

while true; do
  echo "Are we there yet?"
done

false command

A no-op command with an exit status of 1. One application is to comment out code:

if false; then
  start_thermonuclear_war
fi

conditional command

conditional expression

arithmetic expansion

let expression

external expression

arithmetic command

An arithmetic command can be used to test whether an arithmetic expression is zero.

Supports the same type of expressions as $(( )).

eval

How to evaluate a string as a shell command.

Quoting

literal quotes

Literal quotes (aka single quotes) create a word with exactly the characters shown in the source code. There is no escaping mechanism and hence no way to put single quotes in the word.

Literal quotes can be used to put characters that the shell lexer uses to distinguish words inside a single word. For bash these characters are:

| & ; ( ) < > space tab

Literals quotes can also be used to prevent the parameter, brace, pathname, and tilde expansion as well as command substitution. For bash the special characters that trigger these expansions are:

$ { } * ? [ ] ` ~

interpolating quotes

Interpolating quotes (aka double quotes) perform parameter expansion and command substitution of both the $( ) and ` ` variety. They do not perform brace, pathname, or tilde expansion. $ and ` are thus special characters but they can be escaped with a backslash as can the backslash itself, the double quote, and a newline.

quotes with c escapes

interpolating command substitution

Arrays

lookup

a[17] sometimes works for lookup but should be avoided because if the files a1 or a7 exist in the current directory pathname expansion will be performed.

delete value

Deleting elements from a bash array leaves gaps. Deleting elements from a zsh arrays causes higher indexed elements to move to lower index positions.

declare

Associative arrays were added to bash with version 4.0 released in 2009.

Functions

define with parens

How to define a function.

POSIX calls for parens in the declaration, but parameters are not declared inside the parens, nor are parens used when invoking the function. Functions are invoked with the same syntax used to invoke external commands. Defining a function hides a built-in or an external command with the same name, but the built-in or external command can still be invoked with the builtin or command premodiifers.

define with keyword

How to define a function using the function keyword.

parameters

The variables which hold the function parameters.

Outside of a function the variables $1, $2, … refer to the command line arguments provided to the script.

$0 always refers the name of the script in a non-interactive shell.

number of parameters

The variable containing the number of function parameters which were provided.

Outside of a function $# refers to the number of command line arguments.

return

If a function does not have an explicit return statement then the return value is the exit status of the last command executed. If no command executed the return value is 0.

return values

Shell functions can only return integers. Some shells limit the return value to a single byte. This is all the information one can get from the exit status of an external process according to the POSIX standard.

If a shell function needs to return a different type of value, it can write it to a global variable. All variables are global by default. The value in one of the parameters can be used to determine the variable to which the return value will be written. Consider this implementation of setenv:

setenv() {
  eval $1=$2
}

local variables

How to declare and set a local variable.

Local variables are normally defined inside a function. bash throws an error when an attempt is made to define a local outside a function, but dash and zsh do not.

Local variables have lexical, not dynamic scope. If a function recurses, locals in the caller will not be visible in the callee.

list functions

How to list the user defined functions and their definitions.

delete function

How to remove a user defined function.

Command Resolution

alias

Alias expansion is done after history expansion and before all other expansion. A command can be expanded by multiple aliases. For example the following will echo "baz":

alias bar=echo "baz"
alias foo=bar
foo

On the other hand the shells seem smart enough about aliasing to not be put into an infinite loop. The following code causes an error "foo not found":

alias foo=bar
alias bar=foo
foo

Alias definitions are not registered until an entire line of input is read. The following code causes an error "lshome not found":

alias lshome='ls ~'; lshome

User defined functions can replace aliases in the shells which have them; i.e. all shells except tcsh.

The Korn shell has a feature called tracked aliases which are identical to the external command hashes of the other shells.

built-ins

run built-in or external command

When resolving commands, user-defined functions take precedence over shell built-ins which take precedence over external commands. If a user-defined function is hiding a built-in or external command, the command premodifier can be used to run the latter.

run with explicit environment

How to run a command with an explicit environment. env -i clears the environment of exported variables and only provides the external command with the environment variables that are explicitly specified. If the -i option is not specified then the environment is not cleared, which in many cases is no different than if the command had been run directly without the env command. The env command without the -i option is used in shebang scripts to avoid hard-coding the path of the interpreter.

Multiple environment variables can be set with the env command:

env -i VAR1=VAL1 VAR2=VAL2 ... CMD

external command hashes

External command hashes are a mapping from command names to paths on the file system.

The Korn Shell calls external command hashes "tracked aliasaes", and ksh defines hash as an alias for alias -t.

command type

Determine what type a command is. The possible types are alias, shell function, shell builtin, or a path to an external command. If the command is not found an exit status of 1 is returned.

command path

Return the absolute path for an external command. For shell functions and shell builtins the name of the command is returned. For aliases the statement used to define the alias is returned. If the command is not found an exit status of 1 is returned.

command paths

Execution Control

break

Exits the enclosing for, select, until, or while loop.

case

The syntax for a switch statement.

Default clauses, which are indicated by the * pattern in most shells, are optional.

continue

Go to the next iteration of the enclosing for, select, until, or while loop.

for

A loop for iterating over a list of arguments.

zsh has alternate syntax which uses parens instead of the in keyword:

for VAR (ARG ...)
do
  CMD
  ...
done

goto

tcsh supports the goto statement. The target the first line containing just the label followed by a colon. Here's an example:

#/bin/tcsh
goto foo
echo "goto doesn't work!"
exit -1
foo:
echo "goto works"

if

The if statement.

The test which is the argument of if or elif can be any simple command, pipeline, or list of commands. The test executes and if the exit status is zero the corresponding clause is also executed.

Often the test which is the argument of if or elif will be one of the test operators: test, [ ], [[ ]], or (( )).

The elif and else clauses are optional.

tcsh:

The argument of if and elif clauses must be an expression inside parens. Unlike the other shells it cannot be an arbitrary command. One can think of expressions as being built-in to the tcsh shell language rather than being delegated to specialized (albeit built-in) commands such as test and [ ].

Note that the then keyword must be on the same line as the conditional expression. This is different from the POSIX syntax where the then keyword is separated from the test command by a newline or semicolon.

The else if and else clauses are optional.

repeat

Here are a couple of ways to do something 10 times if you aren't using tcsh. Neither technique is POSIX compliant, however:

for i in `seq 1 10`; do echo "la"; done

for i in {1..10}; do echo "la"; done

select

The select statement creates a numbered menu inside an infinite loop. Each time the user selects one of the numbers the corresponding command is executed. The user can use ^D or EOF to exit the loop.

On each iteration var is set to the value corresponding to the number the user chose. The break keyword can be used to give the user a numbered option for exiting the loop.

until

The remarks above on if conditions also apply to the until loop condition.

while

The remarks above on if conditions also apply to the while loop condition.

Redirection

Echo and Read

echo with newline, without newline

How to echo the arguments separated by spaces and followed by a newline; how to suppress the trailing newline.

echo with c escapes, without c escapes

How to make echo interpret C-style backslash escape sequences; how to make echo not interpret escape sequences.

The -e and -E options are not part of the POSIX standard. Nor does the standard define a default behavior for echo. As a result portable scripts use printf for output.

printf

man 3 printf

read

How to read a line of input into one or more variables.

When multiple variables are specified the value of IFS which by default contains the whitespace characters is used to split the input. If there are fewer variables than split values, then the last variable will contain a concatenation of the remaining values with their original separators. If there are fewer values then the extra variables are set to the empty string.

bash and dash use the -p option to set a prompt. ksh and zsh use a ?str suffix appended to the first variable to set the prompt.

The user can put a backslash in front of a newline to split the input up over multiple lines. The backslash and newline are stripped from the input. The user can put backslash into the variable by entering two backslashes. The -r option disables this feature, allowing the user to enter literal backslashes with a single keystroke.

tcsh gets input from the user by reading from the special variable $<. Backslashes are always interpreted literally.

miscellaneous i/o commands

Files and Directories

change current directory

Change the current directory to the specified directory. If the directory starts with a slash '/' then it is taken to be an absolute path. If it does not it is treated as a relative path and CDPATH is used as a colon separated list of starting directories. By default CDPATH is empty in which case the current directory '.' is used as a starting point. See also the section on tilde expansion.

If there is no argument then the current directory is changed to $HOME.

If the argument is a hyphen '-' then the current directory is changed to $OLDPWD which is the most recent former current directory.

When the -P option is used, PWD will be set to the physical path of the current directory; i.e. any symbolic links will be resolved. If the current directory is being displayed in the prompt this will also be set the physical path.

zsh:

When the -s option is used, attempting to change directory into a path containing symlinks will fail.

directory stack

Push a directory provided as an argument onto the directory stack. The directory becomes the current directory.

Pop a directory off the directory stack. The popped directory becomes the current directory.

List the directory stack.

print current directory

Show the current directory. The same as executing:

echo $PWD

source

The source built-in executes the commands in another file using the current shell process and environment.

Some shells will pass arguments to the file being sourced; i.e. the following invocation would set $1, $2, and $3 to bar, baz, and quux while executing foo.sh:

source foo.sh bar baz quux

umask

Set the shell file mode creation mask. umask is a POSIX syscall.

The mask consists of 3 octal digits which apply to the user, group, and other permissions respectively. Each octal digit contains 3 bits of information. In order of most to least significant the bits apply to the read, write, and execute permissions.

Setting a bit in the mask guarantees that the corresponding bit in the file permissions will not be set when a file is created. The logic for computing the file permissions can be expressed with the following shell code:

mask=8#022
perms=8#777

printf "0%o\n" $(( $perms & ~ $mask ))

Here is the same logic in C code:

unsigned int mask = 0022;
unsigned int perms = 0777;

printf("%o\n", perms & ~mask);

If umask is given a numeric argument it is always interpreted as octal; a leading zero is not required.

umask also supports the symbolic notation used by chmod. In this case the argument is one or more 3 character sequences of the format [agou][-+][rwx] separated by commas.

basename

How to extract the basename from a pathname. The following are equivalent:

basename $FILENAME
echo -n ${FILENAME##*/}

dirname

How to extract the dirname from a pathname. The following are equivalent:

dirname $FILENAME
echo -n ${FILENAME%/*}

Process and Job Control

xargs splits standard input on spaces and newlines and feeds the arguments to argument of xargs which is executed as a command. The input delimiter can be changed to null characters with the -0 flag (useful with find -print0) or to the value of the -d flag argument.

By default if the length of the input is more than 4096 characters the input will be broken up and the command run multiple times. This number can be increased with the -s flag up to system configuration variable ARG_MAX. It is also possible to call the command multiple times feeding it a prescribed number of arguments each time using the -n flag. The -t flag will write to standard error the command that is being invoked and its arguments before each invocation.

The -P flag can be used to for parallelization. The argument is the max number of simultaneous processes.

Arguments and Options

options can be set by the script using set. Also set -o (bash) and pipefail.

execute command and exit

Shell executes a single command which is provided on the command line and then exits.

usage

Shell provides list of options and exits.

interactive shell

An interactive shell is one that is not provided a script when invoked as an argument or is not invoked with the -c option. The -i option makes a script interactive regardless. Typically an interactive shell gets its input from and sends its output to a terminal. An interactive shell ignores SIGTERM and will handle but not exit when receiving a SIGINT. Interactive shells display a prompt and enable job control. In an interactive shell the octothorp # causes a syntax error, unlike in non-interactive shells where it is treated as the start of a comment.

login shell

A login shell is a special type of interactive shell. It executes different startup files and will also execute any logout files. When it exits it sends a SIGHUP to all jobs. (is this true?) A login shell ignores the suspend built-in.

make posix compliant

Change the behavior of the shell to be more POSIX compliant.

restricted mode

Shell runs in restricted mode.

show version

Show version and exit.

shift positional parameters

Outside of a function shift operates on the command line arguments. Inside a function shift operates on the function arguments.

set positional parameters

How to set the positional parameters from within a script.

getopts

How to process command line options.

getopts operates on the positional parameters $1, $2, …

The first argument to getopts is a word specifying the options. The options are single characters which cannot be ':' or '?'. The colon ':' indicates that the preceding letter is an option which takes an argument. If an option is encountered which is not in the option word, getopts sets the variable to '?'.

while getopts a:b:c:def OPT
do
    case $OPT in
        a) OPTA=$OPTARG ;;
        b) OPTB=$OPTARG ;;
        c) OPTC=$OPTARG ;;
        d) OPTD=1 ;;
        e) OPTE=1 ;;
        f) OPTF=1 ;;
    esac
done

Startup Files

bash:

When logging in bash will only execute one of ~/.bash_profile, ~/.bash_login, or ~/.profile. It executes the first file that exists.

History Commands and Expansion

command history: listing

How to list recent commands; how to list all commands; how to list commands with the time they were run.

command history: running

How to run a command in the history by command number; how to run the most recent command in the history matching a prefix.

command history: deleting

How to delete a command from the history by command number; how to clear the command history.

command history: fixing

Use the following syntax to edit commands from the history list and run them:

fc [-e EDIT_CMD] [-r] [FIRST [LAST]]

If EDIT_CMD is not specified, the value in the FCEDIT or EDITOR environment variable is used.

If FIRST and LAST are specified, these indicate the numbers of the range of commands to edit. If FIRST is specified but LAST is not, only that command at that number is edited and run. If neither is specified the last command is edited and run.

The -r flag reverses the order of the commands.

To simply list commands the following flags can be used:

fc -l[r] [FROM]
fc -l[r] -NUMBER_CMDS

If neither FROM nor -NUMBER_CMDS is specified the last 16 commands is printed. Use -NUMBER_CMDS (i.e. a negative number) to list the last NUMBER_CMDS commands. Use FROM (i.e. a positive number) to list all commands from FROM on.

The -r flag reverses the order of the commands

To rerun a recent command without editing it use:

fc -s [PAT=REP] [START_OF_CMD]

If START_OF_CMD is specified the last command that starts with START_OF_CMD will be run. If START_OF_CMD is not specified the last command will be run.

If PAT=REP is specified then each occurrence of PAT will be replaced with REP in the command before it is run.

ksh:

hist is a synonym for fc with the sole difference that HISTEDIT is the environment variable that determines the editor instead of FCEDIT.

zsh:

r is an alias for fc -s

command history file

Readline and Tab Completion

Prompt Customization

bash (1989)

bash

The Bourne Again shell is a GNU replacement for the Bourne shell. It can run almost all Bourne scripts and POSIX compliant scripts, and operating systems often use bash as /bin/sh. Because bash has many extensions it is not a good shell to use for determining POSIX compliance.

csh (1978)

csh

The C shell was written by Bill Joy and released as part of the second Berkeley Standard Distribution.

It introduced features that were widely adopted by other shells: history expansion, aliases, tilde notation, and job control.

The C shell was so named because it looked more like C than the Bourne shell. It still used keywords to mark off blocks instead of keywords, but its expressions were delimited by parens instead of square brackets and relational operators such as < and <= could be used instead of -lt and -le. The Unix community nevertheless eventually chose a derivation of the Bourne shell as the standard scripting language. In fact, writing scripts for the C shell is not recommended.

The classic Macintosh operating system had a development environment called The Mac Programmer's Workbench. It included a shell that was derived from the C shell.

dash (2002)

dash

The Debian Almquist shell, dash, was originally a Linux port of the NetBSD Almquist shell, ash. It is POSIX compliant. It is also smaller than the other shells: on Ubuntu Linux the executable is about 100k whereas the other shells are in the 300k-900k range.

dash does not keep a command history or offer command line editing.

ksh (1983)

ksh

The Korn shell added history and job control but otherwise stayed consistent with the Bourne shell. The POSIX standard for the shell was largely based on the Korn shell.

zsh can be used to emulate ksh and both Mac OS X and Ubuntu do this. Switching back and forth between the regular and ksh emulating version of zsh corrupts the .zsh_history file, however. Installing the AT&T version of ksh will prevent this.

sh

POSIX 2008

A succession of shells have been installed at /bin/sh which are known today by the engineers who implemented them: the Thompson shell, the Mashey shell, and the Bourne shell.

The Bourne shell appeared in 1977. It was the first shell that could be used for serious programming. It introduced the execution control structures that are used in most of the modern Unix shells. These control structures, with their distinctive reversed words for marking the end of blocks: fi and esac, were borrowed from Algol 68. However, where Algol 68 uses od the Bourne shell uses done. The Bourne shell also introduced arbitrary length variable names; the Mashey shell by contrast was limited to single letter variable names.

Whatever is installed at /bin/sh should probably be POSIX compliant. Mac OS X uses bash, which changes its behavior somewhat and operates in POSIX mode when invoked as sh. One can also get this behavior by invoking bash with the --posix flag.

Ubuntu makes /bin/sh a symlink to /bin/dash.

tcsh (1981)

tcsh

The TENEX C shell, tcsh, was upgraded version of the C Shell which added tab completion, a feature originally used in the TENEX operating system.

tcsh is backwardly compatible with csh and on many systems csh is simply a symlink to tcsh.

tcsh is the default shell on FreeBSD and it was the default shell on Mac OS X until version 10.3 was introduced in 2003.

The following tcsh built-ins interact with the terminal settings:

• echotc
• settc
• setty
• telltc
• termname

zsh (1990)

The Z shell, zsh, is documented by multiple man pages:

zsh has these builtins for managing the completion module:

• comparguments
• compcall
• compctl
• compdescribe
• compfiles
• compgroups
• compquote
• comptags
• comptry
• compvalues

The following zsh built-ins interact with the terminal settings:

• echotc
• echoti
• getcap
• ttyctl

Special zsh builtins:

• autoload
• zcompile
• zformat
• zmodload
• zparseopts
• zstyle