Venditan, The E-Commerce & Retail Experts
We're Recruiting, PHP Specialists, Enquire Here

Terminal Velocity Part 1 - 4 Great ways to work with automatic expansion in the shell

Come in and take a look inside the venditan hq!

We're recruiting php people, interested?

Show Me More

Terminal Velocity Part 1 - 4 Great ways to work with automatic expansion in the shell

Posted 24/06/2015 by Liam Lord

With so much getting done in the shell, lots of little shortcuts can all add up to make things quicker in the future.  After seeing people doing it the long way round, here’s the first in our series on efficiency hacks.

Note that in this series I'm only going to cover the 3 most common shells people use today, Bash, Ksh and Zsh. If you use something else you may find some of this advice doesn't work or is unreliable. In that case I suggest consulting the manual for your specific shell.

1) Globbing - An oldie but goodie that bears repeating

Globbing is the name given to the pattern matching for files. This should be something most regular users have used, but it's always handy to have a cheat sheet:

Pattern Description Reg Exp Equivalent
* Matches any number of any character .*
? Matches exactly one character .
[AD-Z]

Matches any character in the specified range. Characters may be listed without hyphens to select from the listed characters in the same way RegExps work.

In the listed instance this will match every upper case character except for B and C, since they're not listed, or included in the range D-Z

[AD-Z]
~name Matches a user's home directory  

Globbing in practice:

# Globbing basics
# Create the sample files
touch a ab bc cd de
# * Matches anything and everything
echo *
a ab bc cd de
echo a*
a ab
# ? Matches a single character
echo ?
a
echo ??
ab bc cd de
# Square brackets(and contents) match a single character
echo [abd]*
a ab bc de
echo [a-d]?
ab bc de
echo [ad-z]?
ab de
# ~ on it's own matches your home directory
echo ~
/home/llord
# ~ followed by a username matches that user's home directory
echo ~tom
/home/tom
echo ~root
/root

	

2) Extglob - Expanding on the available options

Almost all shells come with an expansion to the typical glob system that allows even more options, but before it can be used it needs to be turned, although the command to do so depends on your exact shell. If you're unsure you can run

echo $SHELL
/bin/bash

to see what shell you're running at the moment

Shell Command to turn on Command to turn off
Bash shopt -s extglob shopt -u extglob
Ksh N/A - Always on
Zsh set -o ksh_glob set +o ksh_glob

Once enabled:

Pattern Description Reg Exp Equivalent
@(123|456) Matches any specified pattern, eg:  "123" or "456" (123|456)
!(123) Matches anything not contained in the sub group (^123)
?(123) Optionally matches the sub group (123)?
*(123) Matches the sub group any number of times(including 0) (123)*
+(123) Matches the sub group at least once (123)+

Extglob in practice:

# extglob
# Create sample files
touch cheesecake chocolate-cake apple-pie pear-pie rice-pudding
echo *@(cake|pudding)
-bash: syntax error near unexpected token `('
shopt -s extglob
echo *@(cake|pudding)
cheesecake chocolate-cake rice-pudding
# You can even embed other globbing patterns, like ? or *
echo *-@(???|c*)
apple-pie chocolate-cake pear-pie
# Be careful using the not pattern, * is greedy and
# the not operator !(...) could match a single character.
echo *!(pie)
apple-pie cheesecake chocolate-cake pear-pie rice-pudding
echo !(*pie)
cheesecake chocolate-cake rice-pudding

3) More Globbing Options

There are more features than just extglob that can be turned on through shell options that affect the nature of globbing, although not all are available on all possible shells:

Option Description Bash Command Ksh Command Zsh Command
* will match .files like the .git directory, but not the . or .. symlinks. shopt -u dotglob   set -o globdots
Will remove any patterns that don’t match anything instead of keeping the literal. shopt -u nullglob set -o nullglob set -o nullglob
Patterns that don’t match will result in an error that will prevent execution of the command. Very useful if you want to make sure your patterns are correct. shopt -u failglob   set -o nomatch
Enables ** as a pattern to recursively match any directory. shopt -u globstar set -o globstar  

Other options in practice:

# Other glob options
# Set up
mkdir 5th 6th 6th/7th
touch .first .second third fourth 5th/a 5th/b 6th/c 6th/7th/d
# By default * ignores hidden files
echo *
5th 7th fourth third
shopt -s dotglob
# with dotglob they should match
echo *
5th 6th .first fourth .second third
># By default unmatched params are interpreted as literals
echo cat*
cat*
shopt -s nullglob
# with nullglob they're removed from the param list
echo cat*
 
# Turn nullglob back off and turn failglob on
shopt -u nullglob
shopt -s failglob
# with failglob on you get an error
echo cat*
-bash: no match: cat*
# without globstar there are no recursive calls
echo **
5th 6th .first fourth .second third
shopt -s globstar
# With globstar it is a recursive directory search
echo **
5th 5th/a 5th/b 6th 6th/c 6th/7th 6th/7th/d .first .second third fourth

4) Brace Expansion

While globbing will only expand to match filenames that actually exist, brace expansion will always expand to every possible combination of the provided input.

Globbing is useful for working with already existing files. Whereas brace expansion is a general purpose tool that may be used to generate file or folder names that don't exist, as well as a wide range of options not related to the filesystem.

Brace expansion works in two ways:

   1. As a comma separated list:

         {a,b,c}

# Brace expansion - lists
# Brace expansion is really useful for producing cartesian products,
echo {a,b,c}{1,2,3}
a1 a2 a3 b1 b2 b3 c1 c2 c3
# Creating a whole dir structure without repeating names,
echo {application/{controllers,library,modules,plugins,view},pub}/
application/controllers/   application/library/   application/modules/
application/plugins/       application/view       pub/
# Or even just rename a file with a minor change
$ echo V{ei,ie}wHelper
  VeiwHelper   ViewHelper

   2. As a range:

         {a..c}

# Brace expansion - ranges
# Range expansion is good for lists and loops
for i in {1..25};do echo -n "$i "; done
 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
# For example, loop over every letter in the alphabet,
# and count file starting with that letter:
for i in {a..z}; do echo -n "$i "; ls -1 $i* | wc -l; done
a 0
b 1
c 7
d 0
e 12
...

We hope you find these shortcuts useful, next part on making your life easier in directories coming soon.

Back To Posts