cross-platform tool for testing command-line programs. The repo has moved: use . (


title: shelltestrunner: command-line testing

<div id=title>
<h1>Command-line testing</h1>
<img src=“site/title2.png”>

  • toc

Human or Replicant ?
If you have a testing situation such as this<sup><small>1</small></sup>,
we at Joyful Corp. can help.

shelltestrunner is a command-line tool for testing other command-line
programs, or general shell commands, on (eg) GNU/Linux, Mac and Windows.

It reads tests which specify a command to run, some input, and the
expected stdout, stderr, and exit status output. It can run tests in
parallel, selectively, with a timeout, with color output, or with
expected/actual differences highlighted. shelltestrunner is free
software released under GPLv3+.

<a name=“note1”>
<small><sup>1</sup> expressed on the command line</small>

Getting started

Debian, Ubuntu:   | apt-get install shelltestrunner
Gentoo: | emerge shelltestrunner
Elsewhere:<br><br><br> | Get GHC and cabal (or the Haskell Platform),<br>ensure ~/.cabal/bin is in your $PATH,<br>cabal install shelltestrunner

Tests are kept in files with the .test suffix by default. Here's a simple test file:

# echo, given no input, prints nothing and terminates normally
>>>= 0

and another, containing two more tests:

# cat copies its input to stdout, nothing appears on stderr, exit status is 0
>>>= 0

# cat prints an error containing "unrecognized option" if given a bad flag
cat --no-such-flag
>>>2 /unrecognized option/
>>>= !0

Run the tests:

$ shelltest echo.test cat.test
:echo.test: [OK]
:cat.test:1: [OK]
:cat.test:2: [OK]

         Test Cases  Total      
 Passed  3           3          
 Failed  0           0          
 Total   3           3          

That's it!

Test format

<!– Two formats are supported: –>

<!– ### Old format –>

Test files contain one or more tests, which look like this:

# optional comment
a one-line shell command
zero or more lines of standard input
zero or more lines of expected standard output (or /REGEXP/ added to the previous line)
zero or more lines of expected standard error output (or /REGEXP/ added to the previous line)
>>>= STATUS (or /REGEXP/)

The command and the final exit status line are required; the other parts are optional.

A /REGEXP/ pattern may be used instead of specifying the full
output, in which case a match anywhere in the output allows the test
to pass. The regular expression syntax is
regex-tdfa's, plus
you can put ! before /REGEXP/ to negate the match.

STATUS is a numeric exit status
or a /REGEXP/. Again, use a ! prefix to negate the match. Eg !0 matches an unsuccessful exit.

Comment lines beginning with # may be used between tests, but not within them.

<!– more –>


New format (1.4+)

Test files contain one or more test groups consisting of:

  • optional standard input, following < or <<<
  • one or more tests. A test consists of:

  • a one-line command, beginning with $ or $$$
  • optional standard output (following > or >>>) and/or standard error output (following >2 or >>>2) specifications
  • an optional exit status specification (following >= or >>>=)


shelltest accepts one or more test file or directory arguments.
A directory means all files below it which have the test file suffix (.test, by default).

Command-line options:

$ shelltest --help

Common flags:
  -a --all              Show all failure output, even if large
  -c --color            Show colored output if your terminal supports it
  -d --diff             Show failures in diff format
  -p --precise          Show failure output precisely (good for whitespace)
  -x --exclude=STR      Exclude test files whose path contains STR
     --execdir          Run tests from within the test file’s directory
     --extension=EXT    Filename suffix of test files (default: .test)
  -w --with=EXECUTABLE  Replace the first word of (unindented) test commands
     --debug            Show debug info, for troubleshooting
     --debug-parse      Show test file parsing info and stop
     --help-format      Display test format help
  -? --help             Display help message
  -V --version          Print version information
     --numeric-version  Print just the version number

     -- TFOPTIONS       Set extra test-framework options like -j/--threads,
                        -t/--select-tests, -o/--timeout, --hide-successes.
                        Use -- --help for a list. Avoid spaces.

Test commands normally run within your current directory; --execdir
makes them run within the directory where they are defined, instead.

-w/--with replaces the first word of all test commands with something
else, which can be useful for testing alternate versions of a
program. Test commands which have been indented by one or more spaces will
not be affected by this option.

--exclude can be useful to avoid running certain tests, eg
unix-specific tests when on windows and vice-versa.

The test-framework library provides additional options which you can
specify after -- (note: avoid spaces between flags and values here.)
Run shelltest -- --help for a list. Here are some useful ones:

  -j NUM  --threads=NUMBER    number of threads to use to run tests
  -o NUM  --timeout=NUMBER    how many seconds a test should be run for before giving up
  -t PAT  --select-tests=PAT  only tests that match at least one glob pattern given by
                               an instance of this argument will be run
          --hide-successes    hide sucessful tests, and only show failures



  • the tests defined in any *.test file in or below the tests/ directory (tests),
  • in colour if possible (-c),
  • whose names<sup><small>2</small></sup> contain “args” (-- -targs),
  • with up to 8 tests running in parallel (-- -j8),
  • allowing no more than 1 second for each test (-- -o1),
  • reporting only the failures (-- --hide-successes):
$ shelltest tests -c -- -targs -j8 -o1 --hide

<a name=“note2”><small><sup>2</sup>
A test's name is what you see when running tests, ie the file name plus the sequence number within the file.


The released version is on hackage
(Release notes).

The latest code is on darcs hub<sup><small>3</small></sup>
Clone it with:

$ darcs get

<div id=“donate-buttons” style=“float:right; padding-left:1em;”>
<a title=“Donate via Gittip” href=“"><img src=”/site/gittip.png" alt=“Gittip”></a>
<a style=“margin-left:1em;” title=“Donate via Paypal” href=“"><img src=”/site/paypal.gif" alt=“Paypal”></a>
Feedback, testing, code, documentation, packaging, blogging are most welcome.
Here's the
<!– 2012 user survey –>
2012 user survey.
chat me (sm on

<a name=“note3”><small><sup>3</sup>
For help with darcs see here.


Simon Michael wrote shelltestrunner, inspired by John Wiegley's test system for Ledger.

Code contributors include:
John Macfarlane,
Bernie Pope,
Trygve Laugstøl,
Iustin Pop,
Sergei Trofimovich,
Andrés Sicard-Ramírez,
John Chee.

shelltestrunner depends on several fine libraries, in particular Max
Bolingbroke's test-framework, and of course on the Glorious Haskell

The Blade Runner font is by Phil Steinschneider.

<!– –>