hub.darcs.net :: simon -> shelltestrunner -> files

cross-platform tool for testing command-line programs. The repo has moved: use https://github.com/simonmichael/shelltestrunner . (https://github.com/simonmichael/shelltestrunner)

root

<div id=title> <h1>Command-line testing</h1> <img src="site/title2.png"> </div>

  • toc

Human or Replicant ? If you have a testing situation such as this<sup><small>1</small></sup>, we at Joyful Corp. can help.

shelltestrunner is a command-line tool for testing other command-line programs, or general shell commands, on (eg) GNU/Linux, Mac and Windows.

It reads tests which specify a command to run, some input, and the expected stdout, stderr, and exit status output. It can run tests in parallel, selectively, with a timeout, with color output, or with expected/actual differences highlighted. shelltestrunner is free software released under GPLv3+.

<a name="note1"> <small><sup>1</sup> expressed on the command line</small> </a>

Getting started

---------------------------------|-------------------------------------- Debian, Ubuntu:   | apt-get install shelltestrunner Gentoo: | emerge shelltestrunner Elsewhere:<br><br><br> | Get GHC and cabal (or the Haskell Platform),<br>ensure ~/.cabal/bin is in your $PATH,<br>cabal install shelltestrunner

Tests are kept in files with the .test suffix by default. Here's a simple test file:

# echo, given no input, prints nothing and terminates normally
echo
>>>= 0

and another, containing two more tests:

# cat copies its input to stdout, nothing appears on stderr, exit status is 0
cat
<<<
foo
>>>
foo
>>>2
>>>= 0

# cat prints an error containing "unrecognized option" if given a bad flag
cat --no-such-flag
>>>2 /unrecognized option/
>>>= !0

Run the tests:

$ shelltest echo.test cat.test
:echo.test: [OK]
:cat.test:1: [OK]
:cat.test:2: [OK]

         Test Cases  Total      
 Passed  3           3          
 Failed  0           0          
 Total   3           3

That's it!

Test format

<!-- Two formats are supported: -->

<!-- ### Old format -->

Test files contain one or more tests, which look like this:

# optional comment
a one-line shell command
<<<
zero or more lines of standard input
>>>
zero or more lines of expected standard output (or /REGEXP/ added to the previous line)
>>>2
zero or more lines of expected standard error output (or /REGEXP/ added to the previous line)
>>>= STATUS (or /REGEXP/)

The command and the final exit status line are required; the other parts are optional.

A /REGEXP/ pattern may be used instead of specifying the full output, in which case a match anywhere in the output allows the test to pass. The regular expression syntax is regex-tdfa's, plus you can put ! before /REGEXP/ to negate the match.

STATUS is a numeric exit status or a /REGEXP/. Again, use a ! prefix to negate the match. Eg !0 matches an unsuccessful exit.

Comment lines beginning with # may be used between tests, but not within them.

Here are some <!-- more --> real world examples.

<!-- ### New format (1.4+)

Test files contain one or more test groups consisting of:

  • optional standard input, following < or <<<

  • one or more tests. A test consists of:

  • a one-line command, beginning with $ or $$$
  • optional standard output (following > or >>>) and/or standard error output (following >2 or >>>2) specifications

  • an optional exit status specification (following >= or >>>=) -->

Usage

shelltest accepts one or more test file or directory arguments. A directory means all files below it which have the test file suffix (.test, by default).

Command-line options:

$ shelltest --help
shelltest [OPTIONS] [TESTFILES|TESTDIRS]

Common flags:
  -a --all              Show all failure output, even if large
  -c --color            Show colored output if your terminal supports it
  -d --diff             Show failures in diff format
  -p --precise          Show failure output precisely (good for whitespace)
  -x --exclude=STR      Exclude test files whose path contains STR
     --execdir          Run tests from within the test file’s directory
     --extension=EXT    Filename suffix of test files (default: .test)
  -w --with=EXECUTABLE  Replace the first word of (unindented) test commands
     --debug            Show debug info, for troubleshooting
     --debug-parse      Show test file parsing info and stop
     --help-format      Display test format help
  -? --help             Display help message
  -V --version          Print version information
     --numeric-version  Print just the version number

     -- TFOPTIONS       Set extra test-framework options like -j/--threads,
                        -t/--select-tests, -o/--timeout, --hide-successes.
                        Use -- --help for a list. Avoid spaces.

Test commands normally run within your current directory; --execdir makes them run within the directory where they are defined, instead.

-w/--with replaces the first word of all test commands with something else, which can be useful for testing alternate versions of a program. Test commands which have been indented by one or more spaces will not be affected by this option.

--exclude can be useful to avoid running certain tests, eg unix-specific tests when on windows and vice-versa.

The test-framework library provides additional options which you can specify after -- (note: avoid spaces between flags and values here.) Run shelltest -- --help for a list. Here are some useful ones:

  -j NUM  --threads=NUMBER    number of threads to use to run tests
  -o NUM  --timeout=NUMBER    how many seconds a test should be run for before giving up
  -t PAT  --select-tests=PAT  only tests that match at least one glob pattern given by
                               an instance of this argument will be run
          --hide-successes    hide sucessful tests, and only show failures

Example:

Run

  • the tests defined in any *.test file in or below the tests/ directory (tests),
  • in colour if possible (-c),
  • whose names<sup><small>2</small></sup> contain "args" (-- -targs),
  • with up to 8 tests running in parallel (-- -j8),
  • allowing no more than 1 second for each test (-- -o1),
  • reporting only the failures (-- --hide-successes):
$ shelltest tests -c -- -targs -j8 -o1 --hide

<a name="note2"><small><sup>2</sup> A test's name is what you see when running tests, ie the file name plus the sequence number within the file. </small></a>

Contributing

The released version is on hackage (Release notes).

The latest code is on darcs hub<sup><small>3</small></sup> (browse, changes). Clone it with:

$ darcs get http://hub.darcs.net/simon/shelltestrunner

<div id="donate-buttons" style="float:right; padding-left:1em;"> <a title="Donate via Gittip" href="https://www.gittip.com/simonmichael"><img src="/site/gittip.png" alt="Gittip"></a> <a style="margin-left:1em;" title="Donate via Paypal" href="https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=2PLCMR67L4G3E"><img src="/site/paypal.gif" alt="Paypal"></a> </div> Feedback, testing, code, documentation, packaging, blogging are most welcome. Here's the <!-- 2012 user survey --> 2012 user survey. Email or chat me (sm on irc.freenode.net).

<a name="note3"><small><sup>3</sup> For help with darcs see here. </small></a>

Credits

Simon Michael wrote shelltestrunner, inspired by John Wiegley's test system for Ledger.

Code contributors include: John Macfarlane, Bernie Pope, Trygve Laugstøl, Iustin Pop, Sergei Trofimovich, Andrés Sicard-Ramírez, John Chee.

shelltestrunner depends on several fine libraries, in particular Max Bolingbroke's test-framework, and of course on the Glorious Haskell Compiler.

The Blade Runner font is by Phil Steinschneider.

<!-- http://www.explore-science-fiction-movies.com/blade-runner-movie-quotes.html -->

  • darcsden 1.2 beta built Jun 23 201810:48:19
  • report problems to simon