cross-platform tool for testing command-line programs. The repo has moved: use https://github.com/simonmichael/shelltestrunner . (http://joyful.com/shelltestrunner)
<div id=title> <h1>Command-line testing</h1> <img src="site/title2.png"> </div>
Human or Replicant ? If you have a testing situation such as this<sup><small>1</small></sup>, we at Joyful Corp. can help.
shelltestrunner is a command-line tool for testing other command-line programs, or general shell commands, on (eg) GNU/Linux, Mac and Windows.
It reads tests which specify a command to run, some input, and the expected stdout, stderr, and exit status output. It can run tests in parallel, selectively, with a timeout, with color output, or with expected/actual differences highlighted. shelltestrunner is free software released under GPLv3+.
<a name="note1"> <small><sup>1</sup> expressed on the command line</small> </a>
|Elsewhere:<br><br><br>||Get GHC and cabal (or the Haskell Platform),<br>ensure
Tests are kept in files with the
.test suffix by default. Here's a simple test file:
# echo, given no input, prints nothing and terminates normally echo >>>= 0
and another, containing two more tests:
# cat copies its input to stdout, nothing appears on stderr, exit status is 0 cat <<< foo >>> foo >>>2 >>>= 0 # cat prints an error containing "unrecognized option" if given a bad flag cat --no-such-flag >>>2 /unrecognized option/ >>>= !0
Run the tests:
$ shelltest echo.test cat.test :echo.test: [OK] :cat.test:1: [OK] :cat.test:2: [OK] Test Cases Total Passed 3 3 Failed 0 0 Total 3 3
<!-- Two formats are supported: -->
<!-- ### Old format -->
Test files contain one or more tests, which look like this:
# optional comment a one-line shell command <<< zero or more lines of standard input >>> zero or more lines of expected standard output (or /REGEXP/ added to the previous line) >>>2 zero or more lines of expected standard error output (or /REGEXP/ added to the previous line) >>>= STATUS (or /REGEXP/)
The command and the final exit status line are required; the other parts are optional.
/REGEXP/ pattern may be used instead of specifying the full output, in which case a match anywhere in the output allows the test to pass. The regular expression syntax is regex-tdfa's, plus you can put
/REGEXP/ to negate the match.
STATUS is a numeric exit status or a
/REGEXP/. Again, use a
! prefix to negate the match. Eg
!0 matches an unsuccessful exit.
Comment lines beginning with
# may be used between tests, but not within them.
<!-- ### New format (1.4+)
Test files contain one or more test groups consisting of:
- optional standard input, following
one or more tests. A test consists of:
- a one-line command, beginning with
- optional standard output (following
>>>) and/or standard error output (following
an optional exit status specification (following
shelltest accepts one or more test file or directory arguments. A directory means all files below it which have the test file suffix (
.test, by default).
$ shelltest --help
shelltest [OPTIONS] [TESTFILES|TESTDIRS] Common flags: -a --all Show all failure output, even if large -c --color Show colored output if your terminal supports it -d --diff Show failures in diff format -p --precise Show failure output precisely (good for whitespace) -x --exclude=STR Exclude test files whose path contains STR --execdir Run tests from within the test file’s directory --extension=EXT Filename suffix of test files (default: .test) -w --with=EXECUTABLE Replace the first word of (unindented) test commands --debug Show debug info, for troubleshooting --debug-parse Show test file parsing info and stop --help-format Display test format help -? --help Display help message -V --version Print version information --numeric-version Print just the version number -- TFOPTIONS Set extra test-framework options like -j/--threads, -t/--select-tests, -o/--timeout, --hide-successes. Use -- --help for a list. Avoid spaces.
Test commands normally run within your current directory;
--execdir makes them run within the directory where they are defined, instead.
-w/--with replaces the first word of all test commands with something else, which can be useful for testing alternate versions of a program. Test commands which have been indented by one or more spaces will not be affected by this option.
--exclude can be useful to avoid running certain tests, eg unix-specific tests when on windows and vice-versa.
The test-framework library provides additional options which you can specify after
-- (note: avoid spaces between flags and values here.) Run
shelltest -- --help for a list. Here are some useful ones:
-j NUM --threads=NUMBER number of threads to use to run tests -o NUM --timeout=NUMBER how many seconds a test should be run for before giving up -t PAT --select-tests=PAT only tests that match at least one glob pattern given by an instance of this argument will be run --hide-successes hide sucessful tests, and only show failures
- the tests defined in any
*.testfile in or below the
- in colour if possible (
- whose names<sup><small>2</small></sup> contain "
- with up to 8 tests running in parallel (
- allowing no more than 1 second for each test (
- reporting only the failures (
$ shelltest tests -c -- -targs -j8 -o1 --hide
<a name="note2"><small><sup>2</sup> A test's name is what you see when running tests, ie the file name plus the sequence number within the file. </small></a>
$ darcs get http://hub.darcs.net/simon/shelltestrunner
<div id="donate-buttons" style="float:right; padding-left:1em;"> <a title="Donate via Gittip" href="https://www.gittip.com/simonmichael"><img src="/site/gittip.png" alt="Gittip"></a> <a style="margin-left:1em;" title="Donate via Paypal" href="https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=2PLCMR67L4G3E"><img src="/site/paypal.gif" alt="Paypal"></a> </div> Feedback, testing, code, documentation, packaging, blogging are most welcome. Here's the <!-- 2012 user survey --> 2012 user survey. or chat me (
sm on irc.freenode.net).
<a name="note3"><small><sup>3</sup> For help with darcs see here. </small></a>
Simon Michael wrote shelltestrunner, inspired by John Wiegley's test system for Ledger.
Code contributors include: John Macfarlane, Bernie Pope, Trygve Laugstøl, Iustin Pop, Sergei Trofimovich, Andrés Sicard-Ramírez, John Chee.
shelltestrunner depends on several fine libraries, in particular Max Bolingbroke's test-framework, and of course on the Glorious Haskell Compiler.
The Blade Runner font is by Phil Steinschneider.
<!-- http://www.explore-science-fiction-movies.com/blade-runner-movie-quotes.html -->