cross-platform tool for testing command-line programs (http://joyful.com/shelltestrunner)
shelltestrunner tests command-line programs (or arbitrary shell commands.) It reads simple declarative tests specifying a command, some input, and the expected output, and can run them run in parallel, selectively, with a timeout, in color, and/or with differences highlighted. shelltestrunner has been tested on gnu/linux, mac and windows; projects using it include hledger, berp, cblrepo and eddie. shelltestrunner is free software released under GPLv3+.
$ cabal install shelltestrunner
You should now have the
shelltest program in your path.
Test files, typically named
tests/*.test, contain one or more tests consisting of:
- a one-line command
- optional standard input (
<<<), standard output (
>>>) and/or standard error output (
- an exit status (
example.test, a file containing two simple tests:
# 1. let's test that echo runs. Numbering your tests can be helpful. echo >>>= 0 # 2. and now the cat command. On windows, this one should fail. cat <<< foo >>> foo >>>= 0
Run it with
$ shelltest example.test :t.test:1: [OK] :t.test:2: [OK] Test Cases Total Passed 2 2 Failed 0 0 Total 2 2
# optional comment the command to test <<< zero or more lines of standard input >>> zero or more lines of expected standard output (or /REGEXP/ added to the previous line) >>>2 zero or more lines of expected standard error output (or /REGEXP/ added to the previous line) >>>= EXITCODE (or /REGEXP/)
/REGEXP/pattern may be used instead of explicit data. In this case a match anywhere in the output allows the test to pass. The regular expression syntax is regex-tdfa's.
EXITCODEis a numeric exit status, eg
0for a successful exit.
- You can put
EXITCODEto negate the match.
- Comment lines beginning with
#may be used between tests.
shelltest program accepts one or more test file or directory paths. A directory means all files below it with the test file suffix (default:
Example: run all tests defined in or below the
tests directory with
args in their name, up to 8 in parallel, allowing a maximum of 1 second each, showing only failures and the final summary:
$ shelltest tests -- -targs -j8 -o1 --hide Test Cases Total Passed 2 2 Failed 0 0 Total 2 2
$ shelltest --help shelltest 1.2.1 shelltest [OPTIONS] [TESTFILES|TESTDIRS] Common flags: -a --all Show all failure output, even if large -c --color Show colored output if your terminal supports it -d --diff Show failures in diff format -p --precise Show failure output precisely (good for whitespace) -x --exclude=STR Exclude test files whose path contains STR --execdir Run tests from within the test file's directory --extension=EXT Filename suffix of test files (default: .test) -w --with=EXECUTABLE Replace the first word of (unindented) test commands --debug Show debug info, for troubleshooting --debug-parse Show test file parsing info and stop --help-format Display test format help -? --help Display help message -V --version Print version information -- TFOPTIONS Set extra test-framework options like -j/--threads, -t/--select-tests, -o/--timeout, --hide-successes. Use -- --help for a list. Avoid spaces.
Test commands normally run within your current directory;
--execdirmakes them run within the directory where they are defined, instead.
-w/--withreplaces the first word of all test commands with something else, which can be useful for testing alternate versions of a program. Commands which have been indented by one or more spaces will not be affected by this option.
The test-framework library provides additional options which you can specify after
--(note: avoid spaces between flags and values here.) Run
shelltest -- --helpfor a list. Here are some useful ones:
-j NUMBER --threads=NUMBER number of threads to use to run tests -o NUMBER --timeout=NUMBER how many seconds a test should be run for before giving up, by default -t TEST-PATTERN --select-tests=TEST-PATTERN only tests that match at least one glob pattern given by an instance of this argument will be run --hide-successes hide sucessful tests, and only show failures
$ darcs get http://hub.darcs.net/simon/shelltestrunner
Feedback, code, testing, documentation/blogging are most welcome.
Add your experience to the user survey (results).
or chat me (
sm on irc.freenode.net).
For focussed support/faster development/good karma you can hire me or <a href="https://www.wepay.com/donate/39988?ref=widget&utm_medium=widget&utm_campaign=donation" target="blank" style=margin:0 1em;" ><img style="vertical-align:middle;" src="https://www.wepay.com/img/widgets/donatewith_wepay.png" alt="Donate with WePay" /></a>
support latest Diff, cmdargs, test-framework; tested with GHC 7.6.1 (Magnus Therning)
fix unicode handling on GHC >= 7.2
- use the more up-to-date filemanip package for easier Debian packaging
- support latest cmdargs, test-framework, and GHC 7.4
- more readable non-quoted failure output by default; for quoted output, use
--preciseoptions now interact well
- bump process dependency to allow building with GHC 7.2.1
-a/--allflag shows all failure output without truncating
- New home page/docs
>>>=field is now required; you may need to add it to your existing tests
- Input and expected output can now contain lines beginning with
- Multiple tests in a file may now have whitespace between them
-i/--implicitoption has been dropped
-d/--diffoption shows test failures as a unified diff when possible, including line numbers to help locate the problem
-x/--excludeoption skips certain test files (eg platform-specific ones)
- Passing arguments through to test-framework is now more robust
- Fixed: parsing could fail when input contained left angle brackets
- Fixed: some test files generated an extra blank test at the end
- show plain non-ansi output by default, add --color option
- better handling of non-ascii test data. We assume that non-ascii file paths, command-line arguments etc. are UTF-8 encoded on unix systems (cf http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html), and that GHC 6.12 or greater is used. Then:
- non-ascii test file paths should render correctly, eg in failure messages
- non-ascii test commands should run correctly
- non-ascii expected output should match correctly
- non-ascii regular expressions should match correctly. (Caveat: not thoroughly tested, this may break certain regexps, )
- use regex-tdfa instead of pcre-light for better windows compatibility To avoid a memory leak in current regex-tdfa, only regular expressions up to 300 characters in size are supported. Also, DOTALL is no longer enabled and probably fewer regexp constructs are supported. There are still issues on windows/wine but in theory this will help.
- tighten up dependencies
- rename executable to shelltest. The package might also be renamed at some point.
- better built-in help
- shell tests now include a full command line, making them more readable and self-contained. The --with option can be used to replace the first word with something else, unless the test command line begins with a space.
- we also accept directory arguments, searching for test files below them, with two new options: --execdir execute tested command in same directory as test file --extension=EXT file extension of test files (default=.test)
- more robust parsing
- --debug-parse parses test files and stops
- regexps now support escaped forward slash (
- bad regexps now fail at startup
- command-line arguments are required in a test, and may be blank
- a >>>= is no longer required to separate multiple tests in a file
- comments can be appended to delimiter lines
- comments can appear at end of file
- files need not have a final newline
- files containing nothing, all comments, or valid tests are allowed; anything else is rejected
- somewhat better errors
- allow indented input
- support negative (-) and negatively-matched (!) numeric exit codes
- let . in regexps match newline
- warn but continue when a test file fails to parse
- output cleanups, trim large output
- more flexible --implicit flag
- switch to the more robust and faster pcre-light regexp lib
- allow multiple tests per file, handle bad executable better
- show failure output in proper order
- run commands in a more robust way to avoid hangs This fixes hanging when a command generates large output, and hopefully all other deadlocks. The output is consumed strictly. Thanks to Ganesh Sittampalam for his help with this.
- --implicit-tests flag providing implicit tests for omitted fields
- --debug flag
- regular expression matching
- disallow interspersed foreign options which confused parseargs
- change comment character to #
- misc. bugfixes/improvements
- bugfix, build with -threaded
- shelltestrunner, a generic shell command stdout/stderr/exit status tester
Simon Michael wrote and maintains shelltestrunner. John Macfarlane, Bernie Pope and Trygve Laugstøl have contributed code. It relies heavily on Max Bolingbroke's test-framework, other libraries, and of course the Glorious Haskell Compiler. It was inspired by the tests for John Wiegley's ledger.