hub.darcs.net :: simon -> fit-py -> files

This is a python translation of fit, by Simon Michael <simon@joyful.com>.

fit (the Framework for Interactive Testing) is copyright (c) 2002
Cunningham & Cunningham, Inc, and released under the terms of the GNU
General Public License version 2 or later.

Usage: add this directory to your PYTHONPATH and, eg,

  python fit/FileRunner.py Documents/arithmetic.html Reports/arithmetic.html

Notes:
- this is fit-b021014py, based on java fit version b020927
- has not been made python 1.5 compatible
- example Documents have been converted to unix line endings
- types are handled differently from the java version, see below
- "equal-but-different" results are displayed
- Some java class-side methods have become python instance methods
- java signatures preserved in comments, remove later ?
- name changes to avoid clashes with keywords/builtins/methods:
 - Fixture.right, wrong -> rights, wrongs
 - type argument -> atype 
 - Parse.print -> use print str(p) instead
 - ActionFixture.check -> check_
- emulated multimethods variously by
 - a single method with optional arguments
 - multiple methods with differing names
 - hardcoded reference to a specific implementation, eg Fixture.check()

Todo:
- incorporate 021007 enhancements
- why ../fit/fit in tracebacks ?
- verbose/debug option
- figure out issues with use of rexec
- dates support ?
- additional data markup: <, >, !=, +-, ``
- don't sys.exit when invoked from python prompt


Dynamic types, conversion, rendering, matching etc.

Certain issues arise when using fit with a dynamically-typed language like
python. Unlike the java version, which knows the expected type of every
field, method argument and method return value, we must infer this
information from other sources or do without it. 

Some alternatives and difficulties are noted below. Now, here's the
strategy which I think is the most useful and which this version
implements: 

 - Infer all types from the test data, by treating it like a python
   string literal.

 - When checking results, compare expected and actual as typed objects
   using == by default.

 - For passing tests, also compare the repr representations; if they
   differ display both.

 - If you want to test exact repr matches, add a method to your fixture
   which returns the repr of another. Later we'll probably support this
   via `` markup.


Development notes

We are working with two kinds of table cell - input data and expected
results. 

Setting strategies

 a. set everything as strings

 b. choose a type based on the class default for the attribute

 c. choose a type based on the data format

 d. b, otherwise c

 e. other possibilities.. interfaces, markup in the test data

Matching strategies

 1. value matching, type based only on class defaults
    If there's an existing attribute and an adapter to it's type, convert
    expected to that type. Compare with actual's value.
    con:
     - fails a lot (unless your methods accept and return strings) 
       since we often can't infer type
     - expected 1 will match actual 1.0 as per python comparison rules;
       can't specify a test that requires a floating point result

 2. canonical string matching, type based only on class defaults
    If there's an existing attribute and an adapter to it's type, convert
    expected to that type. Now convert both expected & actual to their
    python string literal format (using repr) and compare.
    con:
     - fails a lot when a type cannot be inferred (unless your methods
       accept and return strings) because expected '1' will not match
       actual 1
     - and, in this situation they look the same in the report; expected
       should have quotes to show that it was treated as a string
     - more strict - when type can't be inferred, user must write 0.1 not .1
    pro:
     - on the other hand, if you arrange your methods to accept and return
       strings, this method allows exact control.

 3. value matching, type based on class defaults or data format
    If there's an existing attribute and an adapter corresponding to it's
    type, convert expected to that type. If not, infer the type from
    expected's format (eg by doing a python eval). Compare with actual's value.
    con:
     - expected 1 will match actual 1.0 as per python comparison rules;
       this (and other things ?) may cause confusion. Here the passing
       test should show expected 1, actual 1.0 to reveal what's happening.
     - can't specify a test that requires a floating point result

 4. canonical string matching, type based on class defaults or data format
    If there's an existing attribute and an adapter to it's type, convert
    expected to that type. If not, infer the type from expected's format.
    Then convert both to string literals & compare.
   
 5. value matching, type based only on data format
    pro: simple, intuitive
    con: same as 3

 6. canonical string matching, type based only on data format
    pro: simple, exact
    con: more fragile than value matching

 7. original expected & canonical actual string matching
    For result-matching cells, converting expected to a type and then back to
    string literal loses information in the case of floating point
    numbers. Eg an expected '0.666667' will be changed to it's string
    literal '0.66666700000000001'. When inferring type from data format,
    this would prevent you testing a method which returns the string
    '0.666667'.  Instead, we could treat expected as the literal result we
    expect to see, and simply compare it with the string literal form of
    actual.
    pro: simple, precise
    con: more work to specify, less generic

There seems to be a choice between exact, language/implementation-specific
tests and more lenient tests which could be more robust across different
implementations & platforms.