DIRECTORY CONTENTS:

The subdirectories contain log files from the competition and analyzed
results.

The file min_max_norm_constants.txt contains the normalizing values
for each domain instance (and which planner was responsible for these 
min and max values).

See the all_results.txt file in the MDP and POMDP directories for
tabulated results.

===

IMPORTANT:

(1) It is extremely easy to regenerate these results from the RDDLSim
log files.  The code that produced these files is in the SVN
repository:

  http://code.google.com/p/rddlsim/source/browse/trunk/#trunk%2Fsrc%2Frddl%2Fevaluate

See the README there for how to run the code on these results.

(2) The "truncated normalized scores" min(0,avg-norm-score-instance)
for competitors should be in the range of [0,1].

===

FILE FORMAT for all_results.txt:

These files are tab delimited, so spreadsheet programs like Excel
should be able to load the formatted data.

When reading the file, there are a few sections:

- All results: for each planner and instance provides in order
  * count of runs completed
  * normalized score (see file min_max_norm_constants.txt for 
                      normalizing constants for each domain/instance,
                      score may be below 0, truncation not done yet)
  * raw average score
  * stderr of average score (+/- 95% interval)
  * min score
  * max score

  NOTE: only the last 30 runs are used for above stats if > 30 runs
        submitted.  If < 30 runs submitted then 
        max(noop-policy-avg, random-policy-avg) is used to pad the
        the result list before avg, min, max are determined.
        
- Avg of all normalized rewards by domain
  * count of results averaged - direct average of raw normalized scores
  * avg of normalized scores (can be < 0, not truncated)
  * stderr (+/- 95% interval)

- Avg of min(0,avg-norm-score-instance) by domain
  * count of results averaged - lower count b/c averages of truncated averages
  * avg of normalized scores (cannot be < 0, truncated to 0 if below 0)
  * stderr (+/- 95% interval)

- Avg of all normalized rewards
  * count of results averaged - direct average of raw normalized scores
  * avg of normalized scores (can be < 0, not truncated)
  * stderr (+/- 95% interval)

- Avg of min(0,avg-norm-score-instance)
  * count of results averaged - lower count b/c averages of truncated averages
  * avg of normalized scores (cannot be < 0, truncated to 0 if below 0)
  * stderr (+/- 95% interval)

