Results

To integrate tmt with other tools, tmt should standardize and document the format in which test results are saved on storage.

The following text defines a YAML file structure tmt uses for storing results. tmt itself will use it when saving results of execute step, and custom test results are required to follow it when creating their results.yaml file.

Tests may choose JSON instead of YAML for their custom results file and create results.json file, but tmt itself will always stick to YAML, the final results would be provided in results.yaml file in any case.

Results are saved as a single list of dictionaries, each describing a single test result.

 # String, name of the test.
 name: /test/one

 # fmf ID of the test.
 fmf-id:
   url: http://some.git.host.com/project/tests.git
   ref: main
   name: /test/one
   path: /

 # String, the effective outcome of the test execution.
 result: "pass"|"fail"|"info"|"warn"|"error"|"skip"

 # String, the original outcome of the test execution.
 original-result: "pass"|"fail"|"info"|"warn"|"error"|"skip"

 # List of strings, optional comments to report with the result.
 note:
   - Things were great.

 # List of strings, paths to log files.
 log:
   - path/to/log1.txt
   - path/to/log2.log
     ...

 # Mapping, collection of various test IDs, if there are any to track.
 ids:
   some-id: foo
   another-id: bar

 # String, when the test started, in an ISO 8601 format.
 start-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

 # String, when the test finished, in an ISO 8601 format.
 end-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

 # String, how long did the test run.
 duration: hh:mm:ss

 # Integer, serial number of the test in the sequence of all tests of a plan.
 serial-number: 1

 # Mapping, describes the guest on which the test was executed.
 guest:
   name: client-1
   role: clients

 # String, path to /data directory storing possible test artifacts
 data-path: path/to/test/data

 # Mapping, stores the actual fmf context defined for this test.
 # It's a combination of the context provided via command line
 # and plan's `context` key.
 context:
   some-dimension:
     - its-value
   another-dimension:
     - first-value
     - second-value
       ...

# Represents results of all test checks executed as driven by test's `check`
# key. Fields have the same meaning as fields of the "parent" test result, but
# relate to each check alone.
check:
    # String, the effective outcome of the check execution.
  - result: "pass"|"fail"|"info"|"warn"|"error"|"skip"

    # String, the original outcome of the check execution.
    original-result: "pass"|"fail"|"info"|"warn"|"error"|"skip"

    # List of strings, optional comments to report with the result.
    note:
      - Things were great.

    # List of strings, paths to logs files.
    log:
      - path/to/check/log1.txt
      - path/to/check/log2.log
        ...

    # String, when the check started, in an ISO 8601 format.
    start-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

    # String, when the check finished, in an ISO 8601 format.
    end-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

    # String, how long did the check run.
    duration: hh:mm:ss

    # String, name of the check. Corresponds to the name of the check
    # specified in test metadata.
    name: dummy

    # String, the place in test workflow when the check was executed.
    event: "before-test"|"after-test"

# Represents results of all subtests, if reported by the test. Fields
# have the same meaning as fields of the "parent" test result, but relate
# to each test alone.
subresult:
    # String, name of the test phase.
  - name: First test case

    # String, the effective outcome of the phase execution.
    result: "pass"|"fail"|"info"|"warn"|"error"|"skip"

    # String, the original outcome of the phase execution.
    original-result: "pass"|"fail"|"info"|"warn"|"error"|"skip"

    # List of strings, optional comments to report with the result.
    note:
      - Things were great.

    # List of strings, paths to log files.
    log:
      - path/to/subresult/log1.txt
      - path/to/subresult/log2.log
        ...

    # String, when the subtest started, in an ISO 8601 format.
    start-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

    # String, when the subtest finished, in an ISO 8601 format.
    end-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

    # String, how long did the subtest run.
    duration: hh:mm:ss

    # List, results of checks performed for this subtest.
    # It follows the same structure as the `check` key in the
    # "parent" test result.
    check:
      ...

The result key can have the following values:

pass

Test execution successfully finished and passed.

info

Test finished but only produced an informational message. Represents a soft pass, used for tests with the result attribute set to info. Automation must treat this as a passed test.

warn

A problem appeared during test execution which does not affect test results but might be worth checking and fixing. For example test cleanup phase failed. Automation must treat this as a failed test.

error

Undefined problem encountered during test execution. Human inspection is needed to investigate whether it was a test bug, infrastructure error or a real test failure. Automation must treat it as a failed test.

fail

Test execution successfully finished and failed.

skip

Test was discovered but not executed. Can be used when a single process produces multiple results but not all tests were run.

The name and result keys are required. Also, name, result, and event keys are required for entries under check key, and name and result keys are required for entries under subresult key. Custom result files may omit all other keys, although tmt plugins will strive to provide as many keys as possible.

When importing the custom results file, each test name referenced in the file by the name key would be prefixed by the original test name. A special case, name: /, sets the result for the original test itself.

The log key must list relative paths. Paths in the custom results file are treated as relative to ${TMT_TEST_DATA} path. Paths in the final results file, saved by the execute step, will be relative to the location of the results file itself.

The first log item is considered to be the “main” log, presented to the user by default.

The serial-number, guest and fmf-id keys, if present in the custom results file, will be overwritten by tmt during their import after test completes. This happens on purpose, to assure this vital information is correct.

Similarly, the duration, start-time and end-time keys, if present in the special custom result, representing the original test itself - name: / -, will be overwritten by tmt with actual observed values. This also happens on purpose: while tmt cannot tell how long it took to produce various custom results, it is still able to report the duration of the whole test.

The same applies to context, tmt will set this key for the original test result in the custom result set to the value known to tmt.

The original-result key holds the outcome of a test, check or subresult as reported by the test, check or subresult itself, without any additional influence. A reported outcome may be a subject to interpretation, and tmt may consider additional inputs and eventually report different effective outcome in the result key.

The following rules apply when it comes to test, test check and subresult outcomes and their interpretation and effects:

  • Test outcome is interpreted according to result. It is not yet planned for test check outcomes be interpreted. It is not yet planned for subresult outcomes to be interpreted.

  • Test checks and subresults do not influence the effective test outcome. It is however planned for test check outcomes to affect the test outcome, see https://github.com/teemtee/tmt/issues/3185 for more details. It is not planned for subresults to affect the test outcome, and we do not expect them to gain this effect.

See also the complete JSON schema.

For custom results files in JSON format, the same rules and schema apply.

Changed in version 1.30: fmf context is now saved within results, in the context key.

Changed in version 1.34: phase results are now defined, under the phase key.

Changed in version 1.36: phase results are now renamed to the subresult key.

Changed in version 1.37: original result of test, subtest and check is stored in original-result key.

Changed in version 1.41: note field changed from a string to a list of strings.

Examples:

# Example content of results.yaml
- name: /test/passing
  result: pass
  serial-number: 1
  log:
    - pass_log
  start-time: "2023-03-10T09:44:14.439120+00:00"
  end-time: "2023-03-10T09:44:24.242227+00:00"
  duration: 00:00:09
  note:
    - good result
  ids:
    extra-nitrate: some-nitrate-id
  guest:
    name: default-0

- name: /test/failing
  result: fail
  serial-number: 2
  log:
    - fail_log
    - another_log
  start-time: "2023-03-10T09:44:14.439120+00:00"
  end-time: "2023-03-10T09:44:24.242227+00:00"
  duration: 00:00:09
  note:
    - fail result
  guest:
    name: default-0

# Example content of custom results file
- name: /test/passing
  result: pass
  log:
    - pass_log
  duration: 00:11:22
  note:
    - good result
  ids:
    extra-nitrate: some-nitrate-id

- name: /test/failing
  result: fail
  log:
    - fail_log
    - another_log
  duration: 00:22:33
  note:
    - fail result

# Example of a perfectly valid, yet stingy custom results file
- name: /test/passing
  result: pass

- name: /test/failing
  result: fail

# Example of test check results
- name: /test/passing
  result: pass
  serial-number: 1
  log:
    - pass_log
  start-time: "2023-03-10T09:44:14.439120+00:00"
  end-time: "2023-03-10T09:44:24.242227+00:00"
  duration: 00:00:09
  note:
    - good result
  ids:
    extra-nitrate: some-nitrate-id
  guest:
    name: default-0
  check:
    - name: abrt
      event: after-test
      result: pass
      log: []
      note: []
    - name: kernel-panic
      event: after-test
      result: pass
      log: []
      note: []

/* Example content of custom results.json */
[
  {
    "name": "/test/passing",
    "result": "pass",
    "log": ["pass_log"],
    "duration": "00:11:22",
    "note": ["good result"]
  }
]

Status: implemented and verified