Core

I want to have common core attributes used consistently across all metadata levels.

adjust

Adjust metadata based on the given context

As a user I want to adjust test, plan or story metadata based on the given context such as product, distribution, architecture or trigger.

The adjust attribute allows to modify arbitrary object metadata based on the context in which they are to be used. For example, tests may be relevant only for given environment, the set of required packages may differ across distributions or there might be a different set of plans executed for pull requests versus for the full compose integration testing.

Note

The context currently has to be specified explicitly, see the Context documentation for details. In the future, tmt will detect (at least some of) the needed information from the environment.

The value can be a dictionary or a list of dictionaries which represent individual rules to be applied. Each rule contains metadata to be merged into the original object. The following three keys are reserved for rule evaluation:

when

The condition to be evaluated in order to decide if the metadata should be merged. In the expression, you can use any defined context dimension. See the full condition syntax for details about supported operators. This is a required key.

continue

By default, all provided rules are evaluated. When set to false, the first successful rule finishes the evaluation and the rest is ignored. Must be a boolean.

because

An optional comment with justification of the adjustment. Must be a string.

Note

This covers and extends the original concept of Test Case Relevancy which is now obsoleted.

Examples:

# Disable a test for older distros
enabled: true
adjust:
    enabled: false
    when: distro < fedora-33
    because: the feature was added in Fedora 33

# Adjust the required package name
require: procps-ng
adjust:
  - require: procps
    when: distro == centos-6

# Extend the environment variables, use multiple rules
adjust:
  - environment+:
        SH: bash
    when: component == bash
    continue: true
  - when: distro < centos-6
    enabled: false

# Install the fresh pytest from pip on older distros
adjust:
    prepare+:
     - how: shell
       name: fresh-pytest
       order: 90
       script: 'python3 -m pip install -U pytest'
    when: distro < rhel-8

Status: implemented and verified

description

Detailed description of the object

Multiline string describing all important aspects of the object. Usually spans across several paragraphs. For detailed examples using a dedicated attributes ‘examples’ should be considered.

Status: implemented

enabled

Allow disabling individual tests, plans or stories

As a developer or tester I want selected tests or plans to be skipped during test execution.

When a test or a plan is broken or it is not relevant for given Context it can be disabled so that it’s skipped during the execution. For stories, this attribute might be used to mark stories which should be skipped when generating the documentation.

Must be a boolean. The default value is true.

Examples:

# Mark as disabled
enabled: false

# Disable for older distros
enabled: true
adjust:
    enabled: false
    when: distro < fedora-33
    because: the feature was added in Fedora 33

# List only enabled tests
tmt tests ls --filter enabled:true

Status: implemented and verified

id

Persistent unique identifier of the object

As a user I want to be able to track execution history of a test even if it is renamed or moved to another repository.

Sometimes tests, plans or stories are renamed or moved across the directory structure, into a different git branch or even to another repository. Any of these changes results into an update of the fmf identifier.

In order to identify objects even after they are renamed or moved the id attribute can be used to store a unique identifier which does not change and can be used for example to track the complete test execution history even if the test changed the location or name.

Must be a uniqe string, using a uuid is recommended.

Examples:

id: af994876-1c68-49a7-90e8-c8d2b189189d

Status: implemented and verified

order

Order in which the object should be handled

Sometimes it is important in which order objects are processed. For example when running tests we need to execute some tests first or when exporting stories we need to arrange content in the right sequence to create a meaningful documentation. Must be an integer, negative values are allowed as well. The default value is 50.

Examples:

order: 30

Status: implemented and verified

summary

Concise summary describing purpose of the object

Must be a one-line string, should be up to 50 characters long. It is challenging to be both concise and descriptive, but that is what a well-written summary should do.

Status: implemented

tag

Free form tags for easy filtering

As a user I want to run only a subset of available tests, plans or stories.

Throughout the years, free-form tags proved to be useful for many, many scenarios. Primarily to provide an easy way how to select a subset of objects. Tags are case-sensitive. Using lowercase is recommended. Must be a string or a list of strings. Tag name must not contain spaces or commas.

Examples:

# Single tag
tag: security

# Multiple tags
tag: [security, fast]

# Multiple tags on separate lines
tag:
  - security
  - fast

Status: implemented and verified

tier

Name of the tier set this the object belongs to

As a tester testing a security advisory I want to run the stable set of important tests which cover the most essential functionality and can provide test results in a short time.

It’s quite common to organize tests or plans into “tiers” based on their importance, stability, duration and other aspects. For this tags have been used quite often in the past, now there is a dedicated field for this use case. Should be a string.

Examples:

# Basic tier one definition.
tier: "1"

# Integer values are converted to strings.
tier: 1

# Any custom string can be used as well.
tier: custom

Status: implemented