What is the difference between fmf and tmt?¶
The Flexible Metadata Format or
fmf is a plain text format
yaml used to store data in both human and machine
readable way close to the source code. Thanks to inheritance and
elasticity, metadata are organized in the structure efficiently,
preventing unnecessary duplication.
The Test Management Tool or
tmt is a project which
consists of the Metadata Specification which defines how tests,
plans and stories are organized, python modules implementing the
specification and the command-line tool which provides a
user-friendly way to create, debug and easily run tests.
Using tmt outside of Fedora, CentOS and RHEL distribution¶
The tmt is packaged and tested only for these three flavors, however if one installs tmt from the PyPI it can be run also on other Linux distributions.
The caveat is that installation of required packages depends on
the usage of
dnf. When tmt is executed on
the host none of these commands is necessary so tmt should work
pip install succeeds.
On the other hand - when tmt is used to execute tests on
provisioned guest it depends if the plan will try to install any
packages (either by test require,
recommend or using prepare
install plugin) it will fail as tmt
currently doesn’t work with other package management tools. This
can be worked around by installing the test dependencies (as well
rsync command) using ansible
or shell prepare plugins.
In order to safely run tests under a virtual machine started on
your laptop you only need to install the
package. By default the
session connection is used so no other
steps should be needed, just execute tests using the
If you want to use the
system connection you might need to do
a few steps to set up your box. Here’s just a couple of hints how
to get the virtualization quickly working on your laptop. See the
Getting started with virtualization docs to learn more.
Make sure the
libvirtd is running on your box:
sudo systemctl start libvirtd
Add your user account to the libvirt group:
sudo usermod -a -G libvirt $USER
Note that you might need to restart your desktop session to get it fully working. Or at least start a new login shell:
su - $USER
In some cases you might also need to activate the default network device:
sudo virsh net-start default
Here you can find vm images for download.
Container Package Cache¶
Using containers can speed up your testing. However, fetching package cache can slow things down substantially. Use this set of commands to prepare a container image with a fresh dnf cache:
podman run -itd --name fresh fedora podman exec fresh dnf makecache podman image rm fedora:fresh podman commit fresh fedora:fresh podman container rm -f fresh
Then specify the newly created image in the provision step:
tmt run --all provision --how container --image fedora:fresh
In this way you can save up to several minutes for each plan.
After a nitrate test case is migrated to
fmf git becomes the
canonical source of the test case metadata. All further changes
should be done in git and updates synchronized back to nitrate
tmt test export . --how nitrate command. Otherwise direct
changes in Nitrate might be lost.
A unique identifier of the new test metadata location is stored in
[fmf] section of test case notes. Below is the list of
attributes which are synchronized to corresponding nitrate fields:
component — components tab
contact — default tester
description — purpose-file in the structured field
duration — estimated time
enabled — status
environment — arguments
summary — description in the structured field
tag — tags tab
tier — tags (e.g.
1synced to the
The following attributes, if present, are exported as well:
extra-hardware — hardware in the structured field
extra-pepa — pepa in the structured field
extra-summary — Nitrate test case summary
extra-task — Nitrate test case script
They have the
extra prefix as they are not part of the L1
Metadata Specification and are supposed to be synced temporarily
to keep backward compatibility.
Why is the ‘id’ key added to my test during export?¶
tmt test metadata using
tmt tests export to
other test case management systems, a unique
id is created in
order to provide a persistent way to identify the test even if it
is renamed, moved across the directory structure or into a
different repository. See the id key
specification for more details.
How can I integrate tmt tests with other tools?¶
Each tmt test has a unique fmf identifier which can look like this:
name: /tests/core/docs url: https://github.com/teemtee/tmt.git ref: main
These identifiers can be used for integration with other tools,
for example to execute tmt tests using custom workflows. For this
tmt tests export command can be used to produce a
list of fmf identifiers of selected tests:
tmt tests export --fmf-id | custom-workflow --fmf-id - tmt tests export core/docs --fmf-id | custom-workflow --fmf-id -
Custom workflow can then consume generated ids and perform desired actions such as fetch the tests and execute them.
How do I migrate STI tests to tmt?¶
Standard Test Interface tests are enabled using
Ansible playbooks together with the Standard Test Roles which
make it easier to execute tests, check their results and perform
additional actions such as installing required packages. The
configuration, however, can sometimes be confusing and quite hard
tmt it is possible to achieve the same result with much
more concise and clean syntax. For majority of existing tests the
conversion is relatively straightforward. Let’s demonstrate it on
a couple of real-life examples. Below you can see the original STI
ansible playbooks and their
tmt equivalents for inspiration.
As the first step, initialize the metadata tree using the
init command in the root of the git repository. Then store the
new config files with the
.fmf extension. Naming and location
of the files is up to you. See the Guide for more details.
Running a simple binary using STI:
- hosts: localhost roles: - role: standard-test-basic tags: - classic tests: - simple: dir: . run: binary --help
tmt plan has only two lines:
execute: script: binary --help
Store them for example as
/plans/smoke.fmf and you’re done.
This example prepares testing environment by installing required packages.
- hosts: localhost tags: - atomic - classic - container roles: - role: standard-test-beakerlib tests: - cmd-line-options required_packages: - which - rpm-build - libtool - gettext
tmt example plan (L2 metadata):
summary: Check basic command line options prepare: how: install package: - which - rpm-build - libtool - gettext execute: script: cmd-line-options
Tests in the following example are fetched from a remote repository and filtered by the provided condition.
- hosts: localhost roles: - role: standard-test-beakerlib tags: - classic repositories: - repo: "https://src.fedoraproject.org/tests/shell.git" dest: "shell" fmf_filter: "tier: 1"
tmt example plan (L2 metadata):
summary: Tier 1 shell test plan discover: how: fmf url: https://src.fedoraproject.org/tests/shell.git filter: "tier: 1" execute: how: tmt
In this migration of STI a single plan (L2 metadata) is created and each original test is stored in a separate L1 metadata file (test). This approach allows the setup of different environment variables and required packages for each test.
- hosts: localhost roles: - role: standard-test-basic tags: - classic tests: - smoke27: dir: tests run: VERSION=2.7 METHOD=virtualenv ./venv.sh - smoke37: dir: tests run: VERSION=3.7 ./venv.sh required_packages: - python27 - python37 - python2-virtualenv - python3-virtualenv - python2-devel - python3-devel
tmt example: plan (L2 metadata) and tests (L1 metadata)
discover: how: fmf execute: how: tmt
test: ./venv.sh environment: VERSION: 2.7 METHOD: virtualenv require: - python27 - python2-virtualenv - python2-devel
test: ./venv.sh environment: VERSION: 3.7 require: - python37 - python3-virtualenv - python3-devel
This arrangement can be especially useful when a large number of tests is stored in the repository.
Dist Git Source¶
dist-git-source feature of the
discover step to
extract tests from the (rpm) sources.
- hosts: localhost tags: - classic roles: - role: standard-test-source
tmt example plan (L2 metadata):
discover: how: fmf dist-git-source: true
See the fmf plugin documentation for more details.