Guide
This guide will show you the way through the dense forest of
available tmt
features, commands and options. But don’t be
afraid, we will start slowly, with the simple examples first. And
then, when your eyes get accustomed to the shadow of omni-present
metadata trees, we will slowly dive deeper and deeper so that
you don’t miss any essential functionality which could make your
life smarter, brighter and more joyful. Let’s go, follow me…
The First Steps
Installing the main package with the core functionality is quite straightforward. No worry, there are just a few dependencies:
sudo dnf install -y tmt
Enabling a simple smoke test in the continuous integration should be a joy. Just a couple of concise commands, assuming you are in your project git repository:
tmt init --template mini
vim plans/example.fmf
Open the example plan in your favorite editor and adjust the smoke test script as needed. Your very first plan can look like this:
summary: Basic smoke test
execute:
script: foo --version
Now you’re ready to create a new pull request to check out how it’s working. During push, remote usually shows a direct link to the page with a Create button, so now it’s only two clicks away:
git add .
git checkout -b smoke-test
git commit -m "Enable a simple smoke test"
git push origin -u smoke-test
But perhaps, you are a little bit impatient and would like to see the results faster. Sure, let’s try the smoke test here and now, directly on your localhost:
tmt run --all provision --how local
If you’re a bit afraid that the test could break your machine or just want to keep your environment clean, run it in a container instead:
sudo dnf install -y tmt-provision-container
tmt run -a provision -h container
Or even in a full virtual machine if the container environment is not enough. We’ll use the libvirt to start a new virtual machine on your localhost. Be ready for a bit more dependencies here:
sudo dnf install -y tmt-provision-virtual
tmt run -a provision -h virtual
Don’t care about the disk space? Simply install tmt-all
and
you’ll get all available functionality at hand. Check the help to
list all supported provision methods:
sudo dnf install tmt-all
tmt run provision --help
Now when you’ve met your --help
friend you know everything you
need to get around without getting lost in the forest of available
options:
tmt --help
tmt run --help
tmt run provision --help
tmt run provision --how container --help
Go on and explore. Don’t be shy and ask, --help
is eager to
answer all your questions ;-)
Checking data validity
It is easy to introduce a syntax error to one of the fmf files and
make the whole tree broken. You should run tmt lint
before
pushing changes, ideally even before you commit your changes.
You can set up pre-commit to do it for you. Add to your
repository’s .pre-commit-config.yaml
:
repos:
- repo: https://github.com/teemtee/tmt.git
rev: 1.23.0
hooks:
- id: tmt-lint
This will run tmt lint --source
for all modified fmf files.
There are hooks to just check tests tmt-tests-lint
, plans
tmt-plans-lint
or stories tmt-stories-lint
explicitly.
From time to time you might want to run pre-commit autoupdate
to refresh config to the latest version.
Under The Hood
Now let’s have a brief look under the hood. For storing all config
data we’re using the Flexible Metadata Format. In short, it is
a yaml
format extended with a couple of nice features like
inheritance or virtual hierarchy which help to maintain
even large data efficiently without unnecessary duplication.
The data are organized into trees. Similarly as with git
,
there is a special .fmf
directory which marks the root of the
fmf metadata tree. Use the init
command to initialize it:
tmt init
Do not forget to include this special .fmf
directory in your
commits, it is essential for building the fmf tree structure which
is created from all *.fmf
files discovered under the fmf root.
Plans
As we’ve seen above, in order to enable testing the following plan is just enough:
execute:
script: foo --version
Store these two lines in a *.fmf
file and that’s it. Name and
location of the file is completely up to you, plans are recognized
by the execute
key which is required. Once the newly created
plan is submitted to the CI system test script will be executed.
By the way, there are several basic templates available which can
be applied already during the init
by using the --template
option or the short version -t
. The minimal template, which
includes just a simple plan skeleton, is the fastest way to get
started:
tmt init -t mini
Plans are used to enable testing and group relevant tests together. They describe how to discover tests for execution, how to provision the environment, how to prepare it for testing, how to execute tests, report results and finally how to finish the test job.
Here’s an example of a slightly more complex plan which changes the default provision method to container to speed up the testing process and ensures that an additional package is installed before the testing starts:
provision:
how: container
image: fedora:33
prepare:
how: install
package: wget
execute:
how: tmt
script: wget http://example.org/
Note that each of the steps above uses the how
keyword to
choose the desired method which should be applied. Steps can
provide multiple implementations which enables you to choose the
best one for your use case. For example to prepare the guest it’s
possible to use the install method for
simple package installations, ansible
for more complex system setup or shell
for arbitrary shell commands.
Tests
Very often testing is much more complex than running just a
single shell script. There might be many scenarios covered by
individual scripts. For these cases the discover
step can
be instructed to explore available tests from fmf metadata as
well. The plan will look like this:
discover:
how: fmf
execute:
how: tmt
Tests, identified by the required key test
,
define attributes which are closely related to individual test
cases such as the test script,
framework, directory path
where the test should be executed, maximum test
duration or packages
required to run the test. Here’s an
example of test metadata:
summary: Fetch an example web page
test: wget http://example.org/
require: wget
duration: 1m
Instead of writing the plan and test metadata manualy, you might
want to simply apply the base
template which contains the plan
mentioned above together with a test example including both test
metadata and test script skeleton for inspiration:
tmt init --template base
Similar to plans, it is possible to choose an arbitrary name for
the test. Just make sure the test
key is defined. However, to
organize the metadata efficiently it is recommended to keep tests
and plans under separate folders, e.g. tests
and plans
.
This will also allow you to use inheritance to prevent
unnecessary data duplication.
Stories
It’s always good to start with a “why”. Or, even better, with a
story which can describe more context behind the motivation.
Stories can be used to track implementation, test and
documentation coverage for individual features or requirements.
Thanks to this you can track everything in one place, including
the project implementation progress. Stories are identified by the
story
attribute which every story has to define or inherit.
An example story can look like this:
story:
As a user I want to see more detailed information for
particular command.
example:
- tmt test show -v
- tmt test show -vvv
- tmt test show --verbose
In order to start experimenting with the complete set of examples
covering all metadata levels, use the full
template which
creates a test, a plan and a story:
tmt init -t full
Core
Finally, there are certain metadata keys which can be used across all levels. Core attributes cover general metadata such as summary or description for describing the content, the enabled attribute for disabling and enabling tests, plans and stories and the link key which can be used for tracking relations between objects.
Here’s how the story above could be extended with the core
attributes description
and link
:
description:
Different verbose levels can be enabled by using the
option several times.
link:
- implemented-by: /tmt/cli.py
- documented-by: /tmt/cli.py
- verified-by: /tests/core/dry
Last but not least, the core attribute adjust provides a flexible way to adjust metadata based on the Context. But this is rather a large topic, so let’s keep it for another time. In the next chapter we’ll learn how to comfortably create new tests and plans.