Command Line

As a user I want to have an intuitive command line interface.

common

I want to have common command line options consistently used across all supported commands and subcommands.

debug

Print additional information for debugging

As a tmt developer I want to see as much details about what’s happening during execution so that I can easily reveal bugs in the code.

Debug level can be used to show detailed implementation steps, revealing what’s happening under the hood, so that tool developers are able to more easily find bugs in the code. Use the option multiple times to increase verbosity.

Examples:

tmt run -d

tmt run -ddd

tmt run --debug

Status: implemented, verified and documented

dry

Run in dry mode, just let me know what would be done

As a user I want to run commands in dry mode so that I can see what would happen if I run the command but no actions are performed or changes saved to disk.

Examples:

tmt run -n

tmt run --dry

Status: implemented, verified and documented

force

Force dangerous operations like overwriting files

Examples:

tmt test create -f

tmt test create --force

Status: implemented and documented

format

Provide machine readable output in given format

Examples:

tmt test export --how json

tmt plan export --how yaml

tmt story export --how rst

Status: implemented and documented

quiet

Enable quiet mode of the command

Examples:

tmt test show -q

tmt test show --quiet

Status: implemented and documented

verbose

Enable verbose output of the command

As a user I want to see more detailed information for particular command.

Different verbose levels can be enabled by using the option several times. For example, discover step shows only number of tests by default, list of tests in verbose level one and could show some further test details in higher verbose levels.

Examples:

tmt test show -v

tmt test show -vvv

tmt test show --verbose

Status: implemented, verified and documented

config

As a user I want to store configuration so that I don’t have to always provide my preferred default options on the command line.

Configuration should be stored as an fmf metadata tree according to L2 Metadata specification. Default values from it would be used if value is not specified in the plan.

levels

Configuration levels

Note

This is a draft, the story is not implemented yet.

There are several levels of test execution data configuration.

default

global default settings (common for most instances)

detect

detect from previous steps output (e.g. distro from build)

define

allow to override value by explicit user configuration

Let’s demonstrate these on a simple example with a distribution compose (C) and the amount of memory (M).

  • CI system is configured by default to use compose C1 installed on machines with M1 GB of memory.

  • When inspecting an artifact CI can detect that for this particular build target, compose C2 is a much better choice, and memory is fine as it is.

  • User can still explicitly define in the configuration that at least M2 GB of memory is needed for successful test execution which overrides both default and detected values.

Status: idea

system

System wide configuration

Note

This is a draft, the story is not implemented yet.

Global default values common for all users on the system should be stored under the /etc/tmt folder. It is overridden by user configuration if present.

Examples:

provision:
    how: container

Status: idea

user

User configuration

Note

This is a draft, the story is not implemented yet.

Configuration for individual users should be stored under the ~/.config/tmt folder. Overrides global system configuration.

Examples:

provision:
    image: fedora:33
report:
    how: irc
    channel: #tmt

Status: idea

init

Initialize a new metadata tree

As a user I want to start using tmt in my git repository.

base

Create a tree with some examples

Examples:

tmt init --template base

Status: implemented and verified

empty

Create an empty metadata tree

Examples:

tmt init

Status: implemented and verified

full

Create a tree with full examples

Examples:

tmt init --template full

Status: implemented and verified

mini

Create a tree with simple examples

Examples:

tmt init --template mini

Status: implemented and verified

multiple phases

As a user I want to add, remove and adjust one or more step phases via command line. I should be able to add new phases, without overwriting those loaded from fmf, adjust existing phases and their keys, and remove any of the existing phases. I should be able to adjust or remove either all phases or just a single phase by its name. I should be able to change even ``how`` key of phases, i.e. change the plugin of phases. I should be able to adjust or remove phases no matter what their source was, both phases loaded from fmf or added via CLI should be affected in the same way.

add

Allow adding new phases via command line

Examples:

# Use --insert to add a phase without overwriting phases loaded from fmf.
# Both default `display`` and newly added `html` would run.
tmt run -a report --insert html

# Add an extra report phase to create a junit report, make it run as soon as possible
tmt run -a report --insert --how html --order 1

Status: implemented and verified

remove

Allow removing phases via command line

Note

This is a draft, the story is not implemented yet.

Examples:

# Use --remove to drop an existing phase
tmt run -a report --remove --name dummy-html-report

# Use --remove without --name to drop all existing phases
tmt run -a report --remove

Status: idea

update

Allow adjusting multiple phases via command line

Examples:

# Set the default report phase to be a HTML report
tmt run -a report --how html

# When specified multiple times, only the last phase mentioned would be honored.
# In this case, only a `junit` report would run.
tmt run -a report --how html report --how junit

# Use --update to adjust a particular phase only
tmt run -a report --update --name dummy-html-report --how html --display-guest never

# Use --update without --name to adjust all phases. All phases would turn into
# `html` and would not display guest names in their output.
tmt run -a report --update --how html --display-guest never

Status: implemented and verified

update-missing

Update phase fields only when not already set

In some cases it is useful to only update those fields which are not already set in the configuration. The --update-missing option allows to fill these gaps with information which cannot be hardcoded, for example the current distro image under test.

Examples:

# Use artemis as the provision plugin (unless other method
# is specified in the config) with the given distro image
# (for all guests which do not have the image defined)
tmt run -a provision --update-missing --how artemis --image fedora-38

Status: implemented and verified

plan

As a user I want to comfortably work with plans

create

As a developer I want to easily enable CI

Provide a super-easy and user-friendly way how to enable tests in the CI. Several templates should be supported to cover common use cases.

Examples:

tmt plan create /plans/smoke --template=mini

tmt plan create /plans/features --template=full

Status: implemented, verified and documented

filter

Filter available plans

Search plans using a regular expression or a filter. Use . to select plans under the current directory.

Examples:

tmt plan ls .

tmt plan ls REGEXP

tmt plan show --filter artifact:build

Status: implemented, verified and documented

lint

Check plan against the L2 metadata specification

Verify that plan metadata are aligned with the specification, e.g. that all required attributes are present and that all attributes have correct type.

Examples:

tmt plan lint

Status: implemented and verified

ls

List available plans

Examples:

tmt plan ls

Status: implemented, verified and documented

show

Show plan configuration

Examples:

tmt plan show

Status: implemented, verified and documented

run

As a user I want to execute tests easily

debug

Handsfree debugging

As a test developer I want to automatically execute tests upon saving the updated test code to disk.

Note

This is a draft, the story is not implemented yet.

A very common loop of modifying the source code and re-executing the test should be accessible without subsequent user interaction with the tool.

  • run the tool once, keep it running

  • observe the execution results

  • open an editor in a separate window

  • modify the file, save the changes

  • observe the updated execution results

  • …

Prioritize latency and reuse as much as possible from the previous execution. Ideally, start the re-execution from the modified line.

Examples:

tmt run debug

Status: idea

default

plan

Plan should not be required for test execution

As a tester I want to execute tests in the default environment without having to define a plan.

Even if there are no Plans defined it is still possible to execute tests and custom scripts. In that case the default Plans configuration is applied and tests are executed using the shell method.

In order to discover available tests, the fmf method is used if metadata tree is detected otherwise test discovery defaults to the shell method as well.

Individual steps of the default plan can be adjusted as needed in the same way as if it was a real plans. For example use execute -h beakerlib to execute tests using the beakerlib method.

Examples:

/default/plan:
    summary:
        Default plan when fmf metadata found
    discover:
        how: fmf
    execute:
        how: tmt

/default/plan:
    summary:
        Default plan when no fmf metadata around
    discover:
        how: shell
    execute:
        how: tmt

Status: implemented, verified and documented

run

Default should cover the most common use case

As a user I want to easily and safely execute all available tests in the default environment.

Run all relevant tests in a local virtual machine so that the test environment can be safely prepared and adjusted as needed without affecting or possibly breaking user environment such as laptop.

Examples:

tmt run

Status: implemented and documented

filter

plan

Select plans for execution

Examples:

tmt run plan --name=NAME

tmt run plan --filter=FILTER

tmt run plan --condition=CONDITION

Status: implemented, verified and documented

test

Select tests for execution

Examples:

tmt run test --name=NAME

tmt run test --filter=FILTER

tmt run test --condition=CONDITION

tmt run plan --name=NAME2 test --name=NAME1

Status: implemented, verified and documented

keep

Store test step status, keep machines running

Examples:

tmt run --id ID provision prepare

tmt run --id ID discover execute

tmt run --id ID execute

tmt run --id ID execute

Status: implemented and documented

  • Implemented by /tmp/base.py

  • Documented by Examples

last

As a user I want to rerun tests easily

Execute previous run without the need to specify the previous run id. The command is a shortcut for:

tmt run --id PREVIOUS_RUN

Note that tmt saves last run id on each new execution.

Examples:

tmt run -l

tmt run --last

Status: implemented and documented

login

Easily login into a provisioned guest

As a user I want to log into the provisioned guest so that I can adjust the environment before the test is run or investigate what happened after the test is finished.

experiment

Provision an environment for experimenting

As a user I want to easily provision a clean and safe environment for experimenting and remove it when done.

Examples:

tmt run provision login
tmt run --last finish

Status: implemented, verified and documented

last

Log in at the end of the last enabled step

As a user I want to log into the provisioned guest to investigate once the test execution has finished.

Examples:

tmt run --until execute login

Status: implemented, verified and documented

order

Log in at the selected phase of a step

As a user I want to select the exact phase during the step execution to log into the guest.

Examples:

tmt run login --step prepare:75

Status: implemented, verified and documented

select

Log in at the end of the selected step

As a user I want to finish the guest preparation manually at the end of the prepare step.

Examples:

tmt run login --step prepare
tmt run login --step prepare:end
tmt run login --step prepare:90

Status: implemented, verified and documented

start

Log in at the start of the selected step

As a user I want to manually enable extra package repositories before the prepare step is run.

Examples:

tmt run login --step prepare:start
tmt run login --step prepare:10

Status: implemented, verified and documented

status

Log in when a test finished with given status

As a user I want to quickly investigate what exactly happened on the guest if any test failed.

Examples:

tmt run login --step execute --when fail
tmt run login --step execute --when error
tmt run login --step execute --when fail --when error

Status: implemented, verified and documented

select

Select multiple steps to be executed

all

Run all test steps, customize some

Examples:

tmt run --all provision --how=container

Status: implemented, verified and documented

pick

Choose steps to be executed

Examples:

tmt run provision prepare

tmt run discover provision prepare

Status: implemented, verified and documented

since

Run given step and all following steps.

Examples:

tmt run --since prepare

tmt run --since execute

Status: implemented, verified and documented

skip

Skip given step during execution

Examples:

tmt run --skip prepare

tmt run --skip finish

Status: implemented, verified and documented

until

Run given step and all preceding steps.

Examples:

tmt run --until prepare

tmt run --until execute

Status: implemented, verified and documented

shortcuts

Provide shortcuts for common scenarios

container

Note

This is a draft, the story is not implemented yet.

Examples:

tmt run --container=fedora:rawhide

tmt run --container=fedora:rawhide --cap-add=SYS_ADMIN

Status: idea

mock

Note

This is a draft, the story is not implemented yet.

Examples:

tmt --mock fedora-31-x86_64

tmt --mock fedora-31-x86_64 --no-clean --enable-network

tmt --mock fedora-31-x86_64 --enablerepo=updates-testing

Status: idea

smoke

As a developer I want to do a quick smoke test of my freshly built rpm package.

Examples:

tmt run --all \
prepare -h install -p tmt-0.4-1.fc29.noarch.rpm \
execute -h tmt --script 'tmt --help'

Status: implemented

steps

allowed-how

Allow only a subset of possible step plugins to be used

I wish to limit tmt to only a subset of plugins available for a particular step. This can be achieved by manipulating installed plugins, moving them around to different directories, but that would be impractical.

Instead, every step subcommand offers --allowed-how option which limits methods allowed to be used in the step. The option accepts a regular expression that allowed methods - names set with how key or --how option - must match to be allowed.

By default, the pattern is set to .*, i.e. all plugins and all methods are allowed.

Note

Regular expression accepted by the --allowed-how option is applied in the “match” mode, i.e. it must match from the start of the string.

Added in version 1.31.

Examples:

# Allow only provision plugins using containers.
tmt run -a provision --allowed-how 'container'

# Allow only provision plugins that don't require external services.
tmt run -a provision --allowed-how 'container|local|virtual'

# Allow all provision plugins except one, virtual.libvirt. Note that
# other plugins of a "virtual" family remain allowed, e.g. "virtual.testcloud"
# would be accepted.
tmt run -a provision --allowed-how '^(?!.*virtual\\.libvirt).*$'

Status: implemented

discover

Select or adjust the discover step

Defines which tests should be executed.

Examples:

tmt run discover

tmt run discover --how=fmf

tmt run discover --how=fmf --url=url

tmt run discover --how=shell

Status: implemented

execute

Select or adjust the execute step

Specification of the executor which should run tests.

Examples:

tmt run execute

tmt run execute --how=tmt

tmt run execute --how=restraint

exit-first

Stop execution after a test fails

As a user I want to avoid waiting for all discovered tests to finish if one of them fails.

To interrupt test execution after a test fails use the --exit-first (or -x) option. This option is accepted by both the execute step and the plugins implementing the execute step.

Examples:

tmt run -a execute --exit-first

tmt run -a execute -h tmt -x

Status: implemented and verified

interactive

Interactive test debugging

As a user I want to interactively debug tests in the middle of their execution.

To debug a test use the --interactive option which disables output capturing and allows to interact directly with the test from the terminal. For example, for tests written in shell you can insert a bash command in the middle of the script and investigate. Supported by the tmt executor only.

Examples:

tmt run --all execute --how tmt --interactive

Status: implemented and documented

progress

Watch test execution progress

As a user I want to watch live test execution including the complete test output.

In order to see progress of the testing use the --verbose or -v switch. Applying the option multiple times increases verbosity. This is supported by the tmt executor only.

Examples:

tmt run -v

tmt run -vv

tmt run -vvv

tmt run --all execute -vvv

Status: implemented

finish

Select or adjust the finish step

Additional actions to be performed after the test execution has been completed. Counterpart of the prepare step useful for various cleanup actions.

Examples:

tmt run finish

Status: implemented

prepare

Select or adjust the prepare step

Additional configuration of the provisioned environment needed for testing.

Examples:

tmt run prepare

tmt run prepare --how=install

tmt run prepare --how=install --package=fresh.rpm

tmt run prepare --how=ansible

tmt run prepare --how=ansible --playbook=server.yaml

Status: implemented

provision

Select or adjust the provision step

Describes what environment is needed for testing and how it should provisioned.

Examples:

tmt run provision

tmt run provision --how=virtual

tmt run provision --how=testing-farm --memory=2048MB

tmt run provision --how=container

tmt run provision --how=container --image=fedora:rawhide

tmt run provision --how=local

connect

Connect to a provisioned box

Do not provision a new system. Instead, use provided authentication data to connect to a running machine.

Examples:

tmt run provision --how=connect --guest=name-or-ip --user=login --password=secret

tmt run provision --how=connect --guest=name-or-ip --key=private-key-path

Status: implemented and documented

container

Provision a container

Examples:

tmt run provision --how=container --image=fedora:latest

Status: implemented and documented

default

Use default config or provision a virtual machine

Examples:

tmt run provision

Status: implemented and documented

local

Use localhost for testing

Examples:

tmt run provision --how=local

Status: implemented, verified and documented

virtual

Provision a virtual machine

Examples:

tmt run provision --how=virtual --image=fedora/31-cloud-base

Status: implemented and documented

report

Select or adjust the report step

Adjusting notifications about the test progress and results.

Examples:

tmt run report

tmt run report --how=html

tmt run report --how=html --open

Status: implemented

story

As a developer I want to comfortably work with stories

coverage

Show story coverage

As a developer I want to easily view which stories have been implemented, are covered by tests and documented and which still need work to be done.

code

View code implementation status

Examples:

tmt story coverage -c

tmt story coverage --code

Status: implemented and documented

docs

View documentation status

Examples:

tmt story coverage -d

tmt story coverage --docs

Status: implemented and documented

overview

General coverage overview

Complete overview of the story coverage including implementation, testing and documentation status.

Examples:

tmt story coverage

Status: implemented and documented

test

View test coverage status

Examples:

tmt story coverage -t

tmt story coverage --test

Status: implemented and documented

create

As a developer I want to easily create new story.

Provide an easy way how to create a new story. Several templates should be provided to choose from.

Examples:

tmt story create /stories/area/story

tmt story create /stories/area/story --template mini

tmt story create /stories/area/story --template mini --force

Status: implemented and documented

export

I need to convert stories into a beautiful format

html

Export stories into html

Note

This is a draft, the story is not implemented yet.

So that they can be quickly reviewed in a web browser. Optionally a simple python httpd server could be started to serve the page or pages.

Examples:

tmt story export --html --server

Status: idea

rst

Export stories into reStructuredText

So that they can be included in tmt documentation.

Examples:

tmt story export --rst

Status: implemented

filter

Search available stories

Search stories using a regular expression, a filter or coverage status. Use . to select stories under the current directory.

Examples:

tmt story ls .

tmt story ls REGEXP

tmt story ls --implemented

tmt story ls --unimplemented

tmt story ls --verified

tmt story ls --unverified

tmt story ls --documented

tmt story ls --undocumented

Status: implemented and documented

ls

List available stories

Examples:

tmt story ls

Status: implemented, verified and documented

show

Show story details

Examples:

tmt story show

Status: implemented and documented

test

As a user I want to comfortably work with tests

create

Create a new test skeleton

Provide an easy way how to create a new test. Ideally with support for multiple test skeletons. Similarly as beaker-wizard (which should be obsoleted by this) the tool should prompt for required values if not provided on the command line.

Examples:

tmt test create

tmt test create /tests/area/feature

tmt test create /tests/area/feature --skeleton beakerlib

tmt test create /tests/area/feature --duration=1h

Status: implemented and documented

export

As a tester I want to export test metadata.

format

I want to export test metadata into given format.

Examples:

tmt test export --how yaml

Status: implemented

nitrate

I want to export test metadata into nitrate.

In order to keep metadata in sync with the old test case management system we need to export selected set of attributes back to nitrate. The full fmf object identifier should be added to the structured field under the key fmf. A warning should be added to the structured field informing users that the test metadata are now maintained in git.

Below is the list of supported fmf attributes and corresponding nitrate fields:

  • component — components tab

  • contact — default tester

  • description — purpose-file in the structured field

  • duration — estimated time

  • enabled — status

  • environment — arguments

  • path — not synced

  • result — not synced

  • summary — description in the structured field

  • tag — tags tab

  • test — not synced

  • tier — tags (e.g. 1 synced to the Tier1 tag)

The following attributes, if present, should be exported as well:

  • extra-summary — test case summary

  • extra-hardware — hardware in the structured field

  • extra-pepa — pepa in the structured field

They have the extra prefix as they are not part of the L1 Metadata Specification and are supposed to be synced temporarily to keep backward compatibility.

After a successful export, special tag fmf-export should be added to the nitrate test case in order to allow easy search for migrated test cases directly from the web interface.

Examples:

tmt test export --how nitrate

Status: implemented and documented

polarion

I want to export test metadata into polarion.

In order to synchronize test metadata to the Polarion test case management system set up the Pylero config file ~/.pylero with a valid user token:

[webservice]
url=https://{your polarion web URL}/polarion
svn_repo=https://{your polarion web URL}/repo
default_project={your project name}
token={your personal access token}

See the Pylero Documentation for more details on how to configure the pylero module.

Examples:

tmt test export --how polarion

Status: implemented and verified

filter

Filter available tests

Search tests using a regular expression or a filter. Use . to select tests under the current directory.

Examples:

tmt test ls .

tmt test ls REGEXP

tmt test show --filter tier:1

tmt test show --condition 'int(tier) < 2'

tmt test show --condition '"gcc" in require'

Status: implemented, verified and documented

import

As a tester I want to import test metadata.

Examples:

tmt test import

Status: implemented, verified and documented

lint

Check test against the L1 metadata specification

Verify that test metadata are aligned with the specification, e.g. that all required attributes are present and that all attributes have correct type.

Examples:

tmt test lint

Status: implemented and documented

ls

List available tests

Examples:

tmt test ls

Status: implemented, verified and documented

show

Show test metadata

Examples:

tmt test show

Status: implemented and documented

try

As a tester or developer I want an easy and comfortable way to run tests and experiment with various versions of the operating system.

config

As a tester I usually run tests using my preferred provision method and I don’t want to write it on the command line again and again.

Define your custom user plan to be used by default when using the tmt try command and store it under the config directory ~/.config/tmt. All plans starting with /user/plan will be used for experimenting.

Note that the config directory is a tmt Tree so don’t forget to initialize it. You can copy & paste the following commands to get quickly started:

mkdir -p ~/.config/tmt
cd ~/.config/tmt
tmt init
tmt plan create --template full /user/plan

Added in version 1.30.

Examples:

# ~/.config/tmt/user/plan.fmf
discover:
    how: fmf
provision:
    how: container
    image: "fedora:fresh"
prepare:
    how: install
    package: vim
execute:
    how: tmt

Status: implemented and verified

guest

As a user I want to quickly experiment with a freshly provisioned guest.

Sometimes it’s handy to be able to quickly spawn a new clean guest and start experimenting in the sand box. Run the tmt try command in a directory without tests or explicitly ask for interactive shell with --login.

Added in version 1.30.

Examples:

# Provision default distro, log into it for experimenting
# (make sure there are no tests in the current directory)
tmt try

# Provision desired distro, log into it for experimenting
# (make sure there are no tests in the current directory)
tmt try rhel-9

# Explicitly ask for login only if there are tests around
tmt try --login
tmt try -l

# Do nothing, just provision the guest and ask what to do
tmt try --ask
tmt try -a

Status: implemented and verified

option

As a user I want an easy way to run common options in ``1minutetip`` and ``tmt run``

arch

As a user I want an easy way to specify CPU architecture on the guest.

Specify CPU architecture on the guest.

Added in version 1.39.

Examples:

tmt try fedora@virtual --arch aarch64

Status: implemented

epel

As a user I want an easy way to enable epel repository

Enable EPEL repository

Added in version 1.37.

Examples:

tmt try centos-stream-9@virtual --epel

Status: implemented

install

As a user I want an easy way to install local rpm package on the guest.

Install local rpm package on the guest.

Added in version 1.37.

Examples:

tmt try fedora@virtual --install tree

Status: implemented

test

As a test developer I want an easy way to run a single test on a freshly provisioned guest.

One of the most common scenarios is to run a single test in a given environment so that the test developer can

  • verify the newly created test works as expected

  • debug failures discovered during testing and fix them

Added in version 1.30.

Examples:

# Run test stored in the current working directory using
# the default provision method and image
cd tests/core/smoke
tmt try

# By default we run in verbose level 3, choose a lower
# level if more concise output is enough for you
cd tests/core/smoke
tmt -v try
tmt -vv try

# Run it against the specified distro image
cd tests/core/smoke
tmt try fedora-37

# Run it using given provision method and image
cd tests/core/smoke
tmt try fedora:38@container

# Use custom filter to select tests instead of the current
# working directory
cd tests/core/smoke
tmt try --test feature

# Select plan from the repository instead of default
cd tests/core/smoke
tmt try --plan plan

Status: implemented and verified

usability

As a user I want to type as little as possible.

abbreviation

commands

Allow command abbreviation

Examples:

tmt story show

tmt st sh

tmt s s

Status: implemented

options

Allow option abbreviation

Note

This is a draft, the story is not implemented yet.

Examples:

tmt story ls --implemented

tmt story ls --impl

tmt story ls --i

Status: idea

completion

Support bash completion

Examples:

tmt st<tab> sh<tab>

Status: implemented