Command Line

As a user I want to have an intuitive command line interface.

common

I want to have common command line options consistenly used across all supported commands and subcommands.

debug

Print additional information for debugging

As a tmt developer I want to see as much details about what’s happening during execution so that I can easily reveal bugs in the code.

Debug level can be used to show detailed implementation steps, revealing what’s happening under the hood, so that tool developers are able to more easily find bugs in the code. Use the option multiple times to increase verbosity.

Examples:

tmt run -d
tmt run -ddd
tmt run --debug

Status: implemented, verified and documented

dry

Run in dry mode, just let me know what would be done

As a user I want to run commands in dry mode so that I can see what would happen if I run the command but no actions are performed or changes saved to disk.

Examples:

tmt run -n
tmt run --dry

Status: implemented, verified and documented

force

Force dangerous operations like overwriting files

Examples:

tmt test create -f
tmt test create --force

Status: implemented and documented

format

Provide machine readable output in given format

Examples:

tmt test ls --format json
tmt plan show --format yaml
tmt story export --format rst

Status: implemented and documented

quiet

Enable quiet mode of the command

Examples:

tmt test show -q
tmt test show --quiet

Status: implemented and documented

verbose

Enable verbose output of the command

As a user I want to see more detailed information for particular command.

Different verbose levels can be enabled by using the option several times. For example, discover step shows only number of tests by default, list of tests in verbose level one and could show some further test details in higher verbose levels.

Examples:

tmt test show -v
tmt test show -vvv
tmt test show --verbose

Status: implemented, verified and documented

config

As a user I want to store configuration so that I don’t have to always provide my preferred default options on the command line.

Configuration should be stored as an fmf metadata tree according to L2 Metadata specification. Default values from it would be used if value is not specified in the plan.

levels

Configuration levels

Note

This is a draft, the story is not implemented yet.

There are several levels of test execution data configuration.

default

global default settings (common for most instances)

detect

detect from previous steps output (e.g. distro from build)

define

allow to override value by explicit user configuration

Let’s demonstrate these on a simple example with a distribution compose (C) and the amount of memory (M).

  • CI system is configured by default to use compose C1 installed on machines with M1 GB of memory.

  • When inspecting an artifact CI can detect that for this particular build target, compose C2 is a much better choice, and memory is fine as it is.

  • User can still explicitly define in the configuration that at least M2 GB of memory is needed for successful test execution which overrides both default and detected values.

Status: idea

system

System wide configuration

Note

This is a draft, the story is not implemented yet.

Global default values common for all users on the system should be stored under the /etc/tmt folder. It is overriden by user configuration if present.

Examples:

provision:
    how: container

Status: idea

user

User configuration

Note

This is a draft, the story is not implemented yet.

Configuration for individual users should be stored under the ~/.config/tmt folder. Overrides global system configuration.

Examples:

provision:
    image: fedora:33
report:
    how: irc
    channel: #tmt

Status: idea

init

Initialize a new metadata tree

As a user I want to start using tmt in my git repository.

base

Create a tree with some examples

Examples:

tmt init --template base

Status: implemented and verified

empty

Create an empty metadata tree

Examples:

tmt init

Status: implemented and verified

full

Create a tree with full examples

Examples:

tmt init --template full

Status: implemented and verified

mini

Create a tree with simple examples

Examples:

tmt init --template mini

Status: implemented and verified

plan

As a user I want to comfortably work with plans

create

As a developer I want to easily enable CI

Provide a super-easy and user-friendly way how to enable tests in the CI. Several templates should be supported to cover common use cases.

Examples:

tmt plan create /plans/smoke --template=mini
tmt plan create /plans/features --template=full

Status: implemented, verified and documented

filter

Filter available plans

Search plans using a regular expression or a filter. Use . to select plans under the current directory.

Examples:

tmt plan ls .
tmt plan ls REGEXP
tmt plan show --filter artifact:build

Status: implemented, verified and documented

lint

Check plan against the L2 metadata specification

Verify that plan metadata are aligned with the specification, e.g. that all required attributes are present and that all attributes have correct type.

Examples:

tmt plan lint

Status: implemented and verified

ls

List available plans

Examples:

tmt plan ls

Status: implemented, verified and documented

show

Show plan configuration

Examples:

tmt plan show

Status: implemented, verified and documented

run

As a user I want to execute tests easily

debug

Handsfree debugging

As a test developer I want to automatically execute tests upon saving the updated test code to disk.

Note

This is a draft, the story is not implemented yet.

A very common loop of modifying the source code and re-executing the test should be accessible without subsequent user interaction with the tool.

  • run the tool once, keep it running

  • observe the execution results

  • open an editor in a separate window

  • modify the file, save the changes

  • observe the updated execution results

Prioritize latency and reuse as much as possible from the previous execution. Ideally, start the re-execution from the modified line.

Examples:

tmt run debug

Status: idea

default

plan

Plan should not be required for test execution

As a tester I want to execute tests in the default environment without having to define a plan.

Even if there are no Plans defined it is still possible to execute tests and custom scripts. In that case the default Plans configuration is applied and tests are executed using the shell method.

In order to discover available tests, the fmf method is used if metadata tree is detected otherwise test discovery defaults to the shell method as well.

Individual steps of the default plan can be adjusted as needed in the same way as if it was a real plans. For example use execute -h beakerlib to execute tests using the beakerlib method.

Examples:

/plans/default:
    summary:
        Default plan when fmf metadata found
    discover:
        how: fmf
    execute:
        how: shell

/plans/default:
    summary:
        Default plan when no fmf metadata around
    discover:
        how: shell
    execute:
        how: shell

Status: implemented, verified and documented

run

Default should cover the most common use case

As a user I want to easily and safely execute all available tests in the default environment.

Run all relevant tests in a local virtual machine so that the test environment can be safely prepared and adjusted as needed without affecting or possibly breaking user environment such as laptop.

Examples:

tmt run

Status: implemented and documented

filter

plan

Select plans for execution

Examples:

tmt run plan --name=NAME
tmt run plan --filter=FILTER
tmt run plan --condition=CONDITION

Status: implemented, verified and documented

test

Select tests for execution

Examples:

tmt run test --name=NAME
tmt run test --filter=FILTER
tmt run test --condition=CONDITION
tmt run plan --name=NAME2 test --name=NAME1

Status: implemented, verified and documented

interactive

Provide way similar to git rebase –interactive

Note

This is a draft, the story is not implemented yet.

Provide users with list of steps and let them edit them. Allow adding commands between, or at least interactive “pause”. When user would be finished with single step, just continue recipe. Do not force users to remember all steps when working with them. Make it possible to repeat single step or abort current run. Allow repeating some steps just by repeating lines with task.

Examples:

tmt run --all --interactive
tmt run --continue
tmt run --abort

Status: idea

keep

Store test step status, keep machines running

Examples:

tmt run --id ID provision prepare
tmt run --id ID discover execute
tmt run --id ID execute
tmt run --id ID execute

Status: implemented and documented

last

As a user I want to rerun tests easily

Execute previous run without the need to specify the previous run id. The command is a shortcut for:

tmt run --id PREVIOUS_RUN

Note that tmt saves last run id on each new execution.

Examples:

tmt run -l
tmt run --last

Status: implemented and documented

login

Easily login into a provisioned guest

As a user I want to log into the provisioned guest so that I can adjust the environment before the test is run or investigate what happened after the test is finished.

experiment

Provision an environment for experimenting

As a user I want to easily provision a clean and safe environment for experimenting and remove it when done.

Examples:

tmt run provision login
tmt run --last finish

Status: implemented, verified and documented

last

Log in at the end of the last enabled step

As a user I want to log into the provisioned guest to investigate once the test execution has finished.

Examples:

tmt run --until execute login

Status: implemented, verified and documented

order

Log in at the selected phase of a step

As a user I want to select the exact phase during the step execution to log into the guest.

Examples:

tmt run login --step prepare:75

Status: implemented, verified and documented

select

Log in at the end of the selected step

As a user I want to finish the guest preparation manually at the end of the prepare step.

Examples:

tmt run login --step prepare
tmt run login --step prepare:end
tmt run login --step prepare:90

Status: implemented, verified and documented

start

Log in at the start of the selected step

As a user I want to manually enable extra package repositories before the prepare step is run.

Examples:

tmt run login --step prepare:start
tmt run login --step prepare:10

Status: implemented, verified and documented

status

Log in when a test finished with given status

As a user I want to quickly investigate what exactly happened on the guest if any test failed.

Examples:

tmt run login --step execute --when fail
tmt run login --step execute --when error
tmt run login --step execute --when fail --when error

Status: implemented, verified and documented

restraint

As a tester I want the option to execute tests using the Restraint harness.

Note

This is a draft, the story is not implemented yet.

Restraint is the default harness in beaker for RHEL8 and beyond. In order to provide compatibility with beaker style tests, I would like a way to invoke tmt using the Restaint harness. This would enable Restraint tests to be invoked by tmt without modification. Some common commands include rstrnt-reboot, rstrnt-abort, and rstrnt-report-result.

Examples:

tmt run --all execute --how restraint

Status: idea

select

Select multiple steps to be executed

all

Run all test steps, customize some

Examples:

tmt run --all provision --how=container

Status: implemented, verified and documented

pick

Choose steps to be executed

Examples:

tmt run provision prepare
tmt run discover provision prepare

Status: implemented, verified and documented

since

Run all steps since the given one

Examples:

tmt run --since prepare
tmt run --since execute

Status: implemented, verified and documented

skip

Skip given step during execution

Examples:

tmt run --skip prepare
tmt run --skip finish

Status: implemented, verified and documented

until

Run all steps until the given one

Examples:

tmt run --until prepare
tmt run --until execute

Status: implemented, verified and documented

shortcuts

Provide shortcuts for common scenarios

container

Note

This is a draft, the story is not implemented yet.

Examples:

tmt run --container=fedora:rawhide
tmt run --container=fedora:rawhide --cap-add=SYS_ADMIN

Status: idea

mock

Note

This is a draft, the story is not implemented yet.

Examples:

tmt --mock fedora-31-x86_64
tmt --mock fedora-31-x86_64 --no-clean --enable-network
tmt --mock fedora-31-x86_64 --enablerepo=updates-testing

Status: idea

smoke

As a developer I want to do a quick smoke test of my freshly built rpm package.

Examples:

tmt run --all \
prepare -h install -p tmt-0.4-1.fc29.noarch.rpm \
execute -h tmt --script 'tmt --help'

Status: implemented

upgrade

As a tester I want to verify that a configured application or service still correctly works after the system upgrade.

Note

This is a draft, the story is not implemented yet.

In order to enable developing tests for upgrade testing we need to provide a way how to execute these tests easily. This does not cover unit tests for individual actors but rather system tests which verify the whole upgrade story.

The plan is to provide an environment variable available to the test denoting current phase of the upgrade testing (before or after the upgrade). Based on this variable test can perform appropriate actions.

before

setup, test

after

test, cleanup

without

setup, test, cleanup

Combining test step selection feature and defining extra environment variables together with an external framework to perform the actual system upgrade should give us a nice way how to achieve the desired functionality. In addition, we will also need to implement a way how to aggregate test results from both runs (before and after the upgrade).

Examples:

tmt run --id foo discover provision --image=rhel7 --output=box.json
tmt run --id foo execute --environment='UPGRADE=before'
upgrade-the-system box.json
tmt run --id foo execute --environment='UPGRADE=after'
tmt run --id foo report finish

Status: idea

steps

discover

Select or adjust the discover step

Defines which tests should be executed.

Examples:

tmt run discover
tmt run discover --how=fmf
tmt run discover --how=fmf --url=url
tmt run discover --how=shell

Status: implemented

execute

Select or adjust the execute step

Specification of the executor which should run tests.

Examples:

tmt run execute
tmt run execute --how=tmt
tmt run execute --how=detach
tmt run execute --how=restraint

exit-first

Stop execution after a test fails

As a user I want to avoid waiting for all discovered tests to finish if one of them fails.

To interrupt test execution after a test fails use the --exit-first (or -x) option. This option is accepted by both the execute step and the plugins implementing the execute step. Currently, both tmt and detach executors support this.

Examples:

tmt run -a execute --exit-first
tmt run -a execute -h tmt -x
tmt run -a execute -h detach -x

Status: implemented and verified

interactive

Interactive test debugging

As a user I want to interactively debug tests in the middle of their execution.

To debug a test use the --interactive option which disables output capturing and allows to interact directly with the test from the terminal. For example, for tests written in shell you can insert a bash command in the middle of the script and investigate. Supported by the tmt executor only.

Examples:

tmt run --all execute --how tmt --interactive

Status: implemented and documented

progress

Watch test execution progress

As a user I want to watch live test execution including the complete test output.

In order to see progress of the testing use the --verbose or -v switch. Applying the option multiple times increases verbosity. This is supported by the tmt executor only.

Examples:

tmt run -v
tmt run -vv
tmt run -vvv
tmt run --all execute -vvv

Status: implemented

finish

Select or adjust the finish step

Additional actions to be performed after the test execution has been completed. Counterpart of the prepare step useful for various cleanup actions.

Examples:

tmt run finish

Status: implemented

prepare

Select or adjust the prepare step

Additional configuration of the provisioned environment needed for testing.

Examples:

tmt run prepare
tmt run prepare --how=install
tmt run prepare --how=install --package=fresh.rpm
tmt run prepare --how=ansible
tmt run prepare --how=ansible --playbook=server.yaml

Status: implemented

provision

Select or adjust the provision step

Describes what environment is needed for testing and how it should provisioned.

Examples:

tmt run provision
tmt run provision --how=virtual
tmt run provision --how=testing-farm --memory=2048MB
tmt run provision --how=container
tmt run provision --how=container --image=fedora:rawhide
tmt run provision --how=local

connect

Connect to a provisioned box

Do not provision a new system. Instead, use provided authentication data to connect to a running machine.

Examples:

tmt run provision --how=connect --guest=name-or-ip --user=login --password=secret
tmt run provision --how=connect --guest=name-or-ip --key=private-key-path

Status: implemented and documented

container

Provision a container

Examples:

tmt run provision --how=container --image=fedora:latest

Status: implemented and documented

default

Use default config or provision a virtual machine

Examples:

tmt run provision

Status: implemented and documented

local

Use localhost for testing

Examples:

tmt run provision --how=local

Status: implemented, verified and documented

virtual

Provision a virtual machine

Examples:

tmt run provision --how=virtual --image=fedora/31-cloud-base

Status: implemented and documented

report

Select or adjust the report step

Adjusting notifications about the test progress and results.

Examples:

tmt run report
tmt run report --how=html
tmt run report --how=html --open

Status: implemented

story

As a developer I want to comfortably work with stories

coverage

Show story coverage

As a developer I want to easily view which stories have been implemented, are covered by tests and documented and which still need work to be done.

code

View code implementation status

Examples:

tmt story coverage -c
tmt story coverage --code

Status: implemented and documented

docs

View documentation status

Examples:

tmt story coverage -d
tmt story coverage --docs

Status: implemented and documented

overview

General coverage overview

Complete overview of the story coverage including implementation, testing and documentation status.

Examples:

tmt story coverage

Status: implemented and documented

test

View test coverage status

Examples:

tmt story coverage -t
tmt story coverage --test

Status: implemented and documented

create

As a developer I want to easily create new story.

Provide an easy way how to create a new story. Several templates should be provided to choose from.

Examples:

tmt story create /stories/area/story
tmt story create /stories/area/story --template mini
tmt story create /stories/area/story --template mini --force

Status: implemented and documented

export

I need to convert stories into a beautiful format

html

Export stories into html

Note

This is a draft, the story is not implemented yet.

So that they can be quickly reviewed in a web browser. Optionally a simple python httpd server could be started to serve the page or pages.

Examples:

tmt story export --html --server

Status: idea

rst

Export stories into reStructuredText

So that they can be included in tmt documentation.

Examples:

tmt story export --rst

Status: implemented

filter

Search available stories

Search stories using a regular expression, a filter or coverage status. Use . to select stories under the current directory.

Examples:

tmt story ls .
tmt story ls REGEXP
tmt story ls --implemented
tmt story ls --unimplemented
tmt story ls --verified
tmt story ls --unverified
tmt story ls --documented
tmt story ls --undocumented

Status: implemented and documented

ls

List available stories

Examples:

tmt story ls

Status: implemented, verified and documented

show

Show story details

Examples:

tmt story show

Status: implemented and documented

test

As a user I want to comfortably work with tests

create

Create a new test skeleton

Provide an easy way how to create a new test. Ideally with support for multiple test skeletons. Similarly as beaker-wizard (which should be obsoleted by this) the tool should prompt for required values if not provided on the command line.

Examples:

tmt test create
tmt test create /tests/area/feature
tmt test create /tests/area/feature --skeleton beakerlib
tmt test create /tests/area/feature --duration=1h

Status: implemented and documented

export

As a tester I want to export test metadata.

format

I want to export test metadata into given format.

Examples:

tmt test export --format yaml

Status: implemented

nitrate

I want to export test metadata into nitrate.

In order to keep metadata in sync with the old test case management system we need to export selected set of attributes back to nitrate. The full fmf object identifier should be added to the structured field under the key fmf. A warning should be added to the structured field informing users that the test metadata are now maintained in git.

Below is the list of supported fmf attributes and corresponding nitrate fields:

  • component — components tab

  • contact — default tester

  • description — purpose-file in the structured field

  • duration — estimated time

  • enabled — status

  • environment — arguments

  • path — not synced

  • result — not synced

  • summary — description in the structured field

  • tag — tags tab

  • test — not synced

  • tier — tags (e.g. 1 synced to the Tier1 tag)

The following attributes, if present, should be exported as well:

  • extra-summary — test case summary

  • extra-hardware — hardware in the structured field

  • extra-pepa — pepa in the structured field

They have the extra prefix as they are not part of the L1 Metadata Specification and are supposed to be synced temporarily to keep backward compatibility.

After a successful export, special tag fmf-export should be added to the nitrate test case in order to allow easy search for migrated test cases directly from the web interface.

Examples:

tmt test export --nitrate

Status: implemented and documented

filter

Filter available tests

Search tests using a regular expression or a filter. Use . to select tests under the current directory.

Examples:

tmt test ls .
tmt test ls REGEXP
tmt test show --filter tier:1
tmt test show --condition 'int(tier) < 2'
tmt test show --condition '"gcc" in require'

Status: implemented, verified and documented

import

As a tester I want to import test metadata.

Examples:

tmt test import

Status: implemented, verified and documented

lint

Check test against the L1 metadata specification

Verify that test metadata are aligned with the specification, e.g. that all required attributes are present and that all attributes have correct type.

Examples:

tmt test lint

Status: implemented and documented

ls

List available tests

Examples:

tmt test ls

Status: implemented, verified and documented

show

Show test metadata

Examples:

tmt test show

Status: implemented and documented

usability

As a user I want to type as little as possible.

abbreviation

commands

Allow command abbreviation

Examples:

tmt story show
tmt st sh
tmt s s

Status: implemented

options

Allow option abbreviation

Note

This is a draft, the story is not implemented yet.

Examples:

tmt story ls --implemented
tmt story ls --impl
tmt story ls --i

Status: idea

completion

Support bash completion

Examples:

tmt st<tabl> sh<tab>

Status: implemented