Features¶
Test Coverage¶
As a tester I want to document the test coverage, so that it can be easily tracked which user stories are covered by which test case and vice-versa.
filter¶
repo¶
Filtering based on fmf identifiers
As a tester I want to filter based on reference identifiers, so that I can learn where the tests are implemented/maintained.
Note
This is a draft, the story is not implemented yet.
There should be an easy way to filter based on fmf identifiers.
Examples:
tmt story ls --filter "url:https://gitlab/project"
tmt story ls --filter "url:https://gitlab/project & ref:main name:/tests/smoke"
Status: idea
reference¶
Referencing remote repositories
As a tester I want to reference not only local objects, so that user stories can be covered by test cases from a different repository.
Note
This is a draft, the story is not implemented yet.
It is possible to use a full fmf identifier in the link attribute.
Examples:
verified-by:
url: https://github.com/teemtee/fmf
ref: main
name: /tests/basic/repo
Status: idea
relevancy¶
Story relevancy
As a product owner I want to connect user stories with products or versions, so that I can filter out items relevant (or not relevant) for a particular product.
Note
This is a draft, the story is not implemented yet.
Relevancy is an easy way to bind stories to particular product(s). The same relevancy rules as for tests should be also used for stories.
Examples:
story:
As an admin I want product A doing B, so that I can
use it for C.
adjust:
- enabled: False
when: distro < fedora-28
Status: idea
Reboot during test¶
As a tester I need to reboot the guest during a test and then resume the test execution.
Some tests may require a reboot as a part of them, e.g.
upgrading the system, rebooting and then running some checks.
The tmt-reboot
command can be used to request the guest
reboot from the test and the TMT_REBOOT_COUNT
environment
variable provides number of successfully completed reboots.
Note that this only works with guests provisioned by tmt, e.g.
container
or virtual
, and doesn’t work with the
local
provision method. Support of the connect
provision is on a best effort basis and will not work on
machines where reboot
command is not available.
When a custom command for rebooting is required, the -c
option of the reboot script can be used, e.g. tmt-reboot -c
"dnf system-upgrade reboot"
.
For backward-compatibility with the restraint framework
the rstrnt-reboot
and rhts-reboot
commands are provided
as well together with the RSTRNT_REBOOTCOUNT
and REBOOTCOUNT
environment variables. Calling the script kills the parent process
(the running test). Please note that the content of these
scripts is not preserved, tmt
overwrites them.
Examples:
if [ "$TMT_REBOOT_COUNT" -eq 0 ]; then
rlRun "echo 'Before the reboot'"
rlRun "tmt-reboot" 0 "Reboot the machine"
elif [ "$TMT_REBOOT_COUNT" -eq 1 ]; then
rlRun "echo 'After the reboot'"
fi
Status: implemented and verified
Package Preparation¶
Preparing packages for testing
As a user, I want a flexible and comfortable way to prepare packages on the guest for testing.
all¶
Install all packages from a given source
As a developer I would like to install all packages from a given source.
Note
This is a draft, the story is not implemented yet.
In case of execution from CI, I want all packages from a given Koji build to be installed, even if they have same or lower NVR. In case of COPR repository, I want all packages from a given repository to be installed, in the latest version available in the given repository. In case of already existing repository, I want to install all packages from the given repository.
I don’t have conflicting subpackages in my repository, or I’m excluding packages explicitly to solve those conflicts. All installation conflicts abort the installation and fail the run. If no packages are found the install fails as well. In case of CI, I want to be able override default behaviour, and install all packages instead.
Examples:
/plan/install-all-copr:
prepare:
- how: install-all
copr: psss/tmt
/plan/install-all-repository:
prepare:
- how: install-all
repository: my-enabled-repo
/plan/install-all-ci:
prepare:
- how: install-all
tmt run -a prepare -h install-all -D my_packages/
Status: idea
minimal¶
Install a minimal set of freshly built required packages
As a developer who has just built fresh rpms in given local directory I want to prepare on the guest only the minimal set of required packages.
Note
This is a draft, the story is not implemented yet.
After building new rpm packages locally there should be an easy way to install on the guest only those packages which are actually required by tests or are explicitly requested in the prepare step.
Examples:
make rpm
tmt run test --name smoke \
prepare --how install --directory fresh/rpms
Status: idea
repository¶
Add artifact repository for installation
As a CI system I want to be able to add a repository with high priority from which all subsequent packages will be installed, updated, downgraded or reinstalled.
Note
This is a draft, the story is not implemented yet.
In case of Fedora CI we want the packages to be from the artifact repository which the CI system provides. Due to support of dist-git pull requests we need to support cases when the artifact repository contains packages with lower or the same version, which is quite common in case of scratch builds.
Examples:
tmt run -a prepare --how repository --order 1 \
--base-url file:///repository --priority 999 \
plan --name some-plan
Status: idea
required¶
Install a minimal set of required packages
As a tester who uses VM or container with relevant repositories enabled and whose packages are not part of the default installation I want to prepare the guest with the minimal set of required packages.
Only packages which are actually required by tests or are explicitly requested in the prepare step are installed. This allows to test the installation itself and also verify functionality without hidden requires.
Examples:
tmt run plan --name install/dependencies
Status: implemented
selected¶
Install specific packages
As a developer I would like to install certain packages required for testing.
Note
This is a draft, the story is not implemented yet.
In case of execution from CI I want these packages to be installed from a given Koji build, even if they have same or lower NVR. I have conflicting subpackages in my package. The specification for local execution and CI should be the same.
Examples:
/plan/curl:
prepare:
- how: install
package: curl
/plan/curl-minimal:
prepare:
- how: install
package: curl-minimal
Status: idea