Report Plugins
Common Keys
The following keys are accepted by all plugins of the report step.
- name:
The name of the step phase.
- order:
Order in which the phase should be handled.
Default:
50- summary:
Concise summary describing purpose of the phase.
Default: not set
display
Warning
Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptional.
Show test results on the terminal.
Give a concise summary of test results directly on the terminal. List individual test results in verbose mode.
Configuration
See also Common Keys accepted by the plugin.
- display-guest:
auto|always|never When to display full guest name in report: when more than a single guest was involved (default), always, or never.
Default:
auto
html
Warning
Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptional.
Format test results into an HTML report.
Example config:
report:
how: html
open: true
Configuration
See also Common Keys accepted by the plugin.
- absolute-paths:
true|false Make paths absolute rather than relative to working directory.
Default:
false- display-guest:
auto|always|never When to display full guest name in report: when more than a single guest was involved (default), always, or never.
Default:
auto- open:
true|false Open results in your preferred web browser.
Default:
false
junit
Warning
Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptional.
Save test results in JUnit format.
When file is not specified, output is written into a file
named junit.xml located in the current workdir.
Configuration
See also Common Keys accepted by the plugin.
- file:
PATH Path to the file to store JUnit to.
Default: not set
- include-output-log:
true|false Include full standard output in resulting xml file.
Default:
true
polarion
Warning
Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptional.
Write test results into an xUnit file and upload to Polarion
Configuration
See also Common Keys accepted by the plugin.
- arch:
ARCH Which architecture was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_ARCH.
Default: not set
- assignee:
ASSIGNEE Who is responsible for this test run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_ASSIGNEE.
Default: not set
- build:
BUILD Which build was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_BUILD.
Default: not set
- compose-id:
COMPOSEID Compose ID of image used for this run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_COMPOSE_ID.
Default: not set
- description:
DESCRIPTION Use specific test run description, also uses environment variable TMT_PLUGIN_REPORT_POLARION_DESCRIPTION.
Default: not set
- file:
FILE Path to the file to store xUnit in.
Default: not set
- fips:
true|false FIPS mode enabled or disabled for this run.
Default:
false- include-output-log:
true|false Include full standard output in resulting xml file.
Default:
true- logs:
LOGLOCATION Location of the logs for this test run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_LOGS. Ultimately also uses environment variable TMT_REPORT_ARTIFACTS_URL.
Default: not set
- planned-in:
PLANNEDIN Select a specific release to mark this test run with, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PLANNED_IN.
Default: not set
- platform:
PLATFORM Which platform was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PLATFORM.
Default: not set
- pool-team:
POOLTEAM Which subsystem is this test run relevant for, also uses environment variable TMT_PLUGIN_REPORT_POLARION_POOL_TEAM.
Default: not set
- project-id:
ID Use specific Polarion project ID, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PROJECT_ID.
Default: not set
- sample-image:
SAMPLEIMAGE Which sample image was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_SAMPLE_IMAGE.
Default: not set
- template:
TEMPLATE Use specific test run template, also uses environment variable TMT_PLUGIN_REPORT_POLARION_TEMPLATE.
Default: not set
- title:
TITLE Use specific test run title, also uses environment variable TMT_PLUGIN_REPORT_POLARION_TITLE.
Default: not set
- upload:
true|false Whether to upload results to Polarion.
Default:
true- use-facts:
true|false Use hostname and arch from guest facts.
Default:
false
reportportal
Warning
Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptional.
Report test results to a ReportPortal instance via API.
For communication with Report Portal API is necessary to provide following options:
token for authentication
url of the ReportPortal instance
project name
In addition to command line options it’s possible to use environment variables:
export TMT_PLUGIN_REPORT_REPORTPORTAL_${MY_OPTION}=${MY_VALUE}
Assuming the URL and token are provided by the environment variables, the plan config can look like this:
report:
how: reportportal
project: baseosqe
context:
...
environment:
...
Where the context and environment sections must be filled with corresponding data in order to report context as attributes (arch, component, distro, trigger, compose, etc.) and environment variables as parameters in the Item Details.
Other reported fmf data are summary, id, web link and contact per test.
Two types of data structures are supported for reporting to ReportPortal:
‘launch-per-plan’ mapping (default) that results in launch-test structure.
‘suite-per-plan’ mapping that results in launch-suite-test structure.
Supported report use cases:
Report a new run in launch-suite-test or launch-test structure
Report an additional rerun with ‘launch-rerun’ option and same launch name (->Retry items) or by reusing the run and reporting with ‘again’ option (->append logs)
To see plan progress, discover and report an empty (IDLE) run and reuse the run for execution and updating the report with ‘again’ option
Report contents of a new run to an existing launch via the URL ID in three ways: tests to launch, suites to launch and tests to suite.
Configuration
See also Common Keys accepted by the plugin.
- api-version:
VERSION Override the default reportportal API version (v1).
Default:
v1- artifacts-url:
ARTIFACTS_URL Link to test artifacts provided for report plugins.
Default: not set
- defect-type:
DEFECT_NAME Pass the defect type to be used for failed test, which is defined in the project (e.g. ‘Idle’). ‘To Investigate’ is used by default.
Default: not set
- exclude-variables:
PATTERN Regular expression for excluding environment variables from reporting to ReportPortal (‘^TMT_.*’ used by default). Parameters in ReportPortal get filtered out by the pattern to prevent overloading and to preserve the history aggregation for ReportPortal item if tmt id is not provided.
Default:
^TMT_.*- launch:
LAUNCH_NAME Set the launch name, otherwise name of the plan is used by default. Should be defined with ‘suite-per-plan’ option or it will be named after the first plan.
Default: not set
- launch-description:
DESCRIPTION Pass the description for ReportPortal launch with ‘suite-per-plan’ option or append the original (plan summary) with additional info. Appends test description with ‘upload-to-launch/suite’ options.
Default: not set
- launch-per-plan:
true|false Mapping launch per plan, creating one or more launches with no suite structure.
Default:
false- launch-rerun:
true|false Rerun the last launch based on its name and unique test paths to create Retry item with a new version per each test. Supported in ‘suite-per-plan’ structure only.
Default:
false
launch-url:
Default: not set
launch-uuid:
Default: not set
- project:
PROJECT_NAME Name of the project into which the results should be uploaded.
Default: not set
- suite-per-plan:
true|false Mapping suite per plan, creating one launch and continuous uploading suites into it. Recommended to use with ‘launch’ and ‘launch-description’ options. Can be used with ‘upload-to-launch’ option for an additional upload of new suites.
Default:
false
suite-uuid:
Default: not set
test-uuids:
Default: not set
- token:
TOKEN The token to use for upload to the ReportPortal instance (from the user profile).
Default: not set
- upload-to-launch:
LAUNCH_ID Pass the launch ID for an additional test/suite upload to an existing launch. ID can be found in the launch URL. Keep the launch structure with options ‘launch/suite-per-plan’. To upload specific info into description see also ‘launch-description’.
Default: not set
- upload-to-suite:
SUITE_ID Pass the suite ID for an additional test upload to a suite within an existing launch. ID can be found in the suite URL. To upload specific info into description see also ‘launch-description’.
Default: not set
- url:
URL The URL of the ReportPortal instance where the data should be sent to.
Default: not set