Report Plugins
display
Warning
Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originaly used for command line help only, therefore the formatting is often suboptional.
Show test results on the terminal.
Give a concise summary of test results directly on the terminal. List individual test results in verbose mode.
Configuration
- display-guest:
auto|always|never When to display full guest name in report: when more than a single guest was involved (default), always, or never.
Default:
auto
Common Keys
- name:
The name of the step phase.
- order:
Order in which the phase should be handled.
Default:
50- summary:
Concise summary describing purpose of the phase.
Default: not set
html
Warning
Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originaly used for command line help only, therefore the formatting is often suboptional.
Format test results into an HTML report.
Example config:
report:
how: html
open: true
Configuration
- absolute-paths:
true|false Make paths absolute rather than relative to working directory.
Default:
false- display-guest:
auto|always|never When to display full guest name in report: when more than a single guest was involved (default), always, or never.
Default:
auto- open:
true|false Open results in your preferred web browser.
Default:
false
Common Keys
- name:
The name of the step phase.
- order:
Order in which the phase should be handled.
Default:
50- summary:
Concise summary describing purpose of the phase.
Default: not set
junit
Warning
Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originaly used for command line help only, therefore the formatting is often suboptional.
Save test results in JUnit format.
When file is not specified, output is written into a file
named junit.xml located in the current workdir.
Configuration
- file:
PATH Path to the file to store JUnit to.
Default: not set
Common Keys
- name:
The name of the step phase.
- order:
Order in which the phase should be handled.
Default:
50- summary:
Concise summary describing purpose of the phase.
Default: not set
polarion
Warning
Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originaly used for command line help only, therefore the formatting is often suboptional.
Write test results into an xUnit file and upload to Polarion.
Configuration
- arch:
ARCH Which architecture was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_ARCH.
Default: not set
- assignee:
ASSIGNEE Who is responsible for this test run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_ASSIGNEE.
Default: not set
- build:
BUILD Which build was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_BUILD.
Default: not set
- compose-id:
COMPOSEID Compose ID of image used for this run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_COMPOSE_ID.
Default: not set
- description:
DESCRIPTION Use specific test run description, also uses environment variable TMT_PLUGIN_REPORT_POLARION_DESCRIPTION.
Default: not set
- file:
FILE Path to the file to store xUnit in.
Default: not set
- fips:
true|false FIPS mode enabled or disabled for this run.
Default:
false- logs:
LOGLOCATION Location of the logs for this test run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_LOGS. Ultimately also uses environment variable TMT_REPORT_ARTIFACTS_URL.
Default: not set
- planned-in:
PLANNEDIN Select a specific release to mark this test run with, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PLANNED_IN.
Default: not set
- platform:
PLATFORM Which platform was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PLATFORM.
Default: not set
- pool-team:
POOLTEAM Which subsystem is this test run relevant for, also uses environment variable TMT_PLUGIN_REPORT_POLARION_POOL_TEAM.
Default: not set
- project-id:
ID Use specific Polarion project ID, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PROJECT_ID.
Default: not set
- sample-image:
SAMPLEIMAGE Which sample image was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_SAMPLE_IMAGE.
Default: not set
- template:
TEMPLATE Use specific test run template, also uses environment variable TMT_PLUGIN_REPORT_POLARION_TEMPLATE.
Default: not set
- title:
TITLE Use specific test run title, also uses environment variable TMT_PLUGIN_REPORT_POLARION_TITLE.
Default: not set
- upload:
true|false Whether to upload results to Polarion.
Default:
true- use-facts:
true|false Use hostname and arch from guest facts.
Default:
false
Common Keys
- name:
The name of the step phase.
- order:
Order in which the phase should be handled.
Default:
50- summary:
Concise summary describing purpose of the phase.
Default: not set
reportportal
Warning
Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originaly used for command line help only, therefore the formatting is often suboptional.
Report test results to a ReportPortal instance via API.
For communication with Report Portal API is neccessary to provide following options:
token for authentication
url of the ReportPortal instance
project name
In addition to command line options it’s possible to use environment variables:
export TMT_PLUGIN_REPORT_REPORTPORTAL_${MY_OPTION}=${MY_VALUE}
Assuming the URL and token are provided by the environment variables, the plan config can look like this:
report:
how: reportportal
project: baseosqe
context:
...
environment:
...
Where the context and environment sections must be filled with corresponding data in order to report context as attributes (arch, component, distro, trigger, compose, etc.) and environment variables as parameters in the Item Details.
Other reported fmf data are summary, id, web link and contact per test.
There are supported two ways of mapping plans into ReportPortal
launch-per-plan (default) with reported structure ‘launch > test’, resulting in one or more launches.
suite-per-plan with reported structure ‘launch > suite > test’ resulting in one launch only, and one or more suites within. It is recommended to define launch name and launch description in addition.
Configuration
- api-version:
VERSION Override the default reportportal API version (v1).
Default:
v1- artifacts-url:
ARTIFACTS_URL Link to test artifacts provided for report plugins.
Default: not set
- defect-type:
DEFECT_NAME Pass the defect type to be used for failed test, which is defined in the project (e.g. ‘Idle’). ‘To Investigate’ is used by default.
Default: not set
- exclude-variables:
PATTERN Regular expression for excluding environment variables from reporting to ReportPortal (‘^TMT_.*’ used by default). Parameters in ReportPortal get filtered out by the pattern to prevent overloading and to preserve the history aggregation for ReportPortal item if tmt id is not provided.
Default:
^TMT_.*- launch:
LAUNCH_NAME Set the launch name, otherwise name of the plan is used by default. Should be defined with suite-per-plan option or it will be named after the first plan.
Default: not set
- launch-description:
DESCRIPTION Pass the description for ReportPortal launch with ‘–suite-per-plan’ option or append the original (plan summary) with additional info. Appends test description with upload-to-launch/suite options.
Default: not set
- launch-per-plan:
true|false Mapping launch per plan, creating one or more launches with no suite structure.
Default:
false- launch-rerun:
true|false Rerun the last launch based on its name and unique test paths to create Retry item with a new version per each test. Supported in ‘suite-per-plan’ structure only.
Default:
false
launch-url:
Default: not set
launch-uuid:
Default: not set
- project:
PROJECT_NAME Name of the project into which the results should be uploaded.
Default: not set
- suite-per-plan:
true|false Mapping suite per plan, creating one launch and continous uploading suites into it. Recommended to use with ‘–launch’ and ‘–launch-description’ options. Can be used with ‘–upload-to-launch’ option to avoid creating a new launch.
Default:
false
suite-uuid:
Default: not set
test-uuids:
Default: not set
- token:
TOKEN The token to use for upload to the ReportPortal instance (from the user profile).
Default: not set
- upload-to-launch:
LAUNCH_ID Pass the launch ID for an additional test/suite upload to an existing launch. ID can be found in the launch URL. To upload specific info into description see also launch-description.
Default: not set
- upload-to-suite:
LAUNCH_SUITE Pass the suite ID for an additional test upload to a suite within an existing launch. ID can be found in the suite URL. To upload specific info into description see also launch-description.
Default: not set
- url:
URL The URL of the ReportPortal instance where the data should be sent to.
Default: not set
Common Keys
- name:
The name of the step phase.
- order:
Order in which the phase should be handled.
Default:
50- summary:
Concise summary describing purpose of the phase.
Default: not set