:tocdepth: 0 .. _/plugins/report: Report Plugins ~~~~~~~~~~~~~~ .. _plugins/report/display: display ------- .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptional. Show test results on the terminal. Give a concise summary of test results directly on the terminal. List individual test results in verbose mode. Configuration ^^^^^^^^^^^^^ display-guest: ``auto|always|never`` When to display full guest name in report: when more than a single guest was involved (default), always, or never. Default: ``auto`` Common Keys """"""""""" name: The name of the step phase. order: Order in which the phase should be handled. Default: ``50`` summary: Concise summary describing purpose of the phase. Default: *not set* ---- .. _plugins/report/html: html ---- .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptional. Format test results into an HTML report. Example config: .. code-block:: yaml report: how: html open: true Configuration ^^^^^^^^^^^^^ absolute-paths: ``true|false`` Make paths absolute rather than relative to working directory. Default: ``false`` display-guest: ``auto|always|never`` When to display full guest name in report: when more than a single guest was involved (default), always, or never. Default: ``auto`` open: ``true|false`` Open results in your preferred web browser. Default: ``false`` Common Keys """"""""""" name: The name of the step phase. order: Order in which the phase should be handled. Default: ``50`` summary: Concise summary describing purpose of the phase. Default: *not set* ---- .. _plugins/report/junit: junit ----- .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptional. Save test results in JUnit format. When ``file`` is not specified, output is written into a file named ``junit.xml`` located in the current workdir. Configuration ^^^^^^^^^^^^^ file: ``PATH`` Path to the file to store JUnit to. Default: *not set* Common Keys """"""""""" name: The name of the step phase. order: Order in which the phase should be handled. Default: ``50`` summary: Concise summary describing purpose of the phase. Default: *not set* ---- .. _plugins/report/polarion: polarion -------- .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptional. Write test results into an xUnit file and upload to Polarion. Configuration ^^^^^^^^^^^^^ arch: ``ARCH`` Which architecture was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_ARCH. Default: *not set* assignee: ``ASSIGNEE`` Who is responsible for this test run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_ASSIGNEE. Default: *not set* build: ``BUILD`` Which build was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_BUILD. Default: *not set* compose-id: ``COMPOSEID`` Compose ID of image used for this run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_COMPOSE_ID. Default: *not set* description: ``DESCRIPTION`` Use specific test run description, also uses environment variable TMT_PLUGIN_REPORT_POLARION_DESCRIPTION. Default: *not set* file: ``FILE`` Path to the file to store xUnit in. Default: *not set* fips: ``true|false`` FIPS mode enabled or disabled for this run. Default: ``false`` logs: ``LOGLOCATION`` Location of the logs for this test run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_LOGS. Ultimately also uses environment variable TMT_REPORT_ARTIFACTS_URL. Default: *not set* planned-in: ``PLANNEDIN`` Select a specific release to mark this test run with, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PLANNED_IN. Default: *not set* platform: ``PLATFORM`` Which platform was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PLATFORM. Default: *not set* pool-team: ``POOLTEAM`` Which subsystem is this test run relevant for, also uses environment variable TMT_PLUGIN_REPORT_POLARION_POOL_TEAM. Default: *not set* project-id: ``ID`` Use specific Polarion project ID, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PROJECT_ID. Default: *not set* sample-image: ``SAMPLEIMAGE`` Which sample image was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_SAMPLE_IMAGE. Default: *not set* template: ``TEMPLATE`` Use specific test run template, also uses environment variable TMT_PLUGIN_REPORT_POLARION_TEMPLATE. Default: *not set* title: ``TITLE`` Use specific test run title, also uses environment variable TMT_PLUGIN_REPORT_POLARION_TITLE. Default: *not set* upload: ``true|false`` Whether to upload results to Polarion. Default: ``true`` use-facts: ``true|false`` Use hostname and arch from guest facts. Default: ``false`` Common Keys """"""""""" name: The name of the step phase. order: Order in which the phase should be handled. Default: ``50`` summary: Concise summary describing purpose of the phase. Default: *not set* ---- .. _plugins/report/reportportal: reportportal ------------ .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptional. Report test results to a ReportPortal instance via API. For communication with Report Portal API is necessary to provide following options: * token for authentication * url of the ReportPortal instance * project name In addition to command line options it's possible to use environment variables: .. code-block:: bash export TMT_PLUGIN_REPORT_REPORTPORTAL_${MY_OPTION}=${MY_VALUE} Assuming the URL and token are provided by the environment variables, the plan config can look like this: .. code-block:: yaml report: how: reportportal project: baseosqe context: ... environment: ... Where the context and environment sections must be filled with corresponding data in order to report context as attributes (arch, component, distro, trigger, compose, etc.) and environment variables as parameters in the Item Details. Other reported fmf data are summary, id, web link and contact per test. There are supported two ways of mapping plans into ReportPortal * launch-per-plan (default) with reported structure 'launch > test', resulting in one or more launches. * suite-per-plan with reported structure 'launch > suite > test' resulting in one launch only, and one or more suites within. It is recommended to define launch name and launch description in addition. Configuration ^^^^^^^^^^^^^ api-version: ``VERSION`` Override the default reportportal API version (v1). Default: ``v1`` artifacts-url: ``ARTIFACTS_URL`` Link to test artifacts provided for report plugins. Default: *not set* defect-type: ``DEFECT_NAME`` Pass the defect type to be used for failed test, which is defined in the project (e.g. 'Idle'). 'To Investigate' is used by default. Default: *not set* exclude-variables: ``PATTERN`` Regular expression for excluding environment variables from reporting to ReportPortal ('^TMT_.*' used by default). Parameters in ReportPortal get filtered out by the pattern to prevent overloading and to preserve the history aggregation for ReportPortal item if tmt id is not provided. Default: ``^TMT_.*`` launch: ``LAUNCH_NAME`` Set the launch name, otherwise name of the plan is used by default. Should be defined with suite-per-plan option or it will be named after the first plan. Default: *not set* launch-description: ``DESCRIPTION`` Pass the description for ReportPortal launch with '--suite-per-plan' option or append the original (plan summary) with additional info. Appends test description with upload-to-launch/suite options. Default: *not set* launch-per-plan: ``true|false`` Mapping launch per plan, creating one or more launches with no suite structure. Default: ``false`` launch-rerun: ``true|false`` Rerun the last launch based on its name and unique test paths to create Retry item with a new version per each test. Supported in 'suite-per-plan' structure only. Default: ``false`` launch-url: Default: *not set* launch-uuid: Default: *not set* project: ``PROJECT_NAME`` Name of the project into which the results should be uploaded. Default: *not set* suite-per-plan: ``true|false`` Mapping suite per plan, creating one launch and continuous uploading suites into it. Recommended to use with '--launch' and '--launch-description' options. Can be used with '--upload-to-launch' option to avoid creating a new launch. Default: ``false`` suite-uuid: Default: *not set* test-uuids: Default: *not set* token: ``TOKEN`` The token to use for upload to the ReportPortal instance (from the user profile). Default: *not set* upload-to-launch: ``LAUNCH_ID`` Pass the launch ID for an additional test/suite upload to an existing launch. ID can be found in the launch URL. To upload specific info into description see also launch-description. Default: *not set* upload-to-suite: ``LAUNCH_SUITE`` Pass the suite ID for an additional test upload to a suite within an existing launch. ID can be found in the suite URL. To upload specific info into description see also launch-description. Default: *not set* url: ``URL`` The URL of the ReportPortal instance where the data should be sent to. Default: *not set* Common Keys """"""""""" name: The name of the step phase. order: Order in which the phase should be handled. Default: ``50`` summary: Concise summary describing purpose of the phase. Default: *not set*