:tocdepth: 0 .. _/plugins/report: Report Plugins ~~~~~~~~~~~~~~ .. include:: report-header.inc.rst .. _/plugins/report/common-keys: Common Keys ----------- The following keys are accepted by all plugins of the ``report`` step. name: The name of the step phase. order: Order in which the phase should be handled. Default: ``50`` summary: Concise summary describing purpose of the phase. Default: *not set* .. _/plugins/report/display: display ------- .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal. Show test results on the terminal. Give a concise summary of test results directly on the terminal. List individual test results in verbose mode. Configuration ^^^^^^^^^^^^^ See also :ref:`Common Keys` accepted by the plugin. display-guest: ``auto|always|never`` When to display full guest name in report: when more than a single guest was involved (default), always, or never. Default: ``auto`` ---- .. _/plugins/report/html: html ---- .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal. Format test results into an HTML report. Example config: .. code-block:: yaml report: how: html open: true Configuration ^^^^^^^^^^^^^ See also :ref:`Common Keys` accepted by the plugin. absolute-paths: ``true|false`` Make paths absolute rather than relative to working directory. Default: ``false`` display-guest: ``auto|always|never`` When to display full guest name in report: when more than a single guest was involved (default), always, or never. Default: ``auto`` open: ``true|false`` Open results in your preferred web browser. Default: ``false`` ---- .. _/plugins/report/junit: junit ----- .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal. Save test results in chosen JUnit flavor format. When flavor is set to custom, the ``template-path`` with a path to a custom template must be provided. When ``file`` is not specified, output is written into a file named ``junit.xml`` located in the current workdir. Configuration ^^^^^^^^^^^^^ See also :ref:`Common Keys` accepted by the plugin. file: ``PATH`` Path to the file to store JUnit to. Default: *not set* flavor: ``default|custom`` Name of a JUnit flavor to generate. Default: ``default`` include-output-log: ``true|false`` Include full standard output in resulting xml file. Default: ``true`` prettify: ``true|false`` Enable the XML pretty print for generated JUnit file. This option is always disabled for 'custom' template flavor. Default: ``true`` template-path: ``TEMPLATE_PATH`` Path to a custom template file to use for JUnit creation. Default: *not set* ---- .. _/plugins/report/polarion: polarion -------- .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal. Write test results into an xUnit file and upload to Polarion Configuration ^^^^^^^^^^^^^ See also :ref:`Common Keys` accepted by the plugin. arch: ``ARCH`` Which architecture was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_ARCH. Default: *not set* assignee: ``ASSIGNEE`` Who is responsible for this test run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_ASSIGNEE. Default: *not set* build: ``BUILD`` Which build was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_BUILD. Default: *not set* compose-id: ``COMPOSEID`` Compose ID of image used for this run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_COMPOSE_ID. Default: *not set* description: ``DESCRIPTION`` Use specific test run description, also uses environment variable TMT_PLUGIN_REPORT_POLARION_DESCRIPTION. Default: *not set* file: ``FILE`` Path to the file to store xUnit in. Default: *not set* fips: ``true|false`` FIPS mode enabled or disabled for this run. Default: ``false`` include-output-log: ``true|false`` Include full standard output in resulting xml file. Default: ``true`` logs: ``LOGLOCATION`` Location of the logs for this test run, also uses environment variable TMT_PLUGIN_REPORT_POLARION_LOGS. Ultimately also uses environment variable TMT_REPORT_ARTIFACTS_URL. Default: *not set* planned-in: ``PLANNEDIN`` Select a specific release to mark this test run with, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PLANNED_IN. Default: *not set* platform: ``PLATFORM`` Which platform was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PLATFORM. Default: *not set* pool-team: ``POOLTEAM`` Which subsystem is this test run relevant for, also uses environment variable TMT_PLUGIN_REPORT_POLARION_POOL_TEAM. Default: *not set* prettify: ``true|false`` Enable the XML pretty print for generated XUnit file. Default: ``true`` project-id: ``ID`` Use specific Polarion project ID, also uses environment variable TMT_PLUGIN_REPORT_POLARION_PROJECT_ID. If no project ID is found, the project ID is taken from pylero configuration default project setting as a last resort. Default: *not set* sample-image: ``SAMPLEIMAGE`` Which sample image was this run executed on, also uses environment variable TMT_PLUGIN_REPORT_POLARION_SAMPLE_IMAGE. Default: *not set* template: ``TEMPLATE`` Use specific test run template, also uses environment variable TMT_PLUGIN_REPORT_POLARION_TEMPLATE. Default: *not set* title: ``TITLE`` Use specific test run title, also uses environment variable TMT_PLUGIN_REPORT_POLARION_TITLE. Default: *not set* upload: ``true|false`` Whether to upload results to Polarion, also uses environment variable TMT_PLUGIN_REPORT_POLARION_UPLOAD. Default: ``true`` use-facts: ``true|false`` Use hostname and arch from guest facts, also uses environment variable TMT_PLUGIN_REPORT_POLARION_USE_FACTS. Default: ``false`` ---- .. _/plugins/report/reportportal: reportportal ------------ .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal. Report test results to a ReportPortal instance via API. For communication with Report Portal API is necessary to provide following options: * token for authentication * url of the ReportPortal instance * project name In addition to command line options it's possible to use environment variables: .. code-block:: bash export TMT_PLUGIN_REPORT_REPORTPORTAL_${MY_OPTION}=${MY_VALUE} Assuming the URL and token are provided by the environment variables, the plan config can look like this: .. code-block:: yaml report: how: reportportal project: baseosqe context: ... environment: ... Where the context and environment sections must be filled with corresponding data in order to report context as attributes (arch, component, distro, trigger, compose, etc.) and environment variables as parameters in the Item Details. Other reported fmf data are summary, id, web link and contact per test. Two types of data structures are supported for reporting to ReportPortal: * 'launch-per-plan' mapping (default) that results in launch-test structure. * 'suite-per-plan' mapping that results in launch-suite-test structure. Supported report use cases: * Report a new run in launch-suite-test or launch-test structure * Report an additional rerun with 'launch-rerun' option and same launch name (->Retry items) or by reusing the run and reporting with 'again' option (->append logs) * To see plan progress, discover and report an empty (IDLE) run and reuse the run for execution and updating the report with 'again' option * Report contents of a new run to an existing launch via the URL ID in three ways: tests to launch, suites to launch and tests to suite. Configuration ^^^^^^^^^^^^^ See also :ref:`Common Keys` accepted by the plugin. api-version: ``VERSION`` Override the default reportportal API version (v1). Default: ``v1`` artifacts-url: ``ARTIFACTS_URL`` Link to test artifacts provided for report plugins. Default: *not set* defect-type: ``DEFECT_NAME`` Pass the defect type to be used for failed test, which is defined in the project (e.g. 'Idle'). 'To Investigate' is used by default. Default: *not set* exclude-variables: ``PATTERN`` Regular expression for excluding environment variables from reporting to ReportPortal ('^TMT_.*' used by default). Parameters in ReportPortal get filtered out by the pattern to prevent overloading and to preserve the history aggregation for ReportPortal item if tmt id is not provided. Default: ``^TMT_.*`` launch: ``LAUNCH_NAME`` Set the launch name, otherwise name of the plan is used by default. Should be defined with 'suite-per-plan' option or it will be named after the first plan. Default: *not set* launch-description: ``DESCRIPTION`` Pass the description for ReportPortal launch with 'suite-per-plan' option or append the original (plan summary) with additional info. Appends test description with 'upload-to-launch/suite' options. Default: *not set* launch-per-plan: ``true|false`` Mapping launch per plan, creating one or more launches with no suite structure. Default: ``false`` launch-rerun: ``true|false`` Rerun the last launch based on its name and unique test paths to create Retry item with a new version per each test. Supported in 'suite-per-plan' structure only. Default: ``false`` launch-url: Default: *not set* launch-uuid: Default: *not set* project: ``PROJECT_NAME`` Name of the project into which the results should be uploaded. Default: *not set* suite-per-plan: ``true|false`` Mapping suite per plan, creating one launch and continuous uploading suites into it. Recommended to use with 'launch' and 'launch-description' options. Can be used with 'upload-to-launch' option for an additional upload of new suites. Default: ``false`` suite-uuid: Default: *not set* test-uuids: Default: *not set* token: ``TOKEN`` The token to use for upload to the ReportPortal instance (from the user profile). Default: *not set* upload-to-launch: ``LAUNCH_ID`` Pass the launch ID for an additional test/suite upload to an existing launch. ID can be found in the launch URL. Keep the launch structure with options 'launch/suite-per-plan'. To upload specific info into description see also 'launch-description'. Default: *not set* upload-to-suite: ``SUITE_ID`` Pass the suite ID for an additional test upload to a suite within an existing launch. ID can be found in the suite URL. To upload specific info into description see also 'launch-description'. Default: *not set* url: ``URL`` The URL of the ReportPortal instance where the data should be sent to. Default: *not set*