Signature Classifications

Classification is an optional feature to group non-functional failing Signatures.

Fail Signatures by default are classified as fail (or func), meaning the signature is a generic functional fail

Why use classifications?

Classifications can be used for grouping and filtering non-functional fails.

For example, you can have an infra classification, where a test failed due to an infrastructure problem:

• Not enough disk space
• Out of memory
• License checkout fail
• Network error
• Missing/Lost jobs

Another example is a timeout classification, where a test failed due to:

• Cycle timeout
• Wall clock (queue job) timeout

Another example is an assertion classification, where a test failed due to assertion violation.

Another example is a perf classification, to group performance errors, where a testcase functionally passed, but was flagged as a soft fail, due to not meeting performance criteria.

Another example is an EOT (end-of-test) classification, to group non-functional errors where a test functionally passed, but failed due to end-of-test check.

Another example is a skip classification, where a job was skipped due to missing dependency or unwritten test.

Example Classification

As an example, in following Regression:

• Signature #818 has been classified as Timeout.

→ This now shows two Pass Rates for the regression:

Using classifications

If signatures are classified, this gives you two additional capabilities:

1. When looking at regression details, there is a Raw pass rate vs Net pass rate. The Raw pass rate includes classified fails, while the Net pass rate excludes classified fails. For example, you might have a 95% Net pass rate, vs a 90% Raw pass rate (which includes timeouts).
2. Filter out groups of signatures from charts and signatures. For example, you might want to chart timeouts over time.

Changing Classifications

Note: only Managers and Administrators can change Signature classifications.

To change a signature classification:

• Click the dropdown box next to a signature.
• Select the new classification.

Automatic classifications

Signatures can be automatically classified during job import.

See Job JSON Fields for more details.

Unclassifying

Signatures can be unclassified by clicking the dropdown and selecting (unclassify). This will reset the signature back to a fail classification (or whatever your default classification is set to).

Search: filtering-out by Classification

To filter-out a group of classifications, use the ! prefix.

For example, this search expression:

!timeout|infra

Is a regex which will exclude all timeout and infra classifications from your search.

Classification Configuration: simscope.config

To configure signature classification choices, set them in your simscope.config file:

[server]
classifications = "dv"
classifications = "timeout"
classifications = "infra"
classifications = "perf"
classifications = "assert"

# The default classification is "fail", but can be changed to something like "func"

#defaultclassification = "func"

Recommended classifications

You can set the list of classifications to anything your organization prefers.

Here are a list of recommended signature classifications:

Dynamic Classifications

The Dynamic Classification mode allows setting a signature classification to any string value during job import (as opposed to being limited to the pre-defined list of classifications).

You can enable mode by setting classificationsdynamic = true in the simscope.config file.