Signature Classifications

Classification is an optional feature to group non-functional failing Signatures.

Fail Signatures, by default, are classified as fail (or func), meaning the signature is a generic functional fail.

Why use classifications?

Classifications are used for grouping and filtering non-functional fail signatures.

• They can have any name, but these choices are set by the administrator.

If you want to make a search query to either include or exclude a batch of signatures, this is the purpose of classifications.

• Classifications also allow users to exclude signatures from pass/fail scores.

Example classification: infra

For example, you can have an infra classification, where a job failed due to an infrastructure problem:

• Not enough disk space
• Out of memory
• License checkout fail
• Network error
• Missing/Lost jobs

Example classification: timeout

Another example is a timeout classification, where a job failed due to:

• Cycle timeout
• Wall clock (queue job) timeout

Example classification: assertion

Another example is an assertion classification, where a test failed due to assertion violation.

Example classification: perf

Another example is a perf classification, to group performance errors, where a testcase functionally passed, but was flagged as a soft fail, due to not meeting performance criteria.

Example classification: end of test

Another example is an EOT (end-of-test) classification, to group non-functional errors where a test functionally passed, but failed due to end-of-test check.

Example classification: tb

Another example is distinguishing a signature as an RTL vs Testbench (tb) error.

Example classification: skip

Another example is a skip classification, where a job was skipped due to missing dependency or unwritten test.

Example Scenario

As an example, in following Regression:

• Signature #818 has been classified as Timeout.

→ This now shows two Pass Rates for the regression:

Using classifications

If signatures are classified, this gives you two additional capabilities:

1. When looking at regression details, there is a Raw pass rate vs Net pass rate. The Raw pass rate includes classified fails, while the Net pass rate excludes classified fails. For example, you might have a 95% Net pass rate, vs a 90% Raw pass rate (which includes timeouts).
2. Filter out groups of signatures from charts and signatures. For example, you might want to chart timeouts over time.

Classifying Signatutres

Note: only Managers and Administrators can change Signature classifications.

To classify a signature (ie to change a signature classification):

• Click the dropdown box next to a signature.
• Select the new classification.

Automatic classifications

Signatures can be automatically classified during job import.

See Job JSON Fields for more details.

Unclassifying

Signatures can be unclassified by clicking the dropdown and selecting (unclassify). This will reset the signature back to a fail classification (or whatever your default classification is set to).

Search: filtering-out by Classification

To filter-out a group of classifications, use the ! prefix.

For example, this search expression is a regex which will exclude all timeout and infra classifications from your search.

!timeout|infra

Another common classification filter is just func, which means only show functional errors.

admin: Classification Configuration

To configure signature classification choices, set them in your simscope.config file:

[server]

# List of signature classification choices
classifications = "timeout"
classifications = "infra"
classifications = "perf"
classifications = "assert"
classifications = "eot"

# The default classification is "fail", but can be changed to something like "func"
defaultclassification = "func"

# Optional: Dynamic Classification mode allows setting a signature classification to any
# string value during job import (as opposed to being limited to the pre-defined list
# of classifications).
#classificationsdynamic = true

Recommended classifications

You can set the list of classifications to anything your organization prefers.

Here are a list of recommended signature classifications:

Dynamic Classifications

The Dynamic Classification mode allows setting a signature classification to any string value during job import (as opposed to being limited to the pre-defined list of classifications).

You can enable mode by setting classificationsdynamic = true in the simscope.config file.

Classification Storage (Tuple)

Classifications are described by a three-part-tuple:

• Signature ID
• Component
• Branch

This means each signature can be classified by users in multiple ways (for different components or branches).