Signature Classifications
Classification is an optional feature to group non-functional failing Signatures.
Fail Signatures by default are classified as fail
(or func
), meaning the signature is a
generic functional fail
Why use classifications?
Classifications can be used for grouping and filtering non-functional fails.
For example, you can have an infra
classification, where a test failed due to an
infrastructure problem:
- Not enough disk space
- Out of memory
- License checkout fail
- Network error
- Missing/Lost jobs
Another example is a timeout
classification, where a test failed due to:
- Cycle timeout
- Wall clock (queue job) timeout
Another example is an assertion
classification, where a test failed due to assertion
violation.
Another example is a perf
classification, to group performance errors, where a
testcase functionally passed, but was flagged as a soft fail, due to not meeting performance
criteria.
Another example is an EOT
(end-of-test) classification, to group
non-functional errors where a test functionally passed, but failed due to
end-of-test check.
Another example is a skip
classification, where a job was skipped due to missing dependency
or unwritten test.
Example Classification
As an example, in following Regression:
- Signature
#818
has been classified asTimeout
.
→ This now shows two Pass Rates for the regression:
Pass Rate | Value |
---|---|
Raw (including Timeout ) | 90.6% |
Net (excluding Timeout ) | 92.7% |
Using classifications
If signatures are classified, this gives you two additional capabilities:
- When looking at regression details, there is a Raw pass rate vs Net pass rate.
The Raw pass rate includes classified fails, while the Net pass rate excludes
classified fails. For example, you might have a
95%
Net pass rate, vs a90%
Raw pass rate (which includes timeouts). - Filter out groups of signatures from charts and signatures. For example,
you might want to chart
timeout
s over time.
Changing Classifications
Note: only Managers and Administrators can change Signature classifications.
To change a signature classification:
- Click the dropdown box next to a signature.
- Select the new classification.
Automatic classifications
Signatures can be automatically classified during job import.
See Job JSON Fields for more details.
Unclassifying
Signatures can be unclassified by clicking the dropdown and selecting (unclassify)
.
This will reset the signature back to a fail
classification (or whatever your default
classification is set to).
Search: filtering-out by Classification
To filter-out a group of classifications, use the !
prefix.
For example, this search expression:
!timeout|infra
Is a regex which will exclude all timeout
and infra
classifications from your search.
Classification Configuration: simscope.config
To configure signature classification choices, set them in your
simscope.config
file:
[server]
classifications = "dv"
classifications = "timeout"
classifications = "infra"
classifications = "perf"
classifications = "assert"
# The default classification is "fail", but can be changed to something like "func"
#defaultclassification = "func"
Recommended classifications
You can set the list of classifications to anything your organization prefers.
Here are a list of recommended signature classifications:
Classification | Example Signatures |
---|---|
timeout | Cycle time, wall clock timeout, etc |
infra | Infrastructure error: disk space, license error, etc |
perf | Failure due to not meeting performance requirement (ie too many cycles per transaction) |
assert | Failure due to a design assertion firing |
Dynamic Classifications
The Dynamic Classification mode allows setting a signature classification to any string value during job import (as opposed to being limited to the pre-defined list of classifications).
You can enable mode by setting classificationsdynamic = true
in the
simscope.config
file.