Simscope Release History — v330 — 339

1.339

1. `bugfix` Signature Unclassify

Simscope now supports Signature unclassify.

Note: only managers or administrators can perform this.

This enables removing a Signature classification back to the default value, so that if a future job import occurs, Simscope can automatically re-classify a signature.

In previous Simscope versions, once a Signature has been classified, it can only be manually re-classified.

2. `feature` Regression text report assignments

The regression text report now shows signature assignments (user, state, bug number), for all signatures within the regression that have assignments.

All Simscope table queries limits have been relaxed to at least 1 million records. Some testgroup queries in previous Simscope versions would stop at 50k records (to prevent runaway queries), but this could give users incorrect results.

Also, if queries are terminated early, Simscope will print a warning similar to the following:

[WARN ] Query stopped table=regrmatrix record-limit=1000000

Note: If you see these warnings for common user queries, we may be able to relax the query limits further.

2. `bugfix` Invalid job metadata detection

SIMSCOPE-16

Simscope now validates custom Job metadata keys during server startup.

For example, if you add a metadata key called my transactions to your simscope.config file, this will be rejected, as the key has a space character (you should use a dash, dot, or underscore instead).

3. `bugfix` RabbitMQ bugfix during low disk space

In previous versions, if the Simscope DB directory ran low on space, incoming RabbitMQ jobs would be rejected and discarded.

Now if there is low disk space, the messages will be rejected and re-queued, so that when disk space returns, the jobs can be imported.

1.337

1. `feature` Regression text report improvements

The text report now shows the first line of each job's test config, rather than clipping to 60 characters.
The category/classification summary table at the top is now hyperlinked, so it can be clicked to drill into each section
Fixed line wrapping bug
Renders skip jobs now

2. `bugfix` Prevent regression metric charts with weekly (short term) X axis

In previous versions, you could create a coverage chart with Weekly (short term) X-axis selected.

This will now show users an error message: you need to use a daily or regression scatter plot selection instead.

3. `feature` Coverage DB heatmap

The Coverage DB now shows a heatmap display of the current cover groups.

These can be clicked on to drill into them.

4. `bugfix` Signature Saved search with testgroup

Saved Signature searches with testgroup/build fields are now loaded properly, so you can drill into signature links.

In prior versions, the testgroup search filter would be lost when clicking into a signature details.

1.336

1. `bugfix` Regression text report changes

The Regression text report has been updated:

Copy/paste now works (for example copying the report into an email)
Signatures are now grouped by classification and sorted by number of hits
Changed the timestamp format to HH:MM:SS, so that long jobs (multi-hour) are easier to view the runtime
Added Regression metadata fields to the report

2. `feature` Added start/finish time to Job JSON

SIMSCOPE-14

For debugging purposes, the Simscope Job JSON output now renders start_time and finish_time fields.

3. `bugfix` Tunnel componentize fix

The componentize feature has been corrected to work in the Tunnel.

4. `feature` Simscope RabbitMQ Watchdog

Simscope can now automatically send an email if no RabbitMQ activity occurs within a specific amount of time.

This notifies the administrator something is possibly wrong and needs attention.

To enable, add the following to your simscope.config file:

[rabbitmq]
# Rabbit Health is a watchdog timer, to indicate basic health of the Rabbit connection.
# If there are no Rabbit messages received within 'health' duration,
# Simscope will show a warning, indicating a potential RabbitMQ problem.
# Production Simscope systems should set this to a lower value (ie 60m),
# whereas development Simscope systems may set it higher (ie 12h).
health = "30m"

# Send a notification email if RabbitMQ receives no data within the 'health' interval
healthnotify = true

→ see RabbitMQ docs.

Example notification email

5. `feature` Coverage DB average hits

The Coverage DB now renders average numbers of hits as a column, for each coverage group.

1.335

1. `feature` Regression Text Report

Simscope now can render a text-based regression summary report, by clicking the Text Report button from within a regression.

This text report shows on a single page:

Regression info (branch, model version, command, time, etc.)
Regression statistics grouped by Build
Regression statistics grouped by Test Group
Signatures grouped by Classification
Shortest jobs from each Signature, including their configuration and seed

Example report:

To view the report, click the Text Report link:

2. `feature` Coverage DB location field

The Coverpoint table now renders a Location column.

3. `feature` Coverage post via Tunnel

The simscope-publish-coverage script has now been updated to publish Coverage through the Tunnel using a regr-update RabbitMQ message.

This enables users to publish coverage without needing an API token.

NOTE: you need to edit the simscope-publish-coverage script and point it at your Tunnel executable/config.

1.334

`bugfix` Chart Y-axis rounding bug fixed

In some cases, charts would incorrectly round Y-axis values, leading to ambiguous charts.

For example, if the Y-axis had raw values of 1000, 1500 and 2000, the chart's Y-axis would render as 1k, 2k, and 2k.

This issue is now fixed, so the same chart renders as 1k, 1.5k and 2k.

1.333

1. `feature` Regression Table can be configured to show metadata columns

To enable, add one or more metadata columns to your simscope.config file.

Here is an example with 2 metadata columns:

[regr]
# Show additional regression metadata columns
tablemeta = "P4 Changelist"
tablemeta = ".Coverage %"

2. `bugfix` Charts — Group by Build/Test Group

If using Job Tag mode and grouping by either Build or Test Group, previous versions would incorrectly render each tag inside the chart.

For example, a cycles chart group by Build, would show duplicate data, as each tag was accumulated individually into the chart.

This bug is now fixed.

3. `feature` New Charts — Runtime Days/Weeks

There are two additional charts available: Runtime Days and Runtime Weeks.

Note: runtime is job wallclock time, as opposed to compute, which is job CPU time.

4. `feature` Charts — changed default chart from weekly to 30-days

When building charts, the default selection has been changed from a 52-week chart to a 30-day chart.

Also, when enabling weekly charts, the default has changed to a 26-week (ie half-year) chart.

5. `feature` Coverage DB updates

Two coverage DB updates:

By default, to show a more compact DB tree, Coverage will only show direct ancestors (ie children) by default. This can be overridden with
Coverage points now render on the right side of the page, for convenience.

1.332

When filtering tests by a group of unique regressions, the link now allows additional filtering without losing the regression list.

`bugfix` Signature search — 'user' regex drilldown bugfix

When searching signatures by username, using regular expressions, the drilldown link into signature details works now.

For example: the Signature search for users:

john|sally

will properly drilldown into Signatures with:

user=~john|sally

Natural Sort: Coverage points are sorted using Natural Sort, rather than string sorting, so that numeric indexed points are sorted in a natural order. For example, Natural Sort orders t.a.b[10] after t.a.b[9], whereas common string sorting would order in reverse order.
Hit/Miss: Coverpoint hit/miss now show up as red-light/green-light (instead of 0% and 100%)

`bugfix` Regression Cycle display fix

SIMSCOPE-13

This version fixes a display problem where average-cycles in regression summary clips the HTML.

Now, Simscope only displays min/max cycles for regression summary.

`bugfix` Gantt ignore 'skip' fixes

The Gantt chart now hides skipped jobs from display, as these were not run.

`feature` New Net Pass Rate chart

Simscope Metrics now has a Net Pass Rate chart, which renders pass rates including skip results.

For example, if a Regression has:

80 pass
15 skip
5 fail

Simscope can render two different charts:

Pass Rate     = 94.1% (ie 80/85)
Net Pass Rate = 80.0% (ie 80/100)

1.330

`feature` New 'skip' job type

This release adds an optional skip job result, in addition to the existing pass and fail categories.

→ see the result field in Job JSON

Example skip job scenarios:

If a build failed, and 200 sim jobs were skipped from running, you can send the list of these jobs to Simscope, for users to track.
If you are tracking test plans, you can send skip jobs for tests which have not yet been written.

Skip jobs allow tracking of statistics related to number of jobs skipped within each regression.

You can also filter to find which tests have been skipped.
You can also chart skips.

When importing skipped jobs, Simscope will still calculate normal pass rate:

score = pass / (pass + fail)

But will also have a net score that can calculated net score, including skips:

net_score = pass / (pass + fail + skip)

Example regression with a single skip job:

Example skipped job:

Enabling skips in Simscope

To enable skip job tracking in Simscope, add the following to your simscope.config file:

[sigs]
# Store "skip" job results in Simscope.
# NOTE: enabling skip job tracking reduces the limit of job categories to 5 (instead of 9).
enableskip = true

Simscope Administration Guide