Regression Compare (Diff)

Two or more Regressions can be compared side-by-side, showing the differences between:

• Branch/Commit/Model
• Signatures
• Jobs
• Statistics
  • Pass Rates
  • Compute Time
  • etc.

Example Regression Compare

This example shows comparison between two Regressions.

• Large cell values are highlighted in yellow.
• The Delta column shows percentage changes between the runs (e.g. runtime, pass, fail, etc.).
• This also compares Regression Metadata.
  • If you click the table header, it will sort by largest metadata value change.

Job Diff between Regressions

If you click the Diff Jobs button in the blue bar, this will show differences between the jobs/tests within the regressions.

This compares each individual job, matching them by the config field. Note that if the seeds change, this view will not work, because all jobs will be unique across each regression.

Here is an example Job Diff, showing a job that changed from Pass to Fail across 2 regressions:

Missing / Lost Jobs

If you have Aborted regressions, the Regression Job Diff can also be used to find Missing Jobs, where one regression has lost jobs.

• Simscope will render it as MISSING.

To use this, check one good regression and one bad regression, then Diff them.

Here is an example of missing jobs between regressions:

Job Compute Diff

If you click the Compare dropdown, you can switch to a comparison between compute time, to find performance changes between regressions.

• This highlights differences where the job results are the same, but their time to run has changed by more than a specific percentage of compute time (e.g. 10% slower in a different regression).