Job Metrics Database

Note: Simscope also supports Regression-level Metadata (ie shared across the entire regression). This feature does not require any additional configuration.

Intro

→ Note: Click here for Metrics User documentation and examples on the Knowledge Base.

Simscope has a more powerful database mechanism for Job-level metrics, which allows charting of Job Metadata in:

• Daily charts
• Weekly charts
• Regression scatter plot charts

For example, this chart is a 30-day chart of Average Job gc.heap_objects, across regressions:

Job Metadata types supported

Note that string metadata indexing is useful if you want to view results grouped by a string metadata.

• For example, you can plot #Cycles grouped by CPU_instruction

(optional) Job Metadata Units

If you have unit-based Job Metadata (e.g. KB, MB, etc), you can specify these in your simscope.config file, and charts will scale automatically.

For example, this chart shows disk_usage, in KB (and data is scaled to MB automatically).

Base Units supported

Here are the scientific integer base units supported:

Note: as long as your unit is prefixed by the unit, you can specify any arbitrary unit name. Examples:

• For kilo-hertz (ie frequency), specify the unit as: KHz
• For mega-cycles, specify as: Mcycles
• For plain cycles, specify as cycles

Also note that sub-scale units are not supported. For example, milli-gram (ie divide by 1000) is not supported.

Download/Install

To download/install, choose a data directory and then run:

> bin/metrics-db/install-db.sh INSTALLDIR

# Example:
> bin/metrics-db/install-db.sh /project/metrics-db

Startup

Currently, you must run the database interactively.

• Also, you should run the database on the same machine that the Simscope server is running on.

From a terminal, run:

> INSTALLDIR/bin/start-metrics-db.sh

Note: if you close this terminal, the DB will close too.

Stopping

You can Ctrl-C the terminal to stop the DB process.

Simscope configuration

To enable metrics in Simscope, edit your simscope.config file and add the following:

# Metrics Database
[metrics]
dsn="tcp://localhost:9000"

# Turn on debugging
#dsn="tcp://localhost:9000?debug=true"

# Connect with a specific username/password
#dsn="tcp://localhost:9000?username=abc&password=xyz123"

# List of job metadata fields to be indexed with the metrics database
# Note: this is currently limited to 10 or less metadata fields
metadata = "fail_func"
metadata = "num_cgo_call"
metadata = "total_webtests"
metadata = "gc.num"

Then restart Simscope to activate the database connection.

Manual Table Reindexing

Currently this performs a daily automatic re-index, to push job metadata.

• Note you need to manually re-index after enabling in Simscope the first time (or wait 24 hours).

To manually reindex, click Admin → Repopulate metrics table

• Depending on database size, this should take a few minutes (up to a few hours) to repopulate.

Progress will show up on Simscope's command-line:

[INFO ] progress 11.6% [5m30s remaining]

Disabling Metrics Database

If you have problems starting Simscope due to Metrics DB not connecting, you can disable it temporarily.

[ERROR] Simscope Error: metrics DB connect failed: dial tcp [::1]:9000: connect: connection refused

To disable, comment out the dsn line in your simscope.config file, and then start (or restart) Simscope:

[metrics]
# dsn="tcp://localhost:9000" # Metrics DB disabled

Limitations

• Currently, this limits the number of job metadata indexed to a maximum of 20 fields.
  • This will be increased in future Simscope versions.
• This requires manual reindexing, to get job updates.
  • This will be automated in a future release.

Future features/ideas

Note: please contact VerOps to request DB/chart feature additions.

• Per-job scatter plot charts
• Per-job result table
• MIN/MAX charts?
  • Currently Average() and Sum() are supported
• Custom query expressions: e.g. Cycles/Instruction (CPI) or Instructions/Cycle (IPC)
• Filtering based on job metadata: (ie filter where instr=add)

DB sizing/usage

TBD