Live Jobs

Live Jobs are an optional feature in Simscope, which enable tracking of in-progress (ie running) jobs, by publishing Live Job JSON objects.

This can be a pending simulation, build, etc.

For example, a Live Job can indicate a simulation has started on a specific hostname, with a configuration.

Then this job can either turn into a pass or a fail, once it completes.

Purpose / Use Cases

The main usage of Live jobs is to answer the questions:

Which jobs are remaining in this Regression?
Which host machine is my build running on?
How long has my test been running?

→ Live jobs will show a list of the remaining jobs within each regression automatically.

Example Live Jobs within a Regression

This example shows a Regression with 1 Live job:

If you click on a Live Job in the table, it will show its details.

JSON: Live Job Indicator

Live jobs are indicated by "result": "live" field.

See Job Field Reference for definitions of each field.

NOTE: Live jobs should not have a fail message, compute time, runtime, or cycle count (since the job has not finished).

After the job finishes, you can publish a Job Finish JSON (either pass or fail), and Simscope will: . Remove the Live JSON record. . Replace with a Finished JSON record.

Admin: Enable Live Job tracking in Simscope server

By default, Simscope does not enable live job tracking. To enable in the Simscope server, add the following to your simscope.config file:

[sigs]
# Enable "live" job tracking in Simscope
# > https://admin-docs.simscope.com/job/job-live-json.html
enablelivejobs = true

# If Live Jobs are enabled and regressions time out or finish with live jobs pending, migrate them to this fail signature.
# (Note: this also requires enablelivejobs = true)
livetimeoutsignature = "LIVE Job Timeout or Missing"

Integration: Example Live JSON

Note: Live jobs should not contain compute_ms or runtime_ms fields (since the job is still running).

Required Job Fields (Job key)

The following fields cannot change between a live job and a finish job:

start_time
category
jobdir
seed
config

These fields are used to determine the job key (unique identifier), so they must remain the same in their live and finish messages.

Example Live JSON

This Job is an example of a live simulation.

{
    "start_time": "2021-05-07T02:00:11.123-05:00",
    "jobdir": "/home/runs/myregr/2/job-3",
    "category": "sim",
    "config": "sim +rand +tbuf +seed=123",
    "seed": "123",
    "build": "mybuild",
    "testgroup": "mytestgroup",
    "result": "live"
}

NOTE: you can optionally have other fields in the live JSON, like "host".

Publishing Live Jobs

To publish, use a job-live message:

# NOTE: replace [myrun/123] with your regression name and [job.json] with your live job JSON file
> simscope-tunnel --config=simscope-tunnel.conf --publish job-live --regression=myrun/123 job.json

Notes on Live Jobs

A few notes (limitations) on Live jobs:

Live jobs do not capture statistics into their parent regression: cycles, runtime, etc.
- When Live jobs finish, the statistics will be computed into the regression.
- Also, Live jobs cannot be charted (until they finish).
Live jobs are not fails, so they can not be assigned, nor can they have Rules.
Simscope allows both live and normal regressions simultaneously.
When transitioning from a Live job to a Finished job, the JSON input fields must match. Simscope uses the input fields from the finished job to find and delete the live job automatically. If the inputs change, Simscope will not find the input job.
You can publish multiple live updates on a single job. For example, you can publish metadata or other status fields as the job is running.
Unfinished live jobs automatically get pruned from the database after 7 days.

Example Message flow for a full Regression containing Live Jobs

Here is an example flow for publishing one Regression containing five Jobs, with Live Job tracking.

→ This requires 12 JSON messages:

1 regr-start
5 job-live
5 job-finish
1 regr-finish

JSON Message Summary:

 1. Publish regr-start      — Regression will show as LIVE
    2. Publish job-live    — Job #1 started
    3. Publish job-live    — Job #2 started
    4. Publish job-live    — Job #3 started
    5. Publish job-live    — Job #4 started
    6. Publish job-live    — Job #5 started

    7. Publish job-finish  — after Job #1 finishes
    8. Publish job-finish  — after Job #2 finishes
    9. Publish job-finish  — after Job #3 finishes
   10. Publish job-finish  — after Job #4 finishes
   11. Publish job-finish  — after Job #5 finishes
12. Publish regr-finish    — Regression will show as complete

Regression Live Job Timeout (automatic kill jobs)

If livetimeoutsignature is enabled in simscope.config, and a regression either hits timeout or finishes with pending Live Jobs, these jobs will be migrated intto a fail signature automatically.

This is equivalent to automatically killing the pending jobs.
Note: Tank Regression is a manual mechanism to kill or hide a regression.

Here is an example regression that timed out, with livetimeoutsignature enabled (note this shows up as an Aborted regression):

Live Job Timeout

Simscope Administration Guide