Publish

Tunnel Publish allows posting of new data into Simscope via the Tunnel (using RabbitMQ interface).

Publish Message Types

Tunnel messages formats are documented here.

JSON types supported:

Example regression and job messages

The Quickstart Guide provides a few example messages, showing regression and job imports.

Publish JSON message via simscope-tunnel (aka upload)

Note: this is a high performance publishing interface, requiring zero external dependencies.

The command simscope-tunnel has a builtin client publish interface, which can post messages directly to the tunnel server.

This requires a path to your simscope-tunnel.config file (it may be a read-only path).

Optional: Environment variable overrides

The following OS environment variables can override the tunnel.config values at runtime:

Env Key	Value
`$RABBIT_URL`	RabbitMQ URL (ie `amqp://...`)
`$ROUTING_KEY`	RabbitMQ key for publishing (maps to `InputQueue`)

Example publish JSON command

Here is an example to publish a Regression Start (regr-start) message for regression named my_daily_smoke/123:

> bin/simscope-tunnel --config=PATH/TO/simscope-tunnel.config --publish=regr-start \
--regression=my_daily_smoke/123 message.json

→ You should see a response similar to the following on the command-line:

[INFO ] Published to tunnel regr=my_daily_smoke/123 type=regr-start queue=tunnel-in messages=1

Bulk Publish Job JSON

If using a post-processing flow, you can publish multiple JSON job files from a single publish command, for improved publish performance.

This example publishes 3 job JSON files for my_daily_smoke/2000:

> bin/simscope-tunnel --config=PATH/TO/simscope-tunnel.config --publish=job-finish \
--regression=my_daily_smoke/2000 job1.json job2.json job3.json

Debug: JSON message validation

If you add the --validate CLI option, simscope-tunnel will parse and validate JSON messages (and skip publishing).

This is useful for JSON message debugging.

Example successful JSON parse:

> bin/simscope-tunnel --publish=job-finish tests/job-live-noconfig.json --validate

2023-03-29 15:16:08 [INFO ] Message valid type=job-finish count=1 path=[tests/job-live-noconfig.json]

Example failing parse (missing a JSON result field):

> bin/simscope-tunnel --publish=job-finish tests/missing-result.json --validate

DEBUG JSON body [src/verops/tunnel/tests/missing-result.json]:
────────────────────────────────────────────────────────────
{
    "component": "c_fake",
    "project": "p_fake",
    "model_branch": "master",
    "model_version": "123456",
    "userid": "test_user",
    "model_timestamp": "bozo"
}

────────────────────────────────────────────────────────────
2023-03-29 15:17:25 [ERROR] Tunnel Message publish {type=job-finish path=tests/missing-timestamp.json} failed JSON validation: field {result} is blank

Debug: Publishing with RabbitMQ message replies

For debugging purposes, Simscope can send message responses via RabbitMQ, after each job is received, indicating whether it was accepted or rejected, and if rejected the reason.

This enables debugging script issues without needing access to the Simscope or Tunnel server console/log.

To enable message replies on failure, add the option --reply-on-fail to your publish command. For example:

> bin/simscope-tunnel --config=simscope-tunnel.config --publish job-finish \
--regression=myrun/123 job.json --reply-key simscope-debug-queue --reply-on-fail

Note the reply-key is a queue name. You can either create this queue manually, or use the --declare-reply-queue to create it automatically.

If the import fails, Simscope will post a message to RabbitMQ in a queue called simscope-debug-queue.

Note: by default, reply messages will automatically expire after 24 hours. This enables automatic cleanup, in case the messages were not consumed and inspected.
- You can change this with the replyexpire value in your simscope-tunnel.config file.

Debug Mode: reply on all messages

Note: --reply-DEBUG should only be used for debugging purposes! This will affect performance, so it is not recommended for production use.

To publish reply messages for all messages (ie debug mode):

> bin/simscope-tunnel --config=simscope-tunnel.conf --publish regr-start --regression=myrun/123
regr.json --reply-DEBUG

Remember, when running in production, only use the --reply-on-fail option.

Tail message replies

To dump (view) messages replies, use the --tail option.

For example, this dumps any pending messages from regression simscope-debug-queue:

> bin/simscope-tunnel --config=simscope-tunnel.config --tail=simscope-debug-key

Note that tail will watch the queue forever (or until it is closed). You can Ctrl-C to quit.

Here is a sample error response, which has a single debug message in the queue:

2021-05-04 16:18:41 [INFO ] Message regr=simscope type=regr-start size=1481 is-error=true message=Message contains invalid JSON: invalid character 'a' in literal null (expecting 'u')
body: [name,component_id,model_branch,model_version,queue,pending,userid,host,command,dir,integrate,tool_version,signatures,new_signatures,stats
-----
2021-05-04 16:18:41 [INFO ] End of queue messages=1 ack=true

Advanced: Publish from Python

If you are using a Python-based scripting flow and want to publish results directly from Python, you can use the Simscope Python Tunnel API.

Common Publish Errors

→ For common errors during publish, see RabbitMQ Errors.

Simscope Administration Guide