Tunnel Release History

Tunnel change history, by release version.

1.61

Changed default RabbitMQ connect timeout from 30 seconds to 10 seconds.
Added --timeout-seconds=X option, to override the RabbitMQ connection timeout (in seconds).
Increased maximum message size to 32 MB (for coverage messages)

If you see an error message during publish similar to this (from previous Tunnel versions), this release increases the maximum:

JSON message exceeds maximum size (262144)

1.60

If publishing coverage DBs via tunnel, these messages are now routed directly to Simscope, rather than proxied through the tunnel. This helps to eliminate duplicate traffic in RMQ.

1.59

Improved [componentize] print messages.

1.58

New --log-errors-and-continue option

If publishing batch job data to Simscope, you can now optionally provide the option:

--log-errors-and-continue=FILENAME

With this option enabled, job JSON errors will be appended to FILENAME, and as long as one or more jobs are valid, they will be published (optimistic behavior).

By default (without this option), if any of the jobs is invalid, none of the jobs will be published to Simscope (pessimistic behavior).

1.57

1. `feature` Tunnel batch job finish

Tunnel client now has a batch-job-finish publish mode, which allows publishing an array of jobs with a single JSON publish.

For groups doing a post-processing publish flow into Simscope, where you have many results in a single regression to publish simultaneously, this both improves performance, and shrinks network traffic, due to JSON messages being combined together.

2. `feature` JSON syntax error handling improvements

If publishing JSON files with syntax errors, this now prints context information of where the error is.

For example, here is an example end-of-JSON input error:

Tunnel Message publish failed JSON validation {type=job-finish path=<stdin>}:
───── Line 1, Column 2 ─────
{
 └─── unexpected end of JSON input

1.56

Added [metadata] configuration options, to allow customizing maximum job metadata sizes.

See tunnel.config for more details.

1.55

Long job metadata values are now truncated, instead of being discarded.

Note: the current metadata value limit is 300b (ie bytes) per metadata value.

1.54

This Tunnel release now supports Custom Job Metadata values with comma (ie ,) characters embedded in them.

1.53

For groups running long jobs, increased maximum individual job duration fields (compute_ms and runtime_ms) from 10 days to 24 days.

1.52

Tunnel can now work with RabbitMQ topic exchanges, in addition to standard direct exchanges.

To use with an alternative exchange, add an exchange line to your tunnel.config file.

Here is an example using the amq.topic exchange:

[tunnel]

exchange = "amq.topic"

# Exchange queue input bindings (needed if using topic exchanges)
[exchange "amq.topic"]
key = "tunnel-in" # this should match your input queue name

# Exchange queue output bindings (needed if using topic exchanges)
[outputexchange "amq.topic"]
key = "simscope-in" # this should match your output queue name

Tunnel publish now can send replies to any debug queue name. To set the queue name explicitly, set the --reply-key option:

> bin/simscope-tunnel --config=tunnel.config --reply-key=simscope-debug --reply-on-fail

You can use the optional --declare-reply-queue option, if the queue does not exist.

To view the messages sent to the reply queue:

> bin/simscope-tunnel --config=tunnel.config --tail simscope-debug

2023-05-08 12:23:17 [INFO ] Tailing queue=zzdebug2
2023-05-08 12:23:17 [INFO ] Reply #1 app=simscope-server regression=uart-smoke/1001 id=1/csh2jpz65bc8 type=job-import-ok size=12 is-error=false message= original-body=
-----
uart:default
-----

1.51

Tunnel now has a --validate option, to parse and validate JSON message (and skip publish).

This is useful for JSON message debugging.

Example successful JSON parse:

> simscope-tunnel --publish=job-finish tests/job-live-noconfig.json --validate
2023-03-29 15:16:08 [INFO ] Message valid type=job-finish count=1 path=[tests/job-live-noconfig.json]

Example missing a JSON result field:

> simscope-tunnel --publish=job-finish tests/missing-result.json --validate
DEBUG JSON body [src/verops/tunnel/tests/missing-result.json]:
────────────────────────────────────────────────────────────
{
    "component": "c_fake",
    "project": "p_fake",
    "model_branch": "master",
    "model_version": "123456",
    "userid": "test_user",
    "model_timestamp": "bozo"
}

────────────────────────────────────────────────────────────
2023-03-29 15:17:25 [ERROR] Tunnel Message publish {type=job-finish path=tests/missing-timestamp.json} failed JSON validation: field {result} is blank

1.50

dvsim collection improvements:
- Coverage collection (--cov) now works with Simscope
  - Both cov_merge and cov_report jobs are published.
  - Coverage results from dvsim are extracted into Simscope Job metadata and Regression metadata.
  - Coverage report path gets extracted into Simscope Regression metadata.
- Fixed a dvsim bug where non-local jobs (ie LSF/SGE) were capturing with zero runtime.
Added optional environment variables overrides: $RABBIT_URL and $ROUTING_KEY

1.49

Tunnel publish now detects Regression metadata errors for regr-update messages.
Added simscope_error_fixup.py as an example Python script to apply {{keep}} and <<ignore>> patterns to error messages, before publishing to Simscope.

Here is an example of using it:

KEEP_PATTERNS = [
    'cpu\d+',
    'Vendor error \d+',
]

IGNORE_PATTERNS = [
    'Instr: ".*"',
    'my_checker\d+',
]

error_fixup = SimscopeErrorFixup(keep_patterns=KEEP_PATTERNS, ignore_patterns=IGNORE_PATTERNS)

def publish_to_simscope(job):
    ...

    # Fixup error messages
    job['fail_message'] = error_fixup.auto_fixup_error(job['fail_message'])

    # Now publish the job JSON to the Tunnel...

1.48

Tunnel publish now detects future timestamps and detects Stale Models when publishing Regression JSON.

Example rejection error:

[ERROR] Tunnel Message publish {type=regr-start path=dir/future-timestamp.json} failed JSON validation: model_timestamp in the future: 2040-02-28 00:26:14 +0000 +0000

1.47

Added --quiet option, which turns off INFO messages. Only warnings/errors will be displayed.
Added dvsim Python plugin, to enable automatic publishing from dvsim into Simscope.

1.46

SIMSCOPE-49

Tunnel publish now validates JSON regression fields name, model_branch, project, and component during the publish command.

If they fail, the message will be rejected before publishing.

In prior versions, the message would be published, and then be rejected within the tunnel.

Example rejection error:

[ERROR] Tunnel Message publish {type=regr-start path=../simscope/invalid-component.json} failed
JSON validation: project {"P_FAKE\n"} invalid: whitespace not allowed: '\n'

1.45

Added improvements to help with debugging Tunnel or publish issues.

Run your tunnel command with an additional --debug argument, and Simscope will print additional debugging messages to stdout.

Here is an example (see the DEBUG lines):

> bin/simscope-tunnel --config=docker/config/tunnel-local.config --publish=regr-start \
example-json/example-regr.json --regression=smoke/123 --debug

2022-09-20 19:40:18 [DEBUG] Enabled debug mode
2022-09-20 19:40:18 [DEBUG] Commandline: [bin/simscope-tunnel --config=docker/config/tunnel-local.config --publish=regr-start example-json/example-regr.json --regression=smoke/123 --debug]
2022-09-20 19:46:21 [DEBUG] Publishing num-messages=1 regression=smoke/123 message-type=regr-start url=amqp://guest:password@127.0.0.1:5672/
2022-09-20 19:44:30 [WARN ] Rabbit message was returned code=312 reason=NO_ROUTE route=tunnel-in2
2022-09-20 19:44:30 [ERROR] Received 1 returned messages

1.44

Validation improvements to simscope-tunnel --publish:

Prevents the job category field containing bogus characters, like [/foo]
Validates the job result field only contains one of the following values:
- live
- pass
- fail
- skip
- (blank) — ie field was omitted

1.43

Added size exception for large regression messages.

This allows large Coverage files to be published, instead of being rejected with a large size error message.

Note: the current size limit is 4 MB, for regression JSON messages.

1.42

Large JSON messages are now rejected immediately during a publish command, rather than waiting for the Tunnel server to reject them.

1.41

Added validation that published JSON messages are no larger than 256 kB. These will now be rejected immediately.

1.40

Added Tunnel publish --stdin-json option, to allow publishing JSON messages via Unix Standard Input (stdin pipe).

Example CLI publish:

> PATH/simscope-tunnel --config=PATH/simscope-tunnel.config --publish regr-start --regression=simscope --stdin-json

Then pipe your JSON message to the stdin of the process:

{
    "jobdir": "/home/runs/myregr/2/job-2",
    "category": "sim",
    "config": "sim +rand +tbuf +seed=1794 LOC=4",

... (etc)
}

Added example_tunnel_publish.py script, showing examples of creating and publishing JSON regression/job objects from Python.

1.39

Added Tunnel --flush-exit command option, to process the current queue messages and then exit (without starting a serve thread).
- Note: this is meant to use as a debug command, and not intended for production.

1.38

Added Simscope "Live" job support.
Fixed a race condition where a RabbitMQ channel can error-out during a publish operation, and the error event can either deadlock the Tunnel or produce an infinite-loop of output messages, resulting in a hang condition.

1.37

Minor fix for regression JSON output rendering invalid build field (this is a job-based field).

1.36

Added improved JSON validation during simscope-tunnel --publish

1.35

1. `feature` Tunnel publish retry logic

Tunnel client message publish now makes up to 3 attempts (retries) to publish before failure. This helps to alleviate network issues.

2. `debug` Tunnel server shutdown

Tunnel server now will dump a stack trace for debugging if it does not shut down within 10 seconds.

3. `debug` Tunnel publish validation

The Tunnel publish client command now validates many of the JSON fields (for example a zero model_timestamp), and will reject the file before publishing.

1.34

1. `feature` Tunnel publish JSON validation

The Tunnel publish command now automatically validates JSON input messages, before publishing them.

If they fail basic validation, the publish command will return with an error code back to the shell.

Here is an example publish with an invalid JSON input message:

> bin/simscope-tunnel --config=simscope-tunnel.config --publish job-finish \
--regression=release/11133 bad.json

2021-08-20 15:08:47 [ERROR] Tunnel Message publish [bad.json] contains invalid JSON:
unexpected end of JSON input

2. `bugfix` Tunnel server componentize

The componentize feature now works if using the tunnel without a bucketizer enabled.

1.33

This release contains a bugfix for handling RabbitMQ errors, where the Tunnel server can potentially hang if a channel error is received.

1.32

Fixed a runtime vs compute time bug, related to monotonic clocks.

1.31

Added new optional Job JSON sourcecode field.

1.30

Added tunneld daemon service.

1.29

Minor fixes for message reply debug feature.

1.28

Tunnel now has improved RabbitMQ error handling, in cases of connection or channel error events.

1.27

Tunnel now supports message replies, which enables debugging both message failures (or all messages), without needing access to a Simscope or Tunnel terminal console/log.

To publish reply messages if a message fails, use the --reply-on-fail option when publishing.

For example:

> simscope-tunnel --config=simscope-tunnel.conf --publish regr-start --regression=myrun/123 regr.json --reply-on-fail

To publish reply messages for all messages (ie debug mode):

> simscope-tunnel --config=simscope-tunnel.conf --publish regr-start --regression=myrun/123 regr.json --reply-debug

NOTE: --reply-debug should only be used for debugging purposes! This will affect performance, so it is not recommended for production use.
- Use the --reply-on-fail option instead for production.

→ see Client Publish for more details

1.26

Changed default status interval to 5 minutes.

This can still be overridden via tunnel.config file.

1.25

Added simscope-tunnel --version command to print the tool version.

> simscope-tunnel --version
1.25

1.24

Tunnel now supports an optional extid field for Jobs.

1.23

Tunnel can now publish client json messages from the command-line to a Tunnel server (and into Simscope).

This enables posting client messages without needing Python or the pika library.

1.22

Tunnel now supports the optional regr-update message.
Tunnel now sends heartbeat messages to Simscope, indicating the health of the Tunnel.

1.21

The Tunnel now comes with a JUnit XML (ie from Jenkins) parser.

See the script simscope_junit.py, which will parse a single .xml and .json file and post jobs into a single Regression, using your Tunnel instance.

Tunnel now forwards errors to Simscope, so they can be viewed via the Simscope Web Admin.

Simscope Administration Guide