Simscope Tunnel

Simscope Tunnel allows integration of Simscope with RabbitMQ, a high performance, durable, message queuing server.

This enables simulations from queues/grids, across one or more data centers to all post their results into RabbitMQ queues with low latency, and continue working onto the next job. The compute nodes will perform a "dump and run" of their results to RabbitMQ, and then get back to work.

Without a flow similar to this and a large enough compute infrastructure, you can saturate Simscope with hundreds of simultaneous HTTP connections, which is an inefficient and possibly error-prone workflow.

RabbitMQ workflow:

Performance

Simscope Tunnel with RabbitMQ has been tested to handle processing more than 300 job completions per second (1 million jobs per hour).

RabbitMQ configuration

The RabbitMQ needs to be configured with durable queues enabled, so that messages can be stored in case the server goes down.

This requires RabbitMQ version 3.0 or higher (currently tested with RabbitMQ 3.8.2).

(For mission critical applications, you may want a backup/failsafe RabbitMQ server.)

RabbitMQ+Tunnel can be configured two ways:

1. Direct queue messaging (simplest)
2. Topic-based queue messaging, using exchanges (most configurable and allows multiple processing of messages)

The Tunnel will auto-declare both its input and output queues, based on the configuration parameters (durable, etc).

Input Queue

The Tunnel maintains an input queue, which reads JSON input messages, sent from compute nodes.

The format of these messages is described in Tunnel Message Formats.

Output Queue

The Tunnel writes results into an output queue, which is consumed from directly by Simscope.

Tunnel configuration

Tunnel config example

See tunnel.config

Starting the Simscope Tunnel

Simscope Tunnel runs as a server process, so it should be run under a screen or tmux session.

$ bin/simscope-tunnel simscope-tunnel.config

NOTE: Simscope Tunnel is decoupled from Simscope, so it can be started and stopped before or after Simscope. In other words, there is no ordering dependency on it.

Also if RabbitMQ stops or restarts, the Tunnel will automatically reconnect once RabbitMQ is available again.

Example session:

2020-02-05 20:38:40 [INFO ] Tunnel configuration:
 Name                  | simscope-tunnel.ini
 Tunnel input queue    | tunnel-in
 Input bindings        | -
 Output Simscope queue | simscope-in
 Auto Flush            | 30s (1000 jobs)
 Auto Reconnect        | 5s
 Bucketizer            | -
 Classifications       | -
 Componentize          | -
2020-02-05 20:38:40 [INFO ] Simscope Tunnel connected to Rabbit version=1.13 addr=127.0.0.1:57739
2020-02-05 20:38:40 [INFO ] Tunneling version=1.14 configs=1

Stopping the Simscope Tunnel

To stop the Simscope Tunnel:

• If you are using System Services you can stop or restart from the CLI.
• Otherwise, just <Ctrl-C> the process (or use kill) to stop it.

After shutdown, the tunnel will print statistics of the session:

2020-02-05 20:40:04 [WARN ] <Signal 2> received (interrupt)...
2020-02-05 20:40:04 [INFO ] Tunnel stopped.  Stats:
 Name      | simscope-tunnel.config
 Duration  | 7h30m
 Connects  | 10
 Messages  | 12000
 TotalSize | 8.2MB
 Regrs     | 200
 Jobs      | 11800
 Errors    | 10
2020-02-05 20:40:04 [INFO ] Tunnel OK

Note: To disable the Tunnel, just stop the process.

Input message format: Tunnel

The Tunnel watches its input queue for four types of JSON messages:

How the Tunnel works

Internally, the Tunnel monitors a single input queue from Rabbit, and converts the messages into a CSV format that can be imported into Simscope, and posts these messages to an output queue from Rabbit.

Simscope configuration

The simscope.config has a [rabbitmq] section to process input messages from RabbitMQ.

[rabbitmq]
url = "amqp://USER:PASSWORD@localhost:5672/"
retrytimeout = "5s"
health = "60m"

inputqueue = "simscope-in"
durable    = true
autodelete = false
exclusive  = false

Tunnel Reconnect

The Tunnel will automatically re-connect to RabbitMQ, if its connection is broken.

Here is an example of disconnect and automatic reconnect:

2020-02-06 20:28:06 [WARN ] Rabbit closed with error @error=Exception (501) Reason: "read tcp 127.0.0.1:63622->127.0.0.1:5672: read: connection reset by peer"
2020-02-06 20:28:06 [WARN ] Rabbit queue closed input-queue=simscope-in
2020-02-06 20:28:06 [WARN ] Rabbit disconnected, reconnecting... timeout=5s
2020-02-06 20:28:11 [INFO ] Connected to Rabbit input-queue=simscope-in addr=127.0.0.1:55567