RabbitMQ Errors (Tunnel Publish)

Here are some common RabbitMQ errors, and their possible causes/resolutions.

Health Check Failed/Watchdog errors

If you are getting RabbitMQ health check failed (or unhealthy) notifications, this could either be a harmless notice, or it could be a problem needing to be remedied.

This indicates Simscope has not receied RabbitMQ data publish activity (ie simulation jobs) in XX hours.

Likely causes:

For new or development Simscope installations, this is likely harmless and can be ignored. (You may be only publishing data manually, so this is triggering incorrectly.)
For production Simscope installations, this may mean you need to resolve an infrastructure issue.

→ See the Simscope Rabbit topic for more details.

Error opening: Exclusive use

If you get an exclusive use error during startup of Simscope or Tunnel, similar to the following:

[ERROR] Simscope Error: Rabbit init failed [attempt 1/1]: error opening Simscope input queue:
Exception (403) Reason: "ACCESS_REFUSED - queue 'simscope-in' in vhost '/' in exclusive use"

This likely indicates that Simscope or Tunnel are already running (in another process), and cannot start twice simultaneously.

To remedy, stop (or kill) the other process first.

If you are publishing a JSON client message and get a NO_ROUTE error message, this likely means either Simscope or the Tunnel server are not currently running, and the destination queue does not exist.

Here is an example NO_ROUTE error:

[WARN ] Tunnel publish message was returned code=312 reason=NO_ROUTE
regression=my_daily_smoke/123 type=simscope-regr-start exchange=amq.direct
route=simscope-in size=695

Note: this is a critical error, because it indicates the message could not be routed by RabbitMQ, and was discarded, so the message will be lost.

Normally, if Simscope or Tunnel server are stopped, the Exchange queue still exists, and in this scenario, the messages would just be queued up, and then delivered/processed once Simscope or Tunnel are restarted.

This error is likely caused by:

Publishing to an invalid queue name (from your config file).
A RabbitMQ administrator has deleted the queue.
Your RabbitMQ Exchange has an automatic queue-deletion-policy, after a specified duration.

Network: Check if the firewall is allowing connections remotely

If RabbitMQ server is installed on a virtual machine, you may need to open port 5672 in the firewall configuration.

To verify the port is open, telnet to the hostname of your rabbitmq and press ENTER key multiple times:

> telnet RABBIT_HOST 5672
Trying 123.45.67.89...
Connected to compute-master-01.
Escape character is '^]'.



AMQP    Connection closed by foreign host.

After pressing ENTER multiple times, you should get an AMQP response and then get disconnected automatically.
If you get a hang (ie no response) during the telnet command, this is often caused by a firewall blocking the connection.

I/O Timeout Error: Network Hang

If you get a hang during simscope-tunnel --publish and see an error similar to the following:

[WARN ] Connection retry @error=failed to connect to RabbitMQ: dial tcp 123.45.67.89:5672:
i/o timeout attempt=1

This usually indicates a firewall network issue. See the firewall topic for using telnet to debug this.

Authentication errors

If your AMQP username/password are incorrect, you may get an error similar to this:

pika.exceptions.ProbableAuthenticationError: (403, 'ACCESS_REFUSED -
Login was refused using authentication mechanism PLAIN. For details see the broker logfile.')

From the RABBIT_HOST, you can check the rabbit log file:

> tail -40 /var/log/rabbitmq/rabbit@rhel7.log

Example of invalid authentication:

[error] Error on AMQP connection (127.0.0.1:49401 -> 127.0.0.1:5672, state: starting):
PLAIN login refused: user 'compute' - invalid credentials
[info] closing AMQP connection (127.0.0.1:49401 -> 127.0.0.1:5672)

Example of successful authentication:

[info] connection (127.0.0.1:65466 -> 127.0.0.1:5672): user 'guest' authenticated and
granted access to vhost '/'

Example of user without privileges:

[error] <Error on AMQP connection <0.7657.15> (127.0.0.1:60421 -> 127.0.0.1:5672,
user: 'compute', state: opening):
access to vhost '/' refused for user 'compute'

Simscope Administration Guide