Running a system RabbitMQ on a server with Chef Server

One of the things that I’ve been working on getting set up here at home is Logstash to analyze all the various log files from the home network and the servers that I admin. As far as I can tell, that seems to be the Open Source equivalent of Splunk (which is a great tool, but expensive, and the free version is missing some features that I’d be interested in).

However, I recently migrated my systems from Opscode’s Hosted Chef to the Open Source Chef server running on the box that I had been setting logstash up on. Logstash uses RabbitMQ for messaging, as does Chef. I thought that things would be relatively easy to get working together, but I don’t know much about RabbitMQ. While the ultimate solution was actually trivial, I didn’t find it easy to figure out what to do.

I tried many things that ultimately didn’t pan out. I’m still not entirely sure why, since RabbitMQ is a bit of a black box to me. I tried various combinations of the following settings in /etc/chef-server/chef-server.rb, trying to configure the two systems differently enough so that they wouldn’t conflict with each other:

rabbitmq['consumer_id'] = 'curry'            # Chef default: hotsauce
rabbitmq['nodename'] = '[email protected]'         # Chef default: [email protected]
rabbitmq['node_ip_address'] = '192.168.0.4'  # Chef default: 127.0.0.1
rabbitmq['node_port'] = 5673                 # RabbitMQ default: 5672

None of those did the trick. Even after applying all those settings, I still got this back when I did rabbitmqctl status:

[email protected]:/var/log/rabbitmq# rabbitmqctl status
Status of node [email protected] ...
Error: unable to connect to node [email protected]: nodedown

DIAGNOSTICS
===========

nodes in question: [[email protected]]

hosts, their running nodes and ports:
- rain: [{bookshelf,56170},
         {rabbit,47084},
         {erchef,53705},
         {rabbitmqctl13530,57207}]

current node details:
- node name: [email protected]
- home dir: /var/lib/rabbitmq
- cookie hash: vkNSjkgIyXIKaNLguSEV7A==

[email protected]:/var/log/rabbitmq#

Eventually I came across the solution here: http://www.rabbitmq.com/man/rabbitmq-env.conf.5.man.html. All I had to do was create /etc/rabbitmq/rabbitmq-env.conf and add the following lines:

# I am a complete /etc/rabbitmq/rabbitmq-env.conf file.
# Comment lines start with a hash character.
# This is a /bin/sh script file - use ordinary envt var syntax
NODENAME=hare

This ends up with RabbitMQ operating as a different messaging node on the system and co-existing peacefully with the RabbitMQ setup that Chef is running.

I am still curious as to how rabbitmqctl was able to see the config that the Chef server was using, even when it was running on a different port and the databases are stored in two completely different directories (/var/opt/chef-server/rabbitmq and /var/lib/rabbitmq). If anyone knows the answer to that, I’d love to find out!