Running a system RabbitMQ on a server with Chef Server

One of the things that I’ve been working on getting set up here at home is Logstash to analyze all the various log files from the home network and the servers that I admin. As far as I can tell, that seems to be the Open Source equivalent of Splunk (which is a great tool, but expensive, and the free version is missing some features that I’d be interested in).

However, I recently migrated my systems from Opscode’s Hosted Chef to the Open Source Chef server running on the box that I had been setting logstash up on. Logstash uses RabbitMQ for messaging, as does Chef. I thought that things would be relatively easy to get working together, but I don’t know much about RabbitMQ. While the ultimate solution was actually trivial, I didn’t find it easy to figure out what to do.

I tried many things that ultimately didn’t pan out. I’m still not entirely sure why, since RabbitMQ is a bit of a black box to me. I tried various combinations of the following settings in /etc/chef-server/chef-server.rb, trying to configure the two systems differently enough so that they wouldn’t conflict with each other:

rabbitmq['consumer_id'] = 'curry'            # Chef default: hotsauce
rabbitmq['nodename'] = '[email protected]'         # Chef default: [email protected]
rabbitmq['node_ip_address'] = '192.168.0.4'  # Chef default: 127.0.0.1
rabbitmq['node_port'] = 5673                 # RabbitMQ default: 5672

None of those did the trick. Even after applying all those settings, I still got this back when I did rabbitmqctl status:

[email protected]:/var/log/rabbitmq# rabbitmqctl status
Status of node [email protected] ...
Error: unable to connect to node [email protected]: nodedown

DIAGNOSTICS
===========

nodes in question: [[email protected]]

hosts, their running nodes and ports:
- rain: [{bookshelf,56170},
         {rabbit,47084},
         {erchef,53705},
         {rabbitmqctl13530,57207}]

current node details:
- node name: [email protected]
- home dir: /var/lib/rabbitmq
- cookie hash: vkNSjkgIyXIKaNLguSEV7A==

[email protected]:/var/log/rabbitmq#

Eventually I came across the solution here: http://www.rabbitmq.com/man/rabbitmq-env.conf.5.man.html. All I had to do was create /etc/rabbitmq/rabbitmq-env.conf and add the following lines:

# I am a complete /etc/rabbitmq/rabbitmq-env.conf file.
# Comment lines start with a hash character.
# This is a /bin/sh script file - use ordinary envt var syntax
NODENAME=hare

This ends up with RabbitMQ operating as a different messaging node on the system and co-existing peacefully with the RabbitMQ setup that Chef is running.

I am still curious as to how rabbitmqctl was able to see the config that the Chef server was using, even when it was running on a different port and the databases are stored in two completely different directories (/var/opt/chef-server/rabbitmq and /var/lib/rabbitmq). If anyone knows the answer to that, I’d love to find out!

Tagged , , , , . Bookmark the permalink.

4 Responses to Running a system RabbitMQ on a server with Chef Server

  1. Wes D says:

    You’ve got two things going on. The mnesia database that RMQ uses is tied to the hostname. I think the reason ctrl knew something about Chef is because all RabbitMQ instances running on the machine sit inside the same Erlang VM memory space. It’s probably Erlang Socket Magic that rabbitmqctrl is using. Not every (or many) command on a local ctrl connection use the AMQP socket (5672). So “[email protected]” and “[email protected]” and “[email protected]” are internally routed via the Erlang VM when you’re not using the TCP connection.

    I think. : )

    WD

    • Yeah, I knew it defaulted to the hostname, which is why I was trying to tune Chef to point at a different DB namespace in the first place.

      I think the reason ctrl knew something about Chef is because all RabbitMQ instances running on the machine sit inside the same Erlang VM memory space.

      Do different erlang processes share VM memory space? So the Chef server I have installed is an “omnibus” version, meaning that it’s a standalone package with all of the dependencies embedded into it and installed underneath /opt/chef-server. So, ruby, erlang, nginx, postgresql, the whole works. They did this because so many people were running into dependency problems (particularly with Ruby and Ruby gems, with many distros still shipping out Ruby 1.8, which doesn’t play nicely with certain Chef components). I would tend to think that the memory space would be different between different binary instances of Erlang, but I’ll admit that I know very, very little about Erlang.

      In any case, now that I’m migrating my Linode machines over to a 64-bit install, I’m actually going to move the Chef server over to that and this won’t be an issue anymore… but I hope it helps someone else who runs into a similar conflict.