Running a system RabbitMQ on a server with Chef Server

One of the things that I’ve been working on getting set up here at home is Logstash to analyze all the various log files from the home network and the servers that I admin. As far as I can tell, that seems to be the Open Source equivalent of Splunk (which is a great tool, but expensive, and the free version is missing some features that I’d be interested in).

However, I recently migrated my systems from Opscode’s Hosted Chef to the Open Source Chef server running on the box that I had been setting logstash up on. Logstash uses RabbitMQ for messaging, as does Chef. I thought that things would be relatively easy to get working together, but I don’t know much about RabbitMQ. While the ultimate solution was actually trivial, I didn’t find it easy to figure out what to do.

I tried many things that ultimately didn’t pan out. I’m still not entirely sure why, since RabbitMQ is a bit of a black box to me. I tried various combinations of the following settings in /etc/chef-server/chef-server.rb, trying to configure the two systems differently enough so that they wouldn’t conflict with each other:

rabbitmq['consumer_id'] = 'curry'            # Chef default: hotsauce
rabbitmq['nodename'] = [email protected]'         # Chef default: [email protected]
rabbitmq['node_ip_address'] = '192.168.0.4'  # Chef default: 127.0.0.1
rabbitmq['node_port'] = 5673                 # RabbitMQ default: 5672

None of those did the trick. Even after applying all those settings, I still got this back when I did rabbitmqctl status:

[email protected]:/var/log/rabbitmq# rabbitmqctl status
Status of node [email protected] ...
Error: unable to connect to node [email protected]: nodedown

DIAGNOSTICS
===========

nodes in question: [[email protected]]

hosts, their running nodes and ports:
- rain: [{bookshelf,56170},
         {rabbit,47084},
         {erchef,53705},
         {rabbitmqctl13530,57207}]

current node details:
- node name: [email protected]
- home dir: /var/lib/rabbitmq
- cookie hash: vkNSjkgIyXIKaNLguSEV7A==

[email protected]:/var/log/rabbitmq#

Eventually I came across the solution here: http://www.rabbitmq.com/man/rabbitmq-env.conf.5.man.html. All I had to do was create /etc/rabbitmq/rabbitmq-env.conf and add the following lines:

# I am a complete /etc/rabbitmq/rabbitmq-env.conf file.
# Comment lines start with a hash character.
# This is a /bin/sh script file - use ordinary envt var syntax
NODENAME=hare

This ends up with RabbitMQ operating as a different messaging node on the system and co-existing peacefully with the RabbitMQ setup that Chef is running.

I am still curious as to how rabbitmqctl was able to see the config that the Chef server was using, even when it was running on a different port and the databases are stored in two completely different directories (/var/opt/chef-server/rabbitmq and /var/lib/rabbitmq). If anyone knows the answer to that, I’d love to find out!

Running the cacti cookbook under Ubuntu

So something I’ve been loving lately as I dive into the world of DevOps is the large community that Opscode has built up around Chef. While Puppet and Chef aim towards solving the same problem, and have many similarities in thought towards solutions (and many differences, of course), one of the swaying factors for many people like myself is the community. Puppet mostly gave me the tools to reinvent the wheel for my infrastructure; Chef gives me the tools to make a wheel and a shop full of free wheels already made. Sometimes you need to do a bit of work to make it fit, but sometimes you can just hook it up and go. That’s an invaluable thing in today’s fast paced IT world.

My “itch” to scratch of today was Cacti. I’ve been having some problems with the local Comcast connection, and the temperature has been rising here in the PNW, and as a result my mind has returned towards getting my local network monitoring set back up. And indeed, this is a great chance to set up a local testbed for Chef work unrelated to my day job. So I got Nagios and rsyslog bootstrapped with Chef here at home yesterday and worked on Cacti today.

(I got a little derailed when I found out that for some reason my Linux box’s swap partition had an incorrect entry in /etc/fstab. After fixing that, I got an error when trying to turn swap back on:swapon: /dev/sda5: read swap header failed: Invalid argumentThis article pointed me to the solution:mkswap /dev/sda5; swapon -aThat recreated the swap header and then I was able to enable it and have a stable system again.)

There was already a cookbook for Cacti, but it looks like it was designed for Redhat package names and file paths. I spent some time stepping through things and making it work with Ubuntu. For the most part, it was a matter of taking some hard-coded settings, replacing them with attributes, and setting the default values for those attributes to be the same as the old hard-coded values. This allows me to then override them locally, and anyone else already using the cookbook will see no change. I did add a few platform-specific checks, for things like the Ubuntu package names.

In all likelihood, anyone running a Debian system can probably change the spots I added Ubuntu support and extend them for Debian support as well (since most of the core Ubuntu packages either come from or get merged upstream into Debian). However, I don’t have a Debian test environment yet, so I didn’t want to make assumptions. It’s on the list of things to get up and running in a VM… CentOS, Oracle Linux, and Debian.

Here’s the role that I ended up with, when all was said and done. In my case, this server is only available over the internal network, so I didn’t need SSL support.

{
  "name": "cacti-server",
  "description": "Role to configure Cacti server.",
  "json_class": "Chef::Role",
  "chef_type": "role",
  "default_attributes": {
  },
  "override_attributes": {
    "cacti": {
      "user": "www-data",
      "group": "www-data",
      "cron_minute": "*",
      "apache2": {
        "conf_dir": "/etc/apache2/conf.d",
        "doc_root": "/var/www",
        "ssl": {
          "force": false,
          "enabled": false
        }
      }
    }
  },
  "run_list": [
    "recipe[cacti::server]",
    "recipe[cacti::spine]"
  ],
  "env_run_lists": {
  }
}

I sent over a pull request to get the changes merged in, but until then feel free to grab the cookbook from github (note that you’ll want the ubuntu branch). If you’re using Berkshelf, you can add this to your Berksfile:
cookbook 'cacti', github: 'stormerider/chef-cacti', branch: 'ubuntu'

I hope this helps someone else!