Running a system RabbitMQ on a server with Chef Server

One of the things that I’ve been working on getting set up here at home is Logstash to analyze all the various log files from the home network and the servers that I admin. As far as I can tell, that seems to be the Open Source equivalent of Splunk (which is a great tool, but expensive, and the free version is missing some features that I’d be interested in).

However, I recently migrated my systems from Opscode’s Hosted Chef to the Open Source Chef server running on the box that I had been setting logstash up on. Logstash uses RabbitMQ for messaging, as does Chef. I thought that things would be relatively easy to get working together, but I don’t know much about RabbitMQ. While the ultimate solution was actually trivial, I didn’t find it easy to figure out what to do.

I tried many things that ultimately didn’t pan out. I’m still not entirely sure why, since RabbitMQ is a bit of a black box to me. I tried various combinations of the following settings in /etc/chef-server/chef-server.rb, trying to configure the two systems differently enough so that they wouldn’t conflict with each other:

rabbitmq['consumer_id'] = 'curry'            # Chef default: hotsauce
rabbitmq['nodename'] = [email protected]'         # Chef default: [email protected]
rabbitmq['node_ip_address'] = '192.168.0.4'  # Chef default: 127.0.0.1
rabbitmq['node_port'] = 5673                 # RabbitMQ default: 5672

None of those did the trick. Even after applying all those settings, I still got this back when I did rabbitmqctl status:

[email protected]:/var/log/rabbitmq# rabbitmqctl status
Status of node [email protected] ...
Error: unable to connect to node [email protected]: nodedown

DIAGNOSTICS
===========

nodes in question: [[email protected]]

hosts, their running nodes and ports:
- rain: [{bookshelf,56170},
         {rabbit,47084},
         {erchef,53705},
         {rabbitmqctl13530,57207}]

current node details:
- node name: [email protected]
- home dir: /var/lib/rabbitmq
- cookie hash: vkNSjkgIyXIKaNLguSEV7A==

[email protected]:/var/log/rabbitmq#

Eventually I came across the solution here: http://www.rabbitmq.com/man/rabbitmq-env.conf.5.man.html. All I had to do was create /etc/rabbitmq/rabbitmq-env.conf and add the following lines:

# I am a complete /etc/rabbitmq/rabbitmq-env.conf file.
# Comment lines start with a hash character.
# This is a /bin/sh script file - use ordinary envt var syntax
NODENAME=hare

This ends up with RabbitMQ operating as a different messaging node on the system and co-existing peacefully with the RabbitMQ setup that Chef is running.

I am still curious as to how rabbitmqctl was able to see the config that the Chef server was using, even when it was running on a different port and the databases are stored in two completely different directories (/var/opt/chef-server/rabbitmq and /var/lib/rabbitmq). If anyone knows the answer to that, I’d love to find out!

Running the cacti cookbook under Ubuntu

So something I’ve been loving lately as I dive into the world of DevOps is the large community that Opscode has built up around Chef. While Puppet and Chef aim towards solving the same problem, and have many similarities in thought towards solutions (and many differences, of course), one of the swaying factors for many people like myself is the community. Puppet mostly gave me the tools to reinvent the wheel for my infrastructure; Chef gives me the tools to make a wheel and a shop full of free wheels already made. Sometimes you need to do a bit of work to make it fit, but sometimes you can just hook it up and go. That’s an invaluable thing in today’s fast paced IT world.

My “itch” to scratch of today was Cacti. I’ve been having some problems with the local Comcast connection, and the temperature has been rising here in the PNW, and as a result my mind has returned towards getting my local network monitoring set back up. And indeed, this is a great chance to set up a local testbed for Chef work unrelated to my day job. So I got Nagios and rsyslog bootstrapped with Chef here at home yesterday and worked on Cacti today.

(I got a little derailed when I found out that for some reason my Linux box’s swap partition had an incorrect entry in /etc/fstab. After fixing that, I got an error when trying to turn swap back on:swapon: /dev/sda5: read swap header failed: Invalid argumentThis article pointed me to the solution:mkswap /dev/sda5; swapon -aThat recreated the swap header and then I was able to enable it and have a stable system again.)

There was already a cookbook for Cacti, but it looks like it was designed for Redhat package names and file paths. I spent some time stepping through things and making it work with Ubuntu. For the most part, it was a matter of taking some hard-coded settings, replacing them with attributes, and setting the default values for those attributes to be the same as the old hard-coded values. This allows me to then override them locally, and anyone else already using the cookbook will see no change. I did add a few platform-specific checks, for things like the Ubuntu package names.

In all likelihood, anyone running a Debian system can probably change the spots I added Ubuntu support and extend them for Debian support as well (since most of the core Ubuntu packages either come from or get merged upstream into Debian). However, I don’t have a Debian test environment yet, so I didn’t want to make assumptions. It’s on the list of things to get up and running in a VM… CentOS, Oracle Linux, and Debian.

Here’s the role that I ended up with, when all was said and done. In my case, this server is only available over the internal network, so I didn’t need SSL support.

{
  "name": "cacti-server",
  "description": "Role to configure Cacti server.",
  "json_class": "Chef::Role",
  "chef_type": "role",
  "default_attributes": {
  },
  "override_attributes": {
    "cacti": {
      "user": "www-data",
      "group": "www-data",
      "cron_minute": "*",
      "apache2": {
        "conf_dir": "/etc/apache2/conf.d",
        "doc_root": "/var/www",
        "ssl": {
          "force": false,
          "enabled": false
        }
      }
    }
  },
  "run_list": [
    "recipe[cacti::server]",
    "recipe[cacti::spine]"
  ],
  "env_run_lists": {
  }
}

I sent over a pull request to get the changes merged in, but until then feel free to grab the cookbook from github (note that you’ll want the ubuntu branch). If you’re using Berkshelf, you can add this to your Berksfile:
cookbook 'cacti', github: 'stormerider/chef-cacti', branch: 'ubuntu'

I hope this helps someone else!

Building an Ubuntu AMI with Elastic Beanstalk Support

First off, this would not have been possible without help from Defusal on #eventmachine in Freenode IRC. When I hit the wall and beat my head against it for multiple hours, they were the one to give me the key that allowed me to take the final step forward into completing this conversion.

Secondly, I’ll disclose some biases and assumptions straight up. I’ve done this with Ubuntu 12.04 on a 64-bit platform. I chose this because I wanted to standardize all of my EC2 nodes on Ubuntu as a distribution, and Amazon Linux is Redhat based. Redhat is a great company that does a lot of great work, but I’ve worked with CentOS 5 for some time now and kept running into problems because things were so out of date to ensure stability. I realize that a lot has gotten updated with RHEL 6, but that doesn’t mean that they will keep things up to date moving forward; that’s not the goal of RHEL, that’s more the aim of a distribution like Fedora. I can respect that dedication to stability, but working in a small startup, I want a lot of flexibility and maneuverability from a technical perspective; I don’t want a distribution to limit me to a list of applications that has essentially ossified. However, I’m assuming you’re not here because you like Redhat as a distro, or you wouldn’t be checking this out. 🙂 I also assume that you’ve got your instance set up with Apache2 and ruby-1.9 (in my case, 1.9.3).

First off, boot up an existing ElasticBeanstalk AMI. I did this from an instance that was actually running as part of a EB project. (Yes, I know that everyone says “you can’t do it from a deployed project”. Well, I did.) Tar up the contents of /opt/aws, /opt/elasticbeanstalk, /etc/httpd/sites, and /opt/tomcat7 (or /opt/tomcat6 if that’s what you want to run, but I’ll refer to everything following for Tomcat 7). I also snagged /etc/init.d/functions, /etc/init.d/tomcat7, and /etc/init.d/hostmanager. Copy this file over to your new build host.

Create users for tomcat7 and elasticbeanstalk with the home directories pointed to their respective directories in /opt. Add tomcat7 to the elasticbeanstalk group (this is important). Extract the files into /opt on your new build host. (Do it in this order to preserve file permissions, or chown them appropriately.) Remove /opt/tomcat7/webapps/ROOT.war, /opt/tomcat7/webapps/ROOT/, and /opt/elasticbeanstalk/srv/hostmanager/db/hostmanager.db. These steps are important to be able to deploy your new application code to the instance, and the hostmanager.db file seems to be why most people have problems creating an image from an active EB node.

Copy the init scripts to their correct locations. Note that you could skip the functions file if you want to rewrite the hostmanager init script; I didn’t feel like bothering with it, personally. You’ll need to modify the hostmanager init script slightly; I put the functions file in /opt/aws and updated the hostmanager script to look there for it. Don’t forget to enable them with:

update-rc.d tomcat7 defaults
update-rc.d hostmanager defaults

Hostmanager expects things to be a certain way, and the RedHat way is not the Ubuntu way. The Apache configs obviously go into /etc/apache2/sites-available and then symlinked into /etc/apache2/sites-enabled (either manually or using a2ensite). However, you will need to snap a few symlinks to get hostmanager to recognize the Apache config:

cd /etc ; ln -s apache2 httpd
cd /var/run ; ln -s apache2 httpd
cd apache2 ; ln -s ../apache2.pid httpd.pid

This is the one part I think I might be missing a detail or two on; I was pretty tired when I worked on this so I’m not 100% sure if there were other symlinks to be snapped or just those. It should be fairly obvious in the hostmanager logs if one is missing (look in /opt/elasticbeanstalk/var/log). If you really want, you might be able to avoid these symlinks by customizing the hostmanager config files, but I didn’t trust that Amazon doesn’t have an updating mechanism in place. To a large extent, I have tried to install this in a compatible way that if Amazon pushes an update of some sort that it likely won’t break on your custom image. Not knowing exactly what Amazon can do in regards to that, however, I can’t guarantee anything.

So, one thing about hostmanager and EB. Amazon has embedded an entire Ruby installation into their elasticbeanstalk directory. This means that you don’t have to worry about it conflicting with anything else and you don’t need to install a long list of gem files on your system to get things to work. However, I did run into the fact that Amazon Linux is using a different version of the OpenSSL and libcrypto libraries than Ubuntu 12.04 is (if you’re using an older Ubuntu version, this may or may not apply to you. YMMV). To fix that, I did something along the lines of:

mv /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/openssl.so /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/openssl.so-old
mv /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/digest/rmd160.so /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/digest/rmd160.so-old
mv /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/digest/sha1.so /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/digest/sha1.so-old
mv /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/digest/sha2.so /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/digest/sha2.so-old
mv /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/digest/md5.so /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/digest/md5.so-old
cp /usr/lib/ruby/1.9.1/x86_64-linux/openssl.so /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/
cp /usr/lib/ruby/1.9.1/x86_64-linux/digest/rmd160.so /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/
cp /usr/lib/ruby/1.9.1/x86_64-linux/digest/sha1.so /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/
cp /usr/lib/ruby/1.9.1/x86_64-linux/digest/sha2.so /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/
cp /usr/lib/ruby/1.9.1/x86_64-linux/digest/md5.so /opt/elasticbeanstalk/lib/ruby/1.9.1/x86_64-linux/

I also ran into a HUGE issue that stumped me for hours regarding EventMachine and rb-inotify. I’m not sure why the code & gem versions work perfectly fine on the Amazon setup and not on the Ubuntu host I was working on, but it simply would not work for me. I kept getting errors with hostmanager like so:

[2012-08-09 16:55:15 +0000] Host Manager startup complete
[2012-08-09 16:55:15 +0000] Stopping DaemonManager
>> Exiting!
/opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:795:in `attach_io': undefined method `attach_fd' for EventMachine:Module (NoMethodError)
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:769:in `watch'
from /opt/elasticbeanstalk/srv/hostmanager/lib/elasticbeanstalk/hostmanager/daemon/logdirectorymonitor.rb:101:in `run'
from /opt/elasticbeanstalk/srv/hostmanager/lib/elasticbeanstalk/hostmanager/daemon.rb:36:in `block in start'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:996:in `call'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:996:in `block in run_deferred_callbacks'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:996:in `each'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:996:in `run_deferred_callbacks'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:1449:in `event_callback'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/pr_eventmachine.rb:898:in `eventable_read'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/pr_eventmachine.rb:369:in `block in crank_selectables'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/pr_eventmachine.rb:369:in `each'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/pr_eventmachine.rb:369:in `crank_selectables'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/pr_eventmachine.rb:324:in `block in run'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/pr_eventmachine.rb:318:in `loop'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/pr_eventmachine.rb:318:in `run'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/pr_eventmachine.rb:64:in `run_machine'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/thin-1.3.1/lib/thin/backends/base.rb:61:in `start'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/thin-1.3.1/lib/thin/server.rb:159:in `start'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/thin-1.3.1/lib/thin/controllers/controller.rb:86:in `start'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/thin-1.3.1/lib/thin/runner.rb:185:in `run_command'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/thin-1.3.1/lib/thin/runner.rb:151:in `run!'
from /opt/elasticbeanstalk/lib/ruby/gems/1.9.1/gems/thin-1.3.1/bin/thin:6:in `'
from /opt/elasticbeanstalk/bin/thin:19:in `load'
from /opt/elasticbeanstalk/bin/thin:19:in `'

Ultimately when I hopped into #eventmachine to ask about it, Defusal suggested upgrading eventmachine from the last stable version (which is very, very old) to the newest beta.

[email protected]# /opt/elasticbeanstalk/bin/gem uninstall eventmachine
You have requested to uninstall the gem:
eventmachine-0.12.10
thin-1.3.1 depends on [eventmachine (>= 0.12.6)]
If you remove this gems, one or more dependencies will not be met.
Continue with Uninstall? [Yn] Y
Successfully uninstalled eventmachine-1.0.0.rc.4
[email protected]# /opt/elasticbeanstalk/bin/gem install eventmachine --pre
Building native extensions. This could take a while...
Successfully installed eventmachine-1.0.0.rc.4
1 gem installed
Installing ri documentation for eventmachine-1.0.0.rc.4...
Installing RDoc documentation for eventmachine-1.0.0.rc.4...
[email protected]#

I did that, and hostmanager started up perfectly!

I personally updated the rest of the gems as well, when I was troubleshooting the error with the ssl/crypto libraries, so I don’t know if there are other “gotchas” left if you don’t do that. Your mileage may vary.

At that point, I built a new AMI off of that instance, and once it was done, I set it as a custom AMI in Elastic Beanstalk. It chewed on it a bit, swapping instances around and such, and loaded up fine with the sample AWS Tomcat WAR. Hurray! (Edit: I should also clarify that we’ve been using this image since with no problems on our real webapps, so it’s not just the sample WAR that works.)

I hope this helps some other people, because when I was doing my research on the subject, I couldn’t find anything from anyone else out there about this (outside of being stumped a few steps before what blocked me). I want to thank my work for allowing me to detail the process that I developed while working for them and contribute that knowledge back to the community which has helped us in so many ways.

General Tech Update

I’ve been really bad about keeping this blog up to date, but I’m going to try to post a little bit more frequently. Originally I started it as just a place for me to post things that I was coding, but I’d like to expand that to various different tech-related things that I’m working on or working against.

One of the things that I’ve been playing with lately is the Roku. I picked up one of these (an XD model) on sale at Amazon for around $60, and it’s been great. See, a while back I looked at how much I was paying Comcast and TiVo and the headaches involved, and determined that it wasn’t worth keeping the cable subscription. With the advent of iTunes and Amazon Unbox, I don’t see any need to pay for regular cable service when I can just buy the episodes I want instead. Especially with Netflix to fill in the gaps. The Roku works perfectly with Unbox and Netflix, which are the two primary ways I get content now (I occasionally buy something via iTunes, but in general I try to avoid them– their policy to not allow redownloading is something I don’t like. Video takes up a lot of space, and I’d much rather stream it than have to deal with storing it and backing it up myself. Word on the street is that Apple is looking to change that policy, but they need buy-in from the industry, so we’ll see what happens there.)

However, I do have some local content that I like to play on the TV, and I’ve been looking at the best ways to do that. Before I used to hook up my laptop and play files that way; it worked but now the HDMI port is used by the Roku and I don’t want to deal with plugging and unplugging equipment all the time. Enter two solutions: Gabby and Plex. Both have you set up a local media server and add a channel on the Roku which streams content from the media server. Both Gabby and Plex do transcoding (I believe both use ffmpeg behind the scenes; I know Plex does) so that you can play more than the few media formats the Roku directly supports. This has been an interesting experience as neither is really stable yet. The Plex Media Server for Windows is pretty new, but seems pretty stable; Gabby’s media server has had more than a few glitches and crashes (and I can’t get it to reliably start at boot time due the way it’s implemented in .NET). The Gabby devs are also the devs behind the Gabby Roku channel, since that’s the prime focus for them, whereas the Plex channel is actually developed by someone outside of the core Plex dev team. So I’ve been using both, and liking both, but so far I’m leaning a little bit more towards the Plex camp. Especially since they just announced the availability of the Plex Media Server for Linux. I got it up and running under my dual-boot box (Win Vista / Ubuntu 11.04), but it’s not working properly with the Roku channel. Not sure if that’s a Linux server issue or a Roku channel issue, but I’m sure it’ll get sorted out in a little bit. That gives me one less reason to boot into Windows. 🙂

Speaking of that, I’ve been spending a bit more under Linux in general of late. I find it’s a lot easier on the days that I’m working from home to have a full Linux environment at my fingertips than to run countless PuTTY sessions. Maybe it’s just in my head, but that’s the way it feels to me. I upgraded to Ubuntu Natty Narwhal a while back, which has the upside of Vim 7.3. The downside is that VPN connections seem to make my entire networking stack act weird. I opened a bug on it during the beta, but it’s lingering in limbo at this point. I definitely notice a difference between that system and my laptop, which is a Win Vista / Ubuntu 10.10 dual-boot.

After seeing a presentation on it at LinuxFest NorthWest, I finally buckled down and configured BackupPC on my NAS box (which runs Ubuntu 10.10). I just used the default Ubuntu packages for it, and spent some time configuring all my various machines to work with it. On the Linux side I just use rsync over ssh, and for the Windows boxes I use rsync via DeltaCopy. (The two things I’ll mention about the latter: you need to specifically allow port 873, aka rsyncd, in Windows Firewall. You also need to enable pings in Win Vista/Win7, which can be done via the command “netsh firewall set icmpsetting 8 enable”. Otherwise, even if DeltaCopy is working and the firewall allows it through, BackupPC won’t connect to the host because it’ll think it’s offline.) I’m still fine-tuning things, but it’s backing up around 350GB or so of data for me across like 8-10 machines. I thought for a bit about how to handle my dual-booting computers, and decided to just give them different names and identities between the different OSes. So my main machine is storm under Win Vista and lightning under Ubuntu; my laptop is typhoon under Vista and whirlwind under Ubuntu. I’ll have to figure out something for my netbook once I get the partitions straightened out on that so I can dual-boot that between 7 and Ubuntu.

I’ve also been doing some coding lately, specifically on StormChat. I’ve been working on it for years, but I’ve finally gotten to the point where I have someone else running it on their servers, instead of the only people running it being the people that I maintain it for. Granted, I did install it for Ad Astra, but still… it’s the first install not sitting on my servers. I’m working through the bug list, trying to triage it a bit and get things rolling again. There’s a lot left to be done; right now I wouldn’t really recommend it for anyone that’s not very adventurous or without direct access to me to troubleshoot the things about it that only I know how to fix. I need to work on getting rid of the need for the server, fix/revamp the installer, and fix/revamp the admin console. Those are the three big-ticket items on that project.

I know that my WordPress plugins are really out of date at this point. Some people have reported that some of them still work, which is great to hear. I hope to find the time to sit down in the future and revisit each of them and determine if they need updates or if they can be retired.

I’ve also been playing around with RIFT lately; WoW just hasn’t been grabbing me lately. I tried getting it to run under Wine, but haven’t had much luck yet. If I do get it running, I’ll post how I did so.