In my last post we installed RabbitMQ with Puppet on our CentOS 6.2 box. Today, we continue by installing Sensu, a pretty cool monitoring framework that a lot of people are using these days.
This post follows the Sensu installation guide and points to my GitHub repository where I have committed my Puppet configuration. For the first time, I am not repeating how to build up your box with Vagrant and so forth—just check my previous posts.
For a change, I will not use the command line beyond adapting and extending my Puppet configuration.
So, you are logged into your box as root? Fine. Ready to go!
Before We Get Started
There is a leftover from our RabbitMQ installation. We need to remove a directory and create some symbolic links.
cd /etc/puppet
rm -rf manifests
ln -s /vagrant/manifests
ln -s /vagrant/modules
You need that in case you want to invoke your provisioning run from within the virtual machine with puppet apply. If not, you can always run vagrant provision from outside your box on your host system.
If you start with my Vagrant box, you also need to install git==:
yum -y install git
Generating and Installing SSL Certificates
There is another leftover. I did not add the SSL certificates to the Puppet configuration. Since I already did the manual procedure here, the rest is straightforward.
First, generate the certificates with
git clone git://github.com/joemiller/joemiller.me-intro-to-sensu.git
cd joemiller.me-intro-to-sensu/
./ssl_certs.sh clean
./ssl_certs.sh generate
Then I created the following directory
/vagrant/modules/rabbitmq/files/etc/rabbitmq/ssl/
and copied the following files into it
cp server_key.pem /vagrant/modules/rabbitmq/files/etc/rabbitmq/ssl/
cp server_cert.pem /vagrant/modules/rabbitmq/files/etc/rabbitmq/ssl/
cp testca/cacert.pem /vagrant/modules/rabbitmq/files/etc/rabbitmq/ssl/
Now, we need to extend our Puppet configuration in the file
/vagrant/modules/rabbitmq/manifests/init.pp
For a test I did only one file.
file { '/etc/rabbitmq/server_key.pem':
source => '/etc/puppet/modules/rabbitmq/files/etc/rabbitmq/ssl/server_key.pem',
owner => 'root',
group => 'root',
mode => '644',
notify => Service['rabbitmq-server'],
require => Package['rabbitmq-server'],
}
We also need to tell Puppet that we need the directory /etc/rabbitmq/ssl, since it does not exist yet.
file { '/etc/rabbitmq/ssl/':
source => '/etc/puppet/modules/rabbitmq/files/etc/rabbitmq/ssl/',
owner => 'root',
group => 'root',
mode => '644',
}
Also add this file resource in the require section of the service resource.
Let’s try our new configuration without changing anything by providing the --noop parameter.
cd /etc/puppet
puppet apply --verbose manifests/site.pp --noop
Then I added the other files as well and did another test run. Everything went fine, so I did a real provisioning run.
One more thing we need to do for RabbitMQ is provide a config file that tells RabbitMQ to use SSL and where to find those certificates.
vim /etc/puppet/modules/rabbitmq/files/etc/rabbitmq/rabbitmq.conf
and paste the following into it
[
{rabbit, [
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/server_cert.pem"},
{keyfile,"/etc/rabbitmq/ssl/server_key.pem"},
{verify,verify_peer},
{fail_if_no_peer_cert,true}]}
]}
].
Now add another file resource entry for this configuration file to your init.pp of your Puppet RabbitMQ module and provision it.
To make sure, check our target directory.
You know what? That thing really works! We did this with one server, but we could easily do it with many more servers. It would be no deal at all.
After all this, I committed everything and pushed it up to GitHub on my host system.
git add.
git commit -m "Added the use of SSL to RabbitMQ"
git push origin master
Sensu Puppet Configuration
Finally I added a new module for Sensu to my Puppet configuration and included it in my nodes.pp, which now looks like this
node dev_ops_central {
include cron
include apache
include rabbitmq
include redis
include sensu
}
Then I worked through the installation guide, including the how-to for adding a check and adding a handler, and translated that into Puppet configuration. I’m not going to repeat everything here, since now it is your turn to work through the Puppet code.
Finally I was able to fire up everything, but it did not work at first. When stopping cron, I did not get any notification in my Sensu frontend. As it turned out, I had some issues with my existing RabbitMQ installation. It was not using SSL, because the config file had the wrong name. It is not rabbitmq.conf but rabbitmq.config. It took me a while to figure that out, with rabbitmqctl status showing that it was ignoring my rabbitmq.conf and using its own configuration.
In the meantime, while writing and polishing this post, my Sensu code has been refactored by @linuxaddicted. Also, he added some RSpec tests! Perfect! Thanks a lot. More on this RSpec thing soon.
Beware of the Brownfield
For installing the sensu-plugin with
gem install sensu-plugin
I first had to install the Ruby development libraries with
yum install ruby-devel
I will make that part of my box and not deploy it separately.
Additionally, I had to install gcc. Not an option on potential production boxes.
Feature-Branching with Git
For the first time I used a feature branch called sensu, since I did not want to commit half-baked code to my master branch. I like the idea of Continuous Delivery, where feature branches are managed in a way that supports incremental, releasable updates.
Here is the “process” of how to create a branch with git, add your code, commit it and push it to GitHub.
git checkout -b sensu
git branch
git commit -am "my message"
git push origin sensu
When I was sure everything worked fine, I merged everything back to the master branch and removed the branch sensu locally and remotely on GitHub.
git checkout master
git merge sensu
Now, there is no need to keep the sensu branch, so I removed it locally and remotely with
git branch -d sensu
git push origin:sensu
Having done this work, it was a good time to tag our master branch.
git tag provisioning_sensu_with_puppet
git push --tags
Wrapping It Up
We nearly have a complete automation from box creation to provisioning RabbitMQ and Sensu with Puppet. We can now extend Sensu with more checks and handlers to improve our monitoring and alerting mechanisms. But this is just the beginning of the story. There is much to explore and learn. Stay tuned. For now…
Done for today!