Having installed RabbitMQ and Sensu in the last post we now need a way to visualize all the data we have gathered. For this purpose we start with Graphite to plot continuous data.

Doing so was quite an adventure since it was hard to find an adequate installation procedure. There are no official RPMs out there, so I had to built my own.

Again, with the hint of a colleague, I was able to get the whole thing done with the following three part series blog post

First, I had to get all the dependent software, for building the RPMs and then installing Graphite.

yum install -y gcc zlib-devel curl curl-devel openssl rpm-build gcc-c++ rpm-build python python-ldap python-memcached python-sqlite2 pycairo python-twisted Django django-tagging bitmap bitmap-fonts python-devel glibc-devel gcc-c++ openssl-devel python-zope-interface httpd memcached mod_wsgi

On my virtual machine, I also had to get the wget.

yum install -y wget

An Experiment

Although the blog post used version 0.9.9, I have tried my luck with version 0.9.10. You might want to skip these steps, since it did not work out and I have reverted to 0.9.9 anyways.

As normal user do the following:

cd
mkdir rpmbuild
mkdir rpmbuild/SOURCES
cd rpmbuild/SOURCES
wget http://launchpad.net/graphite/0.9/0.9.10/+download/whisper-0.9.10.tar.gz
wget http://launchpad.net/graphite/0.9/0.9.10/+download/carbon-0.9.10.tar.gz
wget http://launchpad.net/graphite/0.9/0.9.10/+download/graphite-web-0.9.10.tar.gz
cp carbon-0.9.10.tar.gz carbon-0.9.10.tar.gz.orig
cp graphite-web-0.9.10.tar.gz graphite-web-0.9.10.tar.gz.orig
cp whisper-0.9.10.tar.gz whisper-0.9.10.tar.gz.orig
tar -zxvf whisper-0.9.10.tar.gz
cd whisper-0.9.10
python setup.py bdist_rpm
rpm -ivh dist/whisper-0.9.10-1.src.rpm
cd ..
cp whisper-0.9.10.tar.gz.orig whisper-0.9.10.tar.gz
cd ../SPECS/

Edit the spec if you want to add a suffix to the end of the release information and to add a description

vi whisper.spec
rpmbuild -ba whisper.spec
sudo yum --nogpgcheck localinstall -y ../RPMS/noarch/whisper-0.9.10-1.noarch.rpm

Now, let’s do the same for carbon and graphite

cd ~/rpmbuild/SOURCES/
tar -zxvf graphite-web-0.9.10.tar.gz
cd graphite-web-0.9.10
python setup.py bdist_rpm

Here, I got an error saying:

error: error in 'post_install' option: 'distro/redhat/misc/postinstall' does not exist or is not a file

So, I had to find all files containing “distro” by grep -iR 'distro' *. The result was setup.cfg and MANIFEST.in where I changed “distro” to “dist”. Ran the command again and it worked.

But then I have realized, that the directory “dist” did not exist at all. Clever me! What did I do then? Removed everything and continued with the 0.9.9 versions of whisper, carbon and graphite.

Okay, that was the point where I have removed everything and continued with version 0.9.9.

Building the RPMs and installing Graphite

cd ~/rpmbuild/SOURCES
wget http://launchpad.net/graphite/0.9/0.9.9/+download/whisper-0.9.9.tar.gz
wget http://launchpad.net/graphite/0.9/0.9.9/+download/carbon-0.9.9.tar.gz
wget http://launchpad.net/graphite/0.9/0.9.9/+download/graphite-web-0.9.9.tar.gz
cp carbon-0.9.9.tar.gz carbon-0.9.9.tar.gz.orig
cp graphite-web-0.9.9.tar.gz graphite-web-0.9.9.tar.gz.orig
cp whisper-0.9.9.tar.gz whisper-0.9.9.tar.gz.orig
cd ~/rpmbuild/SOURCES/
tar -zxvf whisper-0.9.9.tar.gz
cd whisper-0.9.9
python setup.py bdist_rpm
rpm -ivh dist/whisper-0.9.9-1.src.rpm
cd ..
cp whisper-0.9.9.tar.gz.orig whisper-0.9.9.tar.gz
cd ../SPECS/
rpmbuild -ba whisper.spec
sudo yum --nogpgcheck localinstall -y ../RPMS/noarch/whisper-0.9.9-1.noarch.rpm
cd ~/rpmbuild/SOURCES/
tar -zxvf graphite-web-0.9.9.tar.gz
cd graphite-web-0.9.9
python setup.py bdist_rpm
rpm -ivh dist/graphite-web-0.9.9-1.src.rpm
cd ../
cp graphite-web-0.9.9.tar.gz.orig graphite-web-0.9.9.tar.gz
cd ../SPECS/
rpmbuild -ba graphite-web.spec
sudo yum --nogpgcheck localinstall -y ../RPMS/noarch/graphite-web-0.9.9-1.noarch.rpm
cd ~/rpmbuild/SOURCES/
tar -zxvf carbon-0.9.9.tar.gz
cd carbon-0.9.9
python setup.py bdist_rpm
rpm -ivh dist/carbon-0.9.9-1.src.rpm
cd ..
cp carbon-0.9.9.tar.gz.orig carbon-0.9.9.tar.gz
cd ../SPECS/
rpmbuild -ba carbon.spec
sudo yum --nogpgcheck localinstall -y ../RPMS/noarch/carbon-0.9.9-1.noarch.rpm

Let’s continue with part two of the blog post series where we setup graphite.

Configuration

It starts by installing some dependencies, but I have skipped that, because they are a partial repetition of what we have already installed. Then all RPMS will be installed, but we have already done that as well.

We continue with setting up carbon and graphite-web configuration files.

cd /opt/graphite/conf/
sudo cp graphite.wsgi.example graphite.wsgi
sudo cp storage-schemas.conf.example storage-schemas.conf
sudo cp carbon.conf.example carbon.conf
cd ../webapp/graphite
sudo cp local_settings.py.example local_settings.py
sudo vim local_settings.py

Where I have added/uncommented/edited the following lines:

TIME_ZONE = 'Europe/Berlin'
MEMCACHE_HOSTS = ['127.0.0.1:11211']

Let’s create the Django database with:

sudo python /opt/graphite/webapp/graphite/manage.py syncdb

You will be asked to create a superuser. I have answered that with a ‘no’.

Apache and wsgi is next. I already had those two, but if you want to install it do `sudo yum install -y httpd wsgi.

Next create the file /etc/httpd/conf.d/graphite.conf as descibed in the blog post.

Make /etc/httpd/conf.d/wsgi.conf look like this:

LoadModule wsgi_module modules/mod_wsgi.so
WSGISocketPrefix /var/run/wsgi

Now, set some rights:

chown -R apache:apache /opt/graphite/storage/

Almost done. Create a start/stop-script /etc/init.d/carbon-cache as described in the post and make it executable:

sudo chmod +x /etc/init.d/carbon-cache

Start everything up as root:

service memcached start
service carbon-cache start
service httpd start

And finally: Test if the webapp works by pointing your browser to http://localhost:9080. My ports are a bit different since I am using vagrant with port-forwarding.

Nope, did not work. In /var/log/httpd/graphite_error_log, I got the following:

IOError: [Errno 13] Permission denied: '/opt/graphite/storage/log/webapp/info.log'

The file was not there at all … Anyways, that had something to do with permissions. After some googling I ran into

chown -R apache:apache /opt/graphite/storage

That did it! It worked! Strange, because I was sure I have already did that.

Feeding Data into Graphite

Let’s get some data and visualize it. How this is done is explained in the last and third part of the excellent blog post.

Graphite is listening on ports 2003 and 2004 for incoming data. You can check that by running nmap -v -sT localhost. The format of the data is object value timestamp.

Am not repeating the example from the blog post here, but showing another example from the sensu community plugin:

$ gem install sensu-plugin --no-rdoc --no-ri
$ cd /etc/sensu/plugins
$ wget https://raw.github.com/sensu/sensu-community-plugins/master/plugins/system/cpu-metrics.rb
$ chmod +x cpu-metrics.rb

Plus, I had to add the following section to /etc/sensu/conf.d/client.json:

{
	...
	"checks": {
		...
		"cpu_metrics": {
			"type": "metric",
    		"handlers": ["graphite"],
    		"command": "/etc/sensu/plugins/cpu-metrics.rb --scheme stats.:::name:::",
    		"interval": 60,
    		"subscribers": [ "webservers" ]
		}
	}
}

In Graphite, I was then selecting a few of those measurements to let them be visualized in the dashboard:

And there we finish this article, whereas the original blog post still goes on with more ways on how to gather data and feed it into graphite. Check it out!

Done for today.