Say you are configuring SSL for your website and after enabling mod_ssl, creating self signed certificates and adding necessary configuration options to your virtual host definition you are still getting errors of the sort:

(98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
no listening sockets available, shutting down
Unable to open logs

in your logs, what do you do? You check who is using the port of course,

$ sudo lsof -i :443
$

But that doesn't lead you anywhere because, guess what, nobody is using port 443.

So what is the problem? It might be because there multiple definitions of the same type. In my case it was because of this line:

Listen 443

which got executed twice. One problem of Apache is that the configuration files are all over the place. It used to be that many configuration options were in the same apache.conf file before but there's a restructuring going on and newer releases emphasize this. You might find old articles and when you use them you can get in these kinds of troubles.

Luckily these problems are not that hard to solve. Just be on the lookout for extra definitions and try to understand which declaration is where and does what. You'll avoid most problems by doing that.

We recently started using duplicity for backups. Duplicity is a really nice backup utility which makes it a piece of cake to take full and incremental backups. It uses rsync in the background but adds a whole lot of clever features as well. One can take full backups once a month and then go on to use incremental backups every night so that less resources are used. Duplicity also has many interfaces for the backup service, it can use ssh or ftp or even Amazon's S3 service to store the backups on the cloud.

Having an easy way to restore from the backups is the other side of taking backups. With duplicity this is handled easily as well because you can say "get me this file/directory from 5 days ago". If something goes wrong you can easily restore the files and go on with your work with minimum downtime. Note that backups are not to be used in place of redundancy. In redundancy you can have more than one server doing the same stuff, having the same files but they will both suffer from a failure in the application. So for example, if one of your customers accidentally deletes his account then your redundant servers will not question the action and both will lose all the account info. When that happens you would want your backups to be there.

One trouble with backups is that they are Disk/CPU intensive. This could cause problems because you will generally want to backup important stuff on live servers. There could be workarounds to this but not all the time. For example, if you want to backup your database and you are using database replication, then you can take backups from the slave and not from the live server. But for most other backup types you will not have this flexibility. What do you do with backup processes on live servers?

Two system utilities that come on Linux are "nice" and "ionice". They allow you to fine tune the priority of your programs. Using these two commands you can lower priorities of programs so that they run only when the CPU or disk has less load. On a live server this could be vital.

On my experiments I've found that I don't need encryption duplicity offers because backup server is also our own box and since backups are sent over ssh there's no possibility of eavesdropping. You can disable encryption with the --no-encryption option. The upside of this is that duplicity will not run gpg and use processing power which encryption needs. In the end the duplicity command I ended up using is similar to this:

$ sudo ionice -c3 nice -n19 duplicity -v 4 --no-encryption backupdir scp://myuser@myserver.com//opt/backupdir

"ionice" with -c3 makes it run only when there are no other processes asking for IO. For "nice", I've used the lowest priority of 19 to run the process. You can check out a running process's "ionice" priority value by using -p switch:

$ ionice -pMYPID

and its nice value by using the "renice" tool.

I love the flexibility Linux offers to developers. Having these tools makes taking backups less of a hassle.

Why is monitoring important?

Monitoring the health of your web applications is of utmost importance because, well, measuring puts you on the map and it tells you how well you are doing. Measuring performance, finding about bottlenecks and getting notified of troubles as fast as possible is something every web developer craves about. In addition to this, by having data/statistics about your application you can quickly determine if a modification you have just made is good or bad. In web development -as in many other fields- there are no silver bullets. A cookbook solution which may as well work for 90% of web applications may not be suitable for your case. Your so-called performance improvement might not affect at all, or worse it might start a new set of problems. This is why you should have a weekly/monthly/yearly data for your web application where you can see the patterns and anomalies.

Nagios (and others?)

We needed a flexible web monitoring solution for JotForm. We had used several other commercial products before but they were never good enough. I knew about Nagios (and Ganglia along with it) from my work at METU Computer Center and had good thoughts about it. Nagios in the meantime have moved on to version 3.2 and got quite complicated. I wanted to use Tobi Oetiker's graphing toolbox (MRTG and RRDtool) with Nagios and have some graphics ready as well. This is because it is always much more easier to visualize data using graphics, finding patterns becomes a less cumbersome task. The trouble with this is that it is not easy to configure Nagios to generate and use these graphs.

There are many add-ons for Nagios for generating graphs and trend analysis. Most of these are either really old or lacking documentation or both. I decided to check out other similar software which might make the experience a bit more pleasant. Surely enough there are many packages/programs that use Nagios and promise an easier solution. Opsview for example packages Nagios with a user interface developed with Catalyst Web Framework. There is also a Nagios fork called Icinga which has a really nice GUI developed using ExtJS (the grid view is noticed from ExtJS directly :)). Icinga seemed to be a good alternative to Nagios as well.

Installing Nagios and pnp4nagios Add-on

So why did I choose Nagios and not for example Opsview or Icinga? Well, I thought I didn't really delve into the complexity of Nagios and so I wouldn't really appreciate the simplicity behind these other tools. Also, I would like to be in charge of my software and since these other tools basically hide Nagios' complexity which I didn't like. It would have been a different story if there were a contender to Nagios that was written from scratch. I looked to the Debian repositories for Nagios but it was too old there (1.6 in repository whereas the current version is 3.2!). So, I downloaded Nagios and went to look for add-ons as well.

I came across pnp4nagios add-on which is relatively new but actively maintained and used this add-on to enhance Nagios. I will now be describing my experiences involving the installation of Nagios 3.2 and pnp4nagios 0.6 on a Debian box. I think most of the instructions will fit into any other Linux distribution so I hope these will be helpful for others as well. I think I also will come back to these instructions whenever a need arises.

Download Nagios core and the Plugins. To install Nagios you need a web server (of course!) and gd library if you also want various CGIs too. Repository name for the gd library on the Debians is libgd2-xpm-dev so go on and install it beforehand. Nagios documentation is hidden behind various pages on the website but you wont be experiencing much trouble if you go through quickstart installation guide. I followed the Fedora guide since there were no instructions specific to Debian that was up to date. Try to use the default locations as it will be handy for configuring add-ons later on.

One problem I had with Nagios was with two CGIs called trends.cgi and statusmap.cgi. They didn't get to compile first time and I had to tinker around the configuration options and the Makefile to see what was going on. I'm not sure what else I did but I think I installed libjpeg-dev and libpng-dev that were not installed before. In the end I had them compiled and copied them to the sbin directory under nagios and they worked.

One command that is really helpful is asking nagios to check the configuration file:

$ sudo /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

After the installation you can see what default settings can show you by going to the web interface. If you are looking for some nice graphs at this point you will be disappointed because there will be no graphs waiting for you. For that, you need to install add-ons and pnp4nagios seems to be one good solution for that. I installed PNP 0.6 according to the nice documentation provided which is really nice because documentation is what sucks for most open source products. I did the configuration and integrated the web front end with nagios.

The configuration verification command that I presented for nagios earlier on has a PNP counterpart that is really useful. Like the nagios version, it checks the configuration files for errors and reports them to you. Very useful for catching that typo you made:

$ sudo /usr/local/pnp4nagios/libexec/verify_pnp_config.pl -m "sync"

After you install PNP, you need to restart the Nagios daemon:

$ sudo /etc/init.d/nagios restart

If everything went according to plan, you should have a working installation of Nagios along with some pretty nice graphs. Now go on to the configuration part and start monitoring other servers.