Recipe: Supervising Daemons With daemontools

Up to yesterday, for my Topic Maps REST server (tmipd) I have used startup and shutdown scripts to control it, quite in the Apache tradition:

/etc/init.d/tmipdctrl start|stop|restart

On start, it would detach from the terminal and would daemonize. That worked so far, but I also wanted an automatic server restart in the case it crashes.

Not to reinvent the wheel, I put my Topic Maps REST server (tmipd) under supervision with daemontools (AsTMa fragment attached).

Directory Setup

On a Debian box you would simply install the core tools

root@monda:~# apt-get install daemontools

but eventually I also installed

root@monda:~# apt-get install daemontools-run

which creates an empty

/etc/service

directory, but also starts the service monitoring at boot time (and after install, of course).

According to the "HOWTO" you are supposed to create a subdirectory for your service and put a run shell script into that:

/etc/service/tmipd
/etc/service/tmipd/run

But since I like to have things tidy, I opted to create the supervision part inside my web service:

/var/www/tmip/supervise
/var/www/tmip/supervise/run

To put that under supervision a simple

root@monda:~# update-service --add /var/www/tmip/supervise/ tmipd
Service tmipd added.

does the trick. You can check whether it is there

root@monda:~# update-service --list

and I also observed my server to be started within seconds.

To get rid of the service altogether, you have to use the exact

root@monda:~# update-service --remove /var/www/tmip/supervise/ tmipd

The Run Script

The shell script to start your service looks fairly straightforward:

#!/bin/sh

LIBS=
TMIPD=/usr/sbin/tmipd

# this should be on one line!!
exec perl ${LIBS} ${TMIPD} --daemonize=no
                           --conffile=/var/www/tmip/etc/tmipd.conf
                           --logfile=/var/log/tmipd.log

The only (subtle) thing to make sure is that you use exec as above. If you omit it, the perl process will be started as separate process. Everything works, but you will notice that any signal you send via

root@monda:~# svc -d /etc/service/tmipd/

(to stop it) will simply not arrive at your process. That is definitely not good if you want to use svc for service control.

When you instead use exec, the Perl process will replace the shell script process. As that was a child process of svscan any signal will be propagated to the child, and so to the tmipd service:

905 ?        Ss     0:00 /bin/sh /usr/bin/svscanboot
  913 ?        S      0:04  \_ svscan /etc/service
 6689 ?        S      0:00  |   \_ supervise tmipd
 6690 ?        S      0:00  |       \_ perl /usr/sbin/tmipd --daem....
  914 ?        S      0:00  \_ readproctitle service errors: .....

Use ps axf, or pstree to see the process tree.

Prepping The Service

To make this all work, your server must NOT daemonize automatically. And it must catch and process signals, such as TERM or HUP, properly.

Operating The Service

Starting and stopping the service is then fun:

root@monda:~# svc -u /etc/service/tmipd/
root@monda:~# svc -d /etc/service/tmipd/

To check the uptime, you use

root@monda:~# svstat /etc/service/tmipd/

If you terminate with

root@monda:~# svc -t /etc/service/tmipd/

supervise will simply send your process a TERM signal:

2010/04/04 10:14:56 tmipd 148 tmipd - received TERM
2010/04/04 10:14:57 tmipd 131 tmipd - logging established
2010/04/04 10:14:57 tmipd 153 tmipd - starting listener

and you will notice an immediate restart of the service.

Very nice.

AttachmentSize
daemontools.atm413 bytes
Posted In