DHCP failover / load balancing (and synchronization) with CentOS 6

DHCP is a wonderful piece of software. It keeps our networks running smoothly. For small networks, probably 100 machines or so, one server is enough, but to larger networks, is not a bad idea to have another one, just in case the firts one fails or the load is just to much..

DHCP has some configurations for load balancing and failover, the – failover declaration – that allows us to configure it.

The Servers:

Primary IP address: 10.1.2.1

Secondary IP address: 10.1.2.2

To keep things simple, you can create a new file, and then just insert it in the global dhcpd.conf (configuration below)

Primary DHCP server

Open a new file and put the following lines in it:

vi /etc/dhcp/dhcpd.failover

# Failover specific configurations

failover peer “dhcp” {
primary;
address 10.1.2.1;
port 647;
peer address 10.1.2.2;
peer port 647;
max-response-delay 60;
max-unacked-updates 10;
mclt 600;
split 128;
load balance max seconds 3;
}
Now, in the secondary DHCP server, just to the same:
Secondary Server
failover peer “dhcp” {
secondary;
address 10.1.2.2;
port 647;
peer address 10.1.2.1;
peer port 647;
max-response-delay 60;
max-unacked-updates 10;
load balance max seconds 3;
}
After creating the files, just add the configuration to the global config file (in both servers), somewhere before (groups | share networks | host) definitions
include “/etc/dhcp/dhcpd.failover”;
Now, to the definitions.
To take advantage of the new definitions, we need to create a pool. You can put it in any (subnet| shared-network|etc..) declaration:
 
 
network 10.1.2.0 netmask 255.255.0.0 {
option routers 10.1.2.254;
option subnet-mask 255.255.0.0;
option broadcast-address 10.1.255.255;
pool {
failover peer “dhcp”;
range 10.1.2.1 10.1.254.254;
}
}
 
Note: If you have static declarations (i have all my clients in static declarations), to avoid warnings in the log about dynamic and static mappings, reduce the range to only one client. It’s mandatory the range declaration
IMPORTANT: Is of utmost importance to have both servers with the same date and time. If they are not, DHCP will complain and the secondary server (the whole server) will not work well… You can accomplish this with ntpdate
If you are getting this messages in syslog:
Failover CONNECT to dhcp rejected: Connection rejected, time mismatch too great.

Then the time is not the same.

If you want to know more about the configuration parameters in the failover declarations, go to the ipamworldwide web site
Synchronization
DHCP doesn’t have a synchronization mechanism (as far as i know – please correct me if i’m wrong), so changes you’ll make to the primary server will not reflect in the secondary server. This could be done using scripting or yourself manually copying the changed file over to the secondary server. But sometimes, in the heat of the moment, because something important happened or someone is waiting for your attention, you forget…
There’s a small program that can accomplish the synchronization without you even remember that you must copy the files…
iwatch is a small program that monitors wherever you want (files, directories) and upon changes, it can perform several actions.
A few months ago i wrote about iwatch and how’s installed and configured (portuguese), but i’ll replicate the steps here, using the DHCP files has an example.
The CentOS minimal installation doesn’t have rsync and wget installed, so we need to install those.
yum install rsync wget
 
Note: The steps above are only required in the primary server. The changes are made here and then replicated to the secondary server.
Install the rpmforge repositories. You can get the rpm and instructions here
Install the required perl packages
yum install perl-Event perl-Mail-Sendmail perl-XML-SimpleObject perl-XML-Simple perl-Linux-Inotify2
 
After installing, we can finally install iwatch.
Download it from sourceforge
After download, untar it:
tar -zxvf iwatch-0.2.2.tgz
A new directory is created
cd iwatch
 
In there, you’ll find a few files.
Let’s copy the files to the proper places
cp iwatch /usr/local/bin/
cp iwatch.xml /etc/
cp iwatch.dtd /etc/
A few considerations before continuing:
We want to synchronize changes in the DHCP configurations, so, we’ll monitor the /etc/dhcp directory for:
  • Creation of files
  • changes in files
  • deleting of files
  • add an exception for dhcpd.failover (those are different in the servers – depending of primary or secondary server)
Now that we know what we want, let’s proceed:
Before we edit the configuration file, so we can execute iwatch as a daemon, let’s execute it in command line and edit a file so we can see what’s happening. Open two terminals: One will be used to execute iwatch and some arguments, the other will be to edit a file.
First terminal:
Execute iwatch and see some output:
/usr/local/bin/iwatch -e modify,create,close_write -c “touch /tmp/someaction” -r -v /etc/dhcp/
 

output:
Watch /etc/dhcp
Watch /etc/dhcp/Configs
Watch /etc/dhcp/dhclient.d

 
Second terminal
Let’s edit a file in the watched directory and see what’s happening in terminal 1. You can just open it, no changes, but save the file and watch the output in Terminal 1.
vi /etc/dhcp/dhcpd.conf
In terminal 2, you’ll see:
[14/Mar/2012 16:18:34] IN_CREATE /etc/dhcp/Configs/.dhcp.vlan.swp
[14/Mar/2012 16:18:34] * Command: touch /tmp/someaction
[14/Mar/2012 16:18:34] IN_CREATE /etc/dhcp/Configs/.dhcp.vlan.swx
[14/Mar/2012 16:18:34] * Command: touch /tmp/someaction
[14/Mar/2012 16:18:34] IN_CLOSE_WRITE /etc/dhcp/Configs/.dhcp.vlan.swx
[14/Mar/2012 16:18:34] * Command: touch /tmp/someaction

Now that we saw it working, let’s configure the daemon part.

Edit the file /etc/iwatch.xml. The file syntax is XML. Here’s an example of my configuration.
You can read more in iwatch sourceforge page.
<?xml version=”1.0″ ?>
<!DOCTYPE config SYSTEM “/etc/iwatch.dtd” ><config charset=”utf-8″>

<guard email=”informatica@ulscb.min-saude.pt” name=”IWatch”/>

<watchlist>

<title>DHCP Sync</title>
<contactpoint email=”sysadmin@domain.com” name=”Administrator”/>
<path type=”recursive” syslog=”on” alert=”off” events=”create,delete,close_write” exec=”/root/scripts/syncFiles”>/etc/dhcp</path>
<path type=”regexception”>b4913b</path>
<path type=”exception”>/etc/dhcp/dhcpd.failover</path>
<path type=”exception”>/etc/dhcp/dhclient.d</path>

<path type=”regexception”>.*.swp*</path>

<path type=”regexception”>.*~</path>

</watchlist>

</config>

Now, edit that file and make the changes you want

I’ve added a few exceptions, because there are files i don’t need to sync.

Also, vi creates a few temporary files (directory 4913 and backups with ~ | swp extensions) when you’re editing, and those don’t mind.

We are not also using modify, because if a file is closed with write, it was modified, right ?

The exec  parameter tells iwatch what to do when any of the events occurs. I have a script (syncFiles) that synchronizes with the secondary server and sends and email

#!/bin/bash
# Script to synchronized dhcp changes
# This script will be called by iwatch
# DO NOT EXECUTE THIS SCRIPT – IT WILL BE EXECUTED AUTOMATICALLY
# 15/12/2011
log=”/tmp/synclog.log”
echo “Syncing dhcp from primary server to secondary server” >> $log
# Using rsync so it can only copy different files – Low on bandwith/usr/bin/rsync -avz –delete -e ssh /etc/dhcp/ –exclude dhcpd.failover root@secondary:/etc/dhcp >> $log
# Restart the service with the new configurations
ssh root@secondary -C “service dhcpd restart” >> $log
service dhcpd restart >> $log
# Email
if [ -a $log ]; then
mail sysadmin@domain.com -s “Sync dhcp ” < $log
rm -f $log
fi

I use rsync to perform the copy. I exclude dhcpd.failover because the files are not the same and they depend on the server (primary or secondary)

Notes: A few security issues. iwatch is executed with root privileges – it’s started by /etc/rc.local

If you do nothing, every time the script is executed, you’ll have to give the root password of the secondary server. You can prevent this (if you want) by adding the ssh key to the authorized keys and have a password-less ssh configuration between those two servers (using only keys)

Now, just put iwatch executing when the machine start:

vi /etc/rc.local
# Exec iwatch
/usr/local/bin/iwatch -d

Execute iwatch as daemon

/usr/local/bin/iwatch -d

Now you have a dhcp failover instalation and synchronization

Hope it helps anyone

References

http://www.lithodyne.net/docs/dhcp/dhcp-4.html

http://www.madboa.com/geek/dhcp-failover/

http://www.ipamworldwide.com/dhcp-failover-a-load-balancing/declarations.html