One of the steps required to configure load-balanced cluster as described here was to create an IP alias on the loopback interface. It is needed so that real servers or standby director could see traffic for the virtual IP address.

But when creating a virtual site using BlueQuartz GUI it automatically adds an alias on primary network interface (eth0 in our case) in addition to the alias we already created manually.

Adding a new site using BlueQuartz

Now every server would be configured to respond to packets for virtual IP address 192.168.6.240, which could result in unpredictable behavior.

standby# ip addr sh
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 192.168.6.240/32 brd 255.255.255.255 scope global lo
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:10:e0:05:20:0d brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.2/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0:0

If you have completed all the steps to set-up a cluster you should already have fallback interface alias defined in /etc/sysconfig/network-scripts/ifcfg-lo:0. We no longer need BlueQuartz to manage virtual IP address for us.

In order to prevent BlueQuartz from doing that the following files have to be modified:

...
# handle ip address change
if ($vsite_new->{ipaddr})
{
    # make sure that there is a network interface for the new ip
#    vsite_add_network_interface($cce, $vsite_new->{ipaddr});

    # delete the old interface, this is a no op if another site is using the old ip still
    vsite_del_network_interface($cce, $vsite_old->{ipaddr});
} # end of ip address change specific
...

...
# make sure there is a network interface for this ip
#vsite_add_network_interface($cce, $vsite->{ipaddr});
...
 

This way when you add or modify a virtual site assigned to virtual IP address BlueQuartz won’t add its address to one of the network interfaces because we already took care of it.

Server is configured as director to be a part of high availability and load balancing cluster as described here:
http://www.ultramonkey.org/3/topologies/sl-ha-lb-eg.html

CentOS 4 with Bluequartz from Nuonce.net is used as operating system.

PROBLEM:

As a part of configuration an alias for loopback interface had to be created so server can accept packets for the virtual service:

DEVICE=lo:0
IPADDR=192.168.6.240
NETMASK=255.255.255.255
NETWORK=192.168.6.0
BROADCAST=192.168.6.255
ONBOOT=yes
NAME=loopback

When heartbeat starts on director, it removes the alias on loopback interface lo and creates virtual IP address on interface eth0.

Here is how all interfaces look like before problem occurs:

[root@server ~]# ip addr show
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:19:db:df:15:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.1/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0

However after certain amount of time the alias with virtual IP appears on loopback interface again:

[root@server ~]# ip addr sh
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
    inet 192.168.6.240/32 brd 192.168.6.255 scope global lo:0
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:19:db:df:15:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.1/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0

NOTE:

This problem only affects servers acting as load balancers / directors. Real servers running on CentOS / Bluequartz are not affected.

CAUSE:

It turns out this problem is caused by swatch which is utilized by BlueQuartz. Every 15 minutes it checks the status of all interfaces that have ONBOOT=yes and brings them up if they are down.

See /usr/sausalito/swatch/bin/am_network.sh for more information.

SOLUTION:

In order to prevent swatch from manipulating our virtual IP address, you have to turn off ONBOOT parameter since that what it looks for.

...
ONBOOT=no
...

Then disable heartbeat from starting automatically and make sure it is off.

[root@server ~]# chkconfig heartbeat off
[root@server ~]# chkconfig --list heartbeat | grep heartbeat
heartbeat       0:off   1:off   2:off   3:off   4:off   5:off   6:off

Finally bring the alias and heartbeat up in rc.local. This script file is run once, before all other scripts have run but before the logon prompt appears.

...
ifup lo:0
/etc/init.d/heartbeat start

LINKS:

1. Streamline High Availability and Load Balancing
2. How to install Ultra Monkey LVS in a 2-Node HA/LB Setup on CentOS/RHEL4