One of the steps required to configure load-balanced cluster as described here was to create an IP alias on the loopback interface. It is needed so that real servers or standby director could see traffic for the virtual IP address.

But when creating a virtual site using BlueQuartz GUI it automatically adds an alias on primary network interface (eth0 in our case) in addition to the alias we already created manually.

Adding a new site using BlueQuartz

Now every server would be configured to respond to packets for virtual IP address 192.168.6.240, which could result in unpredictable behavior.

standby# ip addr sh
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 192.168.6.240/32 brd 255.255.255.255 scope global lo
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:10:e0:05:20:0d brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.2/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0:0

If you have completed all the steps to set-up a cluster you should already have fallback interface alias defined in /etc/sysconfig/network-scripts/ifcfg-lo:0. We no longer need BlueQuartz to manage virtual IP address for us.

In order to prevent BlueQuartz from doing that the following files have to be modified:

...
# handle ip address change
if ($vsite_new->{ipaddr})
{
    # make sure that there is a network interface for the new ip
#    vsite_add_network_interface($cce, $vsite_new->{ipaddr});

    # delete the old interface, this is a no op if another site is using the old ip still
    vsite_del_network_interface($cce, $vsite_old->{ipaddr});
} # end of ip address change specific
...

...
# make sure there is a network interface for this ip
#vsite_add_network_interface($cce, $vsite->{ipaddr});
...
 

This way when you add or modify a virtual site assigned to virtual IP address BlueQuartz won’t add its address to one of the network interfaces because we already took care of it.

Server is configured as director to be a part of high availability and load balancing cluster as described here:
http://www.ultramonkey.org/3/topologies/sl-ha-lb-eg.html

CentOS 4 with Bluequartz from Nuonce.net is used as operating system.

PROBLEM:

As a part of configuration an alias for loopback interface had to be created so server can accept packets for the virtual service:

DEVICE=lo:0
IPADDR=192.168.6.240
NETMASK=255.255.255.255
NETWORK=192.168.6.0
BROADCAST=192.168.6.255
ONBOOT=yes
NAME=loopback

When heartbeat starts on director, it removes the alias on loopback interface lo and creates virtual IP address on interface eth0.

Here is how all interfaces look like before problem occurs:

[root@server ~]# ip addr show
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:19:db:df:15:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.1/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0

However after certain amount of time the alias with virtual IP appears on loopback interface again:

[root@server ~]# ip addr sh
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
    inet 192.168.6.240/32 brd 192.168.6.255 scope global lo:0
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:19:db:df:15:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.1/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0

NOTE:

This problem only affects servers acting as load balancers / directors. Real servers running on CentOS / Bluequartz are not affected.

CAUSE:

It turns out this problem is caused by swatch which is utilized by BlueQuartz. Every 15 minutes it checks the status of all interfaces that have ONBOOT=yes and brings them up if they are down.

See /usr/sausalito/swatch/bin/am_network.sh for more information.

SOLUTION:

In order to prevent swatch from manipulating our virtual IP address, you have to turn off ONBOOT parameter since that what it looks for.

...
ONBOOT=no
...

Then disable heartbeat from starting automatically and make sure it is off.

[root@server ~]# chkconfig heartbeat off
[root@server ~]# chkconfig --list heartbeat | grep heartbeat
heartbeat       0:off   1:off   2:off   3:off   4:off   5:off   6:off

Finally bring the alias and heartbeat up in rc.local. This script file is run once, before all other scripts have run but before the logon prompt appears.

...
ifup lo:0
/etc/init.d/heartbeat start

LINKS:

1. Streamline High Availability and Load Balancing
2. How to install Ultra Monkey LVS in a 2-Node HA/LB Setup on CentOS/RHEL4

When starting ColdFusion for the first time connector wizard cannot configure web server correctly.

Starting ColdFusion 8...
The ColdFusion 8 server is starting up and will be available shortly.
======================================================================
Running the ColdFusion 8 connector wizard
======================================================================
Configuring the web server connector (Launched on the first run of the ColdFusion 8 start script)
Running apache connector wizard...
ColdFusion 8 not started, will retry connector 12 more time(s)
ColdFusion 8 not started, will retry connector 11 more time(s)
ColdFusion 8 not started, will retry connector 10 more time(s)
ColdFusion 8 not started, will retry connector 9 more time(s)
ColdFusion 8 not started, will retry connector 8 more time(s)
ColdFusion 8 not started, will retry connector 7 more time(s)
ColdFusion 8 not started, will retry connector 6 more time(s)
ColdFusion 8 not started, will retry connector 5 more time(s)
ColdFusion 8 not started, will retry connector 4 more time(s)
ColdFusion 8 not started, will retry connector 3 more time(s)
ColdFusion 8 not started, will retry connector 2 more time(s)
ColdFusion 8 not started, will retry connector 1 more time(s)
The connector wizard has not been able to contact the local ColdFusion 8, connector installation aborted.
======================================================================
ColdFusion 8 has been started.
ColdFusion 8 will write logs to /opt/coldfusion8/logs/cfserver.log
======================================================================

java.net.UnknownHostException: foo.mydomain.com: foo.mydomain.com

05/06 07:59:07 info Macromedia JRun 4.0 (Build 108487)
05/06 07:59:08 error Could not connect to any JRun/ColdFusion servers on host localhost.
Possible causes:
o Server not running
  -Start Macromedia JRun4 or ColdFusion 8 server
o Server running
  -JNDI listen port in jndi.properties blocked by TCP/IP filtering or firewall
   on server
  -host restriction in security.properties blocking communication with server

CAUSE:

This behavior could be caused by using non-resolvable host name (foo.mydomain.com in our example) along with /etc/hosts file having incorrect permissions.

[root@foo ~]# ls -la /etc/hosts
-rw-r-----  1 root root 1514 May  4 18:04 /etc/hosts

SOLUTION:

To resolve this issue /etc/hosts has to have 0644 access permission:

[root@foo ~]# chmod 0644 /etc/hosts
[root@foo ~]# ls -la /etc/hosts
-rw-r--r--  1 root root 1514 May  4 18:04 /etc/hosts

On distributions with BlueQuartz (Strongbolt, BlueOnyx, etc) it could be caused by /usr/sausalito/handlers/nuonce/fixhosts/fixhosts script (see this post). To prevent this script from resetting permissions again, add the following line at the end of the script:

/bin/chmod 0644 /etc/hosts