grep -E -v '^\s*#|^\s*$|^\s*;' file

or

egrep -v '^\s*#|^\s*$|^\s*;' file

After upgrading PostgreSQL from 7.4 to 8.2 it started crashing sporadically with the following entries in the log file:

LOG:  server process (pid 21460) was terminated by signal 11
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing shared memory and semaphores

CAUSE:

There were changes between PostgreSQL releases in functions definition used by pgcrypto module.

According to [1] and [3] in PostgreSQL 8.0 the pgcrypto functions were non-strict and checked for NULLs, in PostgreSQL 8.1 they were made STRICT, in PostgreSQL 8.2 the NULL checks were removed from code.

SOLUTION:

In order to correct the problem these functions are needed to be recreated. Required SQL commands are defined in pgcrypto.sql which is usually located in
/usr/local/pgsql/share/contrib when pgcrypto is installed.

psql -d DBNAME -f /usr/local/pgsql/share/contrib/pgcrypto.sql

Also it is a good idea to add functions to template1 database if you use pgcrypto often. template1 is the default source database name when using CREATE DATABASE command.

REFERENCES:

1. http://marc.info/?l=postgresql-general&m=118794006505296&w=2
2. http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg109917.html
3. http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/contrib/pgcrypto/pgcrypto.sql.in

One of the steps required to configure load-balanced cluster as described here was to create an IP alias on the loopback interface. It is needed so that real servers or standby director could see traffic for the virtual IP address.

But when creating a virtual site using BlueQuartz GUI it automatically adds an alias on primary network interface (eth0 in our case) in addition to the alias we already created manually.

Adding a new site using BlueQuartz

Now every server would be configured to respond to packets for virtual IP address 192.168.6.240, which could result in unpredictable behavior.

standby# ip addr sh
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 192.168.6.240/32 brd 255.255.255.255 scope global lo
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:10:e0:05:20:0d brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.2/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0:0

If you have completed all the steps to set-up a cluster you should already have fallback interface alias defined in /etc/sysconfig/network-scripts/ifcfg-lo:0. We no longer need BlueQuartz to manage virtual IP address for us.

In order to prevent BlueQuartz from doing that the following files have to be modified:

...
# handle ip address change
if ($vsite_new->{ipaddr})
{
    # make sure that there is a network interface for the new ip
#    vsite_add_network_interface($cce, $vsite_new->{ipaddr});

    # delete the old interface, this is a no op if another site is using the old ip still
    vsite_del_network_interface($cce, $vsite_old->{ipaddr});
} # end of ip address change specific
...

...
# make sure there is a network interface for this ip
#vsite_add_network_interface($cce, $vsite->{ipaddr});
...
 

This way when you add or modify a virtual site assigned to virtual IP address BlueQuartz won’t add its address to one of the network interfaces because we already took care of it.

Server is configured as director to be a part of high availability and load balancing cluster as described here:
http://www.ultramonkey.org/3/topologies/sl-ha-lb-eg.html

CentOS 4 with Bluequartz from Nuonce.net is used as operating system.

PROBLEM:

As a part of configuration an alias for loopback interface had to be created so server can accept packets for the virtual service:

DEVICE=lo:0
IPADDR=192.168.6.240
NETMASK=255.255.255.255
NETWORK=192.168.6.0
BROADCAST=192.168.6.255
ONBOOT=yes
NAME=loopback

When heartbeat starts on director, it removes the alias on loopback interface lo and creates virtual IP address on interface eth0.

Here is how all interfaces look like before problem occurs:

[root@server ~]# ip addr show
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:19:db:df:15:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.1/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0

However after certain amount of time the alias with virtual IP appears on loopback interface again:

[root@server ~]# ip addr sh
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
    inet 192.168.6.240/32 brd 192.168.6.255 scope global lo:0
2: eth0:  mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:19:db:df:15:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.1/24 brd 192.168.6.255 scope global eth0
    inet 192.168.6.240/24 brd 192.168.6.255 scope global secondary eth0

NOTE:

This problem only affects servers acting as load balancers / directors. Real servers running on CentOS / Bluequartz are not affected.

CAUSE:

It turns out this problem is caused by swatch which is utilized by BlueQuartz. Every 15 minutes it checks the status of all interfaces that have ONBOOT=yes and brings them up if they are down.

See /usr/sausalito/swatch/bin/am_network.sh for more information.

SOLUTION:

In order to prevent swatch from manipulating our virtual IP address, you have to turn off ONBOOT parameter since that what it looks for.

...
ONBOOT=no
...

Then disable heartbeat from starting automatically and make sure it is off.

[root@server ~]# chkconfig heartbeat off
[root@server ~]# chkconfig --list heartbeat | grep heartbeat
heartbeat       0:off   1:off   2:off   3:off   4:off   5:off   6:off

Finally bring the alias and heartbeat up in rc.local. This script file is run once, before all other scripts have run but before the logon prompt appears.

...
ifup lo:0
/etc/init.d/heartbeat start

LINKS:

1. Streamline High Availability and Load Balancing
2. How to install Ultra Monkey LVS in a 2-Node HA/LB Setup on CentOS/RHEL4

When starting ColdFusion for the first time connector wizard cannot configure web server correctly.

Starting ColdFusion 8...
The ColdFusion 8 server is starting up and will be available shortly.
======================================================================
Running the ColdFusion 8 connector wizard
======================================================================
Configuring the web server connector (Launched on the first run of the ColdFusion 8 start script)
Running apache connector wizard...
ColdFusion 8 not started, will retry connector 12 more time(s)
ColdFusion 8 not started, will retry connector 11 more time(s)
ColdFusion 8 not started, will retry connector 10 more time(s)
ColdFusion 8 not started, will retry connector 9 more time(s)
ColdFusion 8 not started, will retry connector 8 more time(s)
ColdFusion 8 not started, will retry connector 7 more time(s)
ColdFusion 8 not started, will retry connector 6 more time(s)
ColdFusion 8 not started, will retry connector 5 more time(s)
ColdFusion 8 not started, will retry connector 4 more time(s)
ColdFusion 8 not started, will retry connector 3 more time(s)
ColdFusion 8 not started, will retry connector 2 more time(s)
ColdFusion 8 not started, will retry connector 1 more time(s)
The connector wizard has not been able to contact the local ColdFusion 8, connector installation aborted.
======================================================================
ColdFusion 8 has been started.
ColdFusion 8 will write logs to /opt/coldfusion8/logs/cfserver.log
======================================================================

java.net.UnknownHostException: foo.mydomain.com: foo.mydomain.com

05/06 07:59:07 info Macromedia JRun 4.0 (Build 108487)
05/06 07:59:08 error Could not connect to any JRun/ColdFusion servers on host localhost.
Possible causes:
o Server not running
  -Start Macromedia JRun4 or ColdFusion 8 server
o Server running
  -JNDI listen port in jndi.properties blocked by TCP/IP filtering or firewall
   on server
  -host restriction in security.properties blocking communication with server

CAUSE:

This behavior could be caused by using non-resolvable host name (foo.mydomain.com in our example) along with /etc/hosts file having incorrect permissions.

[root@foo ~]# ls -la /etc/hosts
-rw-r-----  1 root root 1514 May  4 18:04 /etc/hosts

SOLUTION:

To resolve this issue /etc/hosts has to have 0644 access permission:

[root@foo ~]# chmod 0644 /etc/hosts
[root@foo ~]# ls -la /etc/hosts
-rw-r--r--  1 root root 1514 May  4 18:04 /etc/hosts

On distributions with BlueQuartz (Strongbolt, BlueOnyx, etc) it could be caused by /usr/sausalito/handlers/nuonce/fixhosts/fixhosts script (see this post). To prevent this script from resetting permissions again, add the following line at the end of the script:

/bin/chmod 0644 /etc/hosts

I wanted to use Cobalt RaQ4 to test highly available and load balanced cluster configuration. However current Strongbolt kernel 2.6.16i586 doesn’t have IP virtual server support and had to be recompiled. I’ve used the following page as a reference while building a kernel – http://www.digitalhermit.com/linux/Kernel-Build-HOWTO.html.

Below is a list of steps required to compile a kernel with virtual server support. If you don’t need this functionality please ignore steps 6 and 13.

1. Logged in as root, downloaded and extracted kernel source code.

   wget http://www.osoffice.co.uk/linux/raq4-sb-kernel-source/2.6/linux-2.6.16-raq4-source.tar.gz
   tar -xzf linux-2.6.16-raq4-source.tar.gz -C /usr/src
   cd /usr/src/linux-2.6.16-raq4-standard/

2. Installed compiler and required libraries.

   yum install gcc ncurses-devel

3. To prevent overwriting any existing kernel modules on the system modified kernel version to something unique in /usr/src/linux-2.6.16-raq4-standard/Makefile.

   VERSION = 2
   PATCHLEVEL = 6
   SUBLEVEL = 16
   EXTRAVERSION = -ipvs

4. Current kernel configuration settings are stored in /proc/config.gz. Since I only need to make minor changes to the configuration it would be wise to re-use the configuration file.

   zcat /proc/config.gz > /usr/src/linux-2.6.16-raq4-standard/.config

5. Run the configuration utility. Configuration file .config created in previous step will be loaded automatically.

   make menuconfig

6. In the menu, selected Networking -> Networking options -> IP: Virtual Server Configuration and then made the following choices:

       IP virtual server support (EXPERIMENTAL)
      [ ]   IP virtual server debugging (NEW)
      (12)  IPVS connection table size (the Nth power of 2) (NEW)
      ---   IPVS transport protocol load balancing support
      [*]   TCP load balancing support
      [*]   UDP load balancing support
      [ ]   ESP load balancing support (NEW)
      [ ]   AH load balancing support (NEW)
      ---   IPVS scheduler
         round-robin scheduling
         weighted round-robin scheduling
         least-connection scheduling
         weighted least-connection scheduling
         locality-based least-connection scheduling
         locality-based least-connection with replication scheduling
         destination hashing scheduling
         source hashing scheduling
         shortest expected delay scheduling
         never queue scheduling
      ---   IPVS application helper
         FTP protocol helper (NEW)

7. Removed compiled kernel that came with the source files.

   rm vmlinux.bz2

8. Compiled the kernel.

   make vmlinux

9. Compiled and installed all the loadable modules.

   make modules
   make modules_install

10. Made a backup of the current kernel.

    cp /boot/vmlinux.bz2 /boot/vmlinux-`uname -r`.bz2
    chmod 0644 /boot/vmlinux-`uname -r`.bz2

11. Compressed and installed the kernel. Since one of the Strongbolt forum users experienced a problem booting kernel compressed with default settings (see http://www.osoffice.co.uk/forum/viewtopic.php?f=7&t=391), compression method #4 was used.

    bzip2 -4 vmlinux
    cp vmlinux.bz2 /boot

12. Rebooted the server.
13. Loaded virtual server kernel modules.

    modprobe ip_vs ip_vs_rr

]]> http://www.ponticstar.com/blog/2009/04/27/how-to-compile-strongbolt-kernel-on-cobalt-raq4/feed/ 0