Setting up simple clustered service on RedHat

I will show how to set up simple network service (httpd) as clustered service on RedHat. This example will show cluster containing three separate nodes. The first part will describe setting up three node cluster and the second starting httpd on running cluster environment.

So let’s begin!

Prerequisites:

1) Installed packages:

system-config-cluster
perl-Net-Telnet
cman
rgmanager
openais        -> installed with cman
gfs-utils

2) Clustered network:

Is defined, UP and RUNNING on all nodes (i.e. 10.10.111.0/24).

3) Security settings:

iptables -> disabled, SELinux -> disabled or Permissive policy enabled.

Configuring cluster:

All configuration can be done with graphic utility -> system-config-cluster, which operates directly on /etc/cluster/cluster.conf file. You have to:

– type cluster name,

– add each node by name and configure number of votes for each one,

– add fence device with appropiate config.

/etc/cluster/cluster.conf file should be copied on all cluster nodes. After that you can bring up all nodes each after another:

service cman start

If you want cluter to be brought up everytime after boot you should add ‘cman’ service to chkconfig:

chkconfig cman on

Before setting any cluster service you should force each node (or only those which are supposed to handle services) to be ‘Resource Group Manager’ by executing this command on each node:

clurgmgrd

To check cluster status you shoul use:

clustat

Typical output will be like this:

Cluster Status for CentOS_cluster @ Thu Dec  2 13:06:09 2010
Member Status: Quorate

Member Name                                                  ID   Status
—— —-                                                  —- ——
opl01esx12                                                       1 Online, Local, rgmanager
opl01esx13                                                       2 Online, rgmanager
opl01esx14                                                       3 Online, rgmanager

As you can see we have three node cluster, all nodes are capable to run cluster service and all are up and online. Cluster is also quorate.

Configuring simple network clustered service:

You have to modify cluster configuration and:

– add new resource:

If you want to add your own service you should choose ‘Script’ Resource Type, provide unique name and specify path to start script for your service. You have to specify ‘File System’ (FS type, mount point and physical or logical device) and ‘IP Adress’ for the resource (here you should use an adress from clustered network).

Of course both filesystem and IP must be ready to use when service will be started. FS have to be mounted and usable, IP adress must be configured, UP and RUNNING. Additionally for this example also the service itself must be prepared on all cluster nodes for working as clustered service. I.e. Apache must have appropiate variables configured in httpd.conf file. This will be ‘DocumentRoot’ set to clustered FS defined in resources and ‘Listen’ parameter set to clustered IP adress.

– create new service:

Here you can specify if the service should be autostarted and what policy should be use when service is going down (relocating, disabling or restarting this service). You have to add all defined earlier resources (script, FS and IP adress) to your new service.

– create failover domain for the service:

It should contain list of cluster nodes with priority which describe sequence of nodes that will be chosen to start up your service – ‘1’ is the highest priority.

Remember that after making modifications in /etc/cluster/cluster.conf you have to copy this file on all cluster nodes and restart ‘cman’ service!

There is a listing of cluster configuration file shown below:

<?xml version=”1.0″ ?>
<cluster alias=”CentOS_cluster” config_version=”24″ name=”CentOS_cluster”>
<fence_daemon post_fail_delay=”0″ post_join_delay=”3″/>
<clusternodes>
<clusternode name=”opl01esx12″ nodeid=”1″ votes=”1″>
<fence>
<method name=”1″/>
</fence>
</clusternode>
<clusternode name=”opl01esx13″ nodeid=”2″ votes=”1″>
<fence>
<method name=”1″/>
</fence>
</clusternode>
<clusternode name=”opl01esx14″ nodeid=”3″ votes=”1″>
<fence>
<method name=”1″/>
</fence>
</clusternode>
</clusternodes>
<cman/>
<fencedevices>
<fencedevice agent=”fence_xvm” name=”CentOS_cluster_fence”/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name=”httpd-domain” ordered=”1″ restricted=”0″>
<failoverdomainnode name=”opl01esx12″ priority=”1″/>
<failoverdomainnode name=”opl01esx13″ priority=”2″/>
<failoverdomainnode name=”opl01esx14″ priority=”3″/>
</failoverdomain>
</failoverdomains>
<resources>
<script file=”/etc/rc.d/init.d/httpd” name=”httpd – Apache HTTP Server”/>
<fs device=”/dev/mapper/VolGroup00-LogVol02″ force_fsck=”0″ force_unmount=”0″ fsid=”6443″ fstype=”ext3″ mountpoint=”/LVM/” name=”httpd-content” options=”” self_fence=”0″/>
<ip address=”10.10.111.220″ monitor_link=”1″/>
</resources>
<service autostart=”1″ domain=”httpd-domain” name=”httpd-service” recovery=”relocate”>
<script ref=”httpd – Apache HTTP Server”/>
<fs ref=”httpd-content”/>
<ip ref=”10.10.111.220″/>
</service>
</rm>
</cluster>

Cluster commands which should be useful:

clustat – displays cluster and services status

clurgmgrd – enables resource group manager on cluster node

clusvcadm – cluster administration and some examples (enabling. disabling and relocating service to specified node):

clusvcadm -e httpd-service
clusvcadm -d httpd-service
clusvcadm -r httpd-service -m opl01esx13

Now you can check if your clustered service is working by taping in your browser IP adress which you have defined both in httpd.conf file and as a cluster resource. You should see your clustered apache start page :-)

Leave a Response