Oracle Clusterware Installation – Timed out waiting for the CRS stack to start

Terms:

Operating System: Enterprise Linux 4 U5 (RHEL4 U5)

Oracle: 10.2.0.1

Clusterware: 10.2.0.1

Cluster Interconnect: Ethernet (private connection)

Problem:

While running the root.sh script on the last cluster node, during the CLusterware Installation, the following error message occurs.

# /u01/app/oracle/product/10.2.0/crs/root.sh

WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root

WARNING: directory '/u01/app/oracle/product' is not owned by root

Checking to see if Oracle CRS stack is already configured

/etc/oracle does not exist. Creating it now.

Setting the permissions on OCR backup directory

Setting up NS directories

Oracle Cluster Registry configuration upgraded successfully

WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root

WARNING: directory '/u01/app/oracle/product' is not owned by root

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

Successfully accumulated necessary OCR keys.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.

node :
node 1: xen1 xen1-priv xen1

node 2: xen2 xen2-priv xen2

node 3: xen3 xen3-priv xen3

node 4: xen4 xen4-priv xen4

clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster

configuration.

Oracle Cluster Registry for cluster has already been initialized

Startup will be queued to init within 90 seconds.

Adding daemons to inittab

Expecting the CRS daemons to be up within 600 seconds.

CSS is active on these nodes.

xen1

xen2

xen3

xen4

CSS is active on all nodes.

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Timed out waiting for the CRS stack to start.
Solutions:

I found more than one issue that can help solving the problem.


 Use a dedicated Switch for the Cluster Interconnect.
Set the Cluster Interconnetc MTU=1500 (f.e. change later to MTU=9000).
Add only two Cluster Nodes to the CRS with the initial installation and add the other Nodes with addNode.sh Script seperately to the Cluster.

I am not exactly sure which one has solved the probelm on my installation but I considered them all.

                

             

Clean remove Oracle Clusterware (CRS) 10GR2 from a RHEL4

The problem with Oracle Clusterware (also known as Cluster Ready Service – CRS) is, that there is no built-in mechanism from Oracle to clean remove the Clusterware and all of it’s files distributed over the OS filesystem. The follwowing example script removes the Oracle Clusterware completely. The operating system is a RHEL4 U5.

This script has to be edited for personal use. $ORA_CRS_HOME is here under /opt/oracle.


########### script ###############

#!/bin/bash

echo

echo "Remove the Oracle Clusterware Service ?"

echo

echo "Enter y[yes] or n[no] to exit"read comit

if [ $comit == "n" ]; then

echo "Exit from Script without any change..."

exit 1

else

echo "Start to Shutdown and Remove Oracle Clusterware ..."

echo

/etc/init.d/init.evmd stop

/etc/init.d/init.evmd disable

/etc/init.d/init.cssd stop

/etc/init.d/init.cssd disable

/etc/init.d/init.crsd stop

/etc/init.d/init.crsd disable

/etc/init.d/init.crs stop

/etc/init.d/init.crs disable

rm -rf /etc/oracle /etc/oraInst.loc /etc/oratab

rm -rf /etc/init.d/init.crsd /etc/init.d/init.crs /etc/init.d/init.cssd /etc/init.d/init.evmd

rm -rf /etc/rc2.d/K96init.crs /etc/rc2.d/S96init.crs etc/rc3.d/K96init.crs \

/etc/rc3.d/S96init.crs /etc/rc4.d/K96init.crs /etc/rc4.d/S96init.crs \

/etc/rc5.d/K96init.crs /etc/rc5.d/S96init.crs /etc/rc.d/rc0.d/K96init.crs \

/etc/rc.d/rc1.d/K96init.crs /etc/rc.d/rc6.d/K96init.crs /etc/rc.d/rc4.d/K96init.crs

cp /etc/inittab.orig /etc/inittab

rm -rf /etc/inittab.crs /etc/inittab.no_crs

rm -rf /tmp/*

rm -rf /tmp/.oracle

rm -rf /usr/local/bin/dbhome /usr/local/bin/oraenv /usr/local/bin/coraenv

rm -rf /var/tmp/.oracle

rm -rf /opt/oracle/*

echo

echo "Remove on one Node the Shared Devices"

echo "rm -rf /u03/oracrs/*"

echo

fi

########### end script ###############

After running that script on a system it should be possible to reinstall Oracle Clusterware without any problems.