Fun!!! - Changed hostname on GI (Standalone Server)

Changed hostname on Grid Infrastructure for a Standalone Server. why did I talk about this? I just changed hostname in my virtualbox and rebooted it. I got error in /var/log/messages file(The Grid Infrastructure didn't start).

Oct 27 16:20:07 new_mytest logger: Oracle Cluster Ready Services startup disabled.
Oct 27 16:20:07 new_mytest logger: Could not access /etc/oracle/scls_scr/mylinux/oracle/ohasdstr.
Oct 27 16:20:13 new_mytest logger: autorun file for ohasd is missing
Oct 27 16:20:23 new_mytest logger: autorun file for ohasd is missing

So, I just tested/wrote about changed hostname on Grid Infrastructure for a Standalone Server. Not serious, just fun!!!
Example to change hostname from "mytest" to "mylinux" :)
*** Keep some resource informations ***

[root@mytest ~]# mkdir mytest
[root@mytest ~]# for x in ` /u01/app/oracle/product/11.2.0/grid/bin/crsctl stat res | grep NAME\= | awk -F\= '{print $2}'`; do /u01/app/oracle/product/11.2.0/grid/bin/crsctl stat res $x -p > mytest/$x.txt ; done
[root@mytest ~]# ls mytest
ora.asm.txt ora.DATA.dg.txt ora.evmd.txt ora.ons.txt
ora.cssd.txt ora.diskmon.txt ora.LISTENER.lsnr.txt ora.orcl.db.txt

A. Stopped HAS

[root@mytest ~]# /u01/app/oracle/product/11.2.0/grid/bin/crsctl stop has -f

B. Changed hostname

[root@mytest ~]# vi /etc/sysconfig/network
[root@mytest ~]# vi /etc/hosts
[root@mytest ~]# vi /u01/app/oracle/product/11.2.0/grid/network/admin/listener.ora
[root@mytest ~]# hostname
mytest
[root@mytest ~]# hostname mylinux
[root@mylinux ~]# hostname
mylinux

C. Started to change hostname with Grid Infrastructure for a Standalone Server.

[root@mytest ~]# /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl -h
Option hahome requires an argument
Usage:
roothas.pl [-verbose] [-upgrade | -patch]
[-paramfile <parameter-file>] [-deconfig] [-force]
[-unlock [-hahome <path to Oracle Restart home>]]

Options:
-verbose Run this script in verbose mode
-upgrade Oracle HA is being upgraded from previous version
-patch Oracle HA is being upgraded to a patch version
-paramfile Complete path of file specifying HA parameter values
-deconfig To deconfigure Oracle Restart
-force To deconfigure Oracle Restart when configuration is in a good condition
-unlock Unlock Oracle Restart home
-hahome Complete path of Oracle HA home

If neither -upgrade nor -patch is supplied, a new install is performed

To see the full manpage for this program, execute:
perldoc roothas.pl

[root@mytest ~]# /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig -force
Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params

CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Delete failed, or completed with errors.
CLSU-00100: Operating System function: opendir failed with error data: 2
CLSU-00101: Operating System error message: No such file or directory
CLSU-00103: error location: scrsearch1
CLSU-00104: additional error information: cant open scr home dir scls_scr_getval
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully deconfigured Oracle Restart stack

[root@mylinux ~]# /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl -verbose
Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node mylinux successfully pinned.
Adding Clusterware entries to inittab
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
Oracle Database 11g Clusterware Release 11.2.0.3.0 - Production Copyright 1996, 2011 Oracle. All rights reserved.
[client(3619)]CRS-10001:27-Oct-11 18:54 ACFS-9201: Not Supported
2011-10-27 18:56:06.170
[client(3703)]CRS-2101:The OLR was formatted using version 3.
2011-10-27 18:56:09.776
[client(3727)]CRS-1001:The OCR was formatted using version 3.
[client(3770)]CRS-10001:CRS-6021: No msg for has:crs-6021 [l][unlimited]
[client(3771)]CRS-10001:CRS-6021: No msg for has:crs-6021 [n][65536]

*** Fixed them (ulimit) and fixed some... ***

[root@mylinux ~]# ps -aef | grep grid
oracle 3768 1 0 18:56 ? 00:00:00 /u01/app/oracle/product/11.2.0/grid/bin/ohasd.bin reboot
oracle 4213 1 0 19:05 ? 00:00:00 /u01/app/oracle/product/11.2.0/grid/bin/ohasd.bin reboot
oracle 4659 1 0 19:12 ? 00:00:00 /u01/app/oracle/product/11.2.0/grid/bin/ohasd.bin reboot

[root@mylinux ~]# kill -9 3768 4213 4659
[root@mylinux ~]# ps -aef | grep grid
oracle 4875 1 4 19:16 ? 00:00:00 /u01/app/oracle/product/11.2.0/grid/bin/ohasd.bin restart

*** After error and then run script again!!! Just make sure, NO "ohasd.bin reboot" on OS processes ***
*** De-configuration and run "roothas.pl" script again. ***

[root@mylinux ~]# /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig
Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
CRS-2613: Could not find resource 'ora.cssd'.
CRS-4000: Command Stop failed, or completed with errors.
CRS-2613: Could not find resource 'ora.cssd'.
CRS-4000: Command Delete failed, or completed with errors.
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle Restart stack

[root@mylinux ~]# /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl -verbose
Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node mylinux successfully pinned.
Adding Clusterware entries to inittab

mylinux 2011/10/27 19:20:36 /u01/app/oracle/product/11.2.0/grid/cdata/mylinux/backup_20111027_192036.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server

[root@mytest ~]# /u01/app/oracle/product/11.2.0/grid/bin/crsctl check has
CRS-4638: Oracle High Availability Services is online

[root@mytest ~]# /u01/app/oracle/product/11.2.0/grid/bin/crsctl stat res | grep NAME\=
NAME=ora.cssd
NAME=ora.diskmon
NAME=ora.evmd
NAME=ora.ons

*** Checked in backup for resource informations ***

[root@mytest ~]# ls mytest
ora.asm.txt ora.DATA.dg.txt ora.evmd.txt ora.ons.txt
ora.cssd.txt ora.diskmon.txt ora.LISTENER.lsnr.txt ora.orcl.db.txt

Reviewed resources what was not register.
*** Try to use crsctl add res resource_name -type type_name -file mytest/file.txt ***
*** But Error - CRS-0175: Resource type 'type_name' either doesn't exist or cannot be accessed. ***
*** OK use "srvctl" for manual ***

D. Added other resources by manual (srvctl)
- Added Listener (changed listener.ora file before)

[root@mytest ~]# su - oracle
[oracle@mylinux ~]$ srvctl -h | grep listener | grep add
Usage: srvctl add listener [-l <lsnr_name>] [-s] [-p "[TCP:]<port>[, ...][/IPC:<key>][/NMP:<pipe_name>][/TCPS:<s_port>] [/SDP:<port>]"] [-o <oracle_home>]
[oracle@mylinux ~]$ srvctl add listener -l LISTENER -p "TCP:1521" -o /u01/app/oracle/product/11.2.0/grid
[oracle@mylinux ~]$ srvctl start listener
[oracle@mylinux ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): mylinux

- Added ASM

[root@mytest ~]# grep DISK mytest/ora.asm.txt
ASM_DISKSTRING=DATA
[root@mytest ~]# grep ASM_DISKSTRING mytest/ora.asm.txt
ASM_DISKSTRING=DATA
[root@mytest ~]# grep SP mytest/ora.asm.txt
SPFILE=
[root@mytest ~]# su - oracle
[oracle@mylinux ~]$ srvctl -h | grep asm | grep add
Usage: srvctl add asm [-l <lsnr_name>] [-p <spfile>] [-d <asm_diskstring>]
[oracle@mylinux ~]$ srvctl add asm -l LISTENER -p '' -d 'DATA'
[oracle@mylinux ~]$ srvctl start asm
[oracle@mylinux ~]$ srvctl status asm

[oracle@mylinux ~]$ /u01/app/oracle/product/11.2.0/grid/bin/crsctl stat res | grep NAME\=
NAME=ora.LISTENER.lsnr
NAME=ora.asm
NAME=ora.cssd
NAME=ora.diskmon
NAME=ora.evmd
NAME=ora.ons

[oracle@mylinux ~]$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/grid
[oracle@mylinux ~]$ export ORACLE_SID=+ASM
[oracle@mylinux ~]$ sqlplus / as sysasm

SQL> select name from v$asm_diskgroup;

NAME
------------------------------
DATA

SQL> alter diskgroup DATA mount;

Diskgroup altered.

[oracle@mylinux ~]$ /u01/app/oracle/product/11.2.0/grid/bin/crsctl stat res | grep NAME\=
NAME=ora.DATA.dg
NAME=ora.LISTENER.lsnr
NAME=ora.asm
NAME=ora.cssd
NAME=ora.diskmon
NAME=ora.evmd
NAME=ora.ons

- Added Database

[root@mytest ~]# grep SPFILE mytest/ora.orcl.db.txt
SPFILE=/u01/app/oracle/product/11.2.0/dbhome_2/dbs/spfileorcl.ora
[root@mytest ~]# grep ORACLE_HOME\= mytest/ora.orcl.db.txt
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_2

[oracle@mylinux ~]$ srvctl -h | grep database | grep add
Usage: srvctl add database -d <db_unique_name> -o <oracle_home> [-m <domain_name>] [-p <spfile>] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY | SNAPSHOT_STANDBY}] [-s <start_options>] [-t <stop_options>] [-n <db_name>] [-i <inst_name>] [-y {AUTOMATIC | MANUAL | NORESTART}] [-a "<diskgroup_list>"]
[oracle@mylinux ~]$ srvctl add database -d orcl -o /u01/app/oracle/product/11.2.0/dbhome_2 -p /u01/app/oracle/product/11.2.0/dbhome_2/dbs/spfileorcl.ora
[oracle@mylinux ~]$ srvctl start database -d orcl

I learned some when I used "crsctl add res resource_name -type type_name -file mytest/file.txt". I got error "CRS-0175: Resource type 'type_name' either doesn't exist or cannot be accessed", that mean I have to add "resource type" before. But "srvctl" command, that will add "resource type" automatic.

Related Post:
Just Test - Install Oracle GI (standalone server) on OL6
Upgrade Grid Infrastructure for cluster (11.2.0.2 to 11.2.0.3)
11.2.0.3 GI diskmon will be OFFLINE by default in Non-Exadata

Install Grid Infra - clscfg.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory

Nothing special, I just installed Grid Infrastructure on Oracle Linux 6 update 1. I found error during run "root.sh":

clscfg.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory

# /u01/app/oracle/11.2.0/grid/root.sh
Check /u01/app/oracle/11.2.0/grid/install/root_mytest_2011-08-23_18-29-38.log for the output of root script
# cat /u01/app/oracle/11.2.0/grid/install/root_mytest_2011-08-23_18-29-38.log

Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/oracle/11.2.0/grid
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/oracle/11.2.0/grid/crs/install/crsconfig_params
/u01/app/oracle/11.2.0/grid/bin/clscfg.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory
Failed to create keys in the OLR, rc = 127, 32512
/u01/app/oracle/11.2.0/grid/perl/bin/perl -I/u01/app/oracle/11.2.0/grid/perl/lib -I/u01/app/oracle/11.2.0/grid/crs/install /u01/app/oracle/11.2.0/grid/crs/install/roothas.pl execution failed

Check on My Oracle Support for Idea. Clscfg.bin Error While Loading Shared Libraries: libcap.so.1 [ID 952051.1]. Need to install libcap1 and libcap2 packages.

# uname -r
2.6.32-100.34.1.el6uek.x86_64

So, need libcap1 and libcap2 packages for 2.6.32-100.34.1.el6uek.x86_64

# yum list | grep libcap
compat-libcap1.i686 1.10-1 @ol6_ga_base
libcap.i686 2.16-5.2.el6 @ol6_ga_base
libcap.x86_64 2.16-5.2.el6 @anaconda-OracleLinuxServer-201102031546.x86_64/6.0
libcap-devel.i686 2.16-5.2.el6 @ol6_ga_base
libcap-devel.x86_64 2.16-5.2.el6 @ol6_ga_base
libcap-ng.x86_64 0.6.4-3.el6_0.1 @ol6_u1_base
compat-libcap1.x86_64 1.10-1 ol6_ga_base
libcap-ng.i686 0.6.4-3.el6_0.1 ol6_u1_base
libcap-ng-devel.i686 0.6.4-3.el6_0.1 ol6_u1_base
libcap-ng-devel.x86_64 0.6.4-3.el6_0.1 ol6_u1_base
libcap-ng-python.x86_64 0.6.4-3.el6_0.1 ol6_u1_base
libcap-ng-utils.x86_64 0.6.4-3.el6_0.1 ol6_u1_base

Installed...

# yum install libcap.x86_64
Setting up Install Process
Package libcap-2.16-5.2.el6.x86_64 already installed and latest version
Nothing to do

libcap2 package installed, so install libcab1.

# yum install compat-libcap1.x86_64
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package compat-libcap1.x86_64 0:1.10-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================
Package Arch Version Repository Size
=========================================================================================================================
Installing:
compat-libcap1 x86_64 1.10-1 ol6_ga_base 17 k

Transaction Summary
=========================================================================================================================
Install 1 Package(s)

Total download size: 17 k
Installed size: 29 k
Is this ok [y/N]: y
Downloading Packages:
compat-libcap1-1.10-1.x86_64.rpm | 17 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
Installing : compat-libcap1-1.10-1.x86_64 1/1

Installed:
compat-libcap1.x86_64 0:1.10-1

Complete!

OK ... run "root.sh" again.

# /u01/app/oracle/11.2.0/grid/crs/install/roothas.pl -deconfig
Using configuration parameter file: /u01/app/oracle/11.2.0/grid/crs/install/crsconfig_params
# rm /etc/oracle/olr.loc
# /u01/app/oracle/11.2.0/grid/root.sh

...One thing on Oracle Linux 6. It's not support inittab file. You can create file /etc/init/*.conf (created before or modified has script )

#start oracle
start on runlevel [35]
stop on runlevel [016]
respawn
exec /etc/init.d/init.ohasd run > /dev/null 2>&1 < /dev/null

Thank You for Oracle Linux 6.1 And libcap.so.1: cannot open shared object
I learned more...

How to register ASM diskgroup in Grid Infrastructure

It's a cool today, Someone are working, Someone are on vacation.
By the way. Have good holidays ;)

I read/tested Relocating Grid Infrastructure. After I relocated my Grid Infrastructure. I'd like to register ASM diskgroup else.

$ ./crsstat -word dg
HA Resource Type Target State (Host)
----------- ------ ------ -----
ora.DISK00.dg ora.diskgroup.type ONLINE ONLINE on rhel5-test

Some disk groups... not registered in Grid. I had to register by manual, then I checked srvctl command-line, but no "add" diskgroup.

$ srvctl -h | grep diskgroup | grep dg_name
Usage: srvctl start diskgroup -g <dg_name> [-n "<node_list>"]
Usage: srvctl stop diskgroup -g <dg_name> [-n "<node_list>"] [-f]
Usage: srvctl status diskgroup -g <dg_name> [-n "<node_list>"] [-a]
Usage: srvctl enable diskgroup -g <dg_name> [-n "<node_list>"]
Usage: srvctl disable diskgroup -g <dg_name> [-n "<node_list>"]
Usage: srvctl remove diskgroup -g <dg_name> [-f]

Actually, ASM disk groups are registered in Grid by Automatic, After we mounted them ???

$ sqlplus / as sysasm

SYS> alter diskgroup DISK01 mount;

Diskgroup altered.

SYS> alter diskgroup DISK02 mount;

Diskgroup altered.

SYS> alter diskgroup DISK03 mount;

Diskgroup altered.

$ ./crsstat -word dg
HA Resource Type Target State (Host)
----------- ------ ------ -----
ora.DISK00.dg ora.diskgroup.type ONLINE ONLINE on rhel5-test
ora.DISK01.dg ora.diskgroup.type ONLINE ONLINE on rhel5-test
ora.DISK02.dg ora.diskgroup.type ONLINE ONLINE on rhel5-test
ora.DISK03.dg ora.diskgroup.type ONLINE ONLINE on rhel5-test

By the way, It's just fun today. Nothing special.

Reference:
#!/bin/bash
# crsstat v. 0.01
# By surachart
if [ $# -gt 4 ]
then
echo "Please Check arguments."
echo
echo "./crsstat [-word|--word] resource_word [-crshome|--crshome] CRS_HOME "
echo
echo " ./crsstat"
echo " ./crsstat -word orcl"
echo ' ./crsstat -word orcl -crshome $CRS_HOME'
exit 0
fi

while [ $# -gt 0 ]; do
case "$1" in
-h|--h|-help|--help)
echo "./crsstat [-word|--word] resource_word [-crshome|--crshome] CRS_HOME "
echo
echo " ./crsstat"
echo " ./crsstat -word orcl"
echo ' ./crsstat -word orcl -crshome $CRS_HOME'
exit 0
;;
-word|--word)
GREP_KEY=$2
shift
;;
-crshome|--crshome)
ORA_CRS_HOME=$2
shift
;;
esac
shift
done

if [ -z $ORA_CRS_HOME ]
then
ORA_CRS_HOME=$CRS_HOME
if [ ! -d $ORA_CRS_HOME ]
then
echo "Please Check CRS_HOME Environment ($ORA_CRS_HOME)"
exit 1
fi

elif [ ! -d $ORA_CRS_HOME ]
then
ORA_CRS_HOME=$CRS_HOME
if [ ! -d $ORA_CRS_HOME ]
then
echo "Please Check CRS_HOME Environment ($ORA_CRS_HOME)"
exit 1
fi
fi

CRS_STAT=$ORA_CRS_HOME/bin/crs_stat

if [ ! -x $CRS_STAT ]
then
echo "Please Check CRS_HOME Environment ($ORA_CRS_HOME)"
exit 2
fi

AWK=/usr/bin/awk

if [ ! -x $AWK ]
then
AWK=/bin/awk
fi

$AWK \
'BEGIN {printf "%-40s %-25s %-10s %-12s\n", "HA Resource", "Type", "Target", "State (Host)"; printf "%-40s %-25s %-10s %-12s\n", "-----------", "------", "------", "-----";}'
$CRS_STAT | $AWK \
'BEGIN { FS="="; state = 0; }
$1~/NAME/ && $2~/'$GREP_KEY'/ {appname = $2; state=1;}
state == 0 {next;}
$1~/TYPE/ && state == 1 {apptype = $2; state=2;}
$1~/TARGET/ && state == 2 {apptarget = $2; state=3;}
$1~/STATE/ && state == 3 {appstate = $2; state=4;}
state == 4 {printf "%-40s %-25s %-10s %-12s\n", appname,apptype, apptarget, appstate;state=0}'

Disable/Enable Automatic startup Oracle HAS

On 11gR2, Oracle Clusterware consists of two separate stacks: an upper stack anchored by the Cluster Ready Services (CRS) daemon (crsd) and a lower stack anchored by the Oracle High Availability Services daemon (ohasd).

So.. How to disable/enable Oracle HAS.
Use the crsctl disable has command to disable automatic startup of the Oracle High Availability Services stack when the server boots up.

# crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.

How to know Oracle HAS is enabled(if doesn't use "crsctl config has")

# cat /etc/oracle/scls_scr/rhel5-test/root/ohasdstr
enable

# crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.

# crsctl config has
CRS-4621: Oracle High Availability Services autostart is disabled.

# cat /etc/oracle/scls_scr/rhel5-test/root/ohasdstr
disable

Use the crsctl enable has command to enable automatic startup of the Oracle High Availability Services stack when the server boots up.

# crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.

# cat /etc/oracle/scls_scr/rhel5-test/root/ohasdstr
enable

If We just check HAS Disable/Enable status, that uses "crsctl config has" command, it's easier than "ohasdstr" file checking.

How about "crsctl disable/enable crs" on 11gR2?
They disable/enable automatic startup of Oracle HAS.

I posted "check enable/disable the startup of CRS".. that show Oracle Clusterware version <= 11gR1, we can check from "crsstart" file. On 11gR2, crsstart file is not used ???

Use the crsctl disable crs command to prevent the automatic startup of Oracle High Availability Services when the server boots.

Use the crsctl enable crs command to enable automatic startup of Oracle High Availability Services when the server boots.

# crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.

# crsctl config crs
CRS-4622: Oracle High Availability Services autostart is enabled.

# ls -ltr /etc/oracle/scls_scr/rhel5-test/root/
-rw-r--r-- 1 root root 7 Sep 7 00:56 crsstart
-rw-r--r-- 1 root oinstall 5 Nov 22 17:04 ohasdrun
-rw-r--r-- 1 root oinstall 7 Nov 22 17:10 ohasdstr

# cat /etc/oracle/scls_scr/rhel5-test/root/crsstart
enable

# cat /etc/oracle/scls_scr/rhel5-test/root/ohasdstr
enable

# crsctl disable crs
CRS-4621: Oracle High Availability Services autostart is disabled.

# crsctl config crs
CRS-4621: Oracle High Availability Services autostart is disabled.

# crsctl config has
CRS-4621: Oracle High Availability Services autostart is disabled.

# ls -ltr /etc/oracle/scls_scr/rhel5-test/root/
-rw-r--r-- 1 root root 7 Sep 7 00:56 crsstart
-rw-r--r-- 1 root oinstall 5 Nov 22 17:04 ohasdrun
-rw-r--r-- 1 root oinstall 8 Nov 22 17:12 ohasdstr

# cat /etc/oracle/scls_scr/rhel5-test/root/crsstart
enable

# cat /etc/oracle/scls_scr/rhel5-test/root/ohasdstr
disable

However, check CRSCTL Utility Reference

Check enable/disable the startup of the Oracle Clusterware daemons

We can use crsctl commands as follows to enable and disable the startup of the Oracle Clusterware daemons. Run the following command to enable startup for all of the Oracle Clusterware daemons:

crsctl enable crs

Run the following command to disable the startup of all of the Oracle Clusterware daemons:

crsctl disable crs

Indeed... Oracle Document told me like that. But How can I know enable/disable startup status for all of the Oracle Clusterware daemons now?

I know... someone don't need to know because they can run "crsctl enable/disable crs" again and again.

Oracle has scls_scr directory at /etc/oracle path. We can check about enable/disable startup status of the Oracle Clusterware daemons at crsstart file in /etc/oracle/scls_scr/<hostname>/root/ path.

really? example:

root@rac1# cat /etc/oracle/scls_scr/rac1/root/crsstart

enable

root@rac1# cd CRS_HOME/bin

root@rac1# ./crsctl disable crs

root@rac1# cat /etc/oracle/scls_scr/rac1/root/crsstart

disable

after disabled by "crsctl disable crs", crsstart file was changed be "disable"

root@rac1# ./crsctl enable crs

root@rac1# cat /etc/oracle/scls_scr/rac1/root/crsstart

enable

after enabled by "crsctl enable crs", crsstart file was changed be "enable"

that justs idea ;)

Change Oracle ASM resource AUTO_START

After I read otn forums. I felt curious about "ASM resource don't start automatic after server reboot or when Oracle Cluster is started".
so, made simple testing about auto_start not equal 1

# /u01/oracle/product/crs/bin/crsctl start crs
Attempting to start Oracle Clusterware stack
The CRS stack will be started shortly

$ crs_stat -t

Name Type Target State Host
------------------------------------------------------------
ora.oratest01.ASM1.asm OFFLINE OFFLINE
ora.oratest01.LISTENER_ORATEST01.lsnr ONLINE ONLINE on oratest01
ora.oratest01.gsd ONLINE ONLINE on oratest01
ora.oratest01.ons ONLINE ONLINE on oratest01
ora.oratest01.vip ONLINE ONLINE on oratest01

ASM1 didn't start automatic, try to manual start it:

$ srvctl start asm -n oratest01

$ crs_stat -t
ora.oratest01.ASM1.asm ONLINE ONLINE on oratest01

That can start +ASM1 by manual, What's happened?

$ /u01/oracle/product/crs/bin/crs_stat | grep NAME\= | grep asm
NAME=ora.oratest01.ASM1.asm

$ /u01/oracle/product/crs/bin/crs_stat -p ora.oratest01.ASM1.asm
NAME=ora.oratest01.ASM1.asm
TYPE=application
ACTION_SCRIPT=/u01/oracle/product/11.1.0/asm/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=2

that showed auto_start=2: This means, after node reboot or when Oracle Clusterware is started, resources with AUTO_START=2 will need to be started
manually via srvctl.

have to change AUTO_START=1, How? (contact oracle support... Good Idea), Or... change it by manual.

- Stop asm before

$ srvctl stop asm -n oratest01

- copy ASM1 profile, CRS_HOME/crs/public/ PATH and filename = "resource_name".cap

$ /u01/oracle/product/crs/bin/crs_stat -p ora.oratest01.ASM1.asm > /u01/oracle/product/crs/crs/public/ora.oratest01.ASM1.asm.cap

- unregister ASM1 resource

$ /u01/oracle/product/crs/bin/crs_unregister ora.oratest01.ASM1.asm

$ /u01/oracle/product/crs/bin/crs_stat -p ora.oratest01.ASM1.asm
CRS-0210: Could not find resource 'ora.oratest01.ASM1.asm'.

- Edit cap file (AUTO_START=1)

$ vi /u01/oracle/product/crs/crs/public/ora.oratest01.ASM1.asm.cap

AUTO_START=2 -> AUTO_START=1

- register ASM1 resource to Cluster

$ /u01/oracle/product/crs/bin/crs_register ora.oratest01.ASM1.asm

- Check and Test

$ /u01/oracle/product/crs/bin/crs_stat -p ora.oratest01.ASM1.asm

NAME=ora.oratest01.ASM1.asm
TYPE=application
ACTION_SCRIPT=/u01/oracle/product/11.1.0/asm/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1

$ su - root

# /u01/oracle/product/crs/bin/crsctl stop crs
Stopping resources.
This could take several minutes.
Successfully stopped Oracle Clusterware resources
Stopping Cluster Synchronization Services.
Shutting down the Cluster Synchronization Services daemon.
Shutdown request successfully issued.

# /u01/oracle/product/crs/bin/crsctl start crs
Attempting to start Oracle Clusterware stack
The CRS stack will be started shortly

$ crs_stat -t

Name Type Target State Host
------------------------------------------------------------
ora.oratest01.ASM1.asm ONLINE ONLINE on oratest01
ora.oratest01.LISTENER_ORATEST01.lsnr ONLINE ONLINE on oratest01
ora.oratest01.gsd ONLINE ONLINE on oratest01
ora.oratest01.ons ONLINE ONLINE on oratest01
ora.oratest01.vip ONLINE ONLINE on oratest01

Success... ASM automatic started after when Oracle Clusterware is started.

crsctl.bin: error while loading shared libraries: libnnz11.so: cannot restore segment prot after reloc: Permission

When Need to test ASM on Single Node (OLE 5)... But found some error...

# /u01/app/oracle/product/11.1/db/bin/localconfig add

/u01/app/oracle/product/11.1/db/bin/crsctl.bin: error while loading shared libraries: /u01/app/oracle/product/11.1/db/lib/libnnz11.so: cannot restore segment prot after reloc: Permission denied
/u01/app/oracle/product/11.1/db/bin/clscfg.bin: error while loading shared libraries: /u01/app/oracle/product/11.1/db/lib/libnnz11.so: cannot restore segment prot after reloc: Permission denied
Configuration for local CSS has been initialized
Failed configure CSS. Fix the problem and rerun this script from destination Oracle home with 'reset' argument

After Check... Problem about SELinux ...SELINUX=enforcing

/etc/selinux/config file.

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=enforcing
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted

So, Change...
SELINUX=enforcing -> SELINUX=permissive

After change... rebooting help to prove.
Anyway! can switch system into "permissive" mode... no reboot

# cat /selinux/enforce
1
# echo 0 >/selinux/enforce

Prove by run... localconfig again;)

# /u01/app/oracle/product/11.1/db/bin/localconfig reset

Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized

Cleaning up Network socket directories
Setting up Network socket directories
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
node01
Cluster Synchronization Services is active on all the nodes.
Oracle CSS service is installed and running under init(1M)

;)

Why My Oracle Cluster could not start?

When rebooted Oracle RAC Server or..., Oracle Cluster couldn't start anyway;)

that doesn't happen often..., But If!

This just show my idea to solve about it...  If We find out How to resolve ...we can find many... articles on Internet;)

Oracle Cluster wasn't started, and  that we will not find anything at ORA_CRS_HOME/log/HOSTNAME/* to help... So, just find out to Operation System Logs(*.info).

# /u01/oracle/product/crs/bin/crsctl check  crs

Failure 1 contacting Cluster Synchronization Services daemon

Cannot communicate with Cluster Ready Services

Cannot communicate with Event Manager

# /u01/oracle/product/crs/bin/crs_stat 

CRS-0184: Cannot communicate with the CRS daemon.

Start to investigate from Checking init processes;)

# ps -aef | grep "init\."

root      4124     1  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.evmd run

root      4125     1  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.cssd fatal

root      4126     1  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.crsd run

root      4710  4124  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

root      5031  4125  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

root      5289  4126  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

that show some processes was checking to start...

If your crs 's disabled you just find:

# ps -aef | grep "init\."

root      4166     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.evmd run

root      4167     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.cssd fatal

root      4168     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.crsd run

After make sure Cluster have the problem, So, check messages log on (Linux)/var/log/messages

 Apr  3 12:11:59 oratest01 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5031.

Apr  3 12:11:59 oratest01 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5289.

After that, Check /tmp/crsctl.* files

# cat /tmp/crsctl.5289

Oracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]

If your crs 's disabled, that dont' find anything on /var/log/messages file.

Example: Upon problem about OCR File

# cat /etc/oracle/ocr.loc

ocrconfig_loc=/dev/raw/raw11

ocrmirrorconfig_loc=/dev/raw/raw12

local_only=FALSE

that show that use rawdevice ;), So check rawdevice services:

# /etc/init.d/rawdevices status 

Nothing to show, So start rawdevices.

# /etc/init.d/rawdevices start

Assigning devices: 

 .

 .

 .

           /dev/raw/raw11  -->   /dev/loop1

/dev/raw/raw11: bound to major 7, minor 1

           /dev/raw/raw12  -->   /dev/loop2

/dev/raw/raw12: bound to major 7, minor 2

.

.

.

# /etc/init.d/rawdevices status

.

.

.

/dev/raw/raw11: bound to major 7, minor 1

/dev/raw/raw12: bound to major 7, minor 2

.

.

.

That should to resovle this case;)

# /u01/oracle/product/crs/bin/crsctl check  crs

Cluster Synchronization Services appears healthy

Cluster Ready Services appears healthy

Event Manager appears healthy

 >>> What is that mean? I try to show when Oracle Cluster could not start... How can I do ? How can I think to do?

- Check init process 

$ ps -aef | grep "init\."

.

.

.

If don't find any process about cluster, make sure have some scripts on  /etc/inittab file.

>>>

h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1

h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1

h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1

>>>

- Check Error on messages log (/var/log/messages) and /tmp/crsctl.*

Another Case;) to make idea

# /u01/oracle/product/crs/bin/crsctl check  crs

Failure 1 contacting Cluster Synchronization Services daemon

Cannot communicate with Cluster Ready Services

Cannot communicate with Event Manager

# /u01/oracle/product/crs/bin/crs_stat 

CRS-0184: Cannot communicate with the CRS daemon.

# ps -aef | grep "init\."

root      4166     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.evmd run

root      4167     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.cssd fatal

root      4168     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.crsd run

root      9579  4166  0 12:46 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

root     10566  4167  0 12:46 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

root     10585  4168  0 12:46 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

Oracle cluster was checking to start. So, check ... check and check! finally check messages log:

Apr  3 12:47:32 oratest01 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.10566.

Apr  3 12:47:32 oratest01 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.10585.

found something "logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.10585" and then 

# cat /tmp/crsctl.10585

Failed 3 to bind listening endpoint: (ADDRESS=(PROTOCOL=tcp)(HOST=oratest01-priv))

That mean, it had the problem about "oratest01-priv" (InterConnect) , hostname or ..., So check and resolve;)

It's a good thing DBA should to do... check any error on messages log [when cluster could not start or cluster rebooted...].

DBA should to know about Operation System... that helpful to be DBA;)

srvctl start service; PRKP-1030 : Failed to start the service...ORA-44305: service ... is running

After I stoped  S01 service on node4 by srvctl command-line. It's OK .

$ srvctl stop service -d DB -s S01 -i DB4

After that, I need to start S01 service on this (node4) node (DB4)

$ srvctl start service -d DB -s S01 -i DB4

PRKP-1030 : Failed to start the service S01.

CRS-0215: Could not start resource 'ora.DB.S01.DB4.srv'

that's make me!... confused. So, I tried another idea... (try to force stop)

$ srvctl stop  service -d DB -s S01 -i DB4

PRKP-1065 : Service S01 is already stopped on instance DB4.

$ srvctl stop  service -d DB -s S01 -i DB4 -f 

PRKP-1065 : Service S01 is already stopped on instance DB4.

$ srvctl start service -d DB -s S01 -i DB4

PRKP-1030 : Failed to start the service S01.

CRS-0215: Could not start resource 'ora.DB.S01.DB4.srv'

that's not help me anything.

Check... Use "crs_stat"

ora.DB.S01.DB3.srv                         ONLINE     ONLINE on node03  

ora.DB.S01.DB4.srv                         ONLINE     OFFLINE on node04  

ora.DB.S01.cs                                ONLINE     ONLINE on node03  

So, I had some idea... I checked imon_DB.log file at $ORACLE_HOME/log/hostname/racg/* PATH on node, I used "srvctl start service -d DB -s S01 -i DB4"

2009-03-06 15:33:34.912: [    RACG][1273112928] [28413][1273112928][ora.DB.S01.DB4.srv]: clsrcsnstartsrv: service name S01 already exists

2009-03-06 15:33:34.913: [    RACG][1273112928] [28413][1273112928][ora.DB.S01.DB4.srv]: CLSR-0002: Oracle error encountered while executing clsrcsnstartsrv : execute2

2009-03-06 15:33:34.913: [    RACG][1273112928] [28413][1273112928][ora.DB.S01.DB4.srv]: ORA-44305: service S01 is running

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86

ORA-06512: at "SYS.DBMS_SERVICE", line 444

ORA-06512: at "SYS.DBMS_SERVICE", line 365

ORA-44305: service S01 is running 

What's going on?

So, I connected database on DB4 instance (node4)

$ env | grep ORACLE_SID

ORACLE_SID=DB4

$ sqlplus / as sysdba

SQL>  select instance_name from v$instance; 

INSTANCE_NAME

----------------

DB4

SQL> show parameter service_name 

NAME                                 TYPE        VALUE

------------------------- ----------- --------

service_names                        string      S01, DB

Oh, This Instance  has S01 service online. 

My idea used DBMS_SERVICE.STOP_SERVICE to stop service (S01) on this instance.

SQL> exec DBMS_SERVICE.STOP_SERVICE('S01','DB4');

PL/SQL procedure successfully completed.

SQL> show parameter  service_name  

NAME                                 TYPE        VALUE

------------------------- ----------- --------

service_names                        string      DB

And then use "srvctl" command to start again...

$ srvctl start service -d DB -s S01 -i DB4

Wow, It worked... no error

And then check...

Use "crs_stat"

ora.DB.S01.DB3.srv                         ONLINE     ONLINE on node03  

ora.DB.S01.DB4.srv                         ONLINE     ONLINE on node04  

ora.DB.S01.cs                                ONLINE     ONLINE on node03  

And...

$ sqlplus / as sysdba

SQL>  select instance_name from v$instance; 

INSTANCE_NAME

----------------

DB4

SQL> show parameter service_name 

NAME                                 TYPE        VALUE

------------------------- ----------- --------

service_names                        string      S01, DB

... It's OK, thanks OCP & OCE help me this idea!

Enjoy!

Failure at final check of Oracle CRS stack. 10

Who Someone run "root.sh" on second node and get this error:

> Failure at final check of Oracle CRS stack. 10

that is the problem about Firewall or some securities.

With Linux/Unix OS, that should check IPTABLES or Packet Filter.

and make sure on every nodes of Cluster , that can communicate on Public IP (TCP) and Private IP (UDP) with every nodes.

> iptables -L

"Failure at scls_scr_create with code 1" after run rootconfig command

I changed hostnames on RAC (solaris), But my hostnames is upper charactors.

# ./rootconfig
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/oracle/product' is not owned by root
WARNING: directory '/oracle' is not owned by root
Failure at scls_scr_create with code 1
Internal Error Information:
Category: 1234
Operation: scls_scr_create
Location: mkdir
Other: Unable to make user dir
Dep: 2
Failure initializing entries in /var/opt/oracle/scls_scr/HOSTNAME01.


So, I found out some information on metalink.

Cause :
This issue is caused by the host name problem as explained in Bug 4632899

Bug 4632899 -- CSS NOT STARTING UP IF HOSTNAME HAS CAPITAL LETTERS-
SOLARIS

Rediscovery Information:
On Solaris, when the hostname is in
Capital, the following
errors might be noticed when using crs scripts.

Solution :
This basically happens because awk does not have the tolower() function in
the Sun Solaris. The call to tolower() on Solaris 9 and Solaris 10 returns the
unchanged uppercase hostname.

The followings options are available to
resolve this:

1.
Change the host name to lower case.

OR

2. "nawk" has the tolower()
function. Setup a symbolic link to "nawk"

ln -s /usr/bin/nawk
/usr/bin/awk

this case, I used choice II:

mv /usr/bin/awk /usr/bin/awk-org

ln -s /usr/bin/nawk /usr/bin/awk

and then run rootconfig again. It's OK.

CRS-1028: Dependency analysis failed because of: CRS-0223: Resource...

After I changed hostname on my RAC

and then run $ORA_CRS_HOME/install/rootconfig command.

I reconfigured listener, But can not start by RAC (srvctl command-line).

I tried to stop/start, remove, recreate or use netca to recreate.
But can not start by srvctl command-line.

$ srvctl start listener -n test01

CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.test01.LISTENER_TEST01.lsnr' has placement error.


So, I cleaned up OCR and Voting Disks ,then run "$ORA_CRS_HOME/install/rootconfig" and then run netca again to solve it.

Srvctl cannot start instance, I get the following error PRKP-1001 CRS-0215, however sqlplus can start it on both nodes

Today I can not start database cluster by using "srvctl", but start by "sqlplus" command.

$ srvctl start database -d DB
PRKP-1001 : Error starting instance DB1 on node node1
CRS-0215: Could not start resource 'ora.DB.DB1.inst'.
PRKP-1001 : Error starting instance DB2 on node node2
CRS-0215: Could not start resource 'ora.DB.DB2.inst'.

When I find out information:

>>>

Srvctl cannot start instance, I get the following error PRKP-1001 CRS-0215, however sqlplus can start it on both nodes? What is the problem?

This could be many things but a popular issue is when you have a separate ASM Home and the listener is running out of this home (it was the first home installed). Srvctl needs a TNS_ADMIN alias to the network/admin directory in this home instead of using the default ORACLE_HOME/network/admin for the database you are trying to start. For srvctl to work you must

srvctl setenv nodeapps -n node -T TNS_ADMIN=full path

on each node in the cluster.
You cannot rely on a TNS_ADMIN environment variable.

>>>

So, I set ENV on CLUSTER with Database (that make me this idea) and then start again:

$srvctl setenv database -d DB -T TNS_ADMIN=/oracle/network/admin/

$ srvctl start database -d DB

... Start OK...

Oracle crs_stat command

The crs_stat command provide status information for resource on the cluster node. To query resource with the crs_stat command.

Resource are either ONLINE or OFFLINE as shown in the STATE attribute. An application resource in the ONLINE state is running successfully on a cluster node.

The TARGET value shows the state to which Oracle Clusterware attempt to set the resource. If the TARGET value is ONLINE and a cluster node fails, then Oracle Clusterware attempts to restart the application on another node if possible. If there is a condition forcing a resource STATE to be OFFLINE, such as a required resource that is OFFLINE, then the TARGET value remains ONLINE and Oracle Clusterware attempts to start the application or application resource once the condition is corrected.

A TARGET value for all non-application resources should be ONLINE unless the resource has a failure count higher than the failure threshold, in which case the TARGET is changed to OFFLINE. The Oracle Clusterware then treats the resource as if its STATE were OFFLINE. If the STATE is ONLINE and the TARGET is OFFLINE, then you can reset the target value to ONLINE using the crs_start command.

Syntax and Options for crs_stat
Use the crs_stat command with the following syntax:

crs_stat [resource_name [...]] [-v] [-l] [-q] [-c cluster_node]

crs_stat [resource_name [...]] -t [-v] [-q] [-c cluster_node]

crs_stat -p [resource_name [...]] [-q]

crs_stat [-a] resource_name -g

crs_stat [-a] resource_name -r [-c cluster_node]

crs_stat -f [resource_name [...]] [-q] [-c cluster_node]

crs_stat -ls resource_name

* -v— Displays extended information about the status of resources. Extra attributes shown include RESTART_COUNT and FAILURE_COUNT. The RESTART_COUNT attribute shows the number of restarts of the application that have been attempted since it was started. The FAILURE_COUNT attribute shows how many failures have occurred during the last FAILURE_THRESHOLD in seconds. The attribute FAILOVER_STATUS shows the time at which an application resource started while waiting to relocate due to a cluster node failure if the resource has a FAILOVER_DELAY value greater than 0. It is not displayed otherwise.

* -l— Displays the status in a list (non-tabular) format. This is the default format.

* -t— Displays the status in tabular format.

* -p resource_name [...]]— Displays the status of the in-memory profile for the specified resources. If no resources are specified, then the status of the in-memory profile for all registered resources is displayed. If a resource is not registered, its status is not displayed.

* -a resource_name— Used with the -g or -r option to verify whether the specified resource is registered or running under Oracle Clusterware. Specify the name of the resource as listed in its application profile.

* -f— Displays all information about the resource including extended information (-v) and the in-memory profile (-p).

* -g— Returns 0 (zero) if the specified application or application resource is registered with Oracle Clusterware; returns 1 if it is not. This option only works if a single resource is specified with the -a option.

* -r— Returns 0 (zero) if the specified application or application resource is running under Oracle Clusterware; returns 1 if it is not. This option can only be successful if a single resource is specified with the -a option.

* -c cluster_node— Displays information about applications or application resources on the specified cluster node.

Examples of crs_stat
NAME=ora.DB.SDB.DB3.srv
TYPE=application
TARGET=ONLINE
STATE=ONLINE on db04

NAME=ora.DB.SDB.cs
TYPE=application
TARGET=ONLINE
STATE=ONLINE on db03

Examples of crs_stat -v
NAME=ora.DB.SDB.DB3.srv
TYPE=application
RESTART_ATTEMPTS=0
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.DB.SDB.DB4.srv
TYPE=application
RESTART_ATTEMPTS=0
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=OFFLINE
STATE=OFFLINE


Examples of crs_stat -t

Name Type Target State Host
------------------------------------------------------------
ora....DB3.srv application OFFLINE OFFLINE
ora....DB4.srv application OFFLINE OFFLINE
ora....B1.inst application ONLINE ONLINE db01
ora....B2.inst application ONLINE ONLINE db02

Anyway we can write script to easy.
Example "crstat" script>>
#!/usr/bin/ksh
RSC_KEY=$1
QSTAT=-u
AWK=/usr/bin/awk # if not available use /usr/bin/awk
# Table header:echo ""
$AWK \
'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State"; printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}'
# Table body:
$ORA_CRS_HOME/bin/crs_stat $QSTAT | $AWK \
'BEGIN { FS="="; state = 0; }
$1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1};
state == 0 {next;}
$1~/TARGET/ && state == 1 {apptarget = $2; state=2;}
$1~/STATE/ && state == 2 {appstate = $2; state=3;}
state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}'
----------->>>
#./crstat
HA Resource Target State
----------- ------ -----
ora.DB.SDB.DB3.srv OFFLINE OFFLINE
ora.DB.SDB.DB4.srv OFFLINE OFFLINE
ora.DB.DB1.inst ONLINE ONLINE on db01
ora.DB.DB2.inst ONLINE ONLINE on db02

That is a good command help us to provide status information for resource on the cluster node.

Verification of OCR integrity was unsuccessful on all the nodes.

I found Verification of OCR integrity was unsuccessful on all the nodes.

$ cluvfy comp ocr

Verifying OCR integrity


ERROR:
User equivalence unavailable on all the nodes.
Verification cannot proceed.


Verification of OCR integrity was unsuccessful on all the nodes.

So,Check by the oracle software owner can ssh from one node to the other, and back again

#ssh-agent $SHELL
#ssh-add

#$ cluvfy comp ocr

Verifying OCR integrity

Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations.

Uniqueness check for OCR device passed.

Checking the version of OCR...
OCR of correct Version "2" exists.

Checking data integrity of OCR...
Data integrity check for OCR passed.

OCR integrity check passed.

Verification of OCR integrity was successful.

10.2 CRS + X86_64 [PRKN-1008 : Unable to load the shared library "srvmhas10"

On 64bit linux, Startingvipca through root.sh at install time returns following error
PRKH-1010 : Unable to communicate with CRS services.
[PRKH-1000 : Unable to load the SRVM HAS shared library
[PRKN-1008 : Unable to load the shared library "srvmhas10"
or a dependent library, from
LD_LIBRARY_PATH="/oracle/product/10.2.0/crs/jdk/jre/lib/i386/client:
/oracle/product/10.2.0/crs/jdk/jre/lib/i386:/oracle/product/10.2.0/crs/jdk/jre/
../lib/i386:/opt/oracle/product/10.2.0.1/crs/lib32:
/oracle/product/10.2.0/crs/srvm/lib32:/oracle/product/10.2.0/crs/lib:
/oracle/product/10.2.0/crs/srvm/lib:/oracle/product/10.2.0
/crs/lib"


Cause:

Missing library "libclntsh.so.10.1" under $ORA_CRS_HOME/lib32

Bug 4189481 Vipca/Srvctl: Prkn-1008 : Unable To Load The Shared Library "Libsrvmhas.So"


You may try following.

1. shutdown crs on both nodes.
crsctl stop crs

2. backup files under $CRS_HOME/lib
and $CRS_HOME/lib32

3. Log in as root user.

4.set ORACLE_HOME to $CRS_HOME
ie)
export ORACLE_HOME=xxxx/yyyy

5. cd $ORACLE_HOME/network/lib

6. make -f ins_net_client.mk client_sharedlib

7. Change owner,group, permission on files genereed by 6 to oracle,
oinstall ,
755 (check files generated by checking timestamp of files).

8. Check if sybolic link libclntsh.so pointing to
$CRS_HOME/lib/libclntsh.so.10.1 exists under $CRS_HOME/lib as before.
If not,create one.

9. startup crs

Anyway this case you should check pacakges before.... 

So, install glibc-2*.i686.rpm and glibc-devel-2*.i386.rpm and reinstall



*** Just install glibc-2*.i686.rpm and glibc-devel-2*.i386.rpm

>>>

The given interface(s), "eth0" is not public. Public interfaces should be used to configure virtual IPs.

When I install oracle 10G release 2

and i run root.sh, i found below error from some node.

During CRS install while running root.sh, The following messages are displayed
Oracle CRS stack installed and running under init(1M)Running vipca(silent) for configuring nodeappsThe given interface(s), "eth0" is not public. Public interfaces should be used to configure virtual IPs.Changes


You are running root.shCause
When verifying the IP addresses, VIP uses calls to determine if a IP address is valid or not. In this case, VIP finds that the IPs are non routable (For example IP addresses like 192.168.* and 10.10.*.)

Oracle is aware that the IP's can be made public but since mostly such IP's are used for Private, it display this error message.Solution
The workaround is to re-run vipca manually as root
#./vipca

or add the VIP using srvctl add nodeapps

-----------------------