Fun!!! - Changed hostname on GI (Standalone Server)

Changed hostname on Grid Infrastructure for a Standalone Server. why did I talk about this? I just changed hostname in my virtualbox and rebooted it. I got error in /var/log/messages file(The Grid Infrastructure didn't start).

Oct 27 16:20:07 new_mytest logger: Oracle Cluster Ready Services startup disabled.
Oct 27 16:20:07 new_mytest logger: Could not access /etc/oracle/scls_scr/mylinux/oracle/ohasdstr.
Oct 27 16:20:13 new_mytest logger: autorun file for ohasd is missing
Oct 27 16:20:23 new_mytest logger: autorun file for ohasd is missing

So, I just tested/wrote about changed hostname on Grid Infrastructure for a Standalone Server. Not serious, just fun!!!
Example to change hostname from "mytest" to "mylinux" :)
*** Keep some resource informations ***

[root@mytest ~]# mkdir mytest
[root@mytest ~]# for x in ` /u01/app/oracle/product/11.2.0/grid/bin/crsctl stat res | grep NAME\= | awk -F\= '{print $2}'`; do /u01/app/oracle/product/11.2.0/grid/bin/crsctl stat res $x -p > mytest/$x.txt ; done
[root@mytest ~]# ls mytest
ora.asm.txt ora.DATA.dg.txt ora.evmd.txt ora.ons.txt
ora.cssd.txt ora.diskmon.txt ora.LISTENER.lsnr.txt ora.orcl.db.txt

A. Stopped HAS

[root@mytest ~]# /u01/app/oracle/product/11.2.0/grid/bin/crsctl stop has -f

B. Changed hostname

[root@mytest ~]# vi /etc/sysconfig/network
[root@mytest ~]# vi /etc/hosts
[root@mytest ~]# vi /u01/app/oracle/product/11.2.0/grid/network/admin/listener.ora
[root@mytest ~]# hostname
mytest
[root@mytest ~]# hostname mylinux
[root@mylinux ~]# hostname
mylinux

C. Started to change hostname with Grid Infrastructure for a Standalone Server.

[root@mytest ~]# /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl -h
Option hahome requires an argument
Usage:
roothas.pl [-verbose] [-upgrade | -patch]
[-paramfile <parameter-file>] [-deconfig] [-force]
[-unlock [-hahome <path to Oracle Restart home>]]

Options:
-verbose Run this script in verbose mode
-upgrade Oracle HA is being upgraded from previous version
-patch Oracle HA is being upgraded to a patch version
-paramfile Complete path of file specifying HA parameter values
-deconfig To deconfigure Oracle Restart
-force To deconfigure Oracle Restart when configuration is in a good condition
-unlock Unlock Oracle Restart home
-hahome Complete path of Oracle HA home

If neither -upgrade nor -patch is supplied, a new install is performed

To see the full manpage for this program, execute:
perldoc roothas.pl

[root@mytest ~]# /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig -force
Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params

CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Delete failed, or completed with errors.
CLSU-00100: Operating System function: opendir failed with error data: 2
CLSU-00101: Operating System error message: No such file or directory
CLSU-00103: error location: scrsearch1
CLSU-00104: additional error information: cant open scr home dir scls_scr_getval
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully deconfigured Oracle Restart stack

[root@mylinux ~]# /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl -verbose
Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node mylinux successfully pinned.
Adding Clusterware entries to inittab
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
Oracle Database 11g Clusterware Release 11.2.0.3.0 - Production Copyright 1996, 2011 Oracle. All rights reserved.
[client(3619)]CRS-10001:27-Oct-11 18:54 ACFS-9201: Not Supported
2011-10-27 18:56:06.170
[client(3703)]CRS-2101:The OLR was formatted using version 3.
2011-10-27 18:56:09.776
[client(3727)]CRS-1001:The OCR was formatted using version 3.
[client(3770)]CRS-10001:CRS-6021: No msg for has:crs-6021 [l][unlimited]
[client(3771)]CRS-10001:CRS-6021: No msg for has:crs-6021 [n][65536]

*** Fixed them (ulimit) and fixed some... ***

[root@mylinux ~]# ps -aef | grep grid
oracle 3768 1 0 18:56 ? 00:00:00 /u01/app/oracle/product/11.2.0/grid/bin/ohasd.bin reboot
oracle 4213 1 0 19:05 ? 00:00:00 /u01/app/oracle/product/11.2.0/grid/bin/ohasd.bin reboot
oracle 4659 1 0 19:12 ? 00:00:00 /u01/app/oracle/product/11.2.0/grid/bin/ohasd.bin reboot

[root@mylinux ~]# kill -9 3768 4213 4659
[root@mylinux ~]# ps -aef | grep grid
oracle 4875 1 4 19:16 ? 00:00:00 /u01/app/oracle/product/11.2.0/grid/bin/ohasd.bin restart

*** After error and then run script again!!! Just make sure, NO "ohasd.bin reboot" on OS processes ***
*** De-configuration and run "roothas.pl" script again. ***

[root@mylinux ~]# /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig
Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
CRS-2613: Could not find resource 'ora.cssd'.
CRS-4000: Command Stop failed, or completed with errors.
CRS-2613: Could not find resource 'ora.cssd'.
CRS-4000: Command Delete failed, or completed with errors.
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle Restart stack

[root@mylinux ~]# /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl -verbose
Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node mylinux successfully pinned.
Adding Clusterware entries to inittab

mylinux 2011/10/27 19:20:36 /u01/app/oracle/product/11.2.0/grid/cdata/mylinux/backup_20111027_192036.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server

[root@mytest ~]# /u01/app/oracle/product/11.2.0/grid/bin/crsctl check has
CRS-4638: Oracle High Availability Services is online

[root@mytest ~]# /u01/app/oracle/product/11.2.0/grid/bin/crsctl stat res | grep NAME\=
NAME=ora.cssd
NAME=ora.diskmon
NAME=ora.evmd
NAME=ora.ons

*** Checked in backup for resource informations ***

[root@mytest ~]# ls mytest
ora.asm.txt ora.DATA.dg.txt ora.evmd.txt ora.ons.txt
ora.cssd.txt ora.diskmon.txt ora.LISTENER.lsnr.txt ora.orcl.db.txt

Reviewed resources what was not register.
*** Try to use crsctl add res resource_name -type type_name -file mytest/file.txt ***
*** But Error - CRS-0175: Resource type 'type_name' either doesn't exist or cannot be accessed. ***
*** OK use "srvctl" for manual ***

D. Added other resources by manual (srvctl)
- Added Listener (changed listener.ora file before)

[root@mytest ~]# su - oracle
[oracle@mylinux ~]$ srvctl -h | grep listener | grep add
Usage: srvctl add listener [-l <lsnr_name>] [-s] [-p "[TCP:]<port>[, ...][/IPC:<key>][/NMP:<pipe_name>][/TCPS:<s_port>] [/SDP:<port>]"] [-o <oracle_home>]
[oracle@mylinux ~]$ srvctl add listener -l LISTENER -p "TCP:1521" -o /u01/app/oracle/product/11.2.0/grid
[oracle@mylinux ~]$ srvctl start listener
[oracle@mylinux ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): mylinux

- Added ASM

[root@mytest ~]# grep DISK mytest/ora.asm.txt
ASM_DISKSTRING=DATA
[root@mytest ~]# grep ASM_DISKSTRING mytest/ora.asm.txt
ASM_DISKSTRING=DATA
[root@mytest ~]# grep SP mytest/ora.asm.txt
SPFILE=
[root@mytest ~]# su - oracle
[oracle@mylinux ~]$ srvctl -h | grep asm | grep add
Usage: srvctl add asm [-l <lsnr_name>] [-p <spfile>] [-d <asm_diskstring>]
[oracle@mylinux ~]$ srvctl add asm -l LISTENER -p '' -d 'DATA'
[oracle@mylinux ~]$ srvctl start asm
[oracle@mylinux ~]$ srvctl status asm

[oracle@mylinux ~]$ /u01/app/oracle/product/11.2.0/grid/bin/crsctl stat res | grep NAME\=
NAME=ora.LISTENER.lsnr
NAME=ora.asm
NAME=ora.cssd
NAME=ora.diskmon
NAME=ora.evmd
NAME=ora.ons

[oracle@mylinux ~]$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/grid
[oracle@mylinux ~]$ export ORACLE_SID=+ASM
[oracle@mylinux ~]$ sqlplus / as sysasm

SQL> select name from v$asm_diskgroup;

NAME
------------------------------
DATA

SQL> alter diskgroup DATA mount;

Diskgroup altered.

[oracle@mylinux ~]$ /u01/app/oracle/product/11.2.0/grid/bin/crsctl stat res | grep NAME\=
NAME=ora.DATA.dg
NAME=ora.LISTENER.lsnr
NAME=ora.asm
NAME=ora.cssd
NAME=ora.diskmon
NAME=ora.evmd
NAME=ora.ons

- Added Database

[root@mytest ~]# grep SPFILE mytest/ora.orcl.db.txt
SPFILE=/u01/app/oracle/product/11.2.0/dbhome_2/dbs/spfileorcl.ora
[root@mytest ~]# grep ORACLE_HOME\= mytest/ora.orcl.db.txt
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_2

[oracle@mylinux ~]$ srvctl -h | grep database | grep add
Usage: srvctl add database -d <db_unique_name> -o <oracle_home> [-m <domain_name>] [-p <spfile>] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY | SNAPSHOT_STANDBY}] [-s <start_options>] [-t <stop_options>] [-n <db_name>] [-i <inst_name>] [-y {AUTOMATIC | MANUAL | NORESTART}] [-a "<diskgroup_list>"]
[oracle@mylinux ~]$ srvctl add database -d orcl -o /u01/app/oracle/product/11.2.0/dbhome_2 -p /u01/app/oracle/product/11.2.0/dbhome_2/dbs/spfileorcl.ora
[oracle@mylinux ~]$ srvctl start database -d orcl

I learned some when I used "crsctl add res resource_name -type type_name -file mytest/file.txt". I got error "CRS-0175: Resource type 'type_name' either doesn't exist or cannot be accessed", that mean I have to add "resource type" before. But "srvctl" command, that will add "resource type" automatic.

Related Post:
Just Test - Install Oracle GI (standalone server) on OL6
Upgrade Grid Infrastructure for cluster (11.2.0.2 to 11.2.0.3)
11.2.0.3 GI diskmon will be OFFLINE by default in Non-Exadata

Just Learned more - Rename an Oracle database name -)

It's nothing in this post. I just changed Oracle database name and did it something for Grid Infrastructure for standalone server also.
- Check

SQL> select name from v$database;

NAME
---------
ORCLOLD

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area 835104768 bytes
Fixed Size 2217952 bytes
Variable Size 503318560 bytes
Database Buffers 322961408 bytes
Redo Buffers 6606848 bytes
Database mounted.

- Backup before!!!

$ rman target /
RMAN> backup database;
Starting backup at 31-AUG-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=125 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=+DATA/orclold/datafile/system.293.760536553
input datafile file number=00002 name=+DATA/orclold/datafile/sysaux.294.760536553
input datafile file number=00003 name=+DATA/orclold/datafile/undotbs1.295.760536553
input datafile file number=00004 name=+DATA/orclold/datafile/users.296.760536553
channel ORA_DISK_1: starting piece 1 at 31-AUG-11
channel ORA_DISK_1: finished piece 1 at 31-AUG-11
piece handle=+DATA/orclold/backupset/2011_08_31/nnndf0_tag20110831t120028_0.284.760622429 tag=TAG20110831T120028 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:02:25
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 31-AUG-11
channel ORA_DISK_1: finished piece 1 at 31-AUG-11
piece handle=+DATA/orclold/backupset/2011_08_31/ncsnf0_tag20110831t120028_0.285.760622577 tag=TAG20110831T120028 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 31-AUG-11

- Use "nid" to change Oracle database name.

$ nid TARGET=sys/password DBNAME=orcl LOGFILE=/tmp/nid.log
$ cat /tmp/nid.log
DBNEWID: Release 11.2.0.1.0 - Production on Wed Aug 31 12:06:17 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.

Connected to database ORCLOLD (DBID=685834868)

Connected to server version 11.2.0

Control Files in database:
+DATA/orclold/controlfile/current.298.760536693
+DATA/orclold/controlfile/current.297.760536693

Changing database ID from 685834868 to 1288455737
Changing database name from ORCLOLD to ORCL
Control File +DATA/orclold/controlfile/current.298.760536693 - modified
Control File +DATA/orclold/controlfile/current.297.760536693 - modified
Datafile +DATA/orclold/datafile/system.293.76053655 - dbid changed, wrote new name
Datafile +DATA/orclold/datafile/sysaux.294.76053655 - dbid changed, wrote new name
Datafile +DATA/orclold/datafile/undotbs1.295.76053655 - dbid changed, wrote new name
Datafile +DATA/orclold/datafile/users.296.76053655 - dbid changed, wrote new name
Datafile +DATA/orclold/tempfile/temp.305.76053675 - dbid changed, wrote new name
Control File +DATA/orclold/controlfile/current.298.760536693 - dbid changed, wrote new name
Control File +DATA/orclold/controlfile/current.297.760536693 - dbid changed, wrote new name
Instance shut down

Database name changed to ORCL.
Modify parameter file and generate a new password file before restarting.
Database ID for database ORCL changed to 1288455737.
All previous backups and archived redo logs for this database are unusable.
Database is not aware of previous backups and archived logs in Recovery Area.
Database has been shutdown, open database with RESETLOGS option.
Succesfully changed database name and ID.
DBNEWID - Completed succesfully.

- Do others things.

$ mv /u01/app/oracle/11.2.0/dbhome_old/dbs/initorclold.ora /u01/app/oracle/11.2.0/dbhome_old/dbs/initorcl.ora
$ export ORACLE_SID=orcl
$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Wed Aug 31 12:09:30 2011

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.

Total System Global Area 835104768 bytes
Fixed Size 2217952 bytes
Variable Size 461375520 bytes
Database Buffers 364904448 bytes
Redo Buffers 6606848 bytes
SQL> show parameter db_name

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_name string orclold

SQL> alter system set db_name=orcl scope=spfile;

System altered.

SQL> shutdown immediate;
ORA-01507: database not mounted

ORACLE instance shut down.

$ mv /u01/app/oracle/11.2.0/dbhome_old/dbs/orapworclold /u01/app/oracle/11.2.0/dbhome_old/dbs/orapworcl

$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Wed Aug 31 12:20:25 2011

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 835104768 bytes
Fixed Size 2217952 bytes
Variable Size 461375520 bytes
Database Buffers 364904448 bytes
Redo Buffers 6606848 bytes
Database mounted.
SQL> ALTER DATABASE OPEN RESETLOGS;

Database altered.

- Check new Database name and etc...

SQL> select name from v$database;

NAME
---------
ORCL

SQL> !lsnrctl status | grep Service
Service "orcl" has 1 instance(s).

SQL> select name, value from v$parameter where value like '%orclold%';
-- check --

Remark:

Thank You Laurent Schneider.
To rename only :
nid setname=yes

If You need to rename Oracle database name only.
Keyword Description (Default)
----------------------------------------------------
SETNAME Set a new database name only NO

I use Grid Infrastructure also ... have to change something. but I chose to add new - -*

$ crsctl stat res | grep NAME\= | grep db
NAME=ora.orclold.db

$ crsctl stat res ora.orclold.db -p
NAME=ora.orclold.db
TYPE=ora.database.type
ACL=owner:grid:--x,pgrp:asmdba:--x,other::r--,group:oinstall:r-x,user:oracle:rwx
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
ACTIVE_PLACEMENT=1
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CARDINALITY=1
CHECK_INTERVAL=1
CHECK_TIMEOUT=600
CLUSTER_DATABASE=false
DATABASE_TYPE=
DB_UNIQUE_NAME=orclold
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=database) PROPERTY(DB_UNIQUE_NAME= CONCAT(PARSE(%NAME%, ., 2), %USR_ORA_DOMAIN%, .)) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)
DEGREE=1
DESCRIPTION=Oracle Database resource
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=60
FAILURE_THRESHOLD=1
GEN_AUDIT_FILE_DEST=
GEN_START_OPTIONS=
GEN_USR_ORA_INST_NAME=
HOSTING_MEMBERS=
INSTANCE_FAILOVER=0
LOAD=1
LOGGING_LEVEL=1
MANAGEMENT_POLICY=AUTOMATIC
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
ONLINE_RELOCATION_TIMEOUT=0
ORACLE_HOME=/u01/app/oracle/11.2.0/dbhome_old
PLACEMENT=balanced
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=2
ROLE=PRIMARY
SCRIPT_TIMEOUT=60
SERVER_POOLS=
SPFILE=+DATA/orclold/spfileorclold.ora
START_DEPENDENCIES=hard(ora.DATA.dg) weak(type:ora.listener.type,uniform:ora.ons,uniform:ora.eons) pullup(ora.DATA.dg)
START_TIMEOUT=600
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg)
STOP_TIMEOUT=600
TYPE_VERSION=2.2
UPTIME_THRESHOLD=1h
USR_ORA_DB_NAME=orclold
USR_ORA_DOMAIN=
USR_ORA_ENV=
USR_ORA_FLAGS=
USR_ORA_INST_NAME=orclold
USR_ORA_OPEN_MODE=open
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.1.0

$ crsctl stat res ora.orclold.db -p > /tmp/ora.orclold.db.res

*** Modified something in ora.orclold.db.res file.***

- Add new resource (after modified)

$ crsctl add res ora.orcl.db -type ora.database.type -file /tmp/ora.orclold.db.res

$ crsctl stat res | grep NAME\= | grep db
NAME=ora.orcl.db
NAME=ora.orclold.db

$ crsctl start res ora.orcl.db
CRS-2672: Attempting to start 'ora.orcl.db' on 'mytest'
CRS-5010: Update of configuration file "/u01/app/oracle/11.2.0/dbhome_old/srvm/admin/oratab.bak.mytest" failed: details at "(:CLSN00011:)" in "/u01/app/oracle/11.2.0/grid/log/mytest/agent/ohasd/oraagent_grid/oraagent_grid.log"
CRS-2676: Start of 'ora.orcl.db' on 'mytest' succeeded

$ crsctl status res ora.orcl.db
NAME=ora.orcl.db
TYPE=ora.database.type
TARGET=ONLINE
STATE=ONLINE on mytest

$ crsctl stop res ora.orcl.db
CRS-2673: Attempting to stop 'ora.orcl.db' on 'mytest'
CRS-5017: The resource action "ora.orcl.db stop" encountered the following error:
CRS-5003: Invalid attribute value: '' for attribute DATABASE_TYPE
CRS-2675: Stop of 'ora.orcl.db' on 'mytest' failed
CRS-2679: Attempting to clean 'ora.orcl.db' on 'mytest'
CRS-2681: Clean of 'ora.orcl.db' on 'mytest' succeeded

$ crsctl status res ora.orcl.db
NAME=ora.orcl.db
TYPE=ora.database.type
TARGET=OFFLINE
STATE=OFFLINE

So, You can use "srvctl"

srvctl add database -d orcl -o /u01/app/oracle/11.2.0/dbhome_old -p +DATA/orclold/spfileorclold.ora

- Delete old!!!

$ su -
Password:
# /u01/app/oracle/11.2.0/grid/bin/crsctl delete res ora.orclold.db

About CRS-5010 !!! during "crsctl start res ora.orcl.db" - because I use "grid" user for Grid Infrastructure and "oracle" for database software.

$ su - oracle
Password:
$ chmod 775 /u01/app/oracle/11.2.0/dbhome_old/srvm/admin

About "CRS-5003: Invalid attribute value: '' for attribute DATABASE_TYPE"

$ crsctl stat res ora.orcl.db -p | grep DATABASE_TYPE
DATABASE_TYPE=

OK, It should be "DATABASE_TYPE=SINGLE"

$ su -
Password:
# /u01/app/oracle/11.2.0/grid/bin/crsctl modify res ora.orcl.db -attr 'DATABASE_TYPE=SINGLE'

- Test "crsctl"

$ crsctl start res ora.orcl.db
CRS-2672: Attempting to start 'ora.orcl.db' on 'mytest'
CRS-2676: Start of 'ora.orcl.db' on 'mytest' succeeded
$ crsctl stop res ora.orcl.db
CRS-2673: Attempting to stop 'ora.orcl.db' on 'mytest'
CRS-2677: Stop of 'ora.orcl.db' on 'mytest' succeeded
$ crsctl start res ora.orcl.db
CRS-2672: Attempting to start 'ora.orcl.db' on 'mytest'
CRS-2676: Start of 'ora.orcl.db' on 'mytest' succeeded
$ crsctl status res ora.orcl.db
NAME=ora.orcl.db
TYPE=ora.database.type
TARGET=ONLINE
STATE=ONLINE on mytest

No error -)

Creating Database 11gR1 on ASM 11gR2

On Forums - show 10g database creating with 11gR2 ASM failed.
If we check on oracle support, we can use [ID 949073.1] ... That's helpful issue to fix.

To run older databases on 11gR2 ASM and Clusterware we need to create a persistent configuration for the nodes(pinning a node). Because Cluster configuration in 11gR2 is dynamic. This dynamic configuration is incompatible with older database releases.

On Oracle Document about "Pinning Cluster Nodes for Oracle Database Release 10.x or 11.x".

Time To Test It... (Creating Database 11gR1 on ASM 11gR2) and error.

ORA-19624: operation failed, retry possible
ORA-19870: error while restoring backup piece /oracle/product/11.1/db/assistants/dbca/templates/Seed_Database.dfb
ORA-19504: failed to create file "+DISK02"
ORA-17502: ksfdcre:4 Failed to create file +DISK02
ORA-15001: diskgroup "DISK02" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Manager
ORA-06512: at "SYS.X$DBMS_BACKUP_RESTORE", line 5662
ORA-06512: at line 40

Check ASM... about database compatibility.

SQL> select name, compatibility, database_compatibility, state from v$asm_diskgroup_stat;

NAME COMPATIBILITY DATABASE_COMPAT STATE
------------------------------ --------------- --------------- -----------
DISK00 11.2.0.0.0 10.1.0.0.0 MOUNTED
DISK01 11.1.0.0.0 11.1.0.0.0 MOUNTED
DISK02 11.1.0.0.0 11.1.0.0.0 MOUNTED
DISK03 11.2.0.0.0 11.2.0.0.0 MOUNTED

then pin node, by crsctl pin css

# cd /oracle/grid2/
# cd bin/
# ./olsnodes -t -n
rhel5-test 1 Unpinned

# ./crsctl pin css -n rhel5-test
CRS-4664: Node rhel5-test successfully pinned.

# ./olsnodes -t -n
rhel5-test 1 Pinned

then create database again... and no problem.

then register my database(11gR1) in Grid.

$ srvctl add database -d orcl11gr1 -o /oracle/product/11.1/db -p +DISK02/orcl11gr1/spfileorcl11gr1.ora -y AUTOMATIC
$ srvctl add instance -d orcl11gr1 -i orcl11gr1 -n rhel5-test

$ srvctl start database -d orcl11gr1
$ srvctl status database -d orcl11gr1
Instance orcl11gr1 is running on node rhel5-test

$ ./crsstat ora.orcl11gr1
HA Resource Target State (Host)
----------- ------ -----
ora.orcl11gr1.db ONLINE ONLINE on rhel5-test

How to register ASM diskgroup in Grid Infrastructure

It's a cool today, Someone are working, Someone are on vacation.
By the way. Have good holidays ;)

I read/tested Relocating Grid Infrastructure. After I relocated my Grid Infrastructure. I'd like to register ASM diskgroup else.

$ ./crsstat -word dg
HA Resource Type Target State (Host)
----------- ------ ------ -----
ora.DISK00.dg ora.diskgroup.type ONLINE ONLINE on rhel5-test

Some disk groups... not registered in Grid. I had to register by manual, then I checked srvctl command-line, but no "add" diskgroup.

$ srvctl -h | grep diskgroup | grep dg_name
Usage: srvctl start diskgroup -g <dg_name> [-n "<node_list>"]
Usage: srvctl stop diskgroup -g <dg_name> [-n "<node_list>"] [-f]
Usage: srvctl status diskgroup -g <dg_name> [-n "<node_list>"] [-a]
Usage: srvctl enable diskgroup -g <dg_name> [-n "<node_list>"]
Usage: srvctl disable diskgroup -g <dg_name> [-n "<node_list>"]
Usage: srvctl remove diskgroup -g <dg_name> [-f]

Actually, ASM disk groups are registered in Grid by Automatic, After we mounted them ???

$ sqlplus / as sysasm

SYS> alter diskgroup DISK01 mount;

Diskgroup altered.

SYS> alter diskgroup DISK02 mount;

Diskgroup altered.

SYS> alter diskgroup DISK03 mount;

Diskgroup altered.

$ ./crsstat -word dg
HA Resource Type Target State (Host)
----------- ------ ------ -----
ora.DISK00.dg ora.diskgroup.type ONLINE ONLINE on rhel5-test
ora.DISK01.dg ora.diskgroup.type ONLINE ONLINE on rhel5-test
ora.DISK02.dg ora.diskgroup.type ONLINE ONLINE on rhel5-test
ora.DISK03.dg ora.diskgroup.type ONLINE ONLINE on rhel5-test

By the way, It's just fun today. Nothing special.

Reference:
#!/bin/bash
# crsstat v. 0.01
# By surachart
if [ $# -gt 4 ]
then
echo "Please Check arguments."
echo
echo "./crsstat [-word|--word] resource_word [-crshome|--crshome] CRS_HOME "
echo
echo " ./crsstat"
echo " ./crsstat -word orcl"
echo ' ./crsstat -word orcl -crshome $CRS_HOME'
exit 0
fi

while [ $# -gt 0 ]; do
case "$1" in
-h|--h|-help|--help)
echo "./crsstat [-word|--word] resource_word [-crshome|--crshome] CRS_HOME "
echo
echo " ./crsstat"
echo " ./crsstat -word orcl"
echo ' ./crsstat -word orcl -crshome $CRS_HOME'
exit 0
;;
-word|--word)
GREP_KEY=$2
shift
;;
-crshome|--crshome)
ORA_CRS_HOME=$2
shift
;;
esac
shift
done

if [ -z $ORA_CRS_HOME ]
then
ORA_CRS_HOME=$CRS_HOME
if [ ! -d $ORA_CRS_HOME ]
then
echo "Please Check CRS_HOME Environment ($ORA_CRS_HOME)"
exit 1
fi

elif [ ! -d $ORA_CRS_HOME ]
then
ORA_CRS_HOME=$CRS_HOME
if [ ! -d $ORA_CRS_HOME ]
then
echo "Please Check CRS_HOME Environment ($ORA_CRS_HOME)"
exit 1
fi
fi

CRS_STAT=$ORA_CRS_HOME/bin/crs_stat

if [ ! -x $CRS_STAT ]
then
echo "Please Check CRS_HOME Environment ($ORA_CRS_HOME)"
exit 2
fi

AWK=/usr/bin/awk

if [ ! -x $AWK ]
then
AWK=/bin/awk
fi

$AWK \
'BEGIN {printf "%-40s %-25s %-10s %-12s\n", "HA Resource", "Type", "Target", "State (Host)"; printf "%-40s %-25s %-10s %-12s\n", "-----------", "------", "------", "-----";}'
$CRS_STAT | $AWK \
'BEGIN { FS="="; state = 0; }
$1~/NAME/ && $2~/'$GREP_KEY'/ {appname = $2; state=1;}
state == 0 {next;}
$1~/TYPE/ && state == 1 {apptype = $2; state=2;}
$1~/TARGET/ && state == 2 {apptarget = $2; state=3;}
$1~/STATE/ && state == 3 {appstate = $2; state=4;}
state == 4 {printf "%-40s %-25s %-10s %-12s\n", appname,apptype, apptarget, appstate;state=0}'

How to Setup 10gRAC with Solaris (IPMP)

On Solaris, IP network multipathing (IPMP) provides physical interface failure detection and transparent network access failover for a system with multiple interfaces on the same IP link.

On 10gRAC, It's a good question, How to Setup 10gRAC with Solaris (IPMP) ?

Setup Cluster Interconnect (metalink: 368464.1):

rac01:

- Physical IP : 10.10.10.1

- Test IP for ce2 : 10.10.10.11

- Test IP for ce3 : 10.10.10.12

rac02:

- Physical IP : 10.10.10.2

- Test IP for ce2 : 10.10.10.21

- Test IP for ce3 : 10.10.10.22

Check & configure...

oifcfg getif | grep -i cluster_interconnect

If already configuration... with "cluster_interconnect", then delete with 'oifcfg delif'

oifcfg delif [-node <nodename> | -global] [ <if_name> [/ <subnet> ]]

$ oifcfg getif | grep -i cluster_interconnect
ce2 10.10.10.0 global cluster_interconnect
ce3 10.10.10.0 global cluster_interconnect

$ oifcfg delif -global ce2

$ oifcfg delif -global ce3

set the CLUSTER_INTERCONNECTS parameter in the spfile/init.ora (RDBMS/ASM) to the Physical IP (IPMP)... and check on GV$CLUSTER_INTERCONNECTS as well.

rac database:

rac1.CLUSTER_INTERCONNECTS=10.10.10.1

rac2.CLUSTER_INTERCONNECTS=10.10.10.2

SQL> ALTER SYSTEM SET CLUSTER_INTERCONNECTS = '10.10.10.1' scope=spfile sid='rac1';

SQL>ALTER SYSTEM SET CLUSTER_INTERCONNECTS = '10.10.10.2' scope=spfile sid='rac2';

on ASM:

+ASM1.CLUSTER_INTERCONNECTS=10.10.10.1

+ASM2.CLUSTER_INTERCONNECTS=10.10.10.2

.

.

virtual IP (metalink:283107.1):

rac01:

- Physical IP : 192.168.10.10

- Test IP for ce0 : 192.168.10.11

- Test IP for ce1 : 192.168.10.12

- Oracle VIP : 192.168.10.1

rac02:

- Physical IP : 192.168.10.20

- Test IP for ce0 : 192.168.10.21

- Test IP for ce1 : 192.168.10.22

- Oracle VIP : 192.168.10.2

=> New 10gRAC installation:

In vipca (VIP Configuration Assistant, 1 of 2), select all NIC's within the same IPMP group where the VIP should run...

=> Existing 10gRAC installation, use "srvctl" to modify:

http://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/srvctladmin.htm#CHDBJBIG

# srvctl config nodeapps -n rac01 -a

VIP exists.: /rac01-vip/192.168.10.1/255.255.255.0/ce0

# srvctl stop nodeapps -n rac01

# srvctl modify nodeapps -n rac01 -A 192.168.10.1/255.255.255.0/ce0\|ce1

# srvctl config nodeapps -n rac01 -a

VIP exists.: /rac01-vip/192.168.10.1/255.255.255.0/ce0:ce1

# srvctl start nodeapps -n rac01

# srvctl config nodeapps -n rac02 -a

VIP exists.: /rac02-vip/192.168.10.2/255.255.255.0/ce0

# srvctl stop nodeapps -n rac02

# srvctl modify nodeapps -n rac02 -A 192.168.10.2/255.255.255.0/ce0\|ce1

# srvctl config nodeapps -n rac02 -a

VIP exists.: /rac02-vip/192.168.10.2/255.255.255.0/ce0:ce1

# srvctl start nodeapps -n rac02

After modifed... use "srvctl config nodeapps -n nodename -a" to check.

srvctl start service; PRKP-1030 : Failed to start the service...ORA-44305: service ... is running

After I stoped  S01 service on node4 by srvctl command-line. It's OK .

$ srvctl stop service -d DB -s S01 -i DB4

After that, I need to start S01 service on this (node4) node (DB4)

$ srvctl start service -d DB -s S01 -i DB4

PRKP-1030 : Failed to start the service S01.

CRS-0215: Could not start resource 'ora.DB.S01.DB4.srv'

that's make me!... confused. So, I tried another idea... (try to force stop)

$ srvctl stop  service -d DB -s S01 -i DB4

PRKP-1065 : Service S01 is already stopped on instance DB4.

$ srvctl stop  service -d DB -s S01 -i DB4 -f 

PRKP-1065 : Service S01 is already stopped on instance DB4.

$ srvctl start service -d DB -s S01 -i DB4

PRKP-1030 : Failed to start the service S01.

CRS-0215: Could not start resource 'ora.DB.S01.DB4.srv'

that's not help me anything.

Check... Use "crs_stat"

ora.DB.S01.DB3.srv                         ONLINE     ONLINE on node03  

ora.DB.S01.DB4.srv                         ONLINE     OFFLINE on node04  

ora.DB.S01.cs                                ONLINE     ONLINE on node03  

So, I had some idea... I checked imon_DB.log file at $ORACLE_HOME/log/hostname/racg/* PATH on node, I used "srvctl start service -d DB -s S01 -i DB4"

2009-03-06 15:33:34.912: [    RACG][1273112928] [28413][1273112928][ora.DB.S01.DB4.srv]: clsrcsnstartsrv: service name S01 already exists

2009-03-06 15:33:34.913: [    RACG][1273112928] [28413][1273112928][ora.DB.S01.DB4.srv]: CLSR-0002: Oracle error encountered while executing clsrcsnstartsrv : execute2

2009-03-06 15:33:34.913: [    RACG][1273112928] [28413][1273112928][ora.DB.S01.DB4.srv]: ORA-44305: service S01 is running

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86

ORA-06512: at "SYS.DBMS_SERVICE", line 444

ORA-06512: at "SYS.DBMS_SERVICE", line 365

ORA-44305: service S01 is running 

What's going on?

So, I connected database on DB4 instance (node4)

$ env | grep ORACLE_SID

ORACLE_SID=DB4

$ sqlplus / as sysdba

SQL>  select instance_name from v$instance; 

INSTANCE_NAME

----------------

DB4

SQL> show parameter service_name 

NAME                                 TYPE        VALUE

------------------------- ----------- --------

service_names                        string      S01, DB

Oh, This Instance  has S01 service online. 

My idea used DBMS_SERVICE.STOP_SERVICE to stop service (S01) on this instance.

SQL> exec DBMS_SERVICE.STOP_SERVICE('S01','DB4');

PL/SQL procedure successfully completed.

SQL> show parameter  service_name  

NAME                                 TYPE        VALUE

------------------------- ----------- --------

service_names                        string      DB

And then use "srvctl" command to start again...

$ srvctl start service -d DB -s S01 -i DB4

Wow, It worked... no error

And then check...

Use "crs_stat"

ora.DB.S01.DB3.srv                         ONLINE     ONLINE on node03  

ora.DB.S01.DB4.srv                         ONLINE     ONLINE on node04  

ora.DB.S01.cs                                ONLINE     ONLINE on node03  

And...

$ sqlplus / as sysdba

SQL>  select instance_name from v$instance; 

INSTANCE_NAME

----------------

DB4

SQL> show parameter service_name 

NAME                                 TYPE        VALUE

------------------------- ----------- --------

service_names                        string      S01, DB

... It's OK, thanks OCP & OCE help me this idea!

Enjoy!

Change VIP Hostname on Oracle RAC

This note is not intended as a means to change the Public or Private Hostnames themselves.  Public or Private hostnames can only be changed by removing/adding nodes, or reinstalling Oracle Clusterware.   VIP Hostnames can be changed, however.

- Check original  configuration before:

srvctl config nodeapps -n node1 -a

Assume, change from "test01-vip" to "test01-virtual"

Example:

# srvctl  config nodeapps -n test01 -a

VIP exists.: /test01-vip/192.168.254.88/255.255.255.0/eth0

-Stop Resources instance, asm and nodeapps

srvctl stop instance -d db -i db1

srvctl stop asm -n test01

srvctl stop nodeapps -n test01

- After that change DNS or hosts file. (192.168.254.88          test01-virtual) on this node and every nodes (RAC)

- Modify VIP hostname on nodeapps by using srvctl command-line (root user)

srvctl modify nodeapps -n [-o ] [-A ] 

Options Description: 

-n Node name. 

-o Oracle home for the cluster software (CRS-Home). 

-A The node level VIP address (/netmask[/if1[|if2|...]]). 

Example:

$ su - 

# srvctl  modify nodeapps -n test01 -A test01-virtual/255.255.255.0/eth0

- Start Resources  nodeapps, asm and instance

srvct start nodeapps -n test01

srvctl start asm -n test01

srvctl start instance -d db -i db1

- Don't forget to change VIP hostname on applications and client.

- Check... Check and Check..

Start Database by sqlplus, can not by srvctl on Solaris 10 (Oracle RAC 10G)

After I changed project on Solaris 10.

projmod -s -K "project.max-shm-memory=(privileged,13832385536,deny)" oracle

something wrong, i can start instance by sqlplus, but can not by srvctl

srvctl  start instance -d db  -i  db2

PRKP-1001 : Error starting instance db2 on node node02

CRS-0215: Could not start resource 'ora.db.db2.inst'.

When checked log files on ORACLE_HOME/log/{hostname}/racg/imon*log

SQL> ORA-27102: out of memory

SVR4 Error: 22: Invalid argument

SQL> Disconnected

So, found out some informations from matalink.

Cause

Problem is not the oracle project.

This occurs because when we start an instance using srvctl it will inherit the system project, since the crsd.bin is running as root, our racg infrastructure is starting the resource as oracle however racgwrap is initially started as root. It calls  setuid/setgid to change the user & group IDs but unfortunately when we do this we dont call the project ID.

Solution

Easiest fix for this problem is to define a project for system.

% projmod -s -K "project.max-shm-memory=(privileged,2.00TB,deny)" system

% projmod -s -K "process.max-sem-nsems=(privileged,2048,deny)" system

Above assigns root user the system project in order for the CRSD daemon (run under root's user id) to pick up the changes:

% vi /etc/user_attr, add root to project=system:

Finally restart CRS stack

And then I modified project (for system project)

projmod -s -K "project.max-shm-memory=(privileged,13832385536,deny)" oracle

projmod -s -K "project.max-shm-memory=(privileged,15132385536,deny)" system

that helped me solve this case.

srvctl  start instance -d db  -i  db2

.

.

.

Change hostnames oracle RAC

I would like to change hostnames in oracle RAC.

on Metalink (NOTE:220970.1)

Can I change the public hostname in my Oracle Database 10g Cluster using Oracle Clusterware?

Hostname changes are not supported in Oracle Clusterware (CRS), unless you want to perform a deletenode followed by a new addnode operation.
The hostname is used to store among other things the flag files and Oracle Clusterware stack will not start if hostname is changed.

Anyway I found out on internet, I found idea on this site

http://www.pythian.com/blogs/482/changing-hostnames-in-oracle-rac

I like it; From step by step that make me understood and created my step.

this's my step example:

- Capture Resource.

Example script:

mkdir /tmp/resources && for x in `$ORA_CRS_HOME/bin/crs_stat -p  | grep "^NAME=" |awk -F\= '{print $2}'` ; do

    $ORA_CRS_HOME/bin/crs_stat -p $x >/tmp/resources/$x.cap

done

- Stop Cluster every nodes.

$ORA_CRS_HOME/bin/crsctl stop crs

- Change hostnames / (/etc/hosts files) / DNS for every nodes 

- Modified $ORA_CRS_HOME/install/rootconfig (on every nodes)

On variables:

CRS_HOST_NAME_LIST

CRS_NODE_NAME_LIST

CRS_PRIVATE_NAME_LIST

CRS_NODELIST

CRS_NODEVIPS

from "old_hosts" to "ew_hosts"

If use 11G 

should to modified on paramfile.crs file as well.

- Cleanup OCR and Voting Disks

on $ORA_CRS_HOME/install/rootconfig and $ORA_CRS_HOME/install/paramfile.crs (11G) files:

CRS_OCR_LOCATIONS={file},{file}

CRS_VOTING_DISKS={file},{file},{file}

dd if=/dev/zero of={file} bs=1024k

- Run rootconfig script (on every nodes)

$ORA_CRS_HOME/install/rootconfig 

(If Error):

No value set for the CRS parameter CRS_OCR_LOCATIONS. Using Values in paramfile.crs

Checking to see if Oracle CRS stack is already configured

Oracle CRS stack is already configured and will be running under init(1M)

rootconfig script was protected some...

So, remove /etc/oracle/ocr.loc

and run again!

.

.

.

Expecting the CRS daemons to be up within 600 seconds.

Cluster Synchronization Services is active on these nodes. 

.

.

.

Cluster Synchronization Services is active on all the nodes. 

Oracle CRS stack installed and running under init(1M)

Running vipca(silent) for configuring nodeapps

Creating VIP application resource on (1) nodes..

Creating GSD application resource on (1) nodes..

Creating ONS application resource on (1) nodes..

Starting VIP application resource on (1) nodes..

Starting GSD application resource on (1) nodes..

Starting ONS application resource on (1) nodes..

- Verify status.

$ORA_CRS_HOME/bin/crs_stat -t

- Register Listener + ASM + Database + Service to CRS.

use netca for listener or srvctl command-line or crs_register.

.

.

.

And final check that make sure every things working.

CRS-1028: Dependency analysis failed because of: CRS-0223: Resource...

After I changed hostname on my RAC

and then run $ORA_CRS_HOME/install/rootconfig command.

I reconfigured listener, But can not start by RAC (srvctl command-line).

I tried to stop/start, remove, recreate or use netca to recreate.
But can not start by srvctl command-line.

$ srvctl start listener -n test01

CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.test01.LISTENER_TEST01.lsnr' has placement error.


So, I cleaned up OCR and Voting Disks ,then run "$ORA_CRS_HOME/install/rootconfig" and then run netca again to solve it.

[CLUSTER VERIFICATION] SRVM_TRACE environmental variable

When you use "srvctl" to start, stop or get status of database, asm or ... blah blah, How do you turn on tracing?

By setting the environmental variable SRVM_TRACE to TRUE.

Example:

$ srvctl status asm -n db02
ASM instance +ASM2 is running on node db02.

$ export SRVM_TRACE=TRUE

$ srvctl status asm -n db02

/oracle/.../crs/jdk/jre/bin/java -classpath /oracle/.../crs/jlib/netcfg.jar:/oracle/.../crs/jdk/jre/lib/rt.jar:/oracle/.../crs/jdk/jre/lib/i18n.jar:/oracle/.../crs/jlib/srvm.jar:/oracle/.../crs/jlib/srvmhas.jar:/oracle/.../crs/jlib/srvmasm.jar:/oracle/.../crs/srvm/jlib/srvctl.jar -DTRACING.ENABLED=true -DTRACING.LEVEL=2 oracle.ops.opsctl.OPSCTLDriver status asm -n db02

[main] [15:16:8:321] [OPSCTLDriver.setInternalDebugLevel:165] tracing is true at level 2 to file null
[main] [15:16:8:362] [OPSCTLDriver.:95] Security manager is set
[main] [15:16:8:382] [CommandLineParser.parse:173] parsing cmdline args
[main] [15:16:8:383] [CommandLineParser.parse2WordCommandOptions:940] parsing 2-word cmdline
[main] [15:16:8:410] [GetActiveNodes.create:213] Going into GetActiveNodes constructor...
[main] [15:16:8:417] [HASContext.getInstance:190] Module init : 16
[main] [15:16:8:418] [HASContext.getInstance:214] Local Module init : 19
[main] [15:16:8:446] [Library.getInstance:95] Created instance of Library.
[main] [15:16:8:447] [Library.load:195] Loading libsrvmhas10.so...
[main] [15:16:8:447] [Library.load:201] oracleHome null
[main] [15:16:8:448] [Library.load:236] Loading library srvmhas10 from path=
/oracle/.../crs/jdk/jre/lib/i386/client:/oracle/.../crs/jdk/jre/lib/i386:/oracle/.../crs/jdk/jre/../lib/i386:/oracle/.../crs/lib32:/oracle/.../crs/srvm/lib32:/oracle/.../crs/lib:/oracle/.../crs/srvm/lib:/oracle/.../db/lib
[main] [15:16:8:486] [Library.load:244] Loaded library srvmhas10 from path=
/oracle/.../crs/jdk/jre/lib/i386/client:/oracle/.../crs/jdk/jre/lib/i386:/oracle/.../crs/jdk/jre/../lib/i386:/oracle/.../crs/lib32:/oracle/.../crs/srvm/lib32:/oracle/.../crs/lib:/oracle/.../crs/srvm/lib:/oracle/.../db/lib
[main] [15:16:8:488] [has.HASContextNative.Native] prsr_trace: no lsf ctx, line=Native: allocHASContext
[main] [15:16:8:488] [has.HASContextNative.Native] allocHASContext: Came in
[main] [15:16:8:488] [has.HASContextNative.Native] allocHASContext: META context [1]
[main] [15:16:8:489] [has.HASContextNative.Native] allocHASContext: LSF context [1]
[main] [15:16:8:490] [has.HASContextNative.Native] prsr_trace: Native: prsr_initCLSS
[main] [15:16:8:508] [has.HASContextNative.Native] allocHASContext: CLSS context [1]
[main] [15:16:8:509] [has.HASContextNative.Native] allocHASContext: retval [1]
[main] [15:16:8:509] [HASContext.getInstance:249] HAS Context Allocated: 1 to oracle.ops.mgmt.has.ClusterUtil@8a0d5d
[main] [15:16:8:509] [ClusterUtil.:55] ClusterUtil Instance created
[main] [15:16:8:510] [ClusterUtil.:58] ClusterUtil m_bSKGXN=true
.
.
.
[main] [15:16:8:637] [HAOperation.checkCommandVerb:211] Checking an existence of CRS command
[main] [15:16:8:637] [LocalCommand.execute:54] LocalCommand.execute: Calling run method
[main] [15:16:8:637] [HAStatusOperation.run:124] Executing the HA operation /oracle/.../crs/bin/crs_statora.db02.ASM2.asm
[main] [15:16:8:643] [HAOperationImpl.runCommand:1254] CRS cmd is: /oracle/.../crs/bin/crs_stat -u ora.db02.ASM2.asm
[main] [15:16:8:645] [RuntimeExec.runCommand:74] Calling Runtime.exec() with the command
[main] [15:16:8:645] [RuntimeExec.runCommand:76] /oracle/.../crs/bin/crs_stat
[main] [15:16:8:645] [RuntimeExec.runCommand:76] -u
[main] [15:16:8:645] [RuntimeExec.runCommand:76] ora.db02.ASM2.asm
[Thread-1] [15:16:8:650] [StreamReader.run:61] In StreamReader.run
[Thread-0] [15:16:8:651] [StreamReader.run:61] In StreamReader.run
[main] [15:16:8:651] [RuntimeExec.runCommand:131] runCommand: Waiting for the process
[Thread-0] [15:16:8:677] [StreamReader.run:65] OUTPUT>NAME=ora.db02.ASM2.asm
[Thread-0] [15:16:8:677] [StreamReader.run:65] OUTPUT>TYPE=application
[Thread-0] [15:16:8:678] [StreamReader.run:65] OUTPUT>TARGET=ONLINE
[Thread-0] [15:16:8:678] [StreamReader.run:65] OUTPUT>STATE=ONLINE on db02
[Thread-0] [15:16:8:678] [StreamReader.run:65] OUTPUT>
[main] [15:16:8:678] [RuntimeExec.runCommand:133] runCommand: process returns 0
[main] [15:16:8:679] [RuntimeExec.runCommand:147] RunTimeExec: output>
[main] [15:16:8:679] [RuntimeExec.runCommand:150] NAME=ora.db02.ASM2.asm
[main] [15:16:8:679] [RuntimeExec.runCommand:150] TYPE=application
[main] [15:16:8:679] [RuntimeExec.runCommand:150] TARGET=ONLINE
[main] [15:16:8:679] [RuntimeExec.runCommand:150] STATE=ONLINE on db02
[main] [15:16:8:679] [RuntimeExec.runCommand:150]
[main] [15:16:8:680] [RuntimeExec.runCommand:155] RunTimeExec: error>
[main] [15:16:8:680] [RuntimeExec.runCommand:175] Returning from RunTimeExec.runCommand
[main] [15:16:8:680] [HAOperationImpl.runCommand:1283] set status OK
[main] [15:16:8:681] [HAStatusOperation.run:127] Returned from executing the HA Operation
[main] [15:16:8:681] [HAStatusOperation.run:132] OUTPUT> NAME=ora.db02.ASM2.asm
[main] [15:16:8:681] [HAStatusOperation.run:132] OUTPUT> TYPE=application
[main] [15:16:8:681] [HAStatusOperation.run:132] OUTPUT> TARGET=ONLINE
[main] [15:16:8:681] [HAStatusOperation.run:132] OUTPUT> STATE=ONLINE on db02
[main] [15:16:8:682] [HAStatusOperation.run:132] OUTPUT>
[main] [15:16:8:682] [HAStatusOperation.run:138] ERROR>
[main] [15:16:8:682] [LocalCommand.execute:56] LocalCommand.execute: Returned from run method
[main] [15:16:8:684] [HAResourceStatus.parseHAStatusOutput:293] got status: resource=ora.db02.ASM2.asmtarget=ONLINE status=ONLINE on db02 host=db02

ASM instance +ASM2 is running on node db02.

-----------------------------------------------------

This Variable help you get Tracing... , when you find the problem and would like to send (the problem information) to Oracle Support.

[Oracle 10G] Relocate service on RAC

When Some node of Oracle RAC restart. Serices on that node will relocate to other nodes.

We check by crs_stat.

A Resource Target State
----------- ------ -----
ora.H.RADIUS.H1.srv ONLINE ONLINE on h02
ora.H.R.cs ONLINE ONLINE on h01
ora.H.U.H1.srv ONLINE ONLINE on h02
ora.H.U.cs ONLINE ONLINE on h01
ora.H.W.H1.srv ONLINE ONLINE on h02
ora.H.W.cs ONLINE ONLINE on h01
ora.H.H1.inst ONLINE ONLINE on h01
ora.H.H2.inst ONLINE ONLINE on h02
ora.H.db ONLINE ONLINE on h01
ora.h01.ASM1.asm ONLINE ONLINE on h01
ora.h01.LISTENER_H01.lsnr ONLINE ONLINE on h01
ora.h01.gsd ONLINE ONLINE on h01
ora.h01.ons ONLINE ONLINE on h01
ora.h01.vip ONLINE ONLINE on h01
ora.h02.ASM2.asm ONLINE ONLINE on h02
ora.h02.LISTENER_H02.lsnr ONLINE ONLINE on h02
ora.h02.gsd ONLINE ONLINE on h02
ora.h02.ons ONLINE ONLINE on h02
ora.h02.vip ONLINE ONLINE on h02


Services on RAC has stated on h02,that some connections will connect to h02 when pass services.

If we need some connection to pass h01, So we should relocate.

#srvctl relocate service -d H -s U -i H2 -t H1
#srvctl relocate service -d H -s W -i H2 -t H1
#srvctl relocate service -d H -s H -i H2 -t H1


A Resource Target State
----------- ------ -----
ora.H.RADIUS.H1.srv ONLINE ONLINE on h01
ora.H.R.cs ONLINE ONLINE on h01
ora.H.U.H1.srv ONLINE ONLINE on h01
ora.H.U.cs ONLINE ONLINE on h01
ora.H.W.H1.srv ONLINE ONLINE on h01
ora.H.W.cs ONLINE ONLINE on h01
ora.H.H1.inst ONLINE ONLINE on h01
ora.H.H2.inst ONLINE ONLINE on h02
ora.H.db ONLINE ONLINE on h01
ora.h01.ASM1.asm ONLINE ONLINE on h01
ora.h01.LISTENER_H01.lsnr ONLINE ONLINE on h01
ora.h01.gsd ONLINE ONLINE on h01
ora.h01.ons ONLINE ONLINE on h01
ora.h01.vip ONLINE ONLINE on h01
ora.h02.ASM2.asm ONLINE ONLINE on h02
ora.h02.LISTENER_H02.lsnr ONLINE ONLINE on h02
ora.h02.gsd ONLINE ONLINE on h02
ora.h02.ons ONLINE ONLINE on h02
ora.h02.vip ONLINE ONLINE on h02

I want to change IP Address on ORACLE CLUSTER

My Problem ... I must change from 6.9.23.32/255.255.255.240 to 6.9.17.227/255.255.255.240

So, I have to stop all nodes....

1. Stop Oracle Database, ASM , listener and Nodeapps on every nodes

# srvctl stop database -d DB
# srvctl stop ASM -n DB01
# srvctl stop ASM -n DB02
# srvctl stop ASM -n DB03
# srvctl stop ASM -n DB04
# srvctl stop listener -n DB01
# srvctl stop listener -n DB02
# srvctl stop listener -n DB03
# srvctl stop listener -n DB04
# srvctl stop nodeapps -n DB01
# srvctl stop nodeapps -n DB02
# srvctl stop nodeapps -n DB03
# srvctl stop nodeapps -n DB04

2. Change hostname /etc/hosts on every nodes

3. Change IP Address on every nodes

4. Login to root user and export ORACLE_HOME

5. Change IP Network on Cluster and VIP

#/oracle/product/10.2.0/crs/bin/oifcfg getif
bond0 6.9.23.32 global public
bond1 192.168.2.0 global cluster_interconnect

#oifcfg delif -global bond0
#oifcfg setif -global bond0/6.9.17.224:Public

- Change VIPs on nodes (example 4 nodes)

#srvctl modify nodeapps -n DB01 -A 6.9.17.227/255.255.255.240/bond0
#srvctl modify nodeapps -n DB02 -A 6.9.17.229/255.255.255.240/bond0
#srvctl modify nodeapps -n DB03 -A 6.9.17.231/255.255.255.240/bond0
#srvctl modify nodeapps -n DB04 -A 6.9.17.233/255.255.255.240/bond0

remark:
oifcfg setif {-node [nodename] | -global} { [if_name]/[subnet]:[if_type]}...

Be careful "subnet" not mean NETMASK ... It mean Network
Example:
ip address: 192.168.1.1 netmask 255.255.255.0
So, network = 192.168.1.0
#oifcfg setif -global bond1/192.168.1.0:cluster_interconnect