How to upgrade ASM after Install Grid Infrastructure Software Only?

I posted "struggle to Upgrade Oracle Database 11.2.0.1 to 11.2.0.2" topic, and someone comment me "Upgrade Grid Infrastructure to ASM after Installed Grid Infrastructure Software Only"
but I have ASM on 10.2 and need to test:

SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
+ASM

SQL> select name, state from v$asm_diskgroup;
NAME STATE
------------------------------ -----------
ASM_DG01 MOUNTED

SQL> select * from v$version;
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0
PL/SQL Release 10.2.0.5.0 - Production
CORE 10.2.0.5.0 Production
TNS for Linux: Version 10.2.0.5.0 - Production
NLSRTL Version 10.2.0.5.0 - Production

then installed Grid Infrastructure(runInstaller) and chose "Installed Grid Infrastructure Software Only"!!! after that - need to run "root.sh" script.

$ su - root
# /u01/app/oracle/product/11.2.0/grid/root.sh
Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/oracle/product/11.2.0/grid/perl/bin/perl -I/u01/app/oracle/product/11.2.0/grid/perl/lib -I/u01/app/oracle/product/11.2.0/grid/crs/install /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl

To configure Grid Infrastructure for a Cluster execute the following command:
/u01/app/oracle/product/11.2.0/grid/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.

My some answer for question "Upgrade Grid Infrastructure to ASM after Installed Grid Infrastructure Software Only", that show above!!!
My Test is "Grid Infrastructure for a Stand-Alone Server", then

# /u01/app/oracle/product/11.2.0/grid/perl/bin/perl -I/u01/app/oracle/product/11.2.0/grid/perl/lib -I/u01/app/oracle/product/11.2.0/grid/crs/install /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl

Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
CSS appears healthy
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
CRS-4664: Node mytest successfully pinned.
Adding daemon to inittab
ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.32-100.0.19.el5'
mytest 2011/01/12 00:49:30 /u01/app/oracle/product/11.2.0/grid/cdata/mytest/backup_20110112_004930.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server

then check and upgrade ASM (reference)

$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/grid
$ export PATH=$ORACLE_HOME/bin:$PATH
$ crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.2.0]

$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [11.2.0.2.0]

$ export ASMCA_ROLLING_UPGRADE=true
$ asmca

but found error:
So, start LISTENER and then "asmca" again but found
try "asmca" again
ASM upgraded, then check from "sqlplus":

SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
+ASM

SQL> select name, state from v$asm_diskgroup;
NAME STATE
------------------------------ -----------
ASM_DG01 MOUNTED

SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0
PL/SQL Release 11.2.0.2.0 - Production
CORE 11.2.0.2.0 Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

However, ASM Disk Groups compatibility not change:

NAME COMPATIBILITY DATABASE_COMPATIBILITY
------------------------------ ------------------------------ ------------------------------
ASM_DG01 10.1.0.0.0 10.1.0.0.0

OK... this only shows answer, what we should do after "Installed Grid Infrastructure Software Only"

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/oracle/product/11.2.0/grid/perl/bin/perl -I/u01/app/oracle/product/11.2.0/grid/perl/lib -I/u01/app/oracle/product/11.2.0/grid/crs/install /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl

To configure Grid Infrastructure for a Cluster execute the following command:
/u01/app/oracle/product/11.2.0/grid/crs/config/config.sh

!!!

struggle to Upgrade Oracle Database 11.2.0.1 to 11.2.0.2

After I wrote about Oracle Database 11gR2 (11.2.0.2) is available on Linux x86-64, x86. It's time to test for upgrading. On my test system, I install Oracle Grid Infrastructure for Standalone Server (ASM) and Database 11.2.0.1, then test to upgrade 11.2.0.1 => 11.2.0.2:

SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE 11.2.0.1.0 Production
TNS for Linux: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production

I use ASM instance, So I have to upgrade Oracle Grid Infrastructure before, check and stop HAS

# /oracle/product/11.2.0/grid/bin/crsctl stat res
NAME=ora.DATA.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.orcl.db
TYPE=ora.database.type
TARGET=ONLINE
STATE=ONLINE on oel

# /oracle/product/11.2.0/grid/bin/crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel'
.
.
.
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel' has completed
CRS-4133: Oracle High Availability Services has been stopped.

Grid infrastructure Upgrading uses p10098816_112020_Linux-x86-64_3of7.zip file and database use p10098816_112020_Linux-x86-64_1of7.zip, p10098816_112020_Linux-x86-64_2of7.zip files. On step, use "runInstaller" to install (choose Upgrade)
Find warning!!! I can not install to replace old Grid Home.
I have to use new Home.
then press Install, find step to run "rootupgrade.sh" script.

# /oracle/product/11.2.0/grid_new/rootupgrade.sh
Running Oracle 11g root script...
The following environment variables are set as:

ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/11.2.0/grid_new

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/product/11.2.0/grid_new/crs/install/crsconfig_params
Creating trace directory

Error while detecting Oracle Grid Infrastructure. ASMCA needs Oracle Grid Infrastructure to configure ASM.

ASM upgrade failed, see logs for details
Failed to upgrade ASM for Oracle Restart configuration at /oracle/product/11.2.0/grid_new/crs/install/crsconfig_lib.pm line 12653.
/oracle/product/11.2.0/grid_new/perl/bin/perl -I/oracle/product/11.2.0/grid_new/perl/lib -I/oracle/product/11.2.0/grid_new/crs/install /oracle/product/11.2.0/grid_new/crs/install/roothas.pl execution failed

So, run "asmca" but find error.

One idea, copy orapw+ASM file from old grid home :

# cp -a /oracle/product/11.2.0/grid/dbs/orapw+ASM /oracle/product/11.2.0/grid_new/dbs/

then run "rootupgrade.sh" again.

# /oracle/product/11.2.0/grid_new/rootupgrade.sh
Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/11.2.0/grid_new

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/product/11.2.0/grid_new/crs/install/crsconfig_params
Creating trace directory

Error while detecting Oracle Grid Infrastructure. ASMCA needs Oracle Grid Infrastructure to configure ASM.

ASM upgrade failed, see logs for details
Failed to upgrade ASM for Oracle Restart configuration at /oracle/product/11.2.0/grid_new/crs/install/crsconfig_lib.pm line 12653.
/oracle/product/11.2.0/grid_new/perl/bin/perl -I/oracle/product/11.2.0/grid_new/perl/lib -I/oracle/product/11.2.0/grid_new/crs/install /oracle/product/11.2.0/grid_new/crs/install/roothas.pl execution failed

but can not solve, try to start ASM instance to check:

$ export ORACLE_HOME=/oracle/product/11.2.0/grid_new
$ export ORACLE_SID=+ASM
$ /oracle/product/11.2.0/grid_new/bin/sqlplus / as sysasm
SQL*Plus: Release 11.2.0.2.0 Production on Thu Sep 16 03:04:04 2010
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters ORA-29701: unable to connect to Cluster Synchronization Service

From error, I have to start CSS before, then idea "run rootupgrade.sh again"-> fixed by modified /etc/oracle/olr.loc and run "rootupgrade.sh" again:

# vi /etc/oracle/olr.loc

Old:
crs_home=/oracle/product/11.2.0/grid
New:
crs_home=/oracle/product/11.2.0/grid_new

# /oracle/product/11.2.0/grid_new/rootupgrade.sh
Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/11.2.0/grid_new

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/product/11.2.0/grid_new/crs/install/crsconfig_params
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node oel successfully pinned.
Adding daemon to inittab
PRKO-3226 : Upgrade from version 11.2.0.1.0 to version 11.2.0.1.0 using srvctl upgrade model command of version 11.2.0.2.0 is not supported
srvctl upgrade model -first ... failed
PRKO-3226 : Upgrade from version 11.2.0.1.0 to version 11.2.0.1.0 using srvctl upgrade model command of version 11.2.0.2.0 is not supported
srvctl upgrade model -last ... failed
ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.35.4'
oel 2010/09/16 03:17:37 /oracle/product/11.2.0/grid_new/cdata/oel/backup_20100916_031737.olr
oel 2010/09/16 01:00:46 /oracle/product/11.2.0/grid/cdata/oel/backup_20100916_010046.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server

Everything look ok... and press install to continue...

after upgraded Grid Infrastructure, check

# /oracle/product/11.2.0/grid_new/bin/crsctl stat res
NAME=ora.DATA.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.evmd
TYPE=ora.evm.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.ons
TYPE=ora.ons.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.orcl.db
TYPE=ora.database.type
TARGET=ONLINE
STATE=ONLINE on oel

Note: error from rootupgrade.sh, I fixed by 1. copy password ASM file from old home to new home(for start ASM instance ) 2. modified /etc/oracle/olr.loc file (crs_home) after I used "ps -aef" and other idea to check.

Then install Oracle Database 11.2.0.2 by "runInstaller" (use Upgrade)
and found warning(oracle recommends to install out-of-place)
install new home
and then upgrade database.

After upgraded, check

SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE 11.2.0.2.0 Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

but ohasd, init.ohasd files still wrong about CRS_HOME.

# cd /etc/init.d/
# cp ohasd ohasd.org
# cp init.ohasd init.ohasd.org

Change /oracle/product/11.2.0/grid to /oracle/product/11.2.0/grid_new in ohasd, init.ohasd files.

We may find error from "rootupgrade.sh", we should solve it before (try to use "rootupgrade.sh" with no error). If find "The current version of Oracle Grid Infrastructure is 11.2.0.1.0... " while upgrade database(dbua), make sure step rootupgrade.sh done.

ASM Dynamic Volume Manager (DVM) & ACFS

The ASM Dynamic Volume Manager (DVM) is a kernel-loadable device driver that provides a standard device driver interface to clients (for example, ACFS).

after read about DVM & ACFS on 11gR2 Feature, that made my curious. so create testing... ACFS:

start to create Files using "dd":

$ dd if=/dev/zero of=/oracle/asmdisks/_file_disk5 bs=1k count=10000000

10000000+0 records in

10000000+0 records out

10240000000 bytes (10 GB) copied, 303.029 seconds, 33.8 MB/s

Associate Loop Device with the New File:

# losetup /dev/loop5 /oracle/asmdisks/_file_disk5

Create Disk on ASMLib:

# oracleasm createdisk VOL5 /dev/loop5

Create ASM Disk Group with ASM Compatibility "11.2" by "asmca":

$ asmca

create asm Disk Group "DISK03" and then create Volume "asm_vol1" from "DISK03" Disk Group:

and then create ASM Cluster File System(ACFS):

create directory for mount point ,check... and mount:

$ mkdir -p /oracle/product/acfsmounts/disk03_asm_vol1

# ls /dev/asm/asm_vol1*

/dev/asm/asm_vol1-15

# mount -t acfs /dev/asm/asm_vol1-15 /oracle/product/acfsmounts/disk03_asm_vol1

# df

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/asm/asm_vol1-15 5242880 47512 5195368 1% /oracle/product/acfsmounts/disk03_asm_vol1

# cd /oracle/product/acfsmounts/disk03_asm_vol1

# touch file.txt

ASMCMD> volstat

DISKGROUP NUMBER / NAME: 3 / DISK03

---------------------------------------

VOLUME_NAME

READS BYTES_READ READ_TIME READ_ERRS

WRITES BYTES_WRITTEN WRITE_TIME WRITE_ERRS

-------------------------------------------------------------

ASM_VOL1

274 271360 1 0

2936 11739136 263 0

ASMCMD> volinfo -G DISK03 ASM_VOL1

Diskgroup Name: DISK03

Volume Name: ASM_VOL1

Volume Device: /dev/asm/asm_vol1-15

State: ENABLED

Size (MB): 5120

Resize Unit (MB): 256

Redundancy: UNPROT

Stripe Columns: 4

Stripe Width (K): 128

Usage: ACFS

Mountpath: /oracle/product/acfsmounts/disk03_asm_vol1