struggle to Upgrade Oracle Database 11.2.0.1 to 11.2.0.2

After I wrote about Oracle Database 11gR2 (11.2.0.2) is available on Linux x86-64, x86. It's time to test for upgrading. On my test system, I install Oracle Grid Infrastructure for Standalone Server (ASM) and Database 11.2.0.1, then test to upgrade 11.2.0.1 => 11.2.0.2:

SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE 11.2.0.1.0 Production
TNS for Linux: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production

I use ASM instance, So I have to upgrade Oracle Grid Infrastructure before, check and stop HAS

# /oracle/product/11.2.0/grid/bin/crsctl stat res
NAME=ora.DATA.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.orcl.db
TYPE=ora.database.type
TARGET=ONLINE
STATE=ONLINE on oel

# /oracle/product/11.2.0/grid/bin/crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel'
.
.
.
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel' has completed
CRS-4133: Oracle High Availability Services has been stopped.

Grid infrastructure Upgrading uses p10098816_112020_Linux-x86-64_3of7.zip file and database use p10098816_112020_Linux-x86-64_1of7.zip, p10098816_112020_Linux-x86-64_2of7.zip files. On step, use "runInstaller" to install (choose Upgrade)
Find warning!!! I can not install to replace old Grid Home.
I have to use new Home.
then press Install, find step to run "rootupgrade.sh" script.

# /oracle/product/11.2.0/grid_new/rootupgrade.sh
Running Oracle 11g root script...
The following environment variables are set as:

ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/11.2.0/grid_new

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/product/11.2.0/grid_new/crs/install/crsconfig_params
Creating trace directory

Error while detecting Oracle Grid Infrastructure. ASMCA needs Oracle Grid Infrastructure to configure ASM.

ASM upgrade failed, see logs for details
Failed to upgrade ASM for Oracle Restart configuration at /oracle/product/11.2.0/grid_new/crs/install/crsconfig_lib.pm line 12653.
/oracle/product/11.2.0/grid_new/perl/bin/perl -I/oracle/product/11.2.0/grid_new/perl/lib -I/oracle/product/11.2.0/grid_new/crs/install /oracle/product/11.2.0/grid_new/crs/install/roothas.pl execution failed

So, run "asmca" but find error.

One idea, copy orapw+ASM file from old grid home :

# cp -a /oracle/product/11.2.0/grid/dbs/orapw+ASM /oracle/product/11.2.0/grid_new/dbs/

then run "rootupgrade.sh" again.

# /oracle/product/11.2.0/grid_new/rootupgrade.sh
Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/11.2.0/grid_new

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/product/11.2.0/grid_new/crs/install/crsconfig_params
Creating trace directory

Error while detecting Oracle Grid Infrastructure. ASMCA needs Oracle Grid Infrastructure to configure ASM.

ASM upgrade failed, see logs for details
Failed to upgrade ASM for Oracle Restart configuration at /oracle/product/11.2.0/grid_new/crs/install/crsconfig_lib.pm line 12653.
/oracle/product/11.2.0/grid_new/perl/bin/perl -I/oracle/product/11.2.0/grid_new/perl/lib -I/oracle/product/11.2.0/grid_new/crs/install /oracle/product/11.2.0/grid_new/crs/install/roothas.pl execution failed

but can not solve, try to start ASM instance to check:

$ export ORACLE_HOME=/oracle/product/11.2.0/grid_new
$ export ORACLE_SID=+ASM
$ /oracle/product/11.2.0/grid_new/bin/sqlplus / as sysasm
SQL*Plus: Release 11.2.0.2.0 Production on Thu Sep 16 03:04:04 2010
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters ORA-29701: unable to connect to Cluster Synchronization Service

From error, I have to start CSS before, then idea "run rootupgrade.sh again"-> fixed by modified /etc/oracle/olr.loc and run "rootupgrade.sh" again:

# vi /etc/oracle/olr.loc

Old:
crs_home=/oracle/product/11.2.0/grid
New:
crs_home=/oracle/product/11.2.0/grid_new

# /oracle/product/11.2.0/grid_new/rootupgrade.sh
Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/11.2.0/grid_new

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/product/11.2.0/grid_new/crs/install/crsconfig_params
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node oel successfully pinned.
Adding daemon to inittab
PRKO-3226 : Upgrade from version 11.2.0.1.0 to version 11.2.0.1.0 using srvctl upgrade model command of version 11.2.0.2.0 is not supported
srvctl upgrade model -first ... failed
PRKO-3226 : Upgrade from version 11.2.0.1.0 to version 11.2.0.1.0 using srvctl upgrade model command of version 11.2.0.2.0 is not supported
srvctl upgrade model -last ... failed
ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.35.4'
oel 2010/09/16 03:17:37 /oracle/product/11.2.0/grid_new/cdata/oel/backup_20100916_031737.olr
oel 2010/09/16 01:00:46 /oracle/product/11.2.0/grid/cdata/oel/backup_20100916_010046.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server

Everything look ok... and press install to continue...

after upgraded Grid Infrastructure, check

# /oracle/product/11.2.0/grid_new/bin/crsctl stat res
NAME=ora.DATA.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.evmd
TYPE=ora.evm.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.ons
TYPE=ora.ons.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.orcl.db
TYPE=ora.database.type
TARGET=ONLINE
STATE=ONLINE on oel

Note: error from rootupgrade.sh, I fixed by 1. copy password ASM file from old home to new home(for start ASM instance ) 2. modified /etc/oracle/olr.loc file (crs_home) after I used "ps -aef" and other idea to check.

Then install Oracle Database 11.2.0.2 by "runInstaller" (use Upgrade)
and found warning(oracle recommends to install out-of-place)
install new home
and then upgrade database.

After upgraded, check

SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE 11.2.0.2.0 Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

but ohasd, init.ohasd files still wrong about CRS_HOME.

# cd /etc/init.d/
# cp ohasd ohasd.org
# cp init.ohasd init.ohasd.org

Change /oracle/product/11.2.0/grid to /oracle/product/11.2.0/grid_new in ohasd, init.ohasd files.

We may find error from "rootupgrade.sh", we should solve it before (try to use "rootupgrade.sh" with no error). If find "The current version of Oracle Grid Infrastructure is 11.2.0.1.0... " while upgrade database(dbua), make sure step rootupgrade.sh done.