Add Disk with ORA-15054 After ORA-15075

Tested add new disk to ASM Disk Group on 10gR2 RAC 4 nodes, I assumed I have only one node see ASM new Disk (DATA_NEW).

SQL> ALTER DISKGROUP DATA ADD DISK 'ORCL:DATA_NEW';
ALTER DISKGROUP DATA ADD DISK 'ORCL:DATA_NEW'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15075: disk(s) are not visible cluster-wide

In alert log file:

Tue Jun 29 12:47:24 2010
SQL> ALTER DISKGROUP DATA ADD DISK 'ORCL:DATA_NEW'
Tue Jun 29 12:47:24 2010
NOTE: reconfiguration of group 1/0x933863a4 (DATA), full=1
Tue Jun 29 12:47:24 2010
NOTE: initializing header on grp 1 disk DATA_NEW
NOTE: cache opening disk 1 of grp 1: DATA_NEW label:DATA_NEW
NOTE: PST update: grp = 1
NOTE: requesting all-instance disk validation for group=1
Tue Jun 29 12:47:24 2010
NOTE: disk validation pending for group 1/0x933863a4 (DATA)
SUCCESS: validated disks for 1/0x933863a4 (DATA)
Tue Jun 29 12:47:26 2010
NOTE: requesting all-instance PST refresh for group=1
Tue Jun 29 12:47:26 2010
NOTE: PST refresh pending for group 1/0x933863a4 (DATA)
SUCCESS: refreshed PST for 1/0x933863a4 (DATA)
Tue Jun 29 12:47:29 2010
WARNING: offlining disk 1.3915944791 (DATA_NEW) with mask 0x3
NOTE: PST update: grp = 1, dsk = 1, mode = 0x6
Tue Jun 29 12:47:29 2010
NOTE: PST enabling heartbeating (grp 1)
Tue Jun 29 12:47:29 2010
NOTE: PST update: grp = 1, dsk = 1, mode = 0x4
NOTE: cache closing disk 1 of grp 1: DATA_NEW
Tue Jun 29 12:47:31 2010
Reconfiguration started (old inc 16, new inc 17)
List of nodes:

That made ASM other nodes failed... Oops!!!
Then I made all nodes see ASM new Disk (DATA_NEW) and added...

SQL> ALTER DISKGROUP DATA ADD DISK 'ORCL:DATA_NEW';
ALTER DISKGROUP DATA ADD DISK 'ORCL:DATA_NEW'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15033: disk 'ORCL:DATA_NEW' belongs to diskgroup "DATA"

What's wrong?

SQL> ALTER DISKGROUP DATA drop disk DATA_NEW;
ALTER DISKGROUP DATA drop disk DATA_NEW
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15054: disk "DATA_NEW" does not exist in diskgroup "DATA"

Read on How to Confuse ASM, idea to "dd" disk, Great Idea ^ ^
OK ... about "DATA_NEW" (asmlib) = /dev/emcpowerc1

# dd if=/dev/zero of=/dev/emcpowerc1 bs=4k count=20000

then added Disk again:

SQL> ALTER DISKGROUP DATA ADD DISK 'ORCL:DATA_NEW';

Diskgroup altered.

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- -----------
1 REBAL RUN 1 1 96 15988 1680 9

SQL> ALTER DISKGROUP DATA REBALANCE POWER 7;

Diskgroup altered.

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- -----------
1 REBAL RUN 7 7 794 15564 2289 6

OK... we should see new Disk from all nodes before add Disk