11gR2 Add VIP to Node

I'm just curious on Forums.

PRKO-2165 : VIP does not exist on node(s) : linux2

I'd like to know Add VIP to a node idea. So, Tested It.

Virtual IP = 192.168.254.92

/etc/hosts file:
192.168.254.92 RHEL5-TEST-VIP

# ./srvctl status nodeapps
Network is enabled
Network is running on node: rhel5-test
GSD is disabled
GSD is not running on node: rhel5-test
ONS is enabled
ONS daemon is running on node: rhel5-test
eONS is enabled
eONS daemon is running on node: rhel5-test
PRKO-2165 : VIP does not exist on node(s) : rhel5-test

# ./srvctl start nodeapps
PRKO-2421 : Network resource is already started on node(s): rhel5-test
PRKO-2439 : VIP does not exist.
PRKO-2422 : ONS is already started on node(s): rhel5-test
PRKO-2423 : eONS is already started on node(s): rhel5-test

$ ./crsstat vip
HA Resource Target State (Host)
----------- ------ -----
ora.scan1.vip ONLINE ONLINE on rhel5-test

# ping 192.168.254.92
PING 192.168.254.92 (192.168.254.92) 56(84) bytes of data.
From 192.168.254.91 icmp_seq=1 Destination Host Unreachable
From 192.168.254.91 icmp_seq=2 Destination Host Unreachable

--- 192.168.254.92 ping statistics ---
2 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3999ms

Try to add VIP on my node.

srvctl add vip -n node_name -A {name|ip}/netmask[/if1[if2|...]]
   [-k network_number] [-v]

# ./srvctl add vip -n rhel5-test -A 192.168.254.92/255.255.255.0/eth0 -k 2
PRCN-2049 : The network attributes specified (network number: 2, subnet: 192.168.254.0, adapters: eth0) conflict with an already registered network (network number: 1, subnet: 192.168.254.0, adapters: eth0)

So, changed network_number:

# ./srvctl add vip -n rhel5-test -A 192.168.254.92/255.255.255.0/eth0 -k 1

Check again & again.

$ ./crsstat vip
HA Resource Target State (Host)
----------- ------ -----
ora.rhel5-test.vip OFFLINE OFFLINE
ora.scan1.vip ONLINE ONLINE on rhel5-test

# ping 192.168.254.92
PING 192.168.254.92 (192.168.254.92) 56(84) bytes of data.
From 192.168.254.91 icmp_seq=1 Destination Host Unreachable
From 192.168.254.91 icmp_seq=2 Destination Host Unreachable

--- 192.168.254.92 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms

$ srvctl start nodeapps
PRKO-2421 : Network resource is already started on node(s): rhel5-test
PRKO-2422 : ONS is already started on node(s): rhel5-test
PRKO-2423 : eONS is already started on node(s): rhel5-test

Or

$ srvctl start vip -n rhel5-test

Check again.

$ ./crsstat vip
HA Resource Target State (Host)
----------- ------ -----
ora.rhel5-test.vip ONLINE ONLINE on rhel5-test
ora.scan1.vip ONLINE ONLINE on rhel5-test

# ping 192.168.254.92
PING 192.168.254.92 (192.168.254.92) 56(84) bytes of data.
64 bytes from 192.168.254.92: icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from 192.168.254.92: icmp_seq=2 ttl=64 time=0.049 ms

--- 192.168.254.92 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms

# ./srvctl config vip -n rhel5-test
VIP exists.:rhel5-test
VIP exists.: /192.168.254.92/192.168.254.92/255.255.255.0/eth0

That's a example to fun -)

By the way, we should add vip by name.
From /etc/hosts file:
192.168.254.92 RHEL5-TEST-VIP

# ./srvctl stop vip -n rhel5-test -f
# ./srvctl remove vip -i rhel5-test
Please confirm that you intend to remove the VIPs rhel5-test (y/[n]) y

Check:

$ ./crsstat vip
HA Resource Target State (Host)
----------- ------ -----
ora.scan1.vip ONLINE ONLINE on rhel5-test

start add VIP:

# ./srvctl add vip -n rhel5-test -A rhel5-test-vip/255.255.255.0/eth0 -k 1
# ./srvctl config vip -n rhel5-test
VIP exists.:rhel5-test
VIP exists.: /rhel5-test-vip/192.168.254.92/255.255.255.0/eth0

Check:

$ ./crsstat vip
HA Resource Target State (Host)
----------- ------ -----
ora.rhel5-test.vip OFFLINE OFFLINE
ora.scan1.vip ONLINE ONLINE on rhel5-test

then start...

# ./srvctl start vip -n rhel5-test
$ ./crsstat vip
HA Resource Target State (Host)
----------- ------ -----
ora.rhel5-test.vip ONLINE ONLINE on rhel5-test
ora.scan1.vip ONLINE ONLINE on rhel5-test

Nice -)

How to Setup 10gRAC with Solaris (IPMP)

On Solaris, IP network multipathing (IPMP) provides physical interface failure detection and transparent network access failover for a system with multiple interfaces on the same IP link.

On 10gRAC, It's a good question, How to Setup 10gRAC with Solaris (IPMP) ?

Setup Cluster Interconnect (metalink: 368464.1):

rac01:

- Physical IP : 10.10.10.1

- Test IP for ce2 : 10.10.10.11

- Test IP for ce3 : 10.10.10.12

rac02:

- Physical IP : 10.10.10.2

- Test IP for ce2 : 10.10.10.21

- Test IP for ce3 : 10.10.10.22

Check & configure...

oifcfg getif | grep -i cluster_interconnect

If already configuration... with "cluster_interconnect", then delete with 'oifcfg delif'

oifcfg delif [-node <nodename> | -global] [ <if_name> [/ <subnet> ]]

$ oifcfg getif | grep -i cluster_interconnect
ce2 10.10.10.0 global cluster_interconnect
ce3 10.10.10.0 global cluster_interconnect

$ oifcfg delif -global ce2

$ oifcfg delif -global ce3

set the CLUSTER_INTERCONNECTS parameter in the spfile/init.ora (RDBMS/ASM) to the Physical IP (IPMP)... and check on GV$CLUSTER_INTERCONNECTS as well.

rac database:

rac1.CLUSTER_INTERCONNECTS=10.10.10.1

rac2.CLUSTER_INTERCONNECTS=10.10.10.2

SQL> ALTER SYSTEM SET CLUSTER_INTERCONNECTS = '10.10.10.1' scope=spfile sid='rac1';

SQL>ALTER SYSTEM SET CLUSTER_INTERCONNECTS = '10.10.10.2' scope=spfile sid='rac2';

on ASM:

+ASM1.CLUSTER_INTERCONNECTS=10.10.10.1

+ASM2.CLUSTER_INTERCONNECTS=10.10.10.2

.

.

virtual IP (metalink:283107.1):

rac01:

- Physical IP : 192.168.10.10

- Test IP for ce0 : 192.168.10.11

- Test IP for ce1 : 192.168.10.12

- Oracle VIP : 192.168.10.1

rac02:

- Physical IP : 192.168.10.20

- Test IP for ce0 : 192.168.10.21

- Test IP for ce1 : 192.168.10.22

- Oracle VIP : 192.168.10.2

=> New 10gRAC installation:

In vipca (VIP Configuration Assistant, 1 of 2), select all NIC's within the same IPMP group where the VIP should run...

=> Existing 10gRAC installation, use "srvctl" to modify:

http://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/srvctladmin.htm#CHDBJBIG

# srvctl config nodeapps -n rac01 -a

VIP exists.: /rac01-vip/192.168.10.1/255.255.255.0/ce0

# srvctl stop nodeapps -n rac01

# srvctl modify nodeapps -n rac01 -A 192.168.10.1/255.255.255.0/ce0\|ce1

# srvctl config nodeapps -n rac01 -a

VIP exists.: /rac01-vip/192.168.10.1/255.255.255.0/ce0:ce1

# srvctl start nodeapps -n rac01

# srvctl config nodeapps -n rac02 -a

VIP exists.: /rac02-vip/192.168.10.2/255.255.255.0/ce0

# srvctl stop nodeapps -n rac02

# srvctl modify nodeapps -n rac02 -A 192.168.10.2/255.255.255.0/ce0\|ce1

# srvctl config nodeapps -n rac02 -a

VIP exists.: /rac02-vip/192.168.10.2/255.255.255.0/ce0:ce1

# srvctl start nodeapps -n rac02

After modifed... use "srvctl config nodeapps -n nodename -a" to check.

Make Sure! RAC VIP configure, right?

when i implemented Oracle RAC and i used Virtual IP for connecting "but network ;) told me... find packets on both Virtual IP and Public IP"

Example:

service1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac01-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac02-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = service1)
)
)

why? I connected by virtual IPs;)

while connection from client; so i tested "tcpdump" on client and rac01/rac02 servers, and i found traffics/packets on Public IP and Virtual IP.

As "tcpdump" told me client connected by Virtual IP... But used Public IP to work.

client -> rac01-vip:1521
.
.
client -> rac01:1521.
.
.

What wrong?
So, checked on oracle rac "lsnrctl services":

$ lsnrctl services
.
.
.
Service "service1" has 2 instance(s).
Instance "db1", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:2 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=rac01)(PORT=1521))
Instance "db2", status READY, has 2 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=rac02)(PORT=1521))
"DEDICATED" established:8316 refused:0 state:ready
LOCAL SERVER

A mistake configuration about rac vip???
after I read "Oracle RAC — VIP Configuration Mistakes: by Alex Gorbache, i changed configuration:

-> listener.ora

rac01:
LISTENER_rac01 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac01-vip)(PORT = 1521)(IP = FIRST)))
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac01)(PORT = 1521)(IP = FIRST)))
)
)

rac02:
LISTENER_rac02 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac02-vip)(PORT = 1521)(IP = FIRST)))
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac02)(PORT = 1521)(IP = FIRST)))
)
)

-> tnsnames.ora on both nodes

LISTENERS_DB =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac01-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac02-vip)(PORT = 1521))
)

LISTENERS_rac01 =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac01-vip)(PORT = 1521))
)

LISTENERS_rac02 =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac02-vip)(PORT = 1521))
)

-> DATABASE

rac01:
SQL> show parameter listener

NAME TYPE VALUE
-----------------------------------------------------------------
local_listener string LISTENERS_rac01
remote_listener string LISTENERS_DB

rac02:
SQL> show parameter listener

NAME TYPE VALUE
-----------------------------------------------------------------
local_listener string LISTENERS_rac02
remote_listener string LISTENERS_DB

(just example)

and then checked:

$ lsnrctl services
.
.
Service "service1" has 2 instance(s).
Instance "db1", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:2 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=rac01-vip)(PORT=1521))
Instance "db2", status READY, has 2 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=rac02-vip)(PORT=1521))
"DEDICATED" established:8316 refused:0 state:ready
LOCAL SERVER

after that, I tested connection from client by Virtual IPs again, and "tcpdump" dumped traffics/packets.

client -> rac01-vip:1521
.
.
client -> rac01-vip:1521.
.
.

I've not found traffics/packets on Public IP anymore.