Changing Hearbeat in Oracle Rac.

1As of 11.2 Grid Infrastructure, the private network configuration is not only stored in OCR but also in the gpnp profile.  If the private network is not available or its definition is incorrect, the CRSD process will not start and any subsequent changes to the OCR will be impossible. Therefore care needs to be taken when making modifications to the configuration of the private network. It is important to perform the changes in the correct order. Please also note that manual modification of gpnp profile is not supported.
Please take a backup of profile.xml on all cluster nodes before proceeding, as grid user:
cd $GRID_HOME/gpnp/<hostname>/profiles/peer/
cd /app/oracle/product/12.x.x/grid/gpnp/mysrvrahr/profiles/peer
cd /app/oracle/product/12.x.x/grid/gpnp/mysrvrbhr/profiles/peer
cp -p profile.xml profile.xml.bk
2Ensure Oracle Clusterware is running on ALL cluster nodes in the cluster and save current status of resource.
/app/oracle/product/12.x.x/grid/bin/crsctl check cluster -all
/app/oracle/product/12.x.x/grid/bin/crsctl status resource -t>/tmp/beforeNewIps.lst
3As grid user ( always curious who might that b, to me it was the Oracle user btw.):
Get the existing information.
Here you will see only One interconnect in place For example:
/app/oracle/product/12.x.x/grid/bin/oifcfg getif
bond1  global  cluster_interconnect
bond0  global  public
Check the interfaces / subnet address can be identified by command for eth specifically:
/app/oracle/product/12.x.x/grid/bin/oifcfg iflist|grep -i eth|sort
or check  interfaces / subnets in general on OS with oifcfg:
/app/oracle/product/12.x.x/grid/bin/oifcfg iflist|sort
4Add the new cluster_interconnect information:
/app/oracle/product/12.x.x/grid/bin/oifcfg setif -global eth2/,asm
/app/oracle/product/12.x.x/grid/bin/oifcfg setif -global eth6/,asm
5Verify the change:
/app/oracle/product/12.x.x/grid/bin/oifcfg getif

With this information checked and in place it is time for setting up new listeners for asm since the original ASM listener during the installation used eth0 and that eth0 will be dropped  – removed from cluster configuration in steps below:

Existing listener ASMNET1LSNR  will become new one ASMNET122LSNR.
srvctl add listener -asmlistener -l ASMNET1221LSNR -subnet (as mentioned this is the eth2 interface that we are going to use).
srvctl add listener -asmlistener -l ASMNET1222LSNR -subnet (as mentioned this is the eth6 interface that we are going to use).

As always seeing is believing : use 
crsctl status resource -t to see details.
Note: The new ASM listener is created as a resource and it is in a status offline offline on all nodes in the cluster at this point and time.

In the next step we will remove the old ASM listener, and use a -f option to prevent errors – messages with regard to dependencies.

srvctl update listener -listener ASMNET1LSNR_ASM -asm -remove -force
I have checked again with crsctl status resource -t to make sure the old resource is gone now.

Removing the old ASM listener
In the Mos note there is a little inconsistency because it claims  that as a next step the old ASM listener should be stopped.  I was able to grep for the listener ( ps -ef|grep -i inherit)  and i saw it on OS level on the machine(S). But I am not able to stop that listener  since the cluster resource is already gone and lsnrctl did not work. Solution: What I noticed that when I skipped this step and stopped and started the cluster which is mandatory in this scenario, the listener was gone on all nodes.
Should have given this command, but that is NOT working: lsnrctl stop ASMNET1LSNR_ASM
Check configuration before restarting GI:

First command: srvctl config listener -asmlistener Name: ASMNET122LSNR_ASM Type: ASM Listener Owner: oracle Subnet: Home: <CRS home> End points: TCP:1527 Listener is enabled. Listener is individually enabled on nodes: Listener is individually disabled on nodes: Second Command: srvctl config asm ASM home: <CRS home> Password file: +VOTE/orapwASM Backup of Password file: ASM listener: LISTENER ASM instance count: ALL Cluster ASM listener: ASMNET122LSNR_ASM
6In GridInfrastructure: Shutdown Oracle Clusterware on all nodes and disable the Oracle Clusterware as root user ( in my example i was allowed to sudo ):
 sudo su – 
./app/oracle/product/12.x.x/grid/bin/crsctl stop crs
./app/oracle/product/12.x.x/grid/bin/crsctl disable crs
7Make the network configuration change at OS level as required, ensure the new interface is available on all nodes after the change. ( check to ping the interfaces on all nodes ).
for x in 10 11;do for xx in 75 76 77 78;do ping -c2 192.168.${x}.${xx}|egrep ‘icmp_seq|transmitted’;done;echo;done
for x in a b c d; do for xx in 1 2;do ping -c2 mysrvr${x}hr-hb$xx|egrep ‘icmp_seq|transmitted’;done;echo;done 
8Restart Oracle Clusterware on all nodes as root user:
sudo su – 
./app/oracle/product/12.x.x/grid/bin/crsctl start crs
/app/oracle/product/12.x.x/grid/bin/crsctl check cluster -all
/app/oracle/product/12.x.x/grid/bin/crsctl status resource -t>/tmp/afterNewIps.lst
sdiff /tmp/afterNewIps.lst /tmp/beforeNewIps.lst
Enable Oracle Clusterware on all nodes as root user:
./app/oracle/product/12.x.x/grid/bin/crsctl enable crs
10Remove the old interface if required:
/app/oracle/product/12.x.x/grid/bin/oifcfg delif -global bond1/
11Verify the remove:
/app/oracle/product/12.x.x/grid/bin/oifcfg getif

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s