| 1 | As of 11.2 Grid Infrastructure, the private network configuration is not only stored in OCR but also in the gpnp profile. If the private network is not available or its definition is incorrect, the CRSD process will not start and any subsequent changes to the OCR will be impossible. Therefore care needs to be taken when making modifications to the configuration of the private network. It is important to perform the changes in the correct order. Please also note that manual modification of gpnp profile is not supported. |
| Please take a backup of profile.xml on all cluster nodes before proceeding, as grid user: | |
| cd $GRID_HOME/gpnp/<hostname>/profiles/peer/ | |
| cd /app/oracle/product/12.x.x/grid/gpnp/mysrvrahr/profiles/peer | |
| cd /app/oracle/product/12.x.x/grid/gpnp/mysrvrbhr/profiles/peer | |
| cp -p profile.xml profile.xml.bk | |
| 2 | Ensure Oracle Clusterware is running on ALL cluster nodes in the cluster and save current status of resource. |
| /app/oracle/product/12.x.x/grid/bin/crsctl check cluster -all | |
| /app/oracle/product/12.x.x/grid/bin/crsctl status resource -t>/tmp/beforeNewIps.lst | |
| 3 | As grid user ( always curious who might that b, to me it was the Oracle user btw.): |
| Get the existing information. Here you will see only One interconnect in place For example: | |
| /app/oracle/product/12.x.x/grid/bin/oifcfg getif | |
| bond1 172.18.112.208 global cluster_interconnect | |
| bond0 195.233.190.64 global public | |
| ## | |
| Check the interfaces / subnet address can be identified by command for eth specifically: | |
| /app/oracle/product/12.x.x/grid/bin/oifcfg iflist|grep -i eth|sort | |
| eth0 172.18.32.0 | |
| eth2 192.168.10.0 | |
| eth6 192.168.11.0 | |
| or check interfaces / subnets in general on OS with oifcfg: | |
| /app/oracle/product/12.x.x/grid/bin/oifcfg iflist|sort | |
| 4 | Add the new cluster_interconnect information: |
| /app/oracle/product/12.x.x/grid/bin/oifcfg setif -global eth2/192.168.10.0:cluster_interconnect,asm | |
| /app/oracle/product/12.x.x/grid/bin/oifcfg setif -global eth6/192.168.11.0:cluster_interconnect,asm | |
| 5 | Verify the change: |
| /app/oracle/product/12.x.x/grid/bin/oifcfg getif | |
With this information checked and in place it is time for setting up new listeners for asm since the original ASM listener during the installation used eth0 and that eth0 will be dropped – removed from cluster configuration in steps below: Existing listener ASMNET1LSNR will become new one ASMNET122LSNR. srvctl add listener -asmlistener -l ASMNET1221LSNR -subnet 192.168.10.0 (as mentioned this is the eth2 interface that we are going to use). srvctl add listener -asmlistener -l ASMNET1222LSNR -subnet 192.168.11.0 (as mentioned this is the eth6 interface that we are going to use). As always seeing is believing : use crsctl status resource -t to see details. Note: The new ASM listener is created as a resource and it is in a status offline offline on all nodes in the cluster at this point and time. In the next step we will remove the old ASM listener, and use a -f option to prevent errors – messages with regard to dependencies. srvctl update listener -listener ASMNET1LSNR_ASM -asm -remove -force I have checked again with crsctl status resource -t to make sure the old resource is gone now. Removing the old ASM listener In the Mos note there is a little inconsistency because it claims that as a next step the old ASM listener should be stopped. I was able to grep for the listener ( ps -ef|grep -i inherit) and i saw it on OS level on the machine(S). But I am not able to stop that listener since the cluster resource is already gone and lsnrctl did not work. Solution: What I noticed that when I skipped this step and stopped and started the cluster which is mandatory in this scenario, the listener was gone on all nodes. Should have given this command, but that is NOT working: lsnrctl stop ASMNET1LSNR_ASM Check configuration before restarting GI: First command: srvctl config listener -asmlistener Name: ASMNET122LSNR_ASM Type: ASM Listener Owner: oracle Subnet: 192.168.10.0 Home: <CRS home> End points: TCP:1527 Listener is enabled. Listener is individually enabled on nodes: Listener is individually disabled on nodes: Second Command: srvctl config asm ASM home: <CRS home> Password file: +VOTE/orapwASM Backup of Password file: ASM listener: LISTENER ASM instance count: ALL Cluster ASM listener: ASMNET122LSNR_ASM | |
| 6 | In GridInfrastructure: Shutdown Oracle Clusterware on all nodes and disable the Oracle Clusterware as root user ( in my example i was allowed to sudo ): |
| sudo su – | |
| ./app/oracle/product/12.x.x/grid/bin/crsctl stop crs | |
| ./app/oracle/product/12.x.x/grid/bin/crsctl disable crs | |
| 7 | Make the network configuration change at OS level as required, ensure the new interface is available on all nodes after the change. ( check to ping the interfaces on all nodes ). |
| for x in 10 11;do for xx in 75 76 77 78;do ping -c2 192.168.${x}.${xx}|egrep ‘icmp_seq|transmitted’;done;echo;done | |
| for x in a b c d; do for xx in 1 2;do ping -c2 mysrvr${x}hr-hb$xx|egrep ‘icmp_seq|transmitted’;done;echo;done | |
| 8 | Restart Oracle Clusterware on all nodes as root user: |
| sudo su – | |
| ./app/oracle/product/12.x.x/grid/bin/crsctl start crs | |
| 9 | Check |
| /app/oracle/product/12.x.x/grid/bin/crsctl check cluster -all | |
| /app/oracle/product/12.x.x/grid/bin/crsctl status resource -t>/tmp/afterNewIps.lst | |
| sdiff /tmp/afterNewIps.lst /tmp/beforeNewIps.lst | |
| Enable Oracle Clusterware on all nodes as root user: | |
| ./app/oracle/product/12.x.x/grid/bin/crsctl enable crs | |
| 10 | Remove the old interface if required: |
| /app/oracle/product/12.x.x/grid/bin/oifcfg delif -global bond1/172.18.112.208:cluster_interconnect | |
| 11 | Verify the remove: |
| /app/oracle/product/12.x.x/grid/bin/oifcfg getif |
Mathijs Bruggink 