Brief discussion of a case study for mission critical (rac) system(s) in various buildings.
In this case study there are two storage boxes , each one on different location and referred to a mx0530 or mx0531. In assigning the luns in this concept a naming convention as below was thought of as an identifier for them.
If your really want to take it to the next level an additional lun is needed as a quorum disk:
Information used for this case study can be retrieved with a big thank you to Markus for sharing this on the web.:
As of 11.2 Grid Infrastructure, the private network configuration is not only stored in OCR but also in the gpnp profile. If the private network is not available or its definition is incorrect, the CRSD process will not start and any subsequent changes to the OCR will be impossible. Therefore care needs to be taken when making modifications to the configuration of the private network. It is important to perform the changes in the correct order. Please also note that manual modification of gpnp profile is not supported.
Please take a backup of profile.xml on all cluster nodes before proceeding, as grid user:
cd $GRID_HOME/gpnp/<hostname>/profiles/peer/
cd /app/oracle/product/12.x.x/grid/gpnp/mysrvrahr/profiles/peer
cd /app/oracle/product/12.x.x/grid/gpnp/mysrvrbhr/profiles/peer
cp -p profile.xml profile.xml.bk
2
Ensure Oracle Clusterware is running on ALL cluster nodes in the cluster and save current status of resource.
With this information checked and in place it is time for setting up new listeners for asm since the original ASM listener during the installation used eth0 and that eth0 will be dropped – removed from cluster configuration in steps below:
Existing listener ASMNET1LSNR will become new one ASMNET122LSNR. srvctl add listener -asmlistener -l ASMNET1221LSNR -subnet 192.168.10.0 (as mentioned this is the eth2 interface that we are going to use). srvctl add listener -asmlistener -l ASMNET1222LSNR -subnet 192.168.11.0 (as mentioned this is the eth6 interface that we are going to use).
As always seeing is believing : use crsctl status resource -t to see details. Note: The new ASM listener is created as a resource and it is in a status offline offline on all nodes in the cluster at this point and time.
In the next step we will remove the old ASM listener, and use a -f option to prevent errors – messages with regard to dependencies.
srvctl update listener -listener ASMNET1LSNR_ASM -asm -remove -force I have checked again with crsctl status resource -t to make sure the old resource is gone now.
Removing the old ASM listener In the Mos note there is a little inconsistency because it claims that as a next step the old ASM listener should be stopped. I was able to grep for the listener ( ps -ef|grep -i inherit) and i saw it on OS level on the machine(S). But I am not able to stop that listener since the cluster resource is already gone and lsnrctl did not work. Solution: What I noticed that when I skipped this step and stopped and started the cluster which is mandatory in this scenario, the listener was gone on all nodes. Should have given this command, but that is NOT working: lsnrctl stop ASMNET1LSNR_ASM Check configuration before restarting GI:
First command: srvctl config listener -asmlistener Name: ASMNET122LSNR_ASM Type: ASM Listener Owner: oracle Subnet: 192.168.10.0 Home: <CRS home> End points: TCP:1527 Listener is enabled. Listener is individually enabled on nodes: Listener is individually disabled on nodes: Second Command:srvctl config asm ASM home: <CRS home> Password file: +VOTE/orapwASM Backup of Password file: ASM listener: LISTENER ASM instance count: ALL Cluster ASM listener: ASMNET122LSNR_ASM
6
In GridInfrastructure: Shutdown Oracle Clusterware on all nodes and disable the Oracle Clusterware as root user ( in my example i was allowed to sudo ):
Make the network configuration change at OS level as required, ensure the new interface is available on all nodes after the change. ( check to ping the interfaces on all nodes ).
for x in 10 11;do for xx in 75 76 77 78;do ping -c2 192.168.${x}.${xx}|egrep ‘icmp_seq|transmitted’;done;echo;done
for x in a b c d; do for xx in 1 2;do ping -c2 mysrvr${x}hr-hb$xx|egrep ‘icmp_seq|transmitted’;done;echo;done
8
Restart Oracle Clusterware on all nodes as root user:
Autonomous Health Framework (AHF) – Including TFA and ORAchk/EXAChk
AHF-LINUX_v20.1.3.zip
The current installation has to be checked with 2 tools: orachk and cluvfy. Both are included in the 19c software, but it is good practice to download the actual version from MOS (note 1268927.1).
Note: the AHF has 2 requirements: app 5 – 10 GB of storage AND the hierarchy of directory where you will install it needs to be owned by root in full. Recommended to create a FS in such way root:root in /var/SP/ahf and perform install there.
oracle@mysrvr:/app/oracle/stage [MYDB]# ls -al *
-rw-r–r–. 1 oracle dba 165 Jun 24 11:30 status.file
Patch 28553832 was needed on mysrvr since it was not patched since 2018:
total 433452
drwxr-xr-x. 3 oracle dba 4096 Jun 24 11:59 .
drwxr-xr-x. 9 oracle dba 4096 Jun 24 11:58 ..
drwxr-x—. 4 oracle dba 4096 Dec 25 2018 28553832
-rw-r–r–. 1 oracle dba 443838687 Jun 24 11:59 p28553832_12201190115OCWJAN2019RU_Linux-x86-64.zip
Patch 30899722 for april 2020:
total 12
drwxr-xr-x. 3 oracle dba 4096 Jun 19 11:43 .
drwxr-xr-x. 9 oracle dba 4096 Jun 24 11:58 ..
drwxr-x—. 4 oracle dba 4096 Jun 24 11:00 Patch_30899722_GI_RELEASE_UPDATE_19.7.0.0.0__14_Apr_2020
Patch CVU only needed on new cluster:
total 286872
-rwxr-x—. 1 oracle dba 293648959 Jun 24 10:19 cvupack_Linux_x86_64.zip
To install GI:
total 2821488
drwxr-xr-x. 3 oracle dba 4096 Jun 24 10:20 .
drwxr-xr-x. 9 oracle dba 4096 Jun 24 11:58 ..
drwxr-xr-x. 2 oracle dba 4096 Jun 24 10:22 DELETEME
-rwxr-x—. 1 oracle dba 2889184573 Jun 19 11:38 LINUX.X64_193000_grid_home.zip
Latest versions of Opatch:
total 231280
drwxr-xr-x. 2 oracle dba 4096 Jun 24 10:54 .
drwxr-xr-x. 9 oracle dba 4096 Jun 24 11:58 ..
-rwxr-x—. 1 oracle dba 118408624 Jun 24 10:54 p6880880_200000_Linux-x86-64.zip
Database software RDBMS:
total 2988008
drwxr-xr-x. 2 oracle dba 4096 Jun 19 11:40 .
drwxr-xr-x. 9 oracle dba 4096 Jun 24 11:58 ..
-rwxr-x—. 1 oracle dba 3059705302 Jun 19 11:43 LINUX.X64_193000_db_home.zip
Latest TFA:
total 258556
drwxr-xr-x. 2 oracle dba 4096 Jun 19 14:30 .
drwxr-xr-x. 9 oracle dba 4096 Jun 24 11:58 ..
-rw-r–r–. 1 oracle dba 264751391 Jun 19 14:30 TFA-LINUX_v19.2.1.zip
This showed below types of errors, which due to fact this is an existing box with clusterware running was ok ( after checking proceeded with the installation):
Failures were encountered during execution of CVU verification request “stage -pre crsinst”.
Verifying Group Existence: asmadmin …FAILED
mysrvr: PRVG-10461 : Group “asmadmin” selected for privileges “OSASM” does
not exist on node “mysrvr”.
Verifying Group Existence: asmdba …FAILED
mysrvr: PRVG-10461 : Group “asmdba” selected for privileges “OSDBA” does not
exist on node “mysrvr”.
Verifying Group Membership: asmadmin …FAILED
mysrvr: PRVG-10460 : User “oracle” does not belong to group “asmadmin”
selected for privileges “OSASM” on node “mysrvr”.
Verifying Group Membership: asmdba …FAILED
mysrvr: PRVG-10460 : User “oracle” does not belong to group “asmdba” selected
for privileges “OSDBA” on node “mysrvr”.
Verifying Node Connectivity …WARNING
mysrvr: PRVG-11069 : IP address “169.254.0.2” of network interface “idrac” on
oracle@mysrvr:/opt/oracle/diag/asm/+asm/+ASM/trace [+ASM]# crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [19.0.0.0.0]
oracle@mysrvr:/opt/oracle/diag/asm/+asm/+ASM/trace [+ASM]# crsctl query has releasepatch
Oracle Clusterware release patch level is [3633918477] and the complete list of patches [30869156 30869304 30894985 30898856 ] have been applied on the local node. The release patch string is [19.7.0.0.0].
oracle@mysrvr:/opt/oracle/diag/asm/+asm/+ASM/trace [+ASM]# crsctl query has softwarepatch
Oracle Clusterware patch level on node mysrvr is [3633918477].
Mathijs Bruggink 