Introduction:

In the last quarter of 2012 i have set up  a 4 Node Real Application Cluster on Linux which was great to do as always. In day-to-day business I had lost track of this environment  which in itself is not that bad of course. However this week I needed the environment as a preparation for patching so it would be mandatory to check with opatch last patch installed in the grid infra structure and in the rdbms homes that have been set up on those boxes. Well it was to be expected that other dbas would work on those boxes, maybe / likely might have installed new patches so it was really like getting back together with old friends to see what was under the hood.

Well it was a bit of disappointment that the inventory destination  (/opt/oracle/orInventory) was simply empty . Opatch did not like it either cause it  exited with a returncode 73. But then again on the bright side this meant i could take note of the necessary steps  performed  and turn it into a blog.

Work in progress to recreate the Central Repository in a Rac environment:

In the steps below you should be able to recreate your Central repository on the Real Application Cluster:

1-Locate your central inventory.

Well since we are working on  Linux Boxes that meanst  we shall have to check /var/opt/oracle/oraInst.loc  to get the information where Oracle has hmm parked the  Inventory information.

2-Find the Oracle home names & location(s) of Oracle homes by looking in the central inventory’s ContentsXML folder at file. inventory.xml

Well maybe you might be peeking on a different cluster for the setting ( as i wrote  on this box the directory orInventory was empty so that means very little to peek ) .

On Linux you would check to find:

$ cat /var/opt/oracle/oraInst.loc
inventory_loc=/opt/oracle/oraInventory
inst_group=oinstall

## So you would go to the corresponding directory:

$ cd /opt/oracle/oraInventory

##  You would go into the subdirectory ContentsXML

$ cd ContentsXML

## And if present you should check current information on a similar box  by taking a cat of the inventory.xml

$ cat inventory.xml

3-Make a Back up the central inventory by moving it to a different location.

Again this is only useful when you have some inventory present and you would want to save that contents before you start working with  the  recreation .UNIX/Linux example:

mv /opt/oracle/oraInventory /opt/oracle/oraInventory201305  ( added a yyyymm to it).

cd to the OUI bin directory.

cd $ORACLE_HOME/oui/bin

4-Execute runInstaller with attachHome option for OraCrsHome.

Very Important to note is that we will first rebuild the Central inventory for the Grid Infra structure hope. Note. By doing this on the first node  this should trigger the recreate on all  N nodes on your Cluster.

NrunInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=”” ORACLE_HOME_NAME=”” LOCAL_NODE=”” CLUSTER_NODES=”” CRS=true

** For the Grid Infra structure Home it is Mandatory to include CRS=TRUE  ***

5-Execute runInstaller with attachHome option for each  OraDbHome that needs to be added to the central Inventory again.

./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=”” ORACLE_HOME_NAME=”” LOCAL_NODE= CLUSTER_NODES=

6-Check the central inventory location in all the nodes to make sure the inventory was created.

After you have performed the steps 1 – 5 , now is time to check your work. You should be able to do an: opatch lsinventory -oh  on all N nodes and the record should show that that local Node is the one you are running the opatch tool on , and it should list the other nodes as well. If it shows , well then job well done.

Once you are convinced that Central Inventory is showing proper record again for the Grid Infra home, you should check the same for the RDBMS  homes you need to be present in the Central Inventory as well: opatch lsinventory -oh

7-Verify and compare XML files size inside oraInventory/ContentsXML across all the nodes.

Also a good thing to check the xml files after the rebuild both in content and size across all nodes as well.

BE CAREFUL

Do not try attachHome in all the nodes, which may corrupt the inventory. If you follow this procedure on the first Node ( Attaching Home in one node) the cluster will take care that the information is distributed to the other nodes as well.

Always make a backed up the original inventory before you start working. As they say better safe then sorry in these matters.

 Example MYNODE cluster:

## Grid Inventory  recreate

./product/11203_ee_64/db/oui/bin/runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=”/opt/crs/product/112_ee_64/crs” ORACLE_HOME_NAME=”Ora11g_gridinfrahome1″ LOCAL_NODE=”MYNODE1HR” CLUSTER_NODES=”MYNODE1HR,MYNODE2HR,MYNODE3HR,MYNODE4HR” CRS=true

NOTE! it isImportant that for the GI the parameter CRS=TRUE is in place.  BUT  do not use for the RDBMS !!!

## 11.2 Inventory  recreate

./product/11203_ee_64/db/oui/bin/runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=”/opt/oracle/product/11203_ee_64/db” ORACLE_HOME_NAME=”OraDb11g_home1″ LOCAL_NODE=”MYNODE1HR” CLUSTER_NODES=”MYNODE1HR,MYNODE2HR,MYNODE3HR,MYNODE4HR”

## 11.1 Inventory  recreate since a second oracle Home had been installed during  Rac create.

./product/11203_ee_64/db/oui/bin/runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=”/opt/oracle/product/111_ee_64/db” ORACLE_HOME_NAME=”OraDb11g_home2″ LOCAL_NODE=”MYNODE1HR” CLUSTER_NODES=”MYNODE1HR,MYNODE2HR,MYNODE3HR,MYNODE4HR”

Happy reading , and best of Luck

Mathijs