Relink Oracle software after Upgrading Red Hat release

Introduction:

Soon in a theater near you as a main event, the patching of a lot of Linux (database) servers. So it is once more time to dust off the documentation and refresh memory because in the past there has always been a lot of debate was between dbas should we or should we not re-link the Oracle Software once Linux patching has been done? Motivation would be that folks stated “but we have a running version now, and don’t know what misery recompile will bring”.  Plain and simple and with Oracle Support recommendation the answer to that question should always be YES. If required libraries in Linux have changed it is best to find out during that one maintenance window instead of finding out – weeks or months later when most of us have forgotten about the patch activities.

Below steps are a bit specific for my environments where I have setup Asm and Oracle Restart on various single servers. Yet I hope this note will be of benefit as a general overview

Stopping the databases under control in an easy way:

 

Below you will find some nice feature of the srvctl  status – stop – start command in order to easily capture  – stop and start all the databases  that are part of a specific ORACLE_HOME. ( Commands and examples come from Oracle Documentation (https://docs.oracle.com/html/E25494_01/lot.htm)

srvctl status home -o oracle_home -s state_file

srvctl status home Options

Option Description
-o Complete path of the Oracle home
-s Complete path of the state file

srvctl stop home -o oracle_home -s state_file [-t stop_options] [-f]

srvctl stop home Options

Option Description
-o Complete path of the Oracle home
-s Complete path of the state file
-t stop_options SHUTDOWN command options for the database (for example: NORMAL, TRANSACTIONAL, IMMEDIATE, or ABORT). Default is IMMEDIATE.
-f Force stop each component

srvctl start home -o oracle_home -s state_file

srvctl start home Options

Option Description
-o Complete path of the Oracle home
-s Complete path of the state file. The state file contains the current state information for the components in the Oracle home and is created when the srvctl stop home command or the srvctl status home command is run.

 

As prep for the relinking of the software I performed following step:

srvctl status home -o /opt/oracle/product/112_ee_64/db -s /var/tmp/state_file.status

srvctl stop home -o /opt/oracle/product/112_ee_64/db -s /var/tmp/state_file.dmp

Note: in the stop home command a file is created at given location. Oracle will put in all Instances-Databases that are part of that specific Home. The nice part is that this  .dmp file can be used in one command to start all databases again once things are done.

In Order to relink the Rdbms software:

After shutting down the databases (see above):

Had the ORACLE_HOME  set properly

$ORACLE_HOME/bin/relink all

Note: writing relink log to: /opt/oracle/product/112_ee_64/db/install/relink.log

In Order to relink the Oracle Restart software:

Prepare the Oracle Grid Infrastructure for a Standalone Server home for modification using the following procedure:

  1. Log in as the Oracle Grid Infrastructure software owner user and change the directory to the path Grid_home/bin, where Grid_home is the path to the Oracle Grid Infrastructure home:
cd /opt/crs/product/112_ee_64/crs/bin
  1. Shut down the Oracle Restart stack using the following command:
crsctl stop has –f

This will show:

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'myhost01hr'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'myhost01hr'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'myhost01hr' succeeded

CRS-2673: Attempting to stop 'ora.evmd' on 'myhost01hr'

CRS-2677: Stop of 'ora.evmd' on 'myhost01hr' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'myhost01hr' has completed

CRS-4133: Oracle High Availability Services has been stopped.

oracle@myhost01hr:/opt/crs/product/112_ee_64/crs/bin [+ASM]#

Then:

Relink Oracle Grid Infrastructure:

  1. Login as root
    
    cd /opt/crs/product/112_ee_64/crs/crs/install
    
    perl roothas.pl -unlock
    
    Log in as the Oracle Grid Infrastructure for a Standalone Server owner:
    
    $ export ORACLE_HOME=/opt/crs/product/112_ee_64/crs
    
    $ $ORACLE_HOME/bin/relink

Note: This will show: oracle@myhost01hr:/opt/oracle [+ASM]# $ORACLE_HOME/bin/relink
writing relink log to: /opt/crs/product/112_ee_64/crs/install/relink.log

  1. Login as root again:
    
    cd/opt/crs/product/112_ee_64/crs/rdbms/install/
    
    ./rootadd_rdbms.sh 
    
    Note. Rootadd_rdbms came back very fast without any output.
    
    cd /opt/crs/product/112_ee_64/crs/crs/install
    
    perl roothas.pl -patch

Checked with:

  1. export $ORACLE_HOME=/opt/crs/product/112_ee_64/crs/
  2. ./crsctl check has

This showed:CRS-4638: Oracle High Availability Services is online

Starting the databases under control in an easy way:

When the command srvctl stop home was issued a  file was created in /var/tmp as state_file.dmp.  Now we can start all dbs again with one command:

srvctl start home -o /opt/oracle/product/112_ee_64/db -s /var/tmp/state_file.dmp

As always happy reading And happy Dba..

 

The good , the bad and the OPatch (applying PSU April 2014 on Rac and Oracle Restart)

Introduction:

 

Last couple of weeks  I have been busy  patching and upgrading Production , Preproduction en test environment and during those activities OPatch was my friend and tool for that.  Looking back an after talking to colleagues I decided to create a post for this .  In my patching activities I had to apply  a recent PSU patch to both the Grid Infra structure and Rdbms , do an Upgrade of the software  and add the latest PSU patch again. In  your  preparations for OPatch I had issues with regard to storage present on the  mount-point of the Grid Infrastructure . So as part of  activities  you should take a look at your file-system size  Since the  PSU patches will need at least  5 GB free space in the mount.

Preparations:

 

  • As was mentioned in the introduction make sure you have at least  5GB ( more is better in this case) in the mount-point where the Grid infra Structure is located . In my case I had /opt/crs/product/11202/crs as a mount with 15GB of space. In this mount the grid software had been installed and One Psu patch had been applied in the old days ( we are talking October 2012 PSU ). And while applying a required PSU (October 2013) ( required for the upgrade to Oracle 11.2.0.3)  there was not enough space to install the software.
  • Since my current platform is Linux ( this is all about patching Rac environments and Oracle Restart env.) I looked at Metalink and downloaded: p6880880_112000_Linux-x86-64.zip. With every PSU patch you install you should ask yourself is my opatch up to date enough , or should  I download a fresh copy  from Metalink. I tend to  check  and to download a fresh copy every time i am my T-shirt “I-m a patch Dba today and I Like it “.
  • In my environment my software installs look pretty much like this :
    • Grid Infra structure is installed in /opt/crs/product//crs
    • Rdbms is installed in /opt/oracle/product/11202_ee_64/db
    • oh and a bit confusing perhaps my ORACLE_BASE is  the same as the home of the ORACLE user ( which is /opt/oracle)

## tips

•    Make a subdirectory for each psu patch you apply if un unzip N psu patches in same directory opatch will apply them every  time again.
•    Is auto really auto , tend to do it with –oh  which still works fine for me.
•    Keep your Opatch tool up to date .

## Setting up your patching :

oracle@mysrvr:/opt/oracle/stage []# dir
drwxr-xr-x  5 oracle dba     4096 Jun 23 13:29 .
drwxr-xr-x 32 oracle dba     4096 Jun 23 15:22 ..
drwxr-xr-x  2 oracle dba     4096 Jun 11 13:32 OPatchlogs
drwxr-xr-x  2 oracle dba     4096 Jun 23 13:28 psuApr2014
drwxr-xr-x  2 oracle dba     4096 Jun 23 13:29 psuOct2013

## inside psuOct2013

oracle@mysrvr:/opt/oracle/stage/psuOct2013 []# ls -ltr
total 288260
-rw-r–r– 1 oracle dba        21 Apr  4  2013 README.txt
drwxr-xr-x 5 oracle dba      4096 Apr  4  2013 16459322
-rw-r–r– 1 oracle dba       450 Oct  9  2013 bundle.xml
drwxrwxr-x 9 oracle dba      4096 Oct 10  2013 17082367
-rw-rw-r– 1 oracle dba    141496 Jan 20 05:18 README.html
-rw-rw-r– 1 oracle dba    136871 Jan 20 05:18 PatchSearch.xml
-rwxr-xr-x 1 oracle dba 294574955 Jun  4 07:28 p17272753_112020_Linux-x86-64.zip

## Inside psuApr2014

oracle@mysrvr:/opt/oracle/stage/psuApr2014 []# ls -ltr
total 586820
drwxr-xr-x  5 oracle dba      4096 Jan  9 16:27 17592127
drwxrwxr-x 12 oracle dba      4096 Feb  5 07:04 18031683
-rw-r–r–  1 oracle dba       450 Feb 10 10:16 bundle.xml
-rw-r–r–  1 oracle dba         0 Feb 10 10:17 README.txt
-rw-rw-r–  1 oracle dba     59977 Apr 15 12:18 README.html
-rw-rw-r–  1 oracle dba    125015 Apr 15 14:17 PatchSearch.xml
-rwxr-xr-x  1 oracle dba 600096863 May 16 15:33 p18139678_112030_Linux-x86-64.zip

 

## Applying  PSU April 2014

unzip /opt/oracle/stage/p6880880_112000_Linux-x86-64.zip in your GRID_HOME and ORACLE_HOME directory
/opt/oracle/product/11203_ee_64/db/OPatch/ocm/bin/ocm.rsp  set up a response file (and make not of the absolute path for that response file because you will need it during opatch apply.
/opt/crs/product/11203/crs/OPatch/ocm/bin/ocm.rsp : that is my absolute path to the response file
unzip   p18139678_112030_Linux-x86-64.zip ( this was PSU april 2014 )
AS ROOT:export PATH=/opt/crs/product/11203/crs/OPatch:$PATH
export PATH=/opt/oracle/product/11203_ee_64/db/OPatch:$PATH
which opatch ( check if root can run opatch now )
PER NODE in your Cluster as ROOT :
##Crs
opatch auto /opt/oracle/stage/

unzip /opt/oracle/stage/p6880880_112000_Linux-x86-64.zip in your cdora directory
/opt/oracle/product/11203_ee_64/db/OPatch/ocm/bin/ocm.rsp
/opt/crs/product/11203/crs/OPatch/ocm/bin/ocm.rsp
unzip   p18139678_112030_Linux-x86-64.zip
export PATH=/opt/crs/product/11203/crs/OPatch:$PATH
export PATH=/opt/oracle/product/11203_ee_64/db/OPatch:$PATH
which opatch
PER NODE:
##Crs
opatch auto /opt/oracle/stage/

unzip /opt/oracle/stage/p6880880_112000_Linux-x86-64.zip in your cdora directory
/opt/oracle/product/11203_ee_64/db/OPatch/ocm/bin/ocm.rsp
/opt/crs/product/11203/crs/OPatch/ocm/bin/ocm.rsp
unzip   p18139678_112030_Linux-x86-64.zip
export PATH=/opt/crs/product/11203/crs/OPatch:$PATH
export PATH=/opt/oracle/product/11203_ee_64/db/OPatch:$PATH
which opatch
PPER NODE AS ROOT:##Crsopatch auto /opt/oracle/stage/psuApr2014 -ocmrf /opt/crs/product/11203/crs/OPatch/ocm/bin/ocm.rsp -oh /opt/crs/product/11203/crs

##Rdbms

opatch auto /opt/oracle/stage/psuApr2014 -ocmrf /opt/oracle/product/11203_ee_64/db/OPatch/ocm/bin/ocm.rsp -oh /opt/oracle/product/11203_ee_64/db

## Oracle Restart

/opatch auto /opt/oracle/stage/psuApr2014 -ocmrf /opt/crs/product/11203/crs/OPatch/ocm/bin/ocm.rsp -oh /opt/crs/product/11203/crs

/opatch auto /opt/oracle/stage/psuApr2014 -ocmrf /opt/crs/product/11203/crs/OPatch/ocm/bin/ocm.rsp -oh /opt/oracle/product/11203_ee_64/db

-ocmrf /opt/crs/product/11203/crs/OPatch/ocm/bin/ocm.rsp -oh /opt/crs/product/11203/crs
##Rdbms
opatch auto /opt/oracle/stage/11203 -ocmrf /opt/oracle/product/11203_ee_64/db/OPatch/ocm/bin/ocm.rsp -oh /opt/oracle/product/11203_ee_64/db

## Oracle Restart
/opatch auto /opt/oracle/stage/psuApr2014 -ocmrf /opt/crs/product/11203/crs/OPatch/ocm/bin/ocm.rsp -oh /opt/crs/product/11203/crs

/opatch auto /opt/oracle/stage/psuApr2014 -ocmrf /opt/crs/product/11203/crs/OPatch/ocm/bin/ocm.rsp -oh /opt/oracle/product/11203_ee_64/db

 

And as last recommendation . Check the logfiles that are produced during the OPatch in detail  cause i have seen a situation where the OPatch reported “succeeded”  but a detailed look in the logs showed that one of the patches had not been applied due to lack of space !!!!

 

As always happy reading and have a great day,

 

Mathijs