11.2 Rac – Instance unwilling to start with Error ORA-00132

Introduction:

This week  had one of those days where after working together with customer and Oracle Consultant  one  of  the Rac Databases needed a restart due to a variable change.  Together with customer it was decided to restart second  instance first so first took care of the  environment by enabling a Maintenance mode so our monitoring tool would not strike me with tickets.

The second instance was   friendly:

  • the shutdown: srvctl stop instance -d -i went smooth .
  • the startup: srvctl start instance -d -i went smooth .

Then i  performed same steps on first Instance:

  • the shutdown: srvctl stop instance -d -i went smooth .
  • the startup: srvctl start instance -d -i   failed  with an error message that puzzled me a lot .

ORA-00132: syntax error or unresolved network name ‘myserver-scan:33000’

I checked my scan listener settings:

oracle@myserver-s1:/opt/oracle [CRS]#
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node myserver-s4
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node myserver-s2
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node myserver-s3

oracle@myserver-s1:/opt/oracle [CRS]# srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:33000
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:33000
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:33000

Crs agent log showed same error and i could not start the instance via the  srvctl tool. The error suggested that my remote listener ( i am using scan ) was the issue.. But the  remote listener was not even part of  my tnsnames.ora, since my spfile value for the remote listener = myserver-scan:33000.

That was soo confusing. Well  I attempted to start the Instance via sqlplus  and that worked. I checked the cluster and it reported  ONLINE ONLINE on the resource so  the first pressure was off.

To solve it , make sure that  the sqlnet.ora is similar on all environments and it seems to need the ezconnect information in there  ( in names.directory_path).([ID 1318026.1])

Happy reading

Mathijs

Is it really the good – the – bad – and the Gui Dba building Racs?

Introduction

From my experience for some Database Administrators / sites it  still seem to be more tough /  hardcore to use the command line instead of using tools that  could do the same for you only with a lot less effort. And maybe that is also regarded to be a risk cause it is so easy. Well my thoughts on this is that a healthy mix on the use Gui – tools and command lines should  be an asset  to every administrator. And indeed before even starting debate when using the Gui of course you should always know what you are doing .. but then again that is always the case is it not ?

Creating an Rac manually – using DBCA.

Personally i think that the DBCA tool is  your best way forward if you are creating a new Rac environment. Of course  this tool also has also its limits ( such as on a 4 Node environment it will  by default set your cluster_database_instances default to 4 even when you build only a two node Rac)  but in general it is best way to work cause not only will create the Database , and the instances on the various Nodes  but it also adds the information to the 11G Grid infrastructure ( the Cluster ware ). And  for documentation purposes you can / should also always generate the create scripts in a default ( for me always /opt/oracle//admin/create) directory.

Yesterday i had to support a colleague who decided to create a two node RAC by hand . Indeed  even  when i suggested to him that using the DBCA would be helpful and fast in his actions it was his call to do it all by hand. But then he got stuck in the process with various actions.

If you decide to do  a create manually ( as they say maybe you should try to not use the Gui just cause it is written in this blog).  I think following steps are important:

  1. Work accurately ( setting up init oras , one for single Instance – Db) add the Instance specific information (in my case undo tablespace and local_listener ) were needed. Create spfile from that pfile.
  2. Start – create the single instance-db in full with the spfile.
  3. Once the single instance is alive start alter your spfile:
    1. alter system set cluster_database=true scope = spfile:
    2. alter system set instance_number= 1 scope = spfile sid=”instance1″ ;
    3. alter system set instance_number= 2 scope = spfile sid=”instance2″ ;
    4. alter system set thread = 1 scope = spfile sid=”instance1″ ;
    5. alter system set thread = 2 scope = spfile sid=”instance2″ ;
  4. Add at least the 3 redo groups to the second thread( alter database add logfile thread 2 group x:)
  5. set up the remote_listener with the scan listener
  6. alter system enable thread 2 ;
  7. Add a Taf service.
  8. As a check stop and start and stop both instances manually to see if the database is shared as it should be between the two instances.
  9. Add the information to the cluster:

srvctl  add database -d DB -o /opt/oracle/product/oracleVersion

srvctl  add instance -d DB -i DB1 -n Node1 srvctl  add instance -d DB -i DB2 -n Node2

srvctl modify instance -d DB -i DB1 -s +ASM1 srvctl modify instance -d DB -i DB2 -s +ASM2

srvctl  add service -d DB -s DB_TAF.domain -r DB1,DB2 srvctl start database -d DB srvctl start  service -d DB -s DB_TAF.domain

Ok, are  you still with me , do i rest my case here that it is a lot of work to do this the manual way ? And that these steps can be very  lets say error prone if you do not work 100%  accurate..

Yesterdays event where indeed not successful at first . Amongst other issues He got :

PRCR-1079 : Failed to start resource ora.db.db
CRS-5017: The resource action “ora.db.db start” met the following error:
ORA-29760: instance_number parameter not specified

Once again Google was our friend it showed post of a fellow blogger with regard to a Mos note:

MOS note ‘ORA-29760: instance_number parameter not specified’ When Starting the Database with Srvctl [ID 749515.1].

After that . Things worked .

Bottom-line of this story ? Plain and simple use the right and easiest tool for the Job you need to do. If that is a Gui  there is no dishonor in that . If you decide to go manual create , make sure you work accurately  hmm and have Google available for when things come up during the process.

Happy Reading ,

Mathijs