When wallet is not open in TDE Environment (ORA-28365)

Introduction

For one of the new environments i have setup TDE ( transparent Database Encryption) aiming at encryption on the tablespaces level for a Rac database. As part of the preparations i have created a Diskgroup in ASM with normal redundancy and i have created an ACFS filesystem on top of that which will hold the TDE – Keys.

After restarting the instances ( tailing the alert log of both is an old hab it ). This is what i noticed:

In node of instance 2  I saw in alert that:

2019-09-13T09:36:34.512565+02:00

Errors in file /app/oracle/diag/diag/rdbms/MYDBU10a/MYDBU10A2/trace/MYDBU10A2_j001_354937.trc:

ORA-12012: error on auto execute of job 4

ORA-28365: wallet is not open

ORA-06512: at “SYS.ADMIN_AUDIT”, line 102

## On os level in the mount of the Acfs filesystem ( /app/oracle/admin/WALLET ) I Saw this :

total 84

 2  4 drwxrwxr-x. 5 oracle dba   4096 Sep 13 07:58 .

15  4 drwxr-xr-x. 8 oracle dba   4096 Sep 13 05:42 ..

44  4 -rw-r–r–. 1 oracle dba    100 Sep 13 07:58 afiedt.buf

21 64 drwx——. 2 root   root 65536 Sep 13 05:42 lost+found

 ?  ? d?????????? ? ?      ?        ?            ? MYDBU10A

##Followed below steps :

## as root

mysrvr4dr:root:/root $ /sbin/acfsutil registry

Mount Object:

  Device: /dev/asm/MYDBU10avol-125

  Mount Point: /app/oracle/admin/MYDBU10A

  Disk Group: ACFS

  Volume: MYDBU10AVOL

  Options: none

  Nodes: all

  Accelerator Volumes:

Mount Object:

  Device: /dev/asm/netcfg_vol-125

  Mount Point: /app/oracle/admin/NETCFG/12.2.0.1/db/network/admin

  Disk Group: ACFS

  Volume: NETCFG_VOL

  Options: norootsuid

  Nodes: all

  Accelerator Volumes:

Mount Object:

  Device: /dev/asm/tde_vol-125

  Mount Point: /app/oracle/admin/WALLET

  Disk Group: ACFS

  Volume: TDE_VOL

  Options: none

  Nodes: all

  Accelerator Volumes:

## as root :

mysrvr4dr:root:/root $ umount /app/oracle/admin/WALLET

umount: /app/oracle/admin/WALLET: target is busy.

        (In some cases useful info about processes that use

         the device is found by lsof(8) or fuser(1))

umount.acfs: CLSU-00100: operating system function: OfsWaitProc failed with error data: 32

umount.acfs: CLSU-00101: operating system error message: Broken pipe

umount.acfs: CLSU-00103: error location: OWPR_1

umount.acfs: ACFS-04151: unmount of mount point /app/oracle/admin/WALLET failed

## who is out there :

mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET

COMMAND    PID   USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME

ksh      82160 oracle  cwd    DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

ksh      82160 oracle   10r   DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

ksh     240476 oracle  cwd    DIR 252,64003     4096    2 /app/oracle/admin/WALLET

ksh     240476 oracle   11r   DIR 252,64003     4096    2 /app/oracle/admin/WALLET

## kill them all

mysrvr4dr:root:/root $ kill -9 240476

mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET

COMMAND   PID   USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME

ksh     82160 oracle  cwd    DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

ksh     82160 oracle   10r   DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

mysrvr4dr:root:/root $ kill -9 82160

mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET

## still root then do this:

mysrvr4dr:root:/root $ umount /app/oracle/admin/WALLET

# ah but a mount does not work since acfs mounts are not part of fstab.

mysrvr4dr:root:/root $ mount /app/oracle/admin/WALLET

mount: can’t find /app/oracle/admin/WALLET in /etc/fstab

mysrvr4dr:root:/root $ exit

logout

## as oracle

srvctl start filesystem -device /dev/asm/tde_vol-125 -node mysrvr4dr

oracle@mysrvr4dr:/home/oracle []# cd /app/oracle/admin

oracle@mysrvr4dr:/app/oracle/admin []# ls -lisa

total 32

    15 4 drwxr-xr-x.  8 oracle dba 4096 Sep 13 05:42 .

     2 4 drwxr-xr-x. 15 oracle dba 4096 Sep 12 11:36 ..

786508 4 drwxr-xr-x.  3 oracle dba 4096 Sep  9 09:25 +APX

    16 4 drwxr-xr-x.  3 oracle dba 4096 Sep  3 11:37 +ASM

     2 4 drwxrwxr-x. 19 oracle dba 4096 Sep 13 08:15 MYDBU10A

    21 0 lrwxrwxrwx.  1 oracle dba    8 Sep 12 09:43 MYDBU10A1 -> MYDBU10A

    20 0 lrwxrwxrwx.  1 oracle dba    8 Sep 12 09:43 MYDBU10A2 -> MYDBU10A

393302 4 drwx——.  4 oracle dba 4096 Sep  3 15:47 _mgmtdb

917505 4 drwxrwxr-x.  3 oracle dba 4096 Sep 13 05:21 NETCFG

     2 4 drwxrwxr-x.  5 oracle dba 4096 Sep 13 08:52 WALLET

Works again . all is well when it ends well .

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s