Introduction
For one of the new environments i have setup TDE ( transparent Database Encryption) aiming at encryption on the tablespaces level for a Rac database. As part of the preparations i have created a Diskgroup in ASM with normal redundancy and i have created an ACFS filesystem on top of that which will hold the TDE – Keys.
After restarting the instances ( tailing the alert log of both is an old hab it ). This is what i noticed:
In node of instance 2 I saw in alert that:
2019-09-13T09:36:34.512565+02:00
Errors in file /app/oracle/diag/diag/rdbms/MYDBU10a/MYDBU10A2/trace/MYDBU10A2_j001_354937.trc:
ORA-12012: error on auto execute of job 4
ORA-28365: wallet is not open
ORA-06512: at “SYS.ADMIN_AUDIT”, line 102
## On os level in the mount of the Acfs filesystem ( /app/oracle/admin/WALLET ) I Saw this :
total 84
2 4 drwxrwxr-x. 5 oracle dba 4096 Sep 13 07:58 .
15 4 drwxr-xr-x. 8 oracle dba 4096 Sep 13 05:42 ..
44 4 -rw-r–r–. 1 oracle dba 100 Sep 13 07:58 afiedt.buf
21 64 drwx——. 2 root root 65536 Sep 13 05:42 lost+found
? ? d?????????? ? ? ? ? ? MYDBU10A
##Followed below steps :
## as root
mysrvr4dr:root:/root $ /sbin/acfsutil registry
Mount Object:
Device: /dev/asm/MYDBU10avol-125
Mount Point: /app/oracle/admin/MYDBU10A
Disk Group: ACFS
Volume: MYDBU10AVOL
Options: none
Nodes: all
Accelerator Volumes:
Mount Object:
Device: /dev/asm/netcfg_vol-125
Mount Point: /app/oracle/admin/NETCFG/12.2.0.1/db/network/admin
Disk Group: ACFS
Volume: NETCFG_VOL
Options: norootsuid
Nodes: all
Accelerator Volumes:
Mount Object:
Device: /dev/asm/tde_vol-125
Mount Point: /app/oracle/admin/WALLET
Disk Group: ACFS
Volume: TDE_VOL
Options: none
Nodes: all
Accelerator Volumes:
## as root :
mysrvr4dr:root:/root $ umount /app/oracle/admin/WALLET
umount: /app/oracle/admin/WALLET: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount.acfs: CLSU-00100: operating system function: OfsWaitProc failed with error data: 32
umount.acfs: CLSU-00101: operating system error message: Broken pipe
umount.acfs: CLSU-00103: error location: OWPR_1
umount.acfs: ACFS-04151: unmount of mount point /app/oracle/admin/WALLET failed
## who is out there :
mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ksh 82160 oracle cwd DIR 252,64003 4096 74 /app/oracle/admin/WALLET/MYDBU10A (deleted)
ksh 82160 oracle 10r DIR 252,64003 4096 74 /app/oracle/admin/WALLET/MYDBU10A (deleted)
ksh 240476 oracle cwd DIR 252,64003 4096 2 /app/oracle/admin/WALLET
ksh 240476 oracle 11r DIR 252,64003 4096 2 /app/oracle/admin/WALLET
## kill them all
mysrvr4dr:root:/root $ kill -9 240476
mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ksh 82160 oracle cwd DIR 252,64003 4096 74 /app/oracle/admin/WALLET/MYDBU10A (deleted)
ksh 82160 oracle 10r DIR 252,64003 4096 74 /app/oracle/admin/WALLET/MYDBU10A (deleted)
mysrvr4dr:root:/root $ kill -9 82160
mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET
## still root then do this:
mysrvr4dr:root:/root $ umount /app/oracle/admin/WALLET
# ah but a mount does not work since acfs mounts are not part of fstab.
mysrvr4dr:root:/root $ mount /app/oracle/admin/WALLET
mount: can’t find /app/oracle/admin/WALLET in /etc/fstab
mysrvr4dr:root:/root $ exit
logout
## as oracle
srvctl start filesystem -device /dev/asm/tde_vol-125 -node mysrvr4dr
oracle@mysrvr4dr:/home/oracle []# cd /app/oracle/admin
oracle@mysrvr4dr:/app/oracle/admin []# ls -lisa
total 32
15 4 drwxr-xr-x. 8 oracle dba 4096 Sep 13 05:42 .
2 4 drwxr-xr-x. 15 oracle dba 4096 Sep 12 11:36 ..
786508 4 drwxr-xr-x. 3 oracle dba 4096 Sep 9 09:25 +APX
16 4 drwxr-xr-x. 3 oracle dba 4096 Sep 3 11:37 +ASM
2 4 drwxrwxr-x. 19 oracle dba 4096 Sep 13 08:15 MYDBU10A
21 0 lrwxrwxrwx. 1 oracle dba 8 Sep 12 09:43 MYDBU10A1 -> MYDBU10A
20 0 lrwxrwxrwx. 1 oracle dba 8 Sep 12 09:43 MYDBU10A2 -> MYDBU10A
393302 4 drwx——. 4 oracle dba 4096 Sep 3 15:47 _mgmtdb
917505 4 drwxrwxr-x. 3 oracle dba 4096 Sep 13 05:21 NETCFG
2 4 drwxrwxr-x. 5 oracle dba 4096 Sep 13 08:52 WALLET
Works again . all is well when it ends well .
Mathijs Bruggink 