TDE SETUP in a rac environment:

Summary:

This document will share the steps to implement TDE (transparent Database encryption) in the database in a Cluster.  Required Steps that are needed to implement it:

  • have the requirement to have an ACFS file system in place with proper setup there (see below).
  • Have the requirement to have an adapted sqlnet.ora in a centralized location.

Details:

  1. Setup will require preparations. In our case, the first database was MYDB. Steps below will be identical for each database in scope. (Of course you need to change the database name according to the database in your scope).

Mysrv[3-4]dr is holding the MYDB database.

  • Create a Diskgroup in normal Redundancy and call it TDE_KEYS.
  • Prepare the acfs created mountpoint by creating a TDE_VOL.
  • Mount the TDE_VOL as /app/oracle/admin/WALLET
  • On the acfs mount (/app/oracle/admin/WALLET
  • cd /app/oracle/admin/WALLET

mkdir MYDB,

ln -sf MYDB MYDB <1,2>

This acfs filesystem is shared between the clusternode.

With the mkdir you wil create an entry for  the database you are about to encrypt , with the 2 links you will point both instances to the database directory , to keep all in a central place.

  • Make sure you have a centralized Sqlnet.ora and adapt this one. As you can see the ORACLE_SID ( the instance name offers a flexible input in this way):

ENCRYPTION_WALLET_LOCATION =

  (SOURCE =(METHOD = FILE)(METHOD_DATA =

    (DIRECTORY = /app/oracle/admin/WALLET/$ORACLE_SID/ )

                          )

  )

  • In the database first check:

SQL> select * from v$encryption_wallet;

WRL_TYPE

——————–

WRL_PARAMETER

——————————————————————————–

STATUS                                       WALLET_TYPE               WALLET_OR FULLY_BAC

—————————— ——————– ——— ———

    CON_ID

———-

FILE

/app/oracle/admin/WALLET/MYDB1/

NOT_AVAILABLE                                    UNKNOWN                    SINGLE    UNDEFINED

                 0

  • With the below command you will create the keystore  and you will give the keystore a password.

SQL> administer key management create keystore ‘/app/oracle/admin/WALLET/MYDB/’ identified by “mypwd19!”;

## Once that command is given Sqlplus  will report:

keystore altered.

## TIP Be sure the correct sqlnet is read (links?)

  • run below command to open your keystore:

SQL> administer key management set keystore open identified by “mypwd19!”;

## this will report in sqlplus:

keystore altered.

  1. Now is a good time to Check the status of your wallet:

SQL> select * from v$encryption_wallet;

WRL_TYPE

——————–

WRL_PARAMETER

——————————————————————————–

STATUS                                       WALLET_TYPE               WALLET_OR FULLY_BAC

—————————— ——————– ——— ———

    CON_ID

———-

FILE

/app/oracle/admin/WALLET/MYDB1/

OPEN_NO_MASTER_KEY                    PASSWORD                    SINGLE    UNDEFINED

                 0

  1.  Check  your encryption keys.

SQL> select key_id,activation_time from v$encryption_keys;

## sql will 1st time report:

no rows selected

## On Os you should already see the wallet similar to below:

SQL> host

oracle@Mysrv3dr:/app/oracle/admin/WALLET/MYDB []# ls -lisa

77 4 -rw——-. 1 oracle dba 2555 Sep 13 05:51 ewallet.p12

  1. In Sqlplus  now it is time to create the key:

SQL> administer key management create key identified by “mypwd19!” with backup;

  1. When you check your encryption keys again now:

SQL> select key_id,activation_time from v$encryption_keys;

KEY_ID

——————————————————————————

ACTIVATION_TIME

—————————————————————————

NcC1701D+mbkK+6v92xdM/qIxcXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  1. In sqlplus give below command:

SQL> administer key management use key ‘ NcC1701D+mbkK+6v92xdM/qIxcXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

‘ identified by “mypwd19!” with backup;

## sqlplus will report:

keystore altered.

  1. Check again your encryption keys again to get something similar to below:

SQL> select key_id,activation_time from v$encryption_keys;

## now you will see an activation_time Too:

KEY_ID

——————————————————————————

ACTIVATION_TIME

—————————————————————————

NcC1701D+mbkK+6v92xdM/qIxcXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

13-SEP-19 06.01.49.995963 AM +02:00

  1. As a test create a small  tablspace:

SQL> create tablespace ENC datafile ‘+MYDB_DATA’ size 10M encryption using ‘AES256’ default storage(encrypt);

## Check this with:

select tablespace_name,encrypted from dba_tablespaces order by 1;

## Sqlplus should report something like this:

SQL> select tablespace_name,encrypted from dba_tablespaces order by 1;

TABLESPACE_NAME                ENC

—————————— —

ENC                            YES

Now you are all set to create all required application tablespaces with these extra parameters  encryption using ‘AES256’ default storage(encrypt).

Automatically open the wallet

Note:   Basically a wallet needs to be open before you can access the database. This however can and should be altered: When opening your database this should automatically open the wallet:

  1. In sqlplus check:

SQL> select * from v$encryption_wallet;

WRL_TYPE

——————–

WRL_PARAMETER

——————————————————————————–

STATUS                                       WALLET_TYPE               WALLET_OR FULLY_BAC

—————————— ——————– ——— ———

    CON_ID

———-

FILE

/app/oracle/admin/WALLET/MYDB1/

OPEN                                           PASSWORD                    SINGLE    NO

                 0

  • Give  this command:

SQL> administer key management create auto_login keystore from keystore ‘/app/oracle/admin/WALLET/MYDB/’ identified by “mypwd19!”;

## sqlplus will report:

keystore altered.

## On OS you will see :

oracle@Mysrv3dr:/app/oracle/admin/WALLET/MYDB [MYDB1]# ls -lisa

total 48

74  4 drwxr-xr-x. 2 oracle dba 4096 Sep 13 06:37 .

 2  4 drwxrwxr-x. 5 oracle dba 4096 Sep 13 05:44 ..

82 12 -rw——-. 1 oracle dba 5304 Sep 13 06:37 cwallet.sso

78  4 -rw——-. 1 oracle dba 2555 Sep 13 05:55 ewallet_2019091303553484.p12

80  4 -rw——-. 1 oracle dba 3803 Sep 13 05:57 ewallet_2019091303572233.p12

81  8 -rw——-. 1 oracle dba 5067 Sep 13 06:01 ewallet_2019091304014993.p12

77 12 -rw——-. 1 oracle dba 5259 Sep 13 06:01 ewallet.p12

  • And in Sqlplus  too you  will see  a change.

SQL> SELECT * FROM v$encryption_wallet;

WRL_TYPE

——————–

WRL_PARAMETER

——————————————————————————–

STATUS                                       WALLET_TYPE               WALLET_OR FULLY_BAC

—————————— ——————– ——— ———

    CON_ID

———-

FILE

/app/oracle/admin/WALLET/MYDB1/

OPEN                                           AUTOLOGIN                  SINGLE    NO

                 0

Appendix 1 Create an acfs file system:

Idea was to setup acfs via asmca. This however did not work in first implementation due to error because ora.proxy_advm is offline on some nodes.

It should be:

crsctl stat res ora.proxy_advm -t

——————————————————————————–

Name           Target  State        Server                   State details      

——————————————————————————–

Local Resources

——————————————————————————–

ora.proxy_advm

               ONLINE  ONLINE       Mysrv3dr                 STABLE

               ONLINE  ONLINE       Mysrv4dr                 STABLE

If the status is offline check if the acfsmodule are loaded: lsmod | grep acfs

oracleacfs           4840664  2

oracleoks             663240  2 oracleacfs,oracleadvm

if they are not loaded on the local node: log in as root and perform

$GRID_HOME/bin/csrctl stop crs

$GRID_HOME/bin/acfstool install

$GRID_HOME/bin/crsctl start crs

Modules should be loaded now. Volume can be created in asmca (high protection, TDE_VOL1, 1GB)

The volume will show up under /dev/asm (/dev/asm/tde_vol1-301 in this case).

Mkdir  -p /app/oracle/admin/wallet

Cd /app/oracle

Chown –R oracle:dba admin

/sbin/mkfs -t acfs /dev/asm/tde_vol1-302

Register in clusterware:

/sbin/acfsutil registry -a /dev/asm/tde_vol1-302 /app/oracle/admin/WALLET

When wallet is not open in TDE Environment (ORA-28365)

Introduction

For one of the new environments i have setup TDE ( transparent Database Encryption) aiming at encryption on the tablespaces level for a Rac database. As part of the preparations i have created a Diskgroup in ASM with normal redundancy and i have created an ACFS filesystem on top of that which will hold the TDE – Keys.

After restarting the instances ( tailing the alert log of both is an old hab it ). This is what i noticed:

In node of instance 2  I saw in alert that:

2019-09-13T09:36:34.512565+02:00

Errors in file /app/oracle/diag/diag/rdbms/MYDBU10a/MYDBU10A2/trace/MYDBU10A2_j001_354937.trc:

ORA-12012: error on auto execute of job 4

ORA-28365: wallet is not open

ORA-06512: at “SYS.ADMIN_AUDIT”, line 102

## On os level in the mount of the Acfs filesystem ( /app/oracle/admin/WALLET ) I Saw this :

total 84

 2  4 drwxrwxr-x. 5 oracle dba   4096 Sep 13 07:58 .

15  4 drwxr-xr-x. 8 oracle dba   4096 Sep 13 05:42 ..

44  4 -rw-r–r–. 1 oracle dba    100 Sep 13 07:58 afiedt.buf

21 64 drwx——. 2 root   root 65536 Sep 13 05:42 lost+found

 ?  ? d?????????? ? ?      ?        ?            ? MYDBU10A

##Followed below steps :

## as root

mysrvr4dr:root:/root $ /sbin/acfsutil registry

Mount Object:

  Device: /dev/asm/MYDBU10avol-125

  Mount Point: /app/oracle/admin/MYDBU10A

  Disk Group: ACFS

  Volume: MYDBU10AVOL

  Options: none

  Nodes: all

  Accelerator Volumes:

Mount Object:

  Device: /dev/asm/netcfg_vol-125

  Mount Point: /app/oracle/admin/NETCFG/12.2.0.1/db/network/admin

  Disk Group: ACFS

  Volume: NETCFG_VOL

  Options: norootsuid

  Nodes: all

  Accelerator Volumes:

Mount Object:

  Device: /dev/asm/tde_vol-125

  Mount Point: /app/oracle/admin/WALLET

  Disk Group: ACFS

  Volume: TDE_VOL

  Options: none

  Nodes: all

  Accelerator Volumes:

## as root :

mysrvr4dr:root:/root $ umount /app/oracle/admin/WALLET

umount: /app/oracle/admin/WALLET: target is busy.

        (In some cases useful info about processes that use

         the device is found by lsof(8) or fuser(1))

umount.acfs: CLSU-00100: operating system function: OfsWaitProc failed with error data: 32

umount.acfs: CLSU-00101: operating system error message: Broken pipe

umount.acfs: CLSU-00103: error location: OWPR_1

umount.acfs: ACFS-04151: unmount of mount point /app/oracle/admin/WALLET failed

## who is out there :

mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET

COMMAND    PID   USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME

ksh      82160 oracle  cwd    DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

ksh      82160 oracle   10r   DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

ksh     240476 oracle  cwd    DIR 252,64003     4096    2 /app/oracle/admin/WALLET

ksh     240476 oracle   11r   DIR 252,64003     4096    2 /app/oracle/admin/WALLET

## kill them all

mysrvr4dr:root:/root $ kill -9 240476

mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET

COMMAND   PID   USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME

ksh     82160 oracle  cwd    DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

ksh     82160 oracle   10r   DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

mysrvr4dr:root:/root $ kill -9 82160

mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET

## still root then do this:

mysrvr4dr:root:/root $ umount /app/oracle/admin/WALLET

# ah but a mount does not work since acfs mounts are not part of fstab.

mysrvr4dr:root:/root $ mount /app/oracle/admin/WALLET

mount: can’t find /app/oracle/admin/WALLET in /etc/fstab

mysrvr4dr:root:/root $ exit

logout

## as oracle

srvctl start filesystem -device /dev/asm/tde_vol-125 -node mysrvr4dr

oracle@mysrvr4dr:/home/oracle []# cd /app/oracle/admin

oracle@mysrvr4dr:/app/oracle/admin []# ls -lisa

total 32

    15 4 drwxr-xr-x.  8 oracle dba 4096 Sep 13 05:42 .

     2 4 drwxr-xr-x. 15 oracle dba 4096 Sep 12 11:36 ..

786508 4 drwxr-xr-x.  3 oracle dba 4096 Sep  9 09:25 +APX

    16 4 drwxr-xr-x.  3 oracle dba 4096 Sep  3 11:37 +ASM

     2 4 drwxrwxr-x. 19 oracle dba 4096 Sep 13 08:15 MYDBU10A

    21 0 lrwxrwxrwx.  1 oracle dba    8 Sep 12 09:43 MYDBU10A1 -> MYDBU10A

    20 0 lrwxrwxrwx.  1 oracle dba    8 Sep 12 09:43 MYDBU10A2 -> MYDBU10A

393302 4 drwx——.  4 oracle dba 4096 Sep  3 15:47 _mgmtdb

917505 4 drwxrwxr-x.  3 oracle dba 4096 Sep 13 05:21 NETCFG

     2 4 drwxrwxr-x.  5 oracle dba 4096 Sep 13 08:52 WALLET

Works again . all is well when it ends well .