Goldengate when TDE in your database is used.

Introduction

It might sound like a movie trilogy to touch similar topic for the 3rd time, and frankly in my last two topics touched ground on these topics:

Well today the third article has to rise. Simply because there is always an interesting interactions requiring a follow up between Admins and teams using a solution. In such galaxy far far away the question would rise:what will happen if you armor your tablespaces with TDE and then you decide to use Goldengate when classic capture is being used as tool to extract data from that environment. (spoiler alert Goldengate (GG) will break).

On the various capture models in Goldengate check this url:

https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/process_mode.htm#GIORA209

For this blog really all credits needs to be given to a brother in arms, a close friend and a most seasoned dba (Mr. Roel Hendriks) who I had the honor to work with during Project(s). He did the research implemented this solotion first. Since then I was able to implement it on various platforms. For saga completion it is now my goal to complete the trilogy with this blog.

Additional required steps with TDE And GG in place

  • Create package dbms_internal_clkm on database by running @?/rdbms/admin/prvtclkm.plb.
  • Grant this package to database account which extract data from the database: “grant execute on dbms_internal_clkm to ggapp;”. This package enables database accounts to extract data via GoldenGate in a encrypted database.
  • Find out the location of your wallet location via the view v$encryption_wallet or the sqlnet.ora file.

This will show something like :

oracle@mysrvr:/app/oracle/admin/WALLET/MYDB1 [MYDB1]# ls -lisa

total 48

74  4 drwxr-xr-x. 2 oracle dba 4096 Sep 13 06:37 .

 2  4 drwxrwxr-x. 5 oracle dba 4096 Sep 13 08:52 ..

82 12 -rw——-. 1 oracle dba 5304 Sep 13 06:37 cwallet.sso

78  4 -rw——-. 1 oracle dba 2555 Sep 13 05:55 ewallet_2019091303553484.p12

80  4 -rw——-. 1 oracle dba 3803 Sep 13 05:57 ewallet_2019091303572233.p12

81  8 -rw——-. 1 oracle dba 5067 Sep 13 06:01 ewallet_2019091304014993.p12

77 12 -rw——-. 1 oracle dba 5259 Sep 13 06:01 ewallet.p12

  • Execute at wallet location (you have to supply the wallet password) :

(means in this case in:  /app/oracle/admin/WALLET/ MYDB1)

mkstore -wrl ./ -createEntry ORACLE.SECURITY.CL.ENCRYPTION.ORACLEGG

Oracle Secret Store Tool : Version 12.2.0.1.0

Copyright (c) 2004, 2016, Oracle and/or its affiliates. All rights reserved.

 ## you have to enter a password ( twice):

— supplied password: hugo ( this password will be needed in GGSCI)

Enter your secret/Password:

Re-enter your secret/Password: 

Enter wallet password:

  • Verify the result:

mkstore -wrl . -list

Oracle Secret Store Tool : Version 12.2.0.1.0

Copyright (c) 2004, 2016, Oracle and/or its affiliates. All rights reserved.

  • Enter wallet password:

## when wallet password is given you will see similar to below:

Oracle Secret Store entries:

ORACLE.SECURITY.CL.ENCRYPTION.ORACLEGG

ORACLE.SECURITY.DB.ENCRYPTION.Ae/sVNN8L0+iv/uBOenqJsQZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ

ORACLE.SECURITY.DB.ENCRYPTION.AUAh7+KLWk+6v92xdM/qIxcZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ

ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY

ORACLE.SECURITY.ID.ENCRYPTION.

ORACLE.SECURITY.KB.ENCRYPTION.

ORACLE.SECURITY.KM.ENCRYPTION.Ae/sVNN8L0+iv/uBOenqJsQZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ

ORACLE.SECURITY.KM.ENCRYPTION.AUAh7+KLWk+6v92xdM/qIxcZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ

##

  • On os you will now see similar to this:

oracle@mysrvr:/app/oracle/admin/WALLET/MYDB1 [MYDB1]# ls -lisa

total 48

74  4 drwxr-xr-x. 2 oracle dba 4096 Sep 19 11:16 .

 2  4 drwxrwxr-x. 5 oracle dba 4096 Sep 13 08:52 ..

82 12 -rw——-. 1 oracle dba 5453 Sep 19 11:17 cwallet.sso

83  0 -rw——-. 1 oracle dba    0 Sep 19 11:16 cwallet.sso.lck

78  4 -rw——-. 1 oracle dba 2555 Sep 13 05:55 ewallet_2019091303553484.p12

80  4 -rw——-. 1 oracle dba 3803 Sep 13 05:57 ewallet_2019091303572233.p12

81  8 -rw——-. 1 oracle dba 5067 Sep 13 06:01 ewallet_2019091304014993.p12

77 12 -rw——-. 1 oracle dba 5408 Sep 19 11:17 ewallet.p12

79  0 -rw——-. 1 oracle dba    0 Sep 19 11:16 ewallet.p12.lck

  • Switch logfile.
  • Stop / start the database.
  • Be aware:

  The “wallet_type” in the v$encryption_wallet be ‘unknown’  because of using mkstore utility. This is expected behavior: https://docs.oracle.com/database/121/REFRN/GUID-054B6FBC-4263-4A91-AEBD-D6FB50C22107.htm#REFRN30488
7.168 V$ENCRYPTION_WALLET

  • SQL> select * from v$encryption_wallet;
  • WRL_TYPE
  • ——————–
  • WRL_PARAMETER
  • ——————————————————————————–
  • STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC
  • —————————— ——————– ——— ———
  •     CON_ID
  • ———-
  • FILE
  • /app/oracle/admin/WALLET/MYDB1/
  • OPEN                           UNKNOWN              SINGLE    NO
  •          0

Goldengate part, when TDE is involved

  1. Run GGSCI.
  2. Issue the ENCRYPT PASSWORD command to encrypt the shared secret so that it is obfuscated within the Extract parameter file. This is a security requirement.
  3. ENCRYPT PASSWORD hugo {AES128 | AES192 | AES256} ENCRYPTKEY keyname

Where:

  • hugo is the clear-text shared secret. This value is case-sensitive.
  • {AES128 | AES192 | AES256} specifies Advanced Encryption Standard (AES) encryption. Specify one of the values, which represents the desired key length.
  • keyname is the logical name of the encryption key in the ENCKEYS lookup file. Oracle GoldenGate uses this key to look up the actual key in the ENCKEYS file. To create a key and ENCKEYS file, see Administering Oracle GoldenGate for Windows and UNIX.

Example:

ENCRYPT PASSWORD hugo AES256 ENCRYPTKEY mykey1

  • In the Extract parameter file, use the DBOPTIONS parameter with the DECRYPTPASSWORD option. As input, supply the encrypted shared secret and the decryption key.

DBOPTIONS DECRYPTPASSWORD hugo {AES128 | AES192 | AES256} ENCRYPTKEY keyname

In our case the DBOPTIONS was:

DBOPTIONS DECRYPTPASSWORD “AADAAAAAAAAAAAMANAREQEREYDJFOBYALGZHQCLGUCCHQGBDZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZAAA

Happy reading,

Mathijs

TDE SETUP in a rac environment:

Summary:

This document will share the steps to implement TDE (transparent Database encryption) in the database in a Cluster.  Required Steps that are needed to implement it:

  • have the requirement to have an ACFS file system in place with proper setup there (see below).
  • Have the requirement to have an adapted sqlnet.ora in a centralized location.

Details:

  1. Setup will require preparations. In our case, the first database was MYDB. Steps below will be identical for each database in scope. (Of course you need to change the database name according to the database in your scope).

Mysrv[3-4]dr is holding the MYDB database.

  • Create a Diskgroup in normal Redundancy and call it TDE_KEYS.
  • Prepare the acfs created mountpoint by creating a TDE_VOL.
  • Mount the TDE_VOL as /app/oracle/admin/WALLET
  • On the acfs mount (/app/oracle/admin/WALLET
  • cd /app/oracle/admin/WALLET

mkdir MYDB,

ln -sf MYDB MYDB <1,2>

This acfs filesystem is shared between the clusternode.

With the mkdir you wil create an entry for  the database you are about to encrypt , with the 2 links you will point both instances to the database directory , to keep all in a central place.

  • Make sure you have a centralized Sqlnet.ora and adapt this one. As you can see the ORACLE_SID ( the instance name offers a flexible input in this way):

ENCRYPTION_WALLET_LOCATION =

  (SOURCE =(METHOD = FILE)(METHOD_DATA =

    (DIRECTORY = /app/oracle/admin/WALLET/$ORACLE_SID/ )

                          )

  )

  • In the database first check:

SQL> select * from v$encryption_wallet;

WRL_TYPE

——————–

WRL_PARAMETER

——————————————————————————–

STATUS                                       WALLET_TYPE               WALLET_OR FULLY_BAC

—————————— ——————– ——— ———

    CON_ID

———-

FILE

/app/oracle/admin/WALLET/MYDB1/

NOT_AVAILABLE                                    UNKNOWN                    SINGLE    UNDEFINED

                 0

  • With the below command you will create the keystore  and you will give the keystore a password.

SQL> administer key management create keystore ‘/app/oracle/admin/WALLET/MYDB/’ identified by “mypwd19!”;

## Once that command is given Sqlplus  will report:

keystore altered.

## TIP Be sure the correct sqlnet is read (links?)

  • run below command to open your keystore:

SQL> administer key management set keystore open identified by “mypwd19!”;

## this will report in sqlplus:

keystore altered.

  1. Now is a good time to Check the status of your wallet:

SQL> select * from v$encryption_wallet;

WRL_TYPE

——————–

WRL_PARAMETER

——————————————————————————–

STATUS                                       WALLET_TYPE               WALLET_OR FULLY_BAC

—————————— ——————– ——— ———

    CON_ID

———-

FILE

/app/oracle/admin/WALLET/MYDB1/

OPEN_NO_MASTER_KEY                    PASSWORD                    SINGLE    UNDEFINED

                 0

  1.  Check  your encryption keys.

SQL> select key_id,activation_time from v$encryption_keys;

## sql will 1st time report:

no rows selected

## On Os you should already see the wallet similar to below:

SQL> host

oracle@Mysrv3dr:/app/oracle/admin/WALLET/MYDB []# ls -lisa

77 4 -rw——-. 1 oracle dba 2555 Sep 13 05:51 ewallet.p12

  1. In Sqlplus  now it is time to create the key:

SQL> administer key management create key identified by “mypwd19!” with backup;

  1. When you check your encryption keys again now:

SQL> select key_id,activation_time from v$encryption_keys;

KEY_ID

——————————————————————————

ACTIVATION_TIME

—————————————————————————

NcC1701D+mbkK+6v92xdM/qIxcXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  1. In sqlplus give below command:

SQL> administer key management use key ‘ NcC1701D+mbkK+6v92xdM/qIxcXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

‘ identified by “mypwd19!” with backup;

## sqlplus will report:

keystore altered.

  1. Check again your encryption keys again to get something similar to below:

SQL> select key_id,activation_time from v$encryption_keys;

## now you will see an activation_time Too:

KEY_ID

——————————————————————————

ACTIVATION_TIME

—————————————————————————

NcC1701D+mbkK+6v92xdM/qIxcXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

13-SEP-19 06.01.49.995963 AM +02:00

  1. As a test create a small  tablspace:

SQL> create tablespace ENC datafile ‘+MYDB_DATA’ size 10M encryption using ‘AES256’ default storage(encrypt);

## Check this with:

select tablespace_name,encrypted from dba_tablespaces order by 1;

## Sqlplus should report something like this:

SQL> select tablespace_name,encrypted from dba_tablespaces order by 1;

TABLESPACE_NAME                ENC

—————————— —

ENC                            YES

Now you are all set to create all required application tablespaces with these extra parameters  encryption using ‘AES256’ default storage(encrypt).

Automatically open the wallet

Note:   Basically a wallet needs to be open before you can access the database. This however can and should be altered: When opening your database this should automatically open the wallet:

  1. In sqlplus check:

SQL> select * from v$encryption_wallet;

WRL_TYPE

——————–

WRL_PARAMETER

——————————————————————————–

STATUS                                       WALLET_TYPE               WALLET_OR FULLY_BAC

—————————— ——————– ——— ———

    CON_ID

———-

FILE

/app/oracle/admin/WALLET/MYDB1/

OPEN                                           PASSWORD                    SINGLE    NO

                 0

  • Give  this command:

SQL> administer key management create auto_login keystore from keystore ‘/app/oracle/admin/WALLET/MYDB/’ identified by “mypwd19!”;

## sqlplus will report:

keystore altered.

## On OS you will see :

oracle@Mysrv3dr:/app/oracle/admin/WALLET/MYDB [MYDB1]# ls -lisa

total 48

74  4 drwxr-xr-x. 2 oracle dba 4096 Sep 13 06:37 .

 2  4 drwxrwxr-x. 5 oracle dba 4096 Sep 13 05:44 ..

82 12 -rw——-. 1 oracle dba 5304 Sep 13 06:37 cwallet.sso

78  4 -rw——-. 1 oracle dba 2555 Sep 13 05:55 ewallet_2019091303553484.p12

80  4 -rw——-. 1 oracle dba 3803 Sep 13 05:57 ewallet_2019091303572233.p12

81  8 -rw——-. 1 oracle dba 5067 Sep 13 06:01 ewallet_2019091304014993.p12

77 12 -rw——-. 1 oracle dba 5259 Sep 13 06:01 ewallet.p12

  • And in Sqlplus  too you  will see  a change.

SQL> SELECT * FROM v$encryption_wallet;

WRL_TYPE

——————–

WRL_PARAMETER

——————————————————————————–

STATUS                                       WALLET_TYPE               WALLET_OR FULLY_BAC

—————————— ——————– ——— ———

    CON_ID

———-

FILE

/app/oracle/admin/WALLET/MYDB1/

OPEN                                           AUTOLOGIN                  SINGLE    NO

                 0

Appendix 1 Create an acfs file system:

Idea was to setup acfs via asmca. This however did not work in first implementation due to error because ora.proxy_advm is offline on some nodes.

It should be:

crsctl stat res ora.proxy_advm -t

——————————————————————————–

Name           Target  State        Server                   State details      

——————————————————————————–

Local Resources

——————————————————————————–

ora.proxy_advm

               ONLINE  ONLINE       Mysrv3dr                 STABLE

               ONLINE  ONLINE       Mysrv4dr                 STABLE

If the status is offline check if the acfsmodule are loaded: lsmod | grep acfs

oracleacfs           4840664  2

oracleoks             663240  2 oracleacfs,oracleadvm

if they are not loaded on the local node: log in as root and perform

$GRID_HOME/bin/csrctl stop crs

$GRID_HOME/bin/acfstool install

$GRID_HOME/bin/crsctl start crs

Modules should be loaded now. Volume can be created in asmca (high protection, TDE_VOL1, 1GB)

The volume will show up under /dev/asm (/dev/asm/tde_vol1-301 in this case).

Mkdir  -p /app/oracle/admin/wallet

Cd /app/oracle

Chown –R oracle:dba admin

/sbin/mkfs -t acfs /dev/asm/tde_vol1-302

Register in clusterware:

/sbin/acfsutil registry -a /dev/asm/tde_vol1-302 /app/oracle/admin/WALLET

When wallet is not open in TDE Environment (ORA-28365)

Introduction

For one of the new environments i have setup TDE ( transparent Database Encryption) aiming at encryption on the tablespaces level for a Rac database. As part of the preparations i have created a Diskgroup in ASM with normal redundancy and i have created an ACFS filesystem on top of that which will hold the TDE – Keys.

After restarting the instances ( tailing the alert log of both is an old hab it ). This is what i noticed:

In node of instance 2  I saw in alert that:

2019-09-13T09:36:34.512565+02:00

Errors in file /app/oracle/diag/diag/rdbms/MYDBU10a/MYDBU10A2/trace/MYDBU10A2_j001_354937.trc:

ORA-12012: error on auto execute of job 4

ORA-28365: wallet is not open

ORA-06512: at “SYS.ADMIN_AUDIT”, line 102

## On os level in the mount of the Acfs filesystem ( /app/oracle/admin/WALLET ) I Saw this :

total 84

 2  4 drwxrwxr-x. 5 oracle dba   4096 Sep 13 07:58 .

15  4 drwxr-xr-x. 8 oracle dba   4096 Sep 13 05:42 ..

44  4 -rw-r–r–. 1 oracle dba    100 Sep 13 07:58 afiedt.buf

21 64 drwx——. 2 root   root 65536 Sep 13 05:42 lost+found

 ?  ? d?????????? ? ?      ?        ?            ? MYDBU10A

##Followed below steps :

## as root

mysrvr4dr:root:/root $ /sbin/acfsutil registry

Mount Object:

  Device: /dev/asm/MYDBU10avol-125

  Mount Point: /app/oracle/admin/MYDBU10A

  Disk Group: ACFS

  Volume: MYDBU10AVOL

  Options: none

  Nodes: all

  Accelerator Volumes:

Mount Object:

  Device: /dev/asm/netcfg_vol-125

  Mount Point: /app/oracle/admin/NETCFG/12.2.0.1/db/network/admin

  Disk Group: ACFS

  Volume: NETCFG_VOL

  Options: norootsuid

  Nodes: all

  Accelerator Volumes:

Mount Object:

  Device: /dev/asm/tde_vol-125

  Mount Point: /app/oracle/admin/WALLET

  Disk Group: ACFS

  Volume: TDE_VOL

  Options: none

  Nodes: all

  Accelerator Volumes:

## as root :

mysrvr4dr:root:/root $ umount /app/oracle/admin/WALLET

umount: /app/oracle/admin/WALLET: target is busy.

        (In some cases useful info about processes that use

         the device is found by lsof(8) or fuser(1))

umount.acfs: CLSU-00100: operating system function: OfsWaitProc failed with error data: 32

umount.acfs: CLSU-00101: operating system error message: Broken pipe

umount.acfs: CLSU-00103: error location: OWPR_1

umount.acfs: ACFS-04151: unmount of mount point /app/oracle/admin/WALLET failed

## who is out there :

mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET

COMMAND    PID   USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME

ksh      82160 oracle  cwd    DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

ksh      82160 oracle   10r   DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

ksh     240476 oracle  cwd    DIR 252,64003     4096    2 /app/oracle/admin/WALLET

ksh     240476 oracle   11r   DIR 252,64003     4096    2 /app/oracle/admin/WALLET

## kill them all

mysrvr4dr:root:/root $ kill -9 240476

mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET

COMMAND   PID   USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME

ksh     82160 oracle  cwd    DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

ksh     82160 oracle   10r   DIR 252,64003     4096   74 /app/oracle/admin/WALLET/MYDBU10A (deleted)

mysrvr4dr:root:/root $ kill -9 82160

mysrvr4dr:root:/root $ lsof /app/oracle/admin/WALLET

## still root then do this:

mysrvr4dr:root:/root $ umount /app/oracle/admin/WALLET

# ah but a mount does not work since acfs mounts are not part of fstab.

mysrvr4dr:root:/root $ mount /app/oracle/admin/WALLET

mount: can’t find /app/oracle/admin/WALLET in /etc/fstab

mysrvr4dr:root:/root $ exit

logout

## as oracle

srvctl start filesystem -device /dev/asm/tde_vol-125 -node mysrvr4dr

oracle@mysrvr4dr:/home/oracle []# cd /app/oracle/admin

oracle@mysrvr4dr:/app/oracle/admin []# ls -lisa

total 32

    15 4 drwxr-xr-x.  8 oracle dba 4096 Sep 13 05:42 .

     2 4 drwxr-xr-x. 15 oracle dba 4096 Sep 12 11:36 ..

786508 4 drwxr-xr-x.  3 oracle dba 4096 Sep  9 09:25 +APX

    16 4 drwxr-xr-x.  3 oracle dba 4096 Sep  3 11:37 +ASM

     2 4 drwxrwxr-x. 19 oracle dba 4096 Sep 13 08:15 MYDBU10A

    21 0 lrwxrwxrwx.  1 oracle dba    8 Sep 12 09:43 MYDBU10A1 -> MYDBU10A

    20 0 lrwxrwxrwx.  1 oracle dba    8 Sep 12 09:43 MYDBU10A2 -> MYDBU10A

393302 4 drwx——.  4 oracle dba 4096 Sep  3 15:47 _mgmtdb

917505 4 drwxrwxr-x.  3 oracle dba 4096 Sep 13 05:21 NETCFG

     2 4 drwxrwxr-x.  5 oracle dba 4096 Sep 13 08:52 WALLET

Works again . all is well when it ends well .