通过Oracle数据库建立外部表以访问对象存储

简介

本文讲述如何配置Oracle数据库以建立ORACLE_BIGDATA类型的外部表。

实际上此项功能在自治数据库中已提供，依赖于DBMS_CLOUD PL/SQL包。
但对于On-Premise的数据库，按照How To Setup And Use DBMS_CLOUD Package (Doc ID 2748362.1)的说法，在19.9和21.3版本以后，已包含此包。本文讲述DBMS_CLOUD的安装和配置过程。

本文采用的数据库为19.11。

在整个实验中，我们使用的Schema用户为ssb。

参考文档

本文主要参考了以下两篇文档：

How To Setup And Use DBMS_CLOUD Package (Doc ID 2748362.1)
How to access Object Storage from your on-premises Oracle 19c Database
Big Data SQL Compatibility Matrix (Doc ID 2119369.1)

特别说明

在Doc ID 2748362.1中，虽然说明了有无Grid infrastructure的区别，但我在Grid环境下，也是用oracle用户执行的，也成功了：

Configure your Oracle environment to use the new SSL wallet
To have your SSL wallet taken into effect you need to point to the newly created ssl wallet for your Oracle installation by adding it to your sqlnet.ora on the Server side. If you are on a RAC installation then you have to adjust this on all nodes.

For cloud installations without Grid infrastructure, the default location of this file is $ORACLE_HOME/network/admin.
For cloud installations with Grid infrastructure, the default location of this file is $GRID_HOME/network/admin.
If you already had a wallet for SSL certificates and added the certificates to the existing one then this step is not necessary.

步骤

以下操作均以oracle用户。

先建立一个目录：

mkdir /home/oracle/log

使用schema用户建立directory，此directory用于创建外部表语句，用来存放日志。

conn ssb@orclpdb1
drop directory default_dir;
create directory default_dir as '/home/oracle/log';
drop table weather;
CREATE TABLE weather
  ( WEATHER_STATION_ID      VARCHAR2(20),
    WEATHER_STATION_NAME    VARCHAR2(100),
    REPORT_DATE             VARCHAR2(20),
    AVG_WIND                NUMBER,
    PRECIPITATION           NUMBER,
    SNOWFALL                NUMBER,
    SNOW_DEPTH              NUMBER,
    TEMP_AVG                NUMBER,
    TEMP_MAX                NUMBER,
    TEMP_MIN                NUMBER,
    WDF2                    NUMBER,
    WDF5                    NUMBER,
    WESD                    NUMBER,
    WESF                    NUMBER,
    WSF2                    NUMBER,
    WSF5                    NUMBER,
    FOG                     NUMBER,
    HEAVY_FOG               NUMBER(1),
    THUNDER                 NUMBER(1),
    SLEET                   NUMBER(1),
    HAIL                    NUMBER(1),
    GLAZE                   NUMBER(1),
    HAZE                    NUMBER(1),
    DRIFTING_SNOW           NUMBER(1),
    HIGH_WINDS              NUMBER(1)
  )
  ORGANIZATION EXTERNAL
  (TYPE ORACLE_BIGDATA
   DEFAULT DIRECTORY DEFAULT_DIR
   ACCESS PARAMETERS
   (
    com.oracle.bigdata.fileformat = textfile 
    com.oracle.bigdata.csv.skip.header=1
    com.oracle.bigdata.csv.rowformat.fields.terminator = '|'
   )
   location ('https://swiftobjectstorage.us-phoenix-1.oraclecloud.com/v1/adwc4pm/weather/*.csv')
  )  REJECT LIMIT UNLIMITED;

建表虽然成功，但查询出错。原因在于Oracle数据库默认不能与外部通讯。这也是我们需要配置DBMS_CLOUD的原因：

SQL> select count(*) from weather;
select count(*) from weather
*
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout

以下的操作就是按照MOS来做的。

安装DBMS_CLOUD

拷贝MOS中的脚本，在目录~/dbc下存为文件dbms_cloud_install.sql并执行：

mkdir ~/dbc
cd ~/dbc
export SYSPWD=********
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -u sys/$SYSPWD --force_pdb_mode 'READ WRITE' -b dbms_cloud_install -d /home/oracle/dbc -l /home/oracle/dbc dbms_cloud_install.sql

输出如下：

catcon::set_log_file_base_path: ALL catcon-related output will be written to [/home/oracle/dbc/dbms_cloud_install_catcon_80493.lst]

catcon::set_log_file_base_path: catcon: See [/home/oracle/dbc/dbms_cloud_install*.log] files for output generated by scripts

catcon::set_log_file_base_path: catcon: See [/home/oracle/dbc/dbms_cloud_install_*.lst] files for spool files, if any

catcon.pl: completed successfully

确认已安装成功，确认状态为VALID：

$ sqlplus / as sysdba
col owner for a20
col object_name for a20
set lines 120
select con_id, owner, object_name, status, sharing, oracle_maintained from cdb_objects where object_name = 'DBMS_CLOUD' order by con_id;

    CON_ID OWNER                OBJECT_NAME          STATUS  SHARING            O
---------- -------------------- -------------------- ------- ------------------ -
         1 PUBLIC               DBMS_CLOUD           VALID   METADATA LINK      Y
         1 C##CLOUD$SERVICE     DBMS_CLOUD           VALID   METADATA LINK      Y
         1 C##CLOUD$SERVICE     DBMS_CLOUD           VALID   METADATA LINK      Y
         3 PUBLIC               DBMS_CLOUD           VALID   METADATA LINK      Y
         3 C##CLOUD$SERVICE     DBMS_CLOUD           VALID   METADATA LINK      Y
         3 C##CLOUD$SERVICE     DBMS_CLOUD           VALID   METADATA LINK      Y

6 rows selected.

SQL> select owner, object_name, status, sharing, oracle_maintained from dba_objects where object_name = 'DBMS_CLOUD';

OWNER                OBJECT_NAME          STATUS  SHARING            O
-------------------- -------------------- ------- ------------------ -
PUBLIC               DBMS_CLOUD           VALID   METADATA LINK      Y
C##CLOUD$SERVICE     DBMS_CLOUD           VALID   METADATA LINK      Y
C##CLOUD$SERVICE     DBMS_CLOUD           VALID   METADATA LINK      Y

创建Wallet并导入证书

这个wallet是单独建立的，虽然你也可以利用已有的TDE wallet（如果有的话）。

下载证书：

$ wget https://objectstorage.us-phoenix-1.oraclecloud.com/p/QsLX1mx9A-vnjjohcC7TIK6aTDFXVKr0Uogc2DAN-Rd7j6AagsmMaQ3D3Ti4a9yU/n/adwcdemo/b/CERTS/o/dbc_certs.tar

$ tar -tvf dbc_certs.tar
-r-xr-xr-x hbaer/svrtech  1261 2020-10-23 21:22 BaltimoreCyberTrust.cer
-r-xr-xr-x hbaer/svrtech  1360 2020-10-23 21:22 DigiCert.cer
-r-xr-xr-x hbaer/svrtech  1760 2020-10-23 21:22 VeriSign.cer

$ tar -xvf dbc_certs.tar

创建wallet并添加证书：

mkdir -p  /opt/oracle/dcs/commonstore/wallets/ssl
cd /opt/oracle/dcs/commonstore/wallets/ssl
export WALLET_PWD=********
orapki wallet create -wallet . -pwd $WALLET_PWD -auto_login

# 执行下面3个语句
orapki wallet add -wallet . -trusted_cert -cert /home/oracle/dbc/VeriSign.cer -pwd $WALLET_PWD
orapki wallet add -wallet . -trusted_cert -cert /home/oracle/dbc/BaltimoreCyberTrust.cer -pwd $WALLET_PWD
orapki wallet add -wallet . -trusted_cert -cert /home/oracle/dbc/DigiCert.cer -pwd $WALLET_PWD

# 或者按照最新文档，执行以下循环
for i in $(ls /home/oracle/dbc/*.cer)
do
        orapki wallet add -wallet . -trusted_cert -cert $i -pwd $WALLET_PWD
done

确认添加成功：

$  orapki wallet display -wallet .
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Trusted Certificates:
Subject:        CN=VeriSign Class 3 Public Primary Certification Authority - G5,OU=(c) 2006 VeriSign\, Inc. - For authorized use only,OU=VeriSign Trust Network,O=VeriSign\, Inc.,C=US
Subject:        CN=Baltimore CyberTrust Root,OU=CyberTrust,O=Baltimore,C=IE
Subject:        CN=DigiCert Global Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US

配置Oracle数据库以使用刚创建的Wallet

其实就是在sqlnet.ora文件中添加条目，指定wallet的位置。

如果不知道sqlnet.ora所在目录，可以看下面命令的输出：

$ tnsping orclpdb1

...
Used parameter files:
/u01/app/oracle/product/19.0.0.0/dbhome_1/network/admin/sqlnet.ora
...

最终我的文件内容如下, 最后3行WALLET_LOCATION就是需要添加的：

$ cat /u01/app/oracle/product/19.0.0.0/dbhome_1/network/admin/sqlnet.ora
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))

SQLNET.ENCRYPTION_SERVER=REQUIRED
SQLNET.CRYPTO_CHECKSUM_SERVER=REQUIRED
SQLNET.ENCRYPTION_TYPES_SERVER=(AES256,AES192,AES128)
SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER=(SHA1)
SQLNET.ENCRYPTION_CLIENT=REQUIRED
SQLNET.CRYPTO_CHECKSUM_CLIENT=REQUIRED
SQLNET.ENCRYPTION_TYPES_CLIENT=(AES256,AES192,AES128)
SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT=(SHA1)

WALLET_LOCATION=
(SOURCE=(METHOD=FILE)(METHOD_DATA=
(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/ssl)))

配置ACE以让数据库可通过HTTP访问对象存储

将MOS中的命令拷贝为文件dbc_aces.sql，然后按你的实际环境修改文件中以下项目：

define sslwalletdir=/opt/oracle/dcs/commonstore/wallets/ssl

执行时（sqlplus / as sysdba执行即可）会提示输入Proxy信息，如果没有（我的就没有）就一路回车。

确认设置成功：

set lines 120
col PROPERTY_NAME for a12
col PROPERTY_VALUE for a50
col DESCRIPTION for a30
select * from database_properties where property_name in ('SSL_WALLET','HTTP_PROXY');

PROPERTY_NAM PROPERTY_VALUE                                     DESCRIPTION
------------ -------------------------------------------------- ------------------------------
SSL_WALLET   /opt/oracle/dcs/commonstore/wallets/ssl            Location of SSL Wallet

确认DBMS_CLOUD配置成功

如果之前建立的测试表可以访问成功，则本步骤可以跳过，下一步需要设置Schema用户的权限。例如：

SQL> show user
USER is "SYS"

SQL> alter session set container=orclpdb1;

Session altered.

SQL> select count(*) from ssb.weather;

  COUNT(*)
----------
       428

运行MOS中提供的测试语句，假设存为文件dbc_check.sql。此文件有3个地方要根据你的实际环境修改，分别是wallet所在目录，wallet口令，和需要访问的URL。

运行测试脚本（类似以下）。

define sslwalletdir=/opt/oracle/dcs/commonstore/wallets/ssl
define sslwalletpwd=*******
...
&clouduser..GET_PAGE('https://swiftobjectstorage.us-phoenix-1.oraclecloud.com/v1/adwc4pm/weather/weather-newark-airport.csv');
...

输出如果看到valid repsonse，就成功了：

[oracle@dbimsi08 dbc]$ sqlplus / as sysdba

SQL*Plus: Release 21.0.0.0.0 - Production on Tue Aug 23 10:09:27 2022
Version 21.6.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.


Connected to:
Oracle Database 21c EE Extreme Perf Release 21.0.0.0.0 - Production
Version 21.6.0.0.0

SQL> @@dbc_check.sql
old   1: CREATE OR REPLACE PROCEDURE &clouduser..GET_PAGE(url IN VARCHAR2) AS
new   1: CREATE OR REPLACE PROCEDURE C##CLOUD$SERVICE.GET_PAGE(url IN VARCHAR2) AS
old  13: wallet_path => 'file:&sslwalletdir',
new  13: wallet_path => 'file:/opt/oracle/dcs/commonstore/wallets/ssl',
old  14: wallet_password => '&sslwalletpwd');
new  14: wallet_password => 'ABcd123_#');

Procedure created.

old   2: &clouduser..GET_PAGE('https://objectstorage.eu-frankfurt-1.oraclecloud.com');
new   2: C##CLOUD$SERVICE.GET_PAGE('https://objectstorage.eu-frankfurt-1.oraclecloud.com');
valid response

PL/SQL procedure successfully completed.

old   1: drop procedure &clouduser..GET_PAGE
new   1: drop procedure C##CLOUD$SERVICE.GET_PAGE

Procedure dropped.

配置Schema用户可以访问DBMS_CLOUD

如果这个时候Schema用户去访问外部表，报错信息已经跟前面不一样，因为schema用户没有赋权：

SQL> show user
USER is "SSB"

SQL> select count(*) from weather;
select count(*) from weather
*
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-13014: HTTP GET request for
https://swiftobjectstorage.us-phoenix-1.oraclecloud.com/v1/adwc4pm/weather/?form
at=json&limit=10000&prefix= failed: Operation not authorized

因此需要赋权。赋权包括两部分：

赋予访问DBMS_CLOUD的权限
配置ACE（Access Control Entrie）

MOS中提供了user和role两种方法，我选的前者。注意，这两个脚本需在PDB中执行。

第一个脚本保存为文件dbc_user.sql。只需要修改以下行，然后运行：

define username='SSB'

第二个脚本保存为文件dbc_user_ace.sql。只需要修改以下行，然后运行：

define clouduser=SSB

define sslwalletdir=/opt/oracle/dcs/commonstore/wallets/ssl

由于我的URL是public的，因此就无需利用DBMS_CLOUD建立credential等配置了。

测试

然后现在访问外部表就成功了：

$ sqlplus ssb@orclpdb1

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Jul 9 11:31:42 2021
Version 19.11.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.

Last Successful login time: Fri Jul 09 2021 11:03:40 +00:00

Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.11.0.0.0

SQL> select count(*) from weather;

  COUNT(*)
----------
       428

后记

把笔记本的VagrantBox升级到19.11，然后执行了一遍，也成功了。

DBMS_CLOUD 外部表对象存储