如何知道外部表的数据文件是否在增加

问题描述

我有一个 PLsql 应用程序，该应用程序有一个预言机作业 (ABC.AUDIT_LOG_IMPORT_JOB)，它每 10 分钟运行一次，并从日志池中选取文件并使用预言机程序 (ABC.PR_IMPORT_AUDIT_LOG) 对其进行处理。>

有时会在日志池中导入非常大的文件（4 到 6 GB），当成功执行 oracle 作业（ABC.AUDIT_LOG_IMPORT_JOB）时，该过程需要一些时间来处理这些文件。该过程使用了一些外部表。

当程序启动时，会调用一些外部表来从文件中读取数据。由于文件很大，在程序已经启动时导入这些文件需要一些时间，从而导致以下错误。

KUP-05011: Size of file inProgress.txt in directory /opt/app/logpool/auditlog/ has changed from 3630039040 to 3852599296.

如何解决这个问题？

oracle 作业如下图

BEGIN
    DBMS_SCHEDULER.CREATE_JOB (
            job_name => 'ABC.AUDIT_LOG_IMPORT_JOB',job_type => 'STORED_PROCEDURE',job_action => 'ABC.PR_IMPORT_AUDIT_LOG',number_of_arguments => 0,start_date => NULL,repeat_interval => 'FREQ=MINUTELY;INTERVAL=10',end_date => NULL,enabled => FALSE,auto_drop => FALSE,comments => '');

         
     
 
    DBMS_SCHEDULER.SET_ATTRIBUTE( 
             name => 'ABC.AUDIT_LOG_IMPORT_JOB',attribute => 'store_output',value => TRUE);
    DBMS_SCHEDULER.SET_ATTRIBUTE( 
             name => 'ABC.AUDIT_LOG_IMPORT_JOB',attribute => 'logging_level',value => DBMS_SCHEDULER.LOGGING_OFF);
      
   
  
    
    DBMS_SCHEDULER.enable(
             name => 'ABC.AUDIT_LOG_IMPORT_JOB');
END;

使用这个外部表的过程如下图

--------------------------------------------------------
--  DDL for Procedure PR_IMPORT_AUDIT_LOG
--------------------------------------------------------
set define off;

  CREATE OR REPLACE EDITIONABLE PROCEDURE "ABC"."PR_IMPORT_AUDIT_LOG" 
  AS

   CURSOR CUR_FILE_LIST
   IS SELECT FNAME
      FROM EXT_TAB_IMPORT_FILE_LIST 
      WHERE REGEXP_LIKE(FNAME,'AL_\d{2}-\d{2}-\d{2}_\d{8}_\d{6}.txt');

   

   V_INSERT_TIMESTAMP   TIMESTAMP := SYSTIMESTAMP;
BEGIN
    -- This is to prevent the logs from being created in other than English
    EXECUTE IMMEDIATE 'ALTER SESSION SET NLS_LANGUAGE = ''AMERICAN''';

   
    
    FOR REC_FILE_LIST IN CUR_FILE_LIST
    LOOP
      -- copy the original file to the AL_IMPORTBAD directory
    UTL_FILE.FcopY('ABC_AUDITLOG',FNAME,'ABC_AL_IMPORTBAD',FNAME||'-Original_file');

      --RENAME FILE IN inProgress.txt
      UTL_FILE.FRENAME('ABC_AUDITLOG',REC_FILE_LIST.FNAME,'ABC_AUDITLOG','inProgress.txt',TRUE);  
      
      IF PR_AL_IMPORT_CHECK_CORIAID (REC_FILE_LIST.FNAME) --If there are any errors,then there are no records with correct records are saved in the AL_IMPORT folder.
      THEN
      
         --Insert into audit_log
         PR_INSERT_AUDIT_LOG_FROM_FILE(V_INSERT_TIMESTAMP);--Records are inserted from file with correct coridid and during insert just oracle regexp are checked.
         --Check log file
         PR_CHECK_AUDIT_LOG_FILE(REC_FILE_LIST.FNAME);
         --ERROR HANDLING
         PR_IMPORT_CHECK_LOG_FILES( REC_FILE_LIST.FNAME,V_INSERT_TIMESTAMP);

         --move inProgress into archive and rename it with the old name
         UTL_FILE.FRENAME('ABC_AUDITLOG','ABC_AL_IMPORTLOG',TRUE);
         
      ELSE
         --move inProgress into archive and rename it with the old name
--          UTL_FILE.FRENAME('ABC_AUDITLOG',TRUE);
      END IF;     
      
      --remove inProgress.bad
      BEGIN
         UTL_FILE.FREMOVE('ABC_AUDITLOG','inProgress.bad');
      EXCEPTION
         WHEN OTHERS
         THEN
            IF sqlCODE = -29283 -- suppresses no file found exception
            THEN
               NULL;
            ELSE
               RAISE;
            END IF;
      END;         
      --remove inProgress.log      
      BEGIN
         UTL_FILE.FREMOVE('ABC_AUDITLOG','inProgress.log');
      EXCEPTION
         WHEN OTHERS
         THEN
            IF sqlCODE = -29283 -- suppresses no file found exception
            THEN
               NULL;
            ELSE
               RAISE;
            END IF;
      END;

       --remove inProgress.txt
      BEGIN
         UTL_FILE.FREMOVE('ABC_AUDITLOG','inProgress.txt');
      EXCEPTION
         WHEN OTHERS
         THEN
            IF sqlCODE = -29283 -- suppresses no file found exception
            THEN
               NULL;
            ELSE
               RAISE;
            END IF;
      END;
      
      COMMIT;
      
    END LOOP;

EXCEPTION
   WHEN OTHERS
   THEN
      DECLARE
         V_PROCEDURE_NAME  VARCHAR2(30)  := $$PLsql_UNIT;
         V_SUBJECT         VARCHAR2(255) := 'Error on DB procedure '||V_PROCEDURE_NAME||' on '||to_char(sysdate,'dd.mm.yyyy hh24:mi:ss');
         V_BODY            VARCHAR2(10000) := 'Hi all,'||chr(10)||chr(10)||'Procedure '||V_PROCEDURE_NAME||' returned the following error:'||chr(10)||sqlERRM;
      BEGIN
         SEND_MAIL ( 'GENERIC_DB_ERROR',V_SUBJECT,V_BODY);
         RAISE;
      END;      
END;

/

从文件中读取记录的外部表如下

--------------------------------------------------------
--  DDL for Procedure PR_INSERT_AUDIT_LOG_FROM_FILE
--------------------------------------------------------
set define off;

  CREATE OR REPLACE EDITIONABLE PROCEDURE "ABC"."PR_INSERT_AUDIT_LOG_FROM_FILE" (P_IMPORT_TIMESTAMP IN TIMESTAMP)
AS 
BEGIN

   INSERT INTO AUDIT_LOG 
      ( ID,PRIMARY_NAME,PRIMARY_VALUE,TERMINAL_DATE,LAND_ID           
      )
   SELECT ID,TRIM(PRIMARY_NAME),TRIM(PRIMARY_VALUE),LAND_ID,P_IMPORT_TIMESTAMP
   FROM EXT_TAB_AUDIT_LOG AL
   LEFT OUTER JOIN VERFIY_NAME VWZ ON VWZ.VERFIY_TABLE_ID = TRIM(AL.VERFIY_TABLE_ID)
   WHERE REGEXP_LIKE (ID,'\d{2}-\d{2}-\d{2}')
   AND   PRIMARY_NAME IS NOT NULL
   AND   PRIMARY_VALUE IS NOT NULL
   AND   TERMINAL_DATE IS NOT NULL
   AND   LAND_ID IS NOT NULL
 


   COMMIT;

EXCEPTION
   WHEN OTHERS
   THEN
      DECLARE
         V_PROCEDURE_NAME  VARCHAR2(30)  := $$PLsql_UNIT;
         V_SUBJECT         VARCHAR2(255) := 'Error on DB procedure '||V_PROCEDURE_NAME||' on '||to_char(sysdate,V_BODY);
         RAISE;
      END;  
END PR_INSERT_AUDIT_LOG_FROM_FILE;

/

解决方法

根据我的拙见，在有限内存的桌面主机上经常使用 Oracle-SQL*Loader；

您可以根据行数或字节数将大文件或小文件预处理为相同大小的块，

使用 Shell 脚本并在这些脚本中嵌入 PLSQL/SQL*LOADER。