Thursday, December 30, 2010

After suspend trigger to send out email alert to DBA about resumable space suspend

Environment
2-Nodes 11.2.0.2 Oracle RAC on OEL 5.5x64 bit platform.


Problem
During Datapump Import, when a tablespace runs out of space, the import job will be in "ORA-39171: Job is experiencing a resumable wait" state. By default, the database will not send out any alert to DBA about this problem unless DBA is actively monitoring this import job or check the alert log file. 


Solution
To address this problem and free DBAs from constantly monitoring the import job, a "AFTER SUSPEND System Event Trigger" will be created in the database, which will then email DBAs when a "resumable wait" problem happens in the database. 
1 Check system resumable_timeout setting
SQL> show parameters resumable_timeout


2 Set resumable_timeout value
If the above query shows resumable_timeout=0, change it to a desired seconds:
SQL> alter system set resumable_timeout=3600 scope=both;


Note: to understand the usage of this parameter, this is a good information page - Oracle Resumable Space Allocation

3 Enable UTL_MAIL package in order to use utl_mail to send out alert emails
3.1 created the UTL_MAIL package by running the following script under SYS schema
 SQL> @?/rdbms/admin/utlmail.sql
SQL> @?/rdbms/admin/prvtmail.plb

3.2 Grants the execute on UTL_MAIL privilege either to PUBLIC or to the user 
   which will use the package, running one of this statement from SYS:
SQL> GRANT EXECUTE ON UTL_MAIL TO PUBLIC;
     -or-
SQL> GRANT EXECUTE ON UTL_MAIL TO ;

Note: 
-- How to Use the UTL_MAIL Package [ID 269375.1]
-- How to Send Any Type of Attachment with UTL_MAIL [ID 946811.1]

3.3 Make sure the SMTP_OUT_SERVER parameter is defined in the database
SQL> show parameters smtp_out_server
Configure it if it's not set:
SQL>alter system set smtp_out_server=mail.hostname.com scope=both;

Note:
--UTL_MAIL Package


3.4 Test the utl_mail.send procedure to make sure it works

CREATE OR REPLACE PROCEDURE send_email AS
--DECLARE
   CRLF        CHAR(2) := CHR(10) || CHR(13);
BEGIN
   UTL_MAIL.SEND(
        sender => 'dba@email.com'
       ,recipients => 'dba@email.com'
       ,cc => NULL
       ,bcc =>  NULL
       ,subject => 'Test Email'
       ,message => 'Test line1' || CRLF || CRLF
       || 'Test line2'
       ,mime_type => 'text/plain; charset=us-ascii'
       ,priority => 1
   );


   EXCEPTION
   WHEN OTHERS THEN
    --  dbms_output.put_line('Fehler');
     raise_application_error(-20001,'The following error has occured: ' || sqlerrm);
   END;
   /
SHOW ERRORS


exec send_email;


4 After successful testing of the utl_mail.send procedure, create a "AFTER SUSPEND System Event Trigger" in the database:


CREATE OR REPLACE TRIGGER resumable_alert
AFTER SUSPEND
ON DATABASE
DECLARE
CRLF        CHAR(2) := CHR(10) || CHR(13);
BEGIN
   UTL_MAIL.SEND(
        sender => 'dba@email.com'
       ,recipients => 'dba@email.com'
       ,cc => NULL
       ,bcc =>  NULL
       ,subject => 'Space resumable alert found in the database'
       ,message => 'Space resumable alert found in the database' || CRLF || CRLF
       || ' Iusse --select user_id,session_id, status, start_time, suspend_time,resume_time, error_msg from dba_resumable--'
       ,mime_type => 'text/plain; charset=us-ascii'
       ,priority => 1
   );


   EXCEPTION
   WHEN OTHERS THEN
    --  dbms_output.put_line('Fehler');
     raise_application_error(-20001,'The following error has occured: ' || sqlerrm);
   END;
   /
SHOW ERRORS


Note:
--RESUMABLE SPACE ALLOCATION in Oracle 10g
--AFTER SUSPEND Trigger Operation-Suspended Alert
--A Simple Example On AFTER SUSPEND System Event Trigger [ID 258827.1]



After suspend trigger to send out email alert to DBA about resumable space suspend

Environment
2-Nodes 11.2.0.2 Oracle RAC on OEL 5.5x64 bit platform.


Problem
During Datapump Import, when a tablespace runs out of space, the import job will be in "ORA-39171: Job is experiencing a resumable wait" state. By default, the database will not send out any alert to DBA about this problem unless DBA is actively monitoring this import job or check the alert log file. 


Solution
To address this problem and free DBAs from constantly monitoring the import job, a "AFTER SUSPEND System Event Trigger" will be created in the database, which will then email DBAs when a "resumable wait" problem happens in the database. 
1 Check system resumable_timeout setting
SQL> show parameters resumable_timeout


2 Set resumable_timeout value
If the above query shows resumable_timeout=0, change it to a desired seconds:
SQL> alter system set resumable_timeout=3600 scope=both;


Note: to understand the usage of this parameter, this is a good information page - Oracle Resumable Space Allocation

3 Enable UTL_MAIL package in order to use utl_mail to send out alert emails
3.1 created the UTL_MAIL package by running the following script under SYS schema
 SQL> @?/rdbms/admin/utlmail.sql
SQL> @?/rdbms/admin/prvtmail.plb

3.2 Grants the execute on UTL_MAIL privilege either to PUBLIC or to the user 
   which will use the package, running one of this statement from SYS:
SQL> GRANT EXECUTE ON UTL_MAIL TO PUBLIC;
     -or-
SQL> GRANT EXECUTE ON UTL_MAIL TO ;

Note: 
-- How to Use the UTL_MAIL Package [ID 269375.1]
-- How to Send Any Type of Attachment with UTL_MAIL [ID 946811.1]

3.3 Make sure the SMTP_OUT_SERVER parameter is defined in the database
SQL> show parameters smtp_out_server
Configure it if it's not set:
SQL>alter system set smtp_out_server=mail.hostname.com scope=both;

Note:
--UTL_MAIL Package


3.4 Test the utl_mail.send procedure to make sure it works

CREATE OR REPLACE PROCEDURE send_email AS
--DECLARE
   CRLF        CHAR(2) := CHR(10) || CHR(13);
BEGIN
   UTL_MAIL.SEND(
        sender => 'yi.lin@wealthengine.com'
       ,recipients => 'yi.lin@wealthengine.com'
       ,cc => NULL
       ,bcc =>  NULL
       ,subject => 'Test Email'
       ,message => 'Test line1' || CRLF || CRLF
       || 'Test line2'
       ,mime_type => 'text/plain; charset=us-ascii'
       ,priority => 1
   );


   EXCEPTION
   WHEN OTHERS THEN
    --  dbms_output.put_line('Fehler');
     raise_application_error(-20001,'The following error has occured: ' || sqlerrm);
   END;
   /
SHOW ERRORS


exec send_email;


4 After successful testing of the utl_mail.send procedure, create a "AFTER SUSPEND System Event Trigger" in the database:


CREATE OR REPLACE TRIGGER resumable_alert
AFTER SUSPEND
ON DATABASE
DECLARE
CRLF        CHAR(2) := CHR(10) || CHR(13);
BEGIN
   UTL_MAIL.SEND(
        sender => 'sting@wealthengine.com'
       ,recipients => 'yi.lin@wealthengine.com'
       ,cc => NULL
       ,bcc =>  NULL
       ,subject => 'Space resumable alert found in the database'
       ,message => 'Space resumable alert found in the database' || CRLF || CRLF
       || ' Iusse

After suspend trigger to send out email alert to DBA about resumable space suspend

Environment
2-Nodes 11.2.0.2 Oracle RAC on OEL 5.5x64 bit platform.


Problem
During Datapump Import, when a tablespace runs out of space, the import job will be in "ORA-39171: Job is experiencing a resumable wait" state. By default, the database will not send out any alert to DBA about this problem unless DBA is actively monitoring this import job or check the alert log file. 


Solution
To address this problem and free DBAs from constantly monitoring the import job, a "AFTER SUSPEND System Event Trigger" will be created in the database, which will then email DBAs when a "resumable wait" problem happens in the database. 
1 Check system resumable_timeout setting
SQL> show parameters resumable_timeout


2 Set resumable_timeout value
If the above query shows resumable_timeout=0, change it to a desired seconds:
SQL> alter system set resumable_timeout=3600 scope=both;


Note: to understand the usage of this parameter, this is a good information page - Oracle Resumable Space Allocation

3 Enable UTL_MAIL package in order to use utl_mail to send out alert emails
3.1 created the UTL_MAIL package by running the following script under SYS schema
 SQL> @?/rdbms/admin/utlmail.sql
SQL> @?/rdbms/admin/prvtmail.plb

3.2 Grants the execute on UTL_MAIL privilege either to PUBLIC or to the user 
   which will use the package, running one of this statement from SYS:
SQL> GRANT EXECUTE ON UTL_MAIL TO PUBLIC;
     -or-
SQL> GRANT EXECUTE ON UTL_MAIL TO ;

Note: 
-- How to Use the UTL_MAIL Package [ID 269375.1]
-- How to Send Any Type of Attachment with UTL_MAIL [ID 946811.1]

3.3 Make sure the SMTP_OUT_SERVER parameter is defined in the database
SQL> show parameters smtp_out_server
Configure it if it's not set:
SQL>alter system set smtp_out_server=mail.hostname.com scope=both;

Note:
--UTL_MAIL Package


3.4 Test the utl_mail.send procedure to make sure it works

CREATE OR REPLACE PROCEDURE send_email AS
--DECLARE
   CRLF        CHAR(2) := CHR(10) || CHR(13);
BEGIN
   UTL_MAIL.SEND(
        sender => 'yi.lin@wealthengine.com'
       ,recipients => 'yi.lin@wealthengine.com'
       ,cc => NULL
       ,bcc =>  NULL
       ,subject => 'Test Email'
       ,message => 'Test line1' || CRLF || CRLF
       || 'Test line2'
       ,mime_type => 'text/plain; charset=us-ascii'
       ,priority => 1
   );


   EXCEPTION
   WHEN OTHERS THEN
    --  dbms_output.put_line('Fehler');
     raise_application_error(-20001,'The following error has occured: ' || sqlerrm);
   END;
   /
SHOW ERRORS


exec send_email;


4 After successful testing of the utl_mail.send procedure, create a "AFTER SUSPEND System Event Trigger" in the database:

CREATE OR REPLACE TRIGGER resumable_alert
AFTER SUSPEND
ON DATABASE
DECLARE
CRLF        CHAR(2) := CHR(10) || CHR(13);
BEGIN
   UTL_MAIL.SEND(
        sender => 'sting@wealthengine.com'
       ,recipients => 'yi.lin@wealthengine.com'
       ,cc => NULL
       ,bcc =>  NULL
       ,subject => 'Space resumable alert found in the database'
       ,message => 'Space resumable alert found in the database' || CRLF || CRLF
       || ' Iusse

Labels: , , , , ,

11gR2 DataPump Import hang during "Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX" stage

Environment
2-Nodes 11.2.0.2 Oracle RAC on OEL 5.5x64bit platform

Problem
I kicked off Datapump Import around 10 PM at night on 12/29/10, 12 hours later, the import log file shows that it's doing "Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX", the last time the log file was updated was at 23:31 on 12/29/10, so it means the import job has been waiting on this "Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX" for almost 11 hours.

Cause
1 Check v$session
select username,machine,module,sid,state,event,seq#,seconds_in_wait from v$session where type<>'BACKGROUND' and username='FPP'



Or
2 Query v$session_wait view

SELECT sid, event, seconds_in_wait 
FROM v$session_wait WHERE sid = $SID;


Or
3 Query v$system_event or v$session_event view

SELECT event, total_waits, time_waited FROM v$system_event WHERE event like '%suspend%';

select sid, event, total_waits, time_waited FROM v$session_event WHERE event like '%suspend%';

4 Check dba_resumable view to see when did the suspend happened
SQL> select user_id,session_id, status, start_time, suspend_time,resume_time, error_msg from dba_resumable;
So FPPI tablespace is out of space.

5 Check FPPI tablespace usage to check its status.
SQL> select tablespace_name, file_name, maxbytes/1024/1024, user_bytes/1024/1024 from dba_data_files where tablespace_name='FPPI';


So based on the above views, the FPPI tablespace is out of space, add more datafile to this tablespace would resume the Datapump Import job.


Solution
Add more datafiles to FPPI tablespace will resume the Datapump Import job.
Before adding more datafile:

SQL> alter tablespace fppi add datafile '/u08/oradata/racflo/FPPI04.dbf' size 1g autoextend on;


Tablespace altered.

After the above command:

Related Links

Resumable Space Management - (Oracle 9i) by Jeff Hunter


Labels: , , ,

Wednesday, December 29, 2010

Linux script to delete a type of file under multiple directories

Environment
OEL 5.5x64 bit

Problem
There are many old ".txt" files to be deleted under a download directory, which contains many sub-directories, like
$ cd /download
$ ls -d
20090601 20090602 xxx
20090701 20090702 xxx
xxxx

Solution
$ for i in `ls -d 20090[678]*`; do rm $i/*.txt; done

Labels: ,

Enable/Disable Archive Log Mode in 11gR2 RAC database

Environment
2-node 11.2.0.2 Oracle RAC on OEL 5.5x64bit platform

Problem
The RAC database was created with archive log mode enabled, which generated a lot of archive log files during data import.

Cause
Archive log mode generated many archive log files.

Solution
Our business requirements allow us to disable archive log mode in this 2-nodes Oracle RAC database.  We could re-enable it later.
Related links:



[oracle@node2~]$ more typescript

Script started on Tue 28 Dec 2010 03:49:44 PM EST
[oracle@node2~]$ sqlplus / as sysdba


SQL*Plus: Release 11.2.0.1.0 Production on Tue Dec 28 15:49:49 2010


Copyright (c) 1982, 2009, Oracle.  All rights reserved.




Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options


SQL> alter system set cluster_database=false scope=spfile sid='node2';


System altered.


SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
[oracle@node2~]$ ssh node1
srvctl stop database -d node -o immediate


[oracle@node2~]$ sqlplus / as sysdba


SQL*Plus: Release 11.2.0.1.0 Production on Tue Dec 28 15:51:14 2010


Copyright (c) 1982, 2009, Oracle.  All rights reserved.


Connected to an idle instance.


SQL> startup mount;
ORACLE instance started.


Total System Global Area 3.1467E+10 bytes
Fixed Size                  2241960 bytes
Variable Size            1.5234E+10 bytes
Database Buffers         1.6173E+10 bytes
Redo Buffers               57962496 bytes
Database mounted.
SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /u12/ora_arc
Oldest online log sequence     1677
Next log sequence to archive   1678
Current log sequence           1678




SQL> alter database noarchivelog;


Database altered.


SQL> alter system set cluster_database=true scope=spfile sid='node2';


System altered.


SQL> shutdown immediate;
ORA-01109: database not open




Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
[oracle@node2~]$ srvctl start database -d node
[oracle@node2~]$ sqlplus / as sysdba


SQL*Plus: Release 11.2.0.1.0 Production on Tue Dec 28 15:58:31 2010


Copyright (c) 1982, 2009, Oracle.  All rights reserved.




Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options


SQL> archive log list;
Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            /u12/ora_arc
Oldest online log sequence     1679
Current log sequence           1680
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options


Script done on Tue 28 Dec 2010 03:58:46 PM EST

Labels: , , ,

11gR2 DataPump Import Errors: ORA-31693/ORA-31640/ORA-19505/ORA-27037

Environment
2-nodes Oracle 11.2.0.2 RAC on OEL 5.5-64bit platform

Problem
Oracle DataPump Import failed with following errors:
impdp system/******** directory=expimp dumpfile=expdp_fpp_xxx2racflo_%U.dmp logfile=impdp_fpp_xxx2racflo-2010-12-28-1707.log job_name=impdp_xxx2racflo_fpp parallel=4


ORA-31693: Table data object "FPP"."FPP_WALT_MATCH" failed to load/unload and is being skipped due to error:
ORA-31640: unable to open dump file "/opt/DBfileLoad/expdp_fpp_blink2racflo_04.dmp" for read
ORA-19505: failed to identify file "/opt/DBfileLoad/expdp_fpp_blink2racflo_04.dmp"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3


Cause
According to Oracle Support/Metalink Note 1071373.1 - Cannot run DATAPUMP With PARALLEL > 1 ON 11.2 RAC  , "Datapump on 11.2 RAC with PARALLEL > 1 still hits the following errors though the unpublished Bug 8415620 is fixed into 11.2.",  and "From 11.2, Datapump new parameter CLUSTER is introduced". 


Solution
1
"To force Data Pump to use only the instance where the job is started and to replicate pre-Oracle Database 11g release 2 (11.2) behavior, specify CLUSTER=N."

Example:
$ impdp system/******** directory=expimp dumpfile=expdp_fpp_xxx2racflo_%U.dmp logfile=impdp_fpp_xxx2racflo-2010-12-28-1707.log job_name=impdp_xxx2racflo_fpp CLUSTER=N PARALLEL=4



2 Other recommended solutions by Oracle for database between 10.2.0.4 to 11.1.0.7: Impdp Fails with ORA-31693 ORA-29913 ORA-29400 KUP-11010 if Parallel > 1 is Used in RAC Database [ID 762475.1]


2.1. Run the Data Pump import job with parallel=1 (default)

OR

2.2. Run the Data Pump import job if only one instance is started.

Labels: , , ,

Wednesday, December 22, 2010

11.2.0.2 RAC: ora.gsd.type offline

In my 2-nodes 11.2.0.2 RAC database on OEL 5.5 platform, when I used the depreciated command "crs_stat -t" to check CRS status, I got following output:


$crs_stat -t|grep OFFLINE
ora.asm        ora.asm.type   OFFLINE   OFFLINE
ora....SM1.asm application    OFFLINE   OFFLINE
ora.cash.gsd   application    OFFLINE   OFFLINE
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....ry.acfs ora....fs.type OFFLINE   OFFLINE
ora....SM2.asm application    OFFLINE   OFFLINE
ora.sting.gsd  application    OFFLINE   OFFLINE


Since the RAC database is using NetApp as shared storage rather than ASM, it's expected that the ASM related CRS daemons to be offline, so here is a filtered result:

$ crs_stat -t|grep OFFLINE|grep gsd
ora.cash.gsd   application    OFFLINE   OFFLINE
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora.sting.gsd  application    OFFLINE   OFFLINE

According to this post and Metalink note GSD Is Used Only if 9i RAC Database is Present [ID 429966.1] and 1075945.1:
"The function of GSD (10g and above) is to service requests for 9i RAC management clients and therefore when there are no 9i databases present, there is nothing for GSD to do. Consequently, there will be no impact on a RAC cluster if GSD is offline and 9i is not used.
If gsd fails to start due to whetever reasons then best thing is to work with Oracle support to analyze and fix the issue. Until that time, gsd can be temporarily disabled.
In 11.2 GSD is disabled by default and the service will show as target:offline, status:offline."

Labels: , ,