主机:HP-UX essaop1 B.11.31 U ia64 1945507590 unlimited-user license
数据库:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
该系统主机数据库监听服务器,启停不超过5分钟就会自动重启。业务短暂的受到影响。重启过操作系统网卡APA服务后,3个月又开始频繁重启。
vi /oracle/grid/11.2.0/log/essaop2/agent/crsd/orarootagent_root/orarootagent_root.log
2015-08-13 09:42:09.500: [ AGFW][10] {2:56751:2} Agent received the message: AGENT_HB[Engine] ID 12293:2614441
2015-08-13 09:42:12.618: [ default][271023]ICMP Ping from xxx.xxx.89.3 to xx.xx.89.62
2015-08-13 09:42:12.638: [ default][271023]clsicmp_pingdecode recvd other process's packet
2015-08-13 09:42:12.638: [ora.net1.network][271023] {0:2:9230} [check] NetworkAgent::checkLink returned false
2015-08-13 09:42:12.640: [ AGFW][10] {0:2:9230} ora.net1.network essaop2 1 state changed from: ONLINE to: OFFLINE
2015-08-13 09:42:12.640: [ AGFW][10] {0:2:9230} Switching online monitor to offline one
2015-08-13 09:42:12.641: [ AGFW][10] {0:2:9230} Started implicit monitor for [ora.net1.network essaop2 1] interval=60000 delay=60000
meatlink Bug 16039587 。
HP UNIX下需要操作系统打个补丁:
Cause
The issue was investigated in Bug 16039587, the cause is HP-UX bug, basically the contention of address memory range lock on kernel memory causes poll(2) timeout and affects orarootagent process.
Solution
Apply OS kernel patch PHKL_42850.
找到相关补丁,进行了工单审批,到实施完成。AOP数据库监听闪断重大隐患已经解决。
STARTED_AT UPTIME
--------------------------------------------------------------------------------
18-SEP-2015 23:35:29 2 day(s), 10 hour(s), 2 minute(s), 54 seconds 数据库启动时间
sql> !uptime
9:38am up 2 days, 11:15, 1 user, load average: 0.03, 0.03, 0.03 主机启动时间
sql> !lsnrctl status 数据库监听启动时间
LSNRCTL for HPUX: Version 11.2.0.3.0 - Production on 21-SEP-2015 09:50:45
copyright (c) 1991, 2011, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for HPUX: Version 11.2.0.3.0 - Production
Start Date 18-SEP-2015 23:34:01
Uptime 2 days 10 hr. 16 min. 44 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /oracle/grid/11.2.0/network/admin/listener.ora
Listener Log File /oracle/grid/base/diag/tnslsnr/essaop2/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=132.35.89.3)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=132.35.89.4)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "essaop" has 1 instance(s).
Instance "essaop2", status READY, has 1 handler(s) for this service...
Service "essaopXDB" has 1 instance(s).
Instance "essaop2", status READY, has 1 handler(s) for this service...
The command completed successfully