处理因ASM实例异常导致RAC第一节点实例异常终止故障(精)
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
处理因ASM实例异常导致RAC第一节点实例异常终止故障
遭遇RAC第一节点实例由于ASM实例异常导致数据库实例非正常停止，记录在此。
1.故障现象
两节点RAC第一节点实例停止，经检查ASM实例亦异常终止。
2.故障分析
检查数据库实例及ASM实例的的alert寻找处理思路。
1）alert日志内容
Sun May 8 06:59:06 2011
Errors in file /oracle/app/oracle/admin/racdb/bdump/racdb1_asmb_21478.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Sun May 8 06:59:06 2011
ASMB: terminating instance due to error 15064
Sun May 8 06:59:06 2011
Errors in file /oracle/app/oracle/admin/racdb/bdump/racdb1_lms1_21275.trc:
ORA-15064: communication failure with ASM instance
Sun May 8 06:59:06 2011
Errors in file /oracle/app/oracle/admin/racdb/bdump/racdb1_lgwr_21283.trc:
ORA-15064: communication failure with ASM instance
Sun May 8 06:59:06 2011
Errors in file /oracle/app/oracle/admin/racdb/bdump/racdb1_lms0_21271.trc:
ORA-15064: communication failure with ASM instance
Sun May 8 06:59:06 2011
Errors in file /oracle/app/oracle/admin/racdb/bdump/racdb1_lmon_21267.trc: ORA-15064: communication failure with ASM instance
Sun May 8 06:59:06 2011
Errors in file /oracle/app/oracle/admin/racdb/bdump/racdb1_lmd0_21269.trc: ORA-15064: communication failure with ASM instance
Sun May 8 06:59:06 2011
System state dump is made for local instance
System State dumped to trace file
/oracle/app/oracle/admin/racdb/bdump/racdb1_diag_21263.trc
Sun May 8 06:59:06 2011
Errors in file /oracle/app/oracle/admin/racdb/bdump/racdb1_mman_21279.trc: ORA-15064: communication failure with ASM instance
Sun May 8 06:59:07 2011
Shutting down instance (abort
License high water mark = 7
Sun May 8 06:59:07 2011
Trace dumping is performing id=[cdmp_20110508065906]
Sun May 8 06:59:11 2011
Instance terminated by ASMB, pid = 21478
Sun May 8 06:59:12 2011
Instance terminated by USER, pid = 4110
Mon May 9 13:44:05 2011
2）trace文件中截取到如下故障内容
kjctseventdump-end tail 14 heads 0 @ 0 14 @ -1115894656
DEFER MSG QUEUE ON LMS1 IS EMPTY
SEQUENCES:
0:0.0 1:2933.0
error 15064 detected in background process
ORA-15064: communication failure with ASM instance
3）ASM日志中记录了如下内容
Thu Feb 10 19:17:58 2011
NOTE: cache recovered group 1 to fcn 0.20162635
Thu Feb 10 19:17:58 2011
NOTE: opening chunk 1 at fcn 0.20162635 ABA
NOTE: seq=79 blk=1597
Thu Feb 10 19:17:58 2011
NOTE: cache mounting group 1/0xBA97DAE1 (ORADATA succeeded SUCCESS: diskgroup ORADATA was mounted
Thu Feb 10 19:18:01 2011
NOTE: recovering COD for group 1/0xba97dae1 (ORADATA SUCCESS: completed COD recovery for group 1/0xba97dae1 (ORADATA Thu Feb 10 19:18:01 2011
Starting background process ASMB
ASMB started with pid=17, OS id=7767
Thu Feb 10 19:21:06 2011
NOTE: ASMB process exiting due to lack of ASM file activity
Sun May 8 06:48:33 2011
Shutting down instance (abort