RHEL 6.6 + Oracle 11.2.0.4 RAC instance意外terminated
dbdao吧
全部回复
仅看楼主
level 4
maming13 楼主
环境:RHEL 6.6 64bit + Oracle 11.2.0.4 RAC -2nodes
问题现象:RAC节点经常出现terminated by ospid(xxx)。
本次问题时间,下面主要记录出了OS的reboot时间。在reboot之前,已经将instance 终止了。
rac1和rac2的时间有约2min的不同步。
[root@rac1 log]# date
Thu Feb 18 10:48:53 CST 2016
[root@rac1 log]# last|more
root pts/1 10.1.111.188 Thu Feb 18 10:01 still logged in
root pts/0 10.1.111.50 Thu Feb 18 10:00 still logged in
reboot system boot 2.6.32-504.el6.x Tue Feb 16 22:19 - 10:49 (1+12:29)
[root@rac2 log]# date
Thu Feb 18 10:50:14 CST 2016
[root@rac2 log]# last
root pts/0 10.1.111.188 Thu Feb 18 10:02 still logged in
reboot system boot 2.6.32-504.el6.x Tue Feb 16 22:21 - 10:50 (1+12:29)
DB的相关log在云盘中,请Liu大和各位帮忙看看。谢谢!
链接: 百度网盘
密码: 8d2c
2016年02月18日 02点02分 1
level 13
2节点的 ocssd.log
[oracle@vrh8 ~]$ grep -i starting ocssd.log
2016-02-16 09:25:24.890: [ CSSD][42653440]clssscmain: Starting CSS daemon, version 11.2.0.4.0, in (clustered) mode with uniqueness value 1455585924
2016-02-16 09:25:29.301: [GIPCHTHR][4092032768] gipchaWorkerThread: starting worker thread hctx 0x2331990 [0000000000000010] { gipchaContext : host 'rac2', name 'CSS_rac-cluster', luid '3d28d119-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0xc062 }
2016-02-16 09:25:29.303: [GIPCHDEM][22771456] gipchaDaemonThread: starting daemon thread hctx 0x2331990 [0000000000000010] { gipchaContext : host 'rac2', name 'CSS_rac-cluster', luid '3d28d119-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0xc062 }
2016-02-16 09:25:49.685: [ CSSD][4087301888]clssscSAGEInitFencing: kgzf fence initialization starting with GUID 526c872b7db4ffa1bfec178f4e78c222, icin 350042526, node number 2, uniqueness 350042529
2016-02-16 22:17:00.376: [ CSSD][4088878848]clssnmDoSyncUpdate: Starting cluster reconfig with incarnation 350042530
2016-02-16 22:21:57.937: [ CSSD][1254967040]clssscmain: Starting CSS daemon, version 11.2.0.4.0, in (clustered) mode with uniqueness value 1455632517
2016-02-16 22:22:04.356: [GIPCHTHR][1005025024] gipchaWorkerThread: starting worker thread hctx 0xd1e990 [0000000000000010] { gipchaContext : host 'rac2', name 'CSS_rac-cluster', luid 'ac92a349-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0xc062 }
2016-02-16 22:22:04.358: [GIPCHDEM][1235085056] gipchaDaemonThread: starting daemon thread hctx 0xd1e990 [0000000000000010] { gipchaContext : host 'rac2', name 'CSS_rac-cluster', luid 'ac92a349-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0xc062 }
2016-02-16 22:22:22.753: [ CSSD][1209739008]clssscSAGEInitFencing: kgzf fence initialization starting with GUID 526c872b7db4ffa1bfec178f4e78c222, icin 351127240, node number 2, uniqueness 351127241
[oracle@vrh8 ~]$ grep -i manually ocssd.log
2016-02-15 12:09:06.962: [ CSSD][435689216]clssnmSendManualShut: Notifying all nodes that this node has been manually shut down
2016-02-16 22:17:00.377: [ CSSD][4088878848]clssnmRemove: Node 1, rac1 has been shutdown manually
2016-02-16 22:17:00.381: [ CSSD][26892032]clssnmSendManualShut: Notifying all nodes that this node has been manually shut down
核对下几个时间点
2016年02月18日 03点02分 2
1