Quantcast
Channel: High Availability (Clustering) forum
Viewing all 5648 articles
Browse latest View live

Unable to bring cluster group online

0
0

Hi,

I have a Windows 2008 2 node cluster.

We have SQL on node 1.  The cluster group is on node 2 right now.

Over the weekend, the cluster group failed and will not come back online. We were unable to connect via cluster manager through the name.  Able to connect on the server, but only see cluster events and cluster status 'down'.  The cluster service on this node is on.  Here is cluster GROUP output:

Cluster IP Address - Offline

Cluster Name - Offline

Quorum - Failed

When attempting to online the group we receive:

System error 5908 has occurred (0x00001714).
The group is unable to accept the request since it is moving to another node.

The cluster log contains many instances of this error:

00000b30.00001b7c::2014/08/18-16:31:07.502 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_CALL_TIMED_OUT(5910)' because of 'Failed to wait for pending resource control call to Quorum.

The Q: drive is active on node 2 and I am able to browse and write files to it.

Here is the last 10 minutes of cluster log.  I tried to online the group during that time.  Thanks for your review.

-------------------------------------------------------------------------------------------------------------

00000b30.00002728::2014/08/18-16:24:59.737 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_CALL_TIMED_OUT(5910)' because of 'Failed to wait for pending resource control call to Quorum.

'
00000b30.00002728::2014/08/18-16:24:59.737 WARN  [RCM] ResourceControl(GET_COMMON_PROPERTIES) to Quorum returned 5910.
00000b30.0000138c::2014/08/18-16:25:17.883 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_CALL_TIMED_OUT(5910)' because of 'Failed to wait for pending resource control call to Quorum.

'
00000b30.0000138c::2014/08/18-16:25:17.883 WARN  [RCM] ResourceControl(STORAGE_GET_DISK_INFO_EX) to Quorum returned 5910.
000012c4.000018c0::2014/08/18-16:26:07.689 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
00000b30.000022f8::2014/08/18-16:26:07.689 INFO  [RCM] rcm::RcmResType::LoadDll: Got error 126; will attempt to load mqclus.dll via Wow64.
00001330.00001968::2014/08/18-16:26:07.689 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
00000b30.000022f8::2014/08/18-16:26:07.689 WARN  [RCM] Failed to load restype MSMQ: error 126.
000012c4.000018c0::2014/08/18-16:26:07.689 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
00000b30.000022f8::2014/08/18-16:26:07.689 INFO  [RCM] rcm::RcmResType::LoadDll: Got error 126; will attempt to load mqtgclus.dll via Wow64.
00001330.00001968::2014/08/18-16:26:07.689 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
00000b30.000022f8::2014/08/18-16:26:07.689 WARN  [RCM] Failed to load restype MSMQTriggers: error 126.
00000b30.00000f74::2014/08/18-16:26:25.305 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_CALL_TIMED_OUT(5910)' because of 'Failed to wait for pending resource control call to Quorum.

'
00000b30.00000f74::2014/08/18-16:26:25.305 WARN  [RCM] ResourceControl(GET_CLASS_INFO) to Quorum returned 5910.
00000b30.000006dc::2014/08/18-16:26:41.844 INFO  [RCM] rcm::RcmApi::OnlineGroup: (Cluster Group)
00000b30.000006dc::2014/08/18-16:26:41.844 INFO  [GUM] Node 2: Processing RequestLock 2:207
00000b30.00002618::2014/08/18-16:26:41.922 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285421)
00000b30.000006dc::2014/08/18-16:26:41.922 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:26:42.031 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:26:43.046 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:26:43.046 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:26:44.060 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:26:44.060 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:26:45.074 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:26:45.074 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:26:46.088 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:26:46.088 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:26:47.102 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:26:47.102 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:26:48.117 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:26:48.132 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:26:49.146 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:26:49.146 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:26:50.161 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:26:50.176 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:26:51.190 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:26:51.206 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00002618::2014/08/18-16:26:54.295 INFO  [GUM] Node 2: Processing RequestLock 1:22512
00000b30.00002618::2014/08/18-16:26:54.295 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285431)
00000b30.000006dc::2014/08/18-16:27:01.223 INFO  [GUM] Node 2: Processing RequestLock 2:217
00000b30.00002618::2014/08/18-16:27:01.223 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285432)
00000b30.000006dc::2014/08/18-16:27:01.223 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:27:01.223 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:27:11.241 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:27:11.241 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:27:21.258 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:27:21.289 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00002618::2014/08/18-16:27:23.614 INFO  [GUM] Node 2: Processing RequestLock 1:22513
00000b30.00002618::2014/08/18-16:27:23.614 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285435)
00000b30.000006dc::2014/08/18-16:27:31.306 INFO  [GUM] Node 2: Processing RequestLock 2:220
00000b30.00002618::2014/08/18-16:27:31.306 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285436)
00000b30.000006dc::2014/08/18-16:27:31.306 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:27:31.306 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:27:41.323 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:27:41.355 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:27:51.372 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:27:51.372 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00002618::2014/08/18-16:27:54.305 INFO  [GUM] Node 2: Processing RequestLock 1:22514
00000b30.00002618::2014/08/18-16:27:54.305 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285439)
00000b30.000006dc::2014/08/18-16:28:01.389 INFO  [GUM] Node 2: Processing RequestLock 2:223
00000b30.00002618::2014/08/18-16:28:01.389 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285440)
00000b30.000006dc::2014/08/18-16:28:01.389 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:28:01.389 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:28:11.406 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:28:11.422 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:28:21.439 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:28:21.455 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00002618::2014/08/18-16:28:23.624 INFO  [GUM] Node 2: Processing RequestLock 1:22515
00000b30.00002618::2014/08/18-16:28:23.624 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285443)
00000b30.000006dc::2014/08/18-16:28:31.472 INFO  [GUM] Node 2: Processing RequestLock 2:226
00000b30.00002618::2014/08/18-16:28:31.472 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285444)
00000b30.000006dc::2014/08/18-16:28:31.472 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:28:31.488 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:28:41.504 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:28:41.504 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:28:51.520 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:28:51.520 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00002618::2014/08/18-16:28:54.328 INFO  [GUM] Node 2: Processing RequestLock 1:22516
00000b30.00002618::2014/08/18-16:28:54.328 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285447)
00000b30.000006dc::2014/08/18-16:29:01.536 INFO  [GUM] Node 2: Processing RequestLock 2:229
00000b30.00002618::2014/08/18-16:29:01.536 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285448)
00000b30.000006dc::2014/08/18-16:29:01.536 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:29:01.552 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:29:11.568 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:29:11.568 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:29:21.584 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:29:21.600 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00002618::2014/08/18-16:29:23.643 INFO  [GUM] Node 2: Processing RequestLock 1:22517
00000b30.00002618::2014/08/18-16:29:23.643 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285451)
00000b30.000006dc::2014/08/18-16:29:31.616 INFO  [GUM] Node 2: Processing RequestLock 2:232
00000b30.00002618::2014/08/18-16:29:31.616 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285452)
00000b30.000006dc::2014/08/18-16:29:31.616 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:29:31.616 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:29:41.632 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:29:41.632 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:29:51.648 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:29:51.695 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00002618::2014/08/18-16:29:54.331 INFO  [GUM] Node 2: Processing RequestLock 1:22518
00000b30.00002618::2014/08/18-16:29:54.331 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285455)
00000b30.000006dc::2014/08/18-16:30:01.711 INFO  [GUM] Node 2: Processing RequestLock 2:235
00000b30.00002618::2014/08/18-16:30:01.711 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285456)
00000b30.000006dc::2014/08/18-16:30:01.711 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:30:01.711 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:30:11.727 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:30:11.742 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.0000138c::2014/08/18-16:30:17.889 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_CALL_TIMED_OUT(5910)' because of 'Failed to wait for pending resource control call to Quorum.

'
00000b30.0000138c::2014/08/18-16:30:17.889 WARN  [RCM] ResourceControl(GET_COMMON_PROPERTIES) to Quorum returned 5910.
00000b30.00001fdc::2014/08/18-16:30:17.905 INFO  [NM] Received request from client address 10.12.13.8.
00000b30.00000f74::2014/08/18-16:30:17.905 INFO  [NM] Received request from client address 10.12.13.8.
00000b30.0000115c::2014/08/18-16:30:17.952 INFO  [NM] Received request from client address 10.12.13.8.
000012c4.000018c0::2014/08/18-16:30:18.326 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
00000b30.00000f74::2014/08/18-16:30:18.326 INFO  [RCM] rcm::RcmResType::LoadDll: Got error 126; will attempt to load mqclus.dll via Wow64.
00001330.00000a10::2014/08/18-16:30:18.326 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
00000b30.00000f74::2014/08/18-16:30:18.326 WARN  [RCM] Failed to load restype MSMQ: error 126.
00000b30.00000f74::2014/08/18-16:30:18.326 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
000012c4.000018c0::2014/08/18-16:30:18.326 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
00000b30.00002120::2014/08/18-16:30:18.326 INFO  [RCM] rcm::RcmResType::LoadDll: Got error 126; will attempt to load mqclus.dll via Wow64.
00001330.00000a10::2014/08/18-16:30:18.326 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
00000b30.00002120::2014/08/18-16:30:18.326 WARN  [RCM] Failed to load restype MSMQ: error 126.
000012c4.000018c0::2014/08/18-16:30:18.326 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
00000b30.00000f74::2014/08/18-16:30:18.326 INFO  [RCM] rcm::RcmResType::LoadDll: Got error 126; will attempt to load mqtgclus.dll via Wow64.
00001330.00000a10::2014/08/18-16:30:18.326 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
00000b30.00000f74::2014/08/18-16:30:18.326 WARN  [RCM] Failed to load restype MSMQTriggers: error 126.
00000b30.00000f74::2014/08/18-16:30:18.326 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
000012c4.000018c0::2014/08/18-16:30:18.326 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
00000b30.00002120::2014/08/18-16:30:18.326 INFO  [RCM] rcm::RcmResType::LoadDll: Got error 126; will attempt to load mqtgclus.dll via Wow64.
00001330.00000a10::2014/08/18-16:30:18.326 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
00000b30.00002120::2014/08/18-16:30:18.326 WARN  [RCM] Failed to load restype MSMQTriggers: error 126.
00000b30.000006dc::2014/08/18-16:30:21.759 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:30:21.759 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00002618::2014/08/18-16:30:23.646 INFO  [GUM] Node 2: Processing RequestLock 1:22519
00000b30.00002618::2014/08/18-16:30:23.646 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285459)
00000b30.000006dc::2014/08/18-16:30:31.775 INFO  [GUM] Node 2: Processing RequestLock 2:238
00000b30.00002618::2014/08/18-16:30:31.775 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285460)
00000b30.000006dc::2014/08/18-16:30:31.775 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:30:31.775 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000021dc::2014/08/18-16:30:33.974 INFO  [NM] Received request from client address 10.12.13.8.
00000b30.000006dc::2014/08/18-16:30:41.791 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:30:41.822 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:30:51.838 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:30:51.838 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00002618::2014/08/18-16:30:54.334 INFO  [GUM] Node 2: Processing RequestLock 1:22520
00000b30.00002618::2014/08/18-16:30:54.334 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285463)
00000b30.000006dc::2014/08/18-16:31:01.854 INFO  [GUM] Node 2: Processing RequestLock 2:241
00000b30.00002618::2014/08/18-16:31:01.854 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285464)
00000b30.000006dc::2014/08/18-16:31:01.854 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:31:01.870 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00001b7c::2014/08/18-16:31:07.502 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_CALL_TIMED_OUT(5910)' because of 'Failed to wait for pending resource control call to Quorum.

'
00000b30.00001b7c::2014/08/18-16:31:07.502 WARN  [RCM] ResourceControl(STORAGE_GET_DISK_INFO_EX) to Quorum returned 5910.
00000b30.000019f0::2014/08/18-16:31:07.861 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_CALL_TIMED_OUT(5910)' because of 'Failed to wait for pending resource control call to Quorum.

'
00000b30.000019f0::2014/08/18-16:31:07.861 WARN  [RCM] ResourceControl(GET_COMMON_PROPERTIES) to Quorum returned 5910.
00000b30.000006dc::2014/08/18-16:31:11.886 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:31:11.886 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000006dc::2014/08/18-16:31:21.902 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:31:21.902 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.00002618::2014/08/18-16:31:23.665 INFO  [GUM] Node 2: Processing RequestLock 1:22521
00000b30.00002618::2014/08/18-16:31:23.665 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285467)
00000b30.000006dc::2014/08/18-16:31:31.918 INFO  [GUM] Node 2: Processing RequestLock 2:244
00000b30.00002618::2014/08/18-16:31:31.918 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 285468)
00000b30.000006dc::2014/08/18-16:31:31.918 INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(Cluster Group,1)
00000b30.000006dc::2014/08/18-16:31:31.949 WARN  [RCM] rcm::RcmApi::OnlineGroup: retrying: Cluster Group, 5908.
00000b30.000022f8::2014/08/18-16:31:46.427 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_CALL_TIMED_OUT(5910)' because of 'Failed to wait for pending resource control call to Quorum.

'
00000b30.000022f8::2014/08/18-16:31:46.427 WARN  [RCM] ResourceControl(STORAGE_GET_DISK_INFO_EX) to Quorum returned 5910.
00000b30.00002618::2014/08/18-16:31:54.337 INFO  [GUM] Node 2: Processing RequestLock 1:22522
00000b30.00002618::2014/08/18-16:31:54.337 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 285469)



Cluster Networking - Minimal configuration

0
0

I am looking for the best way to configure our 3 node Hyper-V cluster.  The cluster nodes have 2x 1Gbit NICs and 2x 10Gbit nics.  The 2 10Gbit nics are conifgured for iSCSI, so cluster validation warns that they are disabled:
These paths will not be used for cluster communication and will be ignored. This is because interfaces on these networks are connected to an iSCSI target

This made me start looking at the best way to use my 4 network adapters.

My current configuration is:

1Gig1 -Mgmt and Cluster  (10.1.2.0/24 subnet)

1Gig2 -Hyper-V switch for guest VMs (no IP defined on host)

10Gig1 -iSCSI  (10.2.21.0/24 subnet, no gateway)

10Gig2-iSCSI  (10.2.22.0/24 subnet, no gateway)

In Failover cluster manager, I have "Cluster and Client" for the 3 networks that are visible to the host machine (but cluster validation now tells me that the 2 iSCSI adapters can't be used)

A few months ago we had several issues with NIC teaming (errors about MAC addresses and a few BSOD crashes that WinDBG pointed to NIC teaming as the cause), so we moved away from using it.  Not sure if the issues have been resolved.

Is there anything "wrong" with the way it is currently set up?  Is there a better way to set it up using the 4 network cards and still keep things pretty simple?


James Right Size Solutions

2008R2 Hyper-V HA Cluster Issue

0
0

Hi,

Current setup is two DL380 G7 servers, currently in a HA Hyper-V Cluster, storage is an iSCSI HP G4300 G2 SAN. The Hyper-V traffic runs through an HP Team (same on both hosts), this team crashes every 5-28 days, I believe it’s the traffic through the team and not the cluster itself, usually only on one host but has happened on both, which means the VM’s crash, migrate to the other host, then they boot back up.  The event logs when the crash happen all start with a mixture of CPQTeamMP Event ID 435, 388, HP Teaming Software is at the latest software (10.90.0.0) and the latest firmware and drivers for the NIC's are installed from the latest HP SPP (2014.6.0).  This has happened for a good year or so and now it is very frustrating.

I found an article, where it looks like I need to make sure Hyper-V is installed first, then install and setup teaming, which I can’t remember if I did or not when I setup the hosts.

My question is, after I uninstall the teaming, if I uninstall Hyper-V, will this affect the host itself in the cluster and how the host sees the storage? Will I have to setup and configure all that again as well, or can I just uninstall teaming, Hyper-V, then install Hyper-V, then setup the teaming and this won't have an impact on storage or the cluster?

Thanks

Mark

Missing disk on a Hyperv cluster

0
0
HyperV2012 cluster with two nodes. HP servers with 3PAR disk system. 
When I move and delete large disks, 2TB, losing the server I'm working on contact with all SAN disk's. Must take a restart of the disks that will come online again. Sounds like a bug. either?

CAU: The plug-in argument HotfixRootFolderPath has invalid value

0
0

I am trying to apply a hotfix using CAU and have configured the self-updating options.  When I come to preview the update for the cluster and select Microsoft.Hotfixplugin as the plugin, I get the following error:

I am unsure why this is being generate or how to correct the problem.

Any ideas?

How to create a local, non-clustered storage pool

0
0

Hello,

I have setup a two-node Failover Cluster, with a shared SAS DAS. So far so good.

One of the nodes also has internal disks that I wish to use for system backups.

This storage pool should not be clustered, as the disks cannot be seen from the other node. The trouble is that as soon as I create the pool it gets added to the cluster (in failed state).

In fact, the "Storage Pools" window in the server manager will only show me the "clustered storage spaces", with my internal disks in the Primordial pool.

Get-StorageSubSystem will show me both subsystems (Clustered Storage Space on ... + Storage Spaces on node-1) but fails to create a storage pool on the "local" subsystem.

How can I create a local, non clustered storage pool on internal disks ?

Cheers

alex

Failover VM's Servers all grouping together under one server?

0
0

When I add VM server's to high availability using the wizard in the failover cluster manager. They all add under the first server that I add instead off in the top section individually as roles ( See Screenshot ) Ignore partially running, one VM is currently in an off state.

Screenshot
http://i57.tinypic.com/2wgt6p0.jpg

From what I see of other sample setups this isn't normal and they appear individually under roles and we can't migrate individual serves to another node. 

 

The server is setup 2x server, with 1 SAN CSV, and is setup successfully as a failover cluster. 


Cluster dies when 3rd node is on

0
0

Hi,

At work we have 3 servers within a cluster (Windows Server 2012 R2). On Monday the cluster failed and started to live migrate boxes to servers which were rebooting. We had a major site outage, where our proxy, exchange and lync went down. In the Failover Cluster Manager all the virtuals were stuck saying "loading", the only console which was working properly was "Hyper-V Manager". . We managed to get everything back up by rebooting each server to allow it to install Windows Updates.

On Tuesday, we had a similiar outage which was caused by one of the servers trying to take ownership of a store. The cluster then went into a "zombie state", which only occurred when the 3rd node was on. We now have a option where we can evict the node from the cluster and add it back in.

Any ideas why this might have happened?




Resource specific cluster scheduled task not running

0
0

Hi, I've setup a couple of chkdsk cluster scheduled tasks but they're not appearing in the failover cluster section in task scheduler, and not running when the trigger is hit. They were created using the following powershell commands

$actionF = New-ScheduledTaskAction –Execute C:\Windows\System32\chkdsk.exe -Argument /scan -WorkingDirectory "F:"
$triggerF = New-ScheduledTaskTrigger -Weekly -At 12:00 -DaysofWeek Saturday
Register-ClusteredScheduledTask –Cluster ClusterName –TaskName CHKDSK_F –TaskType ResourceSpecific –Resource ClusterResource –Action $actionF –Trigger $triggerF

does the resource need to be brought offline/online for the scheduled task to take effect?

thanks for any assistance

Generic Application fail-over is restarting

0
0

I have setup a two node cluster on 2012. I am trying to use the generic application to fail fail over and to pick up where it left off i.e. Move what is in memory to the fail over server. I can not get this to work. When it fails over it restarts the program and anything in memory is lost. I read that the memory was supposed to be written to disk for the fail over. 

To make it easier to see what is going on I created a program that just counts to a text file on the C drive. It over writes itself so you will only have the last number in the text file. If I run the program and then do a fail over after a few minutes (when the text file should be at 100 or so)it actually starts back at one again.

Any ideas?

NLB Web Front End (WFEs)servers & Clustering IIS Application Pool between the two WFEs

0
0

Hi All,

I have just encountered a very bad experience in my Production environment.My topology is as follows: NLB WFEs, MOSS APP server, Cluster DBs, DCs.

NLB on the WFEs is working fine. However, for some unknown reason, one of my site under the Application Pool in IIS Manager was stopped on one of the WFE and the site was down. NLB did not redirect request to the second WFE server.

My question is: How can I make sites under the Application Pool highly available? If any site is in a stopped state, there should be a mechanism in place to redirect request to the Application Pool on the second NLB WFE.

Kindly help and advise.

Thanks all.


Network Name Resource Availability - failover cluster error 1196

0
0

Hello,

We're getting this error in our even logs of our four node failover cluster, we tried deleting Host A record in DNS management, that did nothing.

Failover cluster event: 1196

"Cluster network name resource 'CAUCrgt8' failed registration of one or more associated DNS name(s) for the following reason: This operation returned because the timeout period expired.
Ensure that the network adapters associated with dependent IP address resources are configured with at least one accessible DNS server."

And this resource http://technet.microsoft.com/en-us/library/cc773529%28v=WS.10%29.aspx did not help in solving this.

Do you guys have any other suggestions we could try to resolve this error?

Validating Windows 2012 R2 Cluster Fail

0
0

Dear Reader,

I am Trying to build a Windows 2012 DC R2 Cluster.

I am having 2 AD Servers in Subnet else than the Subnet where I am trying to build the cluster, there is Firewall Between those 2 Subnets.

We have Limited RPC Port to be from 50000 to 50225, and configured other ports for Name Resolution and AD communication.

I have successfully join those 2 servers to AD, However, when I am trying to validate cluster, I am getting bellow error:

Validate Active Directory Configuration

    Connectivity to a writable domain controller from node XX01.XX.X could not be determined because of this error: Could not get domain controller name from machine XX01.

    Node(s) XX01.XX.X cannot reach a writable domain controller. Please check connectivity of these nodes to the domain controllers.

-----------------------------------------

After checking on Firewall between AD and those 2 Windows Server, I have find that the Cluster Service is trying to communicate on Dynamic Ports which is denied (Coz we configured windows to use dynamic ports between 50000 to 50225),so is there any way to force Failover cluster dynamic ports to be between 50000 to 50225?

Please let me know your suggestion about this.

 


Clustered role 'Availability Role' has exceeded its failover threshold

0
0

I am getting this alert on SQL 2012 R2 SP1. So please kindly tell me the solution of the below given alert on windows failover clustering .

Clustered role 'Availability Role' has exceeded its failover threshold. It has exhausted the configured number of failover attempts within the failover period of time allotted to it and will be left in a failed state. No additional attempts will be made to bring the role online or fail it over to another node in the cluster.Please check the events associated with the failure. After the issues causing the failure are resolved the role can be brought online manually or the cluster may attempt to bring it online again after the restart delay period.

Persistent Reservation not present on Test Disk 0 from node......

0
0

Hi all,

I did a research and find many similar issue in this forum. However, the issue persist.

There is a Windows Server 2012 R2 Hyper-V Failover Cluster in my lab, with HP P2000 storage with Fiber Channel. Everything goes well after a problem with one node(let's call it Node01) in the cluster. So I perform a reinstallation with Node01. After the fresh installation , the LUN for VMs show "unknown" in Disk Management on Node01, you can see the following screenshot. However, the LUN(from the same storage) for quorum is OK.

I try to add Node01 to the existing cluster and the cluster validation give the following warning.

Failure. Persistent Reservation not present on Test Disk 0 from node <<MY_SERVER_FQDN>> after successful call to update reservation

holder’s registration key 0xb.

Test Disk 0 does not support SCSI-3 Persistent Reservations commands needed to support clustered Storage Pools. Some storage devices

require specific firmware versions or settings to function properly with failover clusters. Please contact your storage administrator or

storage vendor to check the configuration of the storage to allow it to function properly with failover clusters.


If I ignore the warning and keep Node01 in the cluster and run the command:get-ClusterSharedVolumeState. It will show:
Node  : Node01
StateInfo : BlockRedirected


Cluster Unavailable

0
0

Hi All,

We got a serious problem here, we got two nodes(2008) in which Node 1 says Unavailable after upgrading the NIC firmware and Node 2 is working fine.

Below is the event occuring in the log,

1573 and 1069

Below is the steps we will be trying to do, but we are unsure as this is so critical node for my environment. Kindly help me in achieving the 100 percent redundancy.

1.       Restart Node 2. So that the  cluster drives will be mapped to Node 1. During this time the complete service will go offline.(approx 10 minutes in ideal situation).

2.       Once the drives are mapped to Node 1, the services will come online and everything should work fine.

3.       Also in the mean time Node 2 will be online and join the cluster.

Kindly confirm if this works fine.


Regards, Pratap

Network Configuration Problems

0
0

Hi,

I am really, really struggling to get networking correct on a Hyper-V cluster. The public LAN (Management and virtual machine network) is fine. This is purely focusing on the network adapters involved in the cluster.

I have 2 cluster nodes, and 4 switches. I have 1 server and 2 switches in each building, the switches are HP 2920.

We will call building 1 ServerRoom, and Building 2, BackupRoom

Switch 1 in ServerRoom is connected to switch 1 in BackupRoom on port A1 of the switch by fibre optic

Switch 2 in Server Room is connected to switch 2 in BackupRoom on port A1 of the switch by fibre optic

Switch 1 and 2 in ServerRoom are stacked

Switch 1 and 2 in BackupRoom are stacked

Below shows the IP addresses on each server and what ports they are connected to on the switches

This is how my cables go from server to the switches

ServerRoom

Switch   Network                           Port       IP Address

sw1        iscsi Primary                       1/a2       10.10.1.1

sw1        Live Mig Primary               1/1         10.10.2.1

sw1        HB Primary                       1/2            10.10.3.1

sw2        iscsi Backup                      2/a2      10.10.5.1

sw2        Live Mig Backup              2/1         10.10.6.1

sw2        HB Backup                         2/2        10.10.4.1

BackupRoom

Switch   Network                              Port       IP Address

sw1        iscsi Primary                      1/a2      10.10.1.2

sw1        live mig Primary               1/1         10.10.2.2

sw1        HB Primary                        1/2         10.10.3.2

sw2        iscsi Backup                       2/a2      10.10.5.2

sw2        Live Mig Backup               2/1         10.10.6.2

sw2        HB Backup                         2/2         10.10.4.2

I have created a trunk between switch 1 and switch 2 on the fibre optic ports 

Configure


Trunk 1/a1,2/a1 trk1 lacp

I have ran the same command on the switch in BackupRoom

Because I have a primary physical network cable and a backup physical cable in each server for each cluster resource I have to put these in different subnets, so they all use subnet mask 255.255.255.0

I have created 6 vlans, on each switch stack to keep the traffic separate, but I have then tagged trk1 into each vlan so I can benefit from the LACP

These commands are on the switch stack, I have used an IP ending in 5 (10.10.1.5 for example) to designate as the “management IP” or whatever HP refers to it as for each vlan (I am not too good with switches), and in BackupRoom I have used an IP ending in 6 (10.10.3.6 for example)

 

Vlan 2

Ip address 10.10.1.5  255.255.255.0

untag 1/a2

tag trk1

 

vlan 3

ip address 10.10.2.5 255.255.255.0

untag 1 /1

tag trk1

 

vlan 4

ip address 10.10.3.5 255.255.255.0

untag 1/2

tag trk1

 

vlan 5

ip address 10.10.5.5 255.255.255.0

untag 2/a2

tag trk1

 

vlan 6

ip address 10.10.6.5 255.255.255.0

untag 2/1

tag trk1

 

vlan 7

ip address 10.10.3.5 255.255.255.0

untag 2/2

tag trk1

I have repeated the above for BackupRoom switch stack.

However, I use a product called starwind to synchronise the storage, and when I use this configuration it seems to crash the product. When I remove the network cables from one of the switches or remove both the ISCSI cables the product frees itself up. This would indicate there is something wrong with my switch configuration, as if the network is being flooded (by the way I enabled spanning tree protocol too). So I’m really stuck on this….. can anyone suggest anything?

Thank you

Steve

Cluster Migration Wizard - network name resource

0
0

Hi!

 

We recently used the Cluster Migration Tool in Server 2008 to sucessfully migrate resources from a W2K3 MSCS Cluster.

Everything went very well, but one of the Network Name resources (totally 5) keeps complaining about the DNS Status. (Netbios and kerberos status is OK).

The network name seems to register fine in DNS.

 

Properties of the network name resource shows:

DNS Status: DNS signature failed to verify

 

Eventlog (system):

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          2008-03-01 11:53:01
Event ID:      1196
Task Category: Network Name Resource
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      xxxx.yyyy.zzz
Description:
Cluster network name resource 'Wins Name' failed registration of one or more associated DNS name(s) for the following reason:
DNS signature failed to verify.


Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          2008-03-01 11:53:01
Event ID:      1119
Task Category: Network Name Resource
Level:         Warning
Keywords:     
User:          SYSTEM
Computer:      xxxx.yyyy.zzz
Description:
Cluster network name resource 'Wins Name' failed to register DNS name 'xxxx.yyyy.zzz' over adapter 'Public' for the following reason:

'DNS signature failed to verify.

Regards,

Gustav

Hyper-V Cluster with Storage Server AH

0
0

Hello Everyone,

I have a question about Clustering&Storage.

I have two servers and what I wanted to do is:

-A cluster of 2 nodes

-The Storage directly on the Nodes

-The storage AH

I don't want a third server as Storage because, if the storage server fail the cluster will stop.

I wanted to know if it is possible to create a 2 node cluster with built-in Storage, so that if NODE1 fail, the VM or Data ar still available on NODE2.

automating failover cluster using batch file and power shell - [cluster disk 1 and cluster disk 2] will these name change in any case.

0
0

hi,

i am trying to automate the whole fail over cluster using set of powershell commands in a single batch file and a properties file, my problem is - after i create a cluster, i have 2 available disk name named cluster disk 1 and cluster disk 2. when i give this name in my properties file, i am afraid that it may change in some cases. also i dont know which one will go for quorum n which 1 for shared disk. can you please confirm if the name remains the same every time. So that i can give the same name in my properties file.

Also i actually wanted to get these cluster Resources using some command and put is by himself in the properties file. but seems not possble. kindly confirm. i used Get-ClusterResources, but it brings all the resources. i just wanted the cluster disk 1 n 2. is there any command for the cluster disk 1 n 2. is there any command for that ?



Viewing all 5648 articles
Browse latest View live




Latest Images