Welcome, Guest
Username: Password: Remember me
This forum is devoted to discussions around virtualization technologies related to cloud computing including Xen Cloud Platform, KVM and VMware.

TOPIC: VM Host Down - Unable to connect via XenCenter

VM Host Down - Unable to connect via XenCenter 7 months 1 week ago #13100

  • yorkds
  • yorkds's Avatar
  • OFFLINE
  • Fresh Boarder
  • Posts: 2
  • Karma: 0
Hi New to the forum,

We have implemented a Cloudstack environment with 2 XenServer hosts. CLVM1 and CLVM2. Last night CLVM1 changed its status to down in Cloudstack and I am unable to bring it back up with the error "Command failed due to Internal Server Error"

Cloudstack states:

2013-03-13 09:20:50,363 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-235:null) Unable to logon in 192.168.241.21 as master in pool(1a8bbe3d-c2b7-9983-7212-83f7b391540f)
2013-03-13 09:20:50,369 WARN [agent.manager.DirectAgentAttache] (DirectAgent-235:null) Seq 2-2072766934: Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: Unable to logon in 192.168.241.21 as master in pool(1a8bbe3d-c2b7-9983-7212-83f7b391540f)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:630)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5341)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:2093)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:435)
at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-03-13 09:20:50,370 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-235:null) Seq 2-2072766934: Response Received:
2013-03-13 09:20:50,370 DEBUG [agent.transport.Request] (DirectAgent-235:null) Seq 2-2072766934: Processing: { Ans: , MgmtId: 126420062913163, via: 2, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: Unable to logon in 192.168.241.21 as master in pool(1a8bbe3d-c2b7-9983-7212-83f7b391540f)","wait":0}}] }
2013-03-13 09:20:50,370 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 2-2072766934: Received: { Ans: , MgmtId: 126420062913163, via: 2, Ver: v1, Flags: 10, { Answer } }
2013-03-13 09:20:50,370 DEBUG [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Details from executing class com.cloud.agent.api.GetHostStatsCommand: com.cloud.utils.exception.CloudRuntimeException: Unable to logon in 192.168.241.21 as master in pool(1a8bbe3d-c2b7-9983-7212-83f7b391540f)
2013-03-13 09:20:50,370 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 2 statistics.
2013-03-13 09:20:50,370 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 2
2013-03-13 09:20:50,684 DEBUG [cloud.server.StatsCollector] (StatsCollector-3:null) VmStatsCollector is running...
2013-03-13 09:20:50,690 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-69:null) Seq 2-2072766935: Executing request
2013-03-13 09:20:50,690 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-69:null) Logging on as the slave to 192.168.241.22
2013-03-13 09:20:50,779 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-69:null) Unable to logon in 192.168.241.21 as master in pool(1a8bbe3d-c2b7-9983-7212-83f7b391540f)
2013-03-13 09:20:50,786 WARN [agent.manager.DirectAgentAttache] (DirectAgent-69:null) Seq 2-2072766935: Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: Unable to logon in 192.168.241.21 as master in pool(1a8bbe3d-c2b7-9983-7212-83f7b391540f)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:630)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5341)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:2183)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:437)
at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-03-13 09:20:50,786 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-69:null) Seq 2-2072766935: Response Received:
2013-03-13 09:20:50,786 DEBUG [agent.transport.Request] (DirectAgent-69:null) Seq 2-2072766935: Processing: { Ans: , MgmtId: 126420062913163, via: 2, Ver: v1, Flags: 10, [{"Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: Unable to logon in 192.168.241.21 as master in pool(1a8bbe3d-c2b7-9983-7212-83f7b391540f)","wait":0}}] }
2013-03-13 09:20:50,786 DEBUG [agent.transport.Request] (StatsCollector-3:null) Seq 2-2072766935: Received: { Ans: , MgmtId: 126420062913163, via: 2, Ver: v1, Flags: 10, { Answer } }
2013-03-13 09:20:50,786 DEBUG [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Details from executing class com.cloud.agent.api.GetVmStatsCommand: com.cloud.utils.exception.CloudRuntimeException: Unable to logon in 192.168.241.21 as master in pool(1a8bbe3d-c2b7-9983-7212-83f7b391540f)
2013-03-13 09:20:50,786 WARN [cloud.vm.UserVmManagerImpl] (StatsCollector-3:null) Unable to obtain VM statistics.

I'm also unable to log into either Hosts with no error (it just sits there for hours)

When I look in /var/log/messages I can see that I have the logs below, these started when the host went down.

Mar 13 04:58:31 yds-clvm1 ovs-vswitchd: 71659|coverage|INFO|Skipping details of duplicate event coverage for hash=c830b7e3 in epoch
174773189
Mar 13 04:58:45 yds-clvm1 xapi: [ warn|yds-clvm1|18 db_gc|DB GC D:50fede636290|db_gc] There are more pending tasks than the maximum
allowed: 316 > 200
Mar 13 04:59:15 yds-clvm1 xapi: [ warn|yds-clvm1|18 db_gc|DB GC D:5e5d3e8ab0db|db_gc] There are more pending tasks than the maximum
allowed: 318 > 200
Mar 13 05:00:15 yds-clvm1 xapi: [ warn|yds-clvm1|18 db_gc|DB GC D:db91d9b8a867|db_gc] There are more pending tasks than the maximum
allowed: 323 > 200
Mar 13 05:01:26 yds-clvm1 xapi: [error|yds-clvm1|5259581 inet-RPC|host.call_plugin R:fe85b72f9dfa|stunnel] get_reusable_stunnel: fre
sh stunnel connection failed with exception: File "stunnel.ml", line 135, characters 20-25: Pattern matching failed: delaying 10.00
seconds before reconnecting to 192.168.241.22:443 (attempt 1 / 10)
Mar 13 05:01:26 yds-clvm1 xapi: [error|yds-clvm1|5259583 inet-RPC|SR.scan R:81ca8e6550a4|xapi] Caught error scanning SR (22a202a1-24
c6-33c7-9c64-44ad06d22fde): INTERNAL_ERROR: [ Sys_error("/dev/urandom: Too many open files") ].
Mar 13 05:01:29 yds-clvm1 /opt/xensource/bin/xapi: xc: error: Could not obtain handle on privileged command interface (24 = Too many
open files): Internal error

Google has given me 0 ideas, all VM's are still running ok so this isn't customer affecting at the moment. For some reason Cloudstack hasn't moved the pool master over to CLVM2 which is maybe why I can't log into that one on XenCenter.

Thanks in advance. - I have also attached screen shot of xsconsole error.
Oliver.

The administrator has disabled public write access.

VM Host Down - Unable to connect via XenCenter 7 months 1 week ago #13101

  • yorkds
  • yorkds's Avatar
  • OFFLINE
  • Fresh Boarder
  • Posts: 2
  • Karma: 0
It seems a problem with xapi at least to start with, when I try to restart the xe-toolstack-restart it just sits as stopping

[root@yds-clvm1 ~]# xe-toolstack-restart
Stopping xapi:

Thanks
The administrator has disabled public write access.
Time to create page: 0.268 seconds

Open@Citrix

Citrix supports the open source community via developer support and evangeslism. We have a number of developers and evangelists that participate actively in the open source community in Apache Cloudstack, OpenDaylight, Xen Project and XenServer. We also conduct educational activities via the Build A Cloud events held all over the world. 

Connect