Welcome, Guest
Username: Password: Remember me
This forum is devoted to discussions around virtualization technologies related to cloud computing including Xen Cloud Platform, KVM and VMware.

TOPIC: Emergency

Emergency 1 year 1 month ago #12542

  • revivex36
  • revivex36's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 71
  • Thank you received: 1
  • Karma: 0
Hi

I am using cloudstack 3.0.2 and 3 xenserver 6.0.2.

2 xens are in same cluster and 3rd one in 2nd pod.
Accidentally all of xens are disconnected and I am trying
re-attach them. So far 3rd xen is OK but other 2 xens in same cluster.

When attaching xen1 and xen2 , cloud returns following error.

Cannot transit agent status with event AgentDisconnected for host 19,
mangement server id is 207377931084,
Unable to transition to a new state from Creating via AgentDisconnected


And I see log,

2012-09-07 06:49:36,105 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentTaskPool-5:null) Disconnected called on 19 with status Alert
2012-09-07 06:49:36,105 DEBUG [cloud.host.Status] (AgentTaskPool-5:null) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 19, name = xenserver02.to-the.net]
2012-09-07 06:49:36,167 DEBUG [cloud.host.Status] (AgentTaskPool-5:null) Agent status update: [id = 19; name = xenserver02.to-the.net; old status = Connecting; event = AgentDisconnected; new status = Alert; old update count = 6; new update count = 7]
2012-09-07 06:49:36,168 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentTaskPool-5:null) Notifying other nodes of to disconnect
2012-09-07 06:49:36,168 WARN [cloud.resource.ResourceManagerImpl] (AgentTaskPool-5:null) Unable to connect due to
com.cloud.exception.ConnectionException: Unable to connect to pool Pool[200|NetworkFilesystem]
at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:91)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:601)
at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1496)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1574)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1600)
at com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1191)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: com.cloud.exception.StorageUnavailableException: Resource [StoragePool:200] is unreachable: Unable establish connection from storage head to storage pool 200 due to ModifyStoragePoolCommand add XenAPIException:Can not see storage pool: e1672cc9-c6ad-3a62-b2d9-6ad810010f72 from on host:dc4635be-10d6-42d9-9ae7-91cf7c31d187 host:dc4635be-10d6-42d9-9ae7-91cf7c31d187 pool: 64.234.225.133/export/primary
at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1567)
at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:88)
... 8 more
2012-09-07 06:49:36,169 DEBUG [cloud.host.Status] (AgentTaskPool-5:null) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 19, name = xenserver02.to-the.net]
2012-09-07 06:49:36,235 DEBUG [cloud.host.Status] (AgentTaskPool-5:null) Agent status update: [id = 19; name = xenserver02.to-the.net; old status = Alert; event = AgentDisconnected; new status = Alert; old update count = 7; new update count = 8]
2012-09-07 06:49:37,890 DEBUG [cloud.server.StatsCollector] (StatsCollector-3:null) VmStatsCollector is running...
2012-09-07 06:49:38,525 DEBUG [cloud.server.StatsCollector] (StatsCollector-3:null) HostStatsCollector is running...
2012-09-07 06:49:38,532 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-67:null) Seq 18-380960787: Executing request
2012-09-07 06:49:38,752 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-67:null) Seq 18-380960787: Response Received:
2012-09-07 06:49:38,753 DEBUG [agent.transport.Request] (StatsCollector-3:null) Seq 18-380960787: Received: { Ans: , MgmtId: 207377931084, via: 18, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2012-09-07 06:49:38,753 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-67:null) Cleanup succeeded. Details null
2012-09-07 06:49:38,753 DEBUG [cloud.vm.VirtualMachineManagerImpl] (StatsCollector-3:null) Cleanup succeeded. Details null
2012-09-07 06:49:40,515 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) StorageCollector is running...
2012-09-07 06:49:40,591 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentManager-Handler-11:null) Cleanup succeeded. Details null
2012-09-07 06:49:40,591 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 20-106889223: Received: { Ans: , MgmtId: 207377931084, via: 20, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2012-09-07 06:49:40,592 DEBUG [cloud.vm.VirtualMachineManagerImpl] (StatsCollector-1:null) Cleanup succeeded. Details null
2012-09-07 06:49:40,598 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-35:null) Seq 18-380960788: Executing request
2012-09-07 06:49:41,054 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-35:null) Seq 18-380960788: Response Received:
2012-09-07 06:49:41,054 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 18-380960788: Received: { Ans: , MgmtId: 207377931084, via: 18, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2012-09-07 06:49:41,054 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-35:null) Cleanup succeeded. Details null
2012-09-07 06:49:41,054 DEBUG [cloud.vm.VirtualMachineManagerImpl] (StatsCollector-1:null) Cleanup succeeded. Details null
2012-09-07 06:49:41,058 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-64:null) Seq 18-380960789: Executing request
2012-09-07 06:49:41,454 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-64:null) Seq 18-380960789: Response Received:
2012-09-07 06:49:41,454 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 18-380960789: Received: { Ans: , MgmtId: 207377931084, via: 18, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2012-09-07 06:49:41,454 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-64:null) Cleanup succeeded. Details null
2012-09-07 06:49:41,454 DEBUG [cloud.vm.VirtualMachineManagerImpl] (StatsCollector-1:null) Cleanup succeeded. Details null


I can not re-attach 2 xen servers. They had SSVM and Console Proxy VM also
router VM.

Please advise and help!
The administrator has disabled public write access.

Re: Emergency 1 year 1 month ago #12663

  • revivex36
  • revivex36's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 71
  • Thank you received: 1
  • Karma: 0
I fixed by myself.

Using xenCenter and mount primary storage with mount name "e1672cc9-c6ad-3a62-b2d9-6ad810010f72" . Then cloud let xens attach again.
The administrator has disabled public write access.

Re: Emergency 1 year 1 month ago #12665

  • revivex36
  • revivex36's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 71
  • Thank you received: 1
  • Karma: 0
Hi

After reattached storage pool, I can not create instance anymore.
console vm, router vm and storage vm are working fine.

Here is error log.

2012-09-23 11:33:22,202 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-5:job-155) Checking if we need to prepare 2 volumes for VM[User|i-2-46-VM]
2012-09-23 11:33:22,300 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-5:job-155) Creating volume: Vol[57|vm=46|ROOT]
2012-09-23 11:33:22,301 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-5:job-155) Trying to create in Pool[200|NetworkFilesystem]
2012-09-23 11:33:22,426 DEBUG [cloud.template.TemplateManagerImpl] (Job-Executor-5:job-155) Template 5 has already been downloaded to pool 200
2012-09-23 11:33:22,430 DEBUG [agent.transport.Request] (Job-Executor-5:job-155) Seq 39-650510386: Sending { Cmd , MgmtId: 207377931084, via: 39, Ver: v1, Fl1, [{"storage.CreateCommand":{"volId":57,"pool":{"id":200,"uuid":"e1672cc9-c6ad-3a62-b2d9-6ad810010f72","host":"64.234.225.133","path":"/export/primary","portpe":"NetworkFilesystem"},"diskCharacteristics":{"size":21474836480,"tags":[],"type":"ROOT","name":"ROOT-46","useLocalStorage":false,"recreatable":true,"diskOf1,"volumeId":57,"hyperType":"XenServer"},"templateUrl":"a153854e-57b3-45d1-b1c9-bc6119c2572c","wait":0}}] }
2012-09-23 11:33:22,430 DEBUG [agent.transport.Request] (Job-Executor-5:job-155) Seq 39-650510386: Executing: { Cmd , MgmtId: 207377931084, via: 39, Ver: v1,0111, [{"storage.CreateCommand":{"volId":57,"pool":{"id":200,"uuid":"e1672cc9-c6ad-3a62-b2d9-6ad810010f72","host":"64.234.225.133","path":"/export/primary","p"type":"NetworkFilesystem"},"diskCharacteristics":{"size":21474836480,"tags":[],"type":"ROOT","name":"ROOT-46","useLocalStorage":false,"recreatable":true,"disd":1,"volumeId":57,"hyperType":"XenServer"},"templateUrl":"a153854e-57b3-45d1-b1c9-bc6119c2572c","wait":0}}] }
2012-09-23 11:33:22,430 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-57:null) Seq 39-650510386: Executing request
2012-09-23 11:33:22,522 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-57:null) SR retrieved for 200
2012-09-23 11:33:22,537 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-57:null) Checking e1672cc9-c6ad-3a62-b2d9-6ad810010f72 or SR 9e9c692a-7d90-6401-2a6db14 on XS[5ce5aa71-c123-4201-b208-4fab52fa44b6-64.234.225.132]
2012-09-23 11:33:22,558 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-57:null) Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid fo53854e-57b3-45d1-b1c9-bc6119c2572c failed due to The uuid you supplied was invalid.
2012-09-23 11:33:22,558 WARN [xen.resource.CitrixResourceBase] (DirectAgent-57:null) Unable to create volume; Pool=Pool[200|64.234.225.133:2049|/export/prima DskChr[ROOT|21474836480|]
com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: a153854e-57b3-45d1-b1c9-bc611iled due to The uuid you supplied was invalid.
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVDIbyUuid(CitrixResourceBase.java:5245)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5100)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:382)
at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: The uuid you supplied was invalid.
at com.xensource.xenapi.Types.checkResponse(Types.java:823)
at com.xensource.xenapi.Connection.dispatch(Connection.java:372)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:905)
at com.xensource.xenapi.VDI.getByUuid(VDI.java:313)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVDIbyUuid(CitrixResourceBase.java:5241)
... 12 more
2012-09-23 11:33:22,559 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-57:null) Seq 39-650510386: Response Received:
2012-09-23 11:33:22,560 DEBUG [agent.transport.Request] (DirectAgent-57:null) Seq 39-650510386: Processing: { Ans: , MgmtId: 207377931084, via: 39, Ver: v1, , [{"storage.CreateAnswer":{"requestTemplateReload":false,"result":false,"details":"Exception: com.cloud.utils.exception.CloudRuntimeException\nMessage: Catch com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: a153854e-57b3-45d1-b1c9-bc6119c2572c failed due to The uuid you supplied was invalid.\nStack:.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: a153854e-57b3-45d1-b1c9-bc6119c2572c fto The uuid you supplied was invalid.\n\tat com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVDIbyUuid(CitrixResourceBase.java:5245)\n\tat com.cloud.hyen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5100)\n\tat com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceB82)\n\tat com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)\n\tat com.cloud.agent.manager.DirectAgentAttache$TrectAgentAttache.java:187)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask$Sync.innerRun(Fjava:334)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:166)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101ThreadPoolExecutor.java:165)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)\n\tat java.urent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat jaread.run(Thread.java:679)\nCaused by: The uuid you supplied was invalid.\n\tat com.xensource.xenapi.Types.checkResponse(Types.java:823)\n\tat com.xensource.xection.dispatch(Connection.java:372)\n\tat com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java: com.xensource.xenapi.VDI.getByUuid(VDI.java:313)\n\tat com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVDIbyUuid(CitrixResourceBase.java:5241)\n\t...","wait":0}}] }
2012-09-23 11:33:22,560 WARN [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-57:null) Cleanup failed due to Exception: com.cloud.utils.exception.CloudRuntin
Message: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: a153854e-57b3-45d1-b1c9-bc6119c2572c failed due to The uuid you supplvalid.
Stack: com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: a153854e-57b3-45d1-b1c572c failed due to The uuid you supplied was invalid.
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVDIbyUuid(CitrixResourceBase.java:5245)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5100)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:382)
at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: The uuid you supplied was invalid.
at com.xensource.xenapi.Types.checkResponse(Types.java:823)
at com.xensource.xenapi.Connection.dispatch(Connection.java:372)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:905)
at com.xensource.xenapi.VDI.getByUuid(VDI.java:313)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVDIbyUuid(CitrixResourceBase.java:5241)
... 12 more

2012-09-23 11:33:22,560 DEBUG [agent.transport.Request] (Job-Executor-5:job-155) Seq 39-650510386: Received: { Ans: , MgmtId: 207377931084, via: 39, Ver: v1,0, { CreateAnswer } }
2012-09-23 11:33:22,560 DEBUG [agent.manager.AgentAttache] (DirectAgent-57:null) Seq 39-650510386: No more commands found
2012-09-23 11:33:22,560 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-5:job-155) Cleanup failed due to Exception: com.cloud.utils.exception.CloudRution
Message: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: a153854e-57b3-45d1-b1c9-bc6119c2572c failed due to The uuid you supplvalid.
Stack: com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: a153854e-57b3-45d1-b1c572c failed due to The uuid you supplied was invalid.
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVDIbyUuid(CitrixResourceBase.java:5245)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5100)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:382)
at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: The uuid you supplied was invalid.
at com.xensource.xenapi.Types.checkResponse(Types.java:823)
at com.xensource.xenapi.Connection.dispatch(Connection.java:372)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:905)
at com.xensource.xenapi.VDI.getByUuid(VDI.java:313)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVDIbyUuid(CitrixResourceBase.java:5241)
... 12 more

2012-09-23 11:33:22,561 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-5:job-155) Unable to create volume Vol[57|vm=46|ROOT]
2012-09-23 11:33:22,684 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-5:job-155) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:200] is unreachable: Unable to create Vol[57|vm=46|ROOT]
at com.cloud.storage.StorageManagerImpl.prepare(StorageManagerImpl.java:2990)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:743)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:461)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2580)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2517)
at com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:32)
at com.cloud.api.commands.DeployVMCmd.execute(DeployVMCmd.java:317)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:132)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:427)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-09-23 11:33:22,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-5:job-155) Cleaning up resources for the vm VM[User|i-2-46-VM] in Starting st
2012-09-23 11:33:22,769 DEBUG [agent.transport.Request] (Job-Executor-5:job-155) Seq 39-650510387: Sending { Cmd , MgmtId: 207377931084, via: 39, Ver: v1, Fl1, [{"StopCommand":{"isProxy":false,"vmName":"i-2-46-VM","wait":0}}] }
2012-09-23 11:33:22,770 DEBUG [agent.transport.Request] (Job-Executor-5:job-155) Seq 39-650510387: Executing: { Cmd , MgmtId: 207377931084, via: 39, Ver: v1,0111, [{"StopCommand":{"isProxy":false,"vmName":"i-2-46-VM","wait":0}}] }
2012-09-23 11:33:22,770 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-80:null) Seq 39-650510387: Executing request
2012-09-23 11:33:22,870 INFO [xen.resource.CitrixResourceBase] (DirectAgent-80:null) VM does not exist on XenServer5ce5aa71-c123-4201-b208-4fab52fa44b6
2012-09-23 11:33:22,871 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-80:null) Seq 39-650510387: Response Received:
2012-09-23 11:33:22,871 DEBUG [agent.transport.Request] (DirectAgent-80:null) Seq 39-650510387: Processing: { Ans: , MgmtId: 207377931084, via: 39, Ver: v1, , [{"StopAnswer":{"vncPort":0,"bytesSent":0,"bytesReceived":0,"result":true,"details":"VM does not exist","wait":0}}] }
2012-09-23 11:33:22,871 DEBUG [agent.transport.Request] (Job-Executor-5:job-155) Seq 39-650510387: Received: { Ans: , MgmtId: 207377931084, via: 39, Ver: v1,0, { StopAnswer } }
2012-09-23 11:33:22,871 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-80:null) Cleanup succeeded. Details VM does not exist
2012-09-23 11:33:22,871 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-5:job-155) Cleanup succeeded. Details VM does not exist
2012-09-23 11:33:22,871 DEBUG [agent.manager.AgentAttache] (DirectAgent-80:null) Seq 39-650510387: No more commands found
2012-09-23 11:33:22,965 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-5:job-155) Changing active number of nics for network id=204 on -1
2012-09-23 11:33:23,060 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-5:job-155) Successfully cleanued up resources for the vm VM[User|i-2-46-VM] i state
2012-09-23 11:33:23,062 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) DeploymentPlanner allocation algorithm: random
2012-09-23 11:33:23,062 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:ested cpu: 500, requested ram: 536870912
2012-09-23 11:33:23,062 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) Is ROOT volume READY (pool already allocated)?: No
2012-09-23 11:33:23,062 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) Searching all possible resources under this Zone: 1
2012-09-23 11:33:23,063 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) Listing clusters in order of aggregate capacity, that have (atleast one enough CPU and RAM capacity under this Zone: 1
2012-09-23 11:33:23,064 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) CPUOverprovisioningFactor considered: 1.0
2012-09-23 11:33:23,069 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) Checking resources in Cluster: 1 under Pod: 1
2012-09-23 11:33:23,069 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) Calling HostAllocators to find suitable hosts
2012-09-23 11:33:23,069 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-5:job-155 FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:
2012-09-23 11:33:23,070 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-5:job-155 FirstFitRoutingAllocator) FirstFitAllocator has 1 hosts to check for : [Host[-39-Routing]]
2012-09-23 11:33:23,072 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-5:job-155 FirstFitRoutingAllocator) Found 1 hosts for allocation after prioritiost[-39-Routing]]
2012-09-23 11:33:23,072 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-5:job-155 FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=512
2012-09-23 11:33:23,072 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-5:job-155 FirstFitRoutingAllocator) Host name: xenserver01.to-the.net, hostId: void set, skipping this and trying other available hosts
2012-09-23 11:33:23,072 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-5:job-155 FirstFitRoutingAllocator) Host Allocator returning 0 suitable hosts
2012-09-23 11:33:23,072 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) No suitable hosts found
2012-09-23 11:33:23,072 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) No suitable hosts found under this Cluster: 1
2012-09-23 11:33:23,072 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-155) Could not find suitable Deployment Destination for this VM under any cluurning.
2012-09-23 11:33:23,564 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-5:job-155) VM state transitted from :Starting to Stopped with event: Operatios original host id: null new host id: null host id before state transition: 39
2012-09-23 11:33:23,567 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-5:job-155) Hosts's actual total CPU: 9576 and CPU after applying overprovisio
2012-09-23 11:33:23,567 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-5:job-155) release cpu from host: 39, old used: 2000,reserved: 0, actual totaotal with overprovisioning: 9576; new used: 1500,reserved:0; movedfromreserved: false,moveToReserveredfalse
2012-09-23 11:33:23,567 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-5:job-155) release mem from host: 39, old used: 2013265920,reserved: 0, total36; new used: 1476395008,reserved:0; movedfromreserved: false,moveToReserveredfalse
2012-09-23 11:33:23,803 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-5:job-155) VM state transitted from :Stopped to Error with event: OperationFarvm's original host id: null new host id: null host id before state transition: null
2012-09-23 11:33:24,434 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-91:null) Ping from 39
2012-09-23 11:33:24,756 INFO [api.commands.DeployVMCmd] (Job-Executor-5:job-155) com.cloud.exception.InsufficientServerCapacityException: Unable to create a for VM[User|i-2-46-VM]Scope=interface com.cloud.dc.DataCenter; id=1
2012-09-23 11:33:24,757 WARN [cloud.api.ApiDispatcher] (Job-Executor-5:job-155) class com.cloud.api.ServerApiException : Unable to create a deployment for VM46-VM]
2012-09-23 11:33:24,757 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-5:job-155) Complete async job-155, jobStatus: 2, resultCode: 530, result: com.clsponse.ExceptionResponse@5c7d517
2012-09-23 11:33:25,045 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-23:null) Async job-155 completed
2012-09-23 11:33:25,413 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-89:null) Seq 39-650510338: Executing request
2012-09-23 11:33:25,725 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-89:null) Seq 39-650510338: Response Received:
2012-09-23 11:33:25,726 DEBUG [agent.transport.Request] (DirectAgent-89:null) Seq 39-650510338: Processing: { Ans: , MgmtId: 207377931084, via: 39, Ver: v1, [{"ClusterSyncAnswer":{"_clusterId":1,"_newStates":{},"_isExecuted":false,"result":true,"wait":0}}] }

Sounds like supplying uuid for storage pool is wrong. I think I messed up
database somehow.

Is anyone can give me a tip to fix this?

Thank you.
The administrator has disabled public write access.

Re: Emergency 1 year 4 weeks ago #12700

  • revivex36
  • revivex36's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 71
  • Thank you received: 1
  • Karma: 0
While fixing this problem, cloud deleted iso and vhd somehow.
And also deleted wrong instance in DB by my handling mistake.

After restore them from backup it start creating vms.
The administrator has disabled public write access.
Time to create page: 0.374 seconds

Open@Citrix

Citrix supports the open source community via developer support and evangeslism. We have a number of developers and evangelists that participate actively in the open source community in Apache Cloudstack, OpenDaylight, Xen Project and XenServer. We also conduct educational activities via the Build A Cloud events held all over the world. 

Connect