Just checked in a fix for this. change xapitimeout to 600 second, looks like we need to change some XAPI calls to Async before shorten the timeout
Anthony From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] Sent: Wednesday, April 16, 2014 7:27 AM To: Murali Reddy Cc: Anthony Xu; dev@cloudstack.apache.org Subject: Re: [4.4]copy_vhd_from_secondarystorage is timing out This is a similar problem to what I sent an e-mail out about last night: XmlRpcException (Read timed out). For me it's happening when CitrixResourceBase issues SR.create. This issue is currently blocking my testing progress and I'm not really sure who might be the right person to look into this. Any thoughts on this anyone? On Wed, Apr 16, 2014 at 1:51 AM, Murali Reddy <murali.re...@citrix.com<mailto:murali.re...@citrix.com>> wrote: System VM's are launching fine now after Anthony's fix. But user VM's are failing to get created with exception. 43275 2014-04-16 12:59:08,647 DEBUG [c.c.h.x.r.XenServerConnectionPool] (DirectAgent-137:ctx-4761cb0c) XmlRpcException for method: VDI.snapshot due to org.apache.xmlrpc.XmlRpcException: Failed to create input stream: Read timed out 43276 2014-04-16 12:59:08,647 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-137:ctx-4761cb0c) Catch Exception org.apache.xmlrpc.XmlRpcException for template + due to org.apache.xmlrpc.XmlRpcException: Failed to create input stream: Read timed out org.apache.xmlrpc.XmlRpcException: Failed to create input stream: Read timed out org.apache.xmlrpc.XmlRpcException: Failed to create input stream: Read timed out at org.apache.xmlrpc.client.XmlRpcSunHttpTransport.getInputStream(XmlRpcSunHttpTransport.java:99) at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:152) at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) at org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunHttpTransport.java:69) at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) at com.xensource.xenapi.Connection.dispatch(Connection.java:285) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456) at com.xensource.xenapi.VDI.snapshot(VDI.java:1155) at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:972) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:77) From: Mike Tutkowski <mike.tutkow...@solidfire.com<mailto:mike.tutkow...@solidfire.com>> Date: Wednesday, 16 April 2014 12:35 AM To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> Cc: Murali Reddy <murali.re...@citrix.com<mailto:murali.re...@citrix.com>> Subject: Re: [4.4]copy_vhd_from_secondarystorage is timing out Hi Anthony, Can you go into a bit more depth about copy_vhd_from_secondarystorage not being used with XenServer 6.2? Does that mean "public Answer copyTemplateToPrimaryStorage(CopyCommand cmd)" in XenServerStorageProcessor is not called when XenServer 6.2 is in use? I'm testing out a feature of mine for 4.4, but have only been using XenServer 6.1 so far. Thanks! Mike On Tue, Apr 15, 2014 at 12:44 PM, Anthony Xu <xuefei...@citrix.com<mailto:xuefei...@citrix.com>> wrote: Hi Murali, It is caused by the commit, I tested the commit on XS 6.2 FOX, which doesn't use copy_vhd_from_secondarystorage, that's why I didn't see this issue. Just disabled XAPI Event in 4.4 branch. Thanks, Anthony -----Original Message----- From: Murali Reddy Sent: Tuesday, April 15, 2014 10:54 AM To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>; Anthony Xu Subject: Re: [4.4]copy_vhd_from_secondarystorage is timing out I am still running into this issue. Alex reverted complete change replacing xapi.jar, which was also resulting XmlRpc exceptions but system VM's were launching fine after revert. I may be wrong but looking at the stack trace and recent commits, commit 9f44909e63a19b2b8034baa13b92e24ea2eb0227 (use event instead of poll for xapi async call in XS 6.2 and above to reduce the pressure on XAPI) may be causing this issue. Anthony can you please take a look? On 14/04/14 8:12 AM, "Mike Tutkowski" <mike.tutkow...@solidfire.com<mailto:mike.tutkow...@solidfire.com>> wrote: >I was seeing a similar issue when XAPI was changed when I tried to >create an SR. > >I wonder if that XAPI issue hasn't been fully resolved. > > >On Sun, Apr 13, 2014 at 4:16 PM, Srikanteswararao Talluri < >srikanteswararao.tall...@citrix.com<mailto:srikanteswararao.tall...@citrix.com>> > wrote: > >> Did anybody see this on latest 4.4? >> >> >> 2014-04-14 00:01:34,305 DEBUG [c.c.h.x.r.XenServerConnectionPool] >> (DirectAgent-71:ctx-12d9be59) XmlRpcException for method: event.from >>due to >> org.apache.xmlrpc.XmlRpcException: Failed to create input stream: >>Read timed out >> >> 2014-04-14 00:01:34,305 WARN [c.c.h.x.r.CitrixResourceBase] >> (DirectAgent-71:ctx-12d9be59) callHostPlugin failed for cmd: >> copy_vhd_from_secondarystorage with args mountpoint: >>10.147.28.7:/export/home/talluri/psec/template/tmpl/1/1/, >> sruuid: b3dd21aa-8e82-a923-ed29-ee58220d158b, namelabel: >> cloud-8498d88f-be6b-41b6-bece-a58bbc1fbc32, due to Failed to create >>input >> stream: Read timed out >> >> org.apache.xmlrpc.XmlRpcException: Failed to create input stream: >> Read timed out >> >> at >> >>org.apache.xmlrpc.client.XmlRpcSunHttpTransport.getInputStream(XmlRpcS >>unH >>ttpTransport.java:99) >> >> at >> >>org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStrea >>mTr >>ansport.java:152) >> >> at >> >>org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTra >>nsp >>ort.java:143) >> >> at >> >>org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunH >>ttp >>Transport.java:69) >> >> at >> >>org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker >>.ja >>va:56) >> >> at >> org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) >> >> at >> org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) >> >> at >> org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) >> >> at >> com.xensource.xenapi.Connection.dispatch(Connection.java:285) >> >> at >> >>com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerCon >>nec >>tion.dispatch(XenServerConnectionPool.java:456) >> >> at com.xensource.xenapi.Event.properFrom(Event.java:310) >> >> at >> >>com.cloud.hypervisor.xen.resource.XenServer620Resource.waitForTask(Xen >>Ser >>ver620Resource.java:125) >> >> at >> >>com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsy >>nc( >>CitrixResourceBase.java:3558) >> >> at >> >>com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_f >>rom >>_secondarystorage(XenServerStorageProcessor.java:826) >> >> at >> >>com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTempla >>teT >>oPrimaryStorage(XenServerStorageProcessor.java:962) >> >> at >> >>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute( >>Sto >>rageSubsystemCommandHandlerBase.java:77) >> >> at >> >>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleSt >>ora >>geCommands(StorageSubsystemCommandHandlerBase.java:52) >> >> at >> >>com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(Ci >>tri >>xResourceBase.java:542) >> >> at >> >>com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(X >>enS >>erver56Resource.java:60) >> >> at >> >>com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest( >>Xen >>Server610Resource.java:92) >> >> at >> >>com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAge >>ntA >>ttache.java:216) >> >> at >> >>org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man >>age >>dContextRunnable.java:49) >> >> at >> >>org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal >>l(D >>efaultManagedContext.java:56) >> >> at >> >>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW >>ith >>Context(DefaultManagedContext.java:103) >> >> >> 547,2-9 >> > > > >-- >*Mike Tutkowski* >*Senior CloudStack Developer, SolidFire Inc.* >e: mike.tutkow...@solidfire.com<mailto:mike.tutkow...@solidfire.com> >o: 303.746.7302<tel:303.746.7302> >Advancing the way the world uses the >cloud<http://solidfire.com/solution/overview/?video=play> >*(tm)* > -- Mike Tutkowski Senior CloudStack Developer, SolidFire Inc. e: mike.tutkow...@solidfire.com<mailto:mike.tutkow...@solidfire.com> o: 303.746.7302<tel:303.746.7302> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>(tm) -- Mike Tutkowski Senior CloudStack Developer, SolidFire Inc. e: mike.tutkow...@solidfire.com<mailto:mike.tutkow...@solidfire.com> o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>(tm)
