Management IP addresses pool exhausted
Hi, How to free exhausted IP addresses pool from cloudstack, Cloudstack was unable to launch systems vms at first and especially ssvm due to strange nfs time out issues so all IP's were used during this issue and now is there any precise way of releasing these IP's and starting system VM's again?. -- Umair Azam
Re: Management IP addresses pool exhausted
SELECT * FROM cloud.vm_instance where (name like s-% or name like v-%); shows me all VM's is expunging state and expunge delay/interval values are set to 86400 Umair Azam On 3/22/2014 6:58 AM, Umair Azam wrote: Hi, How to free exhausted IP addresses pool from cloudstack, Cloudstack was unable to launch systems vms at first and especially ssvm due to strange nfs time out issues so all IP's were used during this issue and now is there any precise way of releasing these IP's and starting system VM's again?.
Re: nfs issue while starting SSVM
I am having exactly this issue. http://discussions.citrix.com/topic/326059-nfs-server-timed-out/ -hypervisor load remains normal -sudden spike in load of ssvm which is being created on hypervisor -nfs server remains up and running -iptables are disabled -replaced switch/ethernet cables/NIC's for testing Umair Azam On 3/14/2014 7:13 AM, Umair Azam wrote: Amin, I cant ssh into it and now unable to deploy it again as all management addresses have been occupied, but the problem at this moment is not why SSVM is not running problem is why xenserver shows nfs server timed out issue in its logs and after that it reboots itself. Umair Azam On 3/14/2014 6:57 AM, Amin Pashapour wrote: Have you tried the ssvm troubleshooting? https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Secondary+storage+troubleshooting of course that won't help if you can't ssh into it :) From: Umair Azam ua...@i2cinc.com To: users@cloudstack.apache.org Sent: Thursday, March 13, 2014 9:33 PM Subject: nfs issue while starting SSVM Hi guys, When cloudstack tries to launch SSVM for the first time nfs server not responding, timed out log entires appear in xenserver 6.2 and then hypervisor reboots itself due to HA enabled. -Iptabes are disabled -sysconfig/nfs is all set up -Repalced machines NIC/switch/ethernet cables. but still xenserver-1 kernel: [ 3096.643040] nfs: server 10.11.17.33 not responding, timed out Mar 14 06:07:56 xenserver-1 kernel: [ 3096.643054] nfs: server 10.11.17.33 not responding, timed out Mar 14 06:07:56 xenserver-1 kernel: [ 3096.643059] nfs: server 10.11.17.33 not responding, timed out Also i m getting following in hypervisor logs Mar 14 06:04:30 xenserver-1 scripts-vif: Called as add vif domid:2 devid:0 mode:bridge Mar 14 06:04:30 xenserver-1 scripts-vif: Called as online vif domid:2 devid:0 mode:bridge Mar 14 06:04:30 xenserver-1 scripts-vif: Setting vif2.0 MTU 1500 Mar 14 06:04:30 xenserver-1 scripts-vif: Adding vif2.0 to xapi0 with address fe:ff:ff:ff:ff:ff Mar 14 06:04:30 xenserver-1 scripts-vif: Failed to ip link set vif2.0 address fe:ff:ff:ff:ff:ff Mar 14 06:04:30 xenserver-1 kernel: [ 2890.509223] device vif2.0 entered promiscuous mode Mar 14 06:04:30 xenserver-1 python: /opt/xensource/libexec/setup-vif-rules[23804] - Called with vif_type=vif, domid=2, devid=0, network_mode=bridge, action=filter Mar 14 06:04:30 xenserver-1 python: /opt/xensource/libexec/setup-vif-rules[23804] - attempting to acquire lock /var/lock/ebtables.lock Mar 14 06:04:30 xenserver-1 python: /opt/xensource/libexec/setup-vif-rules[23804] - acquired lock /var/lock/ebtables.lock Mar 14 06:04:30 xenserver-1 python: /opt/xensource/libexec/setup-vif-rules[23804] - ['/sbin/ip', 'link', 'set', 'vif2.0', 'down'] Mar 14 06:04:30 xenserver-1 python: /opt/xensource/libexec/setup-vif-rules[23804] - ['/sbin/ebtables', '-L', 'FORWARD_vif2.0'] Mar 14 06:04:30 xenserver-1 python: /opt/xensource/libexec/setup-vif-rules[23804] - ['/usr/bin/xenstore-read', '/local/domain/0/backend/vif/2/0/mac'] Mar 14 06:04:30 xenserver-1 python: /opt/xensource/libexec/setup-vif-rules[23804] - ['/usr/bin/xenstore-read', '/xapi/2/private/vif/0/locking-mode'] Mar 14 06:04:30 xenserver-1 python: /opt/xensource/libexec/setup-vif-rules[23804] - ['/usr/bin/xenstore-read', '/xapi/2/private/vif/0/ipv4-allowed'] Mar 14 06:04:30 xenserver-1 python: /opt/xensource/libexec/setup-vif-rules[23804] - ['/usr/bin/xenstore-read', '/xapi/2/private/vif/0/ipv6-allowed'] Mar 14 06:04:30 xenserver-1 python: /opt/xensource/libexec/setup-vif-rules[23804] - Got locking config: MAC=0e:00:a9:fe:00:68; locking_mode=unlocked; ipv4_allowed=; ipv6_ allowed=
Re: Unable to create destination path: /opt/cloud/bin on
Hi, Cloudstack is running with root privileges, hypervisor (17.32) is up and running when following log entries appear in management server. later on heprvisor reboots after NFS timed out issue, Umair Azam On 3/11/2014 12:41 PM, sandeep khandekar wrote: Run with root user privilege Check the system 10.11.17.32 is up and running It is clear from the above output that on 10.11.17.32 it can not create the path On Tue, Mar 11, 2014 at 4:13 AM, Umair Azam ua...@i2cinc.com wrote: Hi, Does anybody know why i m getting this exception in management server logs, I think its related to vhd-util, but cant guess why its not being copied. 2014-03-11 08:05:47,985 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /etc/xapi.d/plugins on 10.11.17.32 but trying anyway 2014-03-11 08:05:48,092 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway 2014-03-11 08:05:48,338 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /etc/xapi.d/plugins on 10.11.17.32 but trying anyway 2014-03-11 08:05:48,675 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway 2014-03-11 08:05:48,759 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway 2014-03-11 08:05:48,843 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/xensource/packages/iso on 10.11.17.32 but trying anyway 2014-03-11 08:05:52,854 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway 2014-03-11 08:05:52,939 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /etc/xapi.d/plugins on 10.11.17.32 but trying anyway 2014-03-11 08:05:53,277 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /etc/xensource on 10.11.17.32 but trying anyway 2014-03-11 08:05:53,360 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway 2014-03-11 08:05:54,119 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /root/.ssh on 10.11.17.32 but trying anyway 2014-03-11 08:05:54,210 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway 2014-03-11 08:05:54,921 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /etc/xapi.d/plugins on 10.11.17.32 but trying anyway 2014-03-11 08:05:55,010 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/xensource/sm on 10.11.17.32 but trying anyway 2014-03-11 08:05:55,095 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway -- Umair Azam
Re: NFS machine not responding, timed out issue
Hi Carlos, I see following management servers logs, what could be the possible reasons for these exceptions. 2014-03-11 08:05:47,985 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /etc/xapi.d/plugins on 10.11.17.32 but trying anyway 2014-03-11 08:05:48,092 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway 2014-03-11 08:05:48,338 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /etc/xapi.d/plugins on 10.11.17.32 but trying anyway 2014-03-11 08:05:48,675 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway 2014-03-11 08:05:48,759 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway 2014-03-11 08:05:48,843 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/xensource/packages/iso on 10.11.17.32 but trying anyway 2014-03-11 08:05:52,854 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /opt/cloud/bin on 10.11.17.32 but trying anyway 2014-03-11 08:05:52,939 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-1:null) Unable to create destination path: /etc/xapi.d/plugins on 10.11.17.32 but trying anyway Umair Azam On 2/13/2014 9:22 AM, Carlos Reategui wrote: If you had it on the management server then it would have been copied over to the hosts when you added them and there is no need for you to do it now. On Feb 12, 2014, at 5:46 PM, Umair Azam ua...@i2cinc.com wrote: Hi, Carlos thanks for the follow up I really appreciate that, Well i have got the vhd-util placed in /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/vhd-util on managment server but missed to place it on XS host in /opt/xensource/bin. I probably missed or couldn't find anything in cloudstack installation guide to place vhd-util in Xen hosts at /opt/xensource/bin. I will do that right away and see if it works or not. Thanks again Umair Azam On 2/13/2014 6:36 AM, Carlos Reategui wrote: Hi Umair (sorry missed the 'i' last time) Haven't been able to digest your logs but one common problem with XS is forgetting to get the specific vhd-util mentioned in the docs: section 4.5.3.3 of https://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/management-server-install-flow.html If you missed this, you can manually copy that vhd-util to the XS hosts to /opt/xensource/bin/ Also if you are using basic network, don't forget to switch the XS hosts to bridge (xe-switch-network-backend bridge) Finally one thing that has helped me is to disable iptables on the XS hosts (service iptables stop; chkconfig iptables off) regards, Carlos On Feb 12, 2014, at 5:13 PM, Umair Azam ua...@i2cinc.com wrote: Hi carlos, I have changed switch ports, cables etc, I am successfully able to mount and copy data. Below logs in the thread are latest after this fix. Umair Azam On 2/13/2014 6:03 AM, Carlos Reategui wrote: Hi Umar, Did you verify your network per Shanker's suggestion? You need to make sure the NIC flapping is fixed first before you can expect anything to run stably. When you say you are able to mount it manually, did you also try copying data over that mount? I had a similar problem once and it ended up being a bad cable. Regards, -Carlos On Feb 12, 2014, at 4:09 PM, Umair Azam ua...@i2cinc.com wrote: Thanks for the follow up shanker. Please go through below details and help me dig out the issue, I am using cloudstack 4.2.1 installed through yum and xenserver 6.2, I am getting very strange errors and i have no idea whats going on. Managment server IP: 10.11.17.30 Primary/secondar Storage IP: 10.11.17.31 PoD IP range 10.11.17.100 to 10.11.17.112 (for system VM's) Guest IP range: 10.11.17.113 to 10.11.17.120 Primary storage path: /export/primary Secondary storage IP: /export/secondary *logs from the management server* (I know i have posted lengthy logs but it would of great help if u help to make me understand the issue: 2014-02-13 09:38:35,497 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-13:null) *Trying to connect to 169.254.1.149* 2014-02-13 09:38:35,542 DEBUG [cloud.api.ApiServlet] (catalina-exec-21:null) ===START=== 10.11.17.22 -- GET command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248334219 2014-02-13 09:38:35,611 DEBUG [cloud.api.ApiServlet] (catalina-exec-21:null) ===END=== 10.11.17.22 -- GET command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248334219 2014-02-13 09:38:38,931 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) StorageCollector is running... 2014-02-13 09:38:38,934 INFO
Re: NFS machine not responding, timed out issue
Hi, Finally After replacing machines NIC's/cables i have figured out the exact issue, It is related to nfs request timed out, this issue has been mentioned in 6.0 release notes but i am using xenserver 6.2 and still having the same problem, somebody can help me out ? Mar 11 02:49:05 xenserver-1 kernel: [ 1848.148548] nfs: server 10.11.17.33 not responding, timed out * In some 10 Gigabit Ethernet environments, occasional performance problems with disk throughput on NFS SRs have been observed. The problem can be identified by a log entry in/var/log/messagessimilar to:kernel: nfs: server 10.0.0.1 not responding, timed out. Citrix continues to investigate this issue with an aim to resolve it in a future release. [CA-59187] http://support.citrix.com/article/CTX130418 Umair Azam On 2/13/2014 9:31 AM, Umair Azam wrote: I have been so much frustrated, couldn't find any solution and unable to resolve the problem, I can mount primary secondary storage manually, copy data also, but hypervisor logs show me nfs server not responding and timed out heartbeat: Problem with heartbeat, no iSCSI or NFS mount defined in /opt/cloud/bin/heartbeat! Umair Azam On 2/13/2014 9:22 AM, Carlos Reategui wrote: If you had it on the management server then it would have been copied over to the hosts when you added them and there is no need for you to do it now. On Feb 12, 2014, at 5:46 PM, Umair Azam ua...@i2cinc.com wrote: Hi, Carlos thanks for the follow up I really appreciate that, Well i have got the vhd-util placed in /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/vhd-util on managment server but missed to place it on XS host in /opt/xensource/bin. I probably missed or couldn't find anything in cloudstack installation guide to place vhd-util in Xen hosts at /opt/xensource/bin. I will do that right away and see if it works or not. Thanks again Umair Azam On 2/13/2014 6:36 AM, Carlos Reategui wrote: Hi Umair (sorry missed the 'i' last time) Haven't been able to digest your logs but one common problem with XS is forgetting to get the specific vhd-util mentioned in the docs: section 4.5.3.3 of https://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/management-server-install-flow.html If you missed this, you can manually copy that vhd-util to the XS hosts to /opt/xensource/bin/ Also if you are using basic network, don't forget to switch the XS hosts to bridge (xe-switch-network-backend bridge) Finally one thing that has helped me is to disable iptables on the XS hosts (service iptables stop; chkconfig iptables off) regards, Carlos On Feb 12, 2014, at 5:13 PM, Umair Azam ua...@i2cinc.com wrote: Hi carlos, I have changed switch ports, cables etc, I am successfully able to mount and copy data. Below logs in the thread are latest after this fix. Umair Azam On 2/13/2014 6:03 AM, Carlos Reategui wrote: Hi Umar, Did you verify your network per Shanker's suggestion? You need to make sure the NIC flapping is fixed first before you can expect anything to run stably. When you say you are able to mount it manually, did you also try copying data over that mount? I had a similar problem once and it ended up being a bad cable. Regards, -Carlos On Feb 12, 2014, at 4:09 PM, Umair Azam ua...@i2cinc.com wrote: Thanks for the follow up shanker. Please go through below details and help me dig out the issue, I am using cloudstack 4.2.1 installed through yum and xenserver 6.2, I am getting very strange errors and i have no idea whats going on. Managment server IP: 10.11.17.30 Primary/secondar Storage IP: 10.11.17.31 PoD IP range 10.11.17.100 to 10.11.17.112 (for system VM's) Guest IP range: 10.11.17.113 to 10.11.17.120 Primary storage path: /export/primary Secondary storage IP: /export/secondary *logs from the management server* (I know i have posted lengthy logs but it would of great help if u help to make me understand the issue: 2014-02-13 09:38:35,497 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-13:null) *Trying to connect to 169.254.1.149* 2014-02-13 09:38:35,542 DEBUG [cloud.api.ApiServlet] (catalina-exec-21:null) ===START=== 10.11.17.22 -- GET command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248334219 2014-02-13 09:38:35,611 DEBUG [cloud.api.ApiServlet] (catalina-exec-21:null) ===END=== 10.11.17.22 -- GET command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248334219 2014-02-13 09:38:38,931 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) StorageCollector is running... 2014-02-13 09:38:38,934 INFO [storage.endpoint.DefaultEndPointSelector] (StatsCollector-1:null) *No running ssvm is found, so command will be sent to LocalHostEndPoint* 2014-02-13 09:38:38,939 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-37:null) Seq 1-535822363: Executing request 2014-02-13 09:38:39,184 DEBUG
Re: Unable to create slave connection to host [Machine gets rebooted]
messages in last 5 seconds (most recently, 5 seconds ago) due to excessive rate Feb 13 04:36:26 xenserver-Host1 ovs-vswitchd: 00226|netdev_linux|WARN|ioctl(SIOCGIFINDEX) *on vif1.1 device failed: No such device** *Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:xs-vm-uuid=3ee2bf12-9ba0-fbc1-4789-c7fd9fc01194 Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:xs-vif-uuid=9779edb4-0ca0-7337-71ad-480ea5fcef9d Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:xs-network-uuid=33831c9a-b2e2-4478-21b7-697da4f9cc4e Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:attached-mac=0e:00:a9:fe:01:95 * * Feb 13 04:36:38 xenserver-Host1 scripts-vif: Called as online vif domid:2 devid:1 mode:openvswitch Feb 13 04:36:38 xenserver-Host1 scripts-vif: Setting vif2.1 MTU 1500 Feb 13 04:36:38 xenserver-Host1 scripts-vif: Adding vif2.1 to xenbr0 with address fe:ff:ff:ff:ff:ff Feb 13 04:36:38 xenserver-Host1 scripts-vif: F*ailed to ip link set vif2.1 address fe:ff:ff:ff:ff:ff* Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.1 external-ids:xs-vm-uuid=3ee2bf12-9ba0-fbc1-4789-c7fd9fc01194 Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.1 external-ids:xs-vif-uuid=5b9f0df6-2756-1aaf-690e-249df678a240 Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.1 external-ids:xs-network-uuid=a67b6e0f-6452-7474-c850-f1ef215d3941 Feb 13 04:36:39 xenserver-Host1 scripts-vif: Called as add vif domid:2 devid:2 mode:openvswitch Feb 13 04:36:39 xenserver-Host1 scripts-vif: Called as online vif domid:2 devid:2 mode:openvswitch Feb 13 04:36:39 xenserver-Host1 scripts-vif: Setting vif2.2 MTU 1500 Feb 13 04:36:39 xenserver-Host1 scripts-vif: Adding vif2.2 to xenbr0 with address fe:ff:ff:ff:ff:ff Feb 13 04:36:39 xenserver-Host1 scripts-vif: *Failed to ip link set vif2.2 address fe:ff:ff:ff:ff:ff* Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:xs-vm-uuid=3ee2bf12-9ba0-fbc1-4789-c7fd9fc01194 Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:xs-vif-uuid=7c4e02c7-658c-4f1f-ea00-d585f052b9c9 Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:xs-network-uuid=a67b6e0f-6452-7474-c850-f1ef215d3941 Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:attached-mac=06:2b:90:00:00:0f Feb 13 04:37:43 xenserver-Host1 scripts-vif: Adding vif3.2 to xenbr0 with address fe:ff:ff:ff:ff:ff Feb 13 04:37:43 xenserver-Host1 scripts-vif: Failed to ip link set vif3.2 address fe:ff:ff:ff:ff:ff Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:xs-vm-uuid=a0ae11c8-af19-1acb-4998-a7de1457a886 Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:xs-vif-uuid=f61262e1-f848-25e2-6c74-338deb0efe59 Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:xs-network-uuid=a67b6e0f-6452-7474-c850-f1ef215d3941 Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:attached-mac=06:e8:4c:00:00:0e Feb 13 04:38:24 xenserver-Host1 xapi: [ info|xenserver-Host1|4856 UNIX /var/xapi/xapi|session.login_with_password D:29f302c62711|xapi] Session.create trackid=7351c4bbf9b985953add83ed4fb5dfe3 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Feb 13 04:38:24 xenserver-Host1 xapi: [ info|xenserver-Host1|4841 INET 0.0.0.0:80|sm_exec D:e58e39494b07|xapi] Session.destroy trackid=a4921a1edef9d2a3f8442836d620e900 Feb 13 04:38:24 xenserver-Host1 xapi: [ info|xenserver-Host1|4874 UNIX /var/xapi/xapi|session.logout D:6f611c2117d0|xapi] Session.destroy trackid=7351c4bbf9b985953add83ed4fb5dfe3 After this hypervisor reboots (reboots might be due to HA enabled) * **logs from the primary/secondary storage:* Feb 13 04:08:30 secondary-storage rpc.idmapd[1767]: nss_getpwnam: name '0' does not map into domain 'localdomain' Feb 13 04:30:29 secondary-storage rpc.mountd[1729]: authenticated mount request from 10.11.17.32:679 for /export/primary (/export) Feb 13 04:30:29 secondary-storage rpc.mountd[1729]: authenticated unmount request from 10.11.17.32:686 for /export/primary (/export) Feb 13 04:30:29 secondary-storage rpc.mountd[1729]: authenticated mount request from 10.11.17.32:710 for /export/primary (/export) Feb 13 04:30:49 secondary-storage rpc.mountd[1729]: authenticated mount request from 10.11.17.32:1007 for /export/secondary/template/tmpl/1/ Umair Azam On 2/12/2014 3:01 PM, sebgoa wrote: Hi Umair, Can you give specifics on which version of cloudstack you are using ? Are you using the packages or compiling from source ? Which version of Xenserver are you using ? -Sebastien On Feb 11, 2014, at 4:29 AM, Umair Azam ua...@i2cinc.com wrote: Is there anybody who can help me out on this issue, I m stuck here since 2weeks, tried re-installation several times, changed machines but still getting this issue :( Umair Azam On 1/27/2014 10:48 PM, Umair Azam wrote: Hi, I am new to cloudstack and testing, evaluating it for future deployment, I
Re: NFS machine not responding, timed out issue
: No such devic*e Feb 13 04:36:26 xenserver-Host1 ovs-vswitchd: 00225|netdev_linux|WARN|Dropped 8 log messages in last 5 seconds (most recently, 5 seconds ago) due to excessive rate Feb 13 04:36:26 xenserver-Host1 ovs-vswitchd: 00226|netdev_linux|WARN|ioctl(SIOCGIFINDEX) *on vif1.1 device failed: No such device** *Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:xs-vm-uuid=3ee2bf12-9ba0-fbc1-4789-c7fd9fc01194 Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:xs-vif-uuid=9779edb4-0ca0-7337-71ad-480ea5fcef9d Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:xs-network-uuid=33831c9a-b2e2-4478-21b7-697da4f9cc4e Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:attached-mac=0e:00:a9:fe:01:95 * * Feb 13 04:36:38 xenserver-Host1 scripts-vif: Called as online vif domid:2 devid:1 mode:openvswitch Feb 13 04:36:38 xenserver-Host1 scripts-vif: Setting vif2.1 MTU 1500 Feb 13 04:36:38 xenserver-Host1 scripts-vif: Adding vif2.1 to xenbr0 with address fe:ff:ff:ff:ff:ff Feb 13 04:36:38 xenserver-Host1 scripts-vif: F*ailed to ip link set vif2.1 address fe:ff:ff:ff:ff:ff* Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.1 external-ids:xs-vm-uuid=3ee2bf12-9ba0-fbc1-4789-c7fd9fc01194 Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.1 external-ids:xs-vif-uuid=5b9f0df6-2756-1aaf-690e-249df678a240 Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.1 external-ids:xs-network-uuid=a67b6e0f-6452-7474-c850-f1ef215d3941 Feb 13 04:36:39 xenserver-Host1 scripts-vif: Called as add vif domid:2 devid:2 mode:openvswitch Feb 13 04:36:39 xenserver-Host1 scripts-vif: Called as online vif domid:2 devid:2 mode:openvswitch Feb 13 04:36:39 xenserver-Host1 scripts-vif: Setting vif2.2 MTU 1500 Feb 13 04:36:39 xenserver-Host1 scripts-vif: Adding vif2.2 to xenbr0 with address fe:ff:ff:ff:ff:ff Feb 13 04:36:39 xenserver-Host1 scripts-vif: *Failed to ip link set vif2.2 address fe:ff:ff:ff:ff:ff* Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:xs-vm-uuid=3ee2bf12-9ba0-fbc1-4789-c7fd9fc01194 Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:xs-vif-uuid=7c4e02c7-658c-4f1f-ea00-d585f052b9c9 Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:xs-network-uuid=a67b6e0f-6452-7474-c850-f1ef215d3941 Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:attached-mac=06:2b:90:00:00:0f Feb 13 04:37:43 xenserver-Host1 scripts-vif: Adding vif3.2 to xenbr0 with address fe:ff:ff:ff:ff:ff Feb 13 04:37:43 xenserver-Host1 scripts-vif: Failed to ip link set vif3.2 address fe:ff:ff:ff:ff:ff Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:xs-vm-uuid=a0ae11c8-af19-1acb-4998-a7de1457a886 Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:xs-vif-uuid=f61262e1-f848-25e2-6c74-338deb0efe59 Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:xs-network-uuid=a67b6e0f-6452-7474-c850-f1ef215d3941 Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:attached-mac=06:e8:4c:00:00:0e Feb 13 04:38:24 xenserver-Host1 xapi: [ info|xenserver-Host1|4856 UNIX /var/xapi/xapi|session.login_with_password D:29f302c62711|xapi] Session.create trackid=7351c4bbf9b985953add83ed4fb5dfe3 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Feb 13 04:38:24 xenserver-Host1 xapi: [ info|xenserver-Host1|4841 INET 0.0.0.0:80|sm_exec D:e58e39494b07|xapi] Session.destroy trackid=a4921a1edef9d2a3f8442836d620e900 Feb 13 04:38:24 xenserver-Host1 xapi: [ info|xenserver-Host1|4874 UNIX /var/xapi/xapi|session.logout D:6f611c2117d0|xapi] Session.destroy trackid=7351c4bbf9b985953add83ed4fb5dfe3 After this hypervisor reboots (reboots might be due to HA enabled) * **logs from the primary/secondary storage:* Feb 13 04:08:30 secondary-storage rpc.idmapd[1767]: nss_getpwnam: name '0' does not map into domain 'localdomain' Feb 13 04:30:29 secondary-storage rpc.mountd[1729]: authenticated mount request from 10.11.17.32:679 for /export/primary (/export) Feb 13 04:30:29 secondary-storage rpc.mountd[1729]: authenticated unmount request from 10.11.17.32:686 for /export/primary (/export) Feb 13 04:30:29 secondary-storage rpc.mountd[1729]: authenticated mount request from 10.11.17.32:710 for /export/primary (/export) Feb 13 04:30:49 secondary-storage rpc.mountd[1729]: authenticated mount request from 10.11.17.32:1007 for /export/secondary/template/tmpl/1/ Umair Azam Associate Systems Administrator Network Operations Center i2c Incorporated 1300 Island Drive, Suite 105 Redwood City, CA 94065-5170 Desk: +1 650.480.5291 PBX: +1 650.593.5400 x 4244 24x7 NOC: +1 650.480.5291 Fax: +1 650.593.5402 URL: www.i2cinc.com ** CONFIDENTIALITY CAUTION This communication (including any accompanying documents) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading
Re: Unable to create slave connection to host [Machine gets rebooted]
|WARN|Dropped 8 log messages in last 5 seconds (most recently, 5 seconds ago) due to excessive rate Feb 13 04:36:26 xenserver-Host1 ovs-vswitchd: 00226|netdev_linux|WARN|ioctl(SIOCGIFINDEX) *on vif1.1 device failed: No such device** *Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:xs-vm-uuid=3ee2bf12-9ba0-fbc1-4789-c7fd9fc01194 Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:xs-vif-uuid=9779edb4-0ca0-7337-71ad-480ea5fcef9d Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:xs-network-uuid=33831c9a-b2e2-4478-21b7-697da4f9cc4e Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.0 external-ids:attached-mac=0e:00:a9:fe:01:95 * * Feb 13 04:36:38 xenserver-Host1 scripts-vif: Called as online vif domid:2 devid:1 mode:openvswitch Feb 13 04:36:38 xenserver-Host1 scripts-vif: Setting vif2.1 MTU 1500 Feb 13 04:36:38 xenserver-Host1 scripts-vif: Adding vif2.1 to xenbr0 with address fe:ff:ff:ff:ff:ff Feb 13 04:36:38 xenserver-Host1 scripts-vif: F*ailed to ip link set vif2.1 address fe:ff:ff:ff:ff:ff* Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.1 external-ids:xs-vm-uuid=3ee2bf12-9ba0-fbc1-4789-c7fd9fc01194 Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.1 external-ids:xs-vif-uuid=5b9f0df6-2756-1aaf-690e-249df678a240 Feb 13 04:36:38 xenserver-Host1 scripts-vif: vif2.1 external-ids:xs-network-uuid=a67b6e0f-6452-7474-c850-f1ef215d3941 Feb 13 04:36:39 xenserver-Host1 scripts-vif: Called as add vif domid:2 devid:2 mode:openvswitch Feb 13 04:36:39 xenserver-Host1 scripts-vif: Called as online vif domid:2 devid:2 mode:openvswitch Feb 13 04:36:39 xenserver-Host1 scripts-vif: Setting vif2.2 MTU 1500 Feb 13 04:36:39 xenserver-Host1 scripts-vif: Adding vif2.2 to xenbr0 with address fe:ff:ff:ff:ff:ff Feb 13 04:36:39 xenserver-Host1 scripts-vif: *Failed to ip link set vif2.2 address fe:ff:ff:ff:ff:ff* Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:xs-vm-uuid=3ee2bf12-9ba0-fbc1-4789-c7fd9fc01194 Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:xs-vif-uuid=7c4e02c7-658c-4f1f-ea00-d585f052b9c9 Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:xs-network-uuid=a67b6e0f-6452-7474-c850-f1ef215d3941 Feb 13 04:36:39 xenserver-Host1 scripts-vif: vif2.2 external-ids:attached-mac=06:2b:90:00:00:0f Feb 13 04:37:43 xenserver-Host1 scripts-vif: Adding vif3.2 to xenbr0 with address fe:ff:ff:ff:ff:ff Feb 13 04:37:43 xenserver-Host1 scripts-vif: Failed to ip link set vif3.2 address fe:ff:ff:ff:ff:ff Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:xs-vm-uuid=a0ae11c8-af19-1acb-4998-a7de1457a886 Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:xs-vif-uuid=f61262e1-f848-25e2-6c74-338deb0efe59 Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:xs-network-uuid=a67b6e0f-6452-7474-c850-f1ef215d3941 Feb 13 04:37:43 xenserver-Host1 scripts-vif: vif3.2 external-ids:attached-mac=06:e8:4c:00:00:0e Feb 13 04:38:24 xenserver-Host1 xapi: [ info|xenserver-Host1|4856 UNIX /var/xapi/xapi|session.login_with_password D:29f302c62711|xapi] Session.create trackid=7351c4bbf9b985953add83ed4fb5dfe3 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Feb 13 04:38:24 xenserver-Host1 xapi: [ info|xenserver-Host1|4841 INET 0.0.0.0:80|sm_exec D:e58e39494b07|xapi] Session.destroy trackid=a4921a1edef9d2a3f8442836d620e900 Feb 13 04:38:24 xenserver-Host1 xapi: [ info|xenserver-Host1|4874 UNIX /var/xapi/xapi|session.logout D:6f611c2117d0|xapi] Session.destroy trackid=7351c4bbf9b985953add83ed4fb5dfe3 After this hypervisor reboots (reboots might be due to HA enabled) * **logs from the primary/secondary storage:* Feb 13 04:08:30 secondary-storage rpc.idmapd[1767]: nss_getpwnam: name '0' does not map into domain 'localdomain' Feb 13 04:30:29 secondary-storage rpc.mountd[1729]: authenticated mount request from 10.11.17.32:679 for /export/primary (/export) Feb 13 04:30:29 secondary-storage rpc.mountd[1729]: authenticated unmount request from 10.11.17.32:686 for /export/primary (/export) Feb 13 04:30:29 secondary-storage rpc.mountd[1729]: authenticated mount request from 10.11.17.32:710 for /export/primary (/export) Feb 13 04:30:49 secondary-storage rpc.mountd[1729]: authenticated mount request from 10.11.17.32:1007 for /export/secondary/template/tmpl/1/ Umair Azam On 2/12/2014 4:02 PM, Geoff Higginbottom wrote: Hi Umair, Can you also provide more info about your environment, as the logs indicate its an issue related to the communication between your management server and the host. Other things to check are the following global settings: 1. host - IP of the CloudStack Management Server (if it has more than 1 interface it might have picked up the wrong IP) 2. management.network.cidr - is the XenServer management IP is this range ? Regards Geoff Higginbottom D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
Re: NFS machine not responding, timed out issue
Hi carlos,
I have changed switch ports, cables etc, I am successfully able to mount
and copy data. Below logs in the thread are latest after this fix.
Umair Azam
On 2/13/2014 6:03 AM, Carlos Reategui wrote:
Hi Umar,
Did you verify your network per Shanker's suggestion? You need to make
sure the NIC flapping is fixed first before you can expect anything to run
stably. When you say you are able to mount it manually, did you also try
copying data over that mount? I had a similar problem once and it ended up
being a bad cable.
Regards,
-Carlos
On Feb 12, 2014, at 4:09 PM, Umair Azam ua...@i2cinc.com wrote:
Thanks for the follow up shanker. Please go through below details and help
me dig out the issue,
I am using cloudstack 4.2.1 installed through yum and xenserver 6.2, I am
getting very strange errors and i have no idea whats going on.
Managment server IP: 10.11.17.30
Primary/secondar Storage IP: 10.11.17.31
PoD IP range 10.11.17.100 to 10.11.17.112 (for system VM's)
Guest IP range: 10.11.17.113 to 10.11.17.120
Primary storage path: /export/primary
Secondary storage IP: /export/secondary
*logs from the management server* (I know i have posted lengthy logs but it
would of great help if u help to make me understand the issue:
2014-02-13 09:38:35,497 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-13:null) *Trying to connect to 169.254.1.149*
2014-02-13 09:38:35,542 DEBUG [cloud.api.ApiServlet]
(catalina-exec-21:null) ===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248334219
2014-02-13 09:38:35,611 DEBUG [cloud.api.ApiServlet]
(catalina-exec-21:null) ===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248334219
2014-02-13 09:38:38,931 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) StorageCollector is running...
2014-02-13 09:38:38,934 INFO [storage.endpoint.DefaultEndPointSelector]
(StatsCollector-1:null) *No running ssvm is found, so command will be sent
to LocalHostEndPoint*
2014-02-13 09:38:38,939 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-37:null) Seq 1-535822363: Executing request
2014-02-13 09:38:39,184 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-37:null) Seq 1-535822363: Response Received:
2014-02-13 09:38:39,184 DEBUG [agent.transport.Request]
(StatsCollector-1:null) Seq 1-535822363: Received: { Ans: , MgmtId:
130189987379, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-02-13 09:38:40,555 DEBUG [cloud.api.ApiServlet]
(catalina-exec-25:null) ===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248339219
2014-02-13 09:38:40,624 DEBUG [cloud.api.ApiServlet]
(catalina-exec-25:null) ===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248339219
2014-02-13 09:38:45,542 DEBUG [cloud.api.ApiServlet]
(catalina-exec-24:null) ===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248344219
2014-02-13 09:38:45,609 DEBUG [cloud.api.ApiServlet]
(catalina-exec-24:null) ===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248344219
2014-02-13 09:38:48,646 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-13:null) *Trying to connect to 169.254.1.149*
2014-02-13 09:38:50,542 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null)
===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248349219
2014-02-13 09:38:50,609 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null)
===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248349219
2014-02-13 09:38:55,542 DEBUG [cloud.api.ApiServlet]
(catalina-exec-23:null) ===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248354219
2014-02-13 09:38:55,608 DEBUG [cloud.api.ApiServlet]
(catalina-exec-23:null) ===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248354219
2014-02-13 09:39:00,542 DEBUG [cloud.api.ApiServlet] (catalina-exec-3:null)
===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248359215
2014-02-13 09:39:00,609 DEBUG [cloud.api.ApiServlet] (catalina-exec-3:null)
===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248359215
2014-02-13 09:39:01,875 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-13:null) *Trying to connect to 169.254.1.149*
2014-02-13 09:39:03,284 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 routers to update status.
2014-02-13 09:39:03,285 DEBUG
Re: NFS machine not responding, timed out issue
Hi,
Carlos thanks for the follow up I really appreciate that, Well i have
got the vhd-util placed in
/usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/vhd-util on
managment server but missed to place it on XS host in
/opt/xensource/bin. I probably missed or couldn't find anything in
cloudstack installation guide to place vhd-util in Xen hosts at
/opt/xensource/bin.
I will do that right away and see if it works or not. Thanks again
Umair Azam
On 2/13/2014 6:36 AM, Carlos Reategui wrote:
Hi Umair (sorry missed the 'i' last time)
Haven't been able to digest your logs but one common problem with XS is
forgetting to get the specific vhd-util mentioned in the docs:
section 4.5.3.3 of
https://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/management-server-install-flow.html
If you missed this, you can manually copy that vhd-util to the XS hosts
to /opt/xensource/bin/
Also if you are using basic network, don't forget to switch the XS hosts to
bridge (xe-switch-network-backend bridge)
Finally one thing that has helped me is to disable iptables on the XS hosts
(service iptables stop; chkconfig iptables off)
regards,
Carlos
On Feb 12, 2014, at 5:13 PM, Umair Azam ua...@i2cinc.com wrote:
Hi carlos,
I have changed switch ports, cables etc, I am successfully able to mount
and copy data. Below logs in the thread are latest after this fix.
Umair Azam
On 2/13/2014 6:03 AM, Carlos Reategui wrote:
Hi Umar,
Did you verify your network per Shanker's suggestion? You need to make
sure the NIC flapping is fixed first before you can expect anything to run
stably. When you say you are able to mount it manually, did you also try
copying data over that mount? I had a similar problem once and it ended up
being a bad cable.
Regards,
-Carlos
On Feb 12, 2014, at 4:09 PM, Umair Azam ua...@i2cinc.com wrote:
Thanks for the follow up shanker. Please go through below details and help
me dig out the issue,
I am using cloudstack 4.2.1 installed through yum and xenserver 6.2, I am
getting very strange errors and i have no idea whats going on.
Managment server IP: 10.11.17.30
Primary/secondar Storage IP: 10.11.17.31
PoD IP range 10.11.17.100 to 10.11.17.112 (for system VM's)
Guest IP range: 10.11.17.113 to 10.11.17.120
Primary storage path: /export/primary
Secondary storage IP: /export/secondary
*logs from the management server* (I know i have posted lengthy logs but it
would of great help if u help to make me understand the issue:
2014-02-13 09:38:35,497 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-13:null) *Trying to connect to 169.254.1.149*
2014-02-13 09:38:35,542 DEBUG [cloud.api.ApiServlet]
(catalina-exec-21:null) ===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248334219
2014-02-13 09:38:35,611 DEBUG [cloud.api.ApiServlet]
(catalina-exec-21:null) ===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248334219
2014-02-13 09:38:38,931 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) StorageCollector is running...
2014-02-13 09:38:38,934 INFO [storage.endpoint.DefaultEndPointSelector]
(StatsCollector-1:null) *No running ssvm is found, so command will be sent
to LocalHostEndPoint*
2014-02-13 09:38:38,939 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-37:null) Seq 1-535822363: Executing request
2014-02-13 09:38:39,184 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-37:null) Seq 1-535822363: Response Received:
2014-02-13 09:38:39,184 DEBUG [agent.transport.Request]
(StatsCollector-1:null) Seq 1-535822363: Received: { Ans: , MgmtId:
130189987379, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-02-13 09:38:40,555 DEBUG [cloud.api.ApiServlet]
(catalina-exec-25:null) ===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248339219
2014-02-13 09:38:40,624 DEBUG [cloud.api.ApiServlet]
(catalina-exec-25:null) ===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248339219
2014-02-13 09:38:45,542 DEBUG [cloud.api.ApiServlet]
(catalina-exec-24:null) ===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248344219
2014-02-13 09:38:45,609 DEBUG [cloud.api.ApiServlet]
(catalina-exec-24:null) ===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248344219
2014-02-13 09:38:48,646 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-13:null) *Trying to connect to 169.254.1.149*
2014-02-13 09:38:50,542 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null)
===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248349219
2014-02-13 09:38:50,609 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null)
===END
Re: NFS machine not responding, timed out issue
I have been so much frustrated, couldn't find any solution and unable to
resolve the problem, I can mount primary secondary storage manually,
copy data also, but hypervisor logs show me nfs server not responding
and timed out
heartbeat: Problem with heartbeat, no iSCSI or NFS mount defined in
/opt/cloud/bin/heartbeat!
Umair Azam
On 2/13/2014 9:22 AM, Carlos Reategui wrote:
If you had it on the management server then it would have been copied over to
the hosts when you added them and there is no need for you to do it now.
On Feb 12, 2014, at 5:46 PM, Umair Azam ua...@i2cinc.com wrote:
Hi,
Carlos thanks for the follow up I really appreciate that, Well i have got the
vhd-util placed in
/usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/vhd-util on
managment server but missed to place it on XS host in /opt/xensource/bin. I
probably missed or couldn't find anything in cloudstack installation guide to
place vhd-util in Xen hosts at /opt/xensource/bin.
I will do that right away and see if it works or not. Thanks again
Umair Azam
On 2/13/2014 6:36 AM, Carlos Reategui wrote:
Hi Umair (sorry missed the 'i' last time)
Haven't been able to digest your logs but one common problem with XS is
forgetting to get the specific vhd-util mentioned in the docs:
section 4.5.3.3 of
https://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/management-server-install-flow.html
If you missed this, you can manually copy that vhd-util to the XS hosts
to /opt/xensource/bin/
Also if you are using basic network, don't forget to switch the XS hosts to
bridge (xe-switch-network-backend bridge)
Finally one thing that has helped me is to disable iptables on the XS hosts
(service iptables stop; chkconfig iptables off)
regards,
Carlos
On Feb 12, 2014, at 5:13 PM, Umair Azam ua...@i2cinc.com wrote:
Hi carlos,
I have changed switch ports, cables etc, I am successfully able to mount
and copy data. Below logs in the thread are latest after this fix.
Umair Azam
On 2/13/2014 6:03 AM, Carlos Reategui wrote:
Hi Umar,
Did you verify your network per Shanker's suggestion? You need to make
sure the NIC flapping is fixed first before you can expect anything to run
stably. When you say you are able to mount it manually, did you also try
copying data over that mount? I had a similar problem once and it ended up
being a bad cable.
Regards,
-Carlos
On Feb 12, 2014, at 4:09 PM, Umair Azam ua...@i2cinc.com wrote:
Thanks for the follow up shanker. Please go through below details and help
me dig out the issue,
I am using cloudstack 4.2.1 installed through yum and xenserver 6.2, I am
getting very strange errors and i have no idea whats going on.
Managment server IP: 10.11.17.30
Primary/secondar Storage IP: 10.11.17.31
PoD IP range 10.11.17.100 to 10.11.17.112 (for system VM's)
Guest IP range: 10.11.17.113 to 10.11.17.120
Primary storage path: /export/primary
Secondary storage IP: /export/secondary
*logs from the management server* (I know i have posted lengthy logs but it
would of great help if u help to make me understand the issue:
2014-02-13 09:38:35,497 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-13:null) *Trying to connect to 169.254.1.149*
2014-02-13 09:38:35,542 DEBUG [cloud.api.ApiServlet]
(catalina-exec-21:null) ===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248334219
2014-02-13 09:38:35,611 DEBUG [cloud.api.ApiServlet]
(catalina-exec-21:null) ===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248334219
2014-02-13 09:38:38,931 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) StorageCollector is running...
2014-02-13 09:38:38,934 INFO [storage.endpoint.DefaultEndPointSelector]
(StatsCollector-1:null) *No running ssvm is found, so command will be sent
to LocalHostEndPoint*
2014-02-13 09:38:38,939 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-37:null) Seq 1-535822363: Executing request
2014-02-13 09:38:39,184 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-37:null) Seq 1-535822363: Response Received:
2014-02-13 09:38:39,184 DEBUG [agent.transport.Request]
(StatsCollector-1:null) Seq 1-535822363: Received: { Ans: , MgmtId:
130189987379, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2014-02-13 09:38:40,555 DEBUG [cloud.api.ApiServlet]
(catalina-exec-25:null) ===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248339219
2014-02-13 09:38:40,624 DEBUG [cloud.api.ApiServlet]
(catalina-exec-25:null) ===END=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248339219
2014-02-13 09:38:45,542 DEBUG [cloud.api.ApiServlet]
(catalina-exec-24:null) ===START=== 10.11.17.22 --
GET
command=listSystemVmsresponse=jsonsessionkey=XDoj6Eke%2Brel612mP%2FBXLiaLwk8%3D_=1392248344219
2014-02-13 09
Re: dom0 configuration of Xenserver
Matthew, Thank you so much for detailed information. Umair Azam On 2/11/2014 10:20 AM, Matthew Kaufman wrote: Hello, By default, XenServer automatically tunes the dom0 settings during installation. IIRC, on a 4G system, ~768MB gets allocated to dom0. The defaults should work just fine for you. dom0_mem = 502MB + int(physical_mem * 0.0205) *Table 2.1. Default Dom0 Memory Size* Physical Memory Dom0 Memory 2 GB 543 MB 4 GB 585 MB 8 GB 669 MB 16 GB 837 MB 32 GB 1173 MB 64 GB 1845 MB 128 GB 3188 MB 256 GB 5875 MB 512 GB 11249 MB 1024 GB 21997 MB 2048 GB 43493 MB Hello, Citrix Xen related questions should probably be asked on the Citrix Xen mailing lists :) Actually; His question is regarding dom0 Sizing WITH His Configuration of 4GB RAM. So it shouldn't be on Citirx Xen; it applies to CloudStack hardware builders too or anyone running Apache CloudStack ON physical hardware (I'd imagine most everybody is here on this mailing list). By default, XenServer automatically tunes the dom0 settings during installation. IIRC, on a 4G system, ~768MB gets allocated to dom0 Here, we use the following sizing computation rules: Rule: dom0_mem = 502 + int(physical_mem * 0.0205) You should always dedicate a fixed amount of RAM for Xen dom0. This can be done by specifying dom0_mem=512M,max:512M option for the Xen hypervisor (usually xen.gz) in grub.conf/menu.lst. This makes sure that the initial amount of memory allocated for dom0 is 512 MB (note: Replace this value with the amount of memory you want to allocate to dom0) and leaves the rest of the host system's RAM available for other guests. See this grub.conf example for GRUB1: *Table 2.1. Default Dom0 Memory Size* Physical Memory Dom0 Memory 2 GB 543 MB 4 GB 585 MB 8 GB 669 MB 16 GB 837 MB 32 GB 1173 MB 64 GB 1845 MB 128 GB 3188 MB 256 GB 5875 MB 512 GB 11249 MB 1024 GB 21997 MB 2048 GB 43493 MB http://wiki.xen.org/wiki/Xen_Best_Practices#Xen_dom0_dedicated_memory_and_preventing_dom0_memory_ballooning
NFS machine not responding, timed out issue
I am facing an error while setting up cloud system, following issue arises when management server machine requests hypervisor to launch CP/SS VM's. After this request hypervisor tries to contact primary storage to access storage but that primary storage server doesn't responds, following are the logs from Xenserver hypervisor. Feb 12 04:58:34 xenserver-Host1 kernel: [74147.731174] e1000e: eth0 NIC Link is Down Feb 12 04:58:38 xenserver-Host1 fe: 21450 (/opt/xensource/bin/tapdisk-cache-stats 91ad2c66-de6b-99f1-b4d9-de264fffbb39) exitted with code 1 Feb 12 04:58:43 xenserver-Host1 fe: 21471 (/opt/xensource/bin/tapdisk-cache-stats 91ad2c66-de6b-99f1-b4d9-de264fffbb39) exitted with code 1 Feb 12 04:58:48 xenserver-Host1 tapdisk[20281]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/64efe7d9-2e45-743b-8a7e-8631a2c36185/a30e8119-0965-49db-9e17-9a3429110d06.vhd: op: 2, lsec: 385070, secs: 2, nbytes: 1024, blk: 94, blk_offset: 65687 Feb 12 04:58:48 xenserver-Host1 kernel: [74161.121077] nfs: server 10.11.17.33 not responding, timed out (hypervisor machine reboots itself after that due to HA enabled) Primary storage server logs are below. (centos machines) Feb 12 04:51:00 storage-server rpc.mountd[1751]: authenticated mount request from 10.11.17.32:788 for /export/primary (/export) Feb 12 04:51:00 storage-server rpc.mountd[1751]: authenticated unmount request from 10.11.17.32:801 for /export/primary (/export) Feb 12 04:51:00 storage-server rpc.mountd[1751]: authenticated mount request from 10.11.17.32:830 for /export/primary (/export) Issue seems to be strange because all necessary NFS ports are open in sysconfig/nfs file, iptables service is also off and both machines can ping each other successfully. -- Umair Azam Associate Systems Administrator Network Operations Center i2c Incorporated 1300 Island Drive, Suite 105 Redwood City, CA 94065-5170 Desk: +1 650.480.5291 PBX: +1 650.593.5400 x 4244 24x7 NOC: +1 650.480.5291 Fax: +1 650.593.5402 URL: www.i2cinc.com ** CONFIDENTIALITY CAUTION This communication (including any accompanying documents) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify us immediately by e-mail, telephone or fax and promptly destroy the original communication. Thank you for your cooperation.
Re: NFS machine not responding, timed out issue
Abhisek, Yes i am able to mount manually. I dont whats the problem i m stuck here for last 2 weeks :/ Umair Azam On 2/12/2014 7:22 AM, abhisek basu wrote: Are we able to mount that NFS manually on the Xen Server? Sent from my iPhone On 12 Feb 2014, at 6:31 am, Umair Azam ua...@i2cinc.com wrote: I am facing an error while setting up cloud system, following issue arises when management server machine requests hypervisor to launch CP/SS VM's. After this request hypervisor tries to contact primary storage to access storage but that primary storage server doesn't responds, following are the logs from Xenserver hypervisor. Feb 12 04:58:34 xenserver-Host1 kernel: [74147.731174] e1000e: eth0 NIC Link is Down Feb 12 04:58:38 xenserver-Host1 fe: 21450 (/opt/xensource/bin/tapdisk-cache-stats 91ad2c66-de6b-99f1-b4d9-de264fffbb39) exitted with code 1 Feb 12 04:58:43 xenserver-Host1 fe: 21471 (/opt/xensource/bin/tapdisk-cache-stats 91ad2c66-de6b-99f1-b4d9-de264fffbb39) exitted with code 1 Feb 12 04:58:48 xenserver-Host1 tapdisk[20281]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/64efe7d9-2e45-743b-8a7e-8631a2c36185/a30e8119-0965-49db-9e17-9a3429110d06.vhd: op: 2, lsec: 385070, secs: 2, nbytes: 1024, blk: 94, blk_offset: 65687 Feb 12 04:58:48 xenserver-Host1 kernel: [74161.121077] nfs: server 10.11.17.33 not responding, timed out (hypervisor machine reboots itself after that due to HA enabled) Primary storage server logs are below. (centos machines) Feb 12 04:51:00 storage-server rpc.mountd[1751]: authenticated mount request from 10.11.17.32:788 for /export/primary (/export) Feb 12 04:51:00 storage-server rpc.mountd[1751]: authenticated unmount request from 10.11.17.32:801 for /export/primary (/export) Feb 12 04:51:00 storage-server rpc.mountd[1751]: authenticated mount request from 10.11.17.32:830 for /export/primary (/export) Issue seems to be strange because all necessary NFS ports are open in sysconfig/nfs file, iptables service is also off and both machines can ping each other successfully. -- Umair Azam Associate Systems Administrator Network Operations Center i2c Incorporated 1300 Island Drive, Suite 105 Redwood City, CA 94065-5170 Desk: +1 650.480.5291 PBX: +1 650.593.5400 x 4244 24x7 NOC: +1 650.480.5291 Fax: +1 650.593.5402 URL: www.i2cinc.com ** CONFIDENTIALITY CAUTION This communication (including any accompanying documents) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify us immediately by e-mail, telephone or fax and promptly destroy the original communication. Thank you for your cooperation.
Re: NFS machine not responding, timed out issue
|network_utils] Call '/sbin/sysctl -q -w net.ipv6.conf.eth0.autoconf=0' exited with code 255 [stdout = ''; stderr = 'error: net.ipv6.conf.eth0.autoconf is an unknown key '] /var/log/messages:Feb 12 05:10:31 xenserver-Host1 kernel: [ 191.878293] e1000e: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None /var/log/messages:Feb 12 05:10:31 xenserver-Host1 kernel: [ 191.878401] e1000e :00:19.0: eth0: 10/100 speed: disabling TSO Umair Azam On 2/12/2014 6:00 AM, Umair Azam wrote: I am facing an error while setting up cloud system, following issue arises when management server machine requests hypervisor to launch CP/SS VM's. After this request hypervisor tries to contact primary storage to access storage but that primary storage server doesn't responds, following are the logs from Xenserver hypervisor. Feb 12 04:58:34 xenserver-Host1 kernel: [74147.731174] e1000e: eth0 NIC Link is Down Feb 12 04:58:38 xenserver-Host1 fe: 21450 (/opt/xensource/bin/tapdisk-cache-stats 91ad2c66-de6b-99f1-b4d9-de264fffbb39) exitted with code 1 Feb 12 04:58:43 xenserver-Host1 fe: 21471 (/opt/xensource/bin/tapdisk-cache-stats 91ad2c66-de6b-99f1-b4d9-de264fffbb39) exitted with code 1 Feb 12 04:58:48 xenserver-Host1 tapdisk[20281]: ERROR: errno -5 at vhd_complete: /var/run/sr-mount/64efe7d9-2e45-743b-8a7e-8631a2c36185/a30e8119-0965-49db-9e17-9a3429110d06.vhd: op: 2, lsec: 385070, secs: 2, nbytes: 1024, blk: 94, blk_offset: 65687 Feb 12 04:58:48 xenserver-Host1 kernel: [74161.121077] nfs: server 10.11.17.33 not responding, timed out (hypervisor machine reboots itself after that due to HA enabled) Primary storage server logs are below. (centos machines) Feb 12 04:51:00 storage-server rpc.mountd[1751]: authenticated mount request from 10.11.17.32:788 for /export/primary (/export) Feb 12 04:51:00 storage-server rpc.mountd[1751]: authenticated unmount request from 10.11.17.32:801 for /export/primary (/export) Feb 12 04:51:00 storage-server rpc.mountd[1751]: authenticated mount request from 10.11.17.32:830 for /export/primary (/export) Issue seems to be strange because all necessary NFS ports are open in sysconfig/nfs file, iptables service is also off and both machines can ping each other successfully.
dom0 configuration of Xenserver
Hi, I have x86 Core2Duo machine with 4GB RAM, what should be the dom0 memory cpu configurations for Citrix xen? i want my SSVM and proxy VM to run on hypervisor of these specs. -- Umair Azam
Unable to determine ssl settings for server.xml
Hi, I am getting this error while installing cloudstack on centos 6.2. I have run cloudstack-setup-management manually, does this solve issue or its still a problem ? Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : cloudstack-management-4.2.1-1.el6.x86_64 1/2 ln: creating symbolic link `/usr/share/cloudstack-management/webapps7080/webapps': File exists Unable to determine ssl settings for server.xml, please run cloudstack-setup-management manually Unable to determine ssl settings for tomcat.conf, please run cloudstack-setup-management manually -- Umair Azam Associate Systems Administrator Network Operations Center i2c Incorporated 1300 Island Drive, Suite 105 Redwood City, CA 94065-5170 Desk: +1 650.480.5291 PBX: +1 650.593.5400 x 4244 24x7 NOC: +1 650.480.5291 Fax: +1 650.593.5402 URL: www.i2cinc.com ** CONFIDENTIALITY CAUTION This communication (including any accompanying documents) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify us immediately by e-mail, telephone or fax and promptly destroy the original communication. Thank you for your cooperation.
Re: Unable to create slave connection to host [Machine gets rebooted]
Is there anybody who can help me out on this issue, I m stuck here since 2weeks, tried re-installation several times, changed machines but still getting this issue :( Umair Azam On 1/27/2014 10:48 PM, Umair Azam wrote: Hi, I am new to cloudstack and testing, evaluating it for future deployment, I am Getting following exception while system VM's creation is in process for the first time, After below exception seen in management server logs, host machine becomes unresponsive and reboots automatically. Host Machine specs: core 2 Duo 4GB RAM with Xenserver 2014-01-27 23:51:55,743 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-15:null) Seq 1-798621700: Executing request 2014-01-27 23:51:55,743 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-15:null) Logging on as the slave to 10.11.17.31 2014-01-27 23:51:57,756 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-15:null) Unable to create slave connection to host(da372d8a-bbeb-486d-a85e-06922cac8da4) due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host 2014-01-27 23:51:58,544 DEBUG [cloud.api.ApiServlet] (catalina-exec-11:null) ===START=== 10.11.17.22 -- GET command=listSystemVmsresponse=jsonsessionkey=k7%2B%2FaPitbX17ZzD9gfUWajyijo8%3D_=1390830753664 2014-01-27 23:51:58,610 DEBUG [cloud.api.ApiServlet] (catalina-exec-11:null) ===END=== 10.11.17.22 -- GET command=listSystemVmsresponse=jsonsessionkey=k7%2B%2FaPitbX17ZzD9gfUWajyijo8%3D_=1390830753664 2014-01-27 23:52:00,757 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-15:null) localLogout has problem Failed to read server's response: No route to host 2014-01-27 23:52:00,757 WARN [agent.manager.DirectAgentAttache] (DirectAgent-15:null) Seq 1-798621700: Exception Caught while executing command com.cloud.utils.exception.CloudRuntimeException: Unable to create slave connection to host(da372d8a-bbeb-486d-a85e-06922cac8da4) due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:580) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5871) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:8098) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:599) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:701) Caused by: org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:161) at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) at org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunHttpTransport.java:69) at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) at com.xensource.xenapi.Connection.dispatch(Connection.java:303) at com.xensource.xenapi.Session.slaveLocalLoginWithPassword(Session.java:587) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.slaveLocalLoginWithPassword(XenServerConnectionPool.java:688) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:574) ... 14 more Caused by: java.net.NoRouteToHostException: No route to host at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java
Re: Basic Zone Configuration
Lopez, You can specify google DNS 8.8.8.8 and internal DNS which is used by your company, I would suggest to google such issues first, you would get solutions as they have been addressed previously many times. Umair Azam Associate Systems Administrator Network Operations Center i2c Incorporated 1300 Island Drive, Suite 105 Redwood City, CA 94065-5170 Desk: +1 650.480.5291 PBX: +1 650.593.5400 x 4244 24x7 NOC: +1 650.480.5291 Fax: +1 650.593.5402 URL: www.i2cinc.com ** CONFIDENTIALITY CAUTION This communication (including any accompanying documents) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify us immediately by e-mail, telephone or fax and promptly destroy the original communication. Thank you for your cooperation. On 2/5/2014 11:58 PM, José Egas López wrote: Hi, I have a doubt about the DNS Servers when adding a zone. I have one DNS (Active Directory) on my datacenter, when adding a zone it asks to enter DNS 1 and 2, and also Internal DNS 1 and 2, what does it exactly means? Do I have to create new DNS servers? or is enough with the one I have? I don't understand it at all, thanks for your help!
Re: What is your setup?
VMware= Lamborghini Xen= BMW KVM= Motor Bike :D Umair Azam Associate Systems Administrator Network Operations Center i2c Incorporated On 2/6/2014 7:24 AM, Maurice Lawler wrote: Just a general question What is YOUR favorite setup? KVM? Xen? VmWare? - Maurice
Re: Restoring CS UI password
Lopez, Switch to mysql user, then use cloud database and reset password. Run these commands. 1)mysql -u root -p 2)use cloud; 3)UPDATE user SET password = 'Temp/123' WHERE username = 'admin'; Umair Azam Associate Systems Administrator Network Operations Center i2c Incorporated 1300 Island Drive, Suite 105 Redwood City, CA 94065-5170 Desk: +1 650.480.5291 PBX: +1 650.593.5400 x 4244 24x7 NOC: +1 650.480.5291 Fax: +1 650.593.5402 URL: www.i2cinc.com ** CONFIDENTIALITY CAUTION This communication (including any accompanying documents) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify us immediately by e-mail, telephone or fax and promptly destroy the original communication. Thank you for your cooperation. On 2/4/2014 1:40 AM, José Egas López wrote: Hi, is there anyway to restore the default CS UI password? I cannot log into the CS UI, first time I put admin and password as default credentials, it worked fine, but before I finish the CS Tour the session expired and when I try to log in again it doesn't work.
Re: Unable to create slave connection to host [Machine gets rebooted]
Hello, It seems the issue is with my xenserver version compatibility. Will install a version recomended by cloudstack and see if it works or not. On 1/28/2014 12:04 AM, Umair Azam wrote: Amogh, All servers are in the same vlan with no restriction on networking side, iptables service is also off, I havent specified any special networking configurations in cloudstack all settings are set to default. On 1/27/2014 11:52 PM, Amogh Vasekar wrote: Hi, Seems like a config issue. Can you share more details on your set-up, on the networking side? Amogh On 1/27/14 9:48 AM, Umair Azam ua...@i2cinc.com wrote: (DirectAgent-15:null) Logging on as the slave to 10.11.17.31 2014-01-27 23:51:57,756 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-15:null) Unable to create slave connection to host(da372d8a-bbeb-486d-a85e-06922cac8da4) due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host
Unable to create slave connection to host [Machine gets rebooted]
Hi, I am new to cloudstack and testing, evaluating it for future deployment, I am Getting following exception while system VM's creation is in process for the first time, After below exception seen in management server logs, host machine becomes unresponsive and reboots automatically. Host Machine specs: core 2 Duo 4GB RAM with Xenserver 2014-01-27 23:51:55,743 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-15:null) Seq 1-798621700: Executing request 2014-01-27 23:51:55,743 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-15:null) Logging on as the slave to 10.11.17.31 2014-01-27 23:51:57,756 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-15:null) Unable to create slave connection to host(da372d8a-bbeb-486d-a85e-06922cac8da4) due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host 2014-01-27 23:51:58,544 DEBUG [cloud.api.ApiServlet] (catalina-exec-11:null) ===START=== 10.11.17.22 -- GET command=listSystemVmsresponse=jsonsessionkey=k7%2B%2FaPitbX17ZzD9gfUWajyijo8%3D_=1390830753664 2014-01-27 23:51:58,610 DEBUG [cloud.api.ApiServlet] (catalina-exec-11:null) ===END=== 10.11.17.22 -- GET command=listSystemVmsresponse=jsonsessionkey=k7%2B%2FaPitbX17ZzD9gfUWajyijo8%3D_=1390830753664 2014-01-27 23:52:00,757 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-15:null) localLogout has problem Failed to read server's response: No route to host 2014-01-27 23:52:00,757 WARN [agent.manager.DirectAgentAttache] (DirectAgent-15:null) Seq 1-798621700: Exception Caught while executing command com.cloud.utils.exception.CloudRuntimeException: Unable to create slave connection to host(da372d8a-bbeb-486d-a85e-06922cac8da4) due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:580) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5871) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:8098) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:599) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:701) Caused by: org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:161) at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) at org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunHttpTransport.java:69) at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) at com.xensource.xenapi.Connection.dispatch(Connection.java:303) at com.xensource.xenapi.Session.slaveLocalLoginWithPassword(Session.java:587) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.slaveLocalLoginWithPassword(XenServerConnectionPool.java:688) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:574) ... 14 more Caused by: java.net.NoRouteToHostException: No route to host at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385) at java.net.Socket.connect(Socket.java:546)
Re: Unable to create slave connection to host [Machine gets rebooted]
Amogh, All servers are in the same vlan with no restriction on networking side, iptables service is also off, I havent specified any special networking configurations in cloudstack all settings are set to default. Umair Azam Associate Systems Administrator Network Operations Center i2c Incorporated 1300 Island Drive, Suite 105 Redwood City, CA 94065-5170 Desk: +1 650.480.5291 PBX: +1 650.593.5400 x 4244 24x7 NOC: +1 650.480.5291 Fax: +1 650.593.5402 URL: www.i2cinc.com ** CONFIDENTIALITY CAUTION This communication (including any accompanying documents) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify us immediately by e-mail, telephone or fax and promptly destroy the original communication. Thank you for your cooperation. On 1/27/2014 11:52 PM, Amogh Vasekar wrote: Hi, Seems like a config issue. Can you share more details on your set-up, on the networking side? Amogh On 1/27/14 9:48 AM, Umair Azam ua...@i2cinc.com wrote: (DirectAgent-15:null) Logging on as the slave to 10.11.17.31 2014-01-27 23:51:57,756 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-15:null) Unable to create slave connection to host(da372d8a-bbeb-486d-a85e-06922cac8da4) due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route to host
