Re: How to prevent DHCp conflict?
As I learned this is a networking issue not a cloudstack issue. See my previous thread, Yipingresponse is what worked for me. https://lists.apache.org/thread.html/51e005eb7730124d1ecb17adb8ca995a4ae5114d88f6e698b2219587@%3Cusers.cloudstack.apache.org%3E HTH, Jesse On Tue, Jul 30, 2019 at 9:40 AM Fariborz Navidan wrote: > Hello, > > Guests have started not to obtain IP from VR. Instead, they are obtaining > IP from a DHCp which does not belong to CloudStack network. It may be > either an external DHCp server in nearby servers/network in data center or > a user's VM which is running Dhcp server. All new VM reach VR to obtain > password and user data. Please advise. > > Thanks >
Re: DHCP instance/vm issue
Yes race condition exists, I have been fortunate it hasn't been seen outside of ACS environment so far.From a network topology, Ideally I should isolate and route traffic to the Pod and use a firewall or other gateway to control traffic. I'll need to re-think my deployment and see if I need additional resources On Tue, Jul 9, 2019 at 12:52 PM Andrija Panic wrote: > Don't kill dhcp client (don't force renew of IP), since again it will NOT > work if you repeat that a few times - a VM will broadcast dhcp discover > messages, all DHCP server will receive it and all DHCP servers will offer a > lease/ip to your VMs - the one DHCP server to be "quicker" to send its dhcp > offer, will "win" and VM will get its IP... you have "race condition" in > any network with more than 1 DHCP server... It's a "wrong" setup > effectively. > > Cheers > > On Tue, Jul 9, 2019, 22:47 Andrija Panic wrote: > > > Jesse, > > > > You can experiment with firewall rules/SG, but in general you should not > > have more than 1 DHCP server in a single network. I assume your VMs would > > be assigned one part of the net/subnet, while your external DHCP server > > should be serving your non-ACS infra - i.e. if your acs network for VMs > is > > 192.168.1.1-128, while 192.168.1.129-254(non-ACS infra) should be served > by > > your external DHCP, then I would think of blocking dhcp ports (dhcp > > discover) from whole 192.168.1.1-128 network on your external DHCP > server - > > i.e. this way your external DHCP SERVER would be "deaf" to all dhcp > > discover messages sent from ACS VMs to itself and thus would not issue > > leases to ACS VMs. > > > > Hope that makes sense. > > > > Best > > Andrija > > > > On Tue, Jul 9, 2019, 21:16 wrote: > > > >> My vm was assigned an ip from our endpoint DHCP server, not from VR. Do > I > >> need to add firewall rule(s) to force DHCP request to VR? I probably > >> missed > >> a part of setup w/KVM hosts and or within management when I defined the > >> zone/pod/... > >> > >> This seems to be correct, VR is running on a different host then the vm. > >> > >> Chain i-2-11-VM-eg (1 references) > >> pkts bytes target prot opt in out source > >> destination > >> 0 0 RETURN all -- * * 0.0.0.0/0 > >> 0.0.0.0/0 > >> > >> Chain i-2-11-def (2 references) > >> pkts bytes target prot opt in out source > >> destination > >> 0 0 ACCEPT all -- * * 0.0.0.0/0 > >> 0.0.0.0/0state RELATED,ESTABLISHED > >> 0 0 ACCEPT udp -- * * 0.0.0.0/0 > >> 0.0.0.0/0PHYSDEV match --physdev-in vnet0 > >> --physdev-is-bridged > >> udp spt:68 dpt:67 > >> 0 0 ACCEPT udp -- * * 0.0.0.0/0 > >> 0.0.0.0/0PHYSDEV match --physdev-out vnet0 > >> --physdev-is-bridged > >> udp spt:67 dpt:68 > >> 0 0 DROP all -- * * 0.0.0.0/0 > >> 0.0.0.0/0PHYSDEV match --physdev-in vnet0 > >> --physdev-is-bridged > >> ! match-set i-2-11-VM src > >> 0 0 RETURN udp -- * * 0.0.0.0/0 > >> 0.0.0.0/0PHYSDEV match --physdev-in vnet0 > >> --physdev-is-bridged > >> match-set i-2-11-VM src udp dpt:53 > >> 0 0 RETURN tcp -- * * 0.0.0.0/0 > >> 0.0.0.0/0PHYSDEV match --physdev-in vnet0 > >> --physdev-is-bridged > >> match-set i-2-11-VM src tcp dpt:53 > >> 0 0 i-2-11-VM-eg all -- * * 0.0.0.0/0 > >> 0.0.0.0/0PHYSDEV match --physdev-in vnet0 > >> --physdev-is-bridged > >> match-set i-2-11-VM src > >>15 1963 i-2-11-VM all -- * * 0.0.0.0/0 > >> 0.0.0.0/0PHYSDEV match --physdev-out vnet0 > >> --physdev-is-bridged > >> > >> > >> > >> Thanks for quick response Andrija! > >> > >> - Jesse > >> > >> > >> > >> > >> On Tue, Jul 9, 2019 at 10:39 AM Andrija Panic > >> wrote: > >> > >> > ACS will only offer DHCP leases to its VMs, via DHCP reservation.. If > >> you > >> > have another DHCP server in your area, than it might be quicker to > >> offer a > >> > lease to a VM. You have to either remove your non-ACS DHCP server > >> > completely, OR make sure it uses reservation for non-ACS servers/hosts > >> i.e. > >> > NOT let it issue leases freely to anyone who asks for it. Pure DHCP > >> > "problem" - i.e. nothing to do with ACS specifically. > >> > > >> > Best, > >> > Andrija > >> > > >> > On Tue, Jul 9, 2019, 20:27 wrote: > >> > > >> > > Have a DHCP issue where vm pulls from ACS proxy properly sometimes > and > >> > > other when it pulls from our normal dhcp server for end-points. > >> > > > >> > > Network layout is flat, and I ACS is using basic network with > security > >> > > groups. IP range for acs is within range of our normal network so > vms > >> > and > >> > > endpoints will flow without additional hardware. How do I ensure > dhcp > >> > > requests are served by router vm and not our normal dhcp server? > >> > > > >> > > TIA, > >> > > Jesse > >> > > >
Re: DHCP instance/vm issue
Interesting proxy in to vm pkill dhclient dhclient -x dhclient eth0 get ip I expected, odd On Tue, Jul 9, 2019 at 11:16 AM wrote: > > My vm was assigned an ip from our endpoint DHCP server, not from VR. Do I > need to add firewall rule(s) to force DHCP request to VR? I probably missed > a part of setup w/KVM hosts and or within management when I defined the > zone/pod/... > > This seems to be correct, VR is running on a different host then the vm. > > Chain i-2-11-VM-eg (1 references) > pkts bytes target prot opt in out source > destination > 0 0 RETURN all -- * * 0.0.0.0/0 > 0.0.0.0/0 > > Chain i-2-11-def (2 references) > pkts bytes target prot opt in out source > destination > 0 0 ACCEPT all -- * * 0.0.0.0/0 > 0.0.0.0/0state RELATED,ESTABLISHED > 0 0 ACCEPT udp -- * * 0.0.0.0/0 > 0.0.0.0/0PHYSDEV match --physdev-in vnet0 > --physdev-is-bridged udp spt:68 dpt:67 > 0 0 ACCEPT udp -- * * 0.0.0.0/0 > 0.0.0.0/0PHYSDEV match --physdev-out vnet0 > --physdev-is-bridged udp spt:67 dpt:68 > 0 0 DROP all -- * * 0.0.0.0/0 > 0.0.0.0/0PHYSDEV match --physdev-in vnet0 > --physdev-is-bridged ! match-set i-2-11-VM src > 0 0 RETURN udp -- * * 0.0.0.0/0 > 0.0.0.0/0PHYSDEV match --physdev-in vnet0 > --physdev-is-bridged match-set i-2-11-VM src udp dpt:53 > 0 0 RETURN tcp -- * * 0.0.0.0/0 > 0.0.0.0/0PHYSDEV match --physdev-in vnet0 > --physdev-is-bridged match-set i-2-11-VM src tcp dpt:53 > 0 0 i-2-11-VM-eg all -- * * 0.0.0.0/0 > 0.0.0.0/0PHYSDEV match --physdev-in vnet0 > --physdev-is-bridged match-set i-2-11-VM src >15 1963 i-2-11-VM all -- * * 0.0.0.0/0 > 0.0.0.0/0PHYSDEV match --physdev-out vnet0 > --physdev-is-bridged > > > > Thanks for quick response Andrija! > > - Jesse > > > > > On Tue, Jul 9, 2019 at 10:39 AM Andrija Panic > wrote: > >> ACS will only offer DHCP leases to its VMs, via DHCP reservation.. If you >> have another DHCP server in your area, than it might be quicker to offer a >> lease to a VM. You have to either remove your non-ACS DHCP server >> completely, OR make sure it uses reservation for non-ACS servers/hosts >> i.e. >> NOT let it issue leases freely to anyone who asks for it. Pure DHCP >> "problem" - i.e. nothing to do with ACS specifically. >> >> Best, >> Andrija >> >> On Tue, Jul 9, 2019, 20:27 wrote: >> >> > Have a DHCP issue where vm pulls from ACS proxy properly sometimes and >> > other when it pulls from our normal dhcp server for end-points. >> > >> > Network layout is flat, and I ACS is using basic network with security >> > groups. IP range for acs is within range of our normal network so vms >> and >> > endpoints will flow without additional hardware. How do I ensure dhcp >> > requests are served by router vm and not our normal dhcp server? >> > >> > TIA, >> > Jesse >> > >> >
Re: DHCP instance/vm issue
My vm was assigned an ip from our endpoint DHCP server, not from VR. Do I need to add firewall rule(s) to force DHCP request to VR? I probably missed a part of setup w/KVM hosts and or within management when I defined the zone/pod/... This seems to be correct, VR is running on a different host then the vm. Chain i-2-11-VM-eg (1 references) pkts bytes target prot opt in out source destination 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 Chain i-2-11-def (2 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0state RELATED,ESTABLISHED 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0PHYSDEV match --physdev-in vnet0 --physdev-is-bridged udp spt:68 dpt:67 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0PHYSDEV match --physdev-out vnet0 --physdev-is-bridged udp spt:67 dpt:68 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0PHYSDEV match --physdev-in vnet0 --physdev-is-bridged ! match-set i-2-11-VM src 0 0 RETURN udp -- * * 0.0.0.0/0 0.0.0.0/0PHYSDEV match --physdev-in vnet0 --physdev-is-bridged match-set i-2-11-VM src udp dpt:53 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0PHYSDEV match --physdev-in vnet0 --physdev-is-bridged match-set i-2-11-VM src tcp dpt:53 0 0 i-2-11-VM-eg all -- * * 0.0.0.0/0 0.0.0.0/0PHYSDEV match --physdev-in vnet0 --physdev-is-bridged match-set i-2-11-VM src 15 1963 i-2-11-VM all -- * * 0.0.0.0/0 0.0.0.0/0PHYSDEV match --physdev-out vnet0 --physdev-is-bridged Thanks for quick response Andrija! - Jesse On Tue, Jul 9, 2019 at 10:39 AM Andrija Panic wrote: > ACS will only offer DHCP leases to its VMs, via DHCP reservation.. If you > have another DHCP server in your area, than it might be quicker to offer a > lease to a VM. You have to either remove your non-ACS DHCP server > completely, OR make sure it uses reservation for non-ACS servers/hosts i.e. > NOT let it issue leases freely to anyone who asks for it. Pure DHCP > "problem" - i.e. nothing to do with ACS specifically. > > Best, > Andrija > > On Tue, Jul 9, 2019, 20:27 wrote: > > > Have a DHCP issue where vm pulls from ACS proxy properly sometimes and > > other when it pulls from our normal dhcp server for end-points. > > > > Network layout is flat, and I ACS is using basic network with security > > groups. IP range for acs is within range of our normal network so vms > and > > endpoints will flow without additional hardware. How do I ensure dhcp > > requests are served by router vm and not our normal dhcp server? > > > > TIA, > > Jesse > > >
DHCP instance/vm issue
Have a DHCP issue where vm pulls from ACS proxy properly sometimes and other when it pulls from our normal dhcp server for end-points. Network layout is flat, and I ACS is using basic network with security groups. IP range for acs is within range of our normal network so vms and endpoints will flow without additional hardware. How do I ensure dhcp requests are served by router vm and not our normal dhcp server? TIA, Jesse
Re: adding new host fails
excuse me, I guess i should have windows fix first. Everything is connected and working. Hate windows fixes, don't know what real issue was. On Mon, Jul 8, 2019 at 9:35 PM wrote: > > Host failes to add > running cloudstack 4.12.0.0 on mgmt & host > > Get these from host setup.log: > DEBUG:root:execute:uuidgen > DEBUG:root:execute:systemctl is-active cloudstack-agent > DEBUG:root:Failed to execute:failed > DEBUG:root:execute:systemctl stop cloudstack-agent > DEBUG:root:execute:sleep 30 > DEBUG:root:execute:systemctl enable cloudstack-agent > DEBUG:root:execute:systemctl is-active cloudstack-agent > DEBUG:root:Failed to execute:failed > > This from management log: > 2019-07-08 20:12:54,575 DEBUG [c.c.u.s.SSHCmdHelper] > (qtp2114650936-12:ctx-ee3d569b ctx-9a9514d8) (logid:fd5e141d) SSH command: > cloudstack-setup-agent -m 10.1.155.14 -z 1 -p 1 -c 1 -g > bf7bc892-ef36-349b-b0de-980fb5947337 -a -s --pubNic=cloudbr0 > --prvNic=cloudbr0 --guestNic=cloudbr0 --hypervisor=kvm > SSH command output:Starting to configure your system: > Configure SElinux ... [OK] > Configure Network ... [OK] > Configure Libvirt ... [OK] > Configure Firewall ...[OK] > Configure Nfs ... [OK] > Configure cloudAgent ... [OK] > CloudStack Agent setup is done! > > Looks like it is failing to create ceph storage-pool in libvirt > Jul 08 16:07:08 kvm02 java[17177]: libvirt: Storage Driver error : Storage > pool not found: no storage pool with matching uuid > '328369ed-6a8a-3f34-a99a-e7340b4fe8a2' > Jul 08 16:07:08 kvm02 java[17177]: WARN > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:) > (logid:61c5ee49) Storage pool 328369ed-6a8a-3f34-a99a-e7340b4fe8a2 was no > t found running in libvirt. Need to create it. > Jul 08 16:07:08 kvm02 java[17177]: INFO > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:) > (logid:61c5ee49) Didn't find an existing storage pool 328369ed-6a8a-3f34- > a99a-e7340b4fe8a2 by UUID, checking for pools with duplicate paths > > Not something I had to do manually with first host > > > Suggestions/help > > TIA, > Jesse >
adding new host fails
Host failes to add running cloudstack 4.12.0.0 on mgmt & host Get these from host setup.log: DEBUG:root:execute:uuidgen DEBUG:root:execute:systemctl is-active cloudstack-agent DEBUG:root:Failed to execute:failed DEBUG:root:execute:systemctl stop cloudstack-agent DEBUG:root:execute:sleep 30 DEBUG:root:execute:systemctl enable cloudstack-agent DEBUG:root:execute:systemctl is-active cloudstack-agent DEBUG:root:Failed to execute:failed This from management log: 2019-07-08 20:12:54,575 DEBUG [c.c.u.s.SSHCmdHelper] (qtp2114650936-12:ctx-ee3d569b ctx-9a9514d8) (logid:fd5e141d) SSH command: cloudstack-setup-agent -m 10.1.155.14 -z 1 -p 1 -c 1 -g bf7bc892-ef36-349b-b0de-980fb5947337 -a -s --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr0 --hypervisor=kvm SSH command output:Starting to configure your system: Configure SElinux ... [OK] Configure Network ... [OK] Configure Libvirt ... [OK] Configure Firewall ...[OK] Configure Nfs ... [OK] Configure cloudAgent ... [OK] CloudStack Agent setup is done! Looks like it is failing to create ceph storage-pool in libvirt Jul 08 16:07:08 kvm02 java[17177]: libvirt: Storage Driver error : Storage pool not found: no storage pool with matching uuid '328369ed-6a8a-3f34-a99a-e7340b4fe8a2' Jul 08 16:07:08 kvm02 java[17177]: WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:) (logid:61c5ee49) Storage pool 328369ed-6a8a-3f34-a99a-e7340b4fe8a2 was no t found running in libvirt. Need to create it. Jul 08 16:07:08 kvm02 java[17177]: INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:) (logid:61c5ee49) Didn't find an existing storage pool 328369ed-6a8a-3f34- a99a-e7340b4fe8a2 by UUID, checking for pools with duplicate paths Not something I had to do manually with first host Suggestions/help TIA, Jesse
VM HA enabled when on ceph
Hi all, Have a ACS 4.12.0.0 running with 2 kvm hosts and ceph for primary storage. How do I enable HA for my VM(s)? The manual hints "HA features work with iSCSI or NFS primary storage", leaves out ceph. Can I just go in db and enable set vm to "HA" yes? mysql> update vm_instance set ha_enabled = 1 where id = ; TIA, Jesse
Re: mgmt node on kvm host
Coming from vmware environments where vcenter has almost always been a hosted vm. You could give mgmt node highest priority over other VMs, but safe approach as you stated is to keep it outside. Environment I am stand up is going to be very light use and wanting to keep infrastructure as light as possible. 2 KVM hosts 1 mgmt hosts (as it seems now) ceph cluster for shared stroage (overkill, but I need redundancy and ability to grow if we expand foot print) On Thu, Jun 20, 2019 at 3:09 PM Andrija Panic wrote: > In short, > > Never, EVER, do such thing (especially in production of any kind). > > Mgmt server (any auxiliary components like standalone DB server, billing > software, monitoring software, load balancers etc - all things that are > needed for production...) can be on KVM nodes, but a standalone KVM nodes, > which are NOT managed by CloudStack. Whether you use VMM from Linux > Desktop, plain/manual management via libvirt and "nano" etc...all good as > long as not managed by ACS - you want to avoid cyclic dependency and mgmt > server issues due to high host usage/overload, etc, etc, etc. > > Regards, > Andrija > > > > > On Thu, Jun 20, 2019, 20:48 wrote: > > > How are most of you running your management node? On a standalone host(s) > > outside hypervisors? On managed hypervisor hosts? > > > > What I want to do is have management node on kvm hosts that it manages. > > What I am not certain is how I get it in there in 1st place. Other than > > standing up a temporary mgmt node, get it managing the kvm hosts. Then > > stand new mgmt node from with in. > > > > Thoughts recommendations, I'm wondering about best practice and worst > case > > scenario where all hosts lose power (dr different issue in itself). How > > will I start mgmt node if there is no mgmt node to start it. virsh start > > mgmtnnode? > > > > TIA, > > > > Jesse > > >
mgmt node on kvm host
How are most of you running your management node? On a standalone host(s) outside hypervisors? On managed hypervisor hosts? What I want to do is have management node on kvm hosts that it manages. What I am not certain is how I get it in there in 1st place. Other than standing up a temporary mgmt node, get it managing the kvm hosts. Then stand new mgmt node from with in. Thoughts recommendations, I'm wondering about best practice and worst case scenario where all hosts lose power (dr different issue in itself). How will I start mgmt node if there is no mgmt node to start it. virsh start mgmtnnode? TIA, Jesse
Re: delete UploadAbandoned template
I did not try your fix but I will be trying to import image again soon and see. My fix was to create directory in template/tmpl/2/209, then using th UI I was able to remove the template. Thanks again, Jesse On Wed, Jun 5, 2019 at 4:55 PM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Hi Jesse, > > As the upload was abandoned, it has never started, so the workaround will > be just deleting the entry from CloudStack as there will be no files on > secondary storage. To do this, please first execute this query on database: > > update template_store_ref set destroyed = 1 where template_id = 209; > > After that, try again to delete the template from CloudStack. > > > Regards, > > Nicolas Vazquez > > > From: jesse.wat...@gmail.com > Sent: Wednesday, June 5, 2019 3:47 PM > To: users@cloudstack.apache.org > Subject: delete UploadAbandoned template > > How do I remove abandined templates? > > While trying to clean up after couple failed attempts of importing a vm > template status is "UploadAbandoned" > When I try to delete file I get an error , "Failed to delete template" > Looking at management.log > 2019-06-05 14:43:17,508 WARN [c.c.t.HypervisorTemplateAdapter] > (API-Job-Executor-27:ctx-1bdcfb28 job-402 ctx-9643b0f7) (logid:334038d9) > Failed to delete the template: > Tmpl[209-QCOW2-209-2-b1f68c47-8b88-3d46-90fa-753f4e8bcc72 from the image > store: atlas_nfs due to: Unable to delete file 207 under Template path > template/tmpl/2/209 > > And directory 209 does not exist due to upload being abandoned. > > Any suggestions? > > TIA, > Jesse > > nicolas.vazq...@shapeblue.com > www.shapeblue.com > Amadeus House, Floral Street, London WC2E 9DPUK > @shapeblue > > > >
delete UploadAbandoned template
How do I remove abandined templates? While trying to clean up after couple failed attempts of importing a vm template status is "UploadAbandoned" When I try to delete file I get an error , "Failed to delete template" Looking at management.log 2019-06-05 14:43:17,508 WARN [c.c.t.HypervisorTemplateAdapter] (API-Job-Executor-27:ctx-1bdcfb28 job-402 ctx-9643b0f7) (logid:334038d9) Failed to delete the template: Tmpl[209-QCOW2-209-2-b1f68c47-8b88-3d46-90fa-753f4e8bcc72 from the image store: atlas_nfs due to: Unable to delete file 207 under Template path template/tmpl/2/209 And directory 209 does not exist due to upload being abandoned. Any suggestions? TIA, Jesse
