If I use a single Management server it works:
Question with the IP addresses assigned: the initial VMs are running: v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y> I do not see the Public IP anywhere, not even advertised on vni1000 or any other, the public VNI is there was created by the system, but nothing is there - No IP addresses advertised I used VLAN/VNI vxlan://1000 vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 master brvx-1000 state forwarding priority There are 2 VNETS assigned virsh domiflist s-285-VM Interface Type Source Model MAC -------------------------------------------------------------- vnet3 bridge cloud0 virtio 0e:00:a9:fe:a1:14 vnet4 bridge cloudbr1 virtio 1e:00:3f:00:03:33 vnet5 bridge brvx-1000 virtio 1e:00:b8:00:0c:fe virsh domifaddr s-285-VM Name MAC address Protocol Address ------------------------------------------------------------------------------- ----- Nothing here ----- the command arp -n, does not showPublic IPaddresses, just the private IP addresses and the <control-ip-169.x.x.x> sh bgp evpn route vni 1000 BGP table version is 1, local router ID is 10.1.1.14 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id] EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP] Network Next Hop Metric LocPrf Weight Path *> [3]:[0]:[32]:[10.1.1.14] 10.1.1.14(hv1) 32768 i ET:8 RT:5641:1000 Displayed 1 prefixes (1 paths) Please Let me know if I should see the Public IP addresses advertised for this VMs or any IP. Tata Y. > On Apr 1, 2025, at 9:35 AM, Wido den Hollander <w...@widodh.nl> wrote: > > > > Op 01-04-2025 om 13:16 schreef Chi vediamo: >> Hello Wido, >> Sorry I meant I only have cloudbr1, there is no cloudbr0 on the hypervisor >> On the management server on the UI interface >> when the VNI/VXLAN for Public is requested then I should enter vxlan://777 >> in your case ? > > Yes, for the first network you create for example. > >> And for the VNI/VXLAN Range for guest leave it blank ? or ALso use a full >> range including VXLAN200 in your case vxlan://1 through vxlan://1000 ? > > I don't think we ever used a VNI range in our setups. We just manually add > the VNI for every network we create. > > Wido > >> Tata Y. >>> On Apr 1, 2025, at 7:06 AM, Wido den Hollander <w...@widodh.nl> wrote: >>> >>> >>> >>> Op 01-04-2025 om 12:16 schreef Chi vediamo: >>>> Thank you Wido for jumping in, as always >>>> So far I only Have cloudbr1 as several VXLAN e-mail trails you indicated. >>>> Then I need to create the cloudbr0 with another VXLAN in addition to >>>> cloudbr1 ? cloudbr1 should have the management per the video you posted >>>> right? >>> >>> In our case we use a static VNI underneath cloudbr1 where it communicates >>> with the other hosts, the mgmt server and the secondary storage (NFS). This >>> is a /22 IPv4 we have allocated for that VNI. >>> >>> vxlan200 device which we created using networkd and is a slave of cloudbr1 >>> >>> cloudbr0 is never used as the VXLAN script creates brvx-XXX devices on the >>> fly. >>> >>>> Hope you recall, should I include in the VNI range the initial VNI/VXLANs >>>> i am using for cloudbr1 and cloudbr0 >>> >>> Not that I'm aware of. We simply specify the broadcast domain when creating >>> a new network: vxlan://777 for example to use VNI 777 for that network. >>> >>> This will result in brvx-777 being created on a host when needed there and >>> the VM being attached to that bridge. >>> >>> Wido >>> >>>> Thank you so much. >>>> Tata Y. >>>>> On Apr 1, 2025, at 6:00 AM, Wido den Hollander <w...@widodh.nl> wrote: >>>>> >>>>> >>>>> >>>>> Op 29-03-2025 om 15:42 schreef Chi vediamo: >>>>>> Hello Wei, >>>>>> The VXLAN with EPVN is poorly documented, then I need all tthe guidance >>>>>> as possible from the community. >>>>> >>>>> There is no silver bullet here as each EVPN+VXLAN setup will differ from >>>>> the other :-) >>>>> >>>>>> Wido, please kindly allow me to use some of your multiple answers to >>>>>> several members of the community as reference. As there is really not >>>>>> much info. >>>>>> * Wei what you suggesting I did try that, already, I did try several >>>>>> options so far. The cloudstack-Agent keeps restarting itself, I copy >>>>>> the logs in the e-mail. >>>>>> * >>>>>> * Saw countless videos, and read the manual, the Manual only states in >>>>>> summary "VXLAN with EVPN is complex", there is nothing else. >>>>>> * >>>>>> * Manual states use and modify "modifyvxlan-evpn.sh" which does not >>>>>> exists. Using the one form github. modifyvxlan.sh >>>>>> * >>>>> >>>>> Yes, the Github one is good. It will be in packages from 4.21 >>>>> >>>>>> * The manual indicates: showing a picture of Management,public and >>>>>> storage on vlans. and indicates - The only thing you have to do is >>>>>> configure the physical Network to use VXLAN as the isolation method >>>>>> for Guest Network - We do not have VLANS at all. >>>>>> * >>>>> >>>>> Don't worry about different networks, VXLAN creates it's own bridges on >>>>> the fly. >>>>> >>>>>> * Per one of the e-mails iIread long time ago from Wido to another >>>>>> member of the community: we do left the UI settings by default. That >>>>>> didn't work. And we start experimenting, and none seems to be working. >>>>> >>>>> It has been a long time since I've set it up. But you should just have >>>>> cloudbr0 and cloudbr1, nothing else. >>>>> >>>>>> Per Wido, we only need cloudbr1, with no VLANs. >>>>>> I only need to know on the cludstack UI, What do we point to the >>>>>> Cloudbr1 My understanding is This is going to be used for Migrations ad >>>>>> that. >>>>>> 4.20.0 does not allow me to add Management alone, nor the Guest Alone, >>>>>> when creating the first zone. >>>>>> I need Public and Management at least. Right? Or please indicate which >>>>>> ones. I try to add only the Guest one, it also fails >>>>>> Additionally, seems like the Cloudstack is not smart enough to see what >>>>>> VNI VXLANs are already configured, then I have to provide a vni range >>>>>> for the Guest. >>>>>> Although the modifyvxlan.sh script should take care of it - that is >>>>>> based on reading the script code. >>>>>> So far we do have VXLAN-EVPN reachability and working. >>>>>> I think we are almost there: >>>>>> * I try adding Guest Only during the zone creation, we do have >>>>>> management, public and storage on VXLANS, all are reacheable by >>>>>> routing, and the cloudstack-agent server reach them and have >>>>>> cloudbr1 created. >>>>>> o Should I have only Managment and Guest on the cloudbr1? >>>>>> * Need help Guidance with The cloudstack-agent keeps Bouncing comes up >>>>>> and immediately down, and even reinstalled removing all files. Still >>>>>> the same. >>>>>> * Any example of the initial agent.properties: Or do not touch the >>>>>> agent.properties >>>>> >>>>> Only set the network devices as you've done below. The guid should match >>>>> what the hypervisor thinks the UUID of this Agent is. Again, long time >>>>> ago since I've set this up for the first time. >>>>> >>>>> Wido >>>>> >>>>>> My agent.properties I have modified specifically 3 parameters: >>>>>> host=<MAMGT-IP1>,<MGMT-IP2> >>>>>> guest.cpu.mode=host-model >>>>>> private.network.device=cloudbr1 >>>>>> Thank you for your guidance. >>>>>> Thank you >>>>>> Tata Y. >>>>>>> On Mar 28, 2025, at 1:43 PM, Wei ZHOU <ustcweiz...@gmail.com> wrote: >>>>>>> >>>>>>> I think you do not need to change agent.properties >>>>>>> Instead, you need to set the network traffic label of each traffic >>>>>>> type, for example cloudbr1. >>>>>>> >>>>>>> By default, management traffic uses cloudbr0, public and guest traffic >>>>>>> use cloudbr1, which might not be what you use. >>>>>>> >>>>>>> >>>>>>> -Wei >>>>>>> >>>>>>> On Friday, March 28, 2025, Chi vediamo <tatay...@gmail.com >>>>>>> <mailto:tatay...@gmail.com>> wrote: >>>>>>> >>>>>>> Hello Apache Cloudstack users, >>>>>>> >>>>>>> >>>>>>> After attempted some options, I am recurring to what should work. >>>>>>> As there is no info in the documentation about this deployment. >>>>>>> >>>>>>> I did replaced the agent server with private.network.device=cloudbr1 >>>>>>> >>>>>>> I have my VXLAN 1 on Cloudbr1 that reaches management and SS, We >>>>>>> only have cloudbr1 created, No other bridge, and everything is BGP. >>>>>>> >>>>>>> >>>>>>> Then at the UI creating the ZONE I got errors: >>>>>>> >>>>>>> We have had tryed the following Options: >>>>>>> >>>>>>> *Option 1: *No tags >>>>>>> I put only [Physical Network 1] [VXLAN] Management, throws a popup >>>>>>> error " All required traffic types should be added and with >>>>>>> multiple physical networks each traffic type should have a label" >>>>>>> >>>>>>> *Option 2:* No Tags, Throws and Error " All required traffic types >>>>>>> should be added and with multiple physical networks each traffic >>>>>>> type should have a label" >>>>>>> I put : >>>>>>> [Physical Network 1] [VXLAN] Management >>>>>>> [Physical Network 2] [VXLAN] Public >>>>>>> [Physical Network 3] [VXLAN] Guest >>>>>>> [Physical Network 4] [VXLAN] Storage >>>>>>> >>>>>>> >>>>>>> *Option 3: * >>>>>>> I put : >>>>>>> [Physical Network 1] [VXLAN] Management, Public, Guest, it goes to >>>>>>> the next window - which is Great >>>>>>> The the system throws exceptions like: >>>>>>> *Something went wrong; please correct the following:* >>>>>>> *Unable to add the host: Cannot find the server resources at >>>>>>> http://<KVM-IP>* >>>>>>> >>>>>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml >>>>>>> * >>>>>>> * >>>>>>> *Management logs shows:* >>>>>>> Succeeded to import certificate in the keystore for agent on the >>>>>>> KVM host: <ip>. Agent secured and trusted >>>>>>> >>>>>>> * >>>>>>> * >>>>>>> *The Agent Logs:* >>>>>>> ** >>>>>>> 2025-03-28 16:48:42,550 DEBUG [commons.beanutils.ConvertUtils] >>>>>>> (main:[]) (logid:) Convert string '<IP-MGMT1>,<IP-MGMT2>' to class >>>>>>> 'java.lang.String' >>>>>>> 2025-03-28 16:48:42,550 DEBUG >>>>>>> [beanutils.converters.StringConverter] (main:[]) (logid:) >>>>>>> Converting 'String' value '<IP-MGMT1>,<IP-MGMT2>' to type 'String' >>>>>>> 2025-03-28 16:48:42,551 DEBUG >>>>>>> [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) >>>>>>> Property [guid] has empty or null value. Using default value [null]. >>>>>>> 2025-03-28 16:48:42,552 DEBUG >>>>>>> [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) >>>>>>> Property [developer] has empty or null value. Using default value >>>>>>> [false]. >>>>>>> 2025-03-28 16:48:53,897 INFO [cloud.agent.AgentShell] (main:[]) >>>>>>> (logid:) Agent started >>>>>>> 2025-03-28 16:48:53,902 INFO [cloud.agent.AgentShell] (main:[]) >>>>>>> (logid:) Implementation Version is 4.20.0.0 >>>>>>> 2025-03-28 16:48:53,904 INFO [cloud.agent.AgentShell] (main:[]) >>>>>>> (logid:) agent.properties found at /etc/cloudstack/agent/ >>>>>>> agent.properties >>>>>>> 2025-03-28 16:48:53,913 DEBUG >>>>>>> [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) >>>>>>> Property [host] was altered. Now using the value >>>>>>> [10.23.123.61,10.23.123.62]. >>>>>>> 2025-03-28 16:48:53,923 DEBUG >>>>>>> [beanutils.converters.BooleanConverter] (main:[]) (logid:) Setting >>>>>>> default value: false >>>>>>> 2025-03-28 16:48:53,923 DEBUG >>>>>>> [beanutils.converters.BooleanConverter] (main:[]) (logid:) >>>>>>> Converting 'Boolean' value 'false' to type 'Boolean' >>>>>>> 2025-03-28 16:48:53,923 DEBUG >>>>>>> [beanutils.converters.BooleanConverter] (main:[]) (logid:) No >>>>>>> conversion required, value is already a Boolean >>>>>>> 2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter] >>>>>>> (main:[]) (logid:) Setting default value: 0 >>>>>>> 2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter] >>>>>>> (main:[]) (logid:) Converting 'Integer' value '0' to type 'Byte' >>>>>>> 2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter] >>>>>>> (main:[]) (logid:) Converted to Byte value '0' >>>>>>> 2025-03-28 16:48:53,926 DEBUG >>>>>>> [beanutils.converters.CharacterConverter] (main:[]) (logid:) >>>>>>> Setting default value: >>>>>>> 2025-03-28 16:48:53,926 DEBUG >>>>>>> [beanutils.converters.CharacterConverter] (main:[]) (logid:) >>>>>>> Converting 'Character' value ' ' to type 'Character' >>>>>>> 2025-03-28 16:48:53,926 DEBUG >>>>>>> [beanutils.converters.CharacterConverter] (main:[]) (logid:) >>>>>>> No conversion required, value is already a Character >>>>>>> 2025-03-28 16:48:53,926 DEBUG >>>>>>> [beanutils.converters.DoubleConverter] (main:[]) (logid:) Setting >>>>>>> default value: 0 >>>>>>> 2025-03-28 16:48:53,926 DEBUG >>>>>>> [beanutils.converters.DoubleConverter] (main:[]) (logid:) >>>>>>> Converting 'Integer' value '0' to type 'Double' >>>>>>> 2025-03-28 16:48:53,927 DEBUG >>>>>>> [beanutils.converters.DoubleConverter] (main:[]) (logid:) >>>>>>> Converted to Double value '0.0' >>>>>>> 2025-03-28 16:48:53,927 DEBUG >>>>>>> [beanutils.converters.FloatConverter] (main:[]) (logid:) Setting >>>>>>> default value: 0 >>>>>>> 2025-03-28 16:48:53,927 DEBUG >>>>>>> [beanutils.converters.FloatConverter] (main:[]) (logid:) >>>>>>> Converting 'Integer' value '0' to type 'Float' >>>>>>> 2025-03-28 16:48:53,927 DEBUG >>>>>>> [beanutils.converters.FloatConverter] (main:[]) (logid:) >>>>>>> Converted to Float value '0.0' >>>>>>> 2025-03-28 16:48:53,928 DEBUG >>>>>>> [beanutils.converters.IntegerConverter] (main:[]) (logid:) Setting >>>>>>> default value: 0 >>>>>>> * >>>>>>> * >>>>>>> AGENT out : >>>>>>> Unable to find the guid >>>>>>> Unable to find the guid >>>>>>> Unable to find the guid >>>>>>> Unable to find the guid >>>>>>> Unable to find the guid >>>>>>> >>>>>>> There is nothing there >>>>>>> >>>>>>> And the agent come up, and immediately goes down!! >>>>>>> >>>>>>> >>>>>>> >>>>>>> Tata Y. >>>>>>> >>>>> >>> >