Re: Experience on GPU Support?
Hi All, Thanks so much for the insights, it’s really eye opening. Im wondering, is vGPU for KVM possible? And does it only work with NVIDIA cards? Or has anyone successfully integrated AMD MI300X before (using vGPU KVM)? Regards, Bryan On 12 Mar 2024 at 3:54 AM +0800, Pierre-Luc Dion , wrote: > There is no issue delivering VM with passthrough GPU with CloudStack, We've > been doing this using XenServer as hypervisor. It requires an enterprise > license from Citrix to enable the GPU and vGPU support features. > > As we don't use KVM , I can't say much about it. > > there is some limitation delivering GPU/vGPU to VMs, on XenServer you can > only share a single GPU per VM in passthrough, so if your server have 4 GPU > cards, you can have 4 VMs with passthrough GPU. 1 VM can support multiple > vGPU, not sure for the count. > > If you plan to deploy vGPU , you also need the RTX driver installed on the > hypervisor, and licensing service in order to deliver the vGPU. vGPU > definition and naming seams standard across hypervisors depending on the > GPU model. > > I would think that you can deliver GPU in passthrough for H100 GPU, > will know more later this year. What would you like to deliver from single > H100 per VMs? > > > Extending vGPU support in cloudstack is easy: > https://github.com/apache/cloudstack/blob/6dc3d06037c39019f29686281856443c37a3e6c0/api/src/main/java/com/cloud/gpu/GPU.java#L27 > > Offering can be created for GPU passthrough that are not listed, this is > mostly for CloudStack UI to list available GPU during compute-offering > creation. > > > > > On Fri, Feb 23, 2024 at 9:04 AM Ivan Kud wrote: > > > Another way to deal with it is to use KVM agent hooks (this is my code > > implemented specifically to deal with GPUs and VM-dedicated drives): > > > > https://github.com/apache/cloudstack/blob/8f6721ed4c4e1b31081a951c62ffbe5331cf16d4/agent/conf/agent.properties#L123 > > > > You can implement the logic in Groovy to modify XML during the start to > > support extra devices out of CloudStack management. > > > > On Fri, Feb 23, 2024 at 2:36 PM Jorge Luiz Correa > > wrote: > > > > > Hi Bryan! We are using here but in a different way, customized for our > > > environment and using how it is possible the features of CloudStack. In > > > documentation we can see support for some GPU models a little bit old > > > today. > > > > > > We are using pci passthrough. All hosts with GPU are configured to boot > > > with IOMMU and vfio-pci, not loading kernel modules for each GPU. > > > > > > Then, we create a serviceoffering to describe VMs that will have GPU. In > > > this serviceoffering we use the serviceofferingdetails[1].value field to > > > insert a block of configuration related to the GPU. It is something like > > > " ... ... address type=pci" that describes the PCI bus > > > from each GPU. Then, we use tags to force this computeoffering to run > > only > > > in hosts with GPUs. > > > > > > We create a Cloudstack cluster with a lot of hosts equipped with GPUs. > > When > > > a user needs a VM with GPU he/she should use the created computeoffering. > > > VM will be instantiated in some host of the cluster and GPUs are > > > passthrough to VM. > > > > > > There are no control executed by cloudstack. For example, it can try to > > > instantiate a VM in a host when a GPU is already being used (will fail). > > > Our management is that the ROOT admin always controls that creation. We > > > launch all VMs using all GPUs from the infrastructure. Then we use a > > queue > > > manager to run jobs in those VMs with GPUs. When a user needs a dedicated > > > VM to develop something, we can shutdown a VM already running (that is > > part > > > of the queue manager as processor node) and then create this dedicated > > VM, > > > that uses the GPUs isolated. > > > > > > There are some possibilities when using GPUs. For example, some models > > > accept virtualization when we can divide a GPU. In that case, Cloudstack > > > would need to support that, so it would manage the driver, creating the > > > virtual GPUs based on information input from the user, as memory size. > > > Then, it should manage the hypervisor to passthrough the virtual gpu to > > VM. > > > > > > Another possibility that would help us in our scenario is to make some > > > control about PCI buses in hosts. For example, if Cloustack could check > > if > > > a PCI is being used in some host an
Re: Options to Monitor Cloudstack Events?
Hi Shiv, Thanks for the input! Yes we use Zabbix. Currently am thinking of using Zabbix Agent to call the Alerts API to get the info. (We already installed an Agent in the Cloudstack Management Server Anyways) Regards, Bryan On 12 Apr 2024 at 12:02 PM +0800, K B Shiv Kumar , wrote: > Hi > > You can use any of the following... > > The events table in the DB. > API for listing events. > RabbitMQ event listener. > > If you're using Zabbix, polling the event table periodically will help you > achieve your goal. > > Regards, > Shiv > (Sent from mobile device. Please excuse brevity and typos.) > > On Fri, 12 Apr 2024, 08:58 Bryan Tiang, wrote: > > > HI All, > > > > How do you guys monitor the Cloudstack Events as an Operator? > > > > Currently, the only way we know how to monitor those events is by logging > > in the GUI as a Root Admin. > > > > But this may not be possible/convenient if we are not in the office or > > travelling. > > > > Is there a way to monitor and send alerts via Telegram? Or Zabbix or > > Grafana Loki? > > > > So we can have the alerts sent to our phone if needed? > > > > Regards, > > Bryan > > > > -- > This message is intended only for the use of the individual or entity to > which it is addressed and may contain confidential and/or privileged > information. If you are not the intended recipient, please delete the > original message and any copy of it from your computer system. You are > hereby notified that any dissemination, distribution or copying of this > communication is strictly prohibited unless proper authorization has been > obtained for such action. If you have received this communication in error, > please notify the sender immediately. Although IndiQus attempts to sweep > e-mail and attachments for viruses, it does not guarantee that both are > virus-free and accepts no liability for any damage sustained as a result of > viruses.
Options to Monitor Cloudstack Events?
HI All, How do you guys monitor the Cloudstack Events as an Operator? Currently, the only way we know how to monitor those events is by logging in the GUI as a Root Admin. But this may not be possible/convenient if we are not in the office or travelling. Is there a way to monitor and send alerts via Telegram? Or Zabbix or Grafana Loki? So we can have the alerts sent to our phone if needed? Regards, Bryan
RE: Storage solution for Cloudstack
Hey Alexandru,
If you have questions on Linbit setup, you could try posting your questions and
logs to the Community Support on Slack here:
https://linbit.com/software-defined-storage/#:~:text=support%20your%20infrastructure.-,Open%20Source,-LINBIT%20SDS
Or, if you are using setting up Cloudstack for Commercial reasons, I suggest
contacting their support. They are very helpful people and can tell you exactly
where you’ve gone wrong, very quickly.
Regards,
Bryan
On 4 Apr 2024 at 10:10 PM +0800, Alexandru Stan
, wrote:
> Hi Bryan,
>
> Not much luck with this I'm afraid. I followed the documentation to the
> letter and I managed to add a 3-node primary storage in CS, I can see the
> random named linstor resources created by CS, yet I cannot create any
> instances, there's always error at startup, like the one bellow.
>
> Unable to orchestrate start VM instance
> {"id":18,"instanceName":"i-2-18-VM","type":"User","uuid":"867503d9-3912-47e4-85be-a30b3e088aa4"}
> due to [Create volume from template (ID = 4) failed: Resource definition
> 'cs-null' not found.].
>
> This is the method I used, except I didn't create a zfs pool, I created a
> thin-lvm one:
> https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-linstor-cloudstack-installing-linstor
>
> Any idea why this is happening?
>
>
> -Original Message-
> From: Bryan Tiang
> Sent: Friday, March 29, 2024 4:38 PM
> To: users@cloudstack.apache.org; users@cloudstack.apache.org
> Subject: RE: Storage solution for Cloudstack
>
> Hey Alexandru,
>
> The architecture is not quite the same because Linstors secret sauce is that
> they use DRBD, which is what allows them to get such high performance and low
> latency.
>
> You can check up more about it here:
> https://linbit.com/blog/how-does-linstor-compare-to-ceph/#:~:text=Ceph's%20use%20of%20the%20CRUSH,Ceph%20isn't%20operating%20normally.
>
> And some performance benchmarks with ceps, here:
> https://blog.palark.com/kubernetes-storage-performance-linstor-ceph-mayastor-vitastor/
>
> Regards,
> Bryan
> On 29 Mar 2024 at 4:11 PM +0800, Bryan Tiang ,
> wrote:
> >
> >
> > Bryan
Ways to Simulate Partial Host Failure for HA Testing
Hi Community, We are trying to test the HA Feature in cloudstack where all VMs which have HA Enabled will failover to a second node, AND cloudstack will fence off the affected node using IPMI when the node is no longer healthy. Of course, the most common way to simulate a host failure is to abruptly kill the power. However, in our experiences, in most cases in production, there scenario is where the node is not dead (cause that would be easy to tell), but it is affected in some way (Dying or sick, but not dead, which is a much harder way to detect.) Does anyone have any recommendation how else I can simulate a host issue where the host is problematic but not fully dead? Just to be able to see if cloudstack does pick it up, fence off the node successfully and restart the VMs in the new host. Regards, Bryan
RE: Storage solution for Cloudstack
Hey Alexandru, The architecture is not quite the same because Linstors secret sauce is that they use DRBD, which is what allows them to get such high performance and low latency. You can check up more about it here: https://linbit.com/blog/how-does-linstor-compare-to-ceph/#:~:text=Ceph's%20use%20of%20the%20CRUSH,Ceph%20isn't%20operating%20normally. And some performance benchmarks with ceps, here: https://blog.palark.com/kubernetes-storage-performance-linstor-ceph-mayastor-vitastor/ Regards, Bryan On 29 Mar 2024 at 4:11 PM +0800, Bryan Tiang , wrote: > > > Bryan
Re: Has anyone hit a performance/throughput limitation with Cloudstack Virtual Router yet?
Hi Community, Just to clarify, am asking this specifically for the Virtual Router. Regards, Bryan On 29 Mar 2024 at 10:04 PM +0800, Bryan Tiang , wrote: > Hi Community, > > My company builds Stock Trading Systems and we are using Cloudstack for > hosting the services we need for distributing Live Market Data Prices to > around 100,000 Concurrent Users. > > Id like to know, has anyone hit any performance/throughput limit with > Cloudstack before? And if so, what was your traffic like? What was your > scenario? How did you overcome it? > > Im asking this because if there was a limit, we’d like to know early on and > see how to avoid it. > > This is our setup, Per Datacenter. > > # 1 VPC with 2 Subnet > # Virtual Router System Offering set at 4Core,4GB memory (Hyperthreaded, no > oversubscription) > # Virtual Router has redundancy turned off > # Data Source -> Autoscale Group 1 (Around 20 VMs) -> Autoscale Group 2 > (Around 40 VMs) -> Autoscale Group 3 (Around 20 VMs) -> End Users via APIs > # Each VM is around 16 Core, 32GB (Hyperthreaded, no oversubscription) > # Because we are using VPCs, I believe Cloudstack only deploys 1 Virtual > Router that is used for all subnets, load balancing, autoscaling, routing, > NAT, Private Gateway etc. > # In our current On Prem Setup, our internet uplink can peak at 5Gb, Per > Datacenter. (We assume the traffic is the same in Cloud). > # Mainly distributing data via RestAPIs and Websocket APIs > > Regards, > Bryan
Re: Storage solution for Cloudstack
Hey Alexandru, We are using Cloudstack + Linstor right now and are live with a few clients. Its open source, performance is pretty great, and they support things like VM Snapshots, Volume Snapshots, Storage Replication, Encryption in Rest/Transit, VM HA and more. I think it suits what you’re looking for. Regards, Bryan On 29 Mar 2024 at 10:04 PM +0800, Alexandru Stan , wrote: > Hi everyone, > > We have a specific scenario in witch we want to have a functional Cloudstack > cluster but we can't seem to find/decide on a working storage solution, so if > anyone can recommend a setup that meets all our requirements, please do. So, > we are using KVM for hypervisors and we want to be able to offer vm snapshots > functionality to the clients and also vm HA - that excludes Ceph. If we go > for NFS, we don't have redundancy, since we're using nvme disks only and I > believe we all know what ar the choices of raid controllers for this right > now. > We're considering Storpool too, but we've yet to receive a price list, and we > assume it's not cheap but who knows. > > Thank you!
Has anyone hit a performance/throughput limitation with Cloudstack Virtual Router yet?
Hi Community, My company builds Stock Trading Systems and we are using Cloudstack for hosting the services we need for distributing Live Market Data Prices to around 100,000 Concurrent Users. Id like to know, has anyone hit any performance/throughput limit with Cloudstack before? And if so, what was your traffic like? What was your scenario? How did you overcome it? Im asking this because if there was a limit, we’d like to know early on and see how to avoid it. This is our setup, Per Datacenter. # 1 VPC with 2 Subnet # Virtual Router System Offering set at 4Core,4GB memory (Hyperthreaded, no oversubscription) # Virtual Router has redundancy turned off # Data Source -> Autoscale Group 1 (Around 20 VMs) -> Autoscale Group 2 (Around 40 VMs) -> Autoscale Group 3 (Around 20 VMs) -> End Users via APIs # Each VM is around 16 Core, 32GB (Hyperthreaded, no oversubscription) # Because we are using VPCs, I believe Cloudstack only deploys 1 Virtual Router that is used for all subnets, load balancing, autoscaling, routing, NAT, Private Gateway etc. # In our current On Prem Setup, our internet uplink can peak at 5Gb, Per Datacenter. (We assume the traffic is the same in Cloud). # Mainly distributing data via RestAPIs and Websocket APIs Regards, Bryan
Experience on GPU Support?
Hi Guys, Anyone running Cloudstack with GPU Support in Production? Say NVIDIA H100 or AMD M1300X? Just want to know if there is any support for this still on going, or anyone who is running a cloud business with GPUs. Regards, Bryan
URGENT: Unstable VM and VR Performance with Cloudstak and Cant seem to find root cause
Hi Community, Urgently need help on this. We are experience unstable performance with Cloudstack and have been having this issue since Monday... We're facing this error frequently and randomly. Unable to get answer that is of class com.cloud.agent.api.StartAnswer We encounter this during the following scenarios: # When VM failover to another host and is not able to start # When creating and starting a new VM # When starting an existing VM from Stopped state # When starting a stopped Virtual Router # When starting a new Virtual Router It happens very randomly and we can’t seem to identify a pattern. For example, when creating a VM fails, we literally just spam repeat the process and suddenly it will work. Or, sometimes we restart the VPC with Virtual Router Cleanup and it will suddenly work again. What we've done #Restarted management server #Removed cloudstack-agent and its directories in all Hypervisors #Increased CPU and memory for Virtual Router offering #Restarted Linstor Storage and Sattelite We are using Cloudstack 4.18.1 + Linstor + Ubuntu. On the host we applied CIS Benchmark hardening for Ubuntu 22.04 and AMD Memory Guard enabled. Regards, Bryan
Re: Cloudstack takes 5 Minutes to Realise Hypervisor is down.
Hi Nux and Wei, We got the 5 minutes by: (1) Killing the Server Power <- Time Start (2) When Cloudstack shows the Host is Offline <- Time Stop So this is what makes us think that 5 minutes for cloudstack to even detect that the host is down takes a long time. However, we notice as well that the moment Cloudstack Detects the host is down, all the VMs have already moved to other hosts. So in this case, if 2 minutes is not enough, what is the general rule of thumb? Regards, Bryan On 24 Jan 2024 at 5:16 PM +0800, Nux , wrote: > When dealing with these timeouts, try not to be too stingy with the > values. > 2 minutes may not be enough for other timeouts in your infra to complete > so that the whole process happens gracefully. > Whoever thought of 5 minutes as default must have had serious > considerations. > > On 2024-01-24 09:02, Wei ZHOU wrote: > > Hi, > > > > You can change the global setting "ping.interval" to 10 (seconds), > > restart > > the management service, and retest. > > > > -Wei > > > > On Wed, 24 Jan 2024 at 09:41, Bryan Tiang > > wrote: > > > > > Hi Community, > > > > > > We are performing failover test to see how quickly a VM will failover > > > to > > > another compute node in the event of a compute node hardware failure. > > > > > > We forcefully power off the compute node where the VM is residing and > > > monitor how long it takes to failover, and after a few tests the > > > result is > > > averagely around 5 mins for the compute node to be declared down and > > > for > > > the VM to failover to another node. > > > > > > Referring to > > > https://cwiki.apache.org/confluence/display/cloudstack/host+ha, we've > > > made a few changes in global settings hoping we can shorten the total > > > time > > > needed to failover but it doesn’t seem to work. > > > > > > We're wondering which settings that will effectively achieve our > > > objective, we're looking at around 2 mins total for failover to > > > complete. > > > > > > We are using Cloudstack 4.18.1 + Linstor + KVM Ubuntu > > > > > > Regards, > > > Bryan > > >
Cloudstack takes 5 Minutes to Realise Hypervisor is down.
Hi Community, We are performing failover test to see how quickly a VM will failover to another compute node in the event of a compute node hardware failure. We forcefully power off the compute node where the VM is residing and monitor how long it takes to failover, and after a few tests the result is averagely around 5 mins for the compute node to be declared down and for the VM to failover to another node. Referring to https://cwiki.apache.org/confluence/display/cloudstack/host+ha, we've made a few changes in global settings hoping we can shorten the total time needed to failover but it doesn’t seem to work. We're wondering which settings that will effectively achieve our objective, we're looking at around 2 mins total for failover to complete. We are using Cloudstack 4.18.1 + Linstor + KVM Ubuntu Regards, Bryan
Re: Unable to Setup Private Gateway
Hey Daan, We did restart the VPC with cleanup, but the new VRs spawned were still in UNKNOWN state We destroyed the VRs and replaced the VR template Regards, Bryan Sent with Spark On 22 Jan 2024 at 2:58 PM +0800, Daan Hoogland , wrote: > glad to hear you got through this. Strange though that the template > should be reinstalled. I am sure you mean that you re-instantiated the > router, right? As in restart network with cleanup? > > On Mon, Jan 22, 2024 at 7:30 AM Bryan Tiang wrote: > > > > Hi Daan, > > > > After few days of troubleshooting, we managed to resolve the issue. > > > > Change 1 - Reconfigure Network Config outside of Cloudstack > > Change 2 - Reinstall Router VR Template (Due to router in an unknown state) > > > > Regards, > > Bryan > > On 19 Jan 2024 at 10:26 PM +0800, Daan Hoogland , > > wrote: > > > Bryan, > > > Did you follow this: > > > http://docs.cloudstack.apache.org/en/4.18.1.0/adminguide/networking/virtual_private_cloud_config.html#adding-a-private-gateway-to-a-vpc? > > > what tools did you use to test it? > > > > > > On Wed, Jan 17, 2024 at 9:49 AM Bryan Tiang > > > wrote: > > > > > > > > Hi All, > > > > > > > > We're setting up Private Gateway on VPC but not able to get the > > > > connection to work. The connection is successful from switch to Private > > > > Gateway IP using VLAN 995. > > > > > > > > However when Private Gateway is created in VPC, the connection failed > > > > when tested from Virtual Router. > > > > > > > > Physical Network 3 is also used for Management & Public Network > > > > > > > > Is there any other config that we could have missed? > > > > > > > > Regards, > > > > Hunter > > > > > > > > > > > > -- > > > Daan > > > > -- > Daan
Re: Miscellaneous Questions about Cloudstack
Hi Dan, I see, my screenshots arent going through. Let me resend the questions and images again. Question 1: How can we get instances created to be done in a Project? It seems whatever we create is not captured within a project. Moreover, we can seem to be move items between projects. Is the only way to do it via direct API calls? Question 2: Refer to screenshot below. It say ‘RX 6.45GB’ and ’TX 4.42GB’. What is this exactly? It doesn’t look like bandwidth. Question 3: What are the out of bounds way to monitor the traffic of the private gateway in the VR? Question 4: We have a large range of public IP addresses. Under VPC > Public IP Addresses, we'd like to acquire a specific IP address that is pretty far down the range, but it's not available in the drop down box. When entered manually, the IP address is also not found. Regards, Bryan On 22 Jan 2024 at 3:08 PM +0800, Daan Hoogland , wrote: > On Mon, Jan 22, 2024 at 7:56 AM Bryan Tiang wrote: > > > Hi Community, > > > > Have some questions, about using cloudstack from UI perspective. > > > > *Question 1:* > > We have created a VPC and instances under ROOT>DOMAIN > > We are wondering if there's a direct way to move the VPC and instances to > > ROOT>DOMAIN>PROJECT instead, without having to create a new VPC in > > ROOT>DOMAIN>PROJ and migrate the instances over one by one > > > there are > https://cloudstack.apache.org/api/apidocs-4.18/apis/assignVirtualMachine.html > and https://cloudstack.apache.org/api/apidocs-4.18/apis/assignVolume.html. > We don't have such an API for networks and I don't think it is implemented > in the UI yet. > > > > *Question 2:* > > When viewing instance details, we can see ingress and egress Network > > utilization. Is this figure the total network usage for the instance? If > > yes, how is it calculated? Since last boot, since instance creation or > > something else? > > [image: WhatsApp Image 2024-01-22 at 1.22.19 PM.jpeg] > > > It is reported from the VR. > > > *Question 3:* > > We have setup a VPC private gateway to connect to a remote data center for > > specific use case. Is there a way to monitor the network utilization of > > this private gateway? Besides that, is there any way to limit the bandwidth > > usage and overall network utilization? > > > There are out-of-bounds way, but this functionality is not implemented in > the VR. > > *Question 4:* > > We have a large range of public IP addresses. Under VPC > Public IP > > Addresses, we'd like to acquire a specific IP address that is pretty far > > down the range, but it's not available in the drop down box. When entered > > manually, the IP address is also not found. > > > Can you try with an command line tool? like cmk? > sounds like either a misconfiguration or a bug. > > I think your images are blocked: > > > [image: WhatsApp Image 2024-01-22 at 1.40.13 PM.jpeg] > > > > Regards, > > Bryan > > > > regards, (and hope I didn't disappoint you too much;) > > -- > Daan
Miscellaneous Questions about Cloudstack
Hi Community, Have some questions, about using cloudstack from UI perspective. *Question 1:* We have created a VPC and instances under ROOT>DOMAIN We are wondering if there's a direct way to move the VPC and instances to ROOT>DOMAIN>PROJECT instead, without having to create a new VPC in ROOT>DOMAIN>PROJ and migrate the instances over one by one *Question 2:* When viewing instance details, we can see ingress and egress Network utilization. Is this figure the total network usage for the instance? If yes, how is it calculated? Since last boot, since instance creation or something else? [image: WhatsApp Image 2024-01-22 at 1.22.19 PM.jpeg] *Question 3:* We have setup a VPC private gateway to connect to a remote data center for specific use case. Is there a way to monitor the network utilization of this private gateway? Besides that, is there any way to limit the bandwidth usage and overall network utilization? *Question 4:* We have a large range of public IP addresses. Under VPC > Public IP Addresses, we'd like to acquire a specific IP address that is pretty far down the range, but it's not available in the drop down box. When entered manually, the IP address is also not found. [image: WhatsApp Image 2024-01-22 at 1.40.13 PM.jpeg] Regards, Bryan
Re: Unable to Setup Private Gateway
Hi Daan, After few days of troubleshooting, we managed to resolve the issue. Change 1 - Reconfigure Network Config outside of Cloudstack Change 2 - Reinstall Router VR Template (Due to router in an unknown state) Regards, Bryan On 19 Jan 2024 at 10:26 PM +0800, Daan Hoogland , wrote: > Bryan, > Did you follow this: > http://docs.cloudstack.apache.org/en/4.18.1.0/adminguide/networking/virtual_private_cloud_config.html#adding-a-private-gateway-to-a-vpc? > what tools did you use to test it? > > On Wed, Jan 17, 2024 at 9:49 AM Bryan Tiang wrote: > > > > Hi All, > > > > We're setting up Private Gateway on VPC but not able to get the connection > > to work. The connection is successful from switch to Private Gateway IP > > using VLAN 995. > > > > However when Private Gateway is created in VPC, the connection failed when > > tested from Virtual Router. > > > > Physical Network 3 is also used for Management & Public Network > > > > Is there any other config that we could have missed? > > > > Regards, > > Hunter > > > > -- > Daan
Re: Unable To Delete Secondary Storage
Morning All, Just to update, we managed to resolve the issue, but it had to be done manually from the db. Manually alter DB to remove templates from old zone Successfully removed old templates > Old Secondary Storage > Old Zone Changed "state" to "inactive" and added date & time to "removed" Regards, Bryan On 16 Jan 2024 at 12:53 PM +0800, Bryan Tiang , wrote: > Hi Community, > > We're trying to remove secondary storage but we need to delete the existing > templates first > However, we're receiving this error Failed to delete template > > Error message from management server > Jan 16 11:23:49 N2NCLOUDMY1MG01 java[87629]: ERROR > [o.a.c.s.i.BaseImageStoreDriverImpl] (API-Job-Executor-1:ctx-259f8594 > job-9377 ctx-9f923096) (logid:c104e949) No remote endpoint to send command, > check if host or ssvm is down? > > Ultimately we're trying to remove the zone, therefore we've also removed > hosts associated to this zone > > > > Regards, > Bryan
Unable to Setup Private Gateway
Hi All, We're setting up Private Gateway on VPC but not able to get the connection to work. The connection is successful from switch to Private Gateway IP using VLAN 995. However when Private Gateway is created in VPC, the connection failed when tested from Virtual Router. Physical Network 3 is also used for Management & Public Network Is there any other config that we could have missed? [image: WhatsApp Image 2024-01-17 at 4.30.14 PM.jpeg] Regards, Hunter
Unable To Delete Secondary Storage
Hi Community, We're trying to remove secondary storage but we need to delete the existing templates first However, we're receiving this error Failed to delete template Error message from management server Jan 16 11:23:49 N2NCLOUDMY1MG01 java[87629]: ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (API-Job-Executor-1:ctx-259f8594 job-9377 ctx-9f923096) (logid:c104e949) No remote endpoint to send command, check if host or ssvm is down? Ultimately we're trying to remove the zone, therefore we've also removed hosts associated to this zone Regards, Bryan
Re: Unable To Delete Physical Network
Hi Guys, After many trial and errors, we managed to delete the physical network. Root Cause: Double entries (namely A & B) in DB's "networks" table when creating Private Gateway. Suspected Reason: Suspect bug causing double entries (namely A & B) in DB's "networks" table when creating Private Gateway. Not sure if this is caused by using VXLAN which is not supported with Private Gateway. When Private Gateway is deleted, A's "state" is changed to "Destroy" but B's "state" did not change. Therefore, Physical Network cannot be deleted due to these records that aren't in "Destroy" state. Solution: Alter DB manually and change these records "state" to "Destroy" and add date & time to "removed" columns But just want to confirm if what we did was a good way of resolving the issue? Worried about side effects (break other things) from this manual change. Regards, Bryan On 12 Jan 2024 at 9:49 PM +0800, Suresh Kumar Anaparti , wrote: > Hi Bryan, > > Are you sure the instances, networks, etc are deleted properly? It's a > different error this time. > > Adding in detail with queries to check from your side. Before deleting a > physical network, the following are checked (in the given order) for the > physical network, and indicated through the respective message if the > check passes. If all checks fail, the physical network is deleted. > > (i) any vlans allocated, in op_dc_vnet_alloc table > SELECT * FROM cloud.op_dc_vnet_alloc WHERE physical_network_id = network integer id> AND taken IS NOT NULL > > (ii) any guest networks exists, in networks table > SELECT * FROM cloud.networks WHERE physical_network_id = integer id> AND removed IS NULL > > (iii) any IP addresses allocated, in user_ip_address table > SELECT * FROM cloud.user_ip_address WHERE physical_network_id = network integer id> AND state != 'Free' > > Get the physical network integer id using => SELECT id FROM > cloud.physical_network WHERE uuid = '' > > Hope this helps. > > Regards, > Suresh > > On Fri, Jan 12, 2024 at 1:29 PM Wei ZHOU wrote: > > > Hi, > > > > As said in the error message, there are some networks associated with this > > physical network. > > > > SELECT * FROM networks WHERE physical_network_id= AND > > removed IS NULL; > > > > -Wei > > > > > > On Fri, 12 Jan 2024 at 08:42, Bryan Tiang > > wrote: > > > > > Hi Suresh, > > > > > > We tried that as well but still got the same result. > > > > > > This is really interesting… We are scratching our heads. > > > > > > > > > > > > Regards, > > > Bryan > > > On 12 Jan 2024 at 1:30 PM +0800, Suresh Kumar Anaparti < > > > sureshkumar.anapa...@gmail.com>, wrote: > > > > > > Hi Bryan, > > > > > > It's the integer id of the physical network, not uuid. > > > > > > with uuid, check with this. > > > SELECT * FROM cloud.op_dc_vnet_alloc WHERE physical_network_id = (SELECT > > > id FROM cloud.physical_network WHERE uuid = '') AND > > > taken IS NOT NULL > > > > > > Regards, > > > Suresh > > > > > > On Fri, Jan 12, 2024 at 10:55 AM Bryan Tiang > > > wrote: > > > > > > > Hi Suresh, > > > > > > > > So sorry for this. But we ran the query and this was the result. > > > > > > > > <> > > > > > > > > Could it be something else? > > > > > > > > Regards, > > > > Bryan > > > > On 11 Jan 2024 at 8:39 PM +0800, Suresh Kumar Anaparti < > > > > sureshkumar.anapa...@gmail.com>, wrote: > > > > > > > > Hi Bryan, > > > > > > > > From the code, the error is due to some allocated vlans found through > > > > the > > > > below query. If the result is empty, that error doesn't appear. > > > > > > > > SELECT * FROM cloud.op_dc_vnet_alloc WHERE physical_network_id = > > > > > > > network id> AND taken IS NOT NULL > > > > > > > > Regards, > > > > Suresh > > > > > > > > On Thu, Jan 11, 2024 at 3:57 PM Bryan Tiang > > > > wrote: > > > > > > > > Hi Suresh, > > > > > > > > Sorry, its Version 4.18.1 > > > > > > > > Yes its the same error message as before: > > > > > > > > Delete physical network: (Physical Network 1) The Physical Ne
Re: Unable To Delete Physical Network
Hi Suresh, We tried that as well but still got the same result. This is really interesting… We are scratching our heads. Regards, Bryan On 12 Jan 2024 at 1:30 PM +0800, Suresh Kumar Anaparti , wrote: > Hi Bryan, > > It's the integer id of the physical network, not uuid. > > with uuid, check with this. > SELECT * FROM cloud.op_dc_vnet_alloc WHERE physical_network_id = (SELECT id > FROM cloud.physical_network WHERE uuid = '') AND taken > IS NOT NULL > > Regards, > Suresh > > > On Fri, Jan 12, 2024 at 10:55 AM Bryan Tiang > > wrote: > > > Hi Suresh, > > > > > > So sorry for this. But we ran the query and this was the result. > > > > > > <> > > > > > > Could it be something else? > > > > > > Regards, > > > Bryan > > > On 11 Jan 2024 at 8:39 PM +0800, Suresh Kumar Anaparti > > > , wrote: > > > > Hi Bryan, > > > > > > > > From the code, the error is due to some allocated vlans found through > > > > the > > > > below query. If the result is empty, that error doesn't appear. > > > > > > > > SELECT * FROM cloud.op_dc_vnet_alloc WHERE physical_network_id = > > > > > > > network id> AND taken IS NOT NULL > > > > > > > > Regards, > > > > Suresh > > > > > > > > On Thu, Jan 11, 2024 at 3:57 PM Bryan Tiang > > > > wrote: > > > > > > > > > Hi Suresh, > > > > > > > > > > Sorry, its Version 4.18.1 > > > > > > > > > > Yes its the same error message as before: > > > > > > > > > > Delete physical network: (Physical Network 1) The Physical Network is > > > > > not > > > > > deletable because there are allocated vnets for this physical network > > > > > > > > > > Version: 4.18.1 > > > > > Hypervisor: Linux Ubuntu > > > > > Storage: Linbit > > > > > > > > > > Regards, > > > > > Bryan > > > > > On 11 Jan 2024 at 5:46 PM +0800, Suresh Kumar Anaparti < > > > > > sureshkumar.anapa...@gmail.com>, wrote: > > > > > > Hi Bryan, > > > > > > > > > > > > If 'taken' / other columns are NULL, you should see different > > > > > > error. Can > > > > > > you confirm if it is still the same error for that physical network? > > > > > What's > > > > > > the cloudstack version you are using? > > > > > > > > > > > > Regards, > > > > > > Suresh > > > > > > > > > > > > On Thu, Jan 11, 2024 at 2:13 PM Bryan Tiang > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Suresh, > > > > > > > > > > > > > > We set it Null and got the same error: > > > > > > > > > > > > > > i. Updated 'taken' to NULL - same error > > > > > > > ii. Updated the other columns to NULL - same error > > > > > > > iii. Restarted management service and retry - same error > > > > > > > > > > > > > > Did we do something wrong? > > > > > > > <> > > > > > > > > > > > > > > Regards, > > > > > > > Bryan > > > > > > > > > > > > On Thu, Jan 11, 2024 at 1:53 PM Suresh Kumar Anaparti < > > > > > > sureshkumar.anapa...@gmail.com> wrote: > > > > > > > > > > > > > Hi Bryan, > > > > > > > > > > > > > > If there are no network / nics for the vlan, you can update > > > > > > > 'taken' to > > > > > > > NULL for that vlan in the op_dc_vnet_alloc table. > > > > > > > > > > > > > > Check network / nics usage with the below queries before update. > > > > > > > SELECT * FROM cloud.networks WHERE broadcast_uri LIKE '% > > > > > > id>%'; > > > > > > > SELECT * FROM cloud.nics where broadcast_uri LIKE '%%'; > > > > > > > > > > > > > > Regards, > > > > > > > Suresh > > > > > > > > > > > > > > On Thu, Jan 11, 2024 at 1:25 PM Bryan Tiang
Re: Unable To Delete Physical Network
Hi Suresh, So sorry for this. But we ran the query and this was the result. Could it be something else? Regards, Bryan On 11 Jan 2024 at 8:39 PM +0800, Suresh Kumar Anaparti , wrote: > Hi Bryan, > > From the code, the error is due to some allocated vlans found through the > below query. If the result is empty, that error doesn't appear. > > SELECT * FROM cloud.op_dc_vnet_alloc WHERE physical_network_id = network id> AND taken IS NOT NULL > > Regards, > Suresh > > On Thu, Jan 11, 2024 at 3:57 PM Bryan Tiang > wrote: > > > Hi Suresh, > > > > Sorry, its Version 4.18.1 > > > > Yes its the same error message as before: > > > > Delete physical network: (Physical Network 1) The Physical Network is not > > deletable because there are allocated vnets for this physical network > > > > Version: 4.18.1 > > Hypervisor: Linux Ubuntu > > Storage: Linbit > > > > Regards, > > Bryan > > On 11 Jan 2024 at 5:46 PM +0800, Suresh Kumar Anaparti < > > sureshkumar.anapa...@gmail.com>, wrote: > > > Hi Bryan, > > > > > > If 'taken' / other columns are NULL, you should see different error. Can > > > you confirm if it is still the same error for that physical network? > > What's > > > the cloudstack version you are using? > > > > > > Regards, > > > Suresh > > > > > > On Thu, Jan 11, 2024 at 2:13 PM Bryan Tiang > > > wrote: > > > > > > > Hi Suresh, > > > > > > > > We set it Null and got the same error: > > > > > > > > i. Updated 'taken' to NULL - same error > > > > ii. Updated the other columns to NULL - same error > > > > iii. Restarted management service and retry - same error > > > > > > > > Did we do something wrong? > > > > <> > > > > > > > > Regards, > > > > Bryan > > > > > > On Thu, Jan 11, 2024 at 1:53 PM Suresh Kumar Anaparti < > > > sureshkumar.anapa...@gmail.com> wrote: > > > > > > > Hi Bryan, > > > > > > > > If there are no network / nics for the vlan, you can update 'taken' to > > > > NULL for that vlan in the op_dc_vnet_alloc table. > > > > > > > > Check network / nics usage with the below queries before update. > > > > SELECT * FROM cloud.networks WHERE broadcast_uri LIKE '%%'; > > > > SELECT * FROM cloud.nics where broadcast_uri LIKE '%%'; > > > > > > > > Regards, > > > > Suresh > > > > > > > > On Thu, Jan 11, 2024 at 1:25 PM Bryan Tiang > > > > wrote: > > > > > > > > > Hi Suresh, > > > > > > > > > > Thank you! In this case, should we clear the data directly? > > > > > > > > > > Clear or amend to something else? > > > > > > > > > > Regards, > > > > > Bryan > > > > > On 11 Jan 2024 at 3:50 PM +0800, Suresh Kumar Anaparti < > > > > > sureshkumar.anapa...@gmail.com>, wrote: > > > > > > Hi Bryan, > > > > > > > > > > > > It seems there are still some vlans allocated. You can manually > > confirm > > > > > from the db table 'op_dc_vnet_alloc'. > > > > > > > > > > > > Check with the query below. > > > > > > SELECT vnet, taken FROM cloud.op_dc_vnet_alloc WHERE > > > > > physical_network_id = (SELECT id FROM cloud.physical_network WHERE > > uuid = > > > > > '') > > > > > > > > > > > > Regards, > > > > > > Suresh > > > > > > > > > > > > > On Thu, Jan 11, 2024 at 11:19 AM Bryan Tiang < > > > > > bryantian...@hotmail.com> wrote: > > > > > > > > Hi Guys, > > > > > > > > > > > > > > > > Our setup is using VXLAN For Guest Networks. > > > > > > > > > > > > > > > > We're trying to delete physical network but encountered this > > error. > > > > > > > > > > > > > > > > We've deleted all the instances, networks, system vms, hosts, > > pods > > > > > and clusters and left with only the zone (disabled). > > > > > > > > > > > > > > > > Can someone please help tell us what we are missing? > > > > > > > > > > > > > > > > <> > > > > > > > > <> > > > > > > > > Regards, > > > > > > > > Bryan > > > > > > > > > > >
Re: Unable To Delete Physical Network
Hi Suresh, Sorry, its Version 4.18.1 Yes its the same error message as before: Delete physical network: (Physical Network 1) The Physical Network is not deletable because there are allocated vnets for this physical network Version: 4.18.1 Hypervisor: Linux Ubuntu Storage: Linbit Regards, Bryan On 11 Jan 2024 at 5:46 PM +0800, Suresh Kumar Anaparti , wrote: > Hi Bryan, > > If 'taken' / other columns are NULL, you should see different error. Can > you confirm if it is still the same error for that physical network? What's > the cloudstack version you are using? > > Regards, > Suresh > > On Thu, Jan 11, 2024 at 2:13 PM Bryan Tiang > wrote: > > > Hi Suresh, > > > > We set it Null and got the same error: > > > > i. Updated 'taken' to NULL - same error > > ii. Updated the other columns to NULL - same error > > iii. Restarted management service and retry - same error > > > > Did we do something wrong? > > <> > > > > Regards, > > Bryan > > On Thu, Jan 11, 2024 at 1:53 PM Suresh Kumar Anaparti < > sureshkumar.anapa...@gmail.com> wrote: > > > Hi Bryan, > > > > If there are no network / nics for the vlan, you can update 'taken' to > > NULL for that vlan in the op_dc_vnet_alloc table. > > > > Check network / nics usage with the below queries before update. > > SELECT * FROM cloud.networks WHERE broadcast_uri LIKE '%%'; > > SELECT * FROM cloud.nics where broadcast_uri LIKE '%%'; > > > > Regards, > > Suresh > > > > On Thu, Jan 11, 2024 at 1:25 PM Bryan Tiang > > wrote: > > > > > Hi Suresh, > > > > > > Thank you! In this case, should we clear the data directly? > > > > > > Clear or amend to something else? > > > > > > Regards, > > > Bryan > > > On 11 Jan 2024 at 3:50 PM +0800, Suresh Kumar Anaparti < > > > sureshkumar.anapa...@gmail.com>, wrote: > > > > Hi Bryan, > > > > > > > > It seems there are still some vlans allocated. You can manually confirm > > > from the db table 'op_dc_vnet_alloc'. > > > > > > > > Check with the query below. > > > > SELECT vnet, taken FROM cloud.op_dc_vnet_alloc WHERE > > > physical_network_id = (SELECT id FROM cloud.physical_network WHERE uuid = > > > '') > > > > > > > > Regards, > > > > Suresh > > > > > > > > > On Thu, Jan 11, 2024 at 11:19 AM Bryan Tiang < > > > bryantian...@hotmail.com> wrote: > > > > > > Hi Guys, > > > > > > > > > > > > Our setup is using VXLAN For Guest Networks. > > > > > > > > > > > > We're trying to delete physical network but encountered this error. > > > > > > > > > > > > We've deleted all the instances, networks, system vms, hosts, pods > > > and clusters and left with only the zone (disabled). > > > > > > > > > > > > Can someone please help tell us what we are missing? > > > > > > > > > > > > <> > > > > > > <> > > > > > > Regards, > > > > > > Bryan > > > > >
Re: Unable To Delete Physical Network
Hi Suresh, Yes its the same error message as before: Delete physical network: (Physical Network 1) The Physical Network is not deletable because there are allocated vnets for this physical network Version: 4.18 Hypervisor: Linux Ubuntu Storage: Linbit Regards, Bryan On 11 Jan 2024 at 5:46 PM +0800, Suresh Kumar Anaparti , wrote: > Hi Bryan, > > If 'taken' / other columns are NULL, you should see different error. Can > you confirm if it is still the same error for that physical network? What's > the cloudstack version you are using? > > Regards, > Suresh > > On Thu, Jan 11, 2024 at 2:13 PM Bryan Tiang > wrote: > > > Hi Suresh, > > > > We set it Null and got the same error: > > > > i. Updated 'taken' to NULL - same error > > ii. Updated the other columns to NULL - same error > > iii. Restarted management service and retry - same error > > > > Did we do something wrong? > > <> > > > > Regards, > > Bryan > > On Thu, Jan 11, 2024 at 1:53 PM Suresh Kumar Anaparti < > sureshkumar.anapa...@gmail.com> wrote: > > > Hi Bryan, > > > > If there are no network / nics for the vlan, you can update 'taken' to > > NULL for that vlan in the op_dc_vnet_alloc table. > > > > Check network / nics usage with the below queries before update. > > SELECT * FROM cloud.networks WHERE broadcast_uri LIKE '%%'; > > SELECT * FROM cloud.nics where broadcast_uri LIKE '%%'; > > > > Regards, > > Suresh > > > > On Thu, Jan 11, 2024 at 1:25 PM Bryan Tiang > > wrote: > > > > > Hi Suresh, > > > > > > Thank you! In this case, should we clear the data directly? > > > > > > Clear or amend to something else? > > > > > > Regards, > > > Bryan > > > On 11 Jan 2024 at 3:50 PM +0800, Suresh Kumar Anaparti < > > > sureshkumar.anapa...@gmail.com>, wrote: > > > > Hi Bryan, > > > > > > > > It seems there are still some vlans allocated. You can manually confirm > > > from the db table 'op_dc_vnet_alloc'. > > > > > > > > Check with the query below. > > > > SELECT vnet, taken FROM cloud.op_dc_vnet_alloc WHERE > > > physical_network_id = (SELECT id FROM cloud.physical_network WHERE uuid = > > > '') > > > > > > > > Regards, > > > > Suresh > > > > > > > > > On Thu, Jan 11, 2024 at 11:19 AM Bryan Tiang < > > > bryantian...@hotmail.com> wrote: > > > > > > Hi Guys, > > > > > > > > > > > > Our setup is using VXLAN For Guest Networks. > > > > > > > > > > > > We're trying to delete physical network but encountered this error. > > > > > > > > > > > > We've deleted all the instances, networks, system vms, hosts, pods > > > and clusters and left with only the zone (disabled). > > > > > > > > > > > > Can someone please help tell us what we are missing? > > > > > > > > > > > > <> > > > > > > <> > > > > > > Regards, > > > > > > Bryan > > > > >
Re: Unable To Delete Physical Network
Hi Suresh, We set it Null and got the same error: i. Updated 'taken' to NULL - same error ii. Updated the other columns to NULL - same error iii. Restarted management service and retry - same error Did we do something wrong? Regards, Bryan On 11 Jan 2024 at 4:24 PM +0800, Suresh Kumar Anaparti , wrote: > Hi Bryan, > > If there are no network / nics for the vlan, you can update 'taken' to NULL > for that vlan in the op_dc_vnet_alloc table. > > Check network / nics usage with the below queries before update. > SELECT * FROM cloud.networks WHERE broadcast_uri LIKE '%%'; > SELECT * FROM cloud.nics where broadcast_uri LIKE '%%'; > > Regards, > Suresh > > On Thu, Jan 11, 2024 at 1:25 PM Bryan Tiang > wrote: > > > Hi Suresh, > > > > Thank you! In this case, should we clear the data directly? > > > > Clear or amend to something else? > > > > Regards, > > Bryan > > On 11 Jan 2024 at 3:50 PM +0800, Suresh Kumar Anaparti < > > sureshkumar.anapa...@gmail.com>, wrote: > > > Hi Bryan, > > > > > > It seems there are still some vlans allocated. You can manually confirm > > from the db table 'op_dc_vnet_alloc'. > > > > > > Check with the query below. > > > SELECT vnet, taken FROM cloud.op_dc_vnet_alloc WHERE physical_network_id > > = (SELECT id FROM cloud.physical_network WHERE uuid = ' > uuid>') > > > > > > Regards, > > > Suresh > > > > > > > On Thu, Jan 11, 2024 at 11:19 AM Bryan Tiang > > wrote: > > > > > Hi Guys, > > > > > > > > > > Our setup is using VXLAN For Guest Networks. > > > > > > > > > > We're trying to delete physical network but encountered this error. > > > > > > > > > > We've deleted all the instances, networks, system vms, hosts, pods > > and clusters and left with only the zone (disabled). > > > > > > > > > > Can someone please help tell us what we are missing? > > > > > > > > > > <> > > > > > <> > > > > > Regards, > > > > > Bryan > >
Re: Unable To Delete Physical Network
Hi Suresh, Thank you! In this case, should we clear the data directly? Clear or amend to something else? Regards, Bryan On 11 Jan 2024 at 3:50 PM +0800, Suresh Kumar Anaparti , wrote: > Hi Bryan, > > It seems there are still some vlans allocated. You can manually confirm from > the db table 'op_dc_vnet_alloc'. > > Check with the query below. > SELECT vnet, taken FROM cloud.op_dc_vnet_alloc WHERE physical_network_id = > (SELECT id FROM cloud.physical_network WHERE uuid = '') > > Regards, > Suresh > > > On Thu, Jan 11, 2024 at 11:19 AM Bryan Tiang > > wrote: > > > Hi Guys, > > > > > > Our setup is using VXLAN For Guest Networks. > > > > > > We're trying to delete physical network but encountered this error. > > > > > > We've deleted all the instances, networks, system vms, hosts, pods and > > > clusters and left with only the zone (disabled). > > > > > > Can someone please help tell us what we are missing? > > > > > > <> > > > <> > > > Regards, > > > Bryan
Unable To Delete Physical Network
Hi Guys,
Our setup is using VXLAN For Guest Networks.
We're trying to delete physical network but encountered this error.
We've deleted all the instances, networks, system vms, hosts, pods and clusters
and left with only the zone (disabled).
Can someone please help tell us what we are missing?
Regards,
Bryan
Jan 11 11:38:39 N2NCLOUDMY1MG01 java[990006]: INFO [c.c.s.StatsCollector]
(StatsCollector-1:ctx-ae5995fc) (logid:5b607178) system memory from /proc:
16744308736
Jan 11 11:38:39 N2NCLOUDMY1MG01 java[990006]: INFO [c.c.s.StatsCollector]
(StatsCollector-1:ctx-ae5995fc) (logid:5b607178) free memory from /proc:
8095809536
Jan 11 11:38:39 N2NCLOUDMY1MG01 java[990006]: INFO [c.c.s.StatsCollector]
(StatsCollector-1:ctx-ae5995fc) (logid:5b607178) used memory from /proc: 932188
Jan 11 11:38:39 N2NCLOUDMY1MG01 java[990006]: ERROR
[c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:ctx-2e14bd30)
(logid:6cad7a80) IOException from : http://10.21.1.38:9090/clusterservice,
method : name=method, value=5
Jan 11 11:38:39 N2NCLOUDMY1MG01 java[990006]: ERROR
[c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:ctx-2e14bd30)
(logid:6cad7a80) IOException from : http://10.21.1.38:9090/clusterservice,
method : name=method, value=5
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1550261631-290:ctx-39820611
ctx-55ae4289) (logid:9ceb8fa2) Account [Account
[{"accountName":"admin","id":2,"uuid":"698819f7-9ef8-11ee-8ec2-5254007314cc"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1550261631-290:ctx-39820611
ctx-55ae4289) (logid:9ceb8fa2) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[deletePhysicalNetwork] for user [User
{"username":"admin","uuid":"6988be55-9ef8-11ee-8ec2-5254007314cc"}.] as allowed.
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: INFO
[o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-34:ctx-54cc2479 job-6743)
(logid:ad061e97) Add job-6743 into job monitoring
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: ERROR
[c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-34:ctx-54cc2479 job-6743)
(logid:1dfb6a24) Unexpected exception while executing
org.apache.cloudstack.api.command.admin.network.DeletePhysicalNetworkCmd
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]:
com.cloud.utils.exception.CloudRuntimeException: The Physical Network is not
deletable because there are allocated vnets for this physical network
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
com.cloud.network.NetworkServiceImpl.checkIfPhysicalNetworkIsDeletable(NetworkServiceImpl.java:4387)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
com.cloud.network.NetworkServiceImpl.deletePhysicalNetwork(NetworkServiceImpl.java:4253)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
jdk.internal.reflect.GeneratedMethodAccessor355.invoke(Unknown Source)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
Jan 11 11:38:41 N2NCLOUDMY1MG01 java[990006]: at
com.sun.proxy.$Proxy278.deletePhysicalNetwork(Unknown Source)
Jan 11
Re: New committer: Alexandre Mattioli
Congratulations Alex, this is well deserved. Regards, Bryan On 11 Jan 2024 at 10:18 AM +0800, Nicolas Vazquez , wrote: > Congratulations Alex! > > Regards, > Nicolas Vazquez > > > From: Nux > Date: Wednesday, 10 January 2024 at 19:28 > To: Dev , Users > Subject: New committer: Alexandre Mattioli > All, > > The Project Management Committee (PMC) for Apache CloudStack > has invited Alexandre Mattioli to become a committer and we are pleased > to announce that they have accepted. > > Alex has been instrumental in many features present today in Cloudstack, > with a focus on networking and VMWare: > - IPv6 static routing > - Edge Zones > - Autoscaling with VR > - VNF appliances > - VMWare NSX support > - Tungsten Fabric / OpenSDN > - Backup & recovery framework > - VLAN trunking and security policies in ESX > and so on. > > > Please join me in congratulating Alex! > > >
Re: Can’t bring up the SystemVM Control Proxy and SSVM.
Hi Suresh,
Resending this because previous attachment was too small to be seen:
Thanks for the help. What we found is that if we use VLAN to setup Cloudstack,
it works fine.
But when we setup using VxLAN/VNI, we encounter the issue where we are unable
to create a network VNI interface to SystemVM (Console Proxy and SSVM).
Anyone set up using VxLAN/VNI before? Below is our configuration:
#Hypervisor KVM: Ubuntu 22.04 LTS with LACP bond (ports 1 and 2), then bridge
interface network as cloudbr1.
#Management OS: Ubuntu 22.04 LTS
Cloudstack version: 4.18.1.0 with VNI network method.
Regards,
Bryan
On 28 Dec 2023 at 9:11 PM +0800, Bryan Tiang , wrote:
> Hi Suresh,
>
> Thanks for the help. What we found is that if we use VLAN to setup
> Cloudstack, it works fine.
>
> But when we setup using VxLAN/VNI, we encounter the issue where we are unable
> to create a network VNI interface to SystemVM (Console Proxy and SSVM).
>
> Anyone set up using VxLAN/VNI before? Below is our configuration:
>
> #Hypervisor KVM: Ubuntu 22.04 LTS with LACP bond (ports 1 and 2), then bridge
> interface network as cloudbr1.
> #Management OS: Ubuntu 22.04 LTS
>
> Cloudstack version: 4.18.1.0 with VNI network method.
>
>
> Regards,
> Bryan
> On 27 Dec 2023 at 3:43 PM +0800, Suresh Kumar Anaparti
> , wrote:
> > Hi Bryan,
> >
> > From the logs, I could see some below (*VNI: integer expression
> > expectedError: argument "VNI" is wrong) * during network configuration.
> > Please check your network config.
> >
> > *2023-12-22 19:09:21,543 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-68:ctx-79cd030e job-215/job-687 ctx-3820f5d5)
> > (logid:3f700e86) Seq 3-3746431940018831724: Received: { Ans: , MgmtId:
> > 90520732211830, via: 3(n2ncloudmy1cp04), Ver: v1, Flags: 10, { StartAnswer,
> > Answer } }*
> >
> > Error here:
> > *2023-12-22 19:09:21,548 INFO [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-68:ctx-79cd030e job-215/job-687 ctx-3820f5d5)
> > (logid:3f700e86) Unable to start VM on Host
> > {"id":3,"name":"n2ncloudmy1cp04","type":"Routing","uuid":"a627f179-865b-49da-98af-61c0b3ff62d4"}
> > due to Failed to create vnet VNI:
> > /usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh: line
> > 173: [: VNI: integer expression expectedError: argument "VNI" is wrong: id
> > is
> > invalid/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh:
> > line 37: /proc/sys/net/ipv6/conf/bond1.VNI/disable_ipv6: No such file or
> > directoryCannot find device "bond1.VNI"Failed to create vlan VNI on pif:
> > bond1.*
> >
> > *2023-12-22 19:09:21,550 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-68:ctx-79cd030e job-215/job-687 ctx-3820f5d5)
> > (logid:3f700e86) Cleaning up resources for the vm VM instance
> > {"id":81,"instanceName":"v-81-VM","type":"ConsoleProxy","uuid":"819c6d02-99f0-4674-bf8f-daf8417eb5fc"}
> > in Starting state*
> >
> > *2023-12-22 19:09:21,570 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-68:ctx-79cd030e job-215/job-687 ctx-3820f5d5)
> > (logid:3f700e86) Seq 3-3746431940018831725: Sending { Cmd , MgmtId:
> > 90520732211830, via: 3(n2ncloudmy1cp04), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.StopCommand":*
> >
> >
> > Regards,
> > Suresh
> >
> >
> > On Wed, Dec 27, 2023 at 12:47 PM Bryan Tiang
> > wrote:
> >
> > > Hi Community,
> > >
> > > We tried re-installing cloudstack and now we can’t bring up the SystemVM
> > > Control Proxy and SSVM.
> > >
> > > Could someone tell us what we’re missing out? Ive attached our management
> > > log files.
> > >
> > > Regards,
> > > Bryan
> > >
Re: Can’t bring up the SystemVM Control Proxy and SSVM.
Hi Suresh,
Thanks for the help. What we found is that if we use VLAN to setup Cloudstack,
it works fine.
But when we setup using VxLAN/VNI, we encounter the issue where we are unable
to create a network VNI interface to SystemVM (Console Proxy and SSVM).
Anyone set up using VxLAN/VNI before? Below is our configuration:
#Hypervisor KVM: Ubuntu 22.04 LTS with LACP bond (ports 1 and 2), then bridge
interface network as cloudbr1.
#Management OS: Ubuntu 22.04 LTS
Cloudstack version: 4.18.1.0 with VNI network method.
Regards,
Bryan
On 27 Dec 2023 at 3:43 PM +0800, Suresh Kumar Anaparti
, wrote:
> Hi Bryan,
>
> From the logs, I could see some below (*VNI: integer expression
> expectedError: argument "VNI" is wrong) * during network configuration.
> Please check your network config.
>
> *2023-12-22 19:09:21,543 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-68:ctx-79cd030e job-215/job-687 ctx-3820f5d5)
> (logid:3f700e86) Seq 3-3746431940018831724: Received: { Ans: , MgmtId:
> 90520732211830, via: 3(n2ncloudmy1cp04), Ver: v1, Flags: 10, { StartAnswer,
> Answer } }*
>
> Error here:
> *2023-12-22 19:09:21,548 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-68:ctx-79cd030e job-215/job-687 ctx-3820f5d5)
> (logid:3f700e86) Unable to start VM on Host
> {"id":3,"name":"n2ncloudmy1cp04","type":"Routing","uuid":"a627f179-865b-49da-98af-61c0b3ff62d4"}
> due to Failed to create vnet VNI:
> /usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh: line
> 173: [: VNI: integer expression expectedError: argument "VNI" is wrong: id
> is
> invalid/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh:
> line 37: /proc/sys/net/ipv6/conf/bond1.VNI/disable_ipv6: No such file or
> directoryCannot find device "bond1.VNI"Failed to create vlan VNI on pif:
> bond1.*
>
> *2023-12-22 19:09:21,550 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-68:ctx-79cd030e job-215/job-687 ctx-3820f5d5)
> (logid:3f700e86) Cleaning up resources for the vm VM instance
> {"id":81,"instanceName":"v-81-VM","type":"ConsoleProxy","uuid":"819c6d02-99f0-4674-bf8f-daf8417eb5fc"}
> in Starting state*
>
> *2023-12-22 19:09:21,570 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-68:ctx-79cd030e job-215/job-687 ctx-3820f5d5)
> (logid:3f700e86) Seq 3-3746431940018831725: Sending { Cmd , MgmtId:
> 90520732211830, via: 3(n2ncloudmy1cp04), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":*
>
>
> Regards,
> Suresh
>
>
> On Wed, Dec 27, 2023 at 12:47 PM Bryan Tiang
> wrote:
>
> > Hi Community,
> >
> > We tried re-installing cloudstack and now we can’t bring up the SystemVM
> > Control Proxy and SSVM.
> >
> > Could someone tell us what we’re missing out? Ive attached our management
> > log files.
> >
> > Regards,
> > Bryan
> >
RE: VM Firewalls In Between Subnets
Thank You, Gentlemen. Will give this a try! On 21 Nov 2023 at 7:14 PM +0800, Alex Mattioli , wrote: > +1 to that > > > > > -Original Message- > From: Stephan Bienek > Sent: Tuesday, November 21, 2023 9:15 AM > To: users@cloudstack.apache.org > Subject: Re: VM Firewalls In Between Subnets > > Hi Bryan, > > sure you can, for example using multiple L2 networks connected to your > virtual appliance. > L2 network will not interfere with any services of your appliance, as there > is no virtual router involved. > > Spreading the VMs to the different L2 networks, their only way to communicate > is via your appliance of choice. > > Use L2 with ConfigDrive network offerings to keep the possibility of > providing UserData for Cloud-Init etc if required. > > Using L2 networks with "specify VLAN" you could even use an maybe already > existing physical or virtual appliance outside of Cloudstack within the > specified VLANs. > > Best regards, > Stephan > > > Bryan Tiang hat am 21.11.2023 07:40 CET > > geschrieben: > > > > > > Hi All, > > > > I have a financial client who requires 3 subnets, each filtered by a > > firewall. > > > > They didnt accept the idea of using Network ACLs. They want packet > > filtering, intrusion prevention systems etc which are all features of a > > full fledged firewall. > > > > Can i install a VM Firewall from Fortinet or Palo Alto, and get achieve the > > subnet segregation? All via cloudstack? > > > > Regards, > > Bryan > > > > Sent with Spark
Bank Client requires VM Firewall in between subnets
Hi All, I have a potential client who is a bank, and requires a VPC, 3 Subnets, with each subnet segregated by a firewall. We proposed the idea of using Network ACLs, but they didn’t accept the idea. They want packet filtering, intrusion prevention features etc which are all features of a full fledge firewall. Is it possible to install a VM Firewall from Fortinet/Palo Alto, use those to segregate the traffic between the 3 subnets instead? All via cloudstack? If not, any idea how I could go around this? Regards, Bryan
VM Firewalls In Between Subnets
Hi All, I have a financial client who requires 3 subnets, each filtered by a firewall. They didnt accept the idea of using Network ACLs. They want packet filtering, intrusion prevention systems etc which are all features of a full fledged firewall. Can i install a VM Firewall from Fortinet or Palo Alto, and get achieve the subnet segregation? All via cloudstack? Regards, Bryan Sent with Spark
Re: Swapping Public IP Addresses
Hi Community, Our current Cloudstack is setup with old public IP addresses is assigned to our zone infrastructure. Ongoing next month, we are going to change telco and require re-assigning all our public IP addresses 1. Can CloudStack do public IP migration from old to new in different zones? 2. Assuming item 1 is feasible, can cloudstack remove all old public instances, system VMs, and VRs that are currently in use 3. As some of my current VPC networks use an old public network (in source-nat mode), can we change to a new public IP to replace the old public network (in source-nat mode) 4. Any docs for items 1, 2, and 3 or workflow on how to make it work? Regards, Bryan On 20 Nov 2023 at 9:12 PM +0800, Bryan Tiang , wrote: > Hi Community, > > Our current Cloudstack is setup with old public IP addresses is assigned to > our zone infrastructure. > > Ongoing next month, we are going to change telco and require re-assigning all > our public IP addresses. > > 1. Can CloudStack do public IP migration from old to new in different zones? > > > > 2. Assuming item 1 is feasible, can cloudstack remove all old public > > instances, system VMs, and VRs that are currently in use? > > > > 3. As some of my current VPC networks use an old public network (in > > source-nat mode), can we change to a new public IP to replace the old > > public network (in source-nat mode)? > > > > 4. Any docs for items 1, 2, and 3 or workflow on how to make it work? > > Regards, > Bryan
Swapping Public IP Addresses
Hi Community, Our current Cloudstack is setup with old public IP addresses is assigned to our zone infrastructure. Ongoing next month, we are going to change telco and require re-assigning all our public IP addresses. > 1. Can CloudStack do public IP migration from old to new in different zones? > > 2. Assuming item 1 is feasible, can cloudstack remove all old public > instances, system VMs, and VRs that are currently in use? > > 3. As some of my current VPC networks use an old public network (in > source-nat mode), can we change to a new public IP to replace the old public > network (in source-nat mode)? > > 4. Any docs for items 1, 2, and 3 or workflow on how to make it work? Regards, Bryan
RE: Anyone used Cloudstack with Accelerator NIC Cards before?
Hey Alex, Amazing! Did you enable TCP Direct or ef_vi? Or did u only use Kernel Bypass (onload)? Was your setup only on the hypervisor? Or the VMs as well? We are trying to achieve a setup where tenants dont need to enable anything or know of the acceleration. Just use as is. Regards Bryan On 17 Nov 2023 at 8:26 AM +0800, Alex Mattioli , wrote: > I have, with VMWare. > The setup of the cards was done between VMWare and the NICs, CloudSstack > didn't really need to know anything about it. > > With KVM you can maybe do the same > > Cheers > Alex > > > > > -----Original Message- > From: Bryan Tiang > Sent: Thursday, November 16, 2023 3:12 PM > To: Vivek Kumar via users > Subject: Anyone used Cloudstack with Accelerator NIC Cards before? > > Hi Guys, > > Has anyone used Cloudstack with Accelerator NIC Cards before? Like Solarflare > X2522. > > They are commonly used in HFT with features such as kernel bypass onload and > vNICs to reduce latency and cpu clock cycles. > > Cant seem to find any test cases with cloudstack , but i did manage to find > some with openstack. > > Regards, > Bryan
Anyone used Cloudstack with Accelerator NIC Cards before?
Hi Guys, Has anyone used Cloudstack with Accelerator NIC Cards before? Like Solarflare X2522. They are commonly used in HFT with features such as kernel bypass onload and vNICs to reduce latency and cpu clock cycles. Cant seem to find any test cases with cloudstack , but i did manage to find some with openstack. Regards, Bryan
Re: AW: Cloudstack Linstor cant do live vm snapshots?
Hi Slavka, Thanks! Another question, I understand that Storage-based Instance Snapshots on KVM does not support memory snapshot. Are there any workaround to allow memory snapshot? Still using KVM and Linstor Regards, Bryan On 16 Nov 2023 at 3:21 PM +0800, Slavka Peleva , wrote: > Hi Bryan, > > Yes, you need qemu-guest-agent to be installed. This operation creates a > volume snapshot for each volume that is attached to the VM while the > virtual machine is frozen. > Storage-based Instance Snapshots on KVM > <https://docs.cloudstack.apache.org/en/latest/adminguide/virtual_machines.html#storage-based-instance-snapshots-on-kvm> > > Best regards, > Slavka > > On Thu, Nov 16, 2023 at 7:47 AM Bryan Tiang > wrote: > > > Hey Swen, > > > > Thanks for this. But for this to work, we need to make sure each VM > > spinner up has the QEMU agent installed, right? > > > > Regards, > > Bryan > > On 10 Nov 2023 at 9:54 PM +0800, m...@swen.io, wrote: > > > Hi Bryan, > > > > > > you can use KVM with linstor to do VM snapshots. You just need to enable > > kvm.snapshot.enabled and kvm.vmstoragesnapshot.enabled in global settings. > > No NFS storage needed at all. > > > > > > Regards, > > > Swen > > > > > > -Ursprüngliche Nachricht- > > > Von: Bryan Tiang > > > Gesendet: Freitag, 10. November 2023 14:12 > > > An: Vivek Kumar via users > > > Betreff: Cloudstack Linstor cant do live vm snapshots? > > > > > > Hi Community, > > > > > > We are using Cloudstack with Linstor. We want to achieve live snapshots > > as this is an important feature for us (can’t live without it). > > > > > > But based on the Cloudstack documentation, KVM Supports VM Snapshots > > only when using NFS Shared Storage. > > > > > > Linstor does support NFS, but there are some limitations to it such as > > not being able to resize the NFS share file (need to create a new NFS and > > do manual migration). > > > > > > How are other users overcoming this issue with Cloudstack + Linstor? Our > > setup details below: > > > > > > Compute Hypervisor: Ubuntu 20/22 LTS KVM Primary Storage: Linstor Raw > > Block Storage > > > > > > Regards, > > > Bryan > > > > > > > >
Re: AW: Cloudstack Linstor cant do live vm snapshots?
Hey Swen, Thanks for this. But for this to work, we need to make sure each VM spinner up has the QEMU agent installed, right? Regards, Bryan On 10 Nov 2023 at 9:54 PM +0800, m...@swen.io, wrote: > Hi Bryan, > > you can use KVM with linstor to do VM snapshots. You just need to enable > kvm.snapshot.enabled and kvm.vmstoragesnapshot.enabled in global settings. No > NFS storage needed at all. > > Regards, > Swen > > -Ursprüngliche Nachricht- > Von: Bryan Tiang > Gesendet: Freitag, 10. November 2023 14:12 > An: Vivek Kumar via users > Betreff: Cloudstack Linstor cant do live vm snapshots? > > Hi Community, > > We are using Cloudstack with Linstor. We want to achieve live snapshots as > this is an important feature for us (can’t live without it). > > But based on the Cloudstack documentation, KVM Supports VM Snapshots only > when using NFS Shared Storage. > > Linstor does support NFS, but there are some limitations to it such as not > being able to resize the NFS share file (need to create a new NFS and do > manual migration). > > How are other users overcoming this issue with Cloudstack + Linstor? Our > setup details below: > > Compute Hypervisor: Ubuntu 20/22 LTS KVM Primary Storage: Linstor Raw Block > Storage > > Regards, > Bryan > >
RE: Palo Alto VM Firewall with Cloudstack
Hey Alex, Thanks for the response. I’ll try to get a trial VM Firewall from Palo Alto and see if it still works with Cloudstack 4.18. Will update our findings. But in the event that this doesn’t work, we were thinking of still using it, but outside of CloudStack. Meaning we will put the Palo Alto Firewall after the router (filtering all traffic into the cloud tenants). Cloudstack will not know of its existence, and we will need to manage this firewall manually. Will this work? Regards, Bryan On 16 Nov 2023 at 1:11 PM +0800, Alex Mattioli , wrote: > I've deployed PaloAlto firewalls as VNFs in CloudStack, but didn't use the > integration, it seems to have been abandoned as it didn't work with the > version of PAN-OS I was using. That was back with ACS 4.11, didn't try with > later versions. > > > > > > > -----Original Message- > From: Bryan Tiang > Sent: Thursday, November 16, 2023 6:08 AM > To: users@cloudstack.apache.org; users@cloudstack.apache.org > Subject: Re: Palo Alto VM Firewall with Cloudstack > > Hi Tesfay, > > Thanks for the links. We are using KVM Ubuntu for our hypervisors. > > I’ll take it as Palo Alto integration with Cloudstack UI and API is still > supported. > > Anyone using this setup in production? Is it stable? Any experiences to share? > > Regards, > Bryan > On 16 Nov 2023 at 1:00 PM +0800, Tesfay Tesfamariam > , wrote: > > Check PaloAlto supported hypervisor page. > > > > https://docs.paloaltonetworks.com/compatibility-matrix/vm-series-firewalls/vms-series-hypervisor-support > > > > CloudStack documentation > > > > http://docs.cloudstack.apache.org/en/4.18.1.0/adminguide/networking/palo_alto_config.html > > > > May be post your question in the community page if none works for you . > > > > https://live.paloaltonetworks.com/ > > > > From: Bryan Tiang > > Sent: Wednesday, November 15, 2023 11:02:23 PM > > To: Vivek Kumar via users > > Subject: Palo Alto VM Firewall with Cloudstack > > > > Hi Guys, > > > > Has anyone used Cloudstack together with Palo Alto VM Firewall? > > > > I can see Palo Alto is supported in the Documentation but I know some > > documents aren’t updated. > > > > We want to be able to manage the Palo Alto VM Firewall via Cloudstack for a > > fully automated solution. > > > > Regards, > > Bryan
Re: Palo Alto VM Firewall with Cloudstack
Hi Tesfay, Thanks for the links. We are using KVM Ubuntu for our hypervisors. I’ll take it as Palo Alto integration with Cloudstack UI and API is still supported. Anyone using this setup in production? Is it stable? Any experiences to share? Regards, Bryan On 16 Nov 2023 at 1:00 PM +0800, Tesfay Tesfamariam , wrote: > Check PaloAlto supported hypervisor page. > > https://docs.paloaltonetworks.com/compatibility-matrix/vm-series-firewalls/vms-series-hypervisor-support > > CloudStack documentation > > http://docs.cloudstack.apache.org/en/4.18.1.0/adminguide/networking/palo_alto_config.html > > May be post your question in the community page if none works for you . > > https://live.paloaltonetworks.com/ > ____ > From: Bryan Tiang > Sent: Wednesday, November 15, 2023 11:02:23 PM > To: Vivek Kumar via users > Subject: Palo Alto VM Firewall with Cloudstack > > Hi Guys, > > Has anyone used Cloudstack together with Palo Alto VM Firewall? > > I can see Palo Alto is supported in the Documentation but I know some > documents aren’t updated. > > We want to be able to manage the Palo Alto VM Firewall via Cloudstack for a > fully automated solution. > > Regards, > Bryan
Palo Alto VM Firewall with Cloudstack
Hi Guys, Has anyone used Cloudstack together with Palo Alto VM Firewall? I can see Palo Alto is supported in the Documentation but I know some documents aren’t updated. We want to be able to manage the Palo Alto VM Firewall via Cloudstack for a fully automated solution. Regards, Bryan
Cloudstack Linstor cant do live vm snapshots?
Hi Community, We are using Cloudstack with Linstor. We want to achieve live snapshots as this is an important feature for us (can’t live without it). But based on the Cloudstack documentation, KVM Supports VM Snapshots only when using NFS Shared Storage. Linstor does support NFS, but there are some limitations to it such as not being able to resize the NFS share file (need to create a new NFS and do manual migration). How are other users overcoming this issue with Cloudstack + Linstor? Our setup details below: Compute Hypervisor: Ubuntu 20/22 LTS KVM Primary Storage: Linstor Raw Block Storage Regards, Bryan
Re: AW: Comparing Hyperconverged + Converged Setup with Cloudstack + Linbit
Hi Pratik, Thanks for the response. Just to confirm, your storage volumes had zero downtime, right? Regards, Bryan On 19 Oct 2023 at 3:18 PM +0800, m...@swen.io, wrote: > Hey Pratik, > > can you elaborate more on this stability problems? We are doing also a CS + > Linstor PoC at the moment and we did a lot of stress testing it without any > problems on linstor side. I am curious if we did miss some tests. > We are using a place count of 2 in a 3 node cluster. > > Regards, > Swen > > -Ursprüngliche Nachricht- > Von: Pratik Chandrakar > Gesendet: Donnerstag, 19. Oktober 2023 07:15 > An: users@cloudstack.apache.org > Betreff: Re: Comparing Hyperconverged + Converged Setup with Cloudstack + > Linbit > > Hi Bryan, > > We did a small PoC with Cloudstack + Linbit SDS(3 Time replica) in a > hyperconverged setup. There was no issue with HA, the VMs successfully > restarted from different nodes. However, we did face stability problems with > Linbit HA, which prevented us from provisioning new storage or virtual > machines. > > On Wed, Oct 18, 2023 at 3:42 PM Bryan Tiang > wrote: > > > Hi Guys, > > > > We are doing some evaluation with Cloudstack + Linbit SDS. > > > > Has anyone had any experience using these with a Converged or > > Hyperconverged setup? > > > > My understanding is that Converged is the best for HA Because: > > > > • If any storage node goes down, there is zero downtime. (3 Time > > Replica) • If any compute node goes down, it will be restarted in > > another node as part of HA feature. > > > > But what about Hyperconverged setup? Can we also set zero downtime > > with storage and fast VM recovery? > > > > Regards, > > Bryan > > > > > -- > *Regards,* > *Pratik Chandrakar* > >
Comparing Hyperconverged + Converged Setup with Cloudstack + Linbit
Hi Guys, We are doing some evaluation with Cloudstack + Linbit SDS. Has anyone had any experience using these with a Converged or Hyperconverged setup? My understanding is that Converged is the best for HA Because: • If any storage node goes down, there is zero downtime. (3 Time Replica) • If any compute node goes down, it will be restarted in another node as part of HA feature. But what about Hyperconverged setup? Can we also set zero downtime with storage and fast VM recovery? Regards, Bryan
Re: AW: Cloudstack VM HA
Hi Nix and Swen, Thanks for the input! Just curious, can VM HA and Host HA be enabled at the same time? In our case, using Cloudstack + Linstor. And to clarify my understanding. Host HA migrates VMs to another Host if Cloudstack detects the physical host to be unhealthy, right? That’s all? Regards, Bryan On 11 Oct 2023 at 7:48 PM +0800, m...@swen.io, wrote: > Hi Bryan, > > we are testing the exact same scenario at the moment! :-) > > As far as I understand CS has 2 different HA. VM HA and Host HA. When talking > about VM HA the VM needs to use an offering with ha is enabled. CS is now > checking if the VM is running and if it is not running it will restart or > recreate it. You can test this when destroying a vm via virsh destroy on KVM > directly. CS will restart this VM. > > Host HA only works, as NUX wrote, with NFS-storage at the moment. As far as I > know StorPool is developing a new framework so other storages can be used for > host ha in the future. I read something on the ccc agenda. > > Regards, > Swen > > -Ursprüngliche Nachricht- > Von: Bryan Tiang > Gesendet: Dienstag, 10. Oktober 2023 12:36 > An: users@cloudstack.apache.org > Betreff: Cloudstack VM HA > > Hi All, > > We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is Ubuntu. > > We are trying to test the VM HA by powering down a physical node at random. > However, the VMs doesn’t seem to be failing over to the other nodes. > > VM HA is enabled already, is there something we are missing? > > Regards, > Bryan > >
Cloudstack VM HA
Hi All, We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is Ubuntu. We are trying to test the VM HA by powering down a physical node at random. However, the VMs doesn’t seem to be failing over to the other nodes. VM HA is enabled already, is there something we are missing? Regards, Bryan
Questions about Provisioning and Networking.
Hi Cloudstack Community, It’s our first time setting up Cloudstack. Currently, we have 6 Physical Servers connected to the same TOR Switch. Using KVM Ubuntu. We noticed that VM instances that are not in the same compute node as the virtual router will not have network connectivity. Is there any configuration needed on the network switches? Or any possible root cause that we are not aware of? Regards, Bryan
Advice on Hardware Rack Design for Cloudstack
Hi Cloudstack Community, Thanks for the help on the previous question regarding how to deploy instances on 2 different physical servers within a cluster. I would like some advice from the community regarding our rack design, its our first time setting up a Private Cloud. We plan to start off our cloud stack journey with a single rack (42u). We have come up with a visual to help understand bit better. This is what we were thinking: * All Infrastructures are Fully Redundant (Eg. Network, Server, Storage etc) * Racks connected to two different PDU Power Sources ( Max 6KW Each) * Each Server is connected to the SAN Storages within the Rack (will not cross out of the Rack) * San Storages shall Mirror Each Other for full storage redundancy. * Additional harddrives to be added to the San Storages as when needed. * Server Capacity shall be configured in the CloudStack Platform to not provision more than 30% (within its cluster) to allow for Auto VM Restart in a new server, should any server fail. * Servers will not have any Local Drives. All Storages are in the San Storage only. * Minimum setup will include minimum 2 Servers. Additional servers (with same CPU Models) to be added as requirements grow. We haven’t decided if we are going to be standardising our servers within the rack to be (2x26pCPU) or (2*64pCPU). Here’s where we need some advice. In both scenarios of the server spec, we aren’t sure what is the recommended (or safe) San Storage spec to go for. We plan to deploy various services in the VMs, but the most intensive would be MSSQL Databases which are very IO Intensive. Our guys have recommended a 2x50TB Hybrid SSD/HDD San Storage at 150k IOPS for the entire rack. But we would like second opinions as we aren’t sure the specifications are insufficient (Afraid that San Storage would be the bottleneck one day). Ideally would be some sort of san storage spec which we can grow the IO over time as the number of servers in the rack increase (We are starting with 2, plan to increase to max 14 over time). Any advice on this area? Even comments on our rack design would be great! We really want to start this on the right footing. Regards, Bryan
Re: General Questions regarding Cloudstack
Hi Vivek, Thank for the response. Which of these would be the most recommended method? In terms of simplification and efficient maintenance? What about CI/CD? Is it possible to integrate our CICD pipeline to deploy 2 instances of the same app to 2 different physical servers? If so, I suppose if I use the Host Tag option, I will somehow need to have the pipeline contain those conditions? Regards, Bryan On 21 Nov 2022, at 6:02 PM, Nux wrote: A
General Questions regarding Cloudstack
Hi Guys, Im a Business Development Manager trying to convince our company to convert our datacenter into a Private Cloud using Cloudstack. Currently things are set up in a Baremetal sort of set up. Im not super technical, more so on a business and risk side of things. And to have this approved by our management, I will need to address some of their concerns. Concern 1: Some of our applications requires High Availability. On a software level, we have covered this. But on a hardware level, is there a way to deploy our applications in a way where each HA Service is deployed in a different physical server (or cluster). This is to avoid any one server taking down the entire service (assuming both HA Applications are deployed as separate VMs, but in the same physical server). Concern 2: Is there any benchmarks only that compares the performance of bare metal and VM? I understand there is a slight drop in performance, but how much is that? Some of the system we currently deploy on bare metal are MSSQL Databases and other services that have very high network IO. Are any benchmarks available online? And can those performance degredations be restored/matched by compensating with more CPU/RAM allocation to the VM? Appreciate the help. Regardfs, Bryan
