Re: Network metrics
> What we'd like to know is realtime stats about the number of NAT/TCP > sessions (i.e. conntrack status), CPU load, memory/swap usage, > network > and I/O bandwidth, and possibly some process metrics. In addition to that, there's more checks that would make sense (and are difficult or impossible to implement on the HV): - filesystem mounted readonly - disk usage The readonly remount can happen if there is an issue with primary storage and will lead to an inaccessible router, no VMs on the attached networks being deployable and no obvious indication what's going on. We recently ran into this issue. Similarly for a full disk. As I mentioned already, we deployed additional monitoring services on each VR, so we could easily integrate them with our existing monitoring system. I would much prefer a fully integrated monitoring solution instead that can be queried through the CloudStack API. This would reduce dependency on the environment and allow router/network health monitoring (and possibly automatic redeploy) at the discretion of CloudStack. As far as I know, there isn't even a simple health indicator that can be queried through the API. Should I rather submit a feature request for this?
Re: Network metrics
> This include stats from virtual router as well since libvirt exporter > use virsh libraries to fetch details. For libvirt, virual router is > just another VM and so it will exporter those stats also. > > Run virsh list and you will see virual router entry and using > dommemstats or other commands you can see other detaila Problem #1: We're using ESX, not KVM. Problem #2: The network is fully virtualised (i.e. with hardware emulation) and it's not possible to gather anything beyond what happens on layer 1 or 2. What we'd like to know is realtime stats about the number of NAT/TCP sessions (i.e. conntrack status), CPU load, memory/swap usage, network and I/O bandwidth, and possibly some process metrics. CPU, memory and I/O metrics are exposed by the CloudStack API for instances, but not for networks or virtual routers. Monitoring via ESX is possible, but tedious and you don't get anything from inside the VM either. To achieve this goal, we used to install a bunch of monitoring daemons inside each VR and poll these from a monitoring node with direct access to the routers. If we could replace this somewhat hackish setup with something offered by CloudStack, that would be very helpful.
Re: Network metrics
This include stats from virtual router as well since libvirt exporter use virsh libraries to fetch details. For libvirt, virual router is just another VM and so it will exporter those stats also. Run virsh list and you will see virual router entry and using dommemstats or other commands you can see other detaila Sent from my iPhone On 05-Apr-2019, at 5:09 PM, Riepl, Gregor (SWISS TXT) wrote: >> You can use "libvirt exporter" to export all stats like cpu utilization, >> memory consumption, disk read/write, network read/write and export it to >> prometheus. From prometheus you can use grafana or other monitoring tools >> to see all stats. We use exporter merger third party tool to merge all the >> data from all the hypervisors and show it on dashboard. This way if any of >> your vm is having issues, you can quickly look at graphs and troubleshoot >> it. > > Good point, but this won't cover metrics that are integral to the virtual > router. > The hypervisor doesn't know about what a VM's network stack does.
Re: Network metrics
What exactly did you have in mind? I mean other than bytes in and out. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro - Original Message - > From: "Riepl, Gregor (SWISS TXT)" > To: "users" > Sent: Friday, 5 April, 2019 16:09:18 > Subject: Re: Network metrics >> You can use "libvirt exporter" to export all stats like cpu utilization, >> memory consumption, disk read/write, network read/write and export it to >> prometheus. From prometheus you can use grafana or other monitoring tools >> to see all stats. We use exporter merger third party tool to merge all the >> data from all the hypervisors and show it on dashboard. This way if any of >> your vm is having issues, you can quickly look at graphs and troubleshoot >> it. > > Good point, but this won't cover metrics that are integral to the virtual > router. > The hypervisor doesn't know about what a VM's network stack does.
Re: Network metrics
> You can use "libvirt exporter" to export all stats like cpu utilization, > memory consumption, disk read/write, network read/write and export it to > prometheus. From prometheus you can use grafana or other monitoring tools > to see all stats. We use exporter merger third party tool to merge all the > data from all the hypervisors and show it on dashboard. This way if any of > your vm is having issues, you can quickly look at graphs and troubleshoot > it. Good point, but this won't cover metrics that are integral to the virtual router. The hypervisor doesn't know about what a VM's network stack does.
Re: Network metrics
Hello Gregor You can use "libvirt exporter" to export all stats like cpu utilization, memory consumption, disk read/write, network read/write and export it to prometheus. From prometheus you can use grafana or other monitoring tools to see all stats. We use exporter merger third party tool to merge all the data from all the hypervisors and show it on dashboard. This way if any of your vm is having issues, you can quickly look at graphs and troubleshoot it. On Fri, Apr 5, 2019 at 10:01 AM Riepl, Gregor (SWISS TXT) < gregor.ri...@swisstxt.ch> wrote: > Hi, > > I recently discovered that CloudStack exposes certain VM metrics like disk > I/O, network bandwidth, CPU load and (if the HV agent is installed) memory > usage. > Presumably, these are fetched directly from the hypervisor. > > However: I found no way to get the same statistics for system VMs, in > particular virtual routers. > Does the CloudStack API offer a way to obtain per-network metrics? > > Regards, > Gregor -- Thanks and regards Rakesh venkatesh
Re: Network metrics
> How I got stats for all VMs, including system ones, was to install > Prometheus, both the cloudstack and the libvirt exporter. > I know it's "outside" Cloudstack, but may prove useful. It looks like the Prometheus exporter is limited to per-zone statistics, though? https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Metrics+Exporter+for+Prometheus But yes, we're probably better off with collecting the metrics directly from the hypervisor. Previously, we installed a bunch of monitoring services inside the VRs, but I'd like to build a cleaner solution that doesn't require messing with the system VMs, if possible. Since I'm also interested in other metrics besides the very basic ones (such as NAT table size, per-network stats in a VPC, etc.), I think it would be best if the VR template offered integrated monitoring capabilities. Should I open a feature request for that?
Re: Network metrics
Hi, How I got stats for all VMs, including system ones, was to install Prometheus, both the cloudstack and the libvirt exporter. I know it's "outside" Cloudstack, but may prove useful. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro - Original Message - > From: "Riepl, Gregor (SWISS TXT)" > To: "users" > Sent: Friday, 5 April, 2019 09:01:22 > Subject: Network metrics > Hi, > > I recently discovered that CloudStack exposes certain VM metrics like disk > I/O, > network bandwidth, CPU load and (if the HV agent is installed) memory usage. > Presumably, these are fetched directly from the hypervisor. > > However: I found no way to get the same statistics for system VMs, in > particular > virtual routers. > Does the CloudStack API offer a way to obtain per-network metrics? > > Regards, > Gregor
Re: Network metrics
Hi Boris, > I don’t think you could directly query for a VR, but you could specify a > network ID and it’ll list you all the metrics of VMs on that network. Then I > suppose you could work out a summary from that list. > https://cloudstack.apache.org/api/apidocs-4.12/apis/listVirtualMachinesMetrics.html > > by > networkid list by network id Thanks, but that isn't quite what I'm looking for. I'm particularly interested in internet traffic that goes through the router. If I just add up the network bandwidth of the individual machines, that won't tell me anything about the incoming/outgoing traffic. There should be a way to do this, as cloud-usage also records total network usage on the virtual router. Regards, Gregor
Re: Network metrics
Hi Gregor, I don’t think you could directly query for a VR, but you could specify a network ID and it’ll list you all the metrics of VMs on that network. Then I suppose you could work out a summary from that list. https://cloudstack.apache.org/api/apidocs-4.12/apis/listVirtualMachinesMetrics.html by networkid list by network id Regards, Bobby. boris.stoya...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue On 5 Apr 2019, at 11:01, Riepl, Gregor (SWISS TXT) mailto:gregor.ri...@swisstxt.ch>> wrote: Hi, I recently discovered that CloudStack exposes certain VM metrics like disk I/O, network bandwidth, CPU load and (if the HV agent is installed) memory usage. Presumably, these are fetched directly from the hypervisor. However: I found no way to get the same statistics for system VMs, in particular virtual routers. Does the CloudStack API offer a way to obtain per-network metrics? Regards, Gregor
