Re: [Ganglia-general] gmetad xml generation time
On Tue, 14 Aug 2012, Chris Burroughs wrote: Sorry, 2000 milliseconds as in 2 seconds. That's from asking for the data to receiving all of it. It does not include waiting between polls. 2 seconds seems a bit excessive. Last time I tested downloading from gmetad on a setup with ~50k metrics it took between 250-300ms to download XML. I will recheck later today. Can you tell me what the size of the downloaded XML is Bytes ie. nc localhost 8651 metrics.xml could affect gmetad's ability to serve XML. (I'm measuring time to respond, not time to update.) Does gmetad block responses while it's updating? No, gmetad does not block. Vladimir -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-general mailing list Ganglia-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general
Re: [Ganglia-general] gmetad xml generation time
On 08/15/2012 09:13 AM, Vladimir Vuksan wrote: 2 seconds seems a bit excessive. Last time I tested downloading from gmetad on a setup with ~50k metrics it took between 250-300ms to download XML. I will recheck later today. Can you tell me what the size of the downloaded XML is Bytes ie. nc localhost 8651 metrics.xml $ time nc localhost 8651 metrics.xml real0m1.921s user0m0.064s sys 0m1.519s $ ll metrics.xml -rw-rw-r-- 1 alerty alerty 24881432 Aug 15 17:42 metrics.xml -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-general mailing list Ganglia-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general
Re: [Ganglia-general] gmetad xml generation time
Rebuild with profiling? Just a thought... On Wed, Aug 15, 2012 at 05:42:52PM -0400, Chris Burroughs wrote: On 08/15/2012 09:13 AM, Vladimir Vuksan wrote: 2 seconds seems a bit excessive. Last time I tested downloading from gmetad on a setup with ~50k metrics it took between 250-300ms to download XML. I will recheck later today. Can you tell me what the size of the downloaded XML is Bytes ie. nc localhost 8651 metrics.xml $ time nc localhost 8651 metrics.xml real 0m1.921s user 0m0.064s sys0m1.519s $ ll metrics.xml -rw-rw-r-- 1 alerty alerty 24881432 Aug 15 17:42 metrics.xml -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-general mailing list Ganglia-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general -- Jesse Becker NHGRI Linux support (Digicon Contractor) -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-general mailing list Ganglia-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general
Re: [Ganglia-general] gmetad xml generation time
On 08/14/2012 02:00 PM, Douglas Wagner wrote: I don't mean to ask stupid questions, maybe I'm not reading you right... Are you saying that it's taking 2K ms (20s) between received data sets? (i.e. Every 2K ms you see XML data come into your application)? Or that it's taking 2K ms between the start of data receipt and end of data receipt? (i.e. It's a 20s wait between the receipt of the first bit of data and the last bit of data.) Sorry, 2000 milliseconds as in 2 seconds. That's from asking for the data to receiving all of it. It does not include waiting between polls. Second question, is it polling the UDP data stream from GMOND (port 8649) or the TCP data stream/xml from GMETAD (port 8651)? (I assume GMETAD since you're suggesting XML data). XML over TCP (port 8651) from gmeatd. Equivalently: $ time nc localhost 8651 /dev/null real0m2.119s user0m0.087s sys 0m1.795s Both questions asked, (and again no attempt to suggest you're stupid or anything) have you taken into account the amount of time you've set in the GMOND poling interval? Is it possible that the ganglia monitoring daemon is only polling data every 15 - 20 seconds and that the time you're seeing is mostly wait time between data polls? I'm not sure I'm following how the gmond or gmetad's polling interval could affect gmetad's ability to serve XML. (I'm measuring time to respond, not time to update.) Does gmetad block responses while it's updating? -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-general mailing list Ganglia-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general
[Ganglia-general] gmetad xml generation time
I have a process that periodically polls gmetad (builds models of some metrics, alerts if things don't look like). To reduce the number of variables I set up a dedicated gmetad on the same host as the poller and set write_rrds off. Unless I'm missing something the only thing it should be doing is polling gmond, and responding to my polls. Polling localhost currently has a mean around 2000 ms, with a stddev around 30. I've seen higher outliers, but right now I'm just trying to figure out of that's normal. 2000 ms to send a request over the loopback interface *seems* like a lot. But I don't really have anything to compare it to. Is that normal? Info: - ganglia 3.4.0 - centos5 [1] - xml size: 25 MiB - hosts: 300 - metrics: 9k - unique host-metric pairs: 80k [1] Same setup in another DC with centos6, so I don't think it's that. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-general mailing list Ganglia-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general
