Re: mesos-collectd-plugin
We would use (and probably contribute back to!) such an improved plugin as well, if you do polish it up be sure to announce to this list :) On Mar 10, 2015, at 2:05 PM, Dan Dong wrote: > Hi, Jeff, > Thanks, is your plugin working together with collectd? It would be great to > publish it! > > A general question for Mesos: Is there a method to monitor CPU/Memory/Disk > usages of jobs from different frameworks(e.g: Hadoop/Mapreduce, Spark etc)? > (Not necessarily to generate figures, text format numbers are quite enough.) > So e.g for a hadoop job, when it's finished, we can collect the general > metrics of it? Ideally although there are many jobs from different frameworks > running at the same time on mesos, > we still could get their metrics respectively. > > Cheers, > Dan > > 2015-03-10 15:46 GMT-05:00 Jeff Schroeder : > I installed it and played with it for a bit but was somewhat underwhelmed > with it. It doesn't support slaves and all of the hardcoding with duplication > isn't my favorite. I ended up writing a single plugin to support both masters > and slaves and putting it on every node in my Mesos cluster. > > Would it be worth polishing up a bit and throwing on github? > > > On Tuesday, March 10, 2015, Dan Dong wrote: > Hi, All, > Does anybody use this mesos-collectd-plugin: > https://github.com/rayrod2030/collectd-mesos > > I have installed collectd and this plugin, then configured it as instructions > and restarted the collectd daemon, why seems nothing happens on the > mesos:5050 web UI( python plugin has been turned on in collectd.conf). > > My question is: > 1. Should I install collectd and this mesos-collectd-plugin on each master > and slave nodes and restart collectd daemon? (This is what I have done.) > 2. Should the config file mesos-master.conf only configured on master node > and > mesos-slave.conf only configured on slave node?(This is what I have done.) > Or both of them should only appear on master node? > 3. Is there an example( or a figure) of what output one is expected to see by > this plugin? > > Cheers, > Dan > > > > -- > Text by Jeff, typos by iPhone >
Re: mesos-collectd-plugin
Hi, Jeff, Thanks, is your plugin working together with collectd? It would be great to publish it! A general question for Mesos: Is there a method to monitor CPU/Memory/Disk usages of jobs from different frameworks(e.g: Hadoop/Mapreduce, Spark etc)? (Not necessarily to generate figures, text format numbers are quite enough.) So e.g for a hadoop job, when it's finished, we can collect the general metrics of it? Ideally although there are many jobs from different frameworks running at the same time on mesos, we still could get their metrics respectively. Cheers, Dan 2015-03-10 15:46 GMT-05:00 Jeff Schroeder : > I installed it and played with it for a bit but was somewhat underwhelmed > with it. It doesn't support slaves and all of the hardcoding with > duplication isn't my favorite. I ended up writing a single plugin to > support both masters and slaves and putting it on every node in my Mesos > cluster. > > Would it be worth polishing up a bit and throwing on github? > > > On Tuesday, March 10, 2015, Dan Dong wrote: > >> Hi, All, >> Does anybody use this mesos-collectd-plugin: >> https://github.com/rayrod2030/collectd-mesos >> >> I have installed collectd and this plugin, then configured it as >> instructions and restarted the collectd daemon, why seems nothing happens >> on the mesos:5050 web UI( python plugin has been turned on in >> collectd.conf). >> >> My question is: >> 1. Should I install collectd and this mesos-collectd-plugin on each >> master and slave nodes and restart collectd daemon? (This is what I have >> done.) >> 2. Should the config file mesos-master.conf only configured on master >> node and >> mesos-slave.conf only configured on slave node?(This is what I have >> done.) >> Or both of them should only appear on master node? >> 3. Is there an example( or a figure) of what output one is expected to >> see by this plugin? >> >> Cheers, >> Dan >> >> > > -- > Text by Jeff, typos by iPhone >
Re: mesos-collectd-plugin
Hi, Dick, 1. Yes, /opt/collectd/etc/collectd.conf is my main and only collectd config file. 2. Have put the block into collectd.conf now. 3. I have set only 1 master, and set the Host line to ip address of master node and restarted collectd. I think I have to install graphite on my cluster too to let the following line works in collectd.conf: LoadPlugin write_graphite Cheers, Dan 2015-03-11 15:09 GMT-05:00 Dick Davies : > Hi Dan > > I can see a couple of things that could be wrong > (NB: not a collectd expert, but these are differences I see from > my working config). > > 1. Is /opt/collectd/etc/collectd.conf your main collectd config file? > > otherwise, it's not being read at all by collectd. > > 2. I configure the plugin in that file i.e. the > > > > block should be in /opt/collectd/etc/collectd.conf , not tucked down > in the python module path > directory. > > 3. Are you sure your master listens on localhost? Mine doesn't, I > needed to set that Host line > to match the IP I set that master to listen on ( e.g. in > /etc/mesos-master/ip ). > > Pretty sure one of those will do the trick > (NB: you'll only get metrics from the elected master; the 'standby' > masters still get polled > but collectd will ignore any data from them unless they're the primary) > > On 11 March 2015 at 19:52, Dan Dong wrote: > > Hi, Dick, > > I put the plugin under: > > $ ls -l /opt/collectd/lib/collectd/plugins/python/ > > total 504 > > -rw-r--r-- 1 root root345 Mar 10 19:40 mesos-master.conf > > -rw-r--r-- 1 root root 1 Mar 10 15:06 mesos-master.py > > -rw-r--r-- 1 root root322 Mar 10 19:44 mesos-slave.conf > > -rw-r--r-- 1 root root 6808 Mar 10 15:06 mesos-slave.py > > -rw-r--r-- 1 root root 288892 Mar 10 19:35 python.a > > -rwxr-xr-x 1 root root969 Mar 10 19:35 python.la > > -rwxr-xr-x 1 root root 188262 Mar 10 19:35 python.so > > > > And in /opt/collectd/etc/collectd.conf, I set: > > > > > > Globals true > > > > . > > > > > > ModulePath "/opt/collectd/lib/collectd/plugins/python/" > > LogTraces true > > > > > > $ cat /opt/collectd/lib/collectd/plugins/python/mesos-master.conf > > > > Globals true > > > > > > > > ModulePath "/opt/collectd/lib/collectd/plugins/python/" > > Import "mesos-master" > > > > Host "localhost" > > Port 5050 > > Verbose false > > Version "0.21.0" > > > > > > > > Anything wrong with the above settings? > > > > Cheers, > > Dan > > > > > > > > 2015-03-10 17:21 GMT-05:00 Dick Davies : > > > >> Hi Dan > >> > >> The .py files (the plugin) live in the collectd python path, > >> it sounds like maybe you're not loading the plugin .conf file into > >> your collectd config? > >> > >> The output will depend on what your collectd is set to write to, I use > >> it with write_graphite. > >> > >> On 10 March 2015 at 20:41, Dan Dong wrote: > >> > Hi, All, > >> > Does anybody use this mesos-collectd-plugin: > >> > https://github.com/rayrod2030/collectd-mesos > >> > > >> > I have installed collectd and this plugin, then configured it as > >> > instructions and restarted the collectd daemon, why seems nothing > >> > happens on > >> > the mesos:5050 web UI( python plugin has been turned on in > >> > collectd.conf). > >> > > >> > My question is: > >> > 1. Should I install collectd and this mesos-collectd-plugin on each > >> > master > >> > and slave nodes and restart collectd daemon? (This is what I have > done.) > >> > 2. Should the config file mesos-master.conf only configured on master > >> > node > >> > and > >> > mesos-slave.conf only configured on slave node?(This is what I > have > >> > done.) > >> > Or both of them should only appear on master node? > >> > 3. Is there an example( or a figure) of what output one is expected to > >> > see > >> > by this plugin? > >> > > >> > Cheers, > >> > Dan > >> > > > > > >
Re: mesos-collectd-plugin
Hi Dan I can see a couple of things that could be wrong (NB: not a collectd expert, but these are differences I see from my working config). 1. Is /opt/collectd/etc/collectd.conf your main collectd config file? otherwise, it's not being read at all by collectd. 2. I configure the plugin in that file i.e. the block should be in /opt/collectd/etc/collectd.conf , not tucked down in the python module path directory. 3. Are you sure your master listens on localhost? Mine doesn't, I needed to set that Host line to match the IP I set that master to listen on ( e.g. in /etc/mesos-master/ip ). Pretty sure one of those will do the trick (NB: you'll only get metrics from the elected master; the 'standby' masters still get polled but collectd will ignore any data from them unless they're the primary) On 11 March 2015 at 19:52, Dan Dong wrote: > Hi, Dick, > I put the plugin under: > $ ls -l /opt/collectd/lib/collectd/plugins/python/ > total 504 > -rw-r--r-- 1 root root345 Mar 10 19:40 mesos-master.conf > -rw-r--r-- 1 root root 1 Mar 10 15:06 mesos-master.py > -rw-r--r-- 1 root root322 Mar 10 19:44 mesos-slave.conf > -rw-r--r-- 1 root root 6808 Mar 10 15:06 mesos-slave.py > -rw-r--r-- 1 root root 288892 Mar 10 19:35 python.a > -rwxr-xr-x 1 root root969 Mar 10 19:35 python.la > -rwxr-xr-x 1 root root 188262 Mar 10 19:35 python.so > > And in /opt/collectd/etc/collectd.conf, I set: > > > Globals true > > . > > > ModulePath "/opt/collectd/lib/collectd/plugins/python/" > LogTraces true > > > $ cat /opt/collectd/lib/collectd/plugins/python/mesos-master.conf > > Globals true > > > > ModulePath "/opt/collectd/lib/collectd/plugins/python/" > Import "mesos-master" > > Host "localhost" > Port 5050 > Verbose false > Version "0.21.0" > > > > Anything wrong with the above settings? > > Cheers, > Dan > > > > 2015-03-10 17:21 GMT-05:00 Dick Davies : > >> Hi Dan >> >> The .py files (the plugin) live in the collectd python path, >> it sounds like maybe you're not loading the plugin .conf file into >> your collectd config? >> >> The output will depend on what your collectd is set to write to, I use >> it with write_graphite. >> >> On 10 March 2015 at 20:41, Dan Dong wrote: >> > Hi, All, >> > Does anybody use this mesos-collectd-plugin: >> > https://github.com/rayrod2030/collectd-mesos >> > >> > I have installed collectd and this plugin, then configured it as >> > instructions and restarted the collectd daemon, why seems nothing >> > happens on >> > the mesos:5050 web UI( python plugin has been turned on in >> > collectd.conf). >> > >> > My question is: >> > 1. Should I install collectd and this mesos-collectd-plugin on each >> > master >> > and slave nodes and restart collectd daemon? (This is what I have done.) >> > 2. Should the config file mesos-master.conf only configured on master >> > node >> > and >> > mesos-slave.conf only configured on slave node?(This is what I have >> > done.) >> > Or both of them should only appear on master node? >> > 3. Is there an example( or a figure) of what output one is expected to >> > see >> > by this plugin? >> > >> > Cheers, >> > Dan >> > > >
Re: mesos-collectd-plugin
Hi, Dick, I put the plugin under: $ ls -l /opt/collectd/lib/collectd/plugins/python/ total 504 -rw-r--r-- 1 root root345 Mar 10 19:40 mesos-master.conf -rw-r--r-- 1 root root 1 Mar 10 15:06 mesos-master.py -rw-r--r-- 1 root root322 Mar 10 19:44 mesos-slave.conf -rw-r--r-- 1 root root 6808 Mar 10 15:06 mesos-slave.py -rw-r--r-- 1 root root 288892 Mar 10 19:35 python.a -rwxr-xr-x 1 root root969 Mar 10 19:35 python.la -rwxr-xr-x 1 root root 188262 Mar 10 19:35 python.so And in /opt/collectd/etc/collectd.conf, I set: Globals true . ModulePath "/opt/collectd/lib/collectd/plugins/python/" LogTraces true $ cat /opt/collectd/lib/collectd/plugins/python/mesos-master.conf Globals true ModulePath "/opt/collectd/lib/collectd/plugins/python/" Import "mesos-master" Host "localhost" Port 5050 Verbose false Version "0.21.0" Anything wrong with the above settings? Cheers, Dan 2015-03-10 17:21 GMT-05:00 Dick Davies : > Hi Dan > > The .py files (the plugin) live in the collectd python path, > it sounds like maybe you're not loading the plugin .conf file into > your collectd config? > > The output will depend on what your collectd is set to write to, I use > it with write_graphite. > > On 10 March 2015 at 20:41, Dan Dong wrote: > > Hi, All, > > Does anybody use this mesos-collectd-plugin: > > https://github.com/rayrod2030/collectd-mesos > > > > I have installed collectd and this plugin, then configured it as > > instructions and restarted the collectd daemon, why seems nothing > happens on > > the mesos:5050 web UI( python plugin has been turned on in > collectd.conf). > > > > My question is: > > 1. Should I install collectd and this mesos-collectd-plugin on each > master > > and slave nodes and restart collectd daemon? (This is what I have done.) > > 2. Should the config file mesos-master.conf only configured on master > node > > and > > mesos-slave.conf only configured on slave node?(This is what I have > > done.) > > Or both of them should only appear on master node? > > 3. Is there an example( or a figure) of what output one is expected to > see > > by this plugin? > > > > Cheers, > > Dan > > >
Re: mesos-collectd-plugin
Hi Dan The .py files (the plugin) live in the collectd python path, it sounds like maybe you're not loading the plugin .conf file into your collectd config? The output will depend on what your collectd is set to write to, I use it with write_graphite. On 10 March 2015 at 20:41, Dan Dong wrote: > Hi, All, > Does anybody use this mesos-collectd-plugin: > https://github.com/rayrod2030/collectd-mesos > > I have installed collectd and this plugin, then configured it as > instructions and restarted the collectd daemon, why seems nothing happens on > the mesos:5050 web UI( python plugin has been turned on in collectd.conf). > > My question is: > 1. Should I install collectd and this mesos-collectd-plugin on each master > and slave nodes and restart collectd daemon? (This is what I have done.) > 2. Should the config file mesos-master.conf only configured on master node > and > mesos-slave.conf only configured on slave node?(This is what I have > done.) > Or both of them should only appear on master node? > 3. Is there an example( or a figure) of what output one is expected to see > by this plugin? > > Cheers, > Dan >
Re: mesos-collectd-plugin
Can you just use the Mesos cluster name? We'd end up preferring that to a configuration tweak anyway. On Mar 10, 2015, at 2:40 PM, Jeff Schroeder wrote: > Alright. You've convinced me I should put this online, but I'll have to clean > it up first. I would have originally but had one weird question. > > In my company, we almost certainly will have multiple potentially large Mesos > clusters as different failure domains. The issue I've got is relatively > simple, the solution is not. The collectd daemon runs on every single Mesos > node (master or slave) in the firm. If it is running on a master, it will get > data from localhost:$master_port/metrics/snapshot. It then checked if > json_data["master/elected"] == 1 and ONLY reports the data if it is the > elected master. By default it uses a namespace like: > > mesos.master.* > > If you have multiple clusters this obviously won't work, so I added an > optional cluster name attribute to the collectd config. That way it will > report to: > > mesos.archive.* where your mesos cluster name is "archive". > > How would you propose a decent way to namespace master stats that are cluster > wide. I had to lie (in the collectd plugin code) and change the host as a > metric such as: > > collectd.$HOSTNAME.master.* seems to make absolutely no sense when there is a > single elected master per cluster. > > > On Tuesday, March 10, 2015, Steven Schlansker > wrote: > We would use (and probably contribute back to!) such an improved plugin as > well, if you do polish it up be sure to announce to this list :) > > On Mar 10, 2015, at 2:05 PM, Dan Dong wrote: > > > Hi, Jeff, > > Thanks, is your plugin working together with collectd? It would be great > > to publish it! > > > > A general question for Mesos: Is there a method to monitor CPU/Memory/Disk > > usages of jobs from different frameworks(e.g: Hadoop/Mapreduce, Spark etc)? > > (Not necessarily to generate figures, text format numbers are quite enough.) > > So e.g for a hadoop job, when it's finished, we can collect the general > > metrics of it? Ideally although there are many jobs from different > > frameworks running at the same time on mesos, > > we still could get their metrics respectively. > > > > Cheers, > > Dan > > > > 2015-03-10 15:46 GMT-05:00 Jeff Schroeder : > > I installed it and played with it for a bit but was somewhat underwhelmed > > with it. It doesn't support slaves and all of the hardcoding with > > duplication isn't my favorite. I ended up writing a single plugin to > > support both masters and slaves and putting it on every node in my Mesos > > cluster. > > > > Would it be worth polishing up a bit and throwing on github? > > > > > > On Tuesday, March 10, 2015, Dan Dong wrote: > > Hi, All, > > Does anybody use this mesos-collectd-plugin: > > https://github.com/rayrod2030/collectd-mesos > > > > I have installed collectd and this plugin, then configured it as > > instructions and restarted the collectd daemon, why seems nothing happens > > on the mesos:5050 web UI( python plugin has been turned on in > > collectd.conf). > > > > My question is: > > 1. Should I install collectd and this mesos-collectd-plugin on each master > > and slave nodes and restart collectd daemon? (This is what I have done.) > > 2. Should the config file mesos-master.conf only configured on master node > > and > > mesos-slave.conf only configured on slave node?(This is what I have > > done.) > > Or both of them should only appear on master node? > > 3. Is there an example( or a figure) of what output one is expected to see > > by this plugin? > > > > Cheers, > > Dan > > > > > > > > -- > > Text by Jeff, typos by iPhone > > > > > > -- > Text by Jeff, typos by iPhone
Re: mesos-collectd-plugin
Alright. You've convinced me I should put this online, but I'll have to clean it up first. I would have originally but had one weird question. In my company, we almost certainly will have multiple potentially large Mesos clusters as different failure domains. The issue I've got is relatively simple, the solution is not. The collectd daemon runs on every single Mesos node (master or slave) in the firm. If it is running on a master, it will get data from localhost:$master_port/metrics/snapshot. It then checked if json_data["master/elected"] == 1 and ONLY reports the data if it is the elected master. By default it uses a namespace like: mesos.master.* If you have multiple clusters this obviously won't work, so I added an optional cluster name attribute to the collectd config. That way it will report to: mesos.archive.* where your mesos cluster name is "archive". How would you propose a decent way to namespace master stats that are cluster wide. I had to lie (in the collectd plugin code) and change the host as a metric such as: collectd.$HOSTNAME.master.* seems to make absolutely no sense when there is a single elected master per cluster. On Tuesday, March 10, 2015, Steven Schlansker wrote: > We would use (and probably contribute back to!) such an improved plugin as > well, if you do polish it up be sure to announce to this list :) > > On Mar 10, 2015, at 2:05 PM, Dan Dong > > wrote: > > > Hi, Jeff, > > Thanks, is your plugin working together with collectd? It would be > great to publish it! > > > > A general question for Mesos: Is there a method to monitor > CPU/Memory/Disk usages of jobs from different frameworks(e.g: > Hadoop/Mapreduce, Spark etc)? (Not necessarily to generate figures, text > format numbers are quite enough.) > > So e.g for a hadoop job, when it's finished, we can collect the general > metrics of it? Ideally although there are many jobs from different > frameworks running at the same time on mesos, > > we still could get their metrics respectively. > > > > Cheers, > > Dan > > > > 2015-03-10 15:46 GMT-05:00 Jeff Schroeder >: > > I installed it and played with it for a bit but was somewhat > underwhelmed with it. It doesn't support slaves and all of the hardcoding > with duplication isn't my favorite. I ended up writing a single plugin to > support both masters and slaves and putting it on every node in my Mesos > cluster. > > > > Would it be worth polishing up a bit and throwing on github? > > > > > > On Tuesday, March 10, 2015, Dan Dong > > wrote: > > Hi, All, > > Does anybody use this mesos-collectd-plugin: > > https://github.com/rayrod2030/collectd-mesos > > > > I have installed collectd and this plugin, then configured it as > instructions and restarted the collectd daemon, why seems nothing happens > on the mesos:5050 web UI( python plugin has been turned on in > collectd.conf). > > > > My question is: > > 1. Should I install collectd and this mesos-collectd-plugin on each > master and slave nodes and restart collectd daemon? (This is what I have > done.) > > 2. Should the config file mesos-master.conf only configured on master > node and > > mesos-slave.conf only configured on slave node?(This is what I have > done.) > > Or both of them should only appear on master node? > > 3. Is there an example( or a figure) of what output one is expected to > see by this plugin? > > > > Cheers, > > Dan > > > > > > > > -- > > Text by Jeff, typos by iPhone > > > > -- Text by Jeff, typos by iPhone
Re: mesos-collectd-plugin
I didn’t use it, but from the README.md, it is a collectd plugin, not mesos plugin. It pull mesos status into collectd world, not other way around. You won’t see anything on mesos UI, but you will have mesos metrics in collectd, which then you can leverage. On Mar 10, 2015, at 4:41 PM, Dan Dong mailto:dongda...@gmail.com>> wrote: Hi, All, Does anybody use this mesos-collectd-plugin: https://github.com/rayrod2030/collectd-mesos I have installed collectd and this plugin, then configured it as instructions and restarted the collectd daemon, why seems nothing happens on the mesos:5050 web UI( python plugin has been turned on in collectd.conf). My question is: 1. Should I install collectd and this mesos-collectd-plugin on each master and slave nodes and restart collectd daemon? (This is what I have done.) 2. Should the config file mesos-master.conf only configured on master node and mesos-slave.conf only configured on slave node?(This is what I have done.) Or both of them should only appear on master node? 3. Is there an example( or a figure) of what output one is expected to see by this plugin? Cheers, Dan
Re: mesos-collectd-plugin
I installed it and played with it for a bit but was somewhat underwhelmed with it. It doesn't support slaves and all of the hardcoding with duplication isn't my favorite. I ended up writing a single plugin to support both masters and slaves and putting it on every node in my Mesos cluster. Would it be worth polishing up a bit and throwing on github? On Tuesday, March 10, 2015, Dan Dong wrote: > Hi, All, > Does anybody use this mesos-collectd-plugin: > https://github.com/rayrod2030/collectd-mesos > > I have installed collectd and this plugin, then configured it as > instructions and restarted the collectd daemon, why seems nothing happens > on the mesos:5050 web UI( python plugin has been turned on in > collectd.conf). > > My question is: > 1. Should I install collectd and this mesos-collectd-plugin on each master > and slave nodes and restart collectd daemon? (This is what I have done.) > 2. Should the config file mesos-master.conf only configured on master node > and > mesos-slave.conf only configured on slave node?(This is what I have > done.) > Or both of them should only appear on master node? > 3. Is there an example( or a figure) of what output one is expected to see > by this plugin? > > Cheers, > Dan > > -- Text by Jeff, typos by iPhone
mesos-collectd-plugin
Hi, All, Does anybody use this mesos-collectd-plugin: https://github.com/rayrod2030/collectd-mesos I have installed collectd and this plugin, then configured it as instructions and restarted the collectd daemon, why seems nothing happens on the mesos:5050 web UI( python plugin has been turned on in collectd.conf). My question is: 1. Should I install collectd and this mesos-collectd-plugin on each master and slave nodes and restart collectd daemon? (This is what I have done.) 2. Should the config file mesos-master.conf only configured on master node and mesos-slave.conf only configured on slave node?(This is what I have done.) Or both of them should only appear on master node? 3. Is there an example( or a figure) of what output one is expected to see by this plugin? Cheers, Dan
