Re: [lustre-discuss] Command line tool to monitor Lustre I/O ?
On Feb 14, 2019, at 08:09, Laifer, Roland (SCC) wrote: > > Hello, > > since I usually just want to do a short check what is going on on > clients and servers the hint for lltop seemed to be the best for my needs. > > I tried it but there was an error message on the MDS. However, I found > an easy fix and just had to replace > "/proc/fs/lustre/mds", > with > "/proc/fs/lustre/mdt", > in lltop-master/serv.c > > It seems like many tools need to be adapted for the structures of new > Lustre versions. > > lltop is a nice tool to show which clients are currently doing most > reads, writes or general requests. Feel free to make a fork of the lltop repo and push a patch? I don't think that John is maintaining it anymore. You could change the link on the wiki page to point at your fork. > Am 21.12.18 um 11:43 schrieb Martin Hecht: >> Hello Roland, >> >> there is a nice collection of lustre monitoring tools on the lustre wiki: >> >> http://wiki.lustre.org/Lustre_Monitoring_and_Statistics_Guide >> >> which also contains a couple of references. One of them is lltop, which >> has already been mentioned a couple of times and that's what came to my >> mind as well when I read your question. >> >> best regards, >> Martin > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Command line tool to monitor Lustre I/O ?
Hello, since I usually just want to do a short check what is going on on clients and servers the hint for lltop seemed to be the best for my needs. I tried it but there was an error message on the MDS. However, I found an easy fix and just had to replace "/proc/fs/lustre/mds", with "/proc/fs/lustre/mdt", in lltop-master/serv.c It seems like many tools need to be adapted for the structures of new Lustre versions. lltop is a nice tool to show which clients are currently doing most reads, writes or general requests. Regards, Roland Am 21.12.18 um 11:43 schrieb Martin Hecht: > Hello Roland, > > there is a nice collection of lustre monitoring tools on the lustre wiki: > > http://wiki.lustre.org/Lustre_Monitoring_and_Statistics_Guide > > which also contains a couple of references. One of them is lltop, which > has already been mentioned a couple of times and that's what came to my > mind as well when I read your question. > > best regards, > Martin ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Command line tool to monitor Lustre I/O ?
I use the exact same setup along with Kapacitor for alerting (alerta as the
dash). We have created have dozens of panels in Grafana that are very
useful for troubleshooting bottlenecks with the OSS nodes, disks, as well
as the clients.
Can't go wrong with it I feel, easy to setup and fun to make graphs :-)
On Thu, Dec 20, 2018 at 12:15 PM Alexander I Kulyavtsev
wrote:
> 1) cerebro + ltop still work.
>
>
> 2) telegraf + inflixdb (collector, time series DB ). Telegraf has
> input plugins for lustre ("lustre2"), zfs, and many others. Grafana to
> plot live data from DB. Also, influxDB integrates with Prometheus.
>
> Basically, each component can feed data to different output types through
> plugins; or take data from multiple type of sources so you can use
> different combination for your monitoring stack.
>
>
> For the simplest tool you may take a look if telegraf from influxdb stack
> has proper output plugin (see influxdata on github).
>
>
> Alex.
> --
> *From:* lustre-discuss on
> behalf of Laifer, Roland (SCC)
> *Sent:* Thursday, December 20, 2018 8:04:55 AM
> *To:* lustre-discuss@lists.lustre.org
> *Subject:* [lustre-discuss] Command line tool to monitor Lustre I/O ?
>
> Dear Lustre administrators,
>
> what is a good command line tool to monitor current Lustre metadata and
> throughput operations on the local client or server? Up to now we had
> used collectl but this no longer works for Lustre 2.10.
>
> Some background about collectl: The Lustre support of collectl was
> removed many years ago but up to Lustre 2.7 it was still possible to
> monitor metadata and throughput operations on clients. In addition,
> there were plugins which also worked for the server side, see
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.lustre.org_Collectl=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=23V5nhLj03jeTboyg6QveA=RpMjhssRJoiP3ANRP6Ze3_nBrliMMPOgQaewqEwRTn4=QmdmoNcRR5A0sOgiJimMo0KtZnc-ne44A4YY8aSWbuI=
> However, it seems that there was no update for these plugins to adapt
> them for Lustre 2.10.
>
> Regards,
> Roland
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=23V5nhLj03jeTboyg6QveA=RpMjhssRJoiP3ANRP6Ze3_nBrliMMPOgQaewqEwRTn4=SXbueuHkxyBAq95D_-bLmBayRVDMtR-l7t0XZfNXEXk=
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Command line tool to monitor Lustre I/O ?
On Fri., 21 Dec. 2018, 01:05 Laifer, Roland (SCC) Dear Lustre administrators, > > what is a good command line tool to monitor current Lustre metadata and > throughput operations on the local client or server? > I wrote a small python script to parse lctl get_param and inject it straight into our influxdb server - As I was dropping this onto a sonnexion (as well as our newer systems which had collectd installed) I didn't want to require any software not already installed on the system. My plan [one of these days in my spare time] is to wrap it properly as a collectd python plugin - would people be interested and I'll probably see if I can find some time to work on it over xmas? Once it's in influx, we can then just plot it with our normal tooling (grafana) - some pictures in the pptx at https://www.dropbox.com/s/rck1lm73wlwlg6v/monitoring.pptx?dl=0 (near end) Andrew > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Command line tool to monitor Lustre I/O ?
Hello Roland, there is a nice collection of lustre monitoring tools on the lustre wiki: http://wiki.lustre.org/Lustre_Monitoring_and_Statistics_Guide which also contains a couple of references. One of them is lltop, which has already been mentioned a couple of times and that's what came to my mind as well when I read your question. best regards, Martin ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Command line tool to monitor Lustre I/O ?
I use ltop heavily:
https://github.com/LLNL/lmt
On 12/20/18 9:15 AM, Alexander I Kulyavtsev wrote:
1) cerebro + ltop still work.
2) telegraf + inflixdb (collector, time series DB ). Telegraf has input plugins
for lustre ("lustre2"), zfs, and many others. Grafana to plot live data from
DB. Also, influxDB integrates with Prometheus.
Basically, each component can feed data to different output types through
plugins; or take data from multiple type of sources so you can use different
combination for your monitoring stack.
For the simplest tool you may take a look if telegraf from influxdb stack has
proper output plugin (see influxdata on github).
Alex.
From: lustre-discuss
<mailto:lustre-discuss-boun...@lists.lustre.org>
on behalf of Laifer, Roland (SCC)
<mailto:roland.lai...@kit.edu>
Sent: Thursday, December 20, 2018 8:04:55 AM
To: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
Subject: [lustre-discuss] Command line tool to monitor Lustre I/O ?
Dear Lustre administrators,
what is a good command line tool to monitor current Lustre metadata and
throughput operations on the local client or server? Up to now we had
used collectl but this no longer works for Lustre 2.10.
Some background about collectl: The Lustre support of collectl was
removed many years ago but up to Lustre 2.7 it was still possible to
monitor metadata and throughput operations on clients. In addition,
there were plugins which also worked for the server side, see
https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.lustre.org_Collectl=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=23V5nhLj03jeTboyg6QveA=RpMjhssRJoiP3ANRP6Ze3_nBrliMMPOgQaewqEwRTn4=QmdmoNcRR5A0sOgiJimMo0KtZnc-ne44A4YY8aSWbuI=
However, it seems that there was no update for these plugins to adapt
them for Lustre 2.10.
Regards,
Roland
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=23V5nhLj03jeTboyg6QveA=RpMjhssRJoiP3ANRP6Ze3_nBrliMMPOgQaewqEwRTn4=SXbueuHkxyBAq95D_-bLmBayRVDMtR-l7t0XZfNXEXk=
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Command line tool to monitor Lustre I/O ?
Hi Roland! > Am 20.12.2018 um 15:04 schrieb Laifer, Roland (SCC) : > > what is a good command line tool to monitor current Lustre metadata and > throughput operations on the local client or server? Up to now we had > used collectl but this no longer works for Lustre 2.10. The Lustre exporter (https://github.com/HewlettPackard/lustre_exporter) for Prometheus copes well with 2.10. Calling it a command-line tool is a bit of a stretch (hey, there’s curl after all!), but it can certainly step in for collect’s non-interactive mode of operation. Kind regards, Daniel -- Daniel Kobras Principal Architect Puzzle ITC Deutschland https://www.puzzle-itc.de -- Puzzle ITC Deutschland GmbH Sitz der Gesellschaft: Jurastr. 27/1, 72072 Tübingen Eingetragen am Amtsgericht Stuttgart HRB 765802 Geschäftsführer: Lukas Kallies ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Command line tool to monitor Lustre I/O ?
1) cerebro + ltop still work.
2) telegraf + inflixdb (collector, time series DB ). Telegraf has input plugins
for lustre ("lustre2"), zfs, and many others. Grafana to plot live data from
DB. Also, influxDB integrates with Prometheus.
Basically, each component can feed data to different output types through
plugins; or take data from multiple type of sources so you can use different
combination for your monitoring stack.
For the simplest tool you may take a look if telegraf from influxdb stack has
proper output plugin (see influxdata on github).
Alex.
From: lustre-discuss on behalf of
Laifer, Roland (SCC)
Sent: Thursday, December 20, 2018 8:04:55 AM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] Command line tool to monitor Lustre I/O ?
Dear Lustre administrators,
what is a good command line tool to monitor current Lustre metadata and
throughput operations on the local client or server? Up to now we had
used collectl but this no longer works for Lustre 2.10.
Some background about collectl: The Lustre support of collectl was
removed many years ago but up to Lustre 2.7 it was still possible to
monitor metadata and throughput operations on clients. In addition,
there were plugins which also worked for the server side, see
https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.lustre.org_Collectl=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=23V5nhLj03jeTboyg6QveA=RpMjhssRJoiP3ANRP6Ze3_nBrliMMPOgQaewqEwRTn4=QmdmoNcRR5A0sOgiJimMo0KtZnc-ne44A4YY8aSWbuI=
However, it seems that there was no update for these plugins to adapt
them for Lustre 2.10.
Regards,
Roland
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=23V5nhLj03jeTboyg6QveA=RpMjhssRJoiP3ANRP6Ze3_nBrliMMPOgQaewqEwRTn4=SXbueuHkxyBAq95D_-bLmBayRVDMtR-l7t0XZfNXEXk=
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
