Re: [LARTC] Qdisc statistics project
* Stephen Hemminger <[EMAIL PROTECTED]> 2004-10-14 13:17 > >netlink errors > > Effort has gone into improving error codes/messages returned > > from netlink users in the kernel. The idea is currently on hold > > as all acceptable solutions would break APIs. Planned for 2.7 > > as of now. > > > > > You probably need to come with another alternative since 2.7 probably > won't start for a long time Agreed, Jamal and myself couldn't find this alternative yet though, it was therefore postponed to after the generic statistic conversion. Ideas are very much appreciated. ___ LARTC mailing list / [EMAIL PROTECTED] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Re: [LARTC] Qdisc statistics project
Thomas Graf wrote: netlink errors Effort has gone into improving error codes/messages returned from netlink users in the kernel. The idea is currently on hold as all acceptable solutions would break APIs. Planned for 2.7 as of now. You probably need to come with another alternative since 2.7 probably won't start for a long time ___ LARTC mailing list / [EMAIL PROTECTED] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Re: [LARTC] Qdisc statistics project
> The development of these scripts is stopped. I'm rebuilding my house, making > a website, I'm a father for 6 months now, .. but if an interesting > project is started, I want to help. In order to not run into duplicated projects: libnl General purpose netlink library for all netlink users with different API levels from raw netlink messaging to abstract processing of already existing netlink users. It supports a generic caching interface with filtering on all levels. All modules may be compiled in or loaded at runtime. Status netlink messaging needs some work to handle special cases link finished and tested neighbour finished and tested qdisc/class nearly finished cbq nearly finished fifofinished and tested priofinished and tested sfq finished and tested ... filters nearly finished u32 finished and tested ... address TLV parsers done route TLV parsers done The reason for not working on LQL are licensing issues not acceptable to me. I will release this library in 2-3 weeks and accept patches from that point. bmon bmon is a portable bandwidth monitor running on various operating systems. It supports various input methods, one of those being the above libnl resulting in support for tc statistics. Variout output modes exist including an interactive curses interface: http://people.suug.ch/~tgr/bmon_ingress.jpg but also lightweight HTML output: http://people.suug.ch/~tgr/stats/axs.4.s.html Statistics may be distributed over a network using multicast or unicast and collected at some point to generate a summary of statistics for a set of nodes. I'm about to release 1.0-pre1 once I fixed the multicast issues on SunOS. netconfig Cisco IOS like iproute2 replacement fully based on netlink supporting full tab completion and command description for those familiar with IOS. It is far from being releaseable as libnl needs to be finished first. Ideas exist to use it as interface for the new ietf netconf protocol. generic netlink network statistics A few of may have noticed the first small set of patches going into 269rc3 introducing generic network statistics which will make the process of adding new statistics to qdiscs/class/filters very easy and removes the issue of backward compatibility. netlink errors Effort has gone into improving error codes/messages returned from netlink users in the kernel. The idea is currently on hold as all acceptable solutions would break APIs. Planned for 2.7 as of now. ___ LARTC mailing list / [EMAIL PROTECTED] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Re: [LARTC] Qdisc statistics project
On Thursday 14 October 2004 10:39, Antonios Chalkiopoulos wrote: > > >As a necessety for my job is to real-time monitor the bytes, packets, > > > packet dropped etc of all the qdiscs working inside the kernel i've > > > tried varius methods: > > > > > >1. Parse tc -s command output and update a round robin database and use > > >rrdtool to graphically display tc statistics. > > > > I have developed myself a similar setup, but i used a perl script with > > snmp pass_persist to retrieve the data via snmp feed it to MRTG and then > > display it with a CGI script, since i changed jobs recently i made some > > changes to the setup and was thinking in creating a sourceforge project. > > I guess you use the qos snmp extentions to achieve the above. The only > drawback to your design is the cpu usage of cgi. > > > But i don't think it is ready for that yet, i mean, it is working > > beautifully for me (and in my previous employer) but there are some > > rough edges to address first. > > I can forward you the software that i use with some documentation. > > Please do so. I did the same, see www.docum.org. Most of the scripts are a mess :) but they work for me. My scripts can: - parse tc output and recreate the tc setup (parent - child relation ship) - use snmp extension to get stats remotly - use rrd to store the stats - a cgi script to generate graphs on the fly - create/edit htb + filters setup and generate commands graphical (cgi-bin script, 1 big hack) - realtime graphs ina browser: written in java, but data is fetched from a webserver so it's not that handy - much more that I forgot The development of these scripts is stopped. I'm rebuilding my house, making a website, I'm a father for 6 months now, .. but if an interesting project is started, I want to help. Stef -- [EMAIL PROTECTED] "Using Linux as bandwidth manager" http://www.docum.org/ ___ LARTC mailing list / [EMAIL PROTECTED] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Re: [LARTC] Qdisc statistics project
> >As a necessety for my job is to real-time monitor the bytes, packets, > > packet dropped etc of all the qdiscs working inside the kernel i've tried > > varius methods: > > > >1. Parse tc -s command output and update a round robin database and use > >rrdtool to graphically display tc statistics. > > I have developed myself a similar setup, but i used a perl script with > snmp pass_persist to retrieve the data via snmp feed it to MRTG and then > display it with a CGI script, since i changed jobs recently i made some > changes to the setup and was thinking in creating a sourceforge project. I guess you use the qos snmp extentions to achieve the above. The only drawback to your design is the cpu usage of cgi. > But i don't think it is ready for that yet, i mean, it is working > beautifully for me (and in my previous employer) but there are some > rough edges to address first. > I can forward you the software that i use with some documentation. Please do so. Thanks, Antonio ___ LARTC mailing list / [EMAIL PROTECTED] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Re: [LARTC] Qdisc statistics project
On Wed, 2004-13-10 at 18:12 -0400, Jason Boxman wrote: > I was playing with that as well, but of late it doesn't seem to be actively > developed. Someone else mentioned LQL[1], but it doesn't seem to have hooks > to let you grab qdisc stats yet. > > [1] http://www.coverfire.com/lql/ > LQL seems to be the only actively developed project currently that could > eventually allow you to plug into netlink and poll for statistics. Right now > it seems you can only get and set parameters. I am currently working on statistics support. These new features are not done yet but should be within a couple of weeks. See below for a bit more information. http://www.coverfire.com/archives/2004/10/13/lql-update/ -- OpenPGP key: http://www.coverfire.com/files/pubkey.txt Key fingerprint: FB0A 2D8A A1E9 11B6 6CA3 0C53 742A 9EA8 891C BD98 ___ LARTC mailing list / [EMAIL PROTECTED] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Re: [LARTC] Qdisc statistics project
On Tuesday 12 October 2004 09:32, Antonios Chalkiopoulos wrote: > As a necessety for my job is to real-time monitor the bytes, packets, > packet dropped etc of all the qdiscs working inside the kernel i've tried > varius methods: > > 1. Parse tc -s command output and update a round robin database and use > rrdtool to graphically display tc statistics. I find that generally works. > [varius perl scripts exist for the above job] > > 2. Unsuccesfully tried QoS SNMP extensions, in order to use a new MIB to > extract qdisc stats from. I was playing with that as well, but of late it doesn't seem to be actively developed. Someone else mentioned LQL[1], but it doesn't seem to have hooks to let you grab qdisc stats yet. [1] http://www.coverfire.com/lql/ > There is also some work in the kernel level that will help reveal qdisc > stats to userspace (thanks Thomas Graf) Are you talking about tcstat[2]? [2] http://reeler.org/tcstat/index.html > To get to the point.. the perl parsing method is hackish and quick&dirty. > A proper open-source tc monitoring tool SHOULD exist ! Maybe inside > iproute2 maybe in sourceforge. LQL seems to be the only actively developed project currently that could eventually allow you to plug into netlink and poll for statistics. Right now it seems you can only get and set parameters. > Are there any volunteers to start working on such a project? I could put > 4-6 weeks full time work on that... any suggestion for the most proper > solution to the above problem is welcome. I wrote yet another script to poll `tc` for statistics and pass that information off to RRDTool via Perl's RRDs module. I am using it for diagnostic purposes. It polls every 10s and updates a graph[3]. For long term tracking I am going to write a simple plugin for Munin[4] to grab information every five minutes and pass that off for graphing. [3] http://edseek.com/foo.png [4] http://www.linpro.no/projects/munin/ -- Jason Boxman Perl Programmer / *NIX Systems Administrator Shimberg Center for Affordable Housing | University of Florida http://edseek.com/ - Linux and FOSS stuff ___ LARTC mailing list / [EMAIL PROTECTED] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Re: [LARTC] Qdisc statistics project
Hi Antonios. Antonios Chalkiopoulos wrote: As a necessety for my job is to real-time monitor the bytes, packets, packet dropped etc of all the qdiscs working inside the kernel i've tried varius methods: 1. Parse tc -s command output and update a round robin database and use rrdtool to graphically display tc statistics. I have developed myself a similar setup, but i used a perl script with snmp pass_persist to retrieve the data via snmp feed it to MRTG and then display it with a CGI script, since i changed jobs recently i made some changes to the setup and was thinking in creating a sourceforge project. But i don't think it is ready for that yet, i mean, it is working beautifully for me (and in my previous employer) but there are some rough edges to address first. Btw, the setup generates TC, iptables and MRTG configuration from a config file. I think it is time to see how can the setup be improved. [varius perl scripts exist for the above job] 2. Unsuccesfully tried QoS SNMP extensions, in order to use a new MIB to extract qdisc stats from. There is also some work in the kernel level that will help reveal qdisc stats to userspace (thanks Thomas Graf) I have never heard of this, must google it. To get to the point.. the perl parsing method is hackish and quick&dirty. A proper open-source tc monitoring tool SHOULD exist ! Maybe inside iproute2 maybe in sourceforge. I really never tried to write a better tool, with less 'tc -s' parsing, don't have the time or incentive to hack the needed code, so can't help you there :-) Are there any volunteers to start working on such a project? I could put 4-6 weeks full time work on that... any suggestion for the most proper solution to the above problem is welcome. I can forward you the software that i use with some documentation. Thanks, Antonio ___ LARTC mailing list / [EMAIL PROTECTED] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ Hope it Helps José Araújo ___ LARTC mailing list / [EMAIL PROTECTED] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
[LARTC] Qdisc statistics project
As a necessety for my job is to real-time monitor the bytes, packets, packet dropped etc of all the qdiscs working inside the kernel i've tried varius methods: 1. Parse tc -s command output and update a round robin database and use rrdtool to graphically display tc statistics. [varius perl scripts exist for the above job] 2. Unsuccesfully tried QoS SNMP extensions, in order to use a new MIB to extract qdisc stats from. There is also some work in the kernel level that will help reveal qdisc stats to userspace (thanks Thomas Graf) To get to the point.. the perl parsing method is hackish and quick&dirty. A proper open-source tc monitoring tool SHOULD exist ! Maybe inside iproute2 maybe in sourceforge. Are there any volunteers to start working on such a project? I could put 4-6 weeks full time work on that... any suggestion for the most proper solution to the above problem is welcome. Thanks, Antonio ___ LARTC mailing list / [EMAIL PROTECTED] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
