Jimmy,
My 10 cents worth.
I agree with Luca that just blindly dumping into an MySQL database (or
any other relational database for that matter) and hoping it will solve
all your problems is not viable (which I'm sure you did not think
anyway!). However if you have enough time to configure various aspects
of the database and are prepared to go to MySQL 5.1 which is not general
release, but which does have partitioning (without which historical
traffic data is a non starter) it is very feasible. Our system stores
flow/conversation level data at 5 minute resolution pretty much for ever
and we get fairly usable reports out of it. We use nProbe rather than
ntop as the source for this data (and are very pleased with it!). There
are many things you can get out of an RDB which you cannot do with RRD
files. For each conversation we store src and dst IP and port, bytes and
packets, and a number of other parameters. This means that with the same
data set and some (admittedly quite processing expensive group by
statements) I can do top 10 reports, protocol breakdowns, IP bandwidth
utilisations and a whole range of other ways of slicing and dicing the
data. RRD will never have this flexibility. I have no problem with RRD
databases, they are extremely efficient and fast, and we also make
extensive use of them, it is just a matter of working out what your
requirements are and picking the best solution.
Not sure if I have helped or hindered, but just wanted to give an
alternative perspective.
Kind Regards,
Guy
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
[EMAIL PROTECTED]
Sent: 14 July 2008 21:09
To: [email protected]
Subject: Ntop Digest, Vol 50, Issue 13
Send Ntop mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
http://listgateway.unipi.it/mailman/listinfo/ntop
or, via email, send a message with subject or body 'help' to
[EMAIL PROTECTED]
You can reach the person managing the list at
[EMAIL PROTECTED]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Ntop digest..."
Today's Topics:
1. Re: nTop Multiple Instance Manager & Persistent mySQLData
Logger (Luca Deri)
2. Unable to build 3.3 (SVN rev 3572) on Fedora 9 (Brock Kuhse)
3. Re: Unable to build 3.3 (SVN rev 3572) on Fedora 9
([EMAIL PROTECTED])
4. Re: Unable to build 3.3 (SVN rev 3572) on Fedo ra 9
([EMAIL PROTECTED])
5. Re: Unable to build 3.3 (SVN rev 3572) on Fedo ra 9
([EMAIL PROTECTED])
6. Re: Unable to build 3.3 (SVN rev 3572) on Fedo
ra 9
([EMAIL PROTECTED])
----------------------------------------------------------------------
Message: 1
Date: Sat, 12 Jul 2008 08:18:26 +0200
From: Luca Deri <[EMAIL PROTECTED]>
Subject: Re: [Ntop] nTop Multiple Instance Manager & Persistent
mySQLData Logger
To: [email protected]
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Dear all
1. first of all Jimmy I would like to see the tool you're developing
and I expect to commit it into the ntop svn as this can be useful to
other people too. This allows the community to grow and also is a way
to cooperate.
2. As of today ntop *can*
> - To obtain top X hosts during a specified period
> - To obtain top X applications during a specified period
You simply have to play with rrdtool a bit. In the ntop traffic
report, ntop shows you already an RRD that contains all apps. You just
need to extend it
3. Mysql is not a solution. Not a solution, full stop. It's an SQL
database used for storing anything but it's not optimized for data nor
for aggregating data. From time to time people ask me about it. If
you're familiar with SQL I understand that learning rrd takes some
time, but please stop thinking in SQL as for networks is more a
problem than a solution.
Luca
----
Luca Deri <[EMAIL PROTECTED]>
You must be the change you want to see in the world
Mahatma Gandhi
On Jul 11, 2008, at 4:48 PM, Panita Pongpaibool wrote:
> Dear Jimmy and all
>
> I understand Jimmy ideas because my team is also developing similar
> features for ntop, namely
> - persistent data storage with mysql
> - decoupled web interface for plotting archived data (so that no
> need to run ntop while looking at history data)
>
> Please correct me if I'm wrong.
> What I think RRD and current NTOP can do for Jimmy's requirement.
> - I think the old data will be stored in RRD even if NTOP crashes or
> restarts. You can still use RRD to plot history data before NTOP
> restarts. (Try clicking on magnifying glass icon next to each RRD
> graph. You will get a page that let you choose to view last 1 hr,
> last 2 hr, ... data. You can go back even to last year depending on
> how to set your RRD preferences.)
> - Since RRD normally plots data in rate. It doesn't matter if the
> NTOP crashes. However if you need raw data like in BYTE units, this
> variable will be reset when NTOP restarts. In this case you will
> need to monitor for restart.
>
> What I think RRD and current NTOP cannot do and why persistent
> storage is needed.
> - To obtain top X hosts during a specified period
> - To obtain top X applications during a specified period
> To do ranking we need to compare archived data of all hosts or all
> app. It's difficult to do with current RRD since each host and each
> app are stored separately.
>
> Anyway, my team has developed a plug-in for storing data into MySQL
> (currently it stores all data RRD stores + some more). Can someone
> let me know how I can contribute plug-in to this project?
> Panita
>
>
> On Fri, Jul 11, 2008 at 1:58 AM, Gary Gatten <[EMAIL PROTECTED]>
> wrote:
> Do you have a router / routers that support netflow or sflow? nTop
> can
> export those streams and they take up MUCH less storage than full
> packet
> captures. If everyone comes back through a hub router you can easily
> monitor usage from that point only. Many options here, but if
> persistent storage is the key AND you need EVERYTHING the web
> interfaces
> display (RRD doesn't store "everything") you may have to write the
> scripts you mentioned. Have you looked at wget? The key thing is to
> accurately identify the requirements and go from there - else you'll
> waste a bunch of time gathering and massaging data that has little if
> any value.
>
> What would be cool is if the web interface was decoupled from the
> data -
> such that you could "easily" change between real time displays and
> those
> of archived files. Of course you can look at "some" history easily
> and
> some not so easy IF RRD is enabled, but if I want to view a dump
> file I
> need to start an instance of ntop to view that dump? - I think?
>
> G
>
> PS: RRD does eat space if you have full detail logging in all
> categories.
>
>
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf
> Of
> Jimmy
> Sent: Thursday, July 10, 2008 11:52 AM
> To: [email protected]
> Subject: Re: [Ntop] nTop Multiple Instance Manager & Persistent
> mySQLData Logger
>
> I really hope that nTop doesn't provide this functionality, and that I
> haven't somehow missed it, or else I have wasted a considerable amount
> of time! My main need is to have persistent storage. I can't lose
> data
> if nTop crashes, or if the box goes down for some other reason. We
> also
> need some of the data in SQL. The only way that I'm aware of to
> accomplish this with nTop, as it is, would be to use the RRD
> plugin. As
> I understand it - even if nTop is storing its data in RRD tables, it
> still won't load that data back in if it is restarted. This means
> that
> the graphs and other data found on the web front-end won't contain
> anything logged before it was started. Then we'd still need to export
> those tables to SQL. Also, in my brief experience with the RRD
> plugin,
> the tables grew wildly out of control when logging hosts on a small
> network.
>
> The other possibility would be to store the pcap traffic dumps, which
> would waste a ton of space, BUT we'd be able to read them back in to
> nTop to take a look at the different hosts, graphs, and other data -
> but
> not across multiple runs of nTop. We'd have a set of dumps for each
> time nTop was running, so if it was restarted three times, we'd have
> three sets of traffic dumps (perhaps we could merge them?).
>
> The only solution that I can see to persistently store nTop's data,
> without regard for nTop being restarted, is to write a collection of
> scripts that monitor the nTop service(s), regularly (currently,
> hourly)
> dump its counters, and generate statistics and graphs similar to the
> embedded front-end, but using our dumped data. This sacrifices the
> possibility of having live data, BUT in a situation where you're
> looking
> for network trends, and per-host totals, that's really not very
> important anyway.
>
> If there is a simpler way to do this, please let me know ASAP!
>
> When I have some more free time, I'd be happy to write some
> documentation. I've become very well acquainted with the software
> over
> the past few weeks, and could write a fairly in-depth paper on the
> different command-line switches, what nTop logs, and what nTop does
> and
> does not do.
>
> -Jimmy
>
>
> On Tue, 2008-07-08 at 17:30 -0500, Gary Gatten wrote:
> > Sounds interesting, but I think what your client wants to accomplish
> can
> > be done without writing anything. I'll read your requirements again
> > more carefully and see what's up. nTop is pretty flexible, but
> would
> > benefit from some written case studies, how to's, etc. I've been
> > meaning to write some doc myself - especially pertaining to
> netflow -
> > but haven't :(
> >
> > G
> >
> >
> > -----Original Message-----
> > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf
> Of
> > Jimmy
> > Sent: Thursday, July 03, 2008 1:33 PM
> > To: [email protected]
> > Subject: [Ntop] nTop Multiple Instance Manager & Persistent mySQL
> Data
> > Logger
> >
> > (I accidentally sent this from an address that's not subscribed to
> this
> > list, so I assume it was discarded - I apologize if it wasn't, and
> this
> > is the second time you're receiving this.)
> >
> > Hello everyone - I'm new to this list, so I'm not sure if I'm
> posting
> > this to the correct one. I wanted this information to reach
> > non-developers, so I figured it would be appropriate to post it
> here.
> >
> > I'm in the process of writing a collection of PHP scripts designed
> to
> > monitor various instances of nTop, perform hourly dumps of ntop's
> > counters, and tie in seamlessly with a web front-end that uses
> Google's
> > charts API to display some pretty charts and graphs of the data it
> has
> > collected. All of the data is stored in mySQL tables.
> >
> > Here is a short list of features:
> > *Monitor different instances of nTop (is the daemon running?)
> > *Automatically restart, and/or e-mail someone about, nTop instances
> that
> > go down unexpectedly.
> > *Persistently store data about each interface and each host, on an
> > hourly basis, and account for the counters being reset due to ntop
> being
> > restarted.
> > *The front-end displays most of the same data that the ntop embedded
> > front-end displays, except for data that ntop doesn't dump (such as
> port
> > distribution, per-host port usage, recent/active per-host sessions,
> > etc.). (In the future, I will write some scripts that extract
> certain
> > data from nTop's front-end, but that's a whole project in itself, so
> > it's pretty much dead last on my list.)
> >
> > Those are the main features - but the project is not complete, and I
> am
> > sure I will add more functionality before I begin using it
> regularly,
> > and before it is released publicly.
> >
> > I decided to write these scripts because I was asked to perform some
> > traffic and bandwidth utilization analysis for a mid-size company.
> nTop
> > is great software, but I needed to be able to permanently store its
> data
> > in a commonly used format, and most importantly, manage multiple
> > instances. My client has multiple sites spread across NYC, all of
> the
> > traffic goes to the central office, and and to their T3. They want
> > statistics for each site, individually. This script suite is
> designed
> > for this exact purpose - to manage multiple instances of nTop,
> > permanently store the data that it accumulates, and do this for a
> week
> > or two week long study. There will be such a large amount of data
> > accumulated throughout this study, that it was simply not reasonable
> to
> > have nTop dump its data to RRD tables, then manually graph different
> > data. This means I'd have to write a front-end that pulled data
> from
> > the RRD tables and displayed them, meaning it'd be easier, quicker,
> and
> > more space-friendly for me to have nTop dump the data, and have my
> > scripts pick and choose what to store permanently.
> >
> > I just wanted to give everyone a heads up, in case you might find
> this
> > at all useful. When this project is complete, and I start using
> these
> > scripts for the actual study, I will release what I have, hopefully
> > along with some decent documentation.
> >
> > Feel free to e-mail me with any questions. If anyone knows of an
> > appropriate location, associated with nTop, for me to post these
> > scripts, I'd be happy to do so.
> >
> > Thanks, and have a great day!
> >
> > -Jimmy Ryan
> >
> >
> > _______________________________________________
> > Ntop mailing list
> > [email protected]
> > http://listgateway.unipi.it/mailman/listinfo/ntop
> >
> >
> >
> >
> >
> > <font size="1">
> > <div style='border:none;border-bottom:double windowtext
> 2.25pt;padding:0in 0in 1.0pt 0in'>
> > </div>
> > "This email is intended to be reviewed by only the intended
> recipient
> > and may contain information that is privileged and/or confidential.
> > If you are not the intended recipient, you are hereby notified that
> > any review, use, dissemination, disclosure or copying of this email
> > and its attachments, if any, is strictly prohibited. If you have
> > received this email in error, please immediately notify the
> sender by
> > return email and delete this email from your system."
> > </font>
> >
> > _______________________________________________
> > Ntop mailing list
> > [email protected]
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> _______________________________________________
> Ntop mailing list
> [email protected]
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext
> 2.25pt;padding:0in 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> [email protected]
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
>
> _______________________________________________
> Ntop mailing list
> [email protected]
> http://listgateway.unipi.it/mailman/listinfo/ntop
------------------------------
Message: 2
Date: Mon, 14 Jul 2008 13:05:34 -0700
From: "Brock Kuhse" <[EMAIL PROTECTED]>
Subject: [Ntop] Unable to build 3.3 (SVN rev 3572) on Fedora 9
To: <[EMAIL PROTECTED]>
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset="us-ascii"
I came across a similar post from Piero Giobbi dated July 1st while
researching my own version of the same problem, but for some reason I
missed
the post in my mailbox (or I'd reply to it to keep the threads
together).
Anyway, I'm trying to build under Fedora 9 (with perl 5.10). Autogen
runs
file, but make dies with the output below. Does anyone have any
thoughts,
ideas, or suggestions? Is there any additional information I can
provide
(library versions, full output, etc)? This problem occurs with SVN
revision
3572.
==========================================
./autogen.sh
make
<snip>
In file included from perl.c:35:
perl/ntop_wrap.c: In function 'SWIG_Perl_ConvertPtr':
perl/ntop_wrap.c:1117: warning: declaration of '_svi' shadows a previous
local
perl/ntop_wrap.c:1117: warning: shadowed declaration is here
perl/ntop_wrap.c:1117: warning: declaration of '_svi' shadows a previous
local
perl/ntop_wrap.c:1117: warning: shadowed declaration is here
perl/ntop_wrap.c:1117: warning: declaration of '_svi' shadows a previous
local
perl/ntop_wrap.c:1117: warning: shadowed declaration is here
perl/ntop_wrap.c:1117: warning: declaration of '_svi' shadows a previous
local
perl/ntop_wrap.c:1117: warning: shadowed declaration is here
perl/ntop_wrap.c:1117: warning: declaration of '_svi' shadows a previous
local
perl/ntop_wrap.c:1117: warning: shadowed declaration is here
perl/ntop_wrap.c:1135: warning: declaration of '_svi' shadows a previous
local
perl/ntop_wrap.c:1135: warning: shadowed declaration is here
perl/ntop_wrap.c: In function 'SWIG_Perl_MakePtr':
perl/ntop_wrap.c:1159: warning: declaration of '_svi' shadows a previous
local
perl/ntop_wrap.c:1159: warning: shadowed declaration is here
In file included from perl.c:35:
perl/ntop_wrap.c: At top level:
perl/ntop_wrap.c:1687: warning: no previous prototype for
'_wrap_ntop_perl_sendString'
perl/ntop_wrap.c:1715: warning: no previous prototype for
'_wrap_ntop_perl_send_http_header'
perl/ntop_wrap.c:1753: warning: no previous prototype for
'_wrap_ntop_perl_send_html_footer'
perl/ntop_wrap.c:1770: warning: no previous prototype for
'_wrap_ntop_perl_loadHost'
perl/ntop_wrap.c:1787: warning: no previous prototype for
'_wrap_ntop_perl_getFirstHost'
perl/ntop_wrap.c:1814: warning: no previous prototype for
'_wrap_ntop_perl_getNextHost'
perl/ntop_wrap.c:1841: warning: no previous prototype for
'_wrap_ntop_perl_sendFile'
perl/ntop_wrap.c: In function '_wrap_ntop_perl_sendFile':
perl/ntop_wrap.c:1866: warning: nested extern declaration of
'ntop_perl_sendFile'
perl/ntop_wrap.c: At top level:
perl/ntop_wrap.c:1879: warning: no previous prototype for
'_wrap_ntop_perl_findHostByNumIP'
perl/ntop_wrap.c:1931: warning: no previous prototype for
'_wrap_ntop_perl_findHostBySerial'
perl/ntop_wrap.c:1973: warning: no previous prototype for
'_wrap_ntop_perl_findHostByMAC'
perl.c:44: warning: declaration of 'my_perl' shadows a global
declaration
perl.c:36: warning: shadowed declaration is here
perl.c:45: warning: declaration of 'my_perl' shadows a global
declaration
perl.c:36: warning: shadowed declaration is here
perl.c:47: warning: declaration of 'my_perl' shadows a global
declaration
perl.c:36: warning: shadowed declaration is here
perl.c:71: warning: no previous prototype for 'ntop_perl_sendFile'
perl.c:71: warning: conflicting types for 'ntop_perl_sendFile'
perl/ntop_wrap.c:1866: warning: previous implicit declaration of
'ntop_perl_sendFile' was here
perl.c:77: warning: no previous prototype for
'ntop_perl_send_html_footer'
perl.c:86: warning: no previous prototype for 'ntop_perl_loadHost'
perl.c: In function 'handlePerlHTTPRequest':
perl.c:167: error: 'argc' undeclared (first use in this function)
perl.c:167: error: (Each undeclared identifier is reported only once
perl.c:167: error: for each function it appears in.)
perl.c:167: error: 'argv' undeclared (first use in this function)
perl.c:167: error: 'env' undeclared (first use in this function)
make[2]: *** [perl.lo] Error 1
make[2]: Leaving directory `/home/brock/rpm/SOURCES/ntop'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/brock/rpm/SOURCES/ntop'
make: *** [all] Error 2
==========================================
- Brock
------------------------------
Message: 3
Date: 14 Jul 2008 17:06:20 -0300
From: [EMAIL PROTECTED]
Subject: Re: [Ntop] Unable to build 3.3 (SVN rev 3572) on Fedora 9
To: [email protected]
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset="UTF-8"
Obrigado pelo Email,
No momento estou de f?rias. Pretendo ler os meus e-mails nesse
per?odo.
Caso seja urgente, mande novamente para [EMAIL PROTECTED]
------------------------------
Message: 4
Date: 14 Jul 2008 17:07:00 -0300
From: [EMAIL PROTECTED]
Subject: Re: [Ntop] Unable to build 3.3 (SVN rev 3572) on Fedo ra 9
To: [email protected]
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset="UTF-8"
Obrigado pelo Email,
No momento estou de f?rias. Pretendo ler os meus e-mails nesse
per?odo.
Caso seja urgente, mande novamente para [EMAIL PROTECTED]
------------------------------
Message: 5
Date: 14 Jul 2008 17:07:43 -0300
From: [EMAIL PROTECTED]
Subject: Re: [Ntop] Unable to build 3.3 (SVN rev 3572) on Fedo
ra 9
To: [email protected]
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset="UTF-8"
Obrigado pelo Email,
No momento estou de f?rias. Pretendo ler os meus e-mails nesse
per?odo.
Caso seja urgente, mande novamente para [EMAIL PROTECTED]
------------------------------
Message: 6
Date: 14 Jul 2008 17:08:14 -0300
From: [EMAIL PROTECTED]
Subject: Re: [Ntop] Unable to build 3.3 (SVN rev 3572) on Fedo
ra 9
To: [email protected]
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset="UTF-8"
Obrigado pelo Email,
No momento estou de f?rias. Pretendo ler os meus e-mails nesse
per?odo.
Caso seja urgente, mande novamente para [EMAIL PROTECTED]
------------------------------
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
End of Ntop Digest, Vol 50, Issue 13
************************************
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
