Hi Adam,
You are entirely correct: those are writers queueing up because
the DB is not able to process the load fast enough. Don't know
if it will help but mongo_refresh_time allows to configure time
intervals to insert into the DB. It can be an idea to let this
coincide with the mongo_history value, ie.
mongo_history: 5m
mongo_refresh_time: 300
The underlying idea is you cache data a bit more in memory (and
sum bytes and packet counters) to alleviate the db. But it's
also true if your aggregation method includes primitives that
defeat the concept, ie. timestamp_start, timestamp_end, then
it might be of no use.
Cheers,
Paolo
On Mon, Dec 02, 2013 at 11:44:30PM -0500, Adam Jacob Muller wrote:
> Hi Paolo,
>
> With things in that configuration I do actually get time-based bins,
> however...
>
> nfacctd seems to start by spawning a single mongodb process:
> pmacct-1.5.0rc1 # ps aux|grep Mongo
> root 15527 2.2 0.1 117040 47720 pts/8 S+ 23:39 0:00
> nfacctd: MongoDB Plugin [all]
>
> After a little bit this grows to two, then three:
> pmacct-1.5.0rc1 # ps aux|grep Mongo
> root 15527 2.8 0.3 120868 78504 pts/8 S+ 23:39 0:00
> nfacctd: MongoDB Plugin [all]
> root 15545 0.6 0.1 123384 44896 pts/8 S+ 23:39 0:00
> nfacctd: MongoDB Plugin [all]
> root 15567 0.0 0.0 8952 664 pts/0 R+ 23:39 0:00
> grep --colour=auto Mongo
> pmacct-1.5.0rc1 # ps aux|grep Mongo
> root 15527 3.0 0.3 121528 82244 pts/8 S+ 23:39 0:00
> nfacctd: MongoDB Plugin [all]
> root 15545 0.6 0.1 123384 44896 pts/8 S+ 23:39 0:00
> nfacctd: MongoDB Plugin [all]
> root 15568 0.0 0.1 128148 49136 pts/8 S+ 23:39 0:00
> nfacctd: MongoDB Plugin [all]
>
> After another minute, this grows very, very quickly until the box will OOM:
> fresh-netflow pmacct-1.5.0rc1 # ps aux|grep -c Mongo
> 72
>
> Is this possibly just because MongoDB can't keep up with the insert
> rate I'm asking it for here?
>
> -Adam
>
> On 12/1/13, 10:24 PM, Paolo Lucente wrote:
> >Hi Adam,
> >
> >You should set mongo_history to something, ie.:
> >
> >mongo_history[all]: 5m
> >mongo_history_roundoff[all]: m
> >
> >This enables historical accounting and time-bins and you should then
> >see collection names being populated with the correct date (and time).
> >Let me know if this works for you.
> >
> >When you say you want 1:1 packets, you mean flow records, correct?
> >
> >Cheers,
> >Paolo
> >
> >On Sun, Dec 01, 2013 at 02:01:12AM -0500, Adam Jacob Muller wrote:
> >>Hi,
> >>I have a relatively simple configuration I'm testing out:
> >>
> >>nfacctd_ip:a.b.c.d
> >>nfacctd_port:2104
> >>plugins: mongodb[all]
> >>
> >>aggregate[all]: dst_host
> >>mongo_table[all]: nfacctd_whk.hist_%Y_%m_%d
> >>mongo_host[all]: e.f.g.h
> >>
> >>
> >>That's a very highly simplified version of my final goal (I really
> >>want nearly 1:1 packets into mongodb records eventually)
> >>
> >>This mostly works, with the exception that the variables in
> >>mongo_table are not properly substituted. Records get put into the
> >>collection "hist_1969_12_31". This is obviously something getting
> >>set to unixtime 0. nfacctd_time_new appears to exist for the purpose
> >>where my netflow sender is not setting the proper date/time headers
> >>(which, actually, I don't think is the case here) but even setting
> >>nfacctd_time_new: true does not fix the issue. The collection is
> >>still created with the wrong date.
> >>
> >>If I include timestamp_start, timestamp_end in the aggregate
> >>definition, the timestamps do appear correct in the resulting
> >>mongodb entries.
> >>
> >>{
> >> "_id" : ObjectId("529ade3f70fd695100000013"),
> >> "ip_dst" : "i.j.k.l",
> >> "timestamp_start" : Timestamp(1385881117, 160000000),
> >> "timestamp_end" : Timestamp(1385881117, 160000000),
> >> "packets" : NumberLong(1),
> >> "bytes" : NumberLong(52)
> >>}
> >>
> >>I am running:
> >>NetFlow Accounting Daemon, nfacctd 1.5.0rc1 (20130829-00)
> >> --enable-mongodb
> >>
> >>Thanks,
> >>-Adam
> >>_______________________________________________
> >>pmacct-discussion mailing list
> >>http://www.pmacct.net/#mailinglists
> >
> >_______________________________________________
> >pmacct-discussion mailing list
> >http://www.pmacct.net/#mailinglists
>
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
