Burton,
if you go on the ntop-ht cvs you will see that there's a mutex for
serializing the access to pcap_XXX. The new ntop-ht is running since
last week on a busy host and it seems to be stable. It losses -50%
packets of the "normal" ntop.
I haven't heard from users yet, but I think it's time to start thinking
to merge the two subtrees.
Cheers, Luca
Burton Strauss wrote:
>Luca:
>
>This whole thing bothers me and I finally figured out why. You are
>depending upon undocumented behavior of libpcap. You are ASSUMING that a
>multi-threaded application, making pcap_loop() calls from each thread
>against the same pcap_t structure will see different packets. A
>load-balancing scenario.
>
>However, that's not documented. pcap_loop is this (0.8.3):
>
>int pcap_loop(pcap_t *p, int cnt, pcap_handler callback, u_char *user) {
>...
> for (;;) {
>...
> do {
> n = p->read_op(p, cnt, callback, user);
> } while (n == 0);
>...
> }
>}
>
>The actual read operation is different for EACH environment:
>
>pcap-dag.c:484: handle->read_op = dag_read;
>pcap-dlpi.c:713: p->read_op = pcap_read_dlpi;
>pcap-linux.c:406: handle->read_op = pcap_read_linux;
>pcap-nit.c:288: p->read_op = pcap_read_nit;
>pcap-pf.c:442: p->read_op = pcap_read_pf;
>pcap-snit.c:347: p->read_op = pcap_read_snit;
>pcap-snoop.c:331: p->read_op = pcap_read_snoop;
>pcap-win32.c:302: p->read_op = pcap_read_win32;
>
>
>It certainly seems to work. However internally:
>
>(1) There is NO serialization. Especially under SMP (dual core), it's going
>to be possible for multiple threads for the same device to be running and
>end up inside (for example under Linux) recvfrom(). pcap_t is an opaque
>structure, but if you look at it in pcap_int.h, it's just a bunch of
>counters, fields and pointers.
>
>This isn't something we can patch/work-around in ntop - it's below the
>pcap_loop() call, where libpcap might be sleeping awaiting packets.
>
>
>(2) Different behavior under different OSes. Linux ends up using
>recvfrom(), bpf uses read(). Win32 and dag use the memory addresses of
>their ring buffers directly.
>
>
>(3) Different behavior under different libpcaps. Between 0.7.x and 0.8.x
>this underwent a major rewrite to make the OS/device specific operations
>work more transparently. C++ style class overloading (read_op), vs. direct
>calls. It shouldn't FUNCTIONALLY be different, but even so, these areas get
>a lot of changes especially with quirks for new OSes.
>
>
>(4) Thread safety. In the mailing list - at least as of a year ago - there
>were still some questions as to whether the DAG, DLPI (HP/Solaris) versions
>were really thread safe.
>http://www.tcpdump.org/lists/workers/2004/04/msg00179.html. Even if they've
>fixed this, I highly doubt that they were thinking of this kind of thread
>safety.
>
>
>I really think you need to create a manager thread which runs pcap_loop()
>and then dispatches the received packets to a number of worker threads.
>That way, we can use your new functions to serialize access to the ntop
>structures and safely update HostTraffic.
>
>
>I'm going to post over on tcpdump-workers and will follow up here with what
>I find. But I think this is just too far a leap for 3.2.
>
>-----Burton
>
>
>
>
>
>
>
>_______________________________________________
>Ntop-dev mailing list
>[email protected]
>http://listgateway.unipi.it/mailman/listinfo/ntop-dev
>
>
--
Luca Deri <[EMAIL PROTECTED]> http://luca.ntop.org/
Hacker: someone who loves to program and enjoys being
clever about it - Richard Stallman
_______________________________________________
Ntop-dev mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop-dev
