Alfredo, One new observation... When pfcount stops recording packets, if I run the load_dna_driver.sh script again and then run pfcount again it will detect packets again for a little while before it "locks up" again. At the very least, this seems like pretty good evidence that it's not a TX problem.
I have two servers: (1) Server A has a Silicom 10GbE NIC (82599-based); and (2) Server B has a NetEffect 10GbE NIC. I have a simple program on Server B that generates UDP packets. This code has been used in other applications and we've never seen this problem before. I performed two experiments to further convince myself that this isn't a TX problem. First, when pfcount (on Server A) stopped counting incoming packets, I rebooted Server A and left the packet generation code running on Server B. When Server A finished booting, pfcount briefly reported packets again. In the same circumstance, I rebooted Server B and the problem persisted. Second, these two servers are also connected through a separate network and I can send use my program to send packets across that interface indefinitely. Moreover, tcpdump on Server B shows that packets are being sent to Server A after pfcount stops recording packets. There are probably explanations of these results that are consistent with a TX issue, but it looks like an RX issue to me. =s= On Fri, Nov 18, 2011 at 1:26 PM, Alfredo Cardigliano <[email protected]> wrote: > Scott > prior to further investigate on this strange issue, are you sure the problem > is on the rx side? how are you generating the traffic? > The card seems to be an X520 (82599), right? > > Regards > Alfredo > > On Nov 18, 2011, at 8:18 PM, Scott L wrote: > >> Alfredo, >> >> I upgraded to 5.2.0 and saw some positive results. I received packets >> for a while and then they mysteriously stopped. So far, the only way >> that I've found to fix the problem is to reboot. The pfcount output >> looks something like this (the duration during which I receive packets >> seems to vary from reboot to reboot). >> >> (at this point, I've just tested connectivity using pings) >> ========================= >> Absolute Stats: [24 pkts rcvd][0 pkts dropped] >> Total Pkts=24/Dropped=0.0 % >> 24 pkts - 2'352 bytes [1.71 pkt/sec - 0.00 Mbit/sec] >> ========================= >> Actual Stats: 0 pkts [1'000.06 ms][0.00 pkt/sec] >> ========================= >> >> (in this interval, I start sending UDP packets as fast as I can from >> the other machine) >> ========================= >> Absolute Stats: [24713 pkts rcvd][12653 pkts dropped] >> Total Pkts=37366/Dropped=33.9 % >> 24'713 pkts - 36'492'694 bytes [1'647.42 pkt/sec - 19.46 Mbit/sec] >> ========================= >> Actual Stats: 24689 pkts [1'000.05 ms][24'687.56 pkt/sec] >> ========================= >> >> ========================= >> Absolute Stats: [164003 pkts rcvd][12653 pkts dropped] >> Total Pkts=176656/Dropped=7.2 % >> 164'003 pkts - 242'363'314 bytes [10'249.49 pkt/sec - 121.17 Mbit/sec] >> ========================= >> Actual Stats: 139290 pkts [1'000.05 ms][139'281.78 pkt/sec] >> ========================= >> >> (mysteriously, I stop receiving packets in this interval) >> ========================= >> Absolute Stats: [164003 pkts rcvd][12653 pkts dropped] >> Total Pkts=176656/Dropped=7.2 % >> 164'003 pkts - 242'363'314 bytes [9'646.57 pkt/sec - 114.05 Mbit/sec] >> ========================= >> Actual Stats: 0 pkts [1'000.07 ms][0.00 pkt/sec] >> ========================= >> >> ========================= >> Absolute Stats: [164003 pkts rcvd][12653 pkts dropped] >> Total Pkts=176656/Dropped=7.2 % >> 164'003 pkts - 242'363'314 bytes [9'110.65 pkt/sec - 107.71 Mbit/sec] >> ========================= >> Actual Stats: 0 pkts [1'000.06 ms][0.00 pkt/sec] >> ========================= >> >> lspci | grep 8259 returns nothing. >> >> The entry for my DNA NICs looks like: >> >> 06:00.0 Ethernet controller: Intel Corporation Device 10fb (rev 01) >> 06:00.1 Ethernet controller: Intel Corporation Device 10fb (rev 01) >> >> ethtool -S dna0 >> ------------------------------ >> NIC statistics: >> rx_packets: 176992 >> tx_packets: 0 >> rx_bytes: 262269024 >> tx_bytes: 0 >> rx_errors: 0 >> tx_errors: 0 >> rx_dropped: 0 >> tx_dropped: 0 >> multicast: 0 >> collisions: 0 >> rx_over_errors: 0 >> rx_crc_errors: 0 >> rx_frame_errors: 0 >> rx_fifo_errors: 0 >> rx_missed_errors: 0 >> tx_aborted_errors: 0 >> tx_carrier_errors: 0 >> tx_fifo_errors: 0 >> tx_heartbeat_errors: 0 >> rx_pkts_nic: 176992 >> tx_pkts_nic: 0 >> rx_bytes_nic: 262269024 >> tx_bytes_nic: 0 >> lsc_int: 2 >> tx_busy: 0 >> non_eop_descs: 0 >> broadcast: 0 >> rx_no_buffer_count: 0 >> tx_timeout_count: 0 >> tx_restart_queue: 0 >> rx_long_length_errors: 0 >> rx_short_length_errors: 0 >> tx_flow_control_xon: 0 >> rx_flow_control_xon: 0 >> tx_flow_control_xoff: 127567 >> rx_flow_control_xoff: 0 >> rx_csum_offload_errors: 0 >> alloc_rx_page_failed: 0 >> alloc_rx_buff_failed: 0 >> lro_aggregated: 0 >> lro_flushed: 0 >> lro_recycled: 0 >> rx_no_dma_resources: 0 >> hw_rsc_aggregated: 0 >> hw_rsc_flushed: 0 >> fdir_match: 0 >> fdir_miss: 0 >> fdir_overflow: 0 >> fcoe_bad_fccrc: 0 >> fcoe_last_errors: 0 >> rx_fcoe_dropped: 0 >> rx_fcoe_packets: 0 >> rx_fcoe_dwords: 0 >> tx_fcoe_packets: 0 >> tx_fcoe_dwords: 0 >> os2bmc_rx_by_bmc: 0 >> os2bmc_tx_by_bmc: 0 >> os2bmc_tx_by_host: 0 >> os2bmc_rx_by_host: 0 >> tx_queue_0_packets: 0 >> tx_queue_0_bytes: 0 >> rx_queue_0_packets: 0 >> rx_queue_0_bytes: 0 >> >> >> Thanks, >> >> =s= >> >> On Fri, Nov 18, 2011 at 2:29 AM, Alfredo Cardigliano >> <[email protected]> wrote: >>> Scott >>> Can you try with the latest stable release? (5.2.0) >>> Can I see the "lspci | grep 8259" and "ethtool -S dna0" output? >>> >>> Alfredo >>> >>> On Nov 18, 2011, at 1:48 AM, Scott L wrote: >>> >>>> Ah. I neglected to update the driver. I'm now seeing dnaX >>>> interfaces. However, I'm still not receiving packets. >>>> >>>> ifconfig dna0 >>>> ------------------ >>>> dna0 Link encap:Ethernet HWaddr 00:E0:ED:FF:18:08 >>>> inet6 addr: fe80::2e0:edff:feff:1808/64 Scope:Link >>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >>>> collisions:0 txqueuelen:1000 >>>> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) >>>> Memory:fd7e0000-fd800000 >>>> >>>> cat /proc/net/pf_ring/<pid>-dna0.8 >>>> -------------------------------------------------- >>>> Bound Device : dna0 >>>> Slot Version : 13 [5.1.0] >>>> Active : 1 >>>> Breed : DNA >>>> Sampling Rate : 1 >>>> Capture Direction : RX+TX >>>> Appl. Name : pfcount >>>> IP Defragment : No >>>> BPF Filtering : Disabled >>>> # Sw Filt. Rules : 0 >>>> # Hw Filt. Rules : 0 >>>> Poll Pkt Watermark : 128 >>>> Num Poll Calls : 1119 >>>> Channel Id : 0 >>>> Num RX Slots : 8192 >>>> Num TX Slots : 8192 >>>> Tot Memory : 34078720 bytes >>>> >>>> cat /proc/net/pf_ring/info >>>> ---------------------------------------- >>>> PF_RING Version : 5.1.0 ($Revision: 4972$) >>>> Ring slots : 4096 >>>> Slot version : 13 >>>> Capture TX : Yes [RX+TX] >>>> IP Defragment : No >>>> Socket Mode : Standard >>>> Transparent mode : Yes (mode 0) >>>> Total rings : 1 >>>> Total plugins : 0 >>>> >>>> ethtool -i dna0 >>>> ------------------------- >>>> driver: ixgbe >>>> version: 3.6.7-DNA >>>> firmware-version: 0x00011bab >>>> bus-info: 0000:06:00.0 >>>> >>>> ethtool dna0 >>>> ----------------------- >>>> Settings for dna0: >>>> Supported ports: [ FIBRE ] >>>> Supported link modes: 10000baseT/Full >>>> Supports auto-negotiation: No >>>> Advertised link modes: 10000baseT/Full >>>> Advertised auto-negotiation: No >>>> Speed: 10000Mb/s >>>> Duplex: Full >>>> Port: FIBRE >>>> PHYAD: 0 >>>> Transceiver: external >>>> Auto-negotiation: off >>>> Supports Wake-on: d >>>> Wake-on: d >>>> Current message level: 0x00000007 (7) >>>> Link detected: yes >>>> >>>> Any thoughts about why it's not working? >>>> >>>> =s= >>>> >>>> >>>> On Mon, Nov 14, 2011 at 1:59 AM, Luca Deri <[email protected]> wrote: >>>>> Hi all >>>>> you are using an old version of the code as now DNA interfaces are named >>>>> dnaX. Please fetch the latest code and drivers from SVN >>>>> >>>>> Luca >>>>> >>>>> On 11/10/2011 06:06 PM, Scott L wrote: >>>>>> >>>>>> Chris, >>>>>> >>>>>> Thanks for your response. I am running pfcount as root. I don't see >>>>>> the error for "pfcount -i eth2" only for "pfcount -i dna:eth2". >>>>>> >>>>>> In my configuration, I don't expect to receive much traffic. I am >>>>>> connecting two machines with 10 GbE NICs back-to-back. So the only >>>>>> traffic I expect to see (for now) are the pings that I am attempting >>>>>> to send; so the number of received packets is on the order of what I >>>>>> would expect. >>>>>> >>>>>> =s= >>>>>> >>>>>> On Wed, Nov 9, 2011 at 1:11 PM, Chris Collord<[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> Hi Scott, >>>>>>> You'll see that error when you run pfcount without su permission (make >>>>>>> sure >>>>>>> you sudo or use root). Not sure if that's your problem or not, i'm >>>>>>> still >>>>>>> figuring this out myself... >>>>>>> On the other hand it doesn't look like you're receiving much. Your RX >>>>>>> packets are 210, whereas mine after an hour is... >>>>>>> eth2 Link encap:Ethernet HWaddr 0xxxxxxxxx:4c >>>>>>> inet addr:0.0.0.1 Bcast:255.255.255.255 Mask:0.0.0.0 >>>>>>> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >>>>>>> RX packets:172969631 errors:0 dropped:1 overruns:0 frame:0 >>>>>>> TX packets:5 errors:0 dropped:0 overruns:0 carrier:0 >>>>>>> collisions:0 txqueuelen:1000 >>>>>>> RX bytes:116951083144 (116.9 GB) TX bytes:398 (398.0 B) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Date: Wed, 9 Nov 2011 12:05:55 -0700 >>>>>>>> From: [email protected] >>>>>>>> To: [email protected] >>>>>>>> Subject: Re: [Ntop-misc] Problems with pfcount >>>>>>>> >>>>>>>> I've updated my PF_RING code and I'm not seeing any difference (i.e., >>>>>>>> pfcount -i eth2 still shows 0 packets received but ifconfig shows >>>>>>>> packets being received). Here's what I'm seeing with the new version: >>>>>>>> >>>>>>>> ifconfig eth2 >>>>>>>> ------------------- >>>>>>>> eth2 Link encap:Ethernet HWaddr 00:E0:ED:FF:18:08 >>>>>>>> inet addr:10.0.2.45 Bcast:10.0.2.255 Mask:255.255.255.0 >>>>>>>> inet6 addr: fe80::2e0:edff:feff:1808/64 Scope:Link >>>>>>>> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >>>>>>>> RX packets:210 errors:0 dropped:0 overruns:0 frame:0 >>>>>>>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >>>>>>>> collisions:0 txqueuelen:1000 >>>>>>>> RX bytes:21420 (20.9 KiB) TX bytes:0 (0.0 b) >>>>>>>> Memory:fd7e0000-fd800000 >>>>>>>> >>>>>>>> cat /proc/net/pfring/info >>>>>>>> ----------------------------------- >>>>>>>> PF_RING Version : 5.1.0 ($Revision: 4972$) >>>>>>>> Ring slots : 4096 >>>>>>>> Slot version : 13 >>>>>>>> Capture TX : Yes [RX+TX] >>>>>>>> IP Defragment : No >>>>>>>> Socket Mode : Standard >>>>>>>> Transparent mode : Yes (mode 0) >>>>>>>> Total rings : 1 >>>>>>>> Total plugins : 0 >>>>>>>> >>>>>>>> cat /proc/net/pfring/<pid>-eth2.1 >>>>>>>> ------------------------------------------------ >>>>>>>> Bound Device : eth2 >>>>>>>> Slot Version : 13 [5.1.0] >>>>>>>> Active : 1 >>>>>>>> Breed : Non-DNA >>>>>>>> Sampling Rate : 1 >>>>>>>> Capture Direction : RX+TX >>>>>>>> Appl. Name : pfcount >>>>>>>> IP Defragment : No >>>>>>>> BPF Filtering : Disabled >>>>>>>> # Sw Filt. Rules : 0 >>>>>>>> # Hw Filt. Rules : 0 >>>>>>>> Poll Pkt Watermark : 128 >>>>>>>> Num Poll Calls : 1503 >>>>>>>> Channel Id : -1 >>>>>>>> Cluster Id : 0 >>>>>>>> Min Num Slots : 6871 >>>>>>>> Bucket Len : 128 >>>>>>>> Slot Len : 304 [bucket+header] >>>>>>>> Tot Memory : 2097152 >>>>>>>> Tot Packets : 0 >>>>>>>> Tot Pkt Lost : 0 >>>>>>>> Tot Insert : 0 >>>>>>>> Tot Read : 0 >>>>>>>> Insert Offset : 0 >>>>>>>> Remove Offset : 0 >>>>>>>> Tot Fwd Ok : 0 >>>>>>>> Tot Fwd Errors : 0 >>>>>>>> Num Free Slots : 6871 >>>>>>>> >>>>>>>> ethtool -i eth2 >>>>>>>> ---------------------- >>>>>>>> driver: ixgbe >>>>>>>> version: 3.3.9-DNA >>>>>>>> firmware-version: 1.3-0 >>>>>>>> bus-info: 0000:06:00.0 >>>>>>>> >>>>>>>> ethtool eth2 >>>>>>>> ---------------------- >>>>>>>> Settings for eth2: >>>>>>>> Supported ports: [ FIBRE ] >>>>>>>> Supported link modes: 10000baseT/Full >>>>>>>> Supports auto-negotiation: No >>>>>>>> Advertised link modes: 10000baseT/Full >>>>>>>> Advertised auto-negotiation: No >>>>>>>> Speed: 10000Mb/s >>>>>>>> Duplex: Full >>>>>>>> Port: FIBRE >>>>>>>> PHYAD: 0 >>>>>>>> Transceiver: external >>>>>>>> Auto-negotiation: off >>>>>>>> Supports Wake-on: d >>>>>>>> Wake-on: d >>>>>>>> Current message level: 0x00000007 (7) >>>>>>>> Link detected: yes >>>>>>>> >>>>>>>> >>>>>>>> I also tried pfcount -i dna:eth2, but got the following error: >>>>>>>> >>>>>>>> pfring_open error (pf_ring not loaded or perhaps you use quick mode >>>>>>>> and have already a socket bound to dna:eth2 ?) >>>>>>>> >>>>>>>> but: >>>>>>>> >>>>>>>> Module Size Used by >>>>>>>> pf_ring 335476 0 >>>>>>>> ip6t_REJECT 5856 2 >>>>>>>> nf_conntrack_ipv6 21880 2 >>>>>>>> ip6table_filter 4016 1 >>>>>>>> ip6_tables 19664 1 ip6table_filter >>>>>>>> ipv6 298864 28 ip6t_REJECT,nf_conntrack_ipv6 >>>>>>>> dm_multipath 17304 0 >>>>>>>> uinput 9248 0 >>>>>>>> ixgbe 202760 0 >>>>>>>> amd64_edac_mod 29632 0 >>>>>>>> i2c_nforce2 9048 0 >>>>>>>> dca 6708 1 ixgbe >>>>>>>> edac_core 43516 4 amd64_edac_mod >>>>>>>> forcedeth 54764 0 >>>>>>>> mlx4_core 84128 0 >>>>>>>> shpchp 34636 0 >>>>>>>> serio_raw 6644 0 >>>>>>>> pata_acpi 5152 0 >>>>>>>> pata_amd 14260 0 >>>>>>>> ata_generic 5924 0 >>>>>>>> sata_nv 25228 2 >>>>>>>> radeon 507616 1 >>>>>>>> ttm 41952 1 radeon >>>>>>>> drm_kms_helper 25360 1 radeon >>>>>>>> drm 171168 3 radeon,ttm,drm_kms_helper >>>>>>>> i2c_algo_bit 6020 1 radeon >>>>>>>> i2c_core 28608 4 i2c_nforce2,radeon,drm,i2c_algo_bit >>>>>>>> >>>>>>>> Any thoughts? Thanks for your help. >>>>>>>> >>>>>>>> =s= >>>>>>>> >>>>>>>> On Tue, Nov 1, 2011 at 2:34 AM, Luca Deri<[email protected]> wrote: >>>>>>>>> >>>>>>>>> Scott >>>>>>>>> please redo your tests using the code in PF_RING's SVN. From the >>>>>>>>> interface names (we now call them dnaX) i see that you are using old >>>>>>>>> stuff >>>>>>>>> >>>>>>>>> Luca >>>>>>>>> >>>>>>>>> On Oct 31, 2011, at 9:21 PM, Scott L wrote: >>>>>>>>> >>>>>>>>>> I'm having problems getting pfcount to work. I have two machines >>>>>>>>>> connected back-to-back: (A) has a Silicom NIC ; and (B) has a >>>>>>>>>> NetEffect 10GbE NIC. I launch pfcount on A. Then on B I start >>>>>>>>>> pinging A. pfcount continually reports zero packets received. >>>>>>>>>> >>>>>>>>>> ========================= >>>>>>>>>> Absolute Stats: [0 pkts rcvd][0 pkts dropped] >>>>>>>>>> Total Pkts=0/Dropped=0.0 % >>>>>>>>>> 0 pkts - 0 bytes [0.00 pkt/sec - 0.00 Mbit/sec] >>>>>>>>>> ========================= >>>>>>>>>> Actual Stats: 0 pkts [1'000.06 ms][0.00 pkt/sec] >>>>>>>>>> ========================= >>>>>>>>>> >>>>>>>>>> The only relevant log messages I see are: >>>>>>>>>> >>>>>>>>>> ixgbe 0000:06:00.0: eth2: NIC Link is Down >>>>>>>>>> ixgbe 0000:06:00.0: eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TX >>>>>>>>>> device eth0 entered promiscuous mode >>>>>>>>>> device eth0 left promiscuous mode >>>>>>>>>> [PF_RING] --> handle_hw_filtering_rule(command=0) >>>>>>>>>> [PF_RING] --> handle_hw_filtering_rule(command=0) >>>>>>>>>> [PF_RING] --> handle_hw_filtering_rule(command=1) >>>>>>>>>> [PF_RING] --> handle_hw_filtering_rule(command=1) >>>>>>>>>> >>>>>>>>>> As I look around I don't see the problem. I can see the packets as >>>>>>>>>> they are transmitted on B. And the RX packet count for eth2 that is >>>>>>>>>> displayed by ifconfig increments appropriately on A. But pfcount >>>>>>>>>> doesn't see them. Anyone have any suggestions? >>>>>>>>>> >>>>>>>>>> ifconfig -i eth2 >>>>>>>>>> ------------------- >>>>>>>>>> eth2 Link encap:Ethernet HWaddr 00:E0:ED:FF:18:08 >>>>>>>>>> inet addr:10.0.2.45 Bcast:10.0.2.255 Mask:255.255.255.0 >>>>>>>>>> inet6 addr: fe80::2e0:edff:feff:1808/64 Scope:Link >>>>>>>>>> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >>>>>>>>>> RX packets:719669104 errors:0 dropped:0 overruns:0 frame:0 >>>>>>>>>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >>>>>>>>>> collisions:0 txqueuelen:1000 >>>>>>>>>> RX bytes:1066541584248 (993.2 GiB) TX bytes:0 (0.0 b) >>>>>>>>>> Memory:fd7e0000-fd800000 >>>>>>>>>> >>>>>>>>>> cat /proc/net/pfring/info >>>>>>>>>> -------------------------- >>>>>>>>>> PF_RING Version : 5.0.0 ($Revision: exported$) >>>>>>>>>> Ring slots : 4096 >>>>>>>>>> Slot version : 13 >>>>>>>>>> Capture TX : Yes [RX+TX] >>>>>>>>>> IP Defragment : No >>>>>>>>>> Socket Mode : Standard >>>>>>>>>> Transparent mode : Yes (mode 0) >>>>>>>>>> Total rings : 1 >>>>>>>>>> Total plugins : 0 >>>>>>>>>> >>>>>>>>>> cat /proc/net/pfring/<pid>-eth2.30 >>>>>>>>>> -------------------------------------------- >>>>>>>>>> Bound Device : eth2 >>>>>>>>>> Slot Version : 13 [5.0.0] >>>>>>>>>> Active : 1 >>>>>>>>>> Breed : DNA >>>>>>>>>> Sampling Rate : 1 >>>>>>>>>> Capture Direction : RX+TX >>>>>>>>>> Appl. Name : pfcount >>>>>>>>>> IP Defragment : No >>>>>>>>>> BPF Filtering : Disabled >>>>>>>>>> # Sw Filt. Rules : 0 >>>>>>>>>> # Hw Filt. Rules : 0 >>>>>>>>>> Poll Pkt Watermark : 128 >>>>>>>>>> Num Poll Calls : 1461 >>>>>>>>>> Channel Id : 0 >>>>>>>>>> Num Slots : 8192 >>>>>>>>>> Tot Memory : 33685504 bytes >>>>>>>>>> >>>>>>>>>> ethtool -i eth2 >>>>>>>>>> ------------------- >>>>>>>>>> driver: ixgbe >>>>>>>>>> version: 3.3.9-DNA >>>>>>>>>> firmware-version: 1.3-0 >>>>>>>>>> bus-info: 0000:06:00.0 >>>>>>>>>> >>>>>>>>>> ethtool eth2 >>>>>>>>>> ---------------- >>>>>>>>>> Settings for eth2: >>>>>>>>>> Supported ports: [ FIBRE ] >>>>>>>>>> Supported link modes: 10000baseT/Full >>>>>>>>>> Supports auto-negotiation: No >>>>>>>>>> Advertised link modes: 10000baseT/Full >>>>>>>>>> Advertised auto-negotiation: No >>>>>>>>>> Speed: 10000Mb/s >>>>>>>>>> Duplex: Full >>>>>>>>>> Port: FIBRE >>>>>>>>>> PHYAD: 0 >>>>>>>>>> Transceiver: external >>>>>>>>>> Auto-negotiation: off >>>>>>>>>> Supports Wake-on: d >>>>>>>>>> Wake-on: d >>>>>>>>>> Current message level: 0x00000007 (7) >>>>>>>>>> Link detected: yes >>>>>>>>>> _______________________________________________ >>>>>>>>>> Ntop-misc mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>>>> >>>>>>>>> --- >>>>>>>>> We can't solve problems by using the same kind of thinking we used >>>>>>>>> when >>>>>>>>> we created them - Albert Einstein >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Ntop-misc mailing list >>>>>>>>> [email protected] >>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Ntop-misc mailing list >>>>>>>> [email protected] >>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Ntop-misc mailing list >>>>>>> [email protected] >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Ntop-misc mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > _______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
