Re: Device polling performance
In a message dated 9/27/04 3:04:22 PM Eastern Daylight Time, [EMAIL PROTECTED] writes: > Mike at sentex.net previously wrote: > > "Given a decent CPU, you wont see very much of a load average at all in the > 200Kpps / 100Mb range." Note that load average and CPU usage are two intirely different things. You could have a huge amount of CPU usage with a load average hovering around zero and somewhat vice versa too - eg high load average without a great deal of CPU usage - though that would be less common. jerry Since device polling is entirely a kernel process (and userland "load average" has nothing to do with it), his statement would have been completely irrelevent if he were, in fact, talking about userland load average. Load average is virtually useless and shouldn't be part of any conversation originating after 1990. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Device polling performance
In a message dated 9/25/04 4:24:07 PM Eastern Daylight Time, [EMAIL PROTECTED] writes: >The EVIDENCE is to the contrary, since it seems that a 2.4Ghz system >will be saturated when bridging ~250Kpps with device-polling enabled, >based on polling stats and userland benchmarking, even though the >system claims to be 100% idle. Interestingly, its about the same with >interrupt enabled. > >The POINT is that since there is no way to measure the performance, >you've got a bunch of guys who think they've figured something out >touting device-polling without having a clue what the performance >advantages (or consequences) are, so it might as well be black magic, >or snake oil, since you are as blind as a bat in your assessments. Hello, Please post your "polling stats and userland benchmarking" results. I would be very interested seeing them as I was thinking of moving to NICs that would benefit from polling. However, because you have "EVIDENCE ... to the contrary", I may hold off. On the other hand, you do go on to say "there is no way to measure the performance" and "you are as blind as a bat in your assessments", so also please post your test methodology. I need to make my decision on reliable, repeatable facts. Also, when you post, would you please wrap your lines to a shorter length? Not everyone on the list uses AOL Reader, like you. -- --- The "evidence" is a bit circumstantial in the absence of working tools, but here are some "observations". There's also an assumption that the knobs associated with polling work as expected. Test machine is a 2.4Ghz celeron box with dual Intel NICs (em driver) on a 32bit, 33Mhz bus, running FreeBSD 4.9. Now I realize that a 133Mhz, 64bit bus is 8x faster and you certainly wouldnt use these NICs on a real network, but for the purpose of a control it doesnt matter, since both tests are on the same MB. Settings: HZ=1000 each_burst=512 max_burst=1000 user_frac=variable RXdescriptors (receive ring size) = 512 (Note that the burst never exceeded 100 at any time) I'm firing a controlled stream of 100K pps through the box (bridging). With only normal userland (idle) usage, the box happily goes about its way. Top shows 0-1.5% usage. I started a cpu intensive userland task (buildworld or something of the sort). The system started to lose packets with a user_frac setting of 78, which implies that the system requires about 22% of the cpu to successfully manage the task, assuming the knob works (it appears to). The same machine, with interrupts enabled, uses about 26%, according to "top". HOWEVER, setting hz back to 100, with interrupts enabled the usage went down under 25%. Given that, it can be argued that there is less than a 5% bonus for polling, which makes a lot more sense than what some of the kooks have been saying. Of course the point here wasn't to prove the difference, which Im still not sure of, but the evidence certainly is that "top" doesn't properly account for CPU usage in device_polling mode.. I'd expect a small bonus, but nothing earth-shattering, as the machine still has to do the same amount of work. Its not like the machine is really servicing an interrupt for every packet, since controllers have hold offs so they don't generate interrupts on top of each other, and multiple events are regularly handled with a single interrupt. Polling gives the appearance of a machine happily going about its business no matter how much traffic you throw at it, but what happens is that you lose packets when it becomes overmatched, which never happens on a system with interrupts enabled before it goes into livelock. While livelock isn't a good thing, if it only happens occasionally, at least you aren't losing packets. Additionally, with a HZ setting of 1000 you're also introducing quite a bit of latency: additional_latency = up to 1ms in receive ring + transmit time for burst-1 frames. Increasing HZ further would reduce the latency, but adds more overhead, which slims the advantages and defeats the purpose of polling in the first place. The bottom line is that there isn't so much difference as to think that polling is going to save the day, but if you don't care about latency or losing packets it can be useful in allocating cpu cycles to user space, if thats your priority. TM ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Device polling performance
> > In a message dated 9/25/04 4:12:52 PM Eastern Daylight Time, [EMAIL PROTECTED] > writes: > >FreeBSD team for developing a stack that uses no resources. > > For the record, what I was saying was that a decent machine (e.g. 2.4 > PIV) should be able to push 200,000 packets per second with decent NICs > (em, or fxp) and with a median packet size (see www.caida.org) of about 540 > bytes, that works out to ~ 100Mb/s. > No you didn't, you said that 200Kpps would show almost no cpu usage, which > is utterly ridiculous. > > Mike at sentex.net previously wrote: > > "Given a decent CPU, you wont see very much of a load average at all in the > 200Kpps / 100Mb range." Note that load average and CPU usage are two intirely different things. You could have a huge amount of CPU usage with a load average hovering around zero and somewhat vice versa too - eg high load average without a great deal of CPU usage - though that would be less common. jerry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Device polling performance
it was said by [EMAIL PROTECTED] : >The EVIDENCE is to the contrary, since it seems that a 2.4Ghz system >will be saturated when bridging ~250Kpps with device-polling enabled, >based on polling stats and userland benchmarking, even though the >system claims to be 100% idle. Interestingly, its about the same with >interrupt enabled. > >The POINT is that since there is no way to measure the performance, >you've got a bunch of guys who think they've figured something out >touting device-polling without having a clue what the performance >advantages (or consequences) are, so it might as well be black magic, >or snake oil, since you are as blind as a bat in your assessments. Hello, Please post your "polling stats and userland benchmarking" results. I would be very interested seeing them as I was thinking of moving to NICs that would benefit from polling. However, because you have "EVIDENCE ... to the contrary", I may hold off. On the other hand, you do go on to say "there is no way to measure the performance" and "you are as blind as a bat in your assessments", so also please post your test methodology. I need to make my decision on reliable, repeatable facts. Also, when you post, would you please wrap your lines to a shorter length? Not everyone on the list uses AOL Reader, like you. Regards, Stheg __ Do you Yahoo!? New and Improved Yahoo! Mail - Send 10MB messages! http://promotions.yahoo.com/new_mail ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Device polling performance
At 11:40 AM 25/09/2004, [EMAIL PROTECTED] wrote: Ah, so the capacity of a FreeBSD router is > 10 million packets per second, since 200K pps only uses .1 % of system resources. Kudos to the FreeBSD team for developing a stack that uses no resources. For the record, what I was saying was that a decent machine (e.g. 2.4 PIV) should be able to push 200,000 packets per second with decent NICs (em, or fxp) and with a median packet size (see www.caida.org) of about 540 bytes, that works out to ~ 100Mb/s. Since its clear you have no idea what you're talking about, perhaps if someone who actually does would like to pipe in it would be useful. I am sure many others will be happy to help you... ---Mike ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Device polling performance
In a message dated 9/25/04 1:06:40 PM Eastern Daylight Time, [EMAIL PROTECTED] writes: > It seems beyond unreasonable that, with interrupts enabled, 55% of the system > is used, and with polling, ~ zero. "Inconceivable!" "Erm...I do not think that word means what you think it means." It's probably easier to claim device polling works by black magic, then to explain just how much useless overhead is added when the CPU has to service tens of thousands of interrupts per second at very high packet rates. If you only need to bridge packets from one NIC to another, modern NICs will do almost all of the work (busmastering DMA, checksum offloading, etc) without needing any CPU time at all. If you've got good hardware, device polling means that the system is usually constrained by PCI bus throughput and the actual network transmission speed of the NICs themselves, not by CPU overhead. If you attempt to use crappy hardware (try digging up some 10Mbs ISA NICs), then the CPU will have to do a lot more work, and device polling will take up more than nearly zero CPU time. If you do routing, have firewall rules (particularly using dynamic stateful rules), or NAT (particularly running userland natd), you'll probably see significant CPU being used there. The EVIDENCE is to the contrary, since it seems that a 2.4Ghz system will be saturated when bridging ~250Kpps with device-polling enabled, based on polling stats and userland benchmarking, even though the system claims to be 100% idle. Interestingly, its about the same with interrupt enabled. The POINT is that since there is no way to measure the performance, you've got a bunch of guys who think they've figured something out touting device-polling without having a clue what the performance advantages (or consequences) are, so it might as well be black magic, or snake oil, since you are as blind as a bat in your assessments. cheers, TM ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Device polling performance
[EMAIL PROTECTED] wrote: [ ... ] Ah, so the capacity of a FreeBSD router is > 10 million packets per second, since 200K pps only uses .1 % of system resources. Kudos to the FreeBSD team for developing a stack that uses no resources. It seems beyond unreasonable that, with interrupts enabled, 55% of the system is used, and with polling, ~ zero. "Inconceivable!" "Erm...I do not think that word means what you think it means." It's probably easier to claim device polling works by black magic, then to explain just how much useless overhead is added when the CPU has to service tens of thousands of interrupts per second at very high packet rates. If you only need to bridge packets from one NIC to another, modern NICs will do almost all of the work (busmastering DMA, checksum offloading, etc) without needing any CPU time at all. If you've got good hardware, device polling means that the system is usually constrained by PCI bus throughput and the actual network transmission speed of the NICs themselves, not by CPU overhead. If you attempt to use crappy hardware (try digging up some 10Mbs ISA NICs), then the CPU will have to do a lot more work, and device polling will take up more than nearly zero CPU time. If you do routing, have firewall rules (particularly using dynamic stateful rules), or NAT (particularly running userland natd), you'll probably see significant CPU being used there. Since its clear you have no idea what you're talking about, perhaps if someone who actually does would like to pipe in it would be useful. It seems obvious that the "system" resource is not accurately monitored with polling enabled, which is what Im trying to get someone to admit, or to tell me when it was or will be repaired. You've got the source code to look at. If you find a mistake in the way the statistics are monitored, great: feel free to fix it. Not only will you earn more karma by submitting a PR or email containing a diff which solves the problem then you get from whining like a petulant child, you might actually resemble a polite human being long enough to encourage other people to respond to your problems instead of ignoring you. Do you understand this point, or am I being too subtle? -- -Chuck ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Device polling performance
In a message dated 9/25/04 10:17:27 AM Eastern Daylight Time, [EMAIL PROTECTED] writes: At 09:57 AM 25/09/2004, [EMAIL PROTECTED] wrote: Hi, > As long as all your interfaces support polling, you should see >hardly see any interrupt usage at all, as that is the whole point of >polling. You can allocate more or less CPU cycles to flinging packets >around via various sysctl settings. See the polling man pages for >more info > > ---Mike > >Thanks, but that doesn't answer the question. Since polling cycles don't >seem to be shown under any usage category, how do you know what your >system usage is when polling is enabled? It seems like a big negative to me. Read the MAN page. There is a whole section there on a number of MIB variables that display various statistics around polling. 50% of the CPU cycles are allocated to the system by default. If that 50% is used up, it will show up in top under system processes in top. Given a decent CPU, you wont see very much of a load average at all in the 200Kpps / 100Mb range. ---Mike Ah, so the capacity of a FreeBSD router is > 10 million packets per second, since 200K pps only uses .1 % of system resources. Kudos to the FreeBSD team for developing a stack that uses no resources. It seems beyond unreasonable that, with interrupts enabled, 55% of the system is used, and with polling, ~ zero. Since its clear you have no idea what you're talking about, perhaps if someone who actually does would like to pipe in it would be useful. It seems obvious that the "system" resource is not accurately monitored with polling enabled, which is what Im trying to get someone to admit, or to tell me when it was or will be repaired. TM PS: and please dont tell me to read the man page again. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Device polling performance
At 09:57 AM 25/09/2004, [EMAIL PROTECTED] wrote: Hi, As long as all your interfaces support polling, you should see hardly see any interrupt usage at all, as that is the whole point of polling. You can allocate more or less CPU cycles to flinging packets around via various sysctl settings. See the polling man pages for more info ---Mike Thanks, but that doesn't answer the question. Since polling cycles don't seem to be shown under any usage category, how do you know what your system usage is when polling is enabled? It seems like a big negative to me. Read the MAN page. There is a whole section there on a number of MIB variables that display various statistics around polling. 50% of the CPU cycles are allocated to the system by default. If that 50% is used up, it will show up in top under system processes in top. Given a decent CPU, you wont see very much of a load average at all in the 200Kpps / 100Mb range. ---Mike Tommy ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Device polling performance
In a message dated 9/24/04 11:28:36 PM Eastern Daylight Time, [EMAIL PROTECTED] writes: >I thought I'd reword my question since no one seemed to understand the first >time. > >Is there a way to measure CPU kernel/interrupt usage when device polling is >enabled on 4.x systems? top and systat both show 100% idle all of the time. > Hi, As long as all your interfaces support polling, you should see hardly see any interrupt usage at all, as that is the whole point of polling. You can allocate more or less CPU cycles to flinging packets around via various sysctl settings. See the polling man pages for more info ---Mike Thanks, but that doesn't answer the question. Since polling cycles don't seem to be shown under any usage category, how do you know what your system usage is when polling is enabled? It seems like a big negative to me. Tommy ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Device polling performance
On Fri, 24 Sep 2004 11:47:52 EDT, in sentex.lists.freebsd.questions you wrote: >I thought I'd reword my question since no one seemed to understand the first >time. > >Is there a way to measure CPU kernel/interrupt usage when device polling is >enabled on 4.x systems? top and systat both show 100% idle all of the time. > Hi, As long as all your interfaces support polling, you should see hardly see any interrupt usage at all, as that is the whole point of polling. You can allocate more or less CPU cycles to flinging packets around via various sysctl settings. See the polling man pages for more info ---Mike ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Device polling performance
I thought I'd reword my question since no one seemed to understand the first time. Is there a way to measure CPU kernel/interrupt usage when device polling is enabled on 4.x systems? top and systat both show 100% idle all of the time. TM ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
