On Sat, Jan 29, 2011 at 02:43:07PM +1100, Bruce Evans wrote:
> On Sat, 29 Jan 2011, Slawa Olhovchenkov wrote:
>
> > On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote:
> >>
> >> To see how much CPU is actually available, run something else and see how
> >> fast it runs. A simple counting loops works well on UP systems.
> >
> > ===
> > #include <stdio.h>
> > #include <sys/time.h>
> >
> > int Dummy;
> >
> > int
> > main(int argc, char *argv[])
> > {
> > long int count,i,dt;
> > struct timeval st,et;
> >
> > count = atol(argv[1]);
> >
> > gettimeofday(&st, NULL);
> > for(i=count;i;i--) Dummy++;
> > gettimeofday(&et, NULL);
> > dt = (et.tv_sec-st.tv_sec)*1000000 + et.tv_usec-st.tv_usec;
> > printf("Elapsed %d us\n",dt);
> > }
> > ===
> >
> > This is ok?
>
> It's better not to compete with the interrupt handler in the kernel by
> spinning making syscalls, but that will do for a start.
In this programm inner loop don't contain any syscall.
Better varian -- loop with syscalls?
> > ./loop 2000000000
> >
> > FreeBSD
> > 1 process: Elapsed 7554193 us
> > 2 process: Elapsed 14493692 us
> > netperf + 1 process: Elapsed 21403644 us
>
> This shows about 35% user 65% network.
>
> > Linux
> > 1 process: Elapsed 7524843 us
> > 2 process: Elapsed 14995866 us
> > netperf + 1 process: Elapsed 14107670 us
>
> This shows about 53% user 47% network.
>
> So FreeBSD has about 18% more network overhead (absolute: 65-47), or
> about 38% more network overhead (relative: (65-47)/47). Not too
> surprising -- the context switches alone might cost that.
For only 14K vs 56K interrupt. 152% more network overhead per one interrupt.
And I see drammaticaly less number of context switches in linux stats
(by dstat).
> BTW, even -current vs my version of FreeBSD-5.2 has 10-20% more network
> overhead (relative) for tx, apparently due to bloat in the network
> stack. This apparently has nothing to do with hardware. The slowdown
> is much the same with bge (heavily modified in my version) and em
> (barely modified). One thing that I modify in both drivers is increase
> the tx ifq length by a massive amount (from about 512 to about 20000).
> This must be bad for overhead because such a large queue cannot fit
> in the L2 cache. A large amount and perhaps more than half of NIC
> overhead consists of waiting for cache misses. The slowdown in -current
> might be caused by minor bloat crossing a threshold and thus causing
> just one more cache miss every packet or 2.
I read disccursion about support DCA in FreeBSD, sound nice.
> >> Normal profiling works poorly (I see you found my old mail about high
> >> resolution profiling). Linux might be misreporting the overhead for
> >
> > I think next server will be support PMC.
> > Report from PMC still poorly?
>
> I should be adequate, but I prefer my version of perfmon which can
> count cache misses precisely for every function. But without patches,
> perfmon is even more broken than high resolution kernel profiling.
Can I use your version of perfmon? How? I don't have expirence with
any kernel profiling.
> >> ...
> >> generate too many interrupts and don't have much or any way to control
> >> this. Linux will certainly be about to handle 56K int/S better than
> >> FreeBSD since it doesn't have heavyweight interrupt threads AFAIK.
> >> FreeBSD also has "fast" interrupts, which are much like normal interrupts
> >> used to be in FreeBSD. I don't know if your NIC driver uses these. I
> >
> > re0: [FILTER]
> >
> > I think this is answer ([FILTER]), but I don't understand this answer :).
>
> [FILTER] means "fast". re used to unconditionally use "fast" interrupts
> and a task queue, which IMO is a bad way to program an interrupt
> handler, but yongari@ recently overhauled re (again :-) so that it now
> doesn't use fast interrupts by default for the MSI/MSIX case . (BTW,
Ineresting, but I don't think this help for this case -- old PCI
chip, old CPU, old RAM, old matherboard -- all old. I don't try to get
wirespeed gigabit performance from this old box, I try to compare
relative performance FreeBSD vs Linux (in last time I got many
feedback about poor network performance FreeBSD vs Linux).
> it still bogusly uses INTR_MPSAFE for the fast interrupt bus_setup_intr().)
> The overhaul probably also reduces interrupt overhead if it works on
> your hardware, just be reducing the interrupt frequency. I don't
> understand what moderates the interrupt frequency in the MSI case.
>
> >> I don't really know if this is low-end, but guess all RealTeks are :-).
> >
> > FreeBSD support interrupt moderation on this chip, and chip support
> > TOE :)
>
> The support was poor according to yongari@'s long messages about
> improving it. With working interrupt moderation, you just don't get
> an interrupt rate of even 14KHz, except transiently. 4KHz would be
> all I would be happy with on a 1-core 1.6GHz CPU. Since re is so
> primitive, yongari@ only managed to limit the rate to 20KHz. (Is this
> only for the non-MSI case, with the MSI case better even before?)
> Linux with its reduced interrupt latency can handled your observed
> 56KHz without losing by so much. My 2-core Athlon (Turion) with its
> low-end bge 5705, whose brokenness consists mainly of:
> completely broken interrupt moderation
> can't handle full 1Gbps -- saturates at about 300Mbps
> CPU and/or DMA resources used for 300Mbps are about the same as for
> a bge 5701 at a full 1Mbps
> saturates at about 100 KHz bge interrupts. This many interrupts takes
> about all of 1 CPU to handle the hardware part and 20% of another to
> generate packets. The system saturates in much the same way under
> WinXP. I only recently got a version of Linux to boot on this system
> and haven't tried network performance tests under Linux on it.
>
> Bruce
> _______________________________________________
> freebsd-performance@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-unsubscr...@freebsd.org"
_______________________________________________
freebsd-performance@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscr...@freebsd.org"
