Re: Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Tom Buskey]

On 6/15/07, Jon 'maddog' Hall <[EMAIL PROTECTED]> wrote:
Interesting stuff I agree with deleted.


Maybe engineering expectations have changed since the days of 10Mbit

ETHERNET, but I do remember having those discussions with the
engineering staff.




Sure, 10base ethernet gets bogged down when there are multiple connections
in the "ether".  The protocol says to talk.  If someone else is talking,
shutup for a random amount of time.  Then talk.  At a certain point you
saturate the ether.  IIRC anything over 70% utilization is the most you can
expect on the back of a napkin.

Switches isolate the traffic between nodes.

Token Ring and FDDI have a token that gets passed around.  You can't talk if
you don't have the token.  You only get to keep the token for a certain
length of time.  You'll get much closer to wire speed and have more
consistent performance.

100base ethernet mainly just drives the chips 10 times faster.  HP had
100baseT4 that was more like token ring IIRC.  It used 4 Cat5 pairs and
incompatible equipment.

For most cases, the shout/shutup/wait protocol works.

I'm not sure how gigabit is different, but it's not 100base ten times
faster.  For example there are not gigabit hubs.  They're all switches.

/me goes to google to update on gigabit while listening here ;-)
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Jon 'maddog' Hall]

On Thu, 2007-06-14 at 22:16 -0400, Paul Lussier wrote:
> "Tom Buskey" <[EMAIL PROTECTED]> writes:
> 
> > On top of that, if hdparm says timed disk writes are around 40MB, what
> >> could you see for sustained download speeds? Maybe a static cached
> >> webpage could saturate a gig connection, sustained 5 gig http download
> >> couldn't right?
> >>
> >> Anyone have real world answers for that stuff?
> >
> > What if you're downloading to RAM disk?
> 
> What if you're building a router?  The traffic never hits the disk, so
> drive performance is irrellevent here.


For each individual node, the speed of the bus and how fast the packet
can be transferred to the disk is important, but for the total bandwidth
of the wire, it is nice to have it as fast as possible while still
preserving things like reliability, reasonable physical length of the
wire for that speed, etc.

If I remember correctly, after a packet comes across, the controller is
supposed to wait some period of time before grabbing the wire again.
This allows some other controller time to grab the wire.  So even a
10Gbit wire is not really going to transfer data at a full 10Gbits
continuously.

I remember a case where Sun was beating Digital's (and other vendors)
pants off on ETHERNET transfers.  Then we did a study of a Sun system on
the wire, and found out that they were not waiting this entire "period"
between packet transfers, so would grab the wire a disproportionate
amount of the time.  When they corrected this "problem", their ETHERNET
capability dropped back to normal.

Digital used to talk about "balance" of a system.  Balancing CPU power,
with size of memory, size of cache, speed of disks and bus.  Putting a
super-fast CPU on a bus that could not deliver the data meant that you
were wasting a lot of power.  Likewise a super-fast ETHERNET controller.

AFAIK it was accepted and anticipated that an ETHERNET controller was
never going to effectively move wire speed at any given time onto or off
of the disk.  That is one of the reasons why there are so many "copy to"
moves inside of the kernel, but it did mean that if you sent the bits
over the wire at "wire speed" the controller would have half a chance of
receiving them correctly and making sure they went where they were
supposed to go, then free up the wire for someone else to use.

Maybe engineering expectations have changed since the days of 10Mbit
ETHERNET, but I do remember having those discussions with the
engineering staff.

And I agree that if you want to have the fastest path to your disk,
having a bus that can support it is a place to start, no matter what
ETHERNET controller you have.

md


___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Ben Scott]

On 6/15/07, Mark Komarinski <[EMAIL PROTECTED]> wrote:
> I haven't checked to see what PCI-X/PCI-E does, but I've hit pretty high
> speeds with it (see below).

  I looked it up last night.  According to the always reliable
Wikipedia, PCI-X brings the bus clock to 133 MHz.  Still 64-bit.  So
8512 megabit/sec before overhead.  PCIe (PCI Express) comes in more
flavors than ice cream, but the "x1" slots are given as 250
megabyte/sec.

  (Note bytes vs bits in the above.  Not knowing the implementations
details, I'm presenting the figures as given.)

-- Ben
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Mark Komarinski]

On 06/14/2007 03:51 PM, Tom Buskey wrote:
> On 6/14/07, *Flaherty, Patrick* <[EMAIL PROTECTED]
> > wrote:
>
> I'm not the best with these bit/byte problems so I might be wrong,
> but.
>
> A PCI bus can pass 1056 bits a second (32 bit, 33 mhz)
> tcp/ip over head is somewhere around %20 (1056 * .8 = 844.8)
>
BTW, that's a standard PCI bus.  Usual performance after taking out PCI
overhead is a bit over 500Mbit (see the 50MB rate below).  PCI-64 bit
and PCI-66Mhz/100Mhz multiply those rates as standard PCI is
32bit/33Mhz, so a PCI-64bit can handle twice the bandwidth of PCI-32bit
and PCI-66Mhz/64bit can handle 4 times.  You're now easily in the Ghz
range, even if you have other devices on the bus.

I haven't checked to see what PCI-X/PCI-E does, but I've hit pretty high
speeds with it (see below).
>
>
> What can you reasonably expect a pci gigabit card to give you for
> through put?
>
>
> The author of O'Reilly "Unix Backup & Restore" says you should expect
> a maximum throughput around 50MB/s for backups over gigabit.
500Mbit is the untuned rate I saw under Linux using netpipe.  With a bit
of kernel tuning I could get to the 825-875Mbit range.
>  
>
> PCI Buses are generally shared (save high end server boards) right?
>
>
> Yep.  Higher end systems will have multiple PCI buses.  The Sun v890
> has 4 seperate buses and you can distribute the cards based on
Most server-class systems have multiple busses or at least have a
separate bus for its Ethernet controllers.  Even on a desktop-class
system, there's not a whole lot else going over that PCI bus unless you
have a second Gig Ethernet card or a SCSI card.
>  
>
> On top of that, if hdparm says timed disk writes are around 40MB, what
> could you see for sustained download speeds? Maybe a static cached
> webpage could saturate a gig connection, sustained 5 gig http download
> couldn't right?
>
> Anyone have real world answers for that stuff?
>
>
> What if you're downloading to RAM disk?
> When I've been doing my network measurements I've been going from
> /dev/zero to /dev/null to eliminate the storage speed effects. 

The reason we did the timing was a few:

- MPI/PVM is usually CPU-CPU (no disk) and we don't want to spend the
additional money for a high speed interconnect, so making the Ethernet
connection really fast was a 0-cost improvement.
- The faster you can shuttle data to a device, the less time that device
(and others trying to contend for bandwidth) are waiting.

-Mark
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Paul Lussier]

"Tom Buskey" <[EMAIL PROTECTED]> writes:

> On top of that, if hdparm says timed disk writes are around 40MB, what
>> could you see for sustained download speeds? Maybe a static cached
>> webpage could saturate a gig connection, sustained 5 gig http download
>> couldn't right?
>>
>> Anyone have real world answers for that stuff?
>
> What if you're downloading to RAM disk?

What if you're building a router?  The traffic never hits the disk, so
drive performance is irrellevent here.
-- 
Seeya,
Paul
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Michael ODonnell]

PCI-32 theoretical maximum throughput would be:

   (((33 million cycles) * 32 bits) / 8 = 132 million bytes ) per second

...but since that's unattainable for more than a dozen ticks or so I'm
guessing that 2/3 of that (88 million) is a more reasonable maximum.

Meanwhile, I (think I) have heard that the rule-of-thumb for
Enet overhead is something like:

   bitrate / 12 = bytes-per-second

...so for GigE we'd get:

   ~1,000,000,000 bits per second / 12 = ~83,333,333

...which is in the same ballpark as that PCI guesstimate.
 
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Ben Scott]

On 6/14/07, Jeff Macdonald <[EMAIL PROTECTED]> wrote:
>> What can you reasonably expect a pci gigabit card to give you for
>> through put?
>
> I thought I read somewhere gig ether should be on the motherboard, not PCI.

  Many modern motherboards do include one or more gig Ether ports.
Sometimes, these are just connected to internal PCI buses.  Sometimes,
they're even all on the same PCI bus.  Sometimes they use internal PCI
Express ports.  Sometimes they hook directly into a special path on
one of the core chips.  You can get varying levels of performance
here, depending on the specifics.

  There's PCI in multiple flavors (32-bit vs 64-bit, 33 MHz vs 66
MHz), PCI-X (some sort of extension to the classic PCI bus), and PCI
Express (totally different electrically; serial rather than parallel).

  I do agree with Patrick Flaherty and Tom Buskey that the NIC may not
be the constraint (although a cheap chipset can certainly kill your
performance).  You need to worry about bus bandwidth (for NIC and disk
controller), disks, disk controller, core logic, IP stack, application
software, kernel tuning, and sometimes even RAM and CPU (depending on
the workload).  And you need to worry about that on both computers.

  Cable and switch quality can also matter, too.  A bad cable can
cause all manner of problems.  And I've heard of cheap gig switches
which couldn't actually forward data that fast.

-- Ben
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Jeff Macdonald]

On 6/14/07, Flaherty, Patrick <[EMAIL PROTECTED]> wrote:


I'm not the best with these bit/byte problems so I might be wrong,
but.

A PCI bus can pass 1056 bits a second (32 bit, 33 mhz)
tcp/ip over head is somewhere around %20 (1056 * .8 = 844.8)

What can you reasonably expect a pci gigabit card to give you for
through put?




I thought I read somewhere gig ether should be on the motherboard, not PCI.
Since then PCI Express (or whatever the faster PCI is called) has come out
and that can handle Gig Ether...


--
Jeff Macdonald
Ayer, MA
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Tom Buskey]

On 6/14/07, Flaherty, Patrick <[EMAIL PROTECTED]> wrote:


I'm not the best with these bit/byte problems so I might be wrong,
but.

A PCI bus can pass 1056 bits a second (32 bit, 33 mhz)
tcp/ip over head is somewhere around %20 (1056 * .8 = 844.8)

What can you reasonably expect a pci gigabit card to give you for
through put?



The author of O'Reilly "Unix Backup & Restore" says you should expect a
maximum throughput around 50MB/s for backups over gigabit.


PCI Buses are generally shared (save high end server boards) right?


Yep.  Higher end systems will have multiple PCI buses.  The Sun v890 has 4
seperate buses and you can distribute the cards based on


On top of that, if hdparm says timed disk writes are around 40MB, what

could you see for sustained download speeds? Maybe a static cached
webpage could saturate a gig connection, sustained 5 gig http download
couldn't right?

Anyone have real world answers for that stuff?



What if you're downloading to RAM disk?
When I've been doing my network measurements I've been going from /dev/zero
to /dev/null to eliminate the storage speed effects.
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

RE: Recommended PCI gigabit ethernet card? OT: PC Gigabit Through put Question

[Flaherty, Patrick]

I'm not the best with these bit/byte problems so I might be wrong,
but.

A PCI bus can pass 1056 bits a second (32 bit, 33 mhz)
tcp/ip over head is somewhere around %20 (1056 * .8 = 844.8)

What can you reasonably expect a pci gigabit card to give you for
through put? 

PCI Buses are generally shared (save high end server boards) right? 

On top of that, if hdparm says timed disk writes are around 40MB, what
could you see for sustained download speeds? Maybe a static cached
webpage could saturate a gig connection, sustained 5 gig http download
couldn't right?

Anyone have real world answers for that stuff?

Patrick




From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Tom Buskey
Sent: Thursday, June 14, 2007 12:36 PM
To: gnhlug-discuss@mail.gnhlug.org
Subject: Recommended PCI gigabit ethernet card?


I have a cheap gigabit nic ($20) in my system and suspect it is slowing
down throughput so I'd like to upgrade it.
I did the google linux thing.  Half were error reports, half were from <
2004, half were sales "reviews", etc (yeah, that > 100%).  The Linux
HOWTOs are 2004 and earlier so there's barely a mention of gigabit
networking. 

It needs to be PCI
I'm running Fedora with Fedora kernels and don't want to compile
drivers.

What do people use, see as fast/compatible?


___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/