All: It appears my initial assumption I can set everything to MTU 9000 was incorrect. Since my initial post, I also found a mail thread of someone else with Linux and Dell PowerConnect 5324 who had the same issues at 9000.
So I played around with the MTUs on both servers. I figured out that 6000 seems to be the best. I tested increments of 500 and every combination up to 9500 (which would give a "SIOCSIFMTU: Invalid argument" error every time, no big deal) I have a lot of data in a table format, which would be too hard to share. But if anyone wants a copy of it, which I guess would be specific to Broadcom ethernet NICs and PowerConnect 5324 switches, feel free to email me. I doubt there will be much need for it right now, so I won't put it up anywhere or make an effort to share it. Anyway, it appears when I would set one machine to 7000 and the other at 7000 or above it would drop down to ~ 25MB/sec. Otherwise, I could get anywhere from 100-117MB/sec. Which isn't a problem - theoretically, gigE can only give me 125MB/sec anyway... so 116-117MB/sec I think is really good, so I am happy now. :) Thanks for the help, and forcing me to walk through the motions! - mike
