I've opened bug 6694625 to track this issue.
Darren Reed wrote:
> Andrew Gallatin wrote:
>
>> Hi,
>>
>> When testing single-stream 10GbE netperf receive performance on both
>> S10U4 and S11b84, I found an odd performance quirk. As I increased
>> the receiver's socket buffer size, I noticed a decrease in performance
>> as the socket buffer sizes approached 512KB, with a crash in
>> performance above 512KB. I also noticed that mpstat was reporting an
>> increasing number of xcals as I increased the socket buffer size.
>> ...
>> Looking at the source of kstrgetmsg(), there is a heuristic which
>> tries to do a pullup(mp, -1) if the first mblk has a length of less
>> than mblk_pull_len, or 64 bytes. In this case, it is pulling up about
>> 460KB.
>>
>> If I disable this by setting mblk_pull_len to 0, then the decrease in
>> performance with an increase in socket buffer size goes away on S10U4
>> (haven't looked at b84 yet).
>>
>> Can somebody explain the "performance degredation from copyout
>> overhead" talked about by the comment near the declaration of
>> mblk_pull_len in common/os/streamio.c. I thought copyout was the
>> normal copy routine plus a trap handler, and should cost the same as
>> the copy in the pullup.
>>
>>
>
> For reference, the two bugs that tie in with this are:
> 4226443
> 4347047
>
> As 4347047 introduced this behaviour, I've copied a section from the
> Comments to the
> Description (nothing confidential or going to cause anyone pain) so you
> should be able
> to check that out when the bug database syncs up. As for 4226443, I
> need to review
> that one more carefully, but I'll add that the behaviour you're seeing
> was introduced by
> '047, while the general problem it was solving goes back to '443.
>
> Since you're experimenting with this, can you try changing the if() from
> if (MBLKL(bp) < mblk_pull_len) {
> to
> if ((bp->b_cont != NULL) && (MBLKL(bp) < mblk_pull_len) &&
> (MBLKL(bp->b_cont) < mblk_pull_len)) {
>
> It's not going to be as good as the "0" case, but it should be better
> than what it
> is today for your case.
>
> This is one of those times when it would be nice if there was a packet
> structure
> that "owned" the mblks and where you track the number of chained mblks,
> thus changing this "guessing game" heuristic into some simple math, e.g:
> if (pkt->dsize / pkt->mblkcnt < mblk_pull_len)
> ...
>
> Has anyone opened a bug for you on this or mentioned if there is a known
> bug associated with this?
>
> Darren
>
> _______________________________________________
> networking-discuss mailing list
> [email protected]
>
_______________________________________________
networking-discuss mailing list
[email protected]
