On 24.03.2014, at 17:23, Christopher Forgeron <csforge...@gmail.com> wrote:
> I think making hw_tsomax a sysctl would be a good patch to commit - It
> could enable easy debugging/performance testing for the masses.
>
> I'm curious to hear how your environment is working with a tso turned off
> on your nics.
This will take some more time. Only one of the affected systems is running the
test kernel with prinfts, additional sysctl, and Rick’s patch. I want to be
able to reproduce the problem with that patch, before changing another variable
(like turning TSO off), but that can take days on one server. I’ll probably be
able to equip some more servers with that kernel soon, and might run a subgroup
without TSO. But first I have to make sure, the new kernel doesn’t add any new
problemes, we can’t afford them on productive servers.
> My testbed just hit the 2 hour mark. With TSO off, I don't get a single
> packet over IP_MAXPACKET. That puts my confidence at around 95% in the
> statement 'turning off tso negates this issue for me'.
>
> I'm now rebooting into a +tso env to see if I can capture the bad packets.
>
> I am also sure that the netstat -m mbuf denied is a completely separate
> issue. I'm going around the lab and powering up different boxes with
> 10.0-RELEASE, and they all have mbuf/mbuf clusters denied on boot, and that
> number increases with network traffic. It's probably not helping the
> IP_MAXPACKET issue.
While we have most symptoms in common, I’ve still not seen any allocation error
in netstat -m. So I tend to agree that this is most probably a different
problem.
Markus
> I'll create a separate thread for that one shortly.
>
>
> On Mon, Mar 24, 2014 at 1:14 PM, Markus Gebert
> <markus.geb...@hostpoint.ch>wrote:
>
>>
>> On 24.03.2014, at 16:21, Christopher Forgeron <csforge...@gmail.com>
>> wrote:
>>
>>> This is regarding the TSO patch that Rick suggested earlier. (With many
>>> thanks for his time and suggestion)
>>>
>>> As I mentioned earlier, it did not fix the issue on a 10.0 system. It did
>>> make it less of a problem on 9.2, but either way, I think it's not
>> needed,
>>> and shouldn't be considered as a patch for testing/etc.
>>>
>>> Patching TSO to anything other than a max value (and by default the code
>>> gives it IP_MAXPACKET) is confusing the matter, as the packet length
>>> ultimately needs to be adjusted for many things on the fly like TCP
>>> Options, etc. Using static header sizes won't be a good idea.
>>>
>>> Additionally, it seems that setting nic TSO will/may be ignored by code
>>> like this in sys/netinet/tcp_output.c:
>>>
>>> 10.0 Code:
>>>
>>> 780 if (len > tp->t_tsomax - hdrlen)
>>> { !!
>>> 781 len = tp->t_tsomax -
>>> hdrlen; !!
>>> 782 sendalot =
>>> 1;
>>> 783 }
>>>
>>>
>>> I've put debugging here, set the nic's max TSO as per Rick's patch ( set
>> to
>>> say 32k), and have seen that tp->t_tsomax == IP_MAXPACKET. It's being set
>>> someplace else, and thus our attempts to set TSO on the nic may be in
>> vain.
>>>
>>> It may have mattered more in 9.2, as I see the code doesn't use
>>> tp->t_tsomax in some locations, and may actually default to what the nic
>> is
>>> set to.
>>>
>>> The NIC may still win, I didn't walk through the code to confirm, it was
>>> enough to suggest to me that setting TSO wouldn't fix this issue.
>>
>>
>> I just applied Rick's ixgbe TSO patch and additionally wanted to be able
>> to easily change the value of hw_tsomax, so I made a sysctl out of it.
>>
>> While doing that, I asked myself the same question. Where and how will
>> this value actually be used and how comes that tcp_output() uses that other
>> value in struct tcpcb.
>>
>> The only place tcpcb->t_tsomax gets set, that I have found so far, is in
>> tcp_input.c's tcp_mss() function. Some subfunctions get called:
>>
>> tcp_mss() -> tcp_mss_update() -> tcp_maxmtu()
>>
>> Then tcp_maxmtu() indeed uses the interface's hw_tsomax value:
>>
>> 1746 cap->tsomax = ifp->if_hw_tsomax;
>>
>> It get's passed back to tcp_mss() where it is set on the connection level
>> which will be used in tcp_output() later on.
>>
>> tcp_mss() gets called from multiple places, I'll look into that later. I
>> will let you know if I find out more.
>>
>>
>> Markus
>>
>>
>>> However, this is still a TSO related issue, it's just not one related to
>>> the setting of TSO's max size.
>>>
>>> A 10.0-STABLE system with tso disabled on ix0 doesn't have a single
>> packet
>>> over IP_MAXPACKET in 1 hour of runtime. I'll let it go a bit longer to
>>> increase confidence in this assertion, but I don't want to waste time on
>>> this when I could be logging problem packets on a system with TSO
>> enabled.
>>>
>>> Comments are very welcome..
>>> _______________________________________________
>>> freebsd-net@freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>>
>>
>>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
