On 29.1.2024. 15:29, Alexander Bluhm wrote:
> On Sat, Jan 27, 2024 at 08:08:35AM +0100, Hrvoje Popovski wrote:
>> On 26.1.2024. 22:47, Alexander Bluhm wrote:
>>> On Fri, Jan 26, 2024 at 11:41:49AM +0100, Hrvoje Popovski wrote:
>>>> I've manage to reproduce TSO em problem on anoter setup, unfortunatly
>>>> production.
>>> What helped debugging a similar issue with ixl(4) and TSO was to
>>> remove all TSO specific code from the driver. Then only this part
>>> remains from the original em(4) TSO diff.
>>>
>>> error = bus_dmamap_create(sc->sc_dmat, EM_TSO_SIZE,
>>> EM_MAX_SCATTER / (sc->pcix_82544 ? 2 : 1),
>>> EM_TSO_SEG_SIZE, 0, BUS_DMA_NOWAIT, &pkt->pkt_map);
>>>
>>> The parameters that changed when adding TSO are:
>>>
>>> bus_size_t size: MAX_JUMBO_FRAME_SIZE 16128 -> EM_TSO_SIZE 65535
>>> bus_size_t maxsegsz: MAX_JUMBO_FRAME_SIZE 16128 -> EM_TSO_SEG_SIZE
>>> 4096
>>>
>>> I suspect that this is the cause for the regression as disabling
>>> TSO did not help. Would it be possible to run the diff below? I
>>> expect that the problem will still be there. But then we know it
>>> must be the change of one of the bus_dmamap_create() arguments.
>>>
>>> bluhm
>>
>> Hi,
>>
>> with this diff em0 seems happy and em watchdog is gone.
>
> This is very interesting. That means that the bus_dmamap_create()
> argument does not cause the regression.
>
> Did you see anywhere "output TSO packets hardware processed in"
> netstat -s. In some iteration of testing you turned TSO off with
> sysctl net.inet.tcp.tso=0, but it did not help. So no TSO packets
> from the stack.
>
> In another mail you mentioned
>
>> Setup is very simple
>> em0 - carp <- uplink
>> em1 - pfsync
>> ix1 - vlans - carp
>
> ix supports LRO. If you forward from ix1 to em0 the LRO packets
> from ix hardware are split by TSO on em hardware. And the ix does
> vlan offloading + LRO, so em must do vlan offloading properly with
> TSO. Or do you use a vlan interface?
>
> Does it help to disable LRO, ifconfig ix1 -tcplro ?
Yes, it helps... Thank you
uplink
em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
hwfeatures=31b7<CSUM_IPv4,CSUM_TCPv4,CSUM_UDPv4,VLAN_MTU,VLAN_HWTAGGING,CSUM_TCPv6,CSUM_UDPv6,TSOv4,TSOv6>
hardmtu 9216
lladdr 0c:c4:7a:da:cd:5a
index 3 priority 0 llprio 3
groups: egress
media: Ethernet autoselect (1000baseT full-duplex,master,rxpause)
status: active
vlans are on ix1 - I've disabled LRO
ix1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
lladdr 90:e2:ba:d7:1b:f5
index 2 priority 0 llprio 3
media: Ethernet autoselect (10GbaseSR full-duplex,rxpause,txpause)
status: active
before I've disabled LRO on ix1 I've got lot of watchdog on em0
bcbnfw1# uptime
9:25AM up 8 mins, 1 user, load averages: 0.14, 0.13, 0.06
bcbnfw1# cat /var/log/messages| grep watchdog
Jan 30 09:18:51 bcbnfw1 /bsd: em0: watchdog: head 148 tail 213 TDH 213
TDT 148
Jan 30 09:19:01 bcbnfw1 /bsd: em0: watchdog: head 160 tail 224 TDH 224
TDT 160
Jan 30 09:19:12 bcbnfw1 /bsd: em0: watchdog: head 163 tail 228 TDH 228
TDT 163
Jan 30 09:19:22 bcbnfw1 /bsd: em0: watchdog: head 128 tail 192 TDH 192
TDT 128
Jan 30 09:19:32 bcbnfw1 /bsd: em0: watchdog: head 309 tail 373 TDH 373
TDT 309
Jan 30 09:19:41 bcbnfw1 /bsd: em0: watchdog: head 113 tail 177 TDH 177
TDT 113
Jan 30 09:19:51 bcbnfw1 /bsd: em0: watchdog: head 402 tail 466 TDH 466
TDT 402
Jan 30 09:20:01 bcbnfw1 /bsd: em0: watchdog: head 114 tail 178 TDH 178
TDT 114
Jan 30 09:20:16 bcbnfw1 /bsd: em0: watchdog: head 111 tail 175 TDH 175
TDT 111
Jan 30 09:20:26 bcbnfw1 /bsd: em0: watchdog: head 199 tail 263 TDH 263
TDT 199
without LRO on ix1 everything seems to work just fine ...
>
> I see this vlan code with mac_type checks. Can we end in a
> configuration where we enable TSO but cannot do VLAN offloading?
>
> #if NVLAN > 0
> /* Find out if we are in VLAN mode */
> if (m->m_flags & M_VLANTAG && (sc->hw.mac_type < em_82575 ||
> sc->hw.mac_type > em_i210)) {
> /* Set the VLAN id */
> desc->upper.fields.special = htole16(m->m_pkthdr.ether_vtag);
>
> /* Tell hardware to add tag */
> desc->lower.data |= htole32(E1000_TXD_CMD_VLE);
> }
> #endif
>
> Hrvoje, I know you do great tests in your lab. Did you try this
> setup:
>
> Send bulk TCP traffic in vlan that will trigger LRO.
> Do VLAN + LRO offloading in ix.
> Forward it to em with TSO.
>
I will prepare one box for this kind of traffic and will contact you and
marcus
> In theory when going through vlan interface it should remove
> M_VLANTAG. But something must be wrong and I wonder what.
>
> bluhm
>
