Re: [PATCH net] ipv4: only create late gso-skb if skb is already set up with CHECKSUM_PARTIAL
Hannes Frederic Sowa wrote: > On 26.02.2016 01:45, Wakko Warner wrote: > >Now there's another one: > >[ 777.315931] [ cut here ] > >[ 777.316099] WARNING: CPU: 0 PID: 1404 at > >/usr/src/linux/dist/4.4-nobklcd/net/ipv4/af_inet.c:155 > >inet_sock_destruct+0x1cb/0x1f0() > >[ 777.316189] Modules linked in: nfsv3 af_packet scsi_transport_iscsi nfsd > >auth_rpcgss oid_registry exportfs nfs lockd grace sunrpc ipv6 ata_piix > >libata evdev virtio_balloon virtio_net unix > >[ 777.316416] CPU: 0 PID: 1404 Comm: kworker/0:1H Not tainted 4.4.0 #2 > >[ 777.316468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > >Debian-1.8.2-1 04/01/2014 > >[ 777.316547] Workqueue: rpciod xprt_autoclose [sunrpc] > >[ 777.316598] 815053f0 811d46ee > >81041383 > >[ 777.316680] 88003d268b40 88003d268cc0 88003d07b3f8 > >88003d07b368 > >[ 777.316763] 8136f53b 88003d268b40 > >8800160f2d40 > >[ 777.316845] Call Trace: > >[ 777.316868] [] ? dump_stack+0x47/0x69 > >[ 777.316911] [] ? warn_slowpath_common+0x73/0xa0 > >[ 777.316963] [] ? inet_sock_destruct+0x1cb/0x1f0 > >[ 777.317018] [] ? sk_destruct+0x13/0xc0 > >[ 777.317061] [] ? inet_release+0x31/0x50 > >[ 777.317108] [] ? sock_release+0x15/0x70 > >[ 777.317153] [] ? xs_close+0x9/0x20 [sunrpc] > >[ 777.317206] [] ? xprt_autoclose+0x2d/0x60 [sunrpc] > >[ 777.317261] [] ? process_one_work+0x129/0x3f0 > >[ 777.317313] [] ? worker_thread+0x42/0x490 > >[ 777.317367] [] ? process_one_work+0x3f0/0x3f0 > >[ 777.317421] [] ? kthread+0xb8/0xd0 > >[ 777.317465] [] ? kthread_worker_fn+0x100/0x100 > >[ 777.317521] [] ? ret_from_fork+0x3f/0x70 > >[ 777.317564] [] ? kthread_worker_fn+0x100/0x100 > >[ 777.317618] ---[ end trace 220e17a0bf3ec971 ]--- > > > >This one happened on the client side VM. There was only 1 NFS mount. The > >server VM didn't show anything nor did the host. > > Can you send me your specific kernel version? There are multiple > WARN_ONs and I want to catch the right one. Same version as before. 4.4.1. -- Microsoft has beaten Volkswagen's world record. Volkswagen only created 22 million bugs.
Re: [PATCH net] ipv4: only create late gso-skb if skb is already set up with CHECKSUM_PARTIAL
Wakko Warner wrote:
> Hannes Frederic Sowa wrote:
> > Otherwise we break the contract with GSO to only pass CHECKSUM_PARTIAL
> > skbs down. This can easily happen with UDP+IPv4 sockets with the first
> > MSG_MORE write smaller than the MTU, second write is a sendfile.
> >
> > Returning -EOPNOTSUPP lets the callers fall back into normal sendmsg path,
> > were we calculate the checksum manually during copying.
> >
> > Commit d749c9cbffd6 ("ipv4: no CHECKSUM_PARTIAL on MSG_MORE corked
> > sockets") started to exposes this bug.
> >
> > Fixes: d749c9cbffd6 ("ipv4: no CHECKSUM_PARTIAL on MSG_MORE corked sockets")
> > Reported-by: Jiri Benc
> > Cc: Jiri Benc
> > Reported-by: Wakko Warner
> > Cc: Wakko Warner
> > Signed-off-by: Hannes Frederic Sowa
>
> I just tested this with 2 of my VMs. It appears to have fixed the issue.
Now there's another one:
[ 777.315931] [ cut here ]
[ 777.316099] WARNING: CPU: 0 PID: 1404 at
/usr/src/linux/dist/4.4-nobklcd/net/ipv4/af_inet.c:155
inet_sock_destruct+0x1cb/0x1f0()
[ 777.316189] Modules linked in: nfsv3 af_packet scsi_transport_iscsi nfsd
auth_rpcgss oid_registry exportfs nfs lockd grace sunrpc ipv6 ata_piix libata
evdev virtio_balloon virtio_net unix
[ 777.316416] CPU: 0 PID: 1404 Comm: kworker/0:1H Not tainted 4.4.0 #2
[ 777.316468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
Debian-1.8.2-1 04/01/2014
[ 777.316547] Workqueue: rpciod xprt_autoclose [sunrpc]
[ 777.316598] 815053f0 811d46ee
81041383
[ 777.316680] 88003d268b40 88003d268cc0 88003d07b3f8
88003d07b368
[ 777.316763] 8136f53b 88003d268b40
8800160f2d40
[ 777.316845] Call Trace:
[ 777.316868] [] ? dump_stack+0x47/0x69
[ 777.316911] [] ? warn_slowpath_common+0x73/0xa0
[ 777.316963] [] ? inet_sock_destruct+0x1cb/0x1f0
[ 777.317018] [] ? sk_destruct+0x13/0xc0
[ 777.317061] [] ? inet_release+0x31/0x50
[ 777.317108] [] ? sock_release+0x15/0x70
[ 777.317153] [] ? xs_close+0x9/0x20 [sunrpc]
[ 777.317206] [] ? xprt_autoclose+0x2d/0x60 [sunrpc]
[ 777.317261] [] ? process_one_work+0x129/0x3f0
[ 777.317313] [] ? worker_thread+0x42/0x490
[ 777.317367] [] ? process_one_work+0x3f0/0x3f0
[ 777.317421] [] ? kthread+0xb8/0xd0
[ 777.317465] [] ? kthread_worker_fn+0x100/0x100
[ 777.317521] [] ? ret_from_fork+0x3f/0x70
[ 777.317564] [] ? kthread_worker_fn+0x100/0x100
[ 777.317618] ---[ end trace 220e17a0bf3ec971 ]---
This one happened on the client side VM. There was only 1 NFS mount. The
server VM didn't show anything nor did the host.
> > ---
> > net/ipv4/ip_output.c | 5 -
> > 1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
> > index 64878efa045c13..565bf64b2b7d60 100644
> > --- a/net/ipv4/ip_output.c
> > +++ b/net/ipv4/ip_output.c
> > @@ -1236,13 +1236,16 @@ ssize_t ip_append_page(struct sock *sk, struct
> > flowi4 *fl4, struct page *page,
> > if (!skb)
> > return -EINVAL;
> >
> > - cork->length += size;
> > if ((size + skb->len > mtu) &&
> > (sk->sk_protocol == IPPROTO_UDP) &&
> > (rt->dst.dev->features & NETIF_F_UFO)) {
> > + if (skb->ip_summed != CHECKSUM_PARTIAL)
> > + return -EOPNOTSUPP;
> > +
> > skb_shinfo(skb)->gso_size = mtu - fragheaderlen;
> > skb_shinfo(skb)->gso_type = SKB_GSO_UDP;
> > }
> > + cork->length += size;
> >
> > while (size > 0) {
> > if (skb_is_gso(skb)) {
> > --
> > 2.5.0
> >
> --
> Microsoft has beaten Volkswagen's world record. Volkswagen only created 22
> million bugs.
--
Microsoft has beaten Volkswagen's world record. Volkswagen only created 22
million bugs.
Re: [PATCH net] ipv4: only create late gso-skb if skb is already set up with CHECKSUM_PARTIAL
Hannes Frederic Sowa wrote:
> Otherwise we break the contract with GSO to only pass CHECKSUM_PARTIAL
> skbs down. This can easily happen with UDP+IPv4 sockets with the first
> MSG_MORE write smaller than the MTU, second write is a sendfile.
>
> Returning -EOPNOTSUPP lets the callers fall back into normal sendmsg path,
> were we calculate the checksum manually during copying.
>
> Commit d749c9cbffd6 ("ipv4: no CHECKSUM_PARTIAL on MSG_MORE corked
> sockets") started to exposes this bug.
>
> Fixes: d749c9cbffd6 ("ipv4: no CHECKSUM_PARTIAL on MSG_MORE corked sockets")
> Reported-by: Jiri Benc
> Cc: Jiri Benc
> Reported-by: Wakko Warner
> Cc: Wakko Warner
> Signed-off-by: Hannes Frederic Sowa
I just tested this with 2 of my VMs. It appears to have fixed the issue.
> ---
> net/ipv4/ip_output.c | 5 -
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
> index 64878efa045c13..565bf64b2b7d60 100644
> --- a/net/ipv4/ip_output.c
> +++ b/net/ipv4/ip_output.c
> @@ -1236,13 +1236,16 @@ ssize_t ip_append_page(struct sock *sk, struct
> flowi4 *fl4, struct page *page,
> if (!skb)
> return -EINVAL;
>
> - cork->length += size;
> if ((size + skb->len > mtu) &&
> (sk->sk_protocol == IPPROTO_UDP) &&
> (rt->dst.dev->features & NETIF_F_UFO)) {
> + if (skb->ip_summed != CHECKSUM_PARTIAL)
> + return -EOPNOTSUPP;
> +
> skb_shinfo(skb)->gso_size = mtu - fragheaderlen;
> skb_shinfo(skb)->gso_type = SKB_GSO_UDP;
> }
> + cork->length += size;
>
> while (size > 0) {
> if (skb_is_gso(skb)) {
> --
> 2.5.0
>
--
Microsoft has beaten Volkswagen's world record. Volkswagen only created 22
million bugs.
Re: 4.4.1 skb_warn_bad_offload+0xc5/0x110
Please keep me in CC. Wakko Warner wrote: > > Hannes Frederic Sowa wrote: > > [full-quote for netdev] > > > > Hello, > > > > On 16.02.2016 01:08, Wakko Warner wrote: > > >I've been seeing the following on some of my VMs ran under qemu. The VMs > > >do > > >not have internet connectivity. This happened when some files were > > >accessed > > >via NFS to another VM (NOTE: Both VMs throw these warnings. Both VMs are > > >running the exact same kernel). The host is also throwing these warnings > > >and is also 4.4.1, but not the same kernel build. > > > > > >The issue appears to have gone away if I issue the following on the guests > > >and on the host (except br0 instead of eth0 on host) > > >ethtool -K eth0 gso off gro off ufo off tso off > > > > > >On the host, br0 does not have any interfaces enslaved except for the > > >interface for the VMs and also does not have an IPv4 address assigned. > > > > > > Can you try the following patch? > > > > --- a/net/ipv4/ip_output.c > > +++ b/net/ipv4/ip_output.c > > @@ -1233,6 +1233,9 @@ ssize_t ip_append_page(struct sock *sk, > > struct flowi4 *fl4, struct page *page, > > if (!skb) > > return -EINVAL; > > > > + if (skb->ip_summed != CHECKSUM_PARTIAL) > > + return -EINVAL; > > + > > cork->length += size; > > if ((size + skb->len > mtu) && > > (sk->sk_protocol == IPPROTO_UDP) && > > Still received the warning. I saw it on the host and on one of the VMs. > The VM in this case was an nfs server. The client did not receive the > warning. I should mention that I'm using v3 and udp on the client for the > mount options. > > I'm not sure if this change effected nfsd on one of the VMs, it isn't working > at all on it. I reverted back to the previous kernel for the VM only and the nfsd on the one VM started working. I should note that when I added this change, I only compiled the kernel and copied the bzImage file. I did not recompile any modules nor copy the system.map over. -- Microsoft has beaten Volkswagen's world record. Volkswagen only created 22 million bugs.
Re: 4.4.1 skb_warn_bad_offload+0xc5/0x110
Please keep me in CC. Hannes Frederic Sowa wrote: > [full-quote for netdev] > > Hello, > > On 16.02.2016 01:08, Wakko Warner wrote: > >I've been seeing the following on some of my VMs ran under qemu. The VMs do > >not have internet connectivity. This happened when some files were accessed > >via NFS to another VM (NOTE: Both VMs throw these warnings. Both VMs are > >running the exact same kernel). The host is also throwing these warnings > >and is also 4.4.1, but not the same kernel build. > > > >The issue appears to have gone away if I issue the following on the guests > >and on the host (except br0 instead of eth0 on host) > >ethtool -K eth0 gso off gro off ufo off tso off > > > >On the host, br0 does not have any interfaces enslaved except for the > >interface for the VMs and also does not have an IPv4 address assigned. > > > Can you try the following patch? > > --- a/net/ipv4/ip_output.c > +++ b/net/ipv4/ip_output.c > @@ -1233,6 +1233,9 @@ ssize_t ip_append_page(struct sock *sk, > struct flowi4 *fl4, struct page *page, > if (!skb) > return -EINVAL; > > + if (skb->ip_summed != CHECKSUM_PARTIAL) > + return -EINVAL; > + > cork->length += size; > if ((size + skb->len > mtu) && > (sk->sk_protocol == IPPROTO_UDP) && Still received the warning. I saw it on the host and on one of the VMs. The VM in this case was an nfs server. The client did not receive the warning. I should mention that I'm using v3 and udp on the client for the mount options. I'm not sure if this change effected nfsd on one of the VMs, it isn't working at all on it. -- Microsoft has beaten Volkswagen's world record. Volkswagen only created 22 million bugs.
Re: 4.4.1 skb_warn_bad_offload+0xc5/0x110
Please keep me in CC. Hannes Frederic Sowa wrote: > [full-quote for netdev] > On 16.02.2016 01:08, Wakko Warner wrote: > >I've been seeing the following on some of my VMs ran under qemu. The VMs do > >not have internet connectivity. This happened when some files were accessed > >via NFS to another VM (NOTE: Both VMs throw these warnings. Both VMs are > >running the exact same kernel). The host is also throwing these warnings > >and is also 4.4.1, but not the same kernel build. > > > >The issue appears to have gone away if I issue the following on the guests > >and on the host (except br0 instead of eth0 on host) > >ethtool -K eth0 gso off gro off ufo off tso off > > > >On the host, br0 does not have any interfaces enslaved except for the > >interface for the VMs and also does not have an IPv4 address assigned. > > > >[ 90.067519] [ cut here ] > >[ 90.067678] WARNING: CPU: 0 PID: 2258 at > >/usr/src/linux/dist/4.4.1-nobklcd/net/core/dev.c:2422 > >skb_warn_bad_offload+0xc5/0x110() > >[ 90.067766] virtio_net: caps=(0x0804001f4a29, 0x) > >len=32934 data_len=32768 gso_size=1480 gso_type=2 ip_summed=0 > >[ 90.067878] Modules linked in: nfsv3 nfsd auth_rpcgss oid_registry > >exportfs nfs lockd grace sunrpc ipv6 virtio_net virtio_balloon evdev unix > >[ 90.068206] CPU: 0 PID: 2258 Comm: kworker/0:1H Not tainted 4.4.1 #1 > >[ 90.068258] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS > >Debian-1.8.2-1 04/01/2014 > >[ 90.068340] Workqueue: rpciod rpc_async_schedule [sunrpc] > >[ 90.068433] 81503288 811d455e 880276ffb9a0 > >81041343 > >[ 90.068575] 88007b85be00 880276ffb9f0 0002 > >880276f87aac > >[ 90.068725] 88027692c000 810413b7 81503498 > >0030 > >[ 90.068846] Call Trace: > >[ 90.06] [] ? dump_stack+0x47/0x69 > >[ 90.068967] [] ? warn_slowpath_common+0x73/0xa0 > >[ 90.069068] [] ? warn_slowpath_fmt+0x47/0x50 > >[ 90.069129] [] ? skb_warn_bad_offload+0xc5/0x110 > >[ 90.069191] [] ? __skb_gso_segment+0x71/0xc0 > >[ 90.069250] [] ? > >validate_xmit_skb.isra.119.part.120+0x100/0x290 > >[ 90.069314] [] ? lock_timer_base.isra.22+0x49/0x60 > >[ 90.069381] [] ? validate_xmit_skb_list+0x31/0x50 > >[ 90.069440] [] ? sch_direct_xmit+0x140/0x1e0 > >[ 90.069497] [] ? __dev_queue_xmit+0x1c8/0x490 > >[ 90.069555] [] ? ip_finish_output2+0x122/0x300 > >[ 90.069613] [] ? release_sock+0xfd/0x160 > >[ 90.069671] [] ? ip_output+0xb5/0xc0 > >[ 90.069720] [] ? ip_reply_glue_bits+0x50/0x50 > >[ 90.069784] [] ? prandom_u32+0x1b/0x30 > >[ 90.069833] [] ? ip_local_out+0x12/0x40 > >[ 90.069877] [] ? ip_send_skb+0x10/0x40 > >[ 90.069922] [] ? udp_send_skb+0x160/0x240 > >[ 90.069990] [] ? udp_push_pending_frames+0x34/0x50 > >[ 90.070050] [] ? udp_sendpage+0xe4/0x150 > >[ 90.070095] [] ? kernel_sendmsg+0x2a/0x40 > >[ 90.070164] [] ? xs_send_kvec+0x83/0x90 [sunrpc] > >[ 90.070223] [] ? inet_sendpage+0x93/0xe0 > >[ 90.070270] [] ? xs_sendpages+0x16f/0x1b0 [sunrpc] > >[ 90.070330] [] ? xs_udp_send_request+0x5e/0x100 > >[sunrpc] > >[ 90.070390] [] ? xprt_transmit+0x47/0x230 [sunrpc] > >[ 90.070449] [] ? call_transmit+0x175/0x220 [sunrpc] > >[ 90.070508] [] ? __rpc_execute+0x4b/0x290 [sunrpc] > >[ 90.070575] [] ? finish_task_switch+0x83/0x1b0 > >[ 90.070653] [] ? process_one_work+0x129/0x3f0 > >[ 90.070711] [] ? worker_thread+0x42/0x490 > >[ 90.070764] [] ? process_one_work+0x3f0/0x3f0 > >[ 90.070816] [] ? kthread+0xb8/0xd0 > >[ 90.070860] [] ? kthread_worker_fn+0x100/0x100 > >[ 90.070925] [] ? ret_from_fork+0x3f/0x70 > >[ 90.070974] [] ? kthread_worker_fn+0x100/0x100 > >[ 90.071035] ---[ end trace ffb4f8c2d24c1959 ]--- > > > > Can you try the following patch? I'll try it tomorrow. I had some disk failures on this system and am in the process of restoring it. > --- a/net/ipv4/ip_output.c > +++ b/net/ipv4/ip_output.c > @@ -1233,6 +1233,9 @@ ssize_t ip_append_page(struct sock *sk, > struct flowi4 *fl4, struct page *page, > if (!skb) > return -EINVAL; > > + if (skb->ip_summed != CHECKSUM_PARTIAL) > + return -EINVAL; > + > cork->length += size; > if ((size + skb->len > mtu) && > (sk->sk_protocol == IPPROTO_UDP) && -- Microsoft has beaten Volkswagen's world record. Volkswagen only created 22 million bugs.
