RE: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
> > When calling drv_tx() the headroom is not big enough for the driver. > > Ok. > > > > > > > Maybe we're adding something else to this skb? > > > > > > I can't find anything in the ath9k_htc driver that's adding more > > > than > > > 23 bytes (it's advertising 24) but clearly the last 8 bytes here are > > > failing: > > > > > > > [ 83.200346] skbuff: skb_under_panic: text:a034c028 > > > > len:154 > > > > put:8 head:880213422e00 data:880213422dfa tail:0x94 > > > > end:0xc0 > > > > dev: > > > > > > Maybe mac80211 is putting something else? It'd have to be > > > > yes mac80211 is adding the security header. > > headroom asked to skb_copy_expand should also take sdata- > > >encrypt_headroom into account. > > I suspected that, since it was about the only place that was adding anything, > but I couldn't test it :) > > Can you send a patch? Sure, I will send a patch. cedric
RE: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
> > Here is the stack trace I get: > > I added a trace before calling skb_copy_expand to get the headroom of > > the buffer before the copy and the headroom asked by the driver. > > > > [ 83.200261] MESH fwd: skb_headroom=154, needed headroom=24 > > Could you also add a similar trace just before calling drv_tx()? When calling drv_tx() the headroom is not big enough for the driver. > > Maybe we're adding something else to this skb? > > I can't find anything in the ath9k_htc driver that's adding more than > 23 bytes (it's advertising 24) but clearly the last 8 bytes here are > failing: > > > [ 83.200346] skbuff: skb_under_panic: text:a034c028 len:154 > > put:8 head:880213422e00 data:880213422dfa tail:0x94 end:0xc0 > > dev: > > Maybe mac80211 is putting something else? It'd have to be yes mac80211 is adding the security header. headroom asked to skb_copy_expand should also take sdata->encrypt_headroom into account. cedric
Re: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
> When calling drv_tx() the headroom is not big enough for the driver. Ok. > > > > Maybe we're adding something else to this skb? > > > > I can't find anything in the ath9k_htc driver that's adding more > > than > > 23 bytes (it's advertising 24) but clearly the last 8 bytes here > > are > > failing: > > > > > [ 83.200346] skbuff: skb_under_panic: text:a034c028 > > > len:154 > > > put:8 head:880213422e00 data:880213422dfa tail:0x94 > > > end:0xc0 > > > dev: > > > > Maybe mac80211 is putting something else? It'd have to be > > yes mac80211 is adding the security header. > headroom asked to skb_copy_expand should also take sdata- > >encrypt_headroom into account. I suspected that, since it was about the only place that was adding anything, but I couldn't test it :) Can you send a patch? johannes
Re: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
> I made a quick test with dongle using ath9k_htc driver and I indeed > reproduce the issue. Thanks. > Here is the stack trace I get: > I added a trace before calling skb_copy_expand to get the headroom of > the buffer before the copy and the headroom asked by the driver. > > [ 83.200261] MESH fwd: skb_headroom=154, needed headroom=24 Could you also add a similar trace just before calling drv_tx()? Maybe we're adding something else to this skb? I can't find anything in the ath9k_htc driver that's adding more than 23 bytes (it's advertising 24) but clearly the last 8 bytes here are failing: > [ 83.200346] skbuff: skb_under_panic: text:a034c028 len:154 > put:8 head:880213422e00 data:880213422dfa tail:0x94 end:0xc0 > dev: Maybe mac80211 is putting something else? It'd have to be johannes
RE: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
> On 2017年01月11日 20:01, Johannes Berg wrote: > > Sure, ssh won't - I was thinking of netconsole: > > https://www.kernel.org/doc/Documentation/networking/netconsole.txt > > Oh, I see. Thanks, I will try. > > Masashi Honma. Hi, I made a quick test with dongle using ath9k_htc driver and I indeed reproduce the issue. Here is the stack trace I get: I added a trace before calling skb_copy_expand to get the headroom of the buffer before the copy and the headroom asked by the driver. [ 83.200261] MESH fwd: skb_headroom=154, needed headroom=24 [ 83.200346] skbuff: skb_under_panic: text:a034c028 len:154 put:8 head:880213422e00 data:880213422dfa tail:0x94 end:0xc0 dev: [ 83.200359] [ cut here ] [ 83.200362] kernel BUG at ../net/core/skbuff.c:105! [ 83.200364] invalid opcode: [#1] SMP [ 83.200366] Modules linked in: ath9k_htc ath9k_common ath9k_hw ath10k_pci ath10k_core mac80211 ath cfg80211 x86_pkg_temp_thermal [ 83.200377] CPU: 4 PID: 29 Comm: ksoftirqd/4 Not tainted 4.9.0+ #3 [ 83.200379] Hardware name: Dell Inc. OptiPlex 990/06D7TR, BIOS A19 08/26/2015 [ 83.200381] task: 880223e55cc0 task.stack: c9db8000 [ 83.200383] RIP: 0010:[] [] skb_panic+0x5c/0x60 [ 83.200391] RSP: 0018:c9dbbba0 EFLAGS: 00010286 [ 83.200393] RAX: 0086 RBX: 0006 RCX: [ 83.200398] RDX: 8802253126d8 RSI: 88022530cb28 RDI: 88022530cb28 [ 83.200400] RBP: c9dbbbc0 R08: 00030e9a R09: 0005 [ 83.200401] R10: R11: 02c8 R12: 880222d0a000 [ 83.200403] R13: 9200 R14: 880221ba7f00 R15: 0010 [ 83.200406] FS: () GS:88022530() knlGS: [ 83.200408] CS: 0010 DS: ES: CR0: 80050033 [ 83.200410] CR2: 7fd414010c98 CR3: 00022247d000 CR4: 000406e0 [ 83.200411] Stack: [ 83.200413] 880213422dfa 0094 00c0 81c308d9 [ 83.200417] c9dbbbd0 817019b7 c9dbbc00 a034c028 [ 83.200420] 880221ba7f00 8802226d15a0 c9dbbcb8 [ 83.200423] Call Trace: [ 83.200429] [] skb_push+0x37/0x40 [ 83.200435] [] htc_issue_send.constprop.2+0x28/0x60 [ath9k_htc] [ 83.200441] [] htc_send+0x11/0x20 [ath9k_htc] [ 83.200445] [] ath9k_htc_tx_start+0xd7/0x2a0 [ath9k_htc] [ 83.200450] [] ath9k_htc_tx+0xa8/0xd0 [ath9k_htc] [ 83.200471] [] ieee80211_tx_frags+0x137/0x1f0 [mac80211] [ 83.200489] [] __ieee80211_tx+0x7c/0x180 [mac80211] [ 83.200506] [] ieee80211_tx+0xe5/0x110 [mac80211] [ 83.200518] [] ieee80211_tx_pending+0x8f/0x1f0 [mac80211] [ 83.200522] [] ? pick_next_task_fair+0x406/0x470 [ 83.200528] [] tasklet_action+0xda/0xf0 [ 83.200532] [] __do_softirq+0xe2/0x270 [ 83.200536] [] run_ksoftirqd+0x17/0x30 [ 83.200540] [] smpboot_thread_fn+0x105/0x160 [ 83.200543] [] ? sort_range+0x20/0x20 [ 83.200547] [] kthread+0xc5/0xe0 [ 83.200550] [] ? kthread_park+0x60/0x60 [ 83.200554] [] ret_from_fork+0x22/0x30 [ 83.200556] Code: c4 00 00 00 48 89 44 24 10 8b 87 c0 00 00 00 48 89 44 24 08 48 8b 87 d0 00 00 00 48 c7 c7 28 57 c3 81 48 89 04 24 e8 15 78 a2 ff <0f> 0b 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 53 65 4c 8b 34 [ 83.200596] RIP [] skb_panic+0x5c/0x60 [ 83.200602] RSP cedric
Re: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
On 2017年01月11日 20:01, Johannes Berg wrote: Sure, ssh won't - I was thinking of netconsole: https://www.kernel.org/doc/Documentation/networking/netconsole.txt Oh, I see. Thanks, I will try. Masashi Honma.
Re: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
On Wed, 2017-01-11 at 19:36 +0900, Masashi Honma wrote: > On 2017年01月11日 19:00, Johannes Berg wrote: > > Nevertheless, I don't have hardware to try to reproduce it, and I > > can't > > see any such issues (even with real forwarding, I even just wrote a > > wpa_s test for that) in hwsim. > > > > Even a photo of the crash on the VT would help. Or maybe you can > > set up > > netconsole on the wired interface? > > Thanks but SSH console via wired interface and laptop display does > not show any log... Sure, ssh won't - I was thinking of netconsole: https://www.kernel.org/doc/Documentation/networking/netconsole.txt johannes
Re: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
On 2017年01月11日 19:00, Johannes Berg wrote: Nevertheless, I don't have hardware to try to reproduce it, and I can't see any such issues (even with real forwarding, I even just wrote a wpa_s test for that) in hwsim. Even a photo of the crash on the VT would help. Or maybe you can set up netconsole on the wired interface? Thanks but SSH console via wired interface and laptop display does not show any log... Masashi Honma.
Re: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
> I will call the mesh peers "STA A" and "STA B". > > Both STA has one physical wireless I/F and wired I/F. Ok. [snip configuration] > Then STA A or STA B crashes, not both. Nevertheless, I don't have hardware to try to reproduce it, and I can't see any such issues (even with real forwarding, I even just wrote a wpa_s test for that) in hwsim. Even a photo of the crash on the VT would help. Or maybe you can set up netconsole on the wired interface? johannes
Re: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
On 2017年01月11日 18:00, Johannes Berg wrote: Ok, that's strange, but maybe there's a reason. Can you extract *any* information whatsoever? Like maybe if you switch to a VT console before running into the crash? I don't have any hardware to run this on, and hwsim doesn't have any issues. I will call the mesh peers "STA A" and "STA B". Both STA has one physical wireless I/F and wired I/F. I have connected to both with SSH via wired I/F and started wpa_supplicant with this command for both. sudo ./hostap/wpa_supplicant/wpa_supplicant -i -D nl80211 -c mesh_sae.conf STA A's mesh_sae.conf is this. -- ctrl_interface=/var/run/wpa_supplicant ap_scan=1 user_mpm=1 update_config=0 network={ ssid="mesh0" key_mgmt=SAE mode=5 frequency=2412 psk="01234567" } -- STA B's mesh_sae.conf is this. The difference is "no_auto_peer=1". -- ctrl_interface=/var/run/wpa_supplicant ap_scan=1 user_mpm=1 update_config=0 network={ ssid="mesh0" key_mgmt=SAE mode=5 frequency=2412 psk="01234567" no_auto_peer=1 } -- Booting the wpa_supplicant finishes successfully. After the successfull peering process, I could see MESH-PEER-CONNECTED message on both side. Then STA A or STA B crashes, not both. Masashi Honma.
Re: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
On Wed, 2017-01-11 at 17:50 +0900, Masashi Honma wrote: > On 2017年01月11日 17:02, Johannes Berg wrote: > > I don't think this makes sense - if you only have two peers then > > you > > shouldn't even run into forwarding code paths? > > > > johannes > > Though it looks odd, the code has run into forwarding code path even > though peer to peer mesh connection. > > fwd_skb = skb_copy(skb, GFP_ATOMIC); > > I checked it with printk(). > > # I know printk() should not be used in the context, just for > checking. Ok, that's strange, but maybe there's a reason. Can you extract *any* information whatsoever? Like maybe if you switch to a VT console before running into the crash? I don't have any hardware to run this on, and hwsim doesn't have any issues. johannes
Re: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
On 2017年01月11日 17:02, Johannes Berg wrote: I don't think this makes sense - if you only have two peers then you shouldn't even run into forwarding code paths? johannes Though it looks odd, the code has run into forwarding code path even though peer to peer mesh connection. fwd_skb = skb_copy(skb, GFP_ATOMIC); I checked it with printk(). # I know printk() should not be used in the context, just for checking. Masashi Honma.
Re: [REGRESSION, bisect] mesh: SAE connection causes kernel crash
On Wed, 2017-01-11 at 08:35 +0900, Masashi Honma wrote: > I have encountered kernel crash when I have used mesh SAE > connection > with ath9k_htc device (Sony UWA-BR100). I have tried to connect 2 > peers > to each other, then only one peer crashes. > > By bisect, this commit looks causes this issue. > > commit d8da0b5d64d58f7775a94bcf12dda50f13a76f22 > Author: Cedric Izoard > Date: Wed Dec 7 09:59:00 2016 + > > mac80211: Ensure enough headroom when forwarding mesh pkt I don't think this makes sense - if you only have two peers then you shouldn't even run into forwarding code paths? johannes
[REGRESSION, bisect] mesh: SAE connection causes kernel crash
I have encountered kernel crash when I have used mesh SAE connection with ath9k_htc device (Sony UWA-BR100). I have tried to connect 2 peers to each other, then only one peer crashes. By bisect, this commit looks causes this issue. commit d8da0b5d64d58f7775a94bcf12dda50f13a76f22 Author: Cedric Izoard Date: Wed Dec 7 09:59:00 2016 + mac80211: Ensure enough headroom when forwarding mesh pkt When a buffer is duplicated during MESH packet forwarding, this patch ensures that the new buffer has enough headroom. I do not have any crash log because the computer was fully uncontrollable. Regards, Masashi Honma.