Re: Fw: [Bug 111771] New: deadlock in ppp/l2tp
On Thu, Feb 04, 2016 at 02:32:48AM +0100, Sorin Manolache wrote: > On 2016-02-03 18:14, Guillaume Nault wrote: > > > >Sorin, it seems like one of your L2TP tunnels is routed to one of its upper > >PPP > >devices. Most likely, the peer address of the PPP device is also the address > >of > >the remote L2TP tunnel endpoint. So L2TP packets are sent back to the upper > >PPP > >device, instead of leaving through the physical interface. > > Thank you. You are right. There's a host route to the peer over the ppp0 > interface in the routing table. I don't know how it gets there. I've checked > the source code of pppd and no such route is added for kernels newer than > 2.1.16. I've grepped /etc for "route" in order to detect a "post-up" script > that would add that route. Nothing. I've double-checked by executing strace > on xl2tpd and its children (i.e. pppd and the initialisation scripts) and I > couldn't find any ioctl SIOCADDRT. So it's a total mystery for me where that > route comes from. Could it come from the kernel? > If that's a /32 IPv4 route to the peer address of the PPP link and has the "proto kernel" attribute, then yes, that's most likely the one generated by the kernel.
Re: Fw: [Bug 111771] New: deadlock in ppp/l2tp
On 2016-02-03 18:14, Guillaume Nault wrote: On Wed, Feb 03, 2016 at 11:04:31AM +1100, Stephen Hemminger wrote: Please excuse URL mangling, my bugzilla address appears to route through stupid corporate firewall. Sorin, it seems like one of your L2TP tunnels is routed to one of its upper PPP devices. Most likely, the peer address of the PPP device is also the address of the remote L2TP tunnel endpoint. So L2TP packets are sent back to the upper PPP device, instead of leaving through the physical interface. Thank you. You are right. There's a host route to the peer over the ppp0 interface in the routing table. I don't know how it gets there. I've checked the source code of pppd and no such route is added for kernels newer than 2.1.16. I've grepped /etc for "route" in order to detect a "post-up" script that would add that route. Nothing. I've double-checked by executing strace on xl2tpd and its children (i.e. pppd and the initialisation scripts) and I couldn't find any ioctl SIOCADDRT. So it's a total mystery for me where that route comes from. Could it come from the kernel? I've found this 9-year-old bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=444180. I've adopted the strategy that the comment proposes: delete the route in an post-up script. Thanks again. Best regards, Sorin
Re: Fw: [Bug 111771] New: deadlock in ppp/l2tp
On Wed, Feb 03, 2016 at 11:04:31AM +1100, Stephen Hemminger wrote:
> Please excuse URL mangling, my bugzilla address appears to route through
> stupid corporate firewall.
>
> Begin forwarded message:
>
> Date: Tue, 2 Feb 2016 18:38:41 +
> From: "bugzilla-dae...@bugzilla.kernel.org"
>
> To: "shemmin...@linux-foundation.org"
> Subject: [Bug 111771] New: deadlock in ppp/l2tp
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_show-5Fbug.cgi-3Fid-3D111771&d=CwICaQ&c=IL_XqQWOjubgfqINi2jTzg&r=q_lvUiVm1uM6QEw9TPH-6jiV__hsrE6xXUAtATPE9x0&m=QRVzJYt9nD-EOW0XdrPpw2-kYmZu0sg62aaPeiiLI_Q&s=l3HC8fgAgyPVwSgMaX2Hjr8GL3P5j2fL1kDXhEW-v9w&e=
>
>
> Bug ID: 111771
>Summary: deadlock in ppp/l2tp
>Product: Networking
>Version: 2.5
> Kernel Version: 4.3.2
> Hardware: x86-64
> OS: Linux
> Tree: Mainline
> Status: NEW
> Severity: normal
> Priority: P1
> Component: Other
> Assignee: shemmin...@linux-foundation.org
> Reporter: sor...@gmail.com
> Regression: No
>
> Created attachment 202771
> -->
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_attachment.cgi-3Fid-3D202771-26action-3Dedit&d=CwICaQ&c=IL_XqQWOjubgfqINi2jTzg&r=q_lvUiVm1uM6QEw9TPH-6jiV__hsrE6xXUAtATPE9x0&m=QRVzJYt9nD-EOW0XdrPpw2-kYmZu0sg62aaPeiiLI_Q&s=2a_KCXgtoQ6NLxDon6_3flSUMpb7Tjj8WhGPZ09E8Vo&e=
>
> kernel.log, config, cpuinfo
>
> I'm getting a deadlock and the computer is unresponsive shortly after setting
> up a xl2tpd/ipsec vpn connection. The deadlock occurs every time, it is very
> reproducible on my computer.
>
> It is the mainline kernel, version 4.3.2.
>
Sorin, it seems like one of your L2TP tunnels is routed to one of its upper PPP
devices. Most likely, the peer address of the PPP device is also the address of
the remote L2TP tunnel endpoint. So L2TP packets are sent back to the upper PPP
device, instead of leaving through the physical interface.
For easier reference, here the trace extracted from bugzilla:
Feb 2 19:02:06 leo-naphta kernel: [ 329.888935]
==
Feb 2 19:02:06 leo-naphta kernel: [ 329.888939] [ INFO: possible circular
locking dependency detected ]
Feb 2 19:02:06 leo-naphta kernel: [ 329.888945] 4.3.2 #5 Not tainted
Feb 2 19:02:06 leo-naphta kernel: [ 329.888948]
---
Feb 2 19:02:06 leo-naphta kernel: [ 329.888952] pppd/4034 is trying to
acquire lock:
Feb 2 19:02:06 leo-naphta kernel: [ 329.888956] (_xmit_PPP#2){+.}, at:
[] sch_direct_xmit+0xf3/0x1f0
Feb 2 19:02:06 leo-naphta kernel: [ 329.888979]
Feb 2 19:02:06 leo-naphta kernel: [ 329.888979] but task is already holding
lock:
Feb 2 19:02:06 leo-naphta kernel: [ 329.888982] (l2tp_sock){+.-...}, at:
[] l2tp_xmit_skb+0x117/0x3d0 [l2tp_core]
Feb 2 19:02:06 leo-naphta kernel: [ 329.888997]
Feb 2 19:02:06 leo-naphta kernel: [ 329.888997] which lock already depends on
the new lock.
Feb 2 19:02:06 leo-naphta kernel: [ 329.888997]
Feb 2 19:02:06 leo-naphta kernel: [ 329.889002]
Feb 2 19:02:06 leo-naphta kernel: [ 329.889002] the existing dependency chain
(in reverse order) is:
Feb 2 19:02:06 leo-naphta kernel: [ 329.889006]
Feb 2 19:02:06 leo-naphta kernel: [ 329.889006] -> #3 (l2tp_sock){+.-...}:
Feb 2 19:02:06 leo-naphta kernel: [ 329.889015][]
lock_acquire+0x60/0x80
Feb 2 19:02:06 leo-naphta kernel: [ 329.889023][]
_raw_spin_lock+0x33/0x50
Feb 2 19:02:06 leo-naphta kernel: [ 329.889033][]
l2tp_xmit_skb+0x117/0x3d0 [l2tp_core]
Feb 2 19:02:06 leo-naphta kernel: [ 329.889041][]
pppol2tp_xmit+0x143/0x1f3 [l2tp_ppp]
Feb 2 19:02:06 leo-naphta kernel: [ 329.889049][]
ppp_channel_push+0x4b/0xc0 [ppp_generic]
Feb 2 19:02:06 leo-naphta kernel: [ 329.889057][]
ppp_write+0xc0/0xf0 [ppp_generic]
Feb 2 19:02:06 leo-naphta kernel: [ 329.889065][]
__vfs_write+0x23/0xe0
Feb 2 19:02:06 leo-naphta kernel: [ 329.889074][]
vfs_write+0x9d/0x180
Feb 2 19:02:06 leo-naphta kernel: [ 329.889080][]
SyS_write+0x44/0xa0
Feb 2 19:02:06 leo-naphta kernel: [ 329.889086][]
entry_SYSCALL_64_fastpath+0x16/0x6f
Feb 2 19:02:06 leo-naphta kernel: [ 329.889094]
Feb 2 19:02:06 leo-naphta kernel: [ 329.889094] -> #2
(&(&pch->downl)->rlock){+.}:
Feb 2 19:02:06 leo-naphta kernel: [ 329.889102][]
lock_acquire+0x60/0x80
Feb 2 19:02:06 leo-naphta kernel: [ 329.889107][]
_raw_spin_lock_bh+0x3a/0x50
Feb 2 19:02:06 leo-naphta kernel: [ 329.889114][]
ppp_push+0x10a/0x5b0 [ppp_generic]
Feb 2 19:02:06 leo-naphta kernel: [ 329.889133][]
ppp_xmit_process+0x3f5/0x5d0 [ppp_generic]
Feb 2 19:02:06 leo-naphta kernel: [ 329.889141][]
ppp_write+0xca/0xf0 [ppp_generic]
Feb 2 19:02:06 leo-naphta kernel: [
Fw: [Bug 111771] New: deadlock in ppp/l2tp
Please excuse URL mangling, my bugzilla address appears to route through stupid corporate firewall. Begin forwarded message: Date: Tue, 2 Feb 2016 18:38:41 + From: "bugzilla-dae...@bugzilla.kernel.org" To: "shemmin...@linux-foundation.org" Subject: [Bug 111771] New: deadlock in ppp/l2tp https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_show-5Fbug.cgi-3Fid-3D111771&d=CwICaQ&c=IL_XqQWOjubgfqINi2jTzg&r=q_lvUiVm1uM6QEw9TPH-6jiV__hsrE6xXUAtATPE9x0&m=QRVzJYt9nD-EOW0XdrPpw2-kYmZu0sg62aaPeiiLI_Q&s=l3HC8fgAgyPVwSgMaX2Hjr8GL3P5j2fL1kDXhEW-v9w&e= Bug ID: 111771 Summary: deadlock in ppp/l2tp Product: Networking Version: 2.5 Kernel Version: 4.3.2 Hardware: x86-64 OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: Other Assignee: shemmin...@linux-foundation.org Reporter: sor...@gmail.com Regression: No Created attachment 202771 --> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_attachment.cgi-3Fid-3D202771-26action-3Dedit&d=CwICaQ&c=IL_XqQWOjubgfqINi2jTzg&r=q_lvUiVm1uM6QEw9TPH-6jiV__hsrE6xXUAtATPE9x0&m=QRVzJYt9nD-EOW0XdrPpw2-kYmZu0sg62aaPeiiLI_Q&s=2a_KCXgtoQ6NLxDon6_3flSUMpb7Tjj8WhGPZ09E8Vo&e= kernel.log, config, cpuinfo I'm getting a deadlock and the computer is unresponsive shortly after setting up a xl2tpd/ipsec vpn connection. The deadlock occurs every time, it is very reproducible on my computer. It is the mainline kernel, version 4.3.2. The processor is given in a copy /proc/cpuinfo that I've attached. uname: Linux version 4.3.2 (gcc version 5.3.1 20160121 (Debian 5.3.1-7) ) #5 SMP PREEMPT Sat Jan 30 00:05:40 CET 2016 ppp is from the debian package of their unstable distribution: ppp_2.4.7-1+2_amd64 I've attached the kernel config file and the kernel log. -- You are receiving this mail because: You are the assignee for the bug.
