Re: [Bugme-new] [Bug 8132] New: pptp server lockup in ppp_asynctty_receive()
On Fri, Mar 09, 2007 at 11:40:04AM +0300, Yuriy N. Shkandybin wrote: ... > .config is at > http://bugzilla.kernel.org/attachment.cgi?id=10660&action=view > Also all information i've provied was recieved by serial console and it's > not hand writing. > > I've checked logs and right before lockup there is oops in syslog > Mar 5 21:50:44 vpn2 skb_under_panic: text:c02248a2 len:207 put:1 > head:db96e22c data:db96e22b tail:db96e2fa end:db96e82c dev: This looks like a real problem with skb and maybe with dev->hard_header_len. I see you are using vlan module, so maybe there is some interaction? I don't know ppp enough, so I CC this message to the ppp list. I'm not sure HZ change will cure this forever (maybe some packets are going to the wrong dev?). If you're willing to experiment, you can try to edit "include/linux/ppp_defs.h" and change it like this: #define PPP_HDRLEN 8 #define PPP_MRU 1496 and "include/linux/if_ppp.h": #define PPP_MTU 1496 plus mru/mtu in your pppd config (and recompile). But I hope ppp people will propose something better. Cheers, Jarek P. > Mar 5 21:50:44 vpn2 [ cut here ] > Mar 5 21:50:44 vpn2 kernel BUG at net/core/skbuff.c:112! > Mar 5 21:50:44 vpn2 invalid opcode: [#1] > Mar 5 21:50:44 vpn2 SMP > Mar 5 21:50:44 vpn2 Modules linked in: 8021q ipt_TCPMSS xt_tcpudp > xt_pkttype iptable_filter ip_tables x_tables i2c_i801 i2c_core > Mar 5 21:50:44 vpn2 CPU:1 > Mar 5 21:50:44 vpn2 EIP:0060:[]Not tainted VLI > Mar 5 21:50:44 vpn2 EFLAGS: 00010092 (2.6.20-gentoo #10) > Mar 5 21:50:44 vpn2 EIP is at skb_under_panic+0x59/0x5d > Mar 5 21:50:44 vpn2 eax: 0072 ebx: db96e22c ecx: 0001 edx: > de20d4d0 > Mar 5 21:50:44 vpn2 esi: edi: db96e2fc ebp: dc5ab2e8 esp: > dcaf5e84 > Mar 5 21:50:44 vpn2 ds: 007b es: 007b ss: 0068 > Mar 5 21:50:44 vpn2 Process pptpctrl (pid: 5203, ti=dcaf4000 task=de20d4d0 > task.ti=dcaf4000) > Mar 5 21:50:44 vpn2 Stack: c0315e34 c02248a2 00cf 0001 db96e22c > db96e22b db96e2fa db96e82c > Mar 5 21:50:44 vpn2 c0301eef 0004 c02248b0 de20d4d0 de20da04 > de20d4d0 > Mar 5 21:50:44 vpn2 db96e22b 00cf de3e28f0 de3e2884 ddae82ac > de3e2854 0296 > Mar 5 21:50:44 vpn2 Call Trace: > Mar 5 21:50:44 vpn2 [] ppp_asynctty_receive+0x6d2/0x710 > Mar 5 21:50:44 vpn2 [] ppp_asynctty_receive+0x6e0/0x710 > Mar 5 21:50:44 vpn2 [] pty_write+0x39/0x41 > Mar 5 21:50:44 vpn2 [] write_chan+0x212/0x320 > Mar 5 21:50:44 vpn2 [] default_wake_function+0x0/0xc > Mar 5 21:50:44 vpn2 [] tty_write+0x11c/0x1d0 > Mar 5 21:50:44 vpn2 [] write_chan+0x0/0x320 > Mar 5 21:50:44 vpn2 [] vfs_write+0x87/0xf0 > Mar 5 21:50:44 vpn2 [] tty_write+0x0/0x1d0 > Mar 5 21:50:44 vpn2 [] sys_write+0x41/0x6a > Mar 5 21:50:44 vpn2 [] sysenter_past_esp+0x5f/0x99 > Mar 5 21:50:44 vpn2 === > Mar 5 21:50:44 vpn2 Code: 00 00 89 5c 24 14 8b 98 8c 00 00 00 89 5c 24 10 > 89 54 24 0c 8b 40 60 89 44 24 08 89 4c 24 04 c7 04 24 34 5e 31 c0 e8 8e e4 > ec ff <0 > f> 0b eb fe 56 53 83 ec 24 8b 70 14 bb ef 1e 30 c0 85 f6 0f 45 > Mar 5 21:50:44 vpn2 EIP: [] skb_under_panic+0x59/0x5d SS:ESP > 0068:dcaf5e84 > > Another thing - when i`ve changed HZ from 100 too 300 there is no such > lockups for few days. > > Jura > - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bugme-new] [Bug 8132] New: pptp server lockup in ppp_asynctty_receive()
On Fri, Mar 09, 2007 at 11:40:04AM +0300, Yuriy N. Shkandybin wrote: ... > I already have > CONFIG_PROVE_LOCKING=y > # CONFIG_4KSTACKS is not set > .config is at > http://bugzilla.kernel.org/attachment.cgi?id=10660&action=view > Also all information i've provied was recieved by serial console and it's > not hand writing. Maybe you cut too much (lines before a bug could be meaningful - if there are some secret data - try to mask only them). And there is no real lockdep diagnose, so I wasn't sure. Dump registers at the end looks incomplete (no ESP info), too. So, this all made me think - it's cut because it's from the screen. Very sorry!!! > I've checked logs and right before lockup there is oops in syslog > Mar 5 21:50:44 vpn2 skb_under_panic: text:c02248a2 len:207 put:1 As a matter of fact the first bug is the most interesting and everything after could be only some aftereffect. This skb_under_panic message could change all diagnose: it's not necessarily a locking problem. On the other hand this could be even harder to diagnose than locking... > head:db96e22c data:db96e22b tail:db96e2fa end:db96e82c dev: > Mar 5 21:50:44 vpn2 [ cut here ] Why?! We don't even know if there was some config pending here. > Mar 5 21:50:44 vpn2 kernel BUG at net/core/skbuff.c:112! ... > Another thing - when i`ve changed HZ from 100 too 300 there is no such > lockups for few days. You wrote it had been since several kernel releases, so it shouldn't be the case, but anyway - it would be interesting to check HZ 100 but with clean and more current kernel version e.g. 2.6.21-rc3. Thanks & regards, Jarek P. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bugme-new] [Bug 8132] New: pptp server lockup in ppp_asynctty_receive()
Could this be done with: CONFIG_PROVE_LOCKING = y CONFIG_4KSTACKS = n and, if possible, to send the first full lockdep error message with some lines before it? It would be nice to try to be more exact when hand writing, too. Regards, Jarek P. I already have CONFIG_PROVE_LOCKING=y # CONFIG_4KSTACKS is not set .config is at http://bugzilla.kernel.org/attachment.cgi?id=10660&action=view Also all information i've provied was recieved by serial console and it's not hand writing. I've checked logs and right before lockup there is oops in syslog Mar 5 21:50:44 vpn2 skb_under_panic: text:c02248a2 len:207 put:1 head:db96e22c data:db96e22b tail:db96e2fa end:db96e82c dev: Mar 5 21:50:44 vpn2 [ cut here ] Mar 5 21:50:44 vpn2 kernel BUG at net/core/skbuff.c:112! Mar 5 21:50:44 vpn2 invalid opcode: [#1] Mar 5 21:50:44 vpn2 SMP Mar 5 21:50:44 vpn2 Modules linked in: 8021q ipt_TCPMSS xt_tcpudp xt_pkttype iptable_filter ip_tables x_tables i2c_i801 i2c_core Mar 5 21:50:44 vpn2 CPU:1 Mar 5 21:50:44 vpn2 EIP:0060:[]Not tainted VLI Mar 5 21:50:44 vpn2 EFLAGS: 00010092 (2.6.20-gentoo #10) Mar 5 21:50:44 vpn2 EIP is at skb_under_panic+0x59/0x5d Mar 5 21:50:44 vpn2 eax: 0072 ebx: db96e22c ecx: 0001 edx: de20d4d0 Mar 5 21:50:44 vpn2 esi: edi: db96e2fc ebp: dc5ab2e8 esp: dcaf5e84 Mar 5 21:50:44 vpn2 ds: 007b es: 007b ss: 0068 Mar 5 21:50:44 vpn2 Process pptpctrl (pid: 5203, ti=dcaf4000 task=de20d4d0 task.ti=dcaf4000) Mar 5 21:50:44 vpn2 Stack: c0315e34 c02248a2 00cf 0001 db96e22c db96e22b db96e2fa db96e82c Mar 5 21:50:44 vpn2 c0301eef 0004 c02248b0 de20d4d0 de20da04 de20d4d0 Mar 5 21:50:44 vpn2 db96e22b 00cf de3e28f0 de3e2884 ddae82ac de3e2854 0296 Mar 5 21:50:44 vpn2 Call Trace: Mar 5 21:50:44 vpn2 [] ppp_asynctty_receive+0x6d2/0x710 Mar 5 21:50:44 vpn2 [] ppp_asynctty_receive+0x6e0/0x710 Mar 5 21:50:44 vpn2 [] pty_write+0x39/0x41 Mar 5 21:50:44 vpn2 [] write_chan+0x212/0x320 Mar 5 21:50:44 vpn2 [] default_wake_function+0x0/0xc Mar 5 21:50:44 vpn2 [] tty_write+0x11c/0x1d0 Mar 5 21:50:44 vpn2 [] write_chan+0x0/0x320 Mar 5 21:50:44 vpn2 [] vfs_write+0x87/0xf0 Mar 5 21:50:44 vpn2 [] tty_write+0x0/0x1d0 Mar 5 21:50:44 vpn2 [] sys_write+0x41/0x6a Mar 5 21:50:44 vpn2 [] sysenter_past_esp+0x5f/0x99 Mar 5 21:50:44 vpn2 === Mar 5 21:50:44 vpn2 Code: 00 00 89 5c 24 14 8b 98 8c 00 00 00 89 5c 24 10 89 54 24 0c 8b 40 60 89 44 24 08 89 4c 24 04 c7 04 24 34 5e 31 c0 e8 8e e4 ec ff <0 f> 0b eb fe 56 53 83 ec 24 8b 70 14 bb ef 1e 30 c0 85 f6 0f 45 Mar 5 21:50:44 vpn2 EIP: [] skb_under_panic+0x59/0x5d SS:ESP 0068:dcaf5e84 Another thing - when i`ve changed HZ from 100 too 300 there is no such lockups for few days. Jura - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bugme-new] [Bug 8132] New: pptp server lockup in ppp_asynctty_receive()
On 06-03-2007 00:13, Andrew Morton wrote: > On Mon, 5 Mar 2007 14:26:30 -0800 > [EMAIL PROTECTED] wrote: > >> http://bugzilla.kernel.org/show_bug.cgi?id=8132 >> >>Summary: pptp server lockup in ppp_asynctty_receive() >> Kernel Version: 2.6.20 >> Status: NEW >> Severity: high >> Owner: [EMAIL PROTECTED] >> Submitter: [EMAIL PROTECTED] >> CC: [EMAIL PROTECTED] >> >> >> Already several kernel releases i've expirienced different lockups of vpn >> (pptp) server. >> There is more then 200 ppp connections sometimes. >> With kernel debug i was able to retrive next information: Could this be done with: CONFIG_PROVE_LOCKING = y CONFIG_4KSTACKS = n and, if possible, to send the first full lockdep error message with some lines before it? It would be nice to try to be more exact when hand writing, too. Regards, Jarek P. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bugme-new] [Bug 8132] New: pptp server lockup in ppp_asynctty_receive()
On Mon, 5 Mar 2007 14:26:30 -0800
[EMAIL PROTECTED] wrote:
> http://bugzilla.kernel.org/show_bug.cgi?id=8132
>
>Summary: pptp server lockup in ppp_asynctty_receive()
> Kernel Version: 2.6.20
> Status: NEW
> Severity: high
> Owner: [EMAIL PROTECTED]
> Submitter: [EMAIL PROTECTED]
> CC: [EMAIL PROTECTED]
>
>
> Already several kernel releases i've expirienced different lockups of vpn
> (pptp) server.
> There is more then 200 ppp connections sometimes.
> With kernel debug i was able to retrive next information:
>
> First:
> Showing all locks held in the system:
> 1 lock held by agetty/4486:
> #0: (&tty->atomic_read_lock){--..}, at: [] read_chan+0x41a/0x60b
> 1 lock held by agetty/4487:
> #0: (&tty->atomic_read_lock){--..}, at: [] read_chan+0x41a/0x60b
> 1 lock held by agetty/4488:
> #0: (&tty->atomic_read_lock){--..}, at: [] read_chan+0x41a/0x60b
> 2 locks held by pptpctrl/4500:
> #0: (&tty->atomic_write_lock){--..}, at: [] tty_write+0x83/0x1d0
> #1: (&ap->recv_lock){}, at: []
> ppp_asynctty_receive+0x2e/0x710
>
> =
> BUG: spinlock lockup on CPU#1, pppd/4504, df5048c4
> [] _raw_spin_lock+0x100/0x134
> [] ppp_async_ioctl+0xa7/0x1d0
> [] ppp_ioctl+0xa5/0xbff
> [] down_read+0x29/0x3a
> [] ppp_async_ioctl+0x0/0x1d0
> [] ppp_ioctl+0xce/0xbff
> [] _spin_unlock+0x14/0x1c
> [] do_wp_page+0x256/0x4ba
> [] __handle_mm_fault+0x74e/0xa22
> [] do_ioctl+0x64/0x6d
> [] vfs_ioctl+0x50/0x273
> [] sys_ioctl+0x34/0x50
> [] sysenter_past_esp+0x5f/0x99
> ===
> BUG: soft lockup detected on CPU#0!
> [] softlockup_tick+0x8d/0xbc
> [] update_process_times+0x28/0x5e
> [] smp_apic_timer_interrupt+0x80/0x9c
> [] apic_timer_interrupt+0x33/0x38
> [] delay_tsc+0x9/0x13
> [] __delay+0x6/0x7
> [] _raw_spin_lock+0xa9/0x134
> [] tty_write+0x83/0x1d0
> [] tty_ldisc_try+0x2f/0x33
> [] lock_kernel+0x19/0x24
> [] tty_write+0x10b/0x1d0
> [] write_chan+0x0/0x320
> [] vfs_write+0x87/0xf0
> [] tty_write+0x0/0x1d0
> [] sys_write+0x41/0x6a
> [] sysenter_past_esp+0x5f/0x99
> ===
>
>
> Second)
> <0>BUG: spinlock lockup on CPU#0, pppd/5209, de3e2884
> [] _raw_spin_lock+0x100/0x134
> BUG: spinlock lockup on CPU#1, ip-down/7524, c0353300
> [] _raw_spin_lock+0x100/0x134
> [] lock_kernel+0x19/0x24
> [] chrdev_open+0x8a/0x16e
> [] chrdev_open+0x0/0x16e
> [] __dentry_open+0xaf/0x1a0
> [] nameidata_to_filp+0x31/0x3a
> [] do_filp_open+0x39/0x40
> [] _spin_unlock+0x14/0x1c
> [] get_unused_fd+0xaa/0xbb
> [] do_sys_open+0x3a/0x6d
> [] sys_open+0x1c/0x20
> [] sysenter_past_esp+0x5f/0x99
> ===
> [] ppp_async_ioctl+0xa7/0x1d0
> [] ppp_ioctl+0xa5/0xbff
> [] down_read+0x29/0x3a
> [] ppp_async_ioctl+0x0/0x1d0
> [] ppp_ioctl+0xce/0xbff
> [] _spin_unlock+0x14/0x1c
> [] do_wp_page+0x256/0x4ba
> [] __handle_mm_fault+0x74e/0xa22
> [] do_ioctl+0x64/0x6d
> [] vfs_ioctl+0x50/0x273
> [] sys_ioctl+0x34/0x50
> [] sysenter_past_esp+0x5f/0x99
> ===
>
> Third)
> BUG: soft lockup detected on CPU#0!
> [] softlockup_tick+0x8d/0xbc
> [] update_process_times+0x28/0x5e
> [] smp_apic_timer_interrupt+0x80/0x9c
> [] apic_timer_interrupt+0x33/0x38
> [] delay_tsc+0x9/0x13
> [] __delay+0x6/0x7
> [] _raw_spin_lock+0xa9/0x134
> [] tty_ldisc_try+0x2f/0x33
> [] lock_kernel+0x19/0x24
> [] tty_read+0x5a/0xbe
> [] vfs_read+0x85/0xee
> [] tty_read+0x0/0xbe
> [] sys_read+0x41/0x6a
> [] sysenter_past_esp+0x5f/0x99
> ===
> BUG: soft lockup detected on CPU#0!
> [] softlockup_tick+0x8d/0xbc
> [] update_process_times+0x28/0x5e
> [] smp_apic_timer_interrupt+0x80/0x9c
> [] apic_timer_interrupt+0x33/0x38
> [] prio_tree_insert+0xe8/0x23b
> [] _raw_spin_lock+0xaf/0x134
> [] tty_ldisc_try+0x2f/0x33
> [] lock_kernel+0x19/0x24
> [] tty_read+0x5a/0xbe
> [] vfs_read+0x85/0xee
> [] tty_read+0x0/0xbe
> [] sys_read+0x41/0x6a
> [] sysenter_past_esp+0x5f/0x99
>
>
> Next via SysRq:
>
> Showing all locks held in the system:
> 1 lock held by agetty/5057:
> #0: (&tty->atomic_read_lock){--..}, at: [] read_chan+0x41a/0x60b
> 1 lock held by agetty/5058:
> #0: (&tty->atomic_read_lock){--..}, at: [] read_chan+0x41a/0x60b
> 1 lock held by agetty/5059:
> #0: (&tty->atomic_read_lock){--..}, at: [] read_chan+0x41a/0x60b
> 2 locks held by pptpctrl/5071:
> #0: (&tty->atomic_write_lock){--..}, at: [] tty_write+0x83/0x1d0
> #1: (&ap->recv_lock){}, at: []
> ppp_asynctty_receive+0x2e/0x710
>
>
> ~#SysRq : Show Blocked State
>
> freesibling
> task PCstack pid father child younger older
> pptpctrl D C02A18E0 0 5071 4646 50745094 5064 (L-TLB)
>df3a3bd0 0082 0029b837 c02a18e0 0246 dd4f131c
> dd563cac
>def86030 c140864c 0009 def86030 2ccaa8e5
> 017d
>
