Re: 'iw events' stops receiving events after a while on 4.9 + hacks
On 05/31/2017 01:18 AM, Bastian Bittorf wrote: * Johannes Berg[31.05.2017 10:09]: Is there any way to dump out the socket information if we reproduce the problem? I have no idea, sorry. If you or Bastian can tell me how to reproduce the problem, I can try to investigate it. there was an interesting fix regarding the shell-builtin 'read' in busybox[1]. I will retest again and report if this changes anything. bye, bastian PS: @ben: are you also using 'iw event | while read -r LINE ...'? I'm using a perl script to read the output, and not using busybox. I have not seen the problem again, so it is not easy for me to reproduce. If you reproduce it, maybe check 'strace' on the 'iw' process to see if it is hung on writing output to the pipe or reading input? In my case, it appeared to be hung reading input from netlink, input that never arrived. Thanks, Ben -- Ben Greear Candela Technologies Inc http://www.candelatech.com
Re: 'iw events' stops receiving events after a while on 4.9 + hacks
* Johannes Berg[31.05.2017 10:09]: > > Is there any way to dump out the socket information if we reproduce > > the problem? > > I have no idea, sorry. > > If you or Bastian can tell me how to reproduce the problem, I can try > to investigate it. there was an interesting fix regarding the shell-builtin 'read' in busybox[1]. I will retest again and report if this changes anything. bye, bastian PS: @ben: are you also using 'iw event | while read -r LINE ...'? [1] https://git.busybox.net/busybox/commit/?id=f5470419404d643070db99d058405b714695b817
Re: 'iw events' stops receiving events after a while on 4.9 + hacks
On Wed, 2017-05-17 at 08:22 -0700, Ben Greear wrote: > > I saw it exactly once so far, and it happened overnight, > but we have not been doing a lot of work with the 4.9 kernel until > recently. > > I don't think there were many messages on this system, and certainly > others have run much longer on systems that should be generating many > more events without trouble. Ok. > Is there any way to dump out the socket information if we reproduce > the problem? I have no idea, sorry. If you or Bastian can tell me how to reproduce the problem, I can try to investigate it. johannes
Re: 'iw events' stops receiving events after a while on 4.9 + hacks
* Johannes Berg[17.05.2017 20:18]: > On Wed, 2017-05-17 at 12:08 +0200, Bastian Bittorf wrote: > > * Ben Greear [17.05.2017 11:51]: [...] > > > kernels, but when testing on 4.9 overnight, I notice that 'iw > > > events' is not showing any input. 'strace' shows > > > that it is waiting on recvmsg. If I start a second 'iw events' > > > then it will get > > > wifi events as expected. > > > > me too, also seen on 4.4 - i'am happy for debug ideas. > > I've never seen this. > > Does it happen when it's very long-running? Or when there are lots of > events? only a couple of hours. hard to say which is the culprit. here i run it like: #!/bin/sh iw event | while read -r LINE; do case "$LINE" in *': new station '*) ... ;; *': del station '*) ... ;; esac done The script marks new stations with "touch /tmp/$mac" and removes this file during 'del station'. What is interesting: i can recognize, that sometimes and somehow i have stations in 'iw dev wlanX station dump' which the script has not seen. When debugging this, the script does not get any new events. A new started 'iw event' can see further events without problems. Hard to say where the error happens. I'am on busybox here. bye, Bastian
Re: 'iw events' stops receiving events after a while on 4.9 + hacks
On 05/17/2017 06:30 AM, Johannes Berg wrote: On Wed, 2017-05-17 at 12:08 +0200, Bastian Bittorf wrote: * Ben Greear[17.05.2017 11:51]: I have been keeping an 'iw events' program running with a perl script gathering its output and post-processing it. This has been working for several years on 4.7 and earlier kernels, but when testing on 4.9 overnight, I notice that 'iw events' is not showing any input. 'strace' shows that it is waiting on recvmsg. If I start a second 'iw events' then it will get wifi events as expected. me too, also seen on 4.4 - i'am happy for debug ideas. I've never seen this. Does it happen when it's very long-running? Or when there are lots of events? Perhaps something in the socket buffer accounting is going wrong, so that it's slowly decreasing to 0? I saw it exactly once so far, and it happened overnight, but we have not been doing a lot of work with the 4.9 kernel until recently. I don't think there were many messages on this system, and certainly others have run much longer on systems that should be generating many more events without trouble. Is there any way to dump out the socket information if we reproduce the problem? Thanks, Ben -- Ben Greear Candela Technologies Inc http://www.candelatech.com
Re: 'iw events' stops receiving events after a while on 4.9 + hacks
On Wed, 2017-05-17 at 12:08 +0200, Bastian Bittorf wrote: > * Ben Greear[17.05.2017 11:51]: > > I have been keeping an 'iw events' program running with a perl > > script gathering its > > output and post-processing it. This has been working for several > > years on 4.7 and earlier > > kernels, but when testing on 4.9 overnight, I notice that 'iw > > events' is not showing any input. 'strace' shows > > that it is waiting on recvmsg. If I start a second 'iw events' > > then it will get > > wifi events as expected. > > me too, also seen on 4.4 - i'am happy for debug ideas. I've never seen this. Does it happen when it's very long-running? Or when there are lots of events? Perhaps something in the socket buffer accounting is going wrong, so that it's slowly decreasing to 0? johannes
Re: 'iw events' stops receiving events after a while on 4.9 + hacks
* Ben Greear[17.05.2017 11:51]: > I have been keeping an 'iw events' program running with a perl script > gathering its > output and post-processing it. This has been working for several years on > 4.7 and earlier > kernels, but when testing on 4.9 overnight, I notice that 'iw events' is not > showing any input. 'strace' shows > that it is waiting on recvmsg. If I start a second 'iw events' then it will > get > wifi events as expected. me too, also seen on 4.4 - i'am happy for debug ideas. bye, bastian