Re: Maintaining connection
On Thu, 2015-03-12 at 09:04 -0500, Alex Ferm wrote: We use cellular modems for telemetry, and they disconnect every so often, and don't come back on-line on their own. I am using dbus via python to send Enable(False) followed by Enable(True) to reset network manager. This makes it reconnect to all the automatic connections that have been configured. What version of NetworkManager? I think 0.9.10 or later added WWAN autoconnect, which should handle this better. In any case, the Enable() mechanism is much better than I was thinking you were going to say :) If you're already using NM 0.9.10 or later and NM isn't attempting to reconnect, then it would be great to see some logs from the device so we can figure out what's going on. One more question: do you mind if the connection starts automatically when NM starts? If that's OK, then autoconnect=true + NM 0.9.10 should do what you want already unless there's a bug. Dan Alex Ferm PetroPower, LLC. 3003 E. 37th Street N. Suite 100 Wichita, KS 67219 Phone: (316) 361-0222 Toll Free: (877) 265-6581 Fax: (316) 361-0967 On 03/11/2015 07:49 PM, Stuart D. Gathman wrote: On Wed, 11 Mar 2015, Dan Williams wrote: On Wed, 2015-03-11 at 15:57 -0500, Alex Ferm wrote: Hello, I'm trying to write a python script that resets NetworkManager when the state is not NM_STATE_CONNECTED_GLOBAL. Does NetworkManager time out and retry automatically during the NM_STATE_CONNECTING state? Also, how is the NM_STATE_CONNECTED_GLOBAL determined(ie: is there a periodic ping or something?)? What's the reason to reset NM when it reports something isn't connected? Just to ensure always-on connectivity as hard as possible? Also, what do you mean by reset, what specific actions are you running to do so? In my case, wpa_supplicant hangs every so many megabytes, and I have to killall wpa_supplicant to restore the network connection. I've been wondering about a way to have NM do that automatically, since fixing wpa_supplicant seems to be difficult. ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: Maintaining connection
On Wed, 2015-03-11 at 20:49 -0400, Stuart D. Gathman wrote: On Wed, 11 Mar 2015, Dan Williams wrote: On Wed, 2015-03-11 at 15:57 -0500, Alex Ferm wrote: Hello, I'm trying to write a python script that resets NetworkManager when the state is not NM_STATE_CONNECTED_GLOBAL. Does NetworkManager time out and retry automatically during the NM_STATE_CONNECTING state? Also, how is the NM_STATE_CONNECTED_GLOBAL determined(ie: is there a periodic ping or something?)? What's the reason to reset NM when it reports something isn't connected? Just to ensure always-on connectivity as hard as possible? Also, what do you mean by reset, what specific actions are you running to do so? In my case, wpa_supplicant hangs every so many megabytes, and I have to killall wpa_supplicant to restore the network connection. I've been wondering about a way to have NM do that automatically, since fixing wpa_supplicant seems to be difficult. You can just kill -9 wpa_supplicant from a script somewhere and NM will restart the supplicant automatically via D-Bus activation. Does the hanging supplicant also block D-Bus communication? Can you gdb the supplicant and find out what function it's hung in? Or is it not actually hanging, but just not doing some requested operation? But obviously, fixing wpa_supplicant would be preferable... Dan ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: Maintaining connection
On 03/12/2015 10:31 AM, Dan Williams wrote: On Wed, 2015-03-11 at 20:49 -0400, Stuart D. Gathman wrote: On Wed, 11 Mar 2015, Dan Williams wrote: What's the reason to reset NM when it reports something isn't connected? Just to ensure always-on connectivity as hard as possible? Also, what do you mean by reset, what specific actions are you running to do so? In my case, wpa_supplicant hangs every so many megabytes, and I have to killall wpa_supplicant to restore the network connection. I've been wondering about a way to have NM do that automatically, since fixing wpa_supplicant seems to be difficult. You can just kill -9 wpa_supplicant from a script somewhere and NM will restart the supplicant automatically via D-Bus activation. Does the hanging supplicant also block D-Bus communication? Can you gdb the supplicant and find out what function it's hung in? Or is it not actually hanging, but just not doing some requested operation? But obviously, fixing wpa_supplicant would be preferable... This has some stacktraces. The problem persists on several different laptops, with different Wifi chipsets. https://bugzilla.redhat.com/show_bug.cgi?id=1119524 ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: Maintaining connection
On 03/12/2015 11:58 AM, Dan Williams wrote: If the packets have an expired key, then that implies that key rotation has not happened. That's done on a timer that's negotiated during the EAP or WPA-PSK setup. If you had wpa_supplicant debug logs you could probably figure out what's going on in the supplicant. It stops logging while hung. This is from Fedora 19 (been reporting this problem for years): wlan0: Trying to associate with 00:1d:7e:2f:bd:0e (SSID='Gathman Home' freq=2462 MHz) wlan0: Associated with 00:1d:7e:2f:bd:0e wlan0: WPA: Key negotiation completed with 00:1d:7e:2f:bd:0e [PTK=CCMP GTK=CCMP] wlan0: CTRL-EVENT-CONNECTED - Connection to 00:1d:7e:2f:bd:0e completed [id=0 id_str=] working away*** dbus: wpas_dbus_bss_signal_prop_changed: Unknown Property value 7 dbus: wpas_dbus_bss_signal_prop_changed: Unknown Property value 7 dbus: wpas_dbus_bss_signal_prop_changed: Unknown Property value 7 repeat hundreds of times dbus: wpas_dbus_bss_signal_prop_changed: Unknown Property value 7 dbus: wpas_dbus_bss_signal_prop_changed: Unknown Property value 7 *hangs* killall wpa_supplicant* wlan0: CTRL-EVENT-DISCONNECTED bssid=00:1d:7e:2f:bd:0e reason=3 locally_generated=1 wlan0: CTRL-EVENT-TERMINATING Successfully initialized wpa_supplicant dbus: wpas_dbus_bss_signal_prop_changed: Unknown Property value 7 wlan0: SME: Trying to authenticate with 00:1d:7e:2f:bd:0e (SSID='Gathman Home' freq=2462 MHz) wlan0: Trying to associate with 00:1d:7e:2f:bd:0e (SSID='Gathman Home' freq=2462 MHz) wlan0: Associated with 00:1d:7e:2f:bd:0e wlan0: WPA: Key negotiation completed with 00:1d:7e:2f:bd:0e [PTK=CCMP GTK=CCMP] wlan0: CTRL-EVENT-CONNECTED - Connection to 00:1d:7e:2f:bd:0e completed [id=0 id_str=] dbus: wpas_dbus_bss_signal_prop_changed: Unknown Property value 7 dbus: wpas_dbus_bss_signal_prop_changed: Unknown Property value 7 dbus: wpas_dbus_bss_signal_prop_changed: Unknown Property value 7 working away again ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: Maintaining connection
On Thu, 2015-03-12 at 11:54 -0400, Stuart Gathman wrote: On 03/11/2015 06:23 PM, Dan Williams wrote: On Wed, 2015-03-11 at 15:57 -0500, Alex Ferm wrote: Hello, I'm trying to write a python script that resets NetworkManager when the state is not NM_STATE_CONNECTED_GLOBAL. Does NetworkManager time out and retry automatically during the NM_STATE_CONNECTING state? Also, how is the NM_STATE_CONNECTED_GLOBAL determined(ie: is there a periodic ping or something?)? What's the reason to reset NM when it reports something isn't connected? Just to ensure always-on connectivity as hard as possible? Also, what do you mean by reset, what specific actions are you running to do so? In general, it would be useful if NM was able to detect that networking was hosed and reset things. How would it know, however? I generally ping the gateway manually to check the connection. Are gateways always pingable? Just checking for a connection is not sufficient - e.g. when wpa_supplicant hangs (radio is operational, but incoming and outgoing packets have expired key and are discarded). Microsoft used to ping one If the packets have an expired key, then that implies that key rotation has not happened. That's done on a timer that's negotiated during the EAP or WPA-PSK setup. If you had wpa_supplicant debug logs you could probably figure out what's going on in the supplicant. of their servers. But that could be a problem anywhere in between the PC and their server. NM does that with it's connectivity checking functionality, but of course since it's phone home functionality more or less, that must be enabled by the user/admin. NM does advertise the result via dbus/nmcli, but does not yet do anything with it internally. It's also not per-interface yet, so you couldn't use the result to determine whether a specific interface was hosted or not. Dan ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: Maintaining connection
On 03/11/2015 06:23 PM, Dan Williams wrote: On Wed, 2015-03-11 at 15:57 -0500, Alex Ferm wrote: Hello, I'm trying to write a python script that resets NetworkManager when the state is not NM_STATE_CONNECTED_GLOBAL. Does NetworkManager time out and retry automatically during the NM_STATE_CONNECTING state? Also, how is the NM_STATE_CONNECTED_GLOBAL determined(ie: is there a periodic ping or something?)? What's the reason to reset NM when it reports something isn't connected? Just to ensure always-on connectivity as hard as possible? Also, what do you mean by reset, what specific actions are you running to do so? In general, it would be useful if NM was able to detect that networking was hosed and reset things. How would it know, however? I generally ping the gateway manually to check the connection. Are gateways always pingable? Just checking for a connection is not sufficient - e.g. when wpa_supplicant hangs (radio is operational, but incoming and outgoing packets have expired key and are discarded). Microsoft used to ping one of their servers. But that could be a problem anywhere in between the PC and their server. ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: Maintaining connection
On Thu, 2015-03-12 at 10:39 -0400, Stuart Gathman wrote: On 03/12/2015 10:31 AM, Dan Williams wrote: On Wed, 2015-03-11 at 20:49 -0400, Stuart D. Gathman wrote: On Wed, 11 Mar 2015, Dan Williams wrote: What's the reason to reset NM when it reports something isn't connected? Just to ensure always-on connectivity as hard as possible? Also, what do you mean by reset, what specific actions are you running to do so? In my case, wpa_supplicant hangs every so many megabytes, and I have to killall wpa_supplicant to restore the network connection. I've been wondering about a way to have NM do that automatically, since fixing wpa_supplicant seems to be difficult. You can just kill -9 wpa_supplicant from a script somewhere and NM will restart the supplicant automatically via D-Bus activation. Does the hanging supplicant also block D-Bus communication? Can you gdb the supplicant and find out what function it's hung in? Or is it not actually hanging, but just not doing some requested operation? But obviously, fixing wpa_supplicant would be preferable... This has some stacktraces. The problem persists on several different laptops, with different Wifi chipsets. https://bugzilla.redhat.com/show_bug.cgi?id=1119524 Ok, we'll need debug logs then, since the stack traces show that the supplicant isn't really hung, it's just that the key change apparently didn't happen correctly... you can do this by: mv /usr/sbin/wpa_supplicant / killall -TERM wpa_supplicant /wpa_supplicant -dddtu (pipe to your favorite logfile) and then go until it hangs. Then send me the logfile since it might contain private information. Move the supplicant back to /usr/sbin/ when you're done to get back to normal. Dan ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: Maintaining connection
On Thu, 2015-03-12 at 14:09 -0400, Stuart Gathman wrote: On 03/12/2015 12:40 PM, Dan Williams wrote: This has some stacktraces. The problem persists on several different laptops, with different Wifi chipsets. https://bugzilla.redhat.com/show_bug.cgi?id=1119524 Ok, we'll need debug logs then, since the stack traces show that the supplicant isn't really hung, it's just that the key change apparently didn't happen correctly... you can do this by: mv /usr/sbin/wpa_supplicant / killall -TERM wpa_supplicant /wpa_supplicant -dddtu (pipe to your favorite logfile) and then go until it hangs. Then send me the logfile since it might contain private information. Move the supplicant back to /usr/sbin/ when you're done to get back to normal. Already did that in f19 (see log in above bugzilla) - it doesn't log anything while hung (maybe I didn't have as many 'd's). But maybe F21 will be different. It takes a little over 3 days to hang with hardware crypto. So hang on ... :-) Yeah, but that log isn't a debug log, it's just the normal messages. -dddtu will add a ton of debug info from which we might be able to find out why rekeying isn't happening. Dan ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: Maintaining connection
On 03/12/2015 12:40 PM, Dan Williams wrote: This has some stacktraces. The problem persists on several different laptops, with different Wifi chipsets. https://bugzilla.redhat.com/show_bug.cgi?id=1119524 Ok, we'll need debug logs then, since the stack traces show that the supplicant isn't really hung, it's just that the key change apparently didn't happen correctly... you can do this by: mv /usr/sbin/wpa_supplicant / killall -TERM wpa_supplicant /wpa_supplicant -dddtu (pipe to your favorite logfile) and then go until it hangs. Then send me the logfile since it might contain private information. Move the supplicant back to /usr/sbin/ when you're done to get back to normal. Already did that in f19 (see log in above bugzilla) - it doesn't log anything while hung (maybe I didn't have as many 'd's). But maybe F21 will be different. It takes a little over 3 days to hang with hardware crypto. So hang on ... :-) ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: Maintaining connection
On Wed, 2015-03-11 at 15:57 -0500, Alex Ferm wrote: Hello, I'm trying to write a python script that resets NetworkManager when the state is not NM_STATE_CONNECTED_GLOBAL. Does NetworkManager time out and retry automatically during the NM_STATE_CONNECTING state? Also, how is the NM_STATE_CONNECTED_GLOBAL determined(ie: is there a periodic ping or something?)? What's the reason to reset NM when it reports something isn't connected? Just to ensure always-on connectivity as hard as possible? Also, what do you mean by reset, what specific actions are you running to do so? Dan ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: Maintaining connection
On Wed, 11 Mar 2015, Dan Williams wrote: On Wed, 2015-03-11 at 15:57 -0500, Alex Ferm wrote: Hello, I'm trying to write a python script that resets NetworkManager when the state is not NM_STATE_CONNECTED_GLOBAL. Does NetworkManager time out and retry automatically during the NM_STATE_CONNECTING state? Also, how is the NM_STATE_CONNECTED_GLOBAL determined(ie: is there a periodic ping or something?)? What's the reason to reset NM when it reports something isn't connected? Just to ensure always-on connectivity as hard as possible? Also, what do you mean by reset, what specific actions are you running to do so? In my case, wpa_supplicant hangs every so many megabytes, and I have to killall wpa_supplicant to restore the network connection. I've been wondering about a way to have NM do that automatically, since fixing wpa_supplicant seems to be difficult. ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
