Re: spamming router with router solicitations
On Tue, Jun 25, 2019 at 10:20:54AM -0400, Brian J. Murrell wrote: > On Mon, 2019-06-24 at 08:52 +0200, Beniamino Galvani via > networkmanager-list wrote: > > > > Hi, > > Hi Beniamino, > > > I checked again the log you sent and I see the problem now. When NM > > receives a RA, it checks whether the parameters > > Which parameters exactly? Because I might be able to shed some light > on this now that this is known. Basically everything in the RA. See how the 'changed' variable is set in: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/blob/1.18.0/src/ndisc/nm-lndp-ndisc.c#L110 > > changed compared to > > the previous RA and if so it applies the new configuration. When it > > does so, it also reapplies the token; this triggers a new router > > solicitation from kernel due to: > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/ipv6/addrconf.c?h=v3.10#n4336 > > Interesting. > > > The new RA received is: > > > > neighbor discovery configuration changed [R]: > > dhcp-level none > > gateway fe80::6eb0:ceff:fef5:1e4a pref high exp 1799.2317 > > address 2001:123:ab:123::2 exp permanent > > route 2001:123:ab:123::/64 via fe80::6eb0:ceff:fef5:1e4a pref > > high exp permanent > > dns_server fd31:aeb1:48df::2 exp 7199.2317 > > > > Note the "changed [R]" part which means that routes changed. This is > > strange because according to log there was no change from previous > > RA. This causes the reapply of token, a new RS, a RA and so on ... > > Here is what an RA from my router looks like: > > [...] > > But three things that the above is not saying: > >1. Until yesterday, the Router Lifetime of one of those RAs was 0 and > the other was 1800 (I don't recall which was which). >2. Until the last week or two, the first prefix was being advertised > with a Router preference of high and the other was medium. >3. Each of those two RAs come in two different packets, one for each > prefix rather than them both being in the same RA which I think is > the typical behaviour. I think all these should be handled well. But perhaps older versions of NM had issues with the 0 lifetime of point 1. > > Apart from this, I think NM should not apply the token when it's > > already set; > > That seems reasonable. This is now fixed in master: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/e4ce9bd7af6a39677ff1a1380906d18062abb24a and stable branches nm-1-18, nm-1-16. Beniamino signature.asc Description: PGP signature ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Mon, 2019-06-24 at 08:52 +0200, Beniamino Galvani via networkmanager-list wrote: > > Hi, Hi Beniamino, > I checked again the log you sent and I see the problem now. When NM > receives a RA, it checks whether the parameters Which parameters exactly? Because I might be able to shed some light on this now that this is known. > changed compared to > the previous RA and if so it applies the new configuration. When it > does so, it also reapplies the token; this triggers a new router > solicitation from kernel due to: > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/ipv6/addrconf.c?h=v3.10#n4336 Interesting. > The new RA received is: > > neighbor discovery configuration changed [R]: > dhcp-level none > gateway fe80::6eb0:ceff:fef5:1e4a pref high exp 1799.2317 > address 2001:123:ab:123::2 exp permanent > route 2001:123:ab:123::/64 via fe80::6eb0:ceff:fef5:1e4a pref > high exp permanent > dns_server fd31:aeb1:48df::2 exp 7199.2317 > > Note the "changed [R]" part which means that routes changed. This is > strange because according to log there was no change from previous > RA. This causes the reapply of token, a new RS, a RA and so on ... Here is what an RA from my router looks like: Soliciting ff02::2 (ff02::2) on enp2s0... Hop limit : 64 ( 0x40) Stateful address conf.: No Stateful other conf. : No Mobile home agent : No Router preference : medium Neighbor discovery proxy : No Router lifetime : 1800 (0x0708) seconds Reachable time: unspecified (0x) Retransmit time : unspecified (0x) Source link-layer address: 6C:B0:CE:F5:1E:4A MTU : 1280 bytes (valid) Prefix : fd31:aeb1:48df::/64 On-link : Yes Autonomous address conf.: Yes Valid time : infinite (0x) Pref. time : infinite (0x) Recursive DNS server : fd31:aeb1:48df::2 DNS server lifetime : 6000 (0x1770) seconds from fe80::6eb0:ceff:fef5:1e4a Hop limit : 64 ( 0x40) Stateful address conf.: No Stateful other conf. : No Mobile home agent : No Router preference : medium Neighbor discovery proxy : No Router lifetime : 1800 (0x0708) seconds Reachable time: unspecified (0x) Retransmit time : unspecified (0x) Source link-layer address: 6C:B0:CE:F5:1E:4A MTU : 1280 bytes (valid) Prefix : 2001:123:ab:123::/64 On-link : Yes Autonomous address conf.: Yes Valid time : infinite (0x) Pref. time : infinite (0x) Recursive DNS server : fd31:aeb1:48df::2 DNS server lifetime : 6000 (0x1770) seconds from fe80::6eb0:ceff:fef5:1e4a But three things that the above is not saying: 1. Until yesterday, the Router Lifetime of one of those RAs was 0 and the other was 1800 (I don't recall which was which). 2. Until the last week or two, the first prefix was being advertised with a Router preference of high and the other was medium. 3. Each of those two RAs come in two different packets, one for each prefix rather than them both being in the same RA which I think is the typical behaviour. > Apart from this, I think NM should not apply the token when it's > already set; That seems reasonable. Cheers, b. signature.asc Description: This is a digitally signed message part ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Fri, Jun 21, 2019 at 01:03:11PM -0400, Brian J. Murrell wrote: > On Fri, 2019-06-21 at 17:58 +0200, Beniamino Galvani via > networkmanager-list wrote: > > > > Sounds ok. > > Thanks. > > > Which kernel version do you have on the EL 7.6 machine? > > 3.10.0-957.21.2.el7.x86_64 Hi, I checked again the log you sent and I see the problem now. When NM receives a RA, it checks whether the parameters changed compared to the previous RA and if so it applies the new configuration. When it does so, it also reapplies the token; this triggers a new router solicitation from kernel due to: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/ipv6/addrconf.c?h=v3.10#n4336 The new RA received is: neighbor discovery configuration changed [R]: dhcp-level none gateway fe80::6eb0:ceff:fef5:1e4a pref high exp 1799.2317 address 2001:123:ab:123::2 exp permanent route 2001:123:ab:123::/64 via fe80::6eb0:ceff:fef5:1e4a pref high exp permanent dns_server fd31:aeb1:48df::2 exp 7199.2317 Note the "changed [R]" part which means that routes changed. This is strange because according to log there was no change from previous RA. This causes the reapply of token, a new RS, a RA and so on ... If you are not able to reproduce the problem with a newer NM version, this means that probably the wrong comparison of RAs was fixed in the meantime. I don't see any commit that seems to directly fix that, though. Apart from this, I think NM should not apply the token when it's already set; and also I'm not sure why it keeps the accept_ra sysctl enabled these days, that allows the kernel to send RAs in parallel with NM. I will check if this can be improved. Thanks, Beniamino signature.asc Description: PGP signature ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Fri, 2019-06-21 at 17:58 +0200, Beniamino Galvani via networkmanager-list wrote: > > Sounds ok. Thanks. > Which kernel version do you have on the EL 7.6 machine? 3.10.0-957.21.2.el7.x86_64 Cheers, b. signature.asc Description: This is a digitally signed message part ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Tue, Jun 18, 2019 at 02:24:09PM -0400, Brian J. Murrell wrote: > On Mon, 2019-06-03 at 14:17 -0400, Brian J. Murrell wrote: > > > > I think I had it set to DEBUG. Let me see if I can reproduce with it > > set to TRACE. > > I have to correct myself. It was set to TRACE. > > > It goes up quite a bit. More indication that it is NM. > > CPU is pegged between NM, journald and one other unrelated process. I > assume without the latter, NM and journald would monopolize the CPU. > > So I don't really know what's going on here. > > But really, given all of the time I have spent on this, the path of > least resistance is probably just rate-limiting the RSes at the router. > > What's a reasonable number of RSes to allow in a given time-frame? > > 1/sec burst 5 seems to be keeping things under control on both the > router and the machine suffering this NM problem. Sounds ok. Which kernel version do you have on the EL 7.6 machine? B. signature.asc Description: PGP signature ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Fri, 2019-05-24 at 13:16 +0200, Till Maas via networkmanager-list wrote: > Hi Brian, Hi. > You can try if it is fixed in a never version. You can find builds > from latest GIT branches for EL 7 here: > > https://copr.fedorainfracloud.org/coprs/networkmanager/ No, the NetworkManager-1.16.3-1.373ddf2907.el7.x86_64 at above has the same problem. But strangely enough NetworkManager-1.16.2-1.fc30.x86_64 in F30 doesn't have this problem. Cheers, b. signature.asc Description: This is a digitally signed message part ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Mon, 2019-06-03 at 14:17 -0400, Brian J. Murrell wrote: > > I think I had it set to DEBUG. Let me see if I can reproduce with it > set to TRACE. I have to correct myself. It was set to TRACE. > It goes up quite a bit. More indication that it is NM. CPU is pegged between NM, journald and one other unrelated process. I assume without the latter, NM and journald would monopolize the CPU. So I don't really know what's going on here. But really, given all of the time I have spent on this, the path of least resistance is probably just rate-limiting the RSes at the router. What's a reasonable number of RSes to allow in a given time-frame? 1/sec burst 5 seems to be keeping things under control on both the router and the machine suffering this NM problem. Probably worth mentioning that the NM version on Fedora doesn't have this problem. Cheers, b. signature.asc Description: This is a digitally signed message part ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Sun, 2019-06-02 at 10:29 +0200, Thomas Haller via networkmanager- list wrote: > > Thanks, that's of course a string indication that NetworkManager is > the > culprit. Seems so to me. As well as (see answer to your other question below)... > Just to check, did you enable level=TRACE I think I had it set to DEBUG. Let me see if I can reproduce with it set to TRACE. > logging and made sure that a > possible burst of logging messages is not rate limited by systemd- > journald? Yes, most certainly. Set journald's rate limiting to 0. > How is the CPU usage of NetworkManager when this happens? It goes up quite a bit. More indication that it is NM. Cheers, b. signature.asc Description: This is a digitally signed message part ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Sat, 2019-06-01 at 07:00 -0400, Brian J. Murrell wrote: > On Thu, 2019-05-30 at 12:23 +0200, Thomas Haller via networkmanager- > list wrote: > > On Wed, 2019-05-29 at 12:23 -0400, Brian J. Murrell wrote: > > > On Wed, 2019-05-29 at 09:28 +0200, Thomas Haller via > > > networkmanager- > > > list wrote: > > > > If this is happening, when you kill NetworkManager with > > > > SIGKILL, > > > > it > > > > would not give NetworkManager to cleanup... > > > > > > > > sudo killall -SIGKILL NetworkManager > > > > > > > > (and veryify that NetworkManager is indeed not running. Maybe > > > > first > > > > `systemctl mask NetworkManager`, so that systemd won't restart > > > > it). > > > > > > Are you suggesting that I kill NM with SIGKILL as a debugging > > > step > > > to > > > see if the RSes still stop when stopping NM? > > > > Yes. > > > > > If so, that was an interesting experiment: > > > > > > # killall -SIGKILL NetworkManager > > > # ps -ef | grep Network > > > root 15001 1 0 12:02 ?00:00:00 > > > /usr/sbin/NetworkManager --no-daemon > > > > > > So clearly systemd restarted it, but it's not flooding RSes after > > > the > > > restart. > > > > Ah ok. That's what I meant with first masking NetworkManager. > > # systemctl mask NetworkManager > Created symlink from /etc/systemd/system/NetworkManager.service to > /dev/null. > # killall -SIGKILL NetworkManager > # systemctl status NetworkManager > ● NetworkManager.service >Loaded: masked (/dev/null; bad) >Active: failed (Result: signal) since Sat 2019-06-01 06:53:23 EDT; > 39s ago > Main PID: 4286 (code=killed, signal=KILL) > > May 31 22:04:18 server.example.com NetworkManager[4286]: > [1559354658.5223] manager: NetworkManager state is now > CONNECTED_SITE > May 31 22:04:18 server.example.com NetworkManager[4286]: > [1559354658.5274] policy: set 'enp2s0' (enp2s0) as default > for IPv6 routing and DNS > May 31 22:04:18 server.example.com NetworkManager[4286]: > [1559354658.5670] device (enp2s0): Activation: successful, > device activated. > May 31 22:04:18 server.example.com NetworkManager[4286]: > [1559354658.5807] manager: NetworkManager state is now > CONNECTED_GLOBAL > May 31 22:04:18 server.example.com NetworkManager[4286]: > [1559354658.5934] manager: startup complete > May 31 22:04:33 server.example.com NetworkManager[4286]: > [1559354673.1941] policy: set 'enp2s0' (enp2s0) as default > for IPv4 routing and DNS > Jun 01 06:53:09 server.example.com systemd[1]: Current command > vanished from the unit file, execution of the command list won't be > resumed. > Jun 01 06:53:23 server.example.com systemd[1]: > NetworkManager.service: main process exited, code=killed, > status=9/KILL > Jun 01 06:53:23 server.example.com systemd[1]: Unit > NetworkManager.service entered failed state. > Jun 01 06:53:23 server.example.com systemd[1]: NetworkManager.service > failed. > > Before killing NM, it was flooding out RSes and after killing it it > stopped. Thanks, that's of course a string indication that NetworkManager is the culprit. But I don't see how. NetworkManager uses libndp for NDP. There is only one place where it sends RS, and thereby it also should log a debug message: [1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/blob/8964dbe8bc9cbe7300a48bffe86faee6b149fbf2/src/ndisc/nm-ndisc.c#L555 I did not see these messages... Just to check, did you enable level=TRACE logging and made sure that a possible burst of logging messages is not rate limited by systemd-journald? See the comments at [2] https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/contrib/fedora/rpm/NetworkManager.conf#n28 about that. Then, in the logfile I presume you do not see the messages about "router solicitation". So, it's unclear how these messages get sent. Maybe the issue is in libndp, but that should also just send one message [3] https://github.com/jpirko/libndp/blob/98bdaa1cb94faff0ccf992abc40a352ea16640fa/libndp/libndp.c#L195 ... unless kernel wrongly fails with errno EINTR, which seems unlikely. It would be interesting to see if this goes away by setting "errno = 0;" right before sendto() in [3]. No ideas so far. How is the CPU usage of NetworkManager when this happens? best, Thomas > > > If you kill NetworkManager with SIGKILL (without letting > > NetworkManager > > restart), it would not give NetworkManager time to do anything. > > If that stops the flodding, then the messages were sent by > > NetworkManager -- otherwise not. > > Indeed. > > > Thanks. It would be most interesting to see them at the moment when > > the > > flodding happen.s > > Damn. I should have gathered these before killing NM because now > that > I have and have restarted it, it's not flooding again. > > I will update here the sysctl content when I see it flooding again. > > Cheers, > b. > > ___ > networkmanager-list mailing list > networkmanager-list@gnome.org >
Re: spamming router with router solicitations
On Thu, 2019-05-30 at 12:23 +0200, Thomas Haller via networkmanager- list wrote: > On Wed, 2019-05-29 at 12:23 -0400, Brian J. Murrell wrote: > > On Wed, 2019-05-29 at 09:28 +0200, Thomas Haller via > > networkmanager- > > list wrote: > > > If this is happening, when you kill NetworkManager with SIGKILL, > > > it > > > would not give NetworkManager to cleanup... > > > > > > sudo killall -SIGKILL NetworkManager > > > > > > (and veryify that NetworkManager is indeed not running. Maybe > > > first > > > `systemctl mask NetworkManager`, so that systemd won't restart > > > it). > > > > Are you suggesting that I kill NM with SIGKILL as a debugging step > > to > > see if the RSes still stop when stopping NM? > > Yes. > > > If so, that was an interesting experiment: > > > > # killall -SIGKILL NetworkManager > > # ps -ef | grep Network > > root 15001 1 0 12:02 ?00:00:00 > > /usr/sbin/NetworkManager --no-daemon > > > > So clearly systemd restarted it, but it's not flooding RSes after > > the > > restart. > > Ah ok. That's what I meant with first masking NetworkManager. # systemctl mask NetworkManager Created symlink from /etc/systemd/system/NetworkManager.service to /dev/null. # killall -SIGKILL NetworkManager # systemctl status NetworkManager ● NetworkManager.service Loaded: masked (/dev/null; bad) Active: failed (Result: signal) since Sat 2019-06-01 06:53:23 EDT; 39s ago Main PID: 4286 (code=killed, signal=KILL) May 31 22:04:18 server.example.com NetworkManager[4286]: [1559354658.5223] manager: NetworkManager state is now CONNECTED_SITE May 31 22:04:18 server.example.com NetworkManager[4286]: [1559354658.5274] policy: set 'enp2s0' (enp2s0) as default for IPv6 routing and DNS May 31 22:04:18 server.example.com NetworkManager[4286]: [1559354658.5670] device (enp2s0): Activation: successful, device activated. May 31 22:04:18 server.example.com NetworkManager[4286]: [1559354658.5807] manager: NetworkManager state is now CONNECTED_GLOBAL May 31 22:04:18 server.example.com NetworkManager[4286]: [1559354658.5934] manager: startup complete May 31 22:04:33 server.example.com NetworkManager[4286]: [1559354673.1941] policy: set 'enp2s0' (enp2s0) as default for IPv4 routing and DNS Jun 01 06:53:09 server.example.com systemd[1]: Current command vanished from the unit file, execution of the command list won't be resumed. Jun 01 06:53:23 server.example.com systemd[1]: NetworkManager.service: main process exited, code=killed, status=9/KILL Jun 01 06:53:23 server.example.com systemd[1]: Unit NetworkManager.service entered failed state. Jun 01 06:53:23 server.example.com systemd[1]: NetworkManager.service failed. Before killing NM, it was flooding out RSes and after killing it it stopped. > If you kill NetworkManager with SIGKILL (without letting > NetworkManager > restart), it would not give NetworkManager time to do anything. > If that stops the flodding, then the messages were sent by > NetworkManager -- otherwise not. Indeed. > Thanks. It would be most interesting to see them at the moment when > the > flodding happen.s Damn. I should have gathered these before killing NM because now that I have and have restarted it, it's not flooding again. I will update here the sysctl content when I see it flooding again. Cheers, b. signature.asc Description: This is a digitally signed message part ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Wed, 2019-05-29 at 12:23 -0400, Brian J. Murrell wrote: > On Wed, 2019-05-29 at 09:28 +0200, Thomas Haller via networkmanager- > list wrote: > > If this is happening, when you kill NetworkManager with SIGKILL, it > > would not give NetworkManager to cleanup... > > > > sudo killall -SIGKILL NetworkManager > > > > (and veryify that NetworkManager is indeed not running. Maybe first > > `systemctl mask NetworkManager`, so that systemd won't restart it). > > Are you suggesting that I kill NM with SIGKILL as a debugging step to > see if the RSes still stop when stopping NM? Yes. > If so, that was an interesting experiment: > > # killall -SIGKILL NetworkManager > # ps -ef | grep Network > root 15001 1 0 12:02 ?00:00:00 > /usr/sbin/NetworkManager --no-daemon > > So clearly systemd restarted it, but it's not flooding RSes after the > restart. Ah ok. That's what I meant with first masking NetworkManager. > I even waited 17 minutes and still no flooding. I want to > leave it like this in case there is anything you want me to > investigate > with it. After restart, NetworkManager will reset all kinds of sysctls and reconfigure the link. So, it's not clear why the flodding stopped. It would be interesting whether those RS were sent by NetworkManager itself (or somebody else, like kernel). If you kill NetworkManager with SIGKILL (without letting NetworkManager restart), it would not give NetworkManager time to do anything. If that stops the flodding, then the messages were sent by NetworkManager -- otherwise not. > Otherwise I will continue on with the second part of the experiment > where I mask the service in systemd and kill it with a KILL and see > what happens. > > > grep -R ^ /proc/sys/net/ipv6/conf/ Thanks. It would be most interesting to see them at the moment when the flodding happen.s > > Currently that's: > > /proc/sys/net/ipv6/conf/all/accept_dad:0 > /proc/sys/net/ipv6/conf/all/accept_ra:1 > /proc/sys/net/ipv6/conf/all/accept_ra_defrtr:1 > /proc/sys/net/ipv6/conf/all/accept_ra_pinfo:1 > /proc/sys/net/ipv6/conf/all/accept_ra_rt_info_max_plen:0 > /proc/sys/net/ipv6/conf/all/accept_ra_rtr_pref:1 > /proc/sys/net/ipv6/conf/all/accept_redirects:1 > /proc/sys/net/ipv6/conf/all/accept_source_route:0 > /proc/sys/net/ipv6/conf/all/autoconf:1 > /proc/sys/net/ipv6/conf/all/dad_transmits:1 > /proc/sys/net/ipv6/conf/all/disable_ipv6:0 > /proc/sys/net/ipv6/conf/all/enhanced_dad:1 > /proc/sys/net/ipv6/conf/all/force_mld_version:0 > /proc/sys/net/ipv6/conf/all/force_tllao:0 > /proc/sys/net/ipv6/conf/all/forwarding:0 > /proc/sys/net/ipv6/conf/all/hop_limit:64 > /proc/sys/net/ipv6/conf/all/keep_addr_on_down:0 > /proc/sys/net/ipv6/conf/all/max_addresses:16 > /proc/sys/net/ipv6/conf/all/max_desync_factor:600 > /proc/sys/net/ipv6/conf/all/mc_forwarding:0 > /proc/sys/net/ipv6/conf/all/mldv1_unsolicited_report_interval:1 > /proc/sys/net/ipv6/conf/all/mldv2_unsolicited_report_interval:1000 > /proc/sys/net/ipv6/conf/all/mtu:1280 > /proc/sys/net/ipv6/conf/all/ndisc_notify:0 > /proc/sys/net/ipv6/conf/all/optimistic_dad:0 > /proc/sys/net/ipv6/conf/all/proxy_ndp:0 > /proc/sys/net/ipv6/conf/all/regen_max_retry:3 > /proc/sys/net/ipv6/conf/all/router_probe_interval:60 > /proc/sys/net/ipv6/conf/all/router_solicitation_delay:1 > /proc/sys/net/ipv6/conf/all/router_solicitation_interval:4 > /proc/sys/net/ipv6/conf/all/router_solicitations:3 > grep: /proc/sys/net/ipv6/conf/all/stable_secret: Input/output error > /proc/sys/net/ipv6/conf/all/temp_prefered_lft:86400 > /proc/sys/net/ipv6/conf/all/temp_valid_lft:604800 > /proc/sys/net/ipv6/conf/all/use_optimistic:0 > /proc/sys/net/ipv6/conf/all/use_tempaddr:0 > /proc/sys/net/ipv6/conf/default/accept_dad:1 > /proc/sys/net/ipv6/conf/default/accept_ra:1 > /proc/sys/net/ipv6/conf/default/accept_ra_defrtr:1 > /proc/sys/net/ipv6/conf/default/accept_ra_pinfo:1 > /proc/sys/net/ipv6/conf/default/accept_ra_rt_info_max_plen:0 > /proc/sys/net/ipv6/conf/default/accept_ra_rtr_pref:1 > /proc/sys/net/ipv6/conf/default/accept_redirects:1 > /proc/sys/net/ipv6/conf/default/accept_source_route:0 > /proc/sys/net/ipv6/conf/default/autoconf:1 > /proc/sys/net/ipv6/conf/default/dad_transmits:1 > /proc/sys/net/ipv6/conf/default/disable_ipv6:0 > /proc/sys/net/ipv6/conf/default/enhanced_dad:1 > /proc/sys/net/ipv6/conf/default/force_mld_version:0 > /proc/sys/net/ipv6/conf/default/force_tllao:0 > /proc/sys/net/ipv6/conf/default/forwarding:0 > /proc/sys/net/ipv6/conf/default/hop_limit:64 > /proc/sys/net/ipv6/conf/default/keep_addr_on_down:0 > /proc/sys/net/ipv6/conf/default/max_addresses:16 > /proc/sys/net/ipv6/conf/default/max_desync_factor:600 > /proc/sys/net/ipv6/conf/default/mc_forwarding:0 > /proc/sys/net/ipv6/conf/default/mldv1_unsolicited_report_interval:100 > 00 > /proc/sys/net/ipv6/conf/default/mldv2_unsolicited_report_interval:100 > 0 > /proc/sys/net/ipv6/conf/default/mtu:1280 > /proc/sys/net/ipv6/conf/default/ndisc_notify:0 >
Re: spamming router with router solicitations
On Wed, 2019-05-29 at 09:28 +0200, Thomas Haller via networkmanager- list wrote: > > If this is happening, when you kill NetworkManager with SIGKILL, it > would not give NetworkManager to cleanup... > > sudo killall -SIGKILL NetworkManager > > (and veryify that NetworkManager is indeed not running. Maybe first > `systemctl mask NetworkManager`, so that systemd won't restart it). Are you suggesting that I kill NM with SIGKILL as a debugging step to see if the RSes still stop when stopping NM? If so, that was an interesting experiment: # killall -SIGKILL NetworkManager # ps -ef | grep Network root 15001 1 0 12:02 ?00:00:00 /usr/sbin/NetworkManager --no-daemon So clearly systemd restarted it, but it's not flooding RSes after the restart. I even waited 17 minutes and still no flooding. I want to leave it like this in case there is anything you want me to investigate with it. Otherwise I will continue on with the second part of the experiment where I mask the service in systemd and kill it with a KILL and see what happens. > grep -R ^ /proc/sys/net/ipv6/conf/ Currently that's: /proc/sys/net/ipv6/conf/all/accept_dad:0 /proc/sys/net/ipv6/conf/all/accept_ra:1 /proc/sys/net/ipv6/conf/all/accept_ra_defrtr:1 /proc/sys/net/ipv6/conf/all/accept_ra_pinfo:1 /proc/sys/net/ipv6/conf/all/accept_ra_rt_info_max_plen:0 /proc/sys/net/ipv6/conf/all/accept_ra_rtr_pref:1 /proc/sys/net/ipv6/conf/all/accept_redirects:1 /proc/sys/net/ipv6/conf/all/accept_source_route:0 /proc/sys/net/ipv6/conf/all/autoconf:1 /proc/sys/net/ipv6/conf/all/dad_transmits:1 /proc/sys/net/ipv6/conf/all/disable_ipv6:0 /proc/sys/net/ipv6/conf/all/enhanced_dad:1 /proc/sys/net/ipv6/conf/all/force_mld_version:0 /proc/sys/net/ipv6/conf/all/force_tllao:0 /proc/sys/net/ipv6/conf/all/forwarding:0 /proc/sys/net/ipv6/conf/all/hop_limit:64 /proc/sys/net/ipv6/conf/all/keep_addr_on_down:0 /proc/sys/net/ipv6/conf/all/max_addresses:16 /proc/sys/net/ipv6/conf/all/max_desync_factor:600 /proc/sys/net/ipv6/conf/all/mc_forwarding:0 /proc/sys/net/ipv6/conf/all/mldv1_unsolicited_report_interval:1 /proc/sys/net/ipv6/conf/all/mldv2_unsolicited_report_interval:1000 /proc/sys/net/ipv6/conf/all/mtu:1280 /proc/sys/net/ipv6/conf/all/ndisc_notify:0 /proc/sys/net/ipv6/conf/all/optimistic_dad:0 /proc/sys/net/ipv6/conf/all/proxy_ndp:0 /proc/sys/net/ipv6/conf/all/regen_max_retry:3 /proc/sys/net/ipv6/conf/all/router_probe_interval:60 /proc/sys/net/ipv6/conf/all/router_solicitation_delay:1 /proc/sys/net/ipv6/conf/all/router_solicitation_interval:4 /proc/sys/net/ipv6/conf/all/router_solicitations:3 grep: /proc/sys/net/ipv6/conf/all/stable_secret: Input/output error /proc/sys/net/ipv6/conf/all/temp_prefered_lft:86400 /proc/sys/net/ipv6/conf/all/temp_valid_lft:604800 /proc/sys/net/ipv6/conf/all/use_optimistic:0 /proc/sys/net/ipv6/conf/all/use_tempaddr:0 /proc/sys/net/ipv6/conf/default/accept_dad:1 /proc/sys/net/ipv6/conf/default/accept_ra:1 /proc/sys/net/ipv6/conf/default/accept_ra_defrtr:1 /proc/sys/net/ipv6/conf/default/accept_ra_pinfo:1 /proc/sys/net/ipv6/conf/default/accept_ra_rt_info_max_plen:0 /proc/sys/net/ipv6/conf/default/accept_ra_rtr_pref:1 /proc/sys/net/ipv6/conf/default/accept_redirects:1 /proc/sys/net/ipv6/conf/default/accept_source_route:0 /proc/sys/net/ipv6/conf/default/autoconf:1 /proc/sys/net/ipv6/conf/default/dad_transmits:1 /proc/sys/net/ipv6/conf/default/disable_ipv6:0 /proc/sys/net/ipv6/conf/default/enhanced_dad:1 /proc/sys/net/ipv6/conf/default/force_mld_version:0 /proc/sys/net/ipv6/conf/default/force_tllao:0 /proc/sys/net/ipv6/conf/default/forwarding:0 /proc/sys/net/ipv6/conf/default/hop_limit:64 /proc/sys/net/ipv6/conf/default/keep_addr_on_down:0 /proc/sys/net/ipv6/conf/default/max_addresses:16 /proc/sys/net/ipv6/conf/default/max_desync_factor:600 /proc/sys/net/ipv6/conf/default/mc_forwarding:0 /proc/sys/net/ipv6/conf/default/mldv1_unsolicited_report_interval:1 /proc/sys/net/ipv6/conf/default/mldv2_unsolicited_report_interval:1000 /proc/sys/net/ipv6/conf/default/mtu:1280 /proc/sys/net/ipv6/conf/default/ndisc_notify:0 /proc/sys/net/ipv6/conf/default/optimistic_dad:0 /proc/sys/net/ipv6/conf/default/proxy_ndp:0 /proc/sys/net/ipv6/conf/default/regen_max_retry:3 /proc/sys/net/ipv6/conf/default/router_probe_interval:60 /proc/sys/net/ipv6/conf/default/router_solicitation_delay:1 /proc/sys/net/ipv6/conf/default/router_solicitation_interval:4 /proc/sys/net/ipv6/conf/default/router_solicitations:3 grep: /proc/sys/net/ipv6/conf/default/stable_secret: Input/output error /proc/sys/net/ipv6/conf/default/temp_prefered_lft:86400 /proc/sys/net/ipv6/conf/default/temp_valid_lft:604800 /proc/sys/net/ipv6/conf/default/use_optimistic:0 /proc/sys/net/ipv6/conf/default/use_tempaddr:0 /proc/sys/net/ipv6/conf/enp2s0/accept_dad:1 /proc/sys/net/ipv6/conf/enp2s0/accept_ra:1 /proc/sys/net/ipv6/conf/enp2s0/accept_ra_defrtr:0 /proc/sys/net/ipv6/conf/enp2s0/accept_ra_pinfo:0 /proc/sys/net/ipv6/conf/enp2s0/accept_ra_rt_info_max_plen:0
Re: spamming router with router solicitations
On Mon, 2019-05-27 at 06:59 -0400, Brian J. Murrell wrote: > On Mon, 2019-05-27 at 10:16 +0200, Beniamino Galvani via > networkmanager-list wrote: > > Are you sure you don't have other daemons like systemd-networkd > > # repoquery -q systemd-networkd > systemd-networkd-0:219-62.el7_6.6.x86_64 > > # rpm -q systemd-networkd > package systemd-networkd is not installed > > > So, not even installed. > > > or > > something else active which are racing with NM? > > Fairly sure there is nothing else. If this is happening, when you kill NetworkManager with SIGKILL, it would not give NetworkManager to cleanup... sudo killall -SIGKILL NetworkManager (and veryify that NetworkManager is indeed not running. Maybe first `systemctl mask NetworkManager`, so that systemd won't restart it). If then the RS flood stops, we know it was sent by NetworkManager. Otherwise, it wasn't. For now, I would guess that kernel is sending these solicitations... if that's the case, the sysctl settings would be interesting: grep -R ^ /proc/sys/net/ipv6/conf/ best, Thomas signature.asc Description: This is a digitally signed message part ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Mon, 2019-05-27 at 10:16 +0200, Beniamino Galvani via networkmanager-list wrote: > > Are you sure you don't have other daemons like systemd-networkd # repoquery -q systemd-networkd systemd-networkd-0:219-62.el7_6.6.x86_64 # rpm -q systemd-networkd package systemd-networkd is not installed So, not even installed. > or > something else active which are racing with NM? Fairly sure there is nothing else. > Can you also attach the output of > > nmcli connection show enp2s0 Sure, happy to: connection.id: enp2s0 connection.uuid:90f5409c-bd23-48f2-b03f-325333198827 connection.stable-id: -- connection.type:802-3-ethernet connection.interface-name: enp2s0 connection.autoconnect: yes connection.autoconnect-priority:0 connection.autoconnect-retries: -1 (default) connection.auth-retries:-1 connection.timestamp: 1558954265 connection.read-only: no connection.permissions: -- connection.zone:public connection.master: -- connection.slave-type: -- connection.autoconnect-slaves: -1 (default) connection.secondaries: -- connection.gateway-ping-timeout:0 connection.metered: unknown connection.lldp:default connection.mdns:-1 (default) 802-3-ethernet.port:-- 802-3-ethernet.speed: 0 802-3-ethernet.duplex: -- 802-3-ethernet.auto-negotiate: no 802-3-ethernet.mac-address: -- 802-3-ethernet.cloned-mac-address: -- 802-3-ethernet.generate-mac-address-mask:-- 802-3-ethernet.mac-address-blacklist: -- 802-3-ethernet.mtu: auto 802-3-ethernet.s390-subchannels:-- 802-3-ethernet.s390-nettype:-- 802-3-ethernet.s390-options:-- 802-3-ethernet.wake-on-lan: default 802-3-ethernet.wake-on-lan-password:-- ipv4.method:manual ipv4.dns: 10.75.22.247 ipv4.dns-search:-- ipv4.dns-options: "" ipv4.dns-priority: 0 ipv4.addresses: 10.75.22.247/24, 10.75.22.5/24, 10.75.22.8/24, 10.75.22.9/24 ipv4.gateway: -- ipv4.routes:-- ipv4.route-metric: -1 ipv4.route-table: 0 (unspec) ipv4.ignore-auto-routes:no ipv4.ignore-auto-dns: no ipv4.dhcp-client-id:-- ipv4.dhcp-timeout: 0 (default) ipv4.dhcp-send-hostname:yes ipv4.dhcp-hostname: -- ipv4.dhcp-fqdn: -- ipv4.never-default: no ipv4.may-fail: yes ipv4.dad-timeout: -1 (default) ipv6.method:auto ipv6.dns: -- ipv6.dns-search:-- ipv6.dns-options: "" ipv6.dns-priority: 0 ipv6.addresses: fd31:aeb1:48df::2/64 ipv6.gateway: fe80::6eb0:ceff:fef5:1e4a ipv6.routes:-- ipv6.route-metric: -1 ipv6.route-table: 0 (unspec) ipv6.ignore-auto-routes:no ipv6.ignore-auto-dns: no ipv6.never-default: no ipv6.may-fail: no ipv6.ip6-privacy: -1 (unknown) ipv6.addr-gen-mode: eui64 ipv6.dhcp-duid: -- ipv6.dhcp-send-hostname:yes ipv6.dhcp-hostname: -- ipv6.token: ::2 proxy.method: none proxy.browser-only: no proxy.pac-url: -- proxy.pac-script: -- GENERAL.NAME: enp2s0 GENERAL.UUID: 90f5409c-bd23-48f2-b03f-325333198827 GENERAL.DEVICES:enp2s0 GENERAL.STATE: activated GENERAL.DEFAULT:yes GENERAL.DEFAULT6: yes GENERAL.SPEC-OBJECT:-- GENERAL.VPN:no GENERAL.DBUS-PATH: /org/freedesktop/NetworkManager/ActiveConnection/1 GENERAL.CON-PATH: /org/freedesktop/NetworkManager/Settings/1 GENERAL.ZONE: public GENERAL.MASTER-PATH:-- IP4.ADDRESS[1]: 10.75.22.247/24 IP4.ADDRESS[2]: 10.75.22.5/24 IP4.ADDRESS[3]:
Re: spamming router with router solicitations
On Sat, May 25, 2019 at 03:47:37PM -0400, Brian J. Murrell wrote: > On Sat, 2019-05-25 at 08:52 +0200, Beniamino Galvani via > networkmanager-list wrote: > > > > Hi, it is not NM sending those RS. > > But they stop as soon as I stop NM. > > > If it were, you would see in the > > journal messages like: > > > > ndisc[0x55a43c15e0e0,"ens9"]: router solicitation sent > > I'm not in a position to argue that point, but if the storm can be > stopped and started by stopping and starting NM, that's pretty strong > evidence that NM is doing it, wouldn't you agree? Are you sure you don't have other daemons like systemd-networkd or something else active which are racing with NM? From logs it is pretty clear that NM is sending exactly one router solicitation at: May 25 15:42:55 server.example.com NetworkManager[10973]: [1558813375.1729] ndisc[0x55a336c550e0,"enp2s0"]: router solicitation sent However it could be that NM is configuring the kernel in a wrong way. > > I see from the log that you set a IPv6 token; can you please try > > without it if it makes any difference? > > But that machine needs to have a predictable, easily remembered > address, which it does not get without a token. > > In any case, as a debugging step, I did disable it and restart NM and > the storm did not start up with it. When I re-enabled the token again > and restarted NM, the storm started when it started, so it does seem to > be related to using a token. > > > Also, please attach a log from > > the beginning of the profile activation. > > There is no "profile activation" because this is a headless server, > hence the reason for the token. I'll attach the NM log from the time > it is started up. Hopefully that provides what you are looking for. Yes, it is ok. Note that you can force a new activation of a connection with: nmcli connection up $con_name where $con_name if the name of the connection obtained through: nmcli connection show In your case the connection name is equal to the device name - enp2s0. The reactivation is useful to apply changes after modifying a connection without the need to restart NM or reboot. Can you also attach the output of nmcli connection show enp2s0 ? Beniamino signature.asc Description: PGP signature ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Sun, May 12, 2019 at 10:41:28AM -0400, Brian J. Murrell wrote: > On Sat, 2019-05-11 at 17:46 -0500, Ian Pilcher via networkmanager-list > wrote: > > > > Presumably you've told NetworkManager to auto-configure the interface > > for IPv6. If you don't want it to do that, set it to link-local > > only. > > But surely NM doesn't need to send dozens of RS's per second to do > that, yes? Hi, it is not NM sending those RS. If it were, you would see in the journal messages like: ndisc[0x55a43c15e0e0,"ens9"]: router solicitation sent I see from the log that you set a IPv6 token; can you please try without it if it makes any difference? Also, please attach a log from the beginning of the profile activation. Thanks, Beniamino signature.asc Description: PGP signature ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
Hi Brian, Am Do., 23. Mai 2019 um 14:46 Uhr schrieb Brian J. Murrell : > > On Thu, 2019-05-09 at 18:25 -0400, Brian J. Murrell wrote: > > NetworkManager 1.12 on EL7.6 is spamming my router with IPv6 router > > solicitations: > > Nobody has any idea about this or what more can be done to debug it? You can try if it is fixed in a never version. You can find builds from latest GIT branches for EL 7 here: https://copr.fedorainfracloud.org/coprs/networkmanager/ Maybe it is already fixed in a newer version. Kind regards Till -- Till Maas Ansible RHEL Networking System Role Maintainer Red Hat GmbH Red Hat GmbH, https://www.redhat.com/de, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael O'Neill, Tom Savage, Eric Shander ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On Sat, 2019-05-11 at 17:46 -0500, Ian Pilcher via networkmanager-list wrote: > > Presumably you've told NetworkManager to auto-configure the interface > for IPv6. If you don't want it to do that, set it to link-local > only. But surely NM doesn't need to send dozens of RS's per second to do that, yes? b. signature.asc Description: This is a digitally signed message part ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
Re: spamming router with router solicitations
On 5/9/19 5:25 PM, Brian J. Murrell wrote: Any idea what's causing it? Presumably you've told NetworkManager to auto-configure the interface for IPv6. If you don't want it to do that, set it to link-local only. -- Ian Pilcher arequip...@gmail.com "I grew up before Mark Zuckerberg invented friendship" ___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
