Re: ntpd segfaults on start
Cy, On 9/6/2019 4:56 PM, Cy Schubert wrote: > ... > > For those who enable ASLR, a better workaround is, to add this to your > ntp.conf: > > rlimit memlock 64 > > Until a more precise default is determined. Should I change the default value for FreeBSD-12 to be 64 for now? I can get this change in place for the upcoming ntp-4.2.8p14 release, and we can change it later if needed. -- Harlan Stenn, Network Time Foundation http://nwtime.org - be a Member! ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
On Mon, Sep 09, 2019 at 03:42:43PM -0600, Warner Losh wrote: > On Mon, Sep 9, 2019 at 3:12 PM Ian Lepore wrote: > > > On Mon, 2019-09-09 at 21:44 +0300, Konstantin Belousov wrote: > > > On Mon, Sep 09, 2019 at 12:13:24PM -0600, Ian Lepore wrote: > > > > On Mon, 2019-09-09 at 09:30 -0700, Rodney W. Grimes wrote: > > > > > > On Sat, 2019-09-07 at 09:28 -0700, Cy Schubert wrote: > > > > > > > In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin > > > > > > > Belousov writes: > > > > > > > > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert > > > > > > > > wrote: > > > > > > > > > [...] > > > > > > > > > > Doesn't locking this memory down also protect ntpd from OOM kills? > > > > > If so that is a MUST preserve functionality, as IMHO killing ntpd > > > > > on a box that has it configured is a total no win situation. > > > > > > > > > > > > > Does it have that effect? I don't know. But I would argue that that's > > > > a separate issue, and we should make that happen by adding > > > > ntpd_oomprotect=YES to /etc/defaults/rc.conf > > > > > > Wiring process memory has no effect on OOM selection. More, because > > > all potentially allocated pages are allocated for real after mlockall(), > > > the size of the vmspace, as accounted by OOM, is the largest possible > > > size from the whole lifetime. > > > > > > On the other hand, the code execution times are not predictable if the > > > process's pages can be paged out. Under severe load next instruction > > > might take several seconds or even minutes to start. It is quite unlike > > > the scheduler delays. That introduces a jitter in the local time > > > measurements and their usage as done in userspace. Wouldn't this affect > > > the accuracy ? > > > > > > > IMO, there is a large gap between "in theory, paging could cause > > indeterminate delays in code execution" and "time will be inaccurate on > > your system". If there were a delay in a part of the code where it > > matters that amounted to "seconds or even minutes", what you'd end up > > with is a measurement that would be discarded by the median filter as > > an outlier. There would be some danger that if that kind of delay > > happened for too many polling cycles in a row, you'd end up with no > > usable measurements after a while and clock accuracy would suffer. > > Sub-second delays would be more worriesome because they might not be > > rejected as outliers. > > > > There are only a couple code paths in freebsd ntpd processing where a > > paging (or scheduling) delay could cause measurement inaccuracy: > > > > - When stepping the clock, the code that runs between calling > > clock_gettime() and calling clock_settime() to apply the step > > adjustment to the clock. > > > > - When beginning an exchange with or replying to a peer, the code that > > runs between obtaining system time for the outgoing Transmit Timestamp > > and actually transmitting that packet. > > > > Stepping the clock typically only happens once at startup. The ntpd > > code itself recognizes that this is a time-critical path (it has > > comments to that effect) but unfortunately the code that runs is > > scattered among several different .c files so it's hard to say what the > > likelyhood is that code in the critical section will all be in the same > > page (or be already-resident because other startup-time code faulted in > > those pages). IMO, the right fix for this would be a kernel interface > > that let you apply a step-delta to the clock with a single syscall > > (perhaps as an extension to the existing ntp_adjtime() using a new mode > > flag). > > > > On freebsd, the Receive timestamps are captured in the kernel and > > delivered along with the packet to userland, and are retrieved by the > > ntpd code from the SCM_BINTIME control message in the packet, so there > > is no latency problem in the receive path. There isn't a corresponding > > kernel mechanism for setting the outgoing timestamps, so whether it's > > originating a request to a peer or replying to a request from a peer, > > the transmit timestamp could be wrong due to: > > > > - paging delays > > - scheduler delays > > - network stack, outgoing queues, and driver delays > > > > So the primary vulnerability is on the transmit path between obtaining > > system time and the packet leaving the system. A quick glance at that > > code makes me think that most of the data being touched has already > > been referenced pretty recently during the process of assembling the > > outgoing packet, so it's unlikely that storing the timestamp into the > > outgoing packet or the other bit of work that happens after that > > triggers a pagein unless the system is pathologically overloaded. > > Naturally, obtaining the timestamp and putting it into the packet is > > one of the last things it does before sending, so the code path is > > relatively short, but it's not clear to me whether it's likely or not > > that the code involved all lives in the same page.
Re: ntpd segfaults on start
On Mon, Sep 9, 2019 at 3:12 PM Ian Lepore wrote: > On Mon, 2019-09-09 at 21:44 +0300, Konstantin Belousov wrote: > > On Mon, Sep 09, 2019 at 12:13:24PM -0600, Ian Lepore wrote: > > > On Mon, 2019-09-09 at 09:30 -0700, Rodney W. Grimes wrote: > > > > > On Sat, 2019-09-07 at 09:28 -0700, Cy Schubert wrote: > > > > > > In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin > > > > > > Belousov writes: > > > > > > > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert > > > > > > > wrote: > > > > > > > > [...] > > > > > > > > Doesn't locking this memory down also protect ntpd from OOM kills? > > > > If so that is a MUST preserve functionality, as IMHO killing ntpd > > > > on a box that has it configured is a total no win situation. > > > > > > > > > > Does it have that effect? I don't know. But I would argue that that's > > > a separate issue, and we should make that happen by adding > > > ntpd_oomprotect=YES to /etc/defaults/rc.conf > > > > Wiring process memory has no effect on OOM selection. More, because > > all potentially allocated pages are allocated for real after mlockall(), > > the size of the vmspace, as accounted by OOM, is the largest possible > > size from the whole lifetime. > > > > On the other hand, the code execution times are not predictable if the > > process's pages can be paged out. Under severe load next instruction > > might take several seconds or even minutes to start. It is quite unlike > > the scheduler delays. That introduces a jitter in the local time > > measurements and their usage as done in userspace. Wouldn't this affect > > the accuracy ? > > > > IMO, there is a large gap between "in theory, paging could cause > indeterminate delays in code execution" and "time will be inaccurate on > your system". If there were a delay in a part of the code where it > matters that amounted to "seconds or even minutes", what you'd end up > with is a measurement that would be discarded by the median filter as > an outlier. There would be some danger that if that kind of delay > happened for too many polling cycles in a row, you'd end up with no > usable measurements after a while and clock accuracy would suffer. > Sub-second delays would be more worriesome because they might not be > rejected as outliers. > > There are only a couple code paths in freebsd ntpd processing where a > paging (or scheduling) delay could cause measurement inaccuracy: > > - When stepping the clock, the code that runs between calling > clock_gettime() and calling clock_settime() to apply the step > adjustment to the clock. > > - When beginning an exchange with or replying to a peer, the code that > runs between obtaining system time for the outgoing Transmit Timestamp > and actually transmitting that packet. > > Stepping the clock typically only happens once at startup. The ntpd > code itself recognizes that this is a time-critical path (it has > comments to that effect) but unfortunately the code that runs is > scattered among several different .c files so it's hard to say what the > likelyhood is that code in the critical section will all be in the same > page (or be already-resident because other startup-time code faulted in > those pages). IMO, the right fix for this would be a kernel interface > that let you apply a step-delta to the clock with a single syscall > (perhaps as an extension to the existing ntp_adjtime() using a new mode > flag). > > On freebsd, the Receive timestamps are captured in the kernel and > delivered along with the packet to userland, and are retrieved by the > ntpd code from the SCM_BINTIME control message in the packet, so there > is no latency problem in the receive path. There isn't a corresponding > kernel mechanism for setting the outgoing timestamps, so whether it's > originating a request to a peer or replying to a request from a peer, > the transmit timestamp could be wrong due to: > > - paging delays > - scheduler delays > - network stack, outgoing queues, and driver delays > > So the primary vulnerability is on the transmit path between obtaining > system time and the packet leaving the system. A quick glance at that > code makes me think that most of the data being touched has already > been referenced pretty recently during the process of assembling the > outgoing packet, so it's unlikely that storing the timestamp into the > outgoing packet or the other bit of work that happens after that > triggers a pagein unless the system is pathologically overloaded. > Naturally, obtaining the timestamp and putting it into the packet is > one of the last things it does before sending, so the code path is > relatively short, but it's not clear to me whether it's likely or not > that the code involved all lives in the same page. Still, it's one of > the heavily exercised paths within ntpd, which should increase the odds > of the pages being resident because of recent use. > > So, I'm not disputing the point that a sufficiently overloaded system > can lead to an
Re: ntpd segfaults on start
On Mon, 2019-09-09 at 12:28 -0700, Cy Schubert wrote: > > > > On the other hand, the code execution times are not predictable if the > > process's pages can be paged out. Under severe load next instruction > > might take several seconds or even minutes to start. It is quite unlike > > the scheduler delays. That introduces a jitter in the local time > > measurements and their usage as done in userspace. Wouldn't this affect > > the accuracy ? > > IMO it would and would be unacceptable when used as a stratum N server or > with some OLTP or database applications. Locking a few megabytes is a small > cost. What I proposed was changing the default to not lock all of ntpd into memory, because I think that would work well for most users. Admins running stratum 1 or 2 servers are probably a bit more savvy about ntp configuration than the average user, and would use the rlimit command in ntp.conf. -- Ian ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
On Mon, 2019-09-09 at 21:44 +0300, Konstantin Belousov wrote: > On Mon, Sep 09, 2019 at 12:13:24PM -0600, Ian Lepore wrote: > > On Mon, 2019-09-09 at 09:30 -0700, Rodney W. Grimes wrote: > > > > On Sat, 2019-09-07 at 09:28 -0700, Cy Schubert wrote: > > > > > In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin > > > > > Belousov writes: > > > > > > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert > > > > > > wrote: > > > > > > > [...] > > > > > > Doesn't locking this memory down also protect ntpd from OOM kills? > > > If so that is a MUST preserve functionality, as IMHO killing ntpd > > > on a box that has it configured is a total no win situation. > > > > > > > Does it have that effect? I don't know. But I would argue that that's > > a separate issue, and we should make that happen by adding > > ntpd_oomprotect=YES to /etc/defaults/rc.conf > > Wiring process memory has no effect on OOM selection. More, because > all potentially allocated pages are allocated for real after mlockall(), > the size of the vmspace, as accounted by OOM, is the largest possible > size from the whole lifetime. > > On the other hand, the code execution times are not predictable if the > process's pages can be paged out. Under severe load next instruction > might take several seconds or even minutes to start. It is quite unlike > the scheduler delays. That introduces a jitter in the local time > measurements and their usage as done in userspace. Wouldn't this affect > the accuracy ? > IMO, there is a large gap between "in theory, paging could cause indeterminate delays in code execution" and "time will be inaccurate on your system". If there were a delay in a part of the code where it matters that amounted to "seconds or even minutes", what you'd end up with is a measurement that would be discarded by the median filter as an outlier. There would be some danger that if that kind of delay happened for too many polling cycles in a row, you'd end up with no usable measurements after a while and clock accuracy would suffer. Sub-second delays would be more worriesome because they might not be rejected as outliers. There are only a couple code paths in freebsd ntpd processing where a paging (or scheduling) delay could cause measurement inaccuracy: - When stepping the clock, the code that runs between calling clock_gettime() and calling clock_settime() to apply the step adjustment to the clock. - When beginning an exchange with or replying to a peer, the code that runs between obtaining system time for the outgoing Transmit Timestamp and actually transmitting that packet. Stepping the clock typically only happens once at startup. The ntpd code itself recognizes that this is a time-critical path (it has comments to that effect) but unfortunately the code that runs is scattered among several different .c files so it's hard to say what the likelyhood is that code in the critical section will all be in the same page (or be already-resident because other startup-time code faulted in those pages). IMO, the right fix for this would be a kernel interface that let you apply a step-delta to the clock with a single syscall (perhaps as an extension to the existing ntp_adjtime() using a new mode flag). On freebsd, the Receive timestamps are captured in the kernel and delivered along with the packet to userland, and are retrieved by the ntpd code from the SCM_BINTIME control message in the packet, so there is no latency problem in the receive path. There isn't a corresponding kernel mechanism for setting the outgoing timestamps, so whether it's originating a request to a peer or replying to a request from a peer, the transmit timestamp could be wrong due to: - paging delays - scheduler delays - network stack, outgoing queues, and driver delays So the primary vulnerability is on the transmit path between obtaining system time and the packet leaving the system. A quick glance at that code makes me think that most of the data being touched has already been referenced pretty recently during the process of assembling the outgoing packet, so it's unlikely that storing the timestamp into the outgoing packet or the other bit of work that happens after that triggers a pagein unless the system is pathologically overloaded. Naturally, obtaining the timestamp and putting it into the packet is one of the last things it does before sending, so the code path is relatively short, but it's not clear to me whether it's likely or not that the code involved all lives in the same page. Still, it's one of the heavily exercised paths within ntpd, which should increase the odds of the pages being resident because of recent use. So, I'm not disputing the point that a sufficiently overloaded system can lead to an indeterminate delay between *any* two instructions executed in userland. What I've said above is more along the lines of considering the usual situation, not the most pathlogical one. In the most pathological cases, either
Re: ntpd segfaults on start
In message <20190909184446.gu2...@kib.kiev.ua>, Konstantin Belousov writes: > On Mon, Sep 09, 2019 at 12:13:24PM -0600, Ian Lepore wrote: > > On Mon, 2019-09-09 at 09:30 -0700, Rodney W. Grimes wrote: > > > > On Sat, 2019-09-07 at 09:28 -0700, Cy Schubert wrote: > > > > > In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin > > > > > Belousov writes: > > > > > > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert wrote: > > > > > > > I've been able to set the memlock rlimit as low as 20 MB. The > > > > > > > issue is > > > > > > > letting it default to 0 which allows ntp to mlockall() > > > > > > > anything it wants. > > > > > > > ntpd on my sandbox is currently using 18 MB. > > > > > > > > > > > > Default stack size on amd64 is 512M, and default stack gap > > > > > > percentage is > > > > > > 3%. This means that the gap can be as large as ~17MB. If 3MB is > > > > > > enough > > > > > > for the stack of the main thread of ntpd, then fine. > > > > > > > > > > The default stack is 200K, which is also tuneable in ntp.conf. > > > > > > > > > > [...] > > > > > > > > I haven't seen anyone ask what I consider to be the crucial > > > > question > > > > yet: why are we locking ntpd into memory by default at all? > > > > > > > > I have seen two rationales for ntpd using mlockall() and > > > > setrlimit(): > > > > > > > >- There are claims that it improves timing performance. > > > > > > > >- Because ntpd is a daemon that can run for months at a time, > > > >setting limits on memory and stack growth can help detect and > > > >mitigate against memory leak problems in the daemon. > > > > > > Doesn't locking this memory down also protect ntpd from OOM kills? > > > If so that is a MUST preserve functionality, as IMHO killing ntpd > > > on a box that has it configured is a total no win situation. > > > > > > > Does it have that effect? I don't know. But I would argue that that's > > a separate issue, and we should make that happen by adding > > ntpd_oomprotect=YES to /etc/defaults/rc.conf > Wiring process memory has no effect on OOM selection. More, because > all potentially allocated pages are allocated for real after mlockall(), > the size of the vmspace, as accounted by OOM, is the largest possible > size from the whole lifetime. > > On the other hand, the code execution times are not predictable if the > process's pages can be paged out. Under severe load next instruction > might take several seconds or even minutes to start. It is quite unlike > the scheduler delays. That introduces a jitter in the local time > measurements and their usage as done in userspace. Wouldn't this affect > the accuracy ? IMO it would and would be unacceptable when used as a stratum N server or with some OLTP or database applications. Locking a few megabytes is a small cost. > > > > > Right now only syslogd has oomprotect set to YES by default. Maybe > > that's a good choice -- once we start declaring one daemon to be more > > important than others, you'll discover there's a whole back lot full of > > bikesheds that need painting. > > > > So maybe we should just document ntpd_oomprotect=YES in some more- > > prominent way. If we were to add a comment block to ntp.conf > > describing rlimit, that might be a good place to mention setting > > ntpd_oomprotect in rc.conf. > > > > -- Ian > > -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
In message <66f80012757134b6317b673f9eeb24db66c996a2.ca...@freebsd.org>, Ian Le pore writes: > On Sat, 2019-09-07 at 09:28 -0700, Cy Schubert wrote: > > In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin Belousov writes: > > > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert wrote: > > > > I've been able to set the memlock rlimit as low as 20 MB. The issue is > > > > letting it default to 0 which allows ntp to mlockall() anything it want > s. > > > > ntpd on my sandbox is currently using 18 MB. > > > > > > Default stack size on amd64 is 512M, and default stack gap percentage is > > > 3%. This means that the gap can be as large as ~17MB. If 3MB is enough > > > for the stack of the main thread of ntpd, then fine. > > > > The default stack is 200K, which is also tuneable in ntp.conf. > > > > [...] > > I haven't seen anyone ask what I consider to be the crucial question > yet: why are we locking ntpd into memory by default at all? > > I have seen two rationales for ntpd using mlockall() and setrlimit(): > >- There are claims that it improves timing performance. > >- Because ntpd is a daemon that can run for months at a time, >setting limits on memory and stack growth can help detect and >mitigate against memory leak problems in the daemon. > > I am *highly* skeptical of claims that locking ntpd in memory will > improve timing performance on freebsd (to the point where I'm inclined > to dismiss them out of hand, but I'd be willing to look at any actual > evidence). > > The second point has at least some validity, but would be better > implemented (IMO) by decoupling the address space limit from the act of > locking down memory, and allowing configuration of RLIMIT_AS separately > from RLIMIT_MEMLOCK. If there's any interest in this, I could try to > put together a patchset and submit it upstream for that. Our upstream is already cc'd on this thread. diff --git a/usr.sbin/ntp/config.h b/usr.sbin/ntp/config.h index 56dbfedba6e..b26dd417885 100644 --- a/usr.sbin/ntp/config.h +++ b/usr.sbin/ntp/config.h @@ -287,7 +287,7 @@ #define DEFAULT_HZ 100 /* Default number of megabytes for RLIMIT_MEMLOCK */ -#define DFLT_RLIMIT_MEMLOCK 32 +#define DFLT_RLIMIT_MEMLOCK -1 /* Default number of 4k pages for RLIMIT_STACK */ #define DFLT_RLIMIT_STACK 50 But I'd wait for Harlan to weigh in on this. I agree with kib@ that this may introduce unwanted jitter. I'd also want to understand why this defaults to -1 for Linux only. > > I would propose that for freebsd, the autoconf-generated value for For the port but in base we control this directly. > DFLT_RLIMIT_MEMLOCK should be -1 to avoid calling setrlimit() and > mlockall() by default. Then in the ntp.conf we distribute we have a > section like: > ># To lock ntpd into memory, uncomment the following rlimit line. ># This should not be necessary on most systems, but may improve ># performance on heavily-loaded servers experiencing enough ># memory pressure to cause ntpd to be paged out. ># rlimit memlock stacksize > > Then we would need to come up with reasonable values for . I haven't had a chance to look at this in depth yet but I suspect that isn't in fact 32 as specified in config.h. It behaves as if this is set to 0 to mlockall() all it asks for. > > -- Ian -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
On Mon, Sep 09, 2019 at 12:13:24PM -0600, Ian Lepore wrote: > On Mon, 2019-09-09 at 09:30 -0700, Rodney W. Grimes wrote: > > > On Sat, 2019-09-07 at 09:28 -0700, Cy Schubert wrote: > > > > In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin > > > > Belousov writes: > > > > > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert wrote: > > > > > > I've been able to set the memlock rlimit as low as 20 MB. The > > > > > > issue is > > > > > > letting it default to 0 which allows ntp to mlockall() > > > > > > anything it wants. > > > > > > ntpd on my sandbox is currently using 18 MB. > > > > > > > > > > Default stack size on amd64 is 512M, and default stack gap > > > > > percentage is > > > > > 3%. This means that the gap can be as large as ~17MB. If 3MB is > > > > > enough > > > > > for the stack of the main thread of ntpd, then fine. > > > > > > > > The default stack is 200K, which is also tuneable in ntp.conf. > > > > > > > > [...] > > > > > > I haven't seen anyone ask what I consider to be the crucial > > > question > > > yet: why are we locking ntpd into memory by default at all? > > > > > > I have seen two rationales for ntpd using mlockall() and > > > setrlimit(): > > > > > >- There are claims that it improves timing performance. > > > > > >- Because ntpd is a daemon that can run for months at a time, > > >setting limits on memory and stack growth can help detect and > > >mitigate against memory leak problems in the daemon. > > > > Doesn't locking this memory down also protect ntpd from OOM kills? > > If so that is a MUST preserve functionality, as IMHO killing ntpd > > on a box that has it configured is a total no win situation. > > > > Does it have that effect? I don't know. But I would argue that that's > a separate issue, and we should make that happen by adding > ntpd_oomprotect=YES to /etc/defaults/rc.conf Wiring process memory has no effect on OOM selection. More, because all potentially allocated pages are allocated for real after mlockall(), the size of the vmspace, as accounted by OOM, is the largest possible size from the whole lifetime. On the other hand, the code execution times are not predictable if the process's pages can be paged out. Under severe load next instruction might take several seconds or even minutes to start. It is quite unlike the scheduler delays. That introduces a jitter in the local time measurements and their usage as done in userspace. Wouldn't this affect the accuracy ? > > Right now only syslogd has oomprotect set to YES by default. Maybe > that's a good choice -- once we start declaring one daemon to be more > important than others, you'll discover there's a whole back lot full of > bikesheds that need painting. > > So maybe we should just document ntpd_oomprotect=YES in some more- > prominent way. If we were to add a comment block to ntp.conf > describing rlimit, that might be a good place to mention setting > ntpd_oomprotect in rc.conf. > > -- Ian > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
> On Mon, 2019-09-09 at 09:30 -0700, Rodney W. Grimes wrote: > > > On Sat, 2019-09-07 at 09:28 -0700, Cy Schubert wrote: > > > > In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin > > > > Belousov writes: > > > > > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert wrote: > > > > > > I've been able to set the memlock rlimit as low as 20 MB. The > > > > > > issue is > > > > > > letting it default to 0 which allows ntp to mlockall() > > > > > > anything it wants. > > > > > > ntpd on my sandbox is currently using 18 MB. > > > > > > > > > > Default stack size on amd64 is 512M, and default stack gap > > > > > percentage is > > > > > 3%. This means that the gap can be as large as ~17MB. If 3MB is > > > > > enough > > > > > for the stack of the main thread of ntpd, then fine. > > > > > > > > The default stack is 200K, which is also tuneable in ntp.conf. > > > > > > > > [...] > > > > > > I haven't seen anyone ask what I consider to be the crucial > > > question > > > yet: why are we locking ntpd into memory by default at all? > > > > > > I have seen two rationales for ntpd using mlockall() and > > > setrlimit(): > > > > > >- There are claims that it improves timing performance. > > > > > >- Because ntpd is a daemon that can run for months at a time, > > >setting limits on memory and stack growth can help detect and > > >mitigate against memory leak problems in the daemon. > > > > Doesn't locking this memory down also protect ntpd from OOM kills? > > If so that is a MUST preserve functionality, as IMHO killing ntpd > > on a box that has it configured is a total no win situation. > > > > Does it have that effect? I don't know. But I would argue that that's > a separate issue, and we should make that happen by adding > ntpd_oomprotect=YES to /etc/defaults/rc.conf > > Right now only syslogd has oomprotect set to YES by default. Maybe > that's a good choice -- once we start declaring one daemon to be more > important than others, you'll discover there's a whole back lot full of > bikesheds that need painting. > > So maybe we should just document ntpd_oomprotect=YES in some more- > prominent way. If we were to add a comment block to ntp.conf > describing rlimit, that might be a good place to mention setting > ntpd_oomprotect in rc.conf. > > -- Ian All very reasonable, thanks Ian. -- Rod Grimes rgri...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
On Mon, 2019-09-09 at 09:30 -0700, Rodney W. Grimes wrote: > > On Sat, 2019-09-07 at 09:28 -0700, Cy Schubert wrote: > > > In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin > > > Belousov writes: > > > > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert wrote: > > > > > I've been able to set the memlock rlimit as low as 20 MB. The > > > > > issue is > > > > > letting it default to 0 which allows ntp to mlockall() > > > > > anything it wants. > > > > > ntpd on my sandbox is currently using 18 MB. > > > > > > > > Default stack size on amd64 is 512M, and default stack gap > > > > percentage is > > > > 3%. This means that the gap can be as large as ~17MB. If 3MB is > > > > enough > > > > for the stack of the main thread of ntpd, then fine. > > > > > > The default stack is 200K, which is also tuneable in ntp.conf. > > > > > > [...] > > > > I haven't seen anyone ask what I consider to be the crucial > > question > > yet: why are we locking ntpd into memory by default at all? > > > > I have seen two rationales for ntpd using mlockall() and > > setrlimit(): > > > >- There are claims that it improves timing performance. > > > >- Because ntpd is a daemon that can run for months at a time, > >setting limits on memory and stack growth can help detect and > >mitigate against memory leak problems in the daemon. > > Doesn't locking this memory down also protect ntpd from OOM kills? > If so that is a MUST preserve functionality, as IMHO killing ntpd > on a box that has it configured is a total no win situation. > Does it have that effect? I don't know. But I would argue that that's a separate issue, and we should make that happen by adding ntpd_oomprotect=YES to /etc/defaults/rc.conf Right now only syslogd has oomprotect set to YES by default. Maybe that's a good choice -- once we start declaring one daemon to be more important than others, you'll discover there's a whole back lot full of bikesheds that need painting. So maybe we should just document ntpd_oomprotect=YES in some more- prominent way. If we were to add a comment block to ntp.conf describing rlimit, that might be a good place to mention setting ntpd_oomprotect in rc.conf. -- Ian ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
> On Sat, 2019-09-07 at 09:28 -0700, Cy Schubert wrote: > > In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin Belousov writes: > > > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert wrote: > > > > I've been able to set the memlock rlimit as low as 20 MB. The issue is > > > > letting it default to 0 which allows ntp to mlockall() anything it > > > > wants. > > > > ntpd on my sandbox is currently using 18 MB. > > > > > > Default stack size on amd64 is 512M, and default stack gap percentage is > > > 3%. This means that the gap can be as large as ~17MB. If 3MB is enough > > > for the stack of the main thread of ntpd, then fine. > > > > The default stack is 200K, which is also tuneable in ntp.conf. > > > > [...] > > I haven't seen anyone ask what I consider to be the crucial question > yet: why are we locking ntpd into memory by default at all? > > I have seen two rationales for ntpd using mlockall() and setrlimit(): > >- There are claims that it improves timing performance. > >- Because ntpd is a daemon that can run for months at a time, >setting limits on memory and stack growth can help detect and >mitigate against memory leak problems in the daemon. Doesn't locking this memory down also protect ntpd from OOM kills? If so that is a MUST preserve functionality, as IMHO killing ntpd on a box that has it configured is a total no win situation. Regards, Rod > I am *highly* skeptical of claims that locking ntpd in memory will > improve timing performance on freebsd (to the point where I'm inclined > to dismiss them out of hand, but I'd be willing to look at any actual > evidence). > > The second point has at least some validity, but would be better > implemented (IMO) by decoupling the address space limit from the act of > locking down memory, and allowing configuration of RLIMIT_AS separately > from RLIMIT_MEMLOCK. If there's any interest in this, I could try to > put together a patchset and submit it upstream for that. > > I would propose that for freebsd, the autoconf-generated value for > DFLT_RLIMIT_MEMLOCK should be -1 to avoid calling setrlimit() and > mlockall() by default. Then in the ntp.conf we distribute we have a > section like: > ># To lock ntpd into memory, uncomment the following rlimit line. ># This should not be necessary on most systems, but may improve ># performance on heavily-loaded servers experiencing enough ># memory pressure to cause ntpd to be paged out. ># rlimit memlock stacksize > > Then we would need to come up with reasonable values for . > > -- Ian > > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > -- Rod Grimes rgri...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
On Sat, 2019-09-07 at 09:28 -0700, Cy Schubert wrote: > In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin Belousov writes: > > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert wrote: > > > I've been able to set the memlock rlimit as low as 20 MB. The issue is > > > letting it default to 0 which allows ntp to mlockall() anything it wants. > > > ntpd on my sandbox is currently using 18 MB. > > > > Default stack size on amd64 is 512M, and default stack gap percentage is > > 3%. This means that the gap can be as large as ~17MB. If 3MB is enough > > for the stack of the main thread of ntpd, then fine. > > The default stack is 200K, which is also tuneable in ntp.conf. > > [...] I haven't seen anyone ask what I consider to be the crucial question yet: why are we locking ntpd into memory by default at all? I have seen two rationales for ntpd using mlockall() and setrlimit(): - There are claims that it improves timing performance. - Because ntpd is a daemon that can run for months at a time, setting limits on memory and stack growth can help detect and mitigate against memory leak problems in the daemon. I am *highly* skeptical of claims that locking ntpd in memory will improve timing performance on freebsd (to the point where I'm inclined to dismiss them out of hand, but I'd be willing to look at any actual evidence). The second point has at least some validity, but would be better implemented (IMO) by decoupling the address space limit from the act of locking down memory, and allowing configuration of RLIMIT_AS separately from RLIMIT_MEMLOCK. If there's any interest in this, I could try to put together a patchset and submit it upstream for that. I would propose that for freebsd, the autoconf-generated value for DFLT_RLIMIT_MEMLOCK should be -1 to avoid calling setrlimit() and mlockall() by default. Then in the ntp.conf we distribute we have a section like: # To lock ntpd into memory, uncomment the following rlimit line. # This should not be necessary on most systems, but may improve # performance on heavily-loaded servers experiencing enough # memory pressure to cause ntpd to be paged out. # rlimit memlock stacksize Then we would need to come up with reasonable values for . -- Ian ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
In message <20190907161749.gj2...@kib.kiev.ua>, Konstantin Belousov writes: > On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert wrote: > > I've been able to set the memlock rlimit as low as 20 MB. The issue is > > letting it default to 0 which allows ntp to mlockall() anything it wants. > > ntpd on my sandbox is currently using 18 MB. > > Default stack size on amd64 is 512M, and default stack gap percentage is > 3%. This means that the gap can be as large as ~17MB. If 3MB is enough > for the stack of the main thread of ntpd, then fine. The default stack is 200K, which is also tuneable in ntp.conf. rlimit [memlock Nmegabytes | stacksize N4kPages filenum Nfiledescriptors] memlock Nmegabytes Specify the number of megabytes of memory that should be allocated and locked. Probably only available under Linux, this option may be useful when dropping root (the -i option). The default is 32 megabytes on non-Linux machines, and -1 under Linux. -1 means "do not lock the process into memory". 0 means "lock whatever memory the process wants into memory". stacksize N4kPages Specifies the maximum size of the process stack on systems with the mlockall() function. Defaults to 50 4k pages (200 4k pages in OpenBSD). filenum Nfiledescriptors Specifies the maximum number of file descriptors ntpd may have open at once. Defaults to the system default. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
On Sat, Sep 07, 2019 at 08:45:21AM -0700, Cy Schubert wrote: > I've been able to set the memlock rlimit as low as 20 MB. The issue is > letting it default to 0 which allows ntp to mlockall() anything it wants. > ntpd on my sandbox is currently using 18 MB. Default stack size on amd64 is 512M, and default stack gap percentage is 3%. This means that the gap can be as large as ~17MB. If 3MB is enough for the stack of the main thread of ntpd, then fine. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
In message <20190907153226.gi2...@kib.kiev.ua>, Konstantin Belousov writes: > On Sat, Sep 07, 2019 at 06:09:16AM -0700, Cy Schubert wrote: > > In message <20190907075619.gg2...@kib.kiev.ua>, Konstantin Belousov writes: > > > On Sat, Sep 07, 2019 at 12:53:19AM -0700, Harlan Stenn wrote: > > > > Cy, > > > > > > > > On 9/6/2019 4:56 PM, Cy Schubert wrote: > > > > > ... > > > > > > > > > > For those who enable ASLR, a better workaround is, to add this to you > r > > > > > ntp.conf: > > > > > > > > > > rlimit memlock 64 > > > > > > > > > > Until a more precise default is determined. > > > > > > > > Should I change the default value for FreeBSD-12 to be 64 for now? > > > > > > > > I can get this change in place for the upcoming ntp-4.2.8p14 release, > > > > and we can change it later if needed. > > > > > > ASLR is disabled by default, so if anybody tweak a system config, she > > > should know better to tweak ntpd as well. I am fine with changing the > > > defaults for ntpd, but I think that more useful would be to update > > > the documentation (but where to put it ?). > > > > I agree. We should update the documentation for now. 64 MB was my first > > successful test but I suspect we can get it lower, like 47 MB. For now we > > can update the documentation to say that if a person enables ASLR they must > > > add this to ntp.conf. I'll find the best number instead of the current > > sledgehammer. > > > > Where to put it? I've added it to the ASLR wiki (https://wiki.freebsd.org/A > S > > LR) for now. An ASLR page should go into the handbook documenting how to > > use up ASLR and gotchas like this and mitigations. > May be in security(7). Maybe. > > There are actually two workarounds, with enabled ASLR. One is the rlimit, > another one is to disable stack base randomization by gap. The latter works but I'm not enamoured with it. I suppose we can list the workarounds and let the user pick the one they want to use. I've been able to set the memlock rlimit as low as 20 MB. The issue is letting it default to 0 which allows ntp to mlockall() anything it wants. ntpd on my sandbox is currently using 18 MB. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
On Sat, Sep 07, 2019 at 06:09:16AM -0700, Cy Schubert wrote: > In message <20190907075619.gg2...@kib.kiev.ua>, Konstantin Belousov writes: > > On Sat, Sep 07, 2019 at 12:53:19AM -0700, Harlan Stenn wrote: > > > Cy, > > > > > > On 9/6/2019 4:56 PM, Cy Schubert wrote: > > > > ... > > > > > > > > For those who enable ASLR, a better workaround is, to add this to your > > > > ntp.conf: > > > > > > > > rlimit memlock 64 > > > > > > > > Until a more precise default is determined. > > > > > > Should I change the default value for FreeBSD-12 to be 64 for now? > > > > > > I can get this change in place for the upcoming ntp-4.2.8p14 release, > > > and we can change it later if needed. > > > > ASLR is disabled by default, so if anybody tweak a system config, she > > should know better to tweak ntpd as well. I am fine with changing the > > defaults for ntpd, but I think that more useful would be to update > > the documentation (but where to put it ?). > > I agree. We should update the documentation for now. 64 MB was my first > successful test but I suspect we can get it lower, like 47 MB. For now we > can update the documentation to say that if a person enables ASLR they must > add this to ntp.conf. I'll find the best number instead of the current > sledgehammer. > > Where to put it? I've added it to the ASLR wiki (https://wiki.freebsd.org/AS > LR) for now. An ASLR page should go into the handbook documenting how to > use up ASLR and gotchas like this and mitigations. May be in security(7). There are actually two workarounds, with enabled ASLR. One is the rlimit, another one is to disable stack base randomization by gap. > > 64 MB is safe for now. I will do more testing. I think it can go below 47. > My sandbox has been running ntpd all night with 47 so far. I will try lower. > > > -- > Cheers, > Cy Schubert > FreeBSD UNIX: Web: http://www.FreeBSD.org > > The need of the many outweighs the greed of the few. > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
In message <20190907075619.gg2...@kib.kiev.ua>, Konstantin Belousov writes: > On Sat, Sep 07, 2019 at 12:53:19AM -0700, Harlan Stenn wrote: > > Cy, > > > > On 9/6/2019 4:56 PM, Cy Schubert wrote: > > > ... > > > > > > For those who enable ASLR, a better workaround is, to add this to your > > > ntp.conf: > > > > > > rlimit memlock 64 > > > > > > Until a more precise default is determined. > > > > Should I change the default value for FreeBSD-12 to be 64 for now? > > > > I can get this change in place for the upcoming ntp-4.2.8p14 release, > > and we can change it later if needed. > > ASLR is disabled by default, so if anybody tweak a system config, she > should know better to tweak ntpd as well. I am fine with changing the > defaults for ntpd, but I think that more useful would be to update > the documentation (but where to put it ?). I agree. We should update the documentation for now. 64 MB was my first successful test but I suspect we can get it lower, like 47 MB. For now we can update the documentation to say that if a person enables ASLR they must add this to ntp.conf. I'll find the best number instead of the current sledgehammer. Where to put it? I've added it to the ASLR wiki (https://wiki.freebsd.org/AS LR) for now. An ASLR page should go into the handbook documenting how to use up ASLR and gotchas like this and mitigations. 64 MB is safe for now. I will do more testing. I think it can go below 47. My sandbox has been running ntpd all night with 47 so far. I will try lower. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
On Sat, Sep 07, 2019 at 12:53:19AM -0700, Harlan Stenn wrote: > Cy, > > On 9/6/2019 4:56 PM, Cy Schubert wrote: > > ... > > > > For those who enable ASLR, a better workaround is, to add this to your > > ntp.conf: > > > > rlimit memlock 64 > > > > Until a more precise default is determined. > > Should I change the default value for FreeBSD-12 to be 64 for now? > > I can get this change in place for the upcoming ntp-4.2.8p14 release, > and we can change it later if needed. ASLR is disabled by default, so if anybody tweak a system config, she should know better to tweak ntpd as well. I am fine with changing the defaults for ntpd, but I think that more useful would be to update the documentation (but where to put it ?). ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
In message <201909060639.x866dj7f090...@slippy.cwsent.com>, Cy Schubert writes: > In message <201909060355.x863trhp089...@slippy.cwsent.com>, Cy Schubert > writes: > > In message <20190905142817.gb2...@kib.kiev.ua>, Konstantin Belousov writes: > > > On Thu, Sep 05, 2019 at 06:07:22AM -0700, Cy Schubert wrote: > > > > In message <20190905063354.qxecqjkafikdtdyq@vzakharov>, Vladimir Zakhar > ov > > > > > > write > > > > s: > > > > > > > > > > --ply3axd74k44nuk3 > > > > > Content-Type: text/plain; charset=utf-8 > > > > > Content-Disposition: inline > > > > > Content-Transfer-Encoding: quoted-printable > > > > > > > > > > On Thu, Sep 05, 2019, Vladimir Zakharov wrote: > > > > > > Hello! > > > > > >=20 > > > > > > Accidentally it turned out that the ntpd does not start on boot. > > > > > > Logs show that it core dumps: > > > > > > 2019-09-05T08:38:43.588563+03:00 vzakharov ntpd 34934 - - ntpd 4.2. > 8p > > 12 > > > -a= > > > > > (1): Starting > > > > > > 2019-09-05T08:38:43.588717+03:00 vzakharov kernel - - - Security po > li > > cy > > > l= > > > > > oaded: MAC/ntpd (mac_ntpd) > > > > > > 2019-09-05T08:38:43.718356+03:00 vzakharov kernel - - - pid 35074 ( > nt > > pd > > > ),= > > > > > jid 0, uid 123: exited on signal 11 > > > > > >=20 > > > > > > Not sure how long does it happen. ktrace and gdb points to `setrlim > it > > ` > > > > > > call. Clean rebuild has no effect. Does anyone have any idea what t > he > > > > > > problem could be? > > > > > > > > > > Accidental insight after sending initial mail: I've disabled > > > > > ASLR related sysctls, and now ntpd starts and runs again. > > > > > # grep aslr /boot/loader.conf > > > > > kern.elf32.aslr.enable=3D0 > > > > > kern.elf64.aslr.enable=3D0 > > > > > # service ntpd status > > > > > ntpd is running as pid 30528. > > > > > > > > I'll take a look at this. When you get a chance can you open a PR for t > hi > > s > > > > and assign the ticket to me, please. > > > > > > See the thread 'ntpd doesn't like ASLR on stable/12 post-r350672' on stab > le > > . > > > Read it to the end. > > > > Same setrlimit, different error NetBSD is experiencing. > > > > I'll investigate. > > Until this can be resolved the workaround is to put the following in your > rc.conf: > > ntpd_prepend="/usr/bin/proccontrol -m aslr -s disable" For those who enable ASLR, a better workaround is, to add this to your ntp.conf: rlimit memlock 64 Until a more precise default is determined. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
In message <201909060355.x863trhp089...@slippy.cwsent.com>, Cy Schubert writes: > In message <20190905142817.gb2...@kib.kiev.ua>, Konstantin Belousov writes: > > On Thu, Sep 05, 2019 at 06:07:22AM -0700, Cy Schubert wrote: > > > In message <20190905063354.qxecqjkafikdtdyq@vzakharov>, Vladimir Zakharov > > > > write > > > s: > > > > > > > > --ply3axd74k44nuk3 > > > > Content-Type: text/plain; charset=utf-8 > > > > Content-Disposition: inline > > > > Content-Transfer-Encoding: quoted-printable > > > > > > > > On Thu, Sep 05, 2019, Vladimir Zakharov wrote: > > > > > Hello! > > > > >=20 > > > > > Accidentally it turned out that the ntpd does not start on boot. > > > > > Logs show that it core dumps: > > > > > 2019-09-05T08:38:43.588563+03:00 vzakharov ntpd 34934 - - ntpd 4.2.8p > 12 > > -a= > > > > (1): Starting > > > > > 2019-09-05T08:38:43.588717+03:00 vzakharov kernel - - - Security poli > cy > > l= > > > > oaded: MAC/ntpd (mac_ntpd) > > > > > 2019-09-05T08:38:43.718356+03:00 vzakharov kernel - - - pid 35074 (nt > pd > > ),= > > > > jid 0, uid 123: exited on signal 11 > > > > >=20 > > > > > Not sure how long does it happen. ktrace and gdb points to `setrlimit > ` > > > > > call. Clean rebuild has no effect. Does anyone have any idea what the > > > > > problem could be? > > > > > > > > Accidental insight after sending initial mail: I've disabled > > > > ASLR related sysctls, and now ntpd starts and runs again. > > > > # grep aslr /boot/loader.conf > > > > kern.elf32.aslr.enable=3D0 > > > > kern.elf64.aslr.enable=3D0 > > > > # service ntpd status > > > > ntpd is running as pid 30528. > > > > > > I'll take a look at this. When you get a chance can you open a PR for thi > s > > > and assign the ticket to me, please. > > > > See the thread 'ntpd doesn't like ASLR on stable/12 post-r350672' on stable > . > > Read it to the end. > > Same setrlimit, different error NetBSD is experiencing. > > I'll investigate. Until this can be resolved the workaround is to put the following in your rc.conf: ntpd_prepend="/usr/bin/proccontrol -m aslr -s disable" -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
In message <20190905142817.gb2...@kib.kiev.ua>, Konstantin Belousov writes: > On Thu, Sep 05, 2019 at 06:07:22AM -0700, Cy Schubert wrote: > > In message <20190905063354.qxecqjkafikdtdyq@vzakharov>, Vladimir Zakharov > > write > > s: > > > > > > --ply3axd74k44nuk3 > > > Content-Type: text/plain; charset=utf-8 > > > Content-Disposition: inline > > > Content-Transfer-Encoding: quoted-printable > > > > > > On Thu, Sep 05, 2019, Vladimir Zakharov wrote: > > > > Hello! > > > >=20 > > > > Accidentally it turned out that the ntpd does not start on boot. > > > > Logs show that it core dumps: > > > > 2019-09-05T08:38:43.588563+03:00 vzakharov ntpd 34934 - - ntpd 4.2.8p12 > -a= > > > (1): Starting > > > > 2019-09-05T08:38:43.588717+03:00 vzakharov kernel - - - Security policy > l= > > > oaded: MAC/ntpd (mac_ntpd) > > > > 2019-09-05T08:38:43.718356+03:00 vzakharov kernel - - - pid 35074 (ntpd > ),= > > > jid 0, uid 123: exited on signal 11 > > > >=20 > > > > Not sure how long does it happen. ktrace and gdb points to `setrlimit` > > > > call. Clean rebuild has no effect. Does anyone have any idea what the > > > > problem could be? > > > > > > Accidental insight after sending initial mail: I've disabled > > > ASLR related sysctls, and now ntpd starts and runs again. > > > # grep aslr /boot/loader.conf > > > kern.elf32.aslr.enable=3D0 > > > kern.elf64.aslr.enable=3D0 > > > # service ntpd status > > > ntpd is running as pid 30528. > > > > I'll take a look at this. When you get a chance can you open a PR for this > > and assign the ticket to me, please. > > See the thread 'ntpd doesn't like ASLR on stable/12 post-r350672' on stable. > Read it to the end. Same setrlimit, different error NetBSD is experiencing. I'll investigate. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
On Thu, Sep 05, 2019 at 06:07:22AM -0700, Cy Schubert wrote: > In message <20190905063354.qxecqjkafikdtdyq@vzakharov>, Vladimir Zakharov > write > s: > > > > --ply3axd74k44nuk3 > > Content-Type: text/plain; charset=utf-8 > > Content-Disposition: inline > > Content-Transfer-Encoding: quoted-printable > > > > On Thu, Sep 05, 2019, Vladimir Zakharov wrote: > > > Hello! > > >=20 > > > Accidentally it turned out that the ntpd does not start on boot. > > > Logs show that it core dumps: > > > 2019-09-05T08:38:43.588563+03:00 vzakharov ntpd 34934 - - ntpd 4.2.8p12-a= > > (1): Starting > > > 2019-09-05T08:38:43.588717+03:00 vzakharov kernel - - - Security policy l= > > oaded: MAC/ntpd (mac_ntpd) > > > 2019-09-05T08:38:43.718356+03:00 vzakharov kernel - - - pid 35074 (ntpd),= > > jid 0, uid 123: exited on signal 11 > > >=20 > > > Not sure how long does it happen. ktrace and gdb points to `setrlimit` > > > call. Clean rebuild has no effect. Does anyone have any idea what the > > > problem could be? > > > > Accidental insight after sending initial mail: I've disabled > > ASLR related sysctls, and now ntpd starts and runs again. > > # grep aslr /boot/loader.conf > > kern.elf32.aslr.enable=3D0 > > kern.elf64.aslr.enable=3D0 > > # service ntpd status > > ntpd is running as pid 30528. > > I'll take a look at this. When you get a chance can you open a PR for this > and assign the ticket to me, please. See the thread 'ntpd doesn't like ASLR on stable/12 post-r350672' on stable. Read it to the end. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
In message <20190905063354.qxecqjkafikdtdyq@vzakharov>, Vladimir Zakharov write s: > > --ply3axd74k44nuk3 > Content-Type: text/plain; charset=utf-8 > Content-Disposition: inline > Content-Transfer-Encoding: quoted-printable > > On Thu, Sep 05, 2019, Vladimir Zakharov wrote: > > Hello! > >=20 > > Accidentally it turned out that the ntpd does not start on boot. > > Logs show that it core dumps: > > 2019-09-05T08:38:43.588563+03:00 vzakharov ntpd 34934 - - ntpd 4.2.8p12-a= > (1): Starting > > 2019-09-05T08:38:43.588717+03:00 vzakharov kernel - - - Security policy l= > oaded: MAC/ntpd (mac_ntpd) > > 2019-09-05T08:38:43.718356+03:00 vzakharov kernel - - - pid 35074 (ntpd),= > jid 0, uid 123: exited on signal 11 > >=20 > > Not sure how long does it happen. ktrace and gdb points to `setrlimit` > > call. Clean rebuild has no effect. Does anyone have any idea what the > > problem could be? > > Accidental insight after sending initial mail: I've disabled > ASLR related sysctls, and now ntpd starts and runs again. > # grep aslr /boot/loader.conf > kern.elf32.aslr.enable=3D0 > kern.elf64.aslr.enable=3D0 > # service ntpd status > ntpd is running as pid 30528. I'll take a look at this. When you get a chance can you open a PR for this and assign the ticket to me, please. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: ntpd segfaults on start
On Thu, Sep 05, 2019, Vladimir Zakharov wrote: > Hello! > > Accidentally it turned out that the ntpd does not start on boot. > Logs show that it core dumps: > 2019-09-05T08:38:43.588563+03:00 vzakharov ntpd 34934 - - ntpd 4.2.8p12-a > (1): Starting > 2019-09-05T08:38:43.588717+03:00 vzakharov kernel - - - Security policy > loaded: MAC/ntpd (mac_ntpd) > 2019-09-05T08:38:43.718356+03:00 vzakharov kernel - - - pid 35074 (ntpd), jid > 0, uid 123: exited on signal 11 > > Not sure how long does it happen. ktrace and gdb points to `setrlimit` > call. Clean rebuild has no effect. Does anyone have any idea what the > problem could be? Accidental insight after sending initial mail: I've disabled ASLR related sysctls, and now ntpd starts and runs again. # grep aslr /boot/loader.conf kern.elf32.aslr.enable=0 kern.elf64.aslr.enable=0 # service ntpd status ntpd is running as pid 30528. -- Regards, | "In theory there is no difference between theory Vladimir Zakharov | and practice. In practice there is."- Yogi Berra signature.asc Description: PGP signature
ntpd segfaults on start
Hello!
Accidentally it turned out that the ntpd does not start on boot.
Logs show that it core dumps:
2019-09-05T08:38:43.588563+03:00 vzakharov ntpd 34934 - - ntpd 4.2.8p12-a (1):
Starting
2019-09-05T08:38:43.588717+03:00 vzakharov kernel - - - Security policy loaded:
MAC/ntpd (mac_ntpd)
2019-09-05T08:38:43.718356+03:00 vzakharov kernel - - - pid 35074 (ntpd), jid
0, uid 123: exited on signal 11
Not sure how long does it happen. ktrace and gdb points to `setrlimit`
call. Clean rebuild has no effect. Does anyone have any idea what the
problem could be?
root@vzakharov:~ # uname -a
FreeBSD vzakharov 13.0-CURRENT FreeBSD 13.0-CURRENT r351834 GENERIC-NODEBUG
amd64
root@vzakharov:~ # ktrace -i sudo -u ntpd `which ntpd`
root@vzakharov:~ # kdump | tail -n 20
29735 ntpd CALL dup2(0,0x2)
29735 ntpd RET dup2 2
29735 ntpd CALL socket(PF_LOCAL,0x1002,0)
29735 ntpd RET socket 3
29735 ntpd CALL connect(0x3,0x7f924f98,0x6a)
29735 ntpd STRU struct sockaddr { AF_LOCAL, /var/run/logpriv }
29735 ntpd NAMI "/var/run/logpriv"
29735 ntpd RET connect -1 errno 13 Permission denied
29735 ntpd CALL connect(0x3,0x7f924f98,0x6a)
29735 ntpd STRU struct sockaddr { AF_LOCAL, /var/run/log }
29735 ntpd NAMI "/var/run/log"
29735 ntpd RET connect 0
29735 ntpd CALL setsid
29735 ntpd RET setsid 29735/0x7427
29735 ntpd CALL getrlimit(RLIMIT_STACK,0x7f9250a0)
29735 ntpd RET getrlimit 0
29735 ntpd CALL setrlimit(RLIMIT_STACK,0x7f9250a0)
29735 ntpd RET setrlimit 0
29735 ntpd PSIG SIGSEGV SIG_DFL code=SEGV_ACCERR
29735 ntpd NAMI "ntpd.core"
root@vzakharov:~ # ntpd
root@vzakharov:~ # gdb -c ntpd.core `which ntpd`
GNU gdb (GDB) 8.3 [GDB v8.3 for FreeBSD]
[...]
Reading symbols from /usr/sbin/ntpd...
Reading symbols from /usr/lib/debug//usr/sbin/ntpd.debug...
[New LWP 100946]
Core was generated by `ntpd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 setrlimit () at setrlimit.S:4
4 RSYSCALL(setrlimit)
(gdb) bt
#0 setrlimit () at setrlimit.S:4
Backtrace stopped: Cannot access memory at address 0x7f3ce7c8
(gdb) quit
# cat /etc/src.conf
WITHOUT_ACCT=
WITHOUT_AMD=
WITHOUT_APM=
WITHOUT_ATM=
WITHOUT_AUTHPF=
WITHOUT_BLACKLIST=
WITHOUT_BLUETOOTH=
WITHOUT_BOOTPARAMD=
WITHOUT_BOOTPD=
WITHOUT_BSNMP=
WITHOUT_CCD=
WITHOUT_CROSS_COMPILER=
WITHOUT_CUSE=
WITHOUT_CXGBETOOL=
# WITHOUT_DEBUG_FILES=
WITHOUT_EE=
WITHOUT_FDT=
WITHOUT_FINGER=
WITHOUT_FLOPPY=
WITHOUT_FREEBSD_UPDATE=
WITHOUT_FTP=
WITHOUT_GDB=
WITHOUT_GNU_DIFF=
WITHOUT_GNU_GREP=
WITHOUT_GNU_GREP_COMPAT=
WITHOUT_GPIO=
WITHOUT_HAST=
WITHOUT_HTML=
WITHOUT_HYPERV=
WITHOUT_IPFILTER=
WITHOUT_ISCSI=
WITHOUT_KVM=
WITHOUT_LEGACY_CONSOLE=
WITHOUT_LIB32=
WITHOUT_LLVM_TARGET_ALL=
WITHOUT_LPR=
WITHOUT_MLX5TOOL=
WITHOUT_NDIS=
WITHOUT_NVME=
WITHOUT_PC_SYSINSTALL=
WITHOUT_PF=
WITHOUT_PMC=
WITHOUT_PPP=
WITHOUT_QUOTAS=
WITHOUT_RADIUS_SUPPORT=
WITHOUT_RBOOTD=
WITHOUT_SHAREDOCS=
WITHOUT_TALK=
WITHOUT_TELNET=
WITHOUT_TESTS=
WITHOUT_TFTP=
WITHOUT_TIMED=
WITH_BSD_GREP=
WITH_CCACHE_BUILD=
# WITH_CLANG_EXTRAS=
# WITH_CTF=
# WITH_RATELIMIT=
WITH_SORT_THREADS=
WITH_SVN=
BOOTSTRAP_ALL_TOOLS=
--
Regards, | "In theory there is no difference between theory
Vladimir Zakharov | and practice. In practice there is."- Yogi Berra
signature.asc
Description: PGP signature
