Re: [systemd-devel] system-wide MemoryMax - possible?
While not directly answering your question, we (facebook) use oomd[0] widely across our fleet to solve the exact problem you have. I'd be happy to answer any questions about it. It should (if configured correctly) be much more reliable than a global memory.max and less heavy handed. In theory, cgooms are subject to the same "livelocks" as with the kernel oom killer. Daniel [0]: https://github.com/facebookincubator/oomd On Sun, Mar 17, 2019, at 9:13 AM, Tomasz Chmielewski wrote: > I think most of us saw the situation when the system becomes > unresponsive - to a point when SSH in doesn't work - because it's out of > memory and kernel's OOM-killer doesn't kick in as fast as it should. > > > I have a server which from time to time - let's say once a week - is > using too much memory. High memory usage can be caused by several > unrelated worker processes. Some of these workers have memory leaks > which are hard to diagnose. > > What happens next - the system becomes very slow for 1-30 minutes, until > kernel's OOM-killer kicks in. Offending process is killed, memory is > released - everything works smooth again. I'm not so worried about the > killed process; I'm more worried that the server is unresponsive for so > long. > > Ideal situation would be - the offending process is killed before the > system becomes very slow. However, OOM in the Linux kernel doesn't seem > to work this way (at least not always). > > > So I thought about "tricking it": > > - move the server to a container (LXD in this case) > - assign the container slightly less RAM than total system RAM (i.e. > 15.5 GB for a container, where the system has 16 GB RAM) > > The result was great - the system is responsive at all times, even if > some processes misbehave and try to use all RAM (OOM-killer kicks in in > container's cgroup, but the system as a whole is never out of memory > from kernel's point of view)! > > > How about achieving a similar result with just systemd? Is there some > system-wide MemoryMax which we could easily set in one place? > > I.e. a desktop system where user opens several browsers, with too many > tabs with too many memory-intensive pages - becomes unresponsive for > long minutes, before OOM-killer finally kills the offender. > > > Tomasz Chmielewski > ___ > systemd-devel mailing list > systemd-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] system-wide MemoryMax - possible?
I think most of us saw the situation when the system becomes unresponsive - to a point when SSH in doesn't work - because it's out of memory and kernel's OOM-killer doesn't kick in as fast as it should. I have a server which from time to time - let's say once a week - is using too much memory. High memory usage can be caused by several unrelated worker processes. Some of these workers have memory leaks which are hard to diagnose. What happens next - the system becomes very slow for 1-30 minutes, until kernel's OOM-killer kicks in. Offending process is killed, memory is released - everything works smooth again. I'm not so worried about the killed process; I'm more worried that the server is unresponsive for so long. Ideal situation would be - the offending process is killed before the system becomes very slow. However, OOM in the Linux kernel doesn't seem to work this way (at least not always). So I thought about "tricking it": - move the server to a container (LXD in this case) - assign the container slightly less RAM than total system RAM (i.e. 15.5 GB for a container, where the system has 16 GB RAM) The result was great - the system is responsive at all times, even if some processes misbehave and try to use all RAM (OOM-killer kicks in in container's cgroup, but the system as a whole is never out of memory from kernel's point of view)! How about achieving a similar result with just systemd? Is there some system-wide MemoryMax which we could easily set in one place? I.e. a desktop system where user opens several browsers, with too many tabs with too many memory-intensive pages - becomes unresponsive for long minutes, before OOM-killer finally kills the offender. Tomasz Chmielewski ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] install ipsec and l2tp problem
On Sa, 16.03.19 13:27, Li Haosen (sanlis3...@gmail.com) wrote: > hello,dear,Engineer,Forgive me,my English not good,i install ipsec and l2tp > has some problem,i can‘t find how to solve,can help me? Please contact the ipsec community about this, not systemd. If ipsec fails to start like it does the failure is inside ipsec. Thank you for understanding, Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Memory used by service reported wrong
On Sat, Mar 16, 2019 at 7:58 PM Manuel Reimer wrote: > I've found my processes: > > # systemctl status session-8.scope > ● session-8.scope - Session 8 of user kodi > Loaded: loaded (/run/systemd/transient/session-8.scope; transient) > Transient: yes > Active: active (running) since Sat 2019-03-16 18:55:16 CET; 2min 1s ago > Tasks: 33 > Memory: 198.3M > CGroup: /user.slice/user-420.slice/session-8.scope > ├─1109 /usr/bin/xinit /usr/bin/dbus-launch > --exit-with-session /usr/> > ├─ (sd-pam) > ├─1112 /usr/lib/Xorg :0 -nolisten tcp vt7 :0 > ├─1117 /bin/sh /usr/bin/kodi-standalone > ├─1120 /usr/bin/dbus-launch --exit-with-session > /usr/bin/kodi-standa> > ├─1121 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 > --print-ad> > ├─1125 /bin/sh /usr/bin/kodi --standalone > └─1133 /usr/lib/kodi/kodi-x11 --standalone > Looks like Kodi startup creates a systemd-logind session for itself, which makes sense, as it seems to work practically the same as any other interactive X11 session; by doing so it gets device access and polkit access. > > Mar 16 18:55:16 manuelsvdr systemd[1]: Started Session 8 of user kodi. > > This scope does not have my memory limitation enabled. How to get this > setting in there? > > Configure the limits in user-420.slice.d/ instead. -- Mantas Mikulėnas ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Kafka Service Fails: Status=200/CHDIR
On Sun, Mar 17, 2019 at 12:22 AM bart schroder wrote: > Hi Mantas, > Please include the mailing list when sending replies, don't turn this into a private conversation. > > *Context:* > I'm taking a Udemy course on Kafka - Kafka for Beginners. > > *Course Objective: Operate (Kafka) Infrastructure as a Service (IaaS)* > We're creating Systemd Services (Docker-Compose, Kafka, Zookeeper, etc.) > to start (stop, etc.) our infrastructure components. > > *Verified:* > I can manually start Kafka using the paths in the ExecStart declaration of > the Kafka.service file. See new attached screenshot of successful start of > kafka from /Home with the path copy/pasted from Kafka.service file in > /etc/systemd/system. > > *Thoughts on cause of my fail response upon attempted start of > Kafka.service?* > Again, you haven't shown any attempts to start kafka.service in your previous post. You've only shown attempts to start docker-compose@kafka.service, a completely different unit. Whether Docker indirectly uses kafka.service is irrelevant at this point – the CHDIR error you've shown is generated directly by the startup of *docker-compose@.service* itself, it did not propagate from some other unit through several thick layers of extra software. -- Mantas Mikulėnas ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
