Re: [lxc-users] Enabling real time support in containers
On 04/05/2017 06:45 AM, Serge E. Hallyn wrote: I correct in assuming LXC *does* provide a means to enable RT The kernel has hardcoded checks (which are not namespaced) that if you are not (global) root, you cannot set or change the rt policy. I suspect there is a way that could be safely relaxed (i.e. if a container has exclusie use of a cpu), but we'd have to talk to the scheduling experts about what would make sense. (see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/sched/core.c?id=refs/tags/v4.11-rc5#n4164 ) Otherwise, as a workaround (assuming this is the only problem you hit) you could simply make sure that the RT policy is correct ahead of time and the priority is high enough that the application is only lowering it, then the kernel wouldn't stop it. Certainly that's more fragile. Or you could get fancier and LD_PRELOAD to catch sys_setscheduler and redirect to an api over a socket to a tiny deamon on the host kernel which sets it up for you... But certainly it would be best for everyone if this was supported in the kernel the right way. Most of our containers do not require real time support. There are a couple of cases though where our software does use real time threads though. We originally were running under libvirt based VMs and real time support was not an issue in this kind of environment (it was fully supported). We ported our software to libvirt lxc based containers and with the appropriate configuration was able to get real time support working under this environment as well. We want to make one more transition now to LXC (mainly due to lack of active support for libvirt-lxc in CentOS). I had assumed this was simply a container configuration issue but your response makes me think that it's not as simple as that. Not sure where that leaves us. Peter ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Enabling real time support in containers
Quoting Peter Steele (pwste...@gmail.com):
> On 03/31/2017 10:16 AM, Peter Steele wrote:
> >As you can see, the sched_setscheduler() call fails with an EPERM
> >error. This same app runs fine on the host.
> >
> >Ultimately I expect this app to fail when run under my container
> >since I have not given the container any real time bandwidth. I
> >had hoped the option
> >
> >lxc.cgroup.cpu.rt_runtime_us = 475000
> >
> >would do the trick but this option is rejected with anything other
> >than "0". So presumably this isn't the correct way to give a
> >container real time bandwidth.
> >
> >I have more experience with the libvirt-lxc framework and I have
> >been able to enable real time support for containers under
> >libvirt. The approach used in this case involves explicitly
> >setting cgroup parameters, specifically
> >
> >/sys/fs/cgroup/cpu/machine.slice/cpu.rt_runtime_us
> >
> >under the host and
> >
> >/sys/fs/cgroup/cpu/cpu.rt_runtime_us
> >
> >under the container. For example, I might do something like this:
> >
> >echo 50 >/sys/fs/cgroup/cpu/machine.slice/cpu.rt_runtime_us
> >--> on the host
> >echo 25000 >/sys/fs/cgroup/cpu/cpu.rt_runtime_us --> on a container
> >
> >These do not work for LXC based containers though.
> >
>
> The test code I'm running can be simplified to just this simple sequence:
>
> #include
> #include
>
> int main() {
> struct sched_param param;
> param.sched_priority = 50;
> const int myself = 0; // 0 is the PID of ourself
> if (0 != sched_setscheduler(myself, SCHED_FIFO, ¶m)) {
> printf("Failure\n");
> return -1;
> }
>
> printf("Success\n");
> return 0;
> }
>
> On a container with RT support enabled, this should print "Success".
>
> Am I correct in assuming LXC *does* provide a means to enable RT
> support? If not, we will need to another approach to this problem.
The kernel has hardcoded checks (which are not namespaced) that
if you are not (global) root, you cannot set or change the rt
policy. I suspect there is a way that could be safely relaxed
(i.e. if a container has exclusie use of a cpu), but we'd have
to talk to the scheduling experts about what would make sense.
(see
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/sched/core.c?id=refs/tags/v4.11-rc5#n4164
)
Otherwise, as a workaround (assuming this is the only problem you
hit) you could simply make sure that the RT policy is correct ahead
of time and the priority is high enough that the application is only
lowering it, then the kernel wouldn't stop it. Certainly that's
more fragile. Or you could get fancier and LD_PRELOAD to catch
sys_setscheduler and redirect to an api over a socket to a tiny
deamon on the host kernel which sets it up for you... But certainly
it would be best for everyone if this was supported in the kernel the
right way.
-serge
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Enabling real time support in containers
On 03/31/2017 10:16 AM, Peter Steele wrote:
As you can see, the sched_setscheduler() call fails with an EPERM
error. This same app runs fine on the host.
Ultimately I expect this app to fail when run under my container since
I have not given the container any real time bandwidth. I had hoped
the option
lxc.cgroup.cpu.rt_runtime_us = 475000
would do the trick but this option is rejected with anything other
than "0". So presumably this isn't the correct way to give a container
real time bandwidth.
I have more experience with the libvirt-lxc framework and I have been
able to enable real time support for containers under libvirt. The
approach used in this case involves explicitly setting cgroup
parameters, specifically
/sys/fs/cgroup/cpu/machine.slice/cpu.rt_runtime_us
under the host and
/sys/fs/cgroup/cpu/cpu.rt_runtime_us
under the container. For example, I might do something like this:
echo 50 >/sys/fs/cgroup/cpu/machine.slice/cpu.rt_runtime_us
--> on the host
echo 25000 >/sys/fs/cgroup/cpu/cpu.rt_runtime_us --> on a container
These do not work for LXC based containers though.
The test code I'm running can be simplified to just this simple sequence:
#include
#include
int main() {
struct sched_param param;
param.sched_priority = 50;
const int myself = 0; // 0 is the PID of ourself
if (0 != sched_setscheduler(myself, SCHED_FIFO, ¶m)) {
printf("Failure\n");
return -1;
}
printf("Success\n");
return 0;
}
On a container with RT support enabled, this should print "Success".
Am I correct in assuming LXC *does* provide a means to enable RT
support? If not, we will need to another approach to this problem.
Peter
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Enabling real time support in containers
On 03/28/2017 07:55 AM, Serge E. Hallyn wrote:
Is this using a user namespace or not?
I am not using a user namespace. This in intended to be a privileged
container with everything running as root. Although I am planning on
using a custom CentOS template I've created, I can reproduce the problem
with the stock "centos" template provided by LXC. I created a container
using the following steps:
lxc-create -n test1 -t centos
chroot /var/lib/lxc/test1/rootfs passwd
sed -i 's/lxcbr0/br0/' /var/lib/lxc/test1/config
sed -i 's/lxc.cap.drop = sys_nice/# lxc.cap.drop = sys_nice/'
/usr/share/lxc/config/centos.common.conf
lxc-start -n test1
lxc-attach -n test1
At this point I can get my container's DHCP assigned IP address and copy
my real-time test app to the container. The strace command shows the
following:
# lxc-attach -n test1
[root@test1 ~]# strace ./rttest
execve("./rttest", ["./rttest"], [/* 25 vars */]) = 0
brk(0) = 0x825000
...
sched_setscheduler(285, SCHED_FIFO, { 92 }) = -1 EPERM (Operation not
permitted)
tgkill(284, 285, SIGRTMIN) = 0
write(1, "Unable to start rt thread, rc=1\n", 32Unable to start rt
thread, rc=1
) = 32
futex(0x7ff9f1a47908, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x7ff9f1a47908, FUTEX_WAKE_PRIVATE, 1) = 0
exit_group(0) = ?
+++ exited with 0 +++
As you can see, the sched_setscheduler() call fails with an EPERM error.
This same app runs fine on the host.
Ultimately I expect this app to fail when run under my container since I
have not given the container any real time bandwidth. I had hoped the option
lxc.cgroup.cpu.rt_runtime_us = 475000
would do the trick but this option is rejected with anything other than
"0". So presumably this isn't the correct way to give a container real
time bandwidth.
I have more experience with the libvirt-lxc framework and I have been
able to enable real time support for containers under libvirt. The
approach used in this case involves explicitly setting cgroup
parameters, specifically
/sys/fs/cgroup/cpu/machine.slice/cpu.rt_runtime_us
under the host and
/sys/fs/cgroup/cpu/cpu.rt_runtime_us
under the container. For example, I might do something like this:
echo 50 >/sys/fs/cgroup/cpu/machine.slice/cpu.rt_runtime_us -->
on the host
echo 25000 >/sys/fs/cgroup/cpu/cpu.rt_runtime_us --> on a container
These do not work for LXC based containers though.
Peter
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Enabling real time support in containers
On Tue, Mar 28, 2017 at 05:32:15AM -0700, Peter Steele wrote: > > >We have a need to create real time threads in some of our > >processes and I've been unable to configure an LXC container to > >support this. One reference I came across was to set a container's > >real time bandwidth via the lxc.cgroup.cpu.rt_runtime_us parameter > >in its config file: > > > >lxc.utsname = test01 > >lxc.include = /var/lib/lxc/centos.conf > >lxc.network.veth.pair = test01 > >lxc.network.hwaddr = fe:d6:e8:e2:fa:db > >lxc.rootfs = /var/lib/lxc/test01/rootfs > >lxc.rootfs.backend = dir > >lxc.cgroup.cpuset.cpus = 0,1 > >lxc.cgroup.memory.limit_in_bytes = 2097152000 > >lxc.cgroup.memory.memsw.limit_in_bytes = 3170893824 > >lxc.cgroup.cpu.rt_runtime_us = 475000 > > > >This container starts up fine if lxc.cgroup.cpu.rt_runtime_us is 0 > >(zero). Anything other than 0 is rejected, which means real time > >threads cannot be created in this container. > > > >What am I missing to get this to work? I am using lxc version > >2.0.6 under CentOS 7.2. The container is being created using a > >custom CentOS 7.2 image. Is this using a user namespace or not? > No takers on this? I assume this is possible? I'm looking for the > configuration options that would allow code such as this to run : > > pthread_attr_setinheritsched(&tattr, PTHREAD_EXPLICIT_SCHED); > pthread_attr_setschedpolicy(&tattr, SCHED_FIFO); > tsparam.sched_priority = sched_get_priority_max(SCHED_FIFO) - 7; > pthread_attr_setschedparam(&tattr, &tsparam); > rc = pthread_create(&test_thread, &tattr, test, NULL); > > While this code runs fine in my host, it returns a non-zero code on > my containers, indicating I am not allowed to create real time > threads. I'm assuming this is a configuration issue in my LXC > containers since I've seen this same code run under other container > frameworks. There doesn't seem to be a lot of hits on this topic > though when it comes to LXC. Can you strace the failing code and get the exact system call which is failing? If you're in a non-init userns, then my guess is in fact the kernel needs an update - see callso to capable(CAP_SYS_NICE) in kernel/sched/core.c. For those to work in a user namespace it'd need to use ns_capable(). ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Enabling real time support in containers
We have a need to create real time threads in some of our processes and I've been unable to configure an LXC container to support this. One reference I came across was to set a container's real time bandwidth via the lxc.cgroup.cpu.rt_runtime_us parameter in its config file: lxc.utsname = test01 lxc.include = /var/lib/lxc/centos.conf lxc.network.veth.pair = test01 lxc.network.hwaddr = fe:d6:e8:e2:fa:db lxc.rootfs = /var/lib/lxc/test01/rootfs lxc.rootfs.backend = dir lxc.cgroup.cpuset.cpus = 0,1 lxc.cgroup.memory.limit_in_bytes = 2097152000 lxc.cgroup.memory.memsw.limit_in_bytes = 3170893824 lxc.cgroup.cpu.rt_runtime_us = 475000 This container starts up fine if lxc.cgroup.cpu.rt_runtime_us is 0 (zero). Anything other than 0 is rejected, which means real time threads cannot be created in this container. What am I missing to get this to work? I am using lxc version 2.0.6 under CentOS 7.2. The container is being created using a custom CentOS 7.2 image. No takers on this? I assume this is possible? I'm looking for the configuration options that would allow code such as this to run : pthread_attr_setinheritsched(&tattr, PTHREAD_EXPLICIT_SCHED); pthread_attr_setschedpolicy(&tattr, SCHED_FIFO); tsparam.sched_priority = sched_get_priority_max(SCHED_FIFO) - 7; pthread_attr_setschedparam(&tattr, &tsparam); rc = pthread_create(&test_thread, &tattr, test, NULL); While this code runs fine in my host, it returns a non-zero code on my containers, indicating I am not allowed to create real time threads. I'm assuming this is a configuration issue in my LXC containers since I've seen this same code run under other container frameworks. There doesn't seem to be a lot of hits on this topic though when it comes to LXC. Peter ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
[lxc-users] Enabling real time support in containers
We have a need to create real time threads in some of our processes and I've been unable to configure an LXC container to support this. One reference I came across was to set a container's real time bandwidth via the lxc.cgroup.cpu.rt_runtime_us parameter in its config file: lxc.utsname = test01 lxc.include = /var/lib/lxc/centos.conf lxc.network.veth.pair = test01 lxc.network.hwaddr = fe:d6:e8:e2:fa:db xc.rootfs = /var/lib/lxc/test01/rootfs lxc.rootfs.backend = dir lxc.cgroup.cpuset.cpus = 0,1 lxc.cgroup.memory.limit_in_bytes = 2097152000 lxc.cgroup.memory.memsw.limit_in_bytes = 3170893824 lxc.cgroup.cpu.rt_runtime_us = 475000 This container starts up fine if lxc.cgroup.cpu.rt_runtime_us is 0 (zero). Anything other than 0 is rejected, which means real time threads cannot be created in this container. What am I missing to get this to work? I am using lxc version 2.0.6 under CentOS 7.2. The container is being created using a custom CentOS 7.2 image. Thanks for the help. Peter Steele ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
