Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
[Petter Reinholdtsen] > I have not tested the change, as I am unable to build the package from > git now. It fail because there is no upstream tag and pristine-tar > content for the latest upstream changes imported into the git > repository. My mistake. I had forgotten to pull the latest changes. Better now. Tested change commited and new version uploaded. -- Happy hacking Petter Reinholdtsen
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
[Lluís Vilanova] > The poor man's solution is to do as you suggested. If perf's configuration is > insufficient, barf at the user and exit, but do not return any error code. Yeah. Something along these lines, perhaps: diff --git a/debian/rules b/debian/rules index d1b0777..87c1f89 100755 --- a/debian/rules +++ b/debian/rules @@ -32,7 +32,11 @@ override_dh_install: $(RM) $(DESTDIR)/usr/share/coz-profiler/viewer/lib/science/LICENSE.txt override_dh_auto_test: - $(MAKE) check + if [ $(cat /proc/sys/kernel/perf_event_paranoid) -le 2 ] ; then \ + $(MAKE) check; \ + else \ + echo error: "; \ + fi override_dh_auto_build: dh_auto_build -- CC=cc CXX=c++ GIT=true I have not tested the change, as I am unable to build the package from git now. It fail because there is no upstream tag and pristine-tar content for the latest upstream changes imported into the git repository. -- Happy hacking Petter Reinholdtsen
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
Petter Reinholdtsen writes: > [Lluís Vilanova] >> H, with grub2 that's easier to do because it's going to test the >> bootloader that is the first thing going to start. In our case we'd >> have to bootstrap a full system and then start it in qemu just to run >> the test. That doesn't sound feasible to me (time and disk space to >> run the test), but I might be wrong. > It is possible to run "foreign" binaries in the current file system > using qemu. I have used it with vmdebootstrap to build arm chroots on > x86. I tested, and this seem to work when installing qemu-user: > qemu-x86_64 /usr/bin/python /usr/bin/coz run --- /bin/ls > I'm not sure if qemu then start its own kernel or passes through the > perf syscalls to the underlying kernel. But I can not test this in > Jessie: > # /sbin/sysctl kernel.perf_event_paranoid=3 > kernel.perf_event_paranoid = 3 > # TMPDIR= TEMP= TEMPDIR= TMP= chroot /scratch/chroot-sid su - > mesg: ttyname failed: Success > # qemu-x86_64 /usr/bin/python /usr/bin/coz run --- /bin/ls > [libcoz.cpp:100] bootstrapping coz > [libcoz.cpp:128] Including MAIN, which is /bin/ls > [inspect.cpp:319] Unable to locate debug information for /bin/ls > [inspect.cpp:325] /lib/x86_64-linux-gnu/ld-2.24.so is not in scope > [inspect.cpp:325] /usr/lib/coz-profiler/libcoz.so is not in scope > [inspect.cpp:325] /lib/x86_64-linux-gnu/libselinux.so.1 is not in scope > [inspect.cpp:325] /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22 is not in > scope > [inspect.cpp:325] /lib/x86_64-linux-gnu/libm-2.24.so is not in scope > [inspect.cpp:325] /lib/x86_64-linux-gnu/libgcc_s.so.1 is not in scope > [inspect.cpp:325] /lib/x86_64-linux-gnu/libpthread-2.24.so is not in scope > [inspect.cpp:325] /lib/x86_64-linux-gnu/libpcre.so.3.13.3 is not in scope > [inspect.cpp:325] /lib/x86_64-linux-gnu/libdl-2.24.so is not in scope > [inspect.cpp:325] /lib/x86_64-linux-gnu/librt-2.24.so is not in scope > [inspect.cpp:325] /usr/lib/x86_64-linux-gnu/libelf++.so.0 is not in scope > [inspect.cpp:325] /usr/lib/x86_64-linux-gnu/libdwarf++.so.0 is not in scope > [inspect.cpp:325] /lib/x86_64-linux-gnu/libc-2.24.so is not in scope > [profiler.cpp:75] Starting profiler thread > profile.coz > # /sbin/sysctl kernel.perf_event_paranoid > kernel.perf_event_paranoid = 3 > # logout > # > At least the value is inherited into qemu. > But if we can not get this to work before Christmas, we should probably > rewrite the test to only test if the value of kernel.perf_event_paranoid > is 2 or lower. When qemu runs user applications, it simply forwards system calls to the host kernel (it does a bit more than that, but that's the basic idea). If we want to be able to control perf's configuration, we have to run a full-blown system in qemu (i.e., a VM). Building a VM image every time coz is built is what I do not consider feasible (although it is perfectly doable). The poor man's solution is to do as you suggested. If perf's configuration is insufficient, barf at the user and exit, but do not return any error code.
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
[Lluís Vilanova] > H, with grub2 that's easier to do because it's going to test the > bootloader that is the first thing going to start. In our case we'd > have to bootstrap a full system and then start it in qemu just to run > the test. That doesn't sound feasible to me (time and disk space to > run the test), but I might be wrong. It is possible to run "foreign" binaries in the current file system using qemu. I have used it with vmdebootstrap to build arm chroots on x86. I tested, and this seem to work when installing qemu-user: qemu-x86_64 /usr/bin/python /usr/bin/coz run --- /bin/ls I'm not sure if qemu then start its own kernel or passes through the perf syscalls to the underlying kernel. But I can not test this in Jessie: # /sbin/sysctl kernel.perf_event_paranoid=3 kernel.perf_event_paranoid = 3 # TMPDIR= TEMP= TEMPDIR= TMP= chroot /scratch/chroot-sid su - mesg: ttyname failed: Success # qemu-x86_64 /usr/bin/python /usr/bin/coz run --- /bin/ls [libcoz.cpp:100] bootstrapping coz [libcoz.cpp:128] Including MAIN, which is /bin/ls [inspect.cpp:319] Unable to locate debug information for /bin/ls [inspect.cpp:325] /lib/x86_64-linux-gnu/ld-2.24.so is not in scope [inspect.cpp:325] /usr/lib/coz-profiler/libcoz.so is not in scope [inspect.cpp:325] /lib/x86_64-linux-gnu/libselinux.so.1 is not in scope [inspect.cpp:325] /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22 is not in scope [inspect.cpp:325] /lib/x86_64-linux-gnu/libm-2.24.so is not in scope [inspect.cpp:325] /lib/x86_64-linux-gnu/libgcc_s.so.1 is not in scope [inspect.cpp:325] /lib/x86_64-linux-gnu/libpthread-2.24.so is not in scope [inspect.cpp:325] /lib/x86_64-linux-gnu/libpcre.so.3.13.3 is not in scope [inspect.cpp:325] /lib/x86_64-linux-gnu/libdl-2.24.so is not in scope [inspect.cpp:325] /lib/x86_64-linux-gnu/librt-2.24.so is not in scope [inspect.cpp:325] /usr/lib/x86_64-linux-gnu/libelf++.so.0 is not in scope [inspect.cpp:325] /usr/lib/x86_64-linux-gnu/libdwarf++.so.0 is not in scope [inspect.cpp:325] /lib/x86_64-linux-gnu/libc-2.24.so is not in scope [profiler.cpp:75] Starting profiler thread profile.coz # /sbin/sysctl kernel.perf_event_paranoid kernel.perf_event_paranoid = 3 # logout # At least the value is inherited into qemu. But if we can not get this to work before Christmas, we should probably rewrite the test to only test if the value of kernel.perf_event_paranoid is 2 or lower. -- Happy hacking Petter Reinholdtsen
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
Petter Reinholdtsen writes: > [Petter Reinholdtsen] >> What options do we have? I can think of two: (1) Stop verifying during >> build that coz when the kernel prohibits it or (2) find another way to >> test coz during build that do not involve the kernel perf interface. >> Any other ideas? > I asked about this on #debian-release, and a third option came up. > Change the build time testing to use qemu and enable the perf interface > there. I do not know how to do this, but they suggested to check out > other packages doing this by searching for them using > https://codesearch.debian.net/search?q=qemu+path%3A%2Fdebian%2Frules >. > Look like at least grub2 is doing this. > Lluís, do you have time to check this out? H, with grub2 that's easier to do because it's going to test the bootloader that is the first thing going to start. In our case we'd have to bootstrap a full system and then start it in qemu just to run the test. That doesn't sound feasible to me (time and disk space to run the test), but I might be wrong.
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
[Petter Reinholdtsen] > What options do we have? I can think of two: (1) Stop verifying during > build that coz when the kernel prohibits it or (2) find another way to > test coz during build that do not involve the kernel perf interface. > Any other ideas? I asked about this on #debian-release, and a third option came up. Change the build time testing to use qemu and enable the perf interface there. I do not know how to do this, but they suggested to check out other packages doing this by searching for them using https://codesearch.debian.net/search?q=qemu+path%3A%2Fdebian%2Frules >. Look like at least grub2 is doing this. Lluís, do you have time to check this out? -- Happy hacking Petter Reinholdtsen
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
The problematic scenario is this: Stretch is released, the autobuilders are upgraded to stretch and a security hole is discovered in coz-profiler (or some other need for an update in Stretch. In this case, the code will no longer build in the autobuilders, and that would be highly problematic. What options do we have? I can think of two: (1) Stop verifying during build that coz when the kernel prohibits it or (2) find another way to test coz during build that do not involve the kernel perf interface. Any other ideas? We need a fix for this very soon for the fix to make it into testing. -- Happy hacking Petter Reinholdtsen
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
> > [ I arrived at this bug because it already fails in my autobuilders ] > > Are you behind the x32 builder? It is the only one failing on > https://buildd.debian.org/status/package.php?p=coz-profiler=unstable > No, I don't run any official buildd, but I have autobuilders running almost all the time, initially to check for "dpkg-buildpackage -A", and then just to ensure that packages in stretch are buildable in stretch. This package will fail to build in any autobuilder running stretch, therefore it does not autobuild by definition. You can't drop the *auto* thing, it's in Release Policy: https://release.debian.org/stretch/rc_policy.txt Again, if you do not believe me, please ask in debian-devel. Thanks.
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
On Fri, Dec 16, 2016 at 10:34:47PM +0100, Petter Reinholdtsen wrote: > > Control: severity -1 important > > [Santiago Vila] > > Based on the above comment, this bug should really be RC, because > > Release Policy says that packages in stretch *must* be buildable in > > stretch (this of course includes a machine running the default kernel > > for stretch). > > All the dependencies are present, and the build if failing will instruct > the person supervising the build to change the content of > /proc/sys/kernel/perf_event_paranoid from 3 to 2. > > To build it, you can for example do like this: > > sudo sh -c "echo 2 > /proc/sys/kernel/perf_event_paranoid" > apt-get build-dep coz-profiler > apt-get source -b coz-profiler > > I believe this demonstrate that it is buildable in Stretch. It is buildable in stretch in an informal sense, but it does not follow Release Policy. Release Policy says "Packages must *auto*build". This means no human intervention should be required. If if needs human intervention, then it no longer autobuild as required by Release Policy. I'm not entering a severity war here, but this is still RC. If you still do not believe me, please ask in debian-devel. Thanks.
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
Control: severity -1 important [Santiago Vila] > Based on the above comment, this bug should really be RC, because > Release Policy says that packages in stretch *must* be buildable in > stretch (this of course includes a machine running the default kernel > for stretch). All the dependencies are present, and the build if failing will instruct the person supervising the build to change the content of /proc/sys/kernel/perf_event_paranoid from 3 to 2. To build it, you can for example do like this: sudo sh -c "echo 2 > /proc/sys/kernel/perf_event_paranoid" apt-get build-dep coz-profiler apt-get source -b coz-profiler I believe this demonstrate that it is buildable in Stretch. So to me that is no release critical bug here. It is at most important. The severity might change when the autobuilders for the release architectures are affected, which is not the case at the moment. > [ I arrived at this bug because it already fails in my autobuilders ] Are you behind the x32 builder? It is the only one failing on https://buildd.debian.org/status/package.php?p=coz-profiler=unstable > > If you absolutely need to run the tests during the build, maybe a > workaround would be to add a build-depends on a package which changes > the kernel value for you in its postinst (as I guess that the ordinary > user used to build packages would not be able to do that). Interesting idea. Not sure if such package will be accepted by the ftpmasters, given that the sudo command above will do the same for those privileged enough to change the level. -- Happy hacking Petter Reinholdtsen
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
Hi. If you absolutely need to run the tests during the build, maybe a workaround would be to add a build-depends on a package which changes the kernel value for you in its postinst (as I guess that the ordinary user used to build packages would not be able to do that). Yes, it would be a little bit hackish, and I don't know of any package doing anything similar. Thanks.
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
severity 844633 serious thanks On Sun, 4 Dec 2016, Petter Reinholdtsen wrote: > It seem to be by accident that the autobuilders work. I asked, and it > is only because they are running Debian stable. The ones running newer > kernels do not work. This will change when they are upgraded, unless > they are willing to adjust the perf_event_paranoid value on the > autobuilders, which seem unlikely given that the kernel maintainer in > Debian recommend against it. Based on the above comment, this bug should really be RC, because Release Policy says that packages in stretch *must* be buildable in stretch (this of course includes a machine running the default kernel for stretch). [ I arrived at this bug because it already fails in my autobuilders ] Thanks.
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
It seem to be by accident that the autobuilders work. I asked, and it is only because they are running Debian stable. The ones running newer kernels do not work. This will change when they are upgraded, unless they are willing to adjust the perf_event_paranoid value on the autobuilders, which seem unlikely given that the kernel maintainer in Debian recommend against it. See https://lists.debian.org/debian-wb-team/2016/12/threads.html#5 > for the discussion. -- Happy hacking Petter Reinholdtsen
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
[Petter Reinholdtsen] > I notice the autobuilders work, the ci.debian.org builders work, but > the reproducable build builders do not. No idea what the difference > is. This is not quite accurate. All autobuilders except the x32 one work. And only of of the two amd64 autobuilders for the reproducable build work, the other fail due to missing perf access. The new message explain the problem quite nicely. I suggest we close this bug as fixed with the new message, as we can not really "fix" coz to work without the kernel feature it require to do the profiling. The message look like this (from the second reproducable build on amd64): Failed to open perf event. Consider tweaking /proc/sys/kernel/perf_event_paranoid to 2 or less (current value is 3), or run coz as a privileged user (with CAP_SYS_ADMIN). I guess it could be made clearer, but it contain enough to google up an explanation about what is going wrong. I found https://lkml.org/lkml/2016/7/27/305 > providing some background information about the value 3. Apparently is block "all access to performance events by users without CAP_SYS_ADMIN", and is the default in Debian and Android. -- Happy hacking Petter Reinholdtsen
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
I've made a pull request to upstream that makes coz be more informative about that error: https://github.com/plasma-umass/coz/pull/86 Cheers, Lluis
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
[Lluís Vilanova] > A possible check could be: > > if [ `/sbin/sysctl kernel.perf_event_paranoid` -gt 2 ]; then > echo "ERROR: perf is too paranoid for us" > exit 1 > fi > > This would build-depend on procps (which installs sysctl). This seem like a good approach. Perhaps explain in the message how to get the build working too? I found http://unix.stackexchange.com/questions/14227/do-i-need-root-admin-permissions-to-run-userspace-perf-tool-perf-events-ar > explaining the values 2 to -1 for perf_event_paraniod. I'm using kernel 3.16 myself, and here the default value seem to be 1. I guess coz should be patched too, to explain why it isn't working instead of returning the fairly incomprehensive message it give today when the perf call fail -- Happy hacking Petter Reinholdtsen
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
Petter Reinholdtsen writes: > [Lluís Vilanova] >> The package currently fails to run its tests during build due to >> insufficient permissions to access Linux's perf interface. > Is there some way to figure out if such permissions are missing or not? > On my machine, where the build work, the value of > /proc/sys/kernel/perf_event_paranoid is '1', so '0' is not required. > I notice the autobuilders work, the ci.debian.org builders work, but the > reproducable build builders do not. No idea what the difference is. I've rechecked, and things seem to be a bit different... For some reason my kernel (4.7.0-1-amd64) ignores changes through the file system. It's more reliable to use: /sbin/sysctl kernel.perf_event_paranoid sudo /sbin/sysctl kernel.perf_event_paranoid= Now, I've tried on a nother machine, and (strangely) the kernel starts with a perf paranoid level of 3, which is not even documented as a valid value on Linux's source code. Getting it down to 2 is sufficient to run the check successfully. A possible check could be: if [ `/sbin/sysctl kernel.perf_event_paranoid` -gt 2 ]; then echo "ERROR: perf is too paranoid for us" exit 1 fi This would build-depend on procps (which installs sysctl). Cheers, Lluis
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
[Lluís Vilanova] > The package currently fails to run its tests during build due to > insufficient permissions to access Linux's perf interface. Is there some way to figure out if such permissions are missing or not? On my machine, where the build work, the value of /proc/sys/kernel/perf_event_paranoid is '1', so '0' is not required. I notice the autobuilders work, the ci.debian.org builders work, but the reproducable build builders do not. No idea what the difference is. -- Happy hacking Petter Reinholdtsen
Bug#844633: coz-profiler: Cannot run build tests (insufficient permissions for perf)
Package: coz-profiler Version: 0.0.git.20161011T1320-3 Severity: normal The package currently fails to run its tests during build due to insufficient permissions to access Linux's perf interface. There's three ways to solve this: * running the checks as root * granting CAP_SYS_ADMIN to the user running the checks * configuring perf's paranoid level: sudo sh -c 'echo 0 >/proc/sys/kernel/perf_event_paranoid' Something I have not yet checked is if there is a subset of the interface, sufficient for coz, that will provide the profiling information without additional permissions. Cheers, Lluis
