Re: [OE-core] [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg
On 05/03/2016 02:49 AM, Patrick Ohly wrote: > But how does opkg know that "config file not present" is an error? It > depends on the the semantic of the file and thus the package providing > the file. It's not declared explicitly, so it is a fairly arbitrary > judgment call to treat "missing" as error. I agree. What I meant when I said that "opkg status" output is pretty rough is that it is making blanket assumptions, and dumping unprocessed metadata. Dpkg -V analyses the md5sum of each file to determine if it changed, and reports changed files. While opkg currently only stores the md5sum of conffiles, I think opkg status should behave similarly to dpkg -V (for conffiles, at least). Do you mind filing a bug ticket on bugzilla for opkg? -- Cheers, Alejandro -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg
On Mon, 2016-05-02 at 11:15 -0500, Alejandro del Castillo wrote: > > On 05/02/2016 08:39 AM, Patrick Ohly wrote: > > On Fri, 2016-04-22 at 14:51 +0100, Richard Purdie wrote: > >> When enabling extra DEBUGFS image generation with opkg, errors are seen > >> like: > >> > >> ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed > >> packages list. Command > >> '/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg -f > >> /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/opkg.conf > >> -o > >> /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs > >> --force_postinstall --prefer-arch-to-version status' returned 0 and > >> stderr: > >> Collected errors: > >> * file_md5sum_alloc: Failed to open > >> file > >> /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/syslog-startup.conf.busybox: > >> No such file or directory. > >> * file_md5sum_alloc: Failed to open > >> file > >> /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/fstab: > >> No such file or directory. > >> > >> basically for all CONFFILES in the image. This is due to the file > >> rearranging > >> the rootfs generation code does. If we preserve the /etc directory, > >> the avoids the problem. > > > > It avoids the problem, but it does not address the root cause (IMHO). > > Should opkg really complain about missing configuration files? It is > > perhaps an edge case, but besides editing a configuration file > > *removing* it entirely may also be a valid user modification. > > > > The usage of opkg here is during image building, but the operation > > itself (opkg status) is not specific to image creation and thus should > > be able to handle arbitrary states of /etc and the config files in > > general. > > Opkg status is supposed to give you the state of all your installed packages. > Currently is pretty rough data: it dumps into stdout the internal metadata > that > tracks package info and promotes to error any inconsistency, like conffiles > not > being present. But how does opkg know that "config file not present" is an error? It depends on the the semantic of the file and thus the package providing the file. It's not declared explicitly, so it is a fairly arbitrary judgment call to treat "missing" as error. > Looking at pacakge_manager.py, opkg status is being called on OpkgPkgsList. > Seems to me that if the purpose is to get all installed packages, then "opkg > list-installed" is the correct command. Looking at dpkg (DpkgPkgsList), > "dpkg-query -W" is being called, which I think maps to opkg list-installed, > not > to opkg status. That indeed seems better: output should be smaller, faster to run and it avoids the ambiguity about missing config files. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg
On 05/02/2016 08:39 AM, Patrick Ohly wrote: > On Fri, 2016-04-22 at 14:51 +0100, Richard Purdie wrote: >> When enabling extra DEBUGFS image generation with opkg, errors are seen like: >> >> ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed >> packages list. Command >> '/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg -f >> /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/opkg.conf >> -o >> /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs >> --force_postinstall --prefer-arch-to-version status' returned 0 and >> stderr: >> Collected errors: >> * file_md5sum_alloc: Failed to open >> file >> /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/syslog-startup.conf.busybox: >> No such file or directory. >> * file_md5sum_alloc: Failed to open >> file >> /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/fstab: >> No such file or directory. >> >> basically for all CONFFILES in the image. This is due to the file >> rearranging >> the rootfs generation code does. If we preserve the /etc directory, >> the avoids the problem. > > It avoids the problem, but it does not address the root cause (IMHO). > Should opkg really complain about missing configuration files? It is > perhaps an edge case, but besides editing a configuration file > *removing* it entirely may also be a valid user modification. > > The usage of opkg here is during image building, but the operation > itself (opkg status) is not specific to image creation and thus should > be able to handle arbitrary states of /etc and the config files in > general. Opkg status is supposed to give you the state of all your installed packages. Currently is pretty rough data: it dumps into stdout the internal metadata that tracks package info and promotes to error any inconsistency, like conffiles not being present. Looking at pacakge_manager.py, opkg status is being called on OpkgPkgsList. Seems to me that if the purpose is to get all installed packages, then "opkg list-installed" is the correct command. Looking at dpkg (DpkgPkgsList), "dpkg-query -W" is being called, which I think maps to opkg list-installed, not to opkg status. -- Cheers, Alejandro -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg
On Fri, 2016-04-22 at 14:51 +0100, Richard Purdie wrote: > When enabling extra DEBUGFS image generation with opkg, errors are seen like: > > ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed packages > list. Command > '/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg -f > /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/opkg.conf > -o > /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs > --force_postinstall --prefer-arch-to-version status' returned 0 and > stderr: > Collected errors: > * file_md5sum_alloc: Failed to open > file > /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/syslog-startup.conf.busybox: > No such file or directory. > * file_md5sum_alloc: Failed to open > file > /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/fstab: > No such file or directory. > > basically for all CONFFILES in the image. This is due to the file > rearranging > the rootfs generation code does. If we preserve the /etc directory, > the avoids the problem. It avoids the problem, but it does not address the root cause (IMHO). Should opkg really complain about missing configuration files? It is perhaps an edge case, but besides editing a configuration file *removing* it entirely may also be a valid user modification. The usage of opkg here is during image building, but the operation itself (opkg status) is not specific to image creation and thus should be able to handle arbitrary states of /etc and the config files in general. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg
Hi, * Richard Purdie[160422 15:52]: > When enabling extra DEBUGFS image generation with opkg, errors are seen like: > ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed packages > list. Command > '/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg -f > /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/opkg.conf > -o > /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs > --force_postinstall --prefer-arch-to-version status' returned 0 and > stderr: > Collected errors: > * file_md5sum_alloc: Failed to open file > /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/syslog-startup.conf.busybox: > No such file or directory. > * file_md5sum_alloc: Failed to open file > /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/fstab: > No such file or directory. > basically for all CONFFILES in the image. This is due to the file rearranging > the rootfs generation code does. If we preserve the /etc directory, > the avoids the problem. > We need to tell copyfile to preserve symlinks since some are present in /etc. Thanks for the patch. It fixes the issue on e.g. core-image-minimal. Unfortunately, on more complex images, we'll still fail with e.g.: ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed packages list. Command '/mnt/cs-builds/anders/poky-play/build/tmp-glibc/sysroots/x86_64-linux/usr/bin/opkg -f /mnt/cs-builds/anders/poky-play/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/opkg.conf -o /mnt/cs-builds/anders/poky-play/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version status' returned 0 and stderr: Collected errors: * file_md5sum_alloc: Failed to open file /mnt/cs-builds/anders/poky-play/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/lib/ssl/openssl.cnf: No such file or directory. I'd guess that all packages that have CONFFILES outside of /etc will suffer from this... I've updated #9490 with this info. Cheers, Anders -- Anders Darander, Senior System Architect ChargeStorm AB / eStorm AB -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg
When enabling extra DEBUGFS image generation with opkg, errors are seen like: ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed packages list. Command '/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg -f /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/opkg.conf -o /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version status' returned 0 and stderr: Collected errors: * file_md5sum_alloc: Failed to open file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/syslog-startup.conf.busybox: No such file or directory. * file_md5sum_alloc: Failed to open file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/fstab: No such file or directory. basically for all CONFFILES in the image. This is due to the file rearranging the rootfs generation code does. If we preserve the /etc directory, the avoids the problem. We need to tell copyfile to preserve symlinks since some are present in /etc. [YOCTO #9490] Signed-off-by: Richard Purdiediff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py index a95e1b7..0546c1e 100644 --- a/meta/lib/oe/rootfs.py +++ b/meta/lib/oe/rootfs.py @@ -122,7 +122,7 @@ class Rootfs(object): bb.note(" Copying back package database...") for dir in dirs: bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir)) -shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir) +shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True) cpath = oe.cachedpath.CachedPath() # Copy files located in /usr/lib/debug or /usr/src/debug @@ -907,7 +907,7 @@ class OpkgRootfs(DpkgOpkgRootfs): self.pm.install_complementary() -self._setup_dbg_rootfs(['/var/lib/opkg']) +self._setup_dbg_rootfs(['/etc', '/var/lib/opkg']) execute_pre_post_process(self.d, opkg_post_process_cmds) -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core