[OE-core] [PATCH] classes/image_types_wic: Reorder do_flush_pseudodb

[Ricardo Ribalda]

When IMAGE_FSTYPES contains more types than wic, it can happen than the
pseudo database is not flushed properly.

This can be solved by changing the order of when do_flush_pseudodb is
launched.

Yocto Bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13898
Fixes: dde90a5dd2b2 ("wic: Fix multi images .wks with bitbake")
Signed-off-by: Ricardo Ribalda 
---
 meta/classes/image_types_wic.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image_types_wic.bbclass 
b/meta/classes/image_types_wic.bbclass
index 96ed0473ee..7b1db50a28 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -142,7 +142,7 @@ python do_rootfs_wicenv () {
 depdir = d.getVar('IMGDEPLOYDIR')
 bb.utils.copyfile(os.path.join(outdir, basename) + '.env', 
os.path.join(depdir, basename) + '.env')
 }
-addtask do_flush_pseudodb after do_image before do_image_wic
+addtask do_flush_pseudodb after do_rootfs before do_image do_image_qa
 addtask do_rootfs_wicenv after do_image before do_image_wic
 do_rootfs_wicenv[vardeps] += "${WICVARS}"
 do_rootfs_wicenv[prefuncs] = 'set_image_size'
-- 
2.26.2

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138399): 
https://lists.openembedded.org/g/openembedded-core/message/138399
Mute This Topic: https://lists.openembedded.org/mt/74288249/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] devtool and multiconfig: Bug or feature?

[Ricardo Ribalda]

Hi

I am trying to use devtool in combination with multiconfig for the
first time, so probably I am doing something wrong.

I have created a very simple multiconfig

cat build/conf/multiconfig/arm.conf
MACHINE = "qemuarm"

And then I want to modify bash:
devtool modify multiconfig:arm:bash

If then I try to run:
devtool build multiconfig:arm:bash
NOTE: Starting bitbake server...
ERROR: No recipe named 'multiconfig:arm:bash' in your workspace

So I run:
devtool build bash

But that builds for x86 instead of for arm...

Simply patching check_workspace_recipe does not do the trick:
@@ -145,6 +145,8 @@ def check_workspace_recipe(workspace, pn,
checksrc=True, bbclassextend=False):
 is present.
 """

+pn = pn.split(":")[-1]
+
 workspacepn = pn

 for recipe, value in workspace.items():


Any idea or documentation to follow?

Thanks!


-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#139339): 
https://lists.openembedded.org/g/openembedded-core/message/139339
Mute This Topic: https://lists.openembedded.org/mt/74753055/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] devtool and multiconfig: Bug or feature?

[Ricardo Ribalda]

Hi Richard
On Tue, Jun 9, 2020 at 11:46 AM Richard Purdie
 wrote:
>
> On Mon, 2020-06-08 at 16:40 +0200, Ricardo Ribalda wrote:
> > Hi
> >
> > I am trying to use devtool in combination with multiconfig for the
> > first time, so probably I am doing something wrong.
> >
> > I have created a very simple multiconfig
> >
> > cat build/conf/multiconfig/arm.conf
> > MACHINE = "qemuarm"
> >
> > And then I want to modify bash:
> > devtool modify multiconfig:arm:bash
> >
> > If then I try to run:
> > devtool build multiconfig:arm:bash
> > NOTE: Starting bitbake server...
> > ERROR: No recipe named 'multiconfig:arm:bash' in your workspace
> >
> > So I run:
> > devtool build bash
> >
> > But that builds for x86 instead of for arm...
> >
> > Simply patching check_workspace_recipe does not do the trick:
> > @@ -145,6 +145,8 @@ def check_workspace_recipe(workspace, pn,
> > checksrc=True, bbclassextend=False):
> >  is present.
> >  """
> >
> > +pn = pn.split(":")[-1]
> > +
> >  workspacepn = pn
> >
> >  for recipe, value in workspace.items():
> >
> >
> > Any idea or documentation to follow?
>
> I think this is something which hasn't been tested before. Its a bug
> but nobody has probably tested or fixed devtool to work with
> multiconfig.

I have opened a bug on bugzilla, with a horrible patch that kind of fixes it...
Lets see how it evolves.

>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#139394): 
https://lists.openembedded.org/g/openembedded-core/message/139394
Mute This Topic: https://lists.openembedded.org/mt/74753055/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 2/2] wic: Add --embed-rootfs argument

[Ricardo Ribalda]

ping?

On Thu, Mar 5, 2020 at 1:26 PM Ricardo Ribalda Delgado
 wrote:
>
> Hi Paul,
>
> On Thu, Mar 5, 2020 at 10:37 AM Paul Barker  wrote:
> >
> > On Wed,  4 Mar 2020 15:49:36 +0100
> > Ricardo Ribalda Delgado  wrote:
> >
> > > This option adds the content of a rootfs on a specific location on the
> > > rootfs.
> > >
> > > It is very useful for making a partition that contains the rootfs for a
> > > host and a target Eg:
> > >
> > > / -> Roofs for the host
> > > /export/ -> Rootfs for the target (which will netboot)
> > >
> > > Although today we support making a partition for "/export" this might
> > > not be compatible with some upgrade systems, or we might be limited by
> > > the number of partitions.
> > >
> > > With this patch we can use something like:
> > >
> > > part / --source rootfs --embed-rootfs target-image /export --embed-rootfs 
> > > target-image2 /export2
> >
> > I like this but it still leaves confusion between `--include-path` and
> > --embed-rootfs` as they're similar but slightly different. Can we just 
> > modify
> > `--include-path` to have this syntax?
>
> I think they are different enough.
>
> - include-path ads a file/folder
> - embed-rootfs adds a rootfs, which is either a folder or a image.bb file.
>
>
> >
> > >
> > > on the .wks file.
> > >
> > > Signed-off-by: Ricardo Ribalda Delgado 
> > > ---
> > >  scripts/lib/wic/help.py  |  8 
> > >  scripts/lib/wic/ksparser.py  |  1 +
> > >  scripts/lib/wic/partition.py |  1 +
> > >  scripts/lib/wic/plugins/source/rootfs.py | 22 +-
> > >  4 files changed, 31 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> > > index 4d342fcf05..140dc504cd 100644
> > > --- a/scripts/lib/wic/help.py
> > > +++ b/scripts/lib/wic/help.py
> > > @@ -979,6 +979,14 @@ DESCRIPTION
> > >   copies. This option only has an effect with the 
> > > rootfs
> > >   source plugin.
> > >
> > > + --embed-rootfs: This option is specific to wic. It embeds a 
> > > rootfs into
> > > + the given path to the resulting image. The 
> > > option
> > > + contains two fields, the roofs and the path, 
> > > separated
> > > + by a space. The rootfs follows the same logic 
> > > as the
> > > + rootfs-dir argument. Multiple options can be 
> > > provided
> > > + in order to embed multiple rootfs. This option 
> > > only has
> > > + an effect with the rootfs source plugin.
> > > +
> > >   --extra-space: This option is specific to wic. It adds extra
> > >  space after the space filled by the content
> > >  of the partition. The final size can go
> > > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> > > index 650b976223..64c8c1175e 100644
> > > --- a/scripts/lib/wic/ksparser.py
> > > +++ b/scripts/lib/wic/ksparser.py
> > > @@ -138,6 +138,7 @@ class KickStart():
> > >  part.add_argument('--align', type=int)
> > >  part.add_argument('--exclude-path', nargs='+')
> > >  part.add_argument('--include-path', nargs='+')
> > > +part.add_argument('--embed-rootfs', nargs=2, action='append')
> > >  part.add_argument("--extra-space", type=sizetype)
> > >  part.add_argument('--fsoptions', dest='fsopts')
> > >  part.add_argument('--fstype', default='vfat',
> > > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> > > index 2d95f78439..13857df82f 100644
> > > --- a/scripts/lib/wic/partition.py
> > > +++ b/scripts/lib/wic/partition.py
> > > @@ -31,6 +31,7 @@ class Partition():
> > >  self.extra_space = args.extra_space
> > >  self.exclude_path = args.exclude_path
> > >  self.include_path = args.include_path
> > > +self.embed_rootfs = args.embed_rootfs
> > >  self.fsopts = args.fsopts
>

Re: [OE-core] [PATCH 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

ping?

On Thu, Mar 5, 2020 at 10:46 AM Ricardo Ribalda Delgado
 wrote:
>
> Hi Paul
>
> On Thu, Mar 5, 2020 at 10:32 AM Paul Barker  wrote:
> >
> > On Wed, 4 Mar 2020 11:02:47 +0100
> > Ricardo Ribalda Delgado  wrote:
> >
> > > Hi Paul
> > >
> > > On Wed, Mar 4, 2020 at 10:53 AM Paul Barker  wrote:
> > > >
> > > > On Wed,  4 Mar 2020 09:34:37 +0100
> > > > Ricardo Ribalda Delgado  wrote:
> > > >
> > > > > When parameters include_path or exclude_path are passed to the rootfs
> > > > > plugin, it will copy the partition content into a folder and make all
> > > > > the modifications there.
> > > > >
> > > > > This is done using copyhardlinktree(), which does not take into
> > > > > consideration the content of the pseudo folder, which contains the
> > > > > information about the right permissions and ownership of the folders.
> > > >
> > > > How are you running wic here? In the do_image_wic task it's executed 
> > > > under
> > > > pseudo so all this is handled already. Executing wic outside of bitbake 
> > > > may
> > > > need some more testing here.
> > >
> > > I am running wic outside bitbake. But even if it is run under bitbake,
> > > it should also fail. The pseudo directory needs to be present on the
> > > target image
> >
> > If you're running wic outside of bitbake, is there any guarantee that pseudo
> > is available?
>
> Yes, the same guarantee that the ext3_tools are available. So I
> believe we are safe here. Actually in my docker pseudo is not
> installed and when I invoke with wic, everything is fine.
>
> >
> > >
> > > >
> > > > >
> > > > > This results in a rootfs owned by the user that is running the wic
> > > > > command (usually UID 1000), which makes some rootfs unbootable.
> > > > >
> > > > > To fix this we copy the content of the pseudo folders to the new 
> > > > > folder
> > > > > and modify the pseudo database using the "pseudo -B" command.
> > > > >
> > > > > Signed-off-by: Ricardo Ribalda Delgado 
> > > > > ---
> > > > >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++---
> > > > >  1 file changed, 19 insertions(+), 3 deletions(-)
> > > > >
> > > > > diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
> > > > > b/scripts/lib/wic/plugins/source/rootfs.py
> > > > > index 705aeb5563..40419a64b3 100644
> > > > > --- a/scripts/lib/wic/plugins/source/rootfs.py
> > > > > +++ b/scripts/lib/wic/plugins/source/rootfs.py
> > > > > @@ -16,11 +16,11 @@ import os
> > > > >  import shutil
> > > > >  import sys
> > > > >
> > > > > -from oe.path import copyhardlinktree
> > > > > +from oe.path import copyhardlinktree, copytree
> > > > >
> > > > >  from wic import WicError
> > > > >  from wic.pluginbase import SourcePlugin
> > > > > -from wic.misc import get_bitbake_var
> > > > > +from wic.misc import get_bitbake_var, exec_native_cmd
> > > > >
> > > > >  logger = logging.getLogger('wic')
> > > > >
> > > > > @@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
> > > > >
> > > > >  return os.path.realpath(image_rootfs_dir)
> > > > >
> > > > > +@staticmethod
> > > > > +def __get_pseudo(native_sysroot, rootfs):
> > > > > +pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
> > > > > +pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % 
> > > > > os.path.join(rootfs, "../pseudo")
> > > > > +pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
> > > > > +pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
> > > > > +pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
> > > > > +return pseudo
> > > > > +
> > > > >  @classmethod
> > > > >  def do_prepare_partition(cls, part, source_params, cr, 
> > > > > cr_workdir,
> > > > >   oe_builddir, bootimg_dir, kernel_dir,
> > > &g

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi Richard

It looks pretty much related to my patch. Will try to replicate locally and
fix it.

Thanks!

On Sun, 5 Apr 2020, 09:01 Richard Purdie, <
richard.pur...@linuxfoundation.org> wrote:

> On Wed, 2020-03-04 at 15:49 +0100, Ricardo Ribalda Delgado wrote:
> > When parameters include_path or exclude_path are passed to the rootfs
> > plugin, it will copy the partition content into a folder and make all
> > the modifications there.
> >
> > This is done using copyhardlinktree(), which does not take into
> > consideration the content of the pseudo folder, which contains the
> > information about the right permissions and ownership of the folders.
> >
> > This results in a rootfs owned by the user that is running the wic
> > command (usually UID 1000), which makes some rootfs unbootable.
> >
> > To fix this we copy the content of the pseudo folders to the new
> > folder
> > and modify the pseudo database using the "pseudo -B" command.
> >
> > Signed-off-by: Ricardo Ribalda Delgado 
> > ---
> >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++---
> >  1 file changed, 19 insertions(+), 3 deletions(-)
>
> I added these two changes to -next but we're seeing a test failure on
> Fedora, twice now:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/834
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/831
>
> I suspect its related to these patches but haven't confirmed that as
> yet...
>
> Cheers,
>
> Richard
>
> 
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137029): 
https://lists.openembedded.org/g/openembedded-core/message/137029
Mute This Topic: https://lists.openembedded.org/mt/72395318/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi Richard

I have tried with master  (0c91fcba446418ad1f71d3df9aa3b186bbd353c7) +
my two patches.
Then I have used crops/poky:fedora-30

and: oe-selftest  --run-tests wic.Wic.test_exclude_path oe-selftest
--run-tests wic.Wic.test_include_path wic.Wic.test_exclude_path

With an OK result :(
[pokyuser@fac53302a692 build]$ oe-selftest  --run-tests
wic.Wic.test_exclude_path oe-selftest  --run-tests
wic.Wic.test_include_path
2020-04-05 07:40:50,186 - oe-selftest - INFO - Adding layer libraries:
2020-04-05 07:40:50,186 - oe-selftest - INFO - /workdir/meta/lib
2020-04-05 07:40:50,186 - oe-selftest - INFO - /workdir/meta-selftest/lib
2020-04-05 07:40:50,187 - oe-selftest - INFO - Running bitbake -e to
test the configuration is valid/parsable
2020-04-05 07:40:52,149 - oe-selftest - INFO - Adding: "include
selftest.inc" in /workdir/build/conf/local.conf
2020-04-05 07:40:52,150 - oe-selftest - INFO - Adding: "include
bblayers.inc" in bblayers.conf
2020-04-05 07:40:52,150 - oe-selftest - INFO - test_include_path (wic.Wic)
2020-04-05 09:04:12,656 - oe-selftest - INFO - Keepalive message
2020-04-05 09:29:51,214 - oe-selftest - INFO -  ... ok
2020-04-05 09:29:51,230 - oe-selftest - INFO -
--
2020-04-05 09:29:51,230 - oe-selftest - INFO - Ran 1 test in 6539.080s
2020-04-05 09:29:51,231 - oe-selftest - INFO - OK
2020-04-05 09:29:52,869 - oe-selftest - INFO - RESULTS:
2020-04-05 09:29:52,870 - oe-selftest - INFO - RESULTS -
wic.Wic.test_include_path: PASSED (6539.07s)
2020-04-05 09:29:52,874 - oe-selftest - INFO - SUMMARY:
2020-04-05 09:29:52,875 - oe-selftest - INFO - oe-selftest () - Ran 1
test in 6540.086s
2020-04-05 09:29:52,875 - oe-selftest - INFO - oe-selftest - OK - All
required tests passed (successes=1, skipped=0, failures=0, errors=0)

I will start a test with mater-next + my two patches. Hopefuly I can
replicate the bug.

It is a bit weird that only crashes on fedora and not on the other distros :S


Thanks again

On Sun, Apr 5, 2020 at 9:21 AM Ricardo Ribalda via
lists.openembedded.org
 wrote:
>
> Hi Richard
>
> It looks pretty much related to my patch. Will try to replicate locally and 
> fix it.
>
> Thanks!
>
> On Sun, 5 Apr 2020, 09:01 Richard Purdie, 
>  wrote:
>>
>> On Wed, 2020-03-04 at 15:49 +0100, Ricardo Ribalda Delgado wrote:
>> > When parameters include_path or exclude_path are passed to the rootfs
>> > plugin, it will copy the partition content into a folder and make all
>> > the modifications there.
>> >
>> > This is done using copyhardlinktree(), which does not take into
>> > consideration the content of the pseudo folder, which contains the
>> > information about the right permissions and ownership of the folders.
>> >
>> > This results in a rootfs owned by the user that is running the wic
>> > command (usually UID 1000), which makes some rootfs unbootable.
>> >
>> > To fix this we copy the content of the pseudo folders to the new
>> > folder
>> > and modify the pseudo database using the "pseudo -B" command.
>> >
>> > Signed-off-by: Ricardo Ribalda Delgado 
>> > ---
>> >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++---
>> >  1 file changed, 19 insertions(+), 3 deletions(-)
>>
>> I added these two changes to -next but we're seeing a test failure on
>> Fedora, twice now:
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/834
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/831
>>
>> I suspect its related to these patches but haven't confirmed that as
>> yet...
>>
>> Cheers,
>>
>> Richard
>>
>>
> 



-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137030): 
https://lists.openembedded.org/g/openembedded-core/message/137030
Mute This Topic: https://lists.openembedded.org/mt/72395318/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi Richard


On Sun, Apr 5, 2020 at 1:04 PM Richard Purdie
 wrote:
>
> On Sun, 2020-04-05 at 11:56 +0200, Ricardo Ribalda Delgado wrote:
> > Hi Richard
> >
> > I have tried with master  (0c91fcba446418ad1f71d3df9aa3b186bbd353c7)
> > +
> > my two patches.
> > Then I have used crops/poky:fedora-30
> >
> > and: oe-selftest  --run-tests wic.Wic.test_exclude_path oe-selftest
> > --run-tests wic.Wic.test_include_path wic.Wic.test_exclude_path
> >
> > With an OK result :(
>
> I worried this may be the case. There definitely is something wrong and
> I could replicate it a third time too.
>
> My guess is that there are perhaps files in /tmp/ which are logged in
> the pseudo database but don't exist when the second rootfs is created.
> Since the files no longer exist, the database integrity check fails.

That is definitely a good hypothesis. Since you can replicate the bug,
can you try to add also the attached patch to your tree and re-run the
test.

What it does, is to remove a file from the database if it has
vanished, during the bitbake move.

If it works I can add it to the beginning of my series, or even trying
to upstream it to pseudo.

Best regards

>
> Why that only occurs on Fedora case on the autobuilder, I don't know,
> I'm just hypothesising right now...
>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda
From bbddec527d9056e1feda0d21edd8ddb9d64623b2 Mon Sep 17 00:00:00 2001
From: Ricardo Ribalda Delgado 
Date: Sun, 5 Apr 2020 13:47:12 +0200
Subject: [PATCH] pseudo: During DB fix, remove files that do not exist

If a file does not exist, either because it has been removed outside
bitbake, or because only some of the files have been moved to a
different location, delete it from the pseudo-db is the user decides to
fix the database.

Signed-off-by: Ricardo Ribalda Delgado 
---
 ...ixup-remove-files-that-do-not-exist-.patch | 48 +++
 meta/recipes-devtools/pseudo/pseudo_git.bb|  1 +
 2 files changed, 49 insertions(+)
 create mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch

diff --git a/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
new file mode 100644
index 00..8aec1c9551
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
@@ -0,0 +1,48 @@
+From b0902e36108b49e6bc88d6b251cc2f8cffcd5a13 Mon Sep 17 00:00:00 2001
+From: Ricardo Ribalda 
+Date: Sun, 5 Apr 2020 11:40:30 +
+Subject: [PATCH] pseudo: On a DB fixup remove files that do not exist anymore
+
+If the user decides to fix a database, remove the files that do not
+exist anymore.
+If only DB test is selected do not change the behaviour (return error).
+
+Signed-off-by: Ricardo Ribalda 
+---
+ pseudo.c | 13 ++---
+ 1 file changed, 10 insertions(+), 3 deletions(-)
+
+diff --git a/pseudo.c b/pseudo.c
+index 0f5850e..98e5b0c 100644
+--- a/pseudo.c
 b/pseudo.c
+@@ -1087,9 +1087,15 @@ pseudo_db_check(int fix) {
+ 			int fixup_needed = 0;
+ 			pseudo_debug(PDBGF_DB, "Checking <%s>\n", m->path);
+ 			if (lstat(m->path, &buf)) {
+-errors = EXIT_FAILURE;
+-pseudo_diag("can't stat <%s>\n", m->path);
+-continue;
++if (!fix) {
++	pseudo_diag("can't stat <%s>\n", m->path);
++	errors = EXIT_FAILURE;
++	continue;
++} else {
++	pseudo_debug("can't stat <%s>\n", m->path);
++	fixup_needed = 2;
++	goto do_fixup;
++}
+ 			}
+ 			/* can't check for device type mismatches, uid/gid, or
+ 			 * permissions, because those are the very things we
+@@ -1125,6 +1131,7 @@ pseudo_db_check(int fix) {
+ 	S_ISDIR(m->mode));
+ fixup_needed = 2;
+ 			}
++			do_fixup:
+ 			if (fixup_needed) {
+ /* in fixup mode, either delete (mismatches) or
+  * correct (dev/ino).
+-- 
+2.21.1
+
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 89e43c5996..a3049c5e6c 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -11,6 +11,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo \
file://0001-realpath.c-Remove-trailing-slashes.patch \
file://0006-xattr-adjust-for-attr-2.4.48-release.patch \
file://seccomp.patch \
+   file://0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch \
"
 
 SRCREV = "060058bb29f70b244e685b3c704eb0641b736f73"
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137032): 
https://lists.openembedded.org/g/openembedded-core/message/137032
Mute This Topic: https://lists.openembedded.org/mt/72395318/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Ups, wrong version

This is the one, sorry.

On Sun, Apr 5, 2020 at 1:53 PM Ricardo Ribalda Delgado
 wrote:
>
> Hi Richard
>
>
> On Sun, Apr 5, 2020 at 1:04 PM Richard Purdie
>  wrote:
> >
> > On Sun, 2020-04-05 at 11:56 +0200, Ricardo Ribalda Delgado wrote:
> > > Hi Richard
> > >
> > > I have tried with master  (0c91fcba446418ad1f71d3df9aa3b186bbd353c7)
> > > +
> > > my two patches.
> > > Then I have used crops/poky:fedora-30
> > >
> > > and: oe-selftest  --run-tests wic.Wic.test_exclude_path oe-selftest
> > > --run-tests wic.Wic.test_include_path wic.Wic.test_exclude_path
> > >
> > > With an OK result :(
> >
> > I worried this may be the case. There definitely is something wrong and
> > I could replicate it a third time too.
> >
> > My guess is that there are perhaps files in /tmp/ which are logged in
> > the pseudo database but don't exist when the second rootfs is created.
> > Since the files no longer exist, the database integrity check fails.
>
> That is definitely a good hypothesis. Since you can replicate the bug,
> can you try to add also the attached patch to your tree and re-run the
> test.
>
> What it does, is to remove a file from the database if it has
> vanished, during the bitbake move.
>
> If it works I can add it to the beginning of my series, or even trying
> to upstream it to pseudo.
>
> Best regards
>
> >
> > Why that only occurs on Fedora case on the autobuilder, I don't know,
> > I'm just hypothesising right now...
> >
> > Cheers,
> >
> > Richard
> >
>
>
> --
> Ricardo Ribalda



-- 
Ricardo Ribalda
From bbddec527d9056e1feda0d21edd8ddb9d64623b2 Mon Sep 17 00:00:00 2001
From: Ricardo Ribalda Delgado 
Date: Sun, 5 Apr 2020 13:47:12 +0200
Subject: [PATCH] pseudo: During DB fix, remove files that do not exist

If a file does not exist, either because it has been removed outside
bitbake, or because only some of the files have been moved to a
different location, delete it from the pseudo-db is the user decides to
fix the database.

Signed-off-by: Ricardo Ribalda Delgado 
---
 ...ixup-remove-files-that-do-not-exist-.patch | 48 +++
 meta/recipes-devtools/pseudo/pseudo_git.bb|  1 +
 2 files changed, 49 insertions(+)
 create mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch

diff --git a/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
new file mode 100644
index 00..8aec1c9551
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
@@ -0,0 +1,48 @@
+From b0902e36108b49e6bc88d6b251cc2f8cffcd5a13 Mon Sep 17 00:00:00 2001
+From: Ricardo Ribalda 
+Date: Sun, 5 Apr 2020 11:40:30 +
+Subject: [PATCH] pseudo: On a DB fixup remove files that do not exist anymore
+
+If the user decides to fix a database, remove the files that do not
+exist anymore.
+If only DB test is selected do not change the behaviour (return error).
+
+Signed-off-by: Ricardo Ribalda 
+---
+ pseudo.c | 13 ++---
+ 1 file changed, 10 insertions(+), 3 deletions(-)
+
+diff --git a/pseudo.c b/pseudo.c
+index 0f5850e..98e5b0c 100644
+--- a/pseudo.c
 b/pseudo.c
+@@ -1087,9 +1087,15 @@ pseudo_db_check(int fix) {
+ 			int fixup_needed = 0;
+ 			pseudo_debug(PDBGF_DB, "Checking <%s>\n", m->path);
+ 			if (lstat(m->path, &buf)) {
+-errors = EXIT_FAILURE;
+-pseudo_diag("can't stat <%s>\n", m->path);
+-continue;
++if (!fix) {
++	pseudo_diag("can't stat <%s>\n", m->path);
++	errors = EXIT_FAILURE;
++	continue;
++} else {
++	pseudo_debug("can't stat <%s>\n", m->path);
++	fixup_needed = 2;
++	goto do_fixup;
++}
+ 			}
+ 			/* can't check for device type mismatches, uid/gid, or
+ 			 * permissions, because those are the very things we
+@@ -1125,6 +1131,7 @@ pseudo_db_check(int fix) {
+ 	S_ISDIR(m->mode));
+ fixup_needed = 2;
+ 			}
++			do_fixup:
+ 			if (fixup_needed) {
+ /* in fixup mode, either delete (mismatches) or
+  * correct (dev/ino).
+-- 
+2.21.1
+
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 89e43c5996..a3049c5e6c 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -11,6 +11,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo \
file://0001-realpath.c-Remove-trailing-slashes.patch \
file://0006-xattr-adjust-for-attr-2.4.48-release.p

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi Richard

Thanks for testing it out. Seems that I am either not very clever or I
should not work on Sundays (or both), v1, and v2 where exactly the
same. I am terribly sorry to spam you.

I know it is abusing your patience, but can you try again. I would
love to try it on my setup , but I cannot replicate the bug.

As we speak I am running oe-selftest--run-tests wic.Wic on poky/fedora-30...

Thanks, and sorry again.

On Sun, Apr 5, 2020 at 5:29 PM Richard Purdie
 wrote:
>
> On Sun, 2020-04-05 at 14:01 +0200, Ricardo Ribalda Delgado wrote:
> > Ups, wrong version
> >
> > This is the one, sorry.
>
> It didn't build unfortunately:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/837
>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda
From 9c2bd9553e81fd16a9e3aad2f20b7a86918de4be Mon Sep 17 00:00:00 2001
From: Ricardo Ribalda Delgado 
Date: Sun, 5 Apr 2020 13:47:12 +0200
Subject: [PATCH v2] pseudo: During DB fix, remove files that do not exist

If a file does not exist, either because it has been removed outside
bitbake, or because only some of the files have been moved to a
different location, delete it from the pseudo-db is the user decides to
fix the database.

Signed-off-by: Ricardo Ribalda Delgado 
---
 ...ixup-remove-files-that-do-not-exist-.patch | 48 +++
 meta/recipes-devtools/pseudo/pseudo_git.bb|  1 +
 2 files changed, 49 insertions(+)
 create mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch

diff --git a/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
new file mode 100644
index 00..26efb0a113
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
@@ -0,0 +1,48 @@
+From b0902e36108b49e6bc88d6b251cc2f8cffcd5a13 Mon Sep 17 00:00:00 2001
+From: Ricardo Ribalda 
+Date: Sun, 5 Apr 2020 11:40:30 +
+Subject: [PATCH] pseudo: On a DB fixup remove files that do not exist anymore
+
+If the user decides to fix a database, remove the files that do not
+exist anymore.
+If only DB test is selected do not change the behaviour (return error).
+
+Signed-off-by: Ricardo Ribalda 
+---
+ pseudo.c | 13 ++---
+ 1 file changed, 10 insertions(+), 3 deletions(-)
+
+diff --git a/pseudo.c b/pseudo.c
+index 0f5850e..98e5b0c 100644
+--- a/pseudo.c
 b/pseudo.c
+@@ -1087,9 +1087,15 @@ pseudo_db_check(int fix) {
+ 			int fixup_needed = 0;
+ 			pseudo_debug(PDBGF_DB, "Checking <%s>\n", m->path);
+ 			if (lstat(m->path, &buf)) {
+-errors = EXIT_FAILURE;
+-pseudo_diag("can't stat <%s>\n", m->path);
+-continue;
++if (!fix) {
++	pseudo_diag("can't stat <%s>\n", m->path);
++	errors = EXIT_FAILURE;
++	continue;
++} else {
++	pseudo_debug(PDBGF_DB, "can't stat <%s>\n", m->path);
++	fixup_needed = 2;
++	goto do_fixup;
++}
+ 			}
+ 			/* can't check for device type mismatches, uid/gid, or
+ 			 * permissions, because those are the very things we
+@@ -1125,6 +1131,7 @@ pseudo_db_check(int fix) {
+ 	S_ISDIR(m->mode));
+ fixup_needed = 2;
+ 			}
++			do_fixup:
+ 			if (fixup_needed) {
+ /* in fixup mode, either delete (mismatches) or
+  * correct (dev/ino).
+-- 
+2.21.1
+
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 89e43c5996..a3049c5e6c 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -11,6 +11,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo \
file://0001-realpath.c-Remove-trailing-slashes.patch \
file://0006-xattr-adjust-for-attr-2.4.48-release.patch \
file://seccomp.patch \
+   file://0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch \
"
 
 SRCREV = "060058bb29f70b244e685b3c704eb0641b736f73"
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137036): 
https://lists.openembedded.org/g/openembedded-core/message/137036
Mute This Topic: https://lists.openembedded.org/mt/72395318/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi

On Mon, Apr 6, 2020 at 11:00 AM Richard Purdie
 wrote:
>
> On Sun, 2020-04-05 at 17:44 +0200, Ricardo Ribalda Delgado wrote:
> > Hi Richard
> >
> > Thanks for testing it out. Seems that I am either not very clever or
> > I
> > should not work on Sundays (or both), v1, and v2 where exactly the
> > same. I am terribly sorry to spam you.
> >
> > I know it is abusing your patience, but can you try again. I would
> > love to try it on my setup , but I cannot replicate the bug.
> >
> > As we speak I am running oe-selftest--run-tests wic.Wic on
> > poky/fedora-30...
>
> I did re-run it, it took a while since a pseudo change invalidates
> sstate. It passed:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/838
>

Thanks a lot for testing it. I will prepare a new series including that patch.

> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137044): 
https://lists.openembedded.org/g/openembedded-core/message/137044
Mute This Topic: https://lists.openembedded.org/mt/72395318/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] pseudo: On a DB fixup remove files that do not exist anymore

[Ricardo Ribalda]

From: Ricardo Ribalda 

If the user decides to fix a database, remove the files that do not
exist anymore.
If only DB test is selected do not change the behaviour (return error).

Signed-off-by: Ricardo Ribalda 
---
 pseudo.c | 13 ++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/pseudo.c b/pseudo.c
index 0f5850e..d430a40 100644
--- a/pseudo.c
+++ b/pseudo.c
@@ -1087,9 +1087,15 @@ pseudo_db_check(int fix) {
int fixup_needed = 0;
pseudo_debug(PDBGF_DB, "Checking <%s>\n", m->path);
if (lstat(m->path, &buf)) {
-   errors = EXIT_FAILURE;
-   pseudo_diag("can't stat <%s>\n", m->path);
-   continue;
+   if (!fix) {
+   pseudo_diag("can't stat <%s>\n", 
m->path);
+   errors = EXIT_FAILURE;
+   continue;
+   } else {
+   pseudo_debug(PDBGF_DB, "can't stat 
<%s>\n", m->path);
+   fixup_needed = 2;
+   goto do_fixup;
+   }
}
/* can't check for device type mismatches, uid/gid, or
 * permissions, because those are the very things we
@@ -1125,6 +1131,7 @@ pseudo_db_check(int fix) {
S_ISDIR(m->mode));
fixup_needed = 2;
}
+   do_fixup:
if (fixup_needed) {
/* in fixup mode, either delete (mismatches) or
 * correct (dev/ino).
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137045): 
https://lists.openembedded.org/g/openembedded-core/message/137045
Mute This Topic: https://lists.openembedded.org/mt/72808260/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi

On Mon, Apr 6, 2020 at 11:04 AM Ricardo Ribalda via
lists.openembedded.org
 wrote:
>
> Hi
>
> On Mon, Apr 6, 2020 at 11:00 AM Richard Purdie
>  wrote:
> >
> > On Sun, 2020-04-05 at 17:44 +0200, Ricardo Ribalda Delgado wrote:
> > > Hi Richard
> > >
> > > Thanks for testing it out. Seems that I am either not very clever or
> > > I
> > > should not work on Sundays (or both), v1, and v2 where exactly the
> > > same. I am terribly sorry to spam you.
> > >
> > > I know it is abusing your patience, but can you try again. I would
> > > love to try it on my setup , but I cannot replicate the bug.
> > >
> > > As we speak I am running oe-selftest--run-tests wic.Wic on
> > > poky/fedora-30...
> >
> > I did re-run it, it took a while since a pseudo change invalidates
> > sstate. It passed:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/838
> >
>
> Thanks a lot for testing it. I will prepare a new series including that patch.

I saw that you already queue it for master-next:

https://git.openembedded.org/openembedded-core/log/?h=master-next

So I will not resend the series ;)



I have send the Individual patch to pseudo, lets see if it is merged.

Thanks!


>
> > Cheers,
> >
> > Richard
> >
>
>
> --
> Ricardo Ribalda
> 



--
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137046): 
https://lists.openembedded.org/g/openembedded-core/message/137046
Mute This Topic: https://lists.openembedded.org/mt/72395318/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 2/2] wic: Add --embed-rootfs argument

[Ricardo Ribalda]

Hi Paul

Yes it got merged
https://git.openembedded.org/openembedded-core/commit/?id=efdcf94801f6abe8e4099e324d9a3deccd8d4384

Even though the functionality looks similar, the devil is on the
details ;). One thing is adding a file, this is  just  file/directory
that is referenced as a path, and the other is a complete rootfs,
which takes care of permissions and can be invoked with a path, or a
bitbake target...

But if you come up with a interface that cover both usecases I can
help implementing it.

Best regards.

On Tue, Apr 7, 2020 at 7:47 PM Paul Barker  wrote:
>
> On Fri, 3 Apr 2020 21:52:52 +0200
> Ricardo Ribalda Delgado  wrote:
>
> > ping?
>
> I'm still concerned that `--embed-rootfs` and `--include-path` are too
> similar and we should instead have one option that can handle both use cases.
>
> Has this already been merged?
>
> >
> > On Thu, Mar 5, 2020 at 1:26 PM Ricardo Ribalda Delgado
> >  wrote:
> > >
> > > Hi Paul,
> > >
> > > On Thu, Mar 5, 2020 at 10:37 AM Paul Barker  wrote:
> > > >
> > > > On Wed,  4 Mar 2020 15:49:36 +0100
> > > > Ricardo Ribalda Delgado  wrote:
> > > >
> > > > > This option adds the content of a rootfs on a specific location on the
> > > > > rootfs.
> > > > >
> > > > > It is very useful for making a partition that contains the rootfs for 
> > > > > a
> > > > > host and a target Eg:
> > > > >
> > > > > / -> Roofs for the host
> > > > > /export/ -> Rootfs for the target (which will netboot)
> > > > >
> > > > > Although today we support making a partition for "/export" this might
> > > > > not be compatible with some upgrade systems, or we might be limited by
> > > > > the number of partitions.
> > > > >
> > > > > With this patch we can use something like:
> > > > >
> > > > > part / --source rootfs --embed-rootfs target-image /export 
> > > > > --embed-rootfs target-image2 /export2
> > > >
> > > > I like this but it still leaves confusion between `--include-path` and
> > > > --embed-rootfs` as they're similar but slightly different. Can we just 
> > > > modify
> > > > `--include-path` to have this syntax?
> > >
> > > I think they are different enough.
> > >
> > > - include-path ads a file/folder
> > > - embed-rootfs adds a rootfs, which is either a folder or a image.bb file.
> > >
> > >
> > > >
> > > > >
> > > > > on the .wks file.
> > > > >
> > > > > Signed-off-by: Ricardo Ribalda Delgado 
> > > > > ---
> > > > >  scripts/lib/wic/help.py  |  8 
> > > > >  scripts/lib/wic/ksparser.py  |  1 +
> > > > >  scripts/lib/wic/partition.py |  1 +
> > > > >  scripts/lib/wic/plugins/source/rootfs.py | 22 +-
> > > > >  4 files changed, 31 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> > > > > index 4d342fcf05..140dc504cd 100644
> > > > > --- a/scripts/lib/wic/help.py
> > > > > +++ b/scripts/lib/wic/help.py
> > > > > @@ -979,6 +979,14 @@ DESCRIPTION
> > > > >   copies. This option only has an effect with 
> > > > > the rootfs
> > > > >   source plugin.
> > > > >
> > > > > + --embed-rootfs: This option is specific to wic. It embeds a 
> > > > > rootfs into
> > > > > + the given path to the resulting image. The 
> > > > > option
> > > > > + contains two fields, the roofs and the 
> > > > > path, separated
> > > > > + by a space. The rootfs follows the same 
> > > > > logic as the
> > > > > + rootfs-dir argument. Multiple options can 
> > > > > be provided
> > > > > + in order to embed multiple rootfs. This 
> > > > > option only has
> > > > > + an effect with the rootfs source plugin.
> > > > > +
> > > > >   --extra-space: This option is specific to wic. It adds extra
> >

Re: [OE-core] [PATCH 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi Paul

Thanks for your review, It has been already merged, but if there is
something wrong we can send a patch fixing it.

https://git.openembedded.org/openembedded-core/commit/?id=36993eea89d1c011397b7692b9b8d61b499d0171

On Tue, Apr 7, 2020 at 8:13 PM Paul Barker  wrote:
>
> On Fri, 3 Apr 2020 21:53:39 +0200
> Ricardo Ribalda Delgado  wrote:
>
> > ping?
>
> I think that '../pseudo' should not be used here. I'll explain inline...
>
> > >
> > > This results in a rootfs owned by the user that is running the wic
> > > command (usually UID 1000), which makes some rootfs unbootable.
> > >
> > > To fix this we copy the content of the pseudo folders to the new folder
> > > and modify the pseudo database using the "pseudo -B" command.
> > >
> > > Signed-off-by: Ricardo Ribalda Delgado 
> > > ---
> > >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++---
> > >  1 file changed, 19 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
> > > b/scripts/lib/wic/plugins/source/rootfs.py
> > > index 705aeb5563..40419a64b3 100644
> > > --- a/scripts/lib/wic/plugins/source/rootfs.py
> > > +++ b/scripts/lib/wic/plugins/source/rootfs.py
> > > @@ -16,11 +16,11 @@ import os
> > >  import shutil
> > >  import sys
> > >
> > > -from oe.path import copyhardlinktree
> > > +from oe.path import copyhardlinktree, copytree
> > >
> > >  from wic import WicError
> > >  from wic.pluginbase import SourcePlugin
> > > -from wic.misc import get_bitbake_var
> > > +from wic.misc import get_bitbake_var, exec_native_cmd
> > >
> > >  logger = logging.getLogger('wic')
> > >
> > > @@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
> > >
> > >  return os.path.realpath(image_rootfs_dir)
> > >
> > > +@staticmethod
> > > +def __get_pseudo(native_sysroot, rootfs):
> > > +pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
> > > +pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % 
> > > os.path.join(rootfs, "../pseudo")
> > > +pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
> > > +pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
> > > +pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
> > > +return pseudo
> > > +
> > >  @classmethod
> > >  def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
> > >   oe_builddir, bootimg_dir, kernel_dir,
> > > @@ -78,9 +87,16 @@ class RootfsPlugin(SourcePlugin):
> > >
> > >  if os.path.lexists(new_rootfs):
> > >  shutil.rmtree(os.path.join(new_rootfs))
> > > -
> > >  copyhardlinktree(part.rootfs_dir, new_rootfs)
> > >
> > > +if os.path.lexists(os.path.join(new_rootfs, "../pseudo")):
>
> new_rootfs is set by the following statement a few lines above:
>
> new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % 
> part.lineno))
>
> Consider that `cr_workdir` may contain multiple rootfs staging directories
> corresponding to multiple lines in the wks file, for example if a rootfs
> image is duplicated into multiple partitions for redundancy. In that case
> `os.path.join(new_rootfs, "../pseudo")` will clash between these different
> rootfs copies.
>
> Let's use an explicit path instead, such as:
>
> new_pseudo = os.path.realpath(os.path.join(cr_workdir, "pseudo%d" % 
> part.lineno))

The reason to have that path was to follow the same structure as the
real image.bb.

If there are multiple partitions on the .wic file the different
partitions are done one by one, not
in parallel.

So
../pseudo will be created  for partition1
then it will be used to generate the partition1

../pseudo will be deleted
../pseudo will be created  for partition2

Even if they use the same partition, the code works (and ../pseudo is
useless once the partition is generated)

>
> > > +shutil.rmtree(os.path.join(new_rootfs, "../pseudo"))
> > > +copytree(os.path.join(part.rootfs_dir, "../pseudo"),
>
> part.rootfs_dir is whatever is given as the option to `--rootfs-dir`. There
> is no guarantee that `../psuedo` is valid or if it corresponds to the rootfs
> directory given. It's unsafe to step up the directory tre

Re: [OE-core] [PATCH 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi

On Tue, Apr 7, 2020 at 9:02 PM Paul Barker  wrote:
>
> On Tue, 7 Apr 2020 20:40:18 +0200
> Ricardo Ribalda Delgado  wrote:
>
> > Hi Paul
> >
> > Thanks for your review, It has been already merged, but if there is
> > something wrong we can send a patch fixing it.
> >
> > https://git.openembedded.org/openembedded-core/commit/?id=36993eea89d1c011397b7692b9b8d61b499d0171
> >
> > On Tue, Apr 7, 2020 at 8:13 PM Paul Barker  wrote:
> > >
> > > On Fri, 3 Apr 2020 21:53:39 +0200
> > > Ricardo Ribalda Delgado  wrote:
> > >
> > > > ping?
> > >
> > > I think that '../pseudo' should not be used here. I'll explain inline...
> > >
> > > > >
> > > > > This results in a rootfs owned by the user that is running the wic
> > > > > command (usually UID 1000), which makes some rootfs unbootable.
> > > > >
> > > > > To fix this we copy the content of the pseudo folders to the new 
> > > > > folder
> > > > > and modify the pseudo database using the "pseudo -B" command.
> > > > >
> > > > > Signed-off-by: Ricardo Ribalda Delgado 
> > > > > ---
> > > > >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++---
> > > > >  1 file changed, 19 insertions(+), 3 deletions(-)
> > > > >
> > > > > diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
> > > > > b/scripts/lib/wic/plugins/source/rootfs.py
> > > > > index 705aeb5563..40419a64b3 100644
> > > > > --- a/scripts/lib/wic/plugins/source/rootfs.py
> > > > > +++ b/scripts/lib/wic/plugins/source/rootfs.py
> > > > > @@ -16,11 +16,11 @@ import os
> > > > >  import shutil
> > > > >  import sys
> > > > >
> > > > > -from oe.path import copyhardlinktree
> > > > > +from oe.path import copyhardlinktree, copytree
> > > > >
> > > > >  from wic import WicError
> > > > >  from wic.pluginbase import SourcePlugin
> > > > > -from wic.misc import get_bitbake_var
> > > > > +from wic.misc import get_bitbake_var, exec_native_cmd
> > > > >
> > > > >  logger = logging.getLogger('wic')
> > > > >
> > > > > @@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
> > > > >
> > > > >  return os.path.realpath(image_rootfs_dir)
> > > > >
> > > > > +@staticmethod
> > > > > +def __get_pseudo(native_sysroot, rootfs):
> > > > > +pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
> > > > > +pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % 
> > > > > os.path.join(rootfs, "../pseudo")
> > > > > +pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
> > > > > +pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
> > > > > +pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
> > > > > +return pseudo
> > > > > +
> > > > >  @classmethod
> > > > >  def do_prepare_partition(cls, part, source_params, cr, 
> > > > > cr_workdir,
> > > > >   oe_builddir, bootimg_dir, kernel_dir,
> > > > > @@ -78,9 +87,16 @@ class RootfsPlugin(SourcePlugin):
> > > > >
> > > > >  if os.path.lexists(new_rootfs):
> > > > >  shutil.rmtree(os.path.join(new_rootfs))
> > > > > -
> > > > >  copyhardlinktree(part.rootfs_dir, new_rootfs)
> > > > >
> > > > > +if os.path.lexists(os.path.join(new_rootfs, 
> > > > > "../pseudo")):
> > >
> > > new_rootfs is set by the following statement a few lines above:
> > >
> > > new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % 
> > > part.lineno))
> > >
> > > Consider that `cr_workdir` may contain multiple rootfs staging directories
> > > corresponding to multiple lines in the wks file, for example if a rootfs
> > > image is duplicated into multiple partitions for redundancy. In that case
> > > `os.path.join(new_rootfs, "../pseudo")` will clash between these different
> > > rootfs copies.
> > >
> > > Let&#x

[OE-core] [PATCH v3 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.

This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.

This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.

This bug can be easily triggerd with the following .wks

part / --source rootfs --fstype=ext4 --exclude-path=home

And this sequence:

$ wic create test-permissions -e core-image-minimal -o test/
$ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
$ ls -la /mnt/etc/shadow

To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.

If the rootfs is not a rootfs generated by bitbake a warning is shown
making the user aware that the permissions on the target might not match
what he expects.

WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions 
will be invalid

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/partition.py |  7 +++--
 scripts/lib/wic/plugins/source/rootfs.py | 35 +---
 2 files changed, 35 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 2d95f78439..b02711be37 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -190,7 +190,7 @@ class Partition():
(self.mountpoint, self.size, self.fixed_size))
 
 def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
-   native_sysroot, real_rootfs = True):
+   native_sysroot, real_rootfs = True, pseudo_dir = None):
 """
 Prepare content for a rootfs partition i.e. create a partition
 and fill it from a /rootfs dir.
@@ -198,8 +198,9 @@ class Partition():
 Currently handles ext2/3/4, btrfs, vfat and squashfs.
 """
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
-p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
- "%s/../pseudo" %  rootfs_dir)
+if (pseudo_dir == None):
+pseudo_dir = "%s/../pseudo" %  rootfs_dir
+p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
 p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
 p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
 pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 705aeb5563..f161d1eae0 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -16,11 +16,11 @@ import os
 import shutil
 import sys
 
-from oe.path import copyhardlinktree
+from oe.path import copyhardlinktree, copytree
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
-from wic.misc import get_bitbake_var
+from wic.misc import get_bitbake_var, exec_native_cmd
 
 logger = logging.getLogger('wic')
 
@@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
 
 return os.path.realpath(image_rootfs_dir)
 
+@staticmethod
+def __get_pseudo(native_sysroot, rootfs, pseudo_dir):
+pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+return pseudo
+
 @classmethod
 def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
  oe_builddir, bootimg_dir, kernel_dir,
@@ -68,8 +77,14 @@ class RootfsPlugin(SourcePlugin):
"it is not a valid path, exiting" % 
part.rootfs_dir)
 
 part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
+if not os.path.lexists(pseudo_dir):
+logger.warn("%s folder does not exist. "
+"Usernames and permissions will be invalid " % 
pseudo_dir)
+pseudo_dir = None
 
 new_rootfs = None
+new_pseudo = None
 # Handle excluded paths.
 if part.exclude_path or part.include_path:
 # We need a new rootfs directory we can delete files from. Copy to
@@ -78,9 +93,20 @@ class RootfsPlugin(SourcePlugin):
 
  

[OE-core] [PATCH v3 2/2] wic: Continue if excluded_path does not exist

[Ricardo Ribalda]

If an excuded path does not exist, continue without an error.
This allows to seamleasly reuse .wks among different projects.

Eg:

part / --source rootfs --fstype=ext4 --exclude-path=/opt/private_keys

Where /opt/private_keys in only populated by some of the image.bb files.

Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/plugins/source/rootfs.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index f161d1eae0..871f349b8c 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -117,6 +117,9 @@ class RootfsPlugin(SourcePlugin):
 sys.exit(1)
 
 full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+if not os.path.lexists(full_path):
+continue
 
 # Disallow climbing outside of parent directory using '..',
 # because doing so could be quite disastrous (we will delete 
the
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137118): 
https://lists.openembedded.org/g/openembedded-core/message/137118
Mute This Topic: https://lists.openembedded.org/mt/72870573/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 2/2] wic: Add --embed-rootfs argument

[Ricardo Ribalda]

Hi Again



On Tue, Apr 7, 2020 at 8:19 PM Ricardo Ribalda via
lists.openembedded.org
 wrote:
>
> Hi Paul
>
> Yes it got merged
> https://git.openembedded.org/openembedded-core/commit/?id=efdcf94801f6abe8e4099e324d9a3deccd8d4384
>
> Even though the functionality looks similar, the devil is on the
> details ;). One thing is adding a file, this is  just  file/directory
> that is referenced as a path, and the other is a complete rootfs,
> which takes care of permissions and can be invoked with a path, or a
> bitbake target...
>
> But if you come up with a interface that cover both usecases I can
> help implementing it.

Looking into merging the functionality of embed-rootfs into include-path

I guess the first step is to modify include-path so it can accept 1 or
2 arguments.

If 1 argument:
the functionality is the same as now

If 2 arguments:
The second argument is considered the destination:
eg: --include-path home  /home/


The second step is to accept not only file/folders but also images.
The challenge here is that we can name an image in multiple ways:
- path: tmp/work/machine/core-image-minimal/1.0-r0/rootfs/
- name: core-image-minimal
- keyword that it is provided by wic: -rootfs-dir target-image=/workir

Which can lead to a name collision. What do we do in this case? We
prefer the directory among the image?


The third step is, as you mentioned on the irc,  to fix also the
permissions of the files/directories, not only for images.
- Maybe leave files owned by root if no pseudo database exists and
clear setuid/setgid bits for safety.

Am I in the right direction ? something I am missing?



>
> Best regards.
>
> On Tue, Apr 7, 2020 at 7:47 PM Paul Barker  wrote:
> >
> > On Fri, 3 Apr 2020 21:52:52 +0200
> > Ricardo Ribalda Delgado  wrote:
> >
> > > ping?
> >
> > I'm still concerned that `--embed-rootfs` and `--include-path` are too
> > similar and we should instead have one option that can handle both use 
> > cases.
> >
> > Has this already been merged?
> >
> > >
> > > On Thu, Mar 5, 2020 at 1:26 PM Ricardo Ribalda Delgado
> > >  wrote:
> > > >
> > > > Hi Paul,
> > > >
> > > > On Thu, Mar 5, 2020 at 10:37 AM Paul Barker  
> > > > wrote:
> > > > >
> > > > > On Wed,  4 Mar 2020 15:49:36 +0100
> > > > > Ricardo Ribalda Delgado  wrote:
> > > > >
> > > > > > This option adds the content of a rootfs on a specific location on 
> > > > > > the
> > > > > > rootfs.
> > > > > >
> > > > > > It is very useful for making a partition that contains the rootfs 
> > > > > > for a
> > > > > > host and a target Eg:
> > > > > >
> > > > > > / -> Roofs for the host
> > > > > > /export/ -> Rootfs for the target (which will netboot)
> > > > > >
> > > > > > Although today we support making a partition for "/export" this 
> > > > > > might
> > > > > > not be compatible with some upgrade systems, or we might be limited 
> > > > > > by
> > > > > > the number of partitions.
> > > > > >
> > > > > > With this patch we can use something like:
> > > > > >
> > > > > > part / --source rootfs --embed-rootfs target-image /export 
> > > > > > --embed-rootfs target-image2 /export2
> > > > >
> > > > > I like this but it still leaves confusion between `--include-path` and
> > > > > --embed-rootfs` as they're similar but slightly different. Can we 
> > > > > just modify
> > > > > `--include-path` to have this syntax?
> > > >
> > > > I think they are different enough.
> > > >
> > > > - include-path ads a file/folder
> > > > - embed-rootfs adds a rootfs, which is either a folder or a image.bb 
> > > > file.
> > > >
> > > >
> > > > >
> > > > > >
> > > > > > on the .wks file.
> > > > > >
> > > > > > Signed-off-by: Ricardo Ribalda Delgado 
> > > > > > ---
> > > > > >  scripts/lib/wic/help.py  |  8 
> > > > > >  scripts/lib/wic/ksparser.py  |  1 +
> > > > > >  scripts/lib/wic/partition.py |  1 +
> > > > > >  scripts/lib/wic/plugins/source/rootfs.py | 22 
> > > > > &

[OE-core] [PATCH v4 4/4] wic: Continue if excluded_path does not exist

[Ricardo Ribalda]

If an excuded path does not exist, continue without an error.
This allows to seamleasly reuse .wks among different projects.

Eg:

part / --source rootfs --fstype=ext4 --exclude-path=/opt/private_keys

Where /opt/private_keys in only populated by some of the image.bb files.

Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/plugins/source/rootfs.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index d3742802bb..8874ac62c4 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -131,6 +131,9 @@ class RootfsPlugin(SourcePlugin):
 sys.exit(1)
 
 full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+if not os.path.lexists(full_path):
+continue
 
 # Disallow climbing outside of parent directory using '..',
 # because doing so could be quite disastrous (we will delete 
the
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137146): 
https://lists.openembedded.org/g/openembedded-core/message/137146
Mute This Topic: https://lists.openembedded.org/mt/72893666/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v4 3/4] wic: Add --change-directory argument

[Ricardo Ribalda]

This option allows to specify which part of a rootfs is going to be
included, the same way the -C argument on tar.

Thanks to this option we can make sure the permissions and usernames
on the target partition are respected, and also simplify the creation of
splitted partitons, not neeting to invoke external vars or using .wks.in
files. Eg:

part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
part /etc --source rootfs --fstype=ext4 --change-directory=/etc

Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/help.py  |  6 ++
 scripts/lib/wic/ksparser.py  |  1 +
 scripts/lib/wic/partition.py |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 17 ++---
 4 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 1e3d06a87b..62a2a90e79 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -980,6 +980,12 @@ DESCRIPTION
  copies. This option only has an effect with the rootfs
  source plugin.
 
+ --change-directory: This option is specific to wic. It changes to the
+ given directory before copying the files. This
+ option is useful when we want to split a rootfs in
+ multiple partitions and we want to keep the right
+ permissions and usernames in all the partitions.
+
  --extra-space: This option is specific to wic. It adds extra
 space after the space filled by the content
 of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 650b976223..c60869d397 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -138,6 +138,7 @@ class KickStart():
 part.add_argument('--align', type=int)
 part.add_argument('--exclude-path', nargs='+')
 part.add_argument('--include-path', nargs='+')
+part.add_argument('--change-directory')
 part.add_argument("--extra-space", type=sizetype)
 part.add_argument('--fsoptions', dest='fsopts')
 part.add_argument('--fstype', default='vfat',
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index d850fbd1b1..3240be072a 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -31,6 +31,7 @@ class Partition():
 self.extra_space = args.extra_space
 self.exclude_path = args.exclude_path
 self.include_path = args.include_path
+self.change_directory = args.change_directory
 self.fsopts = args.fsopts
 self.fstype = args.fstype
 self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index caad9efccc..d3742802bb 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -86,14 +86,25 @@ class RootfsPlugin(SourcePlugin):
 new_rootfs = None
 new_pseudo = None
 # Handle excluded paths.
-if part.exclude_path or part.include_path:
+if part.exclude_path or part.include_path or part.change_directory:
 # We need a new rootfs directory we can delete files from. Copy to
 # workdir.
 new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" 
% part.lineno))
 
 if os.path.lexists(new_rootfs):
 shutil.rmtree(os.path.join(new_rootfs))
-copyhardlinktree(part.rootfs_dir, new_rootfs)
+
+if part.change_directory:
+cd = part.change_directory
+if cd[-1] == '/':
+cd = cd[:-1]
+if os.path.isabs(cd):
+logger.error("Must be relative: --change-directory=%s" % 
cd)
+sys.exit(1)
+orig_dir = os.path.join(part.rootfs_dir, cd)
+else:
+orig_dir = part.rootfs_dir
+copyhardlinktree(orig_dir, new_rootfs)
 
 # Convert the pseudo directory to its new location
 if (pseudo_dir):
@@ -107,7 +118,7 @@ class RootfsPlugin(SourcePlugin):
 pseudo_cmd = "%s -B -m %s -M %s" % 
(cls.__get_pseudo(native_sysroot,
  
new_rootfs,
  
new_pseudo),
-part.rootfs_dir, 
new_rootfs)
+orig_dir, new_rootfs)
 exec_native_cmd(pseudo_cmd, native_sysroot)
 
 for path in part.include_path or []:
-

[OE-core] [PATCH v4 0/4] wic: Fix permissions

[Ricardo Ribalda]

Hi


Today wic behaves differently if we run it from bitbake of directly from the
commandline.

When it is run from bitbake, the permissions/usersnames are handled by the 
pseudo database of the main image.

When it is run from the comandline , it is run outside the main image database.

This results on permissions/usernames not working ok on some usecases on both
bitbake and wic.


This is an attempt to fix all the permission bugs that I am aware from wic.
Using the following usecases


#exclude-path
part / --source rootfs --fstype=ext4 --exclude-path=home

#split_partition
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs 
--rootfs-dir=tmp/work/qt5222-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/ 
--fstype=ext4

#multi_partition
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4


With the current master:
-
#exclude-path
>From Bitbake: OK
wic cmdline: FAIL

#split_partition
>From Bitbake: OK
wic cmdline: FAIL, permissions invalid on both partitions

#multi_partition
>From Bitbake: FAIL second partition
wic cmdline: OK


After:  wic: Fix permissions when using exclude or include path

#exclude-path
>From Bitbake: OK
wic cmdline: OK

#split_partition
>From Bitbake: OK
wic cmdline: FAIL, permissions invalid on second partition

#multi_partition
>From Bitbake: FAIL second partition
wic cmdline: OK


After: wic: Fix multi images .wks with bitbake
--
#exclude-path
>From Bitbake: OK
wic cmdline: OK

#split_partition
>From Bitbake: FAIL, permissions invalid on second partition
wic cmdline: FAIL, permissions invalid on second partition

#multi_partition
>From Bitbake: OK
wic cmdline: OK



After: wic: Add --change-directory argument


we can have a .wks like:
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
part /etc --source rootfs --fstype=ext4 --change-directory=/etc


Wich works fine from bitbake and from cmdline, and has the same functionality as
split_partition.


Ricardo Ribalda Delgado (4):
  wic: Fix permissions when using exclude or include path
  wic: Fix multi images .wks with bitbake
  wic: Add --change-directory argument
  wic: Continue if excluded_path does not exist

 meta/classes/image_types_wic.bbclass |  8 +++-
 scripts/lib/wic/help.py  |  6 +++
 scripts/lib/wic/ksparser.py  |  1 +
 scripts/lib/wic/partition.py | 15 ---
 scripts/lib/wic/plugins/source/rootfs.py | 52 ++--
 5 files changed, 68 insertions(+), 14 deletions(-)

-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137142): 
https://lists.openembedded.org/g/openembedded-core/message/137142
Mute This Topic: https://lists.openembedded.org/mt/72893661/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v4 2/4] wic: Fix multi images .wks with bitbake

[Ricardo Ribalda]

In order to support .wks files with multiple images inside bitbake we
need to explicitly set the pseudo database in use.

Eg: If we try this .mks:
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4

The username for all the files under /export will be set to the runner
of bitbake (usually UID 1000).

Before we run wic, we need to make sure that the pseudo database will be
flushed, and contains all the data needed.

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/classes/image_types_wic.bbclass | 8 ++--
 scripts/lib/wic/partition.py | 9 +++--
 2 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/meta/classes/image_types_wic.bbclass 
b/meta/classes/image_types_wic.bbclass
index b83308b45c..96ed0473ee 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -32,8 +32,7 @@ IMAGE_CMD_wic () {
if [ -z "$wks" ]; then
bbfatal "No kickstart files from WKS_FILES were found: 
${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
fi
-
-   BUILDDIR="${TOPDIR}" wic create "$wks" --vars 
"${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" 
${WIC_CREATE_EXTRA_ARGS}
+   BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars 
"${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" 
${WIC_CREATE_EXTRA_ARGS}
mv "$build_wic/$(basename "${wks%.wks}")"*.direct 
"$out${IMAGE_NAME_SUFFIX}.wic"
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
@@ -86,6 +85,10 @@ python do_write_wks_template () {
 bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + 
os.path.basename(wks_file)))
 }
 
+do_flush_pseudodb() {
+   ${FAKEROOTENV} ${FAKEROOTCMD} -S
+}
+
 python () {
 if d.getVar('USING_WIC'):
 wks_file_u = d.getVar('WKS_FULL_PATH', False)
@@ -139,6 +142,7 @@ python do_rootfs_wicenv () {
 depdir = d.getVar('IMGDEPLOYDIR')
 bb.utils.copyfile(os.path.join(outdir, basename) + '.env', 
os.path.join(depdir, basename) + '.env')
 }
+addtask do_flush_pseudodb after do_image before do_image_wic
 addtask do_rootfs_wicenv after do_image before do_image_wic
 do_rootfs_wicenv[vardeps] += "${WICVARS}"
 do_rootfs_wicenv[prefuncs] = 'set_image_size'
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index b02711be37..d850fbd1b1 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -200,13 +200,10 @@ class Partition():
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
 if (pseudo_dir == None):
 pseudo_dir = "%s/../pseudo" %  rootfs_dir
-p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
-p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
-p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
 pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
-pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
-pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
-pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
 pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
 
 rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137144): 
https://lists.openembedded.org/g/openembedded-core/message/137144
Mute This Topic: https://lists.openembedded.org/mt/72893664/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v4 1/4] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.

This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.

This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.

This bug can be easily triggerd with the following .wks

part / --source rootfs --fstype=ext4 --exclude-path=home

And this sequence:

$ wic create test-permissions -e core-image-minimal -o test/
$ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
$ ls -la /mnt/etc/shadow

To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.

If the rootfs is not a rootfs generated by bitbake a warning is shown
making the user aware that the permissions on the target might not match
what he expects.

WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions 
will be invalid

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/partition.py |  7 +++--
 scripts/lib/wic/plugins/source/rootfs.py | 36 ++--
 2 files changed, 37 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 2d95f78439..b02711be37 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -190,7 +190,7 @@ class Partition():
(self.mountpoint, self.size, self.fixed_size))
 
 def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
-   native_sysroot, real_rootfs = True):
+   native_sysroot, real_rootfs = True, pseudo_dir = None):
 """
 Prepare content for a rootfs partition i.e. create a partition
 and fill it from a /rootfs dir.
@@ -198,8 +198,9 @@ class Partition():
 Currently handles ext2/3/4, btrfs, vfat and squashfs.
 """
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
-p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
- "%s/../pseudo" %  rootfs_dir)
+if (pseudo_dir == None):
+pseudo_dir = "%s/../pseudo" %  rootfs_dir
+p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
 p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
 p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
 pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 705aeb5563..caad9efccc 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -20,7 +20,7 @@ from oe.path import copyhardlinktree
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
-from wic.misc import get_bitbake_var
+from wic.misc import get_bitbake_var, exec_native_cmd
 
 logger = logging.getLogger('wic')
 
@@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
 
 return os.path.realpath(image_rootfs_dir)
 
+@staticmethod
+def __get_pseudo(native_sysroot, rootfs, pseudo_dir):
+pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+return pseudo
+
 @classmethod
 def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
  oe_builddir, bootimg_dir, kernel_dir,
@@ -68,8 +77,14 @@ class RootfsPlugin(SourcePlugin):
"it is not a valid path, exiting" % 
part.rootfs_dir)
 
 part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
+if not os.path.lexists(pseudo_dir):
+logger.warn("%s folder does not exist. "
+"Usernames and permissions will be invalid " % 
pseudo_dir)
+pseudo_dir = None
 
 new_rootfs = None
+new_pseudo = None
 # Handle excluded paths.
 if part.exclude_path or part.include_path:
 # We need a new rootfs directory we can delete files from. Copy to
@@ -78,9 +93,23 @@ class RootfsPlugin(SourcePlugin):
 
 if os.path.lexists(new_rootfs):
 shutil.rmtree(os.path.join(new

Re: [OE-core] [PATCH v4 0/4] wic: Fix permissions

[Ricardo Ribalda]

Hi Paul

On Thu, Apr 9, 2020 at 1:10 PM Paul Barker  wrote:
>
> On Thu,  9 Apr 2020 12:49:47 +0200
> Ricardo Ribalda Delgado  wrote:
>
> > Hi
> >
> >
> > Today wic behaves differently if we run it from bitbake of directly from the
> > commandline.
> >
> > When it is run from bitbake, the permissions/usersnames are handled by the
> > pseudo database of the main image.
> >
> > When it is run from the comandline , it is run outside the main image 
> > database.
> >
> > This results on permissions/usernames not working ok on some usecases on 
> > both
> > bitbake and wic.
> >
> >
> > This is an attempt to fix all the permission bugs that I am aware from wic.
> > Using the following usecases
> >
> >
> > #exclude-path
> > part / --source rootfs --fstype=ext4 --exclude-path=home
> >
> > #split_partition
> > part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
> > part /etc --source rootfs 
> > --rootfs-dir=tmp/work/qt5222-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/
> >  --fstype=ext4
> >
> > #multi_partition
> > part / --source rootfs --ondisk sda --fstype=ext4
> > part /export --source rootfs --rootfs=core-image-minimal-mtdutils 
> > --fstype=ext4
> >
> >
> > With the current master:
> > -
> > #exclude-path
> > From Bitbake: OK
> > wic cmdline: FAIL
> >
> > #split_partition
> > From Bitbake: OK
> > wic cmdline: FAIL, permissions invalid on both partitions
> >
> > #multi_partition
> > From Bitbake: FAIL second partition
> > wic cmdline: OK
> >
> >
> > After:  wic: Fix permissions when using exclude or include path
> > 
> > #exclude-path
> > From Bitbake: OK
> > wic cmdline: OK
> >
> > #split_partition
> > From Bitbake: OK
> > wic cmdline: FAIL, permissions invalid on second partition
> >
> > #multi_partition
> > From Bitbake: FAIL second partition
> > wic cmdline: OK
> >
> >
> > After: wic: Fix multi images .wks with bitbake
> > --
> > #exclude-path
> > From Bitbake: OK
> > wic cmdline: OK
> >
> > #split_partition
> > From Bitbake: FAIL, permissions invalid on second partition
> > wic cmdline: FAIL, permissions invalid on second partition
> >
> > #multi_partition
> > From Bitbake: OK
> > wic cmdline: OK
> >
>
> This looks really good, are you able to automate any of these tests and put
> them in meta/lib/oeqa/selftest/cases/wic.py?

Havent done it before, but for sure I can give it a try.

Any good guide I can use? Last time I was playing with selftest
whenever I changed something on a test I had to run the whole test
from scratch, which is not very productive :).



>
> >
> >
> > After: wic: Add --change-directory argument
> > 
> >
> > we can have a .wks like:
> > part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
> > part /etc --source rootfs --fstype=ext4 --change-directory=/etc
> >
> >
> > Wich works fine from bitbake and from cmdline, and has the same 
> > functionality as
> > split_partition.
> >
> >
> > Ricardo Ribalda Delgado (4):
> >   wic: Fix permissions when using exclude or include path
> >   wic: Fix multi images .wks with bitbake
> >   wic: Add --change-directory argument
> >   wic: Continue if excluded_path does not exist
> >
> >  meta/classes/image_types_wic.bbclass |  8 +++-
> >  scripts/lib/wic/help.py  |  6 +++
> >  scripts/lib/wic/ksparser.py  |  1 +
> >  scripts/lib/wic/partition.py | 15 ---
> >  scripts/lib/wic/plugins/source/rootfs.py | 52 ++--
> >  5 files changed, 68 insertions(+), 14 deletions(-)
> >
>
> I'll give this a detailed review over the weekend. Looks good at first
> glance, though I may suggest renaming `--change-directory` to something else
> so it's more obvious what it's doing when you just read that in a wks file.
> Perhaps `--part-subdir`, I don't know. Let's not bikeshed it too much though.
>

I do not care about the name, I have used tar as "inspiration"

   -C, --directory=DIR
  Change  to DIR before performing any operations.  This option is
  order-sensitive, i.e. it affects all options that follow.

If you have any preference on the name I will take it.

Thanks!

> Thanks,
>
> --
> Paul Barker
> Konsulko Group



-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137152): 
https://lists.openembedded.org/g/openembedded-core/message/137152
Mute This Topic: https://lists.openembedded.org/mt/72893661/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v5 4/6] wic: Continue if excluded_path does not exist

[Ricardo Ribalda]

If an excuded path does not exist, continue without an error.
This allows to seamleasly reuse .wks among different projects.

Eg:

part / --source rootfs --fstype=ext4 --exclude-path=/opt/private_keys

Where /opt/private_keys in only populated by some of the image.bb files.

Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/plugins/source/rootfs.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index d3742802bb..8874ac62c4 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -131,6 +131,9 @@ class RootfsPlugin(SourcePlugin):
 sys.exit(1)
 
 full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+if not os.path.lexists(full_path):
+continue
 
 # Disallow climbing outside of parent directory using '..',
 # because doing so could be quite disastrous (we will delete 
the
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137160): 
https://lists.openembedded.org/g/openembedded-core/message/137160
Mute This Topic: https://lists.openembedded.org/mt/72899597/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v5 3/6] wic: Add --change-directory argument

[Ricardo Ribalda]

This option allows to specify which part of a rootfs is going to be
included, the same way the -C argument on tar.

Thanks to this option we can make sure the permissions and usernames
on the target partition are respected, and also simplify the creation of
splitted partitons, not neeting to invoke external vars or using .wks.in
files. Eg:

part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
part /etc --source rootfs --fstype=ext4 --change-directory=/etc

Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/help.py  |  6 ++
 scripts/lib/wic/ksparser.py  |  1 +
 scripts/lib/wic/partition.py |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 17 ++---
 4 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 1e3d06a87b..62a2a90e79 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -980,6 +980,12 @@ DESCRIPTION
  copies. This option only has an effect with the rootfs
  source plugin.
 
+ --change-directory: This option is specific to wic. It changes to the
+ given directory before copying the files. This
+ option is useful when we want to split a rootfs in
+ multiple partitions and we want to keep the right
+ permissions and usernames in all the partitions.
+
  --extra-space: This option is specific to wic. It adds extra
 space after the space filled by the content
 of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 650b976223..c60869d397 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -138,6 +138,7 @@ class KickStart():
 part.add_argument('--align', type=int)
 part.add_argument('--exclude-path', nargs='+')
 part.add_argument('--include-path', nargs='+')
+part.add_argument('--change-directory')
 part.add_argument("--extra-space", type=sizetype)
 part.add_argument('--fsoptions', dest='fsopts')
 part.add_argument('--fstype', default='vfat',
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index d850fbd1b1..3240be072a 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -31,6 +31,7 @@ class Partition():
 self.extra_space = args.extra_space
 self.exclude_path = args.exclude_path
 self.include_path = args.include_path
+self.change_directory = args.change_directory
 self.fsopts = args.fsopts
 self.fstype = args.fstype
 self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index caad9efccc..d3742802bb 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -86,14 +86,25 @@ class RootfsPlugin(SourcePlugin):
 new_rootfs = None
 new_pseudo = None
 # Handle excluded paths.
-if part.exclude_path or part.include_path:
+if part.exclude_path or part.include_path or part.change_directory:
 # We need a new rootfs directory we can delete files from. Copy to
 # workdir.
 new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" 
% part.lineno))
 
 if os.path.lexists(new_rootfs):
 shutil.rmtree(os.path.join(new_rootfs))
-copyhardlinktree(part.rootfs_dir, new_rootfs)
+
+if part.change_directory:
+cd = part.change_directory
+if cd[-1] == '/':
+cd = cd[:-1]
+if os.path.isabs(cd):
+logger.error("Must be relative: --change-directory=%s" % 
cd)
+sys.exit(1)
+orig_dir = os.path.join(part.rootfs_dir, cd)
+else:
+orig_dir = part.rootfs_dir
+copyhardlinktree(orig_dir, new_rootfs)
 
 # Convert the pseudo directory to its new location
 if (pseudo_dir):
@@ -107,7 +118,7 @@ class RootfsPlugin(SourcePlugin):
 pseudo_cmd = "%s -B -m %s -M %s" % 
(cls.__get_pseudo(native_sysroot,
  
new_rootfs,
  
new_pseudo),
-part.rootfs_dir, 
new_rootfs)
+orig_dir, new_rootfs)
 exec_native_cmd(pseudo_cmd, native_sysroot)
 
 for path in part.include_path or []:
-

[OE-core] [PATCH v5 2/6] wic: Fix multi images .wks with bitbake

[Ricardo Ribalda]

In order to support .wks files with multiple images inside bitbake we
need to explicitly set the pseudo database in use.

Eg: If we try this .mks:
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4

The username for all the files under /export will be set to the runner
of bitbake (usually UID 1000).

Before we run wic, we need to make sure that the pseudo database will be
flushed, and contains all the data needed.

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/classes/image_types_wic.bbclass | 8 ++--
 scripts/lib/wic/partition.py | 9 +++--
 2 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/meta/classes/image_types_wic.bbclass 
b/meta/classes/image_types_wic.bbclass
index b83308b45c..96ed0473ee 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -32,8 +32,7 @@ IMAGE_CMD_wic () {
if [ -z "$wks" ]; then
bbfatal "No kickstart files from WKS_FILES were found: 
${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
fi
-
-   BUILDDIR="${TOPDIR}" wic create "$wks" --vars 
"${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" 
${WIC_CREATE_EXTRA_ARGS}
+   BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars 
"${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" 
${WIC_CREATE_EXTRA_ARGS}
mv "$build_wic/$(basename "${wks%.wks}")"*.direct 
"$out${IMAGE_NAME_SUFFIX}.wic"
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
@@ -86,6 +85,10 @@ python do_write_wks_template () {
 bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + 
os.path.basename(wks_file)))
 }
 
+do_flush_pseudodb() {
+   ${FAKEROOTENV} ${FAKEROOTCMD} -S
+}
+
 python () {
 if d.getVar('USING_WIC'):
 wks_file_u = d.getVar('WKS_FULL_PATH', False)
@@ -139,6 +142,7 @@ python do_rootfs_wicenv () {
 depdir = d.getVar('IMGDEPLOYDIR')
 bb.utils.copyfile(os.path.join(outdir, basename) + '.env', 
os.path.join(depdir, basename) + '.env')
 }
+addtask do_flush_pseudodb after do_image before do_image_wic
 addtask do_rootfs_wicenv after do_image before do_image_wic
 do_rootfs_wicenv[vardeps] += "${WICVARS}"
 do_rootfs_wicenv[prefuncs] = 'set_image_size'
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index b02711be37..d850fbd1b1 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -200,13 +200,10 @@ class Partition():
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
 if (pseudo_dir == None):
 pseudo_dir = "%s/../pseudo" %  rootfs_dir
-p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
-p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
-p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
 pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
-pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
-pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
-pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
 pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
 
 rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137158): 
https://lists.openembedded.org/g/openembedded-core/message/137158
Mute This Topic: https://lists.openembedded.org/mt/72899593/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v5 5/6] wic: Avoid creating invalid pseudo directory

[Ricardo Ribalda]

If the source of the rootfs is not a bitbake cooked image, or it is not
pointing to the root of one, we call pseudo again, which will produce
a new pseudo folder at rootfs/../pseudo

Eg:

part /etc --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4

Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/partition.py | 15 ---
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 3240be072a..7d9dd616a6 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -199,13 +199,14 @@ class Partition():
 Currently handles ext2/3/4, btrfs, vfat and squashfs.
 """
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
-if (pseudo_dir == None):
-pseudo_dir = "%s/../pseudo" %  rootfs_dir
-pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
-pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
-pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
-pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
-pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+if (pseudo_dir):
+pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+else:
+pseudo = None
 
 rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
  self.lineno, self.fstype)
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137161): 
https://lists.openembedded.org/g/openembedded-core/message/137161
Mute This Topic: https://lists.openembedded.org/mt/72899598/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v5 6/6] oeqa: wic: Add tests for permissions and change-directory

[Ricardo Ribalda]

Make sure that the permissions and username are respected when using all
the rootfs modifiers.

Add tests for checnage-directory command

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/lib/oeqa/selftest/cases/wic.py | 91 +
 1 file changed, 91 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
b/meta/lib/oeqa/selftest/cases/wic.py
index 626a217e69..8ef08c3d77 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -62,6 +62,12 @@ def extract_files(debugfs_output):
 return [line.split('/')[5].strip() for line in \
 debugfs_output.strip().split('/\n')]
 
+def files_own_by_root(debugfs_output):
+for line in debugfs_output.strip().split('/\n'):
+if line.split('/')[3:5] != ['0', '0']:
+print(debugfs_output)
+return False
+return True
 
 class WicTestCase(OESelftestTestCase):
 """Wic test class."""
@@ -84,6 +90,7 @@ class WicTestCase(OESelftestTestCase):
 self.skipTest('wic-tools cannot be built due its 
(intltool|gettext)-native dependency and NLS disable')
 
 bitbake('core-image-minimal')
+bitbake('core-image-minimal-mtdutils')
 WicTestCase.image_is_ready = True
 
 rmtree(self.resultdir, ignore_errors=True)
@@ -506,6 +513,90 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 
--include-path %s"""
   % (wks_file, self.resultdir), 
ignore_status=True).status)
 os.remove(wks_file)
 
+def test_permissions(self):
+"""Test permissions are respected"""
+
+oldpath = os.environ['PATH']
+os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+t_normal = """
+part / --source rootfs --fstype=ext4
+"""
+t_exclude = """
+part / --source rootfs --fstype=ext4 --exclude-path=home
+"""
+t_multi = """
+part / --source rootfs --ondisk sda --fstype=ext4
+part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
+"""
+t_change = """
+part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
+part /etc --source rootfs --fstype=ext4 --change-directory=etc
+"""
+tests = [t_normal, t_exclude, t_multi, t_change]
+
+try:
+for test in tests:
+include_path = os.path.join(self.resultdir, 'test-include')
+os.makedirs(include_path)
+wks_file = os.path.join(include_path, 'temp.wks')
+with open(wks_file, 'w') as wks:
+wks.write(test)
+runCmd("wic create %s -e core-image-minimal -o %s" \
+   % (wks_file, self.resultdir))
+
+for part in glob(os.path.join(self.resultdir, 
'temp-*.direct.p*')):
+res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part))
+self.assertEqual(True, files_own_by_root(res.output))
+
+rmtree(self.resultdir, ignore_errors=True)
+
+finally:
+os.environ['PATH'] = oldpath
+
+def test_change_directory(self):
+"""Test --change-directory wks option."""
+
+oldpath = os.environ['PATH']
+os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+try:
+include_path = os.path.join(self.resultdir, 'test-include')
+os.makedirs(include_path)
+wks_file = os.path.join(include_path, 'temp.wks')
+with open(wks_file, 'w') as wks:
+rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+wks.write("part /etc --source rootfs --fstype=ext4 
--change-directory=etc")
+runCmd("wic create %s -e core-image-minimal -o %s" \
+   % (wks_file, self.resultdir))
+
+part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
+
+res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
+files = extract_files(res.output)
+self.assertIn('shadow', files)
+
+finally:
+os.environ['PATH'] = oldpath
+
+def test_change_directory_errors(self):
+"""Test --change-directory wks option error handling."""
+wks_file = 'temp.wks'
+
+# Absolute argument.
+with open(wks

[OE-core] [PATCH v5 1/6] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.

This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.

This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.

This bug can be easily triggerd with the following .wks

part / --source rootfs --fstype=ext4 --exclude-path=home

And this sequence:

$ wic create test-permissions -e core-image-minimal -o test/
$ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
$ ls -la /mnt/etc/shadow

To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.

If the rootfs is not a rootfs generated by bitbake a warning is shown
making the user aware that the permissions on the target might not match
what he expects.

WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions 
will be invalid

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/partition.py |  7 +++--
 scripts/lib/wic/plugins/source/rootfs.py | 36 ++--
 2 files changed, 37 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 2d95f78439..b02711be37 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -190,7 +190,7 @@ class Partition():
(self.mountpoint, self.size, self.fixed_size))
 
 def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
-   native_sysroot, real_rootfs = True):
+   native_sysroot, real_rootfs = True, pseudo_dir = None):
 """
 Prepare content for a rootfs partition i.e. create a partition
 and fill it from a /rootfs dir.
@@ -198,8 +198,9 @@ class Partition():
 Currently handles ext2/3/4, btrfs, vfat and squashfs.
 """
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
-p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
- "%s/../pseudo" %  rootfs_dir)
+if (pseudo_dir == None):
+pseudo_dir = "%s/../pseudo" %  rootfs_dir
+p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
 p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
 p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
 pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 705aeb5563..caad9efccc 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -20,7 +20,7 @@ from oe.path import copyhardlinktree
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
-from wic.misc import get_bitbake_var
+from wic.misc import get_bitbake_var, exec_native_cmd
 
 logger = logging.getLogger('wic')
 
@@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
 
 return os.path.realpath(image_rootfs_dir)
 
+@staticmethod
+def __get_pseudo(native_sysroot, rootfs, pseudo_dir):
+pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+return pseudo
+
 @classmethod
 def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
  oe_builddir, bootimg_dir, kernel_dir,
@@ -68,8 +77,14 @@ class RootfsPlugin(SourcePlugin):
"it is not a valid path, exiting" % 
part.rootfs_dir)
 
 part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
+if not os.path.lexists(pseudo_dir):
+logger.warn("%s folder does not exist. "
+"Usernames and permissions will be invalid " % 
pseudo_dir)
+pseudo_dir = None
 
 new_rootfs = None
+new_pseudo = None
 # Handle excluded paths.
 if part.exclude_path or part.include_path:
 # We need a new rootfs directory we can delete files from. Copy to
@@ -78,9 +93,23 @@ class RootfsPlugin(SourcePlugin):
 
 if os.path.lexists(new_rootfs):
 shutil.rmtree(os.path.join(new

[OE-core] [PATCH v5 0/6] wic: Fix permissions

[Ricardo Ribalda]

Hi

Today wic behaves differently if we run it from bitbake of directly from the
commandline.

When it is run from bitbake, the permissions/usersnames are handled by the 
pseudo database of the main image.

When it is run from the comandline , it is run outside the main image database.

This results on permissions/usernames not working ok on some usecases on both
bitbake and wic.


This is an attempt to fix all the permission bugs that I am aware from wic.
Using the following usecases


#exclude-path
part / --source rootfs --fstype=ext4 --exclude-path=home

#split_partition
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs 
--rootfs-dir=tmp/work/qt5222-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/ 
--fstype=ext4

#multi_partition
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4


With the current master:
-
#exclude-path
>From Bitbake: OK
wic cmdline: FAIL

#split_partition
>From Bitbake: OK
wic cmdline: FAIL, permissions invalid on both partitions

#multi_partition
>From Bitbake: FAIL second partition
wic cmdline: OK


After:  wic: Fix permissions when using exclude or include path

#exclude-path
>From Bitbake: OK
wic cmdline: OK

#split_partition
>From Bitbake: OK
wic cmdline: FAIL, permissions invalid on second partition

#multi_partition
>From Bitbake: FAIL second partition
wic cmdline: OK


After: wic: Fix multi images .wks with bitbake
--
#exclude-path
>From Bitbake: OK
wic cmdline: OK

#split_partition
>From Bitbake: FAIL, permissions invalid on second partition
wic cmdline: FAIL, permissions invalid on second partition

#multi_partition
>From Bitbake: OK
wic cmdline: OK



After: wic: Add --change-directory argument


we can have a .wks like:
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
part /etc --source rootfs --fstype=ext4 --change-directory=/etc


Wich works fine from bitbake and from cmdline, and has the same functionality as
split_partition.


Changelog v4->v5:
  - Add unittesting
  - wic: Avoid creating invalid pseudo directory

Ricardo Ribalda Delgado (6):
  wic: Fix permissions when using exclude or include path
  wic: Fix multi images .wks with bitbake
  wic: Add --change-directory argument
  wic: Continue if excluded_path does not exist
  wic: Avoid creating invalid pseudo directory
  oeqa: wic: Add tests for permissions and change-directory

 meta/classes/image_types_wic.bbclass |  8 ++-
 meta/lib/oeqa/selftest/cases/wic.py  | 91 
 scripts/lib/wic/help.py  |  6 ++
 scripts/lib/wic/ksparser.py  |  1 +
 scripts/lib/wic/partition.py | 20 +++---
 scripts/lib/wic/plugins/source/rootfs.py | 52 --
 6 files changed, 162 insertions(+), 16 deletions(-)

-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137156): 
https://lists.openembedded.org/g/openembedded-core/message/137156
Mute This Topic: https://lists.openembedded.org/mt/72899590/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v6 07/10] wic: misc: Do not find for executables in ALREADY_PROVIDED

[Ricardo Ribalda]

Executables like tar won't be available on the native sysroot, as they
are part of the ALREADY_PROVIDED variable.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/misc.py | 14 --
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 1f199b9f23..91975ba151 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -45,7 +45,8 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
   "parted": "parted",
   "sfdisk": "util-linux",
   "sgdisk": "gptfdisk",
-  "syslinux": "syslinux"
+  "syslinux": "syslinux",
+  "tar": "tar"
  }
 
 def runtool(cmdln_or_args):
@@ -112,6 +113,15 @@ def exec_cmd(cmd_and_args, as_shell=False):
 """
 return _exec_cmd(cmd_and_args, as_shell)[1]
 
+def find_executable(cmd, paths):
+recipe = cmd
+if recipe in NATIVE_RECIPES:
+recipe =  NATIVE_RECIPES[recipe]
+provided = get_bitbake_var("ASSUME_PROVIDED")
+if provided and "%s-native" % recipe in provided:
+return True
+
+return spawn.find_executable(cmd, paths)
 
 def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
 """
@@ -136,7 +146,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, 
pseudo=""):
 logger.debug("exec_native_cmd: %s", native_cmd_and_args)
 
 # If the command isn't in the native sysroot say we failed.
-if spawn.find_executable(args[0], native_paths):
+if find_executable(args[0], native_paths):
 ret, out = _exec_cmd(native_cmd_and_args, True)
 else:
 ret = 127
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137214): 
https://lists.openembedded.org/g/openembedded-core/message/137214
Mute This Topic: https://lists.openembedded.org/mt/73010218/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v6 08/10] wic: root: Add an opt. destination on include-path

[Ricardo Ribalda]

Allow specifying an optional destination to include-path and make the
option aware of permissions and owners.

It is very useful for making a partition that contains the rootfs for a
host and a target Eg:

/ -> Roofs for the host
/export/ -> Rootfs for the target (which will netboot)

Although today we support making a partition for "/export" this might
not be compatible with some upgrade systems, or we might be limited by
the number of partitions.

With this patch we can use something like:

part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils 
export/ --include-path hello

on the .wks file.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-core/meta/wic-tools.bb  |  2 +-
 scripts/lib/wic/help.py  | 20 +
 scripts/lib/wic/ksparser.py  |  2 +-
 scripts/lib/wic/plugins/source/rootfs.py | 54 +++-
 4 files changed, 66 insertions(+), 12 deletions(-)

diff --git a/meta/recipes-core/meta/wic-tools.bb 
b/meta/recipes-core/meta/wic-tools.bb
index 09eb409e87..8aeb942ed2 100644
--- a/meta/recipes-core/meta/wic-tools.bb
+++ b/meta/recipes-core/meta/wic-tools.bb
@@ -6,7 +6,7 @@ DEPENDS = "\
parted-native syslinux-native gptfdisk-native dosfstools-native \
mtools-native bmap-tools-native grub-efi-native cdrtools-native \
btrfs-tools-native squashfs-tools-native pseudo-native \
-   e2fsprogs-native util-linux-native \
+   e2fsprogs-native util-linux-native tar-native\
"
 DEPENDS_append_x86 = " syslinux grub-efi systemd-boot"
 DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot"
diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 62a2a90e79..bd3a2b97df 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -971,14 +971,18 @@ DESCRIPTION
  has an effect with the rootfs source plugin.
 
  --include-path: This option is specific to wic. It adds the contents
- of the given path to the resulting image. The path is
- relative to the directory in which wic is running not
- the rootfs itself so use of an absolute path is
- recommended. This option is most useful when multiple
- copies of the rootfs are added to an image and it is
- required to add extra content to only one of these
- copies. This option only has an effect with the rootfs
- source plugin.
+ of the given path or a rootfs to the resulting image.
+ The option contains two fields, the origin and the
+ destination. When the origin is a rootfs, it follows
+ the same logic as the rootfs-dir argument and the
+ permissions and owners are kept. When the origin is a
+ path, it is relative to the directory in which wic is
+ running not the rootfs itself so use of an absolute
+ path is recommended, and the owner and group is set to
+ root:root. If no destination is given it is
+ automatically set to the root of the rootfs. This
+ option only has an effect with the rootfs source
+ plugin.
 
  --change-directory: This option is specific to wic. It changes to the
  given directory before copying the files. This
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index c60869d397..b8befe78e3 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -137,7 +137,7 @@ class KickStart():
 part.add_argument('--active', action='store_true')
 part.add_argument('--align', type=int)
 part.add_argument('--exclude-path', nargs='+')
-part.add_argument('--include-path', nargs='+')
+part.add_argument('--include-path', nargs='+', action='append')
 part.add_argument('--change-directory')
 part.add_argument("--extra-space", type=sizetype)
 part.add_argument('--fsoptions', dest='fsopts')
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 1d21ec2252..9d0666b638 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -17,6 +17,7 @@ import shutil
 import sys
 
 from oe.path import copyhardlinktree
+from pathlib import Path
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
@@ -125,8 +126,57 @@ class RootfsPlugin(SourcePlugin

[OE-core] [PATCH v6 05/10] wic: Avoid creating invalid pseudo directory

[Ricardo Ribalda]

If the source of the rootfs is not a bitbake cooked image, or it is not
pointing to the root of one, we call pseudo again, which will produce
a new pseudo folder at rootfs/../pseudo

Eg:

part /etc --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/partition.py | 15 ---
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 3240be072a..7d9dd616a6 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -199,13 +199,14 @@ class Partition():
 Currently handles ext2/3/4, btrfs, vfat and squashfs.
 """
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
-if (pseudo_dir == None):
-pseudo_dir = "%s/../pseudo" %  rootfs_dir
-pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
-pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
-pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
-pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
-pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+if (pseudo_dir):
+pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+else:
+pseudo = None
 
 rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
  self.lineno, self.fstype)
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137212): 
https://lists.openembedded.org/g/openembedded-core/message/137212
Mute This Topic: https://lists.openembedded.org/mt/73010213/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v6 03/10] wic: Add --change-directory argument

[Ricardo Ribalda]

This option allows to specify which part of a rootfs is going to be
included, the same way the -C argument on tar.

Thanks to this option we can make sure the permissions and usernames
on the target partition are respected, and also simplify the creation of
splitted partitons, not neeting to invoke external vars or using .wks.in
files. Eg:

part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
part /etc --source rootfs --fstype=ext4 --change-directory=etc

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/help.py  |  6 ++
 scripts/lib/wic/ksparser.py  |  1 +
 scripts/lib/wic/partition.py |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 21 ++---
 4 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 1e3d06a87b..62a2a90e79 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -980,6 +980,12 @@ DESCRIPTION
  copies. This option only has an effect with the rootfs
  source plugin.
 
+ --change-directory: This option is specific to wic. It changes to the
+ given directory before copying the files. This
+ option is useful when we want to split a rootfs in
+ multiple partitions and we want to keep the right
+ permissions and usernames in all the partitions.
+
  --extra-space: This option is specific to wic. It adds extra
 space after the space filled by the content
 of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 650b976223..c60869d397 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -138,6 +138,7 @@ class KickStart():
 part.add_argument('--align', type=int)
 part.add_argument('--exclude-path', nargs='+')
 part.add_argument('--include-path', nargs='+')
+part.add_argument('--change-directory')
 part.add_argument("--extra-space", type=sizetype)
 part.add_argument('--fsoptions', dest='fsopts')
 part.add_argument('--fstype', default='vfat',
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index d850fbd1b1..3240be072a 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -31,6 +31,7 @@ class Partition():
 self.extra_space = args.extra_space
 self.exclude_path = args.exclude_path
 self.include_path = args.include_path
+self.change_directory = args.change_directory
 self.fsopts = args.fsopts
 self.fstype = args.fstype
 self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index caad9efccc..c831829398 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -86,14 +86,29 @@ class RootfsPlugin(SourcePlugin):
 new_rootfs = None
 new_pseudo = None
 # Handle excluded paths.
-if part.exclude_path or part.include_path:
+if part.exclude_path or part.include_path or part.change_directory:
 # We need a new rootfs directory we can delete files from. Copy to
 # workdir.
 new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" 
% part.lineno))
 
 if os.path.lexists(new_rootfs):
 shutil.rmtree(os.path.join(new_rootfs))
-copyhardlinktree(part.rootfs_dir, new_rootfs)
+
+if part.change_directory:
+cd = part.change_directory
+if cd[-1] == '/':
+cd = cd[:-1]
+if os.path.isabs(cd):
+logger.error("Must be relative: --change-directory=%s" % 
cd)
+sys.exit(1)
+orig_dir = os.path.realpath(os.path.join(part.rootfs_dir, cd))
+if not orig_dir.startswith(part.rootfs_dir):
+logger.error("'%s' points to a path outside the rootfs" % 
orig_dir)
+sys.exit(1)
+
+else:
+orig_dir = part.rootfs_dir
+copyhardlinktree(orig_dir, new_rootfs)
 
 # Convert the pseudo directory to its new location
 if (pseudo_dir):
@@ -107,7 +122,7 @@ class RootfsPlugin(SourcePlugin):
 pseudo_cmd = "%s -B -m %s -M %s" % 
(cls.__get_pseudo(native_sysroot,
  
new_rootfs,
  
new_pseudo),
-   

[OE-core] [PATCH v6 09/10] wic: rootfs: Combine path_validation in one function

[Ricardo Ribalda]

Combine all the common path validation in a function to avoid code
duplication.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/plugins/source/rootfs.py | 41 +++-
 1 file changed, 19 insertions(+), 22 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 9d0666b638..53942cc4b7 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -32,6 +32,22 @@ class RootfsPlugin(SourcePlugin):
 
 name = 'rootfs'
 
+@staticmethod
+def __validate_path(cmd, rootfs_dir, path):
+if os.path.isabs(path):
+logger.error("%s: Must be relative: %s" % (cmd, orig_path))
+sys.exit(1)
+
+# Disallow climbing outside of parent directory using '..',
+# because doing so could be quite disastrous (we will delete the
+# directory, or modify a directory outside OpenEmbedded).
+full_path = os.path.realpath(os.path.join(rootfs_dir, path))
+if not full_path.startswith(os.path.realpath(rootfs_dir)):
+logger.error("%s: Must point inside the rootfs:" % (cmd, path))
+sys.exit(1)
+
+return full_path
+
 @staticmethod
 def __get_rootfs_dir(rootfs_dir):
 if os.path.isdir(rootfs_dir):
@@ -99,14 +115,7 @@ class RootfsPlugin(SourcePlugin):
 cd = part.change_directory
 if cd[-1] == '/':
 cd = cd[:-1]
-if os.path.isabs(cd):
-logger.error("Must be relative: --change-directory=%s" % 
cd)
-sys.exit(1)
-orig_dir = os.path.realpath(os.path.join(part.rootfs_dir, cd))
-if not orig_dir.startswith(part.rootfs_dir):
-logger.error("'%s' points to a path outside the rootfs" % 
orig_dir)
-sys.exit(1)
-
+orig_dir = cls.__validate_path("--change-directory", 
part.rootfs_dir, cd)
 else:
 orig_dir = part.rootfs_dir
 copyhardlinktree(orig_dir, new_rootfs)
@@ -161,10 +170,7 @@ class RootfsPlugin(SourcePlugin):
 
 #create destination
 if path:
-destination = os.path.realpath(os.path.join(new_rootfs, 
path))
-if not destination.startswith(new_rootfs):
-logger.error("%s %s" % (destination, new_rootfs))
-sys.exit(1)
+destination = cls.__validate_path("--include-path", 
new_rootfs, path)
 Path(destination).mkdir(parents=True, exist_ok=True)
 else:
 destination = new_rootfs
@@ -180,17 +186,8 @@ class RootfsPlugin(SourcePlugin):
 
 for orig_path in part.exclude_path or []:
 path = orig_path
-if os.path.isabs(path):
-logger.error("Must be relative: --exclude-path=%s" % 
orig_path)
-sys.exit(1)
 
-full_path = os.path.realpath(os.path.join(new_rootfs, path))
-# Disallow climbing outside of parent directory using '..',
-# because doing so could be quite disastrous (we will delete 
the
-# directory).
-if not full_path.startswith(new_rootfs):
-logger.error("'%s' points to a path outside the rootfs" % 
orig_path)
-sys.exit(1)
+full_path = cls.__validate_path("--exclude-path", new_rootfs, 
path)
 
 if not os.path.lexists(full_path):
 continue
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137216): 
https://lists.openembedded.org/g/openembedded-core/message/137216
Mute This Topic: https://lists.openembedded.org/mt/73010220/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v6 02/10] wic: Fix multi images .wks with bitbake

[Ricardo Ribalda]

In order to support .wks files with multiple images inside bitbake we
need to explicitly set the pseudo database in use.

Eg: If we try this .mks:
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4

The username for all the files under /export will be set to the runner
of bitbake (usually UID 1000).

Before we run wic, we need to make sure that the pseudo database will be
flushed, and contains all the data needed.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/classes/image_types_wic.bbclass | 8 ++--
 scripts/lib/wic/partition.py | 9 +++--
 2 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/meta/classes/image_types_wic.bbclass 
b/meta/classes/image_types_wic.bbclass
index b83308b45c..96ed0473ee 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -32,8 +32,7 @@ IMAGE_CMD_wic () {
if [ -z "$wks" ]; then
bbfatal "No kickstart files from WKS_FILES were found: 
${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
fi
-
-   BUILDDIR="${TOPDIR}" wic create "$wks" --vars 
"${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" 
${WIC_CREATE_EXTRA_ARGS}
+   BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars 
"${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" 
${WIC_CREATE_EXTRA_ARGS}
mv "$build_wic/$(basename "${wks%.wks}")"*.direct 
"$out${IMAGE_NAME_SUFFIX}.wic"
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
@@ -86,6 +85,10 @@ python do_write_wks_template () {
 bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + 
os.path.basename(wks_file)))
 }
 
+do_flush_pseudodb() {
+   ${FAKEROOTENV} ${FAKEROOTCMD} -S
+}
+
 python () {
 if d.getVar('USING_WIC'):
 wks_file_u = d.getVar('WKS_FULL_PATH', False)
@@ -139,6 +142,7 @@ python do_rootfs_wicenv () {
 depdir = d.getVar('IMGDEPLOYDIR')
 bb.utils.copyfile(os.path.join(outdir, basename) + '.env', 
os.path.join(depdir, basename) + '.env')
 }
+addtask do_flush_pseudodb after do_image before do_image_wic
 addtask do_rootfs_wicenv after do_image before do_image_wic
 do_rootfs_wicenv[vardeps] += "${WICVARS}"
 do_rootfs_wicenv[prefuncs] = 'set_image_size'
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index b02711be37..d850fbd1b1 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -200,13 +200,10 @@ class Partition():
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
 if (pseudo_dir == None):
 pseudo_dir = "%s/../pseudo" %  rootfs_dir
-p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
-p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
-p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
 pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
-pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
-pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
-pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
 pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
 
 rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137209): 
https://lists.openembedded.org/g/openembedded-core/message/137209
Mute This Topic: https://lists.openembedded.org/mt/73010207/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v6 04/10] wic: Continue if excluded_path does not exist

[Ricardo Ribalda]

If an excuded path does not exist, continue without an error.
This allows to seamleasly reuse .wks among different projects.

Eg:

part / --source rootfs --fstype=ext4 --exclude-path=opt/private_keys

Where /opt/private_keys in only populated by some of the image.bb files.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/plugins/source/rootfs.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index c831829398..1d21ec2252 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -135,7 +135,6 @@ class RootfsPlugin(SourcePlugin):
 sys.exit(1)
 
 full_path = os.path.realpath(os.path.join(new_rootfs, path))
-
 # Disallow climbing outside of parent directory using '..',
 # because doing so could be quite disastrous (we will delete 
the
 # directory).
@@ -143,6 +142,9 @@ class RootfsPlugin(SourcePlugin):
 logger.error("'%s' points to a path outside the rootfs" % 
orig_path)
 sys.exit(1)
 
+if not os.path.lexists(full_path):
+continue
+
 if path.endswith(os.sep):
 # Delete content only.
 for entry in os.listdir(full_path):
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137211): 
https://lists.openembedded.org/g/openembedded-core/message/137211
Mute This Topic: https://lists.openembedded.org/mt/73010210/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v6 01/10] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.

This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.

This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.

This bug can be easily triggerd with the following .wks

part / --source rootfs --fstype=ext4 --exclude-path=home

And this sequence:

$ wic create test-permissions -e core-image-minimal -o test/
$ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
$ ls -la /mnt/etc/shadow

To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.

If the rootfs is not a rootfs generated by bitbake a warning is shown
making the user aware that the permissions on the target might not match
what he expects.

WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions 
will be invalid

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/partition.py |  7 +++--
 scripts/lib/wic/plugins/source/rootfs.py | 36 ++--
 2 files changed, 37 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 2d95f78439..b02711be37 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -190,7 +190,7 @@ class Partition():
(self.mountpoint, self.size, self.fixed_size))
 
 def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
-   native_sysroot, real_rootfs = True):
+   native_sysroot, real_rootfs = True, pseudo_dir = None):
 """
 Prepare content for a rootfs partition i.e. create a partition
 and fill it from a /rootfs dir.
@@ -198,8 +198,9 @@ class Partition():
 Currently handles ext2/3/4, btrfs, vfat and squashfs.
 """
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
-p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
- "%s/../pseudo" %  rootfs_dir)
+if (pseudo_dir == None):
+pseudo_dir = "%s/../pseudo" %  rootfs_dir
+p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
 p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
 p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
 pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 705aeb5563..caad9efccc 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -20,7 +20,7 @@ from oe.path import copyhardlinktree
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
-from wic.misc import get_bitbake_var
+from wic.misc import get_bitbake_var, exec_native_cmd
 
 logger = logging.getLogger('wic')
 
@@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
 
 return os.path.realpath(image_rootfs_dir)
 
+@staticmethod
+def __get_pseudo(native_sysroot, rootfs, pseudo_dir):
+pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+return pseudo
+
 @classmethod
 def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
  oe_builddir, bootimg_dir, kernel_dir,
@@ -68,8 +77,14 @@ class RootfsPlugin(SourcePlugin):
"it is not a valid path, exiting" % 
part.rootfs_dir)
 
 part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
+if not os.path.lexists(pseudo_dir):
+logger.warn("%s folder does not exist. "
+"Usernames and permissions will be invalid " % 
pseudo_dir)
+pseudo_dir = None
 
 new_rootfs = None
+new_pseudo = None
 # Handle excluded paths.
 if part.exclude_path or part.include_path:
 # We need a new rootfs directory we can delete files from. Copy to
@@ -78,9 +93,23 @@ class RootfsPlugin(SourcePlugin):
 
 if os.path.lexists(new_rootfs):
 shutil.rmtree(os.path.join(new

[OE-core] [PATCH v6 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

Today wic behaves differently if we run it from bitbake of directly from the
commandline.

When it is run from bitbake, the permissions/usersnames are handled by the
pseudo database of the main image.

When it is run from the comandline ,it is run outside the main image database.

This results on permissions/usernames not working ok on some usecases on both
bitbake and wic.


This is an attempt to fix all the permission bugs that I am aware from wic.
Using the following usecases:


#exclude-path
part / --source rootfs --fstype=ext4 --exclude-path=home

#split_partition
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs 
--rootfs-dir=tmp/work/qt5222-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/ 
--fstype=ext4

#multi_partition
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4


With the current master:
-
#exclude-path
>From Bitbake: OK
wic cmdline: FAIL

#split_partition
>From Bitbake: OK
wic cmdline: FAIL, permissions invalid on both partitions

#multi_partition
>From Bitbake: FAIL second partition
wic cmdline: OK


After:  wic: Fix permissions when using exclude or include path

#exclude-path
>From Bitbake: OK
wic cmdline: OK

#split_partition
>From Bitbake: OK
wic cmdline: FAIL, permissions invalid on second partition

#multi_partition
>From Bitbake: FAIL second partition
wic cmdline: OK


After: wic: Fix multi images .wks with bitbake
--
#exclude-path
>From Bitbake: OK
wic cmdline: OK

#split_partition
>From Bitbake: FAIL, permissions invalid on second partition
wic cmdline: FAIL, permissions invalid on second partition

#multi_partition
>From Bitbake: OK
wic cmdline: OK



After: wic: Add --change-directory argument


we can have a .wks like:
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs --fstype=ext4 --change-directory=etc


Wich works fine from bitbake and from cmdline, and has the same functionality as
split_partition.


Changelog v5->v6:
  - Allow two arguments to include-path
  - Fix permission on include-path
  - exec_native with ASSUME_PROVIDED
  - More unittesting

Changelog v4->v5:
  - Add unittesting
  - wic: Avoid creating invalid pseudo directory

Ricardo Ribalda Delgado (10):
  wic: Fix permissions when using exclude or include path
  wic: Fix multi images .wks with bitbake
  wic: Add --change-directory argument
  wic: Continue if excluded_path does not exist
  wic: Avoid creating invalid pseudo directory
  oeqa: wic: Add tests for permissions and change-directory
  wic: misc: Do not find for executables in ALREADY_PROVIDED
  wic: root: Add an opt. destination on include-path
  wic: rootfs: Combine path_validation in one function
  oeqa: wic: Add more tests for include_path

 meta/classes/image_types_wic.bbclass |   8 +-
 meta/lib/oeqa/selftest/cases/wic.py  | 155 ++-
 meta/recipes-core/meta/wic-tools.bb  |   2 +-
 scripts/lib/wic/help.py  |  26 ++--
 scripts/lib/wic/ksparser.py  |   3 +-
 scripts/lib/wic/misc.py  |  14 +-
 scripts/lib/wic/partition.py |  20 +--
 scripts/lib/wic/plugins/source/rootfs.py | 126 +++---
 8 files changed, 313 insertions(+), 41 deletions(-)

-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137207): 
https://lists.openembedded.org/g/openembedded-core/message/137207
Mute This Topic: https://lists.openembedded.org/mt/73010205/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v6 06/10] oeqa: wic: Add tests for permissions and change-directory

[Ricardo Ribalda]

Make sure that the permissions and username are respected when using all
the rootfs modifiers.

Add tests for change-directory command

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/lib/oeqa/selftest/cases/wic.py | 91 +
 1 file changed, 91 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
b/meta/lib/oeqa/selftest/cases/wic.py
index 626a217e69..8ef08c3d77 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -62,6 +62,12 @@ def extract_files(debugfs_output):
 return [line.split('/')[5].strip() for line in \
 debugfs_output.strip().split('/\n')]
 
+def files_own_by_root(debugfs_output):
+for line in debugfs_output.strip().split('/\n'):
+if line.split('/')[3:5] != ['0', '0']:
+print(debugfs_output)
+return False
+return True
 
 class WicTestCase(OESelftestTestCase):
 """Wic test class."""
@@ -84,6 +90,7 @@ class WicTestCase(OESelftestTestCase):
 self.skipTest('wic-tools cannot be built due its 
(intltool|gettext)-native dependency and NLS disable')
 
 bitbake('core-image-minimal')
+bitbake('core-image-minimal-mtdutils')
 WicTestCase.image_is_ready = True
 
 rmtree(self.resultdir, ignore_errors=True)
@@ -506,6 +513,90 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 
--include-path %s"""
   % (wks_file, self.resultdir), 
ignore_status=True).status)
 os.remove(wks_file)
 
+def test_permissions(self):
+"""Test permissions are respected"""
+
+oldpath = os.environ['PATH']
+os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+t_normal = """
+part / --source rootfs --fstype=ext4
+"""
+t_exclude = """
+part / --source rootfs --fstype=ext4 --exclude-path=home
+"""
+t_multi = """
+part / --source rootfs --ondisk sda --fstype=ext4
+part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
+"""
+t_change = """
+part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
+part /etc --source rootfs --fstype=ext4 --change-directory=etc
+"""
+tests = [t_normal, t_exclude, t_multi, t_change]
+
+try:
+for test in tests:
+include_path = os.path.join(self.resultdir, 'test-include')
+os.makedirs(include_path)
+wks_file = os.path.join(include_path, 'temp.wks')
+with open(wks_file, 'w') as wks:
+wks.write(test)
+runCmd("wic create %s -e core-image-minimal -o %s" \
+   % (wks_file, self.resultdir))
+
+for part in glob(os.path.join(self.resultdir, 
'temp-*.direct.p*')):
+res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part))
+self.assertEqual(True, files_own_by_root(res.output))
+
+rmtree(self.resultdir, ignore_errors=True)
+
+finally:
+os.environ['PATH'] = oldpath
+
+def test_change_directory(self):
+"""Test --change-directory wks option."""
+
+oldpath = os.environ['PATH']
+os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+try:
+include_path = os.path.join(self.resultdir, 'test-include')
+os.makedirs(include_path)
+wks_file = os.path.join(include_path, 'temp.wks')
+with open(wks_file, 'w') as wks:
+rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+wks.write("part /etc --source rootfs --fstype=ext4 
--change-directory=etc")
+runCmd("wic create %s -e core-image-minimal -o %s" \
+   % (wks_file, self.resultdir))
+
+part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
+
+res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
+files = extract_files(res.output)
+self.assertIn('shadow', files)
+
+finally:
+os.environ['PATH'] = oldpath
+
+def test_change_directory_errors(self):
+"""Test --change-directory wks option error handling."""
+wks_file = 'temp.wks'

[OE-core] [PATCH v6 10/10] oeqa: wic: Add more tests for include_path

[Ricardo Ribalda]

Make sure permissions are respected.

Add new test for orig/destination option.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/lib/oeqa/selftest/cases/wic.py | 64 -
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
b/meta/lib/oeqa/selftest/cases/wic.py
index 8ef08c3d77..1036269b64 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -486,15 +486,77 @@ part /part2 --source rootfs --ondisk mmcblk0 
--fstype=ext4 --include-path %s"""
 res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
 files = extract_files(res.output)
 self.assertNotIn('test-file', files)
+self.assertEqual(True, files_own_by_root(res.output))
 
-# Test partition 2, should not contain 'test-file'
+# Test partition 2, should contain 'test-file'
 res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part2))
 files = extract_files(res.output)
 self.assertIn('test-file', files)
+self.assertEqual(True, files_own_by_root(res.output))
 
 finally:
 os.environ['PATH'] = oldpath
 
+def test_include_path_embeded(self):
+"""Test --include-path wks option."""
+
+oldpath = os.environ['PATH']
+os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+try:
+include_path = os.path.join(self.resultdir, 'test-include')
+os.makedirs(include_path)
+with open(os.path.join(include_path, 'test-file'), 'w') as t:
+t.write("test\n")
+wks_file = os.path.join(include_path, 'temp.wks')
+with open(wks_file, 'w') as wks:
+rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+wks.write("""
+part / --source rootfs  --fstype=ext4 --include-path %s --include-path 
core-image-minimal-mtdutils export/"""
+  % (include_path))
+runCmd("wic create %s -e core-image-minimal -o %s" \
+   % (wks_file, self.resultdir))
+
+part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
+
+res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
+files = extract_files(res.output)
+self.assertIn('test-file', files)
+self.assertEqual(True, files_own_by_root(res.output))
+
+res = runCmd("debugfs -R 'ls -p /export/etc/' %s 2>/dev/null" % 
(part1))
+files = extract_files(res.output)
+self.assertIn('shadow', files)
+self.assertEqual(True, files_own_by_root(res.output))
+
+finally:
+os.environ['PATH'] = oldpath
+
+def test_include_path_errors(self):
+"""Test --include-path wks option error handling."""
+wks_file = 'temp.wks'
+
+# Absolute argument.
+with open(wks_file, 'w') as wks:
+wks.write("part / --source rootfs --fstype=ext4 --include-path 
core-image-minimal-mtdutils /export")
+self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o 
%s" \
+  % (wks_file, self.resultdir), 
ignore_status=True).status)
+os.remove(wks_file)
+
+# Argument pointing to parent directory.
+with open(wks_file, 'w') as wks:
+wks.write("part / --source rootfs --fstype=ext4 --include-path 
core-image-minimal-mtdutils ././..")
+self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o 
%s" \
+  % (wks_file, self.resultdir), 
ignore_status=True).status)
+os.remove(wks_file)
+
+# 3 Argument pointing to parent directory.
+with open(wks_file, 'w') as wks:
+wks.write("part / --source rootfs --fstype=ext4 --include-path 
core-image-minimal-mtdutils export/ dummy")
+self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o 
%s" \
+  % (wks_file, self.resultdir), 
ignore_status=True).status)
+os.remove(wks_file)
+
 def test_exclude_path_errors(self):
 """Test --exclude-path wks option error handling."""
 wks_file = 'temp.wks'
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137217): 
https://lists.openembedded.org/g/openembedded-core/message/137217
Mute This Topic: https://lists.openembedded.org/mt/73010221/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v6 10/10] oeqa: wic: Add more tests for include_path

[Ricardo Ribalda]

 Hi Paul



On Sat, Apr 18, 2020 at 10:10 PM Paul Barker  wrote:
>
> On Tue, 14 Apr 2020 15:36:14 +0200
> Ricardo Ribalda Delgado  wrote:
>
> > Make sure permissions are respected.
> >
> > Add new test for orig/destination option.
> >
> > Cc: Paul Barker 
> > Signed-off-by: Ricardo Ribalda Delgado 
> > ---
> >  meta/lib/oeqa/selftest/cases/wic.py | 64 -
> >  1 file changed, 63 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
> > b/meta/lib/oeqa/selftest/cases/wic.py
> > index 8ef08c3d77..1036269b64 100644
> > --- a/meta/lib/oeqa/selftest/cases/wic.py
> > +++ b/meta/lib/oeqa/selftest/cases/wic.py
> > @@ -486,15 +486,77 @@ part /part2 --source rootfs --ondisk mmcblk0 
> > --fstype=ext4 --include-path %s"""
> >  res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> >  files = extract_files(res.output)
> >  self.assertNotIn('test-file', files)
> > +self.assertEqual(True, files_own_by_root(res.output))
> >
> > -# Test partition 2, should not contain 'test-file'
> > +# Test partition 2, should contain 'test-file'
> >  res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part2))
> >  files = extract_files(res.output)
> >  self.assertIn('test-file', files)
> > +self.assertEqual(True, files_own_by_root(res.output))
> >
> >  finally:
> >  os.environ['PATH'] = oldpath
> >
> > +def test_include_path_embeded(self):
> > +"""Test --include-path wks option."""
> > +
> > +oldpath = os.environ['PATH']
> > +os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
> > +
> > +try:
> > +include_path = os.path.join(self.resultdir, 'test-include')
> > +os.makedirs(include_path)
> > +with open(os.path.join(include_path, 'test-file'), 'w') as t:
> > +t.write("test\n")
> > +wks_file = os.path.join(include_path, 'temp.wks')
> > +with open(wks_file, 'w') as wks:
> > +rootfs_dir = get_bb_var('IMAGE_ROOTFS', 
> > 'core-image-minimal')
> > +wks.write("""
> > +part / --source rootfs  --fstype=ext4 --include-path %s --include-path 
> > core-image-minimal-mtdutils export/"""
> > +  % (include_path))
> > +runCmd("wic create %s -e core-image-minimal -o %s" \
> > +   % (wks_file, self.resultdir))
> > +
> > +part1 = glob(os.path.join(self.resultdir, 
> > 'temp-*.direct.p1'))[0]
> > +
> > +res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> > +files = extract_files(res.output)
> > +self.assertIn('test-file', files)
> > +self.assertEqual(True, files_own_by_root(res.output))
> > +
> > +res = runCmd("debugfs -R 'ls -p /export/etc/' %s 2>/dev/null" 
> > % (part1))
> > +files = extract_files(res.output)
> > +self.assertIn('shadow', files)
>
> This fails if `shadow` isn't part of the image:
>
> 2020-04-18 21:00:01,411 - oe-selftest - INFO - test_include_path_embeded 
> (wic.Wic)
> 2020-04-18 21:00:14,102 - oe-selftest - INFO -  ... FAIL
> 2020-04-18 21:00:14,103 - oe-selftest - INFO - Traceback (most recent call 
> last):
>   File 
> "/home/pbarker/Projects/Yocto/poky/meta/lib/oeqa/selftest/cases/wic.py", line 
> 529, in test_include_path_embeded
> self.assertIn('shadow', files)
> AssertionError: 'shadow' not found in [...]

I though I could assume that /etc/shadow was always part of
core-image-minimal and core-image-minimal-mtdutil. Shall I simply
change it to /etc/passwd?
How did you managed to trigger the error?

Thanks

>
> > +self.assertEqual(True, files_own_by_root(res.output))
> > +
> > +finally:
> > +os.environ['PATH'] = oldpath
> > +
> > +def test_include_path_errors(self):
> > +"""Test --include-path wks option error handling."""
> > + 

Re: [OE-core] [PATCH v6 10/10] oeqa: wic: Add more tests for include_path

[Ricardo Ribalda]

Hi again

On Sat, Apr 18, 2020 at 10:16 PM Ricardo Ribalda via
lists.openembedded.org
 wrote:
>
>  Hi Paul
>
>
>
> On Sat, Apr 18, 2020 at 10:10 PM Paul Barker  wrote:
> >
> > On Tue, 14 Apr 2020 15:36:14 +0200
> > Ricardo Ribalda Delgado  wrote:
> >
> > > Make sure permissions are respected.
> > >
> > > Add new test for orig/destination option.
> > >
> > > Cc: Paul Barker 
> > > Signed-off-by: Ricardo Ribalda Delgado 
> > > ---
> > >  meta/lib/oeqa/selftest/cases/wic.py | 64 -
> > >  1 file changed, 63 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
> > > b/meta/lib/oeqa/selftest/cases/wic.py
> > > index 8ef08c3d77..1036269b64 100644
> > > --- a/meta/lib/oeqa/selftest/cases/wic.py
> > > +++ b/meta/lib/oeqa/selftest/cases/wic.py
> > > @@ -486,15 +486,77 @@ part /part2 --source rootfs --ondisk mmcblk0 
> > > --fstype=ext4 --include-path %s"""
> > >  res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> > >  files = extract_files(res.output)
> > >  self.assertNotIn('test-file', files)
> > > +self.assertEqual(True, files_own_by_root(res.output))
> > >
> > > -# Test partition 2, should not contain 'test-file'
> > > +# Test partition 2, should contain 'test-file'
> > >  res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part2))
> > >  files = extract_files(res.output)
> > >  self.assertIn('test-file', files)
> > > +self.assertEqual(True, files_own_by_root(res.output))
> > >
> > >  finally:
> > >  os.environ['PATH'] = oldpath
> > >
> > > +def test_include_path_embeded(self):
> > > +"""Test --include-path wks option."""
> > > +
> > > +oldpath = os.environ['PATH']
> > > +os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
> > > +
> > > +try:
> > > +include_path = os.path.join(self.resultdir, 'test-include')
> > > +os.makedirs(include_path)
> > > +with open(os.path.join(include_path, 'test-file'), 'w') as t:
> > > +t.write("test\n")
> > > +wks_file = os.path.join(include_path, 'temp.wks')
> > > +with open(wks_file, 'w') as wks:
> > > +rootfs_dir = get_bb_var('IMAGE_ROOTFS', 
> > > 'core-image-minimal')
> > > +wks.write("""
> > > +part / --source rootfs  --fstype=ext4 --include-path %s --include-path 
> > > core-image-minimal-mtdutils export/"""
> > > +  % (include_path))
> > > +runCmd("wic create %s -e core-image-minimal -o %s" \
> > > +   % (wks_file, self.resultdir))
> > > +
> > > +part1 = glob(os.path.join(self.resultdir, 
> > > 'temp-*.direct.p1'))[0]
> > > +
> > > +res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> > > +files = extract_files(res.output)
> > > +self.assertIn('test-file', files)
> > > +self.assertEqual(True, files_own_by_root(res.output))
> > > +
> > > +res = runCmd("debugfs -R 'ls -p /export/etc/' %s 
> > > 2>/dev/null" % (part1))
> > > +files = extract_files(res.output)
> > > +self.assertIn('shadow', files)
> >
> > This fails if `shadow` isn't part of the image:
> >
> > 2020-04-18 21:00:01,411 - oe-selftest - INFO - test_include_path_embeded 
> > (wic.Wic)
> > 2020-04-18 21:00:14,102 - oe-selftest - INFO -  ... FAIL
> > 2020-04-18 21:00:14,103 - oe-selftest - INFO - Traceback (most recent call 
> > last):
> >   File 
> > "/home/pbarker/Projects/Yocto/poky/meta/lib/oeqa/selftest/cases/wic.py", 
> > line 529, in test_include_path_embeded
> > self.assertIn('shadow', files)
> > AssertionError: 'shadow' not found in [...]
>
> I though I could ass

[OE-core] [PATCH v7 08/10] wic: root: Add an opt. destination on include-path

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado 

Allow specifying an optional destination to include-path and make the
option aware of permissions and owners.

It is very useful for making a partition that contains the rootfs for a
host and a target Eg:

/ -> Roofs for the host
/export/ -> Rootfs for the target (which will netboot)

Although today we support making a partition for "/export" this might
not be compatible with some upgrade systems, or we might be limited by
the number of partitions.

With this patch we can use something like:

part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils 
export/ --include-path hello

on the .wks file.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-core/meta/wic-tools.bb  |  2 +-
 scripts/lib/wic/help.py  | 20 
 scripts/lib/wic/ksparser.py  |  2 +-
 scripts/lib/wic/plugins/source/rootfs.py | 60 +++-
 4 files changed, 72 insertions(+), 12 deletions(-)

diff --git a/meta/recipes-core/meta/wic-tools.bb 
b/meta/recipes-core/meta/wic-tools.bb
index 09eb409e87..8aeb942ed2 100644
--- a/meta/recipes-core/meta/wic-tools.bb
+++ b/meta/recipes-core/meta/wic-tools.bb
@@ -6,7 +6,7 @@ DEPENDS = "\
parted-native syslinux-native gptfdisk-native dosfstools-native \
mtools-native bmap-tools-native grub-efi-native cdrtools-native \
btrfs-tools-native squashfs-tools-native pseudo-native \
-   e2fsprogs-native util-linux-native \
+   e2fsprogs-native util-linux-native tar-native\
"
 DEPENDS_append_x86 = " syslinux grub-efi systemd-boot"
 DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot"
diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 62a2a90e79..bd3a2b97df 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -971,14 +971,18 @@ DESCRIPTION
  has an effect with the rootfs source plugin.
 
  --include-path: This option is specific to wic. It adds the contents
- of the given path to the resulting image. The path is
- relative to the directory in which wic is running not
- the rootfs itself so use of an absolute path is
- recommended. This option is most useful when multiple
- copies of the rootfs are added to an image and it is
- required to add extra content to only one of these
- copies. This option only has an effect with the rootfs
- source plugin.
+ of the given path or a rootfs to the resulting image.
+ The option contains two fields, the origin and the
+ destination. When the origin is a rootfs, it follows
+ the same logic as the rootfs-dir argument and the
+ permissions and owners are kept. When the origin is a
+ path, it is relative to the directory in which wic is
+ running not the rootfs itself so use of an absolute
+ path is recommended, and the owner and group is set to
+ root:root. If no destination is given it is
+ automatically set to the root of the rootfs. This
+ option only has an effect with the rootfs source
+ plugin.
 
  --change-directory: This option is specific to wic. It changes to the
  given directory before copying the files. This
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index c60869d397..b8befe78e3 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -137,7 +137,7 @@ class KickStart():
 part.add_argument('--active', action='store_true')
 part.add_argument('--align', type=int)
 part.add_argument('--exclude-path', nargs='+')
-part.add_argument('--include-path', nargs='+')
+part.add_argument('--include-path', nargs='+', action='append')
 part.add_argument('--change-directory')
 part.add_argument("--extra-space", type=sizetype)
 part.add_argument('--fsoptions', dest='fsopts')
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 247f61ff7c..544e868b5e 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -17,6 +17,7 @@ import shutil
 import sys
 
 from oe.path import copyhardlinktree
+from pathlib import Path
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
@@ -126,8 +127,63 @@ class RootfsPlugin(SourcePlu

[OE-core] [PATCH v7 06/10] oeqa: wic: Add tests for permissions and change-directory

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado 

Make sure that the permissions and username are respected when using all
the rootfs modifiers.

Add tests for change-directory command

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/lib/oeqa/selftest/cases/wic.py | 90 +
 1 file changed, 90 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
b/meta/lib/oeqa/selftest/cases/wic.py
index 626a217e69..41cf23f778 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -62,6 +62,12 @@ def extract_files(debugfs_output):
 return [line.split('/')[5].strip() for line in \
 debugfs_output.strip().split('/\n')]
 
+def files_own_by_root(debugfs_output):
+for line in debugfs_output.strip().split('/\n'):
+if line.split('/')[3:5] != ['0', '0']:
+print(debugfs_output)
+return False
+return True
 
 class WicTestCase(OESelftestTestCase):
 """Wic test class."""
@@ -84,6 +90,7 @@ class WicTestCase(OESelftestTestCase):
 self.skipTest('wic-tools cannot be built due its 
(intltool|gettext)-native dependency and NLS disable')
 
 bitbake('core-image-minimal')
+bitbake('core-image-minimal-mtdutils')
 WicTestCase.image_is_ready = True
 
 rmtree(self.resultdir, ignore_errors=True)
@@ -506,6 +513,89 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 
--include-path %s"""
   % (wks_file, self.resultdir), 
ignore_status=True).status)
 os.remove(wks_file)
 
+def test_permissions(self):
+"""Test permissions are respected"""
+
+oldpath = os.environ['PATH']
+os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+t_normal = """
+part / --source rootfs --fstype=ext4
+"""
+t_exclude = """
+part / --source rootfs --fstype=ext4 --exclude-path=home
+"""
+t_multi = """
+part / --source rootfs --ondisk sda --fstype=ext4
+part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
+"""
+t_change = """
+part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
+part /etc --source rootfs --fstype=ext4 --change-directory=etc
+"""
+tests = [t_normal, t_exclude, t_multi, t_change]
+
+try:
+for test in tests:
+include_path = os.path.join(self.resultdir, 'test-include')
+os.makedirs(include_path)
+wks_file = os.path.join(include_path, 'temp.wks')
+with open(wks_file, 'w') as wks:
+wks.write(test)
+runCmd("wic create %s -e core-image-minimal -o %s" \
+   % (wks_file, self.resultdir))
+
+for part in glob(os.path.join(self.resultdir, 
'temp-*.direct.p*')):
+res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part))
+self.assertEqual(True, files_own_by_root(res.output))
+
+rmtree(self.resultdir, ignore_errors=True)
+
+finally:
+os.environ['PATH'] = oldpath
+
+def test_change_directory(self):
+"""Test --change-directory wks option."""
+
+oldpath = os.environ['PATH']
+os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+try:
+include_path = os.path.join(self.resultdir, 'test-include')
+os.makedirs(include_path)
+wks_file = os.path.join(include_path, 'temp.wks')
+with open(wks_file, 'w') as wks:
+wks.write("part /etc --source rootfs --fstype=ext4 
--change-directory=etc")
+runCmd("wic create %s -e core-image-minimal -o %s" \
+   % (wks_file, self.resultdir))
+
+part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
+
+res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
+files = extract_files(res.output)
+self.assertIn('passwd', files)
+
+finally:
+os.environ['PATH'] = oldpath
+
+def test_change_directory_errors(self):
+"""Test --change-directory wks option error handling."""
+wks_file = 'temp.wks'
+
+# Absolute argument.
+with open(wks_file, 'w') as wks:
+wks.write(

[OE-core] [PATCH v7 05/10] wic: Avoid creating invalid pseudo directory

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado 

If the source of the rootfs is not a bitbake cooked image, or it is not
pointing to the root of one, we call pseudo again, which will produce
a new pseudo folder at rootfs/../pseudo

Eg:

part /etc --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/partition.py | 15 ---
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 3240be072a..7d9dd616a6 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -199,13 +199,14 @@ class Partition():
 Currently handles ext2/3/4, btrfs, vfat and squashfs.
 """
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
-if (pseudo_dir == None):
-pseudo_dir = "%s/../pseudo" %  rootfs_dir
-pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
-pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
-pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
-pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
-pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+if (pseudo_dir):
+pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+else:
+pseudo = None
 
 rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
  self.lineno, self.fstype)
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137305): 
https://lists.openembedded.org/g/openembedded-core/message/137305
Mute This Topic: https://lists.openembedded.org/mt/73123276/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

Today wic behaves differently if we run it from bitbake of directly from the
commandline.

When it is run from bitbake, the permissions/usersnames are handled by the
pseudo database of the main image.

When it is run from the comandline ,it is run outside the main image database.

This results on permissions/usernames not working ok on some usecases on both
bitbake and wic.


This is an attempt to fix all the permission bugs that I am aware from wic.
Using the following usecases:


#exclude-path
part / --source rootfs --fstype=ext4 --exclude-path=home

#split_partition
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs 
--rootfs-dir=tmp/work/qt5222-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/ 
--fstype=ext4

#multi_partition
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4


With the current master:
-
#exclude-path
>From Bitbake: OK
wic cmdline: FAIL

#split_partition
>From Bitbake: OK
wic cmdline: FAIL, permissions invalid on both partitions

#multi_partition
>From Bitbake: FAIL second partition
wic cmdline: OK


After:  wic: Fix permissions when using exclude or include path

#exclude-path
>From Bitbake: OK
wic cmdline: OK

#split_partition
>From Bitbake: OK
wic cmdline: FAIL, permissions invalid on second partition

#multi_partition
>From Bitbake: FAIL second partition
wic cmdline: OK


After: wic: Fix multi images .wks with bitbake
--
#exclude-path
>From Bitbake: OK
wic cmdline: OK

#split_partition
>From Bitbake: FAIL, permissions invalid on second partition
wic cmdline: FAIL, permissions invalid on second partition

#multi_partition
>From Bitbake: OK
wic cmdline: OK



After: wic: Add --change-directory argument


we can have a .wks like:
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs --fstype=ext4 --change-directory=etc


Wich works fine from bitbake and from cmdline, and has the same functionality as
split_partition.


Changelog v6->v7 (Thanks Paul):
  - Use passwd instead of shadow for oetests
  - For pseudo folder inside workdir
  - Improve comments on tar
  - Use line number on tar name
  - Allow include-files of files in workdir

Changelog v5->v6:
  - Allow two arguments to include-path
  - Fix permission on include-path
  - exec_native with ASSUME_PROVIDED
  - More unittesting

Changelog v4->v5:
  - Add unittesting
  - wic: Avoid creating invalid pseudo directory

Ricardo Ribalda Delgado (10):
  wic: Fix permissions when using exclude or include path
  wic: Fix multi images .wks with bitbake
  wic: Add --change-directory argument
  wic: Continue if excluded_path does not exist
  wic: Avoid creating invalid pseudo directory
  oeqa: wic: Add tests for permissions and change-directory
  wic: misc: Do not find for executables in ALREADY_PROVIDED
  wic: root: Add an opt. destination on include-path
  wic: rootfs: Combine path_validation in one function
  oeqa: wic: Add more tests for include_path

 meta/classes/image_types_wic.bbclass |   8 +-
 meta/lib/oeqa/selftest/cases/wic.py  | 153 ++-
 meta/recipes-core/meta/wic-tools.bb  |   2 +-
 scripts/lib/wic/help.py  |  26 ++--
 scripts/lib/wic/ksparser.py  |   3 +-
 scripts/lib/wic/misc.py  |  14 ++-
 scripts/lib/wic/partition.py |  20 +--
 scripts/lib/wic/plugins/source/rootfs.py | 133 +---
 8 files changed, 318 insertions(+), 41 deletions(-)

-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137300): 
https://lists.openembedded.org/g/openembedded-core/message/137300
Mute This Topic: https://lists.openembedded.org/mt/73123269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v7 07/10] wic: misc: Do not find for executables in ALREADY_PROVIDED

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado 

Executables like tar won't be available on the native sysroot, as they
are part of the ALREADY_PROVIDED variable.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/misc.py | 14 --
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 1f199b9f23..91975ba151 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -45,7 +45,8 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
   "parted": "parted",
   "sfdisk": "util-linux",
   "sgdisk": "gptfdisk",
-  "syslinux": "syslinux"
+  "syslinux": "syslinux",
+  "tar": "tar"
  }
 
 def runtool(cmdln_or_args):
@@ -112,6 +113,15 @@ def exec_cmd(cmd_and_args, as_shell=False):
 """
 return _exec_cmd(cmd_and_args, as_shell)[1]
 
+def find_executable(cmd, paths):
+recipe = cmd
+if recipe in NATIVE_RECIPES:
+recipe =  NATIVE_RECIPES[recipe]
+provided = get_bitbake_var("ASSUME_PROVIDED")
+if provided and "%s-native" % recipe in provided:
+return True
+
+return spawn.find_executable(cmd, paths)
 
 def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
 """
@@ -136,7 +146,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, 
pseudo=""):
 logger.debug("exec_native_cmd: %s", native_cmd_and_args)
 
 # If the command isn't in the native sysroot say we failed.
-if spawn.find_executable(args[0], native_paths):
+if find_executable(args[0], native_paths):
 ret, out = _exec_cmd(native_cmd_and_args, True)
 else:
 ret = 127
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137307): 
https://lists.openembedded.org/g/openembedded-core/message/137307
Mute This Topic: https://lists.openembedded.org/mt/73123278/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v7 01/10] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado 

When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.

This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.

This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.

This bug can be easily triggerd with the following .wks

part / --source rootfs --fstype=ext4 --exclude-path=home

And this sequence:

$ wic create test-permissions -e core-image-minimal -o test/
$ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
$ ls -la /mnt/etc/shadow

To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.

If the rootfs is not a rootfs generated by bitbake a warning is shown
making the user aware that the permissions on the target might not match
what he expects.

WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions 
will be invalid

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/partition.py |  7 +++--
 scripts/lib/wic/plugins/source/rootfs.py | 37 ++--
 2 files changed, 38 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 2d95f78439..b02711be37 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -190,7 +190,7 @@ class Partition():
(self.mountpoint, self.size, self.fixed_size))
 
 def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
-   native_sysroot, real_rootfs = True):
+   native_sysroot, real_rootfs = True, pseudo_dir = None):
 """
 Prepare content for a rootfs partition i.e. create a partition
 and fill it from a /rootfs dir.
@@ -198,8 +198,9 @@ class Partition():
 Currently handles ext2/3/4, btrfs, vfat and squashfs.
 """
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
-p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
- "%s/../pseudo" %  rootfs_dir)
+if (pseudo_dir == None):
+pseudo_dir = "%s/../pseudo" %  rootfs_dir
+p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
 p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
 p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
 pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 705aeb5563..8b2a067385 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -20,7 +20,7 @@ from oe.path import copyhardlinktree
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
-from wic.misc import get_bitbake_var
+from wic.misc import get_bitbake_var, exec_native_cmd
 
 logger = logging.getLogger('wic')
 
@@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
 
 return os.path.realpath(image_rootfs_dir)
 
+@staticmethod
+def __get_pseudo(native_sysroot, rootfs, pseudo_dir):
+pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+return pseudo
+
 @classmethod
 def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
  oe_builddir, bootimg_dir, kernel_dir,
@@ -68,8 +77,14 @@ class RootfsPlugin(SourcePlugin):
"it is not a valid path, exiting" % 
part.rootfs_dir)
 
 part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
+if not os.path.lexists(pseudo_dir):
+logger.warn("%s folder does not exist. "
+"Usernames and permissions will be invalid " % 
pseudo_dir)
+pseudo_dir = None
 
 new_rootfs = None
+new_pseudo = None
 # Handle excluded paths.
 if part.exclude_path or part.include_path:
 # We need a new rootfs directory we can delete files from. Copy to
@@ -78,9 +93,24 @@ class RootfsPlugin(SourcePlugin):
 
 if os.path.lexists(new_rootfs):
  

[OE-core] [PATCH v7 04/10] wic: Continue if excluded_path does not exist

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado 

If an excuded path does not exist, continue without an error.
This allows to seamleasly reuse .wks among different projects.

Eg:

part / --source rootfs --fstype=ext4 --exclude-path=opt/private_keys

Where /opt/private_keys in only populated by some of the image.bb files.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/plugins/source/rootfs.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 85c634f8a1..247f61ff7c 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -136,7 +136,6 @@ class RootfsPlugin(SourcePlugin):
 sys.exit(1)
 
 full_path = os.path.realpath(os.path.join(new_rootfs, path))
-
 # Disallow climbing outside of parent directory using '..',
 # because doing so could be quite disastrous (we will delete 
the
 # directory).
@@ -144,6 +143,9 @@ class RootfsPlugin(SourcePlugin):
 logger.error("'%s' points to a path outside the rootfs" % 
orig_path)
 sys.exit(1)
 
+if not os.path.lexists(full_path):
+continue
+
 if path.endswith(os.sep):
 # Delete content only.
 for entry in os.listdir(full_path):
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137304): 
https://lists.openembedded.org/g/openembedded-core/message/137304
Mute This Topic: https://lists.openembedded.org/mt/73123273/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v7 09/10] wic: rootfs: Combine path_validation in one function

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado 

Combine all the common path validation in a function to avoid code
duplication.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/plugins/source/rootfs.py | 41 +++-
 1 file changed, 19 insertions(+), 22 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 544e868b5e..f1db83f8a1 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -32,6 +32,22 @@ class RootfsPlugin(SourcePlugin):
 
 name = 'rootfs'
 
+@staticmethod
+def __validate_path(cmd, rootfs_dir, path):
+if os.path.isabs(path):
+logger.error("%s: Must be relative: %s" % (cmd, orig_path))
+sys.exit(1)
+
+# Disallow climbing outside of parent directory using '..',
+# because doing so could be quite disastrous (we will delete the
+# directory, or modify a directory outside OpenEmbedded).
+full_path = os.path.realpath(os.path.join(rootfs_dir, path))
+if not full_path.startswith(os.path.realpath(rootfs_dir)):
+logger.error("%s: Must point inside the rootfs:" % (cmd, path))
+sys.exit(1)
+
+return full_path
+
 @staticmethod
 def __get_rootfs_dir(rootfs_dir):
 if os.path.isdir(rootfs_dir):
@@ -99,14 +115,7 @@ class RootfsPlugin(SourcePlugin):
 cd = part.change_directory
 if cd[-1] == '/':
 cd = cd[:-1]
-if os.path.isabs(cd):
-logger.error("Must be relative: --change-directory=%s" % 
cd)
-sys.exit(1)
-orig_dir = os.path.realpath(os.path.join(part.rootfs_dir, cd))
-if not orig_dir.startswith(part.rootfs_dir):
-logger.error("'%s' points to a path outside the rootfs" % 
orig_dir)
-sys.exit(1)
-
+orig_dir = cls.__validate_path("--change-directory", 
part.rootfs_dir, cd)
 else:
 orig_dir = part.rootfs_dir
 copyhardlinktree(orig_dir, new_rootfs)
@@ -168,10 +177,7 @@ class RootfsPlugin(SourcePlugin):
 
 #create destination
 if path:
-destination = os.path.realpath(os.path.join(new_rootfs, 
path))
-if not destination.startswith(new_rootfs):
-logger.error("%s %s" % (destination, new_rootfs))
-sys.exit(1)
+destination = cls.__validate_path("--include-path", 
new_rootfs, path)
 Path(destination).mkdir(parents=True, exist_ok=True)
 else:
 destination = new_rootfs
@@ -187,17 +193,8 @@ class RootfsPlugin(SourcePlugin):
 
 for orig_path in part.exclude_path or []:
 path = orig_path
-if os.path.isabs(path):
-logger.error("Must be relative: --exclude-path=%s" % 
orig_path)
-sys.exit(1)
 
-full_path = os.path.realpath(os.path.join(new_rootfs, path))
-# Disallow climbing outside of parent directory using '..',
-# because doing so could be quite disastrous (we will delete 
the
-# directory).
-if not full_path.startswith(new_rootfs):
-logger.error("'%s' points to a path outside the rootfs" % 
orig_path)
-sys.exit(1)
+full_path = cls.__validate_path("--exclude-path", new_rootfs, 
path)
 
 if not os.path.lexists(full_path):
 continue
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137309): 
https://lists.openembedded.org/g/openembedded-core/message/137309
Mute This Topic: https://lists.openembedded.org/mt/73123280/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v7 10/10] oeqa: wic: Add more tests for include_path

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado 

Make sure permissions are respected.

Add new test for orig/destination option.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/lib/oeqa/selftest/cases/wic.py | 63 -
 1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
b/meta/lib/oeqa/selftest/cases/wic.py
index 41cf23f778..c8765e5330 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -486,15 +486,76 @@ part /part2 --source rootfs --ondisk mmcblk0 
--fstype=ext4 --include-path %s"""
 res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
 files = extract_files(res.output)
 self.assertNotIn('test-file', files)
+self.assertEqual(True, files_own_by_root(res.output))
 
-# Test partition 2, should not contain 'test-file'
+# Test partition 2, should contain 'test-file'
 res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part2))
 files = extract_files(res.output)
 self.assertIn('test-file', files)
+self.assertEqual(True, files_own_by_root(res.output))
 
 finally:
 os.environ['PATH'] = oldpath
 
+def test_include_path_embeded(self):
+"""Test --include-path wks option."""
+
+oldpath = os.environ['PATH']
+os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+try:
+include_path = os.path.join(self.resultdir, 'test-include')
+os.makedirs(include_path)
+with open(os.path.join(include_path, 'test-file'), 'w') as t:
+t.write("test\n")
+wks_file = os.path.join(include_path, 'temp.wks')
+with open(wks_file, 'w') as wks:
+wks.write("""
+part / --source rootfs  --fstype=ext4 --include-path %s --include-path 
core-image-minimal-mtdutils export/"""
+  % (include_path))
+runCmd("wic create %s -e core-image-minimal -o %s" \
+   % (wks_file, self.resultdir))
+
+part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
+
+res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
+files = extract_files(res.output)
+self.assertIn('test-file', files)
+self.assertEqual(True, files_own_by_root(res.output))
+
+res = runCmd("debugfs -R 'ls -p /export/etc/' %s 2>/dev/null" % 
(part1))
+files = extract_files(res.output)
+self.assertIn('passwd', files)
+self.assertEqual(True, files_own_by_root(res.output))
+
+finally:
+os.environ['PATH'] = oldpath
+
+def test_include_path_errors(self):
+"""Test --include-path wks option error handling."""
+wks_file = 'temp.wks'
+
+# Absolute argument.
+with open(wks_file, 'w') as wks:
+wks.write("part / --source rootfs --fstype=ext4 --include-path 
core-image-minimal-mtdutils /export")
+self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o 
%s" \
+  % (wks_file, self.resultdir), 
ignore_status=True).status)
+os.remove(wks_file)
+
+# Argument pointing to parent directory.
+with open(wks_file, 'w') as wks:
+wks.write("part / --source rootfs --fstype=ext4 --include-path 
core-image-minimal-mtdutils ././..")
+self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o 
%s" \
+  % (wks_file, self.resultdir), 
ignore_status=True).status)
+os.remove(wks_file)
+
+# 3 Argument pointing to parent directory.
+with open(wks_file, 'w') as wks:
+wks.write("part / --source rootfs --fstype=ext4 --include-path 
core-image-minimal-mtdutils export/ dummy")
+self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o 
%s" \
+  % (wks_file, self.resultdir), 
ignore_status=True).status)
+os.remove(wks_file)
+
 def test_exclude_path_errors(self):
 """Test --exclude-path wks option error handling."""
 wks_file = 'temp.wks'
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137310): 
https://lists.openembedded.org/g/openembedded-core/message/137310
Mute This Topic: https://lists.openembedded.org/mt/73123281/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v7 03/10] wic: Add --change-directory argument

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado 

This option allows to specify which part of a rootfs is going to be
included, the same way the -C argument on tar.

Thanks to this option we can make sure the permissions and usernames
on the target partition are respected, and also simplify the creation of
splitted partitons, not neeting to invoke external vars or using .wks.in
files. Eg:

part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
part /etc --source rootfs --fstype=ext4 --change-directory=etc

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/help.py  |  6 ++
 scripts/lib/wic/ksparser.py  |  1 +
 scripts/lib/wic/partition.py |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 21 ++---
 4 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 1e3d06a87b..62a2a90e79 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -980,6 +980,12 @@ DESCRIPTION
  copies. This option only has an effect with the rootfs
  source plugin.
 
+ --change-directory: This option is specific to wic. It changes to the
+ given directory before copying the files. This
+ option is useful when we want to split a rootfs in
+ multiple partitions and we want to keep the right
+ permissions and usernames in all the partitions.
+
  --extra-space: This option is specific to wic. It adds extra
 space after the space filled by the content
 of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 650b976223..c60869d397 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -138,6 +138,7 @@ class KickStart():
 part.add_argument('--align', type=int)
 part.add_argument('--exclude-path', nargs='+')
 part.add_argument('--include-path', nargs='+')
+part.add_argument('--change-directory')
 part.add_argument("--extra-space", type=sizetype)
 part.add_argument('--fsoptions', dest='fsopts')
 part.add_argument('--fstype', default='vfat',
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index d850fbd1b1..3240be072a 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -31,6 +31,7 @@ class Partition():
 self.extra_space = args.extra_space
 self.exclude_path = args.exclude_path
 self.include_path = args.include_path
+self.change_directory = args.change_directory
 self.fsopts = args.fsopts
 self.fstype = args.fstype
 self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py 
b/scripts/lib/wic/plugins/source/rootfs.py
index 8b2a067385..85c634f8a1 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -86,14 +86,29 @@ class RootfsPlugin(SourcePlugin):
 new_rootfs = None
 new_pseudo = None
 # Handle excluded paths.
-if part.exclude_path or part.include_path:
+if part.exclude_path or part.include_path or part.change_directory:
 # We need a new rootfs directory we can delete files from. Copy to
 # workdir.
 new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" 
% part.lineno))
 
 if os.path.lexists(new_rootfs):
 shutil.rmtree(os.path.join(new_rootfs))
-copyhardlinktree(part.rootfs_dir, new_rootfs)
+
+if part.change_directory:
+cd = part.change_directory
+if cd[-1] == '/':
+cd = cd[:-1]
+if os.path.isabs(cd):
+logger.error("Must be relative: --change-directory=%s" % 
cd)
+sys.exit(1)
+orig_dir = os.path.realpath(os.path.join(part.rootfs_dir, cd))
+if not orig_dir.startswith(part.rootfs_dir):
+logger.error("'%s' points to a path outside the rootfs" % 
orig_dir)
+sys.exit(1)
+
+else:
+orig_dir = part.rootfs_dir
+copyhardlinktree(orig_dir, new_rootfs)
 
 # Convert the pseudo directory to its new location
 if (pseudo_dir):
@@ -108,7 +123,7 @@ class RootfsPlugin(SourcePlugin):
 pseudo_cmd = "%s -B -m %s -M %s" % 
(cls.__get_pseudo(native_sysroot,
  
new_rootfs,
  

[OE-core] [PATCH v7 02/10] wic: Fix multi images .wks with bitbake

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado 

In order to support .wks files with multiple images inside bitbake we
need to explicitly set the pseudo database in use.

Eg: If we try this .mks:
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4

The username for all the files under /export will be set to the runner
of bitbake (usually UID 1000).

Before we run wic, we need to make sure that the pseudo database will be
flushed, and contains all the data needed.

Cc: Paul Barker 
Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/classes/image_types_wic.bbclass | 8 ++--
 scripts/lib/wic/partition.py | 9 +++--
 2 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/meta/classes/image_types_wic.bbclass 
b/meta/classes/image_types_wic.bbclass
index b83308b45c..96ed0473ee 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -32,8 +32,7 @@ IMAGE_CMD_wic () {
if [ -z "$wks" ]; then
bbfatal "No kickstart files from WKS_FILES were found: 
${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
fi
-
-   BUILDDIR="${TOPDIR}" wic create "$wks" --vars 
"${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" 
${WIC_CREATE_EXTRA_ARGS}
+   BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars 
"${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" 
${WIC_CREATE_EXTRA_ARGS}
mv "$build_wic/$(basename "${wks%.wks}")"*.direct 
"$out${IMAGE_NAME_SUFFIX}.wic"
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
@@ -86,6 +85,10 @@ python do_write_wks_template () {
 bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + 
os.path.basename(wks_file)))
 }
 
+do_flush_pseudodb() {
+   ${FAKEROOTENV} ${FAKEROOTCMD} -S
+}
+
 python () {
 if d.getVar('USING_WIC'):
 wks_file_u = d.getVar('WKS_FULL_PATH', False)
@@ -139,6 +142,7 @@ python do_rootfs_wicenv () {
 depdir = d.getVar('IMGDEPLOYDIR')
 bb.utils.copyfile(os.path.join(outdir, basename) + '.env', 
os.path.join(depdir, basename) + '.env')
 }
+addtask do_flush_pseudodb after do_image before do_image_wic
 addtask do_rootfs_wicenv after do_image before do_image_wic
 do_rootfs_wicenv[vardeps] += "${WICVARS}"
 do_rootfs_wicenv[prefuncs] = 'set_image_size'
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index b02711be37..d850fbd1b1 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -200,13 +200,10 @@ class Partition():
 p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
 if (pseudo_dir == None):
 pseudo_dir = "%s/../pseudo" %  rootfs_dir
-p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
-p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
-p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
 pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
-pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
-pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
-pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
+pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
 pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
 
 rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
-- 
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137302): 
https://lists.openembedded.org/g/openembedded-core/message/137302
Mute This Topic: https://lists.openembedded.org/mt/73123271/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

Hi Richard

Thanks for the test.

All the errors seem to have the same pattern:

FileNotFoundError: [Errno 2] No such file or directory:
'/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-27135/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-r0/testimage/qemurunner_log.20200423015612'

I do not believe that it is related to my patchset, I have seen that
you are already running the selftest without the patchset
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/850

Lets see if it also crashes or not. I am also trying to replicate locally.

Best regards!

On Thu, Apr 23, 2020 at 10:54 AM Richard Purdie
 wrote:
>
> On Sun, 2020-04-19 at 08:35 +0200, Ricardo Ribalda wrote:
> >
> > Changelog v6->v7 (Thanks Paul):
> >   - Use passwd instead of shadow for oetests
> >   - For pseudo folder inside workdir
> >   - Improve comments on tar
> >   - Use line number on tar name
> >   - Allow include-files of files in workdir
> >
> > Changelog v5->v6:
> >   - Allow two arguments to include-path
> >   - Fix permission on include-path
> >   - exec_native with ASSUME_PROVIDED
> >   - More unittesting
> >
> > Changelog v4->v5:
> >   - Add unittesting
> >   - wic: Avoid creating invalid pseudo directory
> >
> > Ricardo Ribalda Delgado (10):
> >   wic: Fix permissions when using exclude or include path
> >   wic: Fix multi images .wks with bitbake
> >   wic: Add --change-directory argument
> >   wic: Continue if excluded_path does not exist
> >   wic: Avoid creating invalid pseudo directory
> >   oeqa: wic: Add tests for permissions and change-directory
> >   wic: misc: Do not find for executables in ALREADY_PROVIDED
> >   wic: root: Add an opt. destination on include-path
> >   wic: rootfs: Combine path_validation in one function
> >   oeqa: wic: Add more tests for include_path
>
> I did run this through the autobuilder along with other changes and
> there were 5 oe-selftest failures for wic:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/858
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/855
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/861
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/849
>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137410): 
https://lists.openembedded.org/g/openembedded-core/message/137410
Mute This Topic: https://lists.openembedded.org/mt/73123269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

On Thu, Apr 23, 2020 at 12:45 PM Richard Purdie
 wrote:
>
> On Thu, 2020-04-23 at 11:57 +0200, Ricardo Ribalda Delgado wrote:
> > Hi Richard
> >
> > Thanks for the test.
> >
> > All the errors seem to have the same pattern:
> >
> > FileNotFoundError: [Errno 2] No such file or directory:
> > '/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-
> > 27135/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-
> > r0/testimage/qemurunner_log.20200423015612'
> >
> > I do not believe that it is related to my patchset, I have seen that
> > you are already running the selftest without the patchset
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/850
> >
> > Lets see if it also crashes or not. I am also trying to replicate
> > locally.
>
> It did recur:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/862
>
> so your patches are probably not the cause, sorry! :)

No problem, thanks!

I am trying to replicate it anyway here. If I find a problem I will ping you.

>
> I'll continue to try and track it down.
>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137412): 
https://lists.openembedded.org/g/openembedded-core/message/137412
Mute This Topic: https://lists.openembedded.org/mt/73123269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

ver connection (#1)...
Loading cache...done.
rootfs file: 
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
Qemu log file: 
/workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423130652
SSH log file: 
/workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423130652
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:39627
Created listening socket for qemu serial console on: 127.0.0.1:37345
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
tcpserial=39627:37345 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2182265
waiting at most 120 seconds for qemu pid (04/23/20 13:06:55)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231306_qemu
----------
2020-04-23 13:07:15,393 - oe-selftest - INFO - Ran 5 tests in 11724.705s
2020-04-23 13:07:15,393 - oe-selftest - INFO - FAILED
2020-04-23 13:07:15,394 - oe-selftest - INFO -  (errors=5)
2020-04-23 13:07:20,028 - oe-selftest - INFO - RESULTS:
2020-04-23 13:07:20,030 - oe-selftest - INFO - RESULTS -
wic.Wic2.test_biosplusefi_plugin_qemu: ERROR (10504.67s)
2020-04-23 13:07:20,030 - oe-selftest - INFO - RESULTS -
wic.Wic2.test_expand_mbr_image: ERROR (117.71s)
2020-04-23 13:07:20,030 - oe-selftest - INFO - RESULTS -
wic.Wic2.test_qemu: ERROR (122.31s)
2020-04-23 13:07:20,031 - oe-selftest - INFO - RESULTS -
wic.Wic2.test_qemu_efi: ERROR (774.41s)
2020-04-23 13:07:20,031 - oe-selftest - INFO - RESULTS -
wic.Wic2.test_rawcopy_plugin_qemu: ERROR (205.60s)
2020-04-23 13:07:20,033 - oe-selftest - INFO - SUMMARY:
2020-04-23 13:07:20,033 - oe-selftest - INFO - oe-selftest () - Ran 5
tests in 11726.658s
2020-04-23 13:07:20,033 - oe-selftest - INFO - oe-selftest - FAIL -
Required tests failed (successes=0, skipped=0, failures=0, errors=5)

On Thu, Apr 23, 2020 at 12:47 PM Ricardo Ribalda via
lists.openembedded.org
 wrote:
>
> On Thu, Apr 23, 2020 at 12:45 PM Richard Purdie
>  wrote:
> >
> > On Thu, 2020-04-23 at 11:57 +0200, Ricardo Ribalda Delgado wrote:
> > > Hi Richard
> > >
> > > Thanks for the test.
> > >
> > > All the errors seem to have the same pattern:
> > >
> > > FileNotFoundError: [Errno 2] No such file or directory:
> > > '/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-
> > > 27135/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-
> > > r0/testimage/qemurunner_log.20200423015612'
> > >
> > > I do not believe that it is related to my patchset, I have seen that
> > > you are already running the selftest without the patchset
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/850
> > >
> > > Lets see if it also crashes or not. I am also trying to replicate
> > > locally.
> >
> > It did recur:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/862
> >
> > so your patches are probably not the cause, sorry! :)
>
> No problem, thanks!
>
> I am trying to replicate it anyway here. If I find a problem I will ping you.
>
> >
> > I'll continue to try and track it down.
> >
> > Cheers,
> >
> > Richard
> >
>
>
> --
> Ricardo Ribalda
> 



-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137417): 
https://lists.openembedded.org/g/openembedded-core/message/137417
Mute This Topic: https://lists.openembedded.org/mt/73123269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

On Thu, Apr 23, 2020 at 3:23 PM Richard Purdie
 wrote:
>
> On Thu, 2020-04-23 at 15:17 +0200, Ricardo Ribalda Delgado wrote:
> > On my setup, using debian crops the error is even different :(
> >
> > If you want I can keep trying, otherwise I will leave it to the
> > experts ;)
>
> Buried in there I think you have a setup problem:

I launched it inside crops, I was expecting that everything was ok, but

/dev/net/tun is missing :(
pokyuser@e81a7801004c:/workdir/build$ ls -la /dev/net/tun
ls: cannot access '/dev/net/tun': No such file or directory


I can add it and retry, but it takes forever to run every test. Is
there a way to let it rerun, not starting from scratch every time?

2020-04-23 13:18:31,528 - oe-selftest - INFO - Adding layer libraries:
2020-04-23 13:18:31,529 - oe-selftest - INFO - /workdir/meta/lib
2020-04-23 13:18:31,529 - oe-selftest - INFO - /workdir/meta-yocto-bsp/lib
2020-04-23 13:18:31,529 - oe-selftest - INFO - /workdir/meta-selftest/lib
2020-04-23 13:18:31,531 - oe-selftest - INFO - Running bitbake -e to
test the configuration is valid/parsable
2020-04-23 13:18:34,955 - oe-selftest - ERROR - Build directory
/workdir/build-st already exists, aborting


Cheers
>
> Output from runqemu:
> runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
> may need to enable TUN (e.g. sudo modprobe tun)
> runqemu - INFO - Cleaning up
>
> I'm trying to narrow down the failing patch but its looking like it may
> take me a while as its not immediately obvious :(
>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137419): 
https://lists.openembedded.org/g/openembedded-core/message/137419
Mute This Topic: https://lists.openembedded.org/mt/73123269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] RFC wic: Support .wks files for multiple machines

[Ricardo Ribalda]

If you want to make a disk image containing rootfs for different
machines, the only way to do it today is by calling wic and passing the
different rootfs-paths.

Eg:
combined.wks
part /export --source rootfs --rootfs-dir=target-image
part / --source rootfs

bitbake multiconfig:arm:target-image
wic create combined --rootfs-dir 
target-image=/workdir/build/tmp/work/arm-poky-linux/target-image/1.0-r0/rootfs/ 
-e main

This has many drawbacks:
1) You need to know the folder location for the target-image
2) It is easy to forget updating the target (only running wic and not
   bitbake)
3) It is slow
4) It does not scale when you have multiple machines

Instead, wic can be given a hint of what machine to search for:

combined.wks
part /export --source rootfs --rootfs-dir=arm:target-image
part / --source rootfs

bitbake main

If we ensoure the dependency of target-image from main via
do_image[mcdepends], all the dependencies are automatically
handled.

This patch makes wic aware of the machine to use.

Signed-off-by: Ricardo Ribalda Delgado 
---
 scripts/lib/wic/misc.py | 16 +---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 91975ba151..51b43b49aa 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -206,7 +206,12 @@ class BitbakeVars(defaultdict):
 
 if image not in self:
 if image and self.vars_dir:
-fname = os.path.join(self.vars_dir, image + '.env')
+if image.find(":") != -1:
+[machine, img] = image.split(":")
+fname = os.path.join(self.vars_dir, "../../",
+ machine, "imgdata", img + '.env')
+else:
+fname = os.path.join(self.vars_dir, image + '.env')
 if os.path.isfile(fname):
 # parse .env file
 with open(fname) as varsfile:
@@ -220,11 +225,16 @@ class BitbakeVars(defaultdict):
 # Get bitbake -e output
 cmd = "bitbake -e"
 if image:
-cmd += " %s" % image
+if image.find(":") != -1:
+[machine, img] = image.split(":")
+cmd = "MACHINE=%s %s" % (machine, cmd)
+else:
+img = image
+cmd += " %s" % img
 
 log_level = logger.getEffectiveLevel()
 logger.setLevel(logging.INFO)
-ret, lines = _exec_cmd(cmd)
+ret, lines = _exec_cmd(cmd, as_shell=True)
 logger.setLevel(log_level)
 
 if ret:
-- 
2.26.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137446): 
https://lists.openembedded.org/g/openembedded-core/message/137446
Mute This Topic: https://lists.openembedded.org/mt/73241116/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

Hi Richard

Thanks for the update!

On Sat, Apr 25, 2020 at 11:07 PM Richard Purdie
 wrote:
>
> On Thu, 2020-04-23 at 15:17 +0200, Ricardo Ribalda Delgado wrote:
> > On my setup, using debian crops the error is even different :(
> >
> > If you want I can keep trying, otherwise I will leave it to the
> > experts ;)
>
> Just to update, I bisected this down to my sysroot patches in master-
> next. Which makes no sense, how would sysroot changes cause wic to fail
> in runqemu?
>
> I've discovered that you have to run two tests in the same build one
> after the other, specifically:
>
> oe-selftest -r  devtool.DevtoolExtractTests.test_devtool_deploy_target 
> wic.Wic2.test_qemu_efi
>
> and I think my sysroot change "causes" problems as it adds a new test
> class. This reorders tests and triggers this magic order problem. I
> haven't confirmed but I think the above will fail without my patch.
>
> The key is the wic test failing with:
>
> FileNotFoundError: [Errno 2] No such file or directory: 
> '/media/build1/poky/build-st/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-r0/testimage/qemurunner_log.20200425200658'
>
> i.e. why is it writing to oe-selftest-image which is only used by the
> devtool test?
>
> The answer is I think the runqemu code is leaking logfile handles
> somehow.
>
> At least now we have a simpler and consistent reproducer and some idea
> of the problem this should be easier to track down.

I have also been working on the issue, and eventhought I have not been
able to replicate locally the bug, the code insspection was pointing
to the same direction.
My running hypothesis is that the logger keeps writting to the
filehandler because nobody has removed it, so the second time we use
qemu, the log file is missing and all dies


Eg: Here I cannot see where the handler is removed
https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/targetcontrol.py?h=master-next#n122


Eg: Here I can

https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/utils/commands.py#n353


Unfortunately, as I said, I cannot replicate it, so it is a bit frustrating.

Since you can replicate it, could I send you a couple of patches, not
to fix it, but to find out if I am on the right direction?

Can you also publish a git branch somewhere with the current error, so
we work on the same codebase?

Or you just prefer to continue on your own?

Thanks

>
> Cheers,
>
> Richard
>
>
>


-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137487): 
https://lists.openembedded.org/g/openembedded-core/message/137487
Mute This Topic: https://lists.openembedded.org/mt/73123269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] RFC wic: Support .wks files for multiple machines

[Ricardo Ribalda]

Hi Christopher

On Fri, Apr 24, 2020 at 5:56 PM Christopher Larson  wrote:
>
> This will only work the way you expect if hte user's local.conf uses ?= or 
> ??= for MACHINE, otherwise setting it in the environment will do nothing.
>

Good catch, You are right :(,

Any idea of how could I fix it. I think I would have to change
site.conf, or local.conf from wic, and I do not want to do that.

Ignoring this bug we are still covering a big number of usecases:

- bibake
- wic with user provided rootfs-dir
- wic without userprovided rootfs-dir and ?= or ??=

At the end of the day this is just a RFC ;), so i am open to any
implementation. I would love to hear your thoughts

Best regards



> On Fri, Apr 24, 2020 at 6:30 AM Ricardo Ribalda  
> wrote:
>>
>> If you want to make a disk image containing rootfs for different
>> machines, the only way to do it today is by calling wic and passing the
>> different rootfs-paths.
>>
>> Eg:
>> combined.wks
>> part /export --source rootfs --rootfs-dir=target-image
>> part / --source rootfs
>>
>> bitbake multiconfig:arm:target-image
>> wic create combined --rootfs-dir 
>> target-image=/workdir/build/tmp/work/arm-poky-linux/target-image/1.0-r0/rootfs/
>>  -e main
>>
>> This has many drawbacks:
>> 1) You need to know the folder location for the target-image
>> 2) It is easy to forget updating the target (only running wic and not
>>bitbake)
>> 3) It is slow
>> 4) It does not scale when you have multiple machines
>>
>> Instead, wic can be given a hint of what machine to search for:
>>
>> combined.wks
>> part /export --source rootfs --rootfs-dir=arm:target-image
>> part / --source rootfs
>>
>> bitbake main
>>
>> If we ensoure the dependency of target-image from main via
>> do_image[mcdepends], all the dependencies are automatically
>> handled.
>>
>> This patch makes wic aware of the machine to use.
>>
>> Signed-off-by: Ricardo Ribalda Delgado 
>> ---
>>  scripts/lib/wic/misc.py | 16 +---
>>  1 file changed, 13 insertions(+), 3 deletions(-)
>>
>> diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
>> index 91975ba151..51b43b49aa 100644
>> --- a/scripts/lib/wic/misc.py
>> +++ b/scripts/lib/wic/misc.py
>> @@ -206,7 +206,12 @@ class BitbakeVars(defaultdict):
>>
>>  if image not in self:
>>  if image and self.vars_dir:
>> -fname = os.path.join(self.vars_dir, image + '.env')
>> +if image.find(":") != -1:
>> +[machine, img] = image.split(":")
>> +fname = os.path.join(self.vars_dir, "../../",
>> + machine, "imgdata", img + '.env')
>> +else:
>> +fname = os.path.join(self.vars_dir, image + '.env')
>>  if os.path.isfile(fname):
>>  # parse .env file
>>  with open(fname) as varsfile:
>> @@ -220,11 +225,16 @@ class BitbakeVars(defaultdict):
>>  # Get bitbake -e output
>>  cmd = "bitbake -e"
>>  if image:
>> -cmd += " %s" % image
>> +if image.find(":") != -1:
>> +[machine, img] = image.split(":")
>> +cmd = "MACHINE=%s %s" % (machine, cmd)
>> +else:
>> +            img = image
>> +cmd += " %s" % img
>>
>>  log_level = logger.getEffectiveLevel()
>>  logger.setLevel(logging.INFO)
>> -ret, lines = _exec_cmd(cmd)
>> +ret, lines = _exec_cmd(cmd, as_shell=True)
>>  logger.setLevel(log_level)
>>
>>  if ret:
>> --
>> 2.26.1
>>
>> 
>
>
>
> --
> Christopher Larson
> kergoth at gmail dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Senior Software Engineer, Mentor Graphics



-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137488): 
https://lists.openembedded.org/g/openembedded-core/message/137488
Mute This Topic: https://lists.openembedded.org/mt/73241116/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

Hi

On Sat, Apr 25, 2020 at 11:29 PM Richard Purdie
 wrote:
>
> On Sat, 2020-04-25 at 23:25 +0200, Ricardo Ribalda Delgado wrote:
> > I have also been working on the issue, and eventhought I have not
> > been
> > able to replicate locally the bug, the code insspection was pointing
> > to the same direction.
> > My running hypothesis is that the logger keeps writting to the
> > filehandler because nobody has removed it, so the second time we use
> > qemu, the log file is missing and all dies
> >
> >
> > Eg: Here I cannot see where the handler is removed
> > https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/targetcontrol.py?h=master-next#n122
> >
> >
> > Eg: Here I can
> >
> > https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/utils/commands.py#n353
> >
> >
> > Unfortunately, as I said, I cannot replicate it, so it is a bit
> > frustrating.
>
> The line I gave in the last email should reproduce it reliably. The key
> is to have a runqemu session which you then clean the image for
> (devtool does), then a write happens to a directory which no longer
> exists from the logging.
>
> > Since you can replicate it, could I send you a couple of patches, not
> > to fix it, but to find out if I am on the right direction?
>
> Since my other email I couldn't resist poking at the code. Locally this
> patch fixed it so its now also running on the autobuilder:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?h=master-next&id=3e7d64adba65559e9b8af42e973c8524988b216a
>
> > Can you also publish a git branch somewhere with the current error,
> > so we work on the same codebase?
> >
> > Or you just prefer to continue on your own?
>
> Happy to have help but I'm simply using master-next and with any luck I
> might have found it.

If you could replicate it locally, and this fixes it. I am almost sure
you found it. I was looking in the same place ;)

Thanks

>
> Cheers,
>
> Richard
>
>
>


-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#137490): 
https://lists.openembedded.org/g/openembedded-core/message/137490
Mute This Topic: https://lists.openembedded.org/mt/73123269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] RFC wic: Support .wks files for multiple machines

[Ricardo Ribalda]

Hi

Even if it does not cover all the use cases. Can we consider this patch?

Thanks!

On Sat, Apr 25, 2020 at 11:29 PM Ricardo Ribalda Delgado
 wrote:
>
> Hi Christopher
>
> On Fri, Apr 24, 2020 at 5:56 PM Christopher Larson  wrote:
> >
> > This will only work the way you expect if hte user's local.conf uses ?= or 
> > ??= for MACHINE, otherwise setting it in the environment will do nothing.
> >
>
> Good catch, You are right :(,
>
> Any idea of how could I fix it. I think I would have to change
> site.conf, or local.conf from wic, and I do not want to do that.
>
> Ignoring this bug we are still covering a big number of usecases:
>
> - bibake
> - wic with user provided rootfs-dir
> - wic without userprovided rootfs-dir and ?= or ??=
>
> At the end of the day this is just a RFC ;), so i am open to any
> implementation. I would love to hear your thoughts
>
> Best regards
>
>
>
> > On Fri, Apr 24, 2020 at 6:30 AM Ricardo Ribalda  
> > wrote:
> >>
> >> If you want to make a disk image containing rootfs for different
> >> machines, the only way to do it today is by calling wic and passing the
> >> different rootfs-paths.
> >>
> >> Eg:
> >> combined.wks
> >> part /export --source rootfs --rootfs-dir=target-image
> >> part / --source rootfs
> >>
> >> bitbake multiconfig:arm:target-image
> >> wic create combined --rootfs-dir 
> >> target-image=/workdir/build/tmp/work/arm-poky-linux/target-image/1.0-r0/rootfs/
> >>  -e main
> >>
> >> This has many drawbacks:
> >> 1) You need to know the folder location for the target-image
> >> 2) It is easy to forget updating the target (only running wic and not
> >>bitbake)
> >> 3) It is slow
> >> 4) It does not scale when you have multiple machines
> >>
> >> Instead, wic can be given a hint of what machine to search for:
> >>
> >> combined.wks
> >> part /export --source rootfs --rootfs-dir=arm:target-image
> >> part / --source rootfs
> >>
> >> bitbake main
> >>
> >> If we ensoure the dependency of target-image from main via
> >> do_image[mcdepends], all the dependencies are automatically
> >> handled.
> >>
> >> This patch makes wic aware of the machine to use.
> >>
> >> Signed-off-by: Ricardo Ribalda Delgado 
> >> ---
> >>  scripts/lib/wic/misc.py | 16 +---
> >>  1 file changed, 13 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
> >> index 91975ba151..51b43b49aa 100644
> >> --- a/scripts/lib/wic/misc.py
> >> +++ b/scripts/lib/wic/misc.py
> >> @@ -206,7 +206,12 @@ class BitbakeVars(defaultdict):
> >>
> >>  if image not in self:
> >>  if image and self.vars_dir:
> >> -fname = os.path.join(self.vars_dir, image + '.env')
> >> +if image.find(":") != -1:
> >> +[machine, img] = image.split(":")
> >> +fname = os.path.join(self.vars_dir, "../../",
> >> + machine, "imgdata", img + '.env')
> >> +else:
> >> +fname = os.path.join(self.vars_dir, image + '.env')
> >>  if os.path.isfile(fname):
> >>  # parse .env file
> >>  with open(fname) as varsfile:
> >> @@ -220,11 +225,16 @@ class BitbakeVars(defaultdict):
> >>  # Get bitbake -e output
> >>  cmd = "bitbake -e"
> >>  if image:
> >> -cmd += " %s" % image
> >> +if image.find(":") != -1:
> >> +    [machine, img] = image.split(":")
> >> +cmd = "MACHINE=%s %s" % (machine, cmd)
> >> +else:
> >> +img = image
> >> +cmd += " %s" % img
> >>
> >>  log_level = logger.getEffectiveLevel()
> >>  logger.setLevel(logging.INFO)
> >> -ret, lines = _exec_cmd(cmd)
> >> +ret, lines = _exec_cmd(cmd, as_shell=True)
> >>  logger.setLevel(log_level)
> >>
> >>  if ret:
> >> --
> >> 2.26.1
> >>
> >> 
> >
> >
> >
> > --
> > Christopher Larson
> > kergoth at gmail dot com
> > Founder - BitBake, OpenEmbedded, OpenZaurus
> > Senior Software Engineer, Mentor Graphics
>
>
>
> --
> Ricardo Ribalda



-- 
Ricardo Ribalda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#138205): 
https://lists.openembedded.org/g/openembedded-core/message/138205
Mute This Topic: https://lists.openembedded.org/mt/73241116/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] package_deb: Map TARGET_ARCH x86_64 to DPKG_ARCH amd64

[Ricardo Ribalda Delgado]

Without this patch packages are generated as x86_64. Which cannot be
installed by default.

root@qt5022:~# dpkg -i alsa-utils_1.0.27.2-r0_x86-64.deb
dpkg: error processing alsa-utils_1.0.27.2-r0_x86-64.deb (--install):
 package architecture (x86-64) does not match system (amd64)
Errors were encountered while processing:
 alsa-utils_1.0.27.2-r0_x86-64.deb

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/classes/package_deb.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 6a8e080..b391e5c 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -430,6 +430,8 @@ python () {
 darch = d.getVar('DPKG_ARCH', True)
 if darch in ["x86", "i486", "i586", "i686", "pentium"]:
  d.setVar('DPKG_ARCH', 'i386')
+elif darch == "x86_64":
+ d.setVar('DPKG_ARCH', 'amd64')
 elif darch == "arm":
  d.setVar('DPKG_ARCH', 'armel')
 }
-- 
1.9.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] populate_sdk_deb: Fix meta-toolchain-sdk with amd64 ARCH

[Ricardo Ribalda Delgado]

Map SDK_ARCH x86_64 to DEB_SDK_ARCH amd64

Without this patch meta-toolchain-gmae fails to do_populate_sdk

| The following packages have unmet dependencies:
|  nativesdk-packagegroup-sdk-host : Depends: nativesdk-autoconf but it
is not installable
|Depends: nativesdk-libtool but it
is not installable
|Depends: nativesdk-shadow but it is
not installable
|Depends: nativesdk-unfs-server but
it is not installable
|Depends: nativesdk-makedevs but it
is not installable
|Depends: nativesdk-automake but it
is not installable
|Depends: nativesdk-qemu but it is
not installable
|Depends: nativesdk-pkgconfig but it
is not installable
|Depends: nativesdk-pseudo but it is
not installable
|Depends: nativesdk-qemu-helper but
it is not installable
|Depends: nativesdk-opkg but it is
not installable
|  packagegroup-cross-canadian-qt5022 : Depends:
gdb-cross-canadian-x86-64 but it is not installable
|   Depends:
binutils-cross-canadian-x86-64 but it is not installable
|   Depends:
gcc-cross-canadian-x86-64 but it is not installable
|   Depends: meta-environment-qt5022
but it is not installable

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/classes/populate_sdk_deb.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/populate_sdk_deb.bbclass 
b/meta/classes/populate_sdk_deb.bbclass
index ebb842b..aa2b6eb 100644
--- a/meta/classes/populate_sdk_deb.bbclass
+++ b/meta/classes/populate_sdk_deb.bbclass
@@ -6,6 +6,9 @@ DEB_SDK_ARCH = "${@[d.getVar('SDK_ARCH', True), "i386"]\
 [d.getVar('SDK_ARCH', True) in \
 ["x86", "i486", "i586", "i686", "pentium"]]}"
 
+DEB_SDK_ARCH = "${@[d.getVar('SDK_ARCH', True), "amd64"]\
+[d.getVar('SDK_ARCH', True) == "x86_64"]}"
+
 do_populate_sdk[lockfiles] += "${DEPLOY_DIR_DEB}/deb.lock"
 
 populate_sdk_post_deb () {
-- 
1.9.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] apt-native: Install apt-ftparchive

[Ricardo Ribalda Delgado]

apt-ftparchive is needed to create a Release file compatible with
SecureApt.

It is also a more efficient replacement of dpkg-scanpackages.

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-devtools/apt/apt-native.inc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-devtools/apt/apt-native.inc 
b/meta/recipes-devtools/apt/apt-native.inc
index 294ca94..36035b0 100644
--- a/meta/recipes-devtools/apt/apt-native.inc
+++ b/meta/recipes-devtools/apt/apt-native.inc
@@ -39,6 +39,7 @@ do_install_base () {
install -m 0755 bin/apt-cache ${D}${bindir}/
install -m 0755 bin/apt-sortpkgs ${D}${bindir}/
install -m 0755 bin/apt-extracttemplates ${D}${bindir}/
+   install -m 0755 bin/apt-ftparchive ${D}${bindir}/
 
eval `cat environment.mak | grep ^GLIBC_VER | sed -e's, = ,=,'`
eval `cat environment.mak | grep ^LIBSTDCPP_VER | sed -e's, = ,=,'`
-- 
1.9.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] package_deb: Use apt-ftparchive

[Ricardo Ribalda Delgado]

Use apt-ftparchive to create a Release file compatible with SecureApt.

apt-ftparchive is not compatible with libpseudo. The calls to ftw()
returns the path in absolute format instead of relative. This produces
wrong Packages and Release files.

ie:
MD5Sum:
d20227a958f6870137ce0e41b7b84307 1453
/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/all/Release

This is why it is called with PSEUDO_UNLOAD.

apt-ftparchive is also a more efficient replacement of dpkg-scanpackages:

root@neopili:~/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/bobcat
_64# time PSEUDO_UNLOAD=1 apt-ftparchive packages . >/tmp/kkk
real 0m26.873s
user 0m20.968s
sys 0m1.212s

root@neopili:~/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/bobcat
_64# time PSEUDO_UNLOAD=1 dpkg-scanpackages . >/tmp/kkk
dpkg-scanpackages: info: Wrote 6022 entries to output Packages file.
real 0m59.721s
user 0m16.668s
sys 0m11.164s

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/classes/package_deb.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index b391e5c..2ac1aed 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -37,8 +37,10 @@ package_update_index_deb () {
continue;
fi
cd ${DEPLOY_DIR_DEB}/$arch
-   dpkg-scanpackages . | gzip > Packages.gz
+   PSEUDO_UNLOAD=1 apt-ftparchive packages . > Packages
+   gzip Packages -c > Packages.gz
echo "Label: $arch" > Release
+   PSEUDO_UNLOAD=1 apt-ftparchive release . >> Release
found=1
done
if [ "$found" != "1" ]; then
-- 
1.9.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] package_deb: Remove version from Provides Field

[Ricardo Ribalda Delgado]

According to Debian Policy Manual. Chapter 7.1

All of the fields except for Provides may restrict their applicability
to particular versions of each named package. This is done in
parentheses after each individual package name; the parentheses should
contain a relation from the list below followed by a version number, in
the format described in Version, Section 5.6.12.

This Fixes the following warning:

root@qt5022:~# apt-get update

...

Reading package lists... Done
W: Ignoring Provides line with DepCompareOp for package
pkgconfig__pkg-config__
W: You may want to run apt-get update to correct these problems

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/classes/package_deb.bbclass | 8 
 1 file changed, 8 insertions(+)

diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 2d2f3f1..905af24 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -322,6 +322,13 @@ python do_package_deb () {
 
 mapping_rename_hook(localdata)
 
+def debian_cmp_purgeversion(var):
+# dpkg does not allow versions on Provides.
+# From Debian Policy Manual, Chapter 7:
+# All of the fields except for Provides may restrict their 
applicability to particular versions of each named package.
+for key in var:
+var[key] = []
+
 def debian_cmp_remap(var):
 # dpkg does not allow for '(' or ')' in a dependency name
 # replace these instances with '__' and '__'
@@ -359,6 +366,7 @@ python do_package_deb () {
 debian_cmp_remap(rsuggests)
 rprovides = 
bb.utils.explode_dep_versions2(localdata.getVar("RPROVIDES", True) or "")
 debian_cmp_remap(rprovides)
+debian_cmp_purgeversion(rprovides)
 rreplaces = 
bb.utils.explode_dep_versions2(localdata.getVar("RREPLACES", True) or "")
 debian_cmp_remap(rreplaces)
 rconflicts = 
bb.utils.explode_dep_versions2(localdata.getVar("RCONFLICTS", True) or "")
-- 
1.9.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/2] package_deb: Use apt-ftparchive

[Ricardo Ribalda Delgado]

Hello Richard.

I have tested it with dora. I can rebase it.

Shall I rebase it over git://git.yoctoproject.org/poky or over
git://git.openembedded.org/openembedded-core

Thanks!

On Mon, Mar 17, 2014 at 12:01 AM, Richard Purdie
 wrote:
> On Thu, 2014-03-13 at 14:27 +0100, Ricardo Ribalda Delgado wrote:
>> Use apt-ftparchive to create a Release file compatible with SecureApt.
>>
>> apt-ftparchive is not compatible with libpseudo. The calls to ftw()
>> returns the path in absolute format instead of relative. This produces
>> wrong Packages and Release files.
>>
>> ie:
>> MD5Sum:
>> d20227a958f6870137ce0e41b7b84307 1453
>> /home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/all/Release
>>
>> This is why it is called with PSEUDO_UNLOAD.
>>
>> apt-ftparchive is also a more efficient replacement of dpkg-scanpackages:
>>
>> root@neopili:~/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/bobcat
>> _64# time PSEUDO_UNLOAD=1 apt-ftparchive packages . >/tmp/kkk
>> real 0m26.873s
>> user 0m20.968s
>> sys 0m1.212s
>>
>> root@neopili:~/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/bobcat
>> _64# time PSEUDO_UNLOAD=1 dpkg-scanpackages . >/tmp/kkk
>> dpkg-scanpackages: info: Wrote 6022 entries to output Packages file.
>> real 0m59.721s
>> user 0m16.668s
>> sys 0m11.164s
>>
>> Signed-off-by: Ricardo Ribalda Delgado 
>> ---
>>  meta/classes/package_deb.bbclass | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> This doesn't apply against master? What was this tested against?
>
> Cheers,
>
> Richard
>



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2 v2] apt-native: Install apt-ftparchive

[Ricardo Ribalda Delgado]

apt-ftparchive is needed to create a Release file compatible with
SecureApt.

It is also a more efficient replacement of dpkg-scanpackages.

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-devtools/apt/apt-native.inc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-devtools/apt/apt-native.inc 
b/meta/recipes-devtools/apt/apt-native.inc
index 294ca94..36035b0 100644
--- a/meta/recipes-devtools/apt/apt-native.inc
+++ b/meta/recipes-devtools/apt/apt-native.inc
@@ -39,6 +39,7 @@ do_install_base () {
install -m 0755 bin/apt-cache ${D}${bindir}/
install -m 0755 bin/apt-sortpkgs ${D}${bindir}/
install -m 0755 bin/apt-extracttemplates ${D}${bindir}/
+   install -m 0755 bin/apt-ftparchive ${D}${bindir}/
 
eval `cat environment.mak | grep ^GLIBC_VER | sed -e's, = ,=,'`
eval `cat environment.mak | grep ^LIBSTDCPP_VER | sed -e's, = ,=,'`
-- 
1.9.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2 v2] package_manager: Use apt-ftparchive for deb packages

[Ricardo Ribalda Delgado]

Use apt-ftparchive to create a Release file compatible with SecureApt.

apt-ftparchive is also a more efficient replacement of
dpkg-scanpackages:

root@neopili:~/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/bobcat
_64# time PSEUDO_UNLOAD=1 apt-ftparchive packages . >/tmp/kkk
real 0m26.873s
user 0m20.968s
sys 0m1.212s

root@neopili:~/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/bobcat
_64# time PSEUDO_UNLOAD=1 dpkg-scanpackages . >/tmp/kkk
dpkg-scanpackages: info: Wrote 6022 entries to output Packages file.
real 0m59.721s
user 0m16.668s
sys 0m11.164s

apt-ftparchive is not compatible with libpseudo. The calls to ftw()
returns the path in absolute format instead of relative. This
produces
wrong Packages and Release files.

ie:
MD5Sum:
d20227a958f6870137ce0e41b7b84307 1453
/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/all/Release

This is why it is called with PSEUDO_UNLOAD.

Signed-off-by: Ricardo Ribalda Delgado 
---

This v2 is a port of the previous patch (package_deb: Use apt-ftparchive) to 
master

This patch HAS NOT BEEN TESTED on master, just on dora. I do not have a working 
build
system for my platform with master.

 meta/lib/oe/package_manager.py | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 1279b50..ec96bb3 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -180,7 +180,7 @@ class DpkgIndexer(Indexer):
 if sdk_pkg_archs is not None:
 arch_list += sdk_pkg_archs.split()
 
-dpkg_scanpackages = bb.utils.which(os.getenv('PATH'), 
"dpkg-scanpackages")
+apt_ftparchive = bb.utils.which(os.getenv('PATH'), "apt-ftparchive")
 gzip = bb.utils.which(os.getenv('PATH'), "gzip")
 
 index_cmds = []
@@ -190,11 +190,17 @@ class DpkgIndexer(Indexer):
 if not os.path.isdir(arch_dir):
 continue
 
+index_cmds.append("cd %s; PSEUDO_UNLOAD=1 %s packages > Packages" %
+  (arch_dir, apt_ftparchive))
+
+index_cmds.append("cd %s; %s Packages -c > Packages.gz" %
+  (arch_dir, gzip))
+
 with open(os.path.join(arch_dir, "Release"), "w+") as release:
 release.write("Label: %s" % arch)
 
-index_cmds.append("cd %s; %s . | %s > Packages.gz" %
-  (arch_dir, dpkg_scanpackages, gzip))
+index_cmds.append("cd %s; PSEUDO_UNLOAD=1 %s release >> Release" %
+  (arch_dir, apt_ftparchive))
 
 deb_dirs_found = True
 
-- 
1.9.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/2] package_deb: Use apt-ftparchive

[Ricardo Ribalda Delgado]

Hello Robert.

I have just resend the patch. Is there any document describing this
conventions?


Thanks!

On Mon, Mar 17, 2014 at 9:30 AM, Robert Yang  wrote:
>
> Hi Ricardo,
>
>
> On 03/17/2014 03:59 PM, Ricardo Ribalda Delgado wrote:
>>
>> Hello Richard.
>>
>> I have tested it with dora. I can rebase it.
>>
>
> If it is for dora, then the Subject should in include [dora]
>
>
>> Shall I rebase it over git://git.yoctoproject.org/poky or over
>> git://git.openembedded.org/openembedded-core
>>
>
> I think that openembedded-core is preferred.
>
> // Robert
>
>
>> Thanks!
>>
>> On Mon, Mar 17, 2014 at 12:01 AM, Richard Purdie
>>  wrote:
>>>
>>> On Thu, 2014-03-13 at 14:27 +0100, Ricardo Ribalda Delgado wrote:
>>>>
>>>> Use apt-ftparchive to create a Release file compatible with SecureApt.
>>>>
>>>> apt-ftparchive is not compatible with libpseudo. The calls to ftw()
>>>> returns the path in absolute format instead of relative. This produces
>>>> wrong Packages and Release files.
>>>>
>>>> ie:
>>>> MD5Sum:
>>>> d20227a958f6870137ce0e41b7b84307 1453
>>>>
>>>> /home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/all/Release
>>>>
>>>> This is why it is called with PSEUDO_UNLOAD.
>>>>
>>>> apt-ftparchive is also a more efficient replacement of
>>>> dpkg-scanpackages:
>>>>
>>>>
>>>> root@neopili:~/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/bobcat
>>>> _64# time PSEUDO_UNLOAD=1 apt-ftparchive packages . >/tmp/kkk
>>>> real 0m26.873s
>>>> user 0m20.968s
>>>> sys 0m1.212s
>>>>
>>>>
>>>> root@neopili:~/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/deploy/deb/bobcat
>>>> _64# time PSEUDO_UNLOAD=1 dpkg-scanpackages . >/tmp/kkk
>>>> dpkg-scanpackages: info: Wrote 6022 entries to output Packages file.
>>>> real 0m59.721s
>>>> user 0m16.668s
>>>> sys 0m11.164s
>>>>
>>>> Signed-off-by: Ricardo Ribalda Delgado 
>>>> ---
>>>>   meta/classes/package_deb.bbclass | 4 +++-
>>>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>>
>>> This doesn't apply against master? What was this tested against?
>>>
>>> Cheers,
>>>
>>> Richard
>>>
>>
>>
>>
>



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] package_deb: Remove version from Provides Field

[Ricardo Ribalda Delgado]

Ping

On Thu, Mar 13, 2014 at 4:31 PM, Ricardo Ribalda Delgado
 wrote:
> According to Debian Policy Manual. Chapter 7.1
>
> All of the fields except for Provides may restrict their applicability
> to particular versions of each named package. This is done in
> parentheses after each individual package name; the parentheses should
> contain a relation from the list below followed by a version number, in
> the format described in Version, Section 5.6.12.
>
> This Fixes the following warning:
>
> root@qt5022:~# apt-get update
>
> ...
>
> Reading package lists... Done
> W: Ignoring Provides line with DepCompareOp for package
> pkgconfig__pkg-config__
> W: You may want to run apt-get update to correct these problems
>
> Signed-off-by: Ricardo Ribalda Delgado 
> ---
>  meta/classes/package_deb.bbclass | 8 
>  1 file changed, 8 insertions(+)
>
> diff --git a/meta/classes/package_deb.bbclass 
> b/meta/classes/package_deb.bbclass
> index 2d2f3f1..905af24 100644
> --- a/meta/classes/package_deb.bbclass
> +++ b/meta/classes/package_deb.bbclass
> @@ -322,6 +322,13 @@ python do_package_deb () {
>
>  mapping_rename_hook(localdata)
>
> +def debian_cmp_purgeversion(var):
> +# dpkg does not allow versions on Provides.
> +# From Debian Policy Manual, Chapter 7:
> +# All of the fields except for Provides may restrict their 
> applicability to particular versions of each named package.
> +for key in var:
> +var[key] = []
> +
>  def debian_cmp_remap(var):
>  # dpkg does not allow for '(' or ')' in a dependency name
>  # replace these instances with '__' and '__'
> @@ -359,6 +366,7 @@ python do_package_deb () {
>  debian_cmp_remap(rsuggests)
>  rprovides = 
> bb.utils.explode_dep_versions2(localdata.getVar("RPROVIDES", True) or "")
>  debian_cmp_remap(rprovides)
> +debian_cmp_purgeversion(rprovides)
>  rreplaces = 
> bb.utils.explode_dep_versions2(localdata.getVar("RREPLACES", True) or "")
>  debian_cmp_remap(rreplaces)
>  rconflicts = 
> bb.utils.explode_dep_versions2(localdata.getVar("RCONFLICTS", True) or "")
> --
> 1.9.0
>



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] package_deb: Remove version from Provides Field

[Ricardo Ribalda Delgado]

Hello Richard

It is ok to have Provides, but they should not have a version number.
If they have a version number a nasty warning appears (as you can see
on the patch). The provided patch just clears all the versions but
keeps the Provides

We could "fix" dpkg to support versioning, but I believe it is better
to just remove the version number from the Provides.

Cheers!

On Tue, Mar 25, 2014 at 9:50 AM, Richard Purdie
 wrote:
> On Thu, 2014-03-13 at 16:31 +0100, Ricardo Ribalda Delgado wrote:
>> According to Debian Policy Manual. Chapter 7.1
>>
>> All of the fields except for Provides may restrict their applicability
>> to particular versions of each named package. This is done in
>> parentheses after each individual package name; the parentheses should
>> contain a relation from the list below followed by a version number, in
>> the format described in Version, Section 5.6.12.
>>
>> This Fixes the following warning:
>>
>> root@qt5022:~# apt-get update
>>
>> ...
>>
>> Reading package lists... Done
>> W: Ignoring Provides line with DepCompareOp for package
>> pkgconfig__pkg-config__
>> W: You may want to run apt-get update to correct these problems
>>
>> Signed-off-by: Ricardo Ribalda Delgado 
>> ---
>>  meta/classes/package_deb.bbclass | 8 
>>  1 file changed, 8 insertions(+)
>>
>> diff --git a/meta/classes/package_deb.bbclass 
>> b/meta/classes/package_deb.bbclass
>> index 2d2f3f1..905af24 100644
>> --- a/meta/classes/package_deb.bbclass
>> +++ b/meta/classes/package_deb.bbclass
>> @@ -322,6 +322,13 @@ python do_package_deb () {
>>
>>  mapping_rename_hook(localdata)
>>
>> +def debian_cmp_purgeversion(var):
>> +# dpkg does not allow versions on Provides.
>> +# From Debian Policy Manual, Chapter 7:
>> +# All of the fields except for Provides may restrict their 
>> applicability to particular versions of each named package.
>> +for key in var:
>> +var[key] = []
>> +
> Shouldn't this actually make a version comparison, i.e. if 
> then emit the Provides, otherwise don't (looking at the package version
> for comparison)?
>
> I can imagine having multiple versions of a recipe, some with provides
> and some without, all using the same code.
>
> Cheers,
>
> Richard
>
>  def debian_cmp_remap(var):
>>  # dpkg does not allow for '(' or ')' in a dependency name
>>  # replace these instances with '__' and '__'
>> @@ -359,6 +366,7 @@ python do_package_deb () {
>>  debian_cmp_remap(rsuggests)
>>  rprovides = 
>> bb.utils.explode_dep_versions2(localdata.getVar("RPROVIDES", True) or "")
>>  debian_cmp_remap(rprovides)
>> +debian_cmp_purgeversion(rprovides)
>>  rreplaces = 
>> bb.utils.explode_dep_versions2(localdata.getVar("RREPLACES", True) or "")
>>  debian_cmp_remap(rreplaces)
>>  rconflicts = 
>> bb.utils.explode_dep_versions2(localdata.getVar("RCONFLICTS", True) or "")
>
>



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] package_deb: Remove version from Provides Field

[Ricardo Ribalda Delgado]

ok, but for the next release wont be a good idea to fix the warning in dpkg?

On Tue, Mar 25, 2014 at 10:19 AM, Richard Purdie
 wrote:
> On Tue, 2014-03-25 at 09:54 +0100, Ricardo Ribalda Delgado wrote:
>> Hello Richard
>>
>> It is ok to have Provides, but they should not have a version number.
>> If they have a version number a nasty warning appears (as you can see
>> on the patch). The provided patch just clears all the versions but
>> keeps the Provides
>>
>> We could "fix" dpkg to support versioning, but I believe it is better
>> to just remove the version number from the Provides.
>
> We can also do the version comparison in package_deb.bbclass itself. I
> believe there is some code somewhere which can understand the different
> constraints.
>
> Cheers,
>
> Richard
>
>> Cheers!
>>
>> On Tue, Mar 25, 2014 at 9:50 AM, Richard Purdie
>>  wrote:
>> > On Thu, 2014-03-13 at 16:31 +0100, Ricardo Ribalda Delgado wrote:
>> >> According to Debian Policy Manual. Chapter 7.1
>> >>
>> >> All of the fields except for Provides may restrict their applicability
>> >> to particular versions of each named package. This is done in
>> >> parentheses after each individual package name; the parentheses should
>> >> contain a relation from the list below followed by a version number, in
>> >> the format described in Version, Section 5.6.12.
>> >>
>> >> This Fixes the following warning:
>> >>
>> >> root@qt5022:~# apt-get update
>> >>
>> >> ...
>> >>
>> >> Reading package lists... Done
>> >> W: Ignoring Provides line with DepCompareOp for package
>> >> pkgconfig__pkg-config__
>> >> W: You may want to run apt-get update to correct these problems
>> >>
>> >> Signed-off-by: Ricardo Ribalda Delgado 
>> >> ---
>> >>  meta/classes/package_deb.bbclass | 8 
>> >>  1 file changed, 8 insertions(+)
>> >>
>> >> diff --git a/meta/classes/package_deb.bbclass 
>> >> b/meta/classes/package_deb.bbclass
>> >> index 2d2f3f1..905af24 100644
>> >> --- a/meta/classes/package_deb.bbclass
>> >> +++ b/meta/classes/package_deb.bbclass
>> >> @@ -322,6 +322,13 @@ python do_package_deb () {
>> >>
>> >>  mapping_rename_hook(localdata)
>> >>
>> >> +def debian_cmp_purgeversion(var):
>> >> +# dpkg does not allow versions on Provides.
>> >> +# From Debian Policy Manual, Chapter 7:
>> >> +# All of the fields except for Provides may restrict their 
>> >> applicability to particular versions of each named package.
>> >> +for key in var:
>> >> +var[key] = []
>> >> +
>> > Shouldn't this actually make a version comparison, i.e. if 
>> > then emit the Provides, otherwise don't (looking at the package version
>> > for comparison)?
>> >
>> > I can imagine having multiple versions of a recipe, some with provides
>> > and some without, all using the same code.
>> >
>> > Cheers,
>> >
>> > Richard
>> >
>> >  def debian_cmp_remap(var):
>> >>  # dpkg does not allow for '(' or ')' in a dependency name
>> >>  # replace these instances with '__' and '__'
>> >> @@ -359,6 +366,7 @@ python do_package_deb () {
>> >>  debian_cmp_remap(rsuggests)
>> >>  rprovides = 
>> >> bb.utils.explode_dep_versions2(localdata.getVar("RPROVIDES", True) or "")
>> >>  debian_cmp_remap(rprovides)
>> >> +debian_cmp_purgeversion(rprovides)
>> >>  rreplaces = 
>> >> bb.utils.explode_dep_versions2(localdata.getVar("RREPLACES", True) or "")
>> >>  debian_cmp_remap(rreplaces)
>> >>  rconflicts = 
>> >> bb.utils.explode_dep_versions2(localdata.getVar("RCONFLICTS", True) or "")
>> >
>> >
>>
>>
>>
>
>



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] package_manager: Fix apt-ftparchive index creation problems

[Ricardo Ribalda Delgado]

Acked-by: Ricardo Ribalda 

Hello

My fault sorry. I dont have a running system on master. I tried to
port the patch that I tested on dora.

Thanks Richard


On Wed, Mar 26, 2014 at 12:40 PM, Richard Purdie
 wrote:
> a) There were missing parameters to the release and package commands (".")
>
> b) The commands need to be executed as one block since they build upon each 
> other
>
> Signed-off-by: Richard Purdie 
> ---
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index 692528d..824af00 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -192,17 +192,16 @@ class DpkgIndexer(Indexer):
>  if not os.path.isdir(arch_dir):
>  continue
>
> -index_cmds.append("cd %s; PSEUDO_UNLOAD=1 %s packages > 
> Packages" %
> -  (arch_dir, apt_ftparchive))
> +cmd = "cd %s; PSEUDO_UNLOAD=1 %s packages . > Packages;" % 
> (arch_dir, apt_ftparchive)
>
> -index_cmds.append("cd %s; %s Packages -c > Packages.gz" %
> -  (arch_dir, gzip))
> +cmd += "%s -fc Packages > Packages.gz;" % gzip
>
>  with open(os.path.join(arch_dir, "Release"), "w+") as release:
> -release.write("Label: %s" % arch)
> +release.write("Label: %s\n" % arch)
>
> -index_cmds.append("cd %s; PSEUDO_UNLOAD=1 %s release >> Release" 
> %
> -  (arch_dir, apt_ftparchive))
> +cmd += "PSEUDO_UNLOAD=1 %s release . >> Release" % apt_ftparchive
> +
> +index_cmds.append(cmd)
>
>      deb_dirs_found = True
>
>
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] Opencv: Add OpenCV 3.0

[Ricardo Ribalda Delgado]

-Support for new PACKAGECONFIGS
-Merge with opencv-samples

Since it is not backward compatible with 2.x and cannot be installed in
parallel it has a DEFAULT_PREFERENCE of -1.

Signed-off-by: Ricardo Ribalda Delgado 
---
v3: Changes requested by Martin Jansa 
-Select java dependencies via packageconfig

 meta-oe/recipes-support/opencv/opencv_3.0.bb | 128 +++
 1 file changed, 128 insertions(+)
 create mode 100644 meta-oe/recipes-support/opencv/opencv_3.0.bb

diff --git a/meta-oe/recipes-support/opencv/opencv_3.0.bb 
b/meta-oe/recipes-support/opencv/opencv_3.0.bb
new file mode 100644
index ..e929f82c8762
--- /dev/null
+++ b/meta-oe/recipes-support/opencv/opencv_3.0.bb
@@ -0,0 +1,128 @@
+SUMMARY = "Opencv : The Open Computer Vision Library"
+HOMEPAGE = "http://opencv.willowgarage.com/wiki/";
+SECTION = "libs"
+
+LICENSE = "BSD"
+LIC_FILES_CHKSUM = 
"file://include/opencv2/opencv.hpp;endline=41;md5=6d690d8488a6fca7a2c192932466bb14"
+
+ARM_INSTRUCTION_SET = "arm"
+
+DEFAULT_PREFERENCE = "-1"
+
+DEPENDS = "python-numpy libtool swig swig-native python bzip2 zlib glib-2.0"
+
+SRCREV_opencv = "424c2bddb39dae97dc4639a24eaa0e0c8fbb8e69"
+SRCREV_contrib = "844c30e8b2f2f4b34b96a169fafe9beea3c45e87"
+SRCREV_FORMAT = "opencv"
+SRC_URI = "git://github.com/Itseez/opencv.git;name=opencv \
+  
git://github.com/Itseez/opencv_contrib.git;destsuffix=contrib;name=contrib"
+
+PV = "3.0+git${SRCPV}"
+
+S = "${WORKDIR}/git"
+
+EXTRA_OECMAKE = 
"-DPYTHON2_NUMPY_INCLUDE_DIRS:PATH=${STAGING_LIBDIR}/${PYTHON_DIR}/site-packages/numpy/core/include
 \
+-DOPENCV_EXTRA_MODULES_PATH=${WORKDIR}/contrib/modules \
+ -DWITH_1394=OFF \
+ -DCMAKE_SKIP_RPATH=ON \
+ ${@bb.utils.contains("TARGET_CC_ARCH", "-msse3", 
"-DENABLE_SSE=1 -DENABLE_SSE2=1 -DENABLE_SSE3=1 -DENABLE_SSSE3=1", "", d)} \
+ ${@base_conditional("libdir", "/usr/lib64", 
"-DLIB_SUFFIX=64", "", d)} \
+ ${@base_conditional("libdir", "/usr/lib32", 
"-DLIB_SUFFIX=32", "", d)} \
+"
+
+PACKAGECONFIG ??= "eigen jpeg libav png tiff v4l gstreamer samples tbb\
+   ${@bb.utils.contains("DISTRO_FEATURES", "x11", "gtk", "", 
d)}"
+PACKAGECONFIG[eigen] = "-DWITH_EIGEN=ON,-DWITH_EIGEN=OFF,libeigen,"
+PACKAGECONFIG[gtk] = "-DWITH_GTK=ON,-DWITH_GTK=OFF,gtk+3,"
+PACKAGECONFIG[jpeg] = "-DWITH_JPEG=ON,-DWITH_JPEG=OFF,jpeg,"
+PACKAGECONFIG[libav] = "-DWITH_FFMPEG=ON,-DWITH_FFMPEG=OFF,libav,"
+PACKAGECONFIG[png] = "-DWITH_PNG=ON,-DWITH_PNG=OFF,libpng,"
+PACKAGECONFIG[tiff] = "-DWITH_TIFF=ON,-DWITH_TIFF=OFF,tiff,"
+PACKAGECONFIG[v4l] = "-DWITH_V4L=ON,-DWITH_V4L=OFF,v4l-utils,"
+PACKAGECONFIG[jasper] = "-DWITH_JASPER=ON,-DWITH_JASPER=OFF,jasper,"
+PACKAGECONFIG[amdblas] = 
"-DWITH_OPENCLAMDBLAS=ON,-DWITH_OPENCLAMDBLAS=OFF,libclamdblas,"
+PACKAGECONFIG[amdfft] = 
"-DWITH_OPENCLAMDFFT=ON,-DWITH_OPENCLAMDFFT=OFF,libclamdfft,"
+PACKAGECONFIG[opencl] = "-DWITH_OPENCL=ON,-DWITH_OPENCL=OFF,opencl-headers,"
+PACKAGECONFIG[gstreamer] = 
"-DWITH_GSTREAMER=ON,-DWITH_GSTREAMER=OFF,gstreamer1.0 
gstreamer1.0-plugins-base,"
+PACKAGECONFIG[oracle-java] = "-DJAVA_INCLUDE_PATH=${JAVA_HOME}/include 
-DJAVA_INCLUDE_PATH2=${JAVA_HOME}/include/linux 
-DJAVA_AWT_INCLUDE_PATH=${JAVA_HOME}/include 
-DJAVA_AWT_LIBRARY=${JAVA_HOME}/lib/amd64/libjawt.so 
-DJAVA_JVM_LIBRARY=${JAVA_HOME}/lib/amd64/server/libjvm.so,,ant-native 
oracle-jse-jdk oracle-jse-jdk-native,"
+PACKAGECONFIG[samples] = "-DBUILD_EXAMPLES=ON 
-DINSTALL_PYTHON_EXAMPLES=ON,-DBUILD_EXAMPLES=OFF,,"
+PACKAGECONFIG[tbb] = "-DWITH_TBB=ON,-DWITH_TBB=OFF,tbb,"
+
+inherit distutils-base pkgconfig cmake
+
+export BUILD_SYS
+export HOST_SYS
+export PYTHON_CSPEC="-I${STAGING_INCDIR}/${PYTHON_DIR}"
+export PYTHON="${STAGING_BINDIR_NATIVE}/python"
+export JAVA_HOME="${STAGING_DIR_NATIVE}/usr/bin/java"
+export ANT_DIR="${STAGING_DIR_NATIVE}/usr/share/ant/"
+
+TARGET_CC_ARCH += "-I${S}/include "
+
+PACKAGES += "${PN}-java-dbg ${PN}-java ${PN}-samples-dbg ${PN}-samples 
${PN}-apps python-opencv"
+
+python populate_packages_prepend () {
+cv_libdir = d.expand('${libdir}')
+cv_libdir_dbg = d.expand('${libdir}/.debug')
+do_split_packages(d, cv_libdir, '^lib(.*)\.so$', 'lib%s-dev', 'OpenCV %s 
development package', extra_depends='${PN}-dev', allow_links=True)
+do_split_packages(d, cv_libdir, '^lib(.*)\.la$', 'lib

Re: [OE-core] [PATCH] Opencv: Add OpenCV 3.0

[Ricardo Ribalda Delgado]

You are right :)

I repack and send

On Wed, Aug 19, 2015 at 4:00 PM, Nathan Lynch  wrote:
> On 08/19/2015 07:18 AM, Ricardo Ribalda Delgado wrote:
>> --- /dev/null
>> +++ b/meta-oe/recipes-support/opencv/opencv_3.0.bb
>> @@ -0,0 +1,128 @@
>> +SUMMARY = "Opencv : The Open Computer Vision Library"
>> +HOMEPAGE = "http://opencv.willowgarage.com/wiki/";
>
> Is this URL outdated?  It times out for me.
>



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v4] Opencv: Add OpenCV 3.0

[Ricardo Ribalda Delgado]

-Support for new PACKAGECONFIGS
-Merge with opencv-samples

Since it is not backward compatible with 2.x and cannot be installed in
parallel it has a DEFAULT_PREFERENCE of -1.

Signed-off-by: Ricardo Ribalda Delgado 
---
v4: Fix suggested by Nathan Lynch 
  Wrong url on HOMEPAGE

 meta-oe/recipes-support/opencv/opencv_3.0.bb | 128 +++
 1 file changed, 128 insertions(+)
 create mode 100644 meta-oe/recipes-support/opencv/opencv_3.0.bb

diff --git a/meta-oe/recipes-support/opencv/opencv_3.0.bb 
b/meta-oe/recipes-support/opencv/opencv_3.0.bb
new file mode 100644
index ..36ab6271edd6
--- /dev/null
+++ b/meta-oe/recipes-support/opencv/opencv_3.0.bb
@@ -0,0 +1,128 @@
+SUMMARY = "Opencv : The Open Computer Vision Library"
+HOMEPAGE = "http://opencv.org/";
+SECTION = "libs"
+
+LICENSE = "BSD"
+LIC_FILES_CHKSUM = 
"file://include/opencv2/opencv.hpp;endline=41;md5=6d690d8488a6fca7a2c192932466bb14"
+
+ARM_INSTRUCTION_SET = "arm"
+
+DEFAULT_PREFERENCE = "-1"
+
+DEPENDS = "python-numpy libtool swig swig-native python bzip2 zlib glib-2.0"
+
+SRCREV_opencv = "424c2bddb39dae97dc4639a24eaa0e0c8fbb8e69"
+SRCREV_contrib = "844c30e8b2f2f4b34b96a169fafe9beea3c45e87"
+SRCREV_FORMAT = "opencv"
+SRC_URI = "git://github.com/Itseez/opencv.git;name=opencv \
+  
git://github.com/Itseez/opencv_contrib.git;destsuffix=contrib;name=contrib"
+
+PV = "3.0+git${SRCPV}"
+
+S = "${WORKDIR}/git"
+
+EXTRA_OECMAKE = 
"-DPYTHON2_NUMPY_INCLUDE_DIRS:PATH=${STAGING_LIBDIR}/${PYTHON_DIR}/site-packages/numpy/core/include
 \
+-DOPENCV_EXTRA_MODULES_PATH=${WORKDIR}/contrib/modules \
+ -DWITH_1394=OFF \
+ -DCMAKE_SKIP_RPATH=ON \
+ ${@bb.utils.contains("TARGET_CC_ARCH", "-msse3", 
"-DENABLE_SSE=1 -DENABLE_SSE2=1 -DENABLE_SSE3=1 -DENABLE_SSSE3=1", "", d)} \
+ ${@base_conditional("libdir", "/usr/lib64", 
"-DLIB_SUFFIX=64", "", d)} \
+ ${@base_conditional("libdir", "/usr/lib32", 
"-DLIB_SUFFIX=32", "", d)} \
+"
+
+PACKAGECONFIG ??= "eigen jpeg libav png tiff v4l gstreamer samples tbb\
+   ${@bb.utils.contains("DISTRO_FEATURES", "x11", "gtk", "", 
d)}"
+PACKAGECONFIG[eigen] = "-DWITH_EIGEN=ON,-DWITH_EIGEN=OFF,libeigen,"
+PACKAGECONFIG[gtk] = "-DWITH_GTK=ON,-DWITH_GTK=OFF,gtk+3,"
+PACKAGECONFIG[jpeg] = "-DWITH_JPEG=ON,-DWITH_JPEG=OFF,jpeg,"
+PACKAGECONFIG[libav] = "-DWITH_FFMPEG=ON,-DWITH_FFMPEG=OFF,libav,"
+PACKAGECONFIG[png] = "-DWITH_PNG=ON,-DWITH_PNG=OFF,libpng,"
+PACKAGECONFIG[tiff] = "-DWITH_TIFF=ON,-DWITH_TIFF=OFF,tiff,"
+PACKAGECONFIG[v4l] = "-DWITH_V4L=ON,-DWITH_V4L=OFF,v4l-utils,"
+PACKAGECONFIG[jasper] = "-DWITH_JASPER=ON,-DWITH_JASPER=OFF,jasper,"
+PACKAGECONFIG[amdblas] = 
"-DWITH_OPENCLAMDBLAS=ON,-DWITH_OPENCLAMDBLAS=OFF,libclamdblas,"
+PACKAGECONFIG[amdfft] = 
"-DWITH_OPENCLAMDFFT=ON,-DWITH_OPENCLAMDFFT=OFF,libclamdfft,"
+PACKAGECONFIG[opencl] = "-DWITH_OPENCL=ON,-DWITH_OPENCL=OFF,opencl-headers,"
+PACKAGECONFIG[gstreamer] = 
"-DWITH_GSTREAMER=ON,-DWITH_GSTREAMER=OFF,gstreamer1.0 
gstreamer1.0-plugins-base,"
+PACKAGECONFIG[oracle-java] = "-DJAVA_INCLUDE_PATH=${JAVA_HOME}/include 
-DJAVA_INCLUDE_PATH2=${JAVA_HOME}/include/linux 
-DJAVA_AWT_INCLUDE_PATH=${JAVA_HOME}/include 
-DJAVA_AWT_LIBRARY=${JAVA_HOME}/lib/amd64/libjawt.so 
-DJAVA_JVM_LIBRARY=${JAVA_HOME}/lib/amd64/server/libjvm.so,,ant-native 
oracle-jse-jdk oracle-jse-jdk-native,"
+PACKAGECONFIG[samples] = "-DBUILD_EXAMPLES=ON 
-DINSTALL_PYTHON_EXAMPLES=ON,-DBUILD_EXAMPLES=OFF,,"
+PACKAGECONFIG[tbb] = "-DWITH_TBB=ON,-DWITH_TBB=OFF,tbb,"
+
+inherit distutils-base pkgconfig cmake
+
+export BUILD_SYS
+export HOST_SYS
+export PYTHON_CSPEC="-I${STAGING_INCDIR}/${PYTHON_DIR}"
+export PYTHON="${STAGING_BINDIR_NATIVE}/python"
+export JAVA_HOME="${STAGING_DIR_NATIVE}/usr/bin/java"
+export ANT_DIR="${STAGING_DIR_NATIVE}/usr/share/ant/"
+
+TARGET_CC_ARCH += "-I${S}/include "
+
+PACKAGES += "${PN}-java-dbg ${PN}-java ${PN}-samples-dbg ${PN}-samples 
${PN}-apps python-opencv"
+
+python populate_packages_prepend () {
+cv_libdir = d.expand('${libdir}')
+cv_libdir_dbg = d.expand('${libdir}/.debug')
+do_split_packages(d, cv_libdir, '^lib(.*)\.so$', 'lib%s-dev', 'OpenCV %s 
development package', extra_depends='${PN}-dev', allow_links=True)
+do_split_packages(d, cv_libdir, '^lib(.*)\.la$', 'lib%s-dev', 'OpenCV %s 
developmen

[OE-core] [PATCH 1/2] python3: Fix python-distutils variables

[Ricardo Ribalda Delgado]

python-distutils uses the information from this file for setting the
system variables.

Whithout this patch:

root@qt5022:~# pip install pyzmq

...

x86_64-poky-linux-gcc -m64 -march=btver1 -mtune=btver1
--sysroot=/var/lib/jenkins/jobs/qt5022-cesium/workspace/build/tmp/sysroots/qt5022
-fno-strict-aliasing -O2 -pipe -g -feliminate-unused-debug-types
-DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -DNATIVE_LITTLE_ENDIAN=1
-Ibundled/libsodium/src/libsodium/include
-Ibundled/libsodium/src/libsodium/include/sodium
-I/usr/include/python2.7 -c buildutils/initlibsodium.c -o
build/temp.linux-x86_64-2.7/buildutils/initlibsodium.o
In file included from /usr/include/python2.7/Python.h:8:0,
 from buildutils/initlibsodium.c:10:
/usr/include/python2.7/pyconfig.h:24:27: fatal error: bits/wordsize.h: No 
such
file or directory
#include 
root@qt5022:~# ls /usr/include/bits/wordsize.h
/usr/include/bits/wordsize.h

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-devtools/python/python3_3.4.3.bb | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/python/python3_3.4.3.bb 
b/meta/recipes-devtools/python/python3_3.4.3.bb
index 6c2e21d6a9c2..f61d278d79d4 100644
--- a/meta/recipes-devtools/python/python3_3.4.3.bb
+++ b/meta/recipes-devtools/python/python3_3.4.3.bb
@@ -184,8 +184,10 @@ PACKAGE_PREPROCESS_FUNCS += "py_package_preprocess"
 py_package_preprocess () {
# copy back the old Makefile to fix target package
install -m 0644 ${B}/Makefile.orig 
${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
-   # Remove references to buildmachine paths in target Makefile
-   sed -i -e 's:--sysroot=${STAGING_DIR_TARGET}::g' -e 
s:'--with-libtool-sysroot=${STAGING_DIR_TARGET}'::g 
${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
+   # Remove references to buildmachine paths in target Makefile and 
_sysconfigdata
+   sed -i -e 's:--sysroot=${STAGING_DIR_TARGET}::g' -e 
s:'--with-libtool-sysroot=${STAGING_DIR_TARGET}'::g \
+   ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile \
+   ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.py
 }
 
 require python-${PYTHON_MAJMIN}-manifest.inc
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] python: Fix python-distutils variables

[Ricardo Ribalda Delgado]

python-distutils uses the information from this file for setting the
system variables.

Whithout this patch:

root@qt5022:~# pip install pyzmq

...

x86_64-poky-linux-gcc -m64 -march=btver1 -mtune=btver1
--sysroot=/var/lib/jenkins/jobs/qt5022-cesium/workspace/build/tmp/sysroots/qt5022
-fno-strict-aliasing -O2 -pipe -g -feliminate-unused-debug-types
-DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -DNATIVE_LITTLE_ENDIAN=1
-Ibundled/libsodium/src/libsodium/include
-Ibundled/libsodium/src/libsodium/include/sodium
-I/usr/include/python2.7 -c buildutils/initlibsodium.c -o
build/temp.linux-x86_64-2.7/buildutils/initlibsodium.o
In file included from /usr/include/python2.7/Python.h:8:0,
 from buildutils/initlibsodium.c:10:
/usr/include/python2.7/pyconfig.h:24:27: fatal error: bits/wordsize.h: No 
such
file or directory
#include 
root@qt5022:~# ls /usr/include/bits/wordsize.h
/usr/include/bits/wordsize.h

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-devtools/python/python_2.7.9.bb | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/python/python_2.7.9.bb 
b/meta/recipes-devtools/python/python_2.7.9.bb
index 803a1fc7497b..ae455771653e 100644
--- a/meta/recipes-devtools/python/python_2.7.9.bb
+++ b/meta/recipes-devtools/python/python_2.7.9.bb
@@ -137,8 +137,10 @@ py_package_preprocess () {
# copy back the old Makefile to fix target package
install -m 0644 ${B}/Makefile.orig 
${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
 
-   # Remove references to buildmachine paths in target Makefile
-   sed -i -e 's:--sysroot=${STAGING_DIR_TARGET}::g' -e 
s:'--with-libtool-sysroot=${STAGING_DIR_TARGET}'::g 
${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
+   # Remove references to buildmachine paths in target Makefile and 
_sysconfigdata
+   sed -i -e 's:--sysroot=${STAGING_DIR_TARGET}::g' -e 
s:'--with-libtool-sysroot=${STAGING_DIR_TARGET}'::g \
+   ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile \
+   ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.py
 }
 
 require python-${PYTHON_MAJMIN}-manifest.inc
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] ?Regression? Removed linker symbolic link

[Ricardo Ribalda Delgado]

Hello

On my distribution I have a couple of binary packages (java and fglrx)
that expect the dynamic linker on /lib64/ld-linux-x86-64.so.2

Therefore I have added a dummy package that justs creates a link on
/lib/ld-linux-x86-64.so.2 and make both packages depend on this.

This was working ok, but since I updated to  poky 1.6.1 this has
stopped working :( The link seems to be removed from my rootfs.

It is removed when the function _run_ldconfig on rootfs.py is executed.

After some debugging I found that the cause  is this patch:


Date:   Mon May 5 10:30:33 2014 +0800

ldconfig-native: also default to lib32 and lib64 directories

make ldconfig default to /lib+/usr/lib, /lib32+/usr/lib32 and
/lib64+/usr/lib64 on bi-ABI architectures.

(From OE-Core rev: c90bb98d4ac14562ef4882691daed3aaa9d08504)

(From OE-Core rev: 44202d78102fa58ba99040a79d5a61c44033f57d)

Signed-off-by: Ming Liu 
Signed-off-by: Saul Wold 
Signed-off-by: Richard Purdie 

Is there something I can do to avoid the removal of this file?
Any other better method to do this?

Thanks and best regards!



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] package_manager: Fix populate_sdk for dpkg packages

[Ricardo Ribalda Delgado]

Without this patch:

ERROR: Unable to install packages. Command 
'/var/lib/jenkins/jobs/qt5022-cesium/workspace/build/tmp/sysroots/x86_64-linux/usr/bin/apt-get
  install --force-yes --allow-unauthenticated nativesdk-packagegroup-sdk-host 
packagegroup-cross-canadian-qt5022' returned 100:
Reading package lists...
Building dependency tree...
Reading state information...
W: Unable to read 
/var/lib/jenkins/jobs/qt5022-cesium/workspace/build/tmp/work/bobcat_64-poky-linux/meta-toolchain/1.0-r7/apt-sdk/preferences.d/
 - DirectoryExists (2: No such file or directory)
E: Unable to locate package nativesdk-packagegroup-sdk-host
E: Unable to locate package packagegroup-cross-canadian-qt5022

ERROR: Function failed: do_populate_sdk
ERROR: Logfile of failure stored in: 
/var/lib/jenkins/jobs/qt5022-cesium/workspace/build/tmp/work/bobcat_64-poky-linux/meta-toolchain/1.0-r7/temp/log.do_populate_sdk.4926
NOTE: recipe meta-toolchain-1.0-r7: task do_populate_sdk: Failed
ERROR: Task 7 
(/var/lib/jenkins/jobs/qt5022-cesium/workspace/repo/yocto/meta/recipes-core/meta/meta-toolchain.bb,
 do_populate_sdk) failed with exit code '1'
Signed-off-by: Ricardo Ribalda Delgado 
---

Tested on yocto 1.7

Only build test, not tested on target.

 meta/lib/oe/package_manager.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 27fdf26..8828bce 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -188,7 +188,7 @@ class DpkgIndexer(Indexer):
 release.write("Label: %s\n" % arch)
 
 cmd += "PSEUDO_UNLOAD=1 %s release . >> Release" % apt_ftparchive
-
+
 index_cmds.append(cmd)
 
 deb_dirs_found = True
@@ -1467,6 +1467,7 @@ class DpkgPM(PackageManager):
 def __init__(self, d, target_rootfs, archs, base_archs, apt_conf_dir=None):
 super(DpkgPM, self).__init__(d)
 self.target_rootfs = target_rootfs
+self.pkg_archs = archs
 self.deploy_dir = self.d.getVar('DEPLOY_DIR_DEB', True)
 if apt_conf_dir is None:
 self.apt_conf_dir = self.d.expand("${APTCONF_TARGET}/apt")
@@ -1662,7 +1663,7 @@ class DpkgPM(PackageManager):
 bb.utils.mkdirhier(self.apt_conf_dir + "/apt.conf.d/")
 
 arch_list = []
-for arch in self.all_arch_list:
+for arch in self.pkg_archs.split():
 if not os.path.exists(os.path.join(self.deploy_dir, arch)):
 continue
 arch_list.append(arch)
-- 
2.1.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] debianutils: Add recipe from meta-oe

[Ricardo Ribalda Delgado]

This recipe is a running dependency of recipe apt

Signed-off-by: Ricardo Ribalda Delgado 
---
 .../recipes-support/debianutils/debianutils_4.4.bb | 35 ++
 1 file changed, 35 insertions(+)
 create mode 100644 meta/recipes-support/debianutils/debianutils_4.4.bb

diff --git a/meta/recipes-support/debianutils/debianutils_4.4.bb 
b/meta/recipes-support/debianutils/debianutils_4.4.bb
new file mode 100644
index 000..346eaf1
--- /dev/null
+++ b/meta/recipes-support/debianutils/debianutils_4.4.bb
@@ -0,0 +1,35 @@
+SUMMARY = "Miscellaneous utilities specific to Debian"
+SECTION = "base"
+LICENSE = "GPLv2 & SMAIL_GPL"
+LIC_FILES_CHKSUM = 
"file://debian/copyright;md5=f01a5203d50512fc4830b4332b696a9f"
+
+SRC_URI = "${DEBIAN_MIRROR}/main/d/${BPN}/${BPN}_${PV}.tar.gz"
+SRC_URI[md5sum] = "c0cb076754d7f4eb1e3397d00916647f"
+SRC_URI[sha256sum] = 
"190850cdd6b5302e0a1ba1aaed1bc7074d67d3bd8d04c613f242f7145afa53a6"
+
+inherit autotools update-alternatives
+
+do_configure_prepend() {
+sed -i -e 's:tempfile.1 which.1:which.1:g' ${S}/Makefile.am
+}
+
+do_install_append() {
+if [ "${base_bindir}" != "${bindir}" ]; then
+# Debian places some utils into ${base_bindir} as does busybox
+install -d ${D}${base_bindir}
+for app in run-parts tempfile; do
+mv ${D}${bindir}/$app ${D}${base_bindir}/$app
+done
+fi
+}
+
+ALTERNATIVE_PRIORITY="100"
+ALTERNATIVE_${PN} = "add-shell installkernel remove-shell run-parts savelog 
tempfile which"
+
+ALTERNATIVE_LINK_NAME[add-shell]="${sbindir}/add-shell"
+ALTERNATIVE_LINK_NAME[installkernel]="${sbindir}/installkernel"
+ALTERNATIVE_LINK_NAME[remove-shell]="${sbindir}/remove-shell"
+ALTERNATIVE_LINK_NAME[run-parts]="${base_bindir}/run-parts"
+ALTERNATIVE_LINK_NAME[savelog]="${bindir}/savelog"
+ALTERNATIVE_LINK_NAME[tempfile]="${base_bindir}/tempfile"
+ALTERNATIVE_LINK_NAME[which]="${bindir}/which"
-- 
2.1.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] apt: Add missing running depency debianutils

[Ricardo Ribalda Delgado]

apt-file calls run-parts with options --list and --regex:

root@neopili:~/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/work/bobcat_64-poky-linux/apt/0.9.9.4-r0/apt-0.9.9.4#
 grep run-parts cmdline/apt-key -rn
187:for trusted in $(run-parts --list $TRUSTEDPARTS --regex 
'^.*\.gpg$'); do

busybox implementation of run-parts does not support --regex. And --list option 
is not enabled on yocto busybox configuration.

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-devtools/apt/apt_0.9.9.4.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/apt/apt_0.9.9.4.bb 
b/meta/recipes-devtools/apt/apt_0.9.9.4.bb
index 86ffe66..77cbff1 100644
--- a/meta/recipes-devtools/apt/apt_0.9.9.4.bb
+++ b/meta/recipes-devtools/apt/apt_0.9.9.4.bb
@@ -1,5 +1,5 @@
 DEPENDS = "curl db"
-RDEPENDS_${PN} = "dpkg bash"
+RDEPENDS_${PN} = "dpkg bash debianutils"
 LIC_FILES_CHKSUM = "file://COPYING.GPL;md5=0636e73ff0215e8d672dc4c32c317bb3"
 require apt.inc
 
-- 
2.1.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/2] Add missing running depency of apt-key

[Ricardo Ribalda Delgado]

apt-file calls run-parts with options --list and --regex

root@neopili:~/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/work/bobcat_64-poky-linux/apt/0.9.9.4-r0/apt-0.9.9.4#
 grep run-parts cmdline/apt-key -rn 
187:for trusted in $(run-parts --list $TRUSTEDPARTS --regex 
'^.*\.gpg$'); do


busybox implementation of run-parts does not support --regex. And --list option 
is not enabled on yocto busybox configuration.

Adding recipe debianutils from meta-oe fixes the issue.

This fixes bug #6924

https://bugzilla.yoctoproject.org/show_bug.cgi?id=6924

Ricardo Ribalda Delgado (2):
  debianutils: Add recipe from meta-oe
  apt: Add missing running depency debianutils

 meta/recipes-devtools/apt/apt_0.9.9.4.bb   |  2 +-
 .../recipes-support/debianutils/debianutils_4.4.bb | 35 ++
 2 files changed, 36 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-support/debianutils/debianutils_4.4.bb

-- 
2.1.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] cryptodev-module: Fix build on kernel v3.19

[Ricardo Ribalda Delgado]

Kernel commit f938612dd97d481b8b5bf960c992ae577f081c17 removes the 
get_unused_fd_macro().
This patch replaces the macro with its output.

| NOTE: make -j 8 
KERNEL_DIR=/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/sysroots/qt5022/usr/src/kernel
 
PREFIX=/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/work/qt5022-poky-linux/cryptodev-module/1.6-r0/image
 
KERNEL_PATH=/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/sysroots/qt5022/usr/src/kernel
 
KERNEL_SRC=/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/sysroots/qt5022/usr/src/kernel
 KERNEL_VERSION=3.19.0-qtec-standard CC=x86_64-poky-linux-gcc  
LD=x86_64-poky-linux-ld.bfd  AR=x86_64-poky-linux-ar
| make -C 
/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/sysroots/qt5022/usr/src/kernel
 SUBDIRS=`pwd` modules
| make[1]: warning: jobserver unavailable: using -j1.  Add '+' to parent make 
rule.
| make[1]: Entering directory 
'/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/sysroots/qt5022/usr/src/kernel'
|   CC [M]  
/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/work/qt5022-poky-linux/cryptodev-module/1.6-r0/cryptodev-linux-1.6/ioctl.o
| 
/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/work/qt5022-poky-linux/cryptodev-module/1.6-r0/cryptodev-linux-1.6/ioctl.c:
 In function 'clonefd':
| 
/home/ricardo/curro/qtec/qt5022/build-qt5022-cesium/build/tmp/work/qt5022-poky-linux/cryptodev-module/1.6-r0/cryptodev-linux-1.6/ioctl.c:532:2:
 error: implicit declaration of function 'get_unused_fd' 
[-Werror=implicit-function-declaration]
|   ret = get_unused_fd();
|   ^

Signed-off-by: Ricardo Ribalda Delgado 
---
 .../cryptodev/cryptodev-module_1.6.bb  |  1 +
 .../files/0001-ioctl.c-Fix-build-on-3.19.patch | 29 ++
 2 files changed, 30 insertions(+)
 create mode 100644 
meta/recipes-kernel/cryptodev/files/0001-ioctl.c-Fix-build-on-3.19.patch

diff --git a/meta/recipes-kernel/cryptodev/cryptodev-module_1.6.bb 
b/meta/recipes-kernel/cryptodev/cryptodev-module_1.6.bb
index df50e5a..e6b1025 100644
--- a/meta/recipes-kernel/cryptodev/cryptodev-module_1.6.bb
+++ b/meta/recipes-kernel/cryptodev/cryptodev-module_1.6.bb
@@ -10,6 +10,7 @@ DEPENDS += "cryptodev-linux"
 SRC_URI += " \
 file://0001-Disable-installing-header-file-provided-by-another-p.patch \
 file://0002-In-the-3.13-rc1-Linux-kernel-the-INIT_COMPLETION-mac.patch \
+file://0001-ioctl.c-Fix-build-on-3.19.patch \
 "
 
 EXTRA_OEMAKE='KERNEL_DIR="${STAGING_KERNEL_DIR}" PREFIX="${D}"'
diff --git 
a/meta/recipes-kernel/cryptodev/files/0001-ioctl.c-Fix-build-on-3.19.patch 
b/meta/recipes-kernel/cryptodev/files/0001-ioctl.c-Fix-build-on-3.19.patch
new file mode 100644
index 000..74c8972
--- /dev/null
+++ b/meta/recipes-kernel/cryptodev/files/0001-ioctl.c-Fix-build-on-3.19.patch
@@ -0,0 +1,29 @@
+From 7d71124991030f99965e6e6aaed421115f185adb Mon Sep 17 00:00:00 2001
+From: Ricardo Ribalda Delgado 
+Date: Mon, 9 Feb 2015 13:51:56 +0100
+Subject: [PATCH] ioctl.c: Fix build on 3.19
+
+get_unused_fd() MACRO has been removed on kernel 3.19. Replace by the
+actual output of the macro.
+
+Signed-off-by: Ricardo Ribalda Delgado 
+---
+ ioctl.c | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/ioctl.c b/ioctl.c
+index f26cf93..cf8cc17 100644
+--- a/ioctl.c
 b/ioctl.c
+@@ -529,7 +529,7 @@ static int
+ clonefd(struct file *filp)
+ {
+   int ret;
+-  ret = get_unused_fd();
++  ret = get_unused_fd_flags(0);
+   if (ret >= 0) {
+   get_file(filp);
+   fd_install(ret, filp);
+-- 
+2.1.4
+
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] daisy: System crash running grub-efi on 64 bit machine

[Ricardo Ribalda Delgado]

Hello

I have just ported my build system to daisy and I am unable to boot
from grub-efi. As soon as the grub binary is loaded by uefi, the
system freezes. The error can be reproduced on virtualbox.

Since it is even before the kernel is loaded, and no error message is
given I don't know which will be the easiest way to debug the issue. I
hope that the log message from virtualbox can give you a hint of the
issue: https://gist.github.com/anonymous/7ffd78aa3aada41801fe .

It could be a toolchain issue, because if I build the same code (even
with the patches) with Debian's gcc everything works as expected.

If anyone needs the .vdi file I will be more than happy to share it.

Thanks!



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] package_manager: Fix Argument list too long

[Ricardo Ribalda Delgado]

Function buildhistory_list_installed_image fails with error "Argument
list too long". This patch uses a temporal file to pass the package list
to opkg-query-helper.py

File: 
'/var/lib/jenkins/jobs/qt5022-cesium/workspace/repo/yocto/meta/lib/oe/package_manager.py',
 lineno: 421, function: list
 0417:try:
 0418:output = subprocess.check_output("echo -e '%s' | %s" %
 0419: (output, 
opkg_query_cmd),
 0420: 
stderr=subprocess.STDOUT,
 *** 0421: shell=True)
 0422:except subprocess.CalledProcessError as e:
 0423:bb.fatal("Cannot compute packages dependencies. 
Command '%s' "
 0424: "returned %d:\n%s" % (e.cmd, e.returncode, 
e.output))
 0425:
Exception: OSError: [Errno 7] Argument list too long

ERROR: Function failed: buildhistory_list_installed_image
ERROR: Logfile of failure stored in: 
/var/lib/jenkins/jobs/qt5022-cesium/workspace/build/tmp/work/qt5022-poky-linux/qimage-dev/1.0-r0/temp/log.do_rootfs.16747
NOTE: recipe qimage-dev-1.0-r0: task do_rootfs: Failed
ERROR: Task 7 
(/var/lib/jenkins/jobs/qt5022-cesium/workspace/repo/yocto/../qtec/meta-qt5022/recipes-core/images/qimage-dev.bb,
 do_rootfs) failed with exit code '1'
NOTE: Tasks Summary: Attempted 4999 tasks of which 30 didn't need to be rerun 
and 1 failed.
NOTE: Writing buildhistory
Auto packing the repository for optimum performance.

Summary: 1 task failed:
  
/var/lib/jenkins/jobs/qt5022-cesium/workspace/repo/yocto/../qtec/meta-qt5022/recipes-core/images/qimage-dev.bb,
 do_rootfs
Summary: There were 74 WARNING messages shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/lib/oe/package_manager.py | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index a8360fe..3aff9d8 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -6,6 +6,7 @@ import shutil
 import multiprocessing
 import re
 import bb
+import tempfile
 
 
 # this can be used by all PM backends to create the index files in parallel
@@ -411,16 +412,22 @@ class DpkgPkgsList(PkgsList):
 output = tmp_output
 elif format == "deps":
 opkg_query_cmd = bb.utils.which(os.getenv('PATH'), 
"opkg-query-helper.py")
+file_out = tempfile.NamedTemporaryFile()
+file_out.write(output)
+file_out.flush()
 
 try:
-output = subprocess.check_output("echo -e '%s' | %s" %
- (output, opkg_query_cmd),
+output = subprocess.check_output("cat %s | %s" %
+ (file_out.name, 
opkg_query_cmd),
  stderr=subprocess.STDOUT,
  shell=True)
 except subprocess.CalledProcessError as e:
+file_out.close()
 bb.fatal("Cannot compute packages dependencies. Command '%s' "
  "returned %d:\n%s" % (e.cmd, e.returncode, e.output))
 
+file_out.close()
+
 return output
 
 
-- 
2.0.0.rc0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] package_manager: Fix NoneType Object on do_populate_sdk

[Ricardo Ribalda Delgado]

PACKAGE_EXCLUDE can be not defined or empty, leading to a build error.

File: 
'/var/lib/jenkins/jobs/qt5022-cesium/workspace/repo/yocto/meta/lib/oe/package_manager.py',
 lineno: 1649, function: _create_configs
 1645:"Pin-Priority: %d\n\n" % (arch, priority))
 1646:
 1647:priority += 5
 1648:
 *** 1649:for pkg in self.d.getVar('PACKAGE_EXCLUDE', True).split():
 1650:prefs_file.write(
 1651:"Package: %s\n"
 1652:"Pin: release *\n"
 1653:"Pin-Priority: -1\n\n" % pkg)
Exception: AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Function failed: do_populate_sdk
ERROR: Logfile of failure stored in: 
/var/lib/jenkins/jobs/qt5022-cesium/workspace/build/tmp/work/bobcat_64-poky-linux/meta-toolchain/1.0-r7/temp/log.do_populate_sdk.21363
NOTE: recipe meta-toolchain-1.0-r7: task do_populate_sdk: Failed
Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/lib/oe/package_manager.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index a8360fe..b33ee60 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -1637,7 +1637,8 @@ class DpkgPM(PackageManager):
 
 priority += 5
 
-for pkg in self.d.getVar('PACKAGE_EXCLUDE', True).split():
+pkg_exclude = self.d.getVar('PACKAGE_EXCLUDE', True) or ""
+for pkg in pkg_exclude:
 prefs_file.write(
 "Package: %s\n"
 "Pin: release *\n"
-- 
2.0.0.rc0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] daisy: System crash running grub-efi on 64 bit machine

[Ricardo Ribalda Delgado]

I was using dora. But it seems that I was also using all the files
from grub-efi-native to build the grub binary. Of course this is
breaking the cross compilation barrier...

What is pretty strange is that both my distro and poky uses the same
compiler 4.8.2. For exactly the same source code, one build with my
distro gcc via grub-efi-native and the other with the cross compiler I
get one binary that works and another that crashes the machine.

Unfortunately I am not an expert on uefi nor grub, so I dont know the
best way to debug this (except adding printfs...)

Is the output from virtualbox useful some how? Can anybody else
reproduce the error?


Thanks!


On Tue, May 6, 2014 at 7:22 PM, Khem Raj  wrote:
> On Tue, May 6, 2014 at 6:20 AM, Ricardo Ribalda Delgado
>  wrote:
>>
>> I have just ported my build system to daisy and I am unable to boot
>> from grub-efi. As soon as the grub binary is loaded by uefi, the
>> system freezes. The error can be reproduced on virtualbox.
>>
>> Since it is even before the kernel is loaded, and no error message is
>> given I don't know which will be the easiest way to debug the issue. I
>> hope that the log message from virtualbox can give you a hint of the
>> issue: https://gist.github.com/anonymous/7ffd78aa3aada41801fe .
>>
>> It could be a toolchain issue, because if I build the same code (even
>> with the patches) with Debian's gcc everything works as expected.
>>
>> If anyone needs the .vdi file I will be more than happy to share it.
>
> what was the working release you used before and whats the compiler
> version on that release



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] PR_Service. Sync database only on clean stop?

[Ricardo Ribalda Delgado]

Hello

I have been running happily with pr_service for two months.
Unfortunately, due to a power down the computer running the pr server
has stopped.
When I have started the server I have discovered that all the
information about PR was lost :S. I assumed this was due to a db
corruption, but I have later discovered that the database is not
synced after every insert.

I guess this is to improve the performance, but I don't think that
everybody prefer this behaviour, right now I have to remake and
distribute only 10 flashes, something reasonable, but it would have a
disaster later in our development cycle.

Is somebody working on a parameter for the db to be persistent? Would
a patch to implement this be considered?

Regards

-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] xserver-xorg: Make xinerama selectable via PACKAGECONFIG

[Ricardo Ribalda Delgado]

If xinerara can be selected via PACKAGECONFIG, xserver-xorg recipe can
be easier modified via bbappend.

xinerama is needed by the fglrx driver (from AMD).

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-graphics/xorg-xserver/xserver-xorg.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc 
b/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
index f0da488412d5..18b1f896fe3a 100644
--- a/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
+++ b/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
@@ -101,7 +101,6 @@ FILES_${PN}-module-libxf4bpp = 
"${libdir}/xorg/modules/libxf4bpp.so"
 EXTRA_OECONF += "--with-fop=no \
  --with-pic \
  --disable-static \
- --disable-xinerama \
  --disable-record \
  --disable-dmx \
  --disable-xnest \
@@ -128,6 +127,7 @@ PACKAGECONFIG[unwind] = 
"--enable-libunwind,--disable-libunwind,libunwind"
 PACKAGECONFIG[xshmfence] = 
"--enable-xshmfence,--disable-xshmfence,libxshmfence"
 PACKAGECONFIG[xmlto] = "--with-xmlto, --without-xmlto, xmlto-native 
docbook-xml-dtd4-native docbook-xsl-stylesheets-native"
 PACKAGECONFIG[systemd-logind] = 
"--enable-systemd-logind=yes,--enable-systemd-logind=no,dbus,"
+PACKAGECONFIG[xinerama] = "--enable-xinerama,--disable-xinerama"
 
 do_install_append () {
# Its assumed base-files creates this for us
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] gstreamer1.0: Make tests selectable via variable

[Ricardo Ribalda Delgado]

This way, this configuration can be easily changed via .bbappend file
without having to re-define the whole EXTRA_OECONF.

With --disable-check  libgstcheck is not build.

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-multimedia/gstreamer/gstreamer1.0.inc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0.inc 
b/meta/recipes-multimedia/gstreamer/gstreamer1.0.inc
index a01c33ce5ff9..f689617243e1 100644
--- a/meta/recipes-multimedia/gstreamer/gstreamer1.0.inc
+++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0.inc
@@ -10,8 +10,9 @@ DEPENDS = "glib-2.0 libxml2 bison-native flex-native"
 inherit autotools pkgconfig gettext
 
 GSTREAMER_1_DEBUG ?= "--disable-debug"
+GSTREAMER_1_CHECK ?= "--disable-check"
 EXTRA_OECONF = "--disable-docbook --disable-gtk-doc \
---disable-dependency-tracking --disable-check \
+--disable-dependency-tracking ${GSTREAMER_1_CHECK} \
 --disable-examples --disable-tests \
 --disable-valgrind ${GSTREAMER_1_DEBUG} \
 "
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] xserver-xorg: Make xinerama selectable via PACKAGECONFIG

[Ricardo Ribalda Delgado]

If xinerara can be selected via PACKAGECONFIG, xserver-xorg recipe can
be easier modified via bbappend.

xinerama is needed by the fglrx driver (from AMD).

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-graphics/xorg-xserver/xserver-xorg.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc 
b/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
index f0da488412d5..18b1f896fe3a 100644
--- a/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
+++ b/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
@@ -101,7 +101,6 @@ FILES_${PN}-module-libxf4bpp = 
"${libdir}/xorg/modules/libxf4bpp.so"
 EXTRA_OECONF += "--with-fop=no \
  --with-pic \
  --disable-static \
- --disable-xinerama \
  --disable-record \
  --disable-dmx \
  --disable-xnest \
@@ -128,6 +127,7 @@ PACKAGECONFIG[unwind] = 
"--enable-libunwind,--disable-libunwind,libunwind"
 PACKAGECONFIG[xshmfence] = 
"--enable-xshmfence,--disable-xshmfence,libxshmfence"
 PACKAGECONFIG[xmlto] = "--with-xmlto, --without-xmlto, xmlto-native 
docbook-xml-dtd4-native docbook-xsl-stylesheets-native"
 PACKAGECONFIG[systemd-logind] = 
"--enable-systemd-logind=yes,--enable-systemd-logind=no,dbus,"
+PACKAGECONFIG[xinerama] = "--enable-xinerama,--disable-xinerama"
 
 do_install_append () {
# Its assumed base-files creates this for us
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] xserver-xorg: Make xinerama selectable via PACKAGECONFIG

[Ricardo Ribalda Delgado]

If xinerara can be selected via PACKAGECONFIG, xserver-xorg recipe can
be easier modified via bbappend.

xinerama is needed by the fglrx driver (from AMD).

Signed-off-by: Ricardo Ribalda Delgado 
---

v2: Fix by Ross Burton 
 Add missing xineramaproto dep


 meta/recipes-graphics/xorg-xserver/xserver-xorg.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc 
b/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
index f0da488412d5..30299c2921ff 100644
--- a/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
+++ b/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
@@ -101,7 +101,6 @@ FILES_${PN}-module-libxf4bpp = 
"${libdir}/xorg/modules/libxf4bpp.so"
 EXTRA_OECONF += "--with-fop=no \
  --with-pic \
  --disable-static \
- --disable-xinerama \
  --disable-record \
  --disable-dmx \
  --disable-xnest \
@@ -128,6 +127,7 @@ PACKAGECONFIG[unwind] = 
"--enable-libunwind,--disable-libunwind,libunwind"
 PACKAGECONFIG[xshmfence] = 
"--enable-xshmfence,--disable-xshmfence,libxshmfence"
 PACKAGECONFIG[xmlto] = "--with-xmlto, --without-xmlto, xmlto-native 
docbook-xml-dtd4-native docbook-xsl-stylesheets-native"
 PACKAGECONFIG[systemd-logind] = 
"--enable-systemd-logind=yes,--enable-systemd-logind=no,dbus,"
+PACKAGECONFIG[xinerama] = "--enable-xinerama,--disable-xinerama,xineramaproto"
 
 do_install_append () {
# Its assumed base-files creates this for us
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] gstreamer1.0: Make tests selectable via variable

[Ricardo Ribalda Delgado]

Hello Ross

On Fri, May 8, 2015 at 3:47 PM, Burton, Ross  wrote:
>
> On 8 May 2015 at 12:39, Ricardo Ribalda Delgado 
> wrote:
>>
>>  GSTREAMER_1_DEBUG ?= "--disable-debug"
>> +GSTREAMER_1_CHECK ?= "--disable-check"
>>  EXTRA_OECONF = "--disable-docbook --disable-gtk-doc \
>> ---disable-dependency-tracking --disable-check \
>> +--disable-dependency-tracking ${GSTREAMER_1_CHECK} \
>
>
> Seems like a PACKAGECONFIG is a neater solution, can you use that and also
> change _DEBUG to use PACKAGECONFIG too?

sure, give me a sec :)

>
> Ross



-- 
Ricardo Ribalda
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] gstreamer1.0: Make check selectable via PACKAGECONFIG

[Ricardo Ribalda Delgado]

This way, this configuration can be easily changed via .bbappend file
without having to re-define the whole EXTRA_OECONF.

With --disable-check  libgstcheck is not build.

Signed-off-by: Ricardo Ribalda Delgado 
---
 meta/recipes-multimedia/gstreamer/gstreamer1.0.inc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0.inc 
b/meta/recipes-multimedia/gstreamer/gstreamer1.0.inc
index 17a0d7f839c4..be8aa8f9cbb7 100644
--- a/meta/recipes-multimedia/gstreamer/gstreamer1.0.inc
+++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0.inc
@@ -10,9 +10,10 @@ DEPENDS = "glib-2.0 libxml2 bison-native flex-native"
 inherit autotools pkgconfig gettext
 
 PACKAGECONFIG[debug] = "--enable-debug,--disable-debug"
+PACKAGECONFIG[check] = "--enable-check,--disable-check"
 
 EXTRA_OECONF = "--disable-docbook --disable-gtk-doc \
---disable-dependency-tracking --disable-check \
+--disable-dependency-tracking \
 --disable-examples --disable-tests \
 --disable-valgrind \
 "
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core