Re: [yocto] [RFC] devtool: support multiple sources
Le jeu. 23 nov. 2023 à 00:36, Richard Purdie a écrit : > > On Wed, 2023-11-22 at 13:53 +0100, Julien Stephan wrote: > > Le mar. 14 nov. 2023 à 12:21, Alexander Kanavin > > a écrit : > > > > > > I think no one actually anticipated all these corner cases when > > > devtool was written. In general it's simply not designed to handle > > > multiple sources in SRC_URI, it can do (tarball or git)+patches, and > > > not any of the other options. No one is expecting a fix for all of > > > them, perhaps it's best to keep the scope to plain gitsm://. > > > > > > > Hi Alex, > > > > I sent a series to add submodule support. As you suggested I focused > > myself on gitsm recipes, but I had to handle the situation of a git > > tree extracted into S because it was throwing an error now with my > > submodule implementation (while it was not working but not throwing > > any error on master). > > > > > I don't have specific advice for the broader issue; if you have ideas, > > > it would be good to at least document them. > > > > > > Alex > > > > > > For the other cases, I was thinking of using the newly merged API > > "unpack tracer" > > (https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695) > > (suggested by RP in this thread > > https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695). > > I really think that we could leverage this new api to create a devtool > > unpack tracer, so we can have a better idea of what is unpacked and > > where. This could help to solve the corner cases I described > > previously but also the bug 15162 > > (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15162). I think it > > may also help to simplify devtool code in various places. > > Where should I document this? > > Should I open a new ticket? > > Yes please, it would be good to at least document the idea as I do > think it could help devtool with a number of the corner cases. A > bugzilla bug for it is the place to do it. > Hi Richard and Alexander, FYI, here is the bugzilla bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=15294 Cheers Julien > Cheers, > > Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61733): https://lists.yoctoproject.org/g/yocto/message/61733 Mute This Topic: https://lists.yoctoproject.org/mt/102509834/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [RFC] devtool: support multiple sources
On Wed, 2023-11-22 at 13:53 +0100, Julien Stephan wrote: > Le mar. 14 nov. 2023 à 12:21, Alexander Kanavin > a écrit : > > > > I think no one actually anticipated all these corner cases when > > devtool was written. In general it's simply not designed to handle > > multiple sources in SRC_URI, it can do (tarball or git)+patches, and > > not any of the other options. No one is expecting a fix for all of > > them, perhaps it's best to keep the scope to plain gitsm://. > > > > Hi Alex, > > I sent a series to add submodule support. As you suggested I focused > myself on gitsm recipes, but I had to handle the situation of a git > tree extracted into S because it was throwing an error now with my > submodule implementation (while it was not working but not throwing > any error on master). > > > I don't have specific advice for the broader issue; if you have ideas, > > it would be good to at least document them. > > > > Alex > > > For the other cases, I was thinking of using the newly merged API > "unpack tracer" > (https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695) > (suggested by RP in this thread > https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695). > I really think that we could leverage this new api to create a devtool > unpack tracer, so we can have a better idea of what is unpacked and > where. This could help to solve the corner cases I described > previously but also the bug 15162 > (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15162). I think it > may also help to simplify devtool code in various places. > Where should I document this? > Should I open a new ticket? Yes please, it would be good to at least document the idea as I do think it could help devtool with a number of the corner cases. A bugzilla bug for it is the place to do it. Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61732): https://lists.yoctoproject.org/g/yocto/message/61732 Mute This Topic: https://lists.yoctoproject.org/mt/102509834/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [RFC] devtool: support multiple sources
Le mar. 14 nov. 2023 à 12:21, Alexander Kanavin
a écrit :
>
> I think no one actually anticipated all these corner cases when
> devtool was written. In general it's simply not designed to handle
> multiple sources in SRC_URI, it can do (tarball or git)+patches, and
> not any of the other options. No one is expecting a fix for all of
> them, perhaps it's best to keep the scope to plain gitsm://.
>
Hi Alex,
I sent a series to add submodule support. As you suggested I focused
myself on gitsm recipes, but I had to handle the situation of a git
tree extracted into S because it was throwing an error now with my
submodule implementation (while it was not working but not throwing
any error on master).
> I don't have specific advice for the broader issue; if you have ideas,
> it would be good to at least document them.
>
> Alex
For the other cases, I was thinking of using the newly merged API
"unpack tracer"
(https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695)
(suggested by RP in this thread
https://git.yoctoproject.org/poky/commit/?id=ef3e46afd910d4b7727d42c4c18b501525c65695).
I really think that we could leverage this new api to create a devtool
unpack tracer, so we can have a better idea of what is unpacked and
where. This could help to solve the corner cases I described
previously but also the bug 15162
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=15162). I think it
may also help to simplify devtool code in various places.
Where should I document this?
Should I open a new ticket?
Cheers
Julien
>
> On Fri, 10 Nov 2023 at 17:24, Julien Stephan wrote:
> >
> > Hi all,
> >
> > Sorry for the long message here, but I would really appreciate
> > feedback from the community :)
> >
> > I am currently working on bug #14141
> > (https://bugzilla.yoctoproject.org/show_bug.cgi?id=14141). The main
> > goal of this bug is to correctly support git submodules with devtool.
> >
> > I currently have an almost ready branch here:
> > https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule-fix
> >
> > I asked Jon Mason (reporter of the bug) to test the branch, he
> > confirmed that its working for git submodules (i.e recipes using gitsm
> > only) such as vulkan-samples or hafnium (meta-arm), but he also
> > pointed out a new issue highlighted by the recipe edk2-firmware: this
> > recipe is using gitsm for the primary source then extract another git
> > repository inside S.
> >
> > This issue is not related only to gitsm + git recipes, I reproduced
> > this kind of error with cross-localedef-native recipe, using git +
> > git or https + git. But I think the issue is even more global: as of
> > today devtool doesn't properly handle recipes that unpack secondary
> > sources inside S. Let suppose we have:
> >
> > SRC_URI = " \
> > http://.tar.gz \
> > http://.tar.gz;subdir=mysubdir \
> > "
> > If I need to patch something inside mysubdir I would expect to have
> > something like:
> > SRC_URI = " \
> > http://.tar.gz \
> > http://.tar.gz;subdir=mysubdir \
> > file://mypatch.patch;patchdir=mysubdir \
> > "
> > Of course this is working, but if I run devtool modify, create my
> > patch and do a devtool finish I would have something like
> > SRC_URI = " \
> > http://.tar.gz \
> > http://.tar.gz;subdir=mysubdir \
> > file://mypatch.patch \
> > "
> > where mypatch.patch will patch files inside mysubdir as if they were
> > originally part of the primary source.
> > That might be acceptable (but I personally would prefer to have a
> > patchir parameter matching the subdir parameter).
> > Besides that, there is actually a bug here: trying to modify inside
> > devtool a file that was added using file:// will add a new patch AND
> > modify the original patch... (Tested on master branch with bzip2 with
> > the following: devtool modify bzip2, modify Makefile.am, commit the
> > modification, devtool finish bzip2 will create a new patch file AND
> > also update the originale Makefile.am inside the recipe space)
> >
> > But things get really bad when using a git source as secondary source:
> > SRC_URI = " \
> > http://.tar.gz \
> > git ://;destsuffix=mysubdir \
> > "
> > mysubdir gets automatically committed inside the primary source
> > ("Commiting changes from do patch"). This is an issue because doing
> > "git add" on a git directory adds it as a submodule (in a wrong way,
> > the correct way of adding it would be "git submodule add url path").
> > Here we end up with a submodule which is not populated into
> > .gitmodules then "git submodule foreach" commands just fail.
> >
> > My hack is to detect that git directories were added after unpack and
> > properly add them as submodules (" wip: support git tree extracted
> > inside S" in my branch) but I am not 100% satisfied with this because
> > it implies that devtool is managing secondary source differently if
> > they are git directories or not..
> >
> > So the long story made short:
> > * current
Re: [yocto] [RFC] devtool: support multiple sources
I think no one actually anticipated all these corner cases when
devtool was written. In general it's simply not designed to handle
multiple sources in SRC_URI, it can do (tarball or git)+patches, and
not any of the other options. No one is expecting a fix for all of
them, perhaps it's best to keep the scope to plain gitsm://.
I don't have specific advice for the broader issue; if you have ideas,
it would be good to at least document them.
Alex
On Fri, 10 Nov 2023 at 17:24, Julien Stephan wrote:
>
> Hi all,
>
> Sorry for the long message here, but I would really appreciate
> feedback from the community :)
>
> I am currently working on bug #14141
> (https://bugzilla.yoctoproject.org/show_bug.cgi?id=14141). The main
> goal of this bug is to correctly support git submodules with devtool.
>
> I currently have an almost ready branch here:
> https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule-fix
>
> I asked Jon Mason (reporter of the bug) to test the branch, he
> confirmed that its working for git submodules (i.e recipes using gitsm
> only) such as vulkan-samples or hafnium (meta-arm), but he also
> pointed out a new issue highlighted by the recipe edk2-firmware: this
> recipe is using gitsm for the primary source then extract another git
> repository inside S.
>
> This issue is not related only to gitsm + git recipes, I reproduced
> this kind of error with cross-localedef-native recipe, using git +
> git or https + git. But I think the issue is even more global: as of
> today devtool doesn't properly handle recipes that unpack secondary
> sources inside S. Let suppose we have:
>
> SRC_URI = " \
> http://.tar.gz \
> http://.tar.gz;subdir=mysubdir \
> "
> If I need to patch something inside mysubdir I would expect to have
> something like:
> SRC_URI = " \
> http://.tar.gz \
> http://.tar.gz;subdir=mysubdir \
> file://mypatch.patch;patchdir=mysubdir \
> "
> Of course this is working, but if I run devtool modify, create my
> patch and do a devtool finish I would have something like
> SRC_URI = " \
> http://.tar.gz \
> http://.tar.gz;subdir=mysubdir \
> file://mypatch.patch \
> "
> where mypatch.patch will patch files inside mysubdir as if they were
> originally part of the primary source.
> That might be acceptable (but I personally would prefer to have a
> patchir parameter matching the subdir parameter).
> Besides that, there is actually a bug here: trying to modify inside
> devtool a file that was added using file:// will add a new patch AND
> modify the original patch... (Tested on master branch with bzip2 with
> the following: devtool modify bzip2, modify Makefile.am, commit the
> modification, devtool finish bzip2 will create a new patch file AND
> also update the originale Makefile.am inside the recipe space)
>
> But things get really bad when using a git source as secondary source:
> SRC_URI = " \
> http://.tar.gz \
> git ://;destsuffix=mysubdir \
> "
> mysubdir gets automatically committed inside the primary source
> ("Commiting changes from do patch"). This is an issue because doing
> "git add" on a git directory adds it as a submodule (in a wrong way,
> the correct way of adding it would be "git submodule add url path").
> Here we end up with a submodule which is not populated into
> .gitmodules then "git submodule foreach" commands just fail.
>
> My hack is to detect that git directories were added after unpack and
> properly add them as submodules (" wip: support git tree extracted
> inside S" in my branch) but I am not 100% satisfied with this because
> it implies that devtool is managing secondary source differently if
> they are git directories or not..
>
> So the long story made short:
> * current implementation of devtool is buggy (at least with the
> scenario described above with bzip2), I may try to work on this latter
> * is the current behaviour of devtool acceptable ? (i.e not adding
> patchir parameter on patches)
> * is my "hack" acceptable to handle git secondary sources as a real
> git submodule?
>
> This is giving me headaches and I would really appreciate the feedback
> of the community on this.
>
> Cheers
> Julien
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61678): https://lists.yoctoproject.org/g/yocto/message/61678
Mute This Topic: https://lists.yoctoproject.org/mt/102509834/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[yocto] [RFC] devtool: support multiple sources
Hi all,
Sorry for the long message here, but I would really appreciate
feedback from the community :)
I am currently working on bug #14141
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=14141). The main
goal of this bug is to correctly support git submodules with devtool.
I currently have an almost ready branch here:
https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule-fix
I asked Jon Mason (reporter of the bug) to test the branch, he
confirmed that its working for git submodules (i.e recipes using gitsm
only) such as vulkan-samples or hafnium (meta-arm), but he also
pointed out a new issue highlighted by the recipe edk2-firmware: this
recipe is using gitsm for the primary source then extract another git
repository inside S.
This issue is not related only to gitsm + git recipes, I reproduced
this kind of error with cross-localedef-native recipe, using git +
git or https + git. But I think the issue is even more global: as of
today devtool doesn't properly handle recipes that unpack secondary
sources inside S. Let suppose we have:
SRC_URI = " \
http://.tar.gz \
http://.tar.gz;subdir=mysubdir \
"
If I need to patch something inside mysubdir I would expect to have
something like:
SRC_URI = " \
http://.tar.gz \
http://.tar.gz;subdir=mysubdir \
file://mypatch.patch;patchdir=mysubdir \
"
Of course this is working, but if I run devtool modify, create my
patch and do a devtool finish I would have something like
SRC_URI = " \
http://.tar.gz \
http://.tar.gz;subdir=mysubdir \
file://mypatch.patch \
"
where mypatch.patch will patch files inside mysubdir as if they were
originally part of the primary source.
That might be acceptable (but I personally would prefer to have a
patchir parameter matching the subdir parameter).
Besides that, there is actually a bug here: trying to modify inside
devtool a file that was added using file:// will add a new patch AND
modify the original patch... (Tested on master branch with bzip2 with
the following: devtool modify bzip2, modify Makefile.am, commit the
modification, devtool finish bzip2 will create a new patch file AND
also update the originale Makefile.am inside the recipe space)
But things get really bad when using a git source as secondary source:
SRC_URI = " \
http://.tar.gz \
git ://;destsuffix=mysubdir \
"
mysubdir gets automatically committed inside the primary source
("Commiting changes from do patch"). This is an issue because doing
"git add" on a git directory adds it as a submodule (in a wrong way,
the correct way of adding it would be "git submodule add url path").
Here we end up with a submodule which is not populated into
.gitmodules then "git submodule foreach" commands just fail.
My hack is to detect that git directories were added after unpack and
properly add them as submodules (" wip: support git tree extracted
inside S" in my branch) but I am not 100% satisfied with this because
it implies that devtool is managing secondary source differently if
they are git directories or not..
So the long story made short:
* current implementation of devtool is buggy (at least with the
scenario described above with bzip2), I may try to work on this latter
* is the current behaviour of devtool acceptable ? (i.e not adding
patchir parameter on patches)
* is my "hack" acceptable to handle git secondary sources as a real
git submodule?
This is giving me headaches and I would really appreciate the feedback
of the community on this.
Cheers
Julien
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61651): https://lists.yoctoproject.org/g/yocto/message/61651
Mute This Topic: https://lists.yoctoproject.org/mt/102509834/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
