On Thu, 2020-03-05 at 09:42 +0000, Paul Barker wrote: > On Wed, 4 Mar 2020 12:30:10 +0000 > "Freihofer, Adrian" <adrian.freiho...@siemens.com> wrote: > > > -----Original Message----- > > From: Paul Barker <pbar...@konsulko.com> > > To: "Freihofer, Adrian" <adrian.freiho...@siemens.com> > > Cc: openembedded-core@lists.openembedded.org < > > openembedded-core@lists.openembedded.org> > > Subject: Re: [OE-core] [PATCH] bitbake: gitsm: download submodules > > Date: Wed, 04 Mar 2020 09:59:44 +0000 > > > > On Wed, 4 Mar 2020 08:12:27 +0000 > > "Freihofer, Adrian" <adrian.freiho...@siemens.com> wrote: > > > > > The unpack function failed because the submodules were not > > > downloaded. > > > Calling download before unpack for each submodule solves this > > > issue. > > > > > > Signed-off-by: Adrian Freihofer <adrian.freiho...@siemens.com> > > > --- > > > bitbake/lib/bb/fetch2/gitsm.py | 1 + > > > 1 file changed, 1 insertion(+) > > > > > > diff --git a/bitbake/lib/bb/fetch2/gitsm.py > > > b/bitbake/lib/bb/fetch2/gitsm.py > > > index c622771d21..3715e9824f 100644 > > > --- a/bitbake/lib/bb/fetch2/gitsm.py > > > +++ b/bitbake/lib/bb/fetch2/gitsm.py > > > @@ -184,6 +184,7 @@ class GitSM(Git): > > > > > > try: > > > newfetch = Fetch([url], d, cache=False) > > > + newfetch.download() > > > newfetch.unpack(root=os.path.dirname(os.path.joi > > > n(re > > > po > > > _conf, 'modules', module))) > > > except Exception as e: > > > logger.error('gitsm: submodule unpack failed: %s > > > %s' > > > % > > > (type(e).__name__, str(e))) > > > > You shouldn't be trying to download submodules in the do_unpack > > step. > > If > > they're missing at this stage it probably indicates a fetch issue. > > > > Basically true, but the information about which submodules need to > > be > > downloaded is not available before the top level repo has been > > unpacked. I guess the solution must be something like: > > > > fetch top-level > > unpack top-level > > foreach submodule: > > fetch submodule > > unpack submodule > > > > What's the exact error that you're seeing? > > > > Don't remeber exactly. But when I debugged the flow, it became > > obvious, > > that the submodules are not downloaded. > > > > This could be related to the issue I saw when the fetcher uses git > > shallow > > tarballs from a mirror - if that's the case I'm planning to get > > that > > fixed > > this weekend. > > > > Yes, the problem occurs with git shallows. The patch solves it with > > and > > without shallows for us. > > > > Also, your email client seems to have chewed the patch up. It's > > best to > > send > > patches using `git send-email` only. > > > > Sorry, we switched to a new e-mail solution. I have to adapt my > > setup. > > The problem with your patch is that it would require network access > during > do_unpack which then breaks our ability to do offline builds. Fetching in the unpack task is probably really wrong, I agree. However, it also works on our CI infrastructure which runs without Internet connectivity. > > Do you only see this issue when fetching from git shallow mirror > tarballs?
On our CI which works with gitshallow archives, fetching did not work at all. On machines with Internet access we also had problems, but I did not completely analyze what happened. The ovmf package from poky was causing the troubles in our case. > If so, the mirror tarballs need to be extracted during do_fetch but > not placed in the usual 'git2' directory (as future fetches can't > cope with shallow clones in that directory). Do you already have an idea how this could be implemented? > > Fixing this also has some impact on archiver.bbclass as that also > needs to be able to walk the tree of submodules correctly. Yes. > -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core