On Mon, 2025-01-27 at 15:26 -0500, Mark Asselstine via lists.openembedded.org
wrote:
>
>
> On 1/27/2025 8:12 AM, Christian Lindeberg via lists.openembedded.org wrote:
> >
> >
> > On 27/01/2025 13:22, Stefan Herbrechtsmeier via lists.openembedded.org
> > wrote:
> > > Am 27.01.2025 um 12:21 schrieb Stefan Herbrechtsmeier via
> > > lists.openembedded.org:
> > > > Am 27.01.2025 um 10:53 schrieb Richard Purdie:
> > > > > On Mon, 2025-01-27 at 10:07 +0100, Stefan Herbrechtsmeier wrote:
> > > > > > Am 25.01.2025 um 14:17 schrieb Richard Purdie:
> > > > > > > The gomod fetcher is totally broken for mirroring unfortunately.
> > > > > > > It is
> > > > > > > used for the crucible recipe in meta-oe and it is one of the
> > > > > > > reasons
> > > > > > > the autobuilder mirroring test for meta-oe never passes.
> > > > > > >
> > > > > > > The issue is that it downloads files into subdirs of DL_DIR but
> > > > > > > when
> > > > > > > mirroring, that translation is lost. The files are there on the
> > > > > > > mirror,
> > > > > > > the code just doesn't have any way to access them.
> > > > > > Does this mean the downloadfilename doesn't support sub directories
> > > > > > or
> > > > > > is the rewrite of the URI a problem?
> > > > > The rewrite is hard. The syntax doesn't support taking a path
> > > > > component
> > > > > and injecting it into a parameter.
> > > >
> > > > I assume the problem is not the path but the wrong assumption that
> > > > the downloadfilename is used for the replace of the mirror url.
> > > >
> > > > > > Is the npm fetcher broken too?
> > > > > I don't know.
> > > >
> > > > The npm fetcher use a npm2 sub folder in its downloadfilename.
> > > >
> > > > > > > I don't know what to do about this. The source mirroring for
> > > > > > > meta-oe
> > > > > > > has been broken for months. I've tried to fix a couple of the
> > > > > > > issues
> > > > > > > but this one is systemic and not easy to fix.
> > > > > > >
> > > > > > > I'd note that the sanity test in sanity.bbclass doesn't recognise
> > > > > > > the
> > > > > > > gomod and godmogit fetchers anyway. I did try a few different
> > > > > > > urls
> > > > > > > but
> > > > > > > due to the weird way gomod subclasses wget, the normal mirror url
> > > > > > > manipulation techniques don't work.
> > > > > > >
> > > > > > > What do I do?
> > > > > > >
> > > > > > > I could drop meta-oe from source mirroring.
> > > > > > >
> > > > > > > I could ask Khem to drop the recipe.
> > > > > > >
> > > > > > > Neither seem like nice things to do but I don't want to do
> > > > > > > "partial"
> > > > > > > mirrors. Explaining to people that it may or may not work looks
> > > > > > > really
> > > > > > > bad and I'm not doing it.
> > > > > > >
> > > > > > > Is there anyone willing to help fix this? Only stopping it
> > > > > > > running at
> > > > > > > all is really within my own control.
> > > > > > I can look into it. Would it be okay to init the urldata before
> > > > > > resolve
> > > > > > the mirror or should the mirror use the gomod scheme?
> > > > > Currently the mirror being passed the modified url which makes it a
> > > > > challenge as it appears as a normal http url.
> > > > >
> > > > > I posted my workaround that made it work but that has several issues.
> > > > > It needs a new parameter to the wget fetcher, which might be useful
> > > > > elsewhere, not sure. It also requires hardcoding the go proxy url into
> > > > > the PREMIRROR which won't work for any other go proxy servers.
> > > >
> > > > This only works because the downloadfilename match the path of the
> > > > uri. I think it would be better to use the downloadfilename inside
> > > > the build_mirroruris / uri_replace function, add the downloadfilename
> > > > to the mirrortarballs or add an additional downloadfiledir parameter.
> > > >
> > > > The following test data for replaceuris in MirrorUriTest should
> > > > reproduce the problem:
> > > > ("https://somewhere.org/scope/example/1.0.0/
> > > > example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz",
> > > > "https://.*/.*";, "http://somewhere2.org/somedir3";)
> > > > : "http://somewhere2.org/somedir3/subdir/scope-
> > > > example-1.0.0.tgz;downloadfilename=subdir/scope-example-1.0.0.tgz",
> > > >
> > > >
> > > The following patch pass the test cases with the additional test data
> > > above: diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/
> > > bb/fetch2/__init__.py index 1fea02ee9c..fb013160c4 100644 --- a/
> > > bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/
> > > __init__.py @@ -472,7 +472,7 @@ def uri_replace(ud, uri_find,
> > > uri_replace, replacements, d, mirrortarball=None): uri_decoded[5] = {}
> > > uri_find_decoded[5] = {} elif ud.localpath and
> > > ud.method.supports_checksum(ud): - basename =
> > > os.path.basename(ud.localpath) + basename =
> > > ud.localpath.replace(d.getVar("DL_DIR"), "").lstrip("./") if basename:
> > > uri_basename = os.path.basename(uri_decoded[loc]) # Prefix with a
> > > slash as a sentinel in case
> > >
> > > The dot in the lstrip is needed to pass the tests. This need to be
> > > resolved in an other way to avoid side-effects.
> > >
> > Looks like flattening the downloadfilename would solve/workaround the
> > problem then (replace the slashes with a punctuation character other
> > than those valid in a Go module path segment) but I have not had time to
> > try the additional MirrorUriTest case yet. Is this the desired solution
> > for wget based fetchers?
>
> The timing of this is uncanny. I actually spent an hour this weekend
> looking at an unrelated bug associated with `downloadfilename`
> (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15126).
>
> The related test code actually demonstrates some existing confusion
> around `downloadfilename` that could result in some very confusing
> results. Take a look at
> test_fetch_premirror_use_downloadfilename_to_fetch(). The only reason
> this test passes is because the specified `downloadfilename`
> `bitbake-1.0.tar.gz` actually exists in the premirror. The test really
> should use a random `downloadfilename` such as `bitcook-1.0.tar.gz`
> instead of something that can actually exist.
>
> If the behavior is left to stand this way it could definitely lead to
> head scratching as it would be difficult to understand why checksums
> would be failing since what is downloaded is not what is expected.
>
> My thinking was that `downloadfilename` should only come into play when
> fetching from DL_DIR and not when fetching from a PREMIRRORS or MIRRORS,
> but I didn't get that far into reviewing the issue to make any conclusions.
>
> At any rate, I wanted to speak up considering the current discussion is
> around handling of `downloadfilename` and this might just result in even
> more confusion on how it should be handled.
I'm now travelling so I'm not going to get as much time to think about
things as I'd like. I did want to highlight that in theory you can use
DL_DIR as a mirror/premirror pretty directly so the naming does need to
match across them.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2109):
https://lists.openembedded.org/g/openembedded-architecture/message/2109
Mute This Topic: https://lists.openembedded.org/mt/110806536/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
