> -----Original Message----- > From: [email protected] <openembedded- > [email protected]> On Behalf Of Mark Asselstine via > lists.openembedded.org > Sent: den 29 januari 2025 16:28 > To: [email protected]; openembedded- > [email protected] > Subject: Re: [Openembedded-architecture] gomod fetcher totally broken for > mirroring > > > > On 1/29/2025 4:18 AM, Stefan Herbrechtsmeier via lists.openembedded.org > wrote: > > Am 28.01.2025 um 15:47 schrieb Mark Asselstine via > lists.openembedded.org: > >> > >> > >> On 1/28/2025 2:20 AM, Stefan Herbrechtsmeier via > >> lists.openembedded.org wrote: > >>> Am 27.01.2025 um 23:59 schrieb Martin Jansa: > >>>> On Mon, Jan 27, 2025 at 11:11 PM Mark Asselstine via > >>>> lists.openembedded.org > >>>> <[email protected]> wrote: > >>>>> On 1/27/2025 4:11 PM, Richard Purdie wrote: > >>>>>> On Mon, 2025-01-27 at 15:26 -0500, Mark Asselstine via > >>>>>> lists.openembedded.org wrote: > >>>>>>> On 1/27/2025 8:12 AM, Christian Lindeberg via > >>>>>>> lists.openembedded.org wrote: > >>>>>>>> On 27/01/2025 13:22, Stefan Herbrechtsmeier via > >>>>>>>> lists.openembedded.org > >>>>>>>> wrote: > >>>>>>>>> Am 27.01.2025 um 12:21 schrieb Stefan Herbrechtsmeier via > >>>>>>>>> lists.openembedded.org: > >>>>>>>>>> Am 27.01.2025 um 10:53 schrieb Richard Purdie: > >>>>>>>>>>> On Mon, 2025-01-27 at 10:07 +0100, Stefan Herbrechtsmeier > wrote: > >>>>>>>>>>>> Am 25.01.2025 um 14:17 schrieb Richard Purdie: > >>>>>>>>>>>>> The gomod fetcher is totally broken for mirroring > >>>>>>>>>>>>> unfortunately. > >>>>>>>>>>>>> It is > >>>>>>>>>>>>> used for the crucible recipe in meta-oe and it is one of > >>>>>>>>>>>>> the reasons > >>>>>>>>>>>>> the autobuilder mirroring test for meta-oe never passes. > >>>>>>>>>>>>> > >>>>>>>>>>>>> The issue is that it downloads files into subdirs of DL_DIR > >>>>>>>>>>>>> but when > >>>>>>>>>>>>> mirroring, that translation is lost. The files are there on > >>>>>>>>>>>>> the > >>>>>>>>>>>>> mirror, > >>>>>>>>>>>>> the code just doesn't have any way to access them. > >>>>>>>>>>>> Does this mean the downloadfilename doesn't support sub > >>>>>>>>>>>> directories or > >>>>>>>>>>>> is the rewrite of the URI a problem? > >>>>>>>>>>> The rewrite is hard. The syntax doesn't support taking a path > >>>>>>>>>>> component > >>>>>>>>>>> and injecting it into a parameter. > >>>>>>>>>> I assume the problem is not the path but the wrong assumption > >>>>>>>>>> that > >>>>>>>>>> the downloadfilename is used for the replace of the mirror url. > >>>>>>>>>> > >>>>>>>>>>>> Is the npm fetcher broken too? > >>>>>>>>>>> I don't know. > >>>>>>>>>> The npm fetcher use a npm2 sub folder in its downloadfilename. > >>>>>>>>>> > >>>>>>>>>>>>> I don't know what to do about this. The source mirroring > >>>>>>>>>>>>> for meta-oe > >>>>>>>>>>>>> has been broken for months. I've tried to fix a couple of > >>>>>>>>>>>>> the issues > >>>>>>>>>>>>> but this one is systemic and not easy to fix. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I'd note that the sanity test in sanity.bbclass doesn't > >>>>>>>>>>>>> recognise the > >>>>>>>>>>>>> gomod and godmogit fetchers anyway. I did try a few > >>>>>>>>>>>>> different urls > >>>>>>>>>>>>> but > >>>>>>>>>>>>> due to the weird way gomod subclasses wget, the normal > >>>>>>>>>>>>> mirror url > >>>>>>>>>>>>> manipulation techniques don't work. > >>>>>>>>>>>>> > >>>>>>>>>>>>> What do I do? > >>>>>>>>>>>>> > >>>>>>>>>>>>> I could drop meta-oe from source mirroring. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I could ask Khem to drop the recipe. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Neither seem like nice things to do but I don't want to do > >>>>>>>>>>>>> "partial" > >>>>>>>>>>>>> mirrors. Explaining to people that it may or may not work > >>>>>>>>>>>>> looks > >>>>>>>>>>>>> really > >>>>>>>>>>>>> bad and I'm not doing it. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Is there anyone willing to help fix this? Only stopping it > >>>>>>>>>>>>> running at > >>>>>>>>>>>>> all is really within my own control. > >>>>>>>>>>>> I can look into it. Would it be okay to init the urldata > before > >>>>>>>>>>>> resolve > >>>>>>>>>>>> the mirror or should the mirror use the gomod scheme? > >>>>>>>>>>> Currently the mirror being passed the modified url which > >>>>>>>>>>> makes it a > >>>>>>>>>>> challenge as it appears as a normal http url. > >>>>>>>>>>> > >>>>>>>>>>> I posted my workaround that made it work but that has several > >>>>>>>>>>> issues. > >>>>>>>>>>> It needs a new parameter to the wget fetcher, which might be > >>>>>>>>>>> useful > >>>>>>>>>>> elsewhere, not sure. It also requires hardcoding the go proxy > >>>>>>>>>>> url into > >>>>>>>>>>> the PREMIRROR which won't work for any other go proxy servers. > >>>>>>>>>> This only works because the downloadfilename match the path of > >>>>>>>>>> the > >>>>>>>>>> uri. I think it would be better to use the downloadfilename > >>>>>>>>>> inside > >>>>>>>>>> the build_mirroruris / uri_replace function, add the > >>>>>>>>>> downloadfilename > >>>>>>>>>> to the mirrortarballs or add an additional downloadfiledir > >>>>>>>>>> parameter. > >>>>>>>>>> > >>>>>>>>>> The following test data for replaceuris in MirrorUriTest should > >>>>>>>>>> reproduce the problem: > >>>>>>>>>> ("https://somewhere.org/scope/example/1.0.0/ > >>>>>>>>>> example-1.0.0.tgz;downloadfilename=subdir/scope- > >>>>>>>>>> example-1.0.0.tgz", > >>>>>>>>>> "https://.*/.*","http://somewhere2.org/somedir3";) > >>>>>>>>>> :"http://somewhere2.org/somedir3/subdir/scope- > >>>>>>>>>> example-1.0.0.tgz;downloadfilename=subdir/scope- > >>>>>>>>>> example-1.0.0.tgz", > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> The following patch pass the test cases with the additional > >>>>>>>>> test data > >>>>>>>>> above: diff --git a/bitbake/lib/bb/fetch2/__init__.py b/ > >>>>>>>>> bitbake/lib/ > >>>>>>>>> bb/fetch2/__init__.py index 1fea02ee9c..fb013160c4 100644 --- a/ > >>>>>>>>> bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/ > >>>>>>>>> __init__.py @@ -472,7 +472,7 @@ def uri_replace(ud, uri_find, > >>>>>>>>> uri_replace, replacements, d, mirrortarball=None): > >>>>>>>>> uri_decoded[5] = {} > >>>>>>>>> uri_find_decoded[5] = {} elif ud.localpath and > >>>>>>>>> ud.method.supports_checksum(ud): - basename = > >>>>>>>>> os.path.basename(ud.localpath) + basename = > >>>>>>>>> ud.localpath.replace(d.getVar("DL_DIR"), "").lstrip("./") if > >>>>>>>>> basename: > >>>>>>>>> uri_basename = os.path.basename(uri_decoded[loc]) # Prefix with > a > >>>>>>>>> slash as a sentinel in case > >>>>>>>>> > >>>>>>>>> The dot in the lstrip is needed to pass the tests. This need to > be > >>>>>>>>> resolved in an other way to avoid side-effects. > >>>>>>>>> > >>>>>>>> Looks like flattening the downloadfilename would solve/ > >>>>>>>> workaround the > >>>>>>>> problem then (replace the slashes with a punctuation character > >>>>>>>> other > >>>>>>>> than those valid in a Go module path segment) but I have not had > >>>>>>>> time to > >>>>>>>> try the additional MirrorUriTest case yet. Is this the desired > >>>>>>>> solution > >>>>>>>> for wget based fetchers? > >>>>>>> The timing of this is uncanny. I actually spent an hour this > weekend > >>>>>>> looking at an unrelated bug associated with `downloadfilename` > >>>>>>> (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15126). > >>>>>>> > >>>>>>> The related test code actually demonstrates some existing > confusion > >>>>>>> around `downloadfilename` that could result in some very confusing > >>>>>>> results. Take a look at > >>>>>>> test_fetch_premirror_use_downloadfilename_to_fetch(). The only > >>>>>>> reason > >>>>>>> this test passes is because the specified `downloadfilename` > >>>>>>> `bitbake-1.0.tar.gz` actually exists in the premirror. The test > >>>>>>> really > >>>>>>> should use a random `downloadfilename` such as `bitcook- > 1.0.tar.gz` > >>>>>>> instead of something that can actually exist. > >>>>>>> > >>>>>>> If the behavior is left to stand this way it could definitely > >>>>>>> lead to > >>>>>>> head scratching as it would be difficult to understand why > checksums > >>>>>>> would be failing since what is downloaded is not what is expected. > >>>>>>> > >>>>>>> My thinking was that `downloadfilename` should only come into > >>>>>>> play when > >>>>>>> fetching from DL_DIR and not when fetching from a PREMIRRORS or > >>>>>>> MIRRORS, > >>>>>>> but I didn't get that far into reviewing the issue to make any > >>>>>>> conclusions. > >>>>>>> > >>>>>>> At any rate, I wanted to speak up considering the current > >>>>>>> discussion is > >>>>>>> around handling of `downloadfilename` and this might just result > >>>>>>> in even > >>>>>>> more confusion on how it should be handled. > >>>>>> I'm now travelling so I'm not going to get as much time to think > >>>>>> about > >>>>>> things as I'd like. I did want to highlight that in theory you can > >>>>>> use > >>>>>> DL_DIR as a mirror/premirror pretty directly so the naming does > >>>>>> need to > >>>>>> match across them. > >>>>> Ya, it is a bit of a philosophical question, when does `A` actually > >>>>> become `B`. My rationale for saying that PREMIRRORS and MIRRORS > should > >>>>> in fact use the original name and not `downloadfilename` is that > these > >>>>> mirrors may or may not be under the builder's control/ownership, > >>>>> whereas > >>>>> DL_DIR is. Maybe we need to quantify the PREMIRRORS and MIRRORS as > >>>>> being > >>>>> prestine or modified? > >>>> It might depend on the use-case, but in cases like this: > >>>> https://git.openembedded.org/meta-openembedded/commit/? > >>>> id=070b92a3f64e2eee9388d64dc193aaa9d8c115d8 > >>>> I believe you want downloadfilename to be respected by both DL_DIR > and > >>>> in PREMIRRORS/MIRRORS, because you don't want to (PRE)MIRROR random > >>>> version of UCD.zip and get checksum error whenever changing PV in the > >>>> recipe. > >>> > >>> I think we have different use case to support. The common use case is > >>> to use the download folder as a PREMIRRORS and an other use case is > >>> to use a private package manager repository or proxy. Only the first > >>> use case should use the downloadfilename as path. > >> > >> Agreed. > >> > >>> > >>> We could add the value of the downloadfilename parameter to the > >>> mirrortarballs class variable and adapt the usage of its value. At > >>> the moment the mirrortarball is only used if the scheme is changed. > >>> Additional the code removes all parameters. We could use the > >>> mirrortarball if the PREMIRROR doesn't use the PATH variable and only > >>> remove the parameters if the scheme is changed (except https <-> > http). > >>> > >>> Independent of the PREMIRROR use case we have to decide if the > >>> downloadfilename contains a plain filename (basename) or a relative > >>> path. At the moment parts of the code assume a plain filename > >>> (basename) and other parts put a relative path into the > >>> downloadfilename. > >>> > >> > >> If we review the documentation I would argue that `downloadfilename` > >> should be in effect for DL_DIR and the usecase you express above, > >> where DL_DIR is essentially turned into a PREMIRRORS. But MIRRORS > >> should not use `downloadfilename`. Here's the documentation I am > >> referring to: > >> > >> There are two kinds of mirrors: pre-mirrors and regular mirrors. > The > >> PREMIRRORS and MIRRORS variables point to these, respectively. > >> BitBake checks pre-mirrors before looking upstream for any source > >> files. Pre-mirrors are appropriate when you have a shared directory > >> that is not a directory defined by the DL_DIR variable. A Pre- > mirror > >> typically points to a shared directory that is local to your > >> organization. > >> > >> NOTE the term `regular mirrors` and `upstream`. ie. MIRRORS are not a > >> Yocto Project entity, they are truly a backup. > > > > Technically the different between pre-mirrors and mirrors is that the > > pre-mirrors are try before and the mirrors after upstream. > > Understood. > > > > >> > >> A good example of this would be PyPi and a PyPi mirror (something that > >> isn't out of the ordinary to setup). > > > > This is a good example for a pre-mirror with mirror upstream. A private > > PyPi mirror should be used before the normal upstream and not only if > > the upstream failed. > > But this still leaves things open. If I were to create a private PyPi > mirror I would most likely do so using one of the various tools out > there to create it. In that case the names used would be the same as in > PyPi and thus I wouldn't want downloadfilename to be in play for the > fetch. > > MarkA
Doesn't this mean that what we really want is something like a DL_DIR_MIRROR (or PRE_DL_DIR?), which has the same semantics as DL_DIR (basically what PREMIRROR was originally intended for)? And then PREMIRROR should behave like upstream and MIRROR with regards to, e.g., downloadfilename. //Peter
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#2122): https://lists.openembedded.org/g/openembedded-architecture/message/2122 Mute This Topic: https://lists.openembedded.org/mt/110806536/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
