On 29/01/2025 16:27, Mark Asselstine via lists.openembedded.org wrote:
> [You don't often get email from
[email protected]. Learn why this is
important at https://aka.ms/LearnAboutSenderIdentification ]
>
> On 1/29/2025 4:18 AM, Stefan Herbrechtsmeier via lists.openembedded.org
> wrote:
>> Am 28.01.2025 um 15:47 schrieb Mark Asselstine via
lists.openembedded.org:
>>>
>>>
>>> On 1/28/2025 2:20 AM, Stefan Herbrechtsmeier via
>>> lists.openembedded.org wrote:
>>>> Am 27.01.2025 um 23:59 schrieb Martin Jansa:
>>>>> On Mon, Jan 27, 2025 at 11:11 PM Mark Asselstine via
>>>>> lists.openembedded.org
>>>>> <[email protected]> wrote:
>>>>>> On 1/27/2025 4:11 PM, Richard Purdie wrote:
>>>>>>> On Mon, 2025-01-27 at 15:26 -0500, Mark Asselstine via
>>>>>>> lists.openembedded.org wrote:
>>>>>>>> On 1/27/2025 8:12 AM, Christian Lindeberg via
>>>>>>>> lists.openembedded.org wrote:
>>>>>>>>> On 27/01/2025 13:22, Stefan Herbrechtsmeier via
>>>>>>>>> lists.openembedded.org
>>>>>>>>> wrote:
>>>>>>>>>> Am 27.01.2025 um 12:21 schrieb Stefan Herbrechtsmeier via
>>>>>>>>>> lists.openembedded.org:
>>>>>>>>>>> Am 27.01.2025 um 10:53 schrieb Richard Purdie:
>>>>>>>>>>>> On Mon, 2025-01-27 at 10:07 +0100, Stefan Herbrechtsmeier
wrote:
>>>>>>>>>>>>> Am 25.01.2025 um 14:17 schrieb Richard Purdie:
>>>>>>>>>>>>>> The gomod fetcher is totally broken for mirroring
>>>>>>>>>>>>>> unfortunately.
>>>>>>>>>>>>>> It is
>>>>>>>>>>>>>> used for the crucible recipe in meta-oe and it is one of
>>>>>>>>>>>>>> the reasons
>>>>>>>>>>>>>> the autobuilder mirroring test for meta-oe never passes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The issue is that it downloads files into subdirs of DL_DIR
>>>>>>>>>>>>>> but when
>>>>>>>>>>>>>> mirroring, that translation is lost. The files are there on
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> mirror,
>>>>>>>>>>>>>> the code just doesn't have any way to access them.
>>>>>>>>>>>>> Does this mean the downloadfilename doesn't support sub
>>>>>>>>>>>>> directories or
>>>>>>>>>>>>> is the rewrite of the URI a problem?
>>>>>>>>>>>> The rewrite is hard. The syntax doesn't support taking a path
>>>>>>>>>>>> component
>>>>>>>>>>>> and injecting it into a parameter.
>>>>>>>>>>> I assume the problem is not the path but the wrong assumption
>>>>>>>>>>> that
>>>>>>>>>>> the downloadfilename is used for the replace of the mirror url.
>>>>>>>>>>>
>>>>>>>>>>>>> Is the npm fetcher broken too?
>>>>>>>>>>>> I don't know.
>>>>>>>>>>> The npm fetcher use a npm2 sub folder in its downloadfilename.
>>>>>>>>>>>
>>>>>>>>>>>>>> I don't know what to do about this. The source mirroring
>>>>>>>>>>>>>> for meta-oe
>>>>>>>>>>>>>> has been broken for months. I've tried to fix a couple of
>>>>>>>>>>>>>> the issues
>>>>>>>>>>>>>> but this one is systemic and not easy to fix.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd note that the sanity test in sanity.bbclass doesn't
>>>>>>>>>>>>>> recognise the
>>>>>>>>>>>>>> gomod and godmogit fetchers anyway. I did try a few
>>>>>>>>>>>>>> different urls
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> due to the weird way gomod subclasses wget, the normal
>>>>>>>>>>>>>> mirror url
>>>>>>>>>>>>>> manipulation techniques don't work.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do I do?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I could drop meta-oe from source mirroring.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I could ask Khem to drop the recipe.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Neither seem like nice things to do but I don't want to do
>>>>>>>>>>>>>> "partial"
>>>>>>>>>>>>>> mirrors. Explaining to people that it may or may not work
>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>> really
>>>>>>>>>>>>>> bad and I'm not doing it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there anyone willing to help fix this? Only stopping it
>>>>>>>>>>>>>> running at
>>>>>>>>>>>>>> all is really within my own control.
>>>>>>>>>>>>> I can look into it. Would it be okay to init the urldata
before
>>>>>>>>>>>>> resolve
>>>>>>>>>>>>> the mirror or should the mirror use the gomod scheme?
>>>>>>>>>>>> Currently the mirror being passed the modified url which
>>>>>>>>>>>> makes it a
>>>>>>>>>>>> challenge as it appears as a normal http url.
>>>>>>>>>>>>
>>>>>>>>>>>> I posted my workaround that made it work but that has several
>>>>>>>>>>>> issues.
>>>>>>>>>>>> It needs a new parameter to the wget fetcher, which might be
>>>>>>>>>>>> useful
>>>>>>>>>>>> elsewhere, not sure. It also requires hardcoding the go proxy
>>>>>>>>>>>> url into
>>>>>>>>>>>> the PREMIRROR which won't work for any other go proxy
servers.
>>>>>>>>>>> This only works because the downloadfilename match the path of
>>>>>>>>>>> the
>>>>>>>>>>> uri. I think it would be better to use the downloadfilename
>>>>>>>>>>> inside
>>>>>>>>>>> the build_mirroruris / uri_replace function, add the
>>>>>>>>>>> downloadfilename
>>>>>>>>>>> to the mirrortarballs or add an additional downloadfiledir
>>>>>>>>>>> parameter.
>>>>>>>>>>>
>>>>>>>>>>> The following test data for replaceuris in MirrorUriTest should
>>>>>>>>>>> reproduce the problem:
>>>>>>>>>>> ("https://somewhere.org/scope/example/1.0.0/
>>>>>>>>>>> example-1.0.0.tgz;downloadfilename=subdir/scope-
>>>>>>>>>>> example-1.0.0.tgz",
>>>>>>>>>>> "https://.*/.*","http://somewhere2.org/somedir3";)
>>>>>>>>>>> :"http://somewhere2.org/somedir3/subdir/scope-
>>>>>>>>>>> example-1.0.0.tgz;downloadfilename=subdir/scope-
>>>>>>>>>>> example-1.0.0.tgz",
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> The following patch pass the test cases with the additional
>>>>>>>>>> test data
>>>>>>>>>> above: diff --git a/bitbake/lib/bb/fetch2/__init__.py b/
>>>>>>>>>> bitbake/lib/
>>>>>>>>>> bb/fetch2/__init__.py index 1fea02ee9c..fb013160c4 100644 --- a/
>>>>>>>>>> bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/
>>>>>>>>>> __init__.py @@ -472,7 +472,7 @@ def uri_replace(ud, uri_find,
>>>>>>>>>> uri_replace, replacements, d, mirrortarball=None):
>>>>>>>>>> uri_decoded[5] = {}
>>>>>>>>>> uri_find_decoded[5] = {} elif ud.localpath and
>>>>>>>>>> ud.method.supports_checksum(ud): - basename =
>>>>>>>>>> os.path.basename(ud.localpath) + basename =
>>>>>>>>>> ud.localpath.replace(d.getVar("DL_DIR"), "").lstrip("./") if
>>>>>>>>>> basename:
>>>>>>>>>> uri_basename = os.path.basename(uri_decoded[loc]) # Prefix
with a
>>>>>>>>>> slash as a sentinel in case
>>>>>>>>>>
>>>>>>>>>> The dot in the lstrip is needed to pass the tests. This need
to be
>>>>>>>>>> resolved in an other way to avoid side-effects.
>>>>>>>>>>
>>>>>>>>> Looks like flattening the downloadfilename would solve/
>>>>>>>>> workaround the
>>>>>>>>> problem then (replace the slashes with a punctuation character
>>>>>>>>> other
>>>>>>>>> than those valid in a Go module path segment) but I have not had
>>>>>>>>> time to
>>>>>>>>> try the additional MirrorUriTest case yet. Is this the desired
>>>>>>>>> solution
>>>>>>>>> for wget based fetchers?
>>>>>>>> The timing of this is uncanny. I actually spent an hour this
weekend
>>>>>>>> looking at an unrelated bug associated with `downloadfilename`
>>>>>>>> (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15126).
>>>>>>>>
>>>>>>>> The related test code actually demonstrates some existing
confusion
>>>>>>>> around `downloadfilename` that could result in some very confusing
>>>>>>>> results. Take a look at
>>>>>>>> test_fetch_premirror_use_downloadfilename_to_fetch(). The only
>>>>>>>> reason
>>>>>>>> this test passes is because the specified `downloadfilename`
>>>>>>>> `bitbake-1.0.tar.gz` actually exists in the premirror. The test
>>>>>>>> really
>>>>>>>> should use a random `downloadfilename` such as
`bitcook-1.0.tar.gz`
>>>>>>>> instead of something that can actually exist.
>>>>>>>>
>>>>>>>> If the behavior is left to stand this way it could definitely
>>>>>>>> lead to
>>>>>>>> head scratching as it would be difficult to understand why
checksums
>>>>>>>> would be failing since what is downloaded is not what is expected.
>>>>>>>>
>>>>>>>> My thinking was that `downloadfilename` should only come into
>>>>>>>> play when
>>>>>>>> fetching from DL_DIR and not when fetching from a PREMIRRORS or
>>>>>>>> MIRRORS,
>>>>>>>> but I didn't get that far into reviewing the issue to make any
>>>>>>>> conclusions.
>>>>>>>>
>>>>>>>> At any rate, I wanted to speak up considering the current
>>>>>>>> discussion is
>>>>>>>> around handling of `downloadfilename` and this might just result
>>>>>>>> in even
>>>>>>>> more confusion on how it should be handled.
>>>>>>> I'm now travelling so I'm not going to get as much time to think
>>>>>>> about
>>>>>>> things as I'd like. I did want to highlight that in theory you can
>>>>>>> use
>>>>>>> DL_DIR as a mirror/premirror pretty directly so the naming does
>>>>>>> need to
>>>>>>> match across them.
>>>>>> Ya, it is a bit of a philosophical question, when does `A` actually
>>>>>> become `B`. My rationale for saying that PREMIRRORS and MIRRORS
should
>>>>>> in fact use the original name and not `downloadfilename` is that
these
>>>>>> mirrors may or may not be under the builder's control/ownership,
>>>>>> whereas
>>>>>> DL_DIR is. Maybe we need to quantify the PREMIRRORS and MIRRORS as
>>>>>> being
>>>>>> prestine or modified?
>>>>> It might depend on the use-case, but in cases like this:
>>>>> https://git.openembedded.org/meta-openembedded/commit/?
>>>>> id=070b92a3f64e2eee9388d64dc193aaa9d8c115d8
>>>>> I believe you want downloadfilename to be respected by both
DL_DIR and
>>>>> in PREMIRRORS/MIRRORS, because you don't want to (PRE)MIRROR random
>>>>> version of UCD.zip and get checksum error whenever changing PV in the
>>>>> recipe.
>>>>
>>>> I think we have different use case to support. The common use case is
>>>> to use the download folder as a PREMIRRORS and an other use case is
>>>> to use a private package manager repository or proxy. Only the first
>>>> use case should use the downloadfilename as path.
>>>
>>> Agreed.
>>>
>>>>
>>>> We could add the value of the downloadfilename parameter to the
>>>> mirrortarballs class variable and adapt the usage of its value. At
>>>> the moment the mirrortarball is only used if the scheme is changed.
>>>> Additional the code removes all parameters. We could use the
>>>> mirrortarball if the PREMIRROR doesn't use the PATH variable and only
>>>> remove the parameters if the scheme is changed (except https <->
http).
>>>>
>>>> Independent of the PREMIRROR use case we have to decide if the
>>>> downloadfilename contains a plain filename (basename) or a relative
>>>> path. At the moment parts of the code assume a plain filename
>>>> (basename) and other parts put a relative path into the
>>>> downloadfilename.
>>>>
>>>
>>> If we review the documentation I would argue that `downloadfilename`
>>> should be in effect for DL_DIR and the usecase you express above,
>>> where DL_DIR is essentially turned into a PREMIRRORS. But MIRRORS
>>> should not use `downloadfilename`. Here's the documentation I am
>>> referring to:
>>>
>>> There are two kinds of mirrors: pre-mirrors and regular
mirrors. The
>>> PREMIRRORS and MIRRORS variables point to these, respectively.
>>> BitBake checks pre-mirrors before looking upstream for any source
>>> files. Pre-mirrors are appropriate when you have a shared directory
>>> that is not a directory defined by the DL_DIR variable. A
Pre-mirror
>>> typically points to a shared directory that is local to your
>>> organization.
>>>
>>> NOTE the term `regular mirrors` and `upstream`. ie. MIRRORS are not a
>>> Yocto Project entity, they are truly a backup.
>>
>> Technically the different between pre-mirrors and mirrors is that the
>> pre-mirrors are try before and the mirrors after upstream.
>
> Understood.
>
>>
>>>
>>> A good example of this would be PyPi and a PyPi mirror (something that
>>> isn't out of the ordinary to setup).
>>
>> This is a good example for a pre-mirror with mirror upstream. A private
>> PyPi mirror should be used before the normal upstream and not only if
>> the upstream failed.
>
> But this still leaves things open. If I were to create a private PyPi
> mirror I would most likely do so using one of the various tools out
> there to create it. In that case the names used would be the same as in
> PyPi and thus I wouldn't want downloadfilename to be in play for the
fetch.
>
> MarkA
>
So if one would setup a private Go module proxy at goproxy.example.com, the
module proxy can either be pointed to by GO_MOD_PROXY or added as a
pre-mirror
with "https://proxy.golang.org/ https://goproxy.example.com/"; as the
downloadfilename for the module files are currently compatible with the path
structure specified in the GOPROXY protocol section under
https://go.dev/ref/mod#module-proxy.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2124):
https://lists.openembedded.org/g/openembedded-architecture/message/2124
Mute This Topic: https://lists.openembedded.org/mt/110806536/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
