Sebb, I think it would be great to see some result analysis on security/integrity of the proposal. I wonder, did you make any analysis on the integrity aspect? What's the result of it?
I believe originally the requirement for separating file downloads and sha/checksum were to make sure that the checksum/sha files are always served directly from the fully apache-controlled delivery chain. Previously, when there were mirrors, this was the only way to make sure you could validate the provenience of the files - the .sha and .sum files were always served from "https://download.a.o"; so - unless your local certificate authority source were compromised, you could be sure that the downloaded files were not tampered with. It's all too easy to modify both - the file and their checksum/sha to "pretend it is valid". I understand the main reason why this was done this way was that directly the Apache-owned and fully controlled servers were able to serve all those files - because they are very small and can be served directly by the Apache server under the infra control, while using mirrors/cdn is absolutely necessary to serve the binary content which is huge, and we cannot really route all the users to hit directly the apache download server. This is at least when I tried to understand the context and reasoning why it was implemented this way. I am not sure how dlcdn works and how "trustful" it is for the Apache to be 100% sure the content has not been tampered with by the CDN provider. Do you know what is the consequence of your proposal on the integrity check in case they are part of close.lua? Does it mean that it will be served from a "fully owned" apache page ? Is it something that ASF can "delegate" to the CDN provider? Is it something that closer.lua can handle in the "provenience sure way" ? Is the content served by closer.lua fully served by the ASF-owned server and cannot be tampered with by the CDN provider? I do not know the answer to those questions (I do not know too much about the closer.lua scripts and what's the level of trust between the ASF and CDN) - but I am sure that when you proposed the changes, you considered the context and reasoning why it was separated - so can you tell us all if all the integrity and "provenience" aspects are handled well by your proposal? J. On Sun, Apr 17, 2022 at 1:06 PM sebb <seb...@gmail.com> wrote: > > On Sun, 17 Apr 2022 at 11:56, sebb <seb...@gmail.com> wrote: > > > > On Sun, 17 Apr 2022 at 10:36, Greg Stein <gst...@gmail.com> wrote: > > > > > > Hi Craig, > > > > > > The closer.lua, its prior name closer.cgi. and all predecessors have been > > > fine with the Foundation's consumers liking its functionality. This has > > > supported 200+ projects (or more, depending on how you'd like to count). > > > Changes to that system are (thus) demand-based, which we have not seen. > > > Please provide some PRs with the changes you would like to make. > > > > > > I do believe that others agree with your ideas on "automated hashes", > > > though I really have no idea on the follow-on security details. Nor a > > > mechanism for broad-based security chains of GPG keys and hashes for the > > > artifacts. > > > > My idea here was just to link to the sig and hash by adding the > > appropriate suffix to the artifact path name, and checking that it > > exists. > > e.g. look for db/jdo/3.2/jdo-3.2-source-release.tar.gz + .asc also > > .sha256 or .sha512 etc. > > The code already checks for the presence of the artifact. > > > > There aren't that many valid hash types to check; if the code cannot > > find the required files, it could direct users back to the project. > > > > > Figure it out, discuss on the mailing list, file some Pull-Requests, and > > > we can move forward. > > It looks like the code is currently at: > > https://github.com/apache/infrastructure-p6/blob/production/modules/closer_cgi/files/closer.lua > (requires GitHub login linked to ASF login with appropriate karma) > > > > Cheers, > > > Greg > > > InfraAdmin, ASF > > > > > > > > > On Sat, Apr 16, 2022 at 6:23 PM Craig Russell <apache....@gmail.com> > > > wrote: > > >> > > >> I'd like to ask on behalf of the DB PMC for the infra tool closer.lua to > > >> be enhanced a bit for use by projects. > > >> > > >> It will be much easier for us if we can simply link to e.g. > > >> https://www.apache.org/dyn/closer.lua/db/jdo/3.2/jdo-3.2-source-release.tar.gz > > >> and have that page contain the .asc and .sha512 as well. > > >> > > >> Even better if closer.lua can also find the relevant KEYS file and link > > >> to it when describing how to check the release. > > >> > > >> With this approach, it is easier for the project and less error-prone to > > >> maintain the download page(s). We can include the current release as > > >> well as archived releases on the same download page and omit the links > > >> to the .asc and .sha512 and KEYS files. > > >> > > >> Please let us know how we can help. > > >> > > >> Regards, > > >> Craig > > >> > > >> > > >> On Apr 16, 2022, at 2:01 AM, Greg Stein <gst...@gmail.com> wrote: > > >> > > >> On Thu, Apr 14, 2022 at 4:43 PM sebb <seb...@gmail.com> wrote: > > >>> > > >>> On Thu, 14 Apr 2022 at 20:52, Christopher <ctubb...@apache.org> wrote: > > >>> > > > >>> > Doing this would probably restrict INFRA's ability to adapt it to > > >>> > their changing needs, as projects would become dependent on it. > > >>> > > >>> Projects are already dependent on the page. > > >> > > >> > > >> Correct. We *want* projects to use closer.lua. It provides a control > > >> point to direct users to the appropriate location to download. Today, it > > >> primarily sends them to our CDN and to an EU download location. > > >> > > >> Should those decisions ever change, we want projects to be using > > >> closer.lua to effect those changes. > > >> > > >>> > > >>> I think it would make it easier for Infra to make changes, as they > > >>> would be able to adjust it. > > >> > > >> > > >> Yes. > > >> > > >>> > > >>> > Probably best to leave the mirror-management page independent from the > > >>> > project download pages, since they serve the needs of separate groups. > > >>> > > >>> I was not suggesting replacing these, for those projects that want to > > >>> customise the pages. > > >> > > >> > > >> Today, we no longer have a mirror system. But requiring projects to use > > >> closer.lua ensures that we can swap in that future option. > > >> > > >> All that said, as I recall: closer.lua provides support for .ezt pages > > >> so that projects can provide custom download pages. Maybe there is a way > > >> to provide the needed variables to projects' custom download page > > >> templates. > > >> > > >> Or, they can just keep using their pages. > > >> > > >> The download pages are owned by the TLPs. Infra isn't gonna interfere > > >> with them. Should any TLPs want more features, then they can ask. We > > >> haven't seen any requests in years from the TLPs. > > >> > > >> Cheers, > > >> Greg > > >> InfraAdmin, ASF > > >> > > >> > > >> > > >> Craig L Russell > > >> c...@apache.org > > >>
