On Mon, 18 Apr 2022 at 04:28, Craig Russell <apache....@gmail.com> wrote: > > > > On Apr 17, 2022, at 4:02 PM, sebb <seb...@gmail.com> wrote: > > On Sun, 17 Apr 2022 at 23:31, Christopher <ctubb...@apache.org> wrote: > > > I don't think this is proposing to have links to the > checksums/signatures on the CDN, but rather to have the closer.lua > template, which is still controlled by INFRA, include links to those > files at the recommended location at downloads.apache.org. closer.lua > can easily link to $x at dlcdn.apache.org, but $x.sha512 and $x.asc > and KEYS at downloads.apache.org. > > > Yes, the idea is to extend the existing page to create a minimal valid > download page for projects to use. > The links to KEYS, sigs and hashes would use the same URLs as a > regular download page > > > Finding the .asc and either .sha256 or .sha512 (or both) is relatively > straightforward. As Sebb noted earlier, finding the KEYS file may be a bit > harder. > > > FTR, I have got a working Docker image which I will upload shortly. >
https://github.com/apache/infrastructure-p6/tree/sebb/closer Clone the repo branch cd modules/closer_cgi Read comments at end of Dockerfile > Looking forward to seeing the patch to closer.lua and closer.html. > > Craig > > > On Sun, Apr 17, 2022 at 5:46 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Sebb, > > I think it would be great to see some result analysis on > security/integrity of the proposal. I wonder, did you make any > analysis on the integrity aspect? What's the result of it? > > I believe originally the requirement for separating file downloads and > sha/checksum were to make sure that the checksum/sha files are always > served directly from the fully apache-controlled delivery chain. > Previously, when there were mirrors, this was the only way to make > sure you could validate the provenience of the files - the .sha and > .sum files were always served from "https://download.a.o"; so - unless > your local certificate authority source were compromised, you could be > sure that the downloaded files were not tampered with. It's all too > easy to modify both - the file and their checksum/sha to "pretend it > is valid". > > I understand the main reason why this was done this way was that > directly the Apache-owned and fully controlled servers were able to > serve all those files - because they are very small and can be served > directly by the Apache server under the infra control, while using > mirrors/cdn is absolutely necessary to serve the binary content which > is huge, and we cannot really route all the users to hit directly the > apache download server. This is at least when I tried to understand > the context and reasoning why it was implemented this way. > > I am not sure how dlcdn works and how "trustful" it is for the Apache > to be 100% sure the content has not been tampered with by the CDN > provider. > Do you know what is the consequence of your proposal on the integrity > check in case they are part of close.lua? Does it mean that it will be > served from a "fully owned" apache page ? Is it something that ASF can > "delegate" to the CDN provider? Is it something that closer.lua can > handle in the "provenience sure way" ? Is the content served by > closer.lua fully served by the ASF-owned server and cannot be tampered > with by the CDN provider? > > I do not know the answer to those questions (I do not know too much > about the closer.lua scripts and what's the level of trust between the > ASF and CDN) - but I am sure that when you proposed the changes, you > considered the context and reasoning why it was separated - so can you > tell us all if all the integrity and "provenience" aspects are handled > well by your proposal? > > J. > > On Sun, Apr 17, 2022 at 1:06 PM sebb <seb...@gmail.com> wrote: > > > On Sun, 17 Apr 2022 at 11:56, sebb <seb...@gmail.com> wrote: > > > On Sun, 17 Apr 2022 at 10:36, Greg Stein <gst...@gmail.com> wrote: > > > Hi Craig, > > The closer.lua, its prior name closer.cgi. and all predecessors have been > fine with the Foundation's consumers liking its functionality. This has > supported 200+ projects (or more, depending on how you'd like to count). > Changes to that system are (thus) demand-based, which we have not seen. > Please provide some PRs with the changes you would like to make. > > I do believe that others agree with your ideas on "automated hashes", though > I really have no idea on the follow-on security details. Nor a mechanism for > broad-based security chains of GPG keys and hashes for the artifacts. > > > My idea here was just to link to the sig and hash by adding the > appropriate suffix to the artifact path name, and checking that it > exists. > e.g. look for db/jdo/3.2/jdo-3.2-source-release.tar.gz + .asc also > .sha256 or .sha512 etc. > The code already checks for the presence of the artifact. > > There aren't that many valid hash types to check; if the code cannot > find the required files, it could direct users back to the project. > > Figure it out, discuss on the mailing list, file some Pull-Requests, and we > can move forward. > > > It looks like the code is currently at: > > https://github.com/apache/infrastructure-p6/blob/production/modules/closer_cgi/files/closer.lua > (requires GitHub login linked to ASF login with appropriate karma) > > Cheers, > Greg > InfraAdmin, ASF > > > On Sat, Apr 16, 2022 at 6:23 PM Craig Russell <apache....@gmail.com> wrote: > > > I'd like to ask on behalf of the DB PMC for the infra tool closer.lua to be > enhanced a bit for use by projects. > > It will be much easier for us if we can simply link to e.g. > https://www.apache.org/dyn/closer.lua/db/jdo/3.2/jdo-3.2-source-release.tar.gz > and have that page contain the .asc and .sha512 as well. > > Even better if closer.lua can also find the relevant KEYS file and link to it > when describing how to check the release. > > With this approach, it is easier for the project and less error-prone to > maintain the download page(s). We can include the current release as well as > archived releases on the same download page and omit the links to the .asc > and .sha512 and KEYS files. > > Please let us know how we can help. > > Regards, > Craig > > > On Apr 16, 2022, at 2:01 AM, Greg Stein <gst...@gmail.com> wrote: > > On Thu, Apr 14, 2022 at 4:43 PM sebb <seb...@gmail.com> wrote: > > > On Thu, 14 Apr 2022 at 20:52, Christopher <ctubb...@apache.org> wrote: > > > Doing this would probably restrict INFRA's ability to adapt it to > their changing needs, as projects would become dependent on it. > > > Projects are already dependent on the page. > > > > Correct. We *want* projects to use closer.lua. It provides a control point to > direct users to the appropriate location to download. Today, it primarily > sends them to our CDN and to an EU download location. > > Should those decisions ever change, we want projects to be using closer.lua > to effect those changes. > > > I think it would make it easier for Infra to make changes, as they > would be able to adjust it. > > > > Yes. > > > Probably best to leave the mirror-management page independent from the > project download pages, since they serve the needs of separate groups. > > > I was not suggesting replacing these, for those projects that want to > customise the pages. > > > > Today, we no longer have a mirror system. But requiring projects to use > closer.lua ensures that we can swap in that future option. > > All that said, as I recall: closer.lua provides support for .ezt pages so > that projects can provide custom download pages. Maybe there is a way to > provide the needed variables to projects' custom download page templates. > > Or, they can just keep using their pages. > > The download pages are owned by the TLPs. Infra isn't gonna interfere with > them. Should any TLPs want more features, then they can ask. We haven't seen > any requests in years from the TLPs. > > Cheers, > Greg > InfraAdmin, ASF > > > > Craig L Russell > c...@apache.org > > > Craig L Russell > c...@apache.org >
