On Fri, 5 Nov 2021 11:31:34 GMT, Andrew Leonard <aleon...@openjdk.org> wrote:
>> This PR enables reproducible Jars, Jmods and openjdk image zip files >> (eg.src.zip). >> It provides support for SOURCE_DATE_EPOCH for Jar, Jmod and underlying >> ZipOutputStream's. >> It fixes the following keys issues relating to reproducibility: >> - Jar and ZipOutputStream are not SOURCE_DATE_EPOCH aware >> - Jar and ZipOutputStream now detect SOURCE_DATE_EPOCH environment setting >> - Jar and Jmod file content ordering was non-determinsitic >> - Fixes to Jar and Jmod main's to ensure sorted classes content ordering >> - openjdk image zip file generation used "zip" which is non-determinsitic >> - New openjdk build tool "GenerateZip" which produces the final >> determinsitic zip files as part of the build and also detects >> SOURCE_DATE_EPOCH >> >> Signed-off-by: Andrew Leonard <anleo...@redhat.com> > >> I think this is going to require discussion, a PR may be too premature. Is >> your goal to get the JDK build and run-time images creates with jlink to use >> the SOURCE_DATE_EPOCH? Are you looking for project builds that use the >> jar/jmod/etc. tools to use it? Or are you looking to have every library and >> application that uses the java.util.zip APIs to read it? If it includes the >> latter, and the patch in the PR suggests you are, then it will require >> analysis to work through the API spec changes. >> >> One suggestion is to look at JDK-8231640 and PR 5372 [1]. The original >> proposal was to use SOURCE_DATE_EPOCH. After a lengthy discussion we >> converged on the system property java.properties.date rather than the env >> variable. I suspect that much of that discussion applies here. >> >> -Alan >> >> [1] https://github.com/[/pull/5372](https://github.com/openjdk/jdk/pull/5372) > > @AlanBateman Hi Alan, thank you for the comments. So yes, totally agree this > has a way to go, I thought creating a PR would kick things off...! So just to > clarify the initial objectives which you raise above. My current aim is > actually purely an OpenJDK JDK image perspective, building on the work > @magicus has been doing with SOURCE_DATE_EPOCH in the build, hence my natural > approach to this PR. So from that perspective, it's purely a "build" change. > However, the openjdk build obviously uses openjdk Jar, Jmod tooling to build > itself, hence the direction I took in this PR. > > Magnus's PR #5372 is useful thank you, I had not discovered that change, as > you say the direction there is similar to mine, and I understand the > potential pitfalls of adding doPrivileged's etc. So that resolved to a new > system property java.properties.date, which makes sense. > > So with that in mind, and given the objective of JDK image reproducibility, > this would imply presumably an approach of a new cmd line option to Jar and > Jmod tools (eg.--source-date <date>), since these tools are cmds? > > One of the key changes in this PR is to ZipOutputStream where it sets the > ZipEntry.xdostime if not set, to the current time. Now I am thinking given > the "objective", if we fix all upstream tooling to ensure they set ZipEntry > times, using "--source-date" (or whatever mechanism), then we can then avoid > changing ZipOutputStream I believe. JDK tools like jar, jmod generate/update > extra files like Manifests, and in fact I think the current jmod > implementation writes all its content using the current time. > > So something along these lines: > jar -cf myjar.jar --source-date "<date>" * > jmod create --source-date "<date>" ... my.jmod > The jar, jmod,.. tooling then ensure all ZipEntry times are set to > "source-date". > > I chose "source-date" name based on being synonymous with SOURCE_DATE_EPOCH, > implying a similar "use case". > > So taking this approach for JDK tooling should achieve the reproducible JDK > image objective, without a java API behavior change I am hoping. > Thoughts? > > Regards > Andrew @andrew-m-leonard Just to be clear: changing arguments to the command line utilities is still an API change and will need a CSR. My gut feeling is that using a system property that affect all users of ZipStream is preferable. That way a user could run like `java -Djava.properties.date=$(SOURCE_DATE_EPOCH) -Djava.zipfile.date=$(SOURCE_DATE_EPOCH) ...` (or whatever) and get reproducible behavior in all their code, for places that could otherwise be hard to fix. ------------- PR: https://git.openjdk.java.net/jdk/pull/6268
