Re: [DISCUSS] checking reproducible builds
makes sense to me.
Robert
On 30-5-2020 22:56:19, Hervé BOUTEMY wrote:
any objection that I create a new maven-artifact-plugin Git repository
initialized with current maven-buildinfo-plugin Git history?
Regards,
Hervé
Le mercredi 27 mai 2020, 19:26:55 CEST Robert Scholte a écrit :
> maven-studies are just a sandbox, experimental code. Once it has a good
> shape, it can be promoted to a separate project. So no, we're not going to
> release the maven-buildinfo-plugin.
>
> Robert
> On 26-5-2020 23:17:29, Konrad Windszus wrote:
> As creating a new maven-artifact-plugin will probably take some time, maybe
> it would be possible to push a release build of
> https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin to
> Maven Central. Or is there already a rough schedule for coming up with the
> new maven-artifact-plugin?
>
> Thanks,
> Konrad
>
> On 2020/03/08 20:04:56, "Robert Scholte" wrote:
> > I'm thinking of maven-artifact-plugin, having goals related to artifacts.
> > That implies that the save goal should be renamed.
> > A couple of goals of the maven-dependency-plugin are actually more
> > artifact-related are might be worth moving.
> >
> > Robert
> >
> > On 8-3-2020 13:44:07, Michael Osipov wrote:
> >
> > Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> > >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> > >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> > This is expected because I am on 1.8.0_242. I don't have Java 7
> > installed anymore on the server.
> > >>>
> > >>> for the discussion I wanted us to have, just being able to test and
> > >>> see
> > >>> how we detect issues, this is perfect, isn't it?
> > >>
> > >> This is really nice. Here is the diffoscope output:
> > > you're discovering the wonders of diffoscope :)
> > >
> > >>> --- maven-site-plugin-3.9.0.jar
> > >>> +++ reference/maven-site-plugin-3.9.0.jar
> > >>> ├── zipinfo {}
> > >>> │ @@ -1,8 +1,8 @@
> > >
> > > [...]
> > >
> > >>> META-INF/MANIFEST.MF
> > >>> │ @@ -1,10 +1,10 @@
> > >>> │ Manifest-Version: 1.0
> > >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> > >>> │ Implementation-Title: Apache Maven Site Plugin
> > >>> │ Implementation-Version: 3.9.0
> > >>> │ +Build-Jdk-Spec: 1.7^M
> > >>> │ Specification-Vendor: The Apache Software Foundation
> > >>> │ -Specification-Title: Apache Maven Site Plugin^M
> > >>> │ -Build-Jdk-Spec: 1.8^M
> > >>> │ Created-By: Maven Jar Plugin 3.2.0
> > >>> │ +Specification-Title: Apache Maven Site Plugin^M
> > >>> │ Specification-Version: 3.9
> > >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> > >>
> > >> I wonder where the CRs code from...this could be the default
> > >> serialization format on every platform.
> > >
> > > FYI I don't have such CRs in output on my Linux box
> >
> > This cannot be. See
> > https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/s
> > hare/classes/java/util/jar/Manifest.java and search for \r\n. Old Sun code
> > uses *always* CRLF. Plase recheck on your side and run a hexdump on the
> > Manifest file.
> >
> > >>> how did you find the experience? any improvement proposal?
> > >>> and any idea on where to put this goal in the future?
> > >>
> > >> There is room for improvement when I quickly read the code. I will
> > >> write
> > >> separately on this.
> > >
> > > sure, code can be improved: don't hesitate
> > > but I was not asking yet for code improvement (I'm confident, it will
> > > happen) but *experience* improvement
> > >
> > >> I'd leave as a plugin for now.
> > >
> > > you mean a separate plugin? same "buildinfo" name as current? "save"
> > > goal
> > > name?
> >
> > OK, let's talk about experience:
> >
> > * buildinfo may be changed to broader name, e.g.,
> > maven-reproducibility-plugin. Explanain follows
> > * 'save' does too much. It should save only and not compare. Save should
> > either run at initialize or at build-resources phase, imho
> > * Add a 'compare' goal, not phase bound. It performs the actual
> > comparsion.
> >
> > Strictly speaking if the plugin is called buildinfo it should handle the
> > buildinfo files only.
> >
> > >> At least in 3.7.x.
> > >
> > > 3.7.x as Maven 3.7.x?
> > > does that mean that you think it should be one day integrated into Maven
> > > core? what's the rationale?
> >
> > Not really, but if this happens, not before 4.x. I don't have any
> > rationale or entry point for this yet.
> >
> > Michael
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> > For additional commands, e-mail: dev-h...@maven.apache.org
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
-
To
Re: [DISCUSS] checking reproducible builds
any objection that I create a new maven-artifact-plugin Git repository
initialized with current maven-buildinfo-plugin Git history?
Regards,
Hervé
Le mercredi 27 mai 2020, 19:26:55 CEST Robert Scholte a écrit :
> maven-studies are just a sandbox, experimental code. Once it has a good
> shape, it can be promoted to a separate project. So no, we're not going to
> release the maven-buildinfo-plugin.
>
> Robert
> On 26-5-2020 23:17:29, Konrad Windszus wrote:
> As creating a new maven-artifact-plugin will probably take some time, maybe
> it would be possible to push a release build of
> https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin to
> Maven Central. Or is there already a rough schedule for coming up with the
> new maven-artifact-plugin?
>
> Thanks,
> Konrad
>
> On 2020/03/08 20:04:56, "Robert Scholte" wrote:
> > I'm thinking of maven-artifact-plugin, having goals related to artifacts.
> > That implies that the save goal should be renamed.
> > A couple of goals of the maven-dependency-plugin are actually more
> > artifact-related are might be worth moving.
> >
> > Robert
> >
> > On 8-3-2020 13:44:07, Michael Osipov wrote:
> >
> > Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> > >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> > >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> > This is expected because I am on 1.8.0_242. I don't have Java 7
> > installed anymore on the server.
> > >>>
> > >>> for the discussion I wanted us to have, just being able to test and
> > >>> see
> > >>> how we detect issues, this is perfect, isn't it?
> > >>
> > >> This is really nice. Here is the diffoscope output:
> > > you're discovering the wonders of diffoscope :)
> > >
> > >>> --- maven-site-plugin-3.9.0.jar
> > >>> +++ reference/maven-site-plugin-3.9.0.jar
> > >>> ├── zipinfo {}
> > >>> │ @@ -1,8 +1,8 @@
> > >
> > > [...]
> > >
> > >>> META-INF/MANIFEST.MF
> > >>> │ @@ -1,10 +1,10 @@
> > >>> │ Manifest-Version: 1.0
> > >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> > >>> │ Implementation-Title: Apache Maven Site Plugin
> > >>> │ Implementation-Version: 3.9.0
> > >>> │ +Build-Jdk-Spec: 1.7^M
> > >>> │ Specification-Vendor: The Apache Software Foundation
> > >>> │ -Specification-Title: Apache Maven Site Plugin^M
> > >>> │ -Build-Jdk-Spec: 1.8^M
> > >>> │ Created-By: Maven Jar Plugin 3.2.0
> > >>> │ +Specification-Title: Apache Maven Site Plugin^M
> > >>> │ Specification-Version: 3.9
> > >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> > >>
> > >> I wonder where the CRs code from...this could be the default
> > >> serialization format on every platform.
> > >
> > > FYI I don't have such CRs in output on my Linux box
> >
> > This cannot be. See
> > https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/s
> > hare/classes/java/util/jar/Manifest.java and search for \r\n. Old Sun code
> > uses *always* CRLF. Plase recheck on your side and run a hexdump on the
> > Manifest file.
> >
> > >>> how did you find the experience? any improvement proposal?
> > >>> and any idea on where to put this goal in the future?
> > >>
> > >> There is room for improvement when I quickly read the code. I will
> > >> write
> > >> separately on this.
> > >
> > > sure, code can be improved: don't hesitate
> > > but I was not asking yet for code improvement (I'm confident, it will
> > > happen) but *experience* improvement
> > >
> > >> I'd leave as a plugin for now.
> > >
> > > you mean a separate plugin? same "buildinfo" name as current? "save"
> > > goal
> > > name?
> >
> > OK, let's talk about experience:
> >
> > * buildinfo may be changed to broader name, e.g.,
> > maven-reproducibility-plugin. Explanain follows
> > * 'save' does too much. It should save only and not compare. Save should
> > either run at initialize or at build-resources phase, imho
> > * Add a 'compare' goal, not phase bound. It performs the actual
> > comparsion.
> >
> > Strictly speaking if the plugin is called buildinfo it should handle the
> > buildinfo files only.
> >
> > >> At least in 3.7.x.
> > >
> > > 3.7.x as Maven 3.7.x?
> > > does that mean that you think it should be one day integrated into Maven
> > > core? what's the rationale?
> >
> > Not really, but if this happens, not before 4.x. I don't have any
> > rationale or entry point for this yet.
> >
> > Michael
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> > For additional commands, e-mail: dev-h...@maven.apache.org
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail:
Re: [DISCUSS] checking reproducible builds
maven-studies are just a sandbox, experimental code. Once it has a good shape,
it can be promoted to a separate project.
So no, we're not going to release the maven-buildinfo-plugin.
Robert
On 26-5-2020 23:17:29, Konrad Windszus wrote:
As creating a new maven-artifact-plugin will probably take some time, maybe it
would be possible to push a release build of
https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin to Maven
Central. Or is there already a rough schedule for coming up with the new
maven-artifact-plugin?
Thanks,
Konrad
On 2020/03/08 20:04:56, "Robert Scholte" wrote:
> I'm thinking of maven-artifact-plugin, having goals related to artifacts.
> That implies that the save goal should be renamed.
> A couple of goals of the maven-dependency-plugin are actually more
> artifact-related are might be worth moving.
>
> Robert
>
> On 8-3-2020 13:44:07, Michael Osipov wrote:
> Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> This is expected because I am on 1.8.0_242. I don't have Java 7
> installed anymore on the server.
> >>>
> >>> for the discussion I wanted us to have, just being able to test and see
> >>> how we detect issues, this is perfect, isn't it?
> >>
> >> This is really nice. Here is the diffoscope output:
> > you're discovering the wonders of diffoscope :)
> >
> >>> --- maven-site-plugin-3.9.0.jar
> >>> +++ reference/maven-site-plugin-3.9.0.jar
> >>> ├── zipinfo {}
> >>> │ @@ -1,8 +1,8 @@
> > [...]
> >>> META-INF/MANIFEST.MF
> >>> │ @@ -1,10 +1,10 @@
> >>> │ Manifest-Version: 1.0
> >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> >>> │ Implementation-Title: Apache Maven Site Plugin
> >>> │ Implementation-Version: 3.9.0
> >>> │ +Build-Jdk-Spec: 1.7^M
> >>> │ Specification-Vendor: The Apache Software Foundation
> >>> │ -Specification-Title: Apache Maven Site Plugin^M
> >>> │ -Build-Jdk-Spec: 1.8^M
> >>> │ Created-By: Maven Jar Plugin 3.2.0
> >>> │ +Specification-Title: Apache Maven Site Plugin^M
> >>> │ Specification-Version: 3.9
> >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> >>
> >> I wonder where the CRs code from...this could be the default
> >> serialization format on every platform.
> > FYI I don't have such CRs in output on my Linux box
>
> This cannot be. See
> https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/share/classes/java/util/jar/Manifest.java
> and search for \r\n. Old Sun code uses *always* CRLF. Plase recheck on
> your side and run a hexdump on the Manifest file.
>
> >>> how did you find the experience? any improvement proposal?
> >>> and any idea on where to put this goal in the future?
> >>
> >> There is room for improvement when I quickly read the code. I will write
> >> separately on this.
> > sure, code can be improved: don't hesitate
> > but I was not asking yet for code improvement (I'm confident, it will
> > happen)
> > but *experience* improvement
> >
> >> I'd leave as a plugin for now.
> > you mean a separate plugin? same "buildinfo" name as current? "save" goal
> > name?
>
> OK, let's talk about experience:
>
> * buildinfo may be changed to broader name, e.g.,
> maven-reproducibility-plugin. Explanain follows
> * 'save' does too much. It should save only and not compare. Save should
> either run at initialize or at build-resources phase, imho
> * Add a 'compare' goal, not phase bound. It performs the actual comparsion.
>
> Strictly speaking if the plugin is called buildinfo it should handle the
> buildinfo files only.
>
> >> At least in 3.7.x.
> > 3.7.x as Maven 3.7.x?
> > does that mean that you think it should be one day integrated into Maven
> > core?
> > what's the rationale?
>
> Not really, but if this happens, not before 4.x. I don't have any
> rationale or entry point for this yet.
>
> Michael
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
>
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
As creating a new maven-artifact-plugin will probably take some time, maybe it
would be possible to push a release build of
https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin to Maven
Central. Or is there already a rough schedule for coming up with the new
maven-artifact-plugin?
Thanks,
Konrad
On 2020/03/08 20:04:56, "Robert Scholte" wrote:
> I'm thinking of maven-artifact-plugin, having goals related to artifacts.
> That implies that the save goal should be renamed.
> A couple of goals of the maven-dependency-plugin are actually more
> artifact-related are might be worth moving.
>
> Robert
>
> On 8-3-2020 13:44:07, Michael Osipov wrote:
> Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> This is expected because I am on 1.8.0_242. I don't have Java 7
> installed anymore on the server.
> >>>
> >>> for the discussion I wanted us to have, just being able to test and see
> >>> how we detect issues, this is perfect, isn't it?
> >>
> >> This is really nice. Here is the diffoscope output:
> > you're discovering the wonders of diffoscope :)
> >
> >>> --- maven-site-plugin-3.9.0.jar
> >>> +++ reference/maven-site-plugin-3.9.0.jar
> >>> ├── zipinfo {}
> >>> │ @@ -1,8 +1,8 @@
> > [...]
> >>> META-INF/MANIFEST.MF
> >>> │ @@ -1,10 +1,10 @@
> >>> │ Manifest-Version: 1.0
> >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> >>> │ Implementation-Title: Apache Maven Site Plugin
> >>> │ Implementation-Version: 3.9.0
> >>> │ +Build-Jdk-Spec: 1.7^M
> >>> │ Specification-Vendor: The Apache Software Foundation
> >>> │ -Specification-Title: Apache Maven Site Plugin^M
> >>> │ -Build-Jdk-Spec: 1.8^M
> >>> │ Created-By: Maven Jar Plugin 3.2.0
> >>> │ +Specification-Title: Apache Maven Site Plugin^M
> >>> │ Specification-Version: 3.9
> >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> >>
> >> I wonder where the CRs code from...this could be the default
> >> serialization format on every platform.
> > FYI I don't have such CRs in output on my Linux box
>
> This cannot be. See
> https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/share/classes/java/util/jar/Manifest.java
> and search for \r\n. Old Sun code uses *always* CRLF. Plase recheck on
> your side and run a hexdump on the Manifest file.
>
> >>> how did you find the experience? any improvement proposal?
> >>> and any idea on where to put this goal in the future?
> >>
> >> There is room for improvement when I quickly read the code. I will write
> >> separately on this.
> > sure, code can be improved: don't hesitate
> > but I was not asking yet for code improvement (I'm confident, it will
> > happen)
> > but *experience* improvement
> >
> >> I'd leave as a plugin for now.
> > you mean a separate plugin? same "buildinfo" name as current? "save" goal
> > name?
>
> OK, let's talk about experience:
>
> * buildinfo may be changed to broader name, e.g.,
> maven-reproducibility-plugin. Explanain follows
> * 'save' does too much. It should save only and not compare. Save should
> either run at initialize or at build-resources phase, imho
> * Add a 'compare' goal, not phase bound. It performs the actual comparsion.
>
> Strictly speaking if the plugin is called buildinfo it should handle the
> buildinfo files only.
>
> >> At least in 3.7.x.
> > 3.7.x as Maven 3.7.x?
> > does that mean that you think it should be one day integrated into Maven
> > core?
> > what's the rationale?
>
> Not really, but if this happens, not before 4.x. I don't have any
> rationale or entry point for this yet.
>
> Michael
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
>
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
yes, I saw that the main artifact is reproducible, but there are more subtle cases with attached artifacts (-sources.jar and -source-release.zip) If you build with run-its profile, you'll see that the pom.xml injected into these artifacts has less differences: there is still the current directory in it :( It seems it is caused by additional maven-invoker-plugin configuration done in run-its profile, that seems to replace original pom.xml with something generated from invoker: I did not investigate more yet, any help from maven- invoker-plugin experts appreciated FYI I tested current maven-dependency-plugin release and found that it does not suffer from this issue. Regards, Hervé Le mardi 10 mars 2020, 13:11:42 CET Michael Osipov a écrit : > Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY: > > Hi, > > > > Yesterday, I made a key step forward for Reproducible Builds with Maven: I > > wrote code to easily check that your local build produces the same > > binaries as the reference binaries published either to staging or to > > Central repository. > > > > For a live example, see the last paragraph of Maven Site Plugin vote that > > just started [1]. > > > > Process to check build output is based on a single plugin goal, currently > > named buildinfo:save [2]: 1. it creates a buildinfo file during build > > recording output fingerprints, that will eventually in the future be > > published to Central repository 2. it downloads reference artifacts > > and/or reference buildinfo and checks that the output of the local build > > is the same as the reference. > > > > Now I want to discuss: is it clear? can you test and report, please? > > > > If the feedback is positive, the next question will be: in which plugin > > should we put this goal to make a release and add it to our parent pom > > during release, so we publish reference buildinfo along our reference > > binaries to Central repository. > > > > Thanks for your feedback > > > > Regards, > > > > Hervé > > > > [1] > > https://lists.apache.org/thread.html/rd3af15d383ddceeb950cd90569e3dcdd6e5 > > a0f5d3cd653ec534b0609%40%3Cdev.maven.apache.org%3E > > > > [2] https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin > > I have now installed latest OpenJDK 7 from AdoptOpenJDK source. > > > [INFO] --- maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) @ > > maven-site-plugin --- [INFO] Saved info on build to > > /usr/home/mosipov/Projekte/maven-site-plugin/target/maven-site-plugin-3.9 > > .0.buildinfo [INFO] Checking against reference build from > > https://repository.apache.org/content/repositories/maven-1554/... > > [WARNING] Reference buildinfo file not found: it will be generated from > > downloaded reference artifacts [INFO] Minimal buildinfo generated from > > downloaded artifacts: > > /usr/home/mosipov/Projekte/maven-site-plugin/target/reference/maven-site- > > plugin-3.9.0.buildinfo [WARNING] size mismatch > > maven-site-plugin-3.9.0-source-release.zip: diffoscope > > target/reference/maven-site-plugin-3.9.0-source-release.zip > > target/maven-site-plugin-3.9.0-source-release.zip [WARNING] size mismatch > > maven-site-plugin-3.9.0-sources.jar: diffoscope > > target/reference/maven-site-plugin-3.9.0-sources.jar > > target/maven-site-plugin-3.9.0-sources.jar [WARNING] Reproducible Build > > output summary: 1 files ok, 2 different, 0 missing [WARNING] diff > > target/reference/maven-site-plugin-3.9.0.buildinfo > > target/maven-site-plugin-3.9.0.buildinfo > on > > > Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > > Maven home: /usr/local/share/java/maven > > Java version: 1.7.0_251, vendor: Oracle Corporation, runtime: > > /usr/local/openjdk7/jre Default locale: de_DE, platform encoding: UTF-8 > > OS name: "freebsd", version: "11.3-release-p6", arch: "i386", family: > > "unix" > and > > > $ git branch > > * (HEAD losgelöst bei maven-site-plugin-3.9.0) > > > >> diffoscope target/reference/maven-site-plugin-3.9.0-source-release.zip > >> target/maven-site-plugin-3.9.0-source-release.zip> > > There is a diff in maven-site-plugin-3.9.0/dependency-reduced-pom.xml > > > >> diffoscope target/reference/maven-site-plugin-3.9.0-sources.jar > >> target/maven-site-plugin-3.9.0-sources.jar> > > So is here diff in the pom.xml which is actually > > dependency-reduced-pom.xml. > > > > ├── META-INF/maven/org.apache.maven.plugins/maven-site-plugin/pom.xml > > │ ├── META-INF/maven/org.apache.maven.plugins/maven-site-plugin/pom.xml > > │ │ @@ -243,100 +243,40 @@ > > │ │ > > │ │ > > │ │run-its > > │ │ > > │ │ > > │ │ > > │ │ maven-invoker-plugin > > │ │ -3.2.1 > > │ │ - > > │ │ - > > │ │ -integration-test > > │ │ - > > │ │ - install > > │ │ - integration-test > > │ │ - verify > > │ │ - > > │ │ - > > │ │ -
Re: [DISCUSS] checking reproducible builds
Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY:
Hi,
Yesterday, I made a key step forward for Reproducible Builds with Maven: I
wrote code to easily check that your local build produces the same binaries as
the reference binaries published either to staging or to Central repository.
For a live example, see the last paragraph of Maven Site Plugin vote that just
started [1].
Process to check build output is based on a single plugin goal, currently named
buildinfo:save [2]:
1. it creates a buildinfo file during build recording output fingerprints, that
will eventually in the future be published to Central repository
2. it downloads reference artifacts and/or reference buildinfo and checks that
the output of the local build is the same as the reference.
Now I want to discuss: is it clear? can you test and report, please?
If the feedback is positive, the next question will be: in which plugin should
we put this goal to make a release and add it to our parent pom during release,
so we publish reference buildinfo along our reference binaries to Central
repository.
Thanks for your feedback
Regards,
Hervé
[1]
https://lists.apache.org/thread.html/rd3af15d383ddceeb950cd90569e3dcdd6e5a0f5d3cd653ec534b0609%40%3Cdev.maven.apache.org%3E
[2] https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin
I have now installed latest OpenJDK 7 from AdoptOpenJDK source.
[INFO] --- maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) @
maven-site-plugin ---
[INFO] Saved info on build to
/usr/home/mosipov/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.buildinfo
[INFO] Checking against reference build from
https://repository.apache.org/content/repositories/maven-1554/...
[WARNING] Reference buildinfo file not found: it will be generated from
downloaded reference artifacts
[INFO] Minimal buildinfo generated from downloaded artifacts:
/usr/home/mosipov/Projekte/maven-site-plugin/target/reference/maven-site-plugin-3.9.0.buildinfo
[WARNING] size mismatch maven-site-plugin-3.9.0-source-release.zip: diffoscope
target/reference/maven-site-plugin-3.9.0-source-release.zip
target/maven-site-plugin-3.9.0-source-release.zip
[WARNING] size mismatch maven-site-plugin-3.9.0-sources.jar: diffoscope
target/reference/maven-site-plugin-3.9.0-sources.jar
target/maven-site-plugin-3.9.0-sources.jar
[WARNING] Reproducible Build output summary: 1 files ok, 2 different, 0 missing
[WARNING] diff target/reference/maven-site-plugin-3.9.0.buildinfo
target/maven-site-plugin-3.9.0.buildinfo
on
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/share/java/maven
Java version: 1.7.0_251, vendor: Oracle Corporation, runtime:
/usr/local/openjdk7/jre
Default locale: de_DE, platform encoding: UTF-8
OS name: "freebsd", version: "11.3-release-p6", arch: "i386", family: "unix"
and
$ git branch
* (HEAD losgelöst bei maven-site-plugin-3.9.0)
diffoscope target/reference/maven-site-plugin-3.9.0-source-release.zip
target/maven-site-plugin-3.9.0-source-release.zip
There is a diff in maven-site-plugin-3.9.0/dependency-reduced-pom.xml
diffoscope target/reference/maven-site-plugin-3.9.0-sources.jar
target/maven-site-plugin-3.9.0-sources.jar
So is here diff in the pom.xml which is actually dependency-reduced-pom.xml.
├── META-INF/maven/org.apache.maven.plugins/maven-site-plugin/pom.xml
│ ├── META-INF/maven/org.apache.maven.plugins/maven-site-plugin/pom.xml
│ │ @@ -243,100 +243,40 @@
│ │
│ │
│ │run-its
│ │
│ │
│ │
│ │ maven-invoker-plugin
│ │ -3.2.1
│ │ -
│ │ -
│ │ -integration-test
│ │ -
│ │ - install
│ │ - integration-test
│ │ - verify
│ │ -
│ │ -
│ │ - src/it/projects
│ │ - src/it/mrm/settings.xml
│ │ -
│ │ -
${mrm.repository.url}
│ │ -
│ │ -
│ │ -clean
│ │ -
org.apache.maven.plugins:maven-site-plugin:3.9.0:site
│ │ -
│ │ -
│ │ -1.7
│ │ -1.7
│ │ -
TLSv1,TLSv1.1,TLSv1.2
│ │ -
│ │ - true
│ │ -
/home/herve/projets/maven/sources/plugins/core/maven-site-plugin/target/checkout/target/it
│ │ - setup
│ │ - verify
│ │ -
/home/herve/projets/maven/sources/plugins/core/maven-site-plugin/target/checkout/target/local-repo
│ │ -
│ │ -*/pom.xml
│ │ -
│ │ - false
│ │ -
│ │ -
true
│ │ -
│ │ -
│ │ -
│ │ -
│ │
│ │src/it/projects
│ │
Re: [DISCUSS] checking reproducible builds
please "git pull": you're one commit behind HEAD https://github.com/apache/maven-studies/commits/maven-buildinfo-plugin - Mail original - De: "Karl Heinz Marbaise" À: "Maven Developers List" , "Hervé BOUTEMY" Envoyé: Samedi 7 Mars 2020 12:12:08 Objet: Re: [DISCUSS] checking reproducible builds Hi Hervé, I've tried to check my release via the suggested recipe... Downloaded the maven-studies repo and build the following commit: 90b426758363123af6fcc9aa7190b837c0551359 (mvn clean install) Downloaded the source package curl -O https://repository.apache.org/content/repositories/maven-1555/org/apache/maven/plugins/maven-dependency-plugin/3.1.2/maven-dependency-plugin-3.1.2-source-release.zip unzip maven-dependency-plugin-3.1.2-source-release.zip cd maven-dependency-plugin-3.1.2 and tried to run the following: mvn -Papache-release verify buildinfo:save -Dgpg.skip -Dreference.repo=https://repository.apache.org/content/repositories/maven-1555/ and got the following: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) on project maven-dependency-plugin: Error resolving reference artifact org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2: Could not transfer artifact org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2 from/to reference (https://repository.apache.org/content/repositories/maven-1555/): Cannot access https://repository.apache.org/content/repositories/maven-1555/ with type using the available connector factories: BasicRepositoryConnectorFactory: Cannot access https://repository.apache.org/content/repositories/maven-1555/ with type using the available layout factories: Maven2RepositoryLayoutFactory: Unsupported repository layout -> [Help 1] [ERROR] Kind regards Karl Heinz Marbaise On 07.03.20 11:36, Hervé BOUTEMY wrote: > Hi, > > Yesterday, I made a key step forward for Reproducible Builds with Maven: I > wrote code to easily check that your local build produces the same binaries > as the reference binaries published either to staging or to Central > repository. > > For a live example, see the last paragraph of Maven Site Plugin vote that > just started [1]. > > Process to check build output is based on a single plugin goal, currently > named buildinfo:save [2]: > 1. it creates a buildinfo file during build recording output fingerprints, > that will eventually in the future be published to Central repository > 2. it downloads reference artifacts and/or reference buildinfo and checks > that the output of the local build is the same as the reference. > > Now I want to discuss: is it clear? can you test and report, please? > > If the feedback is positive, the next question will be: in which plugin > should we put this goal to make a release and add it to our parent pom during > release, so we publish reference buildinfo along our reference binaries to > Central repository. > > Thanks for your feedback > > Regards, > > Hervé > > [1] > https://lists.apache.org/thread.html/rd3af15d383ddceeb950cd90569e3dcdd6e5a0f5d3cd653ec534b0609%40%3Cdev.maven.apache.org%3E > > [2] https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin > - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
unpack and get (without transitive dependencies) are candidates to me.
Having extra goals makes the plugin more interesting.
Robert
On 8-3-2020 23:25:11, Hervé BOUTEMY wrote:
clearly, save goal is not a good choice: buildinfo would be better
I know buildinfo is not a usual term, but it's widely used in Reproducible
Builds [1] & [2], then it would be nice us Maven not to reinvent a wheel that
has already been invented
on separating checking, I really don't see how this improves experience
I love this idea of maven-artifact-plugin, but I don't see which goals od
maven-dependency-plugin could go in:
https://maven.apache.org/plugins/maven-dependency-plugin/
Regards,
Hervé
[1] https://reproducible-builds.org/docs/jvm/
[2] https://wiki.debian.org/ReproducibleBuilds/BuildinfoFiles
Le dimanche 8 mars 2020, 21:04:56 CET Robert Scholte a écrit :
> I'm thinking of maven-artifact-plugin, having goals related to artifacts.
> That implies that the save goal should be renamed.
> A couple of goals of the maven-dependency-plugin are actually more
> artifact-related are might be worth moving.
>
> Robert
>
> On 8-3-2020 13:44:07, Michael Osipov wrote:
>
> Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> This is expected because I am on 1.8.0_242. I don't have Java 7
> installed anymore on the server.
> >>>
> >>> for the discussion I wanted us to have, just being able to test and see
> >>> how we detect issues, this is perfect, isn't it?
> >>
> >> This is really nice. Here is the diffoscope output:
> > you're discovering the wonders of diffoscope :)
> >
> >>> --- maven-site-plugin-3.9.0.jar
> >>> +++ reference/maven-site-plugin-3.9.0.jar
> >>> ├── zipinfo {}
> >>> │ @@ -1,8 +1,8 @@
> >
> > [...]
> >
> >>> META-INF/MANIFEST.MF
> >>> │ @@ -1,10 +1,10 @@
> >>> │ Manifest-Version: 1.0
> >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> >>> │ Implementation-Title: Apache Maven Site Plugin
> >>> │ Implementation-Version: 3.9.0
> >>> │ +Build-Jdk-Spec: 1.7^M
> >>> │ Specification-Vendor: The Apache Software Foundation
> >>> │ -Specification-Title: Apache Maven Site Plugin^M
> >>> │ -Build-Jdk-Spec: 1.8^M
> >>> │ Created-By: Maven Jar Plugin 3.2.0
> >>> │ +Specification-Title: Apache Maven Site Plugin^M
> >>> │ Specification-Version: 3.9
> >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> >>
> >> I wonder where the CRs code from...this could be the default
> >> serialization format on every platform.
> >
> > FYI I don't have such CRs in output on my Linux box
>
> This cannot be. See
> https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/sha
> re/classes/java/util/jar/Manifest.java and search for \r\n. Old Sun code
> uses *always* CRLF. Plase recheck on your side and run a hexdump on the
> Manifest file.
>
> >>> how did you find the experience? any improvement proposal?
> >>> and any idea on where to put this goal in the future?
> >>
> >> There is room for improvement when I quickly read the code. I will write
> >> separately on this.
> >
> > sure, code can be improved: don't hesitate
> > but I was not asking yet for code improvement (I'm confident, it will
> > happen) but *experience* improvement
> >
> >> I'd leave as a plugin for now.
> >
> > you mean a separate plugin? same "buildinfo" name as current? "save" goal
> > name?
>
> OK, let's talk about experience:
>
> * buildinfo may be changed to broader name, e.g.,
> maven-reproducibility-plugin. Explanain follows
> * 'save' does too much. It should save only and not compare. Save should
> either run at initialize or at build-resources phase, imho
> * Add a 'compare' goal, not phase bound. It performs the actual comparsion.
>
> Strictly speaking if the plugin is called buildinfo it should handle the
> buildinfo files only.
>
> >> At least in 3.7.x.
> >
> > 3.7.x as Maven 3.7.x?
> > does that mean that you think it should be one day integrated into Maven
> > core? what's the rationale?
>
> Not really, but if this happens, not before 4.x. I don't have any
> rationale or entry point for this yet.
>
> Michael
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Hmm, thinking out loud but cant a reproducible build check just build the
project twice staging locally first artifacts and comparing second pass
outputs to the staged ones?
Le dim. 8 mars 2020 à 23:25, Hervé BOUTEMY a écrit :
> clearly, save goal is not a good choice: buildinfo would be better
>
> I know buildinfo is not a usual term, but it's widely used in Reproducible
> Builds [1] & [2], then it would be nice us Maven not to reinvent a wheel
> that
> has already been invented
>
> on separating checking, I really don't see how this improves experience
>
> I love this idea of maven-artifact-plugin, but I don't see which goals od
> maven-dependency-plugin could go in:
> https://maven.apache.org/plugins/maven-dependency-plugin/
>
> Regards,
>
> Hervé
>
> [1] https://reproducible-builds.org/docs/jvm/
>
> [2] https://wiki.debian.org/ReproducibleBuilds/BuildinfoFiles
>
> Le dimanche 8 mars 2020, 21:04:56 CET Robert Scholte a écrit :
> > I'm thinking of maven-artifact-plugin, having goals related to artifacts.
> > That implies that the save goal should be renamed.
> > A couple of goals of the maven-dependency-plugin are actually more
> > artifact-related are might be worth moving.
> >
> > Robert
> >
> > On 8-3-2020 13:44:07, Michael Osipov wrote:
> >
> > Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> > >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> > >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> > This is expected because I am on 1.8.0_242. I don't have Java 7
> > installed anymore on the server.
> > >>>
> > >>> for the discussion I wanted us to have, just being able to test and
> see
> > >>> how we detect issues, this is perfect, isn't it?
> > >>
> > >> This is really nice. Here is the diffoscope output:
> > > you're discovering the wonders of diffoscope :)
> > >
> > >>> --- maven-site-plugin-3.9.0.jar
> > >>> +++ reference/maven-site-plugin-3.9.0.jar
> > >>> ├── zipinfo {}
> > >>> │ @@ -1,8 +1,8 @@
> > >
> > > [...]
> > >
> > >>> META-INF/MANIFEST.MF
> > >>> │ @@ -1,10 +1,10 @@
> > >>> │ Manifest-Version: 1.0
> > >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> > >>> │ Implementation-Title: Apache Maven Site Plugin
> > >>> │ Implementation-Version: 3.9.0
> > >>> │ +Build-Jdk-Spec: 1.7^M
> > >>> │ Specification-Vendor: The Apache Software Foundation
> > >>> │ -Specification-Title: Apache Maven Site Plugin^M
> > >>> │ -Build-Jdk-Spec: 1.8^M
> > >>> │ Created-By: Maven Jar Plugin 3.2.0
> > >>> │ +Specification-Title: Apache Maven Site Plugin^M
> > >>> │ Specification-Version: 3.9
> > >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> > >>
> > >> I wonder where the CRs code from...this could be the default
> > >> serialization format on every platform.
> > >
> > > FYI I don't have such CRs in output on my Linux box
> >
> > This cannot be. See
> >
> https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/sha
> > re/classes/java/util/jar/Manifest.java and search for \r\n. Old Sun code
> > uses *always* CRLF. Plase recheck on your side and run a hexdump on the
> > Manifest file.
> >
> > >>> how did you find the experience? any improvement proposal?
> > >>> and any idea on where to put this goal in the future?
> > >>
> > >> There is room for improvement when I quickly read the code. I will
> write
> > >> separately on this.
> > >
> > > sure, code can be improved: don't hesitate
> > > but I was not asking yet for code improvement (I'm confident, it will
> > > happen) but *experience* improvement
> > >
> > >> I'd leave as a plugin for now.
> > >
> > > you mean a separate plugin? same "buildinfo" name as current? "save"
> goal
> > > name?
> >
> > OK, let's talk about experience:
> >
> > * buildinfo may be changed to broader name, e.g.,
> > maven-reproducibility-plugin. Explanain follows
> > * 'save' does too much. It should save only and not compare. Save should
> > either run at initialize or at build-resources phase, imho
> > * Add a 'compare' goal, not phase bound. It performs the actual
> comparsion.
> >
> > Strictly speaking if the plugin is called buildinfo it should handle the
> > buildinfo files only.
> >
> > >> At least in 3.7.x.
> > >
> > > 3.7.x as Maven 3.7.x?
> > > does that mean that you think it should be one day integrated into
> Maven
> > > core? what's the rationale?
> >
> > Not really, but if this happens, not before 4.x. I don't have any
> > rationale or entry point for this yet.
> >
> > Michael
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> > For additional commands, e-mail: dev-h...@maven.apache.org
>
>
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
>
Re: [DISCUSS] checking reproducible builds
clearly, save goal is not a good choice: buildinfo would be better
I know buildinfo is not a usual term, but it's widely used in Reproducible
Builds [1] & [2], then it would be nice us Maven not to reinvent a wheel that
has already been invented
on separating checking, I really don't see how this improves experience
I love this idea of maven-artifact-plugin, but I don't see which goals od
maven-dependency-plugin could go in:
https://maven.apache.org/plugins/maven-dependency-plugin/
Regards,
Hervé
[1] https://reproducible-builds.org/docs/jvm/
[2] https://wiki.debian.org/ReproducibleBuilds/BuildinfoFiles
Le dimanche 8 mars 2020, 21:04:56 CET Robert Scholte a écrit :
> I'm thinking of maven-artifact-plugin, having goals related to artifacts.
> That implies that the save goal should be renamed.
> A couple of goals of the maven-dependency-plugin are actually more
> artifact-related are might be worth moving.
>
> Robert
>
> On 8-3-2020 13:44:07, Michael Osipov wrote:
>
> Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> This is expected because I am on 1.8.0_242. I don't have Java 7
> installed anymore on the server.
> >>>
> >>> for the discussion I wanted us to have, just being able to test and see
> >>> how we detect issues, this is perfect, isn't it?
> >>
> >> This is really nice. Here is the diffoscope output:
> > you're discovering the wonders of diffoscope :)
> >
> >>> --- maven-site-plugin-3.9.0.jar
> >>> +++ reference/maven-site-plugin-3.9.0.jar
> >>> ├── zipinfo {}
> >>> │ @@ -1,8 +1,8 @@
> >
> > [...]
> >
> >>> META-INF/MANIFEST.MF
> >>> │ @@ -1,10 +1,10 @@
> >>> │ Manifest-Version: 1.0
> >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> >>> │ Implementation-Title: Apache Maven Site Plugin
> >>> │ Implementation-Version: 3.9.0
> >>> │ +Build-Jdk-Spec: 1.7^M
> >>> │ Specification-Vendor: The Apache Software Foundation
> >>> │ -Specification-Title: Apache Maven Site Plugin^M
> >>> │ -Build-Jdk-Spec: 1.8^M
> >>> │ Created-By: Maven Jar Plugin 3.2.0
> >>> │ +Specification-Title: Apache Maven Site Plugin^M
> >>> │ Specification-Version: 3.9
> >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> >>
> >> I wonder where the CRs code from...this could be the default
> >> serialization format on every platform.
> >
> > FYI I don't have such CRs in output on my Linux box
>
> This cannot be. See
> https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/sha
> re/classes/java/util/jar/Manifest.java and search for \r\n. Old Sun code
> uses *always* CRLF. Plase recheck on your side and run a hexdump on the
> Manifest file.
>
> >>> how did you find the experience? any improvement proposal?
> >>> and any idea on where to put this goal in the future?
> >>
> >> There is room for improvement when I quickly read the code. I will write
> >> separately on this.
> >
> > sure, code can be improved: don't hesitate
> > but I was not asking yet for code improvement (I'm confident, it will
> > happen) but *experience* improvement
> >
> >> I'd leave as a plugin for now.
> >
> > you mean a separate plugin? same "buildinfo" name as current? "save" goal
> > name?
>
> OK, let's talk about experience:
>
> * buildinfo may be changed to broader name, e.g.,
> maven-reproducibility-plugin. Explanain follows
> * 'save' does too much. It should save only and not compare. Save should
> either run at initialize or at build-resources phase, imho
> * Add a 'compare' goal, not phase bound. It performs the actual comparsion.
>
> Strictly speaking if the plugin is called buildinfo it should handle the
> buildinfo files only.
>
> >> At least in 3.7.x.
> >
> > 3.7.x as Maven 3.7.x?
> > does that mean that you think it should be one day integrated into Maven
> > core? what's the rationale?
>
> Not really, but if this happens, not before 4.x. I don't have any
> rationale or entry point for this yet.
>
> Michael
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
I'm thinking of maven-artifact-plugin, having goals related to artifacts.
That implies that the save goal should be renamed.
A couple of goals of the maven-dependency-plugin are actually more
artifact-related are might be worth moving.
Robert
On 8-3-2020 13:44:07, Michael Osipov wrote:
Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
>> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
>>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
This is expected because I am on 1.8.0_242. I don't have Java 7
installed anymore on the server.
>>>
>>> for the discussion I wanted us to have, just being able to test and see
>>> how we detect issues, this is perfect, isn't it?
>>
>> This is really nice. Here is the diffoscope output:
> you're discovering the wonders of diffoscope :)
>
>>> --- maven-site-plugin-3.9.0.jar
>>> +++ reference/maven-site-plugin-3.9.0.jar
>>> ├── zipinfo {}
>>> │ @@ -1,8 +1,8 @@
> [...]
>>> META-INF/MANIFEST.MF
>>> │ @@ -1,10 +1,10 @@
>>> │ Manifest-Version: 1.0
>>> │ +Implementation-Vendor: The Apache Software Foundation^M
>>> │ Implementation-Title: Apache Maven Site Plugin
>>> │ Implementation-Version: 3.9.0
>>> │ +Build-Jdk-Spec: 1.7^M
>>> │ Specification-Vendor: The Apache Software Foundation
>>> │ -Specification-Title: Apache Maven Site Plugin^M
>>> │ -Build-Jdk-Spec: 1.8^M
>>> │ Created-By: Maven Jar Plugin 3.2.0
>>> │ +Specification-Title: Apache Maven Site Plugin^M
>>> │ Specification-Version: 3.9
>>> │ -Implementation-Vendor: The Apache Software Foundation^M
>>
>> I wonder where the CRs code from...this could be the default
>> serialization format on every platform.
> FYI I don't have such CRs in output on my Linux box
This cannot be. See
https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/share/classes/java/util/jar/Manifest.java
and search for \r\n. Old Sun code uses *always* CRLF. Plase recheck on
your side and run a hexdump on the Manifest file.
>>> how did you find the experience? any improvement proposal?
>>> and any idea on where to put this goal in the future?
>>
>> There is room for improvement when I quickly read the code. I will write
>> separately on this.
> sure, code can be improved: don't hesitate
> but I was not asking yet for code improvement (I'm confident, it will happen)
> but *experience* improvement
>
>> I'd leave as a plugin for now.
> you mean a separate plugin? same "buildinfo" name as current? "save" goal
> name?
OK, let's talk about experience:
* buildinfo may be changed to broader name, e.g.,
maven-reproducibility-plugin. Explanain follows
* 'save' does too much. It should save only and not compare. Save should
either run at initialize or at build-resources phase, imho
* Add a 'compare' goal, not phase bound. It performs the actual comparsion.
Strictly speaking if the plugin is called buildinfo it should handle the
buildinfo files only.
>> At least in 3.7.x.
> 3.7.x as Maven 3.7.x?
> does that mean that you think it should be one day integrated into Maven core?
> what's the rationale?
Not really, but if this happens, not before 4.x. I don't have any
rationale or entry point for this yet.
Michael
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Il Dom 8 Mar 2020, 13:44 Michael Osipov ha scritto:
> Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> This is expected because I am on 1.8.0_242. I don't have Java 7
> installed anymore on the server.
> >>>
> >>> for the discussion I wanted us to have, just being able to test and see
> >>> how we detect issues, this is perfect, isn't it?
> >>
> >> This is really nice. Here is the diffoscope output:
> > you're discovering the wonders of diffoscope :)
> >
> >>> --- maven-site-plugin-3.9.0.jar
> >>> +++ reference/maven-site-plugin-3.9.0.jar
> >>> ├── zipinfo {}
> >>> │ @@ -1,8 +1,8 @@
> > [...]
> >>> META-INF/MANIFEST.MF
> >>> │ @@ -1,10 +1,10 @@
> >>> │ Manifest-Version: 1.0
> >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> >>> │ Implementation-Title: Apache Maven Site Plugin
> >>> │ Implementation-Version: 3.9.0
> >>> │ +Build-Jdk-Spec: 1.7^M
> >>> │ Specification-Vendor: The Apache Software Foundation
> >>> │ -Specification-Title: Apache Maven Site Plugin^M
> >>> │ -Build-Jdk-Spec: 1.8^M
> >>> │ Created-By: Maven Jar Plugin 3.2.0
> >>> │ +Specification-Title: Apache Maven Site Plugin^M
> >>> │ Specification-Version: 3.9
> >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> >>
> >> I wonder where the CRs code from...this could be the default
> >> serialization format on every platform.
> > FYI I don't have such CRs in output on my Linux box
>
> This cannot be. See
>
> https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/share/classes/java/util/jar/Manifest.java
> and search for \r\n. Old Sun code uses *always* CRLF. Plase recheck on
> your side and run a hexdump on the Manifest file.
>
> >>> how did you find the experience? any improvement proposal?
> >>> and any idea on where to put this goal in the future?
> >>
> >> There is room for improvement when I quickly read the code. I will write
> >> separately on this.
> > sure, code can be improved: don't hesitate
> > but I was not asking yet for code improvement (I'm confident, it will
> happen)
> > but *experience* improvement
> >
> >> I'd leave as a plugin for now.
> > you mean a separate plugin? same "buildinfo" name as current? "save" goal
> > name?
>
> OK, let's talk about experience:
>
> * buildinfo may be changed to broader name, e.g.,
> maven-reproducibility-plugin.
+1
> Explanain follows
> * 'save' does too much. It should save only and not compare. Save should
> either run at initialize or at build-resources phase, imho
> * Add a 'compare' goal, not phase bound. It performs the actual comparsion.
>
+1 for splitting this way
Enrico
> Strictly speaking if the plugin is called buildinfo it should handle the
> buildinfo files only.
>
> >> At least in 3.7.x.
> > 3.7.x as Maven 3.7.x?
> > does that mean that you think it should be one day integrated into Maven
> core?
> > what's the rationale?
>
> Not really, but if this happens, not before 4.x. I don't have any
> rationale or entry point for this yet.
>
> Michael
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
>
Re: [DISCUSS] checking reproducible builds
Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
This is expected because I am on 1.8.0_242. I don't have Java 7
installed anymore on the server.
for the discussion I wanted us to have, just being able to test and see
how we detect issues, this is perfect, isn't it?
This is really nice. Here is the diffoscope output:
you're discovering the wonders of diffoscope :)
--- maven-site-plugin-3.9.0.jar
+++ reference/maven-site-plugin-3.9.0.jar
├── zipinfo {}
│ @@ -1,8 +1,8 @@
[...]
META-INF/MANIFEST.MF
│ @@ -1,10 +1,10 @@
│ Manifest-Version: 1.0
│ +Implementation-Vendor: The Apache Software Foundation^M
│ Implementation-Title: Apache Maven Site Plugin
│ Implementation-Version: 3.9.0
│ +Build-Jdk-Spec: 1.7^M
│ Specification-Vendor: The Apache Software Foundation
│ -Specification-Title: Apache Maven Site Plugin^M
│ -Build-Jdk-Spec: 1.8^M
│ Created-By: Maven Jar Plugin 3.2.0
│ +Specification-Title: Apache Maven Site Plugin^M
│ Specification-Version: 3.9
│ -Implementation-Vendor: The Apache Software Foundation^M
I wonder where the CRs code from...this could be the default
serialization format on every platform.
FYI I don't have such CRs in output on my Linux box
This cannot be. See
https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/share/classes/java/util/jar/Manifest.java
and search for \r\n. Old Sun code uses *always* CRLF. Plase recheck on
your side and run a hexdump on the Manifest file.
how did you find the experience? any improvement proposal?
and any idea on where to put this goal in the future?
There is room for improvement when I quickly read the code. I will write
separately on this.
sure, code can be improved: don't hesitate
but I was not asking yet for code improvement (I'm confident, it will happen)
but *experience* improvement
I'd leave as a plugin for now.
you mean a separate plugin? same "buildinfo" name as current? "save" goal
name?
OK, let's talk about experience:
* buildinfo may be changed to broader name, e.g.,
maven-reproducibility-plugin. Explanain follows
* 'save' does too much. It should save only and not compare. Save should
either run at initialize or at build-resources phase, imho
* Add a 'compare' goal, not phase bound. It performs the actual comparsion.
Strictly speaking if the plugin is called buildinfo it should handle the
buildinfo files only.
At least in 3.7.x.
3.7.x as Maven 3.7.x?
does that mean that you think it should be one day integrated into Maven core?
what's the rationale?
Not really, but if this happens, not before 4.x. I don't have any
rationale or entry point for this yet.
Michael
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> > Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> >> This is expected because I am on 1.8.0_242. I don't have Java 7
> >> installed anymore on the server.
> >
> > for the discussion I wanted us to have, just being able to test and see
> > how we detect issues, this is perfect, isn't it?
>
> This is really nice. Here is the diffoscope output:
you're discovering the wonders of diffoscope :)
> > --- maven-site-plugin-3.9.0.jar
> > +++ reference/maven-site-plugin-3.9.0.jar
> > ├── zipinfo {}
> > │ @@ -1,8 +1,8 @@
[...]
> > META-INF/MANIFEST.MF
> > │ @@ -1,10 +1,10 @@
> > │ Manifest-Version: 1.0
> > │ +Implementation-Vendor: The Apache Software Foundation^M
> > │ Implementation-Title: Apache Maven Site Plugin
> > │ Implementation-Version: 3.9.0
> > │ +Build-Jdk-Spec: 1.7^M
> > │ Specification-Vendor: The Apache Software Foundation
> > │ -Specification-Title: Apache Maven Site Plugin^M
> > │ -Build-Jdk-Spec: 1.8^M
> > │ Created-By: Maven Jar Plugin 3.2.0
> > │ +Specification-Title: Apache Maven Site Plugin^M
> > │ Specification-Version: 3.9
> > │ -Implementation-Vendor: The Apache Software Foundation^M
>
> I wonder where the CRs code from...this could be the default
> serialization format on every platform.
FYI I don't have such CRs in output on my Linux box
>
> > how did you find the experience? any improvement proposal?
> > and any idea on where to put this goal in the future?
>
> There is room for improvement when I quickly read the code. I will write
> separately on this.
sure, code can be improved: don't hesitate
but I was not asking yet for code improvement (I'm confident, it will happen)
but *experience* improvement
> I'd leave as a plugin for now.
you mean a separate plugin? same "buildinfo" name as current? "save" goal
name?
> At least in 3.7.x.
3.7.x as Maven 3.7.x?
does that mean that you think it should be one day integrated into Maven core?
what's the rationale?
Regards,
Hervé
>
> M
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
On Sat, Mar 7, 2020 at 11:39 AM Michael Osipov wrote: > > As note, reproducibility after some time is not always possible if > nessary compilers/tools aren't available anymore -- as you can see. > That's an important point. Some organizations archive their entire build chain including compilers and other tools in the source repository. I haven't seen it done, but I imagine you could go further using Docker images as the source of the reproducible build. -- Elliotte Rusty Harold elh...@ibiblio.org - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Diff on OpenJDK 11: ├── META-INF/MANIFEST.MF │ @@ -1,10 +1,10 @@ │ Manifest-Version: 1.0 │ +Implementation-Vendor: The Apache Software Foundation^M │ +Implementation-Title: Apache Maven Site Plugin^M │ +Implementation-Version: 3.9.0^M │ +Build-Jdk-Spec: 1.7^M │ +Specification-Vendor: The Apache Software Foundation^M │ Created-By: Maven Jar Plugin 3.2.0 │ -Build-Jdk-Spec: 11^M │ Specification-Title: Apache Maven Site Plugin │ Specification-Version: 3.9 │ -Specification-Vendor: The Apache Software Foundation^M │ -Implementation-Title: Apache Maven Site Plugin^M │ -Implementation-Version: 3.9.0^M │ -Implementation-Vendor: The Apache Software Foundation^M It seems like the hash implementation differs from version to version... - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
This is expected because I am on 1.8.0_242. I don't have Java 7
installed anymore on the server.
for the discussion I wanted us to have, just being able to test and see how we
detect issues, this is perfect, isn't it?
This is really nice. Here is the diffoscope output:
--- maven-site-plugin-3.9.0.jar
+++ reference/maven-site-plugin-3.9.0.jar
├── zipinfo {}
│ @@ -1,8 +1,8 @@
│ -Zip file size: 136174 bytes, number of entries: 84
│ +Zip file size: 136331 bytes, number of entries: 84
│ -rw 2.0 fat0 bX defN 20-Mar-06 20:49 META-INF/
│ -rw 2.0 fat 345 bl defN 20-Mar-06 20:49 META-INF/MANIFEST.MF
│ -rw 2.0 fat28157 bl defN 20-Mar-06 20:49 META-INF/DEPENDENCIES
│ -rw 2.0 fat11358 bl defN 20-Mar-06 20:49 META-INF/LICENSE
│ -rw 2.0 fat 181 bl defN 20-Mar-06 20:49 META-INF/NOTICE
│ -rw 2.0 fat0 bl defN 20-Mar-06 20:49 META-INF/maven/
│ -rw 2.0 fat0 bl defN 20-Mar-06 20:49
META-INF/maven/org.apache.maven.plugins/
│ @@ -10,44 +10,44 @@
│ -rw 2.0 fat56112 bl defN 20-Mar-06 20:49
META-INF/maven/org.apache.maven.plugins/maven-site-plugin/plugin-help.xml
│ -rw 2.0 fat 103450 bl defN 20-Mar-06 20:49
META-INF/maven/plugin.xml
│ -rw 2.0 fat0 bl defN 20-Mar-06 20:49 org/
│ -rw 2.0 fat0 bl defN 20-Mar-06 20:49 org/apache/
│ -rw 2.0 fat0 bl defN 20-Mar-06 20:49 org/apache/maven/
│ -rw 2.0 fat0 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/
│ -rw 2.0 fat0 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/
│ --rw 2.0 fat 2983 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/AbstractSiteMojo.class
│ +-rw 2.0 fat 3033 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/AbstractSiteMojo.class
│ -rw 2.0 fat0 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/
│ --rw 2.0 fat 1472 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/AbstractDeployMojo$URIEncoder.class
│ --rw 2.0 fat23211 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/AbstractDeployMojo.class
│ +-rw 2.0 fat 1521 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/AbstractDeployMojo$URIEncoder.class
│ +-rw 2.0 fat23237 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/AbstractDeployMojo.class
│ -rw 2.0 fat 1935 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/AbstractStagingMojo.class
│ --rw 2.0 fat11174 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/HelpMojo.class
│ +-rw 2.0 fat11281 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/HelpMojo.class
│ -rw 2.0 fat 1251 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/SiteDeployMojo.class
│ --rw 2.0 fat 5630 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/SiteStageDeployMojo.class
│ --rw 2.0 fat 3931 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/SiteStageMojo.class
│ +-rw 2.0 fat 5622 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/SiteStageDeployMojo.class
│ +-rw 2.0 fat 3961 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/SiteStageMojo.class
│ -rw 2.0 fat0 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/wagon/
│ --rw 2.0 fat 4927 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/wagon/BugFixedRepository.class
│ --rw 2.0 fat 5604 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/wagon/PathUtils.class
│ +-rw 2.0 fat 4884 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/wagon/BugFixedRepository.class
│ +-rw 2.0 fat 5564 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/deploy/wagon/PathUtils.class
│ -rw 2.0 fat0 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/descriptor/
│ -rw 2.0 fat 4039 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/descriptor/AbstractSiteDescriptorMojo.class
│ --rw 2.0 fat 5734 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/descriptor/EffectiveSiteMojo.class
│ --rw 2.0 fat 4621 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/descriptor/SiteDescriptorArtifactMetadata.class
│ --rw 2.0 fat 4237 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/descriptor/SiteDescriptorAttachMojo.class
│ +-rw 2.0 fat 5780 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/descriptor/EffectiveSiteMojo.class
│ +-rw 2.0 fat 4666 bl defN 20-Mar-06 20:49
org/apache/maven/plugins/site/descriptor/SiteDescriptorArtifactMetadata.class
│ +-rw 2.0 fat 4267 bl defN 20-Mar-06 20:49
Re: [DISCUSS] checking reproducible builds
Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit : > This is expected because I am on 1.8.0_242. I don't have Java 7 > installed anymore on the server. for the discussion I wanted us to have, just being able to test and see how we detect issues, this is perfect, isn't it? how did you find the experience? any improvement proposal? and any idea on where to put this goal in the future? > > As note, reproducibility after some time is not always possible if > nessary compilers/tools aren't available anymore -- as you can see. when we absolutely want to rebuild, this is where containers can ease the job Regards, Hervé > > Michael > > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY: Hi, Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository. For a live example, see the last paragraph of Maven Site Plugin vote that just started [1]. Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]: 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference. Now I want to discuss: is it clear? can you test and report, please? If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository. Made some progress: [INFO] --- maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) @ maven-site-plugin --- [INFO] Saved info on build to /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.buildinfo [INFO] Checking against reference build from https://repository.apache.org/content/repositories/maven-1554/... [WARNING] Reference buildinfo file not found: it will be generated from downloaded reference artifacts [INFO] Minimal buildinfo generated from downloaded artifacts: /var/osipovmi/Projekte/maven-site-plugin/target/reference/maven-site-plugin-3.9.0.buildinfo [WARNING] size mismatch maven-site-plugin-3.9.0.jar: diffoscope target/reference/maven-site-plugin-3.9.0.jar target/maven-site-plugin-3.9.0.jar [WARNING] size mismatch maven-site-plugin-3.9.0-sources.jar: diffoscope target/reference/maven-site-plugin-3.9.0-sources.jar target/maven-site-plugin-3.9.0-sources.jar [WARNING] size mismatch maven-site-plugin-3.9.0-source-release.zip: diffoscope target/reference/maven-site-plugin-3.9.0-source-release.zip target/maven-site-plugin-3.9.0-source-release.zip [WARNING] Reproducible Build output summary: 0 files ok, 3 different, 0 missing [WARNING] diff target/reference/maven-site-plugin-3.9.0.buildinfo target/maven-site-plugin-3.9.0.buildinfo This is expected because I am on 1.8.0_242. I don't have Java 7 installed anymore on the server. As note, reproducibility after some time is not always possible if nessary compilers/tools aren't available anymore -- as you can see. Michael - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
thank you for testing little bug fixed: stupid me, I should have tested this case before asking for feedback: I tested only with central repository, provided as "central" id... please fetch the latest plugin update and retest :) Le samedi 7 mars 2020, 13:12:08 CET Karl Heinz Marbaise a écrit : > Hi Hervé, > > I've tried to check my release via the suggested recipe... > > > Downloaded the maven-studies repo and build the following commit: > 90b426758363123af6fcc9aa7190b837c0551359 (mvn clean install) > > Downloaded the source package > > curl -O > https://repository.apache.org/content/repositories/maven-1555/org/apache/mav > en/plugins/maven-dependency-plugin/3.1.2/maven-dependency-plugin-3.1.2-sourc > e-release.zip > > unzip maven-dependency-plugin-3.1.2-source-release.zip > > cd maven-dependency-plugin-3.1.2 and tried to run the following: > > mvn -Papache-release verify buildinfo:save -Dgpg.skip > -Dreference.repo=https://repository.apache.org/content/repositories/maven-15 > 55/ > > and got the following: > > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save > (default-cli) on project maven-dependency-plugin: Error resolving > reference artifact > org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2: Could > not transfer artifact > org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2 from/to > reference > (https://repository.apache.org/content/repositories/maven-1555/): Cannot > access https://repository.apache.org/content/repositories/maven-1555/ > with type using the available connector factories: > BasicRepositoryConnectorFactory: Cannot access > https://repository.apache.org/content/repositories/maven-1555/ with type > using the available layout factories: Maven2RepositoryLayoutFactory: > Unsupported repository layout -> [Help 1] > [ERROR] > > > > Kind regards > Karl Heinz Marbaise > > On 07.03.20 11:36, Hervé BOUTEMY wrote: > > Hi, > > > > Yesterday, I made a key step forward for Reproducible Builds with Maven: I > > wrote code to easily check that your local build produces the same > > binaries as the reference binaries published either to staging or to > > Central repository. > > > > For a live example, see the last paragraph of Maven Site Plugin vote that > > just started [1]. > > > > Process to check build output is based on a single plugin goal, currently > > named buildinfo:save [2]: 1. it creates a buildinfo file during build > > recording output fingerprints, that will eventually in the future be > > published to Central repository 2. it downloads reference artifacts > > and/or reference buildinfo and checks that the output of the local build > > is the same as the reference. > > > > Now I want to discuss: is it clear? can you test and report, please? > > > > If the feedback is positive, the next question will be: in which plugin > > should we put this goal to make a release and add it to our parent pom > > during release, so we publish reference buildinfo along our reference > > binaries to Central repository. > > > > Thanks for your feedback > > > > Regards, > > > > Hervé > > > > [1] > > https://lists.apache.org/thread.html/rd3af15d383ddceeb950cd90569e3dcdd6e5 > > a0f5d3cd653ec534b0609%40%3Cdev.maven.apache.org%3E > > > > [2] https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Hi, On 07.03.20 14:19, Michael Osipov wrote: Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY: Hi, Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository. For a live example, see the last paragraph of Maven Site Plugin vote that just started [1]. Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]: 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference. Now I want to discuss: is it clear? can you test and report, please? If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository. After even reverting the offending commit from Maven master, I still get: [INFO] --- maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) @ maven-site-plugin --- [INFO] Saved info on build to /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.buildinfo [INFO] Checking against reference build from https://repository.apache.org/content/repositories/maven-1554/... [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:12 min [INFO] Finished at: 2020-03-07T14:16:18+01:00 [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) on project maven-site-plugin: Error resolving reference artifact org.apache.maven.plugins:maven-site-plugin:buildinfo:3.9.0: Could not transfer artifact org.apache.maven.plugins:maven-site-plugin:buildinfo:3.9.0 from/to reference (https://repository.apache.org/content/repositories/maven-1554/): Cannot access https://repository.apache.org/content/repositories/maven-1554/ with type using the available connector factories: BasicRepositoryConnectorFactory: Cannot access https://repository.apache.org/content/repositories/maven-1554/ with type using the available layout factories: Maven2RepositoryLayoutFactory: Unsupported repository layout -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException That's exactly the same issue I have reported with Maven 3.6.3 ... Kind regards Karl Heinz Marbaise - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY: Hi, Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository. For a live example, see the last paragraph of Maven Site Plugin vote that just started [1]. Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]: 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference. Now I want to discuss: is it clear? can you test and report, please? If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository. After even reverting the offending commit from Maven master, I still get: [INFO] --- maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) @ maven-site-plugin --- [INFO] Saved info on build to /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.buildinfo [INFO] Checking against reference build from https://repository.apache.org/content/repositories/maven-1554/... [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:12 min [INFO] Finished at: 2020-03-07T14:16:18+01:00 [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) on project maven-site-plugin: Error resolving reference artifact org.apache.maven.plugins:maven-site-plugin:buildinfo:3.9.0: Could not transfer artifact org.apache.maven.plugins:maven-site-plugin:buildinfo:3.9.0 from/to reference (https://repository.apache.org/content/repositories/maven-1554/): Cannot access https://repository.apache.org/content/repositories/maven-1554/ with type using the available connector factories: BasicRepositoryConnectorFactory: Cannot access https://repository.apache.org/content/repositories/maven-1554/ with type using the available layout factories: Maven2RepositoryLayoutFactory: Unsupported repository layout -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Am 2020-03-07 um 13:45 schrieb Michael Osipov: Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY: Hi, Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository. For a live example, see the last paragraph of Maven Site Plugin vote that just started [1]. Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]: 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference. Now I want to discuss: is it clear? can you test and report, please? If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository. Fails for me with: osipovmi@deblndw011x:~/var/Projekte/maven-site-plugin ((maven-site-plugin-3.9.0) $ ~/apache-maven-3.7.0-SNAPSHOT/bin/mvn -v Apache Maven 3.7.0-SNAPSHOT (f2e9afd788de919646717532d26eca38826e9924) Maven home: /net/home/osipovmi/apache-maven-3.7.0-SNAPSHOT Java version: 1.8.0_242, vendor: Oracle Corporation, runtime: /usr/local/openjdk8/jre Default locale: de_DE, platform encoding: UTF-8 OS name: "freebsd", version: "12.1-stable", arch: "amd64", family: "unix" The build completely stalls at [INFO] Replacing /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.jar with /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0-shaded.jar [INFO] Dependency-reduced POM written at: /var/osipovmi/Projekte/maven-site-plugin/dependency-reduced-pom.xml CPU time is consumed like hell, I killed the process after 10 min. Looking at it with JConsole shows that main thread is heavy working on org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:317) org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229) org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:340) org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:203) org.apache.maven.shared.dependency.graph.internal.Maven31DependencyGraphBuilder.resolveDependencies(Maven31DependencyGraphBuilder.java:124) org.apache.maven.shared.dependency.graph.internal.Maven31DependencyGraphBuilder.buildDependencyGraph(Maven31DependencyGraphBuilder.java:110) org.apache.maven.shared.dependency.graph.internal.DefaultDependencyGraphBuilder.buildDependencyGraph(DefaultDependencyGraphBuilder.java:98) org.apache.maven.shared.dependency.graph.internal.DefaultDependencyGraphBuilder.buildDependencyGraph(DefaultDependencyGraphBuilder.java:67 org.apache.maven.plugins.shade.mojo.ShadeMojo.updateExcludesInDeps(ShadeMojo.java:1266) org.apache.maven.plugins.shade.mojo.ShadeMojo.rewriteDependencyReducedPomIfWeHaveReduction(ShadeMojo.java:1188) org.apache.maven.plugins.shade.mojo.ShadeMojo.createDependencyReducedPom(ShadeMojo.java:1098) org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:599) org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPlug This is a complete contrast to Maven 3.5.4 and not related to this new plugin. A mere "mvn clean verify" on MSITE stalls completely during shade. Need to test more. OK, found it: 716cc1fe02661897232a7cc3e4c1bb3b3df3b832 is the first bad commit commit 716cc1fe02661897232a7cc3e4c1bb3b3df3b832 Author: rfscholte Date: Wed Jan 29 21:18:42 2020 +0100 [MNG-5669] same pom.xml is read multiple times .../java/org/apache/maven/building/FileSource.java | 31 .../org/apache/maven/building/StringSource.java| 33 +++- .../java/org/apache/maven/building/UrlSource.java | 32 +++- .../apache/maven/project/ReactorModelCache.java| 78 +++- .../maven/model/building/ArtifactModelSource.java | 59 ++ .../maven/model/building/DefaultModelBuilder.java | 206 - .../maven/model/building/FileModelSource.java | 9 +- .../apache/maven/model/building/ModelCache.java| 29 +++ .../apache/maven/model/building/ModelCacheTag.java | 26 +++ .../model/superpom/DefaultSuperPomProvider.java| 2 +- .../internal/DefaultArtifactDescriptorReader.java | 7 +- .../repository/internal/DefaultModelResolver.java | 7 +- 12 files changed, 451 insertions(+), 68 deletions(-) create mode 100644 maven-model-builder/src/main/java/org/apache/maven/model/b @Robert, do you want to revert? This requires more testing obviously. Michael
Re: [DISCUSS] checking reproducible builds
Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY: Hi, Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository. For a live example, see the last paragraph of Maven Site Plugin vote that just started [1]. Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]: 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference. Now I want to discuss: is it clear? can you test and report, please? If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository. Fails for me with: osipovmi@deblndw011x:~/var/Projekte/maven-site-plugin ((maven-site-plugin-3.9.0) $ ~/apache-maven-3.7.0-SNAPSHOT/bin/mvn -v Apache Maven 3.7.0-SNAPSHOT (f2e9afd788de919646717532d26eca38826e9924) Maven home: /net/home/osipovmi/apache-maven-3.7.0-SNAPSHOT Java version: 1.8.0_242, vendor: Oracle Corporation, runtime: /usr/local/openjdk8/jre Default locale: de_DE, platform encoding: UTF-8 OS name: "freebsd", version: "12.1-stable", arch: "amd64", family: "unix" The build completely stalls at [INFO] Replacing /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.jar with /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0-shaded.jar [INFO] Dependency-reduced POM written at: /var/osipovmi/Projekte/maven-site-plugin/dependency-reduced-pom.xml CPU time is consumed like hell, I killed the process after 10 min. Looking at it with JConsole shows that main thread is heavy working on org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:317) org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229) org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:340) org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:203) org.apache.maven.shared.dependency.graph.internal.Maven31DependencyGraphBuilder.resolveDependencies(Maven31DependencyGraphBuilder.java:124) org.apache.maven.shared.dependency.graph.internal.Maven31DependencyGraphBuilder.buildDependencyGraph(Maven31DependencyGraphBuilder.java:110) org.apache.maven.shared.dependency.graph.internal.DefaultDependencyGraphBuilder.buildDependencyGraph(DefaultDependencyGraphBuilder.java:98) org.apache.maven.shared.dependency.graph.internal.DefaultDependencyGraphBuilder.buildDependencyGraph(DefaultDependencyGraphBuilder.java:67 org.apache.maven.plugins.shade.mojo.ShadeMojo.updateExcludesInDeps(ShadeMojo.java:1266) org.apache.maven.plugins.shade.mojo.ShadeMojo.rewriteDependencyReducedPomIfWeHaveReduction(ShadeMojo.java:1188) org.apache.maven.plugins.shade.mojo.ShadeMojo.createDependencyReducedPom(ShadeMojo.java:1098) org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:599) org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPlug This is a complete contrast to Maven 3.5.4 and not related to this new plugin. A mere "mvn clean verify" on MSITE stalls completely during shade. Need to test more. Michael - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Am 2020-03-07 um 13:12 schrieb Karl Heinz Marbaise: Hi Hervé, I've tried to check my release via the suggested recipe... Downloaded the maven-studies repo and build the following commit: 90b426758363123af6fcc9aa7190b837c0551359 (mvn clean install) Downloaded the source package curl -O https://repository.apache.org/content/repositories/maven-1555/org/apache/maven/plugins/maven-dependency-plugin/3.1.2/maven-dependency-plugin-3.1.2-source-release.zip unzip maven-dependency-plugin-3.1.2-source-release.zip cd maven-dependency-plugin-3.1.2 and tried to run the following: mvn -Papache-release verify buildinfo:save -Dgpg.skip -Dreference.repo=https://repository.apache.org/content/repositories/maven-1555/ and got the following: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) on project maven-dependency-plugin: Error resolving reference artifact org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2: Could not transfer artifact org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2 from/to reference (https://repository.apache.org/content/repositories/maven-1555/): Cannot access https://repository.apache.org/content/repositories/maven-1555/ with type using the available connector factories: BasicRepositoryConnectorFactory: Cannot access https://repository.apache.org/content/repositories/maven-1555/ with type using the available layout factories: Maven2RepositoryLayoutFactory: Unsupported repository layout -> [Help 1] [ERROR] Same here with Maven 3.5.4. - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] checking reproducible builds
Hi Hervé, I've tried to check my release via the suggested recipe... Downloaded the maven-studies repo and build the following commit: 90b426758363123af6fcc9aa7190b837c0551359 (mvn clean install) Downloaded the source package curl -O https://repository.apache.org/content/repositories/maven-1555/org/apache/maven/plugins/maven-dependency-plugin/3.1.2/maven-dependency-plugin-3.1.2-source-release.zip unzip maven-dependency-plugin-3.1.2-source-release.zip cd maven-dependency-plugin-3.1.2 and tried to run the following: mvn -Papache-release verify buildinfo:save -Dgpg.skip -Dreference.repo=https://repository.apache.org/content/repositories/maven-1555/ and got the following: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) on project maven-dependency-plugin: Error resolving reference artifact org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2: Could not transfer artifact org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2 from/to reference (https://repository.apache.org/content/repositories/maven-1555/): Cannot access https://repository.apache.org/content/repositories/maven-1555/ with type using the available connector factories: BasicRepositoryConnectorFactory: Cannot access https://repository.apache.org/content/repositories/maven-1555/ with type using the available layout factories: Maven2RepositoryLayoutFactory: Unsupported repository layout -> [Help 1] [ERROR] Kind regards Karl Heinz Marbaise On 07.03.20 11:36, Hervé BOUTEMY wrote: Hi, Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository. For a live example, see the last paragraph of Maven Site Plugin vote that just started [1]. Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]: 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference. Now I want to discuss: is it clear? can you test and report, please? If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository. Thanks for your feedback Regards, Hervé [1] https://lists.apache.org/thread.html/rd3af15d383ddceeb950cd90569e3dcdd6e5a0f5d3cd653ec534b0609%40%3Cdev.maven.apache.org%3E [2] https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
