one more "topic" to consider: If we abandon pack200 for compression, we could still use it for repacking, or JAR "normalization".
pack200 has 2 phases; the first does things which change the JAR content/structure: 1. merges/sorts constant-ppol data in the class files and co-locates them 2. removes redundant class attributes 3. stores internal data structures The 2nd phase uses compression technologies to pack. The first phase needs to be done (using the -repack option) before signing; see http://docs.oracle.com/javase/1.5.0/docs/guide/deployment/deployment-guide/pack200.html It works like this: pack (changing structure) -> unpack -> sign -> pack (no change to structure) -> deploy The first pack does the optimizations that change the structure; the 2nd one doesn't because the structures are already in the right way. I'm guessing not too much saving happens in normal JARs due to normalization, alone, but might be worth doing for big jars (like the eclipse uima runtime plugin - 4m). -Marshall On 9/20/2017 11:34 AM, Marshall Schor wrote: > I'm starting to think about redoing our Eclipse Plugin release flow. I'm > hoping > to start a thread where people can post answers to issues (there are many...). > > Motivation: start using the Symantec signing process for Jars (in addition to > the PGP Apache signing process), so that Eclipse, when installing new plugins > doesn't warn about unsigned artifacts. > > There are many things to figure out. One is how to manage the cost of the > signing. Signing costs real money, which Apache covers (see > https://reference.apache.org/pmc/codesigning). > > 1) To find out: is the cost per Jar, or per "signing event" which can include > multiple Jars? This can affect some design choices. > > 2) Another thing is when (in our release flow) to sign. The signing process > alters the JARs. > > Our current flow is 1) generate candidate with PGP signature, 2) Vote 3) > repeat if failed rc, otherwise promote. An Apache principle is to have the > Vote > be on the signed artifacts, which are then promoted without modification. > This > implies the step 1) would need to include signing. > > This means that every failed release candidate costs money. I found out that > some projects "accept" this, but have low failed percentages, like 1 in 10. > In > our process, we seem to frequently have multiple release candidates per > release. > > Perhaps a way forward is to have a process for the Eclipse plugin release > which > is different, and isn't invoked until a "preliminary" candidate review has > been > done; once this has happened, and there's confidence that the signed version > would likely pass the vote, an official Eclipse plugin release candidate could > be posted with signed JARs. > > 3) Another thing to figure out, assuming that signing costs "per jar", is to > insure only new Jars are signed, not the older versions. This could affect > efforts to limit the number of JARs (or not). > > 4) Another thing to figure out is what needs to be signed. I think that only > Eclipse plugin jars need signing, and not the other jars associated with the > update site, like artifacts.jar, content.jar, and for each feature, > feat-name.jar (e.g. org.apache.uima.runtime_3.0.0.alpha02.jar). But maybe > that's wrong. The site > https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F > says multiple things: 1) "by default, every Jar pushed to an update site is > signed", 2) "in standalone zip distributions, all JARed plugins will be > signed". So, my guess is that you don't need to sign the other kinds of > JARs. > > Some websites imply that JARs other than the plugin ones are signed ( > https://wiki.eclipse.org/Platform-releng-signedbuild ), but it's unclear if > they > need signing in order to get the Eclipse install program not to complain. > That > website says the master zip having all the plugins and features is sent to the > signing process. > > 5) Some of our Jars (typically, the runtime Jars) have other Jars inside > them. > This site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F says "JARs > nested at arbitrary depth within other JARs" are signed, too. Is that needed > to > make the Eclipse install program not complain about installing unsigned > artifacts? > > 6) One site ( https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F ) says > there may be legal reasons you are not permitted to sign an embedded JAR. > I've > never seen anything about that, and don't know how to determine if such a > prohibition exists. This page says, under what gets signed, "Some JARs may be > excluded if there are technical or legal reasons why they cannot be signed". > > 7) Signing and pack200 interact. This page > https://wiki.eclipse.org/Tycho/Pack200#Pack200_and_Signing describes a flow: > 1. pack200 operated to "normalize" - this is specified by the "repack" option > 2. sign > 3. pack200 operated to generate the compressed version > 4. p2 metadata generation / updating > > Our current eclipse plugin build flow runs pack200 with both the "-repack" and > "-pack" options. I think this would need to be split into two steps (assuming > we keep pack200 - see following), with the results from -repack being sent for > signing, and then the result subsequently packed. > > 8) I question if we need to do pack200 packing. It has these effects: > 1. for every Jar, it generates 2 Jars: the original and the packed. So there > are 2 jars that need "signing", thus it costs twice as much. > 2. A typical Jar (caseditor.ide) is 69K bytes, packed 28K. A big jar > uima-runtime is 4M, packed: 1.3M > So the result is increased space on the distribution medium, but decreased > bandwidth for downloading. But the download frequency is very low - only when > someone installs the UIMA Eclipse plugins into a new Eclipse instance. I > think > pack200 was envisioned to help websites where the fetch was more frequent, and > web-page load times were important. > > Should we abandon doing pack200 packaging? > > Well, those 8 topics should get us started. Perhaps it will be best to start > a > new thread for each one. > I'm looking forward to the discussion. > > Cheers. -Marshall >
