Re: JEP 238: Multi-Version JAR Files
Note that one possible feature of JDK 9 is a -platform N option to javac which would allow compiling against older versions of the platform libraries: JEP draft: Compile for Specific Platform Version http://openjdk.java.net/jeps/8058150 This feature would address some of the concerns raised below. -Joe On 4/15/2015 2:24 PM, Remi Forax wrote: On 04/15/2015 02:44 PM, Paul Sandoz wrote: Hi Remi, On Apr 15, 2015, at 1:58 PM, Remi Forax wrote: Hi Paul, I think you're seriously underestimate the cost of this JEP. That is why we are asking for feedback :-) I want to understand the impact and get some concrete reasons why certain aspects are difficult. You're asking Java devs to: - be able to maintain several codes with different source level compatibility in the same code tree. Yes. Maintaining one code with a different source level than the installed JDK is still not fully solved, i.e. you still need to have either an old rt.jar or a kind of meta-description of it. This can be solved more easily with modules but we aren't yet there. - add a level of indirection in all tools that crawle/compile Java source code - add a level of indirection in all tools that crawle/rewrite bytecodes How common is it to statically process the byte codes of a JAR file to produce a new JAR file as opposed to doing that dynamically at runtime? (I know of obfuscating tools, which IIRC were more commonly used for mobile?) From my experience, most of tools that do rewriting at runtime are able to do rewriting at compile time (or install time). Are processed JAR files then re-distributed via say maven or kept internally to be used with a specific stack? By example for ASM, we publish several artifacts from the same source, ones are 1.3 compatible (asm.jar), others are 1.5 compatibles (asm-all.jar). So users that run on embedded device can still use ASM. Would such tools process use the JarFile class to get access to the JAR file contents? either jar file or class files directly. all of that to support a feature that: - is not clearly defined To me what is not clearly understood is the potential impact, where as the feature itself is actually fairly simple to grok. I agree, the idea is simple to grok, not it's perimeter. - is supposed to only solve cases where delta between versions is small Yes, it's anticipated there would be a small number of classes that would require changing. - comes with no help from the JDK (how to detect a version, etc). We can propose such things for 9. For example, new public methods to JarFile. There will be a public JDK version query API in 9, is that what you mean by "how to detect a version?", or do you mean how can one analyze an MVJAR? (or perhaps both...) How to analyze a MVJAR given that you usually have only one version of the JDK installed ? Paul. Rémi While I understand the comfort for the end user to have only one big fat jar, it would like to see a prototype that have good answers to all these questions before including it into the JDK. Said simply, tackling such problem requires a JSR not a JEP, because it's a feature which impact a large number of parts of the Java ecosystem. regards, Rémi
Re: JEP 238: Multi-Version JAR Files
On 04/15/2015 02:44 PM, Paul Sandoz wrote: Hi Remi, On Apr 15, 2015, at 1:58 PM, Remi Forax wrote: Hi Paul, I think you're seriously underestimate the cost of this JEP. That is why we are asking for feedback :-) I want to understand the impact and get some concrete reasons why certain aspects are difficult. You're asking Java devs to: - be able to maintain several codes with different source level compatibility in the same code tree. Yes. Maintaining one code with a different source level than the installed JDK is still not fully solved, i.e. you still need to have either an old rt.jar or a kind of meta-description of it. This can be solved more easily with modules but we aren't yet there. - add a level of indirection in all tools that crawle/compile Java source code - add a level of indirection in all tools that crawle/rewrite bytecodes How common is it to statically process the byte codes of a JAR file to produce a new JAR file as opposed to doing that dynamically at runtime? (I know of obfuscating tools, which IIRC were more commonly used for mobile?) From my experience, most of tools that do rewriting at runtime are able to do rewriting at compile time (or install time). Are processed JAR files then re-distributed via say maven or kept internally to be used with a specific stack? By example for ASM, we publish several artifacts from the same source, ones are 1.3 compatible (asm.jar), others are 1.5 compatibles (asm-all.jar). So users that run on embedded device can still use ASM. Would such tools process use the JarFile class to get access to the JAR file contents? either jar file or class files directly. all of that to support a feature that: - is not clearly defined To me what is not clearly understood is the potential impact, where as the feature itself is actually fairly simple to grok. I agree, the idea is simple to grok, not it's perimeter. - is supposed to only solve cases where delta between versions is small Yes, it's anticipated there would be a small number of classes that would require changing. - comes with no help from the JDK (how to detect a version, etc). We can propose such things for 9. For example, new public methods to JarFile. There will be a public JDK version query API in 9, is that what you mean by "how to detect a version?", or do you mean how can one analyze an MVJAR? (or perhaps both...) How to analyze a MVJAR given that you usually have only one version of the JDK installed ? Paul. Rémi While I understand the comfort for the end user to have only one big fat jar, it would like to see a prototype that have good answers to all these questions before including it into the JDK. Said simply, tackling such problem requires a JSR not a JEP, because it's a feature which impact a large number of parts of the Java ecosystem. regards, Rémi
Re: JEP 238: Multi-Version JAR Files
On 04/15/2015 02:59 PM, Paul Sandoz wrote: On Apr 15, 2015, at 2:03 PM, Remi Forax wrote: Of course we cannot feasibly backport every new API in N to N-1, N-2 etc. In selective cases that might be possible, but for the core of the JDK it's like pulling on a string of public dependencies, internal dependencies and perhaps more subtle dependencies between javac and hotspot (Unsafe replacements would be particularly tricky). that why most of the tools like the retroweaver [1] rewrite bytecodes because you can replace a dependency to a JDK class to wire it to a class that will come with your code. IIUC Retroweaver seemed to hit a sweet spot focusing on transforming language features and some minimal API features (i presume some lambda weaving tooling has/will hit a similar sweet spot, there is at least one but i cannot recall the name). It's harder to write a rewriter from 1.8 to 1.7 because lambda means Stream and collection.stream() means default method which is hard to simulate just using bytecode rewriting (the only way I know is to use invokedynamic for that which is fun but will require a lot of testing). It seems for APIs in general this may require more sophisticated code transformation techniques with additional non-core library support? yes Note that in that case, you develop and maintain only one version of the code compatible with the newer version and backport the code to the old one, instead of selectively develop part of the code with the new version as you propose. And either one publishes X JARs for each platform, or the consumer manages that themselves. Or one trusts this to work in some dynamic fashion. -- In my last email i forgot to point out that a tool (jar for example) that transforms a MVJAR into a platform specific JAR before being operated on by a byte code transforming tool may be a reasonable approach, but still requires some additional action. Paul. Rémi
Re: JEP 238: Multi-Version JAR Files
On Apr 15, 2015, at 2:03 PM, Remi Forax wrote: >> Of course we cannot feasibly backport every new API in N to N-1, N-2 etc. In >> selective cases that might be possible, but for the core of the JDK it's >> like pulling on a string of public dependencies, internal dependencies and >> perhaps more subtle dependencies between javac and hotspot (Unsafe >> replacements would be particularly tricky). > > that why most of the tools like the retroweaver [1] rewrite bytecodes because > you can replace a dependency to a JDK class to wire it to a class that will > come with your code. > IIUC Retroweaver seemed to hit a sweet spot focusing on transforming language features and some minimal API features (i presume some lambda weaving tooling has/will hit a similar sweet spot, there is at least one but i cannot recall the name). It seems for APIs in general this may require more sophisticated code transformation techniques with additional non-core library support? > Note that in that case, you develop and maintain only one version of the code > compatible with the newer version and backport the code to the old one, > instead of selectively develop part of the code with the new version as you > propose. > And either one publishes X JARs for each platform, or the consumer manages that themselves. Or one trusts this to work in some dynamic fashion. -- In my last email i forgot to point out that a tool (jar for example) that transforms a MVJAR into a platform specific JAR before being operated on by a byte code transforming tool may be a reasonable approach, but still requires some additional action. Paul.
Re: JEP 238: Multi-Version JAR Files
Hi Remi, On Apr 15, 2015, at 1:58 PM, Remi Forax wrote: > Hi Paul, > I think you're seriously underestimate the cost of this JEP. That is why we are asking for feedback :-) I want to understand the impact and get some concrete reasons why certain aspects are difficult. > > You're asking Java devs to: > - be able to maintain several codes with different source level compatibility > in the same code tree. Yes. > - add a level of indirection in all tools that crawle/compile Java source code > - add a level of indirection in all tools that crawle/rewrite bytecodes > How common is it to statically process the byte codes of a JAR file to produce a new JAR file as opposed to doing that dynamically at runtime? (I know of obfuscating tools, which IIRC were more commonly used for mobile?) Are processed JAR files then re-distributed via say maven or kept internally to be used with a specific stack? Would such tools process use the JarFile class to get access to the JAR file contents? > all of that to support a feature that: > - is not clearly defined To me what is not clearly understood is the potential impact, where as the feature itself is actually fairly simple to grok. > - is supposed to only solve cases where delta between versions is small Yes, it's anticipated there would be a small number of classes that would require changing. > - comes with no help from the JDK (how to detect a version, etc). > We can propose such things for 9. For example, new public methods to JarFile. There will be a public JDK version query API in 9, is that what you mean by "how to detect a version?", or do you mean how can one analyze an MVJAR? (or perhaps both...) Paul. > While I understand the comfort for the end user to have only one big fat jar, > it would like to see a prototype that have good answers to all these > questions before including it into the JDK. > > Said simply, tackling such problem requires a JSR not a JEP, because it's a > feature which impact a large number of parts of the Java ecosystem. > > regards, > Rémi
Re: JEP 238: Multi-Version JAR Files
On 04/15/2015 01:04 PM, Paul Sandoz wrote: Hi Markus, Sorry for the late reply i have been away for the last 3 weeks. You and your colleagues might be interested in discussion of JEP 238 on org.apache.maven.dev: http://markmail.org/thread/v5ywgdpuprntrvfu#query:+page:1+mid:v5ywgdpuprntrvfu+state:results especially when it gets to the technical discussion of how maven could support the production of MVJARs. On Mar 31, 2015, at 1:44 PM, Markus Keller wrote: I've discussed this with the Eclipse JDT team. We're very skeptic and we think that JEP 238 goes into a wrong direction. Eclipse JDT does not intend to add special support for it. Building such MVJARs will be quite complicated, since it requires having separate classpaths and compiling against multiple JDKs to produce the release-specific class files that should end up in the same MVJAR. That means you either need a preprocessor that would allow to keep JDK-release-specific code in the same *.java file, or you need to maintain and sync separate *.java files in separate source folders. Is it really quite so complicated? especially if it is anticipated that only a small number of sources files are likely to be affected? Both options are completely counter Java's "write once, run anywhere" idea, whose foundation is to have only one set of source files and no platform-specific compilation switches. This is one of the greatest features of Java. It's what makes reliable code analysis, dependency tracking, and refactoring possible. The trouble with such maxims is they can be easily co-opted to ones point of view :-) (my comment is in jest, but there is a point too) for example i can reasonably argue that MVJARs are actually increasing the ability to run anywhere, and that benefits all consumers of such JARs. Note that in terms of dependency tracking one is not proposing that the dependencies above the major platform would change. Nor would the API (there may be some form of backward compatible change, see the brief discussion between Moh and myself). I agree that changes would be required to both tooling and IDEs. I am interested in knowing how difficult such changes might be. Are there a series of smallish changes, each of which is not terribly difficult, or is there something fundamentally insurmountable? Furthermore, the general problem is not specific to the JDK, but also occurs for other libraries. Therefore, a JDK-specific solution would be at the wrong level. Just because it's JDK specific does not mean it is de-facto at the wrong level. It's attempting to solve a smaller and much simpler problem. A better solution for such cases is to provide an adapter JAR that makes the new APIs available when running on an older JDK. That way, the problem is solved at the origin, and not again and again for each client. And with "the problem", I mean the problem that the old JDK release didn't offer an API for something that clients should use ASAP, even if they can't require the whole new JDK release right away. That does not solve the problem of internal or deprecated APIs present in N going away in N+1. Layering JDK classes (in JARS) on top of the JDK platform is not very practical. Such classes often have special treatment, we don't want to recommend using the boot classpath, and i don't think developers will be very happy having using such options and manage additional dependencies associated with the platform. Of course we cannot feasibly backport every new API in N to N-1, N-2 etc. In selective cases that might be possible, but for the core of the JDK it's like pulling on a string of public dependencies, internal dependencies and perhaps more subtle dependencies between javac and hotspot (Unsafe replacements would be particularly tricky). that why most of the tools like the retroweaver [1] rewrite bytecodes because you can replace a dependency to a JDK class to wire it to a class that will come with your code. Note that in that case, you develop and maintain only one version of the code compatible with the newer version and backport the code to the old one, instead of selectively develop part of the code with the new version as you propose. Paul. cheers, Rémi [1] http://retroweaver.sourceforge.net/
Re: JEP 238: Multi-Version JAR Files
Hi Paul, I think you're seriously underestimate the cost of this JEP. You're asking Java devs to: - be able to maintain several codes with different source level compatibility in the same code tree. - add a level of indirection in all tools that crawle/compile Java source code - add a level of indirection in all tools that crawle/rewrite bytecodes all of that to support a feature that: - is not clearly defined - is supposed to only solve cases where delta between versions is small - comes with no help from the JDK (how to detect a version, etc). While I understand the comfort for the end user to have only one big fat jar, it would like to see a prototype that have good answers to all these questions before including it into the JDK. Said simply, tackling such problem requires a JSR not a JEP, because it's a feature which impact a large number of parts of the Java ecosystem. regards, Rémi On 04/15/2015 01:04 PM, Paul Sandoz wrote: Hi Markus, Sorry for the late reply i have been away for the last 3 weeks. You and your colleagues might be interested in discussion of JEP 238 on org.apache.maven.dev: http://markmail.org/thread/v5ywgdpuprntrvfu#query:+page:1+mid:v5ywgdpuprntrvfu+state:results especially when it gets to the technical discussion of how maven could support the production of MVJARs. On Mar 31, 2015, at 1:44 PM, Markus Keller wrote: I've discussed this with the Eclipse JDT team. We're very skeptic and we think that JEP 238 goes into a wrong direction. Eclipse JDT does not intend to add special support for it. Building such MVJARs will be quite complicated, since it requires having separate classpaths and compiling against multiple JDKs to produce the release-specific class files that should end up in the same MVJAR. That means you either need a preprocessor that would allow to keep JDK-release-specific code in the same *.java file, or you need to maintain and sync separate *.java files in separate source folders. Is it really quite so complicated? especially if it is anticipated that only a small number of sources files are likely to be affected? Both options are completely counter Java's "write once, run anywhere" idea, whose foundation is to have only one set of source files and no platform-specific compilation switches. This is one of the greatest features of Java. It's what makes reliable code analysis, dependency tracking, and refactoring possible. The trouble with such maxims is they can be easily co-opted to ones point of view :-) (my comment is in jest, but there is a point too) for example i can reasonably argue that MVJARs are actually increasing the ability to run anywhere, and that benefits all consumers of such JARs. Note that in terms of dependency tracking one is not proposing that the dependencies above the major platform would change. Nor would the API (there may be some form of backward compatible change, see the brief discussion between Moh and myself). I agree that changes would be required to both tooling and IDEs. I am interested in knowing how difficult such changes might be. Are there a series of smallish changes, each of which is not terribly difficult, or is there something fundamentally insurmountable? Furthermore, the general problem is not specific to the JDK, but also occurs for other libraries. Therefore, a JDK-specific solution would be at the wrong level. Just because it's JDK specific does not mean it is de-facto at the wrong level. It's attempting to solve a smaller and much simpler problem. A better solution for such cases is to provide an adapter JAR that makes the new APIs available when running on an older JDK. That way, the problem is solved at the origin, and not again and again for each client. And with "the problem", I mean the problem that the old JDK release didn't offer an API for something that clients should use ASAP, even if they can't require the whole new JDK release right away. That does not solve the problem of internal or deprecated APIs present in N going away in N+1. Layering JDK classes (in JARS) on top of the JDK platform is not very practical. Such classes often have special treatment, we don't want to recommend using the boot classpath, and i don't think developers will be very happy having using such options and manage additional dependencies associated with the platform. Of course we cannot feasibly backport every new API in N to N-1, N-2 etc. In selective cases that might be possible, but for the core of the JDK it's like pulling on a string of public dependencies, internal dependencies and perhaps more subtle dependencies between javac and hotspot (Unsafe replacements would be particularly tricky). Paul.
Re: JEP 238: Multi-Version JAR Files
Hi Markus, Sorry for the late reply i have been away for the last 3 weeks. You and your colleagues might be interested in discussion of JEP 238 on org.apache.maven.dev: http://markmail.org/thread/v5ywgdpuprntrvfu#query:+page:1+mid:v5ywgdpuprntrvfu+state:results especially when it gets to the technical discussion of how maven could support the production of MVJARs. On Mar 31, 2015, at 1:44 PM, Markus Keller wrote: > I've discussed this with the Eclipse JDT team. We're very skeptic and we > think that JEP 238 goes into a wrong direction. Eclipse JDT does not intend > to add special support for it. > > Building such MVJARs will be quite complicated, since it requires having > separate classpaths and compiling against multiple JDKs to produce the > release-specific class files that should end up in the same MVJAR. That means > you either need a preprocessor that would allow to keep JDK-release-specific > code in the same *.java file, or you need to maintain and sync separate > *.java files in separate source folders. Is it really quite so complicated? especially if it is anticipated that only a small number of sources files are likely to be affected? > Both options are completely counter Java's "write once, run anywhere" idea, > whose foundation is to have only one set of source files and no > platform-specific compilation switches. This is one of the greatest features > of Java. It's what makes reliable code analysis, dependency tracking, and > refactoring possible. > The trouble with such maxims is they can be easily co-opted to ones point of view :-) (my comment is in jest, but there is a point too) for example i can reasonably argue that MVJARs are actually increasing the ability to run anywhere, and that benefits all consumers of such JARs. Note that in terms of dependency tracking one is not proposing that the dependencies above the major platform would change. Nor would the API (there may be some form of backward compatible change, see the brief discussion between Moh and myself). I agree that changes would be required to both tooling and IDEs. I am interested in knowing how difficult such changes might be. Are there a series of smallish changes, each of which is not terribly difficult, or is there something fundamentally insurmountable? > Furthermore, the general problem is not specific to the JDK, but also occurs > for other libraries. Therefore, a JDK-specific solution would be at the wrong > level. > Just because it's JDK specific does not mean it is de-facto at the wrong level. It's attempting to solve a smaller and much simpler problem. > A better solution for such cases is to provide an adapter JAR that makes the > new APIs available when running on an older JDK. That way, the problem is > solved at the origin, and not again and again for each client. And with "the > problem", I mean the problem that the old JDK release didn't offer an API for > something that clients should use ASAP, even if they can't require the whole > new JDK release right away. > That does not solve the problem of internal or deprecated APIs present in N going away in N+1. Layering JDK classes (in JARS) on top of the JDK platform is not very practical. Such classes often have special treatment, we don't want to recommend using the boot classpath, and i don't think developers will be very happy having using such options and manage additional dependencies associated with the platform. Of course we cannot feasibly backport every new API in N to N-1, N-2 etc. In selective cases that might be possible, but for the core of the JDK it's like pulling on a string of public dependencies, internal dependencies and perhaps more subtle dependencies between javac and hotspot (Unsafe replacements would be particularly tricky). Paul.
Re: JEP 238: Multi-Version JAR Files
I've discussed this with the Eclipse JDT team. We're very skeptic and we think that JEP 238 goes into a wrong direction. Eclipse JDT does not intend to add special support for it. Building such MVJARs will be quite complicated, since it requires having separate classpaths and compiling against multiple JDKs to produce the release-specific class files that should end up in the same MVJAR. That means you either need a preprocessor that would allow to keep JDK-release-specific code in the same *.java file, or you need to maintain and sync separate *.java files in separate source folders. Both options are completely counter Java's "write once, run anywhere" idea, whose foundation is to have only one set of source files and no platform-specific compilation switches. This is one of the greatest features of Java. It's what makes reliable code analysis, dependency tracking, and refactoring possible. Furthermore, the general problem is not specific to the JDK, but also occurs for other libraries. Therefore, a JDK-specific solution would be at the wrong level. A better solution for such cases is to provide an adapter JAR that makes the new APIs available when running on an older JDK. That way, the problem is solved at the origin, and not again and again for each client. And with "the problem", I mean the problem that the old JDK release didn't offer an API for something that clients should use ASAP, even if they can't require the whole new JDK release right away. Thanks, Markus Keller IBM Rational Zurich Research Lab Eclipse JDT UI
Re: JEP 238: Multi-Version JAR Files
Sorry for the late reply, getting through email backlogs...
On Mar 1, 2015, at 11:41 AM, Florian Weimer wrote:
> On 02/27/2015 06:16 PM, Paul Sandoz wrote:
>> On Feb 27, 2015, at 4:47 PM, Florian Weimer wrote:
>>> I really don't think this tooling support will provide sufficient
>>> enticement to developers to maintain separate 7/8/9 source branches of
>>> their libraries. Isn't that the main obstacle, and not the way the bits
>>> are delivered?
>>>
>>
>> What if all the source for 7/8/9 bits were under one project?
>
> Tool support for that is still worse than for separate branches of the
> same project.
>
Separate branches and different library versions with most likely backwards
incompatible APIs, right?
> In general, VCS support for branches is still quite poor because the
> sensible thing to do is to develop a feature on the master branch and
> then backport it to older release branches as needed. But such
> selective backwards-merging very much lacks VCS support because the
> least common ancestor logic build into almost all current tools does not
> apply in this scenario. From a tool perspective, developing the feature
> in the oldest relevant release and then merging forward is the only
> supported way. But it's usually bad engineering because you want new
> features to use current facilities of the code base. So the only thing
> you have left is to cherry-pick individual patches (I think thats what
> “hg transplant” does in Mercurial).
>
> Anyway, you lose the tiny bit of tool support you have for backporting
> if you have everything in the same source repository.
>
>> For example, say there is a source file:
>>
>> src/main/java/foo/Foo.java
>>
>> whose content is:
>>
>> import sun.misc.BASE64Decoder;
>>
>> public class Foo {
>>// does something with sun.misc.BASE64Decoder
>> }
>>
>> There might be another source file located in the 8 area that overrides that
>> in the unversioned area:
>>
>> src/main-8/java/foo/Foo.java
>>
>> whose content is:
>>
>> import java.util.Base64;
>>
>> public class Foo {
>>// does something equivalent with java.util.Base64
>> }
>
> This really screams preprocessor. :-(
That's another approach, generate the sources using a pre-processor, which of
course has it's own issues w.r.t. IDEs. Note i am not trying to dictate a
particular project approach here.
>
>> The public contract of Foo should remain identical across the major Java
>> platform dependent versions, in a more strict sense the public signatures in
>> the byte code should be identical (the jar tool has been modified to check
>> this).
>
> If that's the goal, something like Retroweaver seems far more
> appropriate than forcing library authors to maintain separate sources.
>
Consider the case of library code using Unsafe for lexicographical unsigned
byte[] comparison. It might be quite tricky for a tool to automate without the
library itself telling the tool what to do. We are not proposing something more
sophisticated such as runtime code transformations (discussed further by Remi).
> Multi-version JAR files seem useful because they will eventually allow
> evolution of the class file format.
Sorry, you lost me on that point.
Paul.
> But I think the current use case
> isn't really there, I'm afraid.
>
> --
> Florian Weimer / Red Hat Product Security
Re: JEP 238: Multi-Version JAR Files
On 1 March 2015 at 23:37, Remi Forax wrote: > You only maintain one module which depend on the latest version. > When you ship the module let say as a jar, you also ship the way to > downgrade it as an executable code. > At install time, when all the modules are known, before trying to create the > images, > the code that downgrades the module is executed if necessary. > This code can rewrite the whole bytecodes of all the classes the module thus > ensure backward compatibility without providing binary backward > compatibility. This direction certainly has potential. In many ways, the biggest potential benefit any language could bring is the ability to handle change of code over time (ie. its a huge potential language feature for any new language). While we've always had deprecation, there is no actual mechanism in code/compiler/linker terms to ensure that the client has followed the deprecation (callers just get an error if the code is removed). Thus, what is really needed is a standard compiler/language mechanism to map old code to new code. It would allow a method to be renamed (eg. Duration.getNano() to getNanos()) It would allow a class to be renamed, say java.util.Date to DumbOldTimestampThing It would allow an enum constant to be renamed If done well, it might allow for much more complex refactoring changes, but there would have to be limits This comes under the modules banner in my eyes, because modules is adding a linker step where this kind of stuff could be done. I hope its on the radar (maybe fore JDK 10 rather than 9), as its a lot more useful than MVJARs alone (which seem to have a pretty limited use, one which could be helped over time just by making reflection easier to use). Note that this is slightly different to Remi's suggestion, which is "downgrading" new code to run on an old JDK. Here I'm discussing "upgrading" old code to run against a newer JDK. Its likely that the both mechanisms make sense. Stephen
Re: JEP 238: Multi-Version JAR Files
On 02/25/2015 05:27 PM, Brian Goetz wrote: On 2/12/2015 5:59 PM, Stephen Colebourne wrote: I would expect IDEs to have some considerable work to do. Agree on the "work" part, but I doubt it is "considerable". For creating MV JARs, the 'jar' tool does all the heavy lifting. For running Java apps, the classloader does all the heavy lifting. For tools that have to consume JARs, the JarFile API does all the heavy lifting. If you use JarFile, it's a one-line change to the constructor to get a version-specific view of the JAR. So in each of these cases, the work is limited to: - Figure out how you intend to interact with MVJars; - Configure the appropriate tool (jar tool, JarFile) appropriately; this is generally a matter of new constructor arguments and/or new command line arguments. So I totally agree there will be lots of things that change, but those changes should be individually quite small (and this is consistent with our experience supporting MVJars in the JDK tools.) One aspect that is overlooked here is the "coding phase" which is nowadays performed in IDEs. Refactoring and such. One would like to have all versions of a particular class targeting distinct platforms accessible in IDE not only for primitive text-editing, but also for refactoring. IDEs currently don't like seeing multiple versions of same class at the same time. Supporting this in an IDE is a considerable work to do, I think, and requires IDE to "understand" multiple source files for same class name as distinct versions of the same class which it must keep in sync regarding the public API of the class. I don't think any IDE has this mechanisms in place currently, but I might be wrong. Does anybody know? Peter
Re: JEP 238: Multi-Version JAR Files
Hi Moh, On Feb 27, 2015, at 7:23 PM, "Rezaei, Mohammad A." wrote: > Why do you expect the new classes in the JDK not to be part of the API? An MVJAR is one unit of release. Should it have two or more public APIs? if so what is it's version and set of dependencies? Can it be deployed to maven central? Are those APIs source and/or binary compatibility? So far we have taken a strict approach in terms of compatibility as it simplifies certain things at the expense of restricting other things as you have noted. Perhaps it's possible to relax that approach to some degree of compatibility? In your use-case could CustomFunction be extended to support Function in a source and binary compatible manner? In general, as i tried to allude in the design document, there is this tension between: 1) supporting the same API across multiple versions of the Java platform; and 2) creating a new major version of the API, breaking compatibility with the old API and starting with a newer base Java platform version. This tension is always going to exist, especially for popular libraries. MVJARs is focused on the first case for some definition of "same". Paul. > Simple example: > > I have a library that's 5 years old. The API needed the equivalent of > java.util.Function (from Java 8), which obviously was not there when I wrote > my library. Let's say I had defined CustomFunction and now I want the API to > use Function. > > This sort of useful abstraction has been part and parcel with new JDK's for a > long time (e.g. HashTable -> Map [1.2], String -> CharSequence [1.4], > Generics [1.5], Deque [1.6], AutoClosable [1.7], a dozen useful functional > interfaces [1.8]). > > Currently, my choices are: > 1) Abandon multi-jdk compatibility and release a new version of library for > the new jdk. Keep the new version source compatible by making CustomFunction > extend Function (possibly with a default delegating method). > 2) Have two versions of the code base and release separate jars for each, > porting new stuff between the two versions for a while. > > How does an MV jar give me a third choice? > > Thanks > Moh
Re: JEP 238: Multi-Version JAR Files
On 03/02/2015 12:37 AM, Remi Forax wrote: On 03/01/2015 07:08 PM, Peter Levart wrote: Hi Peter, You can see the whole thing in the opposite way which I think is less disruptive in term of tooling. You only maintain one module which depend on the latest version. When you ship the module let say as a jar, you also ship the way to downgrade it as an executable code. At install time, when all the modules are known, before trying to create the images, the code that downgrades the module is executed if necessary. This code can rewrite the whole bytecodes of all the classes the module thus ensure backward compatibility without providing binary backward compatibility. It's a win-win-win strategy, your code doesn't live in the past, the bytecode rewriter can fail saying there is a problem before runtime (if by example a method is missing), you can break the binary backward compatibility if this is necessary and still being compatible. You can see this mechanism as a kind of annotation processor that will be executed at install time instead of at compile time. cheers, Rémi Such "way to downgrade executable code" is something that doesn't exist (yet)? Is it planned? I imagine this as a kind of compiler. It has to understand executable code, identify "problematic platform-dependent code sequences / API usages" and convert them to semantically equivalent code targeted at specific platform. Who is going to write and support such tool? Should it be extensible, so that the application programmer can extend and "program" it? How are semantically equivalent code sequences targeting specific platform going to be represented? How are "problematic platform-dependent code patterns / API usages" going to be represented? We already have a language-level way to represent semantically equivalent code sequences targeting specific platforms - different implementations of the same interface. By choosing the appropriate implementation at runtime, we can avoid needing a tool that understands executable code and identifies problematic platform-dependent code. In terms of tooling, I don't see any problems as good IDEs and even Maven already support specifying platform dependencies / compilation for different target platforms at the module-level in the same project. But I may be persuaded by a good implementation of such "retroweaver". There was one such tool in JDK 1.4 -> 1.5 times. Regards, Peter On 03/01/2015 12:53 PM, Remi Forax wrote: Currently, there are two ways to solve the Base64 issue: - by loading at runtime either the class that use java.util.Base64 or the class sun.misc.BASE64Decoder and use it through an interface And this, I think, is also the cleanest way to approach a problem that multi-version JAR wants to solve. The only argument against it is that it perhaps requires more work and planing, but that doesn't mean it is less maintainable. In general one has to split the software into "modules": - base module (platform independent code) + facade interfaces to access platform dependent functionality - platform dependent module(s) that depend on base module and contain implementations of facade interfaces published as services through ServiceLoader - one module per "target platform version" / "platform dependent feature" combination Base module is compiled 1st (it does not depend on platform version, so it should be compiled with lowest version javac/rt.jar to be able to run on any target platform) Platform dependent module(s) are compiled against classes of base module and with javac/rt.jar of the desired target platform(s). Now if module system of JDK9 had a feature that would enable/disable the deployed module conditionally, then that condition could be an exact target platform version. But I know, multi-version JAR wants to solve similar problem for pre-JDK9 target platforms too which don't yet have modules. The approach to organizing sources and compiling can be the same as described, just packaging can then use multi-version JARs instead of "conditional" modules. One problem I see with requirering that in multi-version JARs each version sub-path must have same public classes with same public APIs is that IDEs don't like that. They don't like to see multiple versions of same class and they paint them red. Refactoring doesn't work correctly if IDEs see multiple versions of same class and can never be made to work correctly as it would have to keep versions always in sync. I always have problems with IDEA when I present it the unix/windows versions of classes in JDK build at the same time. This can be solved nicely by not requirering that and simply relying on the programmer that (s)he knows what (s)he is doing. With interfaces as services and implementations locateable via ServiceLoader, one does not have to use same names of classes for implementations of interfaces targeting different platforms and IDEs will be happy. Re
Re: JEP 238: Multi-Version JAR Files
On 03/01/2015 07:08 PM, Peter Levart wrote: Hi Peter, You can see the whole thing in the opposite way which I think is less disruptive in term of tooling. You only maintain one module which depend on the latest version. When you ship the module let say as a jar, you also ship the way to downgrade it as an executable code. At install time, when all the modules are known, before trying to create the images, the code that downgrades the module is executed if necessary. This code can rewrite the whole bytecodes of all the classes the module thus ensure backward compatibility without providing binary backward compatibility. It's a win-win-win strategy, your code doesn't live in the past, the bytecode rewriter can fail saying there is a problem before runtime (if by example a method is missing), you can break the binary backward compatibility if this is necessary and still being compatible. You can see this mechanism as a kind of annotation processor that will be executed at install time instead of at compile time. cheers, Rémi On 03/01/2015 12:53 PM, Remi Forax wrote: Currently, there are two ways to solve the Base64 issue: - by loading at runtime either the class that use java.util.Base64 or the class sun.misc.BASE64Decoder and use it through an interface And this, I think, is also the cleanest way to approach a problem that multi-version JAR wants to solve. The only argument against it is that it perhaps requires more work and planing, but that doesn't mean it is less maintainable. In general one has to split the software into "modules": - base module (platform independent code) + facade interfaces to access platform dependent functionality - platform dependent module(s) that depend on base module and contain implementations of facade interfaces published as services through ServiceLoader - one module per "target platform version" / "platform dependent feature" combination Base module is compiled 1st (it does not depend on platform version, so it should be compiled with lowest version javac/rt.jar to be able to run on any target platform) Platform dependent module(s) are compiled against classes of base module and with javac/rt.jar of the desired target platform(s). Now if module system of JDK9 had a feature that would enable/disable the deployed module conditionally, then that condition could be an exact target platform version. But I know, multi-version JAR wants to solve similar problem for pre-JDK9 target platforms too which don't yet have modules. The approach to organizing sources and compiling can be the same as described, just packaging can then use multi-version JARs instead of "conditional" modules. One problem I see with requirering that in multi-version JARs each version sub-path must have same public classes with same public APIs is that IDEs don't like that. They don't like to see multiple versions of same class and they paint them red. Refactoring doesn't work correctly if IDEs see multiple versions of same class and can never be made to work correctly as it would have to keep versions always in sync. I always have problems with IDEA when I present it the unix/windows versions of classes in JDK build at the same time. This can be solved nicely by not requirering that and simply relying on the programmer that (s)he knows what (s)he is doing. With interfaces as services and implementations locateable via ServiceLoader, one does not have to use same names of classes for implementations of interfaces targeting different platforms and IDEs will be happy. Regards, Peter
Re: JEP 238: Multi-Version JAR Files
On 03/01/2015 12:53 PM, Remi Forax wrote: Currently, there are two ways to solve the Base64 issue: - by loading at runtime either the class that use java.util.Base64 or the class sun.misc.BASE64Decoder and use it through an interface And this, I think, is also the cleanest way to approach a problem that multi-version JAR wants to solve. The only argument against it is that it perhaps requires more work and planing, but that doesn't mean it is less maintainable. In general one has to split the software into "modules": - base module (platform independent code) + facade interfaces to access platform dependent functionality - platform dependent module(s) that depend on base module and contain implementations of facade interfaces published as services through ServiceLoader - one module per "target platform version" / "platform dependent feature" combination Base module is compiled 1st (it does not depend on platform version, so it should be compiled with lowest version javac/rt.jar to be able to run on any target platform) Platform dependent module(s) are compiled against classes of base module and with javac/rt.jar of the desired target platform(s). Now if module system of JDK9 had a feature that would enable/disable the deployed module conditionally, then that condition could be an exact target platform version. But I know, multi-version JAR wants to solve similar problem for pre-JDK9 target platforms too which don't yet have modules. The approach to organizing sources and compiling can be the same as described, just packaging can then use multi-version JARs instead of "conditional" modules. One problem I see with requirering that in multi-version JARs each version sub-path must have same public classes with same public APIs is that IDEs don't like that. They don't like to see multiple versions of same class and they paint them red. Refactoring doesn't work correctly if IDEs see multiple versions of same class and can never be made to work correctly as it would have to keep versions always in sync. I always have problems with IDEA when I present it the unix/windows versions of classes in JDK build at the same time. This can be solved nicely by not requirering that and simply relying on the programmer that (s)he knows what (s)he is doing. With interfaces as services and implementations locateable via ServiceLoader, one does not have to use same names of classes for implementations of interfaces targeting different platforms and IDEs will be happy. Regards, Peter
Re: JEP 238: Multi-Version JAR Files
On 03/01/2015 11:41 AM, Florian Weimer wrote:
On 02/27/2015 06:16 PM, Paul Sandoz wrote:
On Feb 27, 2015, at 4:47 PM, Florian Weimer wrote:
I really don't think this tooling support will provide sufficient
enticement to developers to maintain separate 7/8/9 source branches of
their libraries. Isn't that the main obstacle, and not the way the bits
are delivered?
What if all the source for 7/8/9 bits were under one project?
Tool support for that is still worse than for separate branches of the
same project.
In general, VCS support for branches is still quite poor because the
sensible thing to do is to develop a feature on the master branch and
then backport it to older release branches as needed. But such
selective backwards-merging very much lacks VCS support because the
least common ancestor logic build into almost all current tools does not
apply in this scenario. From a tool perspective, developing the feature
in the oldest relevant release and then merging forward is the only
supported way. But it's usually bad engineering because you want new
features to use current facilities of the code base. So the only thing
you have left is to cherry-pick individual patches (I think thats what
“hg transplant” does in Mercurial).
Anyway, you lose the tiny bit of tool support you have for backporting
if you have everything in the same source repository.
For example, say there is a source file:
src/main/java/foo/Foo.java
whose content is:
import sun.misc.BASE64Decoder;
public class Foo {
// does something with sun.misc.BASE64Decoder
}
There might be another source file located in the 8 area that overrides that in
the unversioned area:
src/main-8/java/foo/Foo.java
whose content is:
import java.util.Base64;
public class Foo {
// does something equivalent with java.util.Base64
}
This really screams preprocessor. :-(
The public contract of Foo should remain identical across the major Java
platform dependent versions, in a more strict sense the public signatures in
the byte code should be identical (the jar tool has been modified to check
this).
If that's the goal, something like Retroweaver seems far more
appropriate than forcing library authors to maintain separate sources.
yes, i fully agree,
in that case you need to bundle
- a jar compatible with the lastest version
- and some metadata to help the 'retroweaver' to transform the new code
to the old code.
then you need a special step during the install process where you can
inspect the dependencies
and choose to run the 'retroweaver' or not.
Multi-version JAR files seem useful because they will eventually allow
evolution of the class file format. But I think the current use case
isn't really there, I'm afraid.
I think being able to ship several versions in one jar is a valid use
case but the solution seems causing more troubles than just solving this
use case.
Currently, there are two ways to solve the Base64 issue:
- by loading at runtime either the class that use java.util.Base64 or
the class sun.misc.BASE64Decoder and use it through an interface
- by using using only one class this a code like this:
try {
// use java.util.Base64
} catch(NoClassDefFoundError e) {
// use sun.misc.BASE64Decoder
}
at compile time, you need to provide a fake empty class
sun.misc.BASE64Decoder
that will not be bundled in the destination jar.
Rémi
Re: JEP 238: Multi-Version JAR Files
On 02/27/2015 06:16 PM, Paul Sandoz wrote:
> On Feb 27, 2015, at 4:47 PM, Florian Weimer wrote:
>> I really don't think this tooling support will provide sufficient
>> enticement to developers to maintain separate 7/8/9 source branches of
>> their libraries. Isn't that the main obstacle, and not the way the bits
>> are delivered?
>>
>
> What if all the source for 7/8/9 bits were under one project?
Tool support for that is still worse than for separate branches of the
same project.
In general, VCS support for branches is still quite poor because the
sensible thing to do is to develop a feature on the master branch and
then backport it to older release branches as needed. But such
selective backwards-merging very much lacks VCS support because the
least common ancestor logic build into almost all current tools does not
apply in this scenario. From a tool perspective, developing the feature
in the oldest relevant release and then merging forward is the only
supported way. But it's usually bad engineering because you want new
features to use current facilities of the code base. So the only thing
you have left is to cherry-pick individual patches (I think thats what
“hg transplant” does in Mercurial).
Anyway, you lose the tiny bit of tool support you have for backporting
if you have everything in the same source repository.
> For example, say there is a source file:
>
> src/main/java/foo/Foo.java
>
> whose content is:
>
> import sun.misc.BASE64Decoder;
>
> public class Foo {
> // does something with sun.misc.BASE64Decoder
> }
>
> There might be another source file located in the 8 area that overrides that
> in the unversioned area:
>
> src/main-8/java/foo/Foo.java
>
> whose content is:
>
> import java.util.Base64;
>
> public class Foo {
> // does something equivalent with java.util.Base64
> }
This really screams preprocessor. :-(
> The public contract of Foo should remain identical across the major Java
> platform dependent versions, in a more strict sense the public signatures in
> the byte code should be identical (the jar tool has been modified to check
> this).
If that's the goal, something like Retroweaver seems far more
appropriate than forcing library authors to maintain separate sources.
Multi-version JAR files seem useful because they will eventually allow
evolution of the class file format. But I think the current use case
isn't really there, I'm afraid.
--
Florian Weimer / Red Hat Product Security
Re: JEP 238: Multi-Version JAR Files
Le 27/02/15 16:47, Florian Weimer a écrit : > Maybe I misunderstood the multi-version JAR files proposal, but I > thought that the assumption there is that there are actual *source* > files which use newer features of the platform. > > I really don't think this tooling support will provide sufficient > enticement to developers to maintain separate 7/8/9 source branches of > their libraries. Some projects do, while admittedly maybe only a minority. For example Apache Spatial Information System (SIS) has JDK6, JDK7 and JDK8 branches. Development is done on JDK8, then ported down to JDK6, which is the target JDK platform for current releases [1]. Martin [1] http://sis.apache.org/branches.html
RE: JEP 238: Multi-Version JAR Files
Why do you expect the new classes in the JDK not to be part of the API? Simple
example:
I have a library that's 5 years old. The API needed the equivalent of
java.util.Function (from Java 8), which obviously was not there when I wrote my
library. Let's say I had defined CustomFunction and now I want the API to use
Function.
This sort of useful abstraction has been part and parcel with new JDK's for a
long time (e.g. HashTable -> Map [1.2], String -> CharSequence [1.4], Generics
[1.5], Deque [1.6], AutoClosable [1.7], a dozen useful functional interfaces
[1.8]).
Currently, my choices are:
1) Abandon multi-jdk compatibility and release a new version of library for the
new jdk. Keep the new version source compatible by making CustomFunction extend
Function (possibly with a default delegating method).
2) Have two versions of the code base and release separate jars for each,
porting new stuff between the two versions for a while.
How does an MV jar give me a third choice?
Thanks
Moh
>-Original Message-
>From: core-libs-dev [mailto:core-libs-dev-boun...@openjdk.java.net] On Behalf
>Of Paul Sandoz
>Sent: Friday, February 27, 2015 12:16 PM
>Cc: core-libs-dev@openjdk.java.net
>Subject: Re: JEP 238: Multi-Version JAR Files
>
>On Feb 27, 2015, at 4:47 PM, Florian Weimer wrote:
>> I really don't think this tooling support will provide sufficient
>> enticement to developers to maintain separate 7/8/9 source branches of
>> their libraries. Isn't that the main obstacle, and not the way the bits
>> are delivered?
>>
>
>What if all the source for 7/8/9 bits were under one project?
>
>Hypothetical project layout:
>
> src/main/java
> src/main/resources
> src/test/java
> src/test/resources
> src/main-8/java
> src/main-8/resources
> src/test-8/java
> src/test-8/resources
> src/main-9/java
> src/main-9/resources
> src/test-9/java
> src/test-9/resources
>
>(If this were a maven-kind of project there would be one pom.xml, one version
>and one set of dependencies, and one (MV) JAR produced when packaging.)
>
>I would anticipate most of the source would reside under src/main/java then
>there would be "overriding" source in the versioned areas for classes that use
>replace usage of say internal JDK features with public JDK features.
>
>For example, say there is a source file:
>
> src/main/java/foo/Foo.java
>
>whose content is:
>
> import sun.misc.BASE64Decoder;
>
> public class Foo {
>// does something with sun.misc.BASE64Decoder
> }
>
>There might be another source file located in the 8 area that overrides that
>in the unversioned area:
>
> src/main-8/java/foo/Foo.java
>
>whose content is:
>
> import java.util.Base64;
>
> public class Foo {
>// does something equivalent with java.util.Base64
> }
>
>The public contract of Foo should remain identical across the major Java
>platform dependent versions, in a more strict sense the public signatures in
>the byte code should be identical (the jar tool has been modified to check
>this).
>
>Paul.
Re: JEP 238: Multi-Version JAR Files
On Feb 27, 2015, at 4:47 PM, Florian Weimer wrote:
> I really don't think this tooling support will provide sufficient
> enticement to developers to maintain separate 7/8/9 source branches of
> their libraries. Isn't that the main obstacle, and not the way the bits
> are delivered?
>
What if all the source for 7/8/9 bits were under one project?
Hypothetical project layout:
src/main/java
src/main/resources
src/test/java
src/test/resources
src/main-8/java
src/main-8/resources
src/test-8/java
src/test-8/resources
src/main-9/java
src/main-9/resources
src/test-9/java
src/test-9/resources
(If this were a maven-kind of project there would be one pom.xml, one version
and one set of dependencies, and one (MV) JAR produced when packaging.)
I would anticipate most of the source would reside under src/main/java then
there would be "overriding" source in the versioned areas for classes that use
replace usage of say internal JDK features with public JDK features.
For example, say there is a source file:
src/main/java/foo/Foo.java
whose content is:
import sun.misc.BASE64Decoder;
public class Foo {
// does something with sun.misc.BASE64Decoder
}
There might be another source file located in the 8 area that overrides that in
the unversioned area:
src/main-8/java/foo/Foo.java
whose content is:
import java.util.Base64;
public class Foo {
// does something equivalent with java.util.Base64
}
The public contract of Foo should remain identical across the major Java
platform dependent versions, in a more strict sense the public signatures in
the byte code should be identical (the jar tool has been modified to check
this).
Paul.
Re: JEP 238: Multi-Version JAR Files
On 02/27/2015 03:20 PM, Alan Bateman wrote: > On 27/02/2015 13:27, Florian Weimer wrote: >> : >> I'm wondering how you propose to build such JAR files. Do you think >> library developers will maintain two separate branches, compile one with >> JDK 8, the other one with JDK 9, and then use some (not yet existing?) >> tool to merge the output into a single JAR? > > This submitted JEP is good reading and gives an idea how how to compile > to older JDKs: > http://openjdk.java.net/jeps/8058150 Maybe I misunderstood the multi-version JAR files proposal, but I thought that the assumption there is that there are actual *source* files which use newer features of the platform. I really don't think this tooling support will provide sufficient enticement to developers to maintain separate 7/8/9 source branches of their libraries. Isn't that the main obstacle, and not the way the bits are delivered? -- Florian Weimer / Red Hat Product Security
Re: JEP 238: Multi-Version JAR Files
On Feb 27, 2015, at 2:27 PM, Florian Weimer wrote: > On 02/12/2015 09:52 PM, Paul Sandoz wrote: >> Hi >> >> In connection with the JEP there is also a design document to help the >> discussion: >> >> http://cr.openjdk.java.net/~psandoz/jdk9/MultiVersionJar-8u60-9-design.md >> >> We are especially interesting in hearing feedback from library developers, >> tool/IDE developers, and anyone doing funky stuff with class loading and JAR >> files. > > I'm wondering how you propose to build such JAR files. Do you think > library developers will maintain two separate branches, compile one with > JDK 8, the other one with JDK 9, and then use some (not yet existing?) > tool to merge the output into a single JAR? Is such automated merging > even possible if the bytecode was compiled with different javac versions? > In the last section of the design document i was trying to anticipate a particular approach, a multiversion project, with examples for compiling and packaging. That is out of scope of the MVJAR effort itself but still very important and is dependent on getting buy in from tooling such as maven and gradle. > What about presenting Javadoc in a useful fashion? A MVJAR should only have one API. Paul.
Re: JEP 238: Multi-Version JAR Files
On 27/02/2015 13:27, Florian Weimer wrote: : I'm wondering how you propose to build such JAR files. Do you think library developers will maintain two separate branches, compile one with JDK 8, the other one with JDK 9, and then use some (not yet existing?) tool to merge the output into a single JAR? This submitted JEP is good reading and gives an idea how how to compile to older JDKs: http://openjdk.java.net/jeps/8058150 -Alan.
Re: JEP 238: Multi-Version JAR Files
On 02/27/2015 07:27 AM, Florian Weimer wrote: On 02/12/2015 09:52 PM, Paul Sandoz wrote: Hi In connection with the JEP there is also a design document to help the discussion: http://cr.openjdk.java.net/~psandoz/jdk9/MultiVersionJar-8u60-9-design.md We are especially interesting in hearing feedback from library developers, tool/IDE developers, and anyone doing funky stuff with class loading and JAR files. I'm wondering how you propose to build such JAR files. Do you think library developers will maintain two separate branches, compile one with JDK 8, the other one with JDK 9, and then use some (not yet existing?) tool to merge the output into a single JAR? Is such automated merging even possible if the bytecode was compiled with different javac versions? This is basically what I was getting at in my post. It's interesting that JDK-8058150 (Compile for Specific Platform Version) is surfacing at the same time, because it seems to me that this would make it possible for javac to compile for multiple targets in one pass. Maybe not easy, but possible. :-) What about presenting Javadoc in a useful fashion? Good question... -- - DML
Re: JEP 238: Multi-Version JAR Files
On 02/12/2015 09:52 PM, Paul Sandoz wrote: > Hi > > In connection with the JEP there is also a design document to help the > discussion: > > http://cr.openjdk.java.net/~psandoz/jdk9/MultiVersionJar-8u60-9-design.md > > We are especially interesting in hearing feedback from library developers, > tool/IDE developers, and anyone doing funky stuff with class loading and JAR > files. I'm wondering how you propose to build such JAR files. Do you think library developers will maintain two separate branches, compile one with JDK 8, the other one with JDK 9, and then use some (not yet existing?) tool to merge the output into a single JAR? Is such automated merging even possible if the bytecode was compiled with different javac versions? What about presenting Javadoc in a useful fashion? -- Florian Weimer / Red Hat Product Security
Re: JEP 238: Multi-Version JAR Files
On Feb 27, 2015, at 9:00 AM, Paul Sandoz wrote: > > On Feb 26, 2015, at 5:02 PM, Paul Sandoz wrote: > >> Hi, >> >> If anyone wants to give this a test drive see stuff in here: >> >> http://cr.openjdk.java.net/~psandoz/multiversion-jar/ >> >> produced by Steve (CC'ed) who has done all the development. >> > > Another correction, CC'ing Steve this time... > Hmm it seems the CC field is being removed by the server. Anyone else noticing this? FTR here is Steve's email: Steve Drach Paul.
Re: JEP 238: Multi-Version JAR Files
On Feb 26, 2015, at 5:02 PM, Paul Sandoz wrote: > Hi, > > If anyone wants to give this a test drive see stuff in here: > > http://cr.openjdk.java.net/~psandoz/multiversion-jar/ > > produced by Steve (CC'ed) who has done all the development. > Another correction, CC'ing Steve this time... Paul.
Re: JEP 238: Multi-Version JAR Files
On Feb 26, 2015, at 5:02 PM, Paul Sandoz wrote: > Hi, > > If anyone wants to give this a test drive see stuff in here: > > http://cr.openjdk.java.net/~psandoz/multiversion-jar/ > > produced by Steve (CC'ed) who has done all the development. > > For example: > > multiversion-jar $ java -version > java version "1.7.0_72" > Java(TM) SE Runtime Environment (build 1.7.0_72-b14) > Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode) > multiversion-jar $ > multiversion-jar $ java -jar version.jar > I am running on version 7 > > > multiversion-jar $ java -jar version.jar > I am running on version 7 The above two lines should be: multiversion-jar $ java -version java version "1.8.0" Java(TM) SE Runtime Environment (build 1.8.0-b132) Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode) > multiversion-jar $ > multiversion-jar $ java -jar version.jar > I am running on version 7 > multiversion-jar $ > multiversion-jar $ java -Xbootclasspath/p:URLClassPath.jar -jar version.jar > I am running on version 8 > > > multiversion-jar $ java -version > java version "1.9.0-internal" > Java(TM) SE Runtime Environment (build > 1.9.0-internal-sandoz_2015_01_26_16_48-b00) > Java HotSpot(TM) 64-Bit Server VM (build > 1.9.0-internal-sandoz_2015_01_26_16_48-b00, mixed mode) > multiversion-jar $ > multiversion-jar $ java -jar version.jar > I am running on version 7 > multiversion-jar $ > multiversion-jar $ java -Xbootclasspath/p:URLClassPath.jar -jar version.jar > I am running on version 9 > > Paul. > > On Feb 12, 2015, at 9:52 PM, Paul Sandoz wrote: > >> Hi >> >> In connection with the JEP there is also a design document to help the >> discussion: >> >> http://cr.openjdk.java.net/~psandoz/jdk9/MultiVersionJar-8u60-9-design.md >> >> We are especially interesting in hearing feedback from library developers, >> tool/IDE developers, and anyone doing funky stuff with class loading and JAR >> files. >> >> Paul. >> >> On Feb 12, 2015, at 9:41 PM, mark.reinh...@oracle.com wrote: >> >>> New JEP Candidate: http://openjdk.java.net/jeps/238 >>> >>> - Mark >> >
Re: JEP 238: Multi-Version JAR Files
Hi, If anyone wants to give this a test drive see stuff in here: http://cr.openjdk.java.net/~psandoz/multiversion-jar/ produced by Steve (CC'ed) who has done all the development. For example: multiversion-jar $ java -version java version "1.7.0_72" Java(TM) SE Runtime Environment (build 1.7.0_72-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode) multiversion-jar $ multiversion-jar $ java -jar version.jar I am running on version 7 multiversion-jar $ java -jar version.jar I am running on version 7 multiversion-jar $ multiversion-jar $ java -jar version.jar I am running on version 7 multiversion-jar $ multiversion-jar $ java -Xbootclasspath/p:URLClassPath.jar -jar version.jar I am running on version 8 multiversion-jar $ java -version java version "1.9.0-internal" Java(TM) SE Runtime Environment (build 1.9.0-internal-sandoz_2015_01_26_16_48-b00) Java HotSpot(TM) 64-Bit Server VM (build 1.9.0-internal-sandoz_2015_01_26_16_48-b00, mixed mode) multiversion-jar $ multiversion-jar $ java -jar version.jar I am running on version 7 multiversion-jar $ multiversion-jar $ java -Xbootclasspath/p:URLClassPath.jar -jar version.jar I am running on version 9 Paul. On Feb 12, 2015, at 9:52 PM, Paul Sandoz wrote: > Hi > > In connection with the JEP there is also a design document to help the > discussion: > > http://cr.openjdk.java.net/~psandoz/jdk9/MultiVersionJar-8u60-9-design.md > > We are especially interesting in hearing feedback from library developers, > tool/IDE developers, and anyone doing funky stuff with class loading and JAR > files. > > Paul. > > On Feb 12, 2015, at 9:41 PM, mark.reinh...@oracle.com wrote: > >> New JEP Candidate: http://openjdk.java.net/jeps/238 >> >> - Mark >
Re: JEP 238: Multi-Version JAR Files
On 25/02/2015 18:03, Paul Sandoz wrote: On Feb 25, 2015, at 6:45 PM, Stephen Colebourne wrote: On 25 February 2015 at 13:30, Paul Sandoz wrote: Even in the modular world i will expect class scanning will be used. While we can now iterate reliably over classes in the image i don't believe that functionality is made available for classes in modules on the module path. To be honest, being able to reliably iterate over classes in a module is the most interesting potential feature about modules in general. Really? I would have thought that the most interesting aspects would be formalizing names and dependencies so they can be computed on. Right, plus encapsulation too. I've no doubt that there will be APIs (both new and updates to existing) to introspect the contents of modules but I wouldn't consider this one of the primary goals or benefits of modules, rather it's just something that has to happen anyway as part of bringing modules to the platform. -Alan
Re: JEP 238: Multi-Version JAR Files
On 02/25/2015 11:41 AM, Paul Sandoz wrote: On Feb 25, 2015, at 5:27 PM, Brian Goetz wrote: On 2/12/2015 5:59 PM, Stephen Colebourne wrote: Interesting direction. Reading carefully, the goal is actually very limited in scope, by preventing any public API changes. It doesn't help adoption of JSR-310 for example, but will be useful for Unsafe, which is clearly a motivating factor. I would expect IDEs to have some considerable work to do. Agree on the "work" part, but I doubt it is "considerable". For creating MV JARs, the 'jar' tool does all the heavy lifting. For running Java apps, the classloader does all the heavy lifting. For tools that have to consume JARs, the JarFile API does all the heavy lifting. If you use JarFile, it's a one-line change to the constructor to get a version-specific view of the JAR. So in each of these cases, the work is limited to: - Figure out how you intend to interact with MVJars; - Configure the appropriate tool (jar tool, JarFile) appropriately; this is generally a matter of new constructor arguments and/or new command line arguments. So I totally agree there will be lots of things that change, but those changes should be individually quite small (and this is consistent with our experience supporting MVJars in the JDK tools.) In the design doc i outlined how to explicitly compile an mv-style project. How hard would it be to hack up a new kind of maven project layout and javac compilation task that did something similar? Does anybody with maven expertise know more? I've done a little bit of work on the maven-compiler-plugin internals and its plexus buddies, and overall I don't think this would necessarily be a massive undertaking, though it might have to be, depending on some factors: 1) If the JDK-agnostic sources are required to be fully self-contained (i.e. no "outward" dependencies on JDK-specific code), it would be trivial to compile the JDK-agnostic code first, then the JDK-specific code which references the other code as dependencies. However this arrangement is practically quite useless for the average developer who is doing this just so they could have just one JDK-specific version of some class used by their main code. 2) In the more likely case that the -specific and -agnostic code is interdependent, it gets trickier because the compiler task would have to (I guess) compile the code N times, once for each JDK, while also correlating the -specific files and discarding the redundant -agnostic files (leaving aside weirdness that may occur if the files actually *differ* due to constant expansion weirdness and so on). This thought leaves me feeling not so great - it seems like there are many subtle ways it could fail. 3) The best scenario however is that JavaCompiler will magically do all this work for us, figuring out (for example) the constant stuff and other potential weirdness, and then organizing the resultant .class files in their appropriate places based on some input configuration/cooperation with the JavaFileManager for the task. In this case I'd imagine that the maven-compiler-plugin would work as today, with just a few extra switches to point out the JDK-specific source directories, and Maven would be blissfully ignorant of any other considerations. -- - DML
Re: JEP 238: Multi-Version JAR Files
On Feb 25, 2015, at 6:45 PM, Stephen Colebourne wrote: > On 25 February 2015 at 13:30, Paul Sandoz wrote: >> Even in the modular world i will expect class scanning will be used. While >> we can now iterate reliably over classes in the image i don't believe that >> functionality is made available for classes in modules on the module path. > > To be honest, being able to reliably iterate over classes in a module > is the most interesting potential feature about modules in general. Really? I would have thought that the most interesting aspects would be formalizing names and dependencies so they can be computed on. > Its such an unreliable pain at the moment that it really needs > tackling, and it is a very effective tool in application design in > languages that have it (like Fantom). > How did they solve it? bundled their own class scanning tool? I recall the biggest pain with class scanning was supporting Web/App servers that had their own class loader URL scheme that needed to be reverse engineered. It seems to be a fundamental issue with class loaders. It might be possible if one know the module path in a certain context to start accessing the contents of those modules (the module system API is in flux and i would need to remind myself of it). Paul.
Re: JEP 238: Multi-Version JAR Files
On Feb 25, 2015, at 5:27 PM, Brian Goetz wrote: > > > On 2/12/2015 5:59 PM, Stephen Colebourne wrote: >> Interesting direction. >> >> Reading carefully, the goal is actually very limited in scope, by >> preventing any public API changes. It doesn't help adoption of JSR-310 >> for example, but will be useful for Unsafe, which is clearly a >> motivating factor. >> >> I would expect IDEs to have some considerable work to do. > > Agree on the "work" part, but I doubt it is "considerable". > > For creating MV JARs, the 'jar' tool does all the heavy lifting. > > For running Java apps, the classloader does all the heavy lifting. > > For tools that have to consume JARs, the JarFile API does all the heavy > lifting. If you use JarFile, it's a one-line change to the constructor to > get a version-specific view of the JAR. > > So in each of these cases, the work is limited to: > - Figure out how you intend to interact with MVJars; > - Configure the appropriate tool (jar tool, JarFile) appropriately; this is > generally a matter of new constructor arguments and/or new command line > arguments. > > So I totally agree there will be lots of things that change, but those > changes should be individually quite small (and this is consistent with our > experience supporting MVJars in the JDK tools.) > In the design doc i outlined how to explicitly compile an mv-style project. How hard would it be to hack up a new kind of maven project layout and javac compilation task that did something similar? Does anybody with maven expertise know more? Paul.
Re: JEP 238: Multi-Version JAR Files
On 25 February 2015 at 13:30, Paul Sandoz wrote: > Even in the modular world i will expect class scanning will be used. While we > can now iterate reliably over classes in the image i don't believe that > functionality is made available for classes in modules on the module path. To be honest, being able to reliably iterate over classes in a module is the most interesting potential feature about modules in general. Its such an unreliable pain at the moment that it really needs tackling, and it is a very effective tool in application design in languages that have it (like Fantom). On 25 February 2015 at 16:27, Brian Goetz wrote: > On 2/12/2015 5:59 PM, Stephen Colebourne wrote: >> I would expect IDEs to have some considerable work to do. > Agree on the "work" part, but I doubt it is "considerable". In Eclipse, there is a 1:1 mapping between a project and a JDK compiler/version. Same in Maven. And no doubt elsewhere. I don't think making that one to many instead is likely to be a particularly small change. Stephen
Re: JEP 238: Multi-Version JAR Files
On 2/12/2015 5:59 PM, Stephen Colebourne wrote: Interesting direction. Reading carefully, the goal is actually very limited in scope, by preventing any public API changes. It doesn't help adoption of JSR-310 for example, but will be useful for Unsafe, which is clearly a motivating factor. I would expect IDEs to have some considerable work to do. Agree on the "work" part, but I doubt it is "considerable". For creating MV JARs, the 'jar' tool does all the heavy lifting. For running Java apps, the classloader does all the heavy lifting. For tools that have to consume JARs, the JarFile API does all the heavy lifting. If you use JarFile, it's a one-line change to the constructor to get a version-specific view of the JAR. So in each of these cases, the work is limited to: - Figure out how you intend to interact with MVJars; - Configure the appropriate tool (jar tool, JarFile) appropriately; this is generally a matter of new constructor arguments and/or new command line arguments. So I totally agree there will be lots of things that change, but those changes should be individually quite small (and this is consistent with our experience supporting MVJars in the JDK tools.)
Re: JEP 238: Multi-Version JAR Files
Hi Peter, On Feb 14, 2015, at 8:54 PM, Peter Levart wrote: > > On 02/12/2015 09:52 PM, Paul Sandoz wrote: >> Hi >> >> In connection with the JEP there is also a design document to help the >> discussion: >> >> http://cr.openjdk.java.net/~psandoz/jdk9/MultiVersionJar-8u60-9-design.md >> > > Hi Paul, > > Thinking about this proposal, I can't escape the feeling that the design > favors JDK as the only target for which multi-version jars exist. There is a reason you cannot escape that feeling :-) because as currently designed it is intended only to version a JAR over major Java platform versions. Your maven-inspried profile solution is an interesting, but do think it increases the complexity by expanding versioning to any dependency. A MVJAR (a unit of release) now has two or more sets of dependencies. The developer has to layout the classpath carefully with an understanding of what jar files are associated to what profile in another jar. Managing dependencies in large systems can be rather complex and i fear this will add further to that complexity. I could see how one could more formally manage this with a module system where modules have names and declare the names of modules they depend on. Thus modules can be formally analysed and the module paths verified. But still the multiple sets of dependencies and selection of concerns me (furthermore Project Jigsaw already has a stacked schedule of stuff to do). So at the moment i would prefer to keep this more constrained and focused on the JDK itself. Paul.
Re: JEP 238: Multi-Version JAR Files
HI Peter,
On Feb 14, 2015, at 3:56 PM, Peter Levart wrote:
>
> Hi Paul,
>
> I read through the proposal and couldn't find an explanation of how resources
> placed in versioned paths are going to be visible. For example, if the
> multi-versioned jar contains the following structure:
>
> my.properties
> META-INF/versions/8/my.properties
> META-INF/versions/9/my.properties
>
>
> What will the following code return, when run on JDK9:
>
> URL[] urls = ClassLoader.getSystemClassLoader().getResources("my.properties");
>
> Will we only get 1 URL: META-INF/versions/9/my.properties
> or 3 URLs in order: META-INF/versions/9/my.properties,
> META-INF/versions/8/my.properties, my.properties
>
> Class name -> .class file resolving is easy, since only the 1st one is used,
> but there can be multiple general resources for same path.
>
The intension is only one such "my.properties" resource file will be found
within the MVJAR. There may be some subtleties when using absolute names. The
analogy of placing things on the classpath in the appropriate order does break
down in this respect.
Paul.
Re: JEP 238: Multi-Version JAR Files
On Feb 12, 2015, at 11:59 PM, Stephen Colebourne wrote: > Interesting direction. > Catching up on email after being away last week... > Reading carefully, the goal is actually very limited in scope, by > preventing any public API changes. It doesn't help adoption of JSR-310 > for example, but will be useful for Unsafe, which is clearly a > motivating factor. Yes, in addition to other stuff internal stuff. > > I would expect IDEs to have some considerable work to do. Yes, if the concept of a multi-version project has legs. It "feels" appropriate to have everything under one project rather than spread out between separate projects/repos with some form of merging. > > My biggest concern will be ensuring tools like Reflections (or other > classpath scanning tools) carry on working. Really, classpath scanning > should no longer be necessary with a module system, but since we've > heard nothing to indicate that issue is being tackled, those tools > will continue to be vital (such as to find all classes implementing an > annotation, or all implementations of an interface) > Even in the modular world i will expect class scanning will be used. While we can now iterate reliably over classes in the image i don't believe that functionality is made available for classes in modules on the module path. I am guilty if writing such class scanners :-) The context was JAX-RS/Jersey. In this case i would expect that the annotations should be the same across multiple class files targeted to different platforms, since that is part of the API. But in general i agree there could be issues. I am very interested in hearing feedback on such cases. Please share widely, if not already done so. Paul.
Re: JEP 238: Multi-Version JAR Files
On 02/12/2015 09:52 PM, Paul Sandoz wrote:
Hi
In connection with the JEP there is also a design document to help the
discussion:
http://cr.openjdk.java.net/~psandoz/jdk9/MultiVersionJar-8u60-9-design.md
Hi Paul,
Thinking about this proposal, I can't escape the feeling that the design
favors JDK as the only target for which multi-version jars exist. The
"versions" in the jar are just target JDK versions. Admittedly JDK is
the most important "library" out there, but jar is a general format. So
perhaps at the jar level, the facility could be more general. I'm
thinking about profiles...
For example, taking a layout example from JEP and translating to the
profiles idea:
jar root
- A.class
- B.class
- C.class
- D.class
- META-INF
- MANIFEST.MF
- profiles
- jdk8
- A.class
- B.class
- jdk9
- A.class
With additional metadata in MANIFEST.MF:
Profile-JDK9: jdk9 jdk8
Profile-JDK8: jdk8
A list (ordered) of profile names is associated with each jar file or
expanded directory at runtime. The default list of profile names (if not
specified on classpath) consists of just the platfrom profile name
("JDK9" or "JDK8"), but additional profile names could be encoded as
part of each classpath element.
For example, let's say I want to distribute a multi-profile library jar
that depends on JDK7, JDK8 and JDK9 as platform and on guava 17 and 18.
I could pack it as:
jar root
- A.class
- B.class
- C.class
- D.class
- META-INF
- MANIFEST.MF
- profiles
- jdk8
- A.class
- B.class
- jdk9
- A.class
- guava18
- C.class
- jdk9-guava18
- C.class
MANIFEST.MF:
Profile-JDK9: jdk9-guava18(GUAVA18) jdk9 jdk8
Profile-JDK8: jdk8
Profile-GUAVA18: guava18
Some example usages:
$JDK8_HOME/bin/java -cp guava-18.jar:mylib.jar?profile=+GUAVA18
...in this example, the list of profile names associated with mylib.jar
is: (JDK8, GUAVA18), which yields the following search order inside
mylib.jar:
- META-INF/profiles/jdk8
- META-INF/profiles/guava18
- jar root
$JDK9_HOME/bin/java -cp guava-18.jar:mylib.jar?profile=+GUAVA18
...in this example, the list of profile names associated with mylib.jar
is : (JDK9, GUAVA18), which yields the following search order inside
mylib.jar:
- META-INF/profiles/jdk9-guava18 - this is a conditional entry, included
only when all profile names in brackets are also associated with the jar
- META-INF/profiles/jdk9
- META-INF/profiles/jdk8
- META-INF/profiles/guava18
- jar root
$JDK9_HOME/bin/java -cp guava-17.jar:mylib.jar
...in this example, the list of profile names associated with mylib.jar
is : (JDK9), which yields the following search order inside mylib.jar:
- META-INF/profiles/jdk9
- META-INF/profiles/jdk8
- jar root
The "additional" profile names could be encoded in file: URL(s) as the
the URL "query", for example:
file:/path/to/my.jar?pfofile=+GUAVA18
...so they are part of URL class-path in URLClassLoader.
The JarFile and JarInputStream would have to be extended to support
specifying a list of profile names in addition to current arguments.
This is not much different from proposed JEP. It just adds a layer of
resolving of the internal jar file paths through a list of profile names
as opposed to hard-coding the layout. I know this complicates tooling
support. In particular the additional options for "jar" tool, but
verification part which validates that the exposed public API is same
for all combinations of profiles would not be much more complicated.
The profiles idea is inspired loosely on Maven profiles.
Too complicated?
Regards, Peter
Re: JEP 238: Multi-Version JAR Files
On 02/12/2015 09:52 PM, Paul Sandoz wrote:
Hi
In connection with the JEP there is also a design document to help the
discussion:
http://cr.openjdk.java.net/~psandoz/jdk9/MultiVersionJar-8u60-9-design.md
We are especially interesting in hearing feedback from library developers,
tool/IDE developers, and anyone doing funky stuff with class loading and JAR
files.
Paul.
On Feb 12, 2015, at 9:41 PM, mark.reinh...@oracle.com wrote:
New JEP Candidate: http://openjdk.java.net/jeps/238
- Mark
Hi Paul,
I read through the proposal and couldn't find an explanation of how
resources placed in versioned paths are going to be visible. For
example, if the multi-versioned jar contains the following structure:
my.properties
META-INF/versions/8/my.properties
META-INF/versions/9/my.properties
What will the following code return, when run on JDK9:
URL[] urls =
ClassLoader.getSystemClassLoader().getResources("my.properties");
Will we only get 1 URL: META-INF/versions/9/my.properties
or 3 URLs in order: META-INF/versions/9/my.properties,
META-INF/versions/8/my.properties, my.properties
Class name -> .class file resolving is easy, since only the 1st one is
used, but there can be multiple general resources for same path.
Regards, Peter
Re: JEP 238: Multi-Version JAR Files
Interesting direction. Reading carefully, the goal is actually very limited in scope, by preventing any public API changes. It doesn't help adoption of JSR-310 for example, but will be useful for Unsafe, which is clearly a motivating factor. I would expect IDEs to have some considerable work to do. My biggest concern will be ensuring tools like Reflections (or other classpath scanning tools) carry on working. Really, classpath scanning should no longer be necessary with a module system, but since we've heard nothing to indicate that issue is being tackled, those tools will continue to be vital (such as to find all classes implementing an annotation, or all implementations of an interface) Stephen On 12 February 2015 at 20:52, Paul Sandoz wrote: > Hi > > In connection with the JEP there is also a design document to help the > discussion: > > http://cr.openjdk.java.net/~psandoz/jdk9/MultiVersionJar-8u60-9-design.md > > We are especially interesting in hearing feedback from library developers, > tool/IDE developers, and anyone doing funky stuff with class loading and JAR > files. > > Paul. > > On Feb 12, 2015, at 9:41 PM, mark.reinh...@oracle.com wrote: > >> New JEP Candidate: http://openjdk.java.net/jeps/238 >> >> - Mark >
Re: JEP 238: Multi-Version JAR Files
Hi In connection with the JEP there is also a design document to help the discussion: http://cr.openjdk.java.net/~psandoz/jdk9/MultiVersionJar-8u60-9-design.md We are especially interesting in hearing feedback from library developers, tool/IDE developers, and anyone doing funky stuff with class loading and JAR files. Paul. On Feb 12, 2015, at 9:41 PM, mark.reinh...@oracle.com wrote: > New JEP Candidate: http://openjdk.java.net/jeps/238 > > - Mark
