Re: Better tools for adjusting to strong encapsulation
These new messages are a big problem for an application that interacts with the user via the console. I understand that the messages may be useful to application developers who need to "identify which libraries and frameworks on your class path require illegal access in order to work". For other situations where the reason for the access is known to the application developer and there is no way to avoid it, please consider giving the application developer the option to suppress the messages or redirect them to somewhere other than System.err. Simon On 21/03/2017 14:48, mark.reinh...@oracle.com wrote: By now we've heard quite a bit of feedback, both publicly and privately, from developers who've migrated existing applications to JDK 9. This can be a non-trivial task, involving the iterative construction of a set of command-line workarounds, since some popular libraries and frameworks use the core reflection API to access internal APIs of the JDK. On JDK 9 such components no longer work out-of-the-box because most internal APIs are now strongly encapsulated in order to improve the integrity of the platform. In order to make it easier to adjust to the strong encapsulation of internal APIs in JDK 9, in both the short term and over the long term, we've introduced a temporary "big kill switch", described below. This switch will only be supported in JDK 9. It's already available in the current Jigsaw EA builds [1], and will shortly be merged into JDK 9. Background -- Diagnosing the failures of libraries and frameworks that access internal APIs of the JDK and then working around those failures is possible, but tedious. The initial symptom is usually an `IllegalAccessException` or an `InaccessibleObjectException`. The workaround is to use an `--add-opens` command-line option whose arguments are based upon the information shown in the exception message. A large application that uses several reflective components may suffer many such failures, so this process may have to be repeated many times, resulting in a long list of command-line options that will typically be recorded in a launch script or an @-file. These options are tedious to construct but, additionally, there's a significant risk that once they're baked into a file they'll never be removed, even long after the problems that they work around have been resolved. An application that needs some workarounds today might not benefit as well as it could from strong encapsulation in the long term, after all of the reflective components that it uses have been fixed to replace their use of JDK-internal APIs with proper exported APIs. To help the entire ecosystem migrate to the modular Java platform we want to motivate such fixes as strongly as we possibly can, especially in the near term. We can do that by making it easier for you to get an existing application to work while also ensuring that the run-time system reports illegal reflective-access operations in a way that allows you to identify which libraries and frameworks on your class path require illegal access in order to work. If a component's maintainers have already released a new, fixed version that no longer requires illegal access then you can consider upgrading to that version. If, however, a component still needs to be fixed then we encourage you to contact its maintainers and ask them to stop using JDK-internal APIs [2]. With all this in mind we've introduced a "big kill switch", described below, together with some new warning messages. This switch will be supported in JDK 9, in order to enable application migration and motivate fixes to libraries and frameworks. It will not be supported in JDK 10 or later releases, in order to motivate fixes to libraries and frameworks sooner rather than later. If this switch were to be supported long-term then that would tend to reduce the pressure on maintainers to fix their components. New command-line option: `--permit-illegal-access` -- If this option is present at run time then it will permit illegal access operations by code in any unnamed module (i.e., code on the class path) to members of types in any named module via the standard reflective APIs (`java.lang.reflect` and `java.lang.invoke`), including in particular those that would otherwise result in an `IllegalAccessException` or an `InaccessibleObjectException`. +--+ | This option will only be supported in JDK 9. | | It will be removed in JDK 10.| +--+ This option does not permit illegal access operations by code in named modules to members of types in other named modules; for that you must use the `--add-opens` or `--add-exports` options. Those options can be combined with `--permit-illegal-access`. Warnings of illegal reflective-access operations
Re: Better tools for adjusting to strong encapsulation
I think the distinction between the precise options and the big kill switch is important and I welcome this change. It will solve the immediate problem for my application. I still have reservations about the hard-wired use of System.err for warning messages for the reasons given by Roel Spilker. Michael Rasmussen mentioned other cases of this in JDK 9 but I'm not sure what these are. Simon On 23/03/2017 03:30, mark.reinh...@oracle.com wrote: Thanks to everyone for all the feedback on this topic. It appears that issuing warning messages for illegal-access operations enabled by the precise `--add-opens` and `--add-exports` options is a bit too aggressive, at least for JDK 9. Perhaps we can enable that in JDK 10 after there's been more time for libraries, frameworks, and even the JDK itself to adjust to the realities of strong encapsulation. For now I suggest that we revert to the previous behavior of these two options, so that they do not cause warning messages to be issued. The new `--permit-illegal-access` option will continue to generate warning messages, as proposed. If those messages are a problem in a particular scenario then they can be eliminated by switching to an appropriate set of `--add-opens` options, which can be constructed from the information contained in those messages. Comments? - Mark
Re: Better tools for adjusting to strong encapsulation
On 23/03/2017 19:04, Alan Bateman wrote: On 23/03/2017 18:44, Alan Snyder wrote: If I understand JEP 264 correctly, it should be possible to direct platform generated error messages to locations other than the standard error stream (System.err). Is that correct? If so, would it not make sense for the default to be the (platform dependent) console log rather than System.err, which is used by applications for their own error messages? System.Logger can be configured to send log messages to whatever logging library you are using. However, is is not appropriate here, ditto for several other areas where you don't want arbitrary code to execute. -Alan Would it be possible for the JDK to include a small number of predefined Logger instances that provide alternative destinations for these messages? For example, there could be a logger for System.err, a logger for the platform console (where applicable) and a logger that directs output to a user-specified file. This approach would provide some flexibility for the developer without allowing arbitrary code to execute. Simon
Re: Better tools for adjusting to strong encapsulation
On 24/03/2017 13:02, Alan Bateman wrote: On 24/03/2017 12:01, Peter Levart wrote: : Just a reminder that System.err field is "writable" and therefore System.setErr(PrintStream) can be used to redirect output to execute arbitrary code... That's right, it needs to captured at startup or sent directly to stderr. -Alan I think this means that at startup, an application could do PrintStream sysErr = System.err; System.setErr(myErr); where myErr is an application subclass of PrintStream that filters out any unwanted warnings produced by JDK code and sends all other output to sysErr. Would this work? Simon
Re: Review Request JDK-8175819: OS name and arch in JMOD files should match the values as in the bundle name
On 03/04/2017 21:15, mark.reinh...@oracle.com wrote: 2017/4/3 11:41:03 -0700, mandy.ch...@oracle.com: Webrev: http://cr.openjdk.java.net/~mchung/jdk9/webrevs/8175819/webrev.00/ ... This shows the old and new value of OS_NAME/OS_ARCH properties in the `release` file: JDK 8 JDK 9 - - OS_NAME Linux linux SunOS solaris Darwin osx Windows windows OS_ARCH i386,x86x86 i586,amd64,x86_64 x64 sparcv9 sparcv9 arm arm32 aarch64 arm64 I suggest we use "amd64" instead of "x64" (or any other variants). "x64" is confusingly close to "x86", and while it's used by some companies (including Oracle), it's not used all that widely. "amd64" is better known, matches the long-standing value of the "os.arch" system property (on Linux and Windows, anyway) and, as a side benefit, honors the originators of the architecture. - Mark I am not sure why we would change to osx for Mac when the Mac developers have recently dropped the Mac OS X terminology and changed it to macOS. Simon
Re: Review Request JDK-8175819: OS name and arch in JMOD files should match the values as in the bundle name
On 03/04/2017 22:07, Mandy Chung wrote: On Apr 3, 2017, at 1:35 PM, Simon Nash wrote: On 03/04/2017 21:15, mark.reinh...@oracle.com wrote: I am not sure why we would change to osx for Mac when the Mac developers have recently dropped the Mac OS X terminology and changed it to macOS. Just to be clear, there is no plan to change the value of the ‘os.name’ system property and its value is “Mac OS X”. Changing the value of the system property ‘os.name’ would break existing applications. In JDK 8, OS_NAME in the release file is “Darwin” which is what this patch would change. Mandy I understand that the os.name property value cannot be changed for compatibility reasons and therefore this name needs to stay as the "historic" Mac OS X. My comment was regarding the change of value for OS_NAME. Given that there is no compatibility issue here, does it make sense for the new value to be something that is no longer current in Apple terminology? Simon
Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes
Jason Greene wrote: There is also a disparity here that the JDK itself doesn’t require you to export packages (e.g. I don’t need it for Java serialization). Now I realize that there is an effort underway to de-privilege modules, but I suspect that a portion of the JDK will continue to enjoy special power for precisely the same usability concerns that apply to frameworks / standards which extend the platform. I think this a very important point. If someone wanted to reimplement Java serialization (java.io.ObjectOutputStream, etc.) as an external library (com.foo.ObjectOutputStream, etc.), the new restrictions on reflective access in JDK 9 would prevent this. Simon
Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes
Alan Bateman wrote: On 09/07/2016 08:46, Simon Nash wrote: I think this a very important point. If someone wanted to reimplement Java serialization (java.io.ObjectOutputStream, etc.) as an external library (com.foo.ObjectOutputStream, etc.), the new restrictions on reflective access in JDK 9 would prevent this. If there are types in non-exported packages in the serial form then it could be an issue. Some serialization libraries are based on Unsafe and sun.reflect.ReflectionFactory, both "critical internal APIs" that continue to be available via the jdk.unsupported module. More on this in JEP 260 [1]. -Alan [1] http://openjdk.java.net/jeps/260 Is it really a good idea to encourage packages that currently use reflection to access non-exported private fields (an official part of the Java API) to change to using internal APIs? This seems like a step backwards to me. Simon
Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes
Alan Bateman wrote: On 09/07/2016 09:57, Simon Nash wrote: Is it really a good idea to encourage packages that currently use reflection to access non-exported private fields (an official part of the Java API) to change to using internal APIs? This seems like a step backwards to me. I didn't suggest that. I was just pointing out that some of the 3rd party serialization libraries that we've come across are based on Unsafe and ReflectionFactory and the latter is specifically on critical internal API for that reason. -Alan Yes, I understand. My point was that these APIs might in some cases become a substitute for reflective access to non-exported private fields if the latter is prohibited, Simon
Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes
Paul Benedict wrote: For those who are still supporters of preventing non-exported types from being reflected, I think a compromise can still be found, but it's not in this proposal. Here are two other alternatives I hope the EG will consider: 1) Introduce a new permission type to allow non-exported types to be reflected. I don't find it acceptable the module gets to dictate what can't be reflected. I believe this should be controlled and configured externally -- certainly not the module itself. 2) Allow layers to control if non-exported types can be reflected. Perhaps the JDK sets its own layers to "false", but Containers and what they deploy can be separately configured, each. For example, maybe WebLogic won't allow itself to have its non-exported types reflected, but if each EAR gets its own layers, I could configure WebLogic to allow me to reflect everything within the EAR. PS: I don't see #1 and #2 to be mutually exclusive. Cheers, Paul I would like to propose another possible compromise that might satisfy both of the following requirements that have been expressed on this thread: 1) My module should completely encapsulate some internal classes and make them invisible to the outside world 2) My module or library needs reflective access to internal classes in other modules in order to function correctly At present, a module can mark its packages as: exported at compile time and runtime exported only at runtime via reflection never exported (the default) Instead, these options could change to: exported at compile time and runtime exported at runtime only via reflection (the default) never exported (hidden) This would allow modules to mark certain packages as "hidden" if these packages contain internal methods or fields that should be never be accessible externally under any circumstances. The default would be to allow reflection by other modules or libraries that need such access. Simon
jdk.unsupported module missing from compact profiles
My application uses sun.misc.SignalHandler which is now in the jdk.unsupported module. This class is available when running with the full JRE but not when running with the compact3 JRE. The sun.misc.* classes are available in the compact1/2/3 profiles in JDK 8 and I don't see any reason why the compact1/2/3 profiles should not contain these classes in JDK 9. To fix this, the Images.gmk file should be modified to change line 51 from: COMPACT1_EXTRA_MODULES := jdk.localedata jdk.crypto.pkcs11 jdk.crypto.ec to: COMPACT1_EXTRA_MODULES := jdk.localedata jdk.crypto.pkcs11 jdk.crypto.ec jdk.unsupported Regards, Simon
Re: jdk.unsupported module missing from compact profiles
Alan Bateman wrote: On 29/10/2016 23:09, Simon Nash wrote: My application uses sun.misc.SignalHandler which is now in the jdk.unsupported module. This class is available when running with the full JRE but not when running with the compact3 JRE. The sun.misc.* classes are available in the compact1/2/3 profiles in JDK 8 and I don't see any reason why the compact1/2/3 profiles should not contain these classes in JDK 9. Not an issue if you are using jlink to create the runtime image yourself but fair point that the JDK build should probably include jdk.unsupported in the runtime images for the compact Profiles. I'll create a bug for this. -Alan. Thanks! I have found the bug report and I am starting the process of becoming a contributor so that I can submit a fix. Simon
Accessing internal packages in JDK 9
I am developing an application that is using some internal JDK packages (sun.*) to work around some problems with "leaked" Swing objects that reference my application classloader via their protection domain. This means that my application classloader cannot be garbage collected even if I remove all references to it from within my application. This is a problem because my application uses native DLLs that sometimes need to be replaced by new versions without restarting the application and these DLLs cannot be unloaded and reloaded unless the application classloader that loaded these DLLs has been garbage collected. The workarounds I am using for this problem use a combination of internal APIs together with reflection to locate the fields that are holding these internal "leaked" objects and set these fields to null. With JDK 9 and Project Jigsaw, I understand that it will no longer be possible for my application to directly reference internal sun.* classes. Will it still be possible to use ClassLoader.loadClass to load these sun.* classes and use reflection to invoke their methods? Simon Nash
Re: [SPAM?] Re: Accessing internal packages in JDK 9
mark.reinh...@oracle.com wrote: 2015/7/28 4:21 -0700, Simon Nash : ... With JDK 9 and Project Jigsaw, I understand that it will no longer be possible for my application to directly reference internal sun.* classes. Will it still be possible to use ClassLoader.loadClass to load these sun.* classes and use reflection to invoke their methods? Yes, but you'll have to use a command-line flag to make the classes accessible via the reflection API. - Mark Is this a single option that enables all internal access or does the option need to be repeated for every internal package or class that the application needs to access? If the option is set, can the application use imports for the internal packages or is it required to access them reflectively? Would it be possible to allow the application to set this option programmatically on initialization or set it internally in some other way (e.g., in its jar manifest) so that the users of the application don't need to specify it on the launch command? Simon
Re: Accessing internal packages in JDK 9
mark.reinh...@oracle.com wrote: 2015/7/28 8:12 -0700, Simon Nash : mark.reinh...@oracle.com wrote: 2015/7/28 4:21 -0700, Simon Nash : ... With JDK 9 and Project Jigsaw, I understand that it will no longer be possible for my application to directly reference internal sun.* classes. Will it still be possible to use ClassLoader.loadClass to load these sun.* classes and use reflection to invoke their methods? Yes, but you'll have to use a command-line flag to make the classes accessible via the reflection API. Is this a single option that enables all internal access or does the option need to be repeated for every internal package or class that the application needs to access? If the option is set, can the application use imports for the internal packages or is it required to access them reflectively? Would it be possible to allow the application to set this option programmatically on initialization or set it internally in some other way (e.g., in its jar manifest) so that the users of the application don't need to specify it on the launch command? These are all good questions. We're nearly done drafting a JEP which will answer them, so rather than answer them here can I ask that you read the JEP when it's published? Thanks, - Mark Thanks! I look forward to doing that. :) Simon
Re: [SPAM?] Encapsulating internal APIs in JDK 9 (sun.misc.Unsafe, etc.)
Mark, Thanks for this. In the list of broad categories, one is missing: x Those which are required to enable application code to work around bugs in the JDK such as leaked objects. When do you expect to be able to publish a detailed proposal for how the "last resort" mechanism will work? Simon mark.reinh...@oracle.com wrote: As part of the overall modularization effort [1] we're going to encapsulate most of the JDK's internal APIs within the modules that define and use them so that, by default, they are not accessible to code outside the JDK. This change will improve the integrity of the platform, since many of these internal APIs define privileged, security-sensitive operations. In the long run it will also reduce the costs borne by the maintainers of the JDK itself and by the maintainers of libraries and applications that, knowingly or not, make use of these non-standard, unstable, and unsupported internal APIs. It's well-known that some popular libraries make use of a few of these internal APIs, such as sun.misc.Unsafe, to invoke methods that would be difficult, if not impossible, to implement outside of the JDK. To ensure the broad testing and adoption of the release we propose to treat these critical APIs as follows: - If it has a supported replacement in JDK 8 then we will encapsulate it in JDK 9; - If it does not have a supported replacement in JDK 8 then we will not encapsulate it in JDK 9, so that it remains accessible to outside code; and, further, - If it has a supported replacement in JDK 9 then we will deprecate it in JDK 9 and encapsulate it, or possibly even remove it, in JDK 10. The critical internal APIs proposed to remain accessible in JDK 9 are listed in JEP 260 [2]. Suggested additions to the list, justified by real-world use cases and estimates of developer and end-user impact, are welcome. - Mark [1] http://openjdk.java.net/jeps/200 [2] http://openjdk.java.net/jeps/260
Re: Encapsulating internal APIs in JDK 9 (sun.misc.Unsafe, etc.)
mark.reinh...@oracle.com wrote: 2015/8/4 11:34 -0700, si...@cjnash.com: Thanks for this. In the list of broad categories, one is missing: x Those which are required to enable application code to work around bugs in the JDK such as leaked objects. That could be an awfully broad category. Do you have some specific examples in mind? It is a broad category and it is an important one that deserves mention. I have come across a number of examples of leaked objects in Swing/AWT for which I have implemented workarounds by using internal APIs and reflection to locate the relevant fields and set them to null. At some point, I hope to create formal bug reports for these. The main impact is not memory consumption but the holding of references to my application classloader that prevents it from being garbage collected. This prevents any native libraries that were loaded by this classloader from being replaced with updated versions. When do you expect to be able to publish a detailed proposal for how the "last resort" mechanism will work? That will be an implementation-specific command-line flag, to be proposed in a forthcoming JEP. - Mark I look forward to seeing these details in the forthcoming JEP. Simon
Re: [SPAM?] Re: Encapsulating internal APIs in JDK 9 (sun.misc.Unsafe, etc.)
mark.reinh...@oracle.com wrote: 2015/8/4 12:07 -0700, Simon Nash : mark.reinh...@oracle.com wrote: 2015/8/4 11:34 -0700, si...@cjnash.com: Thanks for this. In the list of broad categories, one is missing: x Those which are required to enable application code to work around bugs in the JDK such as leaked objects. That could be an awfully broad category. Do you have some specific examples in mind? It is a broad category and it is an important one that deserves mention. I have come across a number of examples of leaked objects in Swing/AWT for which I have implemented workarounds by using internal APIs and reflection to locate the relevant fields and set them to null. At some point, I hope to create formal bug reports for these. The main impact is not memory consumption but the holding of references to my application classloader that prevents it from being garbage collected. This prevents any native libraries that were loaded by this classloader from being replaced with updated versions. I understand the problem you're solving, but the difficulty with this as a category is that it arguably includes every internal class, even those never intended to be even internal APIs, since pretty much any internal class can declare a field that winds up retaining some object, somewhere, unnecessarily. If we were to take this on as a category in the context of JEP 260 then we'd wind up encapsulating very little. I wasn't suggesting that this is a reason for not doing encapsulation but rather that this requirement exists and implies the ongoing need for a "last resort" mechanism to bypass encapsulation. This mechanism shouldn't be regarded as a transitional requirement but as something that will still be needed after new public APIs have been created to replace the private APIs that are known to be required by application developers. A better approach here would be to fix the actual bugs, so bug reports would be most welcome (especially if they include suggested fixes). In the meantime you'll be able to use the last-resort command-line workaround to keep your existing code running. - Mark I agree, but it is unrealistic to think that all such bugs will be fixed promptly and no new ones would ever appear. For example, bug 6575402 has been open since 2007 and bug 8029147 has been open since 2013. Just one leaked object causes the application classloader not to be garbage collected and its native DLLs not to be unloaded. This is why it will always be necessary to have some escape mechanism by which application developers can work around such issues. Simon
Re: Feature complete?
Alan Bateman wrote: On 04/12/2015 13:50, Reinier Zwitserloot wrote: : Emphasis mine: "only"? "Just add some command line switches" is not exactly a nice solution. From personal experience, the vast majority of java users do not know how to add configure command line switches to the various compile and build VMs being started by their build tools of choice, for example. Also, not something Alan mentioned here: Adding the command line switches is currently the _ONLY_ solution, so tools that are aware of the issue can't even fix it, other than by mangling modules (bad idea, I think we all agree on that I assume), or advising the userbase not to actually USE any of the modularity features and, preferably, to avoid switching to JDK9 altogether. The context for this discussion, as least my understanding, is that we were talking existing code where you are trying to run it in the same way as you did in JDK 8. If that code is using JDK-internal APIs then the command-line option is the only workaround proposed to keep that code working. I agree that setting the command-line is a challenge in many environments, esp. when it is buried in XML files or scripts. You may be right that it is beyond some developers too. We've tried to make error and exceptions as clear as possible so that it's not too difficult to figure out what is going on. Setting addExports via the command line is very problematical for my application. This application needs to run on 7 different platforms with both graphical and console interfaces. Each platform has different packaging requirements and launch arrangements. In at least one case, (launching via the .jar file association on Windows), it isn't possible to add command-line JVM options. For each platform, the same installable application package needs to run on Java 7, Java 8 and Java 9 with no impact to the user when he/she upgrades to Java 9 from Java 7 or Java 8. Adding the necessary -XaddExports options to the command line for Java 9 for all platforms while satisfying the above constraints is very challenging and would be a large amount of work. If some other way of setting these options could be provided (for example, via the manifest in the main JAR file or by an API call from the application), this would greatly simplify the migration of my application to Java 9. Simon Emphasis mine: "starting out"? This gets back to an earlier issue I raised on this mailing list: Okay, starting out this isn't an issue, but by having no way to opt out of export-based access control, it's a bit like switching from driving on the left side of the road to the right: Unless the entire greater java community switches in the same instant, we're all going to have a bad time. The current design and prototype provides support for migration to modules without waiting for the rest of the world to migrate. We're going to need more experience, help, feedback to make sure that this is useful. The main challenge is of course is the tension between frameworks and strong encapsulation. Existing frameworks don't know about modules and so it will be important for willing maintainers of at least some important frameworks to work with us to get to the right solution. Maybe then we can start switching to the other side of the road, a gradual migrating, starting with the big trucks :-) -Alan
