Re: What does a qualified name mean for a module?

[David M. Lloyd]

On 06/06/2017 12:14 PM, Stephan Herrmann wrote:

On 06.06.2017 18:59, Alex Buckley wrote:
A module name has the same structure as a package name, so 
ModuleElement has the same shape as PackageElement: each inherits 
getSimpleName() from Element, and getQualifiedName() from 
getQualifiedName() from QualifiedNameable.


Syntactically you're right, but ...

Normally, a qualified name denotes two things: a parent element and a 
child.
The package name "java.lang" has a qualifier "java" which denotes a 
top-level package
and "lang" can be used relative to that package to denote a member 
package etc.


For a module - say "java.base" - the qualifier "java" denotes nothing.
And hence, the simple name "base" cannot be resolved in any context.

So the question is: should ModuleElement.getSimpleName() answer the
totally useless last segment of the name, or should it answer the same
as getQualifiedName()?


The answer to that question should apply to PackageElement too, for 
identical reasons.  However, PackageElement is established API already...


--
- DML

Re: Proposal: Allow illegal reflective access by default in JDK 9

[David M. Lloyd]

On 05/19/2017 05:58 AM, Peter Levart wrote:

Hi Nikolai,

On 05/19/2017 12:41 PM, Nicolai Parlog wrote:

  Hi!

I'm too lazy to try right now


It's easy. Take this example:

public class Test {
 public static void main(String[] args) throws Exception {
 Object theUnsafe = Class.forName("jdk.internal.misc.Unsafe")
 .getMethod("getUnsafe")
 .invoke(null);
 }
}

Running this on classpath without any additional JVM options produces:

Exception in thread "main" java.lang.IllegalAccessException: class Test 
cannot access class jdk.internal.misc.Unsafe (in module java.base) 
because module java.base does not export jdk.internal.misc to unnamed 
module @63d4e2ba
 at 
java.base/jdk.internal.reflect.Reflection.throwIllegalAccessException(Reflection.java:445) 

 at 
java.base/jdk.internal.reflect.Reflection.throwIllegalAccessException(Reflection.java:436) 

 at 
java.base/jdk.internal.reflect.Reflection.ensureMemberAccess(Reflection.java:112) 

 at 
java.base/java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:407) 

 at 
java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:399) 


 at java.base/java.lang.reflect.Method.invoke(Method.java:535)
 at Test.main(Test.java:9)

(note that the exception is thrown from Method.invoke)

When run with:

  --add-exports java.base/jdk.internal.misc=ALL-UNNAMED

...the example works (does not throw exception), but when run with:

 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED

...the example still works OOOPS - this is a bug!!! This is 
definitely a bug.


This is seemingly consistent with the code run time code in Module for 
addOpens, which always seems to add export when open is added.  It would 
be quite strange if the descriptor directives and JVM switches didn't 
match the run time code; is this something that the EG should talk 
about?  Or is there a hidden logic to this?


--
- DML

Re: Some suggested patches and improvements

[David M. Lloyd]

On 05/16/2017 06:02 AM, Alan Bateman wrote:

On 12/05/2017 14:31, David M. Lloyd wrote:


:

There is a lot more to #5, something that will become clear when you 
work through all the scenarios. The JSR and spec part are minor 
though but I'd prefer to hold off until there is more discussion on 
this topic in the JSR.


I'd rather not hold off as the JSR essentially only has a couple of 
weeks left to live if there is not a revised PR.  Could you please 
explain what you mean?  Are you referring to jlink, jaotc, or 
something else?


For the most part, this is not a JSR issue.  In any case, some of the 
issues from other exploration and prototypes in this area:


1. Interaction with code on class path, esp. when you have two or more 
modules in the configuration for the boot layer that export the same 
package.


I would say that isolated modules should be inaccessible from the class 
path, and can be specified accordingly.  I don't think this would cause 
any particular usability concern or surprise; isolated modules are 
specifically intended to solve problems like package duplication that 
already means that the class path can't sensibly interact with them in 
an direct way.


Also, if a user is going so far as to isolate modules from one another, 
I think it's a relatively small step to modularize whatever class path 
code they have.


2. Split delegation issues that can arise when explicit modules on the 
module path do qualified exports to upgraded modules or even tool or 
automatic modules defined to the application class loader.


I'll have a look at the upgraded modules code; I didn't realize that 
they would work differently from a normal module path module.


Automatic modules should be subject to the same behavior as regular 
modules, but I'll double-check that as well.


By tool modules, are you referring to jdk.compiler etc.?

3. Visibility of types in non-exported packages, say where you have 
jdk.compiler defined to its own class loader but have code on the class 
path that makes use of types in conjunction with encapsulation busting 
options.


The current patch only allows application modules to be isolated. 
Otherwise this seems to be related to #1: I think it's reasonable to 
prevent class path code from accessing isolated modules.


I think it's important to note that the class path does not need to have 
direct access to the full set of modular features - especially if such 
access would come at the cost of truncating that feature set.



4. TCCL.


For this one I was thinking that TCCL should be set to the class loader 
of the module that was executed.  In the non-isolated case, this would 
be the application class loader, but in the isolated case it would 
depend on whether or not the main module was isolated.


There are other issues that arise from changing visibility but these are 
no different to issues that arise when using graphs of class loaders.


-Alan.




--
- DML

Re: Why not multiple modules of the same name in a class loader?

[David M. Lloyd]

On 05/12/2017 02:09 PM, David M. Lloyd wrote:

On 05/12/2017 01:46 PM, Remi Forax wrote:



On May 12, 2017 8:08:38 PM GMT+02:00, "David M. Lloyd" 
 wrote:

This has come up a couple times now and I'm not sure why the rule
exists: why not allow multiple modules with the same name in the same
class loader?


module names appear in the stack traces,  it will make life of people 
miserable if they have to open several artifacts to find the good one.


I agree; but, I wasn't looking for moral reasons, I was just curious 
about the technical constraint (if it even exists; maybe it does not!).



As far as I can tell, there is no way to locate a module by class
loader, and module namespaces are generally already well-governed by
their layer, so from an API perspective at least, it seems like this is

a harmless scenario (as long as there are no package conflicts,
obviously, but a container generally is able to manage this concern).

Preventing this does block one potential workaround for the lack of a
ModuleLayer.Controller.addPackage() method, which would be to
supplement
a module's content at run time by defining a new Layer that contains a
module of the same name as the original content within the same class
loader, where the container would then take care of such details as
mutual read/opening and cyclic access.

So that made me curious: is there some hotspot-internal dictionary that

I missed, which manages the set of modules within a class loader?


take a look to defineModule :)


I assume you mean Module.defineModule0 -> JVM_DefineModule -> 
Modules::define_module, which is where I looked.  It seemed to care 
about the class loader of the module, and about package names, but 
unless I'm just being blind to this (always a possibility), I don't see 
anyplace where it cares about what modules are defined to the class 
loader.  The closest thing I see is the bootstrap code which reads from 
the jimage, but none of that applies to user ModuleLayers.


In other words, I find no Map, or its C++ equivalent, at 
any level in a class loader that would be capable of such enforcement, 
let alone require it.


I guess I should do some more testing & tracing to see how this part 
works; maybe it's actually allowed!


Found it: there's a module table on ClassLoaderData that you get via its 
modules() method.



--
- DML

Re: Why not multiple modules of the same name in a class loader?

[David M. Lloyd]

On 05/12/2017 01:46 PM, Remi Forax wrote:



On May 12, 2017 8:08:38 PM GMT+02:00, "David M. Lloyd"  
wrote:

This has come up a couple times now and I'm not sure why the rule
exists: why not allow multiple modules with the same name in the same
class loader?


module names appear in the stack traces,  it will make life of people miserable 
if they have to open several artifacts to find the good one.


I agree; but, I wasn't looking for moral reasons, I was just curious 
about the technical constraint (if it even exists; maybe it does not!).



As far as I can tell, there is no way to locate a module by class
loader, and module namespaces are generally already well-governed by
their layer, so from an API perspective at least, it seems like this is

a harmless scenario (as long as there are no package conflicts,
obviously, but a container generally is able to manage this concern).

Preventing this does block one potential workaround for the lack of a
ModuleLayer.Controller.addPackage() method, which would be to
supplement
a module's content at run time by defining a new Layer that contains a
module of the same name as the original content within the same class
loader, where the container would then take care of such details as
mutual read/opening and cyclic access.

So that made me curious: is there some hotspot-internal dictionary that

I missed, which manages the set of modules within a class loader?


take a look to defineModule :)


I assume you mean Module.defineModule0 -> JVM_DefineModule -> 
Modules::define_module, which is where I looked.  It seemed to care 
about the class loader of the module, and about package names, but 
unless I'm just being blind to this (always a possibility), I don't see 
anyplace where it cares about what modules are defined to the class 
loader.  The closest thing I see is the bootstrap code which reads from 
the jimage, but none of that applies to user ModuleLayers.


In other words, I find no Map, or its C++ equivalent, at 
any level in a class loader that would be capable of such enforcement, 
let alone require it.


I guess I should do some more testing & tracing to see how this part 
works; maybe it's actually allowed!


--
- DML

Re: Some suggested patches and improvements

[David M. Lloyd]

On 05/12/2017 01:02 PM, David M. Lloyd wrote:

On 05/12/2017 08:31 AM, David M. Lloyd wrote:

On 05/12/2017 03:22 AM, Alan Bateman wrote:

On 12/05/2017 01:43, David M. Lloyd wrote:

I've proposed five patches to the jpms-spec-experts list [1..5] for 
discussion.  The patches are as follows:



[...]
3. Layer primitive: addPackage() - allows ModuleLayer.Controllers to 
add packages to a module after it has been defined

[...]
However for #3 then you've missed several important error cases, e.g. 
illegal package names, or the package is already in another module 
defined to the class loader. There is impact in other areas too.


I wasn't worried about validating package names at first, because I 
figured that you could never define a class to the package anyway, but 
nevertheless I see the value in error detection (and avoiding piles of 
cruft inside the JVM regardless of whether it might trip any kind of 
internal assertion, which I acknowledge as a possibility as well), so 
I've pushed up a patch [1] to add package name validation.  This also 
detects the empty package name situation.


Detecting package conflict is somewhat more subtle; that's coming up next.

[1] 
https://github.com/dmlloyd/openjdk-modules/commit/a636c8891d6958d7b395a57d17c95699ee549500 


Here [2] is my first pass at detecting package conflict.  As an 
additional precaution, I forbid packages to be added to the bootstrap 
class loader in this way.


[2] 
https://github.com/dmlloyd/openjdk-modules/commit/f02719801ac41c6fd9b8c216f16c3f86b909e5fe



--
- DML

Why not multiple modules of the same name in a class loader?

[David M. Lloyd]

This has come up a couple times now and I'm not sure why the rule 
exists: why not allow multiple modules with the same name in the same 
class loader?


As far as I can tell, there is no way to locate a module by class 
loader, and module namespaces are generally already well-governed by 
their layer, so from an API perspective at least, it seems like this is 
a harmless scenario (as long as there are no package conflicts, 
obviously, but a container generally is able to manage this concern).


Preventing this does block one potential workaround for the lack of a 
ModuleLayer.Controller.addPackage() method, which would be to supplement 
a module's content at run time by defining a new Layer that contains a 
module of the same name as the original content within the same class 
loader, where the container would then take care of such details as 
mutual read/opening and cyclic access.


So that made me curious: is there some hotspot-internal dictionary that 
I missed, which manages the set of modules within a class loader?

--
- DML

Re: Some suggested patches and improvements

[David M. Lloyd]

On 05/12/2017 08:31 AM, David M. Lloyd wrote:

On 05/12/2017 03:22 AM, Alan Bateman wrote:

On 12/05/2017 01:43, David M. Lloyd wrote:

I've proposed five patches to the jpms-spec-experts list [1..5] for 
discussion.  The patches are as follows:



[...]
3. Layer primitive: addPackage() - allows ModuleLayer.Controllers to 
add packages to a module after it has been defined

[...]
However for #3 then you've missed several important error cases, e.g. 
illegal package names, or the package is already in another module 
defined to the class loader. There is impact in other areas too.


I wasn't worried about validating package names at first, because I 
figured that you could never define a class to the package anyway, but 
nevertheless I see the value in error detection (and avoiding piles of 
cruft inside the JVM regardless of whether it might trip any kind of 
internal assertion, which I acknowledge as a possibility as well), so 
I've pushed up a patch [1] to add package name validation.  This also 
detects the empty package name situation.


Detecting package conflict is somewhat more subtle; that's coming up next.

[1] 
https://github.com/dmlloyd/openjdk-modules/commit/a636c8891d6958d7b395a57d17c95699ee549500


--
- DML

Re: Some suggested patches and improvements

[David M. Lloyd]

On 05/12/2017 09:37 AM, Stephen Colebourne wrote:

On 12 May 2017 at 01:43, David M. Lloyd  wrote:

1. Layer primitive: addExports() - mirrors the existing Module.addExports()
method for ModuleLayer.Controllers
2. Layer primitive: addUses() - mirrors the existing Module.addUses() method
for ModuleLayer.Controllers
3. Layer primitive: addPackage() - allows ModuleLayer.Controllers to add
packages to a module after it has been defined


Are these a good idea? I don't know. What I do know is that the use
cases for them seem to be focussed on low-level coders and framework
writers, which is a very small subset of all Java developers, and a
group who can work around difficulties.


I appreciate your confidence in our abilities. :)

However I think these changes will directly lead to a better user 
experience for those people who are using such containers.  And they 
don't impact those who don't.  A "majority rules" sort of metric doesn't 
always give a clear idea of cost/benefit, especially when you consider 
that while most people don't *develop* containers or frameworks, most 
users of Java *use* them.  Frameworks which work smoothly are a key 
ingredient for apps which work smoothly.



4. Make run-time cycle checking optional


My opinion is that run-time cycles are inevitable. The proposed
solutions (refactoring to API vs Impl) is not particularly good in an
open source context. I'm also concerned that "requires static"
(optional dependencies) also naturally leads to cycles

But the question at this point is whether it is worth the effort to
try and live without cycles for this release - after all, no-one wants
cycles in their design.


It's true, cycles sound bad.  But as a point of fact, at run time 
they're not demonstrably harmful, and in fact worrying overmuch about 
cycles detracts from clean design principles: I have an ABI that 
conforms to a particular behavioral contract; it should not matter how I 
meet that contract (and in particular, whether I do so by way of a 
series of other contracts that ultimately may be used by things which 
use *my* ABI is a completely uninteresting distraction).



Since there is a workaround, via the
low-level module/layer code, it feels like we should be willing to
take the punt and try to live without cycles in 9 keeping it under
review to add in 10.


What workaround are you referring to?  Without this patch, there's no 
way I can assemble an application with cycles on the module path without 
the JVM blowing up.


If you're referring to the addReads() API, don't think it's fair to 
users to require that a container or manual module management be 
employed in these cases, nor to forbid theses users from using jlink. 
Think about *why* this is being enforced at run time; what benefits will 
the user receive when they get this error?  What about costs?  With this 
change, the cost can be mitigated, while the benefits (whatever they may 
be) are retained.



5. Add optional class loader isolation for modules on the module path


Again, my opinion is that the isolation of modules is insufficient in
Jigsaw and that hidden packages will sometimes be very messy. However,
the only tool available to solve this appears to be classloaders, and
they have plenty of flaws themselves. While opt-in isolation appears
tempting, I'd prefer to wait and see a better solution in a later
version. Maybe the answer is to deprecated class loaders and come up
with something better!


Among the EC members who voted no, this was the most oft-cited technical 
concern (as far as I could find).  I don't think it is realistic to 
expect a change of vote without addressing the voters' primary concerns; 
what I'm trying to do is to create a compromise solution.


Experience with end user applications running under our modular system 
and OSGi over the past 5+ years seems to indicate that most software can 
cope with class loader isolation with few or no changes.  I would be 
highly skeptical that a piece of software being modularized is likely to 
survive the reflection changes, new service loader behavior, etc., yet 
somehow fail due to class loader isolation; nevertheless, making it 
opt-in means that this risk would be minimized.



Given where we are, I think my preference is to see a JDK 9 release
soon rather than end the JSR over these issues.


The JSR must be resubmitted within 30 days of the conclusion of the 
original vote; my hope is to get enough small changes in there that at 
least the *key* concerns are addressed so that the required 2/3 majority 
vote is likely.


But my purpose of posting these changes here is not to create debate 
over procedure or politics, but rather to discuss the technical 
qualities of the patches: Are they acceptable?  If not, why not, and 
what changes would be needed to make them acceptable, or what 
alternative approaches could solve the same problems in a better way?


--
- DML

Re: Some suggested patches and improvements

[David M. Lloyd]

On 05/12/2017 03:22 AM, Alan Bateman wrote:

On 12/05/2017 01:43, David M. Lloyd wrote:

I've proposed five patches to the jpms-spec-experts list [1..5] for 
discussion.  The patches are as follows:


1. Layer primitive: addExports() - mirrors the existing 
Module.addExports() method for ModuleLayer.Controllers
2. Layer primitive: addUses() - mirrors the existing Module.addUses() 
method for ModuleLayer.Controllers
3. Layer primitive: addPackage() - allows ModuleLayer.Controllers to 
add packages to a module after it has been defined

4. Make run-time cycle checking optional
5. Add optional class loader isolation for modules on the module path

>
#1-#3 are a subset of issue #LayerPrimitives in the JSR. I don't wish to 
comment on whether these make sense or not.


No problem, I'm just looking for technical feedback here.

However for #3 then you've 
missed several important error cases, e.g. illegal package names, or the 
package is already in another module defined to the class loader. There 
is impact in other areas too.


I can look into these two items; can you expand on the other impacts?

#4 seems to be working around the outcome of issue #CyclicDependences in 
the JSR. I also don't wish to comment on that except to say that 
introducing system properties to skip specified checks is highly 
problematic from a conformance perspective.


Can you explain what you mean by that?  I'm more than happy to convert 
these into -X type arguments to the runtime (or, of course, to simply 
make this the standard behavior), but I thought this would be a simpler 
and safer approach (at least for a first pass).


The only other place where I found run time cycles to be relevant is 
jlink and a couple of other tools, which I am in the process of 
developing a similar patch for.


There is a lot more to #5, something that will become clear when you 
work through all the scenarios. The JSR and spec part are minor though 
but I'd prefer to hold off until there is more discussion on this topic 
in the JSR.


I'd rather not hold off as the JSR essentially only has a couple of 
weeks left to live if there is not a revised PR.  Could you please 
explain what you mean?  Are you referring to jlink, jaotc, or something 
else?


--
- DML

Some suggested patches and improvements

[David M. Lloyd]

I've proposed five patches to the jpms-spec-experts list [1..5] for 
discussion.  The patches are as follows:


1. Layer primitive: addExports() - mirrors the existing 
Module.addExports() method for ModuleLayer.Controllers
2. Layer primitive: addUses() - mirrors the existing Module.addUses() 
method for ModuleLayer.Controllers
3. Layer primitive: addPackage() - allows ModuleLayer.Controllers to add 
packages to a module after it has been defined

4. Make run-time cycle checking optional
5. Add optional class loader isolation for modules on the module path

Apart from #3, each of these patches is small and very low-risk, and 
address a clear problem.  In the case of #3, I believe that the risk is 
very low, but am also willing to discuss modifying the patch to move the 
method to a "jdk.unsupported" class or in other ways.


I also am maintaining a mirror of the jigsaw/jake tree as a Git 
repository [6] which contains these patches where they can be viewed and 
fiddled with on a more interactive basis.


It is my belief that these small, low-cost patches will go a very long 
way towards mitigating many of the problems identified and shared by Red 
Hat, by other EG members, by other EC members, and by the community.


Please have a look and let me know if you have any feedback on any of 
these changes.


[1] 
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-May/000705.html
[2] 
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-May/000707.html
[3] 
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-May/000708.html
[4] 
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-May/000711.html
[5] 
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-May/000710.html

[6] https://github.com/dmlloyd/openjdk-modules

--
- DML

Re: Alternatives for naming automatic modules, and a proposal (#AutomaticModuleNames)

[David M. Lloyd]

On 04/26/2017 11:27 AM, mark.reinh...@oracle.com wrote:

2017/4/25 5:08:21 -0700, Stephen Colebourne :

On 24 April 2017 at 19:54,   wrote:

An explicit module that depends upon one or more modules that are
automatic today is, itself, no more stable than those automatic modules.
It could be broken when those automatic modules are modularized
explicitly, and if it `requires transitive` an automatic module then any
modules that depend upon it could be broken when that automatic module
is modularized explicitly.


I find this to be a completely artificial pure view of the world. Most
projects do not have internal packages, and if they do, most end-users
do not use them. When developers upgrade an artifact in Maven, they
expect to have incompatibilities, so a smaller set of packages is just
fine. Its a normal part of software development today, not an
aberration.


If developers expect to have to fix incompatibilities when they upgrade
to a newer version of an artifact then what's wrong, after all, with
publishing an explicit module that depends upon non-modularized
components in the form of automatic modules?

If one of those automatic modules is modularized later on, and given a
different name, then how is having to fix that materially different from
having to fix code that was using a package that's no longer exported?


I'd say the primary difference is that it is almost guaranteed that the 
artifact will have a new name, whereas it's far, far less likely that 
code will be using packages that are not then exported when the 
automatic module is modularized.  The chain of logic where it's OK to 
introduce minor problems because people are used to fixing minor 
problems doesn't hold up.  Minor problems are minor in large part 
because they don't occur frequently.



If anything it might actually be easier to cope with a module-name
change, since all you have to do is edit a `requires` directive; code
that refers to a package that's now encapsulated might require a
non-trivial rewrite.

I think I need to reconsider my previous conclusion that explicit modules
that depend upon automatic modules should never be published for broad
use [2].  Sure, they have unstable names and unstable APIs but if, as you
say, people are generally used to fixing minor problems when they upgrade
then this instability may well be tolerable -- especially since it would
allow maintainers to modularize whenever they want to rather than only
after all of their dependencies have been modularized.


Essentially it means that a fully modularized module *must* publish a 
new version every time an automatic dependency is properly modularized. 
This is an unreasonable extra burden to place on developers.


But, I don't believe that any of the pitfalls of automatic modules will 
really matter in practice, particularly as better migration tooling 
(than automatic modules) emerges.


--
- DML

Re: Alternatives for naming automatic modules, and a proposal (#AutomaticModuleNames)

[David M. Lloyd]

On 04/25/2017 09:15 AM, Brian Fox wrote:


On Tue, Apr 25, 2017 at 9:10 AM, David M. Lloyd mailto:david.ll...@redhat.com>> wrote:

I agree with everything except for this last point.  I think that,
given the amount of evangelism over the past 5 or so years, people
will adopt JPMS whether or not it is a fit for their use case.
Different shops will use different tooling in different ways to work
around these problems; I expect that automatic modules will probably
be mostly ignored except for "hello world" type cases in any event
though.


How would shops do this if everything they need from the ecosystem is
itself unmodularized? Maybe app authors will do so and take the pain
begrudgingly, but all the libraries that make Java such a powerful
ecosystem probably won't because they'll be caught between downstream
and upstream users.


I think that Maven- and Gradle-based tooling will appear to fill this 
need.  The only correct solution (automatic modules or not) to 
modularizing arbitrary groups of artifacts will likely involve a 
separate assembly step which will inevitably include writing or 
rewriting module descriptors.  I expect that the tooling will use the 
artifact dependency graph, module descriptors and/or other metadata 
present on the artifacts, and some manual configuration as input, and 
then produce as output a collection of corrected/re-written module JARs 
(and probably a complete wiring report that can be checked over by human 
eyes, probably with admonitions relating to potential or definite problems).


I really don't see any other way things could possibly work consistently 
for any application which uses external JARs.  Automatic modules are 
just going to be a stumbling block to reaching this conclusion.


OSGi, by the way, avoids this situation somewhat more effectively than 
Jigsaw can, mainly because of three factors: resolving by package 
instead of by bundle name, performing the bulk of resolution work at run 
time, and by the simple fact that it is less invasive towards 
reflection, thus there is more "wiggle room".


This isn't to say that OSGi is an ideal solution of course, but I just 
wanted to point out why the OSGi ecosystem can generally get away with 
publishing artifacts containing descriptor information, while Jigsaw 
probably won't be able to (or at least, not as cleanly).


--
- DML

Huge implications of multiple class loaders for application modules

[David M. Lloyd]

On 04/25/2017 08:31 AM, Alan Bateman wrote:

On 25/04/2017 14:16, Michael Nascimento wrote:


:
and also the fact
packages must be defined by a single module even if they are not
exported,
because it was considered nearly impossible to track in real world
scenarios.


Keep in mind that this is not a module system limitation, instead it is
just a consequence of defining all modules on the application module
path to the application class loader. Containers or applications using
the API doing dynamic configurations and creating module layers can put
modules in their own namespace to avoid conflicts between concealed
packages.

Someday then we might get to the point where modules on the application
module path could be assigned to different class loaders but it has huge
implications and would take an entire release to shake out issues.


I tend to disagree with the assertion that there would be huge 
implications.  Libraries and applications have been coping with multiple 
class loaders for many, many years, so this nothing new for them.  There 
are no real compatibility implications, as the JPMS ecosystem does not 
yet exist.  There are certainly no detrimental security implications (on 
the contrary, security should be marginally improved in the security 
manager situation as getting class loaders for other classes requires 
special permissions).


What exactly *are* the huge implications of this approach?  My attempts 
to get to the bottom of this on the spec experts list have gotten 
nowhere, so hopefully I'll have better luck here.



BTW: The reasons why there are conflicts between concealed packages is a
good discussion point for another thread.

-Alan


--
- DML

Re: Alternatives for naming automatic modules, and a proposal (#AutomaticModuleNames)

[David M. Lloyd]

On 04/25/2017 07:08 AM, Stephen Colebourne wrote:

On 24 April 2017 at 19:54,   wrote:

An explicit module that depends upon one or more modules that are
automatic today is, itself, no more stable than those automatic modules.
It could be broken when those automatic modules are modularized
explicitly, and if it `requires transitive` an automatic module then any
modules that depend upon it could be broken when that automatic module
is modularized explicitly.


I find this to be a completely artificial pure view of the world. Most
projects do not have internal packages, and if they do, most end-users
do not use them. When developers upgrade an artifact in Maven, they
expect to have incompatibilities, so a smaller set of packages is just
fine. Its a normal part of software development today, not an
aberration.

Frankly though, if you think automatic modules are as bad as you make
out, you should simply remove them.


I still agree with this point.  This is a solution looking for a 
matching problem.


[...]

Unfortunately, the concept of bottom-up full modularization simply
won't work, no matter how much the Jigsaw team hopes it will. The
process would take forever, may not be possible for some projects,
will be side-tracked into the release cycles of larger projects, be
blocked by dead projects and for many other reasons just stall. I'd
also note that everyone outside Oracle has given the same message.

[...]

The only approach that makes any sense to migration is to drop the
artificial purity goal. We already have build tools (Maven, Gradle)
that manage versions, locking them in to form a working graph of

[...]

Worrying about getting incomplete "impure" modules in Maven Central is
simply the wrong concern. If we keep the current design, and insist on
no automatic module dependencies in Maven Central, then JPMS will
simply not be adopted outside a few private niches.


I agree with everything except for this last point.  I think that, given 
the amount of evangelism over the past 5 or so years, people will adopt 
JPMS whether or not it is a fit for their use case.  Different shops 
will use different tooling in different ways to work around these 
problems; I expect that automatic modules will probably be mostly 
ignored except for "hello world" type cases in any event though.


Just the fact that there is the *very idea* of a "fit"/"non-fit" for 
JPMS is sad though.  It should have been the ubiquitous thing that 
everyone was expecting.  But denying multiple versions?  Blowing up on 
run time cycles?  Reneging on the idea of being the basis of Java EE 9? 
These are things that people will not be expecting.


--
- DML

Re: How to actually ship JSR-250?

[David M. Lloyd]

To address the first point only... I think if Java EE 8 required Java SE 
9, that would be surprising news for all.  For the last few Java EE 
releases at least, each Java EE release has required the Java SE version 
of the same number.  And there has been plenty of talk over the past 
couple years that seems to indicate that this will continue to 9.  Thus 
I have no expectation that Java EE 8 will do anything special for Java SE 9.


However, we definitely want our implementation to run on Java SE 9.  My 
understanding was that you cannot replace classes in upgradeable modules 
via the class path, but if that's wrong then I suppose it's time to do 
another round of experiments.


On 04/20/2017 11:45 AM, Stephen Felts wrote:

First, I'm not sure that Java EE 8 (corresponding to Java SE 9) will define 
modules.
It seems that the RI will tolerate JDK9 (remove internal JDK API calls) as 
opposed to adopting the new module system.

Since javax.annotation.* classes are now hidden in JDK 9, every project that 
uses these classes will need to ship a copy of these classes.

For a project that is not adopting the module system, these classes will be on 
the classpath.  They can just ship the RI jar file (currently 
glassfish4/glassfish/modules/endorsed/javax.annotation-api.jar) and in JDK9 
endorsed is no longer supported but no longer needed for these classes (that's 
the reason for hiding the classes).
To provide the jax-ws annotations (they seem to live in 
glassfish4/glassfish/modules/endorsed/webservices-api-osgi.jar) also on the 
classpath, it just works.
Both jar files need to be provided by project that uses these annotations and 
added to the classpath directly or indirectly via a manifest classpath.
(As a side note, there are some API's that are not in the endorsed directory 
that will need to be additionally shipped for projects that use them.)

For a project that is adopting the module system, the jar files need to be 
turned into modules with the correct names and the associated module-info.class 
in the root.
Both modules need to be shipped and both jar files need to be in the module 
path or upgrade module path?

If two projects run in the same JVM, one using the classpath and one using the 
module path, the module path wins?




-Original Message-
From: David M. Lloyd [mailto:david.ll...@redhat.com]
Sent: Thursday, April 20, 2017 9:21 AM
To: Alan Bateman; jigsaw-dev
Subject: Re: How to actually ship JSR-250?

On 04/20/2017 08:18 AM, Alan Bateman wrote:

On 20/04/2017 13:57, David M. Lloyd wrote:


Let's say I want to create a module distribution with my own JSR 250
classes.  Let's also assume that I or the spec team want the module
to be named java.annotations.common.

How do I properly upgrade the JDK's java.xml.ws.annotation module
such that java.xml.ws can see it, *and* I have my proper
java.annotations.common module name?  Am I forced to ship my
annotations module as java.xml.ws.annotation?  Or manually replace
java.xml.ws with one that looks for my annotations (and where does
that leave the original java.xml.ws.annotation module)?


java.xml.ws.annotation is upgradable so all you need to do is deploy
the compiled form of the following on the upgrade module path:

module java.xml.ws.annotation {
requires transitive java.annotations.common; }

Separately, I don't know what name that JSR-250 will eventually choose.
It could be indeed be "java.annotations.common", maybe
"java.ee.annotation", maybe something else.


I'm not too worried about that; it has to be decided by the time of Java EE 9 
(obviously) but until then I just want to ensure that it is definitely possible 
to do this, as (unless I missed something) I don't believe that it was 
adequately covered in prior discussions.

--
- DML



--
- DML

Re: How to actually ship JSR-250?

[David M. Lloyd]

On 04/20/2017 08:18 AM, Alan Bateman wrote:

On 20/04/2017 13:57, David M. Lloyd wrote:


Let's say I want to create a module distribution with my own JSR 250
classes.  Let's also assume that I or the spec team want the module to
be named java.annotations.common.

How do I properly upgrade the JDK's java.xml.ws.annotation module such
that java.xml.ws can see it, *and* I have my proper
java.annotations.common module name?  Am I forced to ship my
annotations module as java.xml.ws.annotation?  Or manually replace
java.xml.ws with one that looks for my annotations (and where does
that leave the original java.xml.ws.annotation module)?


java.xml.ws.annotation is upgradable so all you need to do is deploy the
compiled form of the following on the upgrade module path:

module java.xml.ws.annotation {
requires transitive java.annotations.common;
}

Separately, I don't know what name that JSR-250 will eventually choose.
It could be indeed be "java.annotations.common", maybe
"java.ee.annotation", maybe something else.


I'm not too worried about that; it has to be decided by the time of Java 
EE 9 (obviously) but until then I just want to ensure that it is 
definitely possible to do this, as (unless I missed something) I don't 
believe that it was adequately covered in prior discussions.


--
- DML

How to actually ship JSR-250?

[David M. Lloyd]

Let's say I want to create a module distribution with my own JSR 250 
classes.  Let's also assume that I or the spec team want the module to 
be named java.annotations.common.


How do I properly upgrade the JDK's java.xml.ws.annotation module such 
that java.xml.ws can see it, *and* I have my proper 
java.annotations.common module name?  Am I forced to ship my annotations 
module as java.xml.ws.annotation?  Or manually replace java.xml.ws with 
one that looks for my annotations (and where does that leave the 
original java.xml.ws.annotation module)?


--
- DML

Re: Disallowing the dynamic loading of agents by default (revised)

[David M. Lloyd]

On 04/06/2017 02:56 AM, Alan Bateman wrote:

On 05/04/2017 17:55, David M. Lloyd wrote:



This is just plain weird from a security perspective, to say that
unrelated processes have more privilege to control the current process
than processes that are closely related.

Anyway this is yet another case where arbitrary artificial hurdles are
put in place for the purpose of human behavior modification. Such
hurdles can always be bypassed, generally resulting in even uglier
situations that the one you're trying to avoid.  In this case I can
just fire a child process and then attach to it from the parent.  Or
fire off two sibling processes and have one attach to the other.
Nothing is being saved here.

This thread/proposal is concerned with libraries using APIs intended for
tools to do brain surgery in the current VM.


I know, I'm giving examples of how such a library could circumvent this 
restriction.  Another example is to start a child process and a 
grandchild process, and then have the child process exit.


Adding technical hurdles to send a social message is frankly pretty 
lousy engineering.  It never fails to backfire.


--
- DML

Re: Disallowing the dynamic loading of agents by default (revised)

[David M. Lloyd]

On 04/05/2017 11:15 AM, mark.reinh...@oracle.com wrote:

Thanks to everyone for the quick feedback on this topic, and especially
to Andrew for the constructive dialogue.

Here's a revised proposal:

  - Define a new VM option, `-XX:+EnableDynamicAgentLoading`, that's
on by default in JDK 9 but off by default in JDK 10.

This will allow launch scripts that use this option on JDK 10 to
work on JDK 9 without change, and will allow early testing of the
JDK 10 behavior on JDK 9.

  - Revise the `com.sun.tools.attach` API to forbid attachment to the
current process or to an ancestor of the current process, and
define a read-only system property that allows such attachment to
be enabled via the command line.


This is just plain weird from a security perspective, to say that 
unrelated processes have more privilege to control the current process 
than processes that are closely related.


Anyway this is yet another case where arbitrary artificial hurdles are 
put in place for the purpose of human behavior modification.  Such 
hurdles can always be bypassed, generally resulting in even uglier 
situations that the one you're trying to avoid.  In this case I can just 
fire a child process and then attach to it from the parent.  Or fire off 
two sibling processes and have one attach to the other.  Nothing is 
being saved here.



This will discourage the inadvertent use of libraries that, for
better or for worse, intentionally violate strong encapsulation.

  - Enhance the `-jar` launcher option so that if the JAR file being
launched contains a `Premain-Class` attribute then it's launched
as both an application and as an agent for that application.

This will allow `java -jar foo.jar` to be used in place of the
more verbose `java -javaagent:foo.jar -jar foo.jar` [1].

Taken together, these changes are intended to enable the continued use
of legitimate dynamically-loaded agents without change on JDK 9 and with
a small change on JDK 10.  That later change will align the treatment of
such agents with the other means of breaking encapsulation (`--add-opens`,
etc.) in order to ensure integrity by default for all code.

This proposal does not attempt to lock down platform classes as distinct
from user classes.  Many agents have legitimate reasons to transform
platform classes, so an additional mechanism to protect those classes
does not appear to be worthwhile.

Comments?

- Mark


[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-April/012000.html



--
- DML

Re: Review Request JDK-8175819: OS name and arch in JMOD files should match the values as in the bundle name

[David M. Lloyd]

On 04/04/2017 10:12 AM, mark.reinh...@oracle.com wrote:

2017/4/4 1:04:22 -0700, magnus.ihse.bur...@oracle.com:

On 2017-04-03 23:50, Mandy Chung wrote:

...

   JDK 8   JDK 9
   -   -
OS_NAMELinux   linux
   SunOS   solaris
   Darwin  macos
   Windows windows

OS_ARCHi386,x86x86
   i586,amd64,x86_64   amd64
   sparcv9 sparcv9
   arm arm32
   aarch64 arm64


If we are making changes to the original proposal from JDK-8175819, then
I just want to add my few cents:

Why change from the well-established "aarch64" to the virtually unused
"arm64"? As far as I know, using the name "arm64" for the aarch64
platform is something that has only been done in the (recently opened)
closed Oracle port. This change, however, proposes to change the value
in the release file even for the open aarch64 port, which has always
been known by that name.


The trouble here is that "arm64" and "aarch64" are effectively synonyms
for the ISA, but in the JDK we've wound up using them as the names of
two different ports.

A JMOD file built for the 64-bit ARM architecture will (one hopes) run
equally well on either port.  Which name should we use in JMOD files,
"arm64" or "aarch64"?  My sense is that "arm64" is more immediately
understood by developers at large even if "aarch64" is more correct
in the eyes of ARM Holdings plc, but I could be wrong.

For what it's worth, the Linux distros aren't consistent: Debian-based
distros use "arm64", while Red Hat / Fedora seem to prefer "aarch64".


If you use "arm64" instead of "aarch64" then shouldn't you use "sparc64" 
instead of "sparcv9"?  Same logic AFAICT.

--
- DML

Re: Progress report on SLF4J project modularization

[David M. Lloyd]

On 03/24/2017 02:33 PM, Ceki Gülcü wrote:



On 3/24/2017 19:28, David M. Lloyd wrote:


Hi Ceki, we came up with an approach that works for us (though it's a
bit cumbersome).  See: http://word-bits.flurg.com/multrelease-jars/



Hi David,

Thanks for the pointer. I am trying to compile modules-info.java
selectively which seems to be a much harder problem.


It should be possible to do it using our approach if you make it part of 
the Java 9 supplement.  You'd have to add a step to bring that resource 
out separately into the root of the JAR during assembly though.


--
- DML

Re: Progress report on SLF4J project modularization

[David M. Lloyd]

On 03/24/2017 12:41 PM, Ceki Gülcü wrote:


Hi All,

The initial phase of SLF4J project modularization in preparation for
Java 9/Jigsaw went well. I ran into many wrinkles most of which were
due to my own ignorance about Java 9 modules.

You can see the results at

https://github.com/qos-ch/slf4j/tree/1_8_0-SNAPSHOT

Now that the initial phase of modularization is done, I looked at
creating mutlti-release jars for the SLF4J project. However, after
having spent 2 full days on trying to get Maven to do multi-release
jars, I am giving up without having made any measurable progress.


Hi Ceki, we came up with an approach that works for us (though it's a 
bit cumbersome).  See: http://word-bits.flurg.com/multrelease-jars/


--
- DML

Re: #LayerPrimitives

[David M. Lloyd]

On 03/17/2017 11:12 AM, Stephen Colebourne wrote:

I note the discussions on various threads about JPMS primitives.

My first expectation is that OSGI and JBoss modules can run in Java 9
in classpath mode.

I believe this to be true today.


Of course.


My second expectation is that there are sufficient primitive
operations within JPMS to allow a _new_ module system (not OSGI or
JBoss Modules) to be built that permits some but not all of the
features seen in OSGI and JBoss Modules. Thes should include:
- dynamic loading/reloading of modules
- lifecycle callbacks
- multiple versions of a class loaded at the same time
- handling of clashing packages
- some form handling for module cycles

It is clear to me that JPMS does contain an API with some flexibility
here, but it is hard to judge whether my second expectation is met.
This is because the discussion is ledf by OSGI and JBoss Modules
migration concerns, rather than by consideration of what a new module
system would need. Perhaps these concerns are the same, perhaps not. I
would like to see a discussion from the EG about what blocks a
theoretical new extended module system built on top of JPMS.


In the green field, all things are possible.

You could create a variety of module systems that behave in a variety of 
ways and yet still adhere to the current JPMS restrictions.  If you're 
willing to discard, rewrite, or retool all existing software, and change 
best practices (even if the current practices are not particularly 
harmful or problematic, or even useful), and derive your acceptable use 
cases from the implementation restrictions (instead of the other way 
around), you can conform to anything and fit in any situation.


But, accepting the proposals for this and other issues makes that green 
field better for everyone, and increases that variety and decreases the 
number of use cases which would otherwise be unjustly left to obsolescence.



Specifically, I note Thomas Watson's concern over non-lazy class
loader construction as being a concern [1] (point 2) for module
systems.

My third expectation is that it should be possible for bytecode/class
generation tools to generate code inside a module, including in a
package that is not known by the module graph.


A lot can be accomplished through copious amounts of bytecode injection. 
 But I have a hard time accepting any argument to the effect that this 
is an acceptable programming interface for legitimate use cases, common 
or not.  Particularly when the remedy is trivial!



I note that Proxy appears to have a special case for "dynamic
modules", and am concerned that this may over-emphasise Proxy compared
to other similar tools that are not part of the JDK.


Indeed the requirement for these unusual constructs is a huge design red 
flag, yet is generally dismissed on the basis that only special citizens 
need access to it: a self-justifying argument with an ideological basis.



Note that while I understand the desire for OSGI and JBoss Modules to
fit into JPMS and get the benefits, I do not consider that necessary
to complete the JSR - it is more important to close out Java 9 and
move on.


Is it though?  I argue that this importance is a hazardous illusion, and 
furthermore has been and will continue to be used as a tool to undermine 
the community process.  Ten years from now, will anyone care that Java 9 
was delayed another 6 months, or even a year, in order to get a few 
things right?  I strongly doubt it.  Will they care that the community 
process was shunned in favor of expedience?  This seems more likely to me.



These existing systems work today, and provide benefits
today, something that won't change. That they won't get the enhanced
security benefits of JPMS modules is a secondary concern IMO.


It is perhaps secondary to the very large number of vendors and users 
with a substantial investment based on these systems and, much more 
importantly, the development practices they represent.


The thing that concerns me with this line of reasoning is that all of 
these arguments could be used to justify anything at all.  The spec lead 
could just run down the clock or strong-arm the EG into nearly anything. 
 To contrive a humorous example, banning the use of the letter "M" from 
Java could be justified this way.  At what point do we start arguing in 
terms of engineering standards - especially when the remedies are often 
quite simple and even trivial?


--
- DML

Re: Alternatives to automatic modules as a concept

[David M. Lloyd]

On 03/17/2017 09:08 AM, Alan Bateman wrote:

On 17/03/2017 13:36, David M. Lloyd wrote:



SLF4J APIs and implementations are cyclic.  You have to manually
bundle them into a single JAR before they could work with automodules.

BTW: Which of the providers are you using to see the problem? I'm
wondering if you might have converted them to explicit modules rather
than leave them as automatic modules.  FWIW, I don't see any issues when
using the simple/default implementation:

$ java -p mlib --list-modules slf4j.api,slf4j.simple
automatic module slf4j.api@1.7.25 (file:///d/mlib/slf4j-api-1.7.25.jar)
  requires mandated java.base
  contains org.slf4j
  contains org.slf4j.event
  contains org.slf4j.helpers
  contains org.slf4j.spi
automatic module slf4j.simple@1.7.25
(file:///d/mlib/slf4j-simple-1.7.25.jar)
  requires mandated java.base
  contains org.slf4j.impl

$ java -p mlib--add-modules slf4j.api,slf4j.simple -m m/p.Main
[main] INFO p.Main - started


Hmm, so automatic modules explicitly are going to allow cycles then? 
How is that going to not hinder migration, if cycles are disallowed once 
the thing is properly modularized?


--
- DML

Re: Alternatives to automatic modules as a concept

[David M. Lloyd]

On 03/17/2017 08:40 AM, Alan Bateman wrote:

On 17/03/2017 13:36, David M. Lloyd wrote:



SLF4J APIs and implementations are cyclic.  You have to manually
bundle them into a single JAR before they could work with automodules.

They can combined via command line options but would be interesting to
see if you can put the API JAR on the module path so that people can
depend it, leave the implementation on the class path (assuming code in
explicit modules would not reference types in that JAR).


The API itself directly references the implementation via static 
inter-class linkage, so I'm guessing that wouldn't work.

--
- DML

Re: Alternatives to automatic modules as a concept

[David M. Lloyd]

On 03/17/2017 08:34 AM, Alan Bateman wrote:



On 17/03/2017 13:30, David M. Lloyd wrote:

And some auxiliary food for thought: today, it will fail on any JAR
set that includes SLF4J.  Sorry SLF4J fans!


Can you be more specific?


SLF4J APIs and implementations are cyclic.  You have to manually bundle 
them into a single JAR before they could work with automodules.

--
- DML

Re: Alternatives to automatic modules as a concept

[David M. Lloyd]

On 03/17/2017 03:50 AM, Stephen Colebourne wrote:

On 16 March 2017 at 16:21, David M. Lloyd  wrote:

3) Remove the concept of automodules entirely

Migration would be slower, but safer. Tooling might evolve to add
module-info dynamically as part of the build or runtime system, or by
editing all the jar files in Maven Central to have an auto-generated
module name.


Indeed, it would be relatively easy to introduce a tool that would assemble
an arbitrary set of JARs into a module graph, using a configuration
descriptor for only the most advanced cases (pruning the service list,
specifying reflection capabilities if they are more specific than "open
all").  Such a tool would be superior to automatic modules in many ways.

This approach has other important advantages: zero magic (particularly if a
tool can describe what it has done concisely), and choice of implementation.

Even as I was writing this, a colleague of mine sent me a message that he
has already independently created such a tool.  The URL is:
https://github.com/moditect/moditect


Er, I can't think of a viable option 4.



4) Support a mode of operation where the descriptor is external to the
module content.

Containers are already going to be doing this anyway after all.  This is
similar, in a way, to option 3 where tooling (or a person, if we can ever
break away from this idea of requiring bytecode for this purpose) would be
able to establish the modular environment without modifying the artifacts.
This is a model that we use today with success.


I think a solution to automodules cannot _require_ a container or a
tool. Option 3, "remove automodules", merely suggests that by removing
them containers or tools _may_ fill _some_ of the gap. But
tooling/containers can't fill the gap for the key part of automodules,
allowing the author of a module-info.java to depend on something that
has not been modularized. Which is why it is borderline as to whether
option 3 "solves" the specific problem in the original mail.

External descriptors allow an application to be stitched together from
jar files, modularized or not (where guessing module names is fine,
because it is the descriptor for an application, not a public
library). However, external descriptors don't solve the problem where
someone is writing a module-info.java file for a public library and
has to guess a module name for a dependency they don't own.

Note that I'm not disagreeing with the idea of an external descriptor,
just that I think it solves a different problem (and should exist
outside JPMS).


Something to remember is that naming is ultimately the responsibility of 
the distributor of the final module assembly.  The framework author(s) 
can make a good guess about a name but (particularly where split API and 
implementations or specifications are concerned) there's a very good 
chance that any distributor of a nontrivial application is going to have 
to perform substitutions anyway.  Since this is likely to be a common 
occurrence (at least with larger applications), it should be planned 
for.  If users are already expecting to use tooling to create their 
assemblies, then automodules don't actually help because they just add 
another axis of unpredictability to the assembly process.


I expect an assembly workflow to involve fetching artifacts with preset 
versions, either transforming an existing descriptor or establishing a 
new one (probably based at least on the combination of existing module 
descriptors, Maven metadata, and (possibly) some small amount of user 
configuration, which BTW will result in a far better "guess" than 
automodules possibly could), and then finally bundling up the result in 
some way (either jlink or via a more traditional mechanism).


This is why I think automodules should be dropped: I cannot envision a 
real-world non-trivial development scenario where it could possibly be 
helpful.  And some auxiliary food for thought: today, it will fail on 
any JAR set that includes SLF4J.  Sorry SLF4J fans!


--
- DML

Re: Alternatives to automatic modules as a concept

[David M. Lloyd]

On 03/16/2017 10:48 AM, Stephen Colebourne wrote:

Automodules exist to provide a means for gradual migration from a
non-modular world to a modular one. Unfortunately, they have proven
controversial with most of the EG and Jigsaw list. This thread seeks
to bring together some of the proposals to remove automodules.

The key issue is that automodules force a module author to pick the
name of modules that they do not control. Note that while a better
module naming strategy (discussed elsewhere) reduces the potential for
issues, a module author should never, ever be asked to guess somebody
else's name. That is the fundamentally broken concept.

These are the possible alternatives that remove the need for guessing
a name that I can see (without changing JPMS so much that Java 9 is
delayed again):

[...]

3) Remove the concept of automodules entirely

Migration would be slower, but safer. Tooling might evolve to add
module-info dynamically as part of the build or runtime system, or by
editing all the jar files in Maven Central to have an auto-generated
module name.


Indeed, it would be relatively easy to introduce a tool that would 
assemble an arbitrary set of JARs into a module graph, using a 
configuration descriptor for only the most advanced cases (pruning the 
service list, specifying reflection capabilities if they are more 
specific than "open all").  Such a tool would be superior to automatic 
modules in many ways.


This approach has other important advantages: zero magic (particularly 
if a tool can describe what it has done concisely), and choice of 
implementation.


Even as I was writing this, a colleague of mine sent me a message that 
he has already independently created such a tool.  The URL is: 
https://github.com/moditect/moditect



Er, I can't think of a viable option 4.


4) Support a mode of operation where the descriptor is external to the 
module content.


Containers are already going to be doing this anyway after all.  This is 
similar, in a way, to option 3 where tooling (or a person, if we can 
ever break away from this idea of requiring bytecode for this purpose) 
would be able to establish the modular environment without modifying the 
artifacts.  This is a model that we use today with success.


--
- DML

Re: Java SE JSR 250 annotations module renamed to java.xml.ws.annotation?

[David M. Lloyd]

On 02/14/2017 05:52 PM, mark.reinh...@oracle.com wrote:

2017/2/13 9:17:47 -0800, Guillaume Smet :

On Mon, Feb 13, 2017 at 6:10 PM, alan.bate...@oracle.com wrote:

On 13/02/2017 16:58, Guillaume Smet wrote:

On Mon, Feb 13, 2017 at 5:12 PM, alan.bate...@oracle.com wrote:


I agree that @Generated is awkward but I haven't suggested removing it.


What do you suggest then? As far as I understood you, you were suggesting
removing the module in Java 10 so the @Generated annotation would also be
gone? Or did I misunderstand?


That is the proposal. If it goes ahead then it means that tools that rely
on these annotations in the JDK would need to deploy the standalone version
on the class path or as a module on the module path.


Yeah, so basically, it would end up with
http://hg.openjdk.java.net/code-tools/jmh/rev/d74b2861222c .

I don't think it's the best possible outcome for this useful annotation.


I agree.

The `@Generated` annotation falls outside the original charter of the
`java.lang.annotation` package, which was meant for annotations that
directly support the language's annotation facility, but we already
added `@Native` in SE 8, so let's add `@Generated` in SE 9 as David
suggests and encourage people to use it when running on this and later
releases.

The fact that `@Generated` is so widely used is new information to some
of us, so thanks for bringing it up.


I'm glad to hear it... thanks!

--
- DML

Re: Java SE JSR 250 annotations module renamed to java.xml.ws.annotation?

[David M. Lloyd]

On 02/13/2017 11:30 AM, fo...@univ-mlv.fr wrote:

- even as a temporary workaround, it's really weird to have to require a module
named java.xml.ws.annotation for that. It does not really make sense from an
end user perspective.


the module name reflect the fact that it's a part of jax-ws and not annotations 
that you can use for other things.


And here is the problem: these are JSR 250 annotations and I guarantee 
you that >99% of Java users don't know (or care) why they were 
introduced in to the platform.  The fact that they came in as part of 
JAX-WS cannot possibly be less relevant to their modern presence in the 
JDK.  Trying to force people to care (by putting the annotations under 
an otherwise obscure module identifier) is neither helpful nor useful to 
just about anyone; no amount of pretending is going to convince anyone 
that these annotations belong to JAX-WS and not JSR 250.


Nothing in this discussion has changed (at least) my position: the 
classes cannot be removed in the Java 9 time frame; JSR 250 should have 
its own identity regardless of why it was originally introduced; if it 
is to be removed later then there should be a platform replacement for 
@Generated in java.lang.annotation that is ideally introduced now so 
that there is an overlapping transition period.

--
- DML

Re: Java SE JSR 250 annotations module renamed to java.xml.ws.annotation?

[David M. Lloyd]

On 02/13/2017 08:58 AM, David M. Lloyd wrote:

On 02/13/2017 08:49 AM, Alan Bateman wrote:

On 13/02/2017 14:32, David M. Lloyd wrote:


I think this is an error.  It makes more sense to have the
javax.annotation package exist in its own module.  If the long-idle
JSR 250 and specifications like it are really a concern, then this
module should follow the pattern of all other such modules and be
upgradeable.

It is its own module and it is upgradable.



This once again shines a bright light on a few key Jigsaw defects, and
it's getting a bit frustrating watching requirements get reconned to
compensate for implementation problems.  This is purely a hack to make
up for an implementation difficulty and makes no sense when framed
from the perspective of the end user. Let's try to do better.

The technical debt here that a handful of APIs are "shared" between Java
SE and Java EE. The first steps to addressing that technical debt are in
the JSR 379 EDR.


I would agree but for the problem of @Generated.  Would that it had been
put into java.lang.annotation to begin with!  But the reality is that
there has been a long-standing assumption that is broken with Java 9
that this class would be in the JDK, thus code generators have been free
to include this annotation (which, on the surface, seems like a useful
thing to do) without fear of introducing additional dependencies.  Now
it's a compilation error unless you either include a very non-intuitive
module name, or else an external dependency.  The authors of such code
generators can likely figure this out, but users will almost certainly
run into trouble.  It's yet another bump without a very good reason.


Here is what I said to Andrew Haley (our JSR 379 rep):


I think that if this module is to be present in the JDK, it should retain its previous 
name ("java.annotations.common").  If the module is to be deprecated for 
removal from the platform (i.e. permanently replaced with an external dependency), then 
the @Generated annotation should be deprecated from JSR-250 and its related 
specifications, and also (ideally) moved into java.lang.annotation if it is deemed still 
useful, because such an annotation should always be present in the JDK.






--
- DML

Re: Java SE JSR 250 annotations module renamed to java.xml.ws.annotation?

[David M. Lloyd]

On 02/13/2017 08:49 AM, Alan Bateman wrote:

On 13/02/2017 14:32, David M. Lloyd wrote:


I think this is an error.  It makes more sense to have the
javax.annotation package exist in its own module.  If the long-idle
JSR 250 and specifications like it are really a concern, then this
module should follow the pattern of all other such modules and be
upgradeable.

It is its own module and it is upgradable.



This once again shines a bright light on a few key Jigsaw defects, and
it's getting a bit frustrating watching requirements get reconned to
compensate for implementation problems.  This is purely a hack to make
up for an implementation difficulty and makes no sense when framed
from the perspective of the end user. Let's try to do better.

The technical debt here that a handful of APIs are "shared" between Java
SE and Java EE. The first steps to addressing that technical debt are in
the JSR 379 EDR.


I would agree but for the problem of @Generated.  Would that it had been 
put into java.lang.annotation to begin with!  But the reality is that 
there has been a long-standing assumption that is broken with Java 9 
that this class would be in the JDK, thus code generators have been free 
to include this annotation (which, on the surface, seems like a useful 
thing to do) without fear of introducing additional dependencies.  Now 
it's a compilation error unless you either include a very non-intuitive 
module name, or else an external dependency.  The authors of such code 
generators can likely figure this out, but users will almost certainly 
run into trouble.  It's yet another bump without a very good reason.


--
- DML

Re: Java SE JSR 250 annotations module renamed to java.xml.ws.annotation?

[David M. Lloyd]

I think this is an error.  It makes more sense to have the 
javax.annotation package exist in its own module.  If the long-idle JSR 
250 and specifications like it are really a concern, then this module 
should follow the pattern of all other such modules and be upgradeable.


Note that until #AvoidConcealedPackageConflicts is resolved, AFAICT the 
suggestion of depending on the JSR 250 API can't work anyway if 
java.xml.ws.annotations is used because there will be two sets of the 
same package in the module system.


This once again shines a bright light on a few key Jigsaw defects, and 
it's getting a bit frustrating watching requirements get reconned to 
compensate for implementation problems.  This is purely a hack to make 
up for an implementation difficulty and makes no sense when framed from 
the perspective of the end user.  Let's try to do better.


On 02/13/2017 08:08 AM, Claes Redestad wrote:

Hi Guillaume,

On 02/13/2017 02:34 PM, Guillaume Smet wrote:

At the end, I'm wondering if these classes shouldn't be part of the
default
set of classes exposed without requiring an --add-modules option.


one of the main reasons why java.xml.ws.annotations is not resolved
by default is that there are many projects out there with dependencies
on various JSR 305 implementations[1], which is a non-final and dormant
JSR which happen to clash with JSR 250 by adding more annotations to
the same packages... thus if java.xml.ws.annotations is resolved at
boot time using --add-modules then any project with a dependency on
one of the rogue JSR 305 implementations is likely to break[2].

A perhaps less fragile alternative here is to not depend on the
JDK-bundled java.xml.ws.annotations at all but instead depend directly
on the JSR 250 API as published by the JCP[3] as an explicit dependency.

Hope this helps!

/Claes

[1] https://jcp.org/en/jsr/detail?id=305
[2] See http://openjdk.java.net/jeps/261#EE-modules for more info
[3] E.g.:
https://mvnrepository.com/artifact/javax.annotation/jsr250-api/1.0


--
- DML

Re: hg: jigsaw/jake/jdk: Add AccessibleObject::canAccess and trySetAccessible

[David M. Lloyd]

I don't recall any corresponding discussion of this, but the change 
looks like a good idea to me.


On 02/04/2017 05:41 PM, mandy.ch...@oracle.com wrote:

Changeset: 335657c73d0b
Author:mchung
Date:  2017-02-04 15:41 -0800
URL:   http://hg.openjdk.java.net/jigsaw/jake/jdk/rev/335657c73d0b

Add AccessibleObject::canAccess and trySetAccessible

! src/java.base/share/classes/java/lang/SecurityManager.java
! src/java.base/share/classes/java/lang/System.java
! src/java.base/share/classes/java/lang/invoke/MethodHandles.java
! src/java.base/share/classes/java/lang/reflect/AccessibleObject.java
! src/java.base/share/classes/java/security/Security.java
! src/java.base/share/classes/jdk/internal/misc/JavaLangAccess.java
+ test/java/lang/reflect/AccessibleObject/CanAccessTest.java
+ test/java/lang/reflect/AccessibleObject/TrySetAccessibleTest.java



--
- DML

Re: Automatic module names

[David M. Lloyd]

On 02/06/2017 12:51 PM, Alan Bateman wrote:

On 06/02/2017 18:25, David M. Lloyd wrote:



I don't think there is really a substantial increase in risk between
manually modularizing a library and relying on automatic modules.

If several projects take on the responsibility to modularize a library
that they don't maintain then there is no guarantee that they will end
up with equivalent modules. Static analysis (`jdeps
--generate-module-info ...`) gets you so far but it often needs a deeper
understanding of the library to be confident that the module declaration
that they write for the library is complete. Automatic modules take on
much of the behavior of a library on the class path so that it's not
necessary to synthesize everything that an explicit module would declare.


As could a tool, which has the added advantage that the module 
information could easily be incrementally narrowed instead of going 
fully from automatic to fully specified.  To do incremental change from 
automatic to fully specified, you'd need a tool to generate an 
automatic-equivalent descriptor anyway.


--
- DML

Re: Automatic module names

[David M. Lloyd]

On 02/03/2017 11:42 AM, Alan Bateman wrote:

On 03/02/2017 16:42, David M. Lloyd wrote:


:

Sure, but the set of JARs you can't rewrite is really pretty spare.


I've no doubt that many people would be uncomfortable taking
responsibility to modularize a component that they don't maintain or
have full knowledge of.


I don't think there is really a substantial increase in risk between 
manually modularizing a library and relying on automatic modules.  Any 
discomfort is purely illusory, as the present automatic module process 
is a pretty well-defined sequence of steps, and a manual or 
tool-assisted modularization process would be equally well-defined, even 
if not exactly equivalent.  Anything beyond the facts of the processes 
themselves would really be wandering into psychology, but I don't see a 
significant disadvantage with this approach (on the contrary the added 
flexibility seems like an advantage to me).



In the early exploration phase of Project Jigsaw
then we have some basic tooling to help with exactly that approach but
it has long been left behind. One reason is that potential for anarchy
with several people converting the same library to a module.


There's no technical solution for preventing this situation when a 
single module environment is being assembled by multiple parties.  They 
simply must coordinate, or the result may be inconsistent or incorrect. 
I don't see how automatic modules provide any sort of general remedy for 
this.



You might
not like automatic modules but at least the module descriptors that are
derived are very predictable (including the name).


I'm not really arguing about what I like, it's purely about comparative 
quality: in my experience a thing that works for some cases while 
causing potentially subtle or confusing problems in others should 
generally be considered unacceptable.  In the vast majority of cases, 
it's better to do nothing than to do something half-baked.


--
- DML

Re: Automatic module names

[David M. Lloyd]

On 02/03/2017 09:40 AM, Alan Bateman wrote:

On 03/02/2017 11:44, Robert Scholte wrote:


Hi Nicolai,

let's consider that my project depends on the following dependencies:
com.foo.bar:library:1.0 and com.acme:library:2.3.1, both unnamed.

I somehow want to have them both as requirements:
module M.N {
  requires static library; // com.foo.bar:library
  requires library; // com.acme:library
}

How can I define that the 'requires static library' should be mapped
to com.foo.bar:library:1.0 on the modulepath, while 'requires library'
should be mapped to com.acme:library:2.3.1

One ugly solution would be:
  requires static library containing com.foo.bar.baz.SomeClass;
  requires library containing acme.AnotherClass;

We should really wonder if ease-of-transition is worth the minefield
we're creating with the introduction of automodules. IMHO all options
we're trying to add to keep automodules will only over-complicate
things, not even being sure if all edges are covered.

I think automatic modules are critical to migration, otherwise things
move at the pace of the slowest project (which might be 0km/h in case of
unmaintained projects). A lot of the discussion here has been on the
naming but automatic modules but the other important benefit of
automatic modules is that they support bridging to the class path.


I still believe this conclusion is based on a false assumption, or 
rather a pair of them: the assumption that users will generally wish to 
distribute a module as a direct artifact of their build in any cases but 
the most proprietary, and the assumption that modularizing a project is 
something that is done when that project is compiled rather than when it 
is distributed.  All evidence seems to indicate that these assumptions 
are completely unfounded.


Using tooling to migrate to a modularized environment is not only a 
perfectly reasonable alternative, but is very, *very* likely to be quite 
close to the standard build/distribute workflow coming out the gate and 
onward for many years.  Maven and Gradle essentially dominate the world 
of Java project building.  They (and their supporting utilities) already 
take on most if not all distribution tasks: customizing MANIFEST.MF and 
service files, bundling local and even remote resources, source and 
bytecode translation, annotation processing, customized packaging, etc. 
It seems inevitable that such tools will also be used build module 
distributions, performing resolution and wiring tasks on large groups of 
artifacts at a time and producing a module set or even an image at 
output, as a final step before CI and/or distribution.  I cannot 
conceive of any other sane workflow unless as I said your environment is 
highly proprietary (neither consuming nor producing artifacts that are 
ever shared with anyone else nor shared across distibution-wide 
versions) and/or highly constrained (to a very small number of modules 
or using very rigid (and probably brittle) rules for artifact sharing). 
Note that in these cases, most if not all of the value proposition of 
modularity is already lost.



As regards the example naming clash then these two projects might
already get complaints over their poor choice of artifacts, esp. when
artifacts for both projects are in same directory (say where someone
distributes with all JAR files in a `lib` directory).


--
- DML

Re: Automatic module names

[David M. Lloyd]

On 02/03/2017 08:43 AM, Andrew Dinn wrote:

On 03/02/17 14:29, David M. Lloyd wrote:

I think one option we should consider is to perhaps disable automatic
modules for 9 and revisit the idea for 10, as it's late in the day and
still clearly not settled.


I don't think this is thinking about the trade-off correctly.

Automatic modules may not work for some (or maybe many) of the more
complicated cases but those failures can be addressed over time by
adding a module.xml to update releases of jars.

Automatic modules definitely does work for straightforward cases to
provide an easy way of deploying jars you don't own/can't rewrite as
modules.


Sure, but the set of JARs you can't rewrite is really pretty spare. 
And, in the few cases where it might cause a hypothetical problem (say, 
it's signed or something, though I doubt that this actually prevents 
rewriting the descriptor), there is at least one alternative provider 
that can modularize them cleanly (like ours, for example).


Tools like Maven can and probably will easily cover the vast majority of 
cases more effectively than this mechanism.  "It fixes many cases" is 
necessary but not sufficient for acceptance in my view.  "It doesn't 
bungle other cases" is also a necessary condition.



Much as I admit that there are going to be lots of cases where it will
fail I think those where it just works will still be a large subset. So,
automatic modules will definitely be a big help to a lot of users who
want to get started with Jigsaw.

And, well, maybe I need to say this -- yes, an easy start is a /big/
priority.


Yes, however there are more ways than one to bake that particular cake.


That's merely 2 cents, gratis. Your mileage may vary, particularly when
it fails for your app. But I don't the mere possibility of the latter as
a reason to poop on someone (everyone?) else's parade.


I sympathize, mainly because software generally is a Tough Thing to Do, 
however "on the ground" I tend to disagree fundamentally with that 
sentiment, as in my experience this tends to lead immediately to "if the 
first idea works even a bit, run with it", which in turn means that 
quality becomes a lowest common denominator, which in turn invariably 
leads to high costs (of various types).  These ideas simply don't scale, 
and they're the ones that one inevitably looks back at 5 years down the 
road, saying "gee, I really wish we hadn't done that".  Sometimes the 
right answer is "OK, well that idea had some merit, but not enough to 
justify it".


I don't disagree with the goal of providing a rapid on-ramp for users - 
quite the opposite - but I think that the automatic modules feature 
itself is neither necessary nor sufficient in order to meet it.  I think 
tooling can be just as easy and intuitive, and far more effective in 
terms of use cases covered, flexibility, evolvability, etc.


--
- DML

Re: Automatic module names

[David M. Lloyd]

I think one option we should consider is to perhaps disable automatic 
modules for 9 and revisit the idea for 10, as it's late in the day and 
still clearly not settled.


On 02/03/2017 05:44 AM, Robert Scholte wrote:

Hi Nicolai,

let's consider that my project depends on the following dependencies:
com.foo.bar:library:1.0 and com.acme:library:2.3.1, both unnamed.

I somehow want to have them both as requirements:
module M.N {
  requires static library; // com.foo.bar:library
  requires library; // com.acme:library
}

How can I define that the 'requires static library' should be mapped to
com.foo.bar:library:1.0 on the modulepath, while 'requires library'
should be mapped to com.acme:library:2.3.1

One ugly solution would be:
  requires static library containing com.foo.bar.baz.SomeClass;
  requires library containing acme.AnotherClass;

We should really wonder if ease-of-transition is worth the minefield
we're creating with the introduction of automodules. IMHO all options
we're trying to add to keep automodules will only over-complicate
things, not even being sure if all edges are covered.

thanks,
Robert

On Thu, 02 Feb 2017 12:28:13 +0100, Nicolai Parlog  wrote:


 Hi everyone,

after thinking about this a little longer, I came to the conclusion that
compile-time/launch-time aliasing might be the only way out of this (at
least the only I could come up with) that keeps automatic modules alive
and does not introduce a conceptual dependency on Maven.

The idea:

A command line option, let's say `--alias-modules A=X`, maps module name
A to module name X. Every dependency on either A or X will be resolved
to X, implying that there must a module X in the universe of observable
modules. There can be several aliases for the same module
(`--alias-modules A=X,B=X`; X needs to be observable) and they can be
chained (`--alias-modules A=X,X=Y`; Y needs to be observable)

Aliasing would of course have to be applied to qualified exports, opens,
and similar mechanisms as well.

It might be worth adding the rule that no observable module must have an
aliased name. So for `--alias-modules A=X` there must be no observable
module A. This prevents ambiguity and would effectively prevent aliasing
platform modules. That might be a good thing because it looks like
aliasing and upgrading modules has quite some overlap (or is even
identical?)

Unfortunately I could not come up with a way to limit aliasing to
automatic module names (in case that were desirable) without somehow
marking dependencies on automatic modules, likely in the module
declaration. If changing module declaration syntax is still on the
table, it could be changed so that dependencies on automatic modules
must be phrased as something like `requires automatic`.

The obvious semantics would be that only such requires clauses can be
fulfilled with automatic modules and that only such dependencies could
be aliased (this might make it prudent to phrase the aliasing option
accordingly, e.g. `--alias-automatic-modules`).

This could also be used to help developers in keeping their module
declarations clean: The compiler could to emit a warning if a `requires
automatic` clause is fulfilled by a regular module.

I would love to hear some thoughts on this idea, even if it considered
to be stupid, impractical,etc. :)

 so long ... Nicolai



On 27.01.2017 15:11, Stephen Colebourne wrote:

Back in October, I raised the issue of modules names generally and for
automatic modules specifically [1]. The short thread came to no
conclusion, but recent threads have again raised similar problems. The
problem is that automatic modules have magical name creation from a
filename, which is brittle and unlike anything else in Java.

I also recently looked at the Joda-Convert and Joda-Beans libraries,
to see if I could add module-info in preparation for Java 9. I quickly
backed away, again because of the same issue. Put simply, I am
unwilling to write a module-info file that refers to a dependency that
is not yet a module. And I have to advise all open source projects to
do the same. Given this, there can be no simple migration to the JPMS
for open source projects. Each open source project must wait for all
its dependencies to migrate to JPMS (by adding a module-info and
publishing to Maven Central).

The issue is clear. If I write this:

module org.joda.convert {
  requires guava;
}

where guava is an automatic module, I am locking in the name of the
guava dependency, something that I do not control. The name "guava" is
just a guess. The guava authors might choose "com.google.guava" or
something else entirely.

In a closed system of modules, ie. a private application, automatic
modules are fine, because the requires clause can be changed if it
turns out the guess was wrong. But once published as an open source
project to Maven Central or elsewhere, the guess cannot be fixed if it
is wrong (without releasing a new version of the library, which is not
an acceptable solution).

I also str

Re: Automatic module names

[David M. Lloyd]

So far I haven't commented on automatic modules because while I don't 
see them as useful, I also didn't see them as harmful.  However this 
keeps popping up, so, see inline comments.


On 01/27/2017 08:11 AM, Stephen Colebourne wrote:
[...]

The issue is clear. If I write this:

module org.joda.convert {
  requires guava;
}

where guava is an automatic module, I am locking in the name of the
guava dependency, something that I do not control. The name "guava" is
just a guess. The guava authors might choose "com.google.guava" or
something else entirely.

[...]

In a closed system of modules, ie. a private application, automatic
modules are fine, because the requires clause can be changed if it
turns out the guess was wrong. But once published as an open source
project to Maven Central or elsewhere, the guess cannot be fixed if it
is wrong (without releasing a new version of the library, which is not
an acceptable solution).

[...]


- Remove the automatic module concept altogether


This is my vote.  Automatic modules are meant to simplify migration by 
allowing "legacy" artifacts to be included in a modular environment. 
But in real use cases, it is often if not usually the case that manual 
intervention will be required anyway.  Therefore, automatic modules do 
not actually make things any easier (in fact they hide land mines around 
your deployment operation).  It makes more sense to create or use a 
provided tool to easily modularize a JAR by allowing its dependencies to 
be specified.  A system like Maven might allow such a modularization 
process to be largely automatic, thanks to the dependency information 
already at its disposal (though such a case *should* present a report to 
the user allowing them to spot any problems or potential problems that 
may arise from a particular generated wiring; in general, magic is bad 
if it isn't well-defined or easily validated).


As I've expounded on previously, on multiple occasions, it is highly 
unlikely that centrally deployed modules will be universally useful 
without manual work on the part of the consumer.  It is far better to 
make this process as clean, easy, and well-defined as possible than to 
"fudge" it using automatic modules, magical wiring, or anything else of 
the sort, which seems easy at first and then bites back later.



Given that applications can depend on libraries that haven't been
released in years, this has the potential to be a critical problem for
the ecosystem. My preference remains to define a clear mapping from
the widely adopted Maven Central naming strategy to JPMS modules.


This could be very useful.  But I think it should be a Maven function 
rather than adding this to the JPMS, forever.


--
- DML

Re: #CompileTimeDependencies and module resolution

[David M. Lloyd]

On 01/13/2017 06:48 AM, Sander Mak wrote:



The standard use case for the feature is for libraries with optional
dependencies:


That is indeed the use case I was thinking of.


The --add-modules flag is only relevant when using the command line to
turn setup #1 into setup #2, something which should be rare.


Interesting, so what you're saying is if an application wants the optional 
behaviour of Lib through A, the application module itself (or any of its 
modules) has to declare a non-optional dependency in its descriptor on module 
A. Even though, technically, this application module doesn't have any direct 
relation at all with A. If you do that, the application can freely use types 
exported from A. Which is not always what you want (in most cases even, I'd 
say). For example, Lib would be a machine learning library that the application 
uses and A would be a super-fast matrix multiplication library (possibly with 
native code not available on all platforms, so it has to be optional). I won't 
ever use the matrix multiplication lib A directly in my application, but I want 
it to be used by Lib for increased performance.

What I was expecting, is that the mere presence of A on the module path would 
cause it to be resolved when Lib is resolved, without needing a non-static 
requires or --add-modules. Conversely, if A isn't there, Lib would get resolved 
without A just as well. Obviously Lib needs to guard for this situation, as you 
said.


This is the intuitive behavior I also expect of an optional dependency. 
The problem however is that a static dependency isn't an optional 
dependency; it's a compile-time-only dependency, which is not exactly 
the same thing.


We (Jigsaw) don't have a concept of an optional dependency (i.e. one 
that is eagerly used if present but does not cause an error if absent) 
at all.  Maybe we should though, as experience has shown that this is a 
useful operating mode.



Alternatively, you can view optional dependency usage more like 'if the 
application already uses A, then Lib should also use A as well' in which case 
your suggested setup and the current implementation make total sense. This does 
make the example I described above a bit awkward, but I don't have any data on 
which use case is more prevalent.

Thanks for the insights!


Sander



--
- DML

Re: Proposed enhancement to Layer API

[David M. Lloyd]

On 01/04/2017 08:26 AM, Alan Bateman wrote:

On 04/01/2017 14:21, David M. Lloyd wrote:


We need the ability to add content to a module after it is defined to
support our existing deployment architecture.  Is there a material
problem with having this feature, other than transgressing your sense
of "should"?

Can you expand a bit on this? If you are generating modules at run-time,
maybe generating module-info.class files or via the Builder API, then I
would expect that you at least know what packages are in the module and
which packages are exported unconditionally. It sounds a bit strange to
create a module without knowing anything about its contents (assuming
that is what this is about).


In some cases we have the full contents, but in other cases we allow 
dynamic deployment contents to be updated or added at run time (mostly 
for development purposes but it can be practical and useful to add 
contents at run time as well in certain cases).  Because of the way that 
our linkage algorithm is designed, our current Jigsaw integration 
prototype just adds all contents after the fact (mainly because of the 
dependency chain involved in building our modules, in which the contents 
simply aren't available at the time we establish the class loader and 
layer).


It is theoretically possible to rewrite our linkage algorithm to use the 
first known set of packages as the static package set for a module, but 
we'd still need the ability to add contents anyway, or else drop the 
feature entirely, which would be unfortunate.


--
- DML

Re: Proposed enhancement to Layer API

[David M. Lloyd]

We need the ability to add content to a module after it is defined to 
support our existing deployment architecture.  Is there a material 
problem with having this feature, other than transgressing your sense of 
"should"?


On 01/03/2017 05:33 PM, Remi Forax wrote:

I agree with all the proposed methods but the one that add a package, in my 
opinion, a package is part of what structurally define a module, it should not 
be possible to add it after the resolution algorithm.

Rémi

- Mail original -----

De: "David M. Lloyd" 
À: "jigsaw-dev" 
Envoyé: Mardi 3 Janvier 2017 21:45:04
Objet: Proposed enhancement to Layer API



This is a proposed enhancement to the Layer API, a sibling to the
discussion around #NonHierarchicalLayers going on in the JPMS spec list.
 While discussion is ongoing, I want to ensure that there is a "real"
patch somewhere out in the public that can be referenced and discussed.

Essentially this patch allows our module system to achieve a basic level
of functionality in conjunction with Jigsaw, by giving the Layer's
Controller more control over the content and behavior of its modules.

I hereby (on behalf of myself and of my employer) submit the code in the
patch in this email message under the terms of the MIT license, which
(among other things) should allow the OpenJDK project to appropriate and
modify these code changes at will without copyright assignment issues.

Here's the patch:

diff --git
a/jdk/src/java.base/share/classes/java/lang/reflect/Layer.java
b/jdk/src/java.base/share/classes/java/lang/reflect/Layer.java
index 4b4588c..be013e9 100644
--- a/jdk/src/java.base/share/classes/java/lang/reflect/Layer.java
+++ b/jdk/src/java.base/share/classes/java/lang/reflect/Layer.java
@@ -196,6 +196,28 @@ public final class Layer {


 /**
+ * Updates module {@code source} in the layer to contain a new
package.
+ * This method is a no-op if {@code source} already contains
package {@code pn}.
+ *
+ * @param  source
+ * The source module
+ * @param  pn
+ * The package name to add
+ *
+ * @return This controller
+ *
+ * @throws IllegalArgumentException
+ * If {@code source} is not in the layer
+ */
+public Controller addPackage(Module source, String pn) {
+Objects.requireNonNull(source);
+Objects.requireNonNull(pn);
+ensureInLayer(source);
+Modules.addPackage(source, pn);
+return this;
+}
+
+/**
  * Updates module {@code source} in the layer to read module
  * {@code target}. This method is a no-op if {@code source}
already
  * reads {@code target}.
@@ -251,6 +273,160 @@ public final class Layer {
 Modules.addOpens(source, pn, target);
 return this;
 }
+
+/**
+ * Updates module {@code source} in the layer to open a package to
+ * all modules.
+ *
+ * @param  source
+ * The source module
+ * @param  pn
+ * The package name
+ *
+ * @return This controller
+ *
+ * @throws IllegalArgumentException
+ * If {@code source} is not in the layer or the package
is not
+ * in the source module
+ *
+ * @see Module#addOpens
+ */
+public Controller addOpensToAll(Module source, String pn) {
+Objects.requireNonNull(source);
+ensureInLayer(source);
+Modules.addOpensToAll(source, pn);
+return this;
+}
+
+/**
+ * Updates module {@code source} in the layer to open a package to
+ * to all unnamed modules.
+ *
+ * @param  source
+ * The source module
+ * @param  pn
+ * The package name
+ *
+ * @return This controller
+ *
+ * @throws IllegalArgumentException
+ * If {@code source} is not in the layer or the package
is not
+ * in the source module
+ *
+ * @see ModuleDescriptor.Builder#opens(String)
+ */
+public Controller addOpensToAllUnnamed(Module source, String pn) {
+Objects.requireNonNull(source);
+ensureInLayer(source);
+Modules.addOpensToAllUnnamed(source, pn);
+return this;
+}
+
+/**
+ * Updates module {@code source} in the layer to export a
package to
+ * module {@code target}. This method is a no-op if {@code source}
+ * already exports the package to at least {@code target}.
+ *
+ * @param  source
+ * The source module
+ * @param  pn
+ * The package name
+ * @param  target
+ * The target module to re

Re: Valid characters in a module name

[David M. Lloyd]

On 01/03/2017 03:38 PM, Ess Kay wrote:

Java EE and JBoss Modules both allow this

Two points. Firstly there is a big difference between a character being
allowed and the character actually being used in practice. Are you
saying that in practice anyone anywhere is putting spaces, single quotes
or double quotes within a Java EE and JBoss module name?  If the answer
is yes then how common would that be?


I don't have metrics, of course; one only needs to know the API contract 
to know that this is allowed.  But it is completely irrelevant to the 
discussion of the requirement at any rate because you're confusing a few 
things which I'll outline down below.



Also, do Java EE and JBoss module names currently allow the characters
in the range 0x00 to 0x1F?  If the answer is yes then 100% compatibility
with Java 9 module names is already gone.


Yes it's allowed (not 0x00 but the others are), but I don't think there 
is any practical way to actually accomplish injecting most of those 
values (at least in a Java EE situation), nor have I ever seen it happen 
in practice.



The second point is that we are now talking about Java 9 module names
being embedded as identifiers within Java class files where they will
directly affect downstream users.  This was not the case with Java EE
and JBoss Module names.  This is a much, much bigger deal.


No, that's not how it works at all.  Java source modules can only 
reference other Java source modules; nobody is going to be shocked that 
you can't reference a manually-built module from a Java source module. 
There's nothing to affect downstream users.  Only container code will be 
creating or referencing such modules.



There has been plenty of serious thought

In the jpms-spec-experts list it is suggested that "for sanity" Java 9
module names should not contain "any character whose Unicode code point
is less than 0x20".  Yet the DEL (0x127) character is allowed?

Taking a step backwards, it would appear that it was never considered
that Java 9 module names might need be be specified as identifiers in
existing bytecode processing scripting languages.  That is 100%
understandable.  However, now that it is known that that is the case,
doesn't it make sense "for sanity" to not allow a Java 9 module name to
contain spaces, single quotes, double quotes, semicolons or asterixes?


From the perspective of container code, those "sanity" characters are 
selected completely arbitrarily.  They have various usages in various 
contexts, but any given module layer implementation doesn't necessarily 
align with any of those contexts.  For example there may be different 
characters which don't make sense, while some of the examples you listed 
do.  So it makes the most sense to allow anything at the bytecode level, 
and rely on the module layer implementation to apply the appropriate policy.



The problem with spaces, single quotes, double quotes in an identifier
that needs to be parsed from a text file is obvious.


Sure, but the lenient rules only apply to class files that were manually 
generated.  There are zero cases where one would have to parse an 
arbitrary module identifier from a text file; every module layer 
provider is going to have their own syntax and naming policy.



As mentioned
earlier, the problem with semicolons is that they are commonly used as
terminators in the scripting languages which nearly always use a Java
style syntax.  The problem with asterixes is that they are commonly used
as a wildcard character in identifiers in the languages.


In the language, the module identifier is bounded by quite strict 
syntax.  Modules which are distributed for downstream consumption will 
likewise adhere to these criteria (they have no way to avoid it short of 
weird bytecode hacking).  The only time the more general rules come into 
play is when modules are being generated at run time from other module 
systems and setups.


Let me put it another way.

Every module system has its own rules and restrictions for how a module 
can be named.  Those restrictions do not all exactly align.  If you ban 
the union of all the disallowed characters in all module systems, then 
all module systems will break.  However if you only ban the intersection 
of such systems (i.e. control characters), and allow each layer to 
enforce its own policy, then every system will work and interoperate as 
expected.  There is no downside because anyone who "cleverly" hacks 
bytecode to produce a vanilla module with an invalid name will soon 
realize that their module can never be found.  A user has to go to 
extraordinary measures to do so, so there is very little risk of such a 
thing happening nor is there a risk of it impacting users in any 
relevant way.


Everyone has their own notion of what "offensive" characters would be. 
But enforcing these rules is and can only be the job of 

Proposed enhancement to Layer API

[David M. Lloyd]

This is a proposed enhancement to the Layer API, a sibling to the 
discussion around #NonHierarchicalLayers going on in the JPMS spec list. 
 While discussion is ongoing, I want to ensure that there is a "real" 
patch somewhere out in the public that can be referenced and discussed.


Essentially this patch allows our module system to achieve a basic level 
of functionality in conjunction with Jigsaw, by giving the Layer's 
Controller more control over the content and behavior of its modules.


I hereby (on behalf of myself and of my employer) submit the code in the 
patch in this email message under the terms of the MIT license, which 
(among other things) should allow the OpenJDK project to appropriate and 
modify these code changes at will without copyright assignment issues.


Here's the patch:

diff --git 
a/jdk/src/java.base/share/classes/java/lang/reflect/Layer.java 
b/jdk/src/java.base/share/classes/java/lang/reflect/Layer.java

index 4b4588c..be013e9 100644
--- a/jdk/src/java.base/share/classes/java/lang/reflect/Layer.java
+++ b/jdk/src/java.base/share/classes/java/lang/reflect/Layer.java
@@ -196,6 +196,28 @@ public final class Layer {


 /**
+ * Updates module {@code source} in the layer to contain a new 
package.
+ * This method is a no-op if {@code source} already contains 
package {@code pn}.

+ *
+ * @param  source
+ * The source module
+ * @param  pn
+ * The package name to add
+ *
+ * @return This controller
+ *
+ * @throws IllegalArgumentException
+ * If {@code source} is not in the layer
+ */
+public Controller addPackage(Module source, String pn) {
+Objects.requireNonNull(source);
+Objects.requireNonNull(pn);
+ensureInLayer(source);
+Modules.addPackage(source, pn);
+return this;
+}
+
+/**
  * Updates module {@code source} in the layer to read module
  * {@code target}. This method is a no-op if {@code source} 
already

  * reads {@code target}.
@@ -251,6 +273,160 @@ public final class Layer {
 Modules.addOpens(source, pn, target);
 return this;
 }
+
+/**
+ * Updates module {@code source} in the layer to open a package to
+ * all modules.
+ *
+ * @param  source
+ * The source module
+ * @param  pn
+ * The package name
+ *
+ * @return This controller
+ *
+ * @throws IllegalArgumentException
+ * If {@code source} is not in the layer or the package 
is not

+ * in the source module
+ *
+ * @see Module#addOpens
+ */
+public Controller addOpensToAll(Module source, String pn) {
+Objects.requireNonNull(source);
+ensureInLayer(source);
+Modules.addOpensToAll(source, pn);
+return this;
+}
+
+/**
+ * Updates module {@code source} in the layer to open a package to
+ * to all unnamed modules.
+ *
+ * @param  source
+ * The source module
+ * @param  pn
+ * The package name
+ *
+ * @return This controller
+ *
+ * @throws IllegalArgumentException
+ * If {@code source} is not in the layer or the package 
is not

+ * in the source module
+ *
+ * @see ModuleDescriptor.Builder#opens(String)
+ */
+public Controller addOpensToAllUnnamed(Module source, String pn) {
+Objects.requireNonNull(source);
+ensureInLayer(source);
+Modules.addOpensToAllUnnamed(source, pn);
+return this;
+}
+
+/**
+ * Updates module {@code source} in the layer to export a 
package to

+ * module {@code target}. This method is a no-op if {@code source}
+ * already exports the package to at least {@code target}.
+ *
+ * @param  source
+ * The source module
+ * @param  pn
+ * The package name
+ * @param  target
+ * The target module to read
+ *
+ * @return This controller
+ *
+ * @throws IllegalArgumentException
+ * If {@code source} is not in the layer or the package 
is not

+ * in the source module
+ *
+ * @see Module#addExports
+ */
+public Controller addExports(Module source, String pn, Module 
target) {

+Objects.requireNonNull(source);
+Objects.requireNonNull(target);
+ensureInLayer(source);
+Modules.addExports(source, pn, target);
+return this;
+}
+
+/**
+ * Updates module {@code source} in the layer to export a 
package

Re: Suggestion: allow accessible reflection on protected methods of exported types.

[David M. Lloyd]

On 12/29/2016 01:09 PM, Jochen Theodorou wrote:

On 29.12.2016 18:24, Rafael Winterhalter wrote:

Hei Jochen, thank you for your feedback. I must have missed your
posting, I regularly fail to follow up on the things discussed on the
mailing list and its not easy to seatch for topics, sorry for that.


don´t worry, my complaints have not been ignored, but did not change
anything either.


In Byte Buddy, I created an option to use subclassing by creating
dedicated class loaders for proxy classes. There is however a row of
disadvantages:

1. Class loaders are not cheap objects, they require some memory and the
performance cost of creating dedicated class loaders is significant. I
have benchmarked this, as well and it costs you about three times the
time to create such "wrapper class loaders" rather than injecting.


agreed


2. It is close to impossible to define a good life-cycle for classes
loaded by such wrapper class loaders. You need to keep a reference to
the proxy class to prohibit its collection which in turn keeps a
reference to the original class loader making it uncollectable. Using
soft or weak references is less then ideal, too and the only option
would be to use ephemerons which are not supported by Java. With
injection, it is as easy as looking up a previously created class from
the proxied class's class loader which dies and lives with this class
loader.


A ClassValue is no ephemeron, but letting it keep a SoftReference to the
proxy class may work. It means there will be situations in which the
class will be recreated, but thanks to the SoftReference both classes
and loaders should still be garbage collectable.


3. Java packages define their identity by name and class loader at
runtime. Many developers use anonymous classes or nested classes which
are package-private and cannot be subclassed by creating a new class
loader but which must live in the same class loader.


package-private is another story... I would be happy enough to have a
generic solution for protected.

and 4.  if the  loader is defined by a module you will have fun with
creating layers and export/read edges


Those are the main issues that lead us to strongly prefer injection,
especially in a testing library like Mockito and there are some more
complications that we would like to avoid.


the reply will most probably be about asking why these classes are not
made public. They can be exported to only the framework module and all
that.


I am however especially concerned about upgrading. Esepcially with
mature and wide-spread libraries like Mockito or Hibernate, even minor
changes can cause really big, hard-to-debug issues for our users.


been there and I agree. But it is unlikely that something like the
implementation of strong encapsulation will be done without trouble for
bigger projects. Will this lower adoption of Java9? Sure.


Switching from injection to subclassing would be much more than a minor
change. The JVM developers surely know this, otherwise they would not
keep classes like Stack around or fix quirks in classes like ByteBuffer.
Too much code was written on top of these classes and their expected
inner working. If we were however forced to change our APIs that
drastically, I think the real consequences would be much more drastic
than the mentioned changes. At the moment, as a library author, I do
however feel like there are few options to avoid this.


On the other hand... if you think it has to be done at some point, then
why delay it? Of course the question of if it has to be done is
questionable in itself, especially since the slim advantages of the
module system turn into the negative, if weighted against "other
solutions", the work you will have with it and the limitations it imposes.


Here's an important point about adoption (these are my own observations, 
not the EG's or my employer's).  One of the effects of the way Java is 
produced and its historical entrenchment is that no matter what happens, 
adoption is more or less inevitable; every prediction of Java's death 
has proven to be false (which of course doesn't stop pundits from doing 
so anyway).  Even if a future Java was terrible by many objective 
measures, its adoption will probably be eventually inexorable.  It would 
take something highly extraordinary to change these basic truths.  This 
is just the nature of the Java ecosystem today.  Lack of adoption is a 
consequence that essentially nobody is worried about, and is really not 
a deterrent to any given proposed change.


But this just means that it is all the more critical that everyone that 
uses Java in their day to day work to take some degree of responsibility 
to ensure that every language and JDK specification change is reviewed 
for quality.  The JCP EC consists of many parties; it's my personal 
belief and hope that there will always be EC members (hopefully the 
majority of them in fact) that will listen to the practical concerns of 
the community.


It's not a perfect system 

Re: Valid characters in a module name

[David M. Lloyd]

On 01/01/2017 07:44 PM, Ess Kay wrote:

Hi Rémi,


You can update your tool to use an escape character

Sure. However, can you imagine how much work it would be to update a Java
source parser to allow identifiers like package and class names to contain
escaped semi-colons, single quotes or double quotes? My scenario and that
of many others is the same.  It can be done but the result will be ugly.

I repeat my earlier question, are there existing module systems out there
that allow spaces, quotes and semi colons to appear in a module name?


Yes.  Java EE and JBoss Modules both allow this, as do systems where a 
file name is a module name.



All I ask is that serious thought be given to how much flexibility is
really needed in a module name.  There are signs that there has not yet
been much serious thought.  For example backspace is not allowed but DEL
(0x127) is allowed.


There has been plenty of serious thought, and I agree that we should be 
disallowing all Unicode controls of any kind, but my understanding is 
that there are implementation complexities involved which make this 
somehow impractical.  However UTF-8 parsing is not difficult so 
hopefully this can be revisited at some point.


--
- DML

Re: Module exports for java.base: NIO stuff

[David M. Lloyd]

On 12/12/2016 01:11 AM, Alan Bateman wrote:

On 11/12/2016 22:38, David M. Lloyd wrote:



I'm not implementing SelectorProvider, I just want access to all the
possible implementations that are available in the current JDK so we
can select among them on a case-by-case basis.


Are you attempting
to wrap the default implementation - I ask because there isn't anything
in the service type (SelectorProvider in this example) that is useful
for doing selection.


Except for the name of the class, which is what we select by today.

At least in JDK 9 then the only platform in OpenJDK (or Oracle JDK
builds) where there is more than one implementation is Solaris (it has a
/dev/poll and Solaris I/O ports based implementations).  Maybe XNIO wants
to use the legacy poll based Selector on Mondays? It will be out of luck
in JDK 9 as that implementation is excluded by the build (it wasn't
deleted because some people wanted to keep it around for bootstrapping
new ports).


That's unfortunate.


So I will guess that SelectorProvider::provider gets you
most of the way. Maybe this request is rooted in the proposal to add a
poll method to SocketChannel that didn't come to a timely conclusion?


Exactly, and also as a last-ditch option where we can change to 
PollSelectorProvider when things aren't working right.



In any case, what you are asking for would require a spec update.
Specifically SelectorProvider::provider would need to specify that it
ignores "built-in" implementations that are located via ServiceLoader.
It would need an implementation changes too. Nothing too difficult of
course but it just not something that has ever come up before (at least
to my knowledge).


OK, I guess we'll just deal with it.

--
- DML

Re: Module exports for java.base: NIO stuff

[David M. Lloyd]

On 12/11/2016 05:18 AM, Alan Bateman wrote:

On 10/12/2016 15:21, David M. Lloyd wrote:


Would it be possible to have all available selector provider
implementation classes listed in a "provides java.nio.channels.spi
with ..." section of java.base's module descriptor?  My use case is as
follows:

Our I/O library (XNIO) relies on the ability to detect and use
different available selector providers for different purposes in
different circumstances (often to provide alternatives in the event of
platform-specific behavior problems, or to prefer lighter providers
over heavier ones in certain situations).  Right now we directly use
reflection to seek out specific named classes. However, it would be
much better if we could instead use a service loader to discover all
available implementations, which would free us from having to use
reflection for this purpose and also avoid lots of pointless probing.

Today you need to have --add-exports=java.base/sun.nio.ch=xxx which is
definitely not ideal.

Maybe there's another solution to this problem as well; suggestions
welcome.


I don't think we've ever come across alternative implementations of
SelectorProvider, are these complete implementations?


I'm not implementing SelectorProvider, I just want access to all the 
possible implementations that are available in the current JDK so we can 
select among them on a case-by-case basis.



Are you attempting
to wrap the default implementation - I ask because there isn't anything
in the service type (SelectorProvider in this example) that is useful
for doing selection.


Except for the name of the class, which is what we select by today.
--
- DML

Module exports for java.base: NIO stuff

[David M. Lloyd]

Would it be possible to have all available selector provider 
implementation classes listed in a "provides java.nio.channels.spi with 
..." section of java.base's module descriptor?  My use case is as follows:


Our I/O library (XNIO) relies on the ability to detect and use different 
available selector providers for different purposes in different 
circumstances (often to provide alternatives in the event of 
platform-specific behavior problems, or to prefer lighter providers over 
heavier ones in certain situations).  Right now we directly use 
reflection to seek out specific named classes.  However, it would be 
much better if we could instead use a service loader to discover all 
available implementations, which would free us from having to use 
reflection for this purpose and also avoid lots of pointless probing.


Today you need to have --add-exports=java.base/sun.nio.ch=xxx which is 
definitely not ideal.


Maybe there's another solution to this problem as well; suggestions welcome.

--
- DML

Re: uses is useless ?

[David M. Lloyd]

On 11/16/2016 08:06 AM, Alan Bateman wrote:

On 16/11/2016 14:00, David M. Lloyd wrote:



That's really a usability regression though, isn't it?  In the past, I
could always use ServiceLoader with a ClassLoader argument to specify
on whose behalf I'm loading.  In fact we do this quite a lot today.


No change to existing code on the class path, the question only comes up
when migrating to an explicit module (meaning you are writing a module
declaration). It's a reasonable question to ask of course.


Of course, but assuming one ports existing code to be modular and expect 
to live in a modular environment, it's still perfectly reasonable to 
want to load services on behalf of another module.  I disagree with 
using Lookup for this purpose, but a simple method (even on Module 
itself) to create a service loader from that module's perspective would 
be very useful for this.  In fact we do this in JBoss Modules.

--
- DML

Re: uses is useless ?

[David M. Lloyd]

On 11/16/2016 04:44 AM, Alan Bateman wrote:

On 16/11/2016 10:25, fo...@univ-mlv.fr wrote:


:
The whole point of uses is to enable a kind of static analysis on the
service dependencies,
if 'uses' is not required, if you can easily bypass it or it doesn't
provide useful information,
then it should be removed from the spec.

`uses` is used to link the consumers of services to the providers of
services. If nobody `uses S` then the modules that `provides S with ...`
will not be resolved (assuming nobody directly depends on these service
provider modules).

The scenario where a framework uses ServiceLoader on behalf of a
consumer really needs the consumer to declare that it uses the service
type, otherwise there is no guarantee that the modules that provide
implementation of the service will be resolved.


That's really a usability regression though, isn't it?  In the past, I 
could always use ServiceLoader with a ClassLoader argument to specify on 
whose behalf I'm loading.  In fact we do this quite a lot today.


--
- DML

Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages

[David M. Lloyd]

On 11/01/2016 11:02 AM, Andrew Dinn wrote:

On 01/11/16 15:35, David M. Lloyd wrote:

On 11/01/2016 10:09 AM, Andrew Dinn wrote:

There is a very easy way to provide tightly controlled access to a
framework. Export access to e.g. jdk.internal.misc.Unsafe or e.g.
java.lang.[invoke].MethodHandles to a nominated module provided by your
framework then arrange for that module to hand out Lookups (or better
MethodHandles) to framework code as and when they are needed via a
private channel. You can do arrange that with a single addExports option
on the command line and a very small amount of setup code to establish
the private channel


I don't see how this is safer though.  You're proposing to hand all the
keys (or none) to modules which don't need all the keys, whereas I
propose that a module should only gain incremental additional access on
a grant basis.


No, I'm suggesting handing the keys to a single module provided by your
framework -- which module will then use a key to open a room on demand
for your code and your code only. That means the privilege can be
granted using one command line switch with use of that privilege
controlled by a runtime mechanism of your choosing.


Yes, however that means that I'm on the hook to make sure that the keys 
aren't mislaid.  And given the fact of open modules - which already 
semantically grants me privileges - requiring a special command line 
switch (which is itself undesirable) and custom code to enable those 
privileges in the *preferred* manner (i.e. MethodHandles) seems 
backwards when open modules would already grant me the ability to 
utilize a much finer instrument albeit with the legacy approach (i.e. 
reflection).  It seems quite logical to extend this mechanism to the 
preferred MethodHandle approach instead, and forget about Unsafe and 
anything like it, which I think is in everyone's best interests in the 
long term.



At least Rémi's approach requires a specific grant and non-Unsafe
vector, though as I said the weaknesses (of his first approach) are that
the grant must come from the target class (programmatically) instead of
being a static declaration ("opens" is a natural fit for this if I
understand the proposal correctly).  The other weakness is that it
requires class init, but that's purely a consequence of requiring a
programmatic grant as far as I understand.


I did actually suggest a way of avoiding the use of Unsafe. You give
your nominated module full reflective access to java.lang.invoke
allowing it to create Lookup instances (it can actually just create a
single all privileges lookup and use this to clone others). You don't
need to insert a class into java.lang.invoke to do this. You simply add
one exports directive on the command line.


That is marginally better, of course, but I think the rest of my 
objections still stand against this argument.



Rémi's newest Gist uses an annotation, which is closer but not quite on
the target IMO.  An annotation cannot be module-deployment-agnostic in
the way that I outlined previously (hence
#IndirectQualifiedReflectiveAccess); ultimately accessibility
information ought to come from the module configuration itself.


If you implement a module with the ability to hand out Lookup instances
you can use whatever control model you like. For example, you might park
an instance which hands out Lookups in a location only available to your
framework or ensure that any caller asking for a Lookup belongs in a
package implemented by your framework or whatever you want. the point is
that once you have granted a class in our code the ability to create
Lookups you can implement whatever runtime access control you want.
Clearly,as John said, you need to do that responsibly.

regards,


Andrew Dinn
---
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander



--
- DML

Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages

[David M. Lloyd]

On 11/01/2016 10:09 AM, Andrew Dinn wrote:

On 01/11/16 14:39, David M. Lloyd wrote:

On 11/01/2016 09:23 AM, John Rose wrote:

On Nov 1, 2016, at 10:22 AM, Jochen Theodorou  wrote:


Can we clarify "privileged code"? Privileged like in a
SecurityManager in a PrivilegedAction for example, for privileged
like only jdk internal code? Just to see it black on white ;)


Good question:  I mean the basic JDK platform implementation.
Something deep in java.base.  Like Unsafe.


I don't see why this can't be a "regular" API though, rather than a
super-user sledgehammer every single time.  If user code can be
statically granted access, and that user code can deliberately acquire a
narrowly-scoped object which can access those Lookups/*Handles, then
isn't that better than using Unsafe, which not only represents
unrestricted system-wide access, but can undermine even the JVM's
integrity if leaked?


If making this a 'regular' API means allowing any code to use it then
that clearly undermines /all/ the access restrictions provided by
modules. The point is not to unlock all doors in the castle but to open
doors for trusted code when needed (and only then).


Maybe I wasn't clear; I was suggesting that any code be able to use the 
API, but only to access things which were explicitly granted.  I'm 
suggesting *less* access, not *more*.


Saying "suitably privileged" and "Unsafe" seems sufficient until you 
realize that many many frameworks will need this API, which means they 
will need Unsafe, which means they will *get* Unsafe (one way or 
another), which is certainly a far more dangerous thing to have floating 
around than (say) a Lookup that has access to a few specific private 
members.



There is a very easy way to provide tightly controlled access to a
framework. Export access to e.g. jdk.internal.misc.Unsafe or e.g.
java.lang.[invoke].MethodHandles to a nominated module provided by your
framework then arrange for that module to hand out Lookups (or better
MethodHandles) to framework code as and when they are needed via a
private channel. You can do arrange that with a single addExports option
on the command line and a very small amount of setup code to establish
the private channel


I don't see how this is safer though.  You're proposing to hand all the 
keys (or none) to modules which don't need all the keys, whereas I 
propose that a module should only gain incremental additional access on 
a grant basis.


At least Rémi's approach requires a specific grant and non-Unsafe 
vector, though as I said the weaknesses (of his first approach) are that 
the grant must come from the target class (programmatically) instead of 
being a static declaration ("opens" is a natural fit for this if I 
understand the proposal correctly).  The other weakness is that it 
requires class init, but that's purely a consequence of requiring a 
programmatic grant as far as I understand.


Rémi's newest Gist uses an annotation, which is closer but not quite on 
the target IMO.  An annotation cannot be module-deployment-agnostic in 
the way that I outlined previously (hence 
#IndirectQualifiedReflectiveAccess); ultimately accessibility 
information ought to come from the module configuration itself.


--
- DML

Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages

[David M. Lloyd]

On 11/01/2016 09:23 AM, John Rose wrote:

On Nov 1, 2016, at 10:22 AM, Jochen Theodorou  wrote:


Can we clarify "privileged code"? Privileged like in a SecurityManager in a 
PrivilegedAction for example, for privileged like only jdk internal code? Just to see it 
black on white ;)


Good question:  I mean the basic JDK platform implementation.  Something deep 
in java.base.  Like Unsafe.


I don't see why this can't be a "regular" API though, rather than a 
super-user sledgehammer every single time.  If user code can be 
statically granted access, and that user code can deliberately acquire a 
narrowly-scoped object which can access those Lookups/*Handles, then 
isn't that better than using Unsafe, which not only represents 
unrestricted system-wide access, but can undermine even the JVM's 
integrity if leaked?


--
- DML

Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages

[David M. Lloyd]

I just want to point out the two obvious (closely related) major 
problems with this approach which prevent it from being a practical 
replacement for setAcceptable, lest they are forgotten/ignored in the 
excitement around the new ideas:


1. It requires the target class to be initialized
2. It requires the target class to proactively donate MethodHandles or a 
Lookup to the lookup class


Both of these are effectively pretty substantial usability regressions 
compared to the status quo.  These problems are why I'd rather see some 
kind of static grant mechanism, and either a special static API to get 
an extra-privileged Lookup or the appropriate magic to allow a class's 
"normal" Lookup object to "see" everything that was granted to it; but 
really any idea which would solve both of these problems in any way 
should be put forward for consideration IMO.


On 10/31/2016 06:57 PM, Remi Forax wrote:

Hi Paul,
I think you can already sketch an implementation of the proposal now by using 
either an annotation processor + a jlink plugin, or by a jlink plugin + a 
bytecode processor tool.
I believe that you can push the information needed by the meta-framework to 
fulfill requests by adding some method calls inside the static initializer of 
the classes that are annotated.
The runtime part can be something along that line

https://gist.github.com/forax/1511fae2273f04273ff9463c6fbbdfbc

Instead of providing method handles as John suggests, i provide the lookup 
object which is less secure.
And the Class object in the map are stored as strong ref instead of being an 
ephemeron (the lambda used as value in the map has also a strong reference on 
the class).

Rémi

- Mail original -

De: "Paul Bakker" 
À: jigsaw-dev@openjdk.java.net
Envoyé: Lundi 31 Octobre 2016 18:06:27
Objet: Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open   modules 
& open packages



The proposal looks very good! Thanks for listening to the voiced concerns.
When can we expect a new prototype containing the proposed features?

Thanks,

Paul

On Fri, Oct 28, 2016 at 3:38 PM John Rose  wrote:


On Oct 27, 2016, at 9:07 AM, mark.reinh...@oracle.com wrote:


Further comments most welcome, as usual!



+100 for *qualified* opens; this puts an important limit on deep
reflection.

My main concern with reflection is to avoid "falling off the encapsulation
cliff"
the first time a user module wishes to open itself up for deep reflection.
By such a "cliff" I mean that when deep reflection is allowed, potentially
any name in the module can be inspected.  Even of only 0.01% of names
are actually inspected, tools for reorganizing code must assume that 100%
of the names *might* be inspected, at some point in the future, by the
allowed party.

So qualified opens is a partial solution.  And, I think it is exactly
right for
today, because it can be extended tomorrow.  I.e., it is future-friendly.
I'll explain by sketching a concept called "moderate reflection", or "MR".

A fuller solution would allow other tools to make conclusions about more
specific limitations on the actual extent of the deep reflection.   Call
such
an extent-limited reflection moderate reflection, where "moderation" is
declared statically by some sort of flexible declarative rule set.  In
this way
tools could make more detailed conclusions about the encapsulation of
particular names in the module.

 For future JDK releases we can consider layering  of this qualified opens,
by creating a trusted, parameterized meta-framework ("Moderate Reflection")
to which a user module can be qualified-open.  This meta-framework can
then perform deep reflection on behalf of self-describing "moderately
reflecting"
client frameworks.  User modules would not directly open themselves to any
module other than MR.

The advantage would come when the client frameworks explicitly declare
their
self-restraint ("moderation") in reflection.  For example, the "Foo"
framework
might limit itself to names annotated with @FooFrameworkHook.

An AOT compiler (or other jlink-time reorganization tool) would (1) note
that
some user module bar is qualified-open only to Moderate Reflection, and
has an export (maybe qualified) to a Foo framework.  At runtime, (2) the
Foo framework would ask Moderate Reflection to reflect into bar.
Moderate Reflection would (3) check the self-limitation on Foo, and also
ensure that bar exports (though doesn't open) to Foo.  MR would then
gain the requested access and delegate it back to Foo.  Finally, (4) the
AOT compiler would free itself to omit metadata for unexported names
not visible to moderate reflection.

(Method handles are a good way to grant access, since they can
be delegated from MR to Foo without re-authorizing Foo.  Core
Reflection re-authorizes on every access, which awkwardly
requires deep reflection on usage, as well as initial lookup.)

Moderate Reflection might also look at metadata on the bar module
(besides its module exports) to impose further limits on 

Re: Request Review: JDK-6479237 (cl) Add support for classloader names

[David M. Lloyd]

On 10/26/2016 10:43 AM, David M. Lloyd wrote:

On 10/26/2016 10:39 AM, Alan Bateman wrote:

On 26/10/2016 16:28, David M. Lloyd wrote:


:

Some background is in order I guess.  I've been reworking a couple of
our projects to support both Java 9 and Java 8, using MR JARs and also
without MR JARs.  In the MR JAR case it's been easiest (so far) to
push off the JDK-version-specific stuff into a nonpublic "JDKSpecific"
class which gets replaced under Java 9. However, because the
ClassLoader change requires a new constructor to be called, I actually
have to put the whole ClassLoader implementation into the MR
supplement, which is pretty clunky at best and might introduce some
tricky build difficulties at worst.

Can you introduce an intermediate ClassLoader into the type hierarchy?
The 8 version will just invoke super(), the 9+ version will invoke
super(name). That would keep the JDK-specific code to a minimum.


Just wanted to drop a note an say this approach worked perfectly, of 
course.  Thanks for the tip.

--
- DML

Re: Request Review: JDK-6479237 (cl) Add support for classloader names

[David M. Lloyd]

On 10/27/2016 10:02 PM, Mandy Chung wrote:



On Oct 27, 2016, at 8:04 AM, David M. Lloyd  wrote:

OK Thanks.  I was looking at the StackTraceElement#toString JavaDoc and it was 
not very clear if this was the expected output; maybe it's worth pointing out 
explicitly.


http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/javadoc/java/lang/StackTraceElement.html

Is this verion better?


Perfect, thanks.
--
- DML

Re: Request Review: JDK-6479237 (cl) Add support for classloader names

[David M. Lloyd]

On 10/27/2016 09:59 AM, Mandy Chung wrote:



On Oct 27, 2016, at 7:30 AM, David M. Lloyd  wrote:

It looks like if you have a class loader name but not a module (i.e. it's 
unnamed), you get output that looks like this:

   [...]
   at 
org.jboss.as.controller//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:390)
   [...]

In this case the classloader's name is "org.jboss.as.controller".  Is the double-slash 
"//" intended?



Yes.  The format is // and ifthe class 
loader is named but module is unnamed, it will drop .  This can tell if the module is 
named or unnamed from the output.

See the javadoc in StackTraceElement::toLoaderModuleClassName.  I will double 
check if the example in StackTraceElement::toString includes this.


OK Thanks.  I was looking at the StackTraceElement#toString JavaDoc and 
it was not very clear if this was the expected output; maybe it's worth 
pointing out explicitly.

--
- DML

Re: Request Review: JDK-6479237 (cl) Add support for classloader names

[David M. Lloyd]

It looks like if you have a class loader name but not a module (i.e. 
it's unnamed), you get output that looks like this:


[...]
at 
org.jboss.as.controller//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:390)

[...]

In this case the classloader's name is "org.jboss.as.controller".  Is 
the double-slash "//" intended?


On 10/25/2016 06:10 PM, Mandy Chung wrote:

Webrev at:
   http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.00/

Specdiff:
   
http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/specdiff/overview-summary.html

This is a long-standing RFE for adding support for class
loader names.  It's #ClassLoaderNames on JSR 376 issue
list where the proposal [1] has been implemented in jake
for some time.  This patch brings this change to jdk9.

A short summary:
- New constructors are added in ClassLoader, SecureClassLoader
  and URLClassLoader to specify the class loader name.

- New ClassLoader::getName and StackTraceElement::getClassLoaderName
  method

- StackTraceElement::toString is updated to include the name
  of the class loader and module of that frame in this format:
 //(:)

The detail is in StackTraceElement::buildLoaderModuleClassName
that compress the output string for cases when the loader
has no name or the module is unnamed module.  Another thing
to mention is that VM sets the Class object when filling in
a stack trace of a Throwable object.  Then the library will
build a String from the Class object for serialization purpose.

Mandy
[1] 
http://mail.openjdk.java.net/pipermail/jpms-spec-observers/2016-September/000550.html



--
- DML

Re: Request Review: JDK-6479237 (cl) Add support for classloader names

[David M. Lloyd]

On 10/26/2016 10:39 AM, Alan Bateman wrote:

On 26/10/2016 16:28, David M. Lloyd wrote:


:

Some background is in order I guess.  I've been reworking a couple of
our projects to support both Java 9 and Java 8, using MR JARs and also
without MR JARs.  In the MR JAR case it's been easiest (so far) to
push off the JDK-version-specific stuff into a nonpublic "JDKSpecific"
class which gets replaced under Java 9. However, because the
ClassLoader change requires a new constructor to be called, I actually
have to put the whole ClassLoader implementation into the MR
supplement, which is pretty clunky at best and might introduce some
tricky build difficulties at worst.

Can you introduce an intermediate ClassLoader into the type hierarchy?
The 8 version will just invoke super(), the 9+ version will invoke
super(name). That would keep the JDK-specific code to a minimum.


Hmm, that's an idea.  I'll give it a shot, thanks.

--
- DML

Re: Request Review: JDK-6479237 (cl) Add support for classloader names

[David M. Lloyd]

On 10/26/2016 08:52 AM, Alan Bateman wrote:

On 26/10/2016 14:28, David M. Lloyd wrote:



I am not a reviewer.  I have a question though: Does this mean that
you have to establish the name of your class loader in the constructor?

Yes, it's named at creation time. The VM uses the name when filling in
the stack trace, it's not invoking getName to execute arbitrary code
when doing that.


I was afraid that might be the case.  I guess it is impractical 
(performance-wise) to use a StackWalker to do this in Java-space 
instead?  The advantage being that in that case you can just retrieve 
the class loader (and thus its name) off of the class of each frame.



If so, here's what mainly bothers me about this approach:

- With a new constructor being required, you cannot easily write a
class loader which both functions under Java 8 and has a name under
Java 9 without resorting to MR JARs
- The getName() method can be overridden but doing so has no effect on
the real name of the class loader
- ... but you can't make it final or you risk breaking class loaders
that already exist

Maybe instead, the "name" field (or other internal mechanism) could
somehow be lazily populated by calling getName(), which returns null
by default but can be overridden by subclasses?  This way ClassLoaders
can function the same way in Java 8 and 9, and just get extra
functionality when running under Java 9

Ideally getName would be final but it can't for the reason you noted
(there is still the risk that someone has a custom class loader with a
no-arg getName method that is not public or has a different return type
of course).


Yeah, that's not great.  Could be mitigated by having a static 
ClassLoader.getNameOf(classLoader) method I guess, though this doesn't 
really help the concern I already stated (just the opposite in fact)...


Some background is in order I guess.  I've been reworking a couple of 
our projects to support both Java 9 and Java 8, using MR JARs and also 
without MR JARs.  In the MR JAR case it's been easiest (so far) to push 
off the JDK-version-specific stuff into a nonpublic "JDKSpecific" class 
which gets replaced under Java 9.  However, because the ClassLoader 
change requires a new constructor to be called, I actually have to put 
the whole ClassLoader implementation into the MR supplement, which is 
pretty clunky at best and might introduce some tricky build difficulties 
at worst.


Basically I'm hoping there could be some solution which doesn't involve 
requiring a new constructor to be called, so I can avoid having to do this.

--
- DML

Re: Request Review: JDK-6479237 (cl) Add support for classloader names

[David M. Lloyd]

On 10/25/2016 06:10 PM, Mandy Chung wrote:

Webrev at:
   http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.00/

Specdiff:
   
http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/specdiff/overview-summary.html

This is a long-standing RFE for adding support for class
loader names.  It's #ClassLoaderNames on JSR 376 issue
list where the proposal [1] has been implemented in jake
for some time.  This patch brings this change to jdk9.

A short summary:
- New constructors are added in ClassLoader, SecureClassLoader
  and URLClassLoader to specify the class loader name.

- New ClassLoader::getName and StackTraceElement::getClassLoaderName
  method

- StackTraceElement::toString is updated to include the name
  of the class loader and module of that frame in this format:
 //(:)

The detail is in StackTraceElement::buildLoaderModuleClassName
that compress the output string for cases when the loader
has no name or the module is unnamed module.  Another thing
to mention is that VM sets the Class object when filling in
a stack trace of a Throwable object.  Then the library will
build a String from the Class object for serialization purpose.


I am not a reviewer.  I have a question though: Does this mean that you 
have to establish the name of your class loader in the constructor?


If so, here's what mainly bothers me about this approach:

- With a new constructor being required, you cannot easily write a class 
loader which both functions under Java 8 and has a name under Java 9 
without resorting to MR JARs
- The getName() method can be overridden but doing so has no effect on 
the real name of the class loader
- ... but you can't make it final or you risk breaking class loaders 
that already exist


Maybe instead, the "name" field (or other internal mechanism) could 
somehow be lazily populated by calling getName(), which returns null by 
default but can be overridden by subclasses?  This way ClassLoaders can 
function the same way in Java 8 and 9, and just get extra functionality 
when running under Java 9.


--
- DML

Re: Sharing experiences with the latest EA build // #ReflectiveAccessToNonExportedTypes #ResourceEncapsulation

[David M. Lloyd]

On 10/03/2016 10:15 AM, Rafael Winterhalter wrote:

[...] However, for the adoption process, I think it is
crucial that non-modularized code behaves as it did before, i.e. that
non-modularized code shows the same behavior when its run on a Java 9 VM.


I agree with the sentiment here however I disagree with the conclusion. 
I believe it is critical that existing code can be brought into modules 
without having to first rewrite all reflection/internal references/etc. 
These things should all be coverable by (a minimal set of) specific 
declarations to allow applications to be modularized even if they have 
"legacy"-flavored problems.  Otherwise old code will be in a permanent 
bit-rotted state which it will be hard to escape from - especially large 
intertwined code bases.


--
- DML

Re: delegating module access

[David M. Lloyd]

Sanne, weak modules already allow this, but the weak module mechanism is 
not adequate, as has been discussed internally.


The question of whether restricting reflection is useful or not is a 
separate discussion.  Our position is that it is a solution for a 
problem that can be solved in other ways, perhaps more effectively, but 
that is left aside for the purposes of the discussion around the 
#ReflectiveAccessToNonExportedTypes and #AwkwardStrongEncapsulation issues.


If you accept the proposition that restricting reflection is going to 
happen, then it follows from that that (a) modules must be able to 
carefully control who has access, and (b) modules that do have access 
need a way to utilize it.


However it is also important that modules which grant access have a 
simple, non-verbose way to do so.  It is also important that exiting 
code should not have to jump through extra hoops to gain the same access 
they have always had - the module declarations should suffice for this 
purpose.


On 09/30/2016 09:11 AM, Sanne Grinovero wrote:

Hi all,
I think what Peter raises is a very interesting quandary.

Without going in the syntax details of this proposal, the point is
that implementations of "javax.persistence" (and similar APIs) need to
advertise themselves as eligible to getting certain additional
privileges - such as reflective access - which the platform is
attempting to limit (why?).

The problem I see is that it should be fairly easy for any library to
fool the platform to obtain the same privileges; I could certainly
extend Hibernate and make a new JPA provider in just minutes, and do
something else with those additional privileges as well.
This might be a clumsy trick, but sufficiently motivated people - such
as those having malicious intents - could certainly entertain such
plans.
While well intentioned developers who have a legitimate reason to need
reflective access - like plans to make some better framework ? - would
need to write odd looking, time wasting code to get their work done.

Wouldn't it be far easier to just allow reflective access to everyone
without restrictions, at least on weak modules? I'm not seeing the
benefits of preventing this - especially to people using the
Reflection API.

Thanks,
Sanne


On Thu, Sep 29, 2016 at 5:06 PM, Peter Levart  wrote:

Hi,

Building on the idea presented here (the part that talks about method
''assumeAccessOf'):

http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009520.html

...which tries to solve the problem that some modules have when they don't
know in advance the name of the module(s) to which to export its packages
for reflective access and they just want to export them to the selected
module(s) - i.e. use qualified exports.

The canonical example here is an application module, say "app", that is
coded against an API module, say "javax.persistence" which just defines the
API, but at link or assembly time, the module that implements such API, say
"hibernate-core" is added which needs access to "app" classes using
reflection.

The API module ("javax.persistence" in this case) usually also contains a
bootstrap (factory) class, which is responsible to bootstrap the API and
bind it to the implementation. In case of javax.persistence it is
javax.persistence.Persistence class with a static createEntityManagerFactory
method, which finds and instantiates an EntityManagerFactory implementation.
This point is an opportunity for javax.persistence module to delegate its
permissions to access classes in other modules to the module of the
implementation class:

public static EntityManagerFactory createEntityManagerFactory(String
persistenceUnitName, Map properties) {

Class emfClass = ...;

Persistence.class.getModule().delegateAccessTo(emfClass.getModule());

EntityManagerFactory emf = emfClass.newInstance();
...

return emf;
}


So for example, app module could export its entity package(s) to
javax.persistence module only:

module app {
requires javax.persistence;
exports private app.entity to javax.persistence;
}


And the ability to access this package would be transferred to the
implementation module by the javax.persistence module. The implementation
module is in fact just an extension of the API module and the API delegates
to it or is extended by it.

java.lang.Module instance method:

public void delegateAccessTo(Module module) { ... }

...would be similar to addExports or addReads. It would only be possible to
call it from 'this' module (i.e. a module could only delegate access given
to it - not access given to any other module).

When SecurityManager is present, then there would additionally have to be a
permission assigned to codesource of javax.persistence.Persistence class for
it to carry on the delegation.


I think this is safe and sound, and most importantly, self-managing.

What do you think?


Regards, Peter



--
- DML

Solving #ConcealedPackageConflicts

[David M. Lloyd]

By my analysis, the only possible and useful solution for 
#ConcealedPackageConflicts, other than rewriting how class loaders work, 
is to map non-automatic application (i.e. non-JDK) modules in the 
default layer each to their own class loader.  This also potentially 
solves (or provides to means to solve) #MultipleModuleVersions - if you 
take "multiple versions" to be defined as "two modules with the same 
package name", a definition which Red Hat rejects.


We (at Red Hat) know from years of experience that libraries and 
software in the wild generally won't have a problem with this from a 
compatibility perspective.  The widespread adoption of OSGi also proves 
this out, as many applications of OSGi include existing third-party 
libraries.


So, why not?
--
- DML

Re: Alternative mechanism for reflective access control (#ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation)

[David M. Lloyd]

On 09/29/2016 07:31 AM, Peter Levart wrote:

Hi David,

I'm not the one to answer your queries, but may I ask you some questions
about your requirements and also give some comments...

On 09/28/2016 02:13 PM, David M. Lloyd wrote:

Hi all, I've been requested to ask if the OpenJDK development team
have had a chance to review this email, and when we might expect a
response.

Thanks!

On 09/21/2016 11:39 AM, David M. Lloyd wrote:

In our internal discussion of the proposal for
#ReflectiveAccessToNonExportedTypes, we discussed the ins and outs of
various behaviors and have come up with a few ideas or starting points
for solutions that we think would be more workable in conjunction with
existing middleware (ours and others').

For reasons previously explained, we do not think that weak modules are
a good way forward; I won't go into that again here.  But the logical
re-starting point is: If not weak modules, then what?

I will boil it down to a few basic requirements that we have
established.  This list is probably non-exhaustive but hopefully
complete enough to go on for now:

• A module definition must be able to establish that a dependent has (or
all modules have) access to one or more (or all) packages for public
reflection only.
• A module definition must be able to establish that a dependent has (or
all modules have) access to one or more (or all) packages for public or
private reflection only.


Why do you think is important to have a mode that allows reflective
access but disallows compilation/linkage? Because frameworks that use
reflection usually have ability to dynamically adjust to changes of code
that is exposed in that way while compilation/linkage hard-codes access
to the code and therefore breaks when such code is changed?


No, not at all.  It's just that the two concerns (linkage versus 
reflection access permission) are not really coupled.


For example if I have a library that uses reflection to examine 
arbitrary objects that are handed to it, I definitely don't need to (or 
want to) depend on the object class's source module or package for 
linkage purposes.  The code of the given object class might be non-API 
for example, and I may not want to give a permanent compatibility 
guarantee to my consumers, thus I do not want to export it.



I have a concern that this will not prevent breakages. If reflective
access is allowed, you can never be sure whether someone might be using
it and just hard-coding such access in the consuming module. So while
not providing "easy" hard-coded access to such code, it is still
provided, just discouraged by making it more awkward.


Of course not; anyway in this scenario the deployer could als simply 
modify the descriptor to suit their needs.  However it's obvious that in 
doing these things that breakage may occur if the implementation changes.


There is a lot of focus on the security aspect of encapsulation, but 
using encapsulation also encourages clean engineering by presenting 
clear contracts (much like public and non-public types do today). 
Though this is no proof against complaint in the event of breakage - 
especially where the JDK is concerned - out "in the wild" of real-world 
applications and frameworks, these mechanisms really work.



So I'm wondering if there might be an alternative way of "discouraging"
hard coded access without complicating the module system with this
aspect. A standard compile-time TYPE, PACKAGE or even MODULE annotation
comes to mind that would trigger compilation warning if such annotated
type or type in annotated package or type in annotated module would be
referenced in compiled code. A warning can always be turned into an
error for the conscious. I think this could even be implemented as an
annotation processor, outside of javac. There just has to be a standard
annoation to agree upon. What do you think?


It seems pretty weird to have a package be exported so you can use it 
*and* annotated that you *shouldn't* use it.  It seems cleanest to 
export what you want exported, grant what you want granted, and leave 
the rest closed off.  And this talks about javac/compile time, but it is 
our position that in the real world, this is completely irrelevant as 
build tools can and will do whatever they need to do to access what they 
want to access, with complete disregard to what is packaged in a 
descriptor.  The descriptors are more likely to be assembled by a 
distributor or deployer than to have a module developer drop a module 
which is expected to work in any module environment (which requires, 
among other things, completely universal agreement on module namespacing 
- something that has been shown time and again to be an unrealistic 
expectation).


Therefore all these requirements are from the perspective of load-link 
time to run time.



• A module definition must be able to establish that a dependent has (or
all modules hav

Re: Alternative mechanism for reflective access control (#ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation)

[David M. Lloyd]

On 09/28/2016 07:01 AM, Alan Bateman wrote:

On 28/09/2016 10:37, Gunnar Morling wrote:


:

I don't think that is what those folks asking about using the SM had
in mind.

Rather, the idea would be - IIUC - to grant code in module B (say an
ORM tool) reflective access to non-exported (and of course exported)
types in module A (say a module with entities) by default. If needed,
the code in B would then use setAccessible() for making private
members in A accessible before invoking them, allowing it to obtain
the entity state. This is where the SM is part of the picture: if in a
given environment the user wants to restrict who may call
setAccessible(), they could use the SM for it, with a fine grained
policy, e.g. allowing the call only to the ORM module.

I.e. ReflectiveAccessToNonExportedTypes would be granted by default,
only if an SM is enabled at runtime, actions to suppress accessibility
checks would be subject to the current security policy.

This is not to say that I'm behind this idea, I just felt it hasn't
been discussed in the way it may have been meant.

I can see though how JDK devs and library authors want to prevent
access to private code in their modules at all, hence relying on the
SM to be enabled at runtime is not an option to address that requirement.


The original question mentioned "reflective access" and we've said, has
always been specified to do the same access checks as the Java Language
and VM.

On setAccessible, then it wouldn't be hard to have setAccessible
implement fine grain permission checks but not clear how useful this is
given that the SM is opt-in and not widely used. Having setAccessible
ignore module boundaries (assume no SM) goes against the goal of strong
encapsulation and of course makes #AwkwardStrongEncapsulation a lot worse.

In general then I think that we need to find ways to reduce the use of
setAccessible over time. We really need a long term plan to degrade,
deprecate and eventually remove it. This would of course mean working on
some challenging problems and use-cases.

So what would the alternative to setAccessible be? It would be nice if
the framework libraries (you mention ORM tools and maybe the Hibernate
ORM devs could be the guinea pig) would start to make use of the "new
reflection API" that is java.lang.invoke. So rather than bypassing
access checks with legacy core reflection then they would instead use
MH.Lookup objects as capabilities. Think code in a consumer module
creating a Lookup object with the appropriate lookup class + mode that
it cooperatively hands to the framework. This puts the access check in
the consumer module so that the frameworks don't need to break in. This
direction isn't without challenges of course as there may be
initialization issues to deal with. It might, for example, involve
injecting helper code into a submissive consumer module when there isn't
explicit initialization. John Rose has good write-ups in JIRA with ideas
in this area.


Using Lookups are very likely to be an essential piece to solving the 
security conundrum.  It is not without its problems though at present.


Bytecode injection in particular only works in a container setting, for 
example, and only for wholly managed modules.  Relying on cooperative 
code is also a comparatively onerous burden to place on regular 
application developers, and of course few if any existing frameworks and 
standards are set up in this way so it doesn't help the migration story any.


More work definitely needs to be done in this area before we can really 
say that it's a viable alternative to setAccessible().

--
- DML

Re: Alternative mechanism for reflective access control (#ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation)

[David M. Lloyd]

Hi all, I've been requested to ask if the OpenJDK development team have 
had a chance to review this email, and when we might expect a response.


Thanks!

On 09/21/2016 11:39 AM, David M. Lloyd wrote:

In our internal discussion of the proposal for
#ReflectiveAccessToNonExportedTypes, we discussed the ins and outs of
various behaviors and have come up with a few ideas or starting points
for solutions that we think would be more workable in conjunction with
existing middleware (ours and others').

For reasons previously explained, we do not think that weak modules are
a good way forward; I won't go into that again here.  But the logical
re-starting point is: If not weak modules, then what?

I will boil it down to a few basic requirements that we have
established.  This list is probably non-exhaustive but hopefully
complete enough to go on for now:

• A module definition must be able to establish that a dependent has (or
all modules have) access to one or more (or all) packages for public
reflection only.
• A module definition must be able to establish that a dependent has (or
all modules have) access to one or more (or all) packages for public or
private reflection only.
• A module definition must be able to establish that a dependent has (or
all modules have) access to one or more (or all) packages for public
reflection and compilation/linkage (i.e. it's an export by today's
terminology).
• A module definition must be able to establish that a dependent has (or
all modules have) access to one or more (or all) packages for public or
private reflection and compilation/linkage (i.e. it's a "private" export
by today's terminology).
• As today, any packages not declared in one or more of the above
categories is inaccessible outside of the module in any way (note that
as I showed previously we have also concluded that it should continue to
be impossible to export a package for compilation/linkage without public
reflection, as we have not discovered any use for such a mode).

More generally:

• The syntax for all of the above has no particular constraint (in fact
I will try to actively avoid touching what could be a very
bikeshedding-rich discussion), except that it should not be construable
as being pejorative against the usage of reflective frameworks; rather,
it should be clear what level of trust is being established without
raising undue warning.
• Applications should not need gratuitous amounts of declarations in
their module(s) in order to utilize frameworks.
• As previously established, it should not be possible for one
declaration to reduce the scope of access of another declaration in a
module definition.
• Access to a module (for reflective purposes only) must not cause
conflicts if multiple such modules which contain identical packages are
accessible to a single consumer; in other words, reflection-only access
into non-dependency modules is not bound by duplicate package
restrictions as long as each package is unique per class loader, as per
the current (Java 8) class loader rules.

The above cover the useful access modes that we have identified.  This
is _nearly_ adequate to cover the use cases that we are currently
concerned about; for example, I could export all packages for public
reflection only to a specific framework, if only I know the module name
of the implementation.

Unfortunately, this does not work well in the case where a module may
consume a framework whose specification is separate from the
implementation.  An application module may need to use (say) EJB and
JPA; there is presently no clean way to do so without either (a) relying
on a container environment to rewrite the descriptor or (b) opening up
the module and defeating the security mechanism (e.g. "weak").  Without
either of these workarounds, the application developer must have a good
deal of knowledge about what modules provide what services within a
framework-rich environment, possibly resulting in a very verbose (and
error-prone) descriptor; none of these options is really satisfactory.

Thus, apart from the option of redesigning (to an extent) the security
mechanism (thereby eliminating the need to seal off access to public
reflection, which is definitely still an attractive option for various
reasons from our perspective, but which is also a very different
discussion), we need some sort of mechanism which decouples the literal
dependency system from access permission (much like uses/provides does).

For example if I could declare that my module uses "javax.ejb", and, in
so doing, automatically grants public and private reflective access to
the module that provides that service, this would be a good outcome.  A
module which answers to that service name could be responsible for
reflective access to the application module, providing that information
privately to any other framework modules which require it.

The migration story looks much better in thi

Re: Alternative mechanism for reflective access control (#ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation)

[David M. Lloyd]

On 09/28/2016 04:37 AM, Gunnar Morling wrote:

 If I'm compiling a class A that has a reference to a member in type B
then do you really want the compiler calling into a security manager to
ask if this access is allowed?


I don't think that is what those folks asking about using the SM had in
mind.

Rather, the idea would be - IIUC - to grant code in module B (say an ORM
tool) reflective access to non-exported (and of course exported) types in
module A (say a module with entities) by default. If needed, the code in B
would then use setAccessible() for making private members in A accessible
before invoking them, allowing it to obtain the entity state. This is where
the SM is part of the picture: if in a given environment the user wants to
restrict who may call setAccessible(), they could use the SM for it, with a
fine grained policy, e.g. allowing the call only to the ORM module.

I.e. ReflectiveAccessToNonExportedTypes would be granted by default, only
if an SM is enabled at runtime, actions to suppress accessibility checks
would be subject to the current security policy.

This is not to say that I'm behind this idea, I just felt it hasn't been
discussed in the way it may have been meant.

I can see though how JDK devs and library authors want to prevent access to
private code in their modules at all, hence relying on the SM to be enabled
at runtime is not an option to address that requirement.


Note that as a security mechanism for untrusted code, this mechanism is 
not going to be adequate no matter what.  The person who has control of 
the deployment environment can always change the module to accommodate 
their case.  I'm sure it will be a very short time after finalization 
before we see module editing tools in the wild to accomplish exactly 
this.  This is an inevitable consequence of bundling the module 
definition within the module itself.


Only the JDK has any real ability to enforce this kind of security, 
though of course it's always possible to defeat that as well (it's just 
potentially a lot more work).


The chief tangible protection which is expected to be afforded by this 
mechanism is against malicious exploitation of trusted code.

--
- DML

Re: Alternative mechanism for reflective access control (#ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation)

[David M. Lloyd]

On 09/28/2016 05:31 AM, Gunnar Morling wrote:

David,


there is too much existing work out there, and the requirements go far

beyond

what can be satisfied by these simple questions.


What are examples for such requirements? I.e. what from the list of your
requirements cannot be mapped to the mechanism suggested by Stephen? I
find his proposal quite appealing and am wondering whether it couldn't
be tweaked to cater for the remaining requirements.


I would rather in this case ask: how does his proposal solve all of our 
requirements?  The burden is always on the implementer to solve 
requirements, not the other way around.



Btw. I second the concern that the exposure of internal parts for
reflective access should not be implicitly enabled by merely depending
on a module requesting such access. It seems too easy for users to
accidentally give access to internal code without being aware of it.


Possibly.  But that's a usability concern and does not change the 
requirement.


For example it's not hard to imagine that someone importing such a 
module has to acknowledge the requested access explicitly (without 
defining the specifics of it) in order for the module to successfully 
link.  Of course that's just one idea to illustrate that the requirement 
can be addressed; it's not a proposal.





2016-09-27 16:30 GMT+02:00 David M. Lloyd mailto:david.ll...@redhat.com>>:

Hi Stephen,

I just want to point out that this isn't a proposal per se.  I don't
advocate any particular syntax, nor recommendation for patterns that
users should use.  This is merely a set of requirements that we (Red
Hat) have identified as necessary in order to enable the widest
range of existing middleware systems to continue to function with a
minimum of disruption or security compromises.  Any proposal that we
accept will necessarily meet these requirements, though these
requirements may or may not be sufficient for acceptance (only
testing will tell us the full story).

The reason I didn't boil it down to the same set of questions that
you have asked is that those questions are simply not sufficient to
meet the requirements of a modern middleware application.  To quote
the JSR description: "This JSR will define an approachable yet
scalable module system [...] so that developers can use it to
construct and maintain libraries and large applications for both the
Java SE and Java EE Platforms. [...]"

I think it's very possible for the platform module system to be
specified and implemented in such a way as to enable the continued
maintenance of existing libraries and large applications, however in
order to do so, it is our estimation that these requirements must be
met.

In response to the way you've narrowed down the scope of
requirements: I agree that such simplifications work well when
establishing boundaries for a new project and encouraging good usage
patterns.  But I don't think we can really do that here; there is
too much existing work out there, and the requirements go far beyond
what can be satisfied by these simple questions.


On 09/26/2016 05:11 AM, Stephen Colebourne wrote:

Having read this proposal a number of times, and considering how the
talks explained things at JavaOne, I have come to the conclusion
that
this proposal is too complex. FWIW, I like the idea that a module
should be able to declare that it needs reflective access from its
users, however given that the proposal is what results from the
idea,
it doesn't seem as appealing as it should.

The reason why I put forward the exports/exposes approach [1] is
that
it keeps the questions that must be asked when creating a module
simple:
- what do I depend on publicly (requires)
- what do I publish publicly (exports)
- what do I publish privately (exposes)

From a security point of view it also seems that it should be the
responsibility of a module to allow the publishing of its private
details, and simply depending on another module seems very minimal
(and easy to miss) as a mechanism to allow that extra permission.

Stephen

[1]

http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html

<http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html>


On 21 September 2016 at 17:39, David M. Lloyd
mailto:david.ll...@redhat.com>> wrote:

In our internal discussion of the proposal for
#ReflectiveAccessToNonExportedTypes, we discussed the ins
and outs of
various behaviors and have come up with a few ideas or
starting points for
solutions that we think would be

Re: Alternative mechanism for reflective access control (#ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation)

[David M. Lloyd]

Hi Stephen,

I just want to point out that this isn't a proposal per se.  I don't 
advocate any particular syntax, nor recommendation for patterns that 
users should use.  This is merely a set of requirements that we (Red 
Hat) have identified as necessary in order to enable the widest range of 
existing middleware systems to continue to function with a minimum of 
disruption or security compromises.  Any proposal that we accept will 
necessarily meet these requirements, though these requirements may or 
may not be sufficient for acceptance (only testing will tell us the full 
story).


The reason I didn't boil it down to the same set of questions that you 
have asked is that those questions are simply not sufficient to meet the 
requirements of a modern middleware application.  To quote the JSR 
description: "This JSR will define an approachable yet scalable module 
system [...] so that developers can use it to construct and maintain 
libraries and large applications for both the Java SE and Java EE 
Platforms. [...]"


I think it's very possible for the platform module system to be 
specified and implemented in such a way as to enable the continued 
maintenance of existing libraries and large applications, however in 
order to do so, it is our estimation that these requirements must be met.


In response to the way you've narrowed down the scope of requirements: I 
agree that such simplifications work well when establishing boundaries 
for a new project and encouraging good usage patterns.  But I don't 
think we can really do that here; there is too much existing work out 
there, and the requirements go far beyond what can be satisfied by these 
simple questions.


On 09/26/2016 05:11 AM, Stephen Colebourne wrote:

Having read this proposal a number of times, and considering how the
talks explained things at JavaOne, I have come to the conclusion that
this proposal is too complex. FWIW, I like the idea that a module
should be able to declare that it needs reflective access from its
users, however given that the proposal is what results from the idea,
it doesn't seem as appealing as it should.

The reason why I put forward the exports/exposes approach [1] is that
it keeps the questions that must be asked when creating a module
simple:
- what do I depend on publicly (requires)
- what do I publish publicly (exports)
- what do I publish privately (exposes)

From a security point of view it also seems that it should be the
responsibility of a module to allow the publishing of its private
details, and simply depending on another module seems very minimal
(and easy to miss) as a mechanism to allow that extra permission.

Stephen

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html


On 21 September 2016 at 17:39, David M. Lloyd  wrote:

In our internal discussion of the proposal for
#ReflectiveAccessToNonExportedTypes, we discussed the ins and outs of
various behaviors and have come up with a few ideas or starting points for
solutions that we think would be more workable in conjunction with existing
middleware (ours and others').

For reasons previously explained, we do not think that weak modules are a
good way forward; I won't go into that again here.  But the logical
re-starting point is: If not weak modules, then what?

I will boil it down to a few basic requirements that we have established.
This list is probably non-exhaustive but hopefully complete enough to go on
for now:

• A module definition must be able to establish that a dependent has (or all
modules have) access to one or more (or all) packages for public reflection
only.
• A module definition must be able to establish that a dependent has (or all
modules have) access to one or more (or all) packages for public or private
reflection only.
• A module definition must be able to establish that a dependent has (or all
modules have) access to one or more (or all) packages for public reflection
and compilation/linkage (i.e. it's an export by today's terminology).
• A module definition must be able to establish that a dependent has (or all
modules have) access to one or more (or all) packages for public or private
reflection and compilation/linkage (i.e. it's a "private" export by today's
terminology).
• As today, any packages not declared in one or more of the above categories
is inaccessible outside of the module in any way (note that as I showed
previously we have also concluded that it should continue to be impossible
to export a package for compilation/linkage without public reflection, as we
have not discovered any use for such a mode).

More generally:

• The syntax for all of the above has no particular constraint (in fact I
will try to actively avoid touching what could be a very bikeshedding-rich
discussion), except that it should not be construable as being pejorative
against the usage of reflective frameworks; rather, it s

Alternative mechanism for reflective access control (#ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation)

[David M. Lloyd]

In our internal discussion of the proposal for 
#ReflectiveAccessToNonExportedTypes, we discussed the ins and outs of 
various behaviors and have come up with a few ideas or starting points 
for solutions that we think would be more workable in conjunction with 
existing middleware (ours and others').


For reasons previously explained, we do not think that weak modules are 
a good way forward; I won't go into that again here.  But the logical 
re-starting point is: If not weak modules, then what?


I will boil it down to a few basic requirements that we have 
established.  This list is probably non-exhaustive but hopefully 
complete enough to go on for now:


• A module definition must be able to establish that a dependent has (or 
all modules have) access to one or more (or all) packages for public 
reflection only.
• A module definition must be able to establish that a dependent has (or 
all modules have) access to one or more (or all) packages for public or 
private reflection only.
• A module definition must be able to establish that a dependent has (or 
all modules have) access to one or more (or all) packages for public 
reflection and compilation/linkage (i.e. it's an export by today's 
terminology).
• A module definition must be able to establish that a dependent has (or 
all modules have) access to one or more (or all) packages for public or 
private reflection and compilation/linkage (i.e. it's a "private" export 
by today's terminology).
• As today, any packages not declared in one or more of the above 
categories is inaccessible outside of the module in any way (note that 
as I showed previously we have also concluded that it should continue to 
be impossible to export a package for compilation/linkage without public 
reflection, as we have not discovered any use for such a mode).


More generally:

• The syntax for all of the above has no particular constraint (in fact 
I will try to actively avoid touching what could be a very 
bikeshedding-rich discussion), except that it should not be construable 
as being pejorative against the usage of reflective frameworks; rather, 
it should be clear what level of trust is being established without 
raising undue warning.
• Applications should not need gratuitous amounts of declarations in 
their module(s) in order to utilize frameworks.
• As previously established, it should not be possible for one 
declaration to reduce the scope of access of another declaration in a 
module definition.
• Access to a module (for reflective purposes only) must not cause 
conflicts if multiple such modules which contain identical packages are 
accessible to a single consumer; in other words, reflection-only access 
into non-dependency modules is not bound by duplicate package 
restrictions as long as each package is unique per class loader, as per 
the current (Java 8) class loader rules.


The above cover the useful access modes that we have identified.  This 
is _nearly_ adequate to cover the use cases that we are currently 
concerned about; for example, I could export all packages for public 
reflection only to a specific framework, if only I know the module name 
of the implementation.


Unfortunately, this does not work well in the case where a module may 
consume a framework whose specification is separate from the 
implementation.  An application module may need to use (say) EJB and 
JPA; there is presently no clean way to do so without either (a) relying 
on a container environment to rewrite the descriptor or (b) opening up 
the module and defeating the security mechanism (e.g. "weak").  Without 
either of these workarounds, the application developer must have a good 
deal of knowledge about what modules provide what services within a 
framework-rich environment, possibly resulting in a very verbose (and 
error-prone) descriptor; none of these options is really satisfactory.


Thus, apart from the option of redesigning (to an extent) the security 
mechanism (thereby eliminating the need to seal off access to public 
reflection, which is definitely still an attractive option for various 
reasons from our perspective, but which is also a very different 
discussion), we need some sort of mechanism which decouples the literal 
dependency system from access permission (much like uses/provides does).


For example if I could declare that my module uses "javax.ejb", and, in 
so doing, automatically grants public and private reflective access to 
the module that provides that service, this would be a good outcome.  A 
module which answers to that service name could be responsible for 
reflective access to the application module, providing that information 
privately to any other framework modules which require it.


The migration story looks much better in this light: module descriptors 
still can be quite terse and specific.  Applications which use 
reflective frameworks do not need gratuitous exports; in fact it's much 
more fluid for a user to say "I require these 

Re: Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports

[David M. Lloyd]

On 09/21/2016 12:50 AM, Jochen Theodorou wrote:

On 12.09.2016 17:08, Mark Reinhold wrote:

Issue summary
-

   #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
   libraries require reflective access to members of the non-exported
   types of other modules; examples include dependency injection (Guice),
   persistence (JPA), debugging tools, code-automation tools, and
   serialization (XStream).  In some cases the particular library to be
   used is not known until run time (e.g., Hibernate and EclipseLink both
   implement JPA).  This capability is also sometimes used to work around
   bugs in unchangeable code.  Access to non-exported packages can, at
   present, only be done via command-line flags, which is extremely
   awkward.  Provide an easier way for reflective code to access such
   non-exported types. [1]

   #AwkwardStrongEncapsulation --- A non-public element of an exported
   package can still be accessed via the
`AccessibleObject::setAccessible`
   method of the core reflection API.  The only way to strongly
   encapsulate such an element is to move it to a non-exported package.
   This makes it awkward, at best, to encapsulate the internals of a
   package that defines a public API. [2]

[...]

I´d like to give some feedback on this. Our situation was this, that we
finally managed to get the Groovy build running on JDK9 with the jigsaw
version before this change here came live. The situation now is, that
gradle broke and we are not even starting compilation anymore. Well, can
happen with a change like this. But the issue at hand was about using
setAccessible (without exporting private) to get access to a protected
method in java.base. When this proposal mentioned non-public I was
actually automatically thinking private, but of course there are at
least two more visibility options to think of.

So I find another awkward point in #AwkwardStrongEncapsulation in that
you have exported API and you have a protected method in it. You can
write a class in your own module, that will overwrite or use that method
given that it is in a subclass. But if I want to use setAccessible to
invoke the method I cannot do that? This is awkward to me, because it
degrades the former (and often misused) swiss-army-knife setAccessible
to something that is even less capable than what I can do by subclassing
- unless special action are taken. I can understand the idea for private
methods or package private - but protected is for me always part of the
API to program against and as such it makes not sense to me to prevent
setAccessible accessing it here.


It turns out that you don't actually need setAccessible to access a 
superclass protected method from a subclass.


If you have an instance of a subclass of the type containing the 
protected method, any class in the _same package_ as the _subclass_ can 
call invoke() on a Method without first making it accessible, as long as 
you pass in that subclass instance (even if the method is static, which 
is a little weird but necessary to make the check work!).


As to whether this makes sense in 100% of cases... that's a different 
discussion (and a fairly complex one at that - the code for checking 
protected access is (AFAICT) the most complex part of access checking in 
the JDK and the JVM).

--
- DML

Alternatives and fixes to #ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation proposals

[David M. Lloyd]

I've recently sent the below message to the jpms-spec-experts list.  For 
the reasons listed below, we find the current proposals unworkable, and 
would like to propose some refinements and/or alternatives.  I'll follow 
up soon with separate emails to discuss specific alternatives and fixes 
in various areas, and to cover what our minimum requirements are in each 
area.


On 09/16/2016 10:30 AM, David M. Lloyd wrote:

After a fairly detailed review of this proposal, we have determined that
it is not acceptable to Red Hat in its present form.  I will list the
primary problems here, and then I'll start up discussion on jigsaw-dev
of several possible solutions that could work for us.  I'll number these
in case anyone wants to respond piece-wise.

#1) The "weak" designation appears to be pejorative

Under this solution, many existing frameworks and/or the modules which
consume them must be marked "weak" in order to work correctly, or else a
much more complex module descriptor must be used.  While this seems to
be a stepping stone for migration, we find that it rather makes
middleware appear as a second-class citizen.

#2) Weak modules are, in fact, weak

The idea of retaining the Java 8 rules for transition modules appears
good on the surface, however the problem is that users may be forced to
use weak modules indefinitely in order to interoperate with certain
existing middleware and libraries, or else be faced with a potentially
complex migration.  If the security benefits of "strong" modules are
real, then users will be rightly inclined to move from weak modules to
strong modules.  This can be severely problematic if existing
middleware, libraries, and large applications cannot easily operate as
or with "strong" modules or cannot do so in an "easy to learn"/"easy to
use" fashion.  This also means that "weak" modules exist only for the
purposes of maintaining a transition period - at some point they will
inevitably transition from "convenience" to "attractive nuisance".  We
believe that it is feasible to solve compatibility to a satisfactory
degree without relegating users of existing software to a lower tier of
security or functionality.

#3) Controlling reflection access is always bound up with controlling
exports

The proposed exports system is simplistic but not exactly simple, yet
several useful modes remain left unaccounted-for.  Internally we
examined the usefulness of controlling access to public and so-called
"deep" reflection in combination with whether or not a member was
exported, and we came up with this logical table:

 +--+---+
 |  |Module is...   |
 |  Reflection is... ++--+
 |  |  Exported  | Not Exported |
 +--+
 |Forbidden | Not Useful |Useful|
 +--+
 |  Public Only |   Useful   |Useful|
 +--+
 | Public & Private |   Useful   |Useful|
 +--++--+

Some of these useful modes seem not to be available, or are not usable
in an obvious or simple manner.  These modes are useful, possibly even
necessary, to introduce in Java 9 in order to provide a good
compatibility story with the many existing frameworks in the wild, in
the absence of weak modules.

It has been observed that in some cases, the necessity of granting
public or private reflective access to a module is actually bound not
with what that module exports, but rather with what it requires or uses,
which might be useful in consideration of a solution; for example,
"using" a persistence framework which operates only on public types
would imply that the module would grant back public-only reflective
access to the framework which provides the service.

#4) There isn't always a container

The justification for not providing an easy mechanism to restrict
private reflective access to a trusted framework is that a container can
always rewrite a module or inject classes.  However there isn't always a
container present - something which the existence of weak modules
acknowledges.  Some frameworks and middleware are meant to be usable
with or without a container, and it is undesirable to force these use
cases into a weak module situation or to require a strong coupling in
the module descriptor in this case.  Some other solution should be found
that works both for containers and "regular" applications.

Broadly speaking, a container would be able to do additional things that
are useful do to specifically in a container.  But a container should
not be _necessary_ in order to do things that are gene

Re: Revisiting JDK-8161269

[David M. Lloyd]

On 09/16/2016 10:30 AM, Alan Bateman wrote:

On 16/09/2016 07:21, David M. Lloyd wrote:


Hi Alan,

In JDK-8161269 you said [1] that the "null" class loader has never
been specified to contain all Java SE types, using this as a
justification to reject this issue as "Not an Issue", regardless of
the compatibility impact (particularly the common case of a class
loader with a null parent).

The context here is de-privileging non-core components so that they are
no longer defined to the boot loader with all permissions. Modules such
as java.corba, java.xml.ws, java.sql and many more have no business or
need to be defined to the boot loader. It would be nice, if over the
very long term, that we could get to the point where the only module
defined to the boot loader is java.base. I don't know if we will ever
get there.

We of course acknowledge that there is potential compatibility impact
with this change and this is why it is called out in the Risks and
Assumption section of JEP 261 [1] as

"Some Java SE types have been de-privileged and are now loaded by the
platform class loader rather than the bootstrap class loader, as noted
above. Existing custom class loaders that delegate directly to the
bootstrap class loader might not work correctly; they should be updated
to delegate to the platform class loader, which is easily available via
the new ClassLoader::getPlatformClassLoader method."

So far then there hasn't been a lot of feedback on this. The default
loader for delegation is the system class loader so someone creating a
class loader that delegates to the boot loader is probably an advanced
case.

As noted in the bug report, the ClassLoader javadoc has been updated for
Java SE 9 to define the built-in class loaders. The most important part
of this is specifying that all platform classes are visible via the
platform class loader. That works for Java SE 8 and older if you replace
"platform" with "extension". I mention this in case there is concerns
about using a new API in code that needs to be compiled for JDK 8 and
run on 8 or 9.

I don't have cycles just now to get into the topic of radically changing
the class loader hierarchy or how modules on the module path are mapped
to loaders - that topic is an order of magnitude larger than the above,
esp. once you get into all the scenarios around migration.


OK.  For this issue though, would it not make sense to look at the null 
parent class loader case in a specific and separate way: in the past, 
such class loaders had access to all platform classes, so as a 
compatibility factor it would not be unreasonable to take "null" in this 
*specific* context to mean "platform class loader" and do this 
translation inside the constructor?  Since the change that is really 
occurring (in any real, observable sense) is that the "null" parent is 
suddenly shrinking with respect to what it was going back far into 
history, then the _compatible_ change would appear to be to provide a 
new ClassLoader value that indicates "java.base (and optionally a few 
more modules that can decrease over time)", which in reality maps to the 
bootstrap class loader.  This way compatibility is maintained, and the 
new (in the observable sense) functionality of having a more limited 
parent CL is still available.


WDYT?
--
- DML

Revisiting JDK-8161269

[David M. Lloyd]

Hi Alan,

In JDK-8161269 you said [1] that the "null" class loader has never been 
specified to contain all Java SE types, using this as a justification to 
reject this issue as "Not an Issue", regardless of the compatibility 
impact (particularly the common case of a class loader with a null parent).


However in a recent email [2] on the topic of letting module path 
modules each reside in a separate class loader so as to allow for 
duplicate non-exported packages, you seem to imply that the impact of 
changing the class loader of a library, even one on the module path, is 
a compatibility risk though there were no concrete details given.


What is the standard for compatibility?  I contend that there are real 
frameworks today that are in wide usage that have assumed that a null 
class loader contains all platform classes; however, I have yet to run 
across any framework that worked under a class path (i.e. application 
class loader) but failed as soon as it was loaded under its own isolated 
class loader (in our case, as a module under JBoss Modules).  By my 
empirical experience, the former is a significant compatibility risk, 
while the latter is not; by the standard you have set in these two 
places, it appears that the former is not a risk (or not enough of one 
to warrant action) but the latter would be, so I was hoping you'd expand 
a bit so I can get a better understanding of where you're coming from.


Thanks!

[1] 
https://bugs.openjdk.java.net/browse/JDK-8161269?focusedCommentId=13973088#comment-13973088
[2] 
http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009255.html


--
- DML

Re: Using non-parallel custom class loaders for Layer configurations

[David M. Lloyd]

On 09/09/2016 02:05 PM, Mandy Chung wrote:



On Sep 9, 2016, at 8:44 AM, David M. Lloyd  wrote:

On 09/08/2016 06:48 PM, Mandy Chung wrote:



On Sep 8, 2016, at 3:29 PM, David M. Lloyd  wrote:

Would it be possible to include a method like this (pretty old patch I had 
laying around):

diff --git a/jdk/src/java.base/share/classes/java/lang/ClassLoader.java 
b/jdk/src/java.base/share/classes/java/lang/ClassLoader.java
index 1bb1580..3def10e 100644
--- a/jdk/src/java.base/share/classes/java/lang/ClassLoader.java
+++ b/jdk/src/java.base/share/classes/java/lang/ClassLoader.java
@@ -469,6 +477,15 @@ public abstract class ClassLoader {
   return lock;
   }

+/**
+ * Determine whether this class loader is parallel-capable.
+ *
+ * @return {@code true} if the class loader is parallel-capable, {@code 
false} otherwise
+ */
+protected final boolean isParallelCapable() {
+return ParallelLoaders.isRegistered(getClass());
+}
+


This seems a fine method to add.  I think it can be a public method (I don’t 
see any reason it has to be protected)


Great!  Since it's such a small method though, would it be possible to just 
roll this change into some other changeset?  I can't really contribute the 
change in any form other than a patch right now.


I file a RFE and Brent Christian has agreed to work on it and will go through 
JDK 9 FC approval request.
  https://bugs.openjdk.java.net/browse/JDK-8165793


Thanks!
--
- DML

Re: Using non-parallel custom class loaders for Layer configurations

[David M. Lloyd]

On 09/09/2016 11:26 AM, Alan Bateman wrote:

On 08/09/2016 23:29, David M. Lloyd wrote:


Is it not necessary that any class loader in use by a Layer must be
parallel-capable?  Otherwise it seems like deadlocks could occur in
certain situations when there are references that are cyclic with
respect to class loaders mapped by the mapping function.

There aren't cycles in readability graph, at least not at construction
time. They might arise later due to reflective use of addReads but that
shouldn't change the class loader delegation graph. So I'm curious if
you are actually running into an issue or not? I can easily come up with
a mapping to loaders to force this but it would be unusual mapping.

BTW: I'm not opposed to exposing isParallelCapable, just curious if this
is something that you are actually running into or not.


No, I've only run into this with our module system which is separate 
from Jigsaw, but I was examining this code recently and I was surprised 
to see that Jigsaw did not take this into account.


I don't think there needs to be a readability cycle between modules for 
this to be a potential problem; there only has to be a cycle between 
class loaders.  So if you (for whatever reason) put half of your modules 
in one class loader and half in another, and those modules refer to one 
another, I think you might have a problem if those modules refer to one 
another and the class loaders are not parallel capable.  But like I said 
that's just a hypothesis.


--
- DML

Re: Using non-parallel custom class loaders for Layer configurations

[David M. Lloyd]

On 09/08/2016 06:48 PM, Mandy Chung wrote:



On Sep 8, 2016, at 3:29 PM, David M. Lloyd  wrote:

Would it be possible to include a method like this (pretty old patch I had 
laying around):

diff --git a/jdk/src/java.base/share/classes/java/lang/ClassLoader.java 
b/jdk/src/java.base/share/classes/java/lang/ClassLoader.java
index 1bb1580..3def10e 100644
--- a/jdk/src/java.base/share/classes/java/lang/ClassLoader.java
+++ b/jdk/src/java.base/share/classes/java/lang/ClassLoader.java
@@ -469,6 +477,15 @@ public abstract class ClassLoader {
return lock;
}

+/**
+ * Determine whether this class loader is parallel-capable.
+ *
+ * @return {@code true} if the class loader is parallel-capable, {@code 
false} otherwise
+ */
+protected final boolean isParallelCapable() {
+return ParallelLoaders.isRegistered(getClass());
+}
+


This seems a fine method to add.  I think it can be a public method (I don’t 
see any reason it has to be protected)


Great!  Since it's such a small method though, would it be possible to 
just roll this change into some other changeset?  I can't really 
contribute the change in any form other than a patch right now.


Thanks.
--
- DML

Using non-parallel custom class loaders for Layer configurations

[David M. Lloyd]

Is it not necessary that any class loader in use by a Layer must be 
parallel-capable?  Otherwise it seems like deadlocks could occur in 
certain situations when there are references that are cyclic with 
respect to class loaders mapped by the mapping function.


On that note, there is in fact no good way for user code to determine 
whether or not a class loader is indeed parallel-capable, which is often 
a critical question in module systems, especially when users can provide 
specialized class loader implementations.  Right now you can only do 
something (awful) like this (e.g. in your base ClassLoader constructor):


if (getClassLoadingLock("$TEST$") == this) {
throw new Error("Cannot instantiate non-parallel subclass");
}

Would it be possible to include a method like this (pretty old patch I 
had laying around):


diff --git a/jdk/src/java.base/share/classes/java/lang/ClassLoader.java 
b/jdk/src/java.base/share/classes/java/lang/ClassLoader.java

index 1bb1580..3def10e 100644
--- a/jdk/src/java.base/share/classes/java/lang/ClassLoader.java
+++ b/jdk/src/java.base/share/classes/java/lang/ClassLoader.java
@@ -469,6 +477,15 @@ public abstract class ClassLoader {
 return lock;
 }

+/**
+ * Determine whether this class loader is parallel-capable.
+ *
+ * @return {@code true} if the class loader is parallel-capable, 
{@code false} otherwise

+ */
+protected final boolean isParallelCapable() {
+return ParallelLoaders.isRegistered(getClass());
+}
+
 // This method is invoked by the virtual machine to load a class.
 private Class loadClassInternal(String name)
 throws ClassNotFoundException

Alternatively, the method could be made static and protected, only 
returning true if the calling class is a class loader that is parallel 
capable, something like this (against a newer branch):


diff --git a/jdk/src/java.base/share/classes/java/lang/ClassLoader.java 
b/jdk/src/java.base/share/classes/java/lang/ClassLoader.java

index 0139123..06a5909 100644
--- a/jdk/src/java.base/share/classes/java/lang/ClassLoader.java
+++ b/jdk/src/java.base/share/classes/java/lang/ClassLoader.java
@@ -1421,6 +1421,19 @@ public abstract class ClassLoader {
 }

 /**
+ * Determine whether the calling class is a parallel-capable class 
loader.

+ *
+ * @return true if the caller is a parallel-capable class loader 
and false otherwise

+ *
+ * @since  9
+ */
+@CallerSensitive
+protected static boolean isParallelCapable() {
+final Class callerClass = Reflection.getCallerClass();
+return ClassLoader.class.isAssignableFrom(callerClass) && 
ParallelLoaders.isRegistered(callerClass.asSubclass(ClassLoader.class));

+}
+
+/**
  * Find a resource of the specified name from the search path used 
to load

  * classes.  This method locates the resource through the system class
  * loader (see {@link #getSystemClassLoader()}).

WDYT?

--
- DML

Re: Multiple versions of a non-exported dependency

[David M. Lloyd]

On 09/01/2016 08:59 AM, Alan Bateman wrote:

On 01/09/2016 13:29, David M. Lloyd wrote:


It seems like there is no good reason why the application modules
aren't loaded with classloader-per-module now.  The platform stuff
could all be in one, but the application stuff?  Problems like this
are going to come up a lot otherwise; let's consider making that change.

If we were just dealing with a graph of explicit modules they it may be
an option, assuming you get over all the issues that arise when
arranging visibility this way. You might remember "module mode" in the
original exploratory phase of Project Jigsaw for example.

The issue is of course that there is lot more in picture, esp. when you
have the unnamed module (= class path) reading all modules, also
automatic modules that bridge to the class path (and so read the unnamed
module). Then add upgradable modules into the picture, ... and you will
quickly start to see there is a lot more to this, not to mind the risk
of circular delegation.


Risk?  If the modules don't have circular delegation then the class 
loaders won't; but anyway I don't understand *at all* why circular 
delegation in class loaders is a problem (we do this today and it has 
worked great since Java 7, and there's no particular magic necessary to 
do so).  And in any event circular delegation in modules *should* be 
allowed, full stop; there's no good reason not to allow it (especially 
given the very-super-eager-loading behavior of module layers).  Again we 
do this today and it's fine.



So I think what we have ended up with sane and not difficult to explain.
It favors migration and good interop over green field.


Yeah having the class path remain on the legacy application class loader 
is demonstrably better for interop.  But new modules?  Does that make sense?



Sure, there will
be periodic complaints when people try to deploy modules with
overlapping packages on the application module path. Anyone using Maven
Shade Plugin and the like can continue to do this. Finally, it's not
hard to create your own "launcher" that instantiates the configuration
with Layer.defineModulesWithManyLoaders if you really want.


I think many people will do this.  The benefits of modules are 
diminishing if you don't actually get this level of isolation by 
default.  And I think that it's much harder to imagine a real interop 
problem that could arise from it than it is to imagine a real problem 
that will occur from not doing it.


Anyway using interoperability as an argument is very weak as long as 
"export dynamic *" or any variation thereof is considered to be an 
acceptable solution to #ReflectiveAccessToNonExportedTypes.  In order to 
have proper interoperability for any reflection-using or reflected 
module, you have to do this, which defeats the primary security measure 
of modules.  Isn't this a much more likely interop problem than putting 
modules (something that never existed before) into their own class loaders?


--
- DML

Re: Multiple versions of a non-exported dependency

[David M. Lloyd]

You've missed the point I'm afraid.  I'm just talking about having the 
launcher use (the equivalent of) 
java.lang.reflect.Layer#defineModulesWithManyLoaders instead of (the 
equivalent of) java.lang.reflect.Layer#defineModulesWithOneLoader.  (The 
launcher actually uses the slightly lower level defineModules() method I 
think, but really what I'm suggesting is to update the function to 
assign new class loaders for named modules instead of reusing the same one.)


On 09/01/2016 08:23 AM, Gregg Wonderly wrote:

The important detail for me, is that ClassLoader per module, with the current 
Class resolution scheme (this ClassLoader and whatever I can find in the 
parent), provides a lot of issues. The “custom ClassLoaders” or “containers 
like OSGi” remarks point at the “us and them” attitude that is pretty prevalent 
in this conversation.  The majority of developers are looking for a module 
system that is not an “us or them” proposition.   These “all or nothing” 
compromises are what create the “hell” that dominates conversations here.  What 
we all want to be able to do, is write software once, target it to “THE Java 
platform”, and be done.

What Sun and now Oracle are continuing to do, is create more stuff that is 
nothing like what everyone else is doing with modularity and instead create 
something that is orthogonal to most peoples problem spaces and in the end 
creates tremendously more “work” for nothing more than compatibility with the 
new “JVM” environment.

The real goal here needs to be making all of the other module and container 
systems obsolete.  Those systems should “want” to provide support for the 
awesome, new module system that will make in unnecessary for them to roll their 
own details any longer.

Yes, that is a long road and a tall measure for success.  But frankly, even the 
lack of any visibility of the style of modules that Netbeans has used for 
decades makes it clear that this groups view at Oracle is extremely narrow and 
perhaps even more uninformed about what the community actually needs.

Gregg


On Sep 1, 2016, at 7:29 AM, David M. Lloyd  wrote:

It seems like there is no good reason why the application modules aren't loaded 
with classloader-per-module now.  The platform stuff could all be in one, but 
the application stuff?  Problems like this are going to come up a lot 
otherwise; let's consider making that change.

On 08/31/2016 07:45 PM, Neil Bartlett wrote:

Remi,

Actually I don’t think that statically linking will work. This would produce 
modules that have overlapping private (non-exported) packages, and such modules 
also cannot be used in Java 9 on the modulepath.

I tested this in build 9-ea+126-jigsaw-nightly-h5280-20160713 by creating two 
modules both containing a private package org.example.util. The following 
exception resulted:  java.lang.reflect.LayerInstantiationException: Package 
org.example.util in both module a and module b.

Again this could be “solved” by using custom ClassLoaders or a 
ClassLoader-based module system like OSGi on Java 9.

Neil




On 31 Aug 2016, at 20:28, Remi Forax  wrote:

The other solution is to statically link the right version of slf4j inside 
guava and jsoup.
A tool like jarjar can be updated to merge two modular jars (merge two 
module-info).

cheers,
Rémi

- Mail original -

De: "Neil Bartlett" 
À: cow...@bbs.darktech.org, "Alex Buckley" 
Cc: "ZML-OpenJDK-Jigsaw-Developers" 
Envoyé: Mercredi 31 Août 2016 20:54:44
Objet: Re: Multiple versions of a non-exported dependency



Gili,

As Alex points out: your use-case can be supported in Java 9 but only with the
addition of custom ClassLoaders, or by using an existing ClassLoader-based
module system such as OSGi.

The same is also true of Java 8, and Java 7, etc.

Regards,
Neil



On 31 Aug 2016, at 19:29, Alex Buckley  wrote:

On 8/31/2016 10:56 AM, cowwoc wrote:

I recently became aware of the fact that the Jigsaw specification declared
"version-selection" as a non-goal. While I understand how we ended up here,
I am hoping that you were able to support the following (very common)
use-case:

* Module "HelloWorld" depends on modules "Guava" and "JSoup".
* Module "Guava" depends on module slf4j version 1 (requires but does not
export it).
* Module "JSoup" depends on module slf4j version 2 (requires but does not
export it).
* slf4j version 2 and is not backwards-compatible with version 1.

What happens at runtime? Will Jigsaw (out of the box, without 3rd-party
tools like Maven or OSGI) be smart enough to provide different versions of
slf4j to "Guava" and "JSoup"?


(You mean Guava/JSoup requires slf4j version 1/2 and does not "re-export" it
a.k.a. 'requires public'.)

This use case isn't possible on JDK 8 for JARs on the classpath, and it's not
supported on JDK 9 for modular JARs on the modulepath

Re: Multiple versions of a non-exported dependency

[David M. Lloyd]

It seems like there is no good reason why the application modules aren't 
loaded with classloader-per-module now.  The platform stuff could all be 
in one, but the application stuff?  Problems like this are going to come 
up a lot otherwise; let's consider making that change.


On 08/31/2016 07:45 PM, Neil Bartlett wrote:

Remi,

Actually I don’t think that statically linking will work. This would produce 
modules that have overlapping private (non-exported) packages, and such modules 
also cannot be used in Java 9 on the modulepath.

I tested this in build 9-ea+126-jigsaw-nightly-h5280-20160713 by creating two 
modules both containing a private package org.example.util. The following 
exception resulted:  java.lang.reflect.LayerInstantiationException: Package 
org.example.util in both module a and module b.

Again this could be “solved” by using custom ClassLoaders or a 
ClassLoader-based module system like OSGi on Java 9.

Neil




On 31 Aug 2016, at 20:28, Remi Forax  wrote:

The other solution is to statically link the right version of slf4j inside 
guava and jsoup.
A tool like jarjar can be updated to merge two modular jars (merge two 
module-info).

cheers,
Rémi

- Mail original -

De: "Neil Bartlett" 
À: cow...@bbs.darktech.org, "Alex Buckley" 
Cc: "ZML-OpenJDK-Jigsaw-Developers" 
Envoyé: Mercredi 31 Août 2016 20:54:44
Objet: Re: Multiple versions of a non-exported dependency



Gili,

As Alex points out: your use-case can be supported in Java 9 but only with the
addition of custom ClassLoaders, or by using an existing ClassLoader-based
module system such as OSGi.

The same is also true of Java 8, and Java 7, etc.

Regards,
Neil



On 31 Aug 2016, at 19:29, Alex Buckley  wrote:

On 8/31/2016 10:56 AM, cowwoc wrote:

I recently became aware of the fact that the Jigsaw specification declared
"version-selection" as a non-goal. While I understand how we ended up here,
I am hoping that you were able to support the following (very common)
use-case:

* Module "HelloWorld" depends on modules "Guava" and "JSoup".
* Module "Guava" depends on module slf4j version 1 (requires but does not
export it).
* Module "JSoup" depends on module slf4j version 2 (requires but does not
export it).
* slf4j version 2 and is not backwards-compatible with version 1.

What happens at runtime? Will Jigsaw (out of the box, without 3rd-party
tools like Maven or OSGI) be smart enough to provide different versions of
slf4j to "Guava" and "JSoup"?


(You mean Guava/JSoup requires slf4j version 1/2 and does not "re-export" it
a.k.a. 'requires public'.)

This use case isn't possible on JDK 8 for JARs on the classpath, and it's not
supported on JDK 9 for modular JARs on the modulepath:

- If you have two versions of a modular JAR slf4j.jar in different directories
on the modulepath, then the first one to be found will dominate, and that's
what will be resolved for both Guava and JSoup.

- If you have two modular JARs slf4j_v1.jar and slf4j_v2.jar on the modulepath,
and Guava requires slf4j_v1 and JSoup requires slf4j_v2, then launching 'java
-m HelloWorld' will fail. The boot layer will refuse to map the "same" packages
from different slf4j_v* modules to the application class loader.

The use case _is_ supported on JDK 9 for modular JARs loaded into custom loaders
of custom layers. That is, the Java Platform Module System is perfectly capable
of supporting the use case -- please see any of my "Jigsaw: Under The Hood"
presentations. The use case just isn't supported "out of the box" by the 'java'
launcher for JARs on the modulepath.

Alex




--
- DML

Re: Multiple versions of a non-exported dependency

[David M. Lloyd]

There are two dimensions to the version range issue: build 
reproducibility and future-proofing.


In the former category, single versions are generally used for build 
because that ensures that the build will not change over time as new 
versions of things become available.  In other words, I build version X 
of my module on Monday or on Friday and I get the same bits in the end 
(more or less).  Depending on a version range means that a newer version 
of a thing becoming available to the build environment can potentially 
cause a change in the final artifact, so version ranges are clearly not 
good for the build phase.


In the latter category, and this is really the primary motivator for 
having version ranges in the first place, there is a desire for the 
developer to stipulate that a module will work with a range of versions 
of another module.  Thus specifying a range of such versions is 
desirable.  However this is really a test phase concern: you really only 
get an idea of compatibility with some version of a dependency after it 
has been tested or after some other process wherein such a compatibility 
can be reliably asserted.  Furthermore, the range of compatibility may 
be found to have changed over the lifetime of a single version of a 
given artifact, such that one may wish to add or remove versions from 
the known-to-be-compatible set.  Thus this kind of information cannot 
and should not be part and parcel with the artifact itself; rather it 
belongs to the repository or distribution which hosts the artifact, and 
to which (for example) a CI system may refer.


On 08/31/2016 02:21 PM, cowwoc wrote:

Well, this is unfortunate. As I stated earlier, I fail to see how
depending on constant version numbers (not version ranges) fall under
the scope of "version selection". Was this case considered/discussed in
depth?

Not everyone is sold on version ranges (e.g. the vast majority of Maven
artifacts I've seen depend on constant versions) and I think this would
go a long way towards solving the original "classpath hell" problem.

Gili

On 2016-08-31 2:55 PM, Neil Bartlett [via jigsaw-dev] wrote:

Gili,

As Alex points out: your use-case can be supported in Java 9 but only
with the addition of custom ClassLoaders, or by using an existing
ClassLoader-based module system such as OSGi.

The same is also true of Java 8, and Java 7, etc.

Regards,
Neil



On 31 Aug 2016, at 19:29, Alex Buckley <[hidden email]

> wrote:


On 8/31/2016 10:56 AM, cowwoc wrote:

I recently became aware of the fact that the Jigsaw specification

declared

"version-selection" as a non-goal. While I understand how we ended

up here,

I am hoping that you were able to support the following (very common)
use-case:

* Module "HelloWorld" depends on modules "Guava" and "JSoup".
* Module "Guava" depends on module slf4j version 1 (requires but

does not

export it).
* Module "JSoup" depends on module slf4j version 2 (requires but

does not

export it).
* slf4j version 2 and is not backwards-compatible with version 1.

What happens at runtime? Will Jigsaw (out of the box, without

3rd-party

tools like Maven or OSGI) be smart enough to provide different

versions of

slf4j to "Guava" and "JSoup"?


(You mean Guava/JSoup requires slf4j version 1/2 and does not

"re-export" it a.k.a. 'requires public'.)


This use case isn't possible on JDK 8 for JARs on the classpath, and

it's not supported on JDK 9 for modular JARs on the modulepath:


- If you have two versions of a modular JAR slf4j.jar in different

directories on the modulepath, then the first one to be found will
dominate, and that's what will be resolved for both Guava and JSoup.


- If you have two modular JARs slf4j_v1.jar and slf4j_v2.jar on the

modulepath, and Guava requires slf4j_v1 and JSoup requires slf4j_v2,
then launching 'java -m HelloWorld' will fail. The boot layer will
refuse to map the "same" packages from different slf4j_v* modules to
the application class loader.


The use case _is_ supported on JDK 9 for modular JARs loaded into

custom loaders of custom layers. That is, the Java Platform Module
System is perfectly capable of supporting the use case -- please see
any of my "Jigsaw: Under The Hood" presentations. The use case just
isn't supported "out of the box" by the 'java' launcher for JARs on
the modulepath.


Alex





If you reply to this email, your message will be added to the
discussion below:
http://jigsaw-dev.1059479.n5.nabble.com/Multiple-versions-of-a-non-exported-dependency-tp5713364p5713366.html

To unsubscribe from Multiple versions of a non-exported dependency,
click here
.
NAML

Re: Exporting - the wrong default?

[David M. Lloyd]

On 08/01/2016 05:40 AM, dalibor topic wrote:



On 29.07.2016 16:55, David M. Lloyd wrote:

On 07/29/2016 09:20 AM, dalibor topic wrote:


Is it safe to assume that all potentially headache inducing Guns and
Bullets are always kept under lock in non-public classes?


Of course, that's why we had non-public classes in the first place.


OK, let's assume that's correct for the deliberate headache instrument
hiding decisions made by all Java developers for the sake of the argument.

To balance that assumption out, please grant me an assumption as well:
Let's assume for the sake of the argument that not all Java developers
are experts all the time, and therefore some of them might sometimes
produce classes that could be abused as headache inducing Slings and
Stones.

Is it safe to assume that all such involuntarily headache inducing
Slings and Stones are also kept under lock in non-public classes?


Of course not, no more than it is safe to assume that such will be kept 
hidden in non-exported packages under the existing regime.  At some 
point the user has to make the decision to make something public or not 
public; nothing can change that.  But what we can do is to make it easy 
for users: a public class is public, otherwise it is secret and has to 
be explicitly shared.  Contrast with the current system, where whole 
packages must have their definition of "public" defined to mean either 
selectively shared or fully public, which is less intuitive and more 
confusing.  The more confused the user is, the more likely they are to 
make a mistake about security.  With my proposal, if there are any 
doubts about a class, you make it non-public (regardless of what package 
it is in), which is a very simple criterion.  In addition, you make the 
public/non-public choice on a per-class basis, not a per-package basis, 
which is also useful as well as intuitive.


--
- DML

Re: Exporting - the wrong default?

[David M. Lloyd]

On 07/29/2016 10:14 AM, Alan Bateman wrote:

On 29/07/2016 15:55, David M. Lloyd wrote:


This is better than the alternative suggestion: just export everything
to everybody, defeating the point of the security measure in the first
place.


Just to check. Do you mean `exports dynamic *` or some variant of? If so
then it would mean that all public types in the module would be
accessible to code outside of the module. Non-public types would not be
accessible (ignoring wormholes when Lookup objects leak out,
setAccessible, ... for this discussion).


Right, I'm referring to "exports dynamic *".


--
- DML

Re: Exporting - the wrong default?

[David M. Lloyd]

On 07/29/2016 09:20 AM, dalibor topic wrote:



On 28.07.2016 16:59, David M. Lloyd wrote:

You don't have to add new
public packages to the exported list, because exporting all by default
is a safe/sane default as well as being intuitive.


Is it safe to assume that all potentially headache inducing Guns and
Bullets are always kept under lock in non-public classes?


Of course, that's why we had non-public classes in the first place.  The 
only reason users use public for this kind of thing (despite the 
availability of less permissive access modes) is because the language 
gives no useful way to share these classes outside their packages; this 
is why I proposed this exact behavior as an alternative.  This is better 
than the alternative suggestion: just export everything to everybody, 
defeating the point of the security measure in the first place.  Also, I 
propose that this is much closer to the original spirit of the Java 
language than the complex knot we have on the table before us now.


At some point you must have *some* kind of faith in the user.  If you 
can trust the user to know to put the right packages in the module 
descriptor - a task that at best can be described as "side-channel" - 
then surely you can trust the user to know which classes are "safe" to 
be made public and which are not (which, I might add, is something that 
any regular Java user already knows how to do).


--
- DML

Re: Exporting - the wrong default?

[David M. Lloyd]

On 07/28/2016 07:33 AM, dalibor topic wrote:

On 27.07.2016 17:37, Stephen Colebourne wrote:

While the JDK cannot afford to compromise on security, many real-world
systems can and do. Software is mostly a trade-off between security,
quality, features, delivery date, design, resources... etc. As such,
this discussion cannot and should not be shut down solely on the basis
of security.


For an entirely hypothetical straw man example, imagine the expert
authors of a third party component A making a deliberate security trade
off in one class. For the sake of brevity, let's call that class 'Gun'.

Let's also imagine a different set of expert authors of a different
third party component B making a different deliberate security trade off
in another class. Let's call that class 'Bullet'.

Assembling a software system H which, by simply using both component A
and component B, provides both a Gun and a Bullet, could lead to some
pretty bad headaches down the road.

In the "exported by default" world view, the assembler is responsible
for restricting all such headache inducing interactions between these
classes, originating in different components with different trade offs.

Let's hope for their sake they are really awesome experts at that sort
of thing, and especially good at handling the potentially exponential
complexities that can arise from adding new components with further
third party provided Guns and Bullets to their software system.

In the "failsafe by default" world view, they are responsible for
enabling "just" the non-headache inducing interactions between those
classes, of which there in most cases may very well be none.

That's (potentially) a substantial difference in effort necessary to
accurately make such trade offs, in particular over the maintenance life
cycle of a software system.


My proposal gives the best of both worlds: all public types of all 
packages are always accessible, and only the secret/non-public types 
need to be explicitly privately exported.  You don't have to add new 
public packages to the exported list, because exporting all by default 
is a safe/sane default as well as being intuitive.  On the other hand, 
you have to go out of your way to provide additional access to new 
private packages, which is also intuitive.



--
- DML

Re: Exporting - the wrong default?

[David M. Lloyd]

On 07/27/2016 10:26 AM, Remi Forax wrote:

- Mail original -

De: "dalibor topic" 
À: jigsaw-dev@openjdk.java.net
Envoyé: Mercredi 27 Juillet 2016 14:39:05
Objet: Re: Exporting - the wrong default?



On 26.07.2016 18:42, Stephen Colebourne wrote:

In many projects
packages change names frequently during development, everything is
open and locking stuff down is the last thing on peoples minds. While
this of course leads to slightly less secure software, it does achieve
*business value*.


I would recommend Cristina Cifuentes presentation "Are We Ready for
Secure Languages?" from the recent Curry On conference, of which a
recording is available at https://www.youtube.com/watch?v=-fC975HLhyc
for some less anecdotal thoughts on the business value of slightly more
secure software.

It even touches briefly on the utility of modules in JDK 9.


Interesting talk !

to get back to our issue,
there are 4 possibilities when exporting a package, for a public type,
(1) don't see it at compile time, don't see it at runtime (can't reflect on it)
(2) don't see it at compile time, see it at runtime (this is the OSGI/JBoss 
model for not exported)
(3) see it at compile time, may not exist at runtime (so be prepared to get an 
exception then)
(4) see it at compile time and see it at runtime

The default can not be (3) because it's a corner case, it can not be (4) 
because in that case we lost the 'strong encapsulation' that a module should 
provide by default, so the default can be either (1), either (2) or to force 
the user to choose between (1) and (2) when declaring a module.

The problem with (1) is that:
 - it makes most of the code that use reflection not working (and as Stephen 
said, at lot of codes use reflection (or bytecode generation)),
 - it will slow down the adoption of jigsaw (not jdk9 which will be run with a 
-classpth) but the modularization of the already existing jars, so we will end 
up with a module system which will be not used or worst, some jars will be 
modularized, some will not and we will be in the same sad state of Python now 
with 2 mostly compatible worlds *.

The problem of letting users to choose is that the hope to educate them by 
forcing them to make their own choices will be destroyed because in practice 
IDEs will chose for them (e.printStackTrace() anyone ?)

So the only valid choice seem to be (2), which
- still enable JDK and application server implementation modules to not export 
some types at runtime, so the security will improve and by example, it will 
avoid most of the access control bugs Christina talk about.
- the default behavior will make the move to convert their jars to modularized 
jars easier because people will not conflate the problem of the modularization 
itself with the problem of the access control at runtime.
- everybody will be happy and we will not see angry ponies on slides about Java 
9.


This is all rearranging deck chairs.  The critical problem here, lest we 
forget, is that none of these solutions allow reflection to work on 
modules without an export, even for public types, thus breaking 
*everything* eventually.  Any valid choice IMO must allow public types 
to remain public and thus globally accessible.


Since everyone seems to concede that the migration to any more secure 
form for users must be incremental, there is nothing left to be gained 
by the existing Jigsaw public access restriction.  After all, even if 
public is left as public, it is still possible meet any reasonable 
standard of encapsulation.  That aforementioned incremental migration 
could be for users and JDK developers to change public classes to be 
non-public; the mechanism which currently allows for exporting packages 
to modules is conceptually *very* close to a mechanism to share the 
non-public classes in packages with modules.


In other words, users should be able to export packages with a module 
target or a wildcard (which allows compile-time and run-time linkage to 
connect to those packages), and also specify whether each (non-wildcard) 
export should include access to private types (which allows that linkage 
to connect types to those non-public members).


In addition to my aforementioned --safe-mode idea, I think it could also 
be possible/easy to provide a facility for modules to be marked such 
that using setAccessible from that module would be disallowed.


Would this not solve every problem and use case raised thus far, as well 
as being far more intuitive for users?

--
- DML

Re: Should setAccessible be part of Java or not? (was Re: It's not too late for access control)

[David M. Lloyd]

On 07/14/2016 05:28 AM, Alan Bateman wrote:

On 14/07/2016 11:16, Andrew Haley wrote:


:
OK.  But "in the very long term" such a basic language change needs
all stakeholders to be consulted.

I agree (although it's not really a language change in that it's API way
to suppress access checks specified by the language).


:
Yes, indeed, and that is potentially a significant problem.  My
comment stands: there is a serious possibility that his will make it
impossible to use (non-exported) Jigsaw modules for some kinds of
programming.  This is exactly the kind of decision that needs all
stakeholders to be consulted.

It ought to be possible to have some kind of conditional export which
only allows such access by (e.g.)  suitably privileged frameworks or
tools.  But I have no desire to get involved in such design issues: I
am only going to say that this is an issue which requires wider
consultation.


This is #ReflectiveAccessToNonExportedTypes on the JSR 376 issues list.
The problem is reasonably well understood and there are several
proposals and approaches being discussed and considered.


An important aspect of my proposal is that, by separating exports from 
accessibility, any framework which operates on public classes will 
continue to work without any modifications to exports.  Dropping 
'setAccessible' becomes more of a likelihood when frameworks have *some* 
option (like the ability to reflect any public class) available to them; 
even if it means, long-term, that these specifications and frameworks 
slowly have to make their way towards operating only on public classes.


In fact I would say that it makes a hypothetical migration path 
available for this purpose:


Time  Action
  --
   0  Jigsaw introduces friend packages
   1  Frameworks slowly move to hide non-public public things
   n  Java makes setAccessible() disablable via some --safe-mode switch
  Users begin to move to public types for their frameworked classes
   m  Java makes --safe-mode the default, adds --unsafe-mode for legacy 
things

  More users move to public types
   z  Java drops support for --unsafe-mode, setAccessible throws UOE 
forever after


In this example, n could even equal 0 - that is, we could start this 
right now.


--
- DML

Re: It's not too late for access control

[David M. Lloyd]

On 07/13/2016 04:17 PM, mark.reinh...@oracle.com wrote:

2016/7/11 7:21:46 -0700, david.ll...@redhat.com:

...

I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private.  ...


FYI, to jigsaw-dev readers: This approach was discussed on the JPMS EG
list late last year.  Here are links to the relevant messages:

   
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-November/000194.html
   
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000215.html
   
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000219.html
   
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000222.html
   
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000223.html
   
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000227.html
   
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000228.html


Also note that the discussion tapered off inconclusively before really 
discussing the possibility of selectively opening the package-private 
level to friends.  The above links are mostly about the idea of changing 
package-private to mean module-private, which was dismissed as problematic.


Using the selective extension of package-private does not suffer from 
the fatal security problems caused by simple recompilation from -target 
8 to -target 9.  The status quo is maintained in this case; users would 
have to opt in to extending access, just as Jigsaw requires users to opt 
in to make public classes available right now.


--
- DML

Re: It's not too late for access control

[David M. Lloyd]

On 07/12/2016 03:25 PM, Alan Bateman wrote:

On 12/07/2016 18:31, Sanne Grinovero wrote:


:
As a maintainer and contributor to several popular Java open source
libraries my experience is that in practice very few existing
libraries will "just work" in Java 9 out of the box: people will have
to update their code. This is based on my experience, as some of these
projects have dozens of dependencies and it's taking a long time to
identify each problematic point, discuss patches, getting other
communities to release timely, an often there is need for "recursive
releases" and various iterations for each problem as they get
identified, for each dependency.

Going off-topic slightly but when you do run into issues and if they
aren't already listed as compatibility issues then please bring them up.
So far then I think the vast majority of issues that we have heard about
relate to the changes in JEP 220 (tools.jar going away etc), JEP 223 and
the new version-string scheme, the class file version bump in jdk-9+119,
and then all the issues that we have listed in JEP 261. Any help getting
bugs submitted to projects would be appreciated too.

The issue that `public` no longer implies accessible is listed in JEP
261 (first item in the Risks and Assumption) but to be honest, has
barely come up to date. That probably isn't too surprising as it's still
early days for modules and many projects aren't trying out JDK 9 yet.
Anyone trying out modules where a "module unaware" framework gets a
reference to a public type in a non-exported package might run into it
of course but I'm not aware of any reports yet.


Isn't that what this entire thread is about?  And also, what the whole 
#ReflectiveAccessToNonExportedTypes issue is about?  If not, consider 
this the official report that dozens if not hundreds of such frameworks 
are broken under Java 9.  I have been bringing it up for many months, 
and this is a result of testing, not of guesswork.

--
- DML

Re: It's not too late for access control

[David M. Lloyd]

On 07/12/2016 11:07 AM, John Rose wrote:

On Jul 12, 2016, at 9:00 AM, John Rose  wrote:


lurk mode


P.S. I forgot one more point:  As JVM/JIT guy I find the prospect of sealing
packages very appealing.  Maybe it's a false promise, but it would be a
home run if we could reliably seal sub-assemblies for tree-shaking AOT
compilation.  Doing this requires pretty strong assurances that the AOT
engine can enumerate all uses of a "public" class.  Doesn't this use case
interfere with the proposed principle that "public always means global"?
Again, it's a tug-of-war between the class author and an assembly-time
operation.


I'm definitely sympathetic to the AOT argument.  However there is 
absolutely no difference in the guarantees that can be derived from the 
Jigsaw proposal versus my proposal.  Consider these cases:


1) A package is intended to be completely internal and is not exported; 
in Jigsaw, the types in this package would be public, and in my proposal 
they would not be.

2) A package is intended to be shared to certain targets; same as above.
3) A package is meant to be accessible to all; in both cases the types 
would be public; in the Jigsaw case it would have to be additionally 
exported to all.


In all three of these cases, AOT compilation would have to assume that 
this linkage-oriented information would not change between compile and 
assembly, and in all three of these cases, this assumption can be 
trivially defeated by an even mildly determined user.  So in the end, it 
seems to me that AOT is something that would have to be entered into 
with open eyes by the users, and all phases would have to be conscious 
of additional restrictions required to execute a stable run-time image.


--
- DML

Re: It's not too late for access control

[David M. Lloyd]

Not commenting on the entire piece here, just one bit...

On 07/12/2016 11:00 AM, John Rose wrote:

On Jul 12, 2016, at 8:01 AM, Paul Benedict mailto:pbened...@apache.org>> wrote:

This makes great sense to me. Why? Because when I develop a library, I
really don't see any great benefit in hiding a type from another package.
Let me qualify that I really don't see any great benefit in hiding a
type from another package **that I own!** (double emphasis). I have
inspected all my uses of "package private" and it always comes down to one
thing: preventing static linkage to the type from packages **outside my
library**.


To me this looks like a simple power struggle between the class author
expecting a platform for full publicity, and the system assembler expecting
to be able to control inter-library visibility.  Both can't have the
final word.
Shouldn't the assembler have it?  And if the assembler is given absolute
control over some unit of access control, it seems likely to me that this
unit has to look a lot like a Jigsaw module.


In the current model though, the assembler doesn't have the control at 
all.  The module author is the one who decides who can access what. 
Before Java 9, it was just by making some things public and some things 
non-public; now it's a question of ensuring that certain packages (maybe 
all) are accessible to all using explicit mechanisms, but the end result 
is basically the same.


If the assembler wants to add nuance to accessibility, they have to 
modify those lovely binary descriptors at assembly time.  I don't think 
Jigsaw's restriction of "public" adds anything over what I propose in 
this regard (note that what I propose does not involve editing classes 
for any reason though); my proposal doesn't really change this one way 
or another, except that it retains the current well-understood 
difference between visibility and accessibility, with the former acting 
not as a security mechanism but a linking mechanism, just like today.



Note that system assemblers cannot be expected to edit individual classes.
That would be another scale error.  So in the end, the author of a class and
of a package has to provide a provisional, local spec. of what's
visible, while
the assembler (who works with unmodifiable libraries) owns the global spec.


Yeah but think about it: how different is the need to edit classes from 
the need to edit binary module descriptors, encoded *as* class files? 
Really, the assembler should have to do neither thing, because that's an 
undue burden.


--
- DML

Re: It's not too late for access control

[David M. Lloyd]

On 07/12/2016 10:43 AM, John Rose wrote:

On Jul 12, 2016, at 2:00 AM, Remi Forax mailto:fo...@univ-mlv.fr>> wrote:


First, there is already in the Java ecosystem a notion of non-exported
package, packages startings with com.sun or packages containing
internal, it was just a convention and not something enforced by the
VM. What the JPMS spec does is just to normalize how to declare an
exported package and mandate that the VM check this new rule.

Sure it means that public classes are not accessible/visible by
everybody anymore, but a class like sun.misc.Unsafe was never really
accessible by everybody despite being declared public.

Furthermore, declaring if something is exported or not at class level
instead of at package level seems wrong to me, usually, several
classes works together for a purpose and you want these classes to be
exported or not, so it's not something that should be decided at class
level.

So i see the JPMS spec conept of non-exported package as a
standardization of an existing practice not something new that people
will have trouble to understand and reason about.


My $0.02:

As an author of Unsafe I agree enthusiastically with this viewpoint.
  (Well stated, Remi.)  System assemblies (not individual classes or
packages) need to isolate portion A of the global package namespace from
portion B.


I agree, though I disagree with the concept of a "global" package 
namespace, but that's a different discussion.



For me, declaring such assembly instructions inside class definitions
would be a scale error.  The access control in question is not a
relation between individual classes; it's a relation between larger
entities: packages.


Absolutely, and I want to re-emphasize that I do not advocate for 
changing anything inside class definitions in any way compared to what 
is in Java 8.  That is in fact the whole point of my proposal: do not 
change anything over Java 8; just add a new capability which can be used 
to improve security, and which would then subsequently be used within 
the JDK to close the various problems which exist due to internal 
classes being public.



It's a relief to see the old ad hoc package-scale hiding mechanisms
(restricted packages, shared secrets, providers, class loaders, etc.,
etc.) getting supplanted by a first-class formalism.  It's also scary to
see the amount of work involved, but it's worth it.


Sure, and I don't believe my proposal deviates all that far from the 
existing work, but it does solve the reflective access problems that 
everyone keeps hitting, which arise from the redefinition of public to 
relate to exports.


--
- DML

Re: It's not too late for access control

[David M. Lloyd]

On 07/12/2016 04:00 AM, Remi Forax wrote:

- Mail original -

De: "David M. Lloyd" 
À: "jigsaw-dev" 
Envoyé: Lundi 11 Juillet 2016 16:21:46
Objet: It's not too late for access control

The crux of this access control discussion is that, up until JDK 9,
"public" meant "public".  End of story.  If you did not want something
to be visible, you made it not public.  Very simple and very clear.  The
word "public" literally means "accessible to all" after all; that's why
the term was selected in the first place, and it is nearly certain that
this was the intent of the thing up to this date.

What is being required in the JPMS spec, and, I dare to suggest, what
has been generally asked for by the public (and even required by the
JDK), is a way to provide an additional capability - the ability to
selectively share otherwise unshared code.

The existing Java language accessibility model is (of course)
well-understood by experts.  For new developers, the concepts takes a
bit of time to explain but can generally be grasped.  In the end the
accessibility of a member is generally easily determined by examining
the qualifiers of that member.

What has been proposed and implemented in Jigsaw is essentially a
completely new approach to access checking.  Because it is new, and
because it is essentially untried, I predicted that issues would arise
not unlike the ones being currently discussed.  I argue that this
approach is not optimal, for at least reasons discussed on this list and
in this email, but that even now, it's not too late to change the approach.

I propose, once again, that rather than changing the meaning of "public"
to something unintuitive (and indeed counter to the definition of the
actual word), we instead allow the selective extension of
package-private.  Users would make public any type or member which is
*meant* to be public, i.e. accessible by all.  Rather than (at best)
changing their expectations as to the behavior of "public" only to
immediately betray that expectation by forcing them through a backdoor
in order to meet practical needs, we ensure that their expectations
remain: public members are public, and things that are secret are not
public.

Conceptually (and, hopefully, technically) this should not be too far
away from where we've arrived at now in Jigsaw, at least as far as the
package inventory is shared between modules for the purpose of access
control.  Can anyone think of any good reason we should *not* do this,
or ways that this would be substantially weaker than restricting public?
--
- DML



Hi David,
package private means package private :)

One early design idea of jigsaw was to introduce a new modifier "module" with a 
visibility in between public and package private.
It's a bad idea !


I am not suggesting that; we definitely should not do that for reasons 
discussed in the past.



First, there is already in the Java ecosystem a notion of non-exported package, 
packages startings with com.sun or packages containing internal, it was just a 
convention and not something enforced by the VM. What the JPMS spec does is 
just to normalize how to declare an exported package and mandate that the VM 
check this new rule.

Sure it means that public classes are not accessible/visible by everybody 
anymore, but a class like sun.misc.Unsafe was never really accessible by 
everybody despite being declared public.


The key advantage of using package-private is that you can drop the 
access of shared internal classes out of public, and just make that 
package accessible directly to its internal consumers.  From the JDK 
perspective: imagine getting rid of all those "shared secret" classes, 
for example.  In fact many JDK internal classes and APIs could just be 
hidden, closing innumerable security holes.


For many, many existing projects, this approach promises exact 
compatibility with the status quo in terms of accessibility, while also 
opening an avenue forward to improve module security.



Furthermore, declaring if something is exported or not at class level instead 
of at package level seems wrong to me, usually, several classes works together 
for a purpose and you want these classes to be exported or not, so it's not 
something that should be decided at class level.


I did not propose deciding at a class level; I think that would be a bad 
idea.



So i see the JPMS spec conept of non-exported package as a standardization of 
an existing practice not something new that people will have trouble to 
understand and reason about.


Except that there is no existing practice here.  Today, if I can access 
a Class or a ClassLoader, I can access all its public members, and this 
is the crux of the whole thing.  There is no valid compatibility-based 
argument for changing this behavior.  The idea is to take the status 
quo, and add a new capability that

Re: It's not too late for access control

[David M. Lloyd]

On 07/12/2016 04:53 AM, Alan Bateman wrote:

On 11/07/2016 15:21, David M. Lloyd wrote:


The crux of this access control discussion is that, up until JDK 9,
"public" meant "public".  End of story.  If you did not want something
to be visible, you made it not public.  Very simple and very clear.
The word "public" literally means "accessible to all" after all;
that's why the term was selected in the first place, and it is nearly
certain that this was the intent of the thing up to this date.

What is being required in the JPMS spec, and, I dare to suggest, what
has been generally asked for by the public (and even required by the
JDK), is a way to provide an additional capability - the ability to
selectively share otherwise unshared code.

The existing Java language accessibility model is (of course)
well-understood by experts.  For new developers, the concepts takes a
bit of time to explain but can generally be grasped.  In the end the
accessibility of a member is generally easily determined by examining
the qualifiers of that member.

What has been proposed and implemented in Jigsaw is essentially a
completely new approach to access checking.  Because it is new, and
because it is essentially untried, I predicted that issues would arise
not unlike the ones being currently discussed.  I argue that this
approach is not optimal, for at least reasons discussed on this list
and in this email, but that even now, it's not too late to change the
approach.

I propose, once again, that rather than changing the meaning of
"public" to something unintuitive (and indeed counter to the
definition of the actual word), we instead allow the selective
extension of package-private.  Users would make public any type or
member which is *meant* to be public, i.e. accessible by all. Rather
than (at best) changing their expectations as to the behavior of
"public" only to immediately betray that expectation by forcing them
through a backdoor in order to meet practical needs, we ensure that
their expectations remain: public members are public, and things that
are secret are not public.

Conceptually (and, hopefully, technically) this should not be too far
away from where we've arrived at now in Jigsaw, at least as far as the
package inventory is shared between modules for the purpose of access
control.  Can anyone think of any good reason we should *not* do this,
or ways that this would be substantially weaker than restricting public?



It's hard to know where to start on this as what you proposing would
have massive implications. I think what you are proposing is to drop the
notion of exporting packages. You didn't mention the dependency module
or the concept of readability - are you suggesting to drop that too?


No, exporting and dependencies would remain.  The concept of readability 
would go, I think, as it wouldn't add anything (and I'm really not sure 
it adds anything today, as the "direction" of the readability grant 
seems backwards to me in terms of security).



I can't tell if you've grokked that this is the basis for reliable
configuration.


Module linkage can still be reliable using only an exports graph, and 
keeping public as public.  We've been doing it for years.



You haven't said how I could release a library that works
on the class path in some environments and as a module in other
environments.


You would not need to make any changes to most libraries.  If you wanted 
to export package A to package B in a non-module environment, and if we 
wanted to support that, I don't see it being any more difficult than, 
say, supporting *both* module dependencies *and* Class-Path.



I don't see how you proposing to do the equivalent of
qualified exports.


It wouldn't have to be any different; instead of re-enabling public 
access to a dependency, you'd have the option to enable private access, 
thus ensuring that security expectations are consistent: accessibility 
is never weaker than a module author expects.



 I could go on, smarter much people than me could go
further, but I'm not sure that it's worth spending time on now.

At a high level then I think it's important to understand that a lot of
time has been put into the design.


I have sympathy for this, and I also understand (as well as anyone, 
trust me) that it is possible to spend a lot of time thinking about a 
design only to find a problem at the end.  The amount of time spent on a 
design is no proof against changing it.



The proposal in the JPMS has to take
account that the core platform is very mature. This means compatibility,
migration and co-existence with the existing world significantly
influences the design.


Of course!  That's the main point of this: the export access controls 
are so completely incompatible that just about every reflective 
framework - and there are many, many

It's not too late for access control

[David M. Lloyd]

The crux of this access control discussion is that, up until JDK 9, 
"public" meant "public".  End of story.  If you did not want something 
to be visible, you made it not public.  Very simple and very clear.  The 
word "public" literally means "accessible to all" after all; that's why 
the term was selected in the first place, and it is nearly certain that 
this was the intent of the thing up to this date.


What is being required in the JPMS spec, and, I dare to suggest, what 
has been generally asked for by the public (and even required by the 
JDK), is a way to provide an additional capability - the ability to 
selectively share otherwise unshared code.


The existing Java language accessibility model is (of course) 
well-understood by experts.  For new developers, the concepts takes a 
bit of time to explain but can generally be grasped.  In the end the 
accessibility of a member is generally easily determined by examining 
the qualifiers of that member.


What has been proposed and implemented in Jigsaw is essentially a 
completely new approach to access checking.  Because it is new, and 
because it is essentially untried, I predicted that issues would arise 
not unlike the ones being currently discussed.  I argue that this 
approach is not optimal, for at least reasons discussed on this list and 
in this email, but that even now, it's not too late to change the approach.


I propose, once again, that rather than changing the meaning of "public" 
to something unintuitive (and indeed counter to the definition of the 
actual word), we instead allow the selective extension of 
package-private.  Users would make public any type or member which is 
*meant* to be public, i.e. accessible by all.  Rather than (at best) 
changing their expectations as to the behavior of "public" only to 
immediately betray that expectation by forcing them through a backdoor 
in order to meet practical needs, we ensure that their expectations 
remain: public members are public, and things that are secret are not 
public.


Conceptually (and, hopefully, technically) this should not be too far 
away from where we've arrived at now in Jigsaw, at least as far as the 
package inventory is shared between modules for the purpose of access 
control.  Can anyone think of any good reason we should *not* do this, 
or ways that this would be substantially weaker than restricting public?

--
- DML