On 09/29/2016 07:31 AM, Peter Levart wrote:
Hi David,
I'm not the one to answer your queries, but may I ask you some questions
about your requirements and also give some comments...
On 09/28/2016 02:13 PM, David M. Lloyd wrote:
Hi all, I've been requested to ask if the OpenJDK development team
have had a chance to review this email, and when we might expect a
response.
Thanks!
On 09/21/2016 11:39 AM, David M. Lloyd wrote:
In our internal discussion of the proposal for
#ReflectiveAccessToNonExportedTypes, we discussed the ins and outs of
various behaviors and have come up with a few ideas or starting points
for solutions that we think would be more workable in conjunction with
existing middleware (ours and others').
For reasons previously explained, we do not think that weak modules are
a good way forward; I won't go into that again here. But the logical
re-starting point is: If not weak modules, then what?
I will boil it down to a few basic requirements that we have
established. This list is probably non-exhaustive but hopefully
complete enough to go on for now:
• A module definition must be able to establish that a dependent has (or
all modules have) access to one or more (or all) packages for public
reflection only.
• A module definition must be able to establish that a dependent has (or
all modules have) access to one or more (or all) packages for public or
private reflection only.
Why do you think is important to have a mode that allows reflective
access but disallows compilation/linkage? Because frameworks that use
reflection usually have ability to dynamically adjust to changes of code
that is exposed in that way while compilation/linkage hard-codes access
to the code and therefore breaks when such code is changed?
No, not at all. It's just that the two concerns (linkage versus
reflection access permission) are not really coupled.
For example if I have a library that uses reflection to examine
arbitrary objects that are handed to it, I definitely don't need to (or
want to) depend on the object class's source module or package for
linkage purposes. The code of the given object class might be non-API
for example, and I may not want to give a permanent compatibility
guarantee to my consumers, thus I do not want to export it.
I have a concern that this will not prevent breakages. If reflective
access is allowed, you can never be sure whether someone might be using
it and just hard-coding such access in the consuming module. So while
not providing "easy" hard-coded access to such code, it is still
provided, just discouraged by making it more awkward.
Of course not; anyway in this scenario the deployer could als simply
modify the descriptor to suit their needs. However it's obvious that in
doing these things that breakage may occur if the implementation changes.
There is a lot of focus on the security aspect of encapsulation, but
using encapsulation also encourages clean engineering by presenting
clear contracts (much like public and non-public types do today).
Though this is no proof against complaint in the event of breakage -
especially where the JDK is concerned - out "in the wild" of real-world
applications and frameworks, these mechanisms really work.
So I'm wondering if there might be an alternative way of "discouraging"
hard coded access without complicating the module system with this
aspect. A standard compile-time TYPE, PACKAGE or even MODULE annotation
comes to mind that would trigger compilation warning if such annotated
type or type in annotated package or type in annotated module would be
referenced in compiled code. A warning can always be turned into an
error for the conscious. I think this could even be implemented as an
annotation processor, outside of javac. There just has to be a standard
annoation to agree upon. What do you think?
It seems pretty weird to have a package be exported so you can use it
*and* annotated that you *shouldn't* use it. It seems cleanest to
export what you want exported, grant what you want granted, and leave
the rest closed off. And this talks about javac/compile time, but it is
our position that in the real world, this is completely irrelevant as
build tools can and will do whatever they need to do to access what they
want to access, with complete disregard to what is packaged in a
descriptor. The descriptors are more likely to be assembled by a
distributor or deployer than to have a module developer drop a module
which is expected to work in any module environment (which requires,
among other things, completely universal agreement on module namespacing
- something that has been shown time and again to be an unrealistic
expectation).
Therefore all these requirements are from the perspective of load-link
time to run time.
• A module definition must be able to establish that a dependent has (or
all modules have) access to one or more (or all) packages for public
reflection and compilation/linkage (i.e. it's an export by today's
terminology).
• A module definition must be able to establish that a dependent has (or
all modules have) access to one or more (or all) packages for public or
private reflection and compilation/linkage (i.e. it's a "private" export
by today's terminology).
• As today, any packages not declared in one or more of the above
categories is inaccessible outside of the module in any way (note that
as I showed previously we have also concluded that it should continue to
be impossible to export a package for compilation/linkage without public
reflection, as we have not discovered any use for such a mode).
So given above "alternative" for discouraging hard-coded access, current
jigsaw already covers your requirements up to this point. Except maybe a
desire to more easily "export" multiple packages with a single
directive. I don't know why a "wildcard" export couldn't be allowed. I
haven't heard a strong argument against it yet besides the fact that
packages do have hierarchical naming structure, but are otherwise
independent entities so using wildcard exports would be a precedent
which would group a subtree of packages in a hierarchy of names and give
them equal accessibility. But that would just accept the reality which
is that developers usually do group packages in naming hierarchies
according to some property. This might not be THE property which exports
would give them (accessibility) so such feature might in fact push
developers into grouping packages into hierarchies by the accessibility
constraints instead of giving them freedom of unbalanced choice. So this
might be an argument against wildcard exports...
I think that a single '*' wildcard which applies to all packages is
definitely required, however I don't have an opinion about mixing
wildcards with partial package names.
More generally:
• The syntax for all of the above has no particular constraint (in fact
I will try to actively avoid touching what could be a very
bikeshedding-rich discussion), except that it should not be construable
as being pejorative against the usage of reflective frameworks; rather,
it should be clear what level of trust is being established without
raising undue warning.
• Applications should not need gratuitous amounts of declarations in
their module(s) in order to utilize frameworks.
• As previously established, it should not be possible for one
declaration to reduce the scope of access of another declaration in a
module definition.
I have no questions or comments about above three requirements as I
think are not in conflict currently. Are they?
Well, "weak" modules are in direct conflict with the first one, as I
described previously. To use non-"weak" modules seems to mean requiring
gratuitous declarations, in the absence of "exports dynamic". Only the
third point is not currently in conflict; mainly I'm just reaffirming
that to head off any proposals that would be rejected out of hand for
that reason.
• Access to a module (for reflective purposes only) must not cause
conflicts if multiple such modules which contain identical packages are
accessible to a single consumer; in other words, reflection-only access
into non-dependency modules is not bound by duplicate package
restrictions as long as each package is unique per class loader, as per
the current (Java 8) class loader rules.
I think this is more or less not a problem. Access to a module for
reflective purposes is usually performed by frameworks and framework
modules do not "require" modules that expose classes for reflective
access (it's usually the other way around). Frameworks can reflectively
access classes in modules that they don't "require" because reflection
does not need or implies readability. The only concern is when two
independent modules, loaded by different class loaders, export same
packages and they are both "required" by the same module. Maybe I don't
see it, but is this a use case that arises in practice?
No, this is not something that happens with our module system and I
don't expect that it will be an expected use case in the future.
The above cover the useful access modes that we have identified. This
is _nearly_ adequate to cover the use cases that we are currently
concerned about; for example, I could export all packages for public
reflection only to a specific framework, if only I know the module name
of the implementation.
Unfortunately, this does not work well in the case where a module may
consume a framework whose specification is separate from the
implementation. An application module may need to use (say) EJB and
JPA; there is presently no clean way to do so without either (a) relying
on a container environment to rewrite the descriptor or (b) opening up
the module and defeating the security mechanism (e.g. "weak"). Without
either of these workarounds, the application developer must have a good
deal of knowledge about what modules provide what services within a
framework-rich environment, possibly resulting in a very verbose (and
error-prone) descriptor; none of these options is really satisfactory.
This, I think, is a real concern that needs addressing and has its
weight so that complicating jigsaw because of it might be warranted.
Maybe there is an API only solution. If we look at the use-cases, they
are all about a consuming module using some API which is implemented by
a module which is not known in advance and needs reflective access back
to classes of consuming module.
For example, we have a consuming module, say "app", an API module that
just defines the API, say "jpa" and a module that implements the API and
is somehow dynamically resolved at runtime (by ServiceLoader or
similarly), say "hibernate", which name is not known in advance to the
"app" module. What "app" module does know is the name of the API module,
"jpa", so the "app" module could pretend that the "jpa" module is the
module to be performing reflective access to its classes:
module app {
requires jpa;
exports private app.entity to jpa;
}
This, by itself, would only enable accessing app.enity classes from jpa
module which is just a bunch of interfaces without actual code.
Now if there was a SecurityManager checked method in java.lang.Module:
public void assumeAccessOf(Module module) throws SecurityException
{ ... }
...which would somehow merge the reflective access privileges of 'this'
module with privileges of given 'module'. A central SecurityManager
privileged "arbiter" (a container or launcher, for example) would then
invoke:
hibernameModule.assumeAccessOf(jpaModule);
at appropriate time. You could ask why wouldn't hibernate module just
declare that it assumes access of jpa module in its module descriptor?
Because any module could do that and gain access. It has to be an
independent privileged entity that decides and establishes trust towards
'hibernate' in this case.
Simple. Would that satisfy your requirements?
I don't like that this would (seemingly) leave the check to run time. I
agree that, ideally, if I require or use an API which requires
reflective access, if we want to ensure that the usage is not illicit,
then user/requester should somehow have to acknowledge that the grant is
allowed. I'd prefer using a simple syntax (but again I'm not going to
get into that) - something that says "use this service, and yes I give
it permission to access me".
This way the module resolve would actually fail instead of encountering
a much later run time failure.
It doesn't necessarily preclude a more specific export like you describe
though. That would be a requirement beyond what we need but not in
conflict as far as I can see. I don't want to necessitate large amounts
of declarations though. The more succinct, the better - though I'd take
a fairly verbose, working solution over a terse solution which doesn't
meet our other requirements.
Requiring a manual activation of such a grant seems undesirable (in the
same way that adding reads is) because it creates a large (and seemingly
unnecessary) burden on maintainers of existing frameworks to support
modules.
Thus, apart from the option of redesigning (to an extent) the security
mechanism (thereby eliminating the need to seal off access to public
reflection, which is definitely still an attractive option for various
reasons from our perspective, but which is also a very different
discussion), we need some sort of mechanism which decouples the literal
dependency system from access permission (much like uses/provides does).
For example if I could declare that my module uses "javax.ejb", and, in
so doing, automatically grants public and private reflective access to
the module that provides that service, this would be a good outcome. A
module which answers to that service name could be responsible for
reflective access to the application module, providing that information
privately to any other framework modules which require it.
As said in above alternative solution, this is not safe as any module
can declare that it provides any service. There has to be an "arbiter".
I agree, this seems necessary.
Also conflating service uses/provides with access privileges seems
messy. At least from the "syntactical" point of view. You would then
have to conflate declaration of service uses with packages that you wish
to export to the providers of that service. I think it's better to
express access in plain exports directives that are not explicitly tied
to service uses and leave the "binding" to the 3rd party.
Leaving the syntax stuff alone.
The migration story looks much better in this light: module descriptors
still can be quite terse and specific. Applications which use
reflective frameworks do not need gratuitous exports; in fact it's much
more fluid for a user to say "I require these helper libraries; I use
EJB; that's it" which means they don't have to worry about the details
of whatever particular environment they run in. This also has the
advantage of allowing new Java 9-generation specifications to stipulate
standard service names for each specification (e.g. "javax.ejb",
"javax.cdi", that sort of thing).
While this doesn't cover 100% of our remaining issues with Jigsaw (of
course; we'll all continue moving through the issues list as we have
been to get us there), meeting these requirements would go a long way
towards at least having a reflection story that is more practical for
present-day frameworks to move forward with. So the last requirement
would be:
• A module definition must be able to establish that an "indirect"
dependency exists on an otherwise unknown module providing a capability,
wherein that module may require public or public+private reflection
access to some or all packages without compile/link access. This could
possibly exist in conjunction with, or as an evolution of, the current
services mechanism, however a complicating factor is that the current
mechanism is based specifically on types, whereas a purely symbolic
relationship might be better for this purpose (this is not a requirement
though if it can be made to work as-is). Note that any symbolic
relationship system would need some in-code discovery mechanism such
that consumers of the capability are made available to the provider
and/or vice-versa, in order to make practical use of the relationship.
The following example syntax is meant to be unambiguous and
illustrative; no specific attempt is made to reuse existing keywords
(for example), or even to imply an endorsement of the current descriptor
mechanism at all, but to clarify how this might look in practice and
provide a practical application of the ideas herein.
Example 1: A contrived provider of the fictional framework
"javax.fictional.orm" illustrating provides/uses-based access granting
module org.foo.orm.provider {
// Require a module dependency, and give it private reflection
access to everything
requires org.apache.commons.beanutils with private reflection
on *;
// Require a module dependency with no reflection
requires org.apache.commons.logging;
// Provide the framework
provides javax.fictional.orm.ORM
using private reflection
with org.foo.orm.provider.ORMImpl1,
org.foo.orm.provider.ORMImpl2;
}
Example 2: A contrived consumer of #1
module com.mycompany.application {
uses javax.fictional.orm.ORM; // automatically gives private
reflection
}
...as said, giving access on types to unknown modules, letting any
module decide for itself whether it is "qualified" is IMO not a safe
solution.
Example 3: Grant reflection access to a couple of packages to a named
non-dependency module
module com.mycompany.application {
grant public reflection on
com.mycompay.application.package1,
com.mycompay.application.package2
to org.foo.framework;
}
...would be alternatively expressed as:
// module-info.java
module com.mycompany.application {
exports com.mycompay.application.package1 to org.foo.framework;
exports com.mycompay.application.package2 to org.foo.framework;
}
// package-info.java: trigger compiler warnings for referencing types
in package
@DynamicallyAccessed package com.mycompay.application.package1;
// package-info.java: trigger compiler warnings for referencing types
in package
@DynamicallyAccessed package com.mycompay.application.package2;
Example 4: Behave like Java 8
module com.mycompany.application {
grant private reflection on * to *;
}
...alternatively:
// trigger compiler warnings for referencing types in module
@DynamicallyAccessed
module com.mycompany.application {
exports private *; // but see above why wildcard exports might not
be a good idea
}
or:
// trigger compiler warnings for referencing types in module
@DynamicallyAccessed
weak module com.mycompany.application {
}
Example 5: Behave like Java 8, but restrict private access without
requiring a security manager
module com.mycompany.application {
grant public reflection on * to *;
}
...alternatively:
// trigger compiler warnings for referencing types in module
@DynamicallyAccessed
module com.mycompany.application {
exports *; // but see above why wildcard exports might not be a good
idea
}
there's no alternative without using widcards here.
Example 6: An example of using CDI and EJB with symbolic capabilities
module com.mycompany.application {
uses capability javax.ejb, javax.cdi
}
Example 7: An example of providing EJB with symbolic capabilities
module org.foo.ejb.provider {
[...]
provides capability javax.ejb using private reflection;
}
The language of module-info is getting very complicated, hm... Why not
simply:
@DynamicallyAccessed
module com.mycompany.application {
exports private * to javax.ejb;
exports private * to javax.cdi;
}
module javax.ejb {
...
}
module javax.cdi {
...
}
module org.foo.ejb.provider {
requires javax.ejb;
}
Then use a launcher on class-path or in a separate module that requires
all of above like:
public class Launcher {
public static void main(String ... args) {
org.foo.ejb.provider.SomeClass.class.getModule().assumeAccessOf(javax.ejb.SomeClass.class.getModule());
....
}
}
Module::assumeAccessOf could even have a corresponding java launcher
option, in the style of --add-exports etc...
Assuming one regards the necessity of --add-exports as a good thing ;-)
--
- DML
