Tracking OpenJDK 9's usage and adoption rate via Maven Central

[Martijn Verburg]

Hi all,

Several Adopt members have gotten together to look at answering the
questions of what percentage of the most popular (TBD what most popular is)
projects on Maven central can:

1. Run on Java9+
2. Use Automatic-Module-Name
3. Use module-info.java

As a side effect we're also working with Sonatype to improve Maven
central's indexing to include searching for modules.

As a follow on to this the LJC will start a crowdfunding campaign to bring
any popular libraries that aren't Java 9+ compatible up to date and
possible look at applying at least an Automatic-Module-Name as a starting
point to modularity.

See the kickoff doc

for
details.

Cheers,
Martijn

Some points on JPMS/Jigsaw after he modularised some code (from Stephen Colebourne)

[Martijn Verburg]

Hi All,

Stephen's comments are in his blog post: http://blog.joda.org/2018/03/j
pms-negative-benefits.html

He does raise some interesting points.  Perhaps it's worth spending some
time before 11 is out to see if there is a combination of Maven and JPMS
changes / clarifications that can address some of these pain points.

If Stephen is missing something, then that's an area we could document
(and/or publicise better if already documented.  That is something I and
others can help to address through the usual JUG Leaders / Champions / Info
Q etc channels.

@Stephen are you able to share the source code?  It may be that there is
another
path that could be explored to mitigate some of the challenges you faced.

@Robert - I've been thinking about the difficulty of getting enough
engineering time for Maven committers to add support for major features
like JPMS.  I'll start a separate thread with you on that.

Cheers,
Martijn

Re: Disallowing the dynamic loading of agents by default (revised)

[Martijn Verburg]

Hi all,

I've come back into this thread late, but for us (jClarity) we're OK with
the revised proposal, it will give us the time with slow moving enterprise
customers to move to an explicit Java agent approach.

Cheers,
Martijn

On 7 April 2017 at 13:53, Stephen Felts  wrote:

> I have a problem with the second point.
>
> Oracle's application server has "FAST SWAP" functionality similar to
> JRebel built-in so that a developer can speed up the "edit -> build ->
> deploy -> test" cycle.
> It uses an internal agent that attaches to the server so customers don't
> need to see it, configure it, or even know the name of it.
> It only works on application classes in an exploded ear or war file.
> This proposal will make it so the JRebel agent will work but our own agent
> won't.
>
> Ironically, this feature only works in development mode and developers are
> the ones most likely to run without any scripts having the CLASSPATH set up
> and just run java weblogic.Server.
> So as I've said many times I don't want any JDK9 command line options.
>
> In trying to protect against the guilty, you are killing the innocent.
>
> It seems to me that it's late in Phase 2 of ramp-down of JDK 9 and this
> feature is not baked or well received.  IMO no changes should be done in
> this area for JDK 9.
>
>
> -Original Message-
> From: jigsaw-dev [mailto:jigsaw-dev-boun...@openjdk.java.net] On Behalf
> Of mark.reinh...@oracle.com
> Sent: Wednesday, April 5, 2017 12:15 PM
> To: jigsaw-dev@openjdk.java.net
> Subject: Disallowing the dynamic loading of agents by default (revised)
>
> Thanks to everyone for the quick feedback on this topic, and especially to
> Andrew for the constructive dialogue.
>
> Here's a revised proposal:
>
>   - Define a new VM option, `-XX:+EnableDynamicAgentLoading`, that's
> on by default in JDK 9 but off by default in JDK 10.
>
> This will allow launch scripts that use this option on JDK 10 to
> work on JDK 9 without change, and will allow early testing of the
> JDK 10 behavior on JDK 9.
>
>   - Revise the `com.sun.tools.attach` API to forbid attachment to the
> current process or to an ancestor of the current process, and
> define a read-only system property that allows such attachment to
> be enabled via the command line.
>
> This will discourage the inadvertent use of libraries that, for
> better or for worse, intentionally violate strong encapsulation.
>
>   - Enhance the `-jar` launcher option so that if the JAR file being
> launched contains a `Premain-Class` attribute then it's launched
> as both an application and as an agent for that application.
>
> This will allow `java -jar foo.jar` to be used in place of the
> more verbose `java -javaagent:foo.jar -jar foo.jar` [1].
>
> Taken together, these changes are intended to enable the continued use of
> legitimate dynamically-loaded agents without change on JDK 9 and with a
> small change on JDK 10.  That later change will align the treatment of such
> agents with the other means of breaking encapsulation (`--add-opens`,
> etc.) in order to ensure integrity by default for all code.
>
> This proposal does not attempt to lock down platform classes as distinct
> from user classes.  Many agents have legitimate reasons to transform
> platform classes, so an additional mechanism to protect those classes does
> not appear to be worthwhile.
>
> Comments?
>
> - Mark
>
>
> [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-
> April/012000.html
>

Re: Disallowing the dynamic loading of agents by default

[Martijn Verburg]

Hi All,

We're one of the APM tooling vendors (I suspect all of us are in the same
boat) whose customers would be adversely impacted by this change.  In
effect a vital part of our tooling would not function without the proposed
flag being set.

The likely impact is that our customers would stay on Java 8 for longer
until they were prepared to add the extra CL flag or lose a significant set
of analysis features that our tooling provides.

I don't have any proposed solutions right now, but we'll do some digging
and see if we can help propose alternatives that might meet the same goal.


Cheers,
Martijn

On 30 March 2017 at 21:55, Mario Torre 
wrote:

> Hi Mark,
>
> I understand the need for consistency and I agree with your rationale.
>
> I think though that a solution that requires a command line switch for such
> cases is too restrictive.
>
> The problem is that the main users of this API are generally agents that
> are injected after a particular problem may arise. Requiring a command line
> switch like this means that for those use cases you must restart the
> applications, or effectively always turn on the feature, which practically
> invalidates the need for this restriction.
>
> I wonder if the JDK should not instead have a default serviceability agent
> that can have a runtime impact on this flag, something more restricted than
> a generic JMX agent.
>
> I already see support engineers starting to recommend go to customers :)
>
> Seriously though, it seems to me that this restriction should be better
> thought and perhaps left for later (I understand that this means people
> will need to change their command lines twice,but this solution seems very
> rushed).
>
> Cheers,
> Mario
>
> On Thu 30. Mar 2017 at 17:39,  wrote:
>
> > // Moving the general discussion to jigsaw-dev for the record;
> > // bcc'ing {hotspot-runtime,serviceability}-dev for reference.
> >
> > Andrew,
> >
> > Thanks for your feedback on this topic [1][2][3].
> >
> > First, we apologize for the way in which this topic was raised.  Our
> > intent was to post a proposal for discussion prior to code review, but
> > unfortunately that review was posted prematurely (as is evident by its
> > inclusion of Oracle-internal e-mail addresses and URLs).
> >
> > Second, I agree with your earlier analysis as to the security impact of
> > this change.  If an attack is possible via this vector then closing the
> > vector would only slow the attack, not prevent it.
> >
> > The motivation for this change is, however, not merely to improve the
> > security of the platform but to improve its integrity, which is one of
> > the principal goals of the entire modularity effort.  Integrity has
> > benefits for security and also for maintainability, of both the JDK
> > itself and of code that runs upon it, since it makes it clear exactly
> > which APIs are intended for external use, and which are not.
> >
> > To improve platform integrity we've strongly encapsulated (most) of the
> > internal APIs of the JDK.  That can be, as we all know, a source of pain
> > for developers trying to get existing applications to run on JDK 9, so
> > we've provided workarounds via the encapsulation-busting `--add-opens`
> > and `--add-exports` command-line options and, recently, the temporary
> > but more powerful (and verbose) `--permit-illegal-access` option.
> >
> > There are no API equivalents to these command-line options [4].  That's
> > intentional: We want developers and deployers using Java out-of-the-box
> > to be assured that their code is only using APIs actually intended for
> > external use.  They can choose explicitly to expose internal APIs, via
> > the command line, but then that's a deliberate choice that they make, and
> > they own the consequences.  It's reasonable to allow this via the command
> > line since we assume that anyone with access to the command line already
> > has the power to corrupt the JDK in any way they please.
> >
> > The proposal to disable the loading of dynamic agents by default is one
> > more part of the overall integrity story.  As things stand today, code in
> > any JAR file can use the `VirtualMachine.loadAgent` API to load an agent
> > into the JVM in which it's running and, via that agent, break into any
> > module it likes.  Changing the default and providing a new command-line
> > option, in this case `-XX:+EnableDyanmicAgentLoading`, is consistent
> with
> > the other encapsulation-busting options we've introduced.  It allows a
> > developer or deployer to choose to bend, if not violate, the integrity
> > of the platform, yet also to know that by default the platform is whole.
> > (Changing this default is also consistent, as Alan has noted, with the
> > original intent of the `VirtualMachine.loadAgent` API.)
> >
> > I understand your points about the practical difficulties of having to
> > educate users about this new option and enhance startup scripts to use
> > 

Jigsaw Hackday in London - Anything in particular you want us to look at?

[Martijn Verburg]

Hi all,

As part of the Adoption Group we're planning on holding a Hackday in London
for the July/Aug timeframe, primarily as an education/awareness piece
around Jigsaw (there's lots to catch up on).

Is there anything in particular you'd like a group of 'users'/developers to
look at?  I assume basing this off the latest jdk9 forest will be more
stable than the jigsaw forest.

Cheers,
Martijn

Re: Jigsaw Hackday in London - Anything in particular you want us to look at?

[Martijn Verburg]

Hi Alan,

That's brilliant, we'll set up those three broad areas and see if we can
get small groups to work on each.

Cheers,
Martijn

On 23 June 2015 at 12:10, Alan Bateman alan.bate...@oracle.com wrote:

 On 23/06/2015 10:34, Martijn Verburg wrote:

 Hi all,

 As part of the Adoption Group we're planning on holding a Hackday in
 London
 for the July/Aug timeframe, primarily as an education/awareness piece
 around Jigsaw (there's lots to catch up on).

 Is there anything in particular you'd like a group of 'users'/developers
 to
 look at?  I assume basing this off the latest jdk9 forest will be more
 stable than the jigsaw forest.

  I would expect/hope that JSR 376 will be further along by then so there
 may be more to talk about and play with at this event.

 In the mean-time there is a lot to discuss and prepare. We've been warning
 (and javac has emitted warnings) for many years that JDK-internal APIs will
 not be accessible out of the box so anything that reduces or eliminates
 dependences on internal APIs in popular libraries will be a big help. It
 might be fun to use jdeps on the say the 20 most popular libraries to get a
 feel for the problem.

 We've had the initial changes for JEP 220 in JDK 9 for about 6 months.
 This is the JEP that changes the structure of the run-time images. This has
 implications for tools (mostly) that have been used to directly accessing
 rt.jar and other internal files in the legacy images. JEP 220 comes with a
 supported interface for accessing classes and resources in the run-time
 image so that may be worth spending time on. There is a refresh of this in
 flight and it should be in by the time you have this event. It might be fun
 to try out the top 20 tools and plugins to see if they run with JDK 9 and
 maybe even hack on one or two of them to use the FileSystem API.

 Another idea is to look at the module graph in JEP 200, or better still,
 the most up to date graph of modules in the JDK 9 forests. One idea is to
 hack on make/Images.gmk in the top-level repository to force the JDK build
 to produce run-time with a subset of the modules. The primitive image
 building tool that this make file runs will be replaced by a linker tool in
 time but there is enough to hack on to get a feel for the modules and what
 might run (or not run) with a small set of modules. The module graph is
 essentially an API so any feedback or usage of that API would be useful too.

 -Alan

Re: Apache Maven JDeps Plugin

[Martijn Verburg]

From my standpoint that would be amazing - we'd always struggled with how
we're going to try and educate millions of developers about running this on
their code base.  Having it as a core report is a great start!

Cheers,
Martijn

On 20 February 2015 at 03:55, Hervé BOUTEMY herve.bout...@free.fr wrote:

 regarding failing or not failing (that's the question):

 IMHO, it would be useful to create Maven reports: as a first step, adding
 maven-jdeps-report as report will help people be aware of the state of
 their
 code and publishing it inside their site without taking any fix action at
 the
 moment

 concrete use case: I'd like to add this to the Maven parent pom to have
 jdeps
 report for every future release

 And I can even dream one step beyond: then add a dist-tool-plugin report to
 get an overview of the state of our full codebase, like [1] :)

 this dream should not be that hard to happen

 Regards,

 Hervé

 [1]
 https://builds.apache.org/view/M-R/view/Maven/job/dist-tool-plugin/site/dist-tool-prerequisites.html

 Le mardi 17 février 2015 00:04:37 Robert Scholte a écrit :
  Hi Alan,
 
  I've added a flag called failOnWarning (default:true), assuming that the
  usage of jdkinternals is considered a warning and not an error.
  With the following configuration you'll be able to run jdeps multiple
  times within the same build.
 
 plugin
   groupIdorg.apache.maven.plugins/groupId
   artifactIdmaven-jdeps-plugin/artifactId
   version3.0-SNAPSHOT/version
   executions
 execution
   idclasses/id
   goals
 goaljdkinternals/goal
 goaltest-jdkinternals/goal
   /goals
 /execution
 execution
   iddependencies/id
   goals
 goaljdkinternals/goal
 goaltest-jdkinternals/goal
   /goals
   configuration
 recursivetrue/recursive
 failOnWarningfalse/failOnWarning
   /configuration
 /execution
   /executions
 /plugin
 
  This should match your requirements.
 
  Robert
 
 
  Op Mon, 16 Feb 2015 19:45:16 +0100 schreef Alan Bateman
 
  alan.bate...@oracle.com:
   On 16/02/2015 18:28, Robert Scholte wrote:
   Hi Alan,
  
   if you are referring to the -R / -recursive option of the jdeps tool,
   then yes you can.
   See
  
 http://maven.apache.org/plugins-archives/maven-jdeps-plugin-LATEST/maven-
  jdeps-plugin/jdkinternals-mojo.html#recursive I think jdeps is first of
   all interesting for the classes of the
   current Java project, so I've set the default of this parameter to
   'false'. However, if the majority thinks it is better to activate this
   by default, we will consider to change this value.
  
   I could imagine wanting to run it twice: once for the current project
   where I want the build to fail if it makes direct use of JDK-internal
   APIs, and a second time to run with -R and emit warnings if any of the
   transitive dependences (that I don't control) are using JDK internal
   APIs.
  
   -Alan
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
  For additional commands, e-mail: dev-h...@maven.apache.org


 -
 To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
 For additional commands, e-mail: dev-h...@maven.apache.org

Re: JSR 376 (and Capsule)

[Martijn Verburg]

Hi Ron,

The mailing list to discuss jigsaw is currently this one.  The work has
started and has been split into four JEPs (JDK Enhancement Proposals) of
which 2 are complete, one is in progress and doing well and the 4th one is
still to start.

http://openjdk.java.net/projects/jigsaw/ is the summary page for the
activity to date.  You'll likely want to build from the jigsaw forest or
grab the binary builds coming out on an almost weekly basis.


Cheers,
Martijn

On 2 December 2014 at 13:15, Ron Pressler r...@paralleluniverse.co wrote:

   Hi!
 I’m the maintainer of Capsule
 , which is a simple distribution mechanism for Java applications
 (basically, executable JARs on steroids, with support for native libs,
 embedded JARs, and declared Maven dependencies). As such, I’m especially
 interested in the progress of JSR 376. I see that the spec page
  is currently empty and that there hasn’t been much discussion on the
 matter on this mailing list. Is there some draft available? Has the work
 started? Have the “comments” and “observer” mailing lists mentioned in the
 JSR been created?


 Ron

Re: JEP 220 questions

[Martijn Verburg]

Hi Alan,

Thanks - I'll confess to lack of time + ignorance/laziness I hadn't checked
how far advanced things were. Sounds like I'll get to do some solid testing
over the xmas break (don't tell my Board, or my family ;p).

Cheers,
Martijn

On 30 October 2014 20:14, Alan Bateman alan.bate...@oracle.com wrote:

 On 30/10/2014 18:58, Martijn Verburg wrote:

 :

 I note in the testing section there was reference to early access builds.
 Great and we'll work with Rory et al in QA to get that to the wider
 community as soon as possible, but I'm still uncertain whether the time
 frames will long enough to get projects to be ready for the change.

  Just a brief comment on this question.  We do hope to have EA builds
 really soon with a first cut of what is in this JEP. Most of what is
 proposed in the JEP is already in the jigsaw/m2 forest in early form so if
 you build that forest then it allows for some early testing. At this point
 in time then the only things for this JEP that aren't in the jigsaw/m2
 forest are the removal of the extension mechanism and the jrt file system
 provider (there is jimage file system provider in the current forest that
 has to evolve into the jrt file system provider -- more on that item
 soon).

 -Alan.

Re: JEP 220 questions

[Martijn Verburg]

Hi Mark,

Thanks for the clarifications - we'll start with the message at Devoxx in a
few weeks time and progress from there. I'll co-ordinate with Rory and come
back with some further questions for this list (I assume this is the right
place?) with regards to early access builds and concrete things the Jigsaw
team like to see get exercised/tested.

Cheers,
Martijn

On 30 October 2014 20:16, mark.reinh...@oracle.com wrote:

 2014/10/30 11:58 -0700, martijnverb...@gmail.com:
  I've just had a read through JEP 220 and it made for a good read!  I do
  have a couple of questions though. I apologise in advance if my archive
  searching skills failed with regards to any duplicated questions.

 No worries -- they're all good questions!

  1. The endorsed-standards override mechanism  The extension mechanism
 
  ... To help identify any existing uses of this mechanism we will modify
  the compiler and the
  launcher to fail if this system property is set or if the lib/endorsed
  directory exists...
 
  Are these tools intended to be released before Java 9/10 (e.g. Like
 jdeps)
  to help developers prepare,

 The tools are the compiler (javac) and the launcher (java), not some
 other tools.

  or will it be a deprecation warning in 9 (10)
  with view to removal in 10 (11)?

 Removal in 9.

   Most endorsed libraries I've run across
 in
  the wild do tend to be of the legacy kind and sometimes these projects
  don't have their original source code / authors / projects associated.
  Several App servers which serve the Java EE space still use the ext
  mechanism. Getting them shifted may take some time...

 Yes.  We're already aware of a couple of app servers that use the
 extension mechanism.  In many of the cases we've seen, people use that
 mechanism primarily out of laziness -- the extension JAR files can
 (nearly) as easily be placed on the regular class path.

  I note in the testing section there was reference to early access builds.
  Great and we'll work with Rory et al in QA to get that to the wider
  community as soon as possible,

 Great!  The EA builds should be available shortly.

 but I'm still uncertain whether the time
  frames will long enough to get projects to be ready for the change.

 That's part of what we're trying to discover with this exercise, as
 noted at the end of the Risks section.

  2. Removed: rt.jar and tools.jar
 
  Stating/Repeating the obvious but the major IDE manufacturers will need
 to
  get enlisted in an early access program to use the rt.jar and tools.jar
  replacements. They'll likely? have to build a parallel mechanism as
 they'll
  have to support developers working on Java (dare I say it) 6 through to
 9.

 Yes.  Rory already has a list of application and library maintainers to
 whom he'll be reaching out, and the IDEs are at the top of that list.

  I also understand that this is effectively a breaking change for = Java
 7
  developers (jrt-fs.jar support for Java 8). That's OK by me, it helps
  encourage the world to be on a modern/safe Java and it should be less of
 an
  issue by 2016 anyhow given Java 8's adoption rates.

 Agreed, but in principle it wouldn't be difficult to make jrt-fs.jar
 also work on JDK 7 if there's demand for that.  (Any earlier would be
 infeasible, since the NIO file-system API was introduced in 7.)

  3. Some of our products certainly do poke and prod at various JDK/JRE/JVM
  internals/rarer APIs, as do that of other performance tool vendors. I'm
  really not sure how all of these changes will impact us until we get the
  early access builds, if we start seeing those in Q1/2 2015 then I'm
 pretty
  confident we'll be able to figure out the new paths required in time. I
 can
  put together a list of players in the market that try to do 'interesting'
  things here as well and get them testing early also.

 That'd be very helpful.

  =
 
  Overall it looks like a major challenge will be to have the community
 test
  their projects, IDEs, App Servers early enough. I can think of a few
  surveys  research on Maven Central, BlackDuck etc that could go out to
  identify the most important projects to reach out to first. The hardest
  part will be getting to the private organisations (The iceberg under the
  water that you don't see) that aren't always tied in to the channels
 which
  the typical outreach goes to.

 Yes, and any help you can provide along these lines will be greatly
 appreciated!

 - Mark

Re: Jigsaw/Penrose contribution

[Martijn Verburg]

Hi Martin,

The best thing to do is to volunteer on jigsaw-dev (which I've CC'd in) and
see where they need help!  I'm not aware of any other adoption members
currently working on it.

Cheers,
Martijn

On 30 June 2014 22:13, Martin Toshev marto8...@gmail.com wrote:

 Hi, everyone,
 A few days ago we had a session about Jigsaw/Penrose in our local JUG
 and there has been some great interest in contributing to the projects. Is
 anyone from the Adopt OpenJDK group currently contributing to any of the
 two projects and is willing to share some experiences and guidance ?
 Thanks
 in advance.
Best regards,
Martin