Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Laurence Cable]

I think while this discussion is an interesting one, and clearly one the 
elicits strong opinions; I believe that the focus on the proposed change 
on dynamic loading of agents has truly been lost therein, further 
exchanges have little relevance to the topic of Java serviceability and 
the issue at hand.


I would encourage everyone to return their attention to the initial 
discussion topic of agent loading defaults.


Rgds

Larry Cable, Architect, Java Platform Group

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Andrew Dinn]

Hi Gregg,

Thanks for your reply. I only have one small point to make.

On 28/03/2023 16:35, Gregg Wonderly wrote:


Again, the supposition is that somehow users of software systems are always 
surrounded by version planning and management. . . .
No, actually, my supposition is that users of software systems *ought* 
to have measures in place to handle version planning and management.


The fact that many users do not, especially desktop users, merely 
reflects a widespread misunderstanding of how complex a thing it is to 
run a computer. The majority of car drivers, not being mechanically 
minded and skilled, invariably expect (even if they do not welcome) the 
regular expense required to keep their vehicles on the road. What is 
more they can pick and choose from a large number of service garages who 
will perform the necessary maintenance. Sadly, this analogy with running 
a computer breaks down not at the point where expert remedy is needed 
but where it is sought and paid for.


regards,


Andrew Dinn
---
Red Hat Distinguished Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Ron Pressler]

> On 29 Mar 2023, at 01:29, Gregg G Wonderly  wrote:
> 
> This is exactly my point!  Why would any one want to do something like this?  
> This level of workaround and specialized deployment is the kind of breakage 
> that I am referring to.  I just don’t understand how this kind of rigging and 
> customization can even start to feel right.
> 
> Gregg Wonderly

But you do understand, because you yourself have pointed out the problems 
caused by the old approach where the runtime and the application were provided 
separately. The current approach is a result of the JDK evolving to address 
those very problems, and it’s working. Embedded custom runtimes and strong 
encapsulation have greatly alleviated most of them. The alternative, a runtime 
that never changes, is only workable for very limited applications.

The old approach was guided by one primary use case, Applets, which had very 
limited capabilities. Indeed, similarly restricted JavaScript applications are 
delivered separately from their runtime, the web browser, but the more capable 
desktop applications written in JavaScript are delivered with an embedded 
runtime, for similar reasons. In other languages, including “native” ones, the 
trend is to similarly statically link dependencies — including even libc — or 
to bundle them in a container.

These aren’t workarounds, but the best means we have to date to deliver 
applications that are portable, capable, and evolvable, whether it feels right 
or not. Perhaps someday another approach will present itself.

— Ron

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Mike Hearn]

Why, well, you get more features, it's easier for the end user, and
not any harder for the developer. Those are pretty concrete reasons
why people would want to do it that way. I'd suggest trying Conveyor
out yourself before worrying about rigging or customization, because
straightforward Java apps don't actually need any configuration beyond
specifying the URL of the update site. It's easier than Web Start was,
and takes no more lines of code than configuring Gradle to make fat
JARs does. It's free for open source projects so I'd say please just
try it out with an open mind. If you still find it harder than
publishing JARs then I'd be very interested to read an essay or blog
post drilling into the differences. Maybe try following the JavaFX
tutorial here:

https://conveyor.hydraulic.dev/7.2/tutorial/hare/jvm/

Without that kind of concrete detail though, I'm going to feel like
this is about perspectives. If you view Java as a capital-p Platform,
competing on the same level as an operating system, then JARs feel
natural and bundling would feel like a retreat from the glory days. If
you perceive it as a large and fancy library then it's just like any
other library and asking the user to manage it separately makes no
more sense than expecting users to manage the Visual C++ runtime. The
switch to jlinking and bundling in this case is all win, it removes
headaches instead of adding them.

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Gregg G Wonderly]

This is exactly my point!  Why would any one want to do something like this?  
This level of workaround and specialized deployment is the kind of breakage 
that I am referring to.  I just don’t understand how this kind of rigging and 
customization can even start to feel right.

Gregg Wonderly

> On Mar 28, 2023, at 11:11 AM, Mike Hearn  wrote:
> 
> Hi Gregg,
> 
> Distributing little apps as JARs indeed doesn't work well anymore out
> of the box, but it doesn't have to be the end of the line for them.
> I've spent a couple of years writing a tool designed explicitly to
> solve all these problems [1]. You give Conveyor your JARs (or a
> Maven/Gradle build), it'll create and upload self-updating packages
> for Windows, Mac and Linux that bundle a jlinked and minified JVM,
> fully signed and notarized, along with a download HTML page for end
> users to get a big green button. It'll even draw an icon for you. You
> can do this from any OS, you don't need Windows or macOS to ship for
> them.
> 
> This approach has the major downside that unless your app is open
> source it's not free (we gotta make money somehow) BUT if you can put
> that to one side, it works better than the JAR era ever could:
> 
> - No Java compatibility issues by design.
> 
> - Not blocked by browsers/operating system security.
> 
> - Apps can update more smoothly than Web Start ever allowed.
> 
> - You can use OS specific integrations.
> 
> - Clean uninstalls, native code handled better and so on.
> 
> You might object that this is somehow more effort than just making a
> fat jar and sending it to people, but in practice it's not harder. You
> run it, out pops a bunch of files, you make them available to people,
> done.
> 
> W.R.T. corporate deployment, note that Conveyor makes MSIX files which
> are Microsoft's official format and easily deployed across Windows
> networks.
> 
> The difficulty with the send-a-JAR approach is that maintaining
> backwards compatibility at the level you want (Win32, web level) takes
> a massive level of spend, a large library of public programs which can
> be automatically regression tested against, and a commitment to never
> break anything even if it seriously disadvantages later developers,
> and even then things will still break despite best intentions. Decades
> ago this tradeoff made more sense because bandwidth and storage space
> were much tighter, but now it's harder to justify. That's why so few
> platforms do it anymore.
> 
> [1] https://hydraulic.software/

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Mike Hearn]

Hi Gregg,

Distributing little apps as JARs indeed doesn't work well anymore out
of the box, but it doesn't have to be the end of the line for them.
I've spent a couple of years writing a tool designed explicitly to
solve all these problems [1]. You give Conveyor your JARs (or a
Maven/Gradle build), it'll create and upload self-updating packages
for Windows, Mac and Linux that bundle a jlinked and minified JVM,
fully signed and notarized, along with a download HTML page for end
users to get a big green button. It'll even draw an icon for you. You
can do this from any OS, you don't need Windows or macOS to ship for
them.

This approach has the major downside that unless your app is open
source it's not free (we gotta make money somehow) BUT if you can put
that to one side, it works better than the JAR era ever could:

- No Java compatibility issues by design.

- Not blocked by browsers/operating system security.

- Apps can update more smoothly than Web Start ever allowed.

- You can use OS specific integrations.

- Clean uninstalls, native code handled better and so on.

You might object that this is somehow more effort than just making a
fat jar and sending it to people, but in practice it's not harder. You
run it, out pops a bunch of files, you make them available to people,
done.

W.R.T. corporate deployment, note that Conveyor makes MSIX files which
are Microsoft's official format and easily deployed across Windows
networks.

The difficulty with the send-a-JAR approach is that maintaining
backwards compatibility at the level you want (Win32, web level) takes
a massive level of spend, a large library of public programs which can
be automatically regression tested against, and a commitment to never
break anything even if it seriously disadvantages later developers,
and even then things will still break despite best intentions. Decades
ago this tradeoff made more sense because bandwidth and storage space
were much tighter, but now it's harder to justify. That's why so few
platforms do it anymore.

[1] https://hydraulic.software/

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Bernd]

  
  
  

	
	Gregg I don’t understand your problem, nearly all desktop java programs I know (and there are not many left) come with included runtimes. The user just does not care about Java versions. Also i dont really know any enterprise desktops which ship (only) recent plan java runtimes. Your problem with java feels very constructed or local to your organisations deficiencies. And I bet if you are not allowed to install a different runtime, you are not allowed to install a Java application without IT approval/support either. Not to mention that you certainly will not be able or allowed to dynamically Instrument them - that’s simply not a usecase for the change at hand (especially not since it’s just a config option).GreetingsBernd-- http://berndeckenfels.net

  

 Von: serviceability-dev  im Auftrag von Gregg Wonderly Gesendet: Dienstag, März 28, 2023 5:36 PMAn: Andrew Dinn Cc: Ron Pressler ; jigsaw-...@openjdk.org ; serviceability-dev@openjdk.org Betreff: Re: [External] : Re: Disallowing the dynamic loading of agents by default 

> On Mar 28, 2023, at 4:13 AM, Andrew Dinn  wrote:
> 
> Greeg,
> 
> I won't respond point by point to your comments as I cannot see any great value in doing so. I really only want to make one general comment about your account below, which is that you appear to me to be relaying your own experience as a desktop Java user and universalising it to all users and uses of Java. While I acknowledge that you are correct to state that there can be problems with maintaining a consistent desktop setup for Java I'll counter that with two important qualifying arguments which undercut your account.

Again, the supposition is that somehow users of software systems are always surrounded by version planning and management.  This is just not the case. People use software for its functionality not because the platform has certain features.  The platform is what enables software systems to be constructed and used.  When the platform changes it’s feature set or design and removes features or disables the default use of those features, that lack of backward compatibility, when unmanaged by the platform, creates havoc for users and software developers who are providing software systems in vertical markets or into communities of users who are not educated or knowledgable enough to understand what any of these words actually refer to. 

> Firstly, the problems that you describe are general ones that apply when it comes to software configuration management for a highly user-specific and often, in consequence, highly variable environment like a desktop. They are not problems that are specific to Java or other language runtimes Even within that category many runtimes, especially language runtimes, suffer from problems with installed version mismatches and in many cases these have been notably far worse problems than Java (as for example the Python 2/3 fiasco mentioned by Andrew Haley). However, that is not to say that the problem lies with the runtime itself.

Your supposition is again that somehow everyone has the benefit of having a guru or appointed party that manages all of their software and versioning needs.  I used the example of corporate users to draw on that environments “random” upgrade timelines.  Users get new machines as something fails or get new versions of software based on “bugs” or “risks” or other drivers that are completely unrelated to software development timelines of third party suppliers of software systems.

Suggesting that just because this has happened historically, that somehow it provides the relief to do it as well is the point I am trying to counter with my arguments here.

> Version management problems cannot simply be resolved by preserving Java (or any other deployed software) in aspic. At least some minimal level of upgrade of a runtime like Java is needed to deal with emerging security and critical functional problems. However, in the longer term any platform will also need to incorporate larger scale modifications in order to cater for the continuous, dramatic change that we have seen and continue to see in all hardware and operating systems. Java has not stood still over the last 25 years for very good reasons.

Removing features and changing how the platform handles features such that software deployment or software systems have to change is the problem.  There are so many things that have been changed to no real advantage to developers.  Java 9 changes for “modularity” were really only targeted changes to try and manage people plugging into private APIs, because the public APIs were not providing needed features and it was difficult or impossible to use the platform without those changes.  This speaks to the ineffective nature of the platform management overall.  Historically, and I assert this over and over, because I was in the room where it happened, Sun was only interested in selling servers.  Oracle is a

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Gregg Wonderly]

> On Mar 28, 2023, at 4:13 AM, Andrew Dinn  wrote:
> 
> Greeg,
> 
> I won't respond point by point to your comments as I cannot see any great 
> value in doing so. I really only want to make one general comment about your 
> account below, which is that you appear to me to be relaying your own 
> experience as a desktop Java user and universalising it to all users and uses 
> of Java. While I acknowledge that you are correct to state that there can be 
> problems with maintaining a consistent desktop setup for Java I'll counter 
> that with two important qualifying arguments which undercut your account.

Again, the supposition is that somehow users of software systems are always 
surrounded by version planning and management.  This is just not the case. 
People use software for its functionality not because the platform has certain 
features.  The platform is what enables software systems to be constructed and 
used.  When the platform changes it’s feature set or design and removes 
features or disables the default use of those features, that lack of backward 
compatibility, when unmanaged by the platform, creates havoc for users and 
software developers who are providing software systems in vertical markets or 
into communities of users who are not educated or knowledgable enough to 
understand what any of these words actually refer to. 

> Firstly, the problems that you describe are general ones that apply when it 
> comes to software configuration management for a highly user-specific and 
> often, in consequence, highly variable environment like a desktop. They are 
> not problems that are specific to Java or other language runtimes. Even 
> within that category many runtimes, especially language runtimes, suffer from 
> problems with installed version mismatches and in many cases these have been 
> notably far worse problems than Java (as for example the Python 2/3 fiasco 
> mentioned by Andrew Haley). However, that is not to say that the problem lies 
> with the runtime itself.

Your supposition is again that somehow everyone has the benefit of having a 
guru or appointed party that manages all of their software and versioning 
needs.  I used the example of corporate users to draw on that environments 
“random” upgrade timelines.  Users get new machines as something fails or get 
new versions of software based on “bugs” or “risks” or other drivers that are 
completely unrelated to software development timelines of third party suppliers 
of software systems.

Suggesting that just because this has happened historically, that somehow it 
provides the relief to do it as well is the point I am trying to counter with 
my arguments here.

> Version management problems cannot simply be resolved by preserving Java (or 
> any other deployed software) in aspic. At least some minimal level of upgrade 
> of a runtime like Java is needed to deal with emerging security and critical 
> functional problems. However, in the longer term any platform will also need 
> to incorporate larger scale modifications in order to cater for the 
> continuous, dramatic change that we have seen and continue to see in all 
> hardware and operating systems. Java has not stood still over the last 25 
> years for very good reasons.

Removing features and changing how the platform handles features such that 
software deployment or software systems have to change is the problem.  There 
are so many things that have been changed to no real advantage to developers.  
Java 9 changes for “modularity” were really only targeted changes to try and 
manage people plugging into private APIs, because the public APIs were not 
providing needed features and it was difficult or impossible to use the 
platform without those changes.  This speaks to the ineffective nature of the 
platform management overall.  Historically, and I assert this over and over, 
because I was in the room where it happened, Sun was only interested in selling 
servers.  Oracle is again only interested in selling server/backend solutions 
and support.  The OpenJDK community is, in fact providing a great service to 
the community in maintaining JDK versions that avoid forcing the user into 
incompatible places.  

> Your reply suggests that you are unaware of the reality that those who manage 
> non-desktop deployments plan very carefully around this need to adapt 
> deployments to updates. Your lament that (your and others') desktop 
> management does not include such provision may reflect the reality of some 
> (but definitely not all) individuals or organizations. However, that lack of 
> provision attests not to any failing on the part of the developers of Java 
> but rather to a lack of organization, understanding and adequate preparation 
> for *necessary maintenance* on the part of those responsible for managing 
> said desktops.

I’ve done backend development with lots of different languages for decades.  
I’ve been subjected to Windows versioning, Linux Versioning (R

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Ron Pressler]

Applications can now control the Java version available to them (and this is 
something we’ll keep improving), and the JRE, the centralised Java environment 
installed on user machines, has not existed for some time.
Strong encapsulation (which this change is part of) has made compatibility 
better than it’s ever been.

It is true that we cannot now fix the compatibility issues between Java 1.2 and 
1.5, but rather than the reality not being known to decision makers, it looks 
like you may be unfamiliar with the changes made by decision makers (a few 
years ago) to address some of the very issued you raised.

— Ron

> On 28 Mar 2023, at 04:34, Gregg Wonderly  wrote:
> 
> On Mar 27, 2023, at 11:30 AM, Andrew Dinn  wrote:
>> If this is pushed in jdk21 then anyone currently developing or upgrading an 
>> app to target jdk21 will only have been able to test on jdk17-jdk20 where 
>> they will not encounter the issue. So, his nly leaves them a small window to 
>> detect that there will be a problem using agents in jdk21. When jdk21 
>> arrives this may force them to delay deployment or they may even deploy 
>> unaware that the problem exists.
>> 
>> If this is pushed in jdk22 instead of jdk21 then anyone who upgrades from 
>> jdk17 to jdk21 will not have a problem. Anyone working on an app for 
>> deployment on jdk25 will have the opportunity to test on 3 non_LTS releases 
>> which might manifest the potential agent problem before deployment.
> 
> This is, again, where the reality that some Java users live in is very 
> different than what seems to be known my many decision makers here.  Most 
> corporate users of Java don’t control when a particular version of Java is 
> deployed into their environments.  It keeps being proposed, that somehow 
> users are deploying a specific version of Java and getting an appropriate 
> version of an application they use, all at one moment in time.  The 
> supposition that Java is “deployed” with a particular software system that 
> uses it, is summarily false.  Even Linux releases by Redhat for RHEL, Centos 
> or even Fedora don’t let you pick any and every version of Java. Java 
> applications, by the millions were written without needing any particular 
> vision of Java, until a version broke something major like starting at Java2 
> (1.2) and then Java1.4 which need a dozen fixes and then Java 1.5 that broke 
> huge numbers of desktop apps that did not that had not used volatile class 
> values for so many things, including loop control values that kept loops from 
> exiting.  Then we had 1.9 that almost went out the door disabling every 
> single Spring app in existence.  And there are more and more things happening 
> that just do not make much sense in the grand scheme of things.  
> 
> Overall Java is just not a safe place to take people for the first time.  
> Many have had horrid problems and given up on Java.  For Java1.2, Perforce 
> invested huge time to try and create a new desktop app for their SCM system.  
> They got to the point of almost releasing a beta to testers and then 
> summarily threw it all away because they just could not make it work for all 
> the things that got broke in 1.2.  At my business, we have lots of each 
> device applications where it would be a good thing to use, but because of the 
> breakage and issues others have experience with Java over the years, their 
> experiences cause them to just say no to anything Java.
> 
> Java is just randomly upgraded on peoples desktops in their view.  It gets 
> replaced by the IT staff at most corps at unrelated moments that they start 
> using a particular Java application.  Those corps and their IT staff have 
> little to no knowledge of what every Java application is let alone how it 
> might be dependent on features that are being changed at each release of the 
> JDK/JVM. 
> 
> The end result is that it is a surprise, always for this class of user, which 
> version of Java will be available and which application will break this time.
> 
> I still have lots of “jar” file applications that I share with others and 
> they just double click on those to run them.  It’s that class of user that 
> this Java upgrades happen with software updates/distributions process 
> completely overlooks.  The Java licensing was always about you could not use 
> Java as the sole application platform on a computer.  So, all kinds of “free” 
> desktop apps (and applets and Java Web Start) applications were created and 
> used by literally thousands of users that are completely out of sight.
> 
> I continue to see massive migration away from Java as the first choice for 
> new applications amongst developers I talk to. It’s not being taught to most 
> new developers I meet.  They hardly even know that Java exists or what it’s 
> capable of.  Most developers seem to be taught web front end development 
> tools or .net or golang or other languages besides Java for backend dev.
> 
> There seems to be little chance that J

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Andrew Dinn]

Greeg,

I won't respond point by point to your comments as I cannot see any 
great value in doing so. I really only want to make one general comment 
about your account below, which is that you appear to me to be relaying 
your own experience as a desktop Java user and universalising it to all 
users and uses of Java. While I acknowledge that you are correct to 
state that there can be problems with maintaining a consistent desktop 
setup for Java I'll counter that with two important qualifying arguments 
which undercut your account.


Firstly, the problems that you describe are general ones that apply when 
it comes to software configuration management for a highly user-specific 
and often, in consequence, highly variable environment like a desktop. 
They are not problems that are specific to Java or other language 
runtimes. Even within that category many runtimes, especially language 
runtimes, suffer from problems with installed version mismatches and in 
many cases these have been notably far worse problems than Java (as for 
example the Python 2/3 fiasco mentioned by Andrew Haley). However, that 
is not to say that the problem lies with the runtime itself.


Version management problems cannot simply be resolved by preserving Java 
(or any other deployed software) in aspic. At least some minimal level 
of upgrade of a runtime like Java is needed to deal with emerging 
security and critical functional problems. However, in the longer term 
any platform will also need to incorporate larger scale modifications in 
order to cater for the continuous, dramatic change that we have seen and 
continue to see in all hardware and operating systems. Java has not 
stood still over the last 25 years for very good reasons.


Your reply suggests that you are unaware of the reality that those who 
manage non-desktop deployments plan very carefully around this need to 
adapt deployments to updates. Your lament that (your and others') 
desktop management does not include such provision may reflect the 
reality of some (but definitely not all) individuals or organizations. 
However, that lack of provision attests not to any failing on the part 
of the developers of Java but rather to a lack of organization, 
understanding and adequate preparation for *necessary maintenance* on 
the part of those responsible for managing said desktops.


Which brings us to the second point: your complaint omits to allow for 
the enormous efforts that Java developers perform to enable Java users 
to rely on and profit from exactly the sort of continuity that you 
misguidedly claim Java does not provide. We are currently maintaining 
reliable, secure and bug free versions of jdk8, jdk11, jdk17 which allow 
users to continue to run applications that were originally deployed many 
years ago and will do so for many years to come. Indeed, as with many 
other large-scale, organized open source software infrastructure 
projects, this is the primary focus of the OpenJDK team. The number of 
people involved in maintaining legacy releases of Java to support 
existing users far outweighs those involved in developing new releases 
and new features.


Users who put in the work needed to manage the configuration of their 
desktop environments can easily use these legacy releases to maintain 
their own desktop applications. It's not a free lunch -- admins of the 
desktop systems need to have at least a moderate understanding of how to 
configure their systems in order to maintain applications that rely on a 
specific Java platform release. However, to claim that the OpenJDK devs 
have not made this possible, worse to claim that Java has actually 
poisoned the well for desktop users, is a ridiculous and ignorant assertion.


regards,

Andrew Dinn
---
Red Hat Distinguished Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill

On 28/03/2023 04:34, Gregg Wonderly wrote:

On Mar 27, 2023, at 11:30 AM, Andrew Dinn  wrote:

If this is pushed in jdk21 then anyone currently developing or upgrading an app 
to target jdk21 will only have been able to test on jdk17-jdk20 where they will 
not encounter the issue. So, his nly leaves them a small window to detect that 
there will be a problem using agents in jdk21. When jdk21 arrives this may 
force them to delay deployment or they may even deploy unaware that the problem 
exists.

If this is pushed in jdk22 instead of jdk21 then anyone who upgrades from jdk17 
to jdk21 will not have a problem. Anyone working on an app for deployment on 
jdk25 will have the opportunity to test on 3 non_LTS releases which might 
manifest the potential agent problem before deployment.


This is, again, where the reality that some Java users live in is very 
different than what seems to be known my many decision makers here.  Most 
corporate users of Java don’t control when a particular version of Java is 
deployed into their enviro

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Gregg Wonderly]

On Mar 27, 2023, at 11:30 AM, Andrew Dinn  wrote:
> If this is pushed in jdk21 then anyone currently developing or upgrading an 
> app to target jdk21 will only have been able to test on jdk17-jdk20 where 
> they will not encounter the issue. So, his nly leaves them a small window to 
> detect that there will be a problem using agents in jdk21. When jdk21 arrives 
> this may force them to delay deployment or they may even deploy unaware that 
> the problem exists.
> 
> If this is pushed in jdk22 instead of jdk21 then anyone who upgrades from 
> jdk17 to jdk21 will not have a problem. Anyone working on an app for 
> deployment on jdk25 will have the opportunity to test on 3 non_LTS releases 
> which might manifest the potential agent problem before deployment.

This is, again, where the reality that some Java users live in is very 
different than what seems to be known my many decision makers here.  Most 
corporate users of Java don’t control when a particular version of Java is 
deployed into their environments.  It keeps being proposed, that somehow users 
are deploying a specific version of Java and getting an appropriate version of 
an application they use, all at one moment in time.  The supposition that Java 
is “deployed” with a particular software system that uses it, is summarily 
false.  Even Linux releases by Redhat for RHEL, Centos or even Fedora don’t let 
you pick any and every version of Java. Java applications, by the millions were 
written without needing any particular vision of Java, until a version broke 
something major like starting at Java2 (1.2) and then Java1.4 which need a 
dozen fixes and then Java 1.5 that broke huge numbers of desktop apps that did 
not that had not used volatile class values for so many things, including loop 
control values that kept loops from exiting.  Then we had 1.9 that almost went 
out the door disabling every single Spring app in existence.  And there are 
more and more things happening that just do not make much sense in the grand 
scheme of things.  

Overall Java is just not a safe place to take people for the first time.  Many 
have had horrid problems and given up on Java.  For Java1.2, Perforce invested 
huge time to try and create a new desktop app for their SCM system.  They got 
to the point of almost releasing a beta to testers and then summarily threw it 
all away because they just could not make it work for all the things that got 
broke in 1.2.  At my business, we have lots of each device applications where 
it would be a good thing to use, but because of the breakage and issues others 
have experience with Java over the years, their experiences cause them to just 
say no to anything Java.

Java is just randomly upgraded on peoples desktops in their view.  It gets 
replaced by the IT staff at most corps at unrelated moments that they start 
using a particular Java application.  Those corps and their IT staff have 
little to no knowledge of what every Java application is let alone how it might 
be dependent on features that are being changed at each release of the JDK/JVM. 

The end result is that it is a surprise, always for this class of user, which 
version of Java will be available and which application will break this time.

I still have lots of “jar” file applications that I share with others and they 
just double click on those to run them.  It’s that class of user that this Java 
upgrades happen with software updates/distributions process completely 
overlooks.  The Java licensing was always about you could not use Java as the 
sole application platform on a computer.  So, all kinds of “free” desktop apps 
(and applets and Java Web Start) applications were created and used by 
literally thousands of users that are completely out of sight.

I continue to see massive migration away from Java as the first choice for new 
applications amongst developers I talk to. It’s not being taught to most new 
developers I meet.  They hardly even know that Java exists or what it’s capable 
of.  Most developers seem to be taught web front end development tools or .net 
or golang or other languages besides Java for backend dev.

There seems to be little chance that Java will have a place in the landscape 
within the next 5 years or so as those who have used Java since the mid 1990s 
age out of the pool of active developers and are no longer influencing tech 
used.

What a sad tale the last 10 years of Java has been compared to what was 
possible 25 years ago, and should of happened…

Gregg Wonderly 

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Andrew Dinn]

Hi Ron,

Thanks for the reply, I believe we have established a lot of common 
ground here. I'll try to clarify a couple of the things you found 
difficult to follow.


On 27/03/2023 11:32, Ron Pressler wrote:


  - A second, related concern is that flipping the default for this 
configuration in an LTS release as the first exposure to it for most people is 
more likely to derail deployment plans for users than if the default were 
flipped in a non-LTS release. If this change were deferred to jdk22 then that 
would give those planning deployment on (or upgrade to) jdk25 and also those 
planning to upgrade from jdk17 to jdk21 more time to discover and respond to 
the change.


While we should certainly discuss timing after publishing the draft JEP, I’m 
not sure how relevant this particular argument is. Those who upgrade from 17 to 
21 don’t care which of the versions they skipped introduced a change, and even 
the deprecation process does not take into account versions for which Oracle 
and other vendors choose to offer an LTS service. JDK 17 itself also introduced 
a far bigger tightening of strong encapsulation than the one discussed here. 
Furthermore, those who wish to upgrade from one version that has LTS offerings 
to another avail themselves of the LTS service to upgrade not immediately when 
the new version is released, so they are under no time pressure.


It seems to me to be a fairly simple point but I obviously didn't 
express it very well. Here's another try.


If this is pushed in jdk21 then anyone currently developing or upgrading 
an app to target jdk21 will only have been able to test on jdk17-jdk20 
where they will not encounter the issue. So, his nly leaves them a small 
window to detect that there will be a problem using agents in jdk21. 
When jdk21 arrives this may force them to delay deployment or they may 
even deploy unaware that the problem exists.


If this is pushed in jdk22 instead of jdk21 then anyone who upgrades 
from jdk17 to jdk21 will not have a problem. Anyone working on an app 
for deployment on jdk25 will have the opportunity to test on 3 non_LTS 
releases which might manifest the potential agent problem before deployment.


I hope that explains the problem better.


  - A third concern, already pointed out by Volker, is that some users may run 
their Java apps via launcher apps or scripts that mask access to the Java 
command line. For such users the change of default may mean that they lose the 
option to deploy dynamic agents for important ancillary tasks such as 
observability. We are not clear how many of our users this affects but we will 
be looking into this and hope to bring feedback to the JEP review.
  Obviously, this problem can be remedied relatively easily by the supplier of 
the launcher enabling agent use or providing a suitable control switch. Our 
concern is not with how to solve this problem rather how the involvement of two 
parties, supplier and end user, might imply a need for the JEP to be targeted 
to a later release.


This, too, is an argument that’s hard for me to understand. First, many JDK 
releases require changes to the command line, for various reasons. JDK 17 
required bigger changes than the one announced here, and JDK 21 itself may well 
require other such changes that impact even more applications than this one — 
making it an opportunity rather than a liability. Second, such changes are 
normally announced *later* than this one has. If an application under such 
constraints always uses the current JDK release, then surely a six-month notice 
is enough, and if it opts to use an LTS service, then it’s under no pressure.
Of course, I accept that changes to command line options are nothing 
new. However, I don't quite see how to get from there to the implication 
that this specific change cannot therefore raise concerns. I think the 
truth of that conclusion has to depend on details of the change, 
specifically what the effect might be on users.


regards,


Andrew Dinn
---
Red Hat Distinguished Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Andrew Dinn]

Sorry, let me correct some of my mangled grammar

On 27/03/2023 17:06, Andrew Dinn wrote:

On 27/03/2023 13:37, Volker Simonis wrote:
The JEP itself is rather ambitious because I noticed that most of the 
strong encapsulation JEPs lacked a substantial motivation section 
(largely because they had been written before that became the norm) 
and so strong encapsulation has not been motivated in JEP form, and 
this is an opportunity to start rectifying that. This is important 
because some, even in this discussion, are under the impression that 
integrity is about security. Although security is certainly one of 
integrity’s impacts (albeit not in the way hypothesised in this 
discussion) other major impacts are on performance (including, though 
certainly not limited to, link-time optimisations that are of 
interest to Project Leyden) and code evolution (the lack of integrity 
has been, by far, the biggest cause of JDK upgrade issues experienced 
by many applications an libraries).


Thanks for pointing out that "integritiy" and "security" are two
different things and that this discussion is mostly about other
aspects of "integrity" like performance and code evaluation. This is
actually exactly why I already tried to ask several times about the
"real" and/or the long term background of this change. Currently the
JEP only seems to propose the change of the default value for dynamic
agent loading. It is obviously not hard for other JDK vendors to use a
different default and I agree that it is probably still manageable
(though inconvenient) for administrators/operators to change the
default at launch time. BUT, you rightly mention that once higher
integrity is the DEFAULT, this opens the door for future optimizations
(you listed some of them) and even completely different execution
models / semantics for Java applications (e.g. as explored by Project
Leyden). Once such new optimizations will be in place (and only work
for the default, disabled dynamic agent loading setting) it will be
much harder for users who depend on dynamic loading to enable it,
because it will either impact their performance or it will limit their
ability to use certain platform features.


Red Hat's team had discusssed this issue and we did think about raising 
it in the previous reply. It is perhaps better left for the JEP 
discussion but we certainly consider it as important an issue as Volker 
does.


Personally, I will note that this is a point I have been concerned 
by since the associated dilemma was raised by Mark Reinhold during the 
original discussion. Indeed, the possibility that retaining agents might 
necessitate a divergence in JVM (or even JDK) behaviour from the status 
quo when no agent is, nor can be, installed struck me very hard. My 
take-away was not an immediate abreaction. Rather: the opportunity 
agents provide for dynamic adaptation of the runtime is provisional and, 
needs to be balanced against the benefits that might accrue from them 
not being the picture; in the longer term some of the capabilities 
provided through the use of agents may be unsustainable.


As an example of that I considered my agent, Byteman. It is immensely 
useful for injecting faults, validity assertions and monitoring 
capabilities into app code during unit system testing, avoiding the need 
for that code to actually appear in the product. The precise targeting 
of Byteman rules to specific code locations means that the injected code 
only minimally perturbs the whole code base.


Notably, Byteman does not support bulk (online) transformation. Too much 
transformed code means you are effectively testing a very different app 
to the one you deploy, at least as far as execution speed, timings, 
resource use etc are concerned, and possibly even because of too radical 
a change to the app semantics.


Clearly, if simply loading an agent removes opportunities 
for the runtime to operate more 
efficiently and, as a result, causes a significant change in execution 
speed, timings, resource use then a tool like Byteman becomes much less 
useful for this sort of testing. So, for me the writing was already on 
the wall back in jdk9 time. We may well have to trade off of some 
build/test time benefits against the undeniable impetus to improve 
deploy-time performance.



My main concern with the proposed change is not the current proposal
but the impact it will have on the evolution of Java. Java's dynamic
features are one of its biggest strength and a major reason for its
success. Sacrificing some of them or making their usage increasingly
expensive requires a broader discussion in the community and shouldn't
happen "under the hood". I'm happy to continue that discussion on the
actual JEP proposal.

Amen to that.

regards,


Andrew Dinn
---



--
regards,


Andrew Dinn
---
Red Hat Distinguished Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mik

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Andrew Dinn]

On 27/03/2023 13:37, Volker Simonis wrote:

The JEP itself is rather ambitious because I noticed that most of the strong 
encapsulation JEPs lacked a substantial motivation section (largely because 
they had been written before that became the norm) and so strong encapsulation 
has not been motivated in JEP form, and this is an opportunity to start 
rectifying that. This is important because some, even in this discussion, are 
under the impression that integrity is about security. Although security is 
certainly one of integrity’s impacts (albeit not in the way hypothesised in 
this discussion) other major impacts are on performance (including, though 
certainly not limited to, link-time optimisations that are of interest to 
Project Leyden) and code evolution (the lack of integrity has been, by far, the 
biggest cause of JDK upgrade issues experienced by many applications an 
libraries).


Thanks for pointing out that "integritiy" and "security" are two
different things and that this discussion is mostly about other
aspects of "integrity" like performance and code evaluation. This is
actually exactly why I already tried to ask several times about the
"real" and/or the long term background of this change. Currently the
JEP only seems to propose the change of the default value for dynamic
agent loading. It is obviously not hard for other JDK vendors to use a
different default and I agree that it is probably still manageable
(though inconvenient) for administrators/operators to change the
default at launch time. BUT, you rightly mention that once higher
integrity is the DEFAULT, this opens the door for future optimizations
(you listed some of them) and even completely different execution
models / semantics for Java applications (e.g. as explored by Project
Leyden). Once such new optimizations will be in place (and only work
for the default, disabled dynamic agent loading setting) it will be
much harder for users who depend on dynamic loading to enable it,
because it will either impact their performance or it will limit their
ability to use certain platform features.


Red Hat's team had discusssed this issue and we did think about raising 
it in the previous reply. It is perhaps better left for the JEP 
discussion but we certainly consider it as important an issue as Volker 
does.


Personally, I will note that this has been point I have been concerned 
by since the associated dilemma was raised by Mark Reinhold during the 
original discussion. Indeed, the possibility that retaining agents might 
necessitate a divergence in JVM (or even JDK) behaviour from the status 
quo when no agent is, nor can be, installed struck me very hard. My 
take-away was not an immediate abreaction. Rather: the opportunity 
agents provide for dynamic adaptation of the runtime is provisional and, 
needs to be balanced against the benefits that might accrue from them 
not being the picture; in the longer term some of the capabilities 
provided through the use of agents may be unsustainable.


As an example of that I considered my agent, Byteman. It is immensely 
useful for injecting faults, validity assertions and monitoring 
capabilities into app code during unit system testing, avoiding the need 
for that code to actually appear in the product. The precise targeting 
of Byteman rules to specific code locations means that the injected code 
only minimally perturbs the whole code base.


Notably, Byteman does not support bulk (online) transformation. Too much 
transformed code means you are effectively testing a very different app 
to the one you deploy, at least as far as execution speed, timings, 
resource use etc are concerned, and possibly even because of too radical 
a change to the app semantics.


Clearly, if simply loading an agent provides opportunities may 
eventually lead to for the runtime being able to operate more 
efficiently and, as a result, cause in a significant change in execution 
speed, timings, resource use then a tool like Byteman becomes much less 
useful for this sort of testing. So, for me the writing was already on 
the wall back in jdk9 time. We may well have to trade off of some 
build/test time benefits against the undeniable impetus to improve 
deploy-time performance.



My main concern with the proposed change is not the current proposal
but the impact it will have on the evolution of Java. Java's dynamic
features are one of its biggest strength and a major reason for its
success. Sacrificing some of them or making their usage increasingly
expensive requires a broader discussion in the community and shouldn't
happen "under the hood". I'm happy to continue that discussion on the
actual JEP proposal.

Amen to that.

regards,


Andrew Dinn
---

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Volker Simonis]

On Mon, Mar 27, 2023 at 12:32 PM Ron Pressler  wrote:
>
> Hi Andrew!
>
> > On 24 Mar 2023, at 17:21, Andrew Dinn  wrote:
> >
> > Hi Ron,
> >
> > Thank you for providing a heads up on the proposed JEP. The Red Hat Java 
> > team have been discussing this proposal. We have reviewed the original 
> > discussion and also the surrounding debate which established requirements 
> > for adaptation of Jigsaw to incorporate the needs of agents.
> >
> > As an aside, I'll note that a thorough review was necessary /even/ in my 
> > case, despite the fact that I was an active party, because the discussion 
> > occurred, and corresponding decisions were made, quite some time ago. I 
> > mention this because it may explain the air of surprise and the desire to 
> > reiterate some of the original debate on the part of some respondents in 
> > this thread, who perhaps were not party, or only tangentially party, to the 
> > discussion.
> >
> > That also suggests that there may be a lot users who are not aware that the 
> > -XX:+EnableDynamicAgentLoading switch exists or do not really understand 
> > why it exists i.e. that there is a broad education issue at play here.
>
> Understood.
>
> >
> > We do have some concerns about the JEP, specifically about the timing of 
> > its delivery. These are probably best addressed via the normal review 
> > process. In particular that will ensure the discussion happens in a more 
> > suitable and more widely subscribed forum than the Jigsaw list. However, I 
> > will briefly mention our concerns in this reply. Before that let me start 
> > with a few disclaimers:
> >
> > - We acknowledge that there is little to be gained from re-iterating 
> > arguments made in the previous discussion (although that does not imply the 
> > JEP review would not benefit from new arguments, especially from those who 
> > were not involved in that discussion)
> >
> >  - We recognize that the purpose of the -XX:+EnableDynamicAgentLoading 
> > switch is to offer a platform integrity guarantee and that this change of 
> > the default reflects a desire to prioritise integrity over the flexibility 
> > that agents provide
>
> I would qualify that: we want to prioritise integrity *by default*. Integrity 
> is only practical when it is the default, as adding more integrity rules 
> after the fact is hard to the point of being impractical. Agents in 
> themselves don’t impact that default, but dynamically loaded agents do.
>
> >
> >  - We recognize that the proposal is only proposing to flip a configuration 
> > default rather than detract from (or modify) available functionality
> >
> >  - We recognize that changing this default will still allow (*most*) users 
> > to configure the behaviour they desire
> >
> >  - We recognize that this advance notice has been given precisely to ensure 
> > that anyone wishing to deploy on jdk21 an app that relies on use of agents 
> > has time to plan appropriate configuration for their deployment
> >
> >  - We recognize that this change of default is not being proposed for 
> > backport and hence that it will largely only affect the relatively small 
> > number of users who are currently developing for jdk21+
> >
> > So, given that as a base for our comments where is the beef?
> >
> >  - Our main concern is, predictably, timing. Clearly, this is a future, 
> > potential problem rather than a present problem - no one can be deploying 
> > on jdk21 yet and most developers who are currently preparing an app for 
> > deployment on jdk21+ will likely encounter the effect of this change before 
> > actual deployment and be in a position to remedy it. The concern is that 
> > advertising a change like this and getting users prepared to respond to it 
> > has always been difficult to achieve. In particular we expect a long tail 
> > of support problems from users who are trying to upgrade deployments from 
> > earlier releases to jdk21.
> >  So, while it is nice to have such early notice of the proposal we plan to 
> > review its likely impact on our users and how much time we need to prepare 
> > ourselves and our users to negotiate this change in behaviour. Any evidence 
> > we obtain to suggest a delay in targeting is appropriate will be brought to 
> > the JEP review.
>
> Very well, we can certainly discuss timing as part of the JEP discussion.
>
> The JEP itself is rather ambitious because I noticed that most of the strong 
> encapsulation JEPs lacked a substantial motivation section (largely because 
> they had been written before that became the norm) and so strong 
> encapsulation has not been motivated in JEP form, and this is an opportunity 
> to start rectifying that. This is important because some, even in this 
> discussion, are under the impression that integrity is about security. 
> Although security is certainly one of integrity’s impacts (albeit not in the 
> way hypothesised in this discussion) other major impacts are on performance 
> (including, though certainly not 

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Ron Pressler]

> On 25 Mar 2023, at 19:56, Gregg G Wonderly  wrote:
> 
> I understand you may have personal experiences with how you use Java.  In my 
> experience and others, Java has constantly had fundamental breakage in 
> various details due to lack of understanding, on the platform development 
> team(s) of what people actually do with Java.
> 
> Sent from my iPhone


Since at this point it is an established fact that the vast majority of JDK 
upgrade difficulties (certainly weighed by cost) have been the result of lack 
of integrity guarantees made by strong encapsulation, those who care about 
stability should welcome this change and the others to follow that will close 
the remaining loopholes in strong encapsulation.

Strong encapsulation has already proven itself over the past several releases 
in allowing us to make big internal changes to the platform (such as those 
required for virtual threads) with little adverse effect on compatibility, 
something that wasn’t possible before. The success of strong encapsulation in 
that area shows that it’s the right approach for letting Java evolve, even in 
very significant ways, and at the same time reduce upgrade costs.

— Ron

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Ron Pressler]

Hi Andrew!

> On 24 Mar 2023, at 17:21, Andrew Dinn  wrote:
> 
> Hi Ron,
> 
> Thank you for providing a heads up on the proposed JEP. The Red Hat Java team 
> have been discussing this proposal. We have reviewed the original discussion 
> and also the surrounding debate which established requirements for adaptation 
> of Jigsaw to incorporate the needs of agents.
> 
> As an aside, I'll note that a thorough review was necessary /even/ in my 
> case, despite the fact that I was an active party, because the discussion 
> occurred, and corresponding decisions were made, quite some time ago. I 
> mention this because it may explain the air of surprise and the desire to 
> reiterate some of the original debate on the part of some respondents in this 
> thread, who perhaps were not party, or only tangentially party, to the 
> discussion.
> 
> That also suggests that there may be a lot users who are not aware that the 
> -XX:+EnableDynamicAgentLoading switch exists or do not really understand why 
> it exists i.e. that there is a broad education issue at play here.

Understood.

> 
> We do have some concerns about the JEP, specifically about the timing of its 
> delivery. These are probably best addressed via the normal review process. In 
> particular that will ensure the discussion happens in a more suitable and 
> more widely subscribed forum than the Jigsaw list. However, I will briefly 
> mention our concerns in this reply. Before that let me start with a few 
> disclaimers:
> 
> - We acknowledge that there is little to be gained from re-iterating 
> arguments made in the previous discussion (although that does not imply the 
> JEP review would not benefit from new arguments, especially from those who 
> were not involved in that discussion)
> 
>  - We recognize that the purpose of the -XX:+EnableDynamicAgentLoading switch 
> is to offer a platform integrity guarantee and that this change of the 
> default reflects a desire to prioritise integrity over the flexibility that 
> agents provide

I would qualify that: we want to prioritise integrity *by default*. Integrity 
is only practical when it is the default, as adding more integrity rules after 
the fact is hard to the point of being impractical. Agents in themselves don’t 
impact that default, but dynamically loaded agents do.

> 
>  - We recognize that the proposal is only proposing to flip a configuration 
> default rather than detract from (or modify) available functionality
> 
>  - We recognize that changing this default will still allow (*most*) users to 
> configure the behaviour they desire
> 
>  - We recognize that this advance notice has been given precisely to ensure 
> that anyone wishing to deploy on jdk21 an app that relies on use of agents 
> has time to plan appropriate configuration for their deployment
> 
>  - We recognize that this change of default is not being proposed for 
> backport and hence that it will largely only affect the relatively small 
> number of users who are currently developing for jdk21+
> 
> So, given that as a base for our comments where is the beef?
> 
>  - Our main concern is, predictably, timing. Clearly, this is a future, 
> potential problem rather than a present problem - no one can be deploying on 
> jdk21 yet and most developers who are currently preparing an app for 
> deployment on jdk21+ will likely encounter the effect of this change before 
> actual deployment and be in a position to remedy it. The concern is that 
> advertising a change like this and getting users prepared to respond to it 
> has always been difficult to achieve. In particular we expect a long tail of 
> support problems from users who are trying to upgrade deployments from 
> earlier releases to jdk21.
>  So, while it is nice to have such early notice of the proposal we plan to 
> review its likely impact on our users and how much time we need to prepare 
> ourselves and our users to negotiate this change in behaviour. Any evidence 
> we obtain to suggest a delay in targeting is appropriate will be brought to 
> the JEP review.

Very well, we can certainly discuss timing as part of the JEP discussion.

The JEP itself is rather ambitious because I noticed that most of the strong 
encapsulation JEPs lacked a substantial motivation section (largely because 
they had been written before that became the norm) and so strong encapsulation 
has not been motivated in JEP form, and this is an opportunity to start 
rectifying that. This is important because some, even in this discussion, are 
under the impression that integrity is about security. Although security is 
certainly one of integrity’s impacts (albeit not in the way hypothesised in 
this discussion) other major impacts are on performance (including, though 
certainly not limited to, link-time optimisations that are of interest to 
Project Leyden) and code evolution (the lack of integrity has been, by far, the 
biggest cause of JDK upgrade issues experienced by many applications an 
lib

Re: Disallowing the dynamic loading of agents by default

[Gregg G Wonderly]

I understand you may have personal experiences with how you use Java.  In my 
experience and others, Java has constantly had fundamental breakage in various 
details due to lack of understanding, on the platform development team(s) of 
what people actually do with Java.

Sent from my iPhone

> On Mar 25, 2023, at 5:29 AM, Bernd  wrote:
> 
> 
> Gregg i have to disagree, not only is Java one of the most stable platform 
> out there but also the „Enterprise Desktop“ couldn’t care less about the 
> default of this switch. 
> 
> Gruss
> Bernd 
> --
> http://bernd.eckenfels.net
>  
> Von: serviceability-dev  im Auftrag von 
> Gregg Wonderly 
> Gesendet: Freitag, März 24, 2023 11:42 PM
> An: Andrew Dinn 
> Cc: Ron Pressler ; jigsaw-...@openjdk.org 
> ; serviceability-dev@openjdk.org 
> 
> Betreff: Re: Disallowing the dynamic loading of agents by default
>  
> Lot’s of people use Java in places where there is no “release” cycle of Java 
> version in control of the users. These are “corporate users” in most cases 
> and they have Java applications that they are using which will just “stop 
> working” when a new version of Java is installed. 
> 
> Over the years, I’ve watch any favoritism towards java on the desktop or as a 
> general solution programing language wane, because it’s undependable as a 
> platform. You never no when something will break as these “stability” changes 
> occur. People who use software systems are in large part not programers or 
> language/platform experts. The in ability of Oracle and many others to 
> understand how detrimental this behavior has been is just mind blowing. 
> 
> People like myself end up looking like whining babies because we come back 
> every once in a while to see if there is something useful happening in Java 
> development that might finally stabilize the platform on the desktop and 
> other business environments and low and behold write-once-run-anywhere is 
> found to still be unimplemented and basically non appreciated. It’s just a 
> sad, sad thing to see happening. 
> 
> Sun first did this with Java 1.2. The Community beat up on Sun severely and 
> everything quieted down for a while. Then we had the JDK 1.5 release where my 
> much mentioned volatile reachability optimizations broke software all over 
> the place. 
> 
> This is not happening in any other language I am aware of. The people at Sun 
> who were causing all the problems seemed to have gone on to Oracle and there 
> is just a core group of people who just do not understand how horrible Java 
> looks these days because of how much basic functionality got completely 
> broken when a new version of Java showed up on general purpose computing and 
> working software just stopped working… 
> 
> Gregg Wonderly 
> 
> > On Mar 24, 2023, at 12:21 PM, Andrew Dinn  wrote: 
> > 
> > Hi Ron, 
> > 
> > Thank you for providing a heads up on the proposed JEP. The Red Hat Java 
> > team have been discussing this proposal. We have reviewed the original 
> > discussion and also the surrounding debate which established requirements 
> > for adaptation of Jigsaw to incorporate the needs of agents. 
> > 
> > As an aside, I'll note that a thorough review was necessary /even/ in my 
> > case, despite the fact that I was an active party, because the discussion 
> > occurred, and corresponding decisions were made, quite some time ago. I 
> > mention this because it may explain the air of surprise and the desire to 
> > reiterate some of the original debate on the part of some respondents in 
> > this thread, who perhaps were not party, or only tangentially party, to the 
> > discussion. 
> > 
> > That also suggests that there may be a lot users who are not aware that the 
> > -XX:+EnableDynamicAgentLoading switch exists or do not really understand 
> > why it exists i.e. that there is a broad education issue at play here. 
> > 
> > We do have some concerns about the JEP, specifically about the timing of 
> > its delivery. These are probably best addressed via the normal review 
> > process. In particular that will ensure the discussion happens in a more 
> > suitable and more widely subscribed forum than the Jigsaw list. However, I 
> > will briefly mention our concerns in this reply. Before that let me start 
> > with a few disclaimers: 
> > 
> > - We acknowledge that there is little to be gained from re-iterating 
> > arguments made in the previous discussion (although that does not imply the 
> > JEP review would not benefit from new arguments, especially from those who 
> > were not involved in that discussion) 
> > 
> > - We recognize that the pur

Re: Disallowing the dynamic loading of agents by default

[Bernd]

  
  
  

	
	Gregg i have to disagree, not only is Java one of the most stable platform out there but also the „Enterprise Desktop“ couldn’t care less about the default of this switch. GrussBernd -- http://bernd.eckenfels.net

  

 Von: serviceability-dev  im Auftrag von Gregg Wonderly Gesendet: Freitag, März 24, 2023 11:42 PMAn: Andrew Dinn Cc: Ron Pressler ; jigsaw-...@openjdk.org ; serviceability-dev@openjdk.org Betreff: Re: Disallowing the dynamic loading of agents by default Lot’s of people use Java in places where there is no “release” cycle of Java version in control of the users.  These are “corporate users” in most cases and they have Java applications that they are using which will just “stop working” when a new version of Java is installed.

Over the years, I’ve watch any favoritism towards java on the desktop or as a general solution programing language wane, because it’s undependable as a platform.  You never no when something will break as these “stability” changes occur.  People who use software systems are in large part not programers or language/platform experts.  The in ability of Oracle and many others to understand how detrimental this behavior has been is just mind blowing.

People like myself end up looking like whining babies because we come back every once in a while to see if there is something useful happening in Java development that might finally stabilize the platform on the desktop and other business environments and low and behold write-once-run-anywhere is found to still be unimplemented and basically non appreciated. It’s just a sad, sad thing to see happening.

Sun first did this with Java 1.2.  The Community beat up on Sun severely and everything quieted down for a while.  Then we had the JDK 1.5 release where my much mentioned volatile reachability optimizations broke software all over the place.

This is not happening in any other language I am aware of.  The people at Sun who were causing all the problems seemed to have gone on to Oracle and there is just a core group of people who just do not understand how horrible Java looks these days because of how much basic functionality got completely broken when a new version of Java showed up on general purpose computing and working software just stopped working…

Gregg Wonderly

> On Mar 24, 2023, at 12:21 PM, Andrew Dinn  wrote:
> 
> Hi Ron,
> 
> Thank you for providing a heads up on the proposed JEP. The Red Hat Java team have been discussing this proposal. We have reviewed the original discussion and also the surrounding debate which established requirements for adaptation of Jigsaw to incorporate the needs of agents.
> 
> As an aside, I'll note that a thorough review was necessary /even/ in my case, despite the fact that I was an active party, because the discussion occurred, and corresponding decisions were made, quite some time ago. I mention this because it may explain the air of surprise and the desire to reiterate some of the original debate on the part of some respondents in this thread, who perhaps were not party, or only tangentially party, to the discussion.
> 
> That also suggests that there may be a lot users who are not aware that the -XX:+EnableDynamicAgentLoading switch exists or do not really understand why it exists i.e. that there is a broad education issue at play here.
> 
> We do have some concerns about the JEP, specifically about the timing of its delivery. These are probably best addressed via the normal review process. In particular that will ensure the discussion happens in a more suitable and more widely subscribed forum than the Jigsaw list. However, I will briefly mention our concerns in this reply. Before that let me start with a few disclaimers:
> 
> - We acknowledge that there is little to be gained from re-iterating arguments made in the previous discussion (although that does not imply the JEP review would not benefit from new arguments, especially from those who were not involved in that discussion)
> 
>  - We recognize that the purpose of the -XX:+EnableDynamicAgentLoading switch is to offer a platform integrity guarantee and that this change of the default reflects a desire to prioritise integrity over the flexibility that agents provide
> 
>  - We recognize that the proposal is only proposing to flip a configuration default rather than detract from (or modify) available functionality
> 
>  - We recognize that changing this default will still allow (*most*) users to configure the behaviour they desire
> 
>  - We recognize that this advance notice has been given precisely to ensure that anyone wishing to deploy on jdk21 an app that relies on use of agents has time to plan appropriate configuration for their deployment
> 
>  - We recognize that this change of default is not being proposed for backport and hence that it will largely only affect the relatively small number of users who are currently 

Re: Disallowing the dynamic loading of agents by default

[Gregg Wonderly]

Lot’s of people use Java in places where there is no “release” cycle of Java 
version in control of the users.  These are “corporate users” in most cases and 
they have Java applications that they are using which will just “stop working” 
when a new version of Java is installed.

Over the years, I’ve watch any favoritism towards java on the desktop or as a 
general solution programing language wane, because it’s undependable as a 
platform.  You never no when something will break as these “stability” changes 
occur.  People who use software systems are in large part not programers or 
language/platform experts.  The in ability of Oracle and many others to 
understand how detrimental this behavior has been is just mind blowing.

People like myself end up looking like whining babies because we come back 
every once in a while to see if there is something useful happening in Java 
development that might finally stabilize the platform on the desktop and other 
business environments and low and behold write-once-run-anywhere is found to 
still be unimplemented and basically non appreciated. It’s just a sad, sad 
thing to see happening.

Sun first did this with Java 1.2.  The Community beat up on Sun severely and 
everything quieted down for a while.  Then we had the JDK 1.5 release where my 
much mentioned volatile reachability optimizations broke software all over the 
place.

This is not happening in any other language I am aware of.  The people at Sun 
who were causing all the problems seemed to have gone on to Oracle and there is 
just a core group of people who just do not understand how horrible Java looks 
these days because of how much basic functionality got completely broken when a 
new version of Java showed up on general purpose computing and working software 
just stopped working…

Gregg Wonderly

> On Mar 24, 2023, at 12:21 PM, Andrew Dinn  wrote:
> 
> Hi Ron,
> 
> Thank you for providing a heads up on the proposed JEP. The Red Hat Java team 
> have been discussing this proposal. We have reviewed the original discussion 
> and also the surrounding debate which established requirements for adaptation 
> of Jigsaw to incorporate the needs of agents.
> 
> As an aside, I'll note that a thorough review was necessary /even/ in my 
> case, despite the fact that I was an active party, because the discussion 
> occurred, and corresponding decisions were made, quite some time ago. I 
> mention this because it may explain the air of surprise and the desire to 
> reiterate some of the original debate on the part of some respondents in this 
> thread, who perhaps were not party, or only tangentially party, to the 
> discussion.
> 
> That also suggests that there may be a lot users who are not aware that the 
> -XX:+EnableDynamicAgentLoading switch exists or do not really understand why 
> it exists i.e. that there is a broad education issue at play here.
> 
> We do have some concerns about the JEP, specifically about the timing of its 
> delivery. These are probably best addressed via the normal review process. In 
> particular that will ensure the discussion happens in a more suitable and 
> more widely subscribed forum than the Jigsaw list. However, I will briefly 
> mention our concerns in this reply. Before that let me start with a few 
> disclaimers:
> 
> - We acknowledge that there is little to be gained from re-iterating 
> arguments made in the previous discussion (although that does not imply the 
> JEP review would not benefit from new arguments, especially from those who 
> were not involved in that discussion)
> 
>  - We recognize that the purpose of the -XX:+EnableDynamicAgentLoading switch 
> is to offer a platform integrity guarantee and that this change of the 
> default reflects a desire to prioritise integrity over the flexibility that 
> agents provide
> 
>  - We recognize that the proposal is only proposing to flip a configuration 
> default rather than detract from (or modify) available functionality
> 
>  - We recognize that changing this default will still allow (*most*) users to 
> configure the behaviour they desire
> 
>  - We recognize that this advance notice has been given precisely to ensure 
> that anyone wishing to deploy on jdk21 an app that relies on use of agents 
> has time to plan appropriate configuration for their deployment
> 
>  - We recognize that this change of default is not being proposed for 
> backport and hence that it will largely only affect the relatively small 
> number of users who are currently developing for jdk21+
> 
> So, given that as a base for our comments where is the beef?
> 
>  - Our main concern is, predictably, timing. Clearly, this is a future, 
> potential problem rather than a present problem - no one can be deploying on 
> jdk21 yet and most developers who are currently preparing an app for 
> deployment on jdk21+ will likely encounter the effect of this change before 
> actual deployment and be in a position to remedy it. The concern is that 

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Gregg Wonderly]

After all, we do know that Oracle in fact knows about every single Java 
application, where it runs, where it’s deployed and what the future plans are 
for the same.  Otherwise, how else could they know what changes need to be made 
in the platform, right?

Gregg Wonderly

> On Mar 20, 2023, at 5:10 AM, Ron Pressler  wrote:
> 
> Hi.
> 
> The majority of serviceability tools don’t require dynamically loading an 
> agent, and the majority of applications never load an agent dynamically.
> 
> True, there are some tools that will be affected, which is why the decision 
> was to introduce the flag in JDK 9 and to announce this change, but change 
> the default in a later version to give tools ample time to prepare their 
> users. The rationale for this change then hasn’t changed, but will be 
> reiterated in a JEP (we just wanted to announce this ahead of the JEP to give 
> tool authors another reminder more than six months ahead of JDK 21). The only 
> change between then and now is that even fewer use cases require dynamically 
> loaded agents, and so the impact is even smaller.
> 
> It is also true that, when starting an application you don’t know that you 
> *will* need to load an agent, but in most situations you know that you might. 
> E.g. processes that are too critical to bring down even for deep maintenance 
> (although not many of these are written in modern version of Java anyone) or 
> canary services that are under trial. The relatively few sophisticated users 
> who know how to write ad-hoc agents can even opt to enable dynamic agent 
> loading on all their servers; these users are better equipped to can weigh 
> the risks and tradeoffs involved.
> 
> Finally, some tools that require a dynamically loaded JVM TI agents, such as 
> profilers that profile native code, are so tied to the VM's internals that 
> the best place for them is in the JDK. If anything, the bigger problem is not 
> that profilers are used too much in production, but too little, including 
> less advanced ones that don’t require an agent. There is plenty of time to 
> enhance the JDK’s built-in profiling capabilities ahead of demand.
> 
> — Ron
> 
>> On 20 Mar 2023, at 01:21, Andrei Pangin > > wrote:
>> 
>> Hi all,
>> 
>> Serviceability has been one of the biggest Java strengths, but the proposed 
>> change is going to have a large negative impact on it.
>> 
>> Disallowing dynamic agents by default means it will no longer be possible to 
>> attach a profiler to a running app in runtime. JFR cannot close this gap due 
>> to lack of capabilities modern Java profilers have (that's a separate topic 
>> though).
>> 
>> When an issue happens with a live app, it's already too late to add a 
>> command line argument. Furthermore, it may not be even feasible to add an 
>> agent at startup in containerized applications. Starting profiler on demand 
>> from the host OS or from a sidecar is the only viable solution in these 
>> cases.
>> 
>> Next, it's hard to predict beforehand what tools exactly might be useful for 
>> troubleshooting: e.g., one tool may be better for finding memory leaks, a 
>> different one for analyzing CPU performance. Adding all possible tools at 
>> startup does not seem a reasonable approach, especially when tools may 
>> conflict with each other.
>> 
>> The most important aspect of dynamic agents is the possibility to make a 
>> special tool just in time for solving a particular problem. A typical 
>> example is to get a value of some field in a live app without dumping the 
>> entire 60 GB heap. Another common use case is hot patching for fixing 
>> trivial bugs or for adding debug logs dynamically. The prominent example is 
>> when the dynamic agent has proved irreplaceable aid in addressing the 
>> notorious log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046.
>> 
>> I would be grateful to know more about the reasons why we should give up all 
>> the above advantages of dynamic agents in their good and legitimate use 
>> cases.
>> 
>> Thank you,
>> Andrei
>> 
>> чт, 16 мар. 2023 г. в 18:48, Ron Pressler > >:
>>> Hi.
>>> 
>>> In JDK 21 we intend to disallow the dynamic loading of agents by default. 
>>> This
>>> will affect tools that use the Attach API to load an agent into a JVM some 
>>> time
>>> after the JVM has started [1]. There is no change to any of the mechanisms 
>>> that
>>> load an agent at JVM startup (-javaagent/-agentlib on the command line or 
>>> the
>>> Launcher-Agent-Class attribute in the main JAR's manifest).
>>> 
>>> This change in default behavior was proposed in 2017 as part of JEP 261 
>>> [2][3]. 
>>> At that time the consensus was to switch to this default not in JDK 9 but 
>>> in a 
>>> later release to give tool maintainers sufficient time to inform their 
>>> users. 
>>> To allow the dynamic loading of agents, users will need to specify 
>>> -XX:+EnableDynamicAgentLoading on the command line.
>>> 
>>> I'll post a

Re: Disallowing the dynamic loading of agents by default

[Andrew Dinn]

Hi Ron,

Thank you for providing a heads up on the proposed JEP. The Red Hat Java 
team have been discussing this proposal. We have reviewed the original 
discussion and also the surrounding debate which established 
requirements for adaptation of Jigsaw to incorporate the needs of agents.


As an aside, I'll note that a thorough review was necessary /even/ in my 
case, despite the fact that I was an active party, because the 
discussion occurred, and corresponding decisions were made, quite some 
time ago. I mention this because it may explain the air of surprise and 
the desire to reiterate some of the original debate on the part of some 
respondents in this thread, who perhaps were not party, or only 
tangentially party, to the discussion.


That also suggests that there may be a lot users who are not aware that 
the -XX:+EnableDynamicAgentLoading switch exists or do not really 
understand why it exists i.e. that there is a broad education issue at 
play here.


We do have some concerns about the JEP, specifically about the timing of 
its delivery. These are probably best addressed via the normal review 
process. In particular that will ensure the discussion happens in a more 
suitable and more widely subscribed forum than the Jigsaw list. However, 
I will briefly mention our concerns in this reply. Before that let me 
start with a few disclaimers:


- We acknowledge that there is little to be gained from re-iterating 
arguments made in the previous discussion (although that does not imply 
the JEP review would not benefit from new arguments, especially from 
those who were not involved in that discussion)


  - We recognize that the purpose of the -XX:+EnableDynamicAgentLoading 
switch is to offer a platform integrity guarantee and that this change 
of the default reflects a desire to prioritise integrity over the 
flexibility that agents provide


  - We recognize that the proposal is only proposing to flip a 
configuration default rather than detract from (or modify) available 
functionality


  - We recognize that changing this default will still allow (*most*) 
users to configure the behaviour they desire


  - We recognize that this advance notice has been given precisely to 
ensure that anyone wishing to deploy on jdk21 an app that relies on use 
of agents has time to plan appropriate configuration for their deployment


  - We recognize that this change of default is not being proposed for 
backport and hence that it will largely only affect the relatively small 
number of users who are currently developing for jdk21+


So, given that as a base for our comments where is the beef?

  - Our main concern is, predictably, timing. Clearly, this is a 
future, potential problem rather than a present problem - no one can be 
deploying on jdk21 yet and most developers who are currently preparing 
an app for deployment on jdk21+ will likely encounter the effect of this 
change before actual deployment and be in a position to remedy it. The 
concern is that advertising a change like this and getting users 
prepared to respond to it has always been difficult to achieve. In 
particular we expect a long tail of support problems from users who are 
trying to upgrade deployments from earlier releases to jdk21.
  So, while it is nice to have such early notice of the proposal we 
plan to review its likely impact on our users and how much time we need 
to prepare ourselves and our users to negotiate this change in 
behaviour. Any evidence we obtain to suggest a delay in targeting is 
appropriate will be brought to the JEP review.


  - A second, related concern is that flipping the default for this 
configuration in an LTS release as the first exposure to it for most 
people is more likely to derail deployment plans for users than if the 
default were flipped in a non-LTS release. If this change were deferred 
to jdk22 then that would give those planning deployment on (or upgrade 
to) jdk25 and also those planning to upgrade from jdk17 to jdk21 more 
time to discover and respond to the change.


  - A third concern, already pointed out by Volker, is that some users 
may run their Java apps via launcher apps or scripts that mask access to 
the Java command line. For such users the change of default may mean 
that they lose the option to deploy dynamic agents for important 
ancillary tasks such as observability. We are not clear how many of our 
users this affects but we will be looking into this and hope to bring 
feedback to the JEP review.
  Obviously, this problem can be remedied relatively easily by the 
supplier of the launcher enabling agent use or providing a suitable 
control switch. Our concern is not with how to solve this problem rather 
how the involvement of two parties, supplier and end user, might imply a 
need for the JEP to be targeted to a later release.


regards,


Andrew Dinn
---
Red Hat Distinguished Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Ron Pressler]

Hi Volker.

JEP 261 states: "The dynamic loading of JVM TI agents will be disabled by 
default in a future release. To prepare for that change we recommend that 
applications that allow dynamic agents start using the option 
-XX:+EnableDynamicAgentLoading to enable that loading explicitly." The purpose 
of my email was to announce that that change will be put into effect in JDK 21 
and to give a final reminder to those who have not yet done so to follow the 
recommendation in JEP 261 to prepare for that change.

The Project Jigsaw team made that decision at the time after considering the 
perspectives of performance experts, security experts, and tooling experts, but 
unless anyone has some *new* information to present, there is no point in 
replaying the same discussions. You can revisit at least some of the technical 
discussions on jigsaw-dev.

I will summarise the salient aspects (all discussed at the time) in the 
forthcoming JEP but, briefly, dynamically loaded agents -- alongside JNI and 
Unsafe -- break integrity, the ability to guarantee certain invariants, which 
has various implications on performance, security, and code evolution. They 
don't always break integrity in the way a cursory contemplation would suggest, 
which is why you should study those discussions if you're interested in the 
subject. Since JEP 261, the JDK has been evolving under the assumption that 
integrity is preserved unless the application grants explicit consent for it to 
be broken. As far as security in particular is concerned, the point you made is 
*not* the relevant one to the implications that were considered by Project 
Jigsaw at the time. As a member of the Vulnerability Group you may want to 
discuss that particular aspect with the appropriate people.

-- Ron

> On 20 Mar 2023, at 12:16, Volker Simonis  wrote:
> 
> Hi Ron,
> 
> I'm still missing convincing technical arguments for disallowing
> dynamic loading of agents.
> 
> If the argument is security then I can only agree with previous
> answers in that an attacker needs local access with the same
> credentials like the attacked JVM. But once he has that, all bets are
> off anyway.
> 
> If you plan for features/enhancements/optimizations that rely on not
> being able to dynamically load an agent (which I haven't heard off
> yet), I don't understand this change either. Because as long as a
> switch for enabling dynamic loading exists (and I haven' heard that
> you want to completely forbid it) the dynamic loading use case has to
> be supported anyway.
> 
> Dynamic agent loading is one of the features which sets the OpenJDK
> apart from other languages, managed runtimes and even closely related
> platforms like for example GraalVM Native Image which don't support
> such a feature. The mere  existence of tools which rely on it and
> which are in widespread productive use, demonstrates its usefulness.
> And it is always good to know you have this possibility in your
> toolbox for the worst case (e.g. our log4j-hotpatcher [1]).
> 
> I also can't by your argument that "the relatively few sophisticated
> users who know how to write ad-hoc agents can even opt to enable
> dynamic agent loading on all their servers". It is *exactly* not the
> few sophisticated authors of dynamic agents who would need to enable
> them but instead the millions of ingenious end-users and
> administrators who bag for help once they run into trouble. The other
> way round makes much more sense to me - the few sophisticated users
> who know for sure that they will never need the help of dynamic agents
> are free to disable them at startup.
> 
> Given the current arguments, for me the usefulness of dynamic agents
> outweigh their drawbacks by far. Of course every OpenJDK distributor
> is free to change the default settings of command line options at his
> sole discretion, but I don't currently see a compelling reason for
> doing this by default for the whole OpenJDK community. If you have
> future plans which rely on disabling/forbidding dynamic agents please
> let us know.
> 
> Best regards,
> Volker
> 
> [1] 
> https://urldefense.com/v3/__https://aws.amazon.com/blogs/opensource/hotpatch-for-apache-log4j/__;!!ACWV5N9M2RV99hQ!I7QWWsAmQNvmFzektSGaq4lWBWuMxP5R8P6nSwxfugmyEpKOrd_Io64JBX9mD8PBHywYZ7gEbDumhe5MdiQz_QdvpQ$
>  
> 
> On Mon, Mar 20, 2023 at 11:37 AM Jaroslav Bachorik  
> wrote:
>> 
>> Hi,
>> 
>> On Mon, Mar 20, 2023 at 11:11 AM Ron Pressler  
>> wrote:
>>> 
>>> Hi.
>>> 
>>> The majority of serviceability tools don’t require dynamically loading an 
>>> agent, and the majority of applications never load an agent dynamically.
>> 
>> 
>> The majority of the JDK built-in tools, I would say. What about eg. the JMC 
>> agent?
>> 
>>> 
>>> 
>>> True, there are some tools that will be affected, which is why the decision 
>>> was to introduce the flag in JDK 9 and to announce this change, but change 
>>> the default in a later version to give tools ample time to prepare their 
>>> users. The rati

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Ron Pressler]

> On 20 Mar 2023, at 17:53, Ron Pressler  wrote:
> 
> While the JEP will reiterate the relevant considerations (and no one denies 
> that dynamically loaded agents are not useful)

Sorry, no one denies that dynamically loaded agents *are* useful :)

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Ron Pressler]

Hi Kirk.

While the JEP will reiterate the relevant considerations (and no one denies 
that dynamically loaded agents are not useful) that led to this change being 
announced some years ago, the purpose of my email was to announce it will 
finally take effect in JDK 21. All the discussions at time, over all the 
relevant areas, informed the design at the core of the platform and its 
evolution in the past five years, namely that the application must grant 
explicit consent to anything affecting integrity (i.e. guarantees you can 
trust). Unless something has changed dramatically since then, or some new 
information has come to light, reopening the discussions around past decisions 
that have shaped the platform’s current design are unlikely to yield different 
results. Many of those discussions are available on jigsaw-dev.

It is because some people may be affected that JEP 261 postponed the changing 
of that default, so that everyone would have time to prepare and prepare their 
users. We’ve now given an extra six-month advance notice to give those who 
haven’t finished preparing their users the time to do so.

This is an opportunity to remind everyone that other capabilities that 
similarly affect platform and application integrity — such as JNI and Unsafe — 
will also require the application’s consent on the command line — not in JDK 
21, but soon thereafter.

— Ron

On 20 Mar 2023, at 17:02, Kirk Pepperdine 
mailto:kirk.pepperd...@gmail.com>> wrote:

Hi Ron,


On Mar 20, 2023, at 3:10 AM, Ron Pressler 
mailto:ron.press...@oracle.com>> wrote:

Hi.

The majority of serviceability tools don’t require dynamically loading an 
agent, and the majority of applications never load an agent dynamically.

While I wouldn’t be surprised that the majority don’t load agents dynamically, 
I wouldn’t want to diminishes the importance of this capability for those that 
do make use of it. And I believe the number that do dynamically load might 
surprise you. But then, my data on this is likely highly biased. Do you have 
better data to support this view point?


True, there are some tools that will be affected, which is why the decision was 
to introduce the flag in JDK 9 and to announce this change, but change the 
default in a later version to give tools ample time to prepare their users. The 
rationale for this change then hasn’t changed, but will be reiterated in a JEP 
(we just wanted to announce this ahead of the JEP to give tool authors another 
reminder more than six months ahead of JDK 21). The only change between then 
and now is that even fewer use cases require dynamically loaded agents, and so 
the impact is even smaller.

Again, I’m not sure I see the data to support this. But then again, my view 
point remains highly biased. And I see an assumption that tools will be able to 
easily adapt to this change. I’m not sure that is entirely true. At least not 
in a way that in effect returns dynamic attach capabilities with a directly 
loaded proxy.


It is also true that, when starting an application you don’t know that you 
*will* need to load an agent, but in most situations you know that you might. 
E.g. processes that are too critical to bring down even for deep maintenance 
(although not many of these are written in modern version of Java anyone) or 
canary services that are under trial. The relatively few sophisticated users 
who know how to write ad-hoc agents can even opt to enable dynamic agent 
loading on all their servers; these users are better equipped to can weigh the 
risks and tradeoffs involved.

Again, I’m not sure I’d equate numbers to importance. As an analogy, there are 
very few people that know how to build cars and maybe more that know how to fix 
them but, there are certainly many many more than know how to use them.


Finally, some tools that require a dynamically loaded JVM TI agents, such as 
profilers that profile native code, are so tied to the VM's internals that the 
best place for them is in the JDK. If anything, the bigger problem is not that 
profilers are used too much in production, but too little, including less 
advanced ones that don’t require an agent. There is plenty of time to enhance 
the JDK’s built-in profiling capabilities ahead of demand.

At odds is that all profilers come with biases. I’ve always stressed that the 
first thing one needs to do with any profiler is discover it’s biases and then 
determine how that bias affects the results, how one should account for the 
bias, or even should another tool be used. To this point, JFR, a built in 
profiler, has significant biases. For example, allocation profiling, quite 
often, completely misses allocation hotspots for small objects that are not 
scalar replaced in all but trivial examples. But let's not pick on JFR because 
it is the tool of choice for many other things and other allocation profilers 
do have other biases such as altering JIT behavior that may cause EA to fail 
thus preventing otherwise eligible allo

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Kirk Pepperdine]

Hi Ron,


> On Mar 20, 2023, at 3:10 AM, Ron Pressler  wrote:
> 
> Hi.
> 
> The majority of serviceability tools don’t require dynamically loading an 
> agent, and the majority of applications never load an agent dynamically.

While I wouldn’t be surprised that the majority don’t load agents dynamically, 
I wouldn’t want to diminishes the importance of this capability for those that 
do make use of it. And I believe the number that do dynamically load might 
surprise you. But then, my data on this is likely highly biased. Do you have 
better data to support this view point?

> 
> True, there are some tools that will be affected, which is why the decision 
> was to introduce the flag in JDK 9 and to announce this change, but change 
> the default in a later version to give tools ample time to prepare their 
> users. The rationale for this change then hasn’t changed, but will be 
> reiterated in a JEP (we just wanted to announce this ahead of the JEP to give 
> tool authors another reminder more than six months ahead of JDK 21). The only 
> change between then and now is that even fewer use cases require dynamically 
> loaded agents, and so the impact is even smaller.

Again, I’m not sure I see the data to support this. But then again, my view 
point remains highly biased. And I see an assumption that tools will be able to 
easily adapt to this change. I’m not sure that is entirely true. At least not 
in a way that in effect returns dynamic attach capabilities with a directly 
loaded proxy. 

> 
> It is also true that, when starting an application you don’t know that you 
> *will* need to load an agent, but in most situations you know that you might. 
> E.g. processes that are too critical to bring down even for deep maintenance 
> (although not many of these are written in modern version of Java anyone) or 
> canary services that are under trial. The relatively few sophisticated users 
> who know how to write ad-hoc agents can even opt to enable dynamic agent 
> loading on all their servers; these users are better equipped to can weigh 
> the risks and tradeoffs involved.

Again, I’m not sure I’d equate numbers to importance. As an analogy, there are 
very few people that know how to build cars and maybe more that know how to fix 
them but, there are certainly many many more than know how to use them.

> 
> Finally, some tools that require a dynamically loaded JVM TI agents, such as 
> profilers that profile native code, are so tied to the VM's internals that 
> the best place for them is in the JDK. If anything, the bigger problem is not 
> that profilers are used too much in production, but too little, including 
> less advanced ones that don’t require an agent. There is plenty of time to 
> enhance the JDK’s built-in profiling capabilities ahead of demand.

At odds is that all profilers come with biases. I’ve always stressed that the 
first thing one needs to do with any profiler is discover it’s biases and then 
determine how that bias affects the results, how one should account for the 
bias, or even should another tool be used. To this point, JFR, a built in 
profiler, has significant biases. For example, allocation profiling, quite 
often, completely misses allocation hotspots for small objects that are not 
scalar replaced in all but trivial examples. But let's not pick on JFR because 
it is the tool of choice for many other things and other allocation profilers 
do have other biases such as altering JIT behavior that may cause EA to fail 
thus preventing otherwise eligible allocation hotspots from being scalar 
replaced. While one tool is blind, the other generates false positives. Knowing 
this, I can combine the JIT logs with profiler results to help offset the 
effects of the bias for the later profile.. however, I can’t do anything for 
the blind spot.

Finally, if there is anything lesson to be learned from the migrations from 8 
to 9 is that tooling is a huge anchor preventing people from upgrading. That 
JDK 8 is still in as widespread use as it is, is in no small part due to the 
extensive change in the tooling chain. In fact, a number of very useful tools 
simply didn’t survive leaving us with less desirable alternatives. The other 
historical data point that maybe of comparison is the introduction of generics 
into the language. While this slowed the adoption of JDK 5 (from 1.4.2), it had 
no where near the impact that the degradation of the observability/diagnostic 
tool chain had on the migration rates from 7 to 8 and then this huge impact of 
9. In my opinion, we’ve learned enough from this migration to understand that 
we may need to re-evaluate decisions that were made prior to these learning.

Kind regards,
Kirk


> 
> — Ron
> 
>> On 20 Mar 2023, at 01:21, Andrei Pangin > > wrote:
>> 
>> Hi all,
>> 
>> Serviceability has been one of the biggest Java strengths, but the proposed 
>> change is going to have a large negative impact on it.
>> 
>> Disallowi

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Volker Simonis]

Hi Ron,

I'm still missing convincing technical arguments for disallowing
dynamic loading of agents.

If the argument is security then I can only agree with previous
answers in that an attacker needs local access with the same
credentials like the attacked JVM. But once he has that, all bets are
off anyway.

If you plan for features/enhancements/optimizations that rely on not
being able to dynamically load an agent (which I haven't heard off
yet), I don't understand this change either. Because as long as a
switch for enabling dynamic loading exists (and I haven' heard that
you want to completely forbid it) the dynamic loading use case has to
be supported anyway.

Dynamic agent loading is one of the features which sets the OpenJDK
apart from other languages, managed runtimes and even closely related
platforms like for example GraalVM Native Image which don't support
such a feature. The mere  existence of tools which rely on it and
which are in widespread productive use, demonstrates its usefulness.
And it is always good to know you have this possibility in your
toolbox for the worst case (e.g. our log4j-hotpatcher [1]).

I also can't by your argument that "the relatively few sophisticated
users who know how to write ad-hoc agents can even opt to enable
dynamic agent loading on all their servers". It is *exactly* not the
few sophisticated authors of dynamic agents who would need to enable
them but instead the millions of ingenious end-users and
administrators who bag for help once they run into trouble. The other
way round makes much more sense to me - the few sophisticated users
who know for sure that they will never need the help of dynamic agents
are free to disable them at startup.

Given the current arguments, for me the usefulness of dynamic agents
outweigh their drawbacks by far. Of course every OpenJDK distributor
is free to change the default settings of command line options at his
sole discretion, but I don't currently see a compelling reason for
doing this by default for the whole OpenJDK community. If you have
future plans which rely on disabling/forbidding dynamic agents please
let us know.

Best regards,
Volker

[1] https://aws.amazon.com/blogs/opensource/hotpatch-for-apache-log4j/

On Mon, Mar 20, 2023 at 11:37 AM Jaroslav Bachorik  wrote:
>
> Hi,
>
> On Mon, Mar 20, 2023 at 11:11 AM Ron Pressler  wrote:
>>
>> Hi.
>>
>> The majority of serviceability tools don’t require dynamically loading an 
>> agent, and the majority of applications never load an agent dynamically.
>
>
> The majority of the JDK built-in tools, I would say. What about eg. the JMC 
> agent?
>
>>
>>
>> True, there are some tools that will be affected, which is why the decision 
>> was to introduce the flag in JDK 9 and to announce this change, but change 
>> the default in a later version to give tools ample time to prepare their 
>> users. The rationale for this change then hasn’t changed, but will be 
>> reiterated in a JEP (we just wanted to announce this ahead of the JEP to 
>> give tool authors another reminder more than six months ahead of JDK 21). 
>> The only change between then and now is that even fewer use cases require 
>> dynamically loaded agents, and so the impact is even smaller.
>
>
> As a maintainer of one of such tools I can confidently say that this change 
> will either kill the tool as the ease of use will be gone or the workaround 
> (eg. using JAVA_TOOL_OPTIONS) will completely defeat the purpose of this 
> change. Having to put a flag when starting the JVM to allow dynamic loading 
> of agents sounds a bit nonsensical to me - it would be much easier to 
> directly add the agent to the JVM startup and then implement a lightweight 
> control protocol over socket/shared memory to enabled/disable the agent 
> features dynamically.
>
>>
>>
>> It is also true that, when starting an application you don’t know that you 
>> *will* need to load an agent, but in most situations you know that you 
>> might. E.g. processes that are too critical to bring down even for deep 
>> maintenance (although not many of these are written in modern version of 
>> Java anyone) or canary services that are under trial. The relatively few 
>> sophisticated users who know how to write ad-hoc agents can even opt to 
>> enable dynamic agent loading on all their servers; these users are better 
>> equipped to can weigh the risks and tradeoffs involved.
>
>
> Wouldn't having this enabled system-wide actually defeat the purpose of 
> having this flag? Considering that the dynamic attach can be performed only 
> on the same host under the same user as the target process there seems to be 
> a very small chance of loading agents accidentally. In the end people would 
> set up their systems to enabled dynamic agent loading via eg. 
> JAVA_TOOL_OPTIONS and we will be in the same place as before, with the 
> additional hurdle of setting everything up.
>
>>
>> Finally, some tools that require a dynamically loaded JVM TI agents, such as 
>> pr

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Bernd]

  
  
  

	
	Hello,I don’t have a strong feeling towards or against changing the default. While I agree that having ad-hoc access to diagnostics is good, I am also ok with having to prepare that once, even when it means I can’t catch the first instability.Having said that I had a lot of bad experiences with APM tools instrumentations my application without the application owner beeing aware of it (causing a lot of support effort). From that point of view I will probably disable the dynamic loading by default, since most of my tools used in production don’t require ad-hoc instrumentation (mostly only Thread.print or Heapdump). So thanks for the reminder that this option exists (and thanks for fixing it to apply to all loading).GrussBernd-- http://bernd.eckenfels.net

  

 Von: serviceability-dev  im Auftrag von Jaroslav Bachorik Gesendet: Montag, März 20, 2023 11:37 AMAn: Ron Pressler Cc: Andrei Pangin ; jigsaw-...@openjdk.org ; serviceability-dev@openjdk.org Betreff: Re: [External] : Re: Disallowing the dynamic loading of agents by default Hi,On Mon, Mar 20, 2023 at 11:11 AM Ron Pressler <ron.press...@oracle.com> wrote:




Hi.


The majority of serviceability tools don’t require dynamically loading an agent, and the majority of applications never load an agent dynamically.The majority of the JDK built-in tools, I would say. What about eg. the JMC agent? 


True, there are some tools that will be affected, which is why the decision was to introduce the flag in JDK 9 and to announce this change, but change the default in a later version to give tools ample time to prepare their users. The rationale
 for this change then hasn’t changed, but will be reiterated in a JEP (we just wanted to announce this ahead of the JEP to give tool authors another reminder more than six months ahead of JDK 21). The only change between then and now is that even fewer use
 cases require dynamically loaded agents, and so the impact is even smallerAs a maintainer of one of such tools I can confidently say that this change will either kill the tool as the ease of use will be gone or the workaround (eg. using JAVA_TOOL_OPTIONS) will completely defeat the purpose of this change. Having to put a flag when starting the JVM to allow dynamic loading of agents sounds a bit nonsensical to me - it would be much easier to directly add the agent to the JVM startup and then implement a lightweight control protocol over socket/shared memory to enabled/disable the agent features dynamically. 


It is also true that, when starting an application you don’t know that you *will* need to load an agent, but in most situations you know that you might. E.g. processes that are too critical to bring down even for deep maintenance (although not
 many of these are written in modern version of Java anyone) or canary services that are under trial. The relatively few sophisticated users who know how to write ad-hoc agents can even opt to enable dynamic agent loading on all their servers; these users are
 better equipped to can weigh the risks and tradeoffs involved.Wouldn't having this enabled system-wide actually defeat the purpose of having this flag? Considering that the dynamic attach can be performed only on the same host under the same user as the target process there seems to be a very small chance of loading agents accidentally. In the end people would set up their systems to enabled dynamic agent loading via eg. JAVA_TOOL_OPTIONS and we will be in the same place as before, with the additional hurdle of setting everything up.


Finally, some tools that require a dynamically loaded JVM TI agents, such as profilers that profile native code, are so tied to the VM's internals that the best place for them is in the JDK. If anything, the bigger problem is not that profilers
 are used too much in production, but too little, including less advanced ones that don’t require an agent. There is plenty of time to enhance the JDK’s built-in profiling capabilities ahead of demand.I think this is an overly optimistic view. It is *much more* difficult to enhance the JDK's built-in profiling capabilities than do the same in an external profiling agent.Overall, I don't seem to understand the anticipated attack vectors this change is supposed to prevent. AFAIK, in order to do the dynamic agent load one needs to have full access to the target process. That means that there are more convenient and straightforward ways to do anything nefarious than loading a JVMTI agent. Am I missing some other usages where the JVMTI agent would actually give access to something which would be otherwise inaccessible considering that the attacher and attachee must be on the same host and under the same user?Cheers,-JB- 


— Ron


On 20 Mar 2023, at 01:21, Andrei Pangin <andrei.pan...@gmail.com> wrote:


Hi all,

Serviceability has been one of the biggest Java strengths, but the proposed change is going to have a large negative impact on it

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Jaroslav Bachorik]

Hi,

On Mon, Mar 20, 2023 at 11:11 AM Ron Pressler 
wrote:

> Hi.
>
> The majority of serviceability tools don’t require dynamically loading an
> agent, and the majority of applications never load an agent dynamically.
>

The majority of the JDK built-in tools, I would say. What about eg. the JMC
agent?


>
> True, there are some tools that will be affected, which is why the
> decision was to introduce the flag in JDK 9 and to announce this change,
> but change the default in a later version to give tools ample time to
> prepare their users. The rationale for this change then hasn’t changed, but
> will be reiterated in a JEP (we just wanted to announce this ahead of the
> JEP to give tool authors another reminder more than six months ahead of JDK
> 21). The only change between then and now is that even fewer use cases
> require dynamically loaded agents, and so the impact is even smaller.
>

As a maintainer of one of such tools I can confidently say that this change
will either kill the tool as the ease of use will be gone or the workaround
(eg. using JAVA_TOOL_OPTIONS) will completely defeat the purpose of this
change. Having to put a flag when starting the JVM to allow dynamic loading
of agents sounds a bit nonsensical to me - it would be much easier to
directly add the agent to the JVM startup and then implement a lightweight
control protocol over socket/shared memory to enabled/disable the agent
features dynamically.


>
> It is also true that, when starting an application you don’t know that you
> *will* need to load an agent, but in most situations you know that you
> might. E.g. processes that are too critical to bring down even for deep
> maintenance (although not many of these are written in modern version of
> Java anyone) or canary services that are under trial. The relatively few
> sophisticated users who know how to write ad-hoc agents can even opt to
> enable dynamic agent loading on all their servers; these users are better
> equipped to can weigh the risks and tradeoffs involved.
>

Wouldn't having this enabled system-wide actually defeat the purpose of
having this flag? Considering that the dynamic attach can be performed only
on the same host under the same user as the target process there seems to
be a very small chance of loading agents accidentally. In the end people
would set up their systems to enabled dynamic agent loading via eg.
JAVA_TOOL_OPTIONS and we will be in the same place as before, with the
additional hurdle of setting everything up.


> Finally, some tools that require a dynamically loaded JVM TI agents, such
> as profilers that profile native code, are so tied to the VM's internals
> that the best place for them is in the JDK. If anything, the bigger problem
> is not that profilers are used too much in production, but too little,
> including less advanced ones that don’t require an agent. There is plenty
> of time to enhance the JDK’s built-in profiling capabilities ahead of
> demand.
>

I think this is an overly optimistic view. It is *much more* difficult to
enhance the JDK's built-in profiling capabilities than do the same in an
external profiling agent.


Overall, I don't seem to understand the anticipated attack vectors this
change is supposed to prevent. AFAIK, in order to do the dynamic agent load
one needs to have full access to the target process. That means that there
are more convenient and straightforward ways to do anything nefarious than
loading a JVMTI agent. Am I missing some other usages where the JVMTI agent
would actually give access to something which would be otherwise
inaccessible considering that the attacher and attachee must be on the same
host and under the same user?

Cheers,

-JB-


>
> — Ron
>
> On 20 Mar 2023, at 01:21, Andrei Pangin  wrote:
>
> Hi all,
>
> Serviceability has been one of the biggest Java strengths, but the
> proposed change is going to have a large negative impact on it.
>
> Disallowing dynamic agents by default means it will no longer be possible
> to attach a profiler to a running app in runtime. JFR cannot close this gap
> due to lack of capabilities modern Java profilers have (that's a separate
> topic though).
>
> When an issue happens with a live app, it's already too late to add a
> command line argument. Furthermore, it may not be even feasible to add an
> agent at startup in containerized applications. Starting profiler on demand
> from the host OS or from a sidecar is the only viable solution in these
> cases.
>
> Next, it's hard to predict beforehand what tools exactly might be useful
> for troubleshooting: e.g., one tool may be better for finding memory leaks,
> a different one for analyzing CPU performance. Adding all possible tools at
> startup does not seem a reasonable approach, especially when tools may
> conflict with each other.
>
> The most important aspect of dynamic agents is the possibility to make a
> special tool just in time for solving a particular problem. A typical
> example is to get a value of 

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Ron Pressler]

Hi.

The majority of serviceability tools don’t require dynamically loading an 
agent, and the majority of applications never load an agent dynamically.

True, there are some tools that will be affected, which is why the decision was 
to introduce the flag in JDK 9 and to announce this change, but change the 
default in a later version to give tools ample time to prepare their users. The 
rationale for this change then hasn’t changed, but will be reiterated in a JEP 
(we just wanted to announce this ahead of the JEP to give tool authors another 
reminder more than six months ahead of JDK 21). The only change between then 
and now is that even fewer use cases require dynamically loaded agents, and so 
the impact is even smaller.

It is also true that, when starting an application you don’t know that you 
*will* need to load an agent, but in most situations you know that you might. 
E.g. processes that are too critical to bring down even for deep maintenance 
(although not many of these are written in modern version of Java anyone) or 
canary services that are under trial. The relatively few sophisticated users 
who know how to write ad-hoc agents can even opt to enable dynamic agent 
loading on all their servers; these users are better equipped to can weigh the 
risks and tradeoffs involved.

Finally, some tools that require a dynamically loaded JVM TI agents, such as 
profilers that profile native code, are so tied to the VM's internals that the 
best place for them is in the JDK. If anything, the bigger problem is not that 
profilers are used too much in production, but too little, including less 
advanced ones that don’t require an agent. There is plenty of time to enhance 
the JDK’s built-in profiling capabilities ahead of demand.

— Ron

On 20 Mar 2023, at 01:21, Andrei Pangin 
mailto:andrei.pan...@gmail.com>> wrote:

Hi all,

Serviceability has been one of the biggest Java strengths, but the proposed 
change is going to have a large negative impact on it.

Disallowing dynamic agents by default means it will no longer be possible to 
attach a profiler to a running app in runtime. JFR cannot close this gap due to 
lack of capabilities modern Java profilers have (that's a separate topic 
though).

When an issue happens with a live app, it's already too late to add a command 
line argument. Furthermore, it may not be even feasible to add an agent at 
startup in containerized applications. Starting profiler on demand from the 
host OS or from a sidecar is the only viable solution in these cases.

Next, it's hard to predict beforehand what tools exactly might be useful for 
troubleshooting: e.g., one tool may be better for finding memory leaks, a 
different one for analyzing CPU performance. Adding all possible tools at 
startup does not seem a reasonable approach, especially when tools may conflict 
with each other.

The most important aspect of dynamic agents is the possibility to make a 
special tool just in time for solving a particular problem. A typical example 
is to get a value of some field in a live app without dumping the entire 60 GB 
heap. Another common use case is hot patching for fixing trivial bugs or for 
adding debug logs dynamically. The prominent example is when the dynamic agent 
has proved irreplaceable aid in addressing the notorious log4j vulnerabilities 
CVE-2021-44228 and CVE-2021-45046.

I would be grateful to know more about the reasons why we should give up all 
the above advantages of dynamic agents in their good and legitimate use cases.

Thank you,
Andrei

чт, 16 мар. 2023 г. в 18:48, Ron Pressler 
mailto:ron.press...@oracle.com>>:
Hi.

In JDK 21 we intend to disallow the dynamic loading of agents by default. This
will affect tools that use the Attach API to load an agent into a JVM some time
after the JVM has started [1]. There is no change to any of the mechanisms that
load an agent at JVM startup (-javaagent/-agentlib on the command line or the
Launcher-Agent-Class attribute in the main JAR's manifest).

This change in default behavior was proposed in 2017 as part of JEP 261 [2][3].
At that time the consensus was to switch to this default not in JDK 9 but in a
later release to give tool maintainers sufficient time to inform their users.
To allow the dynamic loading of agents, users will need to specify
-XX:+EnableDynamicAgentLoading on the command line.

I'll post a draft JEP for review shortly.

-- Ron

[1]: 
https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html
[2]: https://openjdk.org/jeps/261
[3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html

Re: Disallowing the dynamic loading of agents by default

[Andrei Pangin]

Hi all,

Serviceability has been one of the biggest Java strengths, but the proposed
change is going to have a large negative impact on it.

Disallowing dynamic agents by default means it will no longer be possible
to attach a profiler to a running app in runtime. JFR cannot close this gap
due to lack of capabilities modern Java profilers have (that's a separate
topic though).

When an issue happens with a live app, it's already too late to add a
command line argument. Furthermore, it may not be even feasible to add an
agent at startup in containerized applications. Starting profiler on demand
from the host OS or from a sidecar is the only viable solution in these
cases.

Next, it's hard to predict beforehand what tools exactly might be useful
for troubleshooting: e.g., one tool may be better for finding memory leaks,
a different one for analyzing CPU performance. Adding all possible tools at
startup does not seem a reasonable approach, especially when tools may
conflict with each other.

The most important aspect of dynamic agents is the possibility to make a
special tool just in time for solving a particular problem. A typical
example is to get a value of some field in a live app without dumping the
entire 60 GB heap. Another common use case is hot patching for fixing
trivial bugs or for adding debug logs dynamically. The prominent example is
when the dynamic agent has proved irreplaceable aid in addressing the
notorious log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046.

I would be grateful to know more about the reasons why we should give up
all the above advantages of dynamic agents in their good and legitimate use
cases.

Thank you,
Andrei

чт, 16 мар. 2023 г. в 18:48, Ron Pressler :

> Hi.
>
> In JDK 21 we intend to disallow the dynamic loading of agents by default.
> This
> will affect tools that use the Attach API to load an agent into a JVM some
> time
> after the JVM has started [1]. There is no change to any of the mechanisms
> that
> load an agent at JVM startup (-javaagent/-agentlib on the command line or
> the
> Launcher-Agent-Class attribute in the main JAR's manifest).
>
> This change in default behavior was proposed in 2017 as part of JEP 261
> [2][3].
> At that time the consensus was to switch to this default not in JDK 9 but
> in a
> later release to give tool maintainers sufficient time to inform their
> users.
> To allow the dynamic loading of agents, users will need to specify
> -XX:+EnableDynamicAgentLoading on the command line.
>
> I'll post a draft JEP for review shortly.
>
> -- Ron
>
> [1]:
> https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html
> [2]: https://openjdk.org/jeps/261
> [3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html

Re: Disallowing the dynamic loading of agents by default

[Kirk Pepperdine]

I need to retrace this thread to gain more context but my initial thoughts were 
to all of the tools and techniques that I use and how vulnerable they are to 
this change vs. what the motivation is for this change. My initial assessment 
is that this  is going to heavily impact visibility and wipe out the use of so 
many tools making is so much more difficult than it already is. Especially is 
you prescribe to a top-down methodical targeted approach to trouble-shooting 
rather than a shotgun gather everything you can methodology. The former often 
requires re-instrumentation on the fly. Shutting down to restart when some 
problems may take a couple of weeks to show really isn’t a great option. I 
guess you could just turn things back on but then I’d likely recommend that as 
an across the board setting. Again, I need to dig about to get more context.

Kind regards,
Kirk

> On Mar 19, 2023, at 2:27 AM, Alan Bateman  wrote:
> 
> On 19/03/2023 02:51, Yasumasa Suenaga wrote:
>> :
>> 
>> Can we change flag type of EnableDynamicAgentLoading to `manageable` from 
>> `product`? If so, we can use JVMTI agent without rebooting system when we 
>> encountered some troubles in production system.
> 
> If manageable then it could be enabled at run-time with 
> HotSpotDiagnosticMXBean.setVMOption (or jcmd VM.set_flag), so I think 
> wouldn't change anything. The main issue with JVMTI agents loaded into a 
> running VM is that they can do anything. Even if their capabilities were 
> reduced (and many debugging capabilities are only available in the onload 
> phase) it can still use JNI and bypass access control. So I think a difficult 
> security vs. serviceability trade-off here.
> 
> -Alan.

Re: Disallowing the dynamic loading of agents by default

[Alan Bateman]

On 19/03/2023 02:51, Yasumasa Suenaga wrote:

:

Can we change flag type of EnableDynamicAgentLoading to `manageable` 
from `product`? If so, we can use JVMTI agent without rebooting system 
when we encountered some troubles in production system.


If manageable then it could be enabled at run-time with 
HotSpotDiagnosticMXBean.setVMOption (or jcmd VM.set_flag), so I think 
wouldn't change anything. The main issue with JVMTI agents loaded into a 
running VM is that they can do anything. Even if their capabilities were 
reduced (and many debugging capabilities are only available in the 
onload phase) it can still use JNI and bypass access control. So I think 
a difficult security vs. serviceability trade-off here.


-Alan.

Re: Disallowing the dynamic loading of agents by default

[Yasumasa Suenaga]

HI,

I haven't followed this topic, but I think dynamic loading mechanism of JVMTI 
agent is useful for debugging.

Can we change flag type of EnableDynamicAgentLoading to `manageable` from 
`product`? If so, we can use JVMTI agent without rebooting system when we 
encountered some troubles in production system.


Thanks,

Yasumasa


On 2023/03/17 3:48, Ron Pressler wrote:

Hi.

In JDK 21 we intend to disallow the dynamic loading of agents by default. This
will affect tools that use the Attach API to load an agent into a JVM some time
after the JVM has started [1]. There is no change to any of the mechanisms that
load an agent at JVM startup (-javaagent/-agentlib on the command line or the
Launcher-Agent-Class attribute in the main JAR's manifest).

This change in default behavior was proposed in 2017 as part of JEP 261 [2][3].
At that time the consensus was to switch to this default not in JDK 9 but in a
later release to give tool maintainers sufficient time to inform their users.
To allow the dynamic loading of agents, users will need to specify
-XX:+EnableDynamicAgentLoading on the command line.

I'll post a draft JEP for review shortly.

-- Ron

[1]: 
https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html
[2]: https://openjdk.org/jeps/261
[3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Alan Bateman]

On 17/03/2023 14:11, Thomas Stüfe wrote:

:

Investigation shows that there seems to be a bug in attachListener.cpp 
where we compare AttachOperation::name for "load", but it contains 
"jcmd":


When using the Attach API, the VirtualMachine.loadAgentXXX methods map 
to a "load" command. The Attach API, jstack, jmap ... pre-date the jcmd 
tool and have their own set of commands known to both the tool/client 
side and the VM side.  The "jcmd" command, used by the jcmd tool, works 
a bit like a HTTP upgrade. So different path, and you are right, jcmd 
JVMTI.agent_load was missed when EnableDynamicAgentLoading was added in 
JDK 9. I've created JDK-8304438 to track it.


-Alan

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Ron Pressler]

> On 17 Mar 2023, at 14:11, Thomas Stüfe  wrote:
> 
> Thank you for the clarification.
> 
> Oddly enough, -XX:-EnableDynamicAgentLoading seems to be broken. Tried head 
> (fastdebug, release) and JDK17, even with this switch my sample library loads 
> just fine:
> 
> ```
> thomas@starfish$ ./images/jdk/bin/java -XX:-EnableDynamicAgentLoading 
> -XX:+PrintFlagsFinal  -cp $REPROS_JAR de.stuefe.repros.Simple 
>   
> [Global flags]
> 
> ...
>  bool EnableDynamicAgentLoading= false
>  {product} {command line}
> ...
> 
> OnAttach! Loading JVMTI sample agent
> ```
> 
> Investigation shows that there seems to be a bug in attachListener.cpp where 
> we compare AttachOperation::name for "load", but it contains "jcmd": 
> 
> ```
> Thread 22 "Attach Listener" hit Breakpoint 1, attach_listener_thread_entry 
> (thread=0x7fff94000fd0, __the_thread__=0x7fff94000fd0) at 
> /shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/services/attachListener.cpp:404
> 404 } else if (!EnableDynamicAgentLoading && strcmp(op->name(), 
> "load") == 0) {
> (gdb) p op
> $1 = (AttachOperation *) 0x7fff7401b640
> (gdb) p *op
> $2 = {> = {}, _vptr.AttachOperation = 
> 0x77b61210 , _name = "jcmd\000", 
> '\361' , , _arg = {
> "JVMTI.agent_load /shared/projects/jvmti-sample/sample.so\000", '\361' 
> ..., "\000", '\361' ..., "\000", 
> '\361' ...}}
> (gdb) p op->name()
> $3 = 0x7fff7401b648 "jcmd"
> ```
> 
> This was on Linux x64.
> 
> So if people have been using -XX:-EnableDynamicAgentLoading to check their 
> code, this may not have worked as intended.
> 
> Cheers, Thomas

There may be a missing check in JVMTIAgentLoadDCmd::execute in 
diagnosticCommand.cpp.
Thank you for reporting this!

— Ron

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Thomas Stüfe]

Thank you for the clarification.

Oddly enough, -XX:-EnableDynamicAgentLoading seems to be broken. Tried head
(fastdebug, release) and JDK17, even with this switch my sample library
loads just fine:

```
thomas@starfish$ ./images/jdk/bin/java -XX:-EnableDynamicAgentLoading
-XX:+PrintFlagsFinal  -cp $REPROS_JAR de.stuefe.repros.Simple


[Global flags]

...
 bool EnableDynamicAgentLoading= false
{product} {command line}
...

OnAttach! Loading JVMTI sample agent
```

Investigation shows that there seems to be a bug in attachListener.cpp
where we compare AttachOperation::name for "load", but it contains "jcmd":

```
Thread 22 "Attach Listener" hit Breakpoint 1, attach_listener_thread_entry
(thread=0x7fff94000fd0, __the_thread__=0x7fff94000fd0) at
/shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/services/attachListener.cpp:404
404 } else if (!EnableDynamicAgentLoading && strcmp(op->name(),
"load") == 0) {
(gdb) p op
$1 = (AttachOperation *) 0x7fff7401b640
(gdb) p *op
$2 = {> = {}, _vptr.AttachOperation =
0x77b61210 , _name = "jcmd\000",
'\361' , , _arg = {
"JVMTI.agent_load /shared/projects/jvmti-sample/sample.so\000", '\361'
..., "\000", '\361' ..., "\000",
'\361' ...}}
(gdb) p op->name()
$3 = 0x7fff7401b648 "jcmd"
```

This was on Linux x64.

So if people have been using -XX:-EnableDynamicAgentLoading to check their
code, this may not have worked as intended.

Cheers, Thomas



On Fri, Mar 17, 2023 at 2:42 PM Ron Pressler 
wrote:

>
>
> On 17 Mar 2023, at 13:33, Thomas Stüfe  wrote:
>
> Hi Ron,
>
> Will this affect attaching via jcmd?
>
>
> The Attach mechanism will not be disabled by default, just the ability to
> load agents via the Attach mechanism.
> So the only jcmd command that will be affected is JVMTI.agent_load.
>
> To see the effect of the change today, launch java with
> -XX:-EnableDynamicAgentLoading, which is
> to become the new default.
>
> — Ron
>
>

Re: [External] : Re: Disallowing the dynamic loading of agents by default

[Ron Pressler]

On 17 Mar 2023, at 13:33, Thomas Stüfe 
mailto:thomas.stu...@gmail.com>> wrote:

Hi Ron,

Will this affect attaching via jcmd?

The Attach mechanism will not be disabled by default, just the ability to load 
agents via the Attach mechanism.
So the only jcmd command that will be affected is JVMTI.agent_load.

To see the effect of the change today, launch java with 
-XX:-EnableDynamicAgentLoading, which is
to become the new default.

— Ron

Re: Disallowing the dynamic loading of agents by default

[Thomas Stüfe]

Hi Ron,

Will this affect attaching via jcmd?

Thanks, Thomas

On Thu, Mar 16, 2023 at 7:48 PM Ron Pressler 
wrote:

> Hi.
>
> In JDK 21 we intend to disallow the dynamic loading of agents by default.
> This
> will affect tools that use the Attach API to load an agent into a JVM some
> time
> after the JVM has started [1]. There is no change to any of the mechanisms
> that
> load an agent at JVM startup (-javaagent/-agentlib on the command line or
> the
> Launcher-Agent-Class attribute in the main JAR's manifest).
>
> This change in default behavior was proposed in 2017 as part of JEP 261
> [2][3].
> At that time the consensus was to switch to this default not in JDK 9 but
> in a
> later release to give tool maintainers sufficient time to inform their
> users.
> To allow the dynamic loading of agents, users will need to specify
> -XX:+EnableDynamicAgentLoading on the command line.
>
> I'll post a draft JEP for review shortly.
>
> -- Ron
>
> [1]:
> https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html
> [2]: https://openjdk.org/jeps/261
> [3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html

Disallowing the dynamic loading of agents by default

[Ron Pressler]

Hi.

In JDK 21 we intend to disallow the dynamic loading of agents by default. This
will affect tools that use the Attach API to load an agent into a JVM some time
after the JVM has started [1]. There is no change to any of the mechanisms that
load an agent at JVM startup (-javaagent/-agentlib on the command line or the
Launcher-Agent-Class attribute in the main JAR's manifest).

This change in default behavior was proposed in 2017 as part of JEP 261 [2][3]. 
At that time the consensus was to switch to this default not in JDK 9 but in a 
later release to give tool maintainers sufficient time to inform their users. 
To allow the dynamic loading of agents, users will need to specify 
-XX:+EnableDynamicAgentLoading on the command line.

I'll post a draft JEP for review shortly.

-- Ron

[1]: 
https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html
[2]: https://openjdk.org/jeps/261
[3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html