On May 29, 2018, at 15:54, Frederic Parain <frederic.par...@oracle.com> wrote:
Hi Thomas,
On May 26, 2018, at 04:07, Thomas Stüfe <thomas.stu...@gmail.com> wrote:
<snip>
I think that it is a fundamental piece of the design. Adding a new diagnostic
command to group several commands under a unique safepoint:
1 - requires explicit action from the user (and acknowledgment of the risks)
2 - doesn’t touch the current syntax/semantic at all
Another reason I’m pushing for a specific diagnostic command rather than
changing the syntax is remote execution. jcmd is not the only way to invoke
a diagnostic command, they can also be invoke remotely using JMX and the
com.sun.management.DiagnosticCommand dynamic MBean. Through the
platform MBean, users do not have access to the jcmd syntax, they see
each diagnostic command as a MBean operation (providing that they have
been registered with the DCmd_Source_MBean flag). The initial support
for remote execution was taking a single String and processed it just like
a string passed by jcmd, but people developing consoles for remote
management complained that this was not convenient for them, and asked
for something they could use more programmatically.
If the safepoint bundling is implemented in the syntax, JMX clients won’t
be able to use the feature. If we implement it with a new diagnostic command,
it will be visible from the MBean and they will be able to use it.
That is interesting. Do you have examples for such programs? I was
looking for ways of how to use jcmd (or equivalent) remotely.
If you’re looking for a console application to invoke diagnostic command
on a remote JVM, Mission Control has a dedicated support for that in the
“Diagnostic Commands” tab.
If you’re looking for exemples of source code doing remote invocations,
there’re several unit tests for the DiagnosticCommand MBean in the directory
test/jdk/com/sun/management/DiagnosticCommandMBean.
At last, creating a specific diagnostic command for safepoint bundling would
show the way on how to group commands for consistency. Safepoints is not
the only way to get consistency. For some JVM internals reasons, you might
want to group commands under a single lock, like the Threads_lock. Adding
a diagnostic command for each type of bundling seems more extensible.
You make good points.
Adding a new command has also the advantage that it does not affect
downward compatibility and hence would not require a CSR. CSR seems to
slow down things a bit (I have a very simple CSR for jcmd open
undecided since 12 days:
https://bugs.openjdk.java.net/browse/JDK-8203043). So, while I think
CSR are a necessary process, I am happy to avoid them if possible.
Lets spin this out a bit more:
-> However the user specifies "please execute these commands at a
single safepoint", if one of the enclosed commands cannot be run
inside a safepoint the whole bundle should fail, agreed? So,
specifying e.g. JFR.start as part of such a command bundle would cause
the whole bundle to not being executed.
Yes. The new command N would ask for the execution of a list of
commands { A, B, C, … }, and would fail if:
- any of the specified commands { A, B, C, … } is not a registered
command
- any of the specified commands { A, B, C, … } doesn’t support
execution at safepoint
- any of the specified commands { A, B, C, … } is not exported to
the DCmdSource used to invoke N (*)
(*) Not all commands are exported to all interfaces (jcmd,JMX), for
instance, some commands related to the JMX agent are cannot
be invoked through the JMX. Even if the new command can be
invoked from all interfaces, it must not allow to circumvent this
restriction. This should not be an issue today, because all
JMX agent commands are calling back Java code, and are not
suitable for execution at a safepoint. But this would ensure the
consistency of the model in the long term (and the test is cheap).
-> If we introduce a command like you propose (lets call it "VM.run"
for the sake of the argument) we still need to decide how to delimit
the commands at the command line. That brings us back to the syntax
question: separation by a explicit delimiter, if possible one that
does not clash with common unix shells?
Lets try colon ':' :
jcmd VM.run VM.uptime : VM.metaspace show-loaders show-classes
by-chunktype : VM.classloaders show-classes : VM.info
Of course, we can wrap all arguments to jcmd in quotes, then the
delimiter sign does not affect things as much, unless we use quotes :)
We could e.g. use semicolon:
jcmd ' VM.run VM.uptime ; VM.metaspace show-loaders show-classes
by-chunktype ; VM.classloaders show-classes ; VM.info ‘
I thought about colon and semi-colon too, because they are the most
obvious delimiters. However, they are already as delimiters in classpaths
(semi-colon on Unix, colon on Windows), so parsing to find the right
delimiters would be tricky.
I’d start with ‘!’ or ‘+’ in a first time, to make some progress on the other
aspects of the command, and buy some time to discuss the delimiter
issue in parallel.
The last resort would be to force each command in the bundle to
be specified with quotes (arguments included). But it would make the
syntax very heavy, I’d prefer a simpler delimiter.
Or, we go back to my original idea of letting the command keyword be
the indicator for a new command start. Honestly, at this point I have
no preference. Both approaches seem reasonable.
The only niggle I have with an explicit command is the syntax, which I
find a bit awkward. My thinking before reading your response was that
I'd prefer a jcmd flag to specify the same behavior: that all commands
should be executed at a single safepoint. E.g. [-s/--stacked]
jcmd -s ' VM.uptime ; VM.metaspace show-loaders show-classes
by-chunktype ; VM.classloaders show-classes ; VM.info '
But I see the disadvantage: depending on how it is implemented this
would require consistent changes in both jcmd and hotspot, and not
work with other clients using MBeans. Whereas the hypothetical
"VM.run" would work out of the box for any old jcmd version and any
other client.
Of course one could keep the "VM.run" idea and still give jcmd a
"-s/--stacked" option, as a shallow convenience feature, which just
translates to "VM.run" and runs that.
-> A variation of your "have special commands to run a command bundle
in one particular context" theme could be having only one command,
e.g. "VM.run", specifying the conditions under which to run the bundle
as sub options. E.g. "VM.run -s <commands>" for "run commands at
safepoint", "VM.run -t <commands>" for "run commands under
Threads_lock". That would even allow to combine these contexts. I do
not know if that makes sense.
- Another advantage of having an explicit command just occurring to me
is that it enables more evolved command files, where one can
intersperse command bundles and single commands. For instance, I could
give this to a customer:
<command file>
1 VM.uptime
2 VM.run_commands VM.metaspace show-loaders show-classes by-chunktype
VM.classloaders details VM.classloader_stats
3 GC.run
4 VM.run_commands VM.metaspace show-loaders show-classes by-chunktype
VM.classloaders details VM.classloader_stats
to execute a command bundle before and after triggering a GC.
This would work fine . The parser of the jcmd string will split the
string based on ‘\n’ characters, then each line of command will be executed
sequentially, and lines containing the VM.run_command will then create the
safepoint and trigger the execution of the different commands specified in
its argument list.
We have a three days week-end in the US, but I should be able to code
something next week.
Oh, you want to take over the implementation, sure! Fine by me.
I’m not promising an implementation, I haven’t worked on this code
since I left the serviceability team. I need to code at least a skeleton
of the command to refresh my memory about all aspects of the
diagnostic command framework.
Thank you for your ideas and your interest in diagnostic commands.
Sure. Why I am interested in this: We strongly rely on tools similar
to jcmd for our support. We have quite capable tools in our (licensed,
non-open) VM port, which are similar to jcmd in many aspects. So we
try to bring some of the capabilities upstream, in order to ease the
merge costs for us and to help the OpenJDK community.
The diagnostic command framework was designed to support an
extensible set of commands. Feel free to extend, and thank you
for contributing to the OpenJDK community.
Fred
A new separator, more convenient than newline, would be require to have
a single line syntax.
My 2 cents,
Certainly worth more than 2 cents :) Thanks for your thoughts!
..Thomas
Fred
Erik
Hi Thomas,
very positive suggestion.
One observation:
There already seems to be some different interpretation of the command
line
in different Java versions:
For instance when I try to dump a flight recording in 1.8.0_152-ea-b05 :
I
can use:
jcmd 33014 JFR.dump filename="recording1.jfr" recording=1
but in build 9+181 , the same command must be:
jcmd 33014 JFR.dump filename="recording1.jfr",recording=1
(the comma to separate sub-options you mentioned earlier)
Therefore the suggestion without curly brackets, giving several commands
after one another seems good with regard to backwards compatibility.
Motivation: hawt.io uses the MBean
com.sun.management:type=DiagnosticCommand
to access the same functionality as jcmd. Variations in option syntax
across
Java versions is already an issue (only affecting sub-options though, as
each command is a separate JMX operation). So syntax compatibility is
highly
appreciated :-)
Martin
lør. 12. mai 2018 kl. 20:11 skrev Kirk Pepperdine
<kirk.pepperd...@gmail.com>:
On May 10, 2018, at 11:26 AM, Thomas Stüfe <thomas.stu...@gmail.com>
wrote:
On Thu, May 10, 2018 at 9:13 AM, Kirk Pepperdine
<kirk.pepperd...@gmail.com> wrote:
The stacking at the safe point would be a huge win. Right now thread
dump consistency can really confuse people as the tooling will show
two
threads owning the same lock at seemingly the same time. Reality, it’s
just
that people didn’t realize you get a safe point for each thread
therefor
there is motion in the system as you’re collecting the data.
I am a bit confused. What tooling are you talking about?
jstack. I’ve not seen it with jcmd. But I often see 2 threads holding
the
same lock at the “same time” which is of course non-sense. I can dig one
up
for you if you’re still confused.
As an aside, it’s amazing how many dev’s aren’t aware of jcmd. Just
yesterday after my session at Devoxx I had someone ask me about using
jfr
from the command line, many in that session had not seen jcmd before.
The
feedback was, wow, this is very useful and I wished I had of known
about it
earlier.
Yes, jcmd is quite useful. I also really like the simple design, which
is by default up- and downward compatible (so you can talk to any VM,
new, old, it should not matter). That is just good design. We - Sap -
work to extend jcmd, to help our support folks. E.g. the whole new
VM.metaspace command chain.
And simple it is….well done!!!
— Kirk
