JDK 22 RDP2 & Deprecate sun.misc.Unsafe Memory-Access Methods…
Greetings! We are starting 2024 with JDK 22 as it has just entered Rampdown Phase 2 [1]. And with the initial JDK 22 Release Candidates now less than 2 weeks away (Feb. 8th) [2], it is time to shift our attention to JDK 23. After multiple rounds of incubations and preview, the Foreign Function & Memory API is becoming standard and permanent in JDK 22. If we put its 'Function' angle aside, this API also offers a standard and secure way to access off-heap API. And that brings us to the heads-up below 'Deprecate the memory-access methods in sun.misc.Unsafe for removal in a future release' as developers still using sun.misc.Unsafe for accessing memory are strongly encouraged to start preparing their plans to migrate away from those unsafe methods. [1] https://mail.openjdk.org/pipermail/jdk-dev/2024-January/008675.html [2] https://openjdk.org/projects/jdk/22/ ## Heads-up: Deprecate the Memory-Access Methods in sun.misc.Unsafe for Removal in a Future Release The effort focused on enforcing the integrity of the Java platform [3] continues! The next phase in that long but important initiative will most likely target the sun.misc.Unsafe API used for accessing memory. Those methods alone represent 79 methods out of the 87 sun.misc.Unsafe methods! This draft JEP [4] outlines the plan to deprecate for removal the sun.misc.Unsafe Memory-Access methods, the reasons, and the standard alternatives. As the draft plan suggests, the first step will be to deprecate all memory-access methods (on-heap, off-heap, and bimodal) for removal. This will cause compile-time deprecation warnings for code that refers to the methods, alerting library developers to their forthcoming removal. In addition, a new command-line option will allow users to receive runtime warnings when those methods are used. This command-line will help users to assess if their codebase uses those unsafe API to access memory. It should be mentioned that other tools such as JFR and jdeprscan can also be used to detect the use of those deprecated APIs. Library developers are strongly encouraged to migrate from sun.misc.Unsafe to the supported replacements, so that applications can migrate smoothly to modern JDKs. The initial step will be to conduct investigations to understand if, how, and where sun.misc.Unsafe methods are used to access memory. [3] https://openjdk.org/jeps/8305968 [4] https://openjdk.org/jeps/8323072 ## Heads-up: Java Array Element Alignment - Weakening of Some Methods Guarantees ? Some methods make promises about Java array element alignment that are too strong. There are some ongoing reflexions to change the implementation (and the specification) of `MethodHandles::byteArrayViewVarHandle`, `MethodHandles::byteBufferViewVarHandle`, `ByteBuffer::alignedSlice`, and `ByteBuffer::alignmentOffset` to weaken the guarantees they make about the alignment of Java array elements, in order to bring them in line with the guarantees made by an arbitrary JVM implementation. For more details, make sure to check JDK-8320247 [5] and the related PR [6] but in a nutshell, the new behaviour would be : - The `VarHandle` returned by `MethodHandles::byteArrayViewVarHandle` would only support `get` and `set` methods, and all other access methods would throw an exception. - The `VarHandle` returned by `MethodHandles::byteBufferViewHandle` would only support the `get` and `set` access methods when a heap buffer is used, and all other access methods would throw an exception when used with a heap buffer. Direct byte buffers will continue to work the same way. - The `ByteBuffer::alignmentOffset` and `ByteBuffer::alignedSlice` methods would throw an exception if the buffer is a heap buffer, and the given `unitSize` is greater than 1. If you have relevant feedback about this potential change, please make sure to bring it to the core-libs-dev mailing list [7], or comment on the PR [6]. [5] https://bugs.openjdk.org/browse/JDK-8320247 [6] https://github.com/openjdk/jdk/pull/16681 [7] https://mail.openjdk.org/pipermail/core-libs-dev/ ## JDK 22 Early-Access Builds JDK 22 Early-Access builds 33 are now available [8], and are provided under the GNU General Public License v2, with the Classpath Exception. The Release Notes [9] and the javadocs [10] are also available. ### Changes in recent JDK 22 builds that may be of interest: - JDK-8320597: RSA signature verification fails on signed data that does not encode params correctly [Reported by Apache POI] - JDK-8322214: Return value of XMLInputFactory.getProperty() changed from boolean to String in JDK 22 early access builds [Reported by Apache POI] - JDK-8322725: (tz) Update Timezone Data to 2023d - JDK-8321480: ISO 4217 Amendment 176 Update - JDK-8314468: Improve Compiler loops - JDK-8314295: Enhance verification of verifier - JDK-8316976: Improve signature handling - JDK-8317547: Enhance TLS connection support - JDK-8318971: Better Error Handling for Jar Tool When Process
Re: [Log4j] If and how to document potential OOME
That is kind of my point. Anyone who logs an object that behaves this way is
asking for trouble. It really sounds like an esoteric edge case to me. I don’t
view this as a Log4J problem as the error wouldn’t even have Log4J in the stack
trace.
Ralph
> On Jan 25, 2024, at 9:52 PM, Gary Gregory wrote:
>
> I have no control over this. I think this is a general problem with third
> party objects and toString(). There is no toString(fromHereToThere) or
> toStringReader() so I can't see a general way to deal with it. Even if I
> created a wrapper for the Object and called toString(), truncated and
> cached the result and have the wrapper return the shorter string in its
> toString(), I still would have toString()'d the original...
>
> Gary
>
>> On Thu, Jan 25, 2024, 11:33 PM Ralph Goers
>> wrote:
>>
>> OK. The only good way to handle that is to parse the YAML/JSON file while
>> streaming it and extract just the fields you want to include in the logs.
>>
>> Ralph
>>
>>> On Jan 25, 2024, at 6:40 PM, Gary Gregory
>> wrote:
>>>
>>> Well, it's worse than that because the object is an object created by
>>> parsing a YAML (or JSON) file, then the toString() of that object
>>> renders a String in some other format.
>>>
>>> Gary
>>>
>>> On Thu, Jan 25, 2024 at 7:45 PM Ralph Goers
>> wrote:
Volkan & Matt,
Neither of those is going to help. The issue is that when the toString
>> method is called it reads a whole file in and stores it as a String. This
>> could cause the OOM error. Truncating it in a layout simply limits how much
>> of the String is printed. Even Gary’s proposal of calling substring() is
>> still going to operate on the whole String. He would really need a method
>> that accepts the max number of characters to read from the file.
Ralph
> On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı wrote:
>
> *[Just responding to Matt. I don't have an answer for Gary.]*
>
> `JsonTemplateLayout` has `maxStringLength`, and related with it,
> `truncatedStringSuffix`.
>
> On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker wrote:
>
>> You can use the %maxLength{…}{N} pattern converter with PatternLayout
>> at
>> least. Unfortunately, I don’t think any other layouts support a
>> similar
>> option.
>>
>>> On Jan 25, 2024, at 10:55, Gary D. Gregory
>> wrote:
>>>
>>> Hi All,
>>>
>>> I'd like to ask how to if we can devise advice around an issue I ran
>> into this week.
>>>
>>> One of our test suites processes about 40K files of test fixtures.
>> These
>> inputs are parsed, processed, and asserted. This randomly fails on a
>> call
>> to Logger#debug(), randomly in that it happens usually once per build,
>> somewhere in a logging call. But it usually fails with a call that
>> looks
>> like this:
>>>
>>> logger.debug("This is fun" + myFunObject);
>>>
>>> To simplify things, let's say that it turns out that after an
>> underlying
>> third party jar file version upgrade the call to
>> myFunObject#toString() no
>> longer returns Object#toString() but rather (again to simplify) the
>> contents of the file that was parsed to create myFunObject. This
>> toString()
>> can be megabytes. The solution is obvious:
>>>
>>> logger.debug("This is fun", myFunObject::toString)
>>>
>>> And our CI builds no longer randomly fail since our default logging
>> does
>> not log at the debug level.
>>>
>>> A better solution could be:
>>>
>>> logger.debug("This is fun", () -> myFunObject.toString().substring(0,
>> 100))
>>>
>>> where I still want _some_ information better than making my own
>> toString() with System#identityHashCode(Object) or somesuch. Sure,
>> .toString() is still called but it does not make it down into
>> logging. In
>> my case the OOME happened in myFunObject::toString so the substring()
>> example would not have worked.
>>>
>>> My question is: Should we document some general advice on this
>> pattern
>> and what should the advice be? Would it make sense to have some
>> features
>> where we truncate/reject Strings above a threshold. And yes, calling
>> myFunObject.toString() can still still get me in trouble.
>>>
>>> Gary
>>>
>>
>>
>>
>>
Re: [Log4j] If and how to document potential OOME
Oh, I don't want to suggest this is a Lo4j issue at all. I am wondering if
there is a pattern we should document aside from "know your third party
objects" which is not helpful for us to say. It sounds like there is
nothing for us to do.
Gary
On Fri, Jan 26, 2024, 7:16 AM Apache wrote:
> That is kind of my point. Anyone who logs an object that behaves this way
> is asking for trouble. It really sounds like an esoteric edge case to me. I
> don’t view this as a Log4J problem as the error wouldn’t even have Log4J in
> the stack trace.
>
> Ralph
>
> > On Jan 25, 2024, at 9:52 PM, Gary Gregory
> wrote:
> >
> > I have no control over this. I think this is a general problem with
> third
> > party objects and toString(). There is no toString(fromHereToThere) or
> > toStringReader() so I can't see a general way to deal with it. Even if I
> > created a wrapper for the Object and called toString(), truncated and
> > cached the result and have the wrapper return the shorter string in its
> > toString(), I still would have toString()'d the original...
> >
> > Gary
> >
> >> On Thu, Jan 25, 2024, 11:33 PM Ralph Goers
> >> wrote:
> >>
> >> OK. The only good way to handle that is to parse the YAML/JSON file
> while
> >> streaming it and extract just the fields you want to include in the
> logs.
> >>
> >> Ralph
> >>
> >>> On Jan 25, 2024, at 6:40 PM, Gary Gregory
> >> wrote:
> >>>
> >>> Well, it's worse than that because the object is an object created by
> >>> parsing a YAML (or JSON) file, then the toString() of that object
> >>> renders a String in some other format.
> >>>
> >>> Gary
> >>>
> >>> On Thu, Jan 25, 2024 at 7:45 PM Ralph Goers <
> ralph.go...@dslextreme.com>
> >> wrote:
>
> Volkan & Matt,
>
> Neither of those is going to help. The issue is that when the toString
> >> method is called it reads a whole file in and stores it as a String.
> This
> >> could cause the OOM error. Truncating it in a layout simply limits how
> much
> >> of the String is printed. Even Gary’s proposal of calling substring() is
> >> still going to operate on the whole String. He would really need a
> method
> >> that accepts the max number of characters to read from the file.
>
> Ralph
>
> > On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı wrote:
> >
> > *[Just responding to Matt. I don't have an answer for Gary.]*
> >
> > `JsonTemplateLayout` has `maxStringLength`, and related with it,
> > `truncatedStringSuffix`.
> >
> > On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker
> wrote:
> >
> >> You can use the %maxLength{…}{N} pattern converter with
> PatternLayout
> >> at
> >> least. Unfortunately, I don’t think any other layouts support a
> >> similar
> >> option.
> >>
> >>> On Jan 25, 2024, at 10:55, Gary D. Gregory
> >> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I'd like to ask how to if we can devise advice around an issue I
> ran
> >> into this week.
> >>>
> >>> One of our test suites processes about 40K files of test fixtures.
> >> These
> >> inputs are parsed, processed, and asserted. This randomly fails on a
> >> call
> >> to Logger#debug(), randomly in that it happens usually once per
> build,
> >> somewhere in a logging call. But it usually fails with a call that
> >> looks
> >> like this:
> >>>
> >>> logger.debug("This is fun" + myFunObject);
> >>>
> >>> To simplify things, let's say that it turns out that after an
> >> underlying
> >> third party jar file version upgrade the call to
> >> myFunObject#toString() no
> >> longer returns Object#toString() but rather (again to simplify) the
> >> contents of the file that was parsed to create myFunObject. This
> >> toString()
> >> can be megabytes. The solution is obvious:
> >>>
> >>> logger.debug("This is fun", myFunObject::toString)
> >>>
> >>> And our CI builds no longer randomly fail since our default logging
> >> does
> >> not log at the debug level.
> >>>
> >>> A better solution could be:
> >>>
> >>> logger.debug("This is fun", () ->
> myFunObject.toString().substring(0,
> >> 100))
> >>>
> >>> where I still want _some_ information better than making my own
> >> toString() with System#identityHashCode(Object) or somesuch. Sure,
> >> .toString() is still called but it does not make it down into
> >> logging. In
> >> my case the OOME happened in myFunObject::toString so the
> substring()
> >> example would not have worked.
> >>>
> >>> My question is: Should we document some general advice on this
> >> pattern
> >> and what should the advice be? Would it make sense to have some
> >> features
> >> where we truncate/reject Strings above a threshold. And yes, calling
> >> myFunObject.toString() can still still get me in trouble.
> >>>
> >>> Gary
> >>>
> >>
> >>
>
> >>
> >>
>
>
Re: [Log4j] If and how to document potential OOME
Perhaps mention of StringBuilderFormattable? It helps work around the “load the
string all at once” problem.
> On Jan 26, 2024, at 06:36, Gary Gregory wrote:
>
> Oh, I don't want to suggest this is a Lo4j issue at all. I am wondering if
> there is a pattern we should document aside from "know your third party
> objects" which is not helpful for us to say. It sounds like there is
> nothing for us to do.
>
> Gary
>
> On Fri, Jan 26, 2024, 7:16 AM Apache wrote:
>
>> That is kind of my point. Anyone who logs an object that behaves this way
>> is asking for trouble. It really sounds like an esoteric edge case to me. I
>> don’t view this as a Log4J problem as the error wouldn’t even have Log4J in
>> the stack trace.
>>
>> Ralph
>>
>>> On Jan 25, 2024, at 9:52 PM, Gary Gregory
>> wrote:
>>>
>>> I have no control over this. I think this is a general problem with
>> third
>>> party objects and toString(). There is no toString(fromHereToThere) or
>>> toStringReader() so I can't see a general way to deal with it. Even if I
>>> created a wrapper for the Object and called toString(), truncated and
>>> cached the result and have the wrapper return the shorter string in its
>>> toString(), I still would have toString()'d the original...
>>>
>>> Gary
>>>
On Thu, Jan 25, 2024, 11:33 PM Ralph Goers
wrote:
OK. The only good way to handle that is to parse the YAML/JSON file
>> while
streaming it and extract just the fields you want to include in the
>> logs.
Ralph
> On Jan 25, 2024, at 6:40 PM, Gary Gregory
wrote:
>
> Well, it's worse than that because the object is an object created by
> parsing a YAML (or JSON) file, then the toString() of that object
> renders a String in some other format.
>
> Gary
>
> On Thu, Jan 25, 2024 at 7:45 PM Ralph Goers <
>> ralph.go...@dslextreme.com>
wrote:
>>
>> Volkan & Matt,
>>
>> Neither of those is going to help. The issue is that when the toString
method is called it reads a whole file in and stores it as a String.
>> This
could cause the OOM error. Truncating it in a layout simply limits how
>> much
of the String is printed. Even Gary’s proposal of calling substring() is
still going to operate on the whole String. He would really need a
>> method
that accepts the max number of characters to read from the file.
>>
>> Ralph
>>
>>> On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı wrote:
>>>
>>> *[Just responding to Matt. I don't have an answer for Gary.]*
>>>
>>> `JsonTemplateLayout` has `maxStringLength`, and related with it,
>>> `truncatedStringSuffix`.
>>>
>>> On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker
>> wrote:
>>>
You can use the %maxLength{…}{N} pattern converter with
>> PatternLayout
at
least. Unfortunately, I don’t think any other layouts support a
similar
option.
> On Jan 25, 2024, at 10:55, Gary D. Gregory
wrote:
>
> Hi All,
>
> I'd like to ask how to if we can devise advice around an issue I
>> ran
into this week.
>
> One of our test suites processes about 40K files of test fixtures.
These
inputs are parsed, processed, and asserted. This randomly fails on a
call
to Logger#debug(), randomly in that it happens usually once per
>> build,
somewhere in a logging call. But it usually fails with a call that
looks
like this:
>
> logger.debug("This is fun" + myFunObject);
>
> To simplify things, let's say that it turns out that after an
underlying
third party jar file version upgrade the call to
myFunObject#toString() no
longer returns Object#toString() but rather (again to simplify) the
contents of the file that was parsed to create myFunObject. This
toString()
can be megabytes. The solution is obvious:
>
> logger.debug("This is fun", myFunObject::toString)
>
> And our CI builds no longer randomly fail since our default logging
does
not log at the debug level.
>
> A better solution could be:
>
> logger.debug("This is fun", () ->
>> myFunObject.toString().substring(0,
100))
>
> where I still want _some_ information better than making my own
toString() with System#identityHashCode(Object) or somesuch. Sure,
.toString() is still called but it does not make it down into
logging. In
my case the OOME happened in myFunObject::toString so the
>> substring()
example would not have worked.
>
> My question is: Should we document some general advice on this
pattern
and what should the advice be? Would it make sense to have some
features
where we truncate/rej
