Re: [Log4j] If and how to document potential OOME
Perhaps mention of StringBuilderFormattable? It helps work around the “load the
string all at once” problem.
> On Jan 26, 2024, at 06:36, Gary Gregory wrote:
>
> Oh, I don't want to suggest this is a Lo4j issue at all. I am wondering if
> there is a pattern we should document aside from "know your third party
> objects" which is not helpful for us to say. It sounds like there is
> nothing for us to do.
>
> Gary
>
> On Fri, Jan 26, 2024, 7:16 AM Apache wrote:
>
>> That is kind of my point. Anyone who logs an object that behaves this way
>> is asking for trouble. It really sounds like an esoteric edge case to me. I
>> don’t view this as a Log4J problem as the error wouldn’t even have Log4J in
>> the stack trace.
>>
>> Ralph
>>
>>> On Jan 25, 2024, at 9:52 PM, Gary Gregory
>> wrote:
>>>
>>> I have no control over this. I think this is a general problem with
>> third
>>> party objects and toString(). There is no toString(fromHereToThere) or
>>> toStringReader() so I can't see a general way to deal with it. Even if I
>>> created a wrapper for the Object and called toString(), truncated and
>>> cached the result and have the wrapper return the shorter string in its
>>> toString(), I still would have toString()'d the original...
>>>
>>> Gary
>>>
On Thu, Jan 25, 2024, 11:33 PM Ralph Goers
wrote:
OK. The only good way to handle that is to parse the YAML/JSON file
>> while
streaming it and extract just the fields you want to include in the
>> logs.
Ralph
> On Jan 25, 2024, at 6:40 PM, Gary Gregory
wrote:
>
> Well, it's worse than that because the object is an object created by
> parsing a YAML (or JSON) file, then the toString() of that object
> renders a String in some other format.
>
> Gary
>
> On Thu, Jan 25, 2024 at 7:45 PM Ralph Goers <
>> ralph.go...@dslextreme.com>
wrote:
>>
>> Volkan & Matt,
>>
>> Neither of those is going to help. The issue is that when the toString
method is called it reads a whole file in and stores it as a String.
>> This
could cause the OOM error. Truncating it in a layout simply limits how
>> much
of the String is printed. Even Gary’s proposal of calling substring() is
still going to operate on the whole String. He would really need a
>> method
that accepts the max number of characters to read from the file.
>>
>> Ralph
>>
>>> On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı wrote:
>>>
>>> *[Just responding to Matt. I don't have an answer for Gary.]*
>>>
>>> `JsonTemplateLayout` has `maxStringLength`, and related with it,
>>> `truncatedStringSuffix`.
>>>
>>> On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker
>> wrote:
>>>
You can use the %maxLength{…}{N} pattern converter with
>> PatternLayout
at
least. Unfortunately, I don’t think any other layouts support a
similar
option.
> On Jan 25, 2024, at 10:55, Gary D. Gregory
wrote:
>
> Hi All,
>
> I'd like to ask how to if we can devise advice around an issue I
>> ran
into this week.
>
> One of our test suites processes about 40K files of test fixtures.
These
inputs are parsed, processed, and asserted. This randomly fails on a
call
to Logger#debug(), randomly in that it happens usually once per
>> build,
somewhere in a logging call. But it usually fails with a call that
looks
like this:
>
> logger.debug("This is fun" + myFunObject);
>
> To simplify things, let's say that it turns out that after an
underlying
third party jar file version upgrade the call to
myFunObject#toString() no
longer returns Object#toString() but rather (again to simplify) the
contents of the file that was parsed to create myFunObject. This
toString()
can be megabytes. The solution is obvious:
>
> logger.debug("This is fun", myFunObject::toString)
>
> And our CI builds no longer randomly fail since our default logging
does
not log at the debug level.
>
> A better solution could be:
>
> logger.debug("This is fun", () ->
>> myFunObject.toString().substring(0,
100))
>
> where I still want _some_ information better than making my own
toString() with System#identityHashCode(Object) or somesuch. Sure,
.toString() is still called but it does not make it down into
logging. In
my case the OOME happened in myFunObject::toString so the
>> substring()
example would not have worked.
>
> My question is: Should we document some general advice on this
pattern
and what should the advice be? Would it make sense to have some
features
where we
Re: [Log4j] If and how to document potential OOME
Oh, I don't want to suggest this is a Lo4j issue at all. I am wondering if
there is a pattern we should document aside from "know your third party
objects" which is not helpful for us to say. It sounds like there is
nothing for us to do.
Gary
On Fri, Jan 26, 2024, 7:16 AM Apache wrote:
> That is kind of my point. Anyone who logs an object that behaves this way
> is asking for trouble. It really sounds like an esoteric edge case to me. I
> don’t view this as a Log4J problem as the error wouldn’t even have Log4J in
> the stack trace.
>
> Ralph
>
> > On Jan 25, 2024, at 9:52 PM, Gary Gregory
> wrote:
> >
> > I have no control over this. I think this is a general problem with
> third
> > party objects and toString(). There is no toString(fromHereToThere) or
> > toStringReader() so I can't see a general way to deal with it. Even if I
> > created a wrapper for the Object and called toString(), truncated and
> > cached the result and have the wrapper return the shorter string in its
> > toString(), I still would have toString()'d the original...
> >
> > Gary
> >
> >> On Thu, Jan 25, 2024, 11:33 PM Ralph Goers
> >> wrote:
> >>
> >> OK. The only good way to handle that is to parse the YAML/JSON file
> while
> >> streaming it and extract just the fields you want to include in the
> logs.
> >>
> >> Ralph
> >>
> >>> On Jan 25, 2024, at 6:40 PM, Gary Gregory
> >> wrote:
> >>>
> >>> Well, it's worse than that because the object is an object created by
> >>> parsing a YAML (or JSON) file, then the toString() of that object
> >>> renders a String in some other format.
> >>>
> >>> Gary
> >>>
> >>> On Thu, Jan 25, 2024 at 7:45 PM Ralph Goers <
> ralph.go...@dslextreme.com>
> >> wrote:
>
> Volkan & Matt,
>
> Neither of those is going to help. The issue is that when the toString
> >> method is called it reads a whole file in and stores it as a String.
> This
> >> could cause the OOM error. Truncating it in a layout simply limits how
> much
> >> of the String is printed. Even Gary’s proposal of calling substring() is
> >> still going to operate on the whole String. He would really need a
> method
> >> that accepts the max number of characters to read from the file.
>
> Ralph
>
> > On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı wrote:
> >
> > *[Just responding to Matt. I don't have an answer for Gary.]*
> >
> > `JsonTemplateLayout` has `maxStringLength`, and related with it,
> > `truncatedStringSuffix`.
> >
> > On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker
> wrote:
> >
> >> You can use the %maxLength{…}{N} pattern converter with
> PatternLayout
> >> at
> >> least. Unfortunately, I don’t think any other layouts support a
> >> similar
> >> option.
> >>
> >>> On Jan 25, 2024, at 10:55, Gary D. Gregory
> >> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I'd like to ask how to if we can devise advice around an issue I
> ran
> >> into this week.
> >>>
> >>> One of our test suites processes about 40K files of test fixtures.
> >> These
> >> inputs are parsed, processed, and asserted. This randomly fails on a
> >> call
> >> to Logger#debug(), randomly in that it happens usually once per
> build,
> >> somewhere in a logging call. But it usually fails with a call that
> >> looks
> >> like this:
> >>>
> >>> logger.debug("This is fun" + myFunObject);
> >>>
> >>> To simplify things, let's say that it turns out that after an
> >> underlying
> >> third party jar file version upgrade the call to
> >> myFunObject#toString() no
> >> longer returns Object#toString() but rather (again to simplify) the
> >> contents of the file that was parsed to create myFunObject. This
> >> toString()
> >> can be megabytes. The solution is obvious:
> >>>
> >>> logger.debug("This is fun", myFunObject::toString)
> >>>
> >>> And our CI builds no longer randomly fail since our default logging
> >> does
> >> not log at the debug level.
> >>>
> >>> A better solution could be:
> >>>
> >>> logger.debug("This is fun", () ->
> myFunObject.toString().substring(0,
> >> 100))
> >>>
> >>> where I still want _some_ information better than making my own
> >> toString() with System#identityHashCode(Object) or somesuch. Sure,
> >> .toString() is still called but it does not make it down into
> >> logging. In
> >> my case the OOME happened in myFunObject::toString so the
> substring()
> >> example would not have worked.
> >>>
> >>> My question is: Should we document some general advice on this
> >> pattern
> >> and what should the advice be? Would it make sense to have some
> >> features
> >> where we truncate/reject Strings above a threshold. And yes, calling
> >> myFunObject.toString() can still still get me in trouble.
> >>>
> >>> Gary
> >>>
> >>
> >>
>
> >>
> >>
>
>
Re: [Log4j] If and how to document potential OOME
That is kind of my point. Anyone who logs an object that behaves this way is
asking for trouble. It really sounds like an esoteric edge case to me. I don’t
view this as a Log4J problem as the error wouldn’t even have Log4J in the stack
trace.
Ralph
> On Jan 25, 2024, at 9:52 PM, Gary Gregory wrote:
>
> I have no control over this. I think this is a general problem with third
> party objects and toString(). There is no toString(fromHereToThere) or
> toStringReader() so I can't see a general way to deal with it. Even if I
> created a wrapper for the Object and called toString(), truncated and
> cached the result and have the wrapper return the shorter string in its
> toString(), I still would have toString()'d the original...
>
> Gary
>
>> On Thu, Jan 25, 2024, 11:33 PM Ralph Goers
>> wrote:
>>
>> OK. The only good way to handle that is to parse the YAML/JSON file while
>> streaming it and extract just the fields you want to include in the logs.
>>
>> Ralph
>>
>>> On Jan 25, 2024, at 6:40 PM, Gary Gregory
>> wrote:
>>>
>>> Well, it's worse than that because the object is an object created by
>>> parsing a YAML (or JSON) file, then the toString() of that object
>>> renders a String in some other format.
>>>
>>> Gary
>>>
>>> On Thu, Jan 25, 2024 at 7:45 PM Ralph Goers
>> wrote:
Volkan & Matt,
Neither of those is going to help. The issue is that when the toString
>> method is called it reads a whole file in and stores it as a String. This
>> could cause the OOM error. Truncating it in a layout simply limits how much
>> of the String is printed. Even Gary’s proposal of calling substring() is
>> still going to operate on the whole String. He would really need a method
>> that accepts the max number of characters to read from the file.
Ralph
> On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı wrote:
>
> *[Just responding to Matt. I don't have an answer for Gary.]*
>
> `JsonTemplateLayout` has `maxStringLength`, and related with it,
> `truncatedStringSuffix`.
>
> On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker wrote:
>
>> You can use the %maxLength{…}{N} pattern converter with PatternLayout
>> at
>> least. Unfortunately, I don’t think any other layouts support a
>> similar
>> option.
>>
>>> On Jan 25, 2024, at 10:55, Gary D. Gregory
>> wrote:
>>>
>>> Hi All,
>>>
>>> I'd like to ask how to if we can devise advice around an issue I ran
>> into this week.
>>>
>>> One of our test suites processes about 40K files of test fixtures.
>> These
>> inputs are parsed, processed, and asserted. This randomly fails on a
>> call
>> to Logger#debug(), randomly in that it happens usually once per build,
>> somewhere in a logging call. But it usually fails with a call that
>> looks
>> like this:
>>>
>>> logger.debug("This is fun" + myFunObject);
>>>
>>> To simplify things, let's say that it turns out that after an
>> underlying
>> third party jar file version upgrade the call to
>> myFunObject#toString() no
>> longer returns Object#toString() but rather (again to simplify) the
>> contents of the file that was parsed to create myFunObject. This
>> toString()
>> can be megabytes. The solution is obvious:
>>>
>>> logger.debug("This is fun", myFunObject::toString)
>>>
>>> And our CI builds no longer randomly fail since our default logging
>> does
>> not log at the debug level.
>>>
>>> A better solution could be:
>>>
>>> logger.debug("This is fun", () -> myFunObject.toString().substring(0,
>> 100))
>>>
>>> where I still want _some_ information better than making my own
>> toString() with System#identityHashCode(Object) or somesuch. Sure,
>> .toString() is still called but it does not make it down into
>> logging. In
>> my case the OOME happened in myFunObject::toString so the substring()
>> example would not have worked.
>>>
>>> My question is: Should we document some general advice on this
>> pattern
>> and what should the advice be? Would it make sense to have some
>> features
>> where we truncate/reject Strings above a threshold. And yes, calling
>> myFunObject.toString() can still still get me in trouble.
>>>
>>> Gary
>>>
>>
>>
>>
>>
Re: [Log4j] If and how to document potential OOME
I have no control over this. I think this is a general problem with third
party objects and toString(). There is no toString(fromHereToThere) or
toStringReader() so I can't see a general way to deal with it. Even if I
created a wrapper for the Object and called toString(), truncated and
cached the result and have the wrapper return the shorter string in its
toString(), I still would have toString()'d the original...
Gary
On Thu, Jan 25, 2024, 11:33 PM Ralph Goers
wrote:
> OK. The only good way to handle that is to parse the YAML/JSON file while
> streaming it and extract just the fields you want to include in the logs.
>
> Ralph
>
> > On Jan 25, 2024, at 6:40 PM, Gary Gregory
> wrote:
> >
> > Well, it's worse than that because the object is an object created by
> > parsing a YAML (or JSON) file, then the toString() of that object
> > renders a String in some other format.
> >
> > Gary
> >
> > On Thu, Jan 25, 2024 at 7:45 PM Ralph Goers
> wrote:
> >>
> >> Volkan & Matt,
> >>
> >> Neither of those is going to help. The issue is that when the toString
> method is called it reads a whole file in and stores it as a String. This
> could cause the OOM error. Truncating it in a layout simply limits how much
> of the String is printed. Even Gary’s proposal of calling substring() is
> still going to operate on the whole String. He would really need a method
> that accepts the max number of characters to read from the file.
> >>
> >> Ralph
> >>
> >>> On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı wrote:
> >>>
> >>> *[Just responding to Matt. I don't have an answer for Gary.]*
> >>>
> >>> `JsonTemplateLayout` has `maxStringLength`, and related with it,
> >>> `truncatedStringSuffix`.
> >>>
> >>> On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker wrote:
> >>>
> You can use the %maxLength{…}{N} pattern converter with PatternLayout
> at
> least. Unfortunately, I don’t think any other layouts support a
> similar
> option.
>
> > On Jan 25, 2024, at 10:55, Gary D. Gregory
> wrote:
> >
> > Hi All,
> >
> > I'd like to ask how to if we can devise advice around an issue I ran
> into this week.
> >
> > One of our test suites processes about 40K files of test fixtures.
> These
> inputs are parsed, processed, and asserted. This randomly fails on a
> call
> to Logger#debug(), randomly in that it happens usually once per build,
> somewhere in a logging call. But it usually fails with a call that
> looks
> like this:
> >
> > logger.debug("This is fun" + myFunObject);
> >
> > To simplify things, let's say that it turns out that after an
> underlying
> third party jar file version upgrade the call to
> myFunObject#toString() no
> longer returns Object#toString() but rather (again to simplify) the
> contents of the file that was parsed to create myFunObject. This
> toString()
> can be megabytes. The solution is obvious:
> >
> > logger.debug("This is fun", myFunObject::toString)
> >
> > And our CI builds no longer randomly fail since our default logging
> does
> not log at the debug level.
> >
> > A better solution could be:
> >
> > logger.debug("This is fun", () -> myFunObject.toString().substring(0,
> 100))
> >
> > where I still want _some_ information better than making my own
> toString() with System#identityHashCode(Object) or somesuch. Sure,
> .toString() is still called but it does not make it down into
> logging. In
> my case the OOME happened in myFunObject::toString so the substring()
> example would not have worked.
> >
> > My question is: Should we document some general advice on this
> pattern
> and what should the advice be? Would it make sense to have some
> features
> where we truncate/reject Strings above a threshold. And yes, calling
> myFunObject.toString() can still still get me in trouble.
> >
> > Gary
> >
>
>
> >>
>
>
Re: [Log4j] If and how to document potential OOME
OK. The only good way to handle that is to parse the YAML/JSON file while
streaming it and extract just the fields you want to include in the logs.
Ralph
> On Jan 25, 2024, at 6:40 PM, Gary Gregory wrote:
>
> Well, it's worse than that because the object is an object created by
> parsing a YAML (or JSON) file, then the toString() of that object
> renders a String in some other format.
>
> Gary
>
> On Thu, Jan 25, 2024 at 7:45 PM Ralph Goers
> wrote:
>>
>> Volkan & Matt,
>>
>> Neither of those is going to help. The issue is that when the toString
>> method is called it reads a whole file in and stores it as a String. This
>> could cause the OOM error. Truncating it in a layout simply limits how much
>> of the String is printed. Even Gary’s proposal of calling substring() is
>> still going to operate on the whole String. He would really need a method
>> that accepts the max number of characters to read from the file.
>>
>> Ralph
>>
>>> On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı wrote:
>>>
>>> *[Just responding to Matt. I don't have an answer for Gary.]*
>>>
>>> `JsonTemplateLayout` has `maxStringLength`, and related with it,
>>> `truncatedStringSuffix`.
>>>
>>> On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker wrote:
>>>
You can use the %maxLength{…}{N} pattern converter with PatternLayout at
least. Unfortunately, I don’t think any other layouts support a similar
option.
> On Jan 25, 2024, at 10:55, Gary D. Gregory wrote:
>
> Hi All,
>
> I'd like to ask how to if we can devise advice around an issue I ran
into this week.
>
> One of our test suites processes about 40K files of test fixtures. These
inputs are parsed, processed, and asserted. This randomly fails on a call
to Logger#debug(), randomly in that it happens usually once per build,
somewhere in a logging call. But it usually fails with a call that looks
like this:
>
> logger.debug("This is fun" + myFunObject);
>
> To simplify things, let's say that it turns out that after an underlying
third party jar file version upgrade the call to myFunObject#toString() no
longer returns Object#toString() but rather (again to simplify) the
contents of the file that was parsed to create myFunObject. This toString()
can be megabytes. The solution is obvious:
>
> logger.debug("This is fun", myFunObject::toString)
>
> And our CI builds no longer randomly fail since our default logging does
not log at the debug level.
>
> A better solution could be:
>
> logger.debug("This is fun", () -> myFunObject.toString().substring(0,
100))
>
> where I still want _some_ information better than making my own
toString() with System#identityHashCode(Object) or somesuch. Sure,
.toString() is still called but it does not make it down into logging. In
my case the OOME happened in myFunObject::toString so the substring()
example would not have worked.
>
> My question is: Should we document some general advice on this pattern
and what should the advice be? Would it make sense to have some features
where we truncate/reject Strings above a threshold. And yes, calling
myFunObject.toString() can still still get me in trouble.
>
> Gary
>
>>
Re: [Log4j] If and how to document potential OOME
Well, it's worse than that because the object is an object created by
parsing a YAML (or JSON) file, then the toString() of that object
renders a String in some other format.
Gary
On Thu, Jan 25, 2024 at 7:45 PM Ralph Goers wrote:
>
> Volkan & Matt,
>
> Neither of those is going to help. The issue is that when the toString method
> is called it reads a whole file in and stores it as a String. This could
> cause the OOM error. Truncating it in a layout simply limits how much of the
> String is printed. Even Gary’s proposal of calling substring() is still going
> to operate on the whole String. He would really need a method that accepts
> the max number of characters to read from the file.
>
> Ralph
>
> > On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı wrote:
> >
> > *[Just responding to Matt. I don't have an answer for Gary.]*
> >
> > `JsonTemplateLayout` has `maxStringLength`, and related with it,
> > `truncatedStringSuffix`.
> >
> > On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker wrote:
> >
> >> You can use the %maxLength{…}{N} pattern converter with PatternLayout at
> >> least. Unfortunately, I don’t think any other layouts support a similar
> >> option.
> >>
> >>> On Jan 25, 2024, at 10:55, Gary D. Gregory wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I'd like to ask how to if we can devise advice around an issue I ran
> >> into this week.
> >>>
> >>> One of our test suites processes about 40K files of test fixtures. These
> >> inputs are parsed, processed, and asserted. This randomly fails on a call
> >> to Logger#debug(), randomly in that it happens usually once per build,
> >> somewhere in a logging call. But it usually fails with a call that looks
> >> like this:
> >>>
> >>> logger.debug("This is fun" + myFunObject);
> >>>
> >>> To simplify things, let's say that it turns out that after an underlying
> >> third party jar file version upgrade the call to myFunObject#toString() no
> >> longer returns Object#toString() but rather (again to simplify) the
> >> contents of the file that was parsed to create myFunObject. This toString()
> >> can be megabytes. The solution is obvious:
> >>>
> >>> logger.debug("This is fun", myFunObject::toString)
> >>>
> >>> And our CI builds no longer randomly fail since our default logging does
> >> not log at the debug level.
> >>>
> >>> A better solution could be:
> >>>
> >>> logger.debug("This is fun", () -> myFunObject.toString().substring(0,
> >> 100))
> >>>
> >>> where I still want _some_ information better than making my own
> >> toString() with System#identityHashCode(Object) or somesuch. Sure,
> >> .toString() is still called but it does not make it down into logging. In
> >> my case the OOME happened in myFunObject::toString so the substring()
> >> example would not have worked.
> >>>
> >>> My question is: Should we document some general advice on this pattern
> >> and what should the advice be? Would it make sense to have some features
> >> where we truncate/reject Strings above a threshold. And yes, calling
> >> myFunObject.toString() can still still get me in trouble.
> >>>
> >>> Gary
> >>>
> >>
> >>
>
Re: [Log4j] If and how to document potential OOME
Volkan & Matt,
Neither of those is going to help. The issue is that when the toString method
is called it reads a whole file in and stores it as a String. This could cause
the OOM error. Truncating it in a layout simply limits how much of the String
is printed. Even Gary’s proposal of calling substring() is still going to
operate on the whole String. He would really need a method that accepts the max
number of characters to read from the file.
Ralph
> On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı wrote:
>
> *[Just responding to Matt. I don't have an answer for Gary.]*
>
> `JsonTemplateLayout` has `maxStringLength`, and related with it,
> `truncatedStringSuffix`.
>
> On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker wrote:
>
>> You can use the %maxLength{…}{N} pattern converter with PatternLayout at
>> least. Unfortunately, I don’t think any other layouts support a similar
>> option.
>>
>>> On Jan 25, 2024, at 10:55, Gary D. Gregory wrote:
>>>
>>> Hi All,
>>>
>>> I'd like to ask how to if we can devise advice around an issue I ran
>> into this week.
>>>
>>> One of our test suites processes about 40K files of test fixtures. These
>> inputs are parsed, processed, and asserted. This randomly fails on a call
>> to Logger#debug(), randomly in that it happens usually once per build,
>> somewhere in a logging call. But it usually fails with a call that looks
>> like this:
>>>
>>> logger.debug("This is fun" + myFunObject);
>>>
>>> To simplify things, let's say that it turns out that after an underlying
>> third party jar file version upgrade the call to myFunObject#toString() no
>> longer returns Object#toString() but rather (again to simplify) the
>> contents of the file that was parsed to create myFunObject. This toString()
>> can be megabytes. The solution is obvious:
>>>
>>> logger.debug("This is fun", myFunObject::toString)
>>>
>>> And our CI builds no longer randomly fail since our default logging does
>> not log at the debug level.
>>>
>>> A better solution could be:
>>>
>>> logger.debug("This is fun", () -> myFunObject.toString().substring(0,
>> 100))
>>>
>>> where I still want _some_ information better than making my own
>> toString() with System#identityHashCode(Object) or somesuch. Sure,
>> .toString() is still called but it does not make it down into logging. In
>> my case the OOME happened in myFunObject::toString so the substring()
>> example would not have worked.
>>>
>>> My question is: Should we document some general advice on this pattern
>> and what should the advice be? Would it make sense to have some features
>> where we truncate/reject Strings above a threshold. And yes, calling
>> myFunObject.toString() can still still get me in trouble.
>>>
>>> Gary
>>>
>>
>>
Re: [Log4j] If and how to document potential OOME
*[Just responding to Matt. I don't have an answer for Gary.]*
`JsonTemplateLayout` has `maxStringLength`, and related with it,
`truncatedStringSuffix`.
On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker wrote:
> You can use the %maxLength{…}{N} pattern converter with PatternLayout at
> least. Unfortunately, I don’t think any other layouts support a similar
> option.
>
> > On Jan 25, 2024, at 10:55, Gary D. Gregory wrote:
> >
> > Hi All,
> >
> > I'd like to ask how to if we can devise advice around an issue I ran
> into this week.
> >
> > One of our test suites processes about 40K files of test fixtures. These
> inputs are parsed, processed, and asserted. This randomly fails on a call
> to Logger#debug(), randomly in that it happens usually once per build,
> somewhere in a logging call. But it usually fails with a call that looks
> like this:
> >
> > logger.debug("This is fun" + myFunObject);
> >
> > To simplify things, let's say that it turns out that after an underlying
> third party jar file version upgrade the call to myFunObject#toString() no
> longer returns Object#toString() but rather (again to simplify) the
> contents of the file that was parsed to create myFunObject. This toString()
> can be megabytes. The solution is obvious:
> >
> > logger.debug("This is fun", myFunObject::toString)
> >
> > And our CI builds no longer randomly fail since our default logging does
> not log at the debug level.
> >
> > A better solution could be:
> >
> > logger.debug("This is fun", () -> myFunObject.toString().substring(0,
> 100))
> >
> > where I still want _some_ information better than making my own
> toString() with System#identityHashCode(Object) or somesuch. Sure,
> .toString() is still called but it does not make it down into logging. In
> my case the OOME happened in myFunObject::toString so the substring()
> example would not have worked.
> >
> > My question is: Should we document some general advice on this pattern
> and what should the advice be? Would it make sense to have some features
> where we truncate/reject Strings above a threshold. And yes, calling
> myFunObject.toString() can still still get me in trouble.
> >
> > Gary
> >
>
>
Re: [Log4j] If and how to document potential OOME
You can use the %maxLength{…}{N} pattern converter with PatternLayout at least.
Unfortunately, I don’t think any other layouts support a similar option.
> On Jan 25, 2024, at 10:55, Gary D. Gregory wrote:
>
> Hi All,
>
> I'd like to ask how to if we can devise advice around an issue I ran into
> this week.
>
> One of our test suites processes about 40K files of test fixtures. These
> inputs are parsed, processed, and asserted. This randomly fails on a call to
> Logger#debug(), randomly in that it happens usually once per build, somewhere
> in a logging call. But it usually fails with a call that looks like this:
>
> logger.debug("This is fun" + myFunObject);
>
> To simplify things, let's say that it turns out that after an underlying
> third party jar file version upgrade the call to myFunObject#toString() no
> longer returns Object#toString() but rather (again to simplify) the contents
> of the file that was parsed to create myFunObject. This toString() can be
> megabytes. The solution is obvious:
>
> logger.debug("This is fun", myFunObject::toString)
>
> And our CI builds no longer randomly fail since our default logging does not
> log at the debug level.
>
> A better solution could be:
>
> logger.debug("This is fun", () -> myFunObject.toString().substring(0, 100))
>
> where I still want _some_ information better than making my own toString()
> with System#identityHashCode(Object) or somesuch. Sure, .toString() is still
> called but it does not make it down into logging. In my case the OOME
> happened in myFunObject::toString so the substring() example would not have
> worked.
>
> My question is: Should we document some general advice on this pattern and
> what should the advice be? Would it make sense to have some features where we
> truncate/reject Strings above a threshold. And yes, calling
> myFunObject.toString() can still still get me in trouble.
>
> Gary
>
Re: [Log4j] If and how to document potential OOME
Obvious mistake:
logger.debug("This is fun", myFunObject::toString)
->
logger.debug("This is fun {}", myFunObject::toString)
Gary
On Thu, Jan 25, 2024, 11:55 AM Gary D. Gregory wrote:
> Hi All,
>
> I'd like to ask how to if we can devise advice around an issue I ran into
> this week.
>
> One of our test suites processes about 40K files of test fixtures. These
> inputs are parsed, processed, and asserted. This randomly fails on a call
> to Logger#debug(), randomly in that it happens usually once per build,
> somewhere in a logging call. But it usually fails with a call that looks
> like this:
>
> logger.debug("This is fun" + myFunObject);
>
> To simplify things, let's say that it turns out that after an underlying
> third party jar file version upgrade the call to myFunObject#toString() no
> longer returns Object#toString() but rather (again to simplify) the
> contents of the file that was parsed to create myFunObject. This toString()
> can be megabytes. The solution is obvious:
>
> logger.debug("This is fun", myFunObject::toString)
>
> And our CI builds no longer randomly fail since our default logging does
> not log at the debug level.
>
> A better solution could be:
>
> logger.debug("This is fun", () -> myFunObject.toString().substring(0, 100))
>
> where I still want _some_ information better than making my own toString()
> with System#identityHashCode(Object) or somesuch. Sure, .toString() is
> still called but it does not make it down into logging. In my case the OOME
> happened in myFunObject::toString so the substring() example would not have
> worked.
>
> My question is: Should we document some general advice on this pattern and
> what should the advice be? Would it make sense to have some features where
> we truncate/reject Strings above a threshold. And yes, calling
> myFunObject.toString() can still still get me in trouble.
>
> Gary
>
>
