Well, it seems I have been doing a lot of unnecessary work. In testing it was noticed that within the JSON no “real” newlines were appearing - only “\n” as a string. So I removed the event delimiter and let it default to a newline and everything is working as I would expect in Kibana. This didn’t happen when I was using other Layouts. It seems the JsonWriter class in Json Template Layout handles escaping the line control characters so that they are not a problem. When Kibana gets them as part of the “message” it interprets them correctly.
I didn’t see any mention of this in the JsonTemplateLayout docs. I am going to update the “Logging in the Cloud” page to reflect this. To be honest, I don’t really know why I didn’t catch this before. I re-read our email conversation about this and it says I tested newlines in the message. But my testing now shows they aren’t a problem since they are escaped. Ralph > On Sep 13, 2021, at 9:04 AM, Ralph Goers <ralph.go...@dslextreme.com> wrote: > > I just looked at the stack overflow post you referenced. Although the author > is explicitly > giving permission for everyone to use it everything at StackOverflow is > licensed as > CC-By 4.0. That is covered at > https://www.apache.org/legal/resolved.html#cc-by > <https://www.apache.org/legal/resolved.html#cc-by>. > > It would be best if you could reach out to the author and ask him to post the > code > on GitHub or directly give it to you and bypass StackOverflow. If he posts it > to GitHub > he should include a License.txt with whatever license he wants to use. > Hopefully > something like MIT or Apache. > > Ralph > >> On Sep 13, 2021, at 7:08 AM, Apache <ralph.go...@dslextreme.com> wrote: >> >> I don’t think anyone passing an escape sequence expects it to show up as a >> string. So I would be good with your proposed solution. That method should >> be part of core so we can modify all Layouts to use it. >> >> Ralph >> >>> On Sep 13, 2021, at 12:29 AM, Volkan Yazıcı <volkan.yaz...@gmail.com> wrote: >>> >>> I had addressed this issue in 2020 July in a post titled "String parameter >>> with escapes (Was: Testing Json Template Layout)" >>> <https://mail-archives.apache.org/mod_mbox/logging-dev/202007.mbox/%3cCAP7pH7uWjjn=ZDCeejmqMQHgbFiTV=23J9D3n=whfndb0is...@mail.gmail.com%3e>. >>> For one, *the issue is not specific to JTL*, but applies to almost every >>> layout accepting a parameter to be emitted along with the normal output. >>> Back then, the discussion focused on the null (\0) character needed by >>> SocketAppender for GELF layout in JTL. In the post, I suggested adding >>> something similar to `translateEscapes()` of Java 13. This would allow >>> users to pass escape characters with any string literal in any >>> configuration format; XML, YAML, properties, etc. Contrasting this with >>> your proposal for treating certain special inputs (e.g., "null", "Form >>> Feed") differently, for me, is not different than implementing a >>> `translateEscapes()`. I personally think this is even worse, since we >>> introduce a new syntax, whereas the former sticks to the standard Java >>> string syntax. >>> >>> In conclusion, I propose adding a `translateEscapes()` clone and using that >>> to read `eventDelimiter` and deprecating `nullEventDelimiterEnabled`, since >>> `nullEventDelimiterEnabled` will simply translate to setting >>> `eventDelimiter` to `\0`. This breaks the backward compatibility of >>> `eventDelimiter`, since there might be users out there who set their >>> `eventDelimiter` to `\t\r\n` and was expecting to get that string literal >>> as is. Yet, given the relatively recent introduction of the plugin, I think >>> we can take this risk. >>> >>>> On Mon, Sep 13, 2021 at 6:14 AM Ralph Goers <ralph.go...@dslextreme.com> >>>> wrote: >>>> >>>> I have been working on getting a configuration that works with filebeat >>>> going again and have been >>>> trying to change the line delimiter so I could avoid doing the multiline >>>> crap. I first configured >>>> eventDelimiter=“\f” since filebeat supports that as a replacement for >>>> newline. Except no matter >>>> how hard I tried I couldn’t get filebeat to log anything to stdout. >>>> >>>> I spent hours looking at it before I finally realized that the logs being >>>> written actually had “\f” in >>>> the string instead of a form feed character. It seems that XML must be >>>> passing in the value >>>> treating the “\” as if it is escaped. To remedy this the only solution I >>>> could come up with was >>>> >>>> <JsonTemplateLayout eventTemplateUri="classpath:template.json" >>>> locationInfoEnabled="true"> >>>> <eventDelimiter>…</eventDelimiter> >>>> </JsonTemplateLayout> >>>> It seems that XML doesn’t allow a form feed character specified that way. >>>> This caused filebeat >>>> to start recognizing the log events. I would suggest that we may want to >>>> change eventDelimiter >>>> to accept enum names instead of or in addition to characters, which is >>>> what filebeat does. So >>>> one could specify eventDelimiter=“Form Feed”. >>>> >>>> Ralph >> >> >> >
