I’ve updated the Jira with a second option of enhancing the PatternLayout which I like better.
I will say that I find what DataDog is doing here to be strange. I don’t understand why Datadog doesn’t support passing the API key as a credential on the socket connection instead of validating every log event. That seems really inefficient. Ralph > On Nov 27, 2021, at 11:12 AM, Ralph Goers <[email protected]> wrote: > > https://github.com/logfellow/logstash-logback-encoder#prefixsuffixseparator > explains that the prefix and suffix are before the JSON starts. This would be > incredibly easy to implement as: > > <Appenders> > <Socket name=“JSON_TCP" host=“intake.logs.datadoghq.com" port=“10516" > protocol=“SSL" bufferedIo="true" ignoreExceptions="false”> > <PrefixSuffixLayout prefix=“${env:DD_API_KEY}”> > <JsonTemplateLayout includeLocation=“${includeLocation:-true}” > eventTemplateUri=“classpath:mytemplate.json”/> > </PrefixSuffixLayout> > </Socket> > </Appenders> > > PrefixSuffixLayout would simply emit the prefix before calling the embedded > layout and then emit the suffix afterwards. > > Ralph > > > >> On Nov 27, 2021, at 9:33 AM, Mikael Ståldal <[email protected]> wrote: >> >> JIRA issue created: https://issues.apache.org/jira/browse/LOG4J2-3197 >> >> I think that the API key prefix is outside of the JSON, so the end result is >> not proper JSON. I suppose that the Datadog service will pre-parse the >> received event to extract the API key, and then parse the rest as JSON. Like >> this: >> >> MY_API_KEY_HERE {"some":"json","object":"here"} >> >> /Mikael >> >> >> On 2021-11-27 11:35, Volkan Yazıcı wrote: >>> Thanks for the heads up Mikael! I am pretty sure JSON Template Layout can >>> suffice this goal – if it can't, I will make sure it can. From the Datadog >>> documentation you have shared >>> <https://docs.datadoghq.com/logs/log_collection/java>, the expected JSON >>> structure is not clear. They have used logstash-logback-encoder's prefix >>> feature >>> <https://github.com/logfellow/logstash-logback-encoder#prefixsuffixseparator>, >>> yet I couldn't understand how does it "prefix" a JSON object. For instance, >>> how can you prefix a JSON array with a string!? Nevertheless, I guess I >>> need to run this myself locally and figure that detail out. I will really >>> appreciate it if you can create a JIRA ticket and assign it to my name. >>> After making sure it works, I will get in touch with the Datadog team too. >>> On Sat, Nov 27, 2021 at 11:10 AM Mikael Ståldal <[email protected]> wrote: >>>> The documentation for Datadog contains information on how to setup Log4j >>>> 2 to send logs to Datadog. However, for the agentless configuration, it >>>> says its not possible with Log4j 2 and resorts to bridging to Logback. >>>> >>>> >>>> https://docs.datadoghq.com/logs/log_collection/java/?tab=log4j2#agentless-logging >>>> >>>> The problem is that the JSON formatted log events needs to be prefixed >>>> with an API key (outside of the JSON structure). This is possible with >>>> Logback, but supposedly not with Log4j 2. >>>> >>>> This configuration is arguably a bit strange, but it would be good if >>>> Log4j 2 could support it, since it is likely a relevant use case. >>>> >>>> Maybe this is already possible with recent versions of Log4j 2? If not, >>>> I believe it should be easy to add. >>>> >>>> >>>> For the standard configuration, Datadog recommends the JSONLayout: >>>> >>>> >>>> https://docs.datadoghq.com/logs/log_collection/java/?tab=log4j2#configure-your-logger >>>> >>>> Maybe the new JsonPatternLayout would be more suitable? >>>> >>>> /Mikael >>>> >> > > >
