Re: Lilith and Log4j 2
That test was added as part of LOG4J2-1961 by Gary. On 2017-08-03 16:00, Ralph Goers wrote: The error with the missing artifact happens when you skip tests. The test that failed for you never fails for me on OS X but fails randomly in Jenkins. I’m not sure why no one has looked at it yet. Ralph On Aug 3, 2017, at 6:52 AM, Jörn Huxhorn wrote: OK, everything works for me with 2.9-SNAPSHOT. I can receive JSON, YAML and XML events just fine. I had the following error while building: [ERROR] Failed to execute goal on project log4j-core-its: Could not resolve dependencies for project org.apache.logging.log4j:log4j-core-its:jar:2.9-SNAPSHOT: Could not find artifact org.apache.logging.log4j:log4j-server:jar:tests:2.9-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots), try downloading from https://logging.apache.org/log4j/2.x/download.html -> [Help 1] That wasn’t a serious problem for me since I only need API and core. But I wanted to inform you about this issue since building log4j worked fine for me just a few days ago. I’m sure it worked at d3b722083b8f21ea105a4f01132f08b373cb401e and I think it also worked at b96f5220dbebf6e0a4fb9f7da07c3a9d7d443418 Cheers, Jörn. On 3. August 2017 at 13:30:24, Mikael Ståldal (mi...@apache.org) wrote: Yes, that test is known to be unstablle. Just build with -DskipTests On 2017-08-03 11:24, Jörn Huxhorn wrote: I get test failures building against HEAD (3ac727026f295941fe71a34d6c80afc242f38c0f). Environment: $ mvn -version Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T21:39:06+02:00) Maven home: /Users/huxi/.sdkman/candidates/maven/current Java version: 1.8.0_144, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: “mac" Test failure: --- Test set: org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest --- Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.518 s <<< FAILURE! - in org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest testRollingFileAppenderWithReconfigure(org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest) Time elapsed: 2.518 s <<< FAILURE! java.lang.AssertionError: Expected: is <2> but: was <3> at org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest.testRollingFileAppenderWithReconfigure(RollingAppenderDirectWriteWithReconfigureTest.java:68) On 2. August 2017 at 11:30:47, Mikael Ståldal (mi...@apache.org) wrote: In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout. On 2017-08-02 09:23, Jörn Huxhorn wrote: I started working on the Lilith side of this and I think I’m pretty much done. To test the new functionality in a sandbox application, I’d now need the ability to switch JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with a 0-byte instead of being separated by “, ". On 26. July 2017 at 21:14:41, Mikael Ståldal (mi...@apache.org) wrote: That would make sense. On 2017-07-26 19:51, Jörn Huxhorn wrote: Isn’t this supposed to also use 0-byte terminated events in the future? Because in that case it would make sense to change from String to byte[] processing. On 26. July 2017 at 17:49:07, Mikael Ståldal (mi...@apache.org) wrote: They just need the String version: https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100 On 2017-07-26 17:40, Gary Gregory wrote: What do our XML and JSON server need? Let's make sure we handle those use cases. Gary On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: +1 On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) wrote: Maybe I should remove both the Reader and the InputStream versions, since they cannot parse multiple log events from a stream, and are thus not very useful. On 2017-07-26 10:34, Jörn Huxhorn wrote: It would maybe be a good idea to get rid of the Reader method altogether since it mainly introduces an unnecessary point of failure if the Reader is using an encoding other than UTF-8. Using a FileReader would be an example for this since its limited c'tors are always using Charset.defaultCharset().name(). The correct way to create an UTF-8 file reader is the less than obvious "new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common mistake.
Re: Lilith and Log4j 2
The error with the missing artifact happens when you skip tests. The test that failed for you never fails for me on OS X but fails randomly in Jenkins. I’m not sure why no one has looked at it yet. Ralph > On Aug 3, 2017, at 6:52 AM, Jörn Huxhorn wrote: > > OK, everything works for me with 2.9-SNAPSHOT. > I can receive JSON, YAML and XML events just fine. > > I had the following error while building: > [ERROR] Failed to execute goal on project log4j-core-its: Could not resolve > dependencies for project > org.apache.logging.log4j:log4j-core-its:jar:2.9-SNAPSHOT: Could not find > artifact org.apache.logging.log4j:log4j-server:jar:tests:2.9-SNAPSHOT in > apache.snapshots (https://repository.apache.org/snapshots), try downloading > from https://logging.apache.org/log4j/2.x/download.html -> [Help 1] > > That wasn’t a serious problem for me since I only need API and core. > But I wanted to inform you about this issue since building log4j worked fine > for me just a few days ago. > > I’m sure it worked at d3b722083b8f21ea105a4f01132f08b373cb401e and I think it > also worked at b96f5220dbebf6e0a4fb9f7da07c3a9d7d443418 > > Cheers, > Jörn. > > > On 3. August 2017 at 13:30:24, Mikael Ståldal (mi...@apache.org) wrote: >> Yes, that test is known to be unstablle. >> >> Just build with -DskipTests >> >> >> On 2017-08-03 11:24, Jörn Huxhorn wrote: >>> I get test failures building against HEAD >>> (3ac727026f295941fe71a34d6c80afc242f38c0f). >>> >>> Environment: >>> $ mvn -version >>> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; >>> 2017-04-03T21:39:06+02:00) >>> Maven home: /Users/huxi/.sdkman/candidates/maven/current >>> Java version: 1.8.0_144, vendor: Oracle Corporation >>> Java home: >>> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre >>> Default locale: en_US, platform encoding: UTF-8 >>> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: “mac" >>> >>> Test failure: >>> --- >>> >>> Test set: >>> org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest >>> >>> --- >>> >>> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.518 s <<< >>> FAILURE! >> - in >> org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest >> >>> testRollingFileAppenderWithReconfigure(org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest) >>> >> Time elapsed: 2.518 s <<< FAILURE! >>> java.lang.AssertionError: >>> >>> Expected: is <2> >>> but: was <3> >>> at >>> org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest.testRollingFileAppenderWithReconfigure(RollingAppenderDirectWriteWithReconfigureTest.java:68) >>> >>> >>> >>> >>> >>> On 2. August 2017 at 11:30:47, Mikael Ståldal (mi...@apache.org) wrote: In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout. On 2017-08-02 09:23, Jörn Huxhorn wrote: > I started working on the Lilith side of this and I think I’m pretty much > done. > > To test the new functionality in a sandbox application, I’d now need the > ability to >> switch JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with >> a 0-byte instead of being separated by “, ". > > > On 26. July 2017 at 21:14:41, Mikael Ståldal (mi...@apache.org) wrote: >> That would make sense. >> >> On 2017-07-26 19:51, Jörn Huxhorn wrote: >>> Isn’t this supposed to also use 0-byte terminated events in the future? >>> Because in that case it would make sense to change from String to >>> byte[] processing. >>> >>> On 26. July 2017 at 17:49:07, Mikael Ståldal (mi...@apache.org) wrote: They just need the String version: https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100 On 2017-07-26 17:40, Gary Gregory wrote: > What do our XML and JSON server need? Let's make sure we handle those > use > cases. > > Gary > > On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: > >> +1 >> >> >> On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) >> wrote: >>> Maybe I should remove both the Reader and the InputStream versions, >>> since they cannot parse multiple log events from a stream, and are >>> thus >>> not very useful. >>> >>> On 2017-07-26 10:34, Jörn Huxhorn wrote: It would maybe be a good idea to get rid of the Reader method >> altogether since i
Re: Lilith and Log4j 2
OK, everything works for me with 2.9-SNAPSHOT. I can receive JSON, YAML and XML events just fine. I had the following error while building: [ERROR] Failed to execute goal on project log4j-core-its: Could not resolve dependencies for project org.apache.logging.log4j:log4j-core-its:jar:2.9-SNAPSHOT: Could not find artifact org.apache.logging.log4j:log4j-server:jar:tests:2.9-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots), try downloading from https://logging.apache.org/log4j/2.x/download.html -> [Help 1] That wasn’t a serious problem for me since I only need API and core. But I wanted to inform you about this issue since building log4j worked fine for me just a few days ago. I’m sure it worked at d3b722083b8f21ea105a4f01132f08b373cb401e and I think it also worked at b96f5220dbebf6e0a4fb9f7da07c3a9d7d443418 Cheers, Jörn. On 3. August 2017 at 13:30:24, Mikael Ståldal (mi...@apache.org) wrote: > Yes, that test is known to be unstablle. > > Just build with -DskipTests > > > On 2017-08-03 11:24, Jörn Huxhorn wrote: > > I get test failures building against HEAD > > (3ac727026f295941fe71a34d6c80afc242f38c0f). > > > > Environment: > > $ mvn -version > > Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; > > 2017-04-03T21:39:06+02:00) > > Maven home: /Users/huxi/.sdkman/candidates/maven/current > > Java version: 1.8.0_144, vendor: Oracle Corporation > > Java home: > > /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre > > Default locale: en_US, platform encoding: UTF-8 > > OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: “mac" > > > > Test failure: > > --- > > > > Test set: > > org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest > > > > --- > > > > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.518 s <<< > > FAILURE! > - in > org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest > > > testRollingFileAppenderWithReconfigure(org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest) > > > Time elapsed: 2.518 s <<< FAILURE! > > java.lang.AssertionError: > > > > Expected: is <2> > > but: was <3> > > at > > org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest.testRollingFileAppenderWithReconfigure(RollingAppenderDirectWriteWithReconfigureTest.java:68) > > > > > > > > > > > > On 2. August 2017 at 11:30:47, Mikael Ståldal (mi...@apache.org) wrote: > >> In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout. > >> > >> > >> On 2017-08-02 09:23, Jörn Huxhorn wrote: > >>> I started working on the Lilith side of this and I think I’m pretty much > >>> done. > >>> > >>> To test the new functionality in a sandbox application, I’d now need the > >>> ability to > switch > >> JsonLayout, YamlLayout and XmlLayout into a mode were events are > >> terminated with > a 0-byte > >> instead of being separated by “, ". > >>> > >>> > >>> On 26. July 2017 at 21:14:41, Mikael Ståldal (mi...@apache.org) wrote: > That would make sense. > > On 2017-07-26 19:51, Jörn Huxhorn wrote: > > Isn’t this supposed to also use 0-byte terminated events in the future? > > Because in that case it would make sense to change from String to > > byte[] processing. > > > > On 26. July 2017 at 17:49:07, Mikael Ståldal (mi...@apache.org) wrote: > >> They just need the String version: > >> > >> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100 > >> > >> > >> > >> On 2017-07-26 17:40, Gary Gregory wrote: > >>> What do our XML and JSON server need? Let's make sure we handle those > >>> use > >>> cases. > >>> > >>> Gary > >>> > >>> On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: > >>> > +1 > > > On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) > wrote: > > Maybe I should remove both the Reader and the InputStream versions, > > since they cannot parse multiple log events from a stream, and are > > thus > > not very useful. > > > > On 2017-07-26 10:34, Jörn Huxhorn wrote: > >> It would maybe be a good idea to get rid of the Reader method > altogether since it mainly > > introduces an unnecessary point of failure if the Reader is using an > encoding other than > > UTF-8. > >> > >> Using a FileReader would be an example for this since its limited > c'tors are always using > > Charset.defaultCharset().name(). > >> > >> The
Re: Lilith and Log4j 2
Yes, that test is known to be unstablle. Just build with -DskipTests On 2017-08-03 11:24, Jörn Huxhorn wrote: I get test failures building against HEAD (3ac727026f295941fe71a34d6c80afc242f38c0f). Environment: $ mvn -version Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T21:39:06+02:00) Maven home: /Users/huxi/.sdkman/candidates/maven/current Java version: 1.8.0_144, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: “mac" Test failure: --- Test set: org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest --- Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.518 s <<< FAILURE! - in org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest testRollingFileAppenderWithReconfigure(org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest) Time elapsed: 2.518 s <<< FAILURE! java.lang.AssertionError: Expected: is <2> but: was <3> at org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest.testRollingFileAppenderWithReconfigure(RollingAppenderDirectWriteWithReconfigureTest.java:68) On 2. August 2017 at 11:30:47, Mikael Ståldal (mi...@apache.org) wrote: In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout. On 2017-08-02 09:23, Jörn Huxhorn wrote: I started working on the Lilith side of this and I think I’m pretty much done. To test the new functionality in a sandbox application, I’d now need the ability to switch JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with a 0-byte instead of being separated by “, ". On 26. July 2017 at 21:14:41, Mikael Ståldal (mi...@apache.org) wrote: That would make sense. On 2017-07-26 19:51, Jörn Huxhorn wrote: Isn’t this supposed to also use 0-byte terminated events in the future? Because in that case it would make sense to change from String to byte[] processing. On 26. July 2017 at 17:49:07, Mikael Ståldal (mi...@apache.org) wrote: They just need the String version: https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100 On 2017-07-26 17:40, Gary Gregory wrote: What do our XML and JSON server need? Let's make sure we handle those use cases. Gary On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: +1 On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) wrote: Maybe I should remove both the Reader and the InputStream versions, since they cannot parse multiple log events from a stream, and are thus not very useful. On 2017-07-26 10:34, Jörn Huxhorn wrote: It would maybe be a good idea to get rid of the Reader method altogether since it mainly introduces an unnecessary point of failure if the Reader is using an encoding other than UTF-8. Using a FileReader would be an example for this since its limited c'tors are always using Charset.defaultCharset().name(). The correct way to create an UTF-8 file reader is the less than obvious "new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common mistake.
Re: Lilith and Log4j 2
I get test failures building against HEAD (3ac727026f295941fe71a34d6c80afc242f38c0f). Environment: $ mvn -version Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T21:39:06+02:00) Maven home: /Users/huxi/.sdkman/candidates/maven/current Java version: 1.8.0_144, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: “mac" Test failure: --- Test set: org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest --- Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.518 s <<< FAILURE! - in org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest testRollingFileAppenderWithReconfigure(org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest) Time elapsed: 2.518 s <<< FAILURE! java.lang.AssertionError: Expected: is <2> but: was <3> at org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest.testRollingFileAppenderWithReconfigure(RollingAppenderDirectWriteWithReconfigureTest.java:68) On 2. August 2017 at 11:30:47, Mikael Ståldal (mi...@apache.org) wrote: > In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout. > > > On 2017-08-02 09:23, Jörn Huxhorn wrote: > > I started working on the Lilith side of this and I think I’m pretty much > > done. > > > > To test the new functionality in a sandbox application, I’d now need the > > ability to switch > JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated > with a 0-byte > instead of being separated by “, ". > > > > > > On 26. July 2017 at 21:14:41, Mikael Ståldal (mi...@apache.org) wrote: > >> That would make sense. > >> > >> On 2017-07-26 19:51, Jörn Huxhorn wrote: > >>> Isn’t this supposed to also use 0-byte terminated events in the future? > >>> Because in that case it would make sense to change from String to byte[] > >>> processing. > >>> > >>> On 26. July 2017 at 17:49:07, Mikael Ståldal (mi...@apache.org) wrote: > They just need the String version: > > https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100 > > > > On 2017-07-26 17:40, Gary Gregory wrote: > > What do our XML and JSON server need? Let's make sure we handle those > > use > > cases. > > > > Gary > > > > On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: > > > >> +1 > >> > >> > >> On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) wrote: > >>> Maybe I should remove both the Reader and the InputStream versions, > >>> since they cannot parse multiple log events from a stream, and are > >>> thus > >>> not very useful. > >>> > >>> On 2017-07-26 10:34, Jörn Huxhorn wrote: > It would maybe be a good idea to get rid of the Reader method > >> altogether since it mainly > >>> introduces an unnecessary point of failure if the Reader is using an > >> encoding other than > >>> UTF-8. > > Using a FileReader would be an example for this since its limited > >> c'tors are always using > >>> Charset.defaultCharset().name(). > > The correct way to create an UTF-8 file reader is the less than > >> obvious "new InputStreamReader(new > >>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common > >>> > >> mistake. > >>> > >> > >> > > > > > >>> > >> > >> > > > >
Re: Lilith and Log4j 2
In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout. On 2017-08-02 09:23, Jörn Huxhorn wrote: I started working on the Lilith side of this and I think I’m pretty much done. To test the new functionality in a sandbox application, I’d now need the ability to switch JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with a 0-byte instead of being separated by “, ". On 26. July 2017 at 21:14:41, Mikael Ståldal (mi...@apache.org) wrote: That would make sense. On 2017-07-26 19:51, Jörn Huxhorn wrote: Isn’t this supposed to also use 0-byte terminated events in the future? Because in that case it would make sense to change from String to byte[] processing. On 26. July 2017 at 17:49:07, Mikael Ståldal (mi...@apache.org) wrote: They just need the String version: https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100 On 2017-07-26 17:40, Gary Gregory wrote: What do our XML and JSON server need? Let's make sure we handle those use cases. Gary On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: +1 On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) wrote: Maybe I should remove both the Reader and the InputStream versions, since they cannot parse multiple log events from a stream, and are thus not very useful. On 2017-07-26 10:34, Jörn Huxhorn wrote: It would maybe be a good idea to get rid of the Reader method altogether since it mainly introduces an unnecessary point of failure if the Reader is using an encoding other than UTF-8. Using a FileReader would be an example for this since its limited c'tors are always using Charset.defaultCharset().name(). The correct way to create an UTF-8 file reader is the less than obvious "new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common mistake.
Re: Lilith and Log4j 2
I started working on the Lilith side of this and I think I’m pretty much done. To test the new functionality in a sandbox application, I’d now need the ability to switch JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with a 0-byte instead of being separated by “, ". On 26. July 2017 at 21:14:41, Mikael Ståldal (mi...@apache.org) wrote: > That would make sense. > > On 2017-07-26 19:51, Jörn Huxhorn wrote: > > Isn’t this supposed to also use 0-byte terminated events in the future? > > Because in that case it would make sense to change from String to byte[] > > processing. > > > > On 26. July 2017 at 17:49:07, Mikael Ståldal (mi...@apache.org) wrote: > >> They just need the String version: > >> > >> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100 > >> > >> > >> > >> On 2017-07-26 17:40, Gary Gregory wrote: > >>> What do our XML and JSON server need? Let's make sure we handle those use > >>> cases. > >>> > >>> Gary > >>> > >>> On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: > >>> > +1 > > > On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) wrote: > > Maybe I should remove both the Reader and the InputStream versions, > > since they cannot parse multiple log events from a stream, and are thus > > not very useful. > > > > On 2017-07-26 10:34, Jörn Huxhorn wrote: > >> It would maybe be a good idea to get rid of the Reader method > altogether since it mainly > > introduces an unnecessary point of failure if the Reader is using an > encoding other than > > UTF-8. > >> > >> Using a FileReader would be an example for this since its limited > c'tors are always using > > Charset.defaultCharset().name(). > >> > >> The correct way to create an UTF-8 file reader is the less than > obvious "new InputStreamReader(new > > FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common > mistake. > > > > > >>> > >> > >> > > > >
Re: Lilith and Log4j 2
That would make sense. On 2017-07-26 19:51, Jörn Huxhorn wrote: Isn’t this supposed to also use 0-byte terminated events in the future? Because in that case it would make sense to change from String to byte[] processing. On 26. July 2017 at 17:49:07, Mikael Ståldal (mi...@apache.org) wrote: They just need the String version: https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100 On 2017-07-26 17:40, Gary Gregory wrote: What do our XML and JSON server need? Let's make sure we handle those use cases. Gary On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: +1 On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) wrote: Maybe I should remove both the Reader and the InputStream versions, since they cannot parse multiple log events from a stream, and are thus not very useful. On 2017-07-26 10:34, Jörn Huxhorn wrote: It would maybe be a good idea to get rid of the Reader method altogether since it mainly introduces an unnecessary point of failure if the Reader is using an encoding other than UTF-8. Using a FileReader would be an example for this since its limited c'tors are always using Charset.defaultCharset().name(). The correct way to create an UTF-8 file reader is the less than obvious "new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common mistake.
Re: Lilith and Log4j 2
Isn’t this supposed to also use 0-byte terminated events in the future? Because in that case it would make sense to change from String to byte[] processing. On 26. July 2017 at 17:49:07, Mikael Ståldal (mi...@apache.org) wrote: > They just need the String version: > > https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100 > > > > On 2017-07-26 17:40, Gary Gregory wrote: > > What do our XML and JSON server need? Let's make sure we handle those use > > cases. > > > > Gary > > > > On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: > > > >> +1 > >> > >> > >> On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) wrote: > >>> Maybe I should remove both the Reader and the InputStream versions, > >>> since they cannot parse multiple log events from a stream, and are thus > >>> not very useful. > >>> > >>> On 2017-07-26 10:34, Jörn Huxhorn wrote: > It would maybe be a good idea to get rid of the Reader method > >> altogether since it mainly > >>> introduces an unnecessary point of failure if the Reader is using an > >> encoding other than > >>> UTF-8. > > Using a FileReader would be an example for this since its limited > >> c'tors are always using > >>> Charset.defaultCharset().name(). > > The correct way to create an UTF-8 file reader is the less than > >> obvious "new InputStreamReader(new > >>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common > >> mistake. > >>> > >> > >> > > > >
Re: Lilith and Log4j 2
They just need the String version: https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100 On 2017-07-26 17:40, Gary Gregory wrote: What do our XML and JSON server need? Let's make sure we handle those use cases. Gary On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: +1 On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) wrote: Maybe I should remove both the Reader and the InputStream versions, since they cannot parse multiple log events from a stream, and are thus not very useful. On 2017-07-26 10:34, Jörn Huxhorn wrote: It would maybe be a good idea to get rid of the Reader method altogether since it mainly introduces an unnecessary point of failure if the Reader is using an encoding other than UTF-8. Using a FileReader would be an example for this since its limited c'tors are always using Charset.defaultCharset().name(). The correct way to create an UTF-8 file reader is the less than obvious "new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common mistake.
Re: Lilith and Log4j 2
What do our XML and JSON server need? Let's make sure we handle those use cases. Gary On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote: > +1 > > > On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) wrote: > > Maybe I should remove both the Reader and the InputStream versions, > > since they cannot parse multiple log events from a stream, and are thus > > not very useful. > > > > On 2017-07-26 10:34, Jörn Huxhorn wrote: > > > It would maybe be a good idea to get rid of the Reader method > altogether since it mainly > > introduces an unnecessary point of failure if the Reader is using an > encoding other than > > UTF-8. > > > > > > Using a FileReader would be an example for this since its limited > c'tors are always using > > Charset.defaultCharset().name(). > > > > > > The correct way to create an UTF-8 file reader is the less than > obvious "new InputStreamReader(new > > FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common > mistake. > > > >
Re: Lilith and Log4j 2
+1 On 26. July 2017 at 13:29:27, Mikael Ståldal (mi...@apache.org) wrote: > Maybe I should remove both the Reader and the InputStream versions, > since they cannot parse multiple log events from a stream, and are thus > not very useful. > > On 2017-07-26 10:34, Jörn Huxhorn wrote: > > It would maybe be a good idea to get rid of the Reader method altogether > > since it mainly > introduces an unnecessary point of failure if the Reader is using an encoding > other than > UTF-8. > > > > Using a FileReader would be an example for this since its limited c'tors > > are always using > Charset.defaultCharset().name(). > > > > The correct way to create an UTF-8 file reader is the less than obvious > > "new InputStreamReader(new > FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common > mistake. >
Re: Lilith and Log4j 2
That’s what I meant with too lenient. It was probably a bad idea.
Might make more sense to think about which Mixins, if any, should get a
@JsonIgnoreProperties(ignoreUnknown=true) annotation… but that is unrelated to
the general parsing API. I kind of derailed the topic because this really
doesn’t need an immediate decision right now.
On 26. July 2017 at 13:32:28, Mikael Ståldal (mi...@apache.org) wrote:
> A slightly related question, what should the parser do about missing
> properties?
>
> Currently it is so lenient that it will successfully parse an empty JSON
> object "{}" and just assign null (or other default values) to all fields.
>
>
> On 2017-07-25 23:24, Jörn Huxhorn wrote:
> > Hey Mikael,
> > sorry for the slow response time.
> >
> > GMail thought that - of all the dev mailing list mails - this specific mail
> > looked like
> spam. I just found it hidden in the spam folder...
> >
> > The API looks good to me, assuming that I’ll be fine with the default c’tor
> > versions of
> Log4jJsonObjectMapper, Log4jYamlObjectMapper and Log4jXmlObjectMapper.
> Otherwise
> I'd need matching c’tors in the respective LogEventParser implementations.
> >
> > It may make sense to add this line to the ObjectMapper c'tors:
> > this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true);
> >
> > This would make parsing less fragile. If new fields are added in future
> > log4j versions
> then previous versions could still read the event, just without filling the
> new field.
> > Another example would be StackTraceElement getting new fields in Java 9,
> > making it
> possible to still parse Java 9 events in Java 8.
> >
> > Disclaimer: I didn’t test this. I usually annotate specific Mixins with
> > "@JsonIgnoreProperties(ignoreUnknown=true)”
> where it makes sense. This may be way *too* lenient.
> >
> > It isn’t too critical either and is just meant for your consideration. I’m
> > fine with
> the parser implementations regardless of such a change.
> >
> > Thanks a lot for providing an official parser API!
> >
> > Cheers,
> > Jörn.
> >
> >
> > On 25. July 2017 at 09:51:56, Mikael Ståldal (mi...@apache.org) wrote:
> >> Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you?
> >>
> >> On 2017-07-22 22:49, Mikael Ståldal wrote:
> >>> Work in progress in Git branch LOG4J2-1986.
> >>>
> >>> On 2017-07-22 22:31, Mikael Ståldal wrote:
> Yes, I am putting it in its own Java package.
>
>
> On 2017-07-22 22:13, Gary Gregory wrote:
> > Should this API sit in its own package?
> >
> > On Jul 22, 2017 11:41, "Mikael Ståldal" wrote:
> >
> >> I'll give it a shot:
> >>
> >> https://issues.apache.org/jira/browse/LOG4J2-1986
> >>
> >>
> >> On 2017-07-22 20:28, Ralph Goers wrote:
> >>
> >>> Yes, that is a good idea. Probably XML and YAML as well.
> >>>
> >>> Ralph
> >>>
> >>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
>
> But here we have a concrete use case: a third-party tool, Lilith,
> needs
> to parse log events from our JsonLayout. (Our SocketServer in
> log4j-server
> have the same need, and it uses this class.)
>
> Should we provide a public API in log4j-core for doing so, which
> Lilith,
> our SocketServer, and possibly others (Chainsaw maybe?) can use?
> Maybe such
> public API should be a facade around this which hides the fact
> that we use
> Jackson.
>
Re: Lilith and Log4j 2
A slightly related question, what should the parser do about missing
properties?
Currently it is so lenient that it will successfully parse an empty JSON
object "{}" and just assign null (or other default values) to all fields.
On 2017-07-25 23:24, Jörn Huxhorn wrote:
Hey Mikael,
sorry for the slow response time.
GMail thought that - of all the dev mailing list mails - this specific mail
looked like spam. I just found it hidden in the spam folder...
The API looks good to me, assuming that I’ll be fine with the default c’tor
versions of Log4jJsonObjectMapper, Log4jYamlObjectMapper and
Log4jXmlObjectMapper. Otherwise I'd need matching c’tors in the respective
LogEventParser implementations.
It may make sense to add this line to the ObjectMapper c'tors:
this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true);
This would make parsing less fragile. If new fields are added in future log4j
versions then previous versions could still read the event, just without
filling the new field.
Another example would be StackTraceElement getting new fields in Java 9, making
it possible to still parse Java 9 events in Java 8.
Disclaimer: I didn’t test this. I usually annotate specific Mixins with
"@JsonIgnoreProperties(ignoreUnknown=true)” where it makes sense. This may be
way *too* lenient.
It isn’t too critical either and is just meant for your consideration. I’m fine
with the parser implementations regardless of such a change.
Thanks a lot for providing an official parser API!
Cheers,
Jörn.
On 25. July 2017 at 09:51:56, Mikael Ståldal (mi...@apache.org) wrote:
Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you?
On 2017-07-22 22:49, Mikael Ståldal wrote:
Work in progress in Git branch LOG4J2-1986.
On 2017-07-22 22:31, Mikael Ståldal wrote:
Yes, I am putting it in its own Java package.
On 2017-07-22 22:13, Gary Gregory wrote:
Should this API sit in its own package?
On Jul 22, 2017 11:41, "Mikael Ståldal" wrote:
I'll give it a shot:
https://issues.apache.org/jira/browse/LOG4J2-1986
On 2017-07-22 20:28, Ralph Goers wrote:
Yes, that is a good idea. Probably XML and YAML as well.
Ralph
On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
But here we have a concrete use case: a third-party tool, Lilith,
needs
to parse log events from our JsonLayout. (Our SocketServer in
log4j-server
have the same need, and it uses this class.)
Should we provide a public API in log4j-core for doing so, which
Lilith,
our SocketServer, and possibly others (Chainsaw maybe?) can use?
Maybe such
public API should be a facade around this which hides the fact
that we use
Jackson.
Re: Lilith and Log4j 2
Maybe I should remove both the Reader and the InputStream versions, since they cannot parse multiple log events from a stream, and are thus not very useful. On 2017-07-26 10:34, Jörn Huxhorn wrote: It would maybe be a good idea to get rid of the Reader method altogether since it mainly introduces an unnecessary point of failure if the Reader is using an encoding other than UTF-8. Using a FileReader would be an example for this since its limited c'tors are always using Charset.defaultCharset().name(). The correct way to create an UTF-8 file reader is the less than obvious "new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common mistake.
Re: Lilith and Log4j 2
Whoops... yes, of course. false. m) I'd use the byte[] method in Lilith. It would maybe be a good idea to get rid of the Reader method altogether since it mainly introduces an unnecessary point of failure if the Reader is using an encoding other than UTF-8. Using a FileReader would be an example for this since its limited c'tors are always using Charset.defaultCharset().name(). The correct way to create an UTF-8 file reader is the less than obvious "new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common mistake. On 26. July 2017 at 09:50:21, Mikael Ståldal (mi...@apache.org) wrote: > I guess you mean: > > this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); > > Out of curiosity, which of the methods would you use? Reading > InputStream, Reader, String or byte[] ? > > > On 2017-07-25 23:24, Jörn Huxhorn wrote: > > Hey Mikael, > > sorry for the slow response time. > > > > GMail thought that - of all the dev mailing list mails - this specific mail > > looked like > spam. I just found it hidden in the spam folder... > > > > The API looks good to me, assuming that I’ll be fine with the default c’tor > > versions of > Log4jJsonObjectMapper, Log4jYamlObjectMapper and Log4jXmlObjectMapper. > Otherwise > I'd need matching c’tors in the respective LogEventParser implementations. > > > > It may make sense to add this line to the ObjectMapper c'tors: > > this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true); > > > > This would make parsing less fragile. If new fields are added in future > > log4j versions > then previous versions could still read the event, just without filling the > new field. > > Another example would be StackTraceElement getting new fields in Java 9, > > making it > possible to still parse Java 9 events in Java 8. > > > > Disclaimer: I didn’t test this. I usually annotate specific Mixins with > > "@JsonIgnoreProperties(ignoreUnknown=true)” > where it makes sense. This may be way *too* lenient. > > > > It isn’t too critical either and is just meant for your consideration. I’m > > fine with > the parser implementations regardless of such a change. > > > > Thanks a lot for providing an official parser API! > > > > Cheers, > > Jörn. > > > > > > On 25. July 2017 at 09:51:56, Mikael Ståldal (mi...@apache.org) wrote: > >> Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you? > >> > >> On 2017-07-22 22:49, Mikael Ståldal wrote: > >>> Work in progress in Git branch LOG4J2-1986. > >>> > >>> On 2017-07-22 22:31, Mikael Ståldal wrote: > Yes, I am putting it in its own Java package. > > > On 2017-07-22 22:13, Gary Gregory wrote: > > Should this API sit in its own package? > > > > On Jul 22, 2017 11:41, "Mikael Ståldal" wrote: > > > >> I'll give it a shot: > >> > >> https://issues.apache.org/jira/browse/LOG4J2-1986 > >> > >> > >> On 2017-07-22 20:28, Ralph Goers wrote: > >> > >>> Yes, that is a good idea. Probably XML and YAML as well. > >>> > >>> Ralph > >>> > >>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote: > > But here we have a concrete use case: a third-party tool, Lilith, > needs > to parse log events from our JsonLayout. (Our SocketServer in > log4j-server > have the same need, and it uses this class.) > > Should we provide a public API in log4j-core for doing so, which > Lilith, > our SocketServer, and possibly others (Chainsaw maybe?) can use? > Maybe such > public API should be a facade around this which hides the fact > that we use > Jackson. > >>> > >> > >> > > > >
Re: Lilith and Log4j 2
I guess you mean: this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); Out of curiosity, which of the methods would you use? Reading InputStream, Reader, String or byte[] ? On 2017-07-25 23:24, Jörn Huxhorn wrote: Hey Mikael, sorry for the slow response time. GMail thought that - of all the dev mailing list mails - this specific mail looked like spam. I just found it hidden in the spam folder... The API looks good to me, assuming that I’ll be fine with the default c’tor versions of Log4jJsonObjectMapper, Log4jYamlObjectMapper and Log4jXmlObjectMapper. Otherwise I'd need matching c’tors in the respective LogEventParser implementations. It may make sense to add this line to the ObjectMapper c'tors: this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true); This would make parsing less fragile. If new fields are added in future log4j versions then previous versions could still read the event, just without filling the new field. Another example would be StackTraceElement getting new fields in Java 9, making it possible to still parse Java 9 events in Java 8. Disclaimer: I didn’t test this. I usually annotate specific Mixins with "@JsonIgnoreProperties(ignoreUnknown=true)” where it makes sense. This may be way *too* lenient. It isn’t too critical either and is just meant for your consideration. I’m fine with the parser implementations regardless of such a change. Thanks a lot for providing an official parser API! Cheers, Jörn. On 25. July 2017 at 09:51:56, Mikael Ståldal (mi...@apache.org) wrote: Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you? On 2017-07-22 22:49, Mikael Ståldal wrote: Work in progress in Git branch LOG4J2-1986. On 2017-07-22 22:31, Mikael Ståldal wrote: Yes, I am putting it in its own Java package. On 2017-07-22 22:13, Gary Gregory wrote: Should this API sit in its own package? On Jul 22, 2017 11:41, "Mikael Ståldal" wrote: I'll give it a shot: https://issues.apache.org/jira/browse/LOG4J2-1986 On 2017-07-22 20:28, Ralph Goers wrote: Yes, that is a good idea. Probably XML and YAML as well. Ralph On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote: But here we have a concrete use case: a third-party tool, Lilith, needs to parse log events from our JsonLayout. (Our SocketServer in log4j-server have the same need, and it uses this class.) Should we provide a public API in log4j-core for doing so, which Lilith, our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such public API should be a facade around this which hides the fact that we use Jackson.
Re: Lilith and Log4j 2
Hey Mikael, sorry for the slow response time. GMail thought that - of all the dev mailing list mails - this specific mail looked like spam. I just found it hidden in the spam folder... The API looks good to me, assuming that I’ll be fine with the default c’tor versions of Log4jJsonObjectMapper, Log4jYamlObjectMapper and Log4jXmlObjectMapper. Otherwise I'd need matching c’tors in the respective LogEventParser implementations. It may make sense to add this line to the ObjectMapper c'tors: this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true); This would make parsing less fragile. If new fields are added in future log4j versions then previous versions could still read the event, just without filling the new field. Another example would be StackTraceElement getting new fields in Java 9, making it possible to still parse Java 9 events in Java 8. Disclaimer: I didn’t test this. I usually annotate specific Mixins with "@JsonIgnoreProperties(ignoreUnknown=true)” where it makes sense. This may be way *too* lenient. It isn’t too critical either and is just meant for your consideration. I’m fine with the parser implementations regardless of such a change. Thanks a lot for providing an official parser API! Cheers, Jörn. On 25. July 2017 at 09:51:56, Mikael Ståldal (mi...@apache.org) wrote: > Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you? > > On 2017-07-22 22:49, Mikael Ståldal wrote: > > Work in progress in Git branch LOG4J2-1986. > > > > On 2017-07-22 22:31, Mikael Ståldal wrote: > >> Yes, I am putting it in its own Java package. > >> > >> > >> On 2017-07-22 22:13, Gary Gregory wrote: > >>> Should this API sit in its own package? > >>> > >>> On Jul 22, 2017 11:41, "Mikael Ståldal" wrote: > >>> > I'll give it a shot: > > https://issues.apache.org/jira/browse/LOG4J2-1986 > > > On 2017-07-22 20:28, Ralph Goers wrote: > > > Yes, that is a good idea. Probably XML and YAML as well. > > > > Ralph > > > > On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote: > >> > >> But here we have a concrete use case: a third-party tool, Lilith, > >> needs > >> to parse log events from our JsonLayout. (Our SocketServer in > >> log4j-server > >> have the same need, and it uses this class.) > >> > >> Should we provide a public API in log4j-core for doing so, which > >> Lilith, > >> our SocketServer, and possibly others (Chainsaw maybe?) can use? > >> Maybe such > >> public API should be a facade around this which hides the fact > >> that we use > >> Jackson. > > > >
Re: Lilith and Log4j 2
Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you? On 2017-07-22 22:49, Mikael Ståldal wrote: Work in progress in Git branch LOG4J2-1986. On 2017-07-22 22:31, Mikael Ståldal wrote: Yes, I am putting it in its own Java package. On 2017-07-22 22:13, Gary Gregory wrote: Should this API sit in its own package? On Jul 22, 2017 11:41, "Mikael Ståldal" wrote: I'll give it a shot: https://issues.apache.org/jira/browse/LOG4J2-1986 On 2017-07-22 20:28, Ralph Goers wrote: Yes, that is a good idea. Probably XML and YAML as well. Ralph On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote: But here we have a concrete use case: a third-party tool, Lilith, needs to parse log events from our JsonLayout. (Our SocketServer in log4j-server have the same need, and it uses this class.) Should we provide a public API in log4j-core for doing so, which Lilith, our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such public API should be a facade around this which hides the fact that we use Jackson.
Re: Lilith and Log4j 2
Work in progress in Git branch LOG4J2-1986. On 2017-07-22 22:31, Mikael Ståldal wrote: Yes, I am putting it in its own Java package. On 2017-07-22 22:13, Gary Gregory wrote: Should this API sit in its own package? On Jul 22, 2017 11:41, "Mikael Ståldal" wrote: I'll give it a shot: https://issues.apache.org/jira/browse/LOG4J2-1986 On 2017-07-22 20:28, Ralph Goers wrote: Yes, that is a good idea. Probably XML and YAML as well. Ralph On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote: But here we have a concrete use case: a third-party tool, Lilith, needs to parse log events from our JsonLayout. (Our SocketServer in log4j-server have the same need, and it uses this class.) Should we provide a public API in log4j-core for doing so, which Lilith, our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such public API should be a facade around this which hides the fact that we use Jackson.
Re: Lilith and Log4j 2
Yes, I am putting it in its own Java package.
On 2017-07-22 22:13, Gary Gregory wrote:
Should this API sit in its own package?
On Jul 22, 2017 11:41, "Mikael Ståldal" wrote:
I'll give it a shot:
https://issues.apache.org/jira/browse/LOG4J2-1986
On 2017-07-22 20:28, Ralph Goers wrote:
Yes, that is a good idea. Probably XML and YAML as well.
Ralph
On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
But here we have a concrete use case: a third-party tool, Lilith, needs
to parse log events from our JsonLayout. (Our SocketServer in log4j-server
have the same need, and it uses this class.)
Should we provide a public API in log4j-core for doing so, which Lilith,
our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such
public API should be a facade around this which hides the fact that we use
Jackson.
On 2017-07-22 19:04, Gary Gregory wrote:
Hi All,
Just like anything in log4j-core, we try not break binary compatibility
when we can but we do not bend over backwards like we do for log4j-api.
This class is public probably because Jackson requires it to be,
otherwise
I would have made it package private. The fact that we use Jackson for
JSON
IO is an implementation detail IMO, not an extension point. If anyone
wants
to hard-code a reference to log4j-core's guts then I think its fair to
day
that this comes with caveats.
Gary
On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn
wrote:
Well… it says
/**
* A Jackson JSON {@link ObjectMapper} initialized for Log4j.
*
* Consider this class private.
*
*/
so I wanted to holler about how to continue.
I’ll give it a shot.
On 22. July 2017 at 15:06:28, Mikael Ståldal (mi...@apache.org) wrote:
Would it work to use
org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
public?
See here how it is used:
https://github.com/apache/logging-log4j-tools/blob/
master/log4j-server/src/main/java/org/apache/logging/log4j/server/
JsonInputStreamLogEventBridge.java
On 2017-07-22 13:47, Jörn Huxhorn wrote:
It seems like I don’t get access to the log4j2 ObjectMapper or
Jackson
Module used for
JSON serialization, i.e. I can’t do something like this:
new JacksonFactory.JSON(encodeThreadContextAsList,
includeStacktrace,
false)
It would make sense to reuse your ObjectMapper/Module in my code
instead of reimplementing
them to prevent future compatibility issues down the road.
On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com)
wrote:
I’ll give this a shot over the next view days.
Thanks!
On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org)
wrote:
JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
termination of log events. Will be part of the upcoming 2.9
release.
On 2017-07-18 11:48, Jörn Huxhorn wrote:
A general event header and footer would probably be overkill.
I’m using the following interface in Lilith:
public interface WriteByteStrategy {
void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
throws IOException;
}
One implementation writes an int with the length of bytes before
“bytes", the other
implementation writes a 0-byte after “bytes".
The problem with that approach is that it requires the
intermediate
“bytes” array
and
is thus probably incompatible with your zero garbage approach.
I didn’t try end-of-line (\r\n) mainly because I first wanted to
discuss this with
you.
I would also have to implement a new receiver type whereas I could
just reuse an existing
one in case of 0-byte termination.
I’d prefer using 0-byte termination since it would be less fragile
than end-of-line.
The end-of-line approach would break if compact=“false” was set
erroneously while
you’d still have the option of using either compact=“true” or
compact=“false” in
case
of 0-byte termination. 0-byte termination is also slightly more
efficient while
reading
since the splitting can operate on bytes instead of chars or Strings
- another reason
why I didn’t just implement it and instead wanted to discuss this
with you first.
Joern
On 18. July 2017 at 03:26:32, Ralph Goers (
ralph.go...@dslextreme.com) wrote:
Right now that is all handled by the specific layouts. For
example, by default the
RFC5424Layout
doesn’t append newlines so when writing to a file they will all be
on the same “line”,
but
it has an option to append one if needed. Doing the same would be
another option for
the
JSONLayout.
I’d be ok with event header and footers but ONLY if they have
zero
overhead when not
present.
IOW, when the Layout is initialized it would have to wire in the
appropriate encode
method.
Ralph
On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
Do we want a general event header and footer then?
Gary
On Jul 17, 2017 14:43, "Ralph Goers" wrote:
No. A Footer is only used at end of file. He needs to know how
long each
eve
Re: Lilith and Log4j 2
Should this API sit in its own package?
On Jul 22, 2017 11:41, "Mikael Ståldal" wrote:
> I'll give it a shot:
>
> https://issues.apache.org/jira/browse/LOG4J2-1986
>
>
> On 2017-07-22 20:28, Ralph Goers wrote:
>
>> Yes, that is a good idea. Probably XML and YAML as well.
>>
>> Ralph
>>
>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
>>>
>>> But here we have a concrete use case: a third-party tool, Lilith, needs
>>> to parse log events from our JsonLayout. (Our SocketServer in log4j-server
>>> have the same need, and it uses this class.)
>>>
>>> Should we provide a public API in log4j-core for doing so, which Lilith,
>>> our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such
>>> public API should be a facade around this which hides the fact that we use
>>> Jackson.
>>>
>>>
>>> On 2017-07-22 19:04, Gary Gregory wrote:
>>>
Hi All,
Just like anything in log4j-core, we try not break binary compatibility
when we can but we do not bend over backwards like we do for log4j-api.
This class is public probably because Jackson requires it to be,
otherwise
I would have made it package private. The fact that we use Jackson for
JSON
IO is an implementation detail IMO, not an extension point. If anyone
wants
to hard-code a reference to log4j-core's guts then I think its fair to
day
that this comes with caveats.
Gary
On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn
wrote:
> Well… it says
>
> /**
> * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
> *
> * Consider this class private.
> *
> */
>
> so I wanted to holler about how to continue.
>
> I’ll give it a shot.
>
>
> On 22. July 2017 at 15:06:28, Mikael Ståldal (mi...@apache.org) wrote:
>
>> Would it work to use
>> org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
>> public?
>>
>> See here how it is used:
>>
>> https://github.com/apache/logging-log4j-tools/blob/
>>
> master/log4j-server/src/main/java/org/apache/logging/log4j/server/
> JsonInputStreamLogEventBridge.java
>
>>
>>
>> On 2017-07-22 13:47, Jörn Huxhorn wrote:
>>
>>> It seems like I don’t get access to the log4j2 ObjectMapper or
>>> Jackson
>>>
>> Module used for
>
>> JSON serialization, i.e. I can’t do something like this:
>>
>>> new JacksonFactory.JSON(encodeThreadContextAsList,
>>> includeStacktrace,
>>>
>> false)
>
>>
>>> It would make sense to reuse your ObjectMapper/Module in my code
>>>
>> instead of reimplementing
>
>> them to prevent future compatibility issues down the road.
>>
>>>
>>>
>>> On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com)
>>>
>> wrote:
>
>> I’ll give this a shot over the next view days.
Thanks!
On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org)
>>> wrote:
>
>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
> termination of log events. Will be part of the upcoming 2.9
> release.
>
>
> On 2017-07-18 11:48, Jörn Huxhorn wrote:
>
>> A general event header and footer would probably be overkill.
>>
>> I’m using the following interface in Lilith:
>>
>> public interface WriteByteStrategy {
>> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
>> throws IOException;
>> }
>>
>> One implementation writes an int with the length of bytes before
>>
> “bytes", the other
>
>> implementation writes a 0-byte after “bytes".
>
>>
>> The problem with that approach is that it requires the
>> intermediate
>>
> “bytes” array
>
>> and
> is thus probably incompatible with your zero garbage approach.
>
>>
>> I didn’t try end-of-line (\r\n) mainly because I first wanted to
>>
> discuss this with
>
>> you.
> I would also have to implement a new receiver type whereas I could
>
just reuse an existing
>
>> one in case of 0-byte termination.
>
>>
>> I’d prefer using 0-byte termination since it would be less fragile
>>
> than end-of-line.
>
>> The end-of-line approach would break if compact=“false” was set
>
erroneously while
>
>> you’d still have the option of using either compact=“true” or
>
compact=“false” in
>
>> case
>>
>>> of 0-byte termination. 0-byte termination is also slightly more
>
efficient while
>
>> reading
>>
>>> since the splitting can operate on bytes inste
Re: Lilith and Log4j 2
I'll give it a shot:
https://issues.apache.org/jira/browse/LOG4J2-1986
On 2017-07-22 20:28, Ralph Goers wrote:
Yes, that is a good idea. Probably XML and YAML as well.
Ralph
On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
But here we have a concrete use case: a third-party tool, Lilith, needs to
parse log events from our JsonLayout. (Our SocketServer in log4j-server have
the same need, and it uses this class.)
Should we provide a public API in log4j-core for doing so, which Lilith, our
SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such public
API should be a facade around this which hides the fact that we use Jackson.
On 2017-07-22 19:04, Gary Gregory wrote:
Hi All,
Just like anything in log4j-core, we try not break binary compatibility
when we can but we do not bend over backwards like we do for log4j-api.
This class is public probably because Jackson requires it to be, otherwise
I would have made it package private. The fact that we use Jackson for JSON
IO is an implementation detail IMO, not an extension point. If anyone wants
to hard-code a reference to log4j-core's guts then I think its fair to day
that this comes with caveats.
Gary
On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn
wrote:
Well… it says
/**
* A Jackson JSON {@link ObjectMapper} initialized for Log4j.
*
* Consider this class private.
*
*/
so I wanted to holler about how to continue.
I’ll give it a shot.
On 22. July 2017 at 15:06:28, Mikael Ståldal (mi...@apache.org) wrote:
Would it work to use
org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
public?
See here how it is used:
https://github.com/apache/logging-log4j-tools/blob/
master/log4j-server/src/main/java/org/apache/logging/log4j/server/
JsonInputStreamLogEventBridge.java
On 2017-07-22 13:47, Jörn Huxhorn wrote:
It seems like I don’t get access to the log4j2 ObjectMapper or Jackson
Module used for
JSON serialization, i.e. I can’t do something like this:
new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace,
false)
It would make sense to reuse your ObjectMapper/Module in my code
instead of reimplementing
them to prevent future compatibility issues down the road.
On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com)
wrote:
I’ll give this a shot over the next view days.
Thanks!
On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org)
wrote:
JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
termination of log events. Will be part of the upcoming 2.9 release.
On 2017-07-18 11:48, Jörn Huxhorn wrote:
A general event header and footer would probably be overkill.
I’m using the following interface in Lilith:
public interface WriteByteStrategy {
void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
throws IOException;
}
One implementation writes an int with the length of bytes before
“bytes", the other
implementation writes a 0-byte after “bytes".
The problem with that approach is that it requires the intermediate
“bytes” array
and
is thus probably incompatible with your zero garbage approach.
I didn’t try end-of-line (\r\n) mainly because I first wanted to
discuss this with
you.
I would also have to implement a new receiver type whereas I could
just reuse an existing
one in case of 0-byte termination.
I’d prefer using 0-byte termination since it would be less fragile
than end-of-line.
The end-of-line approach would break if compact=“false” was set
erroneously while
you’d still have the option of using either compact=“true” or
compact=“false” in
case
of 0-byte termination. 0-byte termination is also slightly more
efficient while
reading
since the splitting can operate on bytes instead of chars or Strings
- another reason
why I didn’t just implement it and instead wanted to discuss this
with you first.
Joern
On 18. July 2017 at 03:26:32, Ralph Goers (
ralph.go...@dslextreme.com) wrote:
Right now that is all handled by the specific layouts. For
example, by default the
RFC5424Layout
doesn’t append newlines so when writing to a file they will all be
on the same “line”,
but
it has an option to append one if needed. Doing the same would be
another option for
the
JSONLayout.
I’d be ok with event header and footers but ONLY if they have zero
overhead when not
present.
IOW, when the Layout is initialized it would have to wire in the
appropriate encode
method.
Ralph
On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
Do we want a general event header and footer then?
Gary
On Jul 17, 2017 14:43, "Ralph Goers" wrote:
No. A Footer is only used at end of file. He needs to know how
long each
event is or when it is the start of a new event.
Ralph
On Jul 17, 2017, at 12:32 PM, Gary Gregory
wrote:
Can't you use a footer for any terminator you wish?
Gary
On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
wrote:
Hi.
(Moving this discussion to logging dev mailing list.)
Have yo
Re: Lilith and Log4j 2
Yes, that is a good idea. Probably XML and YAML as well.
Ralph
> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
>
> But here we have a concrete use case: a third-party tool, Lilith, needs to
> parse log events from our JsonLayout. (Our SocketServer in log4j-server have
> the same need, and it uses this class.)
>
> Should we provide a public API in log4j-core for doing so, which Lilith, our
> SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such
> public API should be a facade around this which hides the fact that we use
> Jackson.
>
>
> On 2017-07-22 19:04, Gary Gregory wrote:
>> Hi All,
>> Just like anything in log4j-core, we try not break binary compatibility
>> when we can but we do not bend over backwards like we do for log4j-api.
>> This class is public probably because Jackson requires it to be, otherwise
>> I would have made it package private. The fact that we use Jackson for JSON
>> IO is an implementation detail IMO, not an extension point. If anyone wants
>> to hard-code a reference to log4j-core's guts then I think its fair to day
>> that this comes with caveats.
>> Gary
>> On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn
>> wrote:
>>> Well… it says
>>>
>>> /**
>>> * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
>>> *
>>> * Consider this class private.
>>> *
>>> */
>>>
>>> so I wanted to holler about how to continue.
>>>
>>> I’ll give it a shot.
>>>
>>>
>>> On 22. July 2017 at 15:06:28, Mikael Ståldal (mi...@apache.org) wrote:
Would it work to use
org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
public?
See here how it is used:
https://github.com/apache/logging-log4j-tools/blob/
>>> master/log4j-server/src/main/java/org/apache/logging/log4j/server/
>>> JsonInputStreamLogEventBridge.java
On 2017-07-22 13:47, Jörn Huxhorn wrote:
> It seems like I don’t get access to the log4j2 ObjectMapper or Jackson
>>> Module used for
JSON serialization, i.e. I can’t do something like this:
> new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace,
>>> false)
>
> It would make sense to reuse your ObjectMapper/Module in my code
>>> instead of reimplementing
them to prevent future compatibility issues down the road.
>
>
> On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com)
>>> wrote:
>> I’ll give this a shot over the next view days.
>>
>> Thanks!
>>
>>
>> On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org)
>>> wrote:
>>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
>>> termination of log events. Will be part of the upcoming 2.9 release.
>>>
>>>
>>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
A general event header and footer would probably be overkill.
I’m using the following interface in Lilith:
public interface WriteByteStrategy {
void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
throws IOException;
}
One implementation writes an int with the length of bytes before
>>> “bytes", the other
>>> implementation writes a 0-byte after “bytes".
The problem with that approach is that it requires the intermediate
>>> “bytes” array
>> and
>>> is thus probably incompatible with your zero garbage approach.
I didn’t try end-of-line (\r\n) mainly because I first wanted to
>>> discuss this with
>> you.
>>> I would also have to implement a new receiver type whereas I could
>>> just reuse an existing
>>> one in case of 0-byte termination.
I’d prefer using 0-byte termination since it would be less fragile
>>> than end-of-line.
>>> The end-of-line approach would break if compact=“false” was set
>>> erroneously while
>>> you’d still have the option of using either compact=“true” or
>>> compact=“false” in
case
>>> of 0-byte termination. 0-byte termination is also slightly more
>>> efficient while
reading
>>> since the splitting can operate on bytes instead of chars or Strings
>>> - another reason
>>> why I didn’t just implement it and instead wanted to discuss this
>>> with you first.
Joern
On 18. July 2017 at 03:26:32, Ralph Goers (
>>> ralph.go...@dslextreme.com) wrote:
> Right now that is all handled by the specific layouts. For
>>> example, by default the
>> RFC5424Layout
> doesn’t append newlines so when writing to a file they will all be
>>> on the same “line”,
>>> but
> it has an option to append one if needed. Doing the same would be
>>> another option for
>> the
> JSONLayout.
>
> I’d be ok with event header and footers but ONLY if they have zero
>>> overhead when not
>> present.
> IOW, when the Layout is initialized
Re: Lilith and Log4j 2
But here we have a concrete use case: a third-party tool, Lilith, needs
to parse log events from our JsonLayout. (Our SocketServer in
log4j-server have the same need, and it uses this class.)
Should we provide a public API in log4j-core for doing so, which Lilith,
our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe
such public API should be a facade around this which hides the fact that
we use Jackson.
On 2017-07-22 19:04, Gary Gregory wrote:
Hi All,
Just like anything in log4j-core, we try not break binary compatibility
when we can but we do not bend over backwards like we do for log4j-api.
This class is public probably because Jackson requires it to be, otherwise
I would have made it package private. The fact that we use Jackson for JSON
IO is an implementation detail IMO, not an extension point. If anyone wants
to hard-code a reference to log4j-core's guts then I think its fair to day
that this comes with caveats.
Gary
On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn
wrote:
Well… it says
/**
* A Jackson JSON {@link ObjectMapper} initialized for Log4j.
*
* Consider this class private.
*
*/
so I wanted to holler about how to continue.
I’ll give it a shot.
On 22. July 2017 at 15:06:28, Mikael Ståldal (mi...@apache.org) wrote:
Would it work to use
org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
public?
See here how it is used:
https://github.com/apache/logging-log4j-tools/blob/
master/log4j-server/src/main/java/org/apache/logging/log4j/server/
JsonInputStreamLogEventBridge.java
On 2017-07-22 13:47, Jörn Huxhorn wrote:
It seems like I don’t get access to the log4j2 ObjectMapper or Jackson
Module used for
JSON serialization, i.e. I can’t do something like this:
new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace,
false)
It would make sense to reuse your ObjectMapper/Module in my code
instead of reimplementing
them to prevent future compatibility issues down the road.
On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com)
wrote:
I’ll give this a shot over the next view days.
Thanks!
On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org)
wrote:
JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
termination of log events. Will be part of the upcoming 2.9 release.
On 2017-07-18 11:48, Jörn Huxhorn wrote:
A general event header and footer would probably be overkill.
I’m using the following interface in Lilith:
public interface WriteByteStrategy {
void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
throws IOException;
}
One implementation writes an int with the length of bytes before
“bytes", the other
implementation writes a 0-byte after “bytes".
The problem with that approach is that it requires the intermediate
“bytes” array
and
is thus probably incompatible with your zero garbage approach.
I didn’t try end-of-line (\r\n) mainly because I first wanted to
discuss this with
you.
I would also have to implement a new receiver type whereas I could
just reuse an existing
one in case of 0-byte termination.
I’d prefer using 0-byte termination since it would be less fragile
than end-of-line.
The end-of-line approach would break if compact=“false” was set
erroneously while
you’d still have the option of using either compact=“true” or
compact=“false” in
case
of 0-byte termination. 0-byte termination is also slightly more
efficient while
reading
since the splitting can operate on bytes instead of chars or Strings
- another reason
why I didn’t just implement it and instead wanted to discuss this
with you first.
Joern
On 18. July 2017 at 03:26:32, Ralph Goers (
ralph.go...@dslextreme.com) wrote:
Right now that is all handled by the specific layouts. For
example, by default the
RFC5424Layout
doesn’t append newlines so when writing to a file they will all be
on the same “line”,
but
it has an option to append one if needed. Doing the same would be
another option for
the
JSONLayout.
I’d be ok with event header and footers but ONLY if they have zero
overhead when not
present.
IOW, when the Layout is initialized it would have to wire in the
appropriate encode
method.
Ralph
On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
Do we want a general event header and footer then?
Gary
On Jul 17, 2017 14:43, "Ralph Goers" wrote:
No. A Footer is only used at end of file. He needs to know how
long each
event is or when it is the start of a new event.
Ralph
On Jul 17, 2017, at 12:32 PM, Gary Gregory
wrote:
Can't you use a footer for any terminator you wish?
Gary
On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
wrote:
Hi.
(Moving this discussion to logging dev mailing list.)
Have you tried to use:
Then each log event will be terminated by end-of-line (\r\n).
I think it would be easy to implement 0-byte terminated log
events in
JsonLayout, and that would make sense since we have
implemented support
f
Re: Lilith and Log4j 2
Hi All,
Just like anything in log4j-core, we try not break binary compatibility
when we can but we do not bend over backwards like we do for log4j-api.
This class is public probably because Jackson requires it to be, otherwise
I would have made it package private. The fact that we use Jackson for JSON
IO is an implementation detail IMO, not an extension point. If anyone wants
to hard-code a reference to log4j-core's guts then I think its fair to day
that this comes with caveats.
Gary
On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn
wrote:
> Well… it says
>
> /**
> * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
> *
> * Consider this class private.
> *
> */
>
> so I wanted to holler about how to continue.
>
> I’ll give it a shot.
>
>
> On 22. July 2017 at 15:06:28, Mikael Ståldal (mi...@apache.org) wrote:
> > Would it work to use
> > org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
> > public?
> >
> > See here how it is used:
> >
> > https://github.com/apache/logging-log4j-tools/blob/
> master/log4j-server/src/main/java/org/apache/logging/log4j/server/
> JsonInputStreamLogEventBridge.java
> >
> >
> > On 2017-07-22 13:47, Jörn Huxhorn wrote:
> > > It seems like I don’t get access to the log4j2 ObjectMapper or Jackson
> Module used for
> > JSON serialization, i.e. I can’t do something like this:
> > > new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace,
> false)
> > >
> > > It would make sense to reuse your ObjectMapper/Module in my code
> instead of reimplementing
> > them to prevent future compatibility issues down the road.
> > >
> > >
> > > On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com)
> wrote:
> > >> I’ll give this a shot over the next view days.
> > >>
> > >> Thanks!
> > >>
> > >>
> > >> On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org)
> wrote:
> > >>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
> > >>> termination of log events. Will be part of the upcoming 2.9 release.
> > >>>
> > >>>
> > >>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
> > A general event header and footer would probably be overkill.
> >
> > I’m using the following interface in Lilith:
> >
> > public interface WriteByteStrategy {
> > void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
> > throws IOException;
> > }
> >
> > One implementation writes an int with the length of bytes before
> “bytes", the other
> > >>> implementation writes a 0-byte after “bytes".
> >
> > The problem with that approach is that it requires the intermediate
> “bytes” array
> > >> and
> > >>> is thus probably incompatible with your zero garbage approach.
> >
> > I didn’t try end-of-line (\r\n) mainly because I first wanted to
> discuss this with
> > >> you.
> > >>> I would also have to implement a new receiver type whereas I could
> just reuse an existing
> > >>> one in case of 0-byte termination.
> >
> > I’d prefer using 0-byte termination since it would be less fragile
> than end-of-line.
> > >>> The end-of-line approach would break if compact=“false” was set
> erroneously while
> > >>> you’d still have the option of using either compact=“true” or
> compact=“false” in
> > case
> > >>> of 0-byte termination. 0-byte termination is also slightly more
> efficient while
> > reading
> > >>> since the splitting can operate on bytes instead of chars or Strings
> - another reason
> > >>> why I didn’t just implement it and instead wanted to discuss this
> with you first.
> >
> > Joern
> >
> >
> > On 18. July 2017 at 03:26:32, Ralph Goers (
> ralph.go...@dslextreme.com) wrote:
> > > Right now that is all handled by the specific layouts. For
> example, by default the
> > >> RFC5424Layout
> > > doesn’t append newlines so when writing to a file they will all be
> on the same “line”,
> > >>> but
> > > it has an option to append one if needed. Doing the same would be
> another option for
> > >> the
> > > JSONLayout.
> > >
> > > I’d be ok with event header and footers but ONLY if they have zero
> overhead when not
> > >> present.
> > > IOW, when the Layout is initialized it would have to wire in the
> appropriate encode
> > >>> method.
> > >
> > > Ralph
> > >
> > >> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
> > >>
> > >> Do we want a general event header and footer then?
> > >>
> > >> Gary
> > >>
> > >> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
> > >>
> > >>> No. A Footer is only used at end of file. He needs to know how
> long each
> > >>> event is or when it is the start of a new event.
> > >>>
> > >>> Ralph
> > >>>
> > On Jul 17, 2017, at 12:32 PM, Gary Gregory
> > >>> wrote:
> >
> > Can't you use a footer for any terminator you wish?
> >
> > Gary
> >
> > On Mon, Jul 17, 2017 at
Re: Lilith and Log4j 2
Gary, what do you say? I think you once added that comment.
On 2017-07-22 16:27, Jörn Huxhorn wrote:
Well… it says
/**
* A Jackson JSON {@link ObjectMapper} initialized for Log4j.
*
* Consider this class private.
*
*/
so I wanted to holler about how to continue.
I’ll give it a shot.
On 22. July 2017 at 15:06:28, Mikael Ståldal (mi...@apache.org) wrote:
Would it work to use
org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
public?
See here how it is used:
https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/JsonInputStreamLogEventBridge.java
On 2017-07-22 13:47, Jörn Huxhorn wrote:
It seems like I don’t get access to the log4j2 ObjectMapper or Jackson Module
used for
JSON serialization, i.e. I can’t do something like this:
new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace, false)
It would make sense to reuse your ObjectMapper/Module in my code instead of
reimplementing
them to prevent future compatibility issues down the road.
On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com) wrote:
I’ll give this a shot over the next view days.
Thanks!
On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org) wrote:
JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
termination of log events. Will be part of the upcoming 2.9 release.
On 2017-07-18 11:48, Jörn Huxhorn wrote:
A general event header and footer would probably be overkill.
I’m using the following interface in Lilith:
public interface WriteByteStrategy {
void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
throws IOException;
}
One implementation writes an int with the length of bytes before “bytes", the
other
implementation writes a 0-byte after “bytes".
The problem with that approach is that it requires the intermediate “bytes”
array
and
is thus probably incompatible with your zero garbage approach.
I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this
with
you.
I would also have to implement a new receiver type whereas I could just reuse
an existing
one in case of 0-byte termination.
I’d prefer using 0-byte termination since it would be less fragile than
end-of-line.
The end-of-line approach would break if compact=“false” was set erroneously
while
you’d still have the option of using either compact=“true” or compact=“false” in
case
of 0-byte termination. 0-byte termination is also slightly more efficient while
reading
since the splitting can operate on bytes instead of chars or Strings - another
reason
why I didn’t just implement it and instead wanted to discuss this with you
first.
Joern
On 18. July 2017 at 03:26:32, Ralph Goers (ralph.go...@dslextreme.com) wrote:
Right now that is all handled by the specific layouts. For example, by default
the
RFC5424Layout
doesn’t append newlines so when writing to a file they will all be on the same
“line”,
but
it has an option to append one if needed. Doing the same would be another
option for
the
JSONLayout.
I’d be ok with event header and footers but ONLY if they have zero overhead
when not
present.
IOW, when the Layout is initialized it would have to wire in the appropriate
encode
method.
Ralph
On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
Do we want a general event header and footer then?
Gary
On Jul 17, 2017 14:43, "Ralph Goers" wrote:
No. A Footer is only used at end of file. He needs to know how long each
event is or when it is the start of a new event.
Ralph
On Jul 17, 2017, at 12:32 PM, Gary Gregory
wrote:
Can't you use a footer for any terminator you wish?
Gary
On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
wrote:
Hi.
(Moving this discussion to logging dev mailing list.)
Have you tried to use:
Then each log event will be terminated by end-of-line (\r\n).
I think it would be easy to implement 0-byte terminated log events in
JsonLayout, and that would make sense since we have implemented support
for
that in GelfLayout. Created a JIRA issue for it:
https://issues.apache.org/jira/browse/LOG4J2-1981
As for other tools, the only receivers for Log4j 2 SerializedLayout we
know are Log4j's own SocketServer and Lilith. Chainsaw currently only
support Log4j 1 SerializedLayout. Log4j's own SocketServer support
JsonLayout and XmlLayout as well, and we are changing it to use
JsonLayout
by default.
On 2017-07-17 15:01, Joern Huxhorn wrote:
Hi Mikael,
I've just taken a look at the JsonLayout and see no way to parse its
output event by event.
It's semi-fine for files because then it will result in a well-formed
array at the root level that can be read in one big swoop. It isn't
really ideal for that use case either since it means that the whole
array containing all events needs to be read into memory in one piece,
causing OOM in case of huge files.
But in Lilith, I really need to process events one by one. This is
a
Re: Lilith and Log4j 2
Well… it says
/**
* A Jackson JSON {@link ObjectMapper} initialized for Log4j.
*
* Consider this class private.
*
*/
so I wanted to holler about how to continue.
I’ll give it a shot.
On 22. July 2017 at 15:06:28, Mikael Ståldal (mi...@apache.org) wrote:
> Would it work to use
> org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
> public?
>
> See here how it is used:
>
> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/JsonInputStreamLogEventBridge.java
>
>
>
> On 2017-07-22 13:47, Jörn Huxhorn wrote:
> > It seems like I don’t get access to the log4j2 ObjectMapper or Jackson
> > Module used for
> JSON serialization, i.e. I can’t do something like this:
> > new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace,
> > false)
> >
> > It would make sense to reuse your ObjectMapper/Module in my code instead of
> > reimplementing
> them to prevent future compatibility issues down the road.
> >
> >
> > On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com) wrote:
> >> I’ll give this a shot over the next view days.
> >>
> >> Thanks!
> >>
> >>
> >> On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org) wrote:
> >>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
> >>> termination of log events. Will be part of the upcoming 2.9 release.
> >>>
> >>>
> >>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
> A general event header and footer would probably be overkill.
>
> I’m using the following interface in Lilith:
>
> public interface WriteByteStrategy {
> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
> throws IOException;
> }
>
> One implementation writes an int with the length of bytes before
> “bytes", the other
> >>> implementation writes a 0-byte after “bytes".
>
> The problem with that approach is that it requires the intermediate
> “bytes” array
> >> and
> >>> is thus probably incompatible with your zero garbage approach.
>
> I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss
> this with
> >> you.
> >>> I would also have to implement a new receiver type whereas I could just
> >>> reuse an existing
> >>> one in case of 0-byte termination.
>
> I’d prefer using 0-byte termination since it would be less fragile than
> end-of-line.
> >>> The end-of-line approach would break if compact=“false” was set
> >>> erroneously while
> >>> you’d still have the option of using either compact=“true” or
> >>> compact=“false” in
> case
> >>> of 0-byte termination. 0-byte termination is also slightly more efficient
> >>> while
> reading
> >>> since the splitting can operate on bytes instead of chars or Strings -
> >>> another reason
> >>> why I didn’t just implement it and instead wanted to discuss this with
> >>> you first.
>
> Joern
>
>
> On 18. July 2017 at 03:26:32, Ralph Goers (ralph.go...@dslextreme.com)
> wrote:
> > Right now that is all handled by the specific layouts. For example, by
> > default the
> >> RFC5424Layout
> > doesn’t append newlines so when writing to a file they will all be on
> > the same “line”,
> >>> but
> > it has an option to append one if needed. Doing the same would be
> > another option for
> >> the
> > JSONLayout.
> >
> > I’d be ok with event header and footers but ONLY if they have zero
> > overhead when not
> >> present.
> > IOW, when the Layout is initialized it would have to wire in the
> > appropriate encode
> >>> method.
> >
> > Ralph
> >
> >> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
> >>
> >> Do we want a general event header and footer then?
> >>
> >> Gary
> >>
> >> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
> >>
> >>> No. A Footer is only used at end of file. He needs to know how long
> >>> each
> >>> event is or when it is the start of a new event.
> >>>
> >>> Ralph
> >>>
> On Jul 17, 2017, at 12:32 PM, Gary Gregory
> >>> wrote:
>
> Can't you use a footer for any terminator you wish?
>
> Gary
>
> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
> >>> wrote:
>
> > Hi.
> >
> > (Moving this discussion to logging dev mailing list.)
> >
> > Have you tried to use:
> >
> >
> > Then each log event will be terminated by end-of-line (\r\n).
> >
> > I think it would be easy to implement 0-byte terminated log events
> > in
> > JsonLayout, and that would make sense since we have implemented
> > support
> >>> for
> > that in GelfLayout. Created a JIRA issue for it:
> > https://issues.apache.org/jira/browse/
Re: Lilith and Log4j 2
Would it work to use
org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
public?
See here how it is used:
https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/JsonInputStreamLogEventBridge.java
On 2017-07-22 13:47, Jörn Huxhorn wrote:
It seems like I don’t get access to the log4j2 ObjectMapper or Jackson Module
used for JSON serialization, i.e. I can’t do something like this:
new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace, false)
It would make sense to reuse your ObjectMapper/Module in my code instead of
reimplementing them to prevent future compatibility issues down the road.
On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com) wrote:
I’ll give this a shot over the next view days.
Thanks!
On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org) wrote:
JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
termination of log events. Will be part of the upcoming 2.9 release.
On 2017-07-18 11:48, Jörn Huxhorn wrote:
A general event header and footer would probably be overkill.
I’m using the following interface in Lilith:
public interface WriteByteStrategy {
void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
throws IOException;
}
One implementation writes an int with the length of bytes before “bytes", the
other
implementation writes a 0-byte after “bytes".
The problem with that approach is that it requires the intermediate “bytes”
array
and
is thus probably incompatible with your zero garbage approach.
I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this
with
you.
I would also have to implement a new receiver type whereas I could just reuse
an existing
one in case of 0-byte termination.
I’d prefer using 0-byte termination since it would be less fragile than
end-of-line.
The end-of-line approach would break if compact=“false” was set erroneously
while
you’d still have the option of using either compact=“true” or compact=“false”
in case
of 0-byte termination. 0-byte termination is also slightly more efficient while
reading
since the splitting can operate on bytes instead of chars or Strings - another
reason
why I didn’t just implement it and instead wanted to discuss this with you
first.
Joern
On 18. July 2017 at 03:26:32, Ralph Goers (ralph.go...@dslextreme.com) wrote:
Right now that is all handled by the specific layouts. For example, by default
the
RFC5424Layout
doesn’t append newlines so when writing to a file they will all be on the same
“line”,
but
it has an option to append one if needed. Doing the same would be another
option for
the
JSONLayout.
I’d be ok with event header and footers but ONLY if they have zero overhead
when not
present.
IOW, when the Layout is initialized it would have to wire in the appropriate
encode
method.
Ralph
On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
Do we want a general event header and footer then?
Gary
On Jul 17, 2017 14:43, "Ralph Goers" wrote:
No. A Footer is only used at end of file. He needs to know how long each
event is or when it is the start of a new event.
Ralph
On Jul 17, 2017, at 12:32 PM, Gary Gregory
wrote:
Can't you use a footer for any terminator you wish?
Gary
On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
wrote:
Hi.
(Moving this discussion to logging dev mailing list.)
Have you tried to use:
Then each log event will be terminated by end-of-line (\r\n).
I think it would be easy to implement 0-byte terminated log events in
JsonLayout, and that would make sense since we have implemented support
for
that in GelfLayout. Created a JIRA issue for it:
https://issues.apache.org/jira/browse/LOG4J2-1981
As for other tools, the only receivers for Log4j 2 SerializedLayout we
know are Log4j's own SocketServer and Lilith. Chainsaw currently only
support Log4j 1 SerializedLayout. Log4j's own SocketServer support
JsonLayout and XmlLayout as well, and we are changing it to use
JsonLayout
by default.
On 2017-07-17 15:01, Joern Huxhorn wrote:
Hi Mikael,
I've just taken a look at the JsonLayout and see no way to parse its
output event by event.
It's semi-fine for files because then it will result in a well-formed
array at the root level that can be read in one big swoop. It isn't
really ideal for that use case either since it means that the whole
array containing all events needs to be read into memory in one piece,
causing OOM in case of huge files.
But in Lilith, I really need to process events one by one. This is
absolutely crucial.
The only way to achieve this would essentially require me to
re-implement a JSON parser that can cope with data like this.
I implemented a similar kludge to support reading of log4j xml files
(see
https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
) but this was way easier since i cou
Re: Lilith and Log4j 2
It seems like I don’t get access to the log4j2 ObjectMapper or Jackson Module
used for JSON serialization, i.e. I can’t do something like this:
new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace, false)
It would make sense to reuse your ObjectMapper/Module in my code instead of
reimplementing them to prevent future compatibility issues down the road.
On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com) wrote:
> I’ll give this a shot over the next view days.
>
> Thanks!
>
>
> On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org) wrote:
> > JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
> > termination of log events. Will be part of the upcoming 2.9 release.
> >
> >
> > On 2017-07-18 11:48, Jörn Huxhorn wrote:
> > > A general event header and footer would probably be overkill.
> > >
> > > I’m using the following interface in Lilith:
> > >
> > > public interface WriteByteStrategy {
> > > void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
> > > throws IOException;
> > > }
> > >
> > > One implementation writes an int with the length of bytes before “bytes",
> > > the other
> > implementation writes a 0-byte after “bytes".
> > >
> > > The problem with that approach is that it requires the intermediate
> > > “bytes” array
> and
> > is thus probably incompatible with your zero garbage approach.
> > >
> > > I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss
> > > this with
> you.
> > I would also have to implement a new receiver type whereas I could just
> > reuse an existing
> > one in case of 0-byte termination.
> > >
> > > I’d prefer using 0-byte termination since it would be less fragile than
> > > end-of-line.
> > The end-of-line approach would break if compact=“false” was set erroneously
> > while
> > you’d still have the option of using either compact=“true” or
> > compact=“false” in case
> > of 0-byte termination. 0-byte termination is also slightly more efficient
> > while reading
> > since the splitting can operate on bytes instead of chars or Strings -
> > another reason
> > why I didn’t just implement it and instead wanted to discuss this with you
> > first.
> > >
> > > Joern
> > >
> > >
> > > On 18. July 2017 at 03:26:32, Ralph Goers (ralph.go...@dslextreme.com)
> > > wrote:
> > >> Right now that is all handled by the specific layouts. For example, by
> > >> default the
> RFC5424Layout
> > >> doesn’t append newlines so when writing to a file they will all be on
> > >> the same “line”,
> > but
> > >> it has an option to append one if needed. Doing the same would be
> > >> another option for
> the
> > >> JSONLayout.
> > >>
> > >> I’d be ok with event header and footers but ONLY if they have zero
> > >> overhead when not
> present.
> > >> IOW, when the Layout is initialized it would have to wire in the
> > >> appropriate encode
> > method.
> > >>
> > >> Ralph
> > >>
> > >>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
> > >>>
> > >>> Do we want a general event header and footer then?
> > >>>
> > >>> Gary
> > >>>
> > >>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
> > >>>
> > No. A Footer is only used at end of file. He needs to know how long
> > each
> > event is or when it is the start of a new event.
> >
> > Ralph
> >
> > > On Jul 17, 2017, at 12:32 PM, Gary Gregory
> > wrote:
> > >
> > > Can't you use a footer for any terminator you wish?
> > >
> > > Gary
> > >
> > > On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
> > wrote:
> > >
> > >> Hi.
> > >>
> > >> (Moving this discussion to logging dev mailing list.)
> > >>
> > >> Have you tried to use:
> > >>
> > >>
> > >> Then each log event will be terminated by end-of-line (\r\n).
> > >>
> > >> I think it would be easy to implement 0-byte terminated log events in
> > >> JsonLayout, and that would make sense since we have implemented
> > >> support
> > for
> > >> that in GelfLayout. Created a JIRA issue for it:
> > >> https://issues.apache.org/jira/browse/LOG4J2-1981
> > >>
> > >> As for other tools, the only receivers for Log4j 2 SerializedLayout
> > >> we
> > >> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
> > >> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
> > >> JsonLayout and XmlLayout as well, and we are changing it to use
> > JsonLayout
> > >> by default.
> > >>
> > >>
> > >> On 2017-07-17 15:01, Joern Huxhorn wrote:
> > >>
> > >>> Hi Mikael,
> > >>>
> > >>> I've just taken a look at the JsonLayout and see no way to parse its
> > >>> output event by event.
> > >>>
> > >>> It's semi-fine for files because then it will result in a
> > >>> well-formed
> > >>> array at the root level that can be read in one big swoop. It isn't
> > >>> real
Re: Lilith and Log4j 2
I’ll give this a shot over the next view days.
Thanks!
On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org) wrote:
> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
> termination of log events. Will be part of the upcoming 2.9 release.
>
>
> On 2017-07-18 11:48, Jörn Huxhorn wrote:
> > A general event header and footer would probably be overkill.
> >
> > I’m using the following interface in Lilith:
> >
> > public interface WriteByteStrategy {
> > void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
> > throws IOException;
> > }
> >
> > One implementation writes an int with the length of bytes before “bytes",
> > the other
> implementation writes a 0-byte after “bytes".
> >
> > The problem with that approach is that it requires the intermediate “bytes”
> > array and
> is thus probably incompatible with your zero garbage approach.
> >
> > I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss
> > this with you.
> I would also have to implement a new receiver type whereas I could just reuse
> an existing
> one in case of 0-byte termination.
> >
> > I’d prefer using 0-byte termination since it would be less fragile than
> > end-of-line.
> The end-of-line approach would break if compact=“false” was set erroneously
> while
> you’d still have the option of using either compact=“true” or compact=“false”
> in case
> of 0-byte termination. 0-byte termination is also slightly more efficient
> while reading
> since the splitting can operate on bytes instead of chars or Strings -
> another reason
> why I didn’t just implement it and instead wanted to discuss this with you
> first.
> >
> > Joern
> >
> >
> > On 18. July 2017 at 03:26:32, Ralph Goers (ralph.go...@dslextreme.com)
> > wrote:
> >> Right now that is all handled by the specific layouts. For example, by
> >> default the RFC5424Layout
> >> doesn’t append newlines so when writing to a file they will all be on the
> >> same “line”,
> but
> >> it has an option to append one if needed. Doing the same would be another
> >> option for the
> >> JSONLayout.
> >>
> >> I’d be ok with event header and footers but ONLY if they have zero
> >> overhead when not present.
> >> IOW, when the Layout is initialized it would have to wire in the
> >> appropriate encode
> method.
> >>
> >> Ralph
> >>
> >>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
> >>>
> >>> Do we want a general event header and footer then?
> >>>
> >>> Gary
> >>>
> >>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
> >>>
> No. A Footer is only used at end of file. He needs to know how long each
> event is or when it is the start of a new event.
>
> Ralph
>
> > On Jul 17, 2017, at 12:32 PM, Gary Gregory
> wrote:
> >
> > Can't you use a footer for any terminator you wish?
> >
> > Gary
> >
> > On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
> wrote:
> >
> >> Hi.
> >>
> >> (Moving this discussion to logging dev mailing list.)
> >>
> >> Have you tried to use:
> >>
> >>
> >> Then each log event will be terminated by end-of-line (\r\n).
> >>
> >> I think it would be easy to implement 0-byte terminated log events in
> >> JsonLayout, and that would make sense since we have implemented support
> for
> >> that in GelfLayout. Created a JIRA issue for it:
> >> https://issues.apache.org/jira/browse/LOG4J2-1981
> >>
> >> As for other tools, the only receivers for Log4j 2 SerializedLayout we
> >> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
> >> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
> >> JsonLayout and XmlLayout as well, and we are changing it to use
> JsonLayout
> >> by default.
> >>
> >>
> >> On 2017-07-17 15:01, Joern Huxhorn wrote:
> >>
> >>> Hi Mikael,
> >>>
> >>> I've just taken a look at the JsonLayout and see no way to parse its
> >>> output event by event.
> >>>
> >>> It's semi-fine for files because then it will result in a well-formed
> >>> array at the root level that can be read in one big swoop. It isn't
> >>> really ideal for that use case either since it means that the whole
> >>> array containing all events needs to be read into memory in one piece,
> >>> causing OOM in case of huge files.
> >>>
> >>> But in Lilith, I really need to process events one by one. This is
> >>> absolutely crucial.
> >>>
> >>> The only way to achieve this would essentially require me to
> >>> re-implement a JSON parser that can cope with data like this.
> >>>
> >>> I implemented a similar kludge to support reading of log4j xml files
> >>> (see
> >>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
> >>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
> >>> ) but this was way easier since i c
Re: Lilith and Log4j 2
JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
termination of log events. Will be part of the upcoming 2.9 release.
On 2017-07-18 11:48, Jörn Huxhorn wrote:
A general event header and footer would probably be overkill.
I’m using the following interface in Lilith:
public interface WriteByteStrategy {
void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
throws IOException;
}
One implementation writes an int with the length of bytes before “bytes", the other
implementation writes a 0-byte after “bytes".
The problem with that approach is that it requires the intermediate “bytes”
array and is thus probably incompatible with your zero garbage approach.
I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this
with you. I would also have to implement a new receiver type whereas I could
just reuse an existing one in case of 0-byte termination.
I’d prefer using 0-byte termination since it would be less fragile than
end-of-line. The end-of-line approach would break if compact=“false” was set
erroneously while you’d still have the option of using either compact=“true” or
compact=“false” in case of 0-byte termination. 0-byte termination is also
slightly more efficient while reading since the splitting can operate on bytes
instead of chars or Strings - another reason why I didn’t just implement it and
instead wanted to discuss this with you first.
Joern
On 18. July 2017 at 03:26:32, Ralph Goers (ralph.go...@dslextreme.com) wrote:
Right now that is all handled by the specific layouts. For example, by default
the RFC5424Layout
doesn’t append newlines so when writing to a file they will all be on the same
“line”, but
it has an option to append one if needed. Doing the same would be another
option for the
JSONLayout.
I’d be ok with event header and footers but ONLY if they have zero overhead when not present.
IOW, when the Layout is initialized it would have to wire in the appropriate
encode method.
Ralph
On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
Do we want a general event header and footer then?
Gary
On Jul 17, 2017 14:43, "Ralph Goers" wrote:
No. A Footer is only used at end of file. He needs to know how long each
event is or when it is the start of a new event.
Ralph
On Jul 17, 2017, at 12:32 PM, Gary Gregory
wrote:
Can't you use a footer for any terminator you wish?
Gary
On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
wrote:
Hi.
(Moving this discussion to logging dev mailing list.)
Have you tried to use:
Then each log event will be terminated by end-of-line (\r\n).
I think it would be easy to implement 0-byte terminated log events in
JsonLayout, and that would make sense since we have implemented support
for
that in GelfLayout. Created a JIRA issue for it:
https://issues.apache.org/jira/browse/LOG4J2-1981
As for other tools, the only receivers for Log4j 2 SerializedLayout we
know are Log4j's own SocketServer and Lilith. Chainsaw currently only
support Log4j 1 SerializedLayout. Log4j's own SocketServer support
JsonLayout and XmlLayout as well, and we are changing it to use
JsonLayout
by default.
On 2017-07-17 15:01, Joern Huxhorn wrote:
Hi Mikael,
I've just taken a look at the JsonLayout and see no way to parse its
output event by event.
It's semi-fine for files because then it will result in a well-formed
array at the root level that can be read in one big swoop. It isn't
really ideal for that use case either since it means that the whole
array containing all events needs to be read into memory in one piece,
causing OOM in case of huge files.
But in Lilith, I really need to process events one by one. This is
absolutely crucial.
The only way to achieve this would essentially require me to
re-implement a JSON parser that can cope with data like this.
I implemented a similar kludge to support reading of log4j xml files
(see
https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
) but this was way easier since i could simply search for
"" to find the position at which I could split the stream
of events into a valid XML document for every single event . This was
still kind of nasty but doable, especially since processing an
"offline"
XML file doesn't have the same performance restriction as handling of
live events.
I wouldn't have the luxury of such a unique split signal in JSON,
though.
I would need to detect the "}" at the level of the root array which
would not simply involve counting of opening and closing brackets but
also ignoring of brackets inside of strings. This would boil down to
implementing a custom JSON reader and I won't be doing this.
This isn't just me being lazy either. Such an implementation wouldn't
be
resource (CPU/memory) friendly anyway.
In my own JSON receiver using my own JSON format there are two ways to
send those events:
- write an int containing the amount of bytes repr
Re: Lilith and Log4j 2
A general event header and footer would probably be overkill.
I’m using the following interface in Lilith:
public interface WriteByteStrategy {
void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
throws IOException;
}
One implementation writes an int with the length of bytes before “bytes", the
other implementation writes a 0-byte after “bytes".
The problem with that approach is that it requires the intermediate “bytes”
array and is thus probably incompatible with your zero garbage approach.
I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this
with you. I would also have to implement a new receiver type whereas I could
just reuse an existing one in case of 0-byte termination.
I’d prefer using 0-byte termination since it would be less fragile than
end-of-line. The end-of-line approach would break if compact=“false” was set
erroneously while you’d still have the option of using either compact=“true” or
compact=“false” in case of 0-byte termination. 0-byte termination is also
slightly more efficient while reading since the splitting can operate on bytes
instead of chars or Strings - another reason why I didn’t just implement it and
instead wanted to discuss this with you first.
Joern
On 18. July 2017 at 03:26:32, Ralph Goers (ralph.go...@dslextreme.com) wrote:
> Right now that is all handled by the specific layouts. For example, by
> default the RFC5424Layout
> doesn’t append newlines so when writing to a file they will all be on the
> same “line”, but
> it has an option to append one if needed. Doing the same would be another
> option for the
> JSONLayout.
>
> I’d be ok with event header and footers but ONLY if they have zero overhead
> when not present.
> IOW, when the Layout is initialized it would have to wire in the appropriate
> encode method.
>
> Ralph
>
> > On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
> >
> > Do we want a general event header and footer then?
> >
> > Gary
> >
> > On Jul 17, 2017 14:43, "Ralph Goers" wrote:
> >
> >> No. A Footer is only used at end of file. He needs to know how long each
> >> event is or when it is the start of a new event.
> >>
> >> Ralph
> >>
> >>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
> >> wrote:
> >>>
> >>> Can't you use a footer for any terminator you wish?
> >>>
> >>> Gary
> >>>
> >>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
> >> wrote:
> >>>
> Hi.
>
> (Moving this discussion to logging dev mailing list.)
>
> Have you tried to use:
>
>
> Then each log event will be terminated by end-of-line (\r\n).
>
> I think it would be easy to implement 0-byte terminated log events in
> JsonLayout, and that would make sense since we have implemented support
> >> for
> that in GelfLayout. Created a JIRA issue for it:
> https://issues.apache.org/jira/browse/LOG4J2-1981
>
> As for other tools, the only receivers for Log4j 2 SerializedLayout we
> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
> JsonLayout and XmlLayout as well, and we are changing it to use
> >> JsonLayout
> by default.
>
>
> On 2017-07-17 15:01, Joern Huxhorn wrote:
>
> > Hi Mikael,
> >
> > I've just taken a look at the JsonLayout and see no way to parse its
> > output event by event.
> >
> > It's semi-fine for files because then it will result in a well-formed
> > array at the root level that can be read in one big swoop. It isn't
> > really ideal for that use case either since it means that the whole
> > array containing all events needs to be read into memory in one piece,
> > causing OOM in case of huge files.
> >
> > But in Lilith, I really need to process events one by one. This is
> > absolutely crucial.
> >
> > The only way to achieve this would essentially require me to
> > re-implement a JSON parser that can cope with data like this.
> >
> > I implemented a similar kludge to support reading of log4j xml files
> > (see
> > https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
> > src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
> > ) but this was way easier since i could simply search for
> > "" to find the position at which I could split the stream
> > of events into a valid XML document for every single event . This was
> > still kind of nasty but doable, especially since processing an
> >> "offline"
> > XML file doesn't have the same performance restriction as handling of
> > live events.
> >
> > I wouldn't have the luxury of such a unique split signal in JSON,
> >> though.
> >
> > I would need to detect the "}" at the level of the root array which
> > would not simply involve counting of opening and closing brackets but
> > also ig
Re: Lilith and Log4j 2
Right now that is all handled by the specific layouts. For example, by default the RFC5424Layout doesn’t append newlines so when writing to a file they will all be on the same “line”, but it has an option to append one if needed. Doing the same would be another option for the JSONLayout. I’d be ok with event header and footers but ONLY if they have zero overhead when not present. IOW, when the Layout is initialized it would have to wire in the appropriate encode method. Ralph > On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote: > > Do we want a general event header and footer then? > > Gary > > On Jul 17, 2017 14:43, "Ralph Goers" wrote: > >> No. A Footer is only used at end of file. He needs to know how long each >> event is or when it is the start of a new event. >> >> Ralph >> >>> On Jul 17, 2017, at 12:32 PM, Gary Gregory >> wrote: >>> >>> Can't you use a footer for any terminator you wish? >>> >>> Gary >>> >>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal >> wrote: >>> Hi. (Moving this discussion to logging dev mailing list.) Have you tried to use: Then each log event will be terminated by end-of-line (\r\n). I think it would be easy to implement 0-byte terminated log events in JsonLayout, and that would make sense since we have implemented support >> for that in GelfLayout. Created a JIRA issue for it: https://issues.apache.org/jira/browse/LOG4J2-1981 As for other tools, the only receivers for Log4j 2 SerializedLayout we know are Log4j's own SocketServer and Lilith. Chainsaw currently only support Log4j 1 SerializedLayout. Log4j's own SocketServer support JsonLayout and XmlLayout as well, and we are changing it to use >> JsonLayout by default. On 2017-07-17 15:01, Joern Huxhorn wrote: > Hi Mikael, > > I've just taken a look at the JsonLayout and see no way to parse its > output event by event. > > It's semi-fine for files because then it will result in a well-formed > array at the root level that can be read in one big swoop. It isn't > really ideal for that use case either since it means that the whole > array containing all events needs to be read into memory in one piece, > causing OOM in case of huge files. > > But in Lilith, I really need to process events one by one. This is > absolutely crucial. > > The only way to achieve this would essentially require me to > re-implement a JSON parser that can cope with data like this. > > I implemented a similar kludge to support reading of log4j xml files > (see > https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/ > src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java > ) but this was way easier since i could simply search for > "" to find the position at which I could split the stream > of events into a valid XML document for every single event . This was > still kind of nasty but doable, especially since processing an >> "offline" > XML file doesn't have the same performance restriction as handling of > live events. > > I wouldn't have the luxury of such a unique split signal in JSON, >> though. > > I would need to detect the "}" at the level of the root array which > would not simply involve counting of opening and closing brackets but > also ignoring of brackets inside of strings. This would boil down to > implementing a custom JSON reader and I won't be doing this. > > This isn't just me being lazy either. Such an implementation wouldn't >> be > resource (CPU/memory) friendly anyway. > > In my own JSON receiver using my own JSON format there are two ways to > send those events: > - write an int containing the amount of bytes representing the event, > then the bytes of the event. This type of events also supports > compression. > - write the bytes of the event, followed by a 0-byte. This works fine > and JavaScript/ActionScript is able to produce events like that while > they are unable (or were unable 7 years ago) to count the bytes on >> their > own. This type of events only supports plain text like JSON or XML and > no compression since compressed events could contain the 0-byte while > XML and JSON both won't allow it. > > Both are essentially "binary" formats because of either the raw ints or > the 0-bytes. > > I'm not sure why you deprecate SerializedLayout since the security > issues arise only while deserializing the events, not while serializing > them. Is this just meant to educate the user about the issue? > > I fixed the remote code execution exploit by implementing > https://github.com/huxi/lilith/blob/master/lilith-engine/ > src/main/java/de/huxhorn/lilith/engine/impl/eventproduc > er/WhitelistObjectInputStream.java >>>
Re: Lilith and Log4j 2
Do we want a general event header and footer then? Gary On Jul 17, 2017 14:43, "Ralph Goers" wrote: > No. A Footer is only used at end of file. He needs to know how long each > event is or when it is the start of a new event. > > Ralph > > > On Jul 17, 2017, at 12:32 PM, Gary Gregory > wrote: > > > > Can't you use a footer for any terminator you wish? > > > > Gary > > > > On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal > wrote: > > > >> Hi. > >> > >> (Moving this discussion to logging dev mailing list.) > >> > >> Have you tried to use: > >> > >> > >> Then each log event will be terminated by end-of-line (\r\n). > >> > >> I think it would be easy to implement 0-byte terminated log events in > >> JsonLayout, and that would make sense since we have implemented support > for > >> that in GelfLayout. Created a JIRA issue for it: > >> https://issues.apache.org/jira/browse/LOG4J2-1981 > >> > >> As for other tools, the only receivers for Log4j 2 SerializedLayout we > >> know are Log4j's own SocketServer and Lilith. Chainsaw currently only > >> support Log4j 1 SerializedLayout. Log4j's own SocketServer support > >> JsonLayout and XmlLayout as well, and we are changing it to use > JsonLayout > >> by default. > >> > >> > >> On 2017-07-17 15:01, Joern Huxhorn wrote: > >> > >>> Hi Mikael, > >>> > >>> I've just taken a look at the JsonLayout and see no way to parse its > >>> output event by event. > >>> > >>> It's semi-fine for files because then it will result in a well-formed > >>> array at the root level that can be read in one big swoop. It isn't > >>> really ideal for that use case either since it means that the whole > >>> array containing all events needs to be read into memory in one piece, > >>> causing OOM in case of huge files. > >>> > >>> But in Lilith, I really need to process events one by one. This is > >>> absolutely crucial. > >>> > >>> The only way to achieve this would essentially require me to > >>> re-implement a JSON parser that can cope with data like this. > >>> > >>> I implemented a similar kludge to support reading of log4j xml files > >>> (see > >>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/ > >>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java > >>> ) but this was way easier since i could simply search for > >>> "" to find the position at which I could split the stream > >>> of events into a valid XML document for every single event . This was > >>> still kind of nasty but doable, especially since processing an > "offline" > >>> XML file doesn't have the same performance restriction as handling of > >>> live events. > >>> > >>> I wouldn't have the luxury of such a unique split signal in JSON, > though. > >>> > >>> I would need to detect the "}" at the level of the root array which > >>> would not simply involve counting of opening and closing brackets but > >>> also ignoring of brackets inside of strings. This would boil down to > >>> implementing a custom JSON reader and I won't be doing this. > >>> > >>> This isn't just me being lazy either. Such an implementation wouldn't > be > >>> resource (CPU/memory) friendly anyway. > >>> > >>> In my own JSON receiver using my own JSON format there are two ways to > >>> send those events: > >>> - write an int containing the amount of bytes representing the event, > >>> then the bytes of the event. This type of events also supports > >>> compression. > >>> - write the bytes of the event, followed by a 0-byte. This works fine > >>> and JavaScript/ActionScript is able to produce events like that while > >>> they are unable (or were unable 7 years ago) to count the bytes on > their > >>> own. This type of events only supports plain text like JSON or XML and > >>> no compression since compressed events could contain the 0-byte while > >>> XML and JSON both won't allow it. > >>> > >>> Both are essentially "binary" formats because of either the raw ints or > >>> the 0-bytes. > >>> > >>> I'm not sure why you deprecate SerializedLayout since the security > >>> issues arise only while deserializing the events, not while serializing > >>> them. Is this just meant to educate the user about the issue? > >>> > >>> I fixed the remote code execution exploit by implementing > >>> https://github.com/huxi/lilith/blob/master/lilith-engine/ > >>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc > >>> er/WhitelistObjectInputStream.java > >>> > >>> This still doesn't fix DoS scenarios (endless loops or OOM in case of > >>> malicious data) but is close enough for me, mainly because fixing the > >>> DoS issues would really have to be handled by Java, not by user code. > >>> Manually re-implementing deserialization would likely be fragile and > >>> error prone, possibly even introducing other security issues on its > own. > >>> > >>> I'd be interested how other tools are tackling this issue. Is Chainsaw > >>> officially dead or are they implementing receivers for event streams > >>> like this? > >>> > >>> I'm totally fine with
Re: Lilith and Log4j 2
From comments from Scott Deboy I believe Chainsaw also supports Log4j 2 but not in an official release. Ralph > On Jul 17, 2017, at 12:13 PM, Mikael Ståldal wrote: > > Hi. > > (Moving this discussion to logging dev mailing list.) > > Have you tried to use: > > > Then each log event will be terminated by end-of-line (\r\n). > > I think it would be easy to implement 0-byte terminated log events in > JsonLayout, and that would make sense since we have implemented support for > that in GelfLayout. Created a JIRA issue for it: > https://issues.apache.org/jira/browse/LOG4J2-1981 > > As for other tools, the only receivers for Log4j 2 SerializedLayout we know > are Log4j's own SocketServer and Lilith. Chainsaw currently only support > Log4j 1 SerializedLayout. Log4j's own SocketServer support JsonLayout and > XmlLayout as well, and we are changing it to use JsonLayout by default. > > > On 2017-07-17 15:01, Joern Huxhorn wrote: >> Hi Mikael, >> I've just taken a look at the JsonLayout and see no way to parse its >> output event by event. >> It's semi-fine for files because then it will result in a well-formed >> array at the root level that can be read in one big swoop. It isn't >> really ideal for that use case either since it means that the whole >> array containing all events needs to be read into memory in one piece, >> causing OOM in case of huge files. >> But in Lilith, I really need to process events one by one. This is >> absolutely crucial. >> The only way to achieve this would essentially require me to >> re-implement a JSON parser that can cope with data like this. >> I implemented a similar kludge to support reading of log4j xml files >> (see >> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java >> ) but this was way easier since i could simply search for >> "" to find the position at which I could split the stream >> of events into a valid XML document for every single event . This was >> still kind of nasty but doable, especially since processing an "offline" >> XML file doesn't have the same performance restriction as handling of >> live events. >> I wouldn't have the luxury of such a unique split signal in JSON, though. >> I would need to detect the "}" at the level of the root array which >> would not simply involve counting of opening and closing brackets but >> also ignoring of brackets inside of strings. This would boil down to >> implementing a custom JSON reader and I won't be doing this. >> This isn't just me being lazy either. Such an implementation wouldn't be >> resource (CPU/memory) friendly anyway. >> In my own JSON receiver using my own JSON format there are two ways to >> send those events: >> - write an int containing the amount of bytes representing the event, >> then the bytes of the event. This type of events also supports compression. >> - write the bytes of the event, followed by a 0-byte. This works fine >> and JavaScript/ActionScript is able to produce events like that while >> they are unable (or were unable 7 years ago) to count the bytes on their >> own. This type of events only supports plain text like JSON or XML and >> no compression since compressed events could contain the 0-byte while >> XML and JSON both won't allow it. >> Both are essentially "binary" formats because of either the raw ints or >> the 0-bytes. >> I'm not sure why you deprecate SerializedLayout since the security >> issues arise only while deserializing the events, not while serializing >> them. Is this just meant to educate the user about the issue? >> I fixed the remote code execution exploit by implementing >> https://github.com/huxi/lilith/blob/master/lilith-engine/src/main/java/de/huxhorn/lilith/engine/impl/eventproducer/WhitelistObjectInputStream.java >> This still doesn't fix DoS scenarios (endless loops or OOM in case of >> malicious data) but is close enough for me, mainly because fixing the >> DoS issues would really have to be handled by Java, not by user code. >> Manually re-implementing deserialization would likely be fragile and >> error prone, possibly even introducing other security issues on its own. >> I'd be interested how other tools are tackling this issue. Is Chainsaw >> officially dead or are they implementing receivers for event streams >> like this? >> I'm totally fine with implementing a new receiver as a replacement for >> log4j2 SerializedLayout (even though I won't be able to get rid of the >> Serializable receiver anyway) but the current JsonLayout just isn't up >> to the job at the moment. >> Feel free to share this mail with your fellow developers. I'm open for >> suggestions. >> I hope this doesn't come across in a harsh way. It isn't meant to be. >> Cheers, >> Jörn. > >
Re: Lilith and Log4j 2
No. A Footer is only used at end of file. He needs to know how long each event is or when it is the start of a new event. Ralph > On Jul 17, 2017, at 12:32 PM, Gary Gregory wrote: > > Can't you use a footer for any terminator you wish? > > Gary > > On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal wrote: > >> Hi. >> >> (Moving this discussion to logging dev mailing list.) >> >> Have you tried to use: >> >> >> Then each log event will be terminated by end-of-line (\r\n). >> >> I think it would be easy to implement 0-byte terminated log events in >> JsonLayout, and that would make sense since we have implemented support for >> that in GelfLayout. Created a JIRA issue for it: >> https://issues.apache.org/jira/browse/LOG4J2-1981 >> >> As for other tools, the only receivers for Log4j 2 SerializedLayout we >> know are Log4j's own SocketServer and Lilith. Chainsaw currently only >> support Log4j 1 SerializedLayout. Log4j's own SocketServer support >> JsonLayout and XmlLayout as well, and we are changing it to use JsonLayout >> by default. >> >> >> On 2017-07-17 15:01, Joern Huxhorn wrote: >> >>> Hi Mikael, >>> >>> I've just taken a look at the JsonLayout and see no way to parse its >>> output event by event. >>> >>> It's semi-fine for files because then it will result in a well-formed >>> array at the root level that can be read in one big swoop. It isn't >>> really ideal for that use case either since it means that the whole >>> array containing all events needs to be read into memory in one piece, >>> causing OOM in case of huge files. >>> >>> But in Lilith, I really need to process events one by one. This is >>> absolutely crucial. >>> >>> The only way to achieve this would essentially require me to >>> re-implement a JSON parser that can cope with data like this. >>> >>> I implemented a similar kludge to support reading of log4j xml files >>> (see >>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/ >>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java >>> ) but this was way easier since i could simply search for >>> "" to find the position at which I could split the stream >>> of events into a valid XML document for every single event . This was >>> still kind of nasty but doable, especially since processing an "offline" >>> XML file doesn't have the same performance restriction as handling of >>> live events. >>> >>> I wouldn't have the luxury of such a unique split signal in JSON, though. >>> >>> I would need to detect the "}" at the level of the root array which >>> would not simply involve counting of opening and closing brackets but >>> also ignoring of brackets inside of strings. This would boil down to >>> implementing a custom JSON reader and I won't be doing this. >>> >>> This isn't just me being lazy either. Such an implementation wouldn't be >>> resource (CPU/memory) friendly anyway. >>> >>> In my own JSON receiver using my own JSON format there are two ways to >>> send those events: >>> - write an int containing the amount of bytes representing the event, >>> then the bytes of the event. This type of events also supports >>> compression. >>> - write the bytes of the event, followed by a 0-byte. This works fine >>> and JavaScript/ActionScript is able to produce events like that while >>> they are unable (or were unable 7 years ago) to count the bytes on their >>> own. This type of events only supports plain text like JSON or XML and >>> no compression since compressed events could contain the 0-byte while >>> XML and JSON both won't allow it. >>> >>> Both are essentially "binary" formats because of either the raw ints or >>> the 0-bytes. >>> >>> I'm not sure why you deprecate SerializedLayout since the security >>> issues arise only while deserializing the events, not while serializing >>> them. Is this just meant to educate the user about the issue? >>> >>> I fixed the remote code execution exploit by implementing >>> https://github.com/huxi/lilith/blob/master/lilith-engine/ >>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc >>> er/WhitelistObjectInputStream.java >>> >>> This still doesn't fix DoS scenarios (endless loops or OOM in case of >>> malicious data) but is close enough for me, mainly because fixing the >>> DoS issues would really have to be handled by Java, not by user code. >>> Manually re-implementing deserialization would likely be fragile and >>> error prone, possibly even introducing other security issues on its own. >>> >>> I'd be interested how other tools are tackling this issue. Is Chainsaw >>> officially dead or are they implementing receivers for event streams >>> like this? >>> >>> I'm totally fine with implementing a new receiver as a replacement for >>> log4j2 SerializedLayout (even though I won't be able to get rid of the >>> Serializable receiver anyway) but the current JsonLayout just isn't up >>> to the job at the moment. >>> >>> Feel free to share this mail with your fellow developers. I'm open for
Re: Lilith and Log4j 2
Can't you use a footer for any terminator you wish? Gary On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal wrote: > Hi. > > (Moving this discussion to logging dev mailing list.) > > Have you tried to use: > > > Then each log event will be terminated by end-of-line (\r\n). > > I think it would be easy to implement 0-byte terminated log events in > JsonLayout, and that would make sense since we have implemented support for > that in GelfLayout. Created a JIRA issue for it: > https://issues.apache.org/jira/browse/LOG4J2-1981 > > As for other tools, the only receivers for Log4j 2 SerializedLayout we > know are Log4j's own SocketServer and Lilith. Chainsaw currently only > support Log4j 1 SerializedLayout. Log4j's own SocketServer support > JsonLayout and XmlLayout as well, and we are changing it to use JsonLayout > by default. > > > On 2017-07-17 15:01, Joern Huxhorn wrote: > >> Hi Mikael, >> >> I've just taken a look at the JsonLayout and see no way to parse its >> output event by event. >> >> It's semi-fine for files because then it will result in a well-formed >> array at the root level that can be read in one big swoop. It isn't >> really ideal for that use case either since it means that the whole >> array containing all events needs to be read into memory in one piece, >> causing OOM in case of huge files. >> >> But in Lilith, I really need to process events one by one. This is >> absolutely crucial. >> >> The only way to achieve this would essentially require me to >> re-implement a JSON parser that can cope with data like this. >> >> I implemented a similar kludge to support reading of log4j xml files >> (see >> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/ >> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java >> ) but this was way easier since i could simply search for >> "" to find the position at which I could split the stream >> of events into a valid XML document for every single event . This was >> still kind of nasty but doable, especially since processing an "offline" >> XML file doesn't have the same performance restriction as handling of >> live events. >> >> I wouldn't have the luxury of such a unique split signal in JSON, though. >> >> I would need to detect the "}" at the level of the root array which >> would not simply involve counting of opening and closing brackets but >> also ignoring of brackets inside of strings. This would boil down to >> implementing a custom JSON reader and I won't be doing this. >> >> This isn't just me being lazy either. Such an implementation wouldn't be >> resource (CPU/memory) friendly anyway. >> >> In my own JSON receiver using my own JSON format there are two ways to >> send those events: >> - write an int containing the amount of bytes representing the event, >> then the bytes of the event. This type of events also supports >> compression. >> - write the bytes of the event, followed by a 0-byte. This works fine >> and JavaScript/ActionScript is able to produce events like that while >> they are unable (or were unable 7 years ago) to count the bytes on their >> own. This type of events only supports plain text like JSON or XML and >> no compression since compressed events could contain the 0-byte while >> XML and JSON both won't allow it. >> >> Both are essentially "binary" formats because of either the raw ints or >> the 0-bytes. >> >> I'm not sure why you deprecate SerializedLayout since the security >> issues arise only while deserializing the events, not while serializing >> them. Is this just meant to educate the user about the issue? >> >> I fixed the remote code execution exploit by implementing >> https://github.com/huxi/lilith/blob/master/lilith-engine/ >> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc >> er/WhitelistObjectInputStream.java >> >> This still doesn't fix DoS scenarios (endless loops or OOM in case of >> malicious data) but is close enough for me, mainly because fixing the >> DoS issues would really have to be handled by Java, not by user code. >> Manually re-implementing deserialization would likely be fragile and >> error prone, possibly even introducing other security issues on its own. >> >> I'd be interested how other tools are tackling this issue. Is Chainsaw >> officially dead or are they implementing receivers for event streams >> like this? >> >> I'm totally fine with implementing a new receiver as a replacement for >> log4j2 SerializedLayout (even though I won't be able to get rid of the >> Serializable receiver anyway) but the current JsonLayout just isn't up >> to the job at the moment. >> >> Feel free to share this mail with your fellow developers. I'm open for >> suggestions. >> >> I hope this doesn't come across in a harsh way. It isn't meant to be. >> >> Cheers, >> Jörn. >> > >
Re: Lilith and Log4j 2
Hi. (Moving this discussion to logging dev mailing list.) Have you tried to use: Then each log event will be terminated by end-of-line (\r\n). I think it would be easy to implement 0-byte terminated log events in JsonLayout, and that would make sense since we have implemented support for that in GelfLayout. Created a JIRA issue for it: https://issues.apache.org/jira/browse/LOG4J2-1981 As for other tools, the only receivers for Log4j 2 SerializedLayout we know are Log4j's own SocketServer and Lilith. Chainsaw currently only support Log4j 1 SerializedLayout. Log4j's own SocketServer support JsonLayout and XmlLayout as well, and we are changing it to use JsonLayout by default. On 2017-07-17 15:01, Joern Huxhorn wrote: Hi Mikael, I've just taken a look at the JsonLayout and see no way to parse its output event by event. It's semi-fine for files because then it will result in a well-formed array at the root level that can be read in one big swoop. It isn't really ideal for that use case either since it means that the whole array containing all events needs to be read into memory in one piece, causing OOM in case of huge files. But in Lilith, I really need to process events one by one. This is absolutely crucial. The only way to achieve this would essentially require me to re-implement a JSON parser that can cope with data like this. I implemented a similar kludge to support reading of log4j xml files (see https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java ) but this was way easier since i could simply search for "" to find the position at which I could split the stream of events into a valid XML document for every single event . This was still kind of nasty but doable, especially since processing an "offline" XML file doesn't have the same performance restriction as handling of live events. I wouldn't have the luxury of such a unique split signal in JSON, though. I would need to detect the "}" at the level of the root array which would not simply involve counting of opening and closing brackets but also ignoring of brackets inside of strings. This would boil down to implementing a custom JSON reader and I won't be doing this. This isn't just me being lazy either. Such an implementation wouldn't be resource (CPU/memory) friendly anyway. In my own JSON receiver using my own JSON format there are two ways to send those events: - write an int containing the amount of bytes representing the event, then the bytes of the event. This type of events also supports compression. - write the bytes of the event, followed by a 0-byte. This works fine and JavaScript/ActionScript is able to produce events like that while they are unable (or were unable 7 years ago) to count the bytes on their own. This type of events only supports plain text like JSON or XML and no compression since compressed events could contain the 0-byte while XML and JSON both won't allow it. Both are essentially "binary" formats because of either the raw ints or the 0-bytes. I'm not sure why you deprecate SerializedLayout since the security issues arise only while deserializing the events, not while serializing them. Is this just meant to educate the user about the issue? I fixed the remote code execution exploit by implementing https://github.com/huxi/lilith/blob/master/lilith-engine/src/main/java/de/huxhorn/lilith/engine/impl/eventproducer/WhitelistObjectInputStream.java This still doesn't fix DoS scenarios (endless loops or OOM in case of malicious data) but is close enough for me, mainly because fixing the DoS issues would really have to be handled by Java, not by user code. Manually re-implementing deserialization would likely be fragile and error prone, possibly even introducing other security issues on its own. I'd be interested how other tools are tackling this issue. Is Chainsaw officially dead or are they implementing receivers for event streams like this? I'm totally fine with implementing a new receiver as a replacement for log4j2 SerializedLayout (even though I won't be able to get rid of the Serializable receiver anyway) but the current JsonLayout just isn't up to the job at the moment. Feel free to share this mail with your fellow developers. I'm open for suggestions. I hope this doesn't come across in a harsh way. It isn't meant to be. Cheers, Jörn.
