BSON can be used anywhere, it's a partially compressed JSON format. Right now, I'm using CSV for large local logs I can later process or import in a database.
To save space, I'd like a format that is compressed but can also be processed as is, BSON might help. Yes, I know I can rollover and compress ;-) Gary On Tue, Jan 12, 2016 at 11:26 AM, Matt Sicker <[email protected]> wrote: > Isn't BSON only used by Mongo? > > On 12 January 2016 at 12:51, Gary Gregory <[email protected]> wrote: > >> And I'm wondering if we should have a BSON layout, that would also be >> with Jackson. >> >> Gayr >> >> On Tue, Jan 12, 2016 at 10:51 AM, Gary Gregory <[email protected]> >> wrote: >> >>> Keep in mind that we use Jackson so almost all the code is the same for >>> JSON and XML. >>> >>> Gary >>> >>> On Tue, Jan 12, 2016 at 8:34 AM, Robin Coe <[email protected]> >>> wrote: >>> >>>> I would be happy to test but haven't looked at the writer >>>> implementations, so speedy would not be the effort. But, it's a >>>> performance issue only...maybe...so no hurry anyway. >>>> >>>> On Tue, Jan 12, 2016 at 11:30 AM, Gary Gregory <[email protected]> >>>> wrote: >>>> >>>>> If the reason to do this is performance, it should be proved by a >>>>> benchmark... >>>>> >>>>> Gary >>>>> On Jan 12, 2016 8:28 AM, "Gary Gregory" <[email protected]> >>>>> wrote: >>>>> >>>>>> Maybe something like mdcStyle="this/that/future"? >>>>>> >>>>>> This would also apply to the XML layout? >>>>>> >>>>>> Gary >>>>>> On Jan 12, 2016 8:26 AM, "Mikael Ståldal" <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Ah yes, that's possible. It would be nice. >>>>>>> >>>>>>> On Tue, Jan 12, 2016 at 5:24 PM, Ralph Goers < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> If there is a flag that causes the new structure to be generated >>>>>>>> then he would get the performance gain when it is enabled. The current >>>>>>>> structure would be generated when the flag is not set. >>>>>>>> >>>>>>>> Ralph >>>>>>>> >>>>>>>> On Jan 12, 2016, at 9:14 AM, Mikael Ståldal < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>> But I guess that you won't get any performance gain if we keep the >>>>>>>> old structure besides the new one, since then both will be parsed. >>>>>>>> >>>>>>>> On Tue, Jan 12, 2016 at 3:15 PM, Robin Coe <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I agree that if it were changed there may be some compatibility >>>>>>>>> issues. But, if it's doable, then introducing a new property could >>>>>>>>> bridge >>>>>>>>> the change. Not saying it's doable, because I haven't looked, but a >>>>>>>>> new >>>>>>>>> property and a deprecation warning (in docs, I expect) would allow the >>>>>>>>> change to happen. Very preliminary data showed me that parsing 1000 >>>>>>>>> events >>>>>>>>> slowed my parser from < 500 ms (w/o contextMap) to 2000 ms when each >>>>>>>>> event >>>>>>>>> contained 2 contextMap entries, requiring the list of maps to be >>>>>>>>> converted >>>>>>>>> to a single map. Not sure what the time would be to parse a >>>>>>>>> multi-valued >>>>>>>>> map, though, so I can't be sure of the overhead of walking the list >>>>>>>>> wrapper. >>>>>>>>> >>>>>>>>> On Tue, Jan 12, 2016 at 6:05 AM, Mikael Ståldal < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> I think that the current JSONLayout format is unfortunate, and I >>>>>>>>>> would prefer to have it as you propose. But we cannot change it now >>>>>>>>>> since >>>>>>>>>> that will break backwards compatibility. >>>>>>>>>> >>>>>>>>>> See: https://issues.apache.org/jira/browse/LOG4J2-623 >>>>>>>>>> >>>>>>>>>> Perhaps GELFLayout would work better for you. >>>>>>>>>> >>>>>>>>>> On Mon, Jan 4, 2016 at 10:00 PM, Gary Gregory < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> The point I was trying to make is that you cannot describe what >>>>>>>>>>> you are asking for with a generic XML schema, not sure about JSON >>>>>>>>>>> schema, >>>>>>>>>>> but the idea is the same. Since we use Jackson, that also means we >>>>>>>>>>> use the >>>>>>>>>>> same code to emit JSON and XML. >>>>>>>>>>> >>>>>>>>>>> Gary >>>>>>>>>>> On Jan 4, 2016 12:25 PM, "Robin Coe" <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I can see that XML entities requires conforming to a schema but >>>>>>>>>>>> isn't the writer implementation capable of wrapping the map >>>>>>>>>>>> entries when >>>>>>>>>>>> required? Seems like it's making the JSON representation more >>>>>>>>>>>> complex (and >>>>>>>>>>>> less performant) at the cost of some wrapper code for the xml >>>>>>>>>>>> writer. >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jan 4, 2016 at 3:19 PM, Gary Gregory < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yes, that is because we can define this kind of structure with >>>>>>>>>>>>> XML/JSON schema with ease. >>>>>>>>>>>>> >>>>>>>>>>>>> Gary >>>>>>>>>>>>> On Jan 4, 2016 11:55 AM, "Robin Coe" <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I was trying to deserialize a log event written by the >>>>>>>>>>>>>> JSONLayout appender, which uses Jackson. I therefore also am >>>>>>>>>>>>>> using Jackson >>>>>>>>>>>>>> but with the MrBeanModule, which is a POJO materializer. After >>>>>>>>>>>>>> much >>>>>>>>>>>>>> difficulty with Jackson throwing deserialization exceptions with >>>>>>>>>>>>>> the >>>>>>>>>>>>>> "contextMap" field, I learned that the map is actually written >>>>>>>>>>>>>> out as a >>>>>>>>>>>>>> List of Maps (i.e. List<Map<String,String>>. I've included one >>>>>>>>>>>>>> such event >>>>>>>>>>>>>> here, with unnecessary fields shortened: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> {"timeMillis":...,"thread":"...","level":"OFF","loggerName":"...","message":"...","endOfBatch":false,"loggerFqcn":"...","contextMap":[{"key":"LOGROLL","value":"com.xxx.xxx.handler.event.FailoverHandler"},{"key":"ROUTINGKEY","value":"elasticsearch-rollover"}]} >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm curious why the contextMap is represented as the more >>>>>>>>>>>>>> complex List of single entry Maps, as opposed to a single >>>>>>>>>>>>>> multi-valued >>>>>>>>>>>>>> Map? So, instead of something that looks like: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> {"contextMap":[{"key":"key1"},{"value":"value1"},{"key":"key2"},{"value":"value2"},...] >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would expect the much simpler (and easily parseable): >>>>>>>>>>>>>> {"contextMap":{"key1":"value1","key2":"value2",...}. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is this intended? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Robin. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> [image: MagineTV] >>>>>>>>>> >>>>>>>>>> *Mikael Ståldal* >>>>>>>>>> Senior software developer >>>>>>>>>> >>>>>>>>>> *Magine TV* >>>>>>>>>> [email protected] >>>>>>>>>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com >>>>>>>>>> >>>>>>>>>> Privileged and/or Confidential Information may be contained in >>>>>>>>>> this message. If you are not the addressee indicated in this message >>>>>>>>>> (or responsible for delivery of the message to such a person), >>>>>>>>>> you may not copy or deliver this message to anyone. In such case, >>>>>>>>>> you should destroy this message and kindly notify the sender by >>>>>>>>>> reply email. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> [image: MagineTV] >>>>>>>> >>>>>>>> *Mikael Ståldal* >>>>>>>> Senior software developer >>>>>>>> >>>>>>>> *Magine TV* >>>>>>>> [email protected] >>>>>>>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com >>>>>>>> >>>>>>>> Privileged and/or Confidential Information may be contained in this >>>>>>>> message. If you are not the addressee indicated in this message >>>>>>>> (or responsible for delivery of the message to such a person), you >>>>>>>> may not copy or deliver this message to anyone. In such case, >>>>>>>> you should destroy this message and kindly notify the sender by >>>>>>>> reply email. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> [image: MagineTV] >>>>>>> >>>>>>> *Mikael Ståldal* >>>>>>> Senior software developer >>>>>>> >>>>>>> *Magine TV* >>>>>>> [email protected] >>>>>>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com >>>>>>> >>>>>>> Privileged and/or Confidential Information may be contained in this >>>>>>> message. If you are not the addressee indicated in this message >>>>>>> (or responsible for delivery of the message to such a person), you >>>>>>> may not copy or deliver this message to anyone. In such case, >>>>>>> you should destroy this message and kindly notify the sender by >>>>>>> reply email. >>>>>>> >>>>>> >>>> >>> >>> >>> -- >>> E-Mail: [email protected] | [email protected] >>> Java Persistence with Hibernate, Second Edition >>> <http://www.manning.com/bauer3/> >>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> >>> Spring Batch in Action <http://www.manning.com/templier/> >>> Blog: http://garygregory.wordpress.com >>> Home: http://garygregory.com/ >>> Tweet! http://twitter.com/GaryGregory >>> >> >> >> >> -- >> E-Mail: [email protected] | [email protected] >> Java Persistence with Hibernate, Second Edition >> <http://www.manning.com/bauer3/> >> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> >> Spring Batch in Action <http://www.manning.com/templier/> >> Blog: http://garygregory.wordpress.com >> Home: http://garygregory.com/ >> Tweet! http://twitter.com/GaryGregory >> > > > > -- > Matt Sicker <[email protected]> > -- E-Mail: [email protected] | [email protected] Java Persistence with Hibernate, Second Edition <http://www.manning.com/bauer3/> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> Spring Batch in Action <http://www.manning.com/templier/> Blog: http://garygregory.wordpress.com Home: http://garygregory.com/ Tweet! http://twitter.com/GaryGregory
