Re: Make LogEvent implementations Externalizable
Ralph is an API designer genius. :) On 28 July 2014 16:22, Scott Harrington wrote: > Just to follow up, > > There was no need for an enhancement request. I was able to implement my > own 'PackedLayout' and 'PackedLogEventBridge' that serialize my particular > LogEvents efficiently to/from byte array in exactly the way I wanted (by > making some assumptions not suitable for general-purpose use). > > Thanks for the advice and for the well-designed interfaces! > > Scott > > > > On Fri, 11 Jul 2014, Ralph Goers wrote: > > Sounds good. >> >> This approach allows for many different serialization formats. It might >> be nice to have the CompactBinaryLayout even be “pluggable” to support >> things like Hessian, Protobuf, Thrift, etc. Of course, that would also >> require the LogEventBridge to have support as well. >> >> Ralph >> >> On Jul 11, 2014, at 10:14 AM, Scott Harrington >> wrote: >> >> OK! Agree. So the 2.1 enhancement request will be for a >>> CompactBinaryLayout on the sending side and corresponding LogEventBridge on >>> the receiving side. I'll open it in JIRA when I get back from vacation and >>> have a chance to work through an initial implementation, unless someone >>> else beats me to it. >>> >>> Current 'Serializable' form is a bit chatty but is easier (automatic) to >>> maintain. Forget I ever said anything about Externalizable. >>> >> > > - > To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org > For additional commands, e-mail: log4j-user-h...@logging.apache.org > -- Matt Sicker
Re: Make LogEvent implementations Externalizable
Just to follow up, There was no need for an enhancement request. I was able to implement my own 'PackedLayout' and 'PackedLogEventBridge' that serialize my particular LogEvents efficiently to/from byte array in exactly the way I wanted (by making some assumptions not suitable for general-purpose use). Thanks for the advice and for the well-designed interfaces! Scott On Fri, 11 Jul 2014, Ralph Goers wrote: Sounds good. This approach allows for many different serialization formats. It might be nice to have the CompactBinaryLayout even be “pluggable” to support things like Hessian, Protobuf, Thrift, etc. Of course, that would also require the LogEventBridge to have support as well. Ralph On Jul 11, 2014, at 10:14 AM, Scott Harrington wrote: OK! Agree. So the 2.1 enhancement request will be for a CompactBinaryLayout on the sending side and corresponding LogEventBridge on the receiving side. I'll open it in JIRA when I get back from vacation and have a chance to work through an initial implementation, unless someone else beats me to it. Current 'Serializable' form is a bit chatty but is easier (automatic) to maintain. Forget I ever said anything about Externalizable. - To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org For additional commands, e-mail: log4j-user-h...@logging.apache.org
Re: Make LogEvent implementations Externalizable
Sounds good.
This approach allows for many different serialization formats. It might be
nice to have the CompactBinaryLayout even be “pluggable” to support things like
Hessian, Protobuf, Thrift, etc. Of course, that would also require the
LogEventBridge to have support as well.
Ralph
On Jul 11, 2014, at 10:14 AM, Scott Harrington wrote:
> OK! Agree. So the 2.1 enhancement request will be for a CompactBinaryLayout
> on the sending side and corresponding LogEventBridge on the receiving side.
> I'll open it in JIRA when I get back from vacation and have a chance to work
> through an initial implementation, unless someone else beats me to it.
>
> Current 'Serializable' form is a bit chatty but is easier (automatic) to
> maintain. Forget I ever said anything about Externalizable.
>
>
> On Sat, 12 Jul 2014, Remko Popma wrote:
>
>> Yes that is what I had in mind.
>>
>>
>> On Sat, Jul 12, 2014 at 1:53 AM, Scott Harrington
>> wrote:
>>
>>> Yes it did appear toByteArray would do the job on the /sending/ side but
>>> what about the /receiving/ side? We've got to put some bytes on the wire
>>> that will come out as a LogEvent from the readObject() call in
>>> ObjectInputStreamLogEventBridge. You can't have the sender be
>>> Externalizable and the received class be only Serializable.
>>>
>>> Unless you're proposing we make a totally new LogEventBridge -- i.e. a
>>> 'compact binary' protocol to go alongside the XML and Json versions. If we
>>> go that route then you could achieve even more compression since we know
>>> every object that we're sending is a LogEvent and even Externalizable has
>>> some overhead that we could do away with.
>>>
>>>
>>>
>>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>>
>>> On Sat, Jul 12, 2014 at 12:52 AM, Remko Popma
>>>> wrote:
>>>>
>>>> What prevents you from writing a more compact representation of the
>>>>> LogEvent (including all other LogEvent fields) to the byte array in the
>>>>> {{toByteArray(LogEvent)}} method of an ExternalizableLayout?
>>>>>
>>>>> (Just to clarify, I intended this question literally as it could easily
>>>> be
>>>> that I overlooked something...)
>>>>
>>>>
>>>>
>>>>>
>>>>> On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington >>>>>
>>>>> wrote:
>>>>>
>>>>> I looked at replacing SerializedLayout and/or the toSerializable method
>>>>>> but the real savings comes not from the Message itself but from all the
>>>>>> other fields of the LogEvent, such as the Level, ContextMap,
>>>>>> ContextStack,
>>>>>> and the class descriptor (including superclasses) of the LogEvent
>>>>>> itself.
>>>>>>
>>>>>> For big messages, an ExternalizableLayout would be a fine solution,
>>>>>> because writeUTF would give you roughly 50% compression over Java's
>>>>>> default
>>>>>> 2-byte per char String serialization.
>>>>>>
>>>>>> However for typical log messages the message itself is being dwarfed on
>>>>>> the wire by class descriptor overhead.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>>>>>
>>>>>> That is a nice reduction in size!
>>>>>>
>>>>>>>
>>>>>>> I also think the ExternalizedLayout idea is a very attractive option.
>>>>>>> That
>>>>>>> way there is no pressure to include this in any particular release, we
>>>>>>> can
>>>>>>> release it when we are confident that is ready. I also like the fact
>>>>>>> that
>>>>>>> it does not replace the current serialization and it can be switched on
>>>>>>> and
>>>>>>> off in configuration so if there is a bug, users can fall back to the
>>>>>>> existing plain serialization.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker
>>>>>>> wrote:
>>>>>>>
>>>>>>> I would second the ExternalizedLayout. Layouts are the way t
Re: Make LogEvent implementations Externalizable
It sounds good to go with an initial Serializable format that is easy to
maintain and provide an additional format later that is optimized.
Gary
On Fri, Jul 11, 2014 at 1:14 PM, Scott Harrington
wrote:
> OK! Agree. So the 2.1 enhancement request will be for a
> CompactBinaryLayout on the sending side and corresponding LogEventBridge on
> the receiving side. I'll open it in JIRA when I get back from vacation and
> have a chance to work through an initial implementation, unless someone
> else beats me to it.
>
> Current 'Serializable' form is a bit chatty but is easier (automatic) to
> maintain. Forget I ever said anything about Externalizable.
>
>
>
> On Sat, 12 Jul 2014, Remko Popma wrote:
>
> Yes that is what I had in mind.
>>
>>
>> On Sat, Jul 12, 2014 at 1:53 AM, Scott Harrington
>> wrote:
>>
>> Yes it did appear toByteArray would do the job on the /sending/ side but
>>> what about the /receiving/ side? We've got to put some bytes on the wire
>>> that will come out as a LogEvent from the readObject() call in
>>> ObjectInputStreamLogEventBridge. You can't have the sender be
>>> Externalizable and the received class be only Serializable.
>>>
>>> Unless you're proposing we make a totally new LogEventBridge -- i.e. a
>>> 'compact binary' protocol to go alongside the XML and Json versions. If
>>> we
>>> go that route then you could achieve even more compression since we know
>>> every object that we're sending is a LogEvent and even Externalizable has
>>> some overhead that we could do away with.
>>>
>>>
>>>
>>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>>
>>> On Sat, Jul 12, 2014 at 12:52 AM, Remko Popma
>>>
>>>> wrote:
>>>>
>>>> What prevents you from writing a more compact representation of the
>>>>
>>>>> LogEvent (including all other LogEvent fields) to the byte array in the
>>>>> {{toByteArray(LogEvent)}} method of an ExternalizableLayout?
>>>>>
>>>>> (Just to clarify, I intended this question literally as it could
>>>>> easily
>>>>>
>>>> be
>>>> that I overlooked something...)
>>>>
>>>>
>>>>
>>>>
>>>>> On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington <
>>>>> scott...@sns-usa.com
>>>>>
>>>>>>
>>>>>> wrote:
>>>>>
>>>>> I looked at replacing SerializedLayout and/or the toSerializable
>>>>> method
>>>>>
>>>>>> but the real savings comes not from the Message itself but from all
>>>>>> the
>>>>>> other fields of the LogEvent, such as the Level, ContextMap,
>>>>>> ContextStack,
>>>>>> and the class descriptor (including superclasses) of the LogEvent
>>>>>> itself.
>>>>>>
>>>>>> For big messages, an ExternalizableLayout would be a fine solution,
>>>>>> because writeUTF would give you roughly 50% compression over Java's
>>>>>> default
>>>>>> 2-byte per char String serialization.
>>>>>>
>>>>>> However for typical log messages the message itself is being dwarfed
>>>>>> on
>>>>>> the wire by class descriptor overhead.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>>>>>
>>>>>> That is a nice reduction in size!
>>>>>>
>>>>>>
>>>>>>> I also think the ExternalizedLayout idea is a very attractive option.
>>>>>>> That
>>>>>>> way there is no pressure to include this in any particular release,
>>>>>>> we
>>>>>>> can
>>>>>>> release it when we are confident that is ready. I also like the fact
>>>>>>> that
>>>>>>> it does not replace the current serialization and it can be switched
>>>>>>> on
>>>>>>> and
>>>>>>> off in configuration so if there is a bug, users can fall back to the
>>>>>>> existing plain serialization.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker
>>>>>&
Re: Make LogEvent implementations Externalizable
OK! Agree. So the 2.1 enhancement request will be for a
CompactBinaryLayout on the sending side and corresponding LogEventBridge
on the receiving side. I'll open it in JIRA when I get back from vacation
and have a chance to work through an initial implementation, unless
someone else beats me to it.
Current 'Serializable' form is a bit chatty but is easier (automatic) to
maintain. Forget I ever said anything about Externalizable.
On Sat, 12 Jul 2014, Remko Popma wrote:
Yes that is what I had in mind.
On Sat, Jul 12, 2014 at 1:53 AM, Scott Harrington
wrote:
Yes it did appear toByteArray would do the job on the /sending/ side but
what about the /receiving/ side? We've got to put some bytes on the wire
that will come out as a LogEvent from the readObject() call in
ObjectInputStreamLogEventBridge. You can't have the sender be
Externalizable and the received class be only Serializable.
Unless you're proposing we make a totally new LogEventBridge -- i.e. a
'compact binary' protocol to go alongside the XML and Json versions. If we
go that route then you could achieve even more compression since we know
every object that we're sending is a LogEvent and even Externalizable has
some overhead that we could do away with.
On Sat, 12 Jul 2014, Remko Popma wrote:
On Sat, Jul 12, 2014 at 12:52 AM, Remko Popma
wrote:
What prevents you from writing a more compact representation of the
LogEvent (including all other LogEvent fields) to the byte array in the
{{toByteArray(LogEvent)}} method of an ExternalizableLayout?
(Just to clarify, I intended this question literally as it could easily
be
that I overlooked something...)
On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington
wrote:
I looked at replacing SerializedLayout and/or the toSerializable method
but the real savings comes not from the Message itself but from all the
other fields of the LogEvent, such as the Level, ContextMap,
ContextStack,
and the class descriptor (including superclasses) of the LogEvent
itself.
For big messages, an ExternalizableLayout would be a fine solution,
because writeUTF would give you roughly 50% compression over Java's
default
2-byte per char String serialization.
However for typical log messages the message itself is being dwarfed on
the wire by class descriptor overhead.
On Sat, 12 Jul 2014, Remko Popma wrote:
That is a nice reduction in size!
I also think the ExternalizedLayout idea is a very attractive option.
That
way there is no pressure to include this in any particular release, we
can
release it when we are confident that is ready. I also like the fact
that
it does not replace the current serialization and it can be switched on
and
off in configuration so if there is a bug, users can fall back to the
existing plain serialization.
On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker
wrote:
I would second the ExternalizedLayout. Layouts are the way to go for
compatibility and are simpler. Particularly useful for alternative
serialization protocols, too.
On 11 July 2014 06:41, Gary Gregory wrote:
I understand Ralph ' s concern but now is the time for this kind of
change. Otherwise we will need even more clever solutions to get
this
kind
of size improvement. I'd love to see some performance numbers. The
size
improvement is not negligible, which is great!
Gary
Original message From: Ralph Goers <
ralph.go...@dslextreme.com> Date:07/11/2014 01:46
(GMT-05:00) To: Log4J Users List <
log4j-user@logging.apache.org> Subject: Re: Make LogEvent
implementations Externalizable
I’d be afraid of breaking compatibility even now. However, I
think
what you really want to do is to create an ExternalizedLayout and
then
just
use that instead of the default SerializedLayout. If you want to
supply
that Layout as a patch to a Jira issue it could be added at any time.
Ralph
On Jul 10, 2014, at 9:23 PM, Scott Harrington
wrote:
Ralph & co:
I hear you're gearing up for the release.
Last weekend I scratched an itch of mine relating to SocketAppender
->
SocketServer bandwidth, and was able to reduce a 500-character
message
from
around 1700 bytes to 700 bytes on the wire (it's easy to improve on
Java's
default serialization).
I was going to submit an enhancement request with patch to JIRA but
instead I went on vacation for two weeks.
I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
writeExternal / readExternal methods. I did NOT have time to make
an
equivalent change to Log4jLogEvent, or to write up any performance
tests
or
regression tests.
Should I submit what I have for discussion and hopeful inclusion in
2.0?
Or will it have to wait for 2.1?
If we wait, then due to the necessary serialVersionUID change, v2.0
SocketAppender would not be able to talk to v2.1 SocketServer or
vice
versa
(unles
Re: Make LogEvent implementations Externalizable
Yes that is what I had in mind.
On Sat, Jul 12, 2014 at 1:53 AM, Scott Harrington
wrote:
> Yes it did appear toByteArray would do the job on the /sending/ side but
> what about the /receiving/ side? We've got to put some bytes on the wire
> that will come out as a LogEvent from the readObject() call in
> ObjectInputStreamLogEventBridge. You can't have the sender be
> Externalizable and the received class be only Serializable.
>
> Unless you're proposing we make a totally new LogEventBridge -- i.e. a
> 'compact binary' protocol to go alongside the XML and Json versions. If we
> go that route then you could achieve even more compression since we know
> every object that we're sending is a LogEvent and even Externalizable has
> some overhead that we could do away with.
>
>
>
> On Sat, 12 Jul 2014, Remko Popma wrote:
>
> On Sat, Jul 12, 2014 at 12:52 AM, Remko Popma
>> wrote:
>>
>> What prevents you from writing a more compact representation of the
>>> LogEvent (including all other LogEvent fields) to the byte array in the
>>> {{toByteArray(LogEvent)}} method of an ExternalizableLayout?
>>>
>>> (Just to clarify, I intended this question literally as it could easily
>> be
>> that I overlooked something...)
>>
>>
>>
>>>
>>> On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington >> >
>>> wrote:
>>>
>>> I looked at replacing SerializedLayout and/or the toSerializable method
>>>> but the real savings comes not from the Message itself but from all the
>>>> other fields of the LogEvent, such as the Level, ContextMap,
>>>> ContextStack,
>>>> and the class descriptor (including superclasses) of the LogEvent
>>>> itself.
>>>>
>>>> For big messages, an ExternalizableLayout would be a fine solution,
>>>> because writeUTF would give you roughly 50% compression over Java's
>>>> default
>>>> 2-byte per char String serialization.
>>>>
>>>> However for typical log messages the message itself is being dwarfed on
>>>> the wire by class descriptor overhead.
>>>>
>>>>
>>>>
>>>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>>>
>>>> That is a nice reduction in size!
>>>>
>>>>>
>>>>> I also think the ExternalizedLayout idea is a very attractive option.
>>>>> That
>>>>> way there is no pressure to include this in any particular release, we
>>>>> can
>>>>> release it when we are confident that is ready. I also like the fact
>>>>> that
>>>>> it does not replace the current serialization and it can be switched on
>>>>> and
>>>>> off in configuration so if there is a bug, users can fall back to the
>>>>> existing plain serialization.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker
>>>>> wrote:
>>>>>
>>>>> I would second the ExternalizedLayout. Layouts are the way to go for
>>>>>
>>>>>> compatibility and are simpler. Particularly useful for alternative
>>>>>> serialization protocols, too.
>>>>>>
>>>>>>
>>>>>> On 11 July 2014 06:41, Gary Gregory wrote:
>>>>>>
>>>>>> I understand Ralph ' s concern but now is the time for this kind of
>>>>>>
>>>>>>> change. Otherwise we will need even more clever solutions to get
>>>>>>> this
>>>>>>>
>>>>>>> kind
>>>>>>
>>>>>> of size improvement. I'd love to see some performance numbers. The
>>>>>>> size
>>>>>>> improvement is not negligible, which is great!
>>>>>>>
>>>>>>> Gary
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Original message From: Ralph Goers <
>>>>>>> ralph.go...@dslextreme.com> Date:07/11/2014 01:46
>>>>>>> (GMT-05:00) To: Log4J Users List <
>>>>>>> log4j-user@logging.apache.org> Subject: Re: Make LogEvent
>>>>>>> implementations Externalizable
>>>>>>> I’d be afraid of breaking compatibility even now. However, I
>>>>>>> thin
Re: Make LogEvent implementations Externalizable
Yes it did appear toByteArray would do the job on the /sending/ side but
what about the /receiving/ side? We've got to put some bytes on the wire
that will come out as a LogEvent from the readObject() call in
ObjectInputStreamLogEventBridge. You can't have the sender be
Externalizable and the received class be only Serializable.
Unless you're proposing we make a totally new LogEventBridge -- i.e. a
'compact binary' protocol to go alongside the XML and Json versions. If we
go that route then you could achieve even more compression since we know
every object that we're sending is a LogEvent and even Externalizable has
some overhead that we could do away with.
On Sat, 12 Jul 2014, Remko Popma wrote:
On Sat, Jul 12, 2014 at 12:52 AM, Remko Popma wrote:
What prevents you from writing a more compact representation of the
LogEvent (including all other LogEvent fields) to the byte array in the
{{toByteArray(LogEvent)}} method of an ExternalizableLayout?
(Just to clarify, I intended this question literally as it could easily be
that I overlooked something...)
On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington
wrote:
I looked at replacing SerializedLayout and/or the toSerializable method
but the real savings comes not from the Message itself but from all the
other fields of the LogEvent, such as the Level, ContextMap, ContextStack,
and the class descriptor (including superclasses) of the LogEvent itself.
For big messages, an ExternalizableLayout would be a fine solution,
because writeUTF would give you roughly 50% compression over Java's default
2-byte per char String serialization.
However for typical log messages the message itself is being dwarfed on
the wire by class descriptor overhead.
On Sat, 12 Jul 2014, Remko Popma wrote:
That is a nice reduction in size!
I also think the ExternalizedLayout idea is a very attractive option.
That
way there is no pressure to include this in any particular release, we
can
release it when we are confident that is ready. I also like the fact that
it does not replace the current serialization and it can be switched on
and
off in configuration so if there is a bug, users can fall back to the
existing plain serialization.
On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker wrote:
I would second the ExternalizedLayout. Layouts are the way to go for
compatibility and are simpler. Particularly useful for alternative
serialization protocols, too.
On 11 July 2014 06:41, Gary Gregory wrote:
I understand Ralph ' s concern but now is the time for this kind of
change. Otherwise we will need even more clever solutions to get this
kind
of size improvement. I'd love to see some performance numbers. The size
improvement is not negligible, which is great!
Gary
Original message From: Ralph Goers <
ralph.go...@dslextreme.com> Date:07/11/2014 01:46
(GMT-05:00) To: Log4J Users List <
log4j-user@logging.apache.org> Subject: Re: Make LogEvent
implementations Externalizable
I’d be afraid of breaking compatibility even now. However, I
think
what you really want to do is to create an ExternalizedLayout and then
just
use that instead of the default SerializedLayout. If you want to
supply
that Layout as a patch to a Jira issue it could be added at any time.
Ralph
On Jul 10, 2014, at 9:23 PM, Scott Harrington
wrote:
Ralph & co:
I hear you're gearing up for the release.
Last weekend I scratched an itch of mine relating to SocketAppender ->
SocketServer bandwidth, and was able to reduce a 500-character message
from
around 1700 bytes to 700 bytes on the wire (it's easy to improve on
Java's
default serialization).
I was going to submit an enhancement request with patch to JIRA but
instead I went on vacation for two weeks.
I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
writeExternal / readExternal methods. I did NOT have time to make an
equivalent change to Log4jLogEvent, or to write up any performance
tests
or
regression tests.
Should I submit what I have for discussion and hopeful inclusion in
2.0?
Or will it have to wait for 2.1?
If we wait, then due to the necessary serialVersionUID change, v2.0
SocketAppender would not be able to talk to v2.1 SocketServer or vice
versa
(unless ugly duplicate versions are maintained).
Below is what the added code looks like. I only tested in
RingBufferLogEvent but should be similarly usable in Log4jLogEvent, and
perhaps we should discuss a RingBufferLogEvent.readResolve that makes
them
all become Log4jLogEvents on the SocketServer (receiving) end.
...
public void writeExternal(ObjectOutput out) throws IOException {
getThrownProxy();
out.writeByte(1); // wireFormat
int presenceMap = (loggerName == null ? 0 : 0x1) |
(marker == null ? 0 : 0x2) |
(fqcn == null ? 0 : 0x4) |
(level == n
Re: Make LogEvent implementations Externalizable
On Sat, Jul 12, 2014 at 12:52 AM, Remko Popma wrote:
> What prevents you from writing a more compact representation of the
> LogEvent (including all other LogEvent fields) to the byte array in the
> {{toByteArray(LogEvent)}} method of an ExternalizableLayout?
>
(Just to clarify, I intended this question literally as it could easily be
that I overlooked something...)
>
>
> On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington
> wrote:
>
>> I looked at replacing SerializedLayout and/or the toSerializable method
>> but the real savings comes not from the Message itself but from all the
>> other fields of the LogEvent, such as the Level, ContextMap, ContextStack,
>> and the class descriptor (including superclasses) of the LogEvent itself.
>>
>> For big messages, an ExternalizableLayout would be a fine solution,
>> because writeUTF would give you roughly 50% compression over Java's default
>> 2-byte per char String serialization.
>>
>> However for typical log messages the message itself is being dwarfed on
>> the wire by class descriptor overhead.
>>
>>
>>
>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>
>> That is a nice reduction in size!
>>>
>>> I also think the ExternalizedLayout idea is a very attractive option.
>>> That
>>> way there is no pressure to include this in any particular release, we
>>> can
>>> release it when we are confident that is ready. I also like the fact that
>>> it does not replace the current serialization and it can be switched on
>>> and
>>> off in configuration so if there is a bug, users can fall back to the
>>> existing plain serialization.
>>>
>>>
>>>
>>> On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker wrote:
>>>
>>> I would second the ExternalizedLayout. Layouts are the way to go for
>>>> compatibility and are simpler. Particularly useful for alternative
>>>> serialization protocols, too.
>>>>
>>>>
>>>> On 11 July 2014 06:41, Gary Gregory wrote:
>>>>
>>>> I understand Ralph ' s concern but now is the time for this kind of
>>>>> change. Otherwise we will need even more clever solutions to get this
>>>>>
>>>> kind
>>>>
>>>>> of size improvement. I'd love to see some performance numbers. The size
>>>>> improvement is not negligible, which is great!
>>>>>
>>>>> Gary
>>>>>
>>>>>
>>>>>
>>>>> Original message From: Ralph Goers <
>>>>> ralph.go...@dslextreme.com> Date:07/11/2014 01:46
>>>>> (GMT-05:00) To: Log4J Users List <
>>>>> log4j-user@logging.apache.org> Subject: Re: Make LogEvent
>>>>> implementations Externalizable
>>>>> I’d be afraid of breaking compatibility even now. However, I
>>>>> think
>>>>> what you really want to do is to create an ExternalizedLayout and then
>>>>>
>>>> just
>>>>
>>>>> use that instead of the default SerializedLayout. If you want to
>>>>> supply
>>>>> that Layout as a patch to a Jira issue it could be added at any time.
>>>>>
>>>>> Ralph
>>>>>
>>>>> On Jul 10, 2014, at 9:23 PM, Scott Harrington
>>>>> wrote:
>>>>>
>>>>> Ralph & co:
>>>>>>
>>>>>> I hear you're gearing up for the release.
>>>>>>
>>>>>> Last weekend I scratched an itch of mine relating to SocketAppender ->
>>>>>>
>>>>> SocketServer bandwidth, and was able to reduce a 500-character message
>>>>>
>>>> from
>>>>
>>>>> around 1700 bytes to 700 bytes on the wire (it's easy to improve on
>>>>>
>>>> Java's
>>>>
>>>>> default serialization).
>>>>>
>>>>>>
>>>>>> I was going to submit an enhancement request with patch to JIRA but
>>>>>>
>>>>> instead I went on vacation for two weeks.
>>>>>
>>>>>>
>>>>>> I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
>>>>>>
>>>>> writeExternal / readExternal methods. I did NOT have time to make an
>>>>> equivalent change to Log4jLogEvent, or to write up any performance
&g
Re: Make LogEvent implementations Externalizable
What prevents you from writing a more compact representation of the
LogEvent (including all other LogEvent fields) to the byte array in the
{{toByteArray(LogEvent)}} method of an ExternalizableLayout?
On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington
wrote:
> I looked at replacing SerializedLayout and/or the toSerializable method
> but the real savings comes not from the Message itself but from all the
> other fields of the LogEvent, such as the Level, ContextMap, ContextStack,
> and the class descriptor (including superclasses) of the LogEvent itself.
>
> For big messages, an ExternalizableLayout would be a fine solution,
> because writeUTF would give you roughly 50% compression over Java's default
> 2-byte per char String serialization.
>
> However for typical log messages the message itself is being dwarfed on
> the wire by class descriptor overhead.
>
>
>
> On Sat, 12 Jul 2014, Remko Popma wrote:
>
> That is a nice reduction in size!
>>
>> I also think the ExternalizedLayout idea is a very attractive option. That
>> way there is no pressure to include this in any particular release, we can
>> release it when we are confident that is ready. I also like the fact that
>> it does not replace the current serialization and it can be switched on
>> and
>> off in configuration so if there is a bug, users can fall back to the
>> existing plain serialization.
>>
>>
>>
>> On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker wrote:
>>
>> I would second the ExternalizedLayout. Layouts are the way to go for
>>> compatibility and are simpler. Particularly useful for alternative
>>> serialization protocols, too.
>>>
>>>
>>> On 11 July 2014 06:41, Gary Gregory wrote:
>>>
>>> I understand Ralph ' s concern but now is the time for this kind of
>>>> change. Otherwise we will need even more clever solutions to get this
>>>>
>>> kind
>>>
>>>> of size improvement. I'd love to see some performance numbers. The size
>>>> improvement is not negligible, which is great!
>>>>
>>>> Gary
>>>>
>>>>
>>>>
>>>> Original message From: Ralph Goers <
>>>> ralph.go...@dslextreme.com> Date:07/11/2014 01:46
>>>> (GMT-05:00) To: Log4J Users List <
>>>> log4j-user@logging.apache.org> Subject: Re: Make LogEvent
>>>> implementations Externalizable
>>>> I’d be afraid of breaking compatibility even now. However, I
>>>> think
>>>> what you really want to do is to create an ExternalizedLayout and then
>>>>
>>> just
>>>
>>>> use that instead of the default SerializedLayout. If you want to supply
>>>> that Layout as a patch to a Jira issue it could be added at any time.
>>>>
>>>> Ralph
>>>>
>>>> On Jul 10, 2014, at 9:23 PM, Scott Harrington
>>>> wrote:
>>>>
>>>> Ralph & co:
>>>>>
>>>>> I hear you're gearing up for the release.
>>>>>
>>>>> Last weekend I scratched an itch of mine relating to SocketAppender ->
>>>>>
>>>> SocketServer bandwidth, and was able to reduce a 500-character message
>>>>
>>> from
>>>
>>>> around 1700 bytes to 700 bytes on the wire (it's easy to improve on
>>>>
>>> Java's
>>>
>>>> default serialization).
>>>>
>>>>>
>>>>> I was going to submit an enhancement request with patch to JIRA but
>>>>>
>>>> instead I went on vacation for two weeks.
>>>>
>>>>>
>>>>> I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
>>>>>
>>>> writeExternal / readExternal methods. I did NOT have time to make an
>>>> equivalent change to Log4jLogEvent, or to write up any performance tests
>>>>
>>> or
>>>
>>>> regression tests.
>>>>
>>>>>
>>>>> Should I submit what I have for discussion and hopeful inclusion in
>>>>>
>>>> 2.0?
>>>
>>>>
>>>>> Or will it have to wait for 2.1?
>>>>>
>>>>> If we wait, then due to the necessary serialVersionUID change, v2.0
>>>>>
>>>> SocketAppender would not be able to talk to v2.1 SocketServer or vice
>>>>
>>> versa
>>>
>>>&
Re: Make LogEvent implementations Externalizable
I looked at replacing SerializedLayout and/or the toSerializable method
but the real savings comes not from the Message itself but from all the
other fields of the LogEvent, such as the Level, ContextMap, ContextStack,
and the class descriptor (including superclasses) of the LogEvent itself.
For big messages, an ExternalizableLayout would be a fine solution,
because writeUTF would give you roughly 50% compression over Java's
default 2-byte per char String serialization.
However for typical log messages the message itself is being dwarfed on
the wire by class descriptor overhead.
On Sat, 12 Jul 2014, Remko Popma wrote:
That is a nice reduction in size!
I also think the ExternalizedLayout idea is a very attractive option. That
way there is no pressure to include this in any particular release, we can
release it when we are confident that is ready. I also like the fact that
it does not replace the current serialization and it can be switched on and
off in configuration so if there is a bug, users can fall back to the
existing plain serialization.
On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker wrote:
I would second the ExternalizedLayout. Layouts are the way to go for
compatibility and are simpler. Particularly useful for alternative
serialization protocols, too.
On 11 July 2014 06:41, Gary Gregory wrote:
I understand Ralph ' s concern but now is the time for this kind of
change. Otherwise we will need even more clever solutions to get this
kind
of size improvement. I'd love to see some performance numbers. The size
improvement is not negligible, which is great!
Gary
Original message From: Ralph Goers <
ralph.go...@dslextreme.com> Date:07/11/2014 01:46
(GMT-05:00) To: Log4J Users List <
log4j-user@logging.apache.org> Subject: Re: Make LogEvent
implementations Externalizable
I’d be afraid of breaking compatibility even now. However, I think
what you really want to do is to create an ExternalizedLayout and then
just
use that instead of the default SerializedLayout. If you want to supply
that Layout as a patch to a Jira issue it could be added at any time.
Ralph
On Jul 10, 2014, at 9:23 PM, Scott Harrington
wrote:
Ralph & co:
I hear you're gearing up for the release.
Last weekend I scratched an itch of mine relating to SocketAppender ->
SocketServer bandwidth, and was able to reduce a 500-character message
from
around 1700 bytes to 700 bytes on the wire (it's easy to improve on
Java's
default serialization).
I was going to submit an enhancement request with patch to JIRA but
instead I went on vacation for two weeks.
I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
writeExternal / readExternal methods. I did NOT have time to make an
equivalent change to Log4jLogEvent, or to write up any performance tests
or
regression tests.
Should I submit what I have for discussion and hopeful inclusion in
2.0?
Or will it have to wait for 2.1?
If we wait, then due to the necessary serialVersionUID change, v2.0
SocketAppender would not be able to talk to v2.1 SocketServer or vice
versa
(unless ugly duplicate versions are maintained).
Below is what the added code looks like. I only tested in
RingBufferLogEvent but should be similarly usable in Log4jLogEvent, and
perhaps we should discuss a RingBufferLogEvent.readResolve that makes
them
all become Log4jLogEvents on the SocketServer (receiving) end.
...
public void writeExternal(ObjectOutput out) throws IOException {
getThrownProxy();
out.writeByte(1); // wireFormat
int presenceMap = (loggerName == null ? 0 : 0x1) |
(marker == null ? 0 : 0x2) |
(fqcn == null ? 0 : 0x4) |
(level == null ? 0 : 0x8) |
(message == null ? 0 : (0x10 | (isSerializeAsString(message)
? 0 : 0x20))) |
(thrownProxy == null ? 0 : 0x40) |
(contextMap == null ? 0 : 0x80) |
(contextStack == null ? 0 : 0x100 | (contextStack.getDepth()
== 0 ? 0 : 0x200)) |
(threadName == null ? 0 : 0x400) |
(location == null ? 0 : 0x800);
out.writeShort(presenceMap);
if (loggerName != null) {
out.writeUTF(loggerName);
}
if (marker != null) {
out.writeObject(marker);
}
if (fqcn != null) {
out.writeUTF(fqcn);
}
if (level != null) {
out.writeUTF(level.name());
}
if (message != null) {
if (isSerializeAsString(message)) {
out.writeUTF(message.getFormattedMessage());
}
else {
out.writeObject(message);
}
}
if (thrownProxy != null) {
out.writeObject(thrownProxy);
}
if (contextMap != null) {
writeString2StringMap(out, contextMap);
}
if (contextStack != null && contextStack.getDepth() != 0) {
Re: Make LogEvent implementations Externalizable
That is a nice reduction in size!
I also think the ExternalizedLayout idea is a very attractive option. That
way there is no pressure to include this in any particular release, we can
release it when we are confident that is ready. I also like the fact that
it does not replace the current serialization and it can be switched on and
off in configuration so if there is a bug, users can fall back to the
existing plain serialization.
On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker wrote:
> I would second the ExternalizedLayout. Layouts are the way to go for
> compatibility and are simpler. Particularly useful for alternative
> serialization protocols, too.
>
>
> On 11 July 2014 06:41, Gary Gregory wrote:
>
> > I understand Ralph ' s concern but now is the time for this kind of
> > change. Otherwise we will need even more clever solutions to get this
> kind
> > of size improvement. I'd love to see some performance numbers. The size
> > improvement is not negligible, which is great!
> >
> > Gary
> >
> >
> >
> > Original message From: Ralph Goers <
> > ralph.go...@dslextreme.com> Date:07/11/2014 01:46
> > (GMT-05:00) To: Log4J Users List <
> > log4j-user@logging.apache.org> Subject: Re: Make LogEvent
> > implementations Externalizable
> > I’d be afraid of breaking compatibility even now. However, I think
> > what you really want to do is to create an ExternalizedLayout and then
> just
> > use that instead of the default SerializedLayout. If you want to supply
> > that Layout as a patch to a Jira issue it could be added at any time.
> >
> > Ralph
> >
> > On Jul 10, 2014, at 9:23 PM, Scott Harrington
> > wrote:
> >
> > > Ralph & co:
> > >
> > > I hear you're gearing up for the release.
> > >
> > > Last weekend I scratched an itch of mine relating to SocketAppender ->
> > SocketServer bandwidth, and was able to reduce a 500-character message
> from
> > around 1700 bytes to 700 bytes on the wire (it's easy to improve on
> Java's
> > default serialization).
> > >
> > > I was going to submit an enhancement request with patch to JIRA but
> > instead I went on vacation for two weeks.
> > >
> > > I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
> > writeExternal / readExternal methods. I did NOT have time to make an
> > equivalent change to Log4jLogEvent, or to write up any performance tests
> or
> > regression tests.
> > >
> > > Should I submit what I have for discussion and hopeful inclusion in
> 2.0?
> > >
> > > Or will it have to wait for 2.1?
> > >
> > > If we wait, then due to the necessary serialVersionUID change, v2.0
> > SocketAppender would not be able to talk to v2.1 SocketServer or vice
> versa
> > (unless ugly duplicate versions are maintained).
> > >
> > > Below is what the added code looks like. I only tested in
> > RingBufferLogEvent but should be similarly usable in Log4jLogEvent, and
> > perhaps we should discuss a RingBufferLogEvent.readResolve that makes
> them
> > all become Log4jLogEvents on the SocketServer (receiving) end.
> > >
> > > ...
> > >
> > >public void writeExternal(ObjectOutput out) throws IOException {
> > >getThrownProxy();
> > >out.writeByte(1); // wireFormat
> > >int presenceMap = (loggerName == null ? 0 : 0x1) |
> > >(marker == null ? 0 : 0x2) |
> > >(fqcn == null ? 0 : 0x4) |
> > >(level == null ? 0 : 0x8) |
> > >(message == null ? 0 : (0x10 | (isSerializeAsString(message)
> > ? 0 : 0x20))) |
> > >(thrownProxy == null ? 0 : 0x40) |
> > >(contextMap == null ? 0 : 0x80) |
> > >(contextStack == null ? 0 : 0x100 | (contextStack.getDepth()
> > == 0 ? 0 : 0x200)) |
> > >(threadName == null ? 0 : 0x400) |
> > >(location == null ? 0 : 0x800);
> > >out.writeShort(presenceMap);
> > >if (loggerName != null) {
> > >out.writeUTF(loggerName);
> > >}
> > >if (marker != null) {
> > >out.writeObject(marker);
> > >}
> > >if (fqcn != null) {
> > >out.writeUTF(fqcn);
> > >}
> > >if (level != null) {
> > >out.writeUTF(level.name());
> > >}
> > >if (m
Re: Make LogEvent implementations Externalizable
I would second the ExternalizedLayout. Layouts are the way to go for
compatibility and are simpler. Particularly useful for alternative
serialization protocols, too.
On 11 July 2014 06:41, Gary Gregory wrote:
> I understand Ralph ' s concern but now is the time for this kind of
> change. Otherwise we will need even more clever solutions to get this kind
> of size improvement. I'd love to see some performance numbers. The size
> improvement is not negligible, which is great!
>
> Gary
>
>
>
> Original message From: Ralph Goers <
> ralph.go...@dslextreme.com> Date:07/11/2014 01:46
> (GMT-05:00) To: Log4J Users List <
> log4j-user@logging.apache.org> Subject: Re: Make LogEvent
> implementations Externalizable
> I’d be afraid of breaking compatibility even now. However, I think
> what you really want to do is to create an ExternalizedLayout and then just
> use that instead of the default SerializedLayout. If you want to supply
> that Layout as a patch to a Jira issue it could be added at any time.
>
> Ralph
>
> On Jul 10, 2014, at 9:23 PM, Scott Harrington
> wrote:
>
> > Ralph & co:
> >
> > I hear you're gearing up for the release.
> >
> > Last weekend I scratched an itch of mine relating to SocketAppender ->
> SocketServer bandwidth, and was able to reduce a 500-character message from
> around 1700 bytes to 700 bytes on the wire (it's easy to improve on Java's
> default serialization).
> >
> > I was going to submit an enhancement request with patch to JIRA but
> instead I went on vacation for two weeks.
> >
> > I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
> writeExternal / readExternal methods. I did NOT have time to make an
> equivalent change to Log4jLogEvent, or to write up any performance tests or
> regression tests.
> >
> > Should I submit what I have for discussion and hopeful inclusion in 2.0?
> >
> > Or will it have to wait for 2.1?
> >
> > If we wait, then due to the necessary serialVersionUID change, v2.0
> SocketAppender would not be able to talk to v2.1 SocketServer or vice versa
> (unless ugly duplicate versions are maintained).
> >
> > Below is what the added code looks like. I only tested in
> RingBufferLogEvent but should be similarly usable in Log4jLogEvent, and
> perhaps we should discuss a RingBufferLogEvent.readResolve that makes them
> all become Log4jLogEvents on the SocketServer (receiving) end.
> >
> > ...
> >
> >public void writeExternal(ObjectOutput out) throws IOException {
> >getThrownProxy();
> >out.writeByte(1); // wireFormat
> >int presenceMap = (loggerName == null ? 0 : 0x1) |
> >(marker == null ? 0 : 0x2) |
> >(fqcn == null ? 0 : 0x4) |
> >(level == null ? 0 : 0x8) |
> >(message == null ? 0 : (0x10 | (isSerializeAsString(message)
> ? 0 : 0x20))) |
> >(thrownProxy == null ? 0 : 0x40) |
> >(contextMap == null ? 0 : 0x80) |
> >(contextStack == null ? 0 : 0x100 | (contextStack.getDepth()
> == 0 ? 0 : 0x200)) |
> >(threadName == null ? 0 : 0x400) |
> >(location == null ? 0 : 0x800);
> >out.writeShort(presenceMap);
> >if (loggerName != null) {
> >out.writeUTF(loggerName);
> >}
> >if (marker != null) {
> >out.writeObject(marker);
> >}
> >if (fqcn != null) {
> >out.writeUTF(fqcn);
> >}
> >if (level != null) {
> >out.writeUTF(level.name());
> >}
> >if (message != null) {
> >if (isSerializeAsString(message)) {
> >out.writeUTF(message.getFormattedMessage());
> >}
> >else {
> >out.writeObject(message);
> >}
> >}
> >if (thrownProxy != null) {
> >out.writeObject(thrownProxy);
> >}
> >if (contextMap != null) {
> >writeString2StringMap(out, contextMap);
> >}
> >if (contextStack != null && contextStack.getDepth() != 0) {
> >out.writeObject(contextStack);
> >}
> >if (threadName != null) {
> >out.writeUTF(threadName);
> >}
> >if (location != null) {
> >out.writeUTF(location.getClassName());
> >out.writeUTF(location.getMethodName());
> >if ((presenceMap
Re: Make LogEvent implementations Externalizable
I understand Ralph ' s concern but now is the time for this kind of change.
Otherwise we will need even more clever solutions to get this kind of size
improvement. I'd love to see some performance numbers. The size improvement is
not negligible, which is great!
Gary
Original message From: Ralph Goers
Date:07/11/2014 01:46 (GMT-05:00)
To: Log4J Users List
Subject: Re: Make LogEvent implementations Externalizable
I’d be afraid of breaking compatibility even now. However, I think what
you really want to do is to create an ExternalizedLayout and then just use that
instead of the default SerializedLayout. If you want to supply that Layout as
a patch to a Jira issue it could be added at any time.
Ralph
On Jul 10, 2014, at 9:23 PM, Scott Harrington wrote:
> Ralph & co:
>
> I hear you're gearing up for the release.
>
> Last weekend I scratched an itch of mine relating to SocketAppender ->
> SocketServer bandwidth, and was able to reduce a 500-character message from
> around 1700 bytes to 700 bytes on the wire (it's easy to improve on Java's
> default serialization).
>
> I was going to submit an enhancement request with patch to JIRA but instead I
> went on vacation for two weeks.
>
> I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
> writeExternal / readExternal methods. I did NOT have time to make an
> equivalent change to Log4jLogEvent, or to write up any performance tests or
> regression tests.
>
> Should I submit what I have for discussion and hopeful inclusion in 2.0?
>
> Or will it have to wait for 2.1?
>
> If we wait, then due to the necessary serialVersionUID change, v2.0
> SocketAppender would not be able to talk to v2.1 SocketServer or vice versa
> (unless ugly duplicate versions are maintained).
>
> Below is what the added code looks like. I only tested in RingBufferLogEvent
> but should be similarly usable in Log4jLogEvent, and perhaps we should
> discuss a RingBufferLogEvent.readResolve that makes them all become
> Log4jLogEvents on the SocketServer (receiving) end.
>
> ...
>
>public void writeExternal(ObjectOutput out) throws IOException {
>getThrownProxy();
>out.writeByte(1); // wireFormat
>int presenceMap = (loggerName == null ? 0 : 0x1) |
>(marker == null ? 0 : 0x2) |
>(fqcn == null ? 0 : 0x4) |
>(level == null ? 0 : 0x8) |
>(message == null ? 0 : (0x10 | (isSerializeAsString(message) ? 0 :
> 0x20))) |
>(thrownProxy == null ? 0 : 0x40) |
>(contextMap == null ? 0 : 0x80) |
>(contextStack == null ? 0 : 0x100 | (contextStack.getDepth() == 0
> ? 0 : 0x200)) |
>(threadName == null ? 0 : 0x400) |
>(location == null ? 0 : 0x800);
>out.writeShort(presenceMap);
>if (loggerName != null) {
>out.writeUTF(loggerName);
>}
>if (marker != null) {
>out.writeObject(marker);
>}
>if (fqcn != null) {
>out.writeUTF(fqcn);
>}
>if (level != null) {
>out.writeUTF(level.name());
>}
>if (message != null) {
>if (isSerializeAsString(message)) {
>out.writeUTF(message.getFormattedMessage());
>}
>else {
>out.writeObject(message);
>}
>}
>if (thrownProxy != null) {
>out.writeObject(thrownProxy);
>}
>if (contextMap != null) {
>writeString2StringMap(out, contextMap);
>}
>if (contextStack != null && contextStack.getDepth() != 0) {
>out.writeObject(contextStack);
>}
>if (threadName != null) {
>out.writeUTF(threadName);
>}
>if (location != null) {
>out.writeUTF(location.getClassName());
>out.writeUTF(location.getMethodName());
>if ((presenceMap & 0x1000) != 0) {
>out.writeUTF(location.getFileName());
>}
>out.writeInt(location.getLineNumber());
>}
>out.writeLong(currentTimeMillis);
>out.writeBoolean(endOfBatch);
>out.writeBoolean(includeLocation);
>}
>
>public void readExternal(ObjectInput in) throws IOException,
> ClassNotFoundException {
>int wireFormat = in.readByte();
>if (wireFormat == 1) {
>int presenceMap = in.readShort();
>loggerName = (presenceMap & 0x1) == 0 ? null :
>in.readUTF();
>marker = (presenceMap & 0x2) == 0 ? null :
>(M
RE: Make LogEvent implementations Externalizable
Very clever! I'd love to see the whole implementation and reports you are
describing.
Gary
Original message From: Scott Harrington
Date:07/11/2014 00:23 (GMT-05:00)
To: Log4J Users List
Subject: Make LogEvent implementations Externalizable
Ralph & co:
I hear you're gearing up for the release.
Last weekend I scratched an itch of mine relating to SocketAppender ->
SocketServer bandwidth, and was able to reduce a 500-character message
from around 1700 bytes to 700 bytes on the wire (it's easy to improve on
Java's default serialization).
I was going to submit an enhancement request with patch to JIRA but
instead I went on vacation for two weeks.
I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
writeExternal / readExternal methods. I did NOT have time to make an
equivalent change to Log4jLogEvent, or to write up any performance tests
or regression tests.
Should I submit what I have for discussion and hopeful inclusion in 2.0?
Or will it have to wait for 2.1?
If we wait, then due to the necessary serialVersionUID change, v2.0
SocketAppender would not be able to talk to v2.1 SocketServer or vice
versa (unless ugly duplicate versions are maintained).
Below is what the added code looks like. I only tested in
RingBufferLogEvent but should be similarly usable in Log4jLogEvent, and
perhaps we should discuss a RingBufferLogEvent.readResolve that makes them
all become Log4jLogEvents on the SocketServer (receiving) end.
...
public void writeExternal(ObjectOutput out) throws IOException {
getThrownProxy();
out.writeByte(1); // wireFormat
int presenceMap = (loggerName == null ? 0 : 0x1) |
(marker == null ? 0 : 0x2) |
(fqcn == null ? 0 : 0x4) |
(level == null ? 0 : 0x8) |
(message == null ? 0 : (0x10 | (isSerializeAsString(message) ? 0 :
0x20))) |
(thrownProxy == null ? 0 : 0x40) |
(contextMap == null ? 0 : 0x80) |
(contextStack == null ? 0 : 0x100 | (contextStack.getDepth() == 0
? 0 : 0x200)) |
(threadName == null ? 0 : 0x400) |
(location == null ? 0 : 0x800);
out.writeShort(presenceMap);
if (loggerName != null) {
out.writeUTF(loggerName);
}
if (marker != null) {
out.writeObject(marker);
}
if (fqcn != null) {
out.writeUTF(fqcn);
}
if (level != null) {
out.writeUTF(level.name());
}
if (message != null) {
if (isSerializeAsString(message)) {
out.writeUTF(message.getFormattedMessage());
}
else {
out.writeObject(message);
}
}
if (thrownProxy != null) {
out.writeObject(thrownProxy);
}
if (contextMap != null) {
writeString2StringMap(out, contextMap);
}
if (contextStack != null && contextStack.getDepth() != 0) {
out.writeObject(contextStack);
}
if (threadName != null) {
out.writeUTF(threadName);
}
if (location != null) {
out.writeUTF(location.getClassName());
out.writeUTF(location.getMethodName());
if ((presenceMap & 0x1000) != 0) {
out.writeUTF(location.getFileName());
}
out.writeInt(location.getLineNumber());
}
out.writeLong(currentTimeMillis);
out.writeBoolean(endOfBatch);
out.writeBoolean(includeLocation);
}
public void readExternal(ObjectInput in) throws IOException,
ClassNotFoundException {
int wireFormat = in.readByte();
if (wireFormat == 1) {
int presenceMap = in.readShort();
loggerName = (presenceMap & 0x1) == 0 ? null :
in.readUTF();
marker = (presenceMap & 0x2) == 0 ? null :
(Marker) in.readObject();
fqcn = (presenceMap & 0x4) == 0 ? null :
in.readUTF();
level = (presenceMap & 0x8) == 0 ? null :
Level.valueOf(in.readUTF());
message = (presenceMap & 0x10) == 0 ? null :
(presenceMap & 0x20) == 0 ? new SimpleMessage(in.readUTF()) :
(Message) in.readObject();
thrownProxy = (presenceMap & 0x40) == 0 ? null :
(ThrowableProxy) in.readObject();
contextMap = (presenceMap & 0x80) == 0 ? null :
readString2StringMap(in);
contextStack = (presenceMap & 0x100) == 0 ? null :
(presenceMap & 0x200) == 0 ? ThreadContext.EMPTY_STACK :
(ContextStack) in.readObject();
threadName = (presenceMap & 0x400) == 0 ? null :
in.readUTF();
location = (presenceMap & 0x800) == 0 ? null :
new StackTraceElement(in.readUTF
Re: Make LogEvent implementations Externalizable
I’d be afraid of breaking compatibility even now. However, I think what you
really want to do is to create an ExternalizedLayout and then just use that
instead of the default SerializedLayout. If you want to supply that Layout as
a patch to a Jira issue it could be added at any time.
Ralph
On Jul 10, 2014, at 9:23 PM, Scott Harrington wrote:
> Ralph & co:
>
> I hear you're gearing up for the release.
>
> Last weekend I scratched an itch of mine relating to SocketAppender ->
> SocketServer bandwidth, and was able to reduce a 500-character message from
> around 1700 bytes to 700 bytes on the wire (it's easy to improve on Java's
> default serialization).
>
> I was going to submit an enhancement request with patch to JIRA but instead I
> went on vacation for two weeks.
>
> I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
> writeExternal / readExternal methods. I did NOT have time to make an
> equivalent change to Log4jLogEvent, or to write up any performance tests or
> regression tests.
>
> Should I submit what I have for discussion and hopeful inclusion in 2.0?
>
> Or will it have to wait for 2.1?
>
> If we wait, then due to the necessary serialVersionUID change, v2.0
> SocketAppender would not be able to talk to v2.1 SocketServer or vice versa
> (unless ugly duplicate versions are maintained).
>
> Below is what the added code looks like. I only tested in RingBufferLogEvent
> but should be similarly usable in Log4jLogEvent, and perhaps we should
> discuss a RingBufferLogEvent.readResolve that makes them all become
> Log4jLogEvents on the SocketServer (receiving) end.
>
> ...
>
>public void writeExternal(ObjectOutput out) throws IOException {
>getThrownProxy();
>out.writeByte(1); // wireFormat
>int presenceMap = (loggerName == null ? 0 : 0x1) |
>(marker == null ? 0 : 0x2) |
>(fqcn == null ? 0 : 0x4) |
>(level == null ? 0 : 0x8) |
>(message == null ? 0 : (0x10 | (isSerializeAsString(message) ? 0 :
> 0x20))) |
>(thrownProxy == null ? 0 : 0x40) |
>(contextMap == null ? 0 : 0x80) |
>(contextStack == null ? 0 : 0x100 | (contextStack.getDepth() == 0
> ? 0 : 0x200)) |
>(threadName == null ? 0 : 0x400) |
>(location == null ? 0 : 0x800);
>out.writeShort(presenceMap);
>if (loggerName != null) {
>out.writeUTF(loggerName);
>}
>if (marker != null) {
>out.writeObject(marker);
>}
>if (fqcn != null) {
>out.writeUTF(fqcn);
>}
>if (level != null) {
>out.writeUTF(level.name());
>}
>if (message != null) {
>if (isSerializeAsString(message)) {
>out.writeUTF(message.getFormattedMessage());
>}
>else {
>out.writeObject(message);
>}
>}
>if (thrownProxy != null) {
>out.writeObject(thrownProxy);
>}
>if (contextMap != null) {
>writeString2StringMap(out, contextMap);
>}
>if (contextStack != null && contextStack.getDepth() != 0) {
>out.writeObject(contextStack);
>}
>if (threadName != null) {
>out.writeUTF(threadName);
>}
>if (location != null) {
>out.writeUTF(location.getClassName());
>out.writeUTF(location.getMethodName());
>if ((presenceMap & 0x1000) != 0) {
>out.writeUTF(location.getFileName());
>}
>out.writeInt(location.getLineNumber());
>}
>out.writeLong(currentTimeMillis);
>out.writeBoolean(endOfBatch);
>out.writeBoolean(includeLocation);
>}
>
>public void readExternal(ObjectInput in) throws IOException,
> ClassNotFoundException {
>int wireFormat = in.readByte();
>if (wireFormat == 1) {
>int presenceMap = in.readShort();
>loggerName = (presenceMap & 0x1) == 0 ? null :
>in.readUTF();
>marker = (presenceMap & 0x2) == 0 ? null :
>(Marker) in.readObject();
>fqcn = (presenceMap & 0x4) == 0 ? null :
>in.readUTF();
>level = (presenceMap & 0x8) == 0 ? null :
>Level.valueOf(in.readUTF());
>message = (presenceMap & 0x10) == 0 ? null :
>(presenceMap & 0x20) == 0 ? new SimpleMessage(in.readUTF()) :
> (Message) in.readObject();
>thrownProxy = (presenceMap & 0x40) == 0 ? null :
>(ThrowableProxy) in.readObject();
>contextMap = (presenceMap & 0x80) == 0 ? null :
>readString2StringMap(in);
>contextStack = (presenceMap & 0x100) == 0 ? null :
>(presenceMap & 0x200) == 0 ? ThreadContext.EMPTY_STACK :
> (ContextStack) in.readObject();
>threadName = (presenceMap & 0
