What prevents you from writing a more compact representation of the
LogEvent (including all other LogEvent fields) to the byte array in the
{{toByteArray(LogEvent)}} method of an ExternalizableLayout?
On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington <scott...@sns-usa.com>
wrote:
> I looked at replacing SerializedLayout and/or the toSerializable method
> but the real savings comes not from the Message itself but from all the
> other fields of the LogEvent, such as the Level, ContextMap, ContextStack,
> and the class descriptor (including superclasses) of the LogEvent itself.
>
> For big messages, an ExternalizableLayout would be a fine solution,
> because writeUTF would give you roughly 50% compression over Java's default
> 2-byte per char String serialization.
>
> However for typical log messages the message itself is being dwarfed on
> the wire by class descriptor overhead.
>
>
>
> On Sat, 12 Jul 2014, Remko Popma wrote:
>
> That is a nice reduction in size!
>>
>> I also think the ExternalizedLayout idea is a very attractive option. That
>> way there is no pressure to include this in any particular release, we can
>> release it when we are confident that is ready. I also like the fact that
>> it does not replace the current serialization and it can be switched on
>> and
>> off in configuration so if there is a bug, users can fall back to the
>> existing plain serialization.
>>
>>
>>
>> On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker <boa...@gmail.com> wrote:
>>
>> I would second the ExternalizedLayout. Layouts are the way to go for
>>> compatibility and are simpler. Particularly useful for alternative
>>> serialization protocols, too.
>>>
>>>
>>> On 11 July 2014 06:41, Gary Gregory <garydgreg...@gmail.com> wrote:
>>>
>>> I understand Ralph ' s concern but now is the time for this kind of
>>>> change. Otherwise we will need even more clever solutions to get this
>>>>
>>> kind
>>>
>>>> of size improvement. I'd love to see some performance numbers. The size
>>>> improvement is not negligible, which is great!
>>>>
>>>> Gary
>>>>
>>>>
>>>>
>>>> <div>-------- Original message --------</div><div>From: Ralph Goers <
>>>> ralph.go...@dslextreme.com> </div><div>Date:07/11/2014 01:46
>>>> (GMT-05:00) </div><div>To: Log4J Users List <
>>>> log4j-user@logging.apache.org> </div><div>Subject: Re: Make LogEvent
>>>> implementations Externalizable </div><div>
>>>> </div>I’d be afraid of breaking compatibility even now. However, I
>>>> think
>>>> what you really want to do is to create an ExternalizedLayout and then
>>>>
>>> just
>>>
>>>> use that instead of the default SerializedLayout. If you want to supply
>>>> that Layout as a patch to a Jira issue it could be added at any time.
>>>>
>>>> Ralph
>>>>
>>>> On Jul 10, 2014, at 9:23 PM, Scott Harrington <scott...@sns-usa.com>
>>>> wrote:
>>>>
>>>> Ralph & co:
>>>>>
>>>>> I hear you're gearing up for the release.
>>>>>
>>>>> Last weekend I scratched an itch of mine relating to SocketAppender ->
>>>>>
>>>> SocketServer bandwidth, and was able to reduce a 500-character message
>>>>
>>> from
>>>
>>>> around 1700 bytes to 700 bytes on the wire (it's easy to improve on
>>>>
>>> Java's
>>>
>>>> default serialization).
>>>>
>>>>>
>>>>> I was going to submit an enhancement request with patch to JIRA but
>>>>>
>>>> instead I went on vacation for two weeks.
>>>>
>>>>>
>>>>> I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
>>>>>
>>>> writeExternal / readExternal methods. I did NOT have time to make an
>>>> equivalent change to Log4jLogEvent, or to write up any performance tests
>>>>
>>> or
>>>
>>>> regression tests.
>>>>
>>>>>
>>>>> Should I submit what I have for discussion and hopeful inclusion in
>>>>>
>>>> 2.0?
>>>
>>>>
>>>>> Or will it have to wait for 2.1?
>>>>>
>>>>> If we wait, then due to the necessary serialVersionUID change, v2.0
>>>>>
>>>> SocketAppender would not be able to talk to v2.1 SocketServer or vice
>>>>
>>> versa
>>>
>>>> (unless ugly duplicate versions are maintained).
>>>>
>>>>>
>>>>> Below is what the added code looks like. I only tested in
>>>>>
>>>> RingBufferLogEvent but should be similarly usable in Log4jLogEvent, and
>>>> perhaps we should discuss a RingBufferLogEvent.readResolve that makes
>>>>
>>> them
>>>
>>>> all become Log4jLogEvents on the SocketServer (receiving) end.
>>>>
>>>>>
>>>>> ...
>>>>>
>>>>> public void writeExternal(ObjectOutput out) throws IOException {
>>>>> getThrownProxy();
>>>>> out.writeByte(1); // wireFormat
>>>>> int presenceMap = (loggerName == null ? 0 : 0x1) |
>>>>> (marker == null ? 0 : 0x2) |
>>>>> (fqcn == null ? 0 : 0x4) |
>>>>> (level == null ? 0 : 0x8) |
>>>>> (message == null ? 0 : (0x10 | (isSerializeAsString(message)
>>>>>
>>>> ? 0 : 0x20))) |
>>>>
>>>>> (thrownProxy == null ? 0 : 0x40) |
>>>>> (contextMap == null ? 0 : 0x80) |
>>>>> (contextStack == null ? 0 : 0x100 | (contextStack.getDepth()
>>>>>
>>>> == 0 ? 0 : 0x200)) |
>>>>
>>>>> (threadName == null ? 0 : 0x400) |
>>>>> (location == null ? 0 : 0x800);
>>>>> out.writeShort(presenceMap);
>>>>> if (loggerName != null) {
>>>>> out.writeUTF(loggerName);
>>>>> }
>>>>> if (marker != null) {
>>>>> out.writeObject(marker);
>>>>> }
>>>>> if (fqcn != null) {
>>>>> out.writeUTF(fqcn);
>>>>> }
>>>>> if (level != null) {
>>>>> out.writeUTF(level.name());
>>>>> }
>>>>> if (message != null) {
>>>>> if (isSerializeAsString(message)) {
>>>>> out.writeUTF(message.getFormattedMessage());
>>>>> }
>>>>> else {
>>>>> out.writeObject(message);
>>>>> }
>>>>> }
>>>>> if (thrownProxy != null) {
>>>>> out.writeObject(thrownProxy);
>>>>> }
>>>>> if (contextMap != null) {
>>>>> writeString2StringMap(out, contextMap);
>>>>> }
>>>>> if (contextStack != null && contextStack.getDepth() != 0) {
>>>>> out.writeObject(contextStack);
>>>>> }
>>>>> if (threadName != null) {
>>>>> out.writeUTF(threadName);
>>>>> }
>>>>> if (location != null) {
>>>>> out.writeUTF(location.getClassName());
>>>>> out.writeUTF(location.getMethodName());
>>>>> if ((presenceMap & 0x1000) != 0) {
>>>>> out.writeUTF(location.getFileName());
>>>>> }
>>>>> out.writeInt(location.getLineNumber());
>>>>> }
>>>>> out.writeLong(currentTimeMillis);
>>>>> out.writeBoolean(endOfBatch);
>>>>> out.writeBoolean(includeLocation);
>>>>> }
>>>>>
>>>>> public void readExternal(ObjectInput in) throws IOException,
>>>>>
>>>> ClassNotFoundException {
>>>>
>>>>> int wireFormat = in.readByte();
>>>>> if (wireFormat == 1) {
>>>>> int presenceMap = in.readShort();
>>>>> loggerName = (presenceMap & 0x1) == 0 ? null :
>>>>> in.readUTF();
>>>>> marker = (presenceMap & 0x2) == 0 ? null :
>>>>> (Marker) in.readObject();
>>>>> fqcn = (presenceMap & 0x4) == 0 ? null :
>>>>> in.readUTF();
>>>>> level = (presenceMap & 0x8) == 0 ? null :
>>>>> Level.valueOf(in.readUTF());
>>>>> message = (presenceMap & 0x10) == 0 ? null :
>>>>> (presenceMap & 0x20) == 0 ? new
>>>>>
>>>> SimpleMessage(in.readUTF()) : (Message) in.readObject();
>>>>
>>>>> thrownProxy = (presenceMap & 0x40) == 0 ? null :
>>>>> (ThrowableProxy) in.readObject();
>>>>> contextMap = (presenceMap & 0x80) == 0 ? null :
>>>>> readString2StringMap(in);
>>>>> contextStack = (presenceMap & 0x100) == 0 ? null :
>>>>> (presenceMap & 0x200) == 0 ? ThreadContext.EMPTY_STACK :
>>>>>
>>>> (ContextStack) in.readObject();
>>>>
>>>>> threadName = (presenceMap & 0x400) == 0 ? null :
>>>>> in.readUTF();
>>>>> location = (presenceMap & 0x800) == 0 ? null :
>>>>> new StackTraceElement(in.readUTF(), in.readUTF(),
>>>>>
>>>> (presenceMap & 0x1000) == 0 ? null : in.readUTF(), in.readInt());
>>>>
>>>>> currentTimeMillis = in.readLong();
>>>>> endOfBatch = in.readBoolean();
>>>>> includeLocation = in.readBoolean();
>>>>> }
>>>>> else {
>>>>> throw new StreamCorruptedException("Unsupported LogEvent
>>>>>
>>>> wire
>>>
>>>> format " + wireFormat);
>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> private static boolean isSerializeAsString(Message message)
>>>>> {
>>>>> return message instanceof SimpleMessage || message instanceof
>>>>>
>>>> ObjectMessage;
>>>>
>>>>> }
>>>>>
>>>>> private void writeString2StringMap(ObjectOutput out, Map<String,
>>>>>
>>>> String> map) throws IOException
>>>>
>>>>> {
>>>>> out.writeInt(map.size());
>>>>> for (Map.Entry<String, String> entry : map.entrySet()) {
>>>>> out.writeUTF(entry.getKey());
>>>>> out.writeUTF(entry.getValue());
>>>>> }
>>>>> }
>>>>>
>>>>> private static Map<String, String> readString2StringMap(ObjectInput
>>>>>
>>>> in) throws ClassNotFoundException, IOException
>>>>
>>>>> {
>>>>> int size = in.readInt();
>>>>> if (size == 0) {
>>>>> return Collections.emptyMap();
>>>>> }
>>>>> Map<String, String> map = new HashMap<String, String>(size);
>>>>> while (size-- > 0) {
>>>>> map.put(in.readUTF(), in.readUTF());
>>>>> }
>>>>> return map;
>>>>> }
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
>>>>> For additional commands, e-mail: log4j-user-h...@logging.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
>>>> For additional commands, e-mail: log4j-user-h...@logging.apache.org
>>>>
>>>>
>>>>
>>>
>>> --
>>> Matt Sicker <boa...@gmail.com>
>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
> For additional commands, e-mail: log4j-user-h...@logging.apache.org
>
