[
https://issues.apache.org/jira/browse/SOLR-18415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrzej Bialecki updated SOLR-18415:
------------------------------------
Description:
{{MirroredSolrRequestSerializer}} uses a subclass of {{ByteArrayOutputStream}}
in an attempt to optimize data transfer by avoiding array copy - instead it
passes the whole buffer directly as the output bytes from {{{}serialize(){}}}.
This sounds like a plausible optimization. However, the initial size of the
buffer (32 bytes) is doubled on each overflow as the codec writes to the
stream, which means that on average the actual serialized data occupies around
half of the buffer, the rest are zeroes. Deserialization will handle these
trailing zeros just fine, it's the cost of transmitting roughly twice as much
data over the wire that is questionable - this buffer is then passed as-is to
the output stream and eventually ends up as network traffic.
This "optimization" avoids one array copy cost (which reduces GC) but generates
multiple costs on the way - not only the empty data has to be sent over the
network but also it needs to be stored in source Kafka, in MirrorMaker and in
target Kafka, to be finally retrieved by the Consumer (which has to allocate a
buffer sufficiently large to fit also the zeroes), only to discard the zeroes
during deserialization.
I propose to get rid of this optimization and use the plain
{{ByteArrayOutputStream}} instead.
was:
{{MirroringSolrRequestSerializer}} uses a subclass of {{ByteArrayOutputStream}}
in an attempt to optimize data transfer by avoiding array copy - instead it
passes the whole buffer directly as the output bytes from {{{}serialize(){}}}.
This sounds like a plausible optimization. However, the initial size of the
buffer (32 bytes) is doubled on each overflow as the codec writes to the
stream, which means that on average the actual serialized data occupies around
half of the buffer, the rest are zeroes. Deserialization will handle these
trailing zeros just fine, it's the cost of transmitting roughly twice as much
data over the wire that is questionable - this buffer is then passed as-is to
the output stream and eventually ends up as network traffic.
This "optimization" avoids one array copy cost (which reduces GC) but generates
multiple costs on the way - not only the empty data has to be sent over the
network but also it needs to be stored in source Kafka, in MirrorMaker and in
target Kafka, to be finally retrieved by the Consumer (which has to allocate a
buffer sufficiently large to fit also the zeroes), only to discard the zeroes
during deserialization.
I propose to get rid of this optimization and use the plain
{{ByteArrayOutputStream}} instead.
> CrossDC Common: ExposedByteArrayOutputStream optimizes wrong aspect
> -------------------------------------------------------------------
>
> Key: SOLR-18415
> URL: https://issues.apache.org/jira/browse/SOLR-18415
> Project: Solr
> Issue Type: Bug
> Components: module - crossDC
> Affects Versions: 10.0, 9.10.1
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Priority: Major
> Fix For: main(11.0), 10.1
>
>
> {{MirroredSolrRequestSerializer}} uses a subclass of
> {{ByteArrayOutputStream}} in an attempt to optimize data transfer by avoiding
> array copy - instead it passes the whole buffer directly as the output bytes
> from {{{}serialize(){}}}.
> This sounds like a plausible optimization. However, the initial size of the
> buffer (32 bytes) is doubled on each overflow as the codec writes to the
> stream, which means that on average the actual serialized data occupies
> around half of the buffer, the rest are zeroes. Deserialization will handle
> these trailing zeros just fine, it's the cost of transmitting roughly twice
> as much data over the wire that is questionable - this buffer is then passed
> as-is to the output stream and eventually ends up as network traffic.
> This "optimization" avoids one array copy cost (which reduces GC) but
> generates multiple costs on the way - not only the empty data has to be sent
> over the network but also it needs to be stored in source Kafka, in
> MirrorMaker and in target Kafka, to be finally retrieved by the Consumer
> (which has to allocate a buffer sufficiently large to fit also the zeroes),
> only to discard the zeroes during deserialization.
> I propose to get rid of this optimization and use the plain
> {{ByteArrayOutputStream}} instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
