[
https://issues.apache.org/jira/browse/KAFKA-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500347#comment-16500347
]
Chia-Ping Tsai commented on KAFKA-5761:
---------------------------------------
+1(non-binding) to this idea(if the ByteBuffer overkill, we can enhance the
org.apache.kafka.common.utils.Bytes to accept the offset and length). Our
application requires this feature also. The serializer, currently, forces us to
do a deep copy from our custom off-heap pool in order to return a "byte[]"...
> Serializer API should support ByteBuffer
> ----------------------------------------
>
> Key: KAFKA-5761
> URL: https://issues.apache.org/jira/browse/KAFKA-5761
> Project: Kafka
> Issue Type: Improvement
> Components: clients
> Affects Versions: 0.11.0.0
> Reporter: Bhaskar Gollapudi
> Priority: Major
> Labels: features, performance
>
> Consider the Serializer : Its main method is :
> byte[] serialize(String topic, T data);
> Producer applications create a implementation that takes in an instance (
> of T ) and convert that to a byte[]. This byte array is allocated a new for
> this message.This byte array then is handed over to Kafka Producer API
> internals that write the bytes to buffer/ network socket. When the next
> message arrives , the serializer instead of creating a new byte[] , should
> try to reuse the existing byte[] for the new message. This requires two
> things :
> 1. The process of handing off the bytes to the buffer/socket and reusing
> the byte[] must happen on the same thread.
> 2 There should be a way for marking the end of available bytes in the
> byte[].
> The first is reasonably simple to understand. If this does not happen , and
> without other necessary synchrinization , the byte[] get corrupted and so
> is the message written to buffer/socket.However , this requirement is easy
> to meet for a producer application , because it controls the threads on
> which the serializer is invoked.
> The second is where the problem lies with the current API. It does not
> allow a variable size of bytes to be read from a container. It is limited
> by the byte[]'s length. This forces the producer to
> 1 either create a new byte[] for a message that is bigger than the previous
> one.
> OR
> 2. Decide a max size and use a padding .
> Both are cumbersome and error prone, and may cause wasting of network
> bandwidth.
> Instead , if there is an Serializer with this method :
> ByteBuffer serialize(String topic, T data);
> This helps to implements a reusable bytes container for clients to avoid
> allocations for each message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
