Hi Matthias,
Indeed, you are right. I missed your email, I had a problem with my mail
server, so I guess I didn’t receive it.
1) Here is what I came up with
In my ListSerializer.java I have the following:
try (final DataOutputStream out = new DataOutputStream(baos)) {
// I’m encoding the class name at the top of the byte array
out.writeUTF(data.getClass().getCanonicalName());
out.writeInt(size);
for (T entry : data) {
final byte[] bytes = serializer.serialize(topic, entry);
out.writeInt(bytes.length);
out.write(bytes);
}
return baos.toByteArray();
}
Then in ListDeserializer.java:
try (final DataInputStream dis = new DataInputStream(new
ByteArrayInputStream(data))) {
final String listClassName = dis.readUTF();
final int size = dis.readInt();
List<T> deserializedList = getListInstance(listClassName, size);
for (int i = 0; i < size; i++) {
byte[] payload = new byte[dis.readInt()];
dis.read(payload);
deserializedList.add(deserializer.deserialize(topic, payload));
}
return deserializedList;
}
private List<T> getListInstance(String listClassName, int listSize) {
try {
Class<?> listClass = Class.forName(listClassName);
Constructor<?> listConstructor = listClass.getConstructor(Integer.TYPE);
return (List<T>) listConstructor.newInstance(listSize);
} catch (Exception e) {
throw new RuntimeException("Could not construct a list instance of \""
+ listClassName + "\"", e);
}
}
It works just fine, except for one edge case. If you try to pass
Arrays.asList(...) to the (de)serializer which is ArrayList instance, it will
throw: java.lang.ClassNotFoundException: java.util.Arrays.ArrayList
Because it is expecting java.util.ArrayList.
The problem is described here:
https://stackoverflow.com/questions/28851652/java-lang-classcastexception-java-util-arraysarraylist-cannot-be-cast-to-java
Arrays.asList() produces an instance of a List implementation
(java.util.Arrays$ArrayList) that is not java.util.ArrayList
What do you think of this? Based on your experience, what is the best approach
in this case? Maybe we modify a String? Is there any other cases where similar
problem happens?
Still working on (2)
Thank you Matthias!
Best,
Daniyar Yeralin
> On Jun 11, 2019, at 6:05 PM, Matthias J. Sax <matth...@confluent.io> wrote:
>
> Seems you missed my reply from 31/6. C&P below:
>
>> (1) The current PR suggests to always instantiate an `ArrayList` --
>> however, if a user wants to use any other list implementation, they have
>> no way to specify this. It might be good to either allow users to
>> specify the list-type on the deserializer, or encode the list type
>> directly in the bytes, and hence, whatever type the serialized list was,
>> the same type will be used on deserialization (might only work for Java
>> build-it list types).
>>
>> Personally, I thinks its better/more flexible to specify the list-type
>> on the deserializer, as it also allows to plug-in any custom list types.
>>
>> This could of course be opt-in and for the case users don't care, we
>> just default to `ArrayList`.
>>
>>
>> (2) For Java built-in types, we could check the type via `instanceof` --
>> if the type is unknown, we fall back to per-element length encoding. As
>> an alternative, we could also add a constructor taking an `enum` with
>> two values `fixed-size` and `variable-size`, or a config instead of a
>> constructor element.
>>
>>
>> Just bounding off ideas -- maybe there are good reasons (too
>> complicated?) to not support either of them.
>>
>
>
> -Matthias
>
> On 6/10/19 8:44 AM, Development wrote:
>> Bump
>>
>>> On May 24, 2019, at 2:09 PM, Development <d...@yeralin.net> wrote:
>>>
>>> Hey,
>>>
>>> - did we consider to make the return type (ie, ArrayList, vs
>>> LinkesList) configurable or encode it the serialized bytes?
>>>
>>> Not sure about this one. Could you elaborate?
>>>
>>> - atm the size of each element is encoded individually; did we consider
>>> an optimization for fixed size elements (like Long) to avoid this overhead?
>>>
>>> I cannot think of any clean way to do so. How would you see it?
>>>
>>> Btw I resolved all your comments under PR
>>>
>>> Best,
>>> Daniyar Yeralin
>>>
>>>> On May 24, 2019, at 12:01 AM, Matthias J. Sax <matth...@confluent.io>
>>>> wrote:
>>>>
>>>> Thanks for the KIP. I also had a look into the PR and have two follow up
>>>> question:
>>>>
>>>>
>>>> - did we consider to make the return type (ie, ArrayList, vs
>>>> LinkesList) configurable or encode it the serialized bytes?
>>>>
>>>> - atm the size of each element is encoded individually; did we consider
>>>> an optimization for fixed size elements (like Long) to avoid this overhead?
>>>>
>>>>
>>>>
>>>> -Matthias
>>>>
>>>> On 5/15/19 6:05 PM, John Roesler wrote:
>>>>> Sounds good!
>>>>>
>>>>> On Tue, May 14, 2019 at 9:21 AM Development <d...@yeralin.net> wrote:
>>>>>>
>>>>>> Hey,
>>>>>>
>>>>>> I think it the proposal is finalized, no one raised any concerns. Shall
>>>>>> we call it for a [VOTE]?
>>>>>>
>>>>>> Best,
>>>>>> Daniyar Yeralin
>>>>>>
>>>>>>> On May 10, 2019, at 10:17 AM, John Roesler <j...@confluent.io> wrote:
>>>>>>>
>>>>>>> Good observation, Daniyar.
>>>>>>>
>>>>>>> Maybe we should just not implement support for serdeFrom.
>>>>>>>
>>>>>>> We can always add it later, but I think you're right, we need some
>>>>>>> kind of more sophisticated support, or at least a second argument for
>>>>>>> the inner class.
>>>>>>>
>>>>>>> For now, it seems like most use cases would be satisfied without
>>>>>>> serdeFrom(...List...)
>>>>>>>
>>>>>>> -John
>>>>>>>
>>>>>>> On Fri, May 10, 2019 at 8:57 AM Development <d...@yeralin.net> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I was trying to add some test cases for the list serde, and it led me
>>>>>>>> to this class
>>>>>>>> `org.apache.kafka.common.serialization.SerializationTest`. I saw that
>>>>>>>> it relies on method
>>>>>>>> `org.apache.kafka.common.serialization.serdeFrom(Class<T> type)`
>>>>>>>>
>>>>>>>> Now, I’m not sure how to adapt List<T> serde for this method, since it
>>>>>>>> will be a “nested class”. What is the best approach in this case?
>>>>>>>>
>>>>>>>> I remember that in Jackson for example, one uses a TypeFactory, and
>>>>>>>> constructs “collectionType” of two classes. For example,
>>>>>>>> `constructCollectionType(List.class, String.class).getClass()`. I
>>>>>>>> don’t think it applies here.
>>>>>>>>
>>>>>>>> Any ideas?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Daniyar Yeralin
>>>>>>>>
>>>>>>>>> On May 9, 2019, at 2:10 PM, Development <d...@yeralin.net> wrote:
>>>>>>>>>
>>>>>>>>> Hey Sophie,
>>>>>>>>>
>>>>>>>>> Thank you for your input. I think I’d rather finish this KIP as is,
>>>>>>>>> and then open a new one for the Collections (if everyone agrees). I
>>>>>>>>> don’t want to extend the current KIP-466, since most of the work is
>>>>>>>>> already done for it.
>>>>>>>>>
>>>>>>>>> Meanwhile, I’ll start adding some test cases for this new list serde
>>>>>>>>> since this discussion seems to be approaching its logical end.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Daniyar Yeralin
>>>>>>>>>
>>>>>>>>>> On May 9, 2019, at 1:35 PM, Sophie Blee-Goldman
>>>>>>>>>> <sop...@confluent.io> wrote:
>>>>>>>>>>
>>>>>>>>>> Good point about serdes for other Collections. On the one hand I'd
>>>>>>>>>> guess
>>>>>>>>>> that non-List Collections are probably relatively rare in practice
>>>>>>>>>> (if
>>>>>>>>>> anyone disagrees please correct me!) but on the other hand, a) even
>>>>>>>>>> if just
>>>>>>>>>> a small number of people benefit I think it's worth the extra effort
>>>>>>>>>> and b)
>>>>>>>>>> if we do end up needing/wanting them in the future it would save us
>>>>>>>>>> a KIP
>>>>>>>>>> to just add them now. Personally I feel it would make sense to
>>>>>>>>>> expand the
>>>>>>>>>> scope of this KIP a bit to include all Collections as a logical
>>>>>>>>>> unit, but
>>>>>>>>>> the ROI could be low..
>>>>>>>>>>
>>>>>>>>>> (I know of at least one instance in the unit tests where a Set serde
>>>>>>>>>> could
>>>>>>>>>> be useful, and there may be more)
>>>>>>>>>>
>>>>>>>>>> On Thu, May 9, 2019 at 7:27 AM Development <d...@yeralin.net> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey,
>>>>>>>>>>>
>>>>>>>>>>> I don’t see any replies. Seems like this proposal can be finalized
>>>>>>>>>>> and
>>>>>>>>>>> called for a vote?
>>>>>>>>>>>
>>>>>>>>>>> Also I’ve been thinking. Do we need more serdes for other
>>>>>>>>>>> Collections?
>>>>>>>>>>> Like queue or set for example
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Daniyar Yeralin
>>>>>>>>>>>
>>>>>>>>>>>> On May 8, 2019, at 2:28 PM, John Roesler <j...@confluent.io> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Daniyar,
>>>>>>>>>>>>
>>>>>>>>>>>> No worries about the procedural stuff. Prior experience with KIPs
>>>>>>>>>>>> is
>>>>>>>>>>>> not required :)
>>>>>>>>>>>>
>>>>>>>>>>>> I was just trying to help you propose this stuff in a way that the
>>>>>>>>>>>> others will find easy to review.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for updating the KIP. Thanks to the others for helping out
>>>>>>>>>>>> with
>>>>>>>>>>>> the syntax.
>>>>>>>>>>>>
>>>>>>>>>>>> Given these updates, I'm curious if anyone else has feedback about
>>>>>>>>>>>> this proposal. Personally, I think it sounds fine!
>>>>>>>>>>>>
>>>>>>>>>>>> -John
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 8, 2019 at 1:01 PM Development <d...@yeralin.net>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hey,
>>>>>>>>>>>>>
>>>>>>>>>>>>> That worked! I certainly lack Java generics knowledge. Thanks for
>>>>>>>>>>>>> the
>>>>>>>>>>> snippet. I’ll update KIP again.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Daniyar Yeralin
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On May 8, 2019, at 1:39 PM, Chris Egerton <chr...@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Daniyar,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think you may want to tweak your syntax a little:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public static <T> Serde<List<T>> List(Serde<T> innerSerde) {
>>>>>>>>>>>>>> return new ListSerde<T>(innerSerde);
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Does that work?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, May 8, 2019 at 10:29 AM Development <d...@yeralin.net
>>>>>>>>>>>>>> <mailto:
>>>>>>>>>>> d...@yeralin.net>> wrote:
>>>>>>>>>>>>>> Hi John,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I updated JIRA and KIP.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I didn’t know about the process, and created PR before I knew
>>>>>>>>>>>>>> about
>>>>>>>>>>> KIPs :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As per static declaration, I don’t think Java allows that:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Daniyar Yeralin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On May 7, 2019, at 2:22 PM, John Roesler <j...@confluent.io
>>>>>>>>>>>>>>> <mailto:
>>>>>>>>>>> j...@confluent.io>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for that update. Do you mind making changes primarily on
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> KIP document ? (
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization
>>>>>>>>>>> <
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization
>>>>>>>>>>>> )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is the design document that we have to agree on and vote
>>>>>>>>>>>>>>> for, the
>>>>>>>>>>>>>>> PR comes later. It can be nice to have an implementation to
>>>>>>>>>>>>>>> look at,
>>>>>>>>>>>>>>> but the KIP is the main artifact for this discussion.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> With this in mind, it will help get more reviewers to look at
>>>>>>>>>>>>>>> it if
>>>>>>>>>>>>>>> you can tidy up the KIP document so that it stands on its own.
>>>>>>>>>>>>>>> People
>>>>>>>>>>>>>>> shouldn't have to look at any other document to understand the
>>>>>>>>>>>>>>> motivation of the proposal, and they shouldn't have to look at
>>>>>>>>>>>>>>> a PR to
>>>>>>>>>>>>>>> see what the public API will look like. If it helps, you can
>>>>>>>>>>>>>>> take a
>>>>>>>>>>>>>>> look at some other recent KIPs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Given that the list serde needs an inner serde, I agree you
>>>>>>>>>>>>>>> can't have
>>>>>>>>>>>>>>> a zero-argument static factory method for it, but it seems you
>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>> still have a static method:
>>>>>>>>>>>>>>> `public static Serde<List<T>> List(Serde<T> innerSerde)`.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, May 7, 2019 at 12:18 PM Development <d...@yeralin.net
>>>>>>>>>>>>>>> <mailto:
>>>>>>>>>>> d...@yeralin.net>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Absolutely agree. Already pushed another commit to remove
>>>>>>>>>>>>>>>> comparator
>>>>>>>>>>> argument: https://github.com/apache/kafka/pull/6592 <
>>>>>>>>>>> https://github.com/apache/kafka/pull/6592> <
>>>>>>>>>>> https://github.com/apache/kafka/pull/6592 <
>>>>>>>>>>> https://github.com/apache/kafka/pull/6592>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you for your input John! I really appreciate it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What about this point I made:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. Since type for List serde needs to be declared before hand,
>>>>>>>>>>>>>>>> I
>>>>>>>>>>> could not create a static method for List Serde under
>>>>>>>>>>> org.apache.kafka.common.serialization.Serdes. I addressed it in the
>>>>>>>>>>> KIP:
>>>>>>>>>>>>>>>> P.S. Static method corresponding to ListSerde under
>>>>>>>>>>> org.apache.kafka.common.serialization.Serdes (something like static
>>>>>>>>>>> public
>>>>>>>>>>> Serde<List<T>> List() {...}
>>>>>>>>>>> inorg.apache.kafka.common.serialization.Serdes)
>>>>>>>>>>> class cannot be added because type needs to be defined beforehand.
>>>>>>>>>>> That's
>>>>>>>>>>> why one needs to create List Serde in the following fashion:
>>>>>>>>>>>>>>>> new Serdes.ListSerde<String>(Serdes.String(),
>>>>>>>>>>> Comparator.comparing(String::length));
>>>>>>>>>>>>>>>> (can possibly be simplified by declaring import static
>>>>>>>>>>> org.apache.kafka.common.serialization.Serdes.ListSerde)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On May 7, 2019, at 11:50 AM, John Roesler <j...@confluent.io
>>>>>>>>>>> <mailto:j...@confluent.io>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the reply Daniyar,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That makes much more sense! I thought I must be missing
>>>>>>>>>>>>>>>>> something,
>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>> couldn't for the life of me figure it out.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What do you think about just taking an argument, instead of
>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>> Comparator, for the Serde of the inner type? That way, the
>>>>>>>>>>>>>>>>> user can
>>>>>>>>>>> control
>>>>>>>>>>>>>>>>> how exactly the inner data gets serialized, while also
>>>>>>>>>>>>>>>>> bounding the
>>>>>>>>>>> generic
>>>>>>>>>>>>>>>>> parameter properly. As for the order, since the list is
>>>>>>>>>>>>>>>>> already in a
>>>>>>>>>>>>>>>>> specific order, which the user themselves controls, it
>>>>>>>>>>>>>>>>> doesn't seem
>>>>>>>>>>>>>>>>> strictly necessary to offer an option to sort the data during
>>>>>>>>>>> serialization.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, May 6, 2019 at 8:47 PM Development <d...@yeralin.net
>>>>>>>>>>> <mailto:d...@yeralin.net>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi John,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I’m really sorry for the confusion. I cloned that JIRA
>>>>>>>>>>>>>>>>>> ticket from
>>>>>>>>>>> an old
>>>>>>>>>>>>>>>>>> one about introducing UUID Serde, and I guess was too hasty
>>>>>>>>>>>>>>>>>> while
>>>>>>>>>>> editing
>>>>>>>>>>>>>>>>>> the copy to notice the mistake. Just edited the ticket.
>>>>>>>>>>>>>>>>>> Sorry for
>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>> inconvenience .
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As per comparator, I agree. Let’s make user be responsible
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> implementing comparable interface. I was just thinking to
>>>>>>>>>>>>>>>>>> make the
>>>>>>>>>>> serde a
>>>>>>>>>>>>>>>>>> little more flexible (i.e. let user decide in which order
>>>>>>>>>>>>>>>>>> records
>>>>>>>>>>> is going
>>>>>>>>>>>>>>>>>> to be inserted into a change log topic).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Daniyar Yeralin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On May 6, 2019, at 5:37 PM, John Roesler <j...@confluent.io
>>>>>>>>>>> <mailto:j...@confluent.io>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Daniyar,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the proposal!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If I understand the point about the comparator, is it just
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>> capture the
>>>>>>>>>>>>>>>>>>> generic type parameter? If so, then anything that
>>>>>>>>>>>>>>>>>>> implements a
>>>>>>>>>>> known
>>>>>>>>>>>>>>>>>>> interface would work just as well, right? I've been
>>>>>>>>>>>>>>>>>>> considering
>>>>>>>>>>> adding
>>>>>>>>>>>>>>>>>>> something like the Jackson TypeReference (or similar
>>>>>>>>>>>>>>>>>>> classes in
>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>> projects). Would this be a good time to do it?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Note that it's not necessary to actually require that the
>>>>>>>>>>> captured type
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> Comparable (as this proposal currently does), it's just a
>>>>>>>>>>>>>>>>>>> way to
>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>> sure
>>>>>>>>>>>>>>>>>>> there is some method that makes use of the generic type
>>>>>>>>>>> parameter, to
>>>>>>>>>>>>>>>>>> force
>>>>>>>>>>>>>>>>>>> the compiler to capture the type.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Just to make sure I understand the motivation... You
>>>>>>>>>>>>>>>>>>> expressed a
>>>>>>>>>>> desire
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> be able to serialize UUIDs, which I didn't follow, since
>>>>>>>>>>>>>>>>>>> there is
>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> built-in UUID serde:
>>>>>>>>>>> org.apache.kafka.common.serialization.Serdes#UUID,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> also, a UUID isn't a List. Did you mean that you need to use
>>>>>>>>>>> *lists of*
>>>>>>>>>>>>>>>>>>> UUIDs?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, May 6, 2019 at 11:49 AM Development
>>>>>>>>>>>>>>>>>>> <d...@yeralin.net
>>>>>>>>>>> <mailto:d...@yeralin.net>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Starting a discussion for KIP-466 adding support for List
>>>>>>>>>>>>>>>>>>>> Serde.
>>>>>>>>>>> PR is
>>>>>>>>>>>>>>>>>>>> created under https://github.com/apache/kafka/pull/6592 <
>>>>>>>>>>> https://github.com/apache/kafka/pull/6592> <
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/6592 <
>>>>>>>>>>> https://github.com/apache/kafka/pull/6592>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> There are two topics I would like to discuss:
>>>>>>>>>>>>>>>>>>>> 1. Since type for List serve needs to be declared before
>>>>>>>>>>>>>>>>>>>> hand, I
>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> create a static method for List Serde under
>>>>>>>>>>>>>>>>>>>> org.apache.kafka.common.serialization.Serdes. I addressed
>>>>>>>>>>>>>>>>>>>> it in
>>>>>>>>>>> the KIP:
>>>>>>>>>>>>>>>>>>>> P.S. Static method corresponding to ListSerde under
>>>>>>>>>>>>>>>>>>>> org.apache.kafka.common.serialization.Serdes (something
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>> static
>>>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>>>>>> Serde<List<T>> List() {...}
>>>>>>>>>>>>>>>>>> inorg.apache.kafka.common.serialization.Serdes)
>>>>>>>>>>>>>>>>>>>> class cannot be added because type needs to be defined
>>>>>>>>>>> beforehand.
>>>>>>>>>>>>>>>>>> That's
>>>>>>>>>>>>>>>>>>>> why one needs to create List Serde in the following
>>>>>>>>>>>>>>>>>>>> fashion:
>>>>>>>>>>>>>>>>>>>> new Serdes.ListSerde<String>(Serdes.String(),
>>>>>>>>>>>>>>>>>>>> Comparator.comparing(String::length));
>>>>>>>>>>>>>>>>>>>> (can possibly be simplified by declaring import static
>>>>>>>>>>>>>>>>>>>> org.apache.kafka.common.serialization.Serdes.ListSerde)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2. @miguno Michael G. Noll <https://github.com/miguno <
>>>>>>>>>>> https://github.com/miguno>> is questioning
>>>>>>>>>>>>>>>>>>>> whether I need to pass a comparator to ListDeserializer.
>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>> certainly
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> not required. Feel free to add your input:
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/kafka/pull/6592#discussion_r281152067
>>>>>>>>>>> <https://github.com/apache/kafka/pull/6592#discussion_r281152067>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> Daniyar Yeralin
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On May 6, 2019, at 11:59 AM, Daniyar Yeralin (JIRA) <
>>>>>>>>>>> j...@apache.org <mailto:j...@apache.org>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Daniyar Yeralin created KAFKA-8326:
>>>>>>>>>>>>>>>>>>>>> --------------------------------------
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Summary: Add List<T> Serde
>>>>>>>>>>>>>>>>>>>>> Key: KAFKA-8326
>>>>>>>>>>>>>>>>>>>>> URL:
>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-8326 <
>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-8326>
>>>>>>>>>>>>>>>>>>>>> Project: Kafka
>>>>>>>>>>>>>>>>>>>>> Issue Type: Improvement
>>>>>>>>>>>>>>>>>>>>> Components: clients, streams
>>>>>>>>>>>>>>>>>>>>> Reporter: Daniyar Yeralin
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I propose adding serializers and deserializers for the
>>>>>>>>>>> java.util.List
>>>>>>>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I have many use cases where I want to set the key of a
>>>>>>>>>>>>>>>>>>>>> Kafka
>>>>>>>>>>> message to
>>>>>>>>>>>>>>>>>>>> be a UUID. Currently, I need to turn UUIDs into strings or
>>>>>>>>>>>>>>>>>>>> byte
>>>>>>>>>>> arrays
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> use their associated Serdes, but it would be more
>>>>>>>>>>>>>>>>>>>> convenient to
>>>>>>>>>>>>>>>>>> serialize
>>>>>>>>>>>>>>>>>>>> and deserialize UUIDs directly.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I believe there are many use cases where one would want
>>>>>>>>>>>>>>>>>>>>> to have
>>>>>>>>>>> a List
>>>>>>>>>>>>>>>>>>>> serde. Ex. [
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>> https://stackoverflow.com/questions/41427174/aggregate-java-objects-in-a-list-with-kafka-streams-dsl-windows
>>>>>>>>>>> <
>>>>>>>>>>> https://stackoverflow.com/questions/41427174/aggregate-java-objects-in-a-list-with-kafka-streams-dsl-windows
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ],
>>>>>>>>>>>>>>>>>>>> [
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>> https://stackoverflow.com/questions/46365884/issue-with-arraylist-serde-in-kafka-streams-api
>>>>>>>>>>> <
>>>>>>>>>>> https://stackoverflow.com/questions/46365884/issue-with-arraylist-serde-in-kafka-streams-api
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> KIP Link: [
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization
>>>>>>>>>>> <
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> This message was sent by Atlassian JIRA
>>>>>>>>>>>>>>>>>>>>> (v7.6.3#76005)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>
>
