Re: Need help in understanding PojoSerializer
Hi Sachin,
Check the last few comments I wrote in this thread:
https://lists.apache.org/thread/l71d1cqo9xv8rsw0gfjo19kb1pct2xj1
Regards,
Alexis.
On Wed, 20 Mar 2024, 18:51 Sachin Mittal, wrote:
> Hi,
> I saw the post but I did not understand how I would configure these fields
> to use those serializers. (I can change the set type to a list type for
> now).
> As per the docs I see that we can annotate fields with @TypeInfo
>
> But what I did not get is how using this annotation I can use ListSerializer
> and *MapSerializer.*
>
> Thanks
> Sachin
>
>
> On Wed, Mar 20, 2024 at 10:47 PM Ken Krugler
> wrote:
>
>> Flink doesn’t have built-in support for serializing Sets.
>>
>> See this (stale) issue about the same:
>> https://issues.apache.org/jira/browse/FLINK-16729
>>
>> You could create a custom serializer for sets, see
>> https://stackoverflow.com/questions/59800851/flink-serialization-of-java-util-list-and-java-util-map
>> and
>> https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/ListSerializer.html
>> for details on how this was done for a list, but it’s not trivial.
>>
>> Or as a hack, use a Map and the existing support for map
>> serialization via
>> https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/MapSerializer.html
>>
>> — Ken
>>
>>
>> On Mar 20, 2024, at 10:04 AM, Sachin Mittal wrote:
>>
>> Hi,
>> I have a Pojo class like this
>>
>> public class A {
>>
>> public String str;
>>
>> public Set aSet;
>>
>> public Map dMap;
>>
>> }
>>
>> However when I start the flink program I get this message:
>>
>> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
>> dMap will be processed as GenericType. Please read the Flink
>> documentation on "Data Types & Serialization" for details of the effect on
>> performance and schema evolution.
>>
>> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
>> aSet will be processed as GenericType. Please read the Flink
>> documentation on "Data Types & Serialization" for details of the effect on
>> performance and schema evolution.
>>
>> Also in my code I have added
>>
>> env.getConfig().disableGenericTypes();
>>
>> So I don't understand when I use Maps and Sets of primitive types why is
>> Flink not
>>
>> able to use PojoSerializer for these fields and even when I have disabled
>> generics types.
>>
>> why I am getting message that it will be processed as GenericType?
>>
>>
>> Any help in understanding what I need to do to ensure all the fields of my
>> object are handled using PojoSerializer.
>>
>>
>> Thanks
>>
>> Sachin
>>
>>
>>
>> --
>> Ken Krugler
>> http://www.scaleunlimited.com
>> Custom big data solutions
>> Flink & Pinot
>>
>>
>>
>>
Re: Need help in understanding PojoSerializer
Hi,
I saw the post but I did not understand how I would configure these fields
to use those serializers. (I can change the set type to a list type for
now).
As per the docs I see that we can annotate fields with @TypeInfo
But what I did not get is how using this annotation I can use ListSerializer
and *MapSerializer.*
Thanks
Sachin
On Wed, Mar 20, 2024 at 10:47 PM Ken Krugler
wrote:
> Flink doesn’t have built-in support for serializing Sets.
>
> See this (stale) issue about the same:
> https://issues.apache.org/jira/browse/FLINK-16729
>
> You could create a custom serializer for sets, see
> https://stackoverflow.com/questions/59800851/flink-serialization-of-java-util-list-and-java-util-map
> and
> https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/ListSerializer.html
> for details on how this was done for a list, but it’s not trivial.
>
> Or as a hack, use a Map and the existing support for map
> serialization via
> https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/MapSerializer.html
>
> — Ken
>
>
> On Mar 20, 2024, at 10:04 AM, Sachin Mittal wrote:
>
> Hi,
> I have a Pojo class like this
>
> public class A {
>
> public String str;
>
> public Set aSet;
>
> public Map dMap;
>
> }
>
> However when I start the flink program I get this message:
>
> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
> dMap will be processed as GenericType. Please read the Flink
> documentation on "Data Types & Serialization" for details of the effect on
> performance and schema evolution.
>
> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
> aSet will be processed as GenericType. Please read the Flink
> documentation on "Data Types & Serialization" for details of the effect on
> performance and schema evolution.
>
> Also in my code I have added
>
> env.getConfig().disableGenericTypes();
>
> So I don't understand when I use Maps and Sets of primitive types why is
> Flink not
>
> able to use PojoSerializer for these fields and even when I have disabled
> generics types.
>
> why I am getting message that it will be processed as GenericType?
>
>
> Any help in understanding what I need to do to ensure all the fields of my
> object are handled using PojoSerializer.
>
>
> Thanks
>
> Sachin
>
>
>
> --
> Ken Krugler
> http://www.scaleunlimited.com
> Custom big data solutions
> Flink & Pinot
>
>
>
>
Re: Need help in understanding PojoSerializer
Flink doesn’t have built-in support for serializing Sets.
See this (stale) issue about the same:
https://issues.apache.org/jira/browse/FLINK-16729
You could create a custom serializer for sets, see
https://stackoverflow.com/questions/59800851/flink-serialization-of-java-util-list-and-java-util-map
and
https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/ListSerializer.html
for details on how this was done for a list, but it’s not trivial.
Or as a hack, use a Map and the existing support for map
serialization via
https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/MapSerializer.html
— Ken
> On Mar 20, 2024, at 10:04 AM, Sachin Mittal wrote:
>
> Hi,
> I have a Pojo class like this
>
> public class A {
> public String str;
> public Set aSet;
> public Map dMap;
> }
>
> However when I start the flink program I get this message:
>
> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field
> A#dMap will be processed as GenericType. Please read the Flink documentation
> on "Data Types & Serialization" for details of the effect on performance and
> schema evolution.
>
> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field
> A#aSet will be processed as GenericType. Please read the Flink documentation
> on "Data Types & Serialization" for details of the effect on performance and
> schema evolution.
>
> Also in my code I have added
> env.getConfig().disableGenericTypes();
> So I don't understand when I use Maps and Sets of primitive types why is
> Flink not
> able to use PojoSerializer for these fields and even when I have disabled
> generics types.
> why I am getting message that it will be processed as GenericType?
>
> Any help in understanding what I need to do to ensure all the fields of my
> object are handled using PojoSerializer.
>
> Thanks
> Sachin
>
>
--
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink & Pinot
Need help in understanding PojoSerializer
Hi,
I have a Pojo class like this
public class A {
public String str;
public Set aSet;
public Map dMap;
}
However when I start the flink program I get this message:
org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
dMap will be processed as GenericType. Please read the Flink documentation
on "Data Types & Serialization" for details of the effect on performance
and schema evolution.
org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
aSet will be processed as GenericType. Please read the Flink documentation
on "Data Types & Serialization" for details of the effect on performance
and schema evolution.
Also in my code I have added
env.getConfig().disableGenericTypes();
So I don't understand when I use Maps and Sets of primitive types why
is Flink not
able to use PojoSerializer for these fields and even when I have
disabled generics types.
why I am getting message that it will be processed as GenericType?
Any help in understanding what I need to do to ensure all the fields
of my object are handled using PojoSerializer.
Thanks
Sachin
