Re: HybridMemorySegment index out of bounds exception

2021-01-28 Thread Timo Walther
FYI: Yuval and I scheduled a call to investigate this serialization issue remotely on Monday. If you have any idea by looking at the code beforehand, let us know. On 28.01.21 16:57, Yuval Itzchakov wrote: Hi Timo, The code example I posted doesn't really match the code that is causing this

Re: HybridMemorySegment index out of bounds exception

2021-01-28 Thread Yuval Itzchakov
Hi Timo, The code example I posted doesn't really match the code that is causing this issue. I tried to extend it a bit but couldn't make the reproduction work there. I am no longer using the serialized strings, but registering the custom serializers with the runtime during bootstrap and

Re: HybridMemorySegment index out of bounds exception

2021-01-28 Thread Timo Walther
This is helpful information. So I guess the problem must be in the flink-table module and not in flink-core. I will try to reserve some time tomorrow to look into the code again. How did you express RawType(Array[String])? Again with fully serialized type string? Could it be related to

Re: HybridMemorySegment index out of bounds exception

2021-01-28 Thread Yuval Itzchakov
Hi Timo, I tried replacing it with an ordinary ARRAY DataType, which doesn't reproduce the issue. If I use a RawType(Array[String]), the problem still manifests, so I assume it's not directly related to a Kryo serialization of the specific underlying type (io.circe.Json), but something in the way

Re: HybridMemorySegment index out of bounds exception

2021-01-28 Thread Timo Walther
Hi Yuval, we should definitely find the root cause of this issue. It helps if the exception happens frequently to nail down the problem. Have you tried to replace the JSON object with a regular String? If the exception is gone after this change. I believe it must be the serialization and