Hi Dave,
maybe it would be better to execute your tests against a local cluster
instead of the mini cluster. Also object reuse should be disabled and
chaining should be disabled to force serialization.
Maybe others have better ideas.
Regards,
Timo
On 30.04.21 10:25, Dave Maughan wrote:
Hi,
I recently encountered a scenario where the data type being passed
between operators in my streaming job was modified such that it broke
serialization. This was due to a non-Avro top-level data type containing
an Avro field. The existing integration test (mini cluster) continued to
work and unit tests that attempted to cover Kryo serialization continued
to work, but when deployed to a real cluster it failed. The problem was
easily solved but in future I'd like to catch problems like this in my
testing.
Is there a way to force serialization always between all operators in
the mini-cluster? Or is there another strategy I can apply to exercise
the serialization of my data types?
Thanks,
Dave
