Github user ggevay commented on the issue:
https://github.com/apache/flink/pull/2211
Hi,
There were a number of bugs and painful parts in the code, and
unfortunately I'm not really sure whether they are major blockers or easily
fixable. It surely wouldn't be an 1-2 day work, but some weeks might be
realistic.
As an example from just off the top of my head, we choke in situations
where the POJO has a generic argument T, and a field of type T, and we tell
Flink with a `.returns` about the generic argument.
There were some other random test failures in older tests, which would
require some debugging.
I think we also have the problem that we don't correctly clean up the old
generated classes, so they cause a memory leak across Flink jobs.
Another potential issue is that since this PR, there was a new mechanism
added to the Flink serializers for dealing with compatibility of serializers
(e.g., to ensure that savepoints work across Flink versions), and I'm not sure
whether it would be straightforward to update this PR for this. (For example,
`PojoSerializer.ensureCompatibility` did not exist back then, and it looks a
bit scary at first glance.)
One of the pain points was shipping generated code from the driver program
to the TMs. The problem is that if you simply serialize an instance of a
generated class in the driver (for shipping the job to the cluster), the TM
cannot just simply deserialize it, since the generated class does not exist in
that JVM. The good news here is that since this PR I accidentally stumbled upon
a new solution to this (using `writeReplace` and `readResolve`), which would be
nicer than the current solution of this PR.
Note that there is also this PR for similar code generation for the sorters:
https://github.com/apache/flink/pull/3511
This has a lot of similarities, but less pain points, because the sorters
are better isolated from the other parts of Flink than the serializers, and
also because there we don't have to ship generated code from the driver to the
TMs (because the sorters are created on the TMs). So I think if we wanted to
push this serializer codegen PR, then first we should start with the sorter
codegen PR instead, because it is the easier one of the two. I think that one
is actually quite close to being mergable. (Note, that I think we solved the
"cleanup between Flink jobs" issue in that PR, and that solution could be
adapted to this PR.)
Btw. we are in the planning phase of an MSc thesis with (@mukrram-bajwa),
whose work might subsume both of these PRs. The idea is to use Graal to do many
of the specializations that are in these PRs. The nice thing would be that the
modifications to Flink's code would be much less compared to these PRs, as the
magic would happen mostly inside a Graal compilation plugin as custom
optimization passes.
Best,
Gábor
---
