GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/1308
[FLINK-2800] [kryo] Fix Kryo serialization to clear buffered data
The Kryo serializer uses Kryo's Output class to buffer individual write
operations before
it is written to the underlying output stream. This Output class is flushed
by Flink's
KryoSerializer upon finishing its serialize call. However, in case of an
exception when
flushing the Output, the buffered data is kept in the buffer. Since Flink
uses EOFExceptions
to mark that an underlying buffer is full and has to be spilled, for
example, it can happen
that the record triggering the spilling is written twice after it is
rewritten. The reason
is that Kryo's Output buffer still contains the serialization data of the
failed attempt which
is also flushed to the emptied output stream.
This duplication of records can lead to corrupted data which eventually
let's the Flink program
crash. The problem is solved by clearing Kryo's Output when the flush
operation was not successful.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink fixKryoSerialization
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1308.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1308
----
commit 5618cf7ff65c972dc33c77ba953966224e8c2a1e
Author: Till Rohrmann <[email protected]>
Date: 2015-10-28T17:40:41Z
[FLINK-2800] [kryo] Fix Kryo serialization to clear buffered data
The Kryo serializer uses Kryo's Output class to buffer individual write
operations before
it is written to the underlying output stream. This Output class is flushed
by Flink's
KryoSerializer upon finishing its serialize call. However, in case of an
exception when
flushing the Output, the buffered data is kept in the buffer. Since Flink
uses EOFExceptions
to mark that an underlying buffer is full and has to be spilled, for
example, it can happen
that the record triggering the spilling is written twice after it is
rewritten. The reason
is that Kryo's Output buffer still contains the serialization data of the
failed attempt which
is also flushed to the emptied output stream.
This duplication of records can lead to corrupted data which eventually
let's the Flink program
crash. The problem is solved by clearing Kryo's Output when the flush
operation was not successful.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
