[
https://issues.apache.org/jira/browse/FLINK-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15081171#comment-15081171
]
ASF GitHub Bot commented on FLINK-3188:
---------------------------------------
GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/1484
[FLINK-3188] Pass deletes to KeyedDeserializationSchema
I've converted the two patches from
https://issues.apache.org/jira/browse/FLINK-3188 into this pull request.
I removed the manual serializer creation from the patches and replaced them
by a `TypeInformationKeyValueSerializationSchema`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink flink3188
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1484.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1484
----
commit 4bd36b45ab53940e63dbbc98e21fe51ac4f7a3ad
Author: Sebastian Klemke <[email protected]>
Date: 2016-01-04T13:52:52Z
[FLINK-3188] Pass deletes to KeyedDeserializationSchema
----
> Deletes in Kafka source should be passed on to KeyedDeserializationSchema
> -------------------------------------------------------------------------
>
> Key: FLINK-3188
> URL: https://issues.apache.org/jira/browse/FLINK-3188
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
> Affects Versions: 1.0.0
> Reporter: Sebastian Klemke
> Assignee: Robert Metzger
> Attachments: kafka-deletions.patch, testcase.patch
>
>
> When keys are deleted in the kafka queue, they show up as keys with null
> payload. Currently in Flink 1.0-SNAPSHOT, these deletions are silently
> skipped, without increasing current offset.
> This leads to two problems:
> 1. When a fetch window contains only deletions, LegacyFetcher gets stuck
> 2. For KeyedDeserializationSchemas, it would make sense to pass deletions to
> the deserializer, so that it can decide to wrap deleted keys as a deletion
> command. This is also more consistent with the semantics of keys in Kafka
> queues: When compaction is activated, only the latest message with the same
> key needs to be kept by Kafka.
> We propose the attached patch as a workaround for both issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
