Jamie Grier created FLINK-3679:
----------------------------------
Summary: DeserializationSchema should handle zero or more outputs
for every input
Key: FLINK-3679
URL: https://issues.apache.org/jira/browse/FLINK-3679
Project: Flink
Issue Type: Bug
Components: DataStream API
Reporter: Jamie Grier
There are a couple of issues with the DeserializationSchema API that I think
should be improved. This request has come to me via an existing Flink user.
The main issue is simply that the API assumes that there is a one-to-one
mapping between input and outputs. In reality there are scenarios where one
input message (say from Kafka) might actually map to zero or more logical
elements in the pipeline.
Particularly important here is the case where you receive a message from a
source (such as Kafka) and say the raw bytes don't deserialize properly. Right
now the only recourse is to throw IOException and therefore fail the job.
This is definitely not good since bad data is a reality and failing the job is
not the right option. If the job fails we'll just end up replaying the bad
data and the whole thing will start again.
Instead in this case it would be best if the user could just return the empty
set.
The other case is where one input message should logically be multiple output
messages. This case is probably less important since there are other ways to
do this but in general it might be good to make the
DeserializationSchema.deserialize() method return a collection rather than a
single element.
Maybe we need to support a DeserializationSchema variant that has semantics
more like that of FlatMap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
