[jira] [Commented] (FLINK-4796) Add new Sink interface with access to more meta data
[ https://issues.apache.org/jira/browse/FLINK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166443#comment-16166443 ] Erik van Oosten commented on FLINK-4796: I am not sure why this is marked as a duplicate. The problem here is inconsistent handling of the runtime context inside the different layers under FlinkKafkaProducer: method {{getRuntimeContext}} gives {{null}} even though {{setRuntimeContext}} was called. How does that relate to the addition of a new interface? > Add new Sink interface with access to more meta data > > > Key: FLINK-4796 > URL: https://issues.apache.org/jira/browse/FLINK-4796 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Affects Versions: 1.2.0 >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > > The current {{SinkFunction}} cannot access the timestamps of elements which > resulted in the (somewhat hacky) {{FlinkKafkaProducer010}}. Due to other > limitations {{GenericWriteAheadSink}} is currently also a {{StreamOperator}} > and not a {{SinkFunction}}. > We should add a new interface for sinks that takes a context parameter, > similar to {{ProcessFunction}}. This will allow sinks to query additional > meta data about the element that they're receiving. > This is one ML thread where a user ran into a problem caused by this: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Why-I-am-getting-Null-pointer-exception-while-accessing-RuntimeContext-in-FlinkKafkaProducer010-td12633.html#a12635 > h3. Original Text (that is still valid but not general) > The Kafka 0.10 connector supports writing event timestamps to Kafka. > Currently, the regular DataStream APIs don't allow user code to access the > event timestamp easily. That's why the Kafka connector is using a custom > operator ({{transform()}}) to access the event time. > With this JIRA, I would like to provide the event timestamp in the regular > DataStream APIs. > Once I'll look into the issue, I'll post some proposals how to add the > timestamp. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-4796) Add new Sink interface with access to more meta data
[ https://issues.apache.org/jira/browse/FLINK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166394#comment-16166394 ] Aljoscha Krettek commented on FLINK-4796: - Ah I see, the workaround is for the NPE. > Add new Sink interface with access to more meta data > > > Key: FLINK-4796 > URL: https://issues.apache.org/jira/browse/FLINK-4796 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Affects Versions: 1.2.0 >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > > The current {{SinkFunction}} cannot access the timestamps of elements which > resulted in the (somewhat hacky) {{FlinkKafkaProducer010}}. Due to other > limitations {{GenericWriteAheadSink}} is currently also a {{StreamOperator}} > and not a {{SinkFunction}}. > We should add a new interface for sinks that takes a context parameter, > similar to {{ProcessFunction}}. This will allow sinks to query additional > meta data about the element that they're receiving. > This is one ML thread where a user ran into a problem caused by this: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Why-I-am-getting-Null-pointer-exception-while-accessing-RuntimeContext-in-FlinkKafkaProducer010-td12633.html#a12635 > h3. Original Text (that is still valid but not general) > The Kafka 0.10 connector supports writing event timestamps to Kafka. > Currently, the regular DataStream APIs don't allow user code to access the > event timestamp easily. That's why the Kafka connector is using a custom > operator ({{transform()}}) to access the event time. > With this JIRA, I would like to provide the event timestamp in the regular > DataStream APIs. > Once I'll look into the issue, I'll post some proposals how to add the > timestamp. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-4796) Add new Sink interface with access to more meta data
[ https://issues.apache.org/jira/browse/FLINK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166391#comment-16166391 ] Aljoscha Krettek commented on FLINK-4796: - Sorry, I didn't get what this workaround is for. Could you please go into a bit more detail? > Add new Sink interface with access to more meta data > > > Key: FLINK-4796 > URL: https://issues.apache.org/jira/browse/FLINK-4796 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Affects Versions: 1.2.0 >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > > The current {{SinkFunction}} cannot access the timestamps of elements which > resulted in the (somewhat hacky) {{FlinkKafkaProducer010}}. Due to other > limitations {{GenericWriteAheadSink}} is currently also a {{StreamOperator}} > and not a {{SinkFunction}}. > We should add a new interface for sinks that takes a context parameter, > similar to {{ProcessFunction}}. This will allow sinks to query additional > meta data about the element that they're receiving. > This is one ML thread where a user ran into a problem caused by this: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Why-I-am-getting-Null-pointer-exception-while-accessing-RuntimeContext-in-FlinkKafkaProducer010-td12633.html#a12635 > h3. Original Text (that is still valid but not general) > The Kafka 0.10 connector supports writing event timestamps to Kafka. > Currently, the regular DataStream APIs don't allow user code to access the > event timestamp easily. That's why the Kafka connector is using a custom > operator ({{transform()}}) to access the event time. > With this JIRA, I would like to provide the event timestamp in the regular > DataStream APIs. > Once I'll look into the issue, I'll post some proposals how to add the > timestamp. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-4796) Add new Sink interface with access to more meta data
[ https://issues.apache.org/jira/browse/FLINK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160810#comment-16160810 ] Erik van Oosten commented on FLINK-4796: A workaround is to override {{setRuntimeContext}} (make sure to call {{super.setRuntimeContext}}), and use the passed in context. Possibly store it in a private field for later access. > Add new Sink interface with access to more meta data > > > Key: FLINK-4796 > URL: https://issues.apache.org/jira/browse/FLINK-4796 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Affects Versions: 1.2.0 >Reporter: Robert Metzger > > The current {{SinkFunction}} cannot access the timestamps of elements which > resulted in the (somewhat hacky) {{FlinkKafkaProducer010}}. Due to other > limitations {{GenericWriteAheadSink}} is currently also a {{StreamOperator}} > and not a {{SinkFunction}}. > We should add a new interface for sinks that takes a context parameter, > similar to {{ProcessFunction}}. This will allow sinks to query additional > meta data about the element that they're receiving. > This is one ML thread where a user ran into a problem caused by this: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Why-I-am-getting-Null-pointer-exception-while-accessing-RuntimeContext-in-FlinkKafkaProducer010-td12633.html#a12635 > h3. Original Text (that is still valid but not general) > The Kafka 0.10 connector supports writing event timestamps to Kafka. > Currently, the regular DataStream APIs don't allow user code to access the > event timestamp easily. That's why the Kafka connector is using a custom > operator ({{transform()}}) to access the event time. > With this JIRA, I would like to provide the event timestamp in the regular > DataStream APIs. > Once I'll look into the issue, I'll post some proposals how to add the > timestamp. -- This message was sent by Atlassian JIRA (v6.4.14#64029)