[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526936#comment-16526936 ] Richard Yu commented on SPARK-18258: {quote} * We need agreement on whether it is worth making a

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525883#comment-16525883 ] Richard Yu commented on SPARK-18258: [~c...@koeninger.org] I have a bit of a concern regarding where

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525779#comment-16525779 ] Apache Spark commented on SPARK-18258: -- User 'ConcurrencyPractitioner' has created a pull request

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525137#comment-16525137 ] Cody Koeninger commented on SPARK-18258: [~Yohan123] This ticket is about giving implementors of

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524921#comment-16524921 ] Richard Yu commented on SPARK-18258: Just a question, I noticed that in {{KafkaSink}}'s particular

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15639002#comment-15639002 ] Liwei Lin commented on SPARK-18258: --- I think Cody refers to the {{OffsetSeq}} from

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638887#comment-15638887 ] Reynold Xin commented on SPARK-18258: - What is OffsetSeq? > Sinks need access to offset

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638555#comment-15638555 ] Cody Koeninger commented on SPARK-18258: Sure, added, let me know if I'm missing something or can

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638470#comment-15638470 ] Reynold Xin commented on SPARK-18258: - This makes sense. It's just extra information you want to be

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637621#comment-15637621 ] Cody Koeninger commented on SPARK-18258: So one obvious one is that if wherever checkpoint data

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637595#comment-15637595 ] Michael Armbrust commented on SPARK-18258: -- What sort of failures are you anticipating here? >

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637576#comment-15637576 ] Cody Koeninger commented on SPARK-18258: The sink doesn't have to reason about equality of the

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637491#comment-15637491 ] Michael Armbrust commented on SPARK-18258: -- I agree that we don't want to lock people in, which