[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-09 Thread Ofir Manor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560906#comment-15560906 ] Ofir Manor commented on SPARK-17815: I totally understand your concerns, and yes, I hope you will

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-09 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560837#comment-15560837 ] Cody Koeninger commented on SPARK-17815: My personal concerns about complexity are because I'm

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-09 Thread Ofir Manor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560749#comment-15560749 ] Ofir Manor commented on SPARK-17815: Thanks Cody, this is much clearer. (BTW - I've been bitten

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-09 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560684#comment-15560684 ] Cody Koeninger commented on SPARK-17815: Regarding kafka consumer behavior, I'm not saying it's

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-09 Thread Ofir Manor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560661#comment-15560661 ] Ofir Manor commented on SPARK-17815: I hope others with better understanding of Spark internals can

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-09 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559911#comment-15559911 ] Cody Koeninger commented on SPARK-17815: The WAL cannot be the only source of truth, because it

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-09 Thread Ofir Manor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559549#comment-15559549 ] Ofir Manor commented on SPARK-17815: Good, we are on the same page, no argument that SPARK-16963

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-09 Thread Ofir Manor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559541#comment-15559541 ] Ofir Manor commented on SPARK-17815: As far as I understand, there is a clear single-point-of-truth:

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-08 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558528#comment-15558528 ] Cody Koeninger commented on SPARK-17815: So if you start committing offsets to kafka, there are

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553900#comment-15553900 ] Michael Armbrust commented on SPARK-17815: -- The only reason its blocked is because setting it

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-06 Thread Ofir Manor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553662#comment-15553662 ] Ofir Manor commented on SPARK-17815: I think this is a good idea.There is a minor confusion here,