[jira] [Commented] (SPARK-12306) Add an option to ignore BlockRDD partition data loss
[ https://issues.apache.org/jira/browse/SPARK-12306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15054995#comment-15054995 ] Liwei Lin commented on SPARK-12306: --- Sorry for the inconvenience; I've added some detailed description, [~srowen] would you mind reopen it? Thanks. :-) > Add an option to ignore BlockRDD partition data loss > > > Key: SPARK-12306 > URL: https://issues.apache.org/jira/browse/SPARK-12306 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.5.2 >Reporter: Liwei Lin > > Currently in Spark Streaming, a Receiver stores the received data into some > BlockManager and then later the data will be used by a BlockRDD. If this > BlockManager were to lost because of some failure, then this BlockRDD would > throw a SparkException saying "Could not compute split, block not found". > In most cases this is the right thing to do. However, in a streaming scenario > where it can tolerant small pieces of data loss, maybe just move on silently > -- instead of throwing an exception -- is more preferable. > This issue proposes to add such an "spark.streaming.ignoreBlockNotFound" > option, which defaults to false, to tell whether to throw an exception or > just move on when a block is not found. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12306) Add an option to ignore BlockRDD partition data loss
[ https://issues.apache.org/jira/browse/SPARK-12306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15054184#comment-15054184 ] Liwei Lin commented on SPARK-12306: --- This is reported by me, and I'm working on it. :-) > Add an option to ignore BlockRDD partition data loss > > > Key: SPARK-12306 > URL: https://issues.apache.org/jira/browse/SPARK-12306 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.5.2 >Reporter: Liwei Lin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12306) Add an option to ignore BlockRDD partition data loss
[ https://issues.apache.org/jira/browse/SPARK-12306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15054199#comment-15054199 ] Sean Owen commented on SPARK-12306: --- [~proflin] there's no detail here. It doesn't sound like something that should be optionally ignored. I'd have to close this unless you can make a case for this, or at least describe it. > Add an option to ignore BlockRDD partition data loss > > > Key: SPARK-12306 > URL: https://issues.apache.org/jira/browse/SPARK-12306 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.5.2 >Reporter: Liwei Lin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
