If you have a reproduction you should open a JIRA. It would be great if there is a fix. I'm just saying I know a similar issue does not exist in structured streaming.
On Fri, Mar 10, 2017 at 7:46 AM, Justin Miller < justin.mil...@protectwise.com> wrote: > Hi Michael, > > I'm experiencing a similar issue. Will this not be fixed in Spark > Streaming? > > Best, > Justin > > On Mar 10, 2017, at 8:34 AM, Michael Armbrust <mich...@databricks.com> > wrote: > > One option here would be to try Structured Streaming. We've added an > option "failOnDataLoss" that will cause Spark to just skip a head when this > exception is encountered (its off by default though so you don't silently > miss data). > > On Fri, Mar 18, 2016 at 4:16 AM, Ramkumar Venkataraman < > ram.the.m...@gmail.com> wrote: > >> I am using Spark streaming and reading data from Kafka using >> KafkaUtils.createDirectStream. I have the "auto.offset.reset" set to >> smallest. >> >> But in some Kafka partitions, I get kafka.common.OffsetOutOfRangeE >> xception >> and my spark job crashes. >> >> I want to understand if there is a graceful way to handle this failure and >> not kill the job. I want to keep ignoring these exceptions, as some other >> partitions are fine and I am okay with data loss. >> >> Is there any way to handle this and not have my spark job crash? I have no >> option of increasing the kafka retention period. >> >> I tried to have the DStream returned by createDirectStream() wrapped in a >> Try construct, but since the exception happens in the executor, the Try >> construct didn't take effect. Do you have any ideas of how to handle this? >> >> >> >> -- >> View this message in context: http://apache-spark-user-list. >> 1001560.n3.nabble.com/How-to-gracefully-handle-Kafka-OffsetO >> utOfRangeException-tp26534.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > >
