We had a similar issue while working on one of our usecase where we were
processing at a moderate throughput (around 500MB/S). When the processing
time exceeds the batch duration, it started to throw up blocknotfound
exceptions, i made a workaround for that issue and is explained over here
http://a
Hi guys,
We[1] are doing a bit of work on Spark Streaming, to help it face
situations where the throughput of data on an InputStream is (momentarily)
susceptible to overwhelm the Receiver(s) memory.
The JIRA & design doc is here:
https://issues.apache.org/jira/browse/SPARK-7398
We'd sure appreci
