I mean the `StreamingExecution` generated a proper error message: 2017-08-26 07:05:00,641 ERROR StreamExecution:? - Query [id = 8597ae0b-2183-407f-8300-239a24eb68ab, runId = c1fe627d-bcf4-4462-bbd9-b178ffaca860] terminated with error org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 4 times, most recent failure: Lost task 2.3 in stage 1.0 (TID 33, XX.XX.XX.XX, executor 8): java.io.EOFException: Stream ended prematurely
But the driver was still alive, thus the application was still running. When you say "it should not affect the termination of the application whether the queries are active or not", do you mean that if the streaming query engine is no longer running, the application itself should not be affected? That sounds counter intuitive if all the application is doing is consuming from the source to apply transformations on incoming data. On Tue, Aug 29, 2017 at 12:21 PM, Tathagata Das <tathagata.das1...@gmail.com > wrote: > When you say "the application remained alive", do you mean the > StreamingQuery stayed alive, or the whole process stayed alive? The > StreamingQuery should be terminated immediately. And the stream execution > threads are all daemon threads, so it should not affect the termination of > the application whether the queries are active or not. May be something > else is keeping the application alive? > > > > On Tue, Aug 29, 2017 at 2:09 AM, Yuval Itzchakov <yuva...@gmail.com> > wrote: > >> I wasn't sure if this would be a proper bug or not. >> >> Today, the behavior of Structured Streaming is such that if a source >> fails with an exception, the `StreamExecution` class halts reading further >> from the source, but the application is remained alive. For applications >> where the sole purpose is to transform data from a non static source (such >> as Kafka), this is rather useless and might be surprising. >> >> For example, if you have a simple monitor which checks whether the >> application is alive or not, you'll still get reports that the application >> is alive and running, but actually it isn't consuming anything from the >> source and is logically dead. >> >> Should this be the behavior? I think that perhaps there should be a >> configuration that asks whether to completely shutdown the application on >> source failure. >> >> What do you guys think? >> >> -- >> Best Regards, >> Yuval Itzchakov. >> > > -- Best Regards, Yuval Itzchakov.
