Re: Spark Streaming - graceful shutdown when stream has no more data

t; > > > > > Back to original exception, blindly calling “Option.get” is always not a > > good practice. It would be better to pre-validate or use > > getOption/getOrElse. > > > > > > > > Thanks, > > > > William > > > > >

RE: Spark Streaming - graceful shutdown when stream has no more data

; Ted Yu; Femi Anthony; user Subject: RE: Spark Streaming - graceful shutdown when stream has no more data This is very interesting, how to shutdown the streaming job gracefully once no input data for some time. A doable solution probably you can count the input data by using the Accumulator

RE: Spark Streaming - graceful shutdown when stream has no more data

ih...@gmail.com>; Femi Anthony <femib...@gmail.com>; user <user@spark.apache.org> Subject: Re: Spark Streaming - graceful shutdown when stream has no more data During testing you will typically be using some finite data. You want the stream to shut down automatically when that data

Re: Spark Streaming - graceful shutdown when stream has no more data

During testing you will typically be using some finite data. You want the stream to shut down automatically when that data has been consumed so your test shuts down gracefully. Of course once the code is running in production you'll want it to keep waiting for new records. So whether the stream

Re: Spark Streaming - graceful shutdown when stream has no more data

Just out of curiosity I will like to know why a streaming program should shutdown when no new data is arriving? I think it should keep waiting for arrival of new records. Thanks Ashutosh On Tue, Feb 23, 2016 at 9:17 PM, Hemant Bhanawat wrote: > A guess - parseRecord is

Re: Spark Streaming - graceful shutdown when stream has no more data

A guess - parseRecord is returning None in some case (probaly empty lines). And then entry.get is throwing the exception. You may want to filter the None values from accessLogDStream before you run the map function over it. Hemant Hemant Bhanawat

Re: Spark Streaming - graceful shutdown when stream has no more data

Which line is line 42 in your code ? When variable lines becomes empty, you can stop your program. Cheers > On Feb 23, 2016, at 12:25 AM, Femi Anthony wrote: > > I am working on Spark Streaming API and I wish to stream a set of > pre-downloaded web log files continuously