t; >
> >
> > Back to original exception, blindly calling “Option.get” is always not a
> > good practice. It would be better to pre-validate or use
> > getOption/getOrElse.
> >
> >
> >
> > Thanks,
> >
> > William
> >
> >
>
; Ted Yu; Femi Anthony; user
Subject: RE: Spark Streaming - graceful shutdown when stream has no more data
This is very interesting, how to shutdown the streaming job gracefully once no
input data for some time.
A doable solution probably you can count the input data by using the
Accumulator
ih...@gmail.com>; Femi
Anthony <femib...@gmail.com>; user <user@spark.apache.org>
Subject: Re: Spark Streaming - graceful shutdown when stream has no more data
During testing you will typically be using some finite data. You want the
stream to shut down automatically when that data
During testing you will typically be using some finite data. You want the
stream to shut down automatically when that data has been consumed so your
test shuts down gracefully.
Of course once the code is running in production you'll want it to keep
waiting for new records. So whether the stream
Just out of curiosity I will like to know why a streaming program should
shutdown when no new data is arriving? I think it should keep waiting for
arrival of new records.
Thanks
Ashutosh
On Tue, Feb 23, 2016 at 9:17 PM, Hemant Bhanawat
wrote:
> A guess - parseRecord is
A guess - parseRecord is returning None in some case (probaly empty lines).
And then entry.get is throwing the exception.
You may want to filter the None values from accessLogDStream before you run
the map function over it.
Hemant
Hemant Bhanawat
Which line is line 42 in your code ?
When variable lines becomes empty, you can stop your program.
Cheers
> On Feb 23, 2016, at 12:25 AM, Femi Anthony wrote:
>
> I am working on Spark Streaming API and I wish to stream a set of
> pre-downloaded web log files continuously
I am working on Spark Streaming API and I wish to stream a set of
pre-downloaded web log files continuously to simulate a real-time stream. I
wrote a script that gunzips the compressed logs and pipes the output to nc
on port .
The script looks like this: