See inline.
On Tue, Sep 8, 2015 at 9:02 PM, Dmitry Goldenberg <dgoldenberg...@gmail.com>
wrote:
> What's wrong with creating a checkpointed context?? We WANT
> checkpointing, first of all. We therefore WANT the checkpointed context.
>
> Second of all, it's not true that we're loading the checkpointed context
> independent of whether params.isCheckpointed() is true. I'm quoting the
> code again:
>
> // This is NOT loading a checkpointed context if isCheckpointed() is false.
> JavaStreamingContext jssc = params.isCheckpointed() ?
> createCheckpointedContext(sparkConf, params) : createContext(sparkConf,
> params);
>
> private JavaStreamingContext createCheckpointedContext(SparkConf
> sparkConf, Parameters params) {
> JavaStreamingContextFactory factory = new
> JavaStreamingContextFactory() {
> @Override
> public JavaStreamingContext create() {
> return createContext(sparkConf, params);
> }
> };
> return *JavaStreamingContext.getOrCreate(params.getCheckpointDir(),
> factory);*
>
^^^^^ when you use getOrCreate, and there exists a valid checkpoint, it
will always return the context from the checkpoint and not call the
factory. Simple way to see whats going on is to print something in the
factory to verify whether it is ever called.
> }
>
> private JavaStreamingContext createContext(SparkConf sparkConf,
> Parameters params) {
> // Create context with the specified batch interval, in milliseconds.
> JavaStreamingContext jssc = new JavaStreamingContext(sparkConf,
> Durations.milliseconds(params.getBatchDurationMillis()));
> // Set the checkpoint directory, if we're checkpointing
> if (params.isCheckpointed()) {
> jssc.checkpoint(params.getCheckpointDir());
>
> }
> ...............
> Again, this is *only* calling context.checkpoint() if isCheckpointed() is
> true. And we WANT it to be true.
>
> What am I missing here?
>
>
>
