Spark Streaming can also generate RDDs with zero partitions if no data was
received i.e. no data blocks were created in a batch interval. The
resultant BlockRDD for that batch interval will have zero partitions.

This RDD still need to be cogrouped with another RDD (state RDD) and
finally new state RDD needs to be computed (even if there was no input
data).
However, as far as I think, since we are doing a shuffle on the BlockRDD
(for the cogroup) the ShuffledRDD (or whatever its now called) will
non-zero partitions (even though its parent RDD has zero-poartitions). So
this is not really a case of zero partitions and it will actually get
computed in the current code.

TL;DR: From Spark Streaming point of view, even though there are
0-partition RDDs possible, I dont see a point of handling zero partitions
beyond runjob.

TD


On Fri, Aug 23, 2013 at 11:07 AM, Reynold Xin <[email protected]> wrote:

> But is there any reason to do the handling of those beyond runJob?
>
>
> On Fri, Aug 23, 2013 at 11:04 AM, Charles Reiss
> <[email protected]>wrote:
>
> > On 8/22/13 22:57 , Reynold Xin wrote:
> > > I actually don't think there is any reason to have 0 partition stages,
> > be it
> > > either result stage or shufflemap.
> > >
> > > It looks like Charles added those. Charles, any comments?
> >
> > One can get 0-partition RDDs (and thus 0-partition stages of either type)
> > pretty easily with PartitionPruningRDD. Given that, e.g., Shark uses this
> > with
> > partition statistics, I imagine that real programs can hit the
> 0-partition
> > stage case this way.
> >
> > One can also get a 0-partition RDD from sc.textFile() on an empty
> > directory,
> > and presumably some uses of hadoopFile/etc., though I won't claim that
> > these
> > are important to support.
> >
> > - Charles
> >
> > >
> > >
> > >
> > > On Thu, Aug 22, 2013 at 10:51 PM, Mark Hamstra <
> [email protected]
> > > <mailto:[email protected]>> wrote:
> > >
> > >     We already do a quick, no-op return from DAGScheduler.runJob when
> > there are
> > >     no partitions submitted with the job, so running a job with no
> > partitions
> > >     in the usual way isn't a problem.  That still leaves at least the
> > "zero
> > >     split job" in the DAGSchedulerSuite and the possibility of
> shuffleMap
> > >     stages with no partitions.  Is "zero split job" testing anything
> > >     meaningful, or is its only purpose to cause me headaches?  Can
> > shuffleMap
> > >     stages actually have no partitions, or is this (also) a distraction
> > posing
> > >     as a legitimate problem?
> > >
> > >     In short, when are RDDs with no partitions real things that we
> > actually
> > >     have to deal with?
> > >
> > >
> > >
> > >     On Thu, Aug 22, 2013 at 9:20 PM, Reynold Xin <[email protected]
> > >     <mailto:[email protected]>> wrote:
> > >
> > >     > Being the guy that added the empty partition rdd, I second your
> > idea that
> > >     > we should just short-circuit those in DAGScheduler.runJob.
> > >     >
> > >     >
> > >     >
> > >     >
> > >     > On Thu, Aug 22, 2013 at 8:26 PM, Mark Hamstra <
> > [email protected]
> > >     <mailto:[email protected]>
> > >     > >wrote:
> > >     >
> > >     > > So how do these get created, and are we really handling them
> > correctly?
> > >     > >  What is prompting my questions is that I'm looking at making
> > sure that
> > >     > the
> > >     > > various data structures in the DAGScheduler shrink when
> > appropriate
> > >     > instead
> > >     > > of growing without bounds.  Jobs with no partitions and the
> > "zero split
> > >     > > job" test in the DAGSchedulerSuite really throw a wrench into
> > the works.
> > >     > >  That's because in the DAGScheduler we go part way along in
> > handling this
> > >     > > weird case as though it were a normal job submission, we start
> > >     > initializing
> > >     > > or adding to various data structures, etc.; then we pretty much
> > bail out
> > >     > in
> > >     > > submitMissingTasks when we find out that there actually are no
> > tasks to
> > >     > be
> > >     > > done.  We remove the stage from the set of running stages, but
> > we don't
> > >     > > ever clean up pendingTasks, activeJobs, stageIdToStage,
> > stageToInfos, and
> > >     > > others because no tasks are ever submitted for the stage, so
> > there are
> > >     > > never any completion events, nor is the stage aborted -- i.e.
> > the normal
> > >     > > paths to cleanup are never taken.  The end result is that
> > shuffleMap
> > >     > stages
> > >     > > with no partitions (can these even occur?) never complete, and
> > job's with
> > >     > > no partitions would seem also to persist forever.
> > >     > >
> > >     > > In short, RDDs with no partitions do really weird things to the
> > >     > > DAGScheduler.
> > >     > >
> > >     > > So, if there is no way to effectively prevent the creation of
> > RDDs with
> > >     > no
> > >     > > partitions, is there any reason why we can't short-circuit
> their
> > handling
> > >     > > within the DAGScheduler so that data structures are never built
> > or
> > >     > > populated for these weird things, or must we add a bunch of
> > special-case
> > >     > > cleanup code to submitMissingStages?
> > >     > >
> > >     >
> > >
> > >
> >
> >
>

Reply via email to