On 8/22/13 22:57 , Reynold Xin wrote:
> I actually don't think there is any reason to have 0 partition stages, be it
> either result stage or shufflemap. 
> 
> It looks like Charles added those. Charles, any comments?

One can get 0-partition RDDs (and thus 0-partition stages of either type)
pretty easily with PartitionPruningRDD. Given that, e.g., Shark uses this with
partition statistics, I imagine that real programs can hit the 0-partition
stage case this way.

One can also get a 0-partition RDD from sc.textFile() on an empty directory,
and presumably some uses of hadoopFile/etc., though I won't claim that these
are important to support.

- Charles

> 
> 
> 
> On Thu, Aug 22, 2013 at 10:51 PM, Mark Hamstra <[email protected]
> <mailto:[email protected]>> wrote:
> 
>     We already do a quick, no-op return from DAGScheduler.runJob when there 
> are
>     no partitions submitted with the job, so running a job with no partitions
>     in the usual way isn't a problem.  That still leaves at least the "zero
>     split job" in the DAGSchedulerSuite and the possibility of shuffleMap
>     stages with no partitions.  Is "zero split job" testing anything
>     meaningful, or is its only purpose to cause me headaches?  Can shuffleMap
>     stages actually have no partitions, or is this (also) a distraction posing
>     as a legitimate problem?
> 
>     In short, when are RDDs with no partitions real things that we actually
>     have to deal with?
> 
> 
> 
>     On Thu, Aug 22, 2013 at 9:20 PM, Reynold Xin <[email protected]
>     <mailto:[email protected]>> wrote:
> 
>     > Being the guy that added the empty partition rdd, I second your idea 
> that
>     > we should just short-circuit those in DAGScheduler.runJob.
>     >
>     >
>     >
>     >
>     > On Thu, Aug 22, 2013 at 8:26 PM, Mark Hamstra <[email protected]
>     <mailto:[email protected]>
>     > >wrote:
>     >
>     > > So how do these get created, and are we really handling them 
> correctly?
>     > >  What is prompting my questions is that I'm looking at making sure 
> that
>     > the
>     > > various data structures in the DAGScheduler shrink when appropriate
>     > instead
>     > > of growing without bounds.  Jobs with no partitions and the "zero 
> split
>     > > job" test in the DAGSchedulerSuite really throw a wrench into the 
> works.
>     > >  That's because in the DAGScheduler we go part way along in handling 
> this
>     > > weird case as though it were a normal job submission, we start
>     > initializing
>     > > or adding to various data structures, etc.; then we pretty much bail 
> out
>     > in
>     > > submitMissingTasks when we find out that there actually are no tasks 
> to
>     > be
>     > > done.  We remove the stage from the set of running stages, but we 
> don't
>     > > ever clean up pendingTasks, activeJobs, stageIdToStage, stageToInfos, 
> and
>     > > others because no tasks are ever submitted for the stage, so there are
>     > > never any completion events, nor is the stage aborted -- i.e. the 
> normal
>     > > paths to cleanup are never taken.  The end result is that shuffleMap
>     > stages
>     > > with no partitions (can these even occur?) never complete, and job's 
> with
>     > > no partitions would seem also to persist forever.
>     > >
>     > > In short, RDDs with no partitions do really weird things to the
>     > > DAGScheduler.
>     > >
>     > > So, if there is no way to effectively prevent the creation of RDDs 
> with
>     > no
>     > > partitions, is there any reason why we can't short-circuit their 
> handling
>     > > within the DAGScheduler so that data structures are never built or
>     > > populated for these weird things, or must we add a bunch of 
> special-case
>     > > cleanup code to submitMissingStages?
>     > >
>     >
> 
> 

Reply via email to