But when you talk about optimizing the DAG, it really doesn't make sense to also talk about transformation steps as separate entities. The DAGScheduler knows about Jobs, Stages, TaskSets and Tasks. The TaskScheduler knows about TaskSets ad Tasks. Neither of them understands the transformation steps that you used to define your RDD -- at least not as separable, distinct steps. To give the kind of transformation-step-oriented information that you want would require parts of Spark that don't currently concern themselves at all with RDD transformation steps to start tracking them and how they map to Jobs, Stages, TaskSets and Tasks -- and when you start talking about Datasets and Spark SQL, you then needing to start talking about tracking and mapping concepts like Plans, Schemas and Queries. It would introduce significant new complexity.
On Wed, May 25, 2016 at 6:59 PM, Nirav Patel <npa...@xactlycorp.com> wrote: > Hi Mark, > > I might have said stage instead of step in my last statement "UI just > says Collect failed but in fact it could be any stage in that lazy chain of > evaluation." > > Anyways even you agree that this visibility of underlaying steps wont't be > available. which does pose difficulties in terms of troubleshooting as well > as optimizations at step level. I think users will have hard time without > this. Its great that spark community working on different levels of > internal optimizations but its also important to give enough visibility > to users to enable them to debug issues and resolve bottleneck. > There is also no visibility into how spark utilizes shuffle memory space > vs user memory space vs cache space. It's a separate topic though. If > everything is working magically as a black box then it's fine but when you > have large number of people on this site complaining about OOM and shuffle > error all the time you need to start providing some transparency to > address that. > > Thanks > > > On Wed, May 25, 2016 at 6:41 PM, Mark Hamstra <m...@clearstorydata.com> > wrote: > >> You appear to be misunderstanding the nature of a Stage. Individual >> transformation steps such as `map` do not define the boundaries of Stages. >> Rather, a sequence of transformations in which there is only a >> NarrowDependency between each of the transformations will be pipelined into >> a single Stage. It is only when there is a ShuffleDependency that a new >> Stage will be defined -- i.e. shuffle boundaries define Stage boundaries. >> With whole stage code gen in Spark 2.0, there will be even less opportunity >> to treat individual transformations within a sequence of narrow >> dependencies as though they were discrete, separable entities. The Failed >> Stages portion of the Web UI will tell you which Stage in a Job failed, and >> the accompanying error log message will generally also give you some idea >> of which Tasks failed and why. Tracing the error back further and at a >> different level of abstraction to lay blame on a particular transformation >> wouldn't be particularly easy. >> >> On Wed, May 25, 2016 at 5:28 PM, Nirav Patel <npa...@xactlycorp.com> >> wrote: >> >>> It's great that spark scheduler does optimized DAG processing and only >>> does lazy eval when some action is performed or shuffle dependency is >>> encountered. Sometime it goes further after shuffle dep before executing >>> anything. e.g. if there are map steps after shuffle then it doesn't stop at >>> shuffle to execute anything but goes to that next map steps until it finds >>> a reason(spark action) to execute. As a result stage that spark is running >>> can be internally series of (map -> shuffle -> map -> map -> collect) and >>> spark UI just shows its currently running 'collect' stage. SO if job fails >>> at that point spark UI just says Collect failed but in fact it could be any >>> stage in that lazy chain of evaluation. Looking at executor logs gives some >>> insights but that's not always straightforward. >>> Correct me if I am wrong here but I think we need more visibility into >>> what's happening underneath so we can easily troubleshoot as well as >>> optimize our DAG. >>> >>> THanks >>> >>> >>> >>> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> >>> >>> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] >>> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] >>> <https://twitter.com/Xactly> [image: Facebook] >>> <https://www.facebook.com/XactlyCorp> [image: YouTube] >>> <http://www.youtube.com/xactlycorporation> >> >> >> > > > > [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> > > <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] > <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] > <https://twitter.com/Xactly> [image: Facebook] > <https://www.facebook.com/XactlyCorp> [image: YouTube] > <http://www.youtube.com/xactlycorporation> >
