Re: [SQL] Unresolved reference with chained window functions.

2017-03-24 Thread Herman van Hövell tot Westerflier
This is definitely a bug in the CollapseWindow optimizer rule. I think we can use SPARK-20086 to track this. On Fri, Mar 24, 2017 at 9:28 PM, Maciej Szymkiewicz wrote: > Forwarded from SO

[SQL] Unresolved reference with chained window functions.

2017-03-24 Thread Maciej Szymkiewicz
Forwarded from SO (http://stackoverflow.com/q/43007433). Looks like regression compared to 2.0.2. scala> import org.apache.spark.sql.expressions.Window import org.apache.spark.sql.expressions.Window scala> val win_spec_max =

Re: RFC: deprecate SparkStatusTracker, remove JobProgressListener

2017-03-24 Thread Marcelo Vanzin
On Fri, Mar 24, 2017 at 1:18 PM, Ryan Blue wrote: > For the status tracker, I'd like to see it replaced with something better > before deprecating it. I've been looking at it for implementing better > feedback in notebooks, and it looks sufficient for that at the moment. Is >

Re: RFC: deprecate SparkStatusTracker, remove JobProgressListener

2017-03-24 Thread Marcelo Vanzin
On Fri, Mar 24, 2017 at 1:13 PM, Josh Rosen wrote: > Let's not deprecate SparkStatusTracker (at least not until there's a > suitable replacement that's just as easy to use). Deprecating it now would > leave users with an unactionable warning and that's not great for

Re: RFC: deprecate SparkStatusTracker, remove JobProgressListener

2017-03-24 Thread Marcelo Vanzin
On Fri, Mar 24, 2017 at 12:07 PM, Josh Rosen wrote: > I think that it should be safe to remove JobProgressListener but I'd like to > keep the SparkStatusTracker API. Thanks Josh. I can work with that. My main concern would be keeping the listener around. Is it worth it

Re: RFC: deprecate SparkStatusTracker, remove JobProgressListener

2017-03-24 Thread Josh Rosen
I think that it should be safe to remove JobProgressListener but I'd like to keep the SparkStatusTracker API. SparkStatusTracker was originally developed to provide a stable programmatic status API for use by Hive on Spark. SparkStatusTracker predated the Spark REST APIs for status tracking which

Re: planning & discussion for larger scheduler changes

2017-03-24 Thread Reynold Xin
On Fri, Mar 24, 2017 at 4:41 PM, Imran Rashid wrote: > Kay and I were discussing some of the bigger scheduler changes getting > proposed lately, and realized there is a broader discussion to have with > the community, outside of any single jira. I'll start by sharing my >

Re: Fwd: [SparkSQL] Project using NamedExpression

2017-03-24 Thread Aviral Agarwal
Hi , Can you please point me on how to resolve the expression ? I was looking into LogicalPlan.Resolve expression() that takes a Partial Function but I am not sure how to use that. Thanks, Aviral Agarwal On Mar 24, 2017 09:20, "Liang-Chi Hsieh" wrote: Hi, You need to