Re: Fwd: [SparkSQL] Project using NamedExpression

2017-03-23 Thread Liang-Chi Hsieh
Hi, You need to resolve the expressions before passing into creating UnsafeProjection. Aviral Agarwal wrote > Hi guys, > > I want transform Row using NamedExpression. > > Below is the code snipped that I am using : > > > def apply(dataFrame: DataFrame, selectExpressions: >

RFC: deprecate SparkStatusTracker, remove JobProgressListener

2017-03-23 Thread Marcelo Vanzin
Hello all, For those not following, I'm working on SPARK-18085, where my goal is to decouple the storage of UI data from the actual UI implementation. This is mostly targeted at the history server, so that it's possible to quickly load a "database" with UI information instead of the existing way

[SQL] Registering custom Rule[LogicalPlan] using extendedResolutionRules by overriding SparkSession, SessionState, and Analyzer only?

2017-03-23 Thread Jacek Laskowski
Hi, We had a Spark-a-thon in Warsaw, Poland [1] where we set on learning QueryPlan API. My initial idea was to start with Analyzer and register a custom Rule[LogicalPlan] using extendedResolutionRules [2]. We were glad to have seen the scaladoc: "Override to provide additional rules for the