Re: `Project` not preserving child partitioning ?

2016-10-12 Thread Tejas Patil
Sure :) Thanks, Tejas On Wed, Oct 12, 2016 at 11:26 AM, Reynold Xin wrote: > It actually does -- but do it through a really weird way. > > UnaryNodeExec actually defines: > > trait UnaryExecNode extends SparkPlan { > def child: SparkPlan > > override final def children: Seq[SparkPlan] = chi

Re: `Project` not preserving child partitioning ?

2016-10-12 Thread Reynold Xin
It actually does -- but do it through a really weird way. UnaryNodeExec actually defines: trait UnaryExecNode extends SparkPlan { def child: SparkPlan override final def children: Seq[SparkPlan] = child :: Nil override def outputPartitioning: Partitioning = child.outputPartitioning } I

`Project` not preserving child partitioning ?

2016-10-12 Thread Tejas Patil
See https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala#L80 Project operator preserves child's sort ordering but for output partitioning, it does not. I don't see any way projection would alter the partitioning of the chil