[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r180778937
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: String
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+
+  def isPrimitive: Boolean = CodeGenerator.isPrimitiveType(javaType)
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: String) extends 
ExprValue {
--- End diff --

Yes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r180725444
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: String
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+
+  def isPrimitive: Boolean = CodeGenerator.isPrimitiveType(javaType)
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: String) extends 
ExprValue {
--- End diff --

We currently have case objects for `TrueLiteral` and `FalseLiteral` which 
extends `LiteralValue`.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r180724505
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: String
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+
+  def isPrimitive: Boolean = CodeGenerator.isPrimitiveType(javaType)
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: String) extends 
ExprValue {
--- End diff --

why not a case class?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20043


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171557425
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: ExprType
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+
+  def isPrimitive(ctx: CodegenContext): Boolean = javaType.isPrimitive(ctx)
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+  def apply(value: String, javaType: ExprType): LiteralValue = new 
LiteralValue(value, javaType)
+  def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType) extends ExprValue {
+  override def toString: String = variableName
+  override val canDirectAccess: Boolean = false
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+  override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean"))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean"))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String) {
+  def isPrimitive(ctx: CodegenContext): Boolean = 
ctx.isPrimitiveType(typeName)
+}
+
+object ExprType {
+  def apply(ctx: CodegenContext, dataType: DataType): ExprType = 
ExprType(ctx.javaType(dataType))
--- End diff --

I don't think this method is very useful. If needed, the caller can do 
`ctx.javaType(dataType)` I think


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171557192
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: ExprType
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+
+  def isPrimitive(ctx: CodegenContext): Boolean = javaType.isPrimitive(ctx)
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+  def apply(value: String, javaType: ExprType): LiteralValue = new 
LiteralValue(value, javaType)
+  def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType) extends ExprValue {
+  override def toString: String = variableName
+  override val canDirectAccess: Boolean = false
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+  override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean"))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean"))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String) {
+  def isPrimitive(ctx: CodegenContext): Boolean = 
ctx.isPrimitiveType(typeName)
--- End diff --

here following the approach in #20700 we can avoid to pass `CodegenContext` 
as a parameter. We can move the method to `CodeGenerator`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171556576
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -70,13 +72,14 @@ case class GlobalValue(val value: String, val javaType: 
ExprType) extends ExprVa
   override val canDirectAccess: Boolean = true
 }
 
-case object TrueLiteral extends LiteralValue("true", ExprType("boolean", 
true))
-case object FalseLiteral extends LiteralValue("false", ExprType("boolean", 
true))
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean"))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean"))
 
 // Represents the java type of an evaluation.
-case class ExprType(val typeName: String, val isPrimitive: Boolean)
+case class ExprType(val typeName: String) {
+  def isPrimitive(ctx: CodegenContext): Boolean = 
ctx.isPrimitiveType(typeName)
--- End diff --

I see that @kiszk created a PR to move all the "static" methods to 
`CodeGenerator`. Using that approach here we do not need to pass `ctx` as an 
argument.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171556291
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -70,13 +72,14 @@ case class GlobalValue(val value: String, val javaType: 
ExprType) extends ExprVa
   override val canDirectAccess: Boolean = true
 }
 
-case object TrueLiteral extends LiteralValue("true", ExprType("boolean", 
true))
-case object FalseLiteral extends LiteralValue("false", ExprType("boolean", 
true))
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean"))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean"))
 
 // Represents the java type of an evaluation.
-case class ExprType(val typeName: String, val isPrimitive: Boolean)
+case class ExprType(val typeName: String) {
+  def isPrimitive(ctx: CodegenContext): Boolean = 
ctx.isPrimitiveType(typeName)
+}
 
 object ExprType {
-  def apply(ctx: CodegenContext, dataType: DataType): ExprType = 
ExprType(ctx.javaType(dataType),
-ctx.isPrimitiveType(dataType))
+  def apply(ctx: CodegenContext, dataType: DataType): ExprType = 
ExprType(ctx.javaType(dataType))
--- End diff --

I am not sure this is useful. `ctx.javaType(dataType)` can be done by the 
caller


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171536167
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: ExprType
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+  def apply(value: String, javaType: ExprType): LiteralValue = new 
LiteralValue(value, javaType)
+  def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
--- End diff --

Ok. I'd let it as fixed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171517146
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: ExprType
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+  def apply(value: String, javaType: ExprType): LiteralValue = new 
LiteralValue(value, javaType)
+  def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
--- End diff --

a static variable is a `GlobalValue`, isn't it? Considering that we should 
be able to access also from methods in other internal classes I don't see any 
use case where this flexibility is required, honestly...


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171516452
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: ExprType
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+  def apply(value: String, javaType: ExprType): LiteralValue = new 
LiteralValue(value, javaType)
+  def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+  override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+  override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean", 
true))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean", 
true))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String, val isPrimitive: Boolean)
--- End diff --

than can't we move the method from `CodegenContext` to a static method and 
use that? Currently this information is never used, and I feel this is hard to 
maintain (even though I don't expect it to change frequently).

We can add a method like
```
def isPrimitive: Boolean = CodegenContext.isPrimitive(typeName)
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171447686
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: ExprType
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+  def apply(value: String, javaType: ExprType): LiteralValue = new 
LiteralValue(value, javaType)
+  def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
--- End diff --

I want to give it a bit flexibility for something like static variable.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171444357
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
 ---
@@ -31,7 +31,7 @@ import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.ScalaReflection.universe.TermName
 import org.apache.spark.sql.catalyst.encoders.RowEncoder
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, 
ExprCode}
+import org.apache.spark.sql.catalyst.expressions.codegen._
--- End diff --

It will list too many classes `CodegenContext`, `ExprCode`, `ExprValue`, 
`GlobalValue`, `FalseLiteral`, `VariableValue`.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171443786
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: ExprType
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+  def apply(value: String, javaType: ExprType): LiteralValue = new 
LiteralValue(value, javaType)
+  def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+  override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+  override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean", 
true))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean", 
true))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String, val isPrimitive: Boolean)
--- End diff --

Here the idea is to include java type information for an evaluation. Then 
we don't need to consult `CodegenContext`.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171268197
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
 ---
@@ -22,7 +22,7 @@ import scala.collection.mutable.ArrayBuffer
 import org.apache.spark.rdd.RDD
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, 
ExprCode}
+import org.apache.spark.sql.catalyst.expressions.codegen._
--- End diff --

ditto


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171264271
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: ExprType
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+  def apply(value: String, javaType: ExprType): LiteralValue = new 
LiteralValue(value, javaType)
+  def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
--- End diff --

why isn't this fixed like for `GlobalValue`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171264038
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
 ---
@@ -31,7 +31,7 @@ import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.ScalaReflection.universe.TermName
 import org.apache.spark.sql.catalyst.encoders.RowEncoder
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, 
ExprCode}
+import org.apache.spark.sql.catalyst.expressions.codegen._
--- End diff --

can we list the needed classes instead?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171263916
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+  val javaType: ExprType
+
+  // Whether we can directly access the evaluation value anywhere.
+  // For example, a variable created outside a method can not be accessed 
inside the method.
+  // For such cases, we may need to pass the evaluation as parameter.
+  val canDirectAccess: Boolean
+}
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+  def apply(value: String, javaType: ExprType): LiteralValue = new 
LiteralValue(value, javaType)
+  def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+  override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+  override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends 
ExprValue {
+  override def toString: String = value
+  override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean", 
true))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean", 
true))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String, val isPrimitive: Boolean)
--- End diff --

why is this `isPrimitive` needed? If I am not wrong, we have somewhere a 
method to check whether a type is primitive or not. I think we can get rid of 
this and use that method when needed, or at least store this using that method 
instead of passing it every time.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r171262081
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -323,7 +323,8 @@ class CodegenContext {
   case _: StructType | _: ArrayType | _: MapType => s"$value = 
$initCode.copy();"
   case _ => s"$value = $initCode;"
 }
-ExprCode(code, "false", value)
+ExprCode(code, FalseLiteral,
+  GlobalValue(value, ExprType(this, dataType)))
--- End diff --

nit: this can go on one line


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r169317284
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BoundAttribute.scala
 ---
@@ -75,7 +75,7 @@ case class BoundReference(ordinal: Int, dataType: 
DataType, nullable: Boolean)
  |$javaType ${ev.value} = ${ev.isNull} ? 
${ctx.defaultValue(dataType)} : ($value);
""".stripMargin)
   } else {
-ev.copy(code = s"$javaType ${ev.value} = $value;", isNull = 
"false")
+ev.copy(code = s"$javaType ${ev.value} = $value;", isNull = 
LiteralValue("false"))
--- End diff --

Looks like a good idea.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r169223825
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BoundAttribute.scala
 ---
@@ -75,7 +75,7 @@ case class BoundReference(ordinal: Int, dataType: 
DataType, nullable: Boolean)
  |$javaType ${ev.value} = ${ev.isNull} ? 
${ctx.defaultValue(dataType)} : ($value);
""".stripMargin)
   } else {
-ev.copy(code = s"$javaType ${ev.value} = $value;", isNull = 
"false")
+ev.copy(code = s"$javaType ${ev.value} = $value;", isNull = 
LiteralValue("false"))
--- End diff --

I think it should be useful to the `isNull` field.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r169223570
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BoundAttribute.scala
 ---
@@ -75,7 +75,7 @@ case class BoundReference(ordinal: Int, dataType: 
DataType, nullable: Boolean)
  |$javaType ${ev.value} = ${ev.isNull} ? 
${ctx.defaultValue(dataType)} : ($value);
""".stripMargin)
   } else {
-ev.copy(code = s"$javaType ${ev.value} = $value;", isNull = 
"false")
+ev.copy(code = s"$javaType ${ev.value} = $value;", isNull = 
LiteralValue("false"))
--- End diff --

nit: shall we introduce a `TrueLiteral` and `FalseLiteral`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r159227015
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --

OK let's keep it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158598549
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --

@cloud-fan What do you think?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158441506
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --

If no strong preference for combining them, I'd keep it as two concepts for 
now, if we foresee the need to distinguish them. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158426351
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --

In summary, I have no strong preference.

In the future, we will want to distinguish `Literal` and `Global` for some 
optimizations. 
[This](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L121)
 is already one of optimizations for `Literal`.

If this PR just focuses on classifying types between arguments and 
non-arguments, it is fine to combine `Literal` and `Global`. Then, another PR 
will separate one type into `Literal` and `Global`. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158403352
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --

@kiszk WDYT?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158403030
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
 ---
@@ -321,7 +322,12 @@ case class IsNull(child: Expression) extends 
UnaryExpression with Predicate {
 
   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
 val eval = child.genCode(ctx)
-ExprCode(code = eval.code, isNull = "false", value = eval.isNull)
+val value = if ("true" == s"${eval.isNull}" || "false" == 
s"${eval.isNull}") {
--- End diff --

We can do `eval.isNull.instanceOf[LiteralValue]` as suggested by @cloud-fan 
below.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158401512
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --

For now `LiteralValue` and `GlobalValue` can be seen as the same 
effectively, as they are all accessible anywhere and we don't need to carry it 
via method parameters.

I don't have strong preference here.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158400953
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
 ---
@@ -118,10 +118,10 @@ abstract class Expression extends 
TreeNode[Expression] {
   private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
 // TODO: support whole stage codegen too
 if (eval.code.trim.length > 1024 && ctx.INPUT_ROW != null && 
ctx.currentVars == null) {
-  val setIsNull = if (eval.isNull != "false" && eval.isNull != "true") 
{
+  val setIsNull = if ("false" != s"${eval.isNull}" && "true" != 
s"${eval.isNull}") {
--- End diff --

oh, yea.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158361417
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --

IMHO I prefer this approach because in the future we might need to 
distinguish these two cases, thus I think is a good thing to let them be 
distinct.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158361160
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
 ---
@@ -321,7 +322,12 @@ case class IsNull(child: Expression) extends 
UnaryExpression with Predicate {
 
   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
 val eval = child.genCode(ctx)
-ExprCode(code = eval.code, isNull = "false", value = eval.isNull)
+val value = if ("true" == s"${eval.isNull}" || "false" == 
s"${eval.isNull}") {
--- End diff --

or `eval.isNull == Literal("true")`? Or even better we can create a 
`LiteralTrue = Literal("true")` and equivalent for false and use them?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158329715
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --

I think we should classify `ExprValue` by our needs, not by java 
definitions. Thinking about the needs, we wanna know: 1) if this value is 
accessible anywhere and we don't need to carry it via method parameters. 2) if 
this value needs to be carried with parameters, do we need to generate a 
parameter name or use this value directly?

So basically we can combine `LiteralValue` and `GlobalValue`.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158328435
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
 ---
@@ -118,10 +118,10 @@ abstract class Expression extends 
TreeNode[Expression] {
   private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
 // TODO: support whole stage codegen too
 if (eval.code.trim.length > 1024 && ctx.INPUT_ROW != null && 
ctx.currentVars == null) {
-  val setIsNull = if (eval.isNull != "false" && eval.isNull != "true") 
{
+  val setIsNull = if ("false" != s"${eval.isNull}" && "true" != 
s"${eval.isNull}") {
--- End diff --

this can be simplified to `!eval.isNull.instanceOf[LiteralValue]`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158299743
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
 ---
@@ -347,7 +353,14 @@ case class IsNotNull(child: Expression) extends 
UnaryExpression with Predicate {
 
   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
 val eval = child.genCode(ctx)
-ExprCode(code = eval.code, isNull = "false", value = 
s"(!(${eval.isNull}))")
+val value = if ("true" == s"${eval.isNull}") {
--- End diff --

ditto


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158299679
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
 ---
@@ -321,7 +322,12 @@ case class IsNull(child: Expression) extends 
UnaryExpression with Predicate {
 
   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
 val eval = child.genCode(ctx)
-ExprCode(code = eval.code, isNull = "false", value = eval.isNull)
+val value = if ("true" == s"${eval.isNull}" || "false" == 
s"${eval.isNull}") {
--- End diff --

`if ("true" == eval.isNull || "false" == eval.isNull) {`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158210849
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+case class LiteralValue(val value: String) extends ExprValue {
+  override def toString: String = value
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(val variableName: String) extends ExprValue {
+  override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(val statement: String) extends ExprValue {
+  override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String) extends ExprValue {
--- End diff --

It is considered as global variable now, as it can be accessed globally and 
don't/can't be parameterized.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158209659
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
+
+object ExprValue {
+  implicit def exprValueToString(exprValue: ExprValue): String = 
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+case class LiteralValue(val value: String) extends ExprValue {
+  override def toString: String = value
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(val variableName: String) extends ExprValue {
+  override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(val statement: String) extends ExprValue {
+  override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String) extends ExprValue {
--- End diff --

for compacted global variables, we may get something like `arr[1]` while 
`arr` is a global variable. Is `arr[1]` a statement or global variable?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

GitHub user viirya opened a pull request:

https://github.com/apache/spark/pull/20043

[SPARK-22856][SQL] Add wrappers for codegen output and nullability

## What changes were proposed in this pull request?

The codegen output of `Expression`, aka `ExprCode`, now encapsulates only 
strings of output value (`value`) and nullability (`isNull`). It makes 
difficulty for us to know what the output really is. I think it is better if we 
can add wrappers for the value and nullability that let us to easily know that.

## How was this patch tested?

Existing tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/viirya/spark-1 SPARK-22856

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20043.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20043


commit 78680cc60c27d645c349451090bfa056380e89f6
Author: Liang-Chi Hsieh 
Date:   2017-12-21T04:22:10Z

Add wrappers for codegen output.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org