[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r180778937
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: String
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+
+ def isPrimitive: Boolean = CodeGenerator.isPrimitiveType(javaType)
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: String) extends
ExprValue {
--- End diff --
Yes.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r180725444
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: String
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+
+ def isPrimitive: Boolean = CodeGenerator.isPrimitiveType(javaType)
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: String) extends
ExprValue {
--- End diff --
We currently have case objects for `TrueLiteral` and `FalseLiteral` which
extends `LiteralValue`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r180724505
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: String
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+
+ def isPrimitive: Boolean = CodeGenerator.isPrimitiveType(javaType)
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: String) extends
ExprValue {
--- End diff --
why not a case class?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20043 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171557425
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+
+ def isPrimitive(ctx: CodegenContext): Boolean = javaType.isPrimitive(ctx)
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType) extends ExprValue {
+ override def toString: String = variableName
+ override val canDirectAccess: Boolean = false
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+ override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean"))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean"))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String) {
+ def isPrimitive(ctx: CodegenContext): Boolean =
ctx.isPrimitiveType(typeName)
+}
+
+object ExprType {
+ def apply(ctx: CodegenContext, dataType: DataType): ExprType =
ExprType(ctx.javaType(dataType))
--- End diff --
I don't think this method is very useful. If needed, the caller can do
`ctx.javaType(dataType)` I think
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171557192
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+
+ def isPrimitive(ctx: CodegenContext): Boolean = javaType.isPrimitive(ctx)
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType) extends ExprValue {
+ override def toString: String = variableName
+ override val canDirectAccess: Boolean = false
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+ override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean"))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean"))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String) {
+ def isPrimitive(ctx: CodegenContext): Boolean =
ctx.isPrimitiveType(typeName)
--- End diff --
here following the approach in #20700 we can avoid to pass `CodegenContext`
as a parameter. We can move the method to `CodeGenerator`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171556576
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -70,13 +72,14 @@ case class GlobalValue(val value: String, val javaType:
ExprType) extends ExprVa
override val canDirectAccess: Boolean = true
}
-case object TrueLiteral extends LiteralValue("true", ExprType("boolean",
true))
-case object FalseLiteral extends LiteralValue("false", ExprType("boolean",
true))
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean"))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean"))
// Represents the java type of an evaluation.
-case class ExprType(val typeName: String, val isPrimitive: Boolean)
+case class ExprType(val typeName: String) {
+ def isPrimitive(ctx: CodegenContext): Boolean =
ctx.isPrimitiveType(typeName)
--- End diff --
I see that @kiszk created a PR to move all the "static" methods to
`CodeGenerator`. Using that approach here we do not need to pass `ctx` as an
argument.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171556291
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -70,13 +72,14 @@ case class GlobalValue(val value: String, val javaType:
ExprType) extends ExprVa
override val canDirectAccess: Boolean = true
}
-case object TrueLiteral extends LiteralValue("true", ExprType("boolean",
true))
-case object FalseLiteral extends LiteralValue("false", ExprType("boolean",
true))
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean"))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean"))
// Represents the java type of an evaluation.
-case class ExprType(val typeName: String, val isPrimitive: Boolean)
+case class ExprType(val typeName: String) {
+ def isPrimitive(ctx: CodegenContext): Boolean =
ctx.isPrimitiveType(typeName)
+}
object ExprType {
- def apply(ctx: CodegenContext, dataType: DataType): ExprType =
ExprType(ctx.javaType(dataType),
-ctx.isPrimitiveType(dataType))
+ def apply(ctx: CodegenContext, dataType: DataType): ExprType =
ExprType(ctx.javaType(dataType))
--- End diff --
I am not sure this is useful. `ctx.javaType(dataType)` can be done by the
caller
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171536167
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
--- End diff --
Ok. I'd let it as fixed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171517146
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
--- End diff --
a static variable is a `GlobalValue`, isn't it? Considering that we should
be able to access also from methods in other internal classes I don't see any
use case where this flexibility is required, honestly...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171516452
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+ override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+ override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean",
true))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean",
true))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String, val isPrimitive: Boolean)
--- End diff --
than can't we move the method from `CodegenContext` to a static method and
use that? Currently this information is never used, and I feel this is hard to
maintain (even though I don't expect it to change frequently).
We can add a method like
```
def isPrimitive: Boolean = CodegenContext.isPrimitive(typeName)
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171447686
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
--- End diff --
I want to give it a bit flexibility for something like static variable.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171444357
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -31,7 +31,7 @@ import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.ScalaReflection.universe.TermName
import org.apache.spark.sql.catalyst.encoders.RowEncoder
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
+import org.apache.spark.sql.catalyst.expressions.codegen._
--- End diff --
It will list too many classes `CodegenContext`, `ExprCode`, `ExprValue`,
`GlobalValue`, `FalseLiteral`, `VariableValue`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171443786
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+ override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+ override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean",
true))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean",
true))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String, val isPrimitive: Boolean)
--- End diff --
Here the idea is to include java type information for an evaluation. Then
we don't need to consult `CodegenContext`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171268197
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
---
@@ -22,7 +22,7 @@ import scala.collection.mutable.ArrayBuffer
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
+import org.apache.spark.sql.catalyst.expressions.codegen._
--- End diff --
ditto
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171264271
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
--- End diff --
why isn't this fixed like for `GlobalValue`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171264038
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -31,7 +31,7 @@ import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.ScalaReflection.universe.TermName
import org.apache.spark.sql.catalyst.encoders.RowEncoder
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
+import org.apache.spark.sql.catalyst.expressions.codegen._
--- End diff --
can we list the needed classes instead?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171263916
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+ override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+ override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean",
true))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean",
true))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String, val isPrimitive: Boolean)
--- End diff --
why is this `isPrimitive` needed? If I am not wrong, we have somewhere a
method to check whether a type is primitive or not. I think we can get rid of
this and use that method when needed, or at least store this using that method
instead of passing it every time.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171262081
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -323,7 +323,8 @@ class CodegenContext {
case _: StructType | _: ArrayType | _: MapType => s"$value =
$initCode.copy();"
case _ => s"$value = $initCode;"
}
-ExprCode(code, "false", value)
+ExprCode(code, FalseLiteral,
+ GlobalValue(value, ExprType(this, dataType)))
--- End diff --
nit: this can go on one line
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r169317284
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BoundAttribute.scala
---
@@ -75,7 +75,7 @@ case class BoundReference(ordinal: Int, dataType:
DataType, nullable: Boolean)
|$javaType ${ev.value} = ${ev.isNull} ?
${ctx.defaultValue(dataType)} : ($value);
""".stripMargin)
} else {
-ev.copy(code = s"$javaType ${ev.value} = $value;", isNull =
"false")
+ev.copy(code = s"$javaType ${ev.value} = $value;", isNull =
LiteralValue("false"))
--- End diff --
Looks like a good idea.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r169223825
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BoundAttribute.scala
---
@@ -75,7 +75,7 @@ case class BoundReference(ordinal: Int, dataType:
DataType, nullable: Boolean)
|$javaType ${ev.value} = ${ev.isNull} ?
${ctx.defaultValue(dataType)} : ($value);
""".stripMargin)
} else {
-ev.copy(code = s"$javaType ${ev.value} = $value;", isNull =
"false")
+ev.copy(code = s"$javaType ${ev.value} = $value;", isNull =
LiteralValue("false"))
--- End diff --
I think it should be useful to the `isNull` field.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r169223570
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BoundAttribute.scala
---
@@ -75,7 +75,7 @@ case class BoundReference(ordinal: Int, dataType:
DataType, nullable: Boolean)
|$javaType ${ev.value} = ${ev.isNull} ?
${ctx.defaultValue(dataType)} : ($value);
""".stripMargin)
} else {
-ev.copy(code = s"$javaType ${ev.value} = $value;", isNull =
"false")
+ev.copy(code = s"$javaType ${ev.value} = $value;", isNull =
LiteralValue("false"))
--- End diff --
nit: shall we introduce a `TrueLiteral` and `FalseLiteral`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r159227015
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
* @param value A term for a (possibly primitive) value of the result of
the evaluation. Not
* valid if `isNull` is set to `true`.
*/
-case class ExprCode(var code: String, var isNull: String, var value:
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value:
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --
OK let's keep it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158598549
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
* @param value A term for a (possibly primitive) value of the result of
the evaluation. Not
* valid if `isNull` is set to `true`.
*/
-case class ExprCode(var code: String, var isNull: String, var value:
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value:
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --
@cloud-fan What do you think?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158441506
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
* @param value A term for a (possibly primitive) value of the result of
the evaluation. Not
* valid if `isNull` is set to `true`.
*/
-case class ExprCode(var code: String, var isNull: String, var value:
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value:
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --
If no strong preference for combining them, I'd keep it as two concepts for
now, if we foresee the need to distinguish them.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158426351
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
* @param value A term for a (possibly primitive) value of the result of
the evaluation. Not
* valid if `isNull` is set to `true`.
*/
-case class ExprCode(var code: String, var isNull: String, var value:
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value:
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --
In summary, I have no strong preference.
In the future, we will want to distinguish `Literal` and `Global` for some
optimizations.
[This](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L121)
is already one of optimizations for `Literal`.
If this PR just focuses on classifying types between arguments and
non-arguments, it is fine to combine `Literal` and `Global`. Then, another PR
will separate one type into `Literal` and `Global`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158403352
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
* @param value A term for a (possibly primitive) value of the result of
the evaluation. Not
* valid if `isNull` is set to `true`.
*/
-case class ExprCode(var code: String, var isNull: String, var value:
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value:
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --
@kiszk WDYT?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158403030
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
---
@@ -321,7 +322,12 @@ case class IsNull(child: Expression) extends
UnaryExpression with Predicate {
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val eval = child.genCode(ctx)
-ExprCode(code = eval.code, isNull = "false", value = eval.isNull)
+val value = if ("true" == s"${eval.isNull}" || "false" ==
s"${eval.isNull}") {
--- End diff --
We can do `eval.isNull.instanceOf[LiteralValue]` as suggested by @cloud-fan
below.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158401512
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
* @param value A term for a (possibly primitive) value of the result of
the evaluation. Not
* valid if `isNull` is set to `true`.
*/
-case class ExprCode(var code: String, var isNull: String, var value:
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value:
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --
For now `LiteralValue` and `GlobalValue` can be seen as the same
effectively, as they are all accessible anywhere and we don't need to carry it
via method parameters.
I don't have strong preference here.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158400953
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -118,10 +118,10 @@ abstract class Expression extends
TreeNode[Expression] {
private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
// TODO: support whole stage codegen too
if (eval.code.trim.length > 1024 && ctx.INPUT_ROW != null &&
ctx.currentVars == null) {
- val setIsNull = if (eval.isNull != "false" && eval.isNull != "true")
{
+ val setIsNull = if ("false" != s"${eval.isNull}" && "true" !=
s"${eval.isNull}") {
--- End diff --
oh, yea.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158361417
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
* @param value A term for a (possibly primitive) value of the result of
the evaluation. Not
* valid if `isNull` is set to `true`.
*/
-case class ExprCode(var code: String, var isNull: String, var value:
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value:
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --
IMHO I prefer this approach because in the future we might need to
distinguish these two cases, thus I think is a good thing to let them be
distinct.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158361160
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
---
@@ -321,7 +322,12 @@ case class IsNull(child: Expression) extends
UnaryExpression with Predicate {
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val eval = child.genCode(ctx)
-ExprCode(code = eval.code, isNull = "false", value = eval.isNull)
+val value = if ("true" == s"${eval.isNull}" || "false" ==
s"${eval.isNull}") {
--- End diff --
or `eval.isNull == Literal("true")`? Or even better we can create a
`LiteralTrue = Literal("true")` and equivalent for false and use them?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158329715
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
* @param value A term for a (possibly primitive) value of the result of
the evaluation. Not
* valid if `isNull` is set to `true`.
*/
-case class ExprCode(var code: String, var isNull: String, var value:
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value:
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --
I think we should classify `ExprValue` by our needs, not by java
definitions. Thinking about the needs, we wanna know: 1) if this value is
accessible anywhere and we don't need to carry it via method parameters. 2) if
this value needs to be carried with parameters, do we need to generate a
parameter name or use this value directly?
So basically we can combine `LiteralValue` and `GlobalValue`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158328435
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -118,10 +118,10 @@ abstract class Expression extends
TreeNode[Expression] {
private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
// TODO: support whole stage codegen too
if (eval.code.trim.length > 1024 && ctx.INPUT_ROW != null &&
ctx.currentVars == null) {
- val setIsNull = if (eval.isNull != "false" && eval.isNull != "true")
{
+ val setIsNull = if ("false" != s"${eval.isNull}" && "true" !=
s"${eval.isNull}") {
--- End diff --
this can be simplified to `!eval.isNull.instanceOf[LiteralValue]`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158299743
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
---
@@ -347,7 +353,14 @@ case class IsNotNull(child: Expression) extends
UnaryExpression with Predicate {
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val eval = child.genCode(ctx)
-ExprCode(code = eval.code, isNull = "false", value =
s"(!(${eval.isNull}))")
+val value = if ("true" == s"${eval.isNull}") {
--- End diff --
ditto
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158299679
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
---
@@ -321,7 +322,12 @@ case class IsNull(child: Expression) extends
UnaryExpression with Predicate {
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val eval = child.genCode(ctx)
-ExprCode(code = eval.code, isNull = "false", value = eval.isNull)
+val value = if ("true" == s"${eval.isNull}" || "false" ==
s"${eval.isNull}") {
--- End diff --
`if ("true" == eval.isNull || "false" == eval.isNull) {`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158210849
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
* @param value A term for a (possibly primitive) value of the result of
the evaluation. Not
* valid if `isNull` is set to `true`.
*/
-case class ExprCode(var code: String, var isNull: String, var value:
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value:
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+case class LiteralValue(val value: String) extends ExprValue {
+ override def toString: String = value
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(val variableName: String) extends ExprValue {
+ override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(val statement: String) extends ExprValue {
+ override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String) extends ExprValue {
--- End diff --
It is considered as global variable now, as it can be accessed globally and
don't/can't be parameterized.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r158209659
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
* @param value A term for a (possibly primitive) value of the result of
the evaluation. Not
* valid if `isNull` is set to `true`.
*/
-case class ExprCode(var code: String, var isNull: String, var value:
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value:
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+case class LiteralValue(val value: String) extends ExprValue {
+ override def toString: String = value
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(val variableName: String) extends ExprValue {
+ override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(val statement: String) extends ExprValue {
+ override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String) extends ExprValue {
--- End diff --
for compacted global variables, we may get something like `arr[1]` while
`arr` is a global variable. Is `arr[1]` a statement or global variable?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/20043 [SPARK-22856][SQL] Add wrappers for codegen output and nullability ## What changes were proposed in this pull request? The codegen output of `Expression`, aka `ExprCode`, now encapsulates only strings of output value (`value`) and nullability (`isNull`). It makes difficulty for us to know what the output really is. I think it is better if we can add wrappers for the value and nullability that let us to easily know that. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/viirya/spark-1 SPARK-22856 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20043.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20043 commit 78680cc60c27d645c349451090bfa056380e89f6 Author: Liang-Chi Hsieh Date: 2017-12-21T04:22:10Z Add wrappers for codegen output. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
