[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16610 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r103891461 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -99,6 +99,8 @@ object functions { * The passed in object is returned directly if it is already a [[Column]]. * If the object is a Scala Symbol, it is converted into a [[Column]] also. * Otherwise, a new [[Column]] is created to represent the literal value. + * Different from [[lit]], this functions can handle all the types in Scala, --- End diff -- okay, updated. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r103890828 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -99,6 +99,8 @@ object functions { * The passed in object is returned directly if it is already a [[Column]]. * If the object is a Scala Symbol, it is converted into a [[Column]] also. * Otherwise, a new [[Column]] is created to represent the literal value. + * Different from [[lit]], this functions can handle all the types in Scala, --- End diff -- `This different between this function and [[lit]] is that this function can handle parameterized scala types e.g.: List, Seq and Map.` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r103888025 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -91,15 +91,22 @@ object functions { * @group normal_funcs * @since 1.3.0 */ - def lit(literal: Any): Column = { -literal match { - case c: Column => return c - case s: Symbol => return new ColumnName(literal.asInstanceOf[Symbol].name) - case _ => // continue -} + def lit(literal: Any): Column = typedLit(literal) -val literalExpr = Literal(literal) -Column(literalExpr) + /** + * Creates a [[Column]] of literal value. --- End diff -- updated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r103885180 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -91,15 +91,22 @@ object functions { * @group normal_funcs * @since 1.3.0 */ - def lit(literal: Any): Column = { -literal match { - case c: Column => return c - case s: Symbol => return new ColumnName(literal.asInstanceOf[Symbol].name) - case _ => // continue -} + def lit(literal: Any): Column = typedLit(literal) -val literalExpr = Literal(literal) -Column(literalExpr) + /** + * Creates a [[Column]] of literal value. --- End diff -- okay, I'll update --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r103879925 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -91,15 +91,22 @@ object functions { * @group normal_funcs * @since 1.3.0 */ - def lit(literal: Any): Column = { -literal match { - case c: Column => return c - case s: Symbol => return new ColumnName(literal.asInstanceOf[Symbol].name) - case _ => // continue -} + def lit(literal: Any): Column = typedLit(literal) -val literalExpr = Literal(literal) -Column(literalExpr) + /** + * Creates a [[Column]] of literal value. + * + * The passed in object is returned directly if it is already a [[Column]]. + * If the object is a Scala Symbol, it is converted into a [[Column]] also. + * Otherwise, a new [[Column]] is created to represent the literal value. + * + * @group normal_funcs + * @since 2.2.0 + */ + def typedLit[T : TypeTag](literal: T): Column = literal match { --- End diff -- cc @cloud-fan wdyt about the naming? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r103879881 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -91,15 +91,22 @@ object functions { * @group normal_funcs * @since 1.3.0 */ - def lit(literal: Any): Column = { -literal match { - case c: Column => return c - case s: Symbol => return new ColumnName(literal.asInstanceOf[Symbol].name) - case _ => // continue -} + def lit(literal: Any): Column = typedLit(literal) -val literalExpr = Literal(literal) -Column(literalExpr) + /** + * Creates a [[Column]] of literal value. --- End diff -- We should explain when you want to use this method instead of `lit`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r101985694 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -102,6 +102,27 @@ object functions { Column(literalExpr) } + /** + * Creates a [[Column]] of literal value. + * + * The passed in object is returned directly if it is already a [[Column]]. + * If the object is a Scala Symbol, it is converted into a [[Column]] also. + * Otherwise, a new [[Column]] is created to represent the literal value. + * + * @group normal_funcs + * @since 2.2.0 + */ + def typedLit[T : TypeTag](literal: T): Column = { +literal match { --- End diff -- @hvanhovell How about the fix? https://github.com/apache/spark/pull/16610/commits/4f65fe2315e35272aecbea9e2e3ef1f15398da31 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r101977589 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -102,6 +102,27 @@ object functions { Column(literalExpr) } + /** + * Creates a [[Column]] of literal value. + * + * The passed in object is returned directly if it is already a [[Column]]. + * If the object is a Scala Symbol, it is converted into a [[Column]] also. + * Otherwise, a new [[Column]] is created to represent the literal value. + * + * @group normal_funcs + * @since 2.2.0 + */ + def typedLit[T : TypeTag](literal: T): Column = { +literal match { --- End diff -- okay, I'll update --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r101974913 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -102,6 +102,27 @@ object functions { Column(literalExpr) } + /** + * Creates a [[Column]] of literal value. + * + * The passed in object is returned directly if it is already a [[Column]]. + * If the object is a Scala Symbol, it is converted into a [[Column]] also. + * Otherwise, a new [[Column]] is created to represent the literal value. + * + * @group normal_funcs + * @since 2.2.0 + */ + def typedLit[T : TypeTag](literal: T): Column = { +literal match { --- End diff -- This match statement is slightly hair raising (I know this is copied from `lit(...), how about: ```scala literal match { case c: Column => c case s: Symbol => new ColumnName(s.name) case _ => Column(Literal.create(literal)) } ``` You could also consider mapping the untyped `lit(..)` function to this function. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r101974502 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -102,6 +102,27 @@ object functions { Column(literalExpr) } + /** + * Creates a [[Column]] of literal value. + * + * The passed in object is returned directly if it is already a [[Column]]. + * If the object is a Scala Symbol, it is converted into a [[Column]] also. + * Otherwise, a new [[Column]] is created to represent the literal value. + * + * @group normal_funcs + * @since 2.2.0 + */ + def typedLit[T : TypeTag](literal: T): Column = { --- End diff -- cc @cloud-fan WDYT? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r100398880 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala --- @@ -153,6 +154,12 @@ object Literal { Literal(CatalystTypeConverters.convertToCatalyst(v), dataType) } + def create[T : TypeTag](v: T): Literal = { +val ScalaReflection.Schema(dataType, _) = ScalaReflection.schemaFor[T] +val convert = CatalystTypeConverters.createToCatalystConverter(dataType) --- End diff -- haha, actually I'm joining the dev talk now ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r100397761 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala --- @@ -153,6 +154,12 @@ object Literal { Literal(CatalystTypeConverters.convertToCatalyst(v), dataType) } + def create[T : TypeTag](v: T): Literal = { +val ScalaReflection.Schema(dataType, _) = ScalaReflection.schemaFor[T] +val convert = CatalystTypeConverters.createToCatalystConverter(dataType) --- End diff -- Shouldn't you be going to talks :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r100397651 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala --- @@ -153,6 +154,12 @@ object Literal { Literal(CatalystTypeConverters.convertToCatalyst(v), dataType) } + def create[T : TypeTag](v: T): Literal = { +val ScalaReflection.Schema(dataType, _) = ScalaReflection.schemaFor[T] +val convert = CatalystTypeConverters.createToCatalystConverter(dataType) --- End diff -- Ah ok. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r100395948 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala --- @@ -153,6 +154,12 @@ object Literal { Literal(CatalystTypeConverters.convertToCatalyst(v), dataType) } + def create[T : TypeTag](v: T): Literal = { +val ScalaReflection.Schema(dataType, _) = ScalaReflection.schemaFor[T] +val convert = CatalystTypeConverters.createToCatalystConverter(dataType) --- End diff -- Oh, my bad. I found the code you suggested made some tests fail. Actually, `CatalystTypeConverters.convertToCatalyst` does not do the same thing with `CatalystTypeConverters.createToCatalystConverter(dataType)`. For instance, `CatalystTypeConverters.convertToCatalyst` does not convert `TupleX` to `GenericInternalRow`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r100366614 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala --- @@ -153,6 +154,12 @@ object Literal { Literal(CatalystTypeConverters.convertToCatalyst(v), dataType) } + def create[T : TypeTag](v: T): Literal = { +val ScalaReflection.Schema(dataType, _) = ScalaReflection.schemaFor[T] +val convert = CatalystTypeConverters.createToCatalystConverter(dataType) --- End diff -- Aha, you're right. I'll fix this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r100365381 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -102,6 +102,27 @@ object functions { Column(literalExpr) } + /** + * Creates a [[Column]] of literal value. + * + * The passed in object is returned directly if it is already a [[Column]]. + * If the object is a Scala Symbol, it is converted into a [[Column]] also. + * Otherwise, a new [[Column]] is created to represent the literal value. + * + * @group normal_funcs + * @since 2.2.0 + */ + def lit2[T : TypeTag](literal: T): Column = { --- End diff -- I just named this without strong thoughts, so `typedLit` looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r100360787 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala --- @@ -153,6 +154,12 @@ object Literal { Literal(CatalystTypeConverters.convertToCatalyst(v), dataType) } + def create[T : TypeTag](v: T): Literal = { +val ScalaReflection.Schema(dataType, _) = ScalaReflection.schemaFor[T] +val convert = CatalystTypeConverters.createToCatalystConverter(dataType) --- End diff -- Call `create(v, dataType)` instead? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16610#discussion_r100360526 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -102,6 +102,27 @@ object functions { Column(literalExpr) } + /** + * Creates a [[Column]] of literal value. + * + * The passed in object is returned directly if it is already a [[Column]]. + * If the object is a Scala Symbol, it is converted into a [[Column]] also. + * Otherwise, a new [[Column]] is created to represent the literal value. + * + * @group normal_funcs + * @since 2.2.0 + */ + def lit2[T : TypeTag](literal: T): Column = { --- End diff -- Do you think there is a way we can actually avoid this? If we must why not name it `typedLit`? cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16610: [SPARK-19254][SQL] Support Seq, Map, and Struct i...
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/16610 [SPARK-19254][SQL] Support Seq, Map, and Struct in functions.lit ## What changes were proposed in this pull request? This pr is to support Seq, Map, and Struct in functions.lit; it adds a new IF named `lit2` with `TypeTag` for avoiding type erasure. ## How was this patch tested? Added tests in `LiteralExpressionSuite` You can merge this pull request into a Git repository by running: $ git pull https://github.com/maropu/spark SPARK-19254 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16610.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16610 commit 6a02490745952bd2a5c5b0c84482b5cd874ae820 Author: Takeshi YAMAMURO Date: 2016-11-14T13:21:09Z Add a new create with TypeTag in Literal --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org