[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[
https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715979#comment-16715979
]
ASF GitHub Bot commented on SPARK-26323:
cloud-fan commented on a change in pull request #23275: [SPARK-26323][SQL]
Scala UDF should still check input types even if some inputs are of type Any
URL: https://github.com/apache/spark/pull/23275#discussion_r240451835
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala
##
@@ -88,68 +88,49 @@ sealed trait UserDefinedFunction {
private[sql] case class SparkUserDefinedFunction(
f: AnyRef,
dataType: DataType,
-inputTypes: Option[Seq[DataType]],
-nullableTypes: Option[Seq[Boolean]],
+inputSchemas: Seq[Option[ScalaReflection.Schema]],
name: Option[String] = None,
nullable: Boolean = true,
deterministic: Boolean = true) extends UserDefinedFunction {
@scala.annotation.varargs
- override def apply(exprs: Column*): Column = {
-// TODO: make sure this class is only instantiated through
`SparkUserDefinedFunction.create()`
-// and `nullableTypes` is always set.
-if (inputTypes.isDefined) {
- assert(inputTypes.get.length == nullableTypes.get.length)
-}
-
-val inputsNullSafe = nullableTypes.getOrElse {
- ScalaReflection.getParameterTypeNullability(f)
-}
+ override def apply(cols: Column*): Column = {
+Column(createScalaUDF(cols.map(_.expr)))
+ }
-Column(ScalaUDF(
+ private[sql] def createScalaUDF(exprs: Seq[Expression]): ScalaUDF = {
+// It's possible that some of the inputs don't have a specific type(e.g.
`Any`), skip type
+// check and null check for them.
+val inputTypes = inputSchemas.map(_.map(_.dataType).getOrElse(AnyDataType))
+val inputsNullSafe = inputSchemas.map(_.map(_.nullable).getOrElse(true))
Review comment:
Here `getOrElse` maybe better, as it matches the previous line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> check input types of ScalaUDF even if some inputs are of Any type
> -
>
> Key: SPARK-26323
> URL: https://issues.apache.org/jira/browse/SPARK-26323
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Wenchen Fan
>Assignee: Wenchen Fan
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715966#comment-16715966 ] ASF GitHub Bot commented on SPARK-26323: cloud-fan commented on a change in pull request #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#discussion_r240449840 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ## @@ -47,25 +47,13 @@ case class ScalaUDF( function: AnyRef, dataType: DataType, children: Seq[Expression], -inputsNullSafe: Seq[Boolean], -inputTypes: Seq[DataType] = Nil, +@transient inputsNullSafe: Seq[Boolean], Review comment: expressions are usually serialized to executor side. Previously it's fine, as all data types are case class, which is serializable. But `AnyDataType` is normal scala object, which is not serializable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[
https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715114#comment-16715114
]
ASF GitHub Bot commented on SPARK-26323:
srowen commented on a change in pull request #23275: [SPARK-26323][SQL] Scala
UDF should still check input types even if some inputs are of type Any
URL: https://github.com/apache/spark/pull/23275#discussion_r240298939
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala
##
@@ -88,68 +88,49 @@ sealed trait UserDefinedFunction {
private[sql] case class SparkUserDefinedFunction(
f: AnyRef,
dataType: DataType,
-inputTypes: Option[Seq[DataType]],
-nullableTypes: Option[Seq[Boolean]],
+inputSchemas: Seq[Option[ScalaReflection.Schema]],
name: Option[String] = None,
nullable: Boolean = true,
deterministic: Boolean = true) extends UserDefinedFunction {
@scala.annotation.varargs
- override def apply(exprs: Column*): Column = {
-// TODO: make sure this class is only instantiated through
`SparkUserDefinedFunction.create()`
-// and `nullableTypes` is always set.
-if (inputTypes.isDefined) {
- assert(inputTypes.get.length == nullableTypes.get.length)
-}
-
-val inputsNullSafe = nullableTypes.getOrElse {
- ScalaReflection.getParameterTypeNullability(f)
-}
+ override def apply(cols: Column*): Column = {
+Column(createScalaUDF(cols.map(_.expr)))
+ }
-Column(ScalaUDF(
+ private[sql] def createScalaUDF(exprs: Seq[Expression]): ScalaUDF = {
+// It's possible that some of the inputs don't have a specific type(e.g.
`Any`), skip type
+// check and null check for them.
+val inputTypes = inputSchemas.map(_.map(_.dataType).getOrElse(AnyDataType))
+val inputsNullSafe = inputSchemas.map(_.map(_.nullable).getOrElse(true))
Review comment:
Ah right. I'm neutral on whether it's clearer than getOrElse; I think we end
up using the latter in the code more. I know IJ suggests forall though.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> check input types of ScalaUDF even if some inputs are of Any type
> -
>
> Key: SPARK-26323
> URL: https://issues.apache.org/jira/browse/SPARK-26323
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Wenchen Fan
>Assignee: Wenchen Fan
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715043#comment-16715043 ] ASF GitHub Bot commented on SPARK-26323: SparkQA removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445840083 **[Test build #99920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99920/testReport)** for PR 23275 at commit [`92466d4`](https://github.com/apache/spark/commit/92466d486734f3904be31e45b85e49654eb39255). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715051#comment-16715051 ] ASF GitHub Bot commented on SPARK-26323: AmplabJenkins removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445882601 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99920/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715048#comment-16715048 ] ASF GitHub Bot commented on SPARK-26323: AmplabJenkins removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445882594 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715045#comment-16715045 ] ASF GitHub Bot commented on SPARK-26323: AmplabJenkins commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445882594 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715046#comment-16715046 ] ASF GitHub Bot commented on SPARK-26323: AmplabJenkins commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445882601 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99920/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715041#comment-16715041 ] ASF GitHub Bot commented on SPARK-26323: SparkQA commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445882225 **[Test build #99920 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99920/testReport)** for PR 23275 at commit [`92466d4`](https://github.com/apache/spark/commit/92466d486734f3904be31e45b85e49654eb39255). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715028#comment-16715028 ] ASF GitHub Bot commented on SPARK-26323: AmplabJenkins removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445878940 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99919/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715026#comment-16715026 ] ASF GitHub Bot commented on SPARK-26323: AmplabJenkins removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445878926 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715025#comment-16715025 ] ASF GitHub Bot commented on SPARK-26323: SparkQA removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445836177 **[Test build #99919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99919/testReport)** for PR 23275 at commit [`8582607`](https://github.com/apache/spark/commit/8582607195f12a4c133fb28b59e8a7fce7a97fbb). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715023#comment-16715023 ] ASF GitHub Bot commented on SPARK-26323: AmplabJenkins commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445878926 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715022#comment-16715022 ] ASF GitHub Bot commented on SPARK-26323: SparkQA commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445878673 **[Test build #99919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99919/testReport)** for PR 23275 at commit [`8582607`](https://github.com/apache/spark/commit/8582607195f12a4c133fb28b59e8a7fce7a97fbb). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715024#comment-16715024 ] ASF GitHub Bot commented on SPARK-26323: AmplabJenkins commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-445878940 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99919/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[
https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715013#comment-16715013
]
ASF GitHub Bot commented on SPARK-26323:
mgaido91 commented on a change in pull request #23275: [SPARK-26323][SQL] Scala
UDF should still check input types even if some inputs are of type Any
URL: https://github.com/apache/spark/pull/23275#discussion_r240278143
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala
##
@@ -88,68 +88,49 @@ sealed trait UserDefinedFunction {
private[sql] case class SparkUserDefinedFunction(
f: AnyRef,
dataType: DataType,
-inputTypes: Option[Seq[DataType]],
-nullableTypes: Option[Seq[Boolean]],
+inputSchemas: Seq[Option[ScalaReflection.Schema]],
name: Option[String] = None,
nullable: Boolean = true,
deterministic: Boolean = true) extends UserDefinedFunction {
@scala.annotation.varargs
- override def apply(exprs: Column*): Column = {
-// TODO: make sure this class is only instantiated through
`SparkUserDefinedFunction.create()`
-// and `nullableTypes` is always set.
-if (inputTypes.isDefined) {
- assert(inputTypes.get.length == nullableTypes.get.length)
-}
-
-val inputsNullSafe = nullableTypes.getOrElse {
- ScalaReflection.getParameterTypeNullability(f)
-}
+ override def apply(cols: Column*): Column = {
+Column(createScalaUDF(cols.map(_.expr)))
+ }
-Column(ScalaUDF(
+ private[sql] def createScalaUDF(exprs: Seq[Expression]): ScalaUDF = {
+// It's possible that some of the inputs don't have a specific type(e.g.
`Any`), skip type
+// check and null check for them.
+val inputTypes = inputSchemas.map(_.map(_.dataType).getOrElse(AnyDataType))
+val inputsNullSafe = inputSchemas.map(_.map(_.nullable).getOrElse(true))
Review comment:
I mean `inputSchemas.map(_.forall(_.nullable))`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> check input types of ScalaUDF even if some inputs are of Any type
> -
>
> Key: SPARK-26323
> URL: https://issues.apache.org/jira/browse/SPARK-26323
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Wenchen Fan
>Assignee: Wenchen Fan
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[
https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715003#comment-16715003
]
ASF GitHub Bot commented on SPARK-26323:
srowen commented on a change in pull request #23275: [SPARK-26323][SQL] Scala
UDF should still check input types even if some inputs are of type Any
URL: https://github.com/apache/spark/pull/23275#discussion_r240275041
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala
##
@@ -88,68 +88,49 @@ sealed trait UserDefinedFunction {
private[sql] case class SparkUserDefinedFunction(
f: AnyRef,
dataType: DataType,
-inputTypes: Option[Seq[DataType]],
-nullableTypes: Option[Seq[Boolean]],
+inputSchemas: Seq[Option[ScalaReflection.Schema]],
name: Option[String] = None,
nullable: Boolean = true,
deterministic: Boolean = true) extends UserDefinedFunction {
@scala.annotation.varargs
- override def apply(exprs: Column*): Column = {
-// TODO: make sure this class is only instantiated through
`SparkUserDefinedFunction.create()`
-// and `nullableTypes` is always set.
-if (inputTypes.isDefined) {
- assert(inputTypes.get.length == nullableTypes.get.length)
-}
-
-val inputsNullSafe = nullableTypes.getOrElse {
- ScalaReflection.getParameterTypeNullability(f)
-}
+ override def apply(cols: Column*): Column = {
+Column(createScalaUDF(cols.map(_.expr)))
+ }
-Column(ScalaUDF(
+ private[sql] def createScalaUDF(exprs: Seq[Expression]): ScalaUDF = {
+// It's possible that some of the inputs don't have a specific type(e.g.
`Any`), skip type
+// check and null check for them.
+val inputTypes = inputSchemas.map(_.map(_.dataType).getOrElse(AnyDataType))
+val inputsNullSafe = inputSchemas.map(_.map(_.nullable).getOrElse(true))
Review comment:
I'm missing it, how could you write this more simply with `forall` to get
from `Option[Schema]` to `Boolean`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> check input types of ScalaUDF even if some inputs are of Any type
> -
>
> Key: SPARK-26323
> URL: https://issues.apache.org/jira/browse/SPARK-26323
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Wenchen Fan
>Assignee: Wenchen Fan
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[
https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714906#comment-16714906
]
ASF GitHub Bot commented on SPARK-26323:
mgaido91 commented on a change in pull request #23275: [SPARK-26323][SQL] Scala
UDF should still check input types even if some inputs are of type Any
URL: https://github.com/apache/spark/pull/23275#discussion_r240250904
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala
##
@@ -88,68 +88,49 @@ sealed trait UserDefinedFunction {
private[sql] case class SparkUserDefinedFunction(
f: AnyRef,
dataType: DataType,
-inputTypes: Option[Seq[DataType]],
-nullableTypes: Option[Seq[Boolean]],
+inputSchemas: Seq[Option[ScalaReflection.Schema]],
name: Option[String] = None,
nullable: Boolean = true,
deterministic: Boolean = true) extends UserDefinedFunction {
@scala.annotation.varargs
- override def apply(exprs: Column*): Column = {
-// TODO: make sure this class is only instantiated through
`SparkUserDefinedFunction.create()`
-// and `nullableTypes` is always set.
-if (inputTypes.isDefined) {
- assert(inputTypes.get.length == nullableTypes.get.length)
-}
-
-val inputsNullSafe = nullableTypes.getOrElse {
- ScalaReflection.getParameterTypeNullability(f)
-}
+ override def apply(cols: Column*): Column = {
+Column(createScalaUDF(cols.map(_.expr)))
+ }
-Column(ScalaUDF(
+ private[sql] def createScalaUDF(exprs: Seq[Expression]): ScalaUDF = {
+// It's possible that some of the inputs don't have a specific type(e.g.
`Any`), skip type
+// check and null check for them.
+val inputTypes = inputSchemas.map(_.map(_.dataType).getOrElse(AnyDataType))
+val inputsNullSafe = inputSchemas.map(_.map(_.nullable).getOrElse(true))
Review comment:
nit: forall?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> check input types of ScalaUDF even if some inputs are of Any type
> -
>
> Key: SPARK-26323
> URL: https://issues.apache.org/jira/browse/SPARK-26323
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Wenchen Fan
>Assignee: Wenchen Fan
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714789#comment-16714789 ] Apache Spark commented on SPARK-26323: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/23275 > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26323) check input types of ScalaUDF even if some inputs are of Any type
[ https://issues.apache.org/jira/browse/SPARK-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714790#comment-16714790 ] Apache Spark commented on SPARK-26323: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/23275 > check input types of ScalaUDF even if some inputs are of Any type > - > > Key: SPARK-26323 > URL: https://issues.apache.org/jira/browse/SPARK-26323 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
