[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4186 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72725706 Thanks! Merged to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72590949 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26586/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72590945 [Test build #26586 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26586/consoleFull) for PR 4186 at commit [`adc8311`](https://github.com/apache/spark/commit/adc8311ec8627e2033b163fde0c9768f24872c70). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * `case class MultiAlias(child: Expression, names: Seq[String])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72590366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26584/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72590357 [Test build #26584 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26584/consoleFull) for PR 4186 at commit [`2783aed`](https://github.com/apache/spark/commit/2783aede525f64b0af19b84d526d8a452be031dd). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * `case class MultiAlias(child: Expression, names: Seq[String])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72589331 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26588/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72589324 [Test build #26588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26588/consoleFull) for PR 4186 at commit [`c35e922`](https://github.com/apache/spark/commit/c35e92220efb8f20e4d922b1725aed95a0f76660). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72585466 @marmbrus how about this version? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72583855 [Test build #26588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26588/consoleFull) for PR 4186 at commit [`c35e922`](https://github.com/apache/spark/commit/c35e92220efb8f20e4d922b1725aed95a0f76660). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72582984 [Test build #26586 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26586/consoleFull) for PR 4186 at commit [`adc8311`](https://github.com/apache/spark/commit/adc8311ec8627e2033b163fde0c9768f24872c70). * This patch **does not merge cleanly**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72582547 [Test build #26584 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26584/consoleFull) for PR 4186 at commit [`2783aed`](https://github.com/apache/spark/commit/2783aede525f64b0af19b84d526d8a452be031dd). * This patch **does not merge cleanly**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72548543 Thanks for working on this! I agree with @chenghao-intel, can't you just convert it to a Generate instead of leaving it inside of a `Project` clause? Would be great if this could be updated soon so I can include it in 1.3. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/4186#discussion_r23963316
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala
---
@@ -321,6 +324,18 @@ private[hive] case class HiveGenericUdtf(
override def toString =
s"$nodeName#${funcWrapper.functionClassName}(${children.mkString(",")})"
}
+/**
+ * Resolve Udtfs Alias.
+ */
+private[spark] object ResolveUdtfsAlias extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan) = plan transform {
+case q: LogicalPlan => q transformExpressions {
+ case MultiAlias(udtf@HiveGenericUdtf(_, _, _), names) =>
--- End diff --
When matching without doing extraction you should use this syntax `udtf:
HiveGenericUdtf`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72163260 @scwf oh, I saw that, but it should convert into `Generate`, instead of `SELECT` clause. See https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L419 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72156869 [Test build #26379 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26379/consoleFull) for PR 4186 at commit [`a87668a`](https://github.com/apache/spark/commit/a87668a19a34f5329bb8e46580e39f772aa867df). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class MultiAlias(child: Expression, names: Seq[String])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72156871 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26379/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72152909 [Test build #26379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26379/consoleFull) for PR 4186 at commit [`a87668a`](https://github.com/apache/spark/commit/a87668a19a34f5329bb8e46580e39f772aa867df). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72152519 [Test build #26378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26378/consoleFull) for PR 4186 at commit [`360cb4f`](https://github.com/apache/spark/commit/360cb4f8fcc5101bea2598915443568d1e33a37d). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class MultiAlias(child: Generator, names: Seq[String])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/4186#issuecomment-72152552
If we eliminate the Project too, we can not check "only single UDTF
supported for SELECT clause"
in Batch("Check Analysis").
So here do not change this version, just added a check rule for "single
generator supported for select clause" to CheckMultiAlias.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72152522 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26378/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72152304 [Test build #26378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26378/consoleFull) for PR 4186 at commit [`360cb4f`](https://github.com/apache/spark/commit/360cb4f8fcc5101bea2598915443568d1e33a37d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72150780 Or also eliminate the Project when eliminate the MultiAlias? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72150031 Hey, @chenghao-intel, I think we can not eliminate the `MultiAlias` since Generator (UDTF) is not namedExpression which will leads to java.lang.ClassCastException: org.apache.spark.sql.hive.HiveGenericUdtf cannot be cast to org.apache.spark.sql.catalyst.expressions.NamedExpression Project require the projectList must be NamedExpression. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72144349 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26358/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72144340 [Test build #26358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26358/consoleFull) for PR 4186 at commit [`b25d9b3`](https://github.com/apache/spark/commit/b25d9b30a45234f9631e1ab9501507bd9f2585a4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class MultiAlias(child: Expression, names: Seq[String])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72141891 @chenghao-intel , yeah now i got i. i think this is reasonable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/4186#discussion_r23819673
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala
---
@@ -321,6 +324,18 @@ private[hive] case class HiveGenericUdtf(
override def toString =
s"$nodeName#${funcWrapper.functionClassName}(${children.mkString(",")})"
}
+/**
+ * Resolve Udtfs Alias.
+ */
+private[spark] object ResolveUdtfsAlias extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan) = plan transform {
+case q: LogicalPlan => q transformExpressions {
+ case MultiAlias(udtf@HiveGenericUdtf(_, _, _), names) =>
--- End diff --
Aha, that's good, thanks:)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72141501 @scwf Sorry, I didn't make it clearer, my original idea is: 1) Create a `MultiAlias` (like the `UnresolvedAttribute`), and it carrys the `Expression` and multiple aliases. 2) Resolve the `MultiAlias` during analyzing, and create the real UDTF and set the aliases for UDTF. 3) We also need to `CheckResolution`, only single UDTF supported for `SELECT` clause. The reason I want to eliminate the `MultiAlias` is, all of the `Generator` (UDTF) function actually supports multiple names, the only thing we need to do is passing the right alias names to them (during the analyzing). What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4186#discussion_r23819350
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala
---
@@ -321,6 +324,18 @@ private[hive] case class HiveGenericUdtf(
override def toString =
s"$nodeName#${funcWrapper.functionClassName}(${children.mkString(",")})"
}
+/**
+ * Resolve Udtfs Alias.
+ */
+private[spark] object ResolveUdtfsAlias extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan) = plan transform {
+case q: LogicalPlan => q transformExpressions {
+ case MultiAlias(udtf@HiveGenericUdtf(_, _, _), names) =>
--- End diff --
A better idea is to convert the `MultiAlias(udtf@HiveGenericUdtf(_, _, _),
names)` => `HiveGenericUdtf(_, names, _)`? And the `MultiAlias` just like the
`UnresolvedAttribute`, which will be eliminated after `Analyzing`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72138899 /cc @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-72138018 [Test build #26358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26358/consoleFull) for PR 4186 at commit [`b25d9b3`](https://github.com/apache/spark/commit/b25d9b30a45234f9631e1ab9501507bd9f2585a4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71424264 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26081/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71424257 [Test build #26081 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26081/consoleFull) for PR 4186 at commit [`d38f041`](https://github.com/apache/spark/commit/d38f041261f7bd1c8fa5bc7fa298e52b767db1bf). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class MultiAlias(child: Expression, names: Seq[String])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71420740 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26080/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71420232 [Test build #26081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26081/consoleFull) for PR 4186 at commit [`d38f041`](https://github.com/apache/spark/commit/d38f041261f7bd1c8fa5bc7fa298e52b767db1bf). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71420162 [Test build #26077 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26077/consoleFull) for PR 4186 at commit [`cafbf37`](https://github.com/apache/spark/commit/cafbf37a55fd69f9ecf176878df96520fe85dc2a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class MultiAlias(child: Expression, names: Seq[String])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71420167 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26077/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71416926 hi @chenghao-intel, can you review this again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71416907 [Test build #26077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26077/consoleFull) for PR 4186 at commit [`cafbf37`](https://github.com/apache/spark/commit/cafbf37a55fd69f9ecf176878df96520fe85dc2a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71411316 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26072/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71411314 [Test build #26072 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26072/consoleFull) for PR 4186 at commit [`324150d`](https://github.com/apache/spark/commit/324150df86a567af2baa5528e2e627dd09870bd4). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class MultiAlias(child: Expression, names: Seq[String])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71411257 [Test build #26072 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26072/consoleFull) for PR 4186 at commit [`324150d`](https://github.com/apache/spark/commit/324150df86a567af2baa5528e2e627dd09870bd4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71407130 @scwf yeah, we think we can do that in `Analyzer` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71406698 >And since only single UDTF expression is supported the SELECT clause, we'd better also add a semantic checking in analyzer. @chenghao-intel , you mean check multialias for its children must be udtf? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71404705 @chenghao-intel, actually i have considered your solution and why i use this one is that only UDTF support multi alias in my decription and maybe a liitle redundancy to add a new node for it. But as you said, this one is a little hacky, so i will change it to the solution you suggested to make it more clear, thanks:) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71402394 I believe this will work, but seems a little hacky to me. A better way probably is create a new expression node like MultiAlias. And since only single UDTF expression is supported the SELECT clause, we'd better also add a semantic checking in analyzer. What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4186#discussion_r23509410
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala
---
@@ -321,6 +323,18 @@ private[hive] case class HiveGenericUdtf(
override def toString =
s"$nodeName#${funcWrapper.functionClassName}(${children.mkString(",")})"
}
+/**
+ * Resolve Udtfs Alias.
+ */
+private[spark] object ResolveUdtfsAlias extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan) = plan transform {
+case q: LogicalPlan => q transformExpressions {
+ case Alias(udtf@HiveGenericUdtf(_, _, _), names) if
names.contains("|") =>
+Alias(udtf.copy(aliasNames = names.split("\\|")), names)()
--- End diff --
I believe this will work, but seems a little hacky to me. A better way
probably is create a new expression node like `MultiAlias`. And since only
single UDTF expression is supported the SELECT clause, we'd better also add a
semantic checking in `analyzer`.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71355052 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26062/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71355050 [Test build #26062 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26062/consoleFull) for PR 4186 at commit [`74f5a81`](https://github.com/apache/spark/commit/74f5a8147aac554c53d0fdb28b26535d335038b4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71352726 [Test build #26062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26062/consoleFull) for PR 4186 at commit [`74f5a81`](https://github.com/apache/spark/commit/74f5a8147aac554c53d0fdb28b26535d335038b4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71323103 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26048/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71323100 [Test build #26048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26048/consoleFull) for PR 4186 at commit [`5bc3f59`](https://github.com/apache/spark/commit/5bc3f597cce88c6971336542f3cf6836cafb0d78). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4186#issuecomment-71319773 [Test build #26048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26048/consoleFull) for PR 4186 at commit [`5bc3f59`](https://github.com/apache/spark/commit/5bc3f597cce88c6971336542f3cf6836cafb0d78). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5383][SQL] Support alias for udtfs
GitHub user scwf opened a pull request: https://github.com/apache/spark/pull/4186 [SPARK-5383][SQL] Support alias for udtfs Add support for alias of udtfs, such as ``` select stack(2, key, value, key, value) as (a, b) from src limit 5; select a, b from (select stack(2, key, value, key, value) as (a, b) from src) t limit 5 ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/scwf/spark multi-alias-names Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4186.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4186 commit 3daec28453a400f1057abce622649ff930f931c6 Author: scwf Date: 2015-01-24T14:22:12Z support alias for udfs with multi output columns commit 5bc3f597cce88c6971336542f3cf6836cafb0d78 Author: scwf Date: 2015-01-24T14:25:38Z style fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
