[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11501 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193901130 Added comment, merging this into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193897913 LGTM - but can you add comment explaining why we need the reverse in the code itself? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193892560 @rxin @nongli Is this ready to go? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193671630 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193671638 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52641/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193670547 **[Test build #2617 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2617/consoleFull)** for PR 11501 at commit [`ea41707`](https://github.com/apache/spark/commit/ea417071f468ba3cdda6a6d273abd2fed2d4cf2b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193670461 **[Test build #52641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52641/consoleFull)** for PR 11501 at commit [`ea41707`](https://github.com/apache/spark/commit/ea417071f468ba3cdda6a6d273abd2fed2d4cf2b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193669240 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193669255 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52640/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193668901 **[Test build #52640 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52640/consoleFull)** for PR 11501 at commit [`5765b09`](https://github.com/apache/spark/commit/5765b096b105688af83311b429168c3df834c1d2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193643544 **[Test build #52641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52641/consoleFull)** for PR 11501 at commit [`ea41707`](https://github.com/apache/spark/commit/ea417071f468ba3cdda6a6d273abd2fed2d4cf2b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193642117 **[Test build #2617 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2617/consoleFull)** for PR 11501 at commit [`ea41707`](https://github.com/apache/spark/commit/ea417071f468ba3cdda6a6d273abd2fed2d4cf2b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193641668 **[Test build #52640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52640/consoleFull)** for PR 11501 at commit [`5765b09`](https://github.com/apache/spark/commit/5765b096b105688af83311b429168c3df834c1d2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193637383 Oops there is a conflict. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193637100 **[Test build #2616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2616/consoleFull)** for PR 11501 at commit [`c187554`](https://github.com/apache/spark/commit/c187554382ed1921c29b3a017a8ea6270eeec50f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193595713 **[Test build #2616 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2616/consoleFull)** for PR 11501 at commit [`c187554`](https://github.com/apache/spark/commit/c187554382ed1921c29b3a017a8ea6270eeec50f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193481937 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193481941 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52593/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193481323 **[Test build #52593 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52593/consoleFull)** for PR 11501 at commit [`c187554`](https://github.com/apache/spark/commit/c187554382ed1921c29b3a017a8ea6270eeec50f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193436688 **[Test build #52593 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52593/consoleFull)** for PR 11501 at commit [`c187554`](https://github.com/apache/spark/commit/c187554382ed1921c29b3a017a8ea6270eeec50f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/11501#issuecomment-193436132 @nongli Before this PR, the query will fail to parse if you specify so many predicates for partition columns. In order to run it, you have to remove those predicates, then the number of partitions will go much higher. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11501#discussion_r55263449
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/CatalystQl.scala
---
@@ -661,6 +664,36 @@
https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C
val INTEGRAL = "[+-]?\\d+".r
val DECIMAL = "[+-]?((\\d+(\\.\\d*)?)|(\\.\\d+))".r
+ /**
+ * Flattens the left deep tree with the specified pattern into a list.
+ */
+ private def flattenLeftDeepTree(node: ASTNode, pattern: Regex):
Seq[ASTNode] = {
+val collected = ArrayBuffer[ASTNode]()
+var rest = node
+while (rest match {
+ case Token(pattern(), l :: r :: Nil) =>
+collected += r
+rest = l
+true
+ case _ => false
+}) {}
--- End diff --
maybe add an explicit "// do nothing" comment would help
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11501#discussion_r55254711
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/CatalystQl.scala
---
@@ -661,6 +664,36 @@
https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C
val INTEGRAL = "[+-]?\\d+".r
val DECIMAL = "[+-]?((\\d+(\\.\\d*)?)|(\\.\\d+))".r
+ /**
+ * Flattens the left deep tree with the specified pattern into a list.
+ */
+ private def flattenLeftDeepTree(node: ASTNode, pattern: Regex):
Seq[ASTNode] = {
+val collected = ArrayBuffer[ASTNode]()
+var rest = node
+while (rest match {
+ case Token(pattern(), l :: r :: Nil) =>
+collected += r
+rest = l
+true
+ case _ => false
+}) {}
--- End diff --
Yeah, we already do the dirty work in the condition, to avoid another
match-case in the body of while.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/11501#discussion_r55246967
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/CatalystQl.scala
---
@@ -661,6 +664,36 @@
https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C
val INTEGRAL = "[+-]?\\d+".r
val DECIMAL = "[+-]?((\\d+(\\.\\d*)?)|(\\.\\d+))".r
+ /**
+ * Flattens the left deep tree with the specified pattern into a list.
+ */
+ private def flattenLeftDeepTree(node: ASTNode, pattern: Regex):
Seq[ASTNode] = {
+val collected = ArrayBuffer[ASTNode]()
+var rest = node
+while (rest match {
+ case Token(pattern(), l :: r :: Nil) =>
+collected += r
+rest = l
+true
+ case _ => false
+}) {}
--- End diff --
`{}`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
