[GitHub] [spark] maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE cannot add directories from sql CLI
maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE cannot add directories from sql CLI URL: https://github.com/apache/spark/pull/26863#discussion_r359200277 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ResourcesSuite.scala ## @@ -0,0 +1,49 @@ +/* Review comment: How about naming `RunnableCommandSuite` instead of `ResourcesSuite`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE cannot add directories from sql CLI
maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE
cannot add directories from sql CLI
URL: https://github.com/apache/spark/pull/26863#discussion_r359199567
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/ResourcesSuite.scala
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.io.File
+
+import org.apache.spark.{SparkException, SparkFiles, SparkFunSuite}
+import org.apache.spark.sql.SparkSession
+
+class ResourcesSuite extends SparkFunSuite{
Review comment:
nit: `SparkFunSuite {`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26934: [SPARK-30293][SQL] Remove closeTimestamp!=0 statement since we add a close status
AmplabJenkins commented on issue #26934: [SPARK-30293][SQL] Remove closeTimestamp!=0 statement since we add a close status URL: https://github.com/apache/spark/pull/26934#issuecomment-566919977 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE cannot add directories from sql CLI
maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE
cannot add directories from sql CLI
URL: https://github.com/apache/spark/pull/26863#discussion_r359198977
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/ResourcesSuite.scala
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.io.File
+
+import org.apache.spark.{SparkException, SparkFiles, SparkFunSuite}
+import org.apache.spark.sql.SparkSession
+
+class ResourcesSuite extends SparkFunSuite{
+
+ val sparkSession =
SparkSession.builder().master("local").appName("test_session").getOrCreate()
+
+ test("Add Directory when ADD_DIRECTORY_USING_RECURSIVE not set to true") {
+withTempDir { dir =>
+ val dirPath = dir.getAbsolutePath
+intercept[SparkException] {
+ sparkSession.sql(s"ADD FILE $dir")
+}.getMessage.contains(s" Added file $dirPath is a directory and
recursive is not turned on")
+}
+ }
+
+ test("Add Directory when ADD_DIRECTORY_USING_RECURSIVE set to true") {
+withTempDir { testDir =>
+ val testFile1 = File.createTempFile("testFile", "1", testDir)
+ sparkSession.sql("set spark.sql.addDirectory.recursive=true")
Review comment:
plz use `withSQLConf`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #26934: [SPARK-30293][SQL] Remove closeTimestamp!=0 statement since we add a close status
AngersZh commented on issue #26934: [SPARK-30293][SQL] Remove closeTimestamp!=0 statement since we add a close status URL: https://github.com/apache/spark/pull/26934#issuecomment-566919665 @juliuszsompolski @wangyum This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE cannot add directories from sql CLI
maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE
cannot add directories from sql CLI
URL: https://github.com/apache/spark/pull/26863#discussion_r359198761
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/ResourcesSuite.scala
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.io.File
+
+import org.apache.spark.{SparkException, SparkFiles, SparkFunSuite}
+import org.apache.spark.sql.SparkSession
+
+class ResourcesSuite extends SparkFunSuite{
+
+ val sparkSession =
SparkSession.builder().master("local").appName("test_session").getOrCreate()
+
+ test("Add Directory when ADD_DIRECTORY_USING_RECURSIVE not set to true") {
+withTempDir { dir =>
+ val dirPath = dir.getAbsolutePath
+intercept[SparkException] {
+ sparkSession.sql(s"ADD FILE $dir")
+}.getMessage.contains(s" Added file $dirPath is a directory and
recursive is not turned on")
+}
+ }
+
+ test("Add Directory when ADD_DIRECTORY_USING_RECURSIVE set to true") {
+withTempDir { testDir =>
+ val testFile1 = File.createTempFile("testFile", "1", testDir)
+ sparkSession.sql("set spark.sql.addDirectory.recursive=true")
+ sparkSession.sql(s"ADD FILE $testDir")
+ val sep = File.separator
+ if(!new File(SparkFiles.get(testDir.getName + sep +
testFile1.getName)).exists()) {
+throw new SparkException("TestFile1 Not found.")
Review comment:
plz use `assert`here, too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu opened a new pull request #26934: [SPARK-30293][SQL] Remove closeTimestamp!=0 statement since we add a close status
AngersZh opened a new pull request #26934: [SPARK-30293][SQL] Remove closeTimestamp!=0 statement since we add a close status URL: https://github.com/apache/spark/pull/26934 ### What changes were proposed in this pull request? Since we have add a close status for each statement, here we should use closeTimestamp. If not when too many statement , we may remove statement that not closed, then if client call CloseStatement, we will get an error when call `onStatementClosed()` since we don't have this statement's execution info in executionList. ### Why are the changes needed? Fix bug ### Does this PR introduce any user-facing change? NO ### How was this patch tested? Don't need UT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE cannot add directories from sql CLI
maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE
cannot add directories from sql CLI
URL: https://github.com/apache/spark/pull/26863#discussion_r359198588
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -2098,6 +2098,13 @@ object SQLConf {
"defined by `from` and `to`.")
.booleanConf
.createWithDefault(false)
+
+ val ADD_DIRECTORY_USING_RECURSIVE =
+buildConf("spark.sql.addDirectory.recursive")
+ .doc(" When true, users can add directory by passing path of a directory
to ADD FILE" +
Review comment:
Also, it seems we have a space in the end.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE cannot add directories from sql CLI
maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE
cannot add directories from sql CLI
URL: https://github.com/apache/spark/pull/26863#discussion_r359198707
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/ResourcesSuite.scala
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.io.File
+
+import org.apache.spark.{SparkException, SparkFiles, SparkFunSuite}
+import org.apache.spark.sql.SparkSession
+
+class ResourcesSuite extends SparkFunSuite{
+
+ val sparkSession =
SparkSession.builder().master("local").appName("test_session").getOrCreate()
+
+ test("Add Directory when ADD_DIRECTORY_USING_RECURSIVE not set to true") {
+withTempDir { dir =>
+ val dirPath = dir.getAbsolutePath
+intercept[SparkException] {
+ sparkSession.sql(s"ADD FILE $dir")
+}.getMessage.contains(s" Added file $dirPath is a directory and
recursive is not turned on")
Review comment:
plz use `assert`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE cannot add directories from sql CLI
maropu commented on a change in pull request #26863: [SPARK-30234]ADD FILE
cannot add directories from sql CLI
URL: https://github.com/apache/spark/pull/26863#discussion_r359198300
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -2098,6 +2098,13 @@ object SQLConf {
"defined by `from` and `to`.")
.booleanConf
.createWithDefault(false)
+
+ val ADD_DIRECTORY_USING_RECURSIVE =
+buildConf("spark.sql.addDirectory.recursive")
+ .doc(" When true, users can add directory by passing path of a directory
to ADD FILE" +
Review comment:
nit: drop a space in the head.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extendedto be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566915417 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115492/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
SparkQA removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566914862 **[Test build #115492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115492/testReport)** for PR 26927 at commit [`af0336f`](https://github.com/apache/spark/commit/af0336f0ca461a20a0e4b2f83c362fdd6605bd45). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extendedto be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566915410 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extendedto be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566915209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20295/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
SparkQA commented on issue #26927: [SPARK-29505][SQL] desc extendedto be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566915400 **[Test build #115492 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115492/testReport)** for PR 26927 at commit [`af0336f`](https://github.com/apache/spark/commit/af0336f0ca461a20a0e4b2f83c362fdd6605bd45). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566915417 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115492/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566915410 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extendedto be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566915204 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566915204 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566915209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20295/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
SparkQA commented on issue #26927: [SPARK-29505][SQL] desc extendedto be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566914862 **[Test build #115492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115492/testReport)** for PR 26927 at commit [`af0336f`](https://github.com/apache/spark/commit/af0336f0ca461a20a0e4b2f83c362fdd6605bd45). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangshuo128 commented on a change in pull request #26924: [SPARK-30285][CORE]Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError
wangshuo128 commented on a change in pull request #26924:
[SPARK-30285][CORE]Fix deadlock between LiveListenerBus#stop and
AsyncEventQueue#removeListenerOnError
URL: https://github.com/apache/spark/pull/26924#discussion_r359193483
##
File path: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala
##
@@ -201,10 +201,24 @@ private class AsyncEventQueue(
true
}
+ override def doPostEvent(listener: SparkListenerInterface, event:
SparkListenerEvent): Unit = {
+// If listener is dead, we don't post any event to it.
+if (!listener.dead) {
+ super.doPostEvent(listener, event)
+}
+ }
+
override def removeListenerOnError(listener: SparkListenerInterface): Unit =
{
-// the listener failed in an unrecoverably way, we want to remove it from
the entire
-// LiveListenerBus (potentially stopping a queue if it is empty)
-bus.removeListener(listener)
+if (bus.isInStop) {
+ // If bus is in the progress of stop, we just mark the listener as dead
instead of removing
+ // via calling `bus.removeListener` to avoid race condition
+ // dead listeners will be removed eventually in `bus.stop`
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26512: [SPARK-29493][SQL] Arrow MapType support
SparkQA commented on issue #26512: [SPARK-29493][SQL] Arrow MapType support URL: https://github.com/apache/spark/pull/26512#issuecomment-566912642 **[Test build #115491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115491/testReport)** for PR 26512 at commit [`76b2a93`](https://github.com/apache/spark/commit/76b2a93f92c6e29ad8889811d60ea7c608421963). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26512: [SPARK-29493][SQL] Arrow MapType support
AmplabJenkins commented on issue #26512: [SPARK-29493][SQL] Arrow MapType support URL: https://github.com/apache/spark/pull/26512#issuecomment-566905786 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26512: [SPARK-29493][SQL] Arrow MapType support
AmplabJenkins removed a comment on issue #26512: [SPARK-29493][SQL] Arrow MapType support URL: https://github.com/apache/spark/pull/26512#issuecomment-566905786 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26512: [SPARK-29493][SQL] Arrow MapType support
AmplabJenkins removed a comment on issue #26512: [SPARK-29493][SQL] Arrow MapType support URL: https://github.com/apache/spark/pull/26512#issuecomment-566905812 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20294/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26512: [SPARK-29493][SQL] Arrow MapType support
AmplabJenkins commented on issue #26512: [SPARK-29493][SQL] Arrow MapType support URL: https://github.com/apache/spark/pull/26512#issuecomment-566905812 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20294/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jalpan-randeri commented on a change in pull request #26512: [SPARK-29493][SQL] Arrow MapType support
jalpan-randeri commented on a change in pull request #26512: [SPARK-29493][SQL]
Arrow MapType support
URL: https://github.com/apache/spark/pull/26512#discussion_r359189091
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1525,7 +1525,7 @@ object SQLConf {
"3. dapply " +
"4. gapply " +
"The following data types are unsupported: " +
-"FloatType, BinaryType, ArrayType, StructType and MapType.")
+"FloatType, ArrayType, and StructType.")
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jalpan-randeri commented on a change in pull request #26512: [SPARK-29493][SQL] Arrow MapType support
jalpan-randeri commented on a change in pull request #26512: [SPARK-29493][SQL] Arrow MapType support URL: https://github.com/apache/spark/pull/26512#discussion_r35914 ## File path: docs/sql-pyspark-pandas-with-arrow.md ## @@ -217,7 +217,7 @@ For detailed usage, please see [`pyspark.sql.functions.pandas_udf`](api/python/p ### Supported SQL Types -Currently, all Spark SQL data types are supported by Arrow-based conversion except `MapType`, +Currently, all Spark SQL data types are supported by Arrow-based conversion except Review comment: Done. Reverted last 2 commits This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
maropu commented on a change in pull request #26918: [SPARK-30279][SQL] Support
32 or more grouping attributes for GROUPING_ID
URL: https://github.com/apache/spark/pull/26918#discussion_r359188374
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -563,15 +570,44 @@ class Analyzer(
}
}
+private object EmptyGroupingIDExtractor {
+
+ private def hasEmptyGroupingID(aggExprs: Seq[Expression]): Boolean = {
+aggExprs.exists { p => p.collectFirst { case GroupingID(Nil) => true
}.isDefined }
+ }
+
+ def unapply(p: LogicalPlan): Option[Seq[Expression]] = p match {
+case Aggregate(Seq(Cube(groupByExprs)), aggExprs, _) if
hasEmptyGroupingID(aggExprs) =>
+ Some(constructGroupBy(cubeExprs(groupByExprs)))
+case Aggregate(Seq(Rollup(groupByExprs)), aggExprs, _) if
hasEmptyGroupingID(aggExprs) =>
+ Some(constructGroupBy(rollupExprs(groupByExprs)))
+case x: GroupingSets if hasEmptyGroupingID(x.aggregations) =>
+ Some(constructGroupBy(x.selectedGroupByExprs))
+case Filter(cond, child) if hasEmptyGroupingID(cond :: Nil) =>
+ Some(findGroupingExprs(child))
+case Sort(order, _, child) if hasEmptyGroupingID(order) =>
+ Some(findGroupingExprs(child))
+case _ =>
+ None
+ }
+}
+
// This require transformUp to replace grouping()/grouping_id() in
resolved Filter/Sort
def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
case a if !a.childrenResolved => a // be sure all of the children are
resolved.
+ // If a plan has unresolved grouping IDs, we resolve them first
+ case p @ EmptyGroupingIDExtractor(groupByExprs) =>
+p.transformExpressions {
+ // In case that `GroupingID` has an empty group-by expressions
+ case gid: GroupingID if !gid.resolved => GroupingID(groupByExprs)
+}
Review comment:
As you said, I think now we don't need this handling for string grouping
IDs. Rather, we need to use `bigint` for grouping IDs intead of `int`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
maropu commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566900211 I checked the hive behaivour; hive uses not `int` but `bigint` for grouping IDs, so hive supports less than 64 grouping sets. And then, if a query has more than 65 grouping sets, hive thorws an unsupported exception below; ``` hive> select GROUPING__ID, sum(v) from test8 group by c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,c14,c15,c16,c17,c18,c19,c20,c21,c22,c23,c24,c25,c26,c27,c28,c29,c30,c31,c32,c33,c34,c35,c36,c37,c38,c39,c40,c41,c42,c43,c44,c45,c46,c47,c48,c49,c50,c51,c52,c53,c54,c55,c56,c57,c58,c59,c60,c61,c62,c63,c64,c65,c66,c67,c68,c69,c70,c71,c72,c73,c74,c75,c76,c77,c78,c79 grouping sets ((c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,c14,c15,c16,c17,c18,c19,c20,c21,c22,c23,c24,c25,c26,c27,c28,c29,c30,c31,c32,c33,c34,c35,c36,c37,c38,c39,c40,c41,c42,c43,c44,c45,c46,c47,c48,c49,c50,c51,c52,c53,c54,c55,c56,c57,c58,c59,c60,c61,c62,c63,c64,c65,c66,c67,c68,c69,c70,c71,c72,c73,c74,c75,c76,c77,c78,c79), (c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,c14,c15,c16,c17,c18,c19,c20,c21,c22,c23,c24,c25,c26,c27,c28,c29,c30,c31,c32,c33,c34,c35,c36,c37,c38,c39,c40,c41,c42,c43,c44,c45,c46,c47,c48,c49,c50,c51,c52,c53,c54,c55,c56,c57,c58,c59,c60,c61,c62,c63,c64,c65,c66,c67,c68,c69,c70,c71,c72,c73,c74,c75,c76,c77,c78), (c0)); FAILED: SemanticException [Error 10411]: Grouping sets size cannot be greater than 64 ``` This looks reasonable to me, If nobody is againt this hive behaivour, I'll fix this pr in that way. WDYT, @viirya ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #22316: [SPARK-25048][SQL] Pivoting by multiple columns in Scala/Java
HyukjinKwon commented on issue #22316: [SPARK-25048][SQL] Pivoting by multiple
columns in Scala/Java
URL: https://github.com/apache/spark/pull/22316#issuecomment-566888624
You can try:
```python
df.groupby(...).pivot(..., values=[F.struct(F.lit("..."))._jc])
```
for now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
viirya commented on a change in pull request #26918: [SPARK-30279][SQL] Support
32 or more grouping attributes for GROUPING_ID
URL: https://github.com/apache/spark/pull/26918#discussion_r359170592
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -563,15 +570,44 @@ class Analyzer(
}
}
+private object EmptyGroupingIDExtractor {
+
+ private def hasEmptyGroupingID(aggExprs: Seq[Expression]): Boolean = {
+aggExprs.exists { p => p.collectFirst { case GroupingID(Nil) => true
}.isDefined }
+ }
+
+ def unapply(p: LogicalPlan): Option[Seq[Expression]] = p match {
+case Aggregate(Seq(Cube(groupByExprs)), aggExprs, _) if
hasEmptyGroupingID(aggExprs) =>
+ Some(constructGroupBy(cubeExprs(groupByExprs)))
+case Aggregate(Seq(Rollup(groupByExprs)), aggExprs, _) if
hasEmptyGroupingID(aggExprs) =>
+ Some(constructGroupBy(rollupExprs(groupByExprs)))
+case x: GroupingSets if hasEmptyGroupingID(x.aggregations) =>
+ Some(constructGroupBy(x.selectedGroupByExprs))
+case Filter(cond, child) if hasEmptyGroupingID(cond :: Nil) =>
+ Some(findGroupingExprs(child))
+case Sort(order, _, child) if hasEmptyGroupingID(order) =>
+ Some(findGroupingExprs(child))
+case _ =>
+ None
+ }
+}
+
// This require transformUp to replace grouping()/grouping_id() in
resolved Filter/Sort
def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
case a if !a.childrenResolved => a // be sure all of the children are
resolved.
+ // If a plan has unresolved grouping IDs, we resolve them first
+ case p @ EmptyGroupingIDExtractor(groupByExprs) =>
+p.transformExpressions {
+ // In case that `GroupingID` has an empty group-by expressions
+ case gid: GroupingID if !gid.resolved => GroupingID(groupByExprs)
+}
Review comment:
Why is this needed? We decide gid datatype by `GroupingID.groupIdDataType`
above, isn't it enough?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc
HyukjinKwon edited a comment on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc URL: https://github.com/apache/spark/pull/26922#issuecomment-566886930 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc
HyukjinKwon commented on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc URL: https://github.com/apache/spark/pull/26922#issuecomment-566886930 Merged to master and branch-2.4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26933: Throw Exception when invalid string is cast to decimal in ANSI mode
AmplabJenkins commented on issue #26933: Throw Exception when invalid string is cast to decimal in ANSI mode URL: https://github.com/apache/spark/pull/26933#issuecomment-566886144 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-566886284 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-566886284 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26933: Throw Exception when invalid string is cast to decimal in ANSI mode
AmplabJenkins removed a comment on issue #26933: Throw Exception when invalid string is cast to decimal in ANSI mode URL: https://github.com/apache/spark/pull/26933#issuecomment-566885779 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-566886292 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20293/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-566886292 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20293/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
SparkQA commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-566885900 **[Test build #115490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115490/testReport)** for PR 26656 at commit [`b3584c8`](https://github.com/apache/spark/commit/b3584c868321a8732d760e8c71587dab15ff9259). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26933: Throw Exception when invalid string is cast to decimal in ANSI mode
AmplabJenkins commented on issue #26933: Throw Exception when invalid string is cast to decimal in ANSI mode URL: https://github.com/apache/spark/pull/26933#issuecomment-566885779 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] iRakson edited a comment on issue #26933: Throw Exception when invalid string is cast to decimal in ANSI mode
iRakson edited a comment on issue #26933: Throw Exception when invalid string is cast to decimal in ANSI mode URL: https://github.com/apache/spark/pull/26933#issuecomment-566884491 cc @cloud-fan #26518 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] iRakson commented on issue #26933: Throw Exception when invalid string is cast to decimal in ANSI mode
iRakson commented on issue #26933: Throw Exception when invalid string is cast to decimal in ANSI mode URL: https://github.com/apache/spark/pull/26933#issuecomment-566884491 cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] iRakson opened a new pull request #26933: Throw Exception when invalid string is cast to decimal in ANSI mode
iRakson opened a new pull request #26933: Throw Exception when invalid string
is cast to decimal in ANSI mode
URL: https://github.com/apache/spark/pull/26933
### What changes were proposed in this pull request?
If spark.sql.ansi.enabled is set,
Throw analysis exception when cast to decimal do not follow the ANSI casting
standards.
### Why are the changes needed?
ANSI SQL standards do not allow invalid strings to get casted into decimal
and throw exception for that. Currently spark sql gives NULL in such cases.
Before:
`select cast('str' as decimal) => NULL`
After :
`select cast('str' as decimal) => invalid input syntax for type numeric: str`
These results are after setting `spark.sql.ansi.enabled=true`
### Does this PR introduce any user-facing change?
Yes. Now when ansi mode is on users will get analysis exception for invalid
strings.
### How was this patch tested?
Manually. Test cases will added soon.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
imback82 commented on a change in pull request #26923: [SPARK-30284][SQL]
CREATE VIEW should keep the current catalog and namespace
URL: https://github.com/apache/spark/pull/26923#discussion_r358961919
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
##
@@ -1929,6 +1929,21 @@ class DataSourceV2SQLSuite
}
}
+ test("SPARK-30284: CREATE VIEW should track the current catalog and
namespace") {
+// unset this config to use the default v2 session catalog.
+spark.conf.unset(V2_SESSION_CATALOG_IMPLEMENTATION.key)
+
+withView("v") {
+ sql("USE testcat.ns1.ns2")
+ sql("CREATE TABLE t USING foo AS SELECT 1")
Review comment:
Can we also create temp view `t` and make sure it's not used? (may not be
the scope of this PR if it doesn't work)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #26921: [SPARK-30282][SQL] UnresolvedV2Relation should be resolved to temp view first
imback82 commented on a change in pull request #26921: [SPARK-30282][SQL]
UnresolvedV2Relation should be resolved to temp view first
URL: https://github.com/apache/spark/pull/26921#discussion_r359164909
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -759,22 +761,29 @@ class Analyzer(
.getOrElse(i)
case desc @ DescribeTable(u: UnresolvedV2Relation, _) =>
-CatalogV2Util.loadRelation(u.catalog, u.tableName)
-.map(rel => desc.copy(table = rel))
-.getOrElse(desc)
+resolveV2Relation(u).map(rel => desc.copy(table = rel)).getOrElse(desc)
case alter @ AlterTable(_, _, u: UnresolvedV2Relation, _) =>
-CatalogV2Util.loadRelation(u.catalog, u.tableName)
-.map(rel => alter.copy(table = rel))
-.getOrElse(alter)
+resolveV2Relation(u).map(rel => alter.copy(table =
rel)).getOrElse(alter)
case show @ ShowTableProperties(u: UnresolvedV2Relation, _) =>
-CatalogV2Util.loadRelation(u.catalog, u.tableName)
- .map(rel => show.copy(table = rel))
- .getOrElse(show)
+resolveV2Relation(u).map(rel => show.copy(table = rel)).getOrElse(show)
case u: UnresolvedV2Relation =>
-CatalogV2Util.loadRelation(u.catalog, u.tableName).getOrElse(u)
+resolveV2Relation(u).getOrElse(u)
+}
+
+private def resolveV2Relation(unresolved: UnresolvedV2Relation) :
Option[NamedRelation] = {
+ val maybeTempView = unresolved.originalNameParts match {
+case Seq(part) => v1SessionCatalog.lookupTempView(part)
+case _ => None
+ }
+ if (maybeTempView.isDefined) {
+unresolved.failAnalysis(
+ s"Invalid command: '${unresolved.originalNameParts.quoted}' is a
view not a table.")
Review comment:
I left this to the same message used in checkAnalysis to minimize the
changes in tests. Please let me know if this neds to updated to more
descriptive message such as 'A temp view 't' cannot be handled by V2 commands'
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on issue #26921: [SPARK-30282][SQL] UnresolvedV2Relation should be resolved to temp view first
imback82 commented on issue #26921: [SPARK-30282][SQL] UnresolvedV2Relation should be resolved to temp view first URL: https://github.com/apache/spark/pull/26921#issuecomment-566881435 cc: @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu edited a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
maropu edited a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566880237 > Should we just use string type for grouping id? A consistent type of grouping id sounds more reliable Yea, I think so. But, I have no idea now about the way to keep the type consistent while supporting it. > Do other systems (Hive, Presto...) have similar issue? ok, I will check later. (plz let me know if someone already knows it) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu edited a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
maropu edited a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566880237 > Should we just use string type for grouping id? A consistent type of grouping id sounds more reliable Yea, I think so. But, I have no idea now about the way to keep the type consistent while supporting it. > Do other systems (Hive, Presto...) have similar issue? We need to check that behaviour. I will check later. (plz let me know if someone already knows it) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
maropu commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566880237 > Should we just use string type for grouping id? A consistent type of grouping id sounds more reliable Yea, I think so. But, I have no idea about keeping the type consistent while supporting it. > Do other systems (Hive, Presto...) have similar issue? We need to check that behaviour. I will check later. (plz let me know if someone already knows it) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
imback82 commented on a change in pull request #26923: [SPARK-30284][SQL]
CREATE VIEW should keep the current catalog and namespace
URL: https://github.com/apache/spark/pull/26923#discussion_r359157149
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
##
@@ -368,7 +388,27 @@ case class CatalogTable(
}
object CatalogTable {
+ // Starting from Spark 3.0, we don't use this property any more.
`VIEW_TABLE_NAME_CONTEXT` is
Review comment:
Did you define `VIEW_TABLE_NAME_CONTEXT` somewhere or did you mean
`VIEW_CATALOG_AND_NAMESPACE`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
imback82 commented on a change in pull request #26923: [SPARK-30284][SQL]
CREATE VIEW should keep the current catalog and namespace
URL: https://github.com/apache/spark/pull/26923#discussion_r358960206
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -795,116 +807,60 @@ class Analyzer(
* Replaces [[UnresolvedRelation]]s with concrete relations from the catalog.
*/
object ResolveRelations extends Rule[LogicalPlan] {
+def apply(plan: LogicalPlan): LogicalPlan =
ResolveTempViews(plan).resolveOperatorsUp {
+ case i @ InsertIntoStatement(table, _, _, _, _) if i.query.resolved =>
+val relation = table match {
+ case u: UnresolvedRelation =>
+lookupRelation(u.multipartIdentifier).getOrElse(u)
+ case other => other
+}
-// If an unresolved relation is given, it is looked up from the session
catalog and either v1
-// or v2 relation is returned. Otherwise, we look up the table from catalog
-// and change the default database name (in AnalysisContext) if it is a
view.
-// We usually look up a table from the default database if the table
identifier has an empty
-// database part, for a view the default database should be the currentDb
when the view was
-// created. When the case comes to resolving a nested view, the view may
have different default
-// database with that the referenced view has, so we need to use
-// `AnalysisContext.defaultDatabase` to track the current default database.
-// When the relation we resolve is a view, we fetch the view.desc(which is
a CatalogTable), and
-// then set the value of `CatalogTable.viewDefaultDatabase` to
-// `AnalysisContext.defaultDatabase`, we look up the relations that the
view references using
-// the default database.
-// For example:
-// |- view1 (defaultDatabase = db1)
-// |- operator
-// |- table2 (defaultDatabase = db1)
-// |- view2 (defaultDatabase = db2)
-//|- view3 (defaultDatabase = db3)
-// |- view4 (defaultDatabase = db4)
-// In this case, the view `view1` is a nested view, it directly references
`table2`, `view2`
-// and `view4`, the view `view2` references `view3`. On resolving the
table, we look up the
-// relations `table2`, `view2`, `view4` using the default database `db1`,
and look up the
-// relation `view3` using the default database `db2`.
-//
-// Note this is compatible with the views defined by older versions of
Spark(before 2.2), which
-// have empty defaultDatabase and all the relations in viewText have
database part defined.
-def resolveRelation(plan: LogicalPlan): LogicalPlan = plan match {
- case u @ UnresolvedRelation(SessionCatalogAndIdentifier(catalog, ident))
=>
-lookupRelation(catalog, ident, recurse = true).getOrElse(u)
+EliminateSubqueryAliases(relation) match {
+ case v: View =>
+table.failAnalysis(s"Inserting into a view is not allowed. View:
${v.desc.identifier}.")
+ case other => i.copy(table = other)
+}
+
+ case u: UnresolvedRelation =>
+lookupRelation(u.multipartIdentifier).map(tryResolveViews).getOrElse(u)
+}
+
+private def tryResolveViews(plan: LogicalPlan): LogicalPlan = plan match {
+ case p @ SubqueryAlias(_, view: View) =>
+p.copy(child = tryResolveViews(view))
// The view's child should be a logical plan parsed from the
`desc.viewText`, the variable
// `viewText` should be defined, or else we throw an error on the
generation of the View
// operator.
case view @ View(desc, _, child) if !child.resolved =>
// Resolve all the UnresolvedRelations and Views in the child.
-val newChild =
AnalysisContext.withAnalysisContext(desc.viewDefaultDatabase) {
+val newChild =
AnalysisContext.withAnalysisContext(desc.viewCatalogAndNamespace) {
if (AnalysisContext.get.nestedViewDepth > conf.maxNestedViewDepth) {
-view.failAnalysis(s"The depth of view ${view.desc.identifier}
exceeds the maximum " +
+view.failAnalysis(s"The depth of view ${desc.identifier} exceeds
the maximum " +
s"view resolution depth (${conf.maxNestedViewDepth}). Analysis
is aborted to " +
s"avoid errors. Increase the value of
${SQLConf.MAX_NESTED_VIEW_DEPTH.key} to work " +
"around this.")
}
executeSameContext(child)
}
view.copy(child = newChild)
- case p @ SubqueryAlias(_, view: View) =>
-val newChild = resolveRelation(view)
-p.copy(child = newChild)
- case _ => plan
-}
-def apply(plan: LogicalPlan): LogicalPlan =
ResolveTempViews(plan).resolveOperatorsUp {
- case i @ InsertIntoStatement(table, _, _, _, _) if i.query.resolved =>
-val relation = table match {
-
[GitHub] [spark] imback82 commented on a change in pull request #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
imback82 commented on a change in pull request #26923: [SPARK-30284][SQL]
CREATE VIEW should keep the current catalog and namespace
URL: https://github.com/apache/spark/pull/26923#discussion_r358961919
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
##
@@ -1929,6 +1929,21 @@ class DataSourceV2SQLSuite
}
}
+ test("SPARK-30284: CREATE VIEW should track the current catalog and
namespace") {
+// unset this config to use the default v2 session catalog.
+spark.conf.unset(V2_SESSION_CATALOG_IMPLEMENTATION.key)
+
+withView("v") {
+ sql("USE testcat.ns1.ns2")
+ sql("CREATE TABLE t USING foo AS SELECT 1")
Review comment:
Can we also create temp view `t` and make sure it's not used?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
imback82 commented on a change in pull request #26923: [SPARK-30284][SQL]
CREATE VIEW should keep the current catalog and namespace
URL: https://github.com/apache/spark/pull/26923#discussion_r359162122
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -795,116 +807,60 @@ class Analyzer(
* Replaces [[UnresolvedRelation]]s with concrete relations from the catalog.
*/
object ResolveRelations extends Rule[LogicalPlan] {
+def apply(plan: LogicalPlan): LogicalPlan =
ResolveTempViews(plan).resolveOperatorsUp {
+ case i @ InsertIntoStatement(table, _, _, _, _) if i.query.resolved =>
+val relation = table match {
+ case u: UnresolvedRelation =>
+lookupRelation(u.multipartIdentifier).getOrElse(u)
+ case other => other
+}
-// If an unresolved relation is given, it is looked up from the session
catalog and either v1
-// or v2 relation is returned. Otherwise, we look up the table from catalog
-// and change the default database name (in AnalysisContext) if it is a
view.
-// We usually look up a table from the default database if the table
identifier has an empty
-// database part, for a view the default database should be the currentDb
when the view was
-// created. When the case comes to resolving a nested view, the view may
have different default
-// database with that the referenced view has, so we need to use
-// `AnalysisContext.defaultDatabase` to track the current default database.
-// When the relation we resolve is a view, we fetch the view.desc(which is
a CatalogTable), and
-// then set the value of `CatalogTable.viewDefaultDatabase` to
-// `AnalysisContext.defaultDatabase`, we look up the relations that the
view references using
-// the default database.
-// For example:
-// |- view1 (defaultDatabase = db1)
-// |- operator
-// |- table2 (defaultDatabase = db1)
-// |- view2 (defaultDatabase = db2)
-//|- view3 (defaultDatabase = db3)
-// |- view4 (defaultDatabase = db4)
-// In this case, the view `view1` is a nested view, it directly references
`table2`, `view2`
-// and `view4`, the view `view2` references `view3`. On resolving the
table, we look up the
-// relations `table2`, `view2`, `view4` using the default database `db1`,
and look up the
-// relation `view3` using the default database `db2`.
-//
-// Note this is compatible with the views defined by older versions of
Spark(before 2.2), which
-// have empty defaultDatabase and all the relations in viewText have
database part defined.
-def resolveRelation(plan: LogicalPlan): LogicalPlan = plan match {
- case u @ UnresolvedRelation(SessionCatalogAndIdentifier(catalog, ident))
=>
-lookupRelation(catalog, ident, recurse = true).getOrElse(u)
+EliminateSubqueryAliases(relation) match {
+ case v: View =>
+table.failAnalysis(s"Inserting into a view is not allowed. View:
${v.desc.identifier}.")
+ case other => i.copy(table = other)
+}
+
+ case u: UnresolvedRelation =>
+lookupRelation(u.multipartIdentifier).map(tryResolveViews).getOrElse(u)
+}
+
+private def tryResolveViews(plan: LogicalPlan): LogicalPlan = plan match {
Review comment:
nit: just `resolveViews` sounds more consistent in this context.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
imback82 commented on a change in pull request #26923: [SPARK-30284][SQL]
CREATE VIEW should keep the current catalog and namespace
URL: https://github.com/apache/spark/pull/26923#discussion_r358962641
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -344,9 +337,9 @@ object ViewHelper {
fieldNames, "in the view definition", session.sessionState.conf.resolver)
// Generate the view default database name.
-val viewDefaultDatabase = session.sessionState.catalog.getCurrentDatabase
+val manager = session.sessionState.catalogManager
removeQueryColumnNames(properties) ++
- generateViewDefaultDatabase(viewDefaultDatabase) ++
+ catalogAndNamespaceToProps(manager.currentCatalog.name(),
manager.currentNamespace) ++
Review comment:
nit: `name` instead of `name()` to be consistent?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE
AmplabJenkins commented on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE URL: https://github.com/apache/spark/pull/26931#issuecomment-566877443 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115481/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE
AmplabJenkins commented on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE URL: https://github.com/apache/spark/pull/26931#issuecomment-566877439 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE
AmplabJenkins removed a comment on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE URL: https://github.com/apache/spark/pull/26931#issuecomment-566877439 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE
AmplabJenkins removed a comment on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE URL: https://github.com/apache/spark/pull/26931#issuecomment-566877443 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115481/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE
SparkQA removed a comment on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE URL: https://github.com/apache/spark/pull/26931#issuecomment-566831365 **[Test build #115481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115481/testReport)** for PR 26931 at commit [`4b71858`](https://github.com/apache/spark/commit/4b718581c73494481f8743c7c637eb69be99dafa). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
AmplabJenkins commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566877001 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
AmplabJenkins removed a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566877001 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
AmplabJenkins removed a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566877008 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20292/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
AmplabJenkins commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566877008 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20292/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE
SparkQA commented on issue #26931: [SPARK-30291] catch the exception when doing materialize in AQE URL: https://github.com/apache/spark/pull/26931#issuecomment-566877035 **[Test build #115481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115481/testReport)** for PR 26931 at commit [`4b71858`](https://github.com/apache/spark/commit/4b718581c73494481f8743c7c637eb69be99dafa). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
SparkQA commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566876665 **[Test build #115489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115489/testReport)** for PR 26918 at commit [`85f931c`](https://github.com/apache/spark/commit/85f931c1d6de1bbe88235d7498c9ef1fc61bb517). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
viirya commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566876700 Do other systems (Hive, Presto...) have similar issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc
AmplabJenkins removed a comment on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc URL: https://github.com/apache/spark/pull/26922#issuecomment-566876646 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115482/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc
AmplabJenkins removed a comment on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc URL: https://github.com/apache/spark/pull/26922#issuecomment-566876634 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc
AmplabJenkins commented on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc URL: https://github.com/apache/spark/pull/26922#issuecomment-566876646 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115482/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on issue #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer commented on issue #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#issuecomment-566876474 @cloud-fan find the comment below https://issues.apache.org/jira/browse/SPARK-29600?focusedCommentId=16998413=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16998413 Thanks all. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc
AmplabJenkins commented on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc URL: https://github.com/apache/spark/pull/26922#issuecomment-566876634 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
viirya commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-566876398 Should we just use string type for grouping id? A consistent type of grouping id sounds more reliable. cc @cloud-fan @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc
SparkQA commented on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc URL: https://github.com/apache/spark/pull/26922#issuecomment-566876277 **[Test build #115482 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115482/testReport)** for PR 26922 at commit [`2c1b3e2`](https://github.com/apache/spark/commit/2c1b3e273c7028ec1f0cdc041a5ab54c7f76d048). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc
SparkQA removed a comment on issue #26922: [MINOR][SQL][DOC] Fix some format issues in Dataset API Doc URL: https://github.com/apache/spark/pull/26922#issuecomment-566831392 **[Test build #115482 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115482/testReport)** for PR 26922 at commit [`2c1b3e2`](https://github.com/apache/spark/commit/2c1b3e273c7028ec1f0cdc041a5ab54c7f76d048). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer removed a comment on issue #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer removed a comment on issue #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#issuecomment-59319 Thanks all. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer removed a comment on issue #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer removed a comment on issue #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#issuecomment-58130 Sure I will do. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
maropu commented on a change in pull request #26918: [SPARK-30279][SQL] Support
32 or more grouping attributes for GROUPING_ID
URL: https://github.com/apache/spark/pull/26918#discussion_r359158750
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -563,15 +570,43 @@ class Analyzer(
}
}
+private object EmptyGroupingIDExtractor {
+
+ private def hasEmptyGroupingID(aggExprs: Seq[Expression]): Boolean = {
+aggExprs.exists { p => p.collectFirst { case GroupingID(Nil) => true
}.isDefined }
+ }
+
+ def unapply(p: LogicalPlan): Option[Seq[Expression]] = p match {
+case Aggregate(Seq(Cube(groupByExprs)), aggExprs, _) if
hasEmptyGroupingID(aggExprs) =>
+ Some(constructGroupBy(cubeExprs(groupByExprs)))
+case Aggregate(Seq(Rollup(groupByExprs)), aggExprs, _) if
hasEmptyGroupingID(aggExprs) =>
+ Some(constructGroupBy(rollupExprs(groupByExprs)))
+case x: GroupingSets if hasEmptyGroupingID(x.aggregations) =>
+ Some(constructGroupBy(x.selectedGroupByExprs))
+case Filter(cond, child) if hasEmptyGroupingID(cond :: Nil) =>
+ Some(findGroupingExprs(child))
+case Sort(order, _, child) if hasEmptyGroupingID(order) =>
+ Some(findGroupingExprs(child))
+case _ =>
+ None
+ }
+}
+
// This require transformUp to replace grouping()/grouping_id() in
resolved Filter/Sort
def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
case a if !a.childrenResolved => a // be sure all of the children are
resolved.
+ // If a plan has grouping IDs with an empty group-by, we resolve them
first
+ case p @ EmptyGroupingIDExtractor(groupByExprs) =>
+p.transformExpressions {
+ case GroupingID(Nil) => GroupingID(groupByExprs)
+}
+
Review comment:
I added this new rule for not refactoring but supporting string grouping
IDs. This pr makes a `GroupingID` dataType depend on the number of group-by
expressions (int for a few attributes and string for many ones). So, in case of
the empty group-by case, we need to resolve it (by filling in group-by exprs)
here before analyzing grouping analytics (cube, roll-up, and grouping set).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats
AmplabJenkins commented on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats URL: https://github.com/apache/spark/pull/26016#issuecomment-566868919 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115478/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats
AmplabJenkins commented on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats URL: https://github.com/apache/spark/pull/26016#issuecomment-566868912 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats
AmplabJenkins removed a comment on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats URL: https://github.com/apache/spark/pull/26016#issuecomment-566868919 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115478/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats
AmplabJenkins removed a comment on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats URL: https://github.com/apache/spark/pull/26016#issuecomment-566868912 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures
AmplabJenkins removed a comment on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures URL: https://github.com/apache/spark/pull/26803#issuecomment-566868470 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures
AmplabJenkins removed a comment on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures URL: https://github.com/apache/spark/pull/26803#issuecomment-566868473 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115486/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats
SparkQA commented on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats URL: https://github.com/apache/spark/pull/26016#issuecomment-566868538 **[Test build #115478 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115478/testReport)** for PR 26016 at commit [`cb37e43`](https://github.com/apache/spark/commit/cb37e43b071237bccf15109f89e173aaa70f2729). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures
AmplabJenkins commented on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures URL: https://github.com/apache/spark/pull/26803#issuecomment-566868470 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures
AmplabJenkins commented on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures URL: https://github.com/apache/spark/pull/26803#issuecomment-566868473 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115486/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats
SparkQA removed a comment on issue #26016: [SPARK-24914][SQL] New statistic to improve data size estimate for columnar storage formats URL: https://github.com/apache/spark/pull/26016#issuecomment-566820169 **[Test build #115478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115478/testReport)** for PR 26016 at commit [`cb37e43`](https://github.com/apache/spark/commit/cb37e43b071237bccf15109f89e173aaa70f2729). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] iRakson edited a comment on issue #26863: [SPARK-30234]ADD FILE cannot add directories from sql CLI
iRakson edited a comment on issue #26863: [SPARK-30234]ADD FILE cannot add directories from sql CLI URL: https://github.com/apache/spark/pull/26863#issuecomment-566835810 Can you please review the test suite cc @maropu @cloud-fan @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures
SparkQA removed a comment on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures URL: https://github.com/apache/spark/pull/26803#issuecomment-566856194 **[Test build #115486 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115486/testReport)** for PR 26803 at commit [`8ef2b1b`](https://github.com/apache/spark/commit/8ef2b1bf49e0c8853aed03c2902ba3e07d0c54a6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures
SparkQA commented on issue #26803: [SPARK-30178][ML] RobustScaler support large numFeatures URL: https://github.com/apache/spark/pull/26803#issuecomment-566868260 **[Test build #115486 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115486/testReport)** for PR 26803 at commit [`8ef2b1b`](https://github.com/apache/spark/commit/8ef2b1bf49e0c8853aed03c2902ba3e07d0c54a6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extendedto be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566866941 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115488/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extendedto be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566866938 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
SparkQA removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566866591 **[Test build #115488 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115488/testReport)** for PR 26927 at commit [`9557a10`](https://github.com/apache/spark/commit/9557a10680b745830a8cd9a2a955216a4248988f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins removed a comment on issue #26927: [SPARK-29505][SQL] desc extendedto be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566866911 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20291/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
SparkQA commented on issue #26927: [SPARK-29505][SQL] desc extendedto be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566866929 **[Test build #115488 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115488/testReport)** for PR 26927 at commit [`9557a10`](https://github.com/apache/spark/commit/9557a10680b745830a8cd9a2a955216a4248988f). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566866903 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive
AmplabJenkins commented on issue #26927: [SPARK-29505][SQL] desc extended to be case insensitive URL: https://github.com/apache/spark/pull/26927#issuecomment-566866941 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115488/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
