[GitHub] [spark] dongjoon-hyun commented on issue #24597: [SPARK-27698][SQL] Add new method for getting pushed down filters in Parquet file reader
dongjoon-hyun commented on issue #24597: [SPARK-27698][SQL] Add new method for getting pushed down filters in Parquet file reader URL: https://github.com/apache/spark/pull/24597#issuecomment-493848222 Thank you for pinging me, @gengliangwang . Shall we wait for one day? Currently, after SPARK-27699, `HiveOrcFilterSuite` failure is reported in `Hadoop 3.2` profile. The fix is under testing and will be merged tomorrow. @wangyum and @HyukjinKwon is working on that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #24597: [SPARK-27698][SQL] Add new method for getting pushed down filters in Parquet file reader
dongjoon-hyun edited a comment on issue #24597: [SPARK-27698][SQL] Add new method for getting pushed down filters in Parquet file reader URL: https://github.com/apache/spark/pull/24597#issuecomment-493848222 Thank you for pinging me, @gengliangwang . Shall we wait for one day? Currently, after SPARK-27699, `HiveOrcFilterSuite` failure is reported in `Hadoop 3.2` profile. The fix is under testing and will be merged tomorrow. @wangyum and @HyukjinKwon is actively working on that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive
SparkQA commented on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive URL: https://github.com/apache/spark/pull/23992#issuecomment-493847713 **[Test build #105550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105550/testReport)** for PR 23992 at commit [`7cf2875`](https://github.com/apache/spark/commit/7cf2875aa71094bf2db6f7d24991533a91c6ad29). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive
AmplabJenkins commented on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive URL: https://github.com/apache/spark/pull/23992#issuecomment-493847373 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive
AmplabJenkins commented on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive URL: https://github.com/apache/spark/pull/23992#issuecomment-493847375 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10804/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive
AmplabJenkins removed a comment on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive URL: https://github.com/apache/spark/pull/23992#issuecomment-493847375 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10804/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive
AmplabJenkins removed a comment on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive URL: https://github.com/apache/spark/pull/23992#issuecomment-493847373 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive
dongjoon-hyun commented on issue #23992: [SPARK-27074][SQL] Hive 3.1 metastore support HiveClientImpl.runHive URL: https://github.com/apache/spark/pull/23992#issuecomment-493847124 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
SparkQA commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493846374 **[Test build #105549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105549/testReport)** for PR 24626 at commit [`b380f1d`](https://github.com/apache/spark/commit/b380f1dc39f0e4c7701b1a46603b0b62d1f94f96). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
AmplabJenkins commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493846055 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10803/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
AmplabJenkins removed a comment on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493846049 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
AmplabJenkins removed a comment on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493846055 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10803/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maryannxue commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
maryannxue commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493846041 LGTM, pending Jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
AmplabJenkins commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493846049 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts
dongjoon-hyun commented on a change in pull request #24601: [SPARK-27702][K8S]
Allow using some alternatives for service accounts
URL: https://github.com/apache/spark/pull/24601#discussion_r285434004
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
##
@@ -50,10 +50,21 @@ private[spark] class KubernetesClusterManager extends
ExternalClusterManager wit
require(sc.conf.get(KUBERNETES_DRIVER_POD_NAME).isDefined,
"If the application is deployed using spark-submit in cluster mode,
the driver pod name " +
"must be provided.")
+ val serviceAccountToken =
+Some(new
File(Config.KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH)).filter(_.exists)
+ val serviceAccountCaCrt =
+Some(new
File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)).filter(_.exists)
+ val apiServerUri = if (serviceAccountToken.nonEmpty ||
serviceAccountCaCrt.nonEmpty) {
+KUBERNETES_MASTER_INTERNAL_URL
+ } else {
+val masterUri = KubernetesUtils.parseMasterUrl(masterURL)
+logWarning(s"Service account has been disabled. Using master:
$masterUri")
+masterUri
+ }
(KUBERNETES_AUTH_DRIVER_MOUNTED_CONF_PREFIX,
-KUBERNETES_MASTER_INTERNAL_URL,
-Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH)),
-Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)))
+apiServerUri,
+serviceAccountToken,
+serviceAccountCaCrt)
Review comment:
@Udbhav30 . Could you add a test case this your PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
SparkQA commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493845094 **[Test build #105548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105548/testReport)** for PR 24626 at commit [`b033f55`](https://github.com/apache/spark/commit/b033f55560c7f2fc26b898a21cad444d619fab92). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
AmplabJenkins removed a comment on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493844771 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
AmplabJenkins commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493844775 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10802/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
AmplabJenkins commented on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493844771 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan
AmplabJenkins removed a comment on issue #24626: [SPARK-27747][SQL] add a logical plan link in the physical plan URL: https://github.com/apache/spark/pull/24626#issuecomment-493844775 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10802/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #24596: [SPARK-27694][SQL] Support auto-updating table statistics for data source CTAS command
dongjoon-hyun closed pull request #24596: [SPARK-27694][SQL] Support auto-updating table statistics for data source CTAS command URL: https://github.com/apache/spark/pull/24596 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
SparkQA commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493843871 **[Test build #105547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105547/testReport)** for PR 24644 at commit [`92abff8`](https://github.com/apache/spark/commit/92abff88ea626812c2b6e161122bc4d0766be4a7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493843572 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493843576 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10801/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493843576 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10801/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493843572 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24233: [SPARK-26356][SQL] remove SaveMode from data source v2
cloud-fan commented on a change in pull request #24233: [SPARK-26356][SQL]
remove SaveMode from data source v2
URL: https://github.com/apache/spark/pull/24233#discussion_r285431892
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2Suite.scala
##
@@ -219,14 +219,14 @@ class DataSourceV2Suite extends QueryTest with
SharedSQLContext {
assert(spark.read.format(cls.getName).option("path",
path).load().collect().isEmpty)
spark.range(10).select('id as 'i, -'id as 'j).write.format(cls.getName)
- .option("path", path).save()
+ .option("path", path).mode("append").save()
Review comment:
Before my PR, we write the files twice: once without the save mode, once
with append mode.
Now I switch order, to make sure that the second write doesn't specify save
mode, and prove the default mode is append.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
HyukjinKwon commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493843148 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24594: [SPARK-27693][SQL] Add default catalog property
cloud-fan commented on a change in pull request #24594: [SPARK-27693][SQL] Add
default catalog property
URL: https://github.com/apache/spark/pull/24594#discussion_r285431382
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1767,6 +1767,11 @@ object SQLConf {
"with String")
.booleanConf
.createWithDefault(false)
+
+ val DEFAULT_V2_CATALOG = buildConf("spark.sql.default.catalog")
+ .doc("Name of the default v2 catalog, used when an catalog is not
identified in queries")
Review comment:
as we discussed in the DS v2 meeting, we should clearly point out which
places this default catalog is used. View/Function resolution definitely
doesn't use this default catalog for now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24594: [SPARK-27693][SQL] Add default catalog property
cloud-fan commented on a change in pull request #24594: [SPARK-27693][SQL] Add
default catalog property
URL: https://github.com/apache/spark/pull/24594#discussion_r285431212
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1767,6 +1767,11 @@ object SQLConf {
"with String")
.booleanConf
.createWithDefault(false)
+
+ val DEFAULT_V2_CATALOG = buildConf("spark.sql.default.catalog")
Review comment:
nit: `spark.sql.catalog.default` is more consistent with other SQL config
names: `spark.sql.componentName.featureName`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
viirya commented on a change in pull request #24637: [SPARK-27707][SQL] Prune
unnecessary nested fields from Generate to address performance issue in explode
URL: https://github.com/apache/spark/pull/24637#discussion_r285430439
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/MiscBenchmark.scala
##
@@ -130,6 +130,18 @@ object MiscBenchmark extends SqlBasedBenchmark {
df.selectExpr("*", "explode(arr) as arr_col")
.select("col", "arr_col.*").count
}
+
+ codegenBenchmark("generate big nested struct array", M) {
+import spark.implicits._
+val df = spark.sparkContext.parallelize(Seq(("1",
+ Array.fill(M)({
+val i = math.random
+(i.toString, (i + 1).toString, (i + 2).toString, (i + 3).toString)
+ }.toDF("col", "arr")
+ .selectExpr("col", "struct(col, arr) as st")
+ .selectExpr("col", "st.col as col1", "explode(st.arr) as arr_col")
+df.collect()
+ }
Review comment:
Yea, I posted the benchmark result for the added one. Because I think the
all benchmark results should be run on r3.xlarge machine, I plan to post it
later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
viirya commented on a change in pull request #24637: [SPARK-27707][SQL] Prune
unnecessary nested fields from Generate to address performance issue in explode
URL: https://github.com/apache/spark/pull/24637#discussion_r285430503
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/MiscBenchmark.scala
##
@@ -130,6 +130,18 @@ object MiscBenchmark extends SqlBasedBenchmark {
df.selectExpr("*", "explode(arr) as arr_col")
.select("col", "arr_col.*").count
}
+
+ codegenBenchmark("generate big nested struct array", M) {
+import spark.implicits._
+val df = spark.sparkContext.parallelize(Seq(("1",
+ Array.fill(M)({
+val i = math.random
+(i.toString, (i + 1).toString, (i + 2).toString, (i + 3).toString)
+ }.toDF("col", "arr")
+ .selectExpr("col", "struct(col, arr) as st")
+ .selectExpr("col", "st.col as col1", "explode(st.arr) as arr_col")
+df.collect()
+ }
Review comment:
Will update `MiscBenchmark-results` soon.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493840306 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105545/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
SparkQA removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493827618 **[Test build #105545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105545/testReport)** for PR 24644 at commit [`92abff8`](https://github.com/apache/spark/commit/92abff88ea626812c2b6e161122bc4d0766be4a7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493840303 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
dongjoon-hyun commented on a change in pull request #24637: [SPARK-27707][SQL]
Prune unnecessary nested fields from Generate to address performance issue in
explode
URL: https://github.com/apache/spark/pull/24637#discussion_r285429699
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/MiscBenchmark.scala
##
@@ -130,6 +130,18 @@ object MiscBenchmark extends SqlBasedBenchmark {
df.selectExpr("*", "explode(arr) as arr_col")
.select("col", "arr_col.*").count
}
+
+ codegenBenchmark("generate big nested struct array", M) {
+import spark.implicits._
+val df = spark.sparkContext.parallelize(Seq(("1",
+ Array.fill(M)({
+val i = math.random
+(i.toString, (i + 1).toString, (i + 2).toString, (i + 3).toString)
+ }.toDF("col", "arr")
+ .selectExpr("col", "struct(col, arr) as st")
+ .selectExpr("col", "st.col as col1", "explode(st.arr) as arr_col")
+df.collect()
+ }
Review comment:
This one, `sql/core/benchmarks/MiscBenchmark-results.txt`~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
dongjoon-hyun commented on a change in pull request #24637: [SPARK-27707][SQL]
Prune unnecessary nested fields from Generate to address performance issue in
explode
URL: https://github.com/apache/spark/pull/24637#discussion_r285429711
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -32,9 +32,7 @@ object NestedColumnAliasing {
def unapply(plan: LogicalPlan)
: Option[(Map[GetStructField, Alias], Map[ExprId, Seq[Alias]])] = plan
match {
-case Project(projectList, child)
-if SQLConf.get.nestedSchemaPruningEnabled &&
canProjectPushThrough(child) =>
- getAliasSubMap(projectList)
+case Project(projectList, child) => getAliasSubMap(projectList)
Review comment:
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493840306 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105545/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493840303 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
dongjoon-hyun commented on a change in pull request #24637: [SPARK-27707][SQL]
Prune unnecessary nested fields from Generate to address performance issue in
explode
URL: https://github.com/apache/spark/pull/24637#discussion_r285429576
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/MiscBenchmark.scala
##
@@ -130,6 +130,18 @@ object MiscBenchmark extends SqlBasedBenchmark {
df.selectExpr("*", "explode(arr) as arr_col")
.select("col", "arr_col.*").count
}
+
+ codegenBenchmark("generate big nested struct array", M) {
+import spark.implicits._
+val df = spark.sparkContext.parallelize(Seq(("1",
+ Array.fill(M)({
+val i = math.random
+(i.toString, (i + 1).toString, (i + 2).toString, (i + 3).toString)
+ }.toDF("col", "arr")
+ .selectExpr("col", "struct(col, arr) as st")
+ .selectExpr("col", "st.col as col1", "explode(st.arr) as arr_col")
+df.collect()
+ }
Review comment:
Please add benchmark result in this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
SparkQA commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493840155 **[Test build #105545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105545/testReport)** for PR 24644 at commit [`92abff8`](https://github.com/apache/spark/commit/92abff88ea626812c2b6e161122bc4d0766be4a7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
viirya commented on a change in pull request #24637: [SPARK-27707][SQL] Prune
unnecessary nested fields from Generate to address performance issue in explode
URL: https://github.com/apache/spark/pull/24637#discussion_r285429526
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -32,9 +32,7 @@ object NestedColumnAliasing {
def unapply(plan: LogicalPlan)
: Option[(Map[GetStructField, Alias], Map[ExprId, Seq[Alias]])] = plan
match {
-case Project(projectList, child)
-if SQLConf.get.nestedSchemaPruningEnabled &&
canProjectPushThrough(child) =>
- getAliasSubMap(projectList)
+case Project(projectList, child) => getAliasSubMap(projectList)
Review comment:
I see. If so, I need to make a little change to prevent it. Will change it
later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
dongjoon-hyun commented on a change in pull request #24637: [SPARK-27707][SQL]
Prune unnecessary nested fields from Generate to address performance issue in
explode
URL: https://github.com/apache/spark/pull/24637#discussion_r285429248
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -32,9 +32,7 @@ object NestedColumnAliasing {
def unapply(plan: LogicalPlan)
: Option[(Map[GetStructField, Alias], Map[ExprId, Seq[Alias]])] = plan
match {
-case Project(projectList, child)
-if SQLConf.get.nestedSchemaPruningEnabled &&
canProjectPushThrough(child) =>
- getAliasSubMap(projectList)
+case Project(projectList, child) => getAliasSubMap(projectList)
Review comment:
@viirya . Sorry, but this is a regression on all the existing code. We
should avoid `getAliasSubMap` invocation.
https://github.com/apache/spark/pull/24637/files#diff-a636a87d8843eeccca90140be91d4fafR635
doesn't prevent `getAliasSubMap` invocation inside `unapply`, does it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jzhuge commented on issue #24594: [SPARK-27693][SQL] Add default catalog property
jzhuge commented on issue #24594: [SPARK-27693][SQL] Add default catalog property URL: https://github.com/apache/spark/pull/24594#issuecomment-493839634 Thanks @dongjoon-hyun ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
dongjoon-hyun commented on a change in pull request #24637: [SPARK-27707][SQL]
Prune unnecessary nested fields from Generate to address performance issue in
explode
URL: https://github.com/apache/spark/pull/24637#discussion_r285429248
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -32,9 +32,7 @@ object NestedColumnAliasing {
def unapply(plan: LogicalPlan)
: Option[(Map[GetStructField, Alias], Map[ExprId, Seq[Alias]])] = plan
match {
-case Project(projectList, child)
-if SQLConf.get.nestedSchemaPruningEnabled &&
canProjectPushThrough(child) =>
- getAliasSubMap(projectList)
+case Project(projectList, child) => getAliasSubMap(projectList)
Review comment:
@viirya . Sorry, but this is a regression on all the existing code. We
should avoid `getAliasSubMap` invocation.
https://github.com/apache/spark/pull/24637/files#diff-a636a87d8843eeccca90140be91d4fafR635
doesn't prevent `getAliasSubMap` invocation inside `unapply`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-493838695 cc @dbtsai This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
HyukjinKwon commented on a change in pull request #24644:
[SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test
for hadoop3.2 for now
URL: https://github.com/apache/spark/pull/24644#discussion_r285428395
##
File path: dev/sparktestsupport/modules.py
##
@@ -568,6 +558,15 @@ def __hash__(self):
]
)
+# TODO: Skip hive-thriftserver module for hadoop-3.2. remove this once
hadoop-3.2 support it
+if os.environ.get("AMPLAB_JENKINS"):
+hadoop_version = os.environ.get("AMPLAB_JENKINS_BUILD_PROFILE",
"hadoop2.7")
+else:
+hadoop_version = os.environ.get("HADOOP_PROFILE", "hadoop2.7")
+if hadoop_version == "hadoop3.2":
+print("[info] Skip unsupported module:", "hive-thriftserver")
+all_modules = [m for m in all_modules if m.name != "hive-thriftserver"]
Review comment:
yea I just moved.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24631: [MINOR][CORE] Avoid hardcoded configs
dongjoon-hyun commented on issue #24631: [MINOR][CORE] Avoid hardcoded configs URL: https://github.com/apache/spark/pull/24631#issuecomment-493838177 @wenxuanguan . Thank you for improving the PR. Now, this PR seems need a JIRA issue because it touches many files. Could you file a Apache Spark JIRA issue? Please enumerate the configs in the JIRA which you replaced. Also, please use that JIRA id instead of `[MINOR]`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #24497: [SPARK-27630][CORE]Stage retry causes totalRunningTasks calculation to be negative
cxzl25 commented on a change in pull request #24497: [SPARK-27630][CORE]Stage
retry causes totalRunningTasks calculation to be negative
URL: https://github.com/apache/spark/pull/24497#discussion_r285428024
##
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
##
@@ -646,10 +646,9 @@ private[spark] class ExecutorAllocationManager(
private[spark] class ExecutorAllocationListener extends SparkListener {
private val stageIdToNumTasks = new mutable.HashMap[Int, Int]
-// Number of running tasks per stage including speculative tasks.
-// Should be 0 when no stages are active.
-private val stageIdToNumRunningTask = new mutable.HashMap[Int, Int]
private val stageIdToTaskIndices = new mutable.HashMap[Int,
mutable.HashSet[Int]]
+private val liveTaskIds = new mutable.HashSet[Long]
Review comment:
The taskid is actually not repeated.
Only the same taskid appears in some test cases, and the task has different
types.
```scala
createTaskInfo(0, 0, "executor-1")
createTaskInfo(0, 0, "executor-2", true)
```
https://github.com/apache/spark/blob/cd4a284030e5142bdb405ff5b71735ac8cee2dde/core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala#L260-L267
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493837335 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105544/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493837330 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24335: [SPARK-27425][SQL] Add count_if functions
dongjoon-hyun commented on issue #24335: [SPARK-27425][SQL] Add count_if functions URL: https://github.com/apache/spark/pull/24335#issuecomment-493837370 Hi, @cryeo . Did you ask the questions to the community as @HyukjinKwon recommended? I'm just wondering if the decision was made. If we are not going to proceed with this, we had better close this PR and JIRA issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493837335 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105544/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493837330 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
SparkQA commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493837230 **[Test build #105544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105544/testReport)** for PR 24644 at commit [`92abff8`](https://github.com/apache/spark/commit/92abff88ea626812c2b6e161122bc4d0766be4a7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
SparkQA removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493826550 **[Test build #105544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105544/testReport)** for PR 24644 at commit [`92abff8`](https://github.com/apache/spark/commit/92abff88ea626812c2b6e161122bc4d0766be4a7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #24594: [SPARK-27693][SQL] Add default catalog property
dongjoon-hyun closed pull request #24594: [SPARK-27693][SQL] Add default catalog property URL: https://github.com/apache/spark/pull/24594 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24594: [SPARK-27693][SQL] Add default catalog property
SparkQA commented on issue #24594: [SPARK-27693][SQL] Add default catalog property URL: https://github.com/apache/spark/pull/24594#issuecomment-493835187 **[Test build #105546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105546/testReport)** for PR 24594 at commit [`03cf48b`](https://github.com/apache/spark/commit/03cf48ba8c9e13fb55cfe409b477b3a70a5a1760). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24594: [SPARK-27693][SQL] Add default catalog property
AmplabJenkins removed a comment on issue #24594: [SPARK-27693][SQL] Add default catalog property URL: https://github.com/apache/spark/pull/24594#issuecomment-493834930 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24594: [SPARK-27693][SQL] Add default catalog property
AmplabJenkins removed a comment on issue #24594: [SPARK-27693][SQL] Add default catalog property URL: https://github.com/apache/spark/pull/24594#issuecomment-493834935 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10800/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24594: [SPARK-27693][SQL] Add default catalog property
AmplabJenkins commented on issue #24594: [SPARK-27693][SQL] Add default catalog property URL: https://github.com/apache/spark/pull/24594#issuecomment-493834935 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10800/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24594: [SPARK-27693][SQL] Add default catalog property
AmplabJenkins commented on issue #24594: [SPARK-27693][SQL] Add default catalog property URL: https://github.com/apache/spark/pull/24594#issuecomment-493834930 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24594: [SPARK-27693][SQL] Add default catalog property
dongjoon-hyun commented on issue #24594: [SPARK-27693][SQL] Add default catalog property URL: https://github.com/apache/spark/pull/24594#issuecomment-493834677 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24594: [SPARK-27693][SQL] Add default catalog property
dongjoon-hyun commented on a change in pull request #24594: [SPARK-27693][SQL]
Add default catalog property
URL: https://github.com/apache/spark/pull/24594#discussion_r285424373
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1767,6 +1767,11 @@ object SQLConf {
"with String")
.booleanConf
.createWithDefault(false)
+
+ val DEFAULT_V2_CATALOG = buildConf("spark.sql.default.catalog")
+ .doc("Name of the default v2 catalog, used when an catalog is not
identified in queries")
Review comment:
nit. `an catalog` -> `a catalog`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #24641: [SPARK-27610][FOLLOW-UP][YARN] Remove duplicate declaration of plugin maven-antrun-plugin
dongjoon-hyun closed pull request #24641: [SPARK-27610][FOLLOW-UP][YARN] Remove duplicate declaration of plugin maven-antrun-plugin URL: https://github.com/apache/spark/pull/24641 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
viirya commented on a change in pull request #24644:
[SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test
for hadoop3.2 for now
URL: https://github.com/apache/spark/pull/24644#discussion_r285423349
##
File path: dev/sparktestsupport/modules.py
##
@@ -568,6 +558,15 @@ def __hash__(self):
]
)
+# TODO: Skip hive-thriftserver module for hadoop-3.2. remove this once
hadoop-3.2 support it
+if os.environ.get("AMPLAB_JENKINS"):
+hadoop_version = os.environ.get("AMPLAB_JENKINS_BUILD_PROFILE",
"hadoop2.7")
+else:
+hadoop_version = os.environ.get("HADOOP_PROFILE", "hadoop2.7")
+if hadoop_version == "hadoop3.2":
+print("[info] Skip unsupported module:", "hive-thriftserver")
+all_modules = [m for m in all_modules if m.name != "hive-thriftserver"]
Review comment:
This just does the same thing as before, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
HyukjinKwon commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493830628 okie profiles look good. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285422373
##
File path: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala
##
@@ -985,4 +976,99 @@ class CachedTableSuite extends QueryTest with
SQLTestUtils with SharedSQLContext
val queryStats3 = query().queryExecution.optimizedPlan.stats.attributeStats
assert(queryStats3.map(_._1.name).toSet === Set("c0", "v1", "v2"))
}
+
+ test("SPARK-27248 refreshTable should recreate cache with same cache name
and storage level") {
+
+// This section tests when a table is cached with its qualified name but
it is refreshed with
+// its unqualified name.
+withTempDatabase { db =>
+ withTempPath { path =>
+withTable(s"$db.cachedTable") {
+ // Create table 'cachedTable' in temp db for testing purpose.
+ spark.catalog.createTable(
+s"$db.cachedTable",
+"PARQUET",
+StructType(Array(StructField("key", StringType))),
+Map("LOCATION" -> path.toURI.toString))
+
+ withCache(s"$db.cachedTable") {
+
Review comment:
ditto.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285422109
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala
##
@@ -49,16 +49,7 @@ class CachedTableSuite extends QueryTest with SQLTestUtils
with TestHiveSingleto
maybeBlock.nonEmpty
}
- // Blocking uncache table for tests
- private def uncacheTable(tableName: String): Unit = {
-val tableIdent =
spark.sessionState.sqlParser.parseTableIdentifier(tableName)
-val cascade = !spark.sessionState.catalog.isTemporaryTable(tableIdent)
-spark.sharedState.cacheManager.uncacheQuery(
- spark,
- spark.table(tableName).logicalPlan,
- cascade = cascade,
- blocking = true)
- }
+
Review comment:
Remove the blank line 52. Removing the existing function is enough and we
don't need to add another blank line here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24631: [MINOR][CORE] Avoid hardcoded configs
AmplabJenkins removed a comment on issue #24631: [MINOR][CORE] Avoid hardcoded configs URL: https://github.com/apache/spark/pull/24631#issuecomment-493830155 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105540/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285422259
##
File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
##
@@ -299,6 +299,27 @@ private[sql] trait SQLTestUtilsBase
}
}
+ /**
+ * Drops cache `cacheName` after calling `f`.
+ */
+ protected def withCache(cacheNames: String*)(f: => Unit): Unit = {
+try f finally {
+ cacheNames.foreach(uncacheTable)
+}
+ }
Review comment:
The above is required when we use `withCache("a", "b", "c")`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24631: [MINOR][CORE] Avoid hardcoded configs
AmplabJenkins removed a comment on issue #24631: [MINOR][CORE] Avoid hardcoded configs URL: https://github.com/apache/spark/pull/24631#issuecomment-493830153 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285422353
##
File path: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala
##
@@ -985,4 +976,99 @@ class CachedTableSuite extends QueryTest with
SQLTestUtils with SharedSQLContext
val queryStats3 = query().queryExecution.optimizedPlan.stats.attributeStats
assert(queryStats3.map(_._1.name).toSet === Set("c0", "v1", "v2"))
}
+
+ test("SPARK-27248 refreshTable should recreate cache with same cache name
and storage level") {
+
Review comment:
Remove a blank line here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24631: [MINOR][CORE] Avoid hardcoded configs
AmplabJenkins commented on issue #24631: [MINOR][CORE] Avoid hardcoded configs URL: https://github.com/apache/spark/pull/24631#issuecomment-493830153 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24631: [MINOR][CORE] Avoid hardcoded configs
AmplabJenkins commented on issue #24631: [MINOR][CORE] Avoid hardcoded configs URL: https://github.com/apache/spark/pull/24631#issuecomment-493830155 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105540/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285422138
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala
##
@@ -361,4 +352,84 @@ class CachedTableSuite extends QueryTest with
SQLTestUtils with TestHiveSingleto
assert(spark.sharedState.cacheManager.lookupCachedData(samePlan).isDefined)
}
}
+
+ test("SPARK-27248 refreshTable should recreate cache with same cache name
and storage level") {
+
Review comment:
Please remove blank line here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285422162
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala
##
@@ -361,4 +352,84 @@ class CachedTableSuite extends QueryTest with
SQLTestUtils with TestHiveSingleto
assert(spark.sharedState.cacheManager.lookupCachedData(samePlan).isDefined)
}
}
+
+ test("SPARK-27248 refreshTable should recreate cache with same cache name
and storage level") {
+
+// This section tests when a table is cached with its qualified name but
its is refreshed with
+// its unqualified name.
+withTempDatabase { db =>
+ withTable(s"$db.cachedTable") {
+withCache(s"$db.cachedTable") {
+
+ // Create table 'cachedTable' in default db for testing purpose.
+ sql(s"CREATE TABLE $db.cachedTable AS SELECT 1 AS key")
+
+ // Cache the table 'cachedTable' in temp db with qualified table
name,
+ // and then check whether the table is cached with expected name
+ sql(s"CACHE TABLE $db.cachedTable")
+ assertCached(sql(s"select * from $db.cachedTable"),
s"`$db`.`cachedTable`")
+ assert(spark.catalog.isCached(s"$db.cachedTable"),
+s"Table '$db.cachedTable' should be cached.")
+
+ // Refresh the table 'cachedTable' in temp db with qualified table
name, and then check
+ // whether the table is still cached with the same name and storage
level.
+ sql(s"REFRESH TABLE $db.cachedTable")
+ assertCached(sql(s"select * from $db.cachedTable"),
s"`$db`.`cachedTable`")
+ assert(spark.catalog.isCached(s"$db.cachedTable"),
+s"Table '$db.cachedTable' should be cached after refreshing with
its qualified name.")
+
+ // Change the active database to the temp db and refresh the table
with unqualified
+ // table name, and then check whether the table is still cached with
the same name and
+ // storage level.
+ // Without bug fix 'SPARK-27248', the recreated cache name will be
changed to
+ // 'cachedTable', instead of '$db.cachedTable'
+ activateDatabase(db) {
+sql("REFRESH TABLE cachedTable")
+assertCached(sql("select * from cachedTable"),
s"`$db`.`cachedTable`")
+assert(spark.catalog.isCached("cachedTable"),
+ s"Table '$db.cachedTable' should be cached after refreshing with
its " +
+"unqualified name.")
+ }
+}
+ }
+}
+
+
+// This section tests when a table is cached with its unqualified name but
it is refreshed
+// with its qualified name.
+withTempDatabase { db =>
+ withTable("cachedTable") {
+withCache("cachedTable") {
+
Review comment:
ditto.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285422146
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala
##
@@ -361,4 +352,84 @@ class CachedTableSuite extends QueryTest with
SQLTestUtils with TestHiveSingleto
assert(spark.sharedState.cacheManager.lookupCachedData(samePlan).isDefined)
}
}
+
+ test("SPARK-27248 refreshTable should recreate cache with same cache name
and storage level") {
+
+// This section tests when a table is cached with its qualified name but
its is refreshed with
+// its unqualified name.
+withTempDatabase { db =>
+ withTable(s"$db.cachedTable") {
+withCache(s"$db.cachedTable") {
+
Review comment:
ditto.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285422109
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala
##
@@ -49,16 +49,7 @@ class CachedTableSuite extends QueryTest with SQLTestUtils
with TestHiveSingleto
maybeBlock.nonEmpty
}
- // Blocking uncache table for tests
- private def uncacheTable(tableName: String): Unit = {
-val tableIdent =
spark.sessionState.sqlParser.parseTableIdentifier(tableName)
-val cascade = !spark.sessionState.catalog.isTemporaryTable(tableIdent)
-spark.sharedState.cacheManager.uncacheQuery(
- spark,
- spark.table(tableName).logicalPlan,
- cascade = cascade,
- blocking = true)
- }
+
Review comment:
Remove the blank line 52. Removing the existing one is enough and we don't
need to add another blank line here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285422007
##
File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
##
@@ -299,6 +299,27 @@ private[sql] trait SQLTestUtilsBase
}
}
+ /**
+ * Drops cache `cacheName` after calling `f`.
+ */
+ protected def withCache(cacheNames: String*)(f: => Unit): Unit = {
+try f finally {
+ cacheNames.foreach(uncacheTable)
+}
+ }
+
+
Review comment:
Remove line 311. One blank line will be enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24631: [MINOR][CORE] Avoid hardcoded configs
SparkQA removed a comment on issue #24631: [MINOR][CORE] Avoid hardcoded configs URL: https://github.com/apache/spark/pull/24631#issuecomment-493812040 **[Test build #105540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105540/testReport)** for PR 24631 at commit [`ecef09f`](https://github.com/apache/spark/commit/ecef09fe8f901b4e1595faeca715894cbf62126c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24631: [MINOR][CORE] Avoid hardcoded configs
SparkQA commented on issue #24631: [MINOR][CORE] Avoid hardcoded configs URL: https://github.com/apache/spark/pull/24631#issuecomment-493829903 **[Test build #105540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105540/testReport)** for PR 24631 at commit [`ecef09f`](https://github.com/apache/spark/commit/ecef09fe8f901b4e1595faeca715894cbf62126c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285421984
##
File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
##
@@ -299,6 +299,27 @@ private[sql] trait SQLTestUtilsBase
}
}
+ /**
+ * Drops cache `cacheName` after calling `f`.
+ */
+ protected def withCache(cacheNames: String*)(f: => Unit): Unit = {
+try f finally {
+ cacheNames.foreach(uncacheTable)
+}
+ }
Review comment:
Could you update like the following? Like the other `withXXX` functions,
`withCache` should clean up multiple cache names in the `finally` clause.
```scala
protected def withCache(cacheNames: String*)(f: => Unit): Unit = {
try f finally {
- cacheNames.foreach(uncacheTable)
+ cacheNames.foreach { cacheName =>
+try uncacheTable(cacheName) catch {
+ case _: AnalysisException =>
+}
+ }
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
SparkQA commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493827618 **[Test build #105545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105545/testReport)** for PR 24644 at commit [`92abff8`](https://github.com/apache/spark/commit/92abff88ea626812c2b6e161122bc4d0766be4a7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493827310 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins removed a comment on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493827311 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10799/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493827311 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10799/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
AmplabJenkins commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493827310 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
HyukjinKwon commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493826857 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
HyukjinKwon commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493826505 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
SparkQA commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493826550 **[Test build #105544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105544/testReport)** for PR 24644 at commit [`92abff8`](https://github.com/apache/spark/commit/92abff88ea626812c2b6e161122bc4d0766be4a7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #24644: [SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
HyukjinKwon commented on a change in pull request #24644:
[SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2] Exclude 'hive-thriftserver' in
modules to test for hadoop3.2 for now
URL: https://github.com/apache/spark/pull/24644#discussion_r285417372
##
File path: dev/run-tests.py
##
@@ -122,9 +123,15 @@ def determine_modules_to_test(changed_modules):
# If we need to run all of the tests, then we should short-circuit and
return 'root'
if modules.root in modules_to_test:
return [modules.root]
-return toposort_flatten(
+changed_modules = toposort_flatten(
{m: set(m.dependencies).intersection(modules_to_test) for m in
modules_to_test}, sort=True)
+# TODO: Skip hive-thriftserver module for hadoop-3.2. remove this once
hadoop-3.2 support it
Review comment:
To be clear, this fix itself is a temp fix too in order to make the affected
scope isolated and minimised. To fix it properly, we should maybe be able to
fix it within `Module` at `modules.py` (and remove this entire module
conditionally)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver
AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-493826145 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105538/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver
AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-493826139 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver
AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-493826139 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver
SparkQA removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-493802167 **[Test build #105538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105538/testReport)** for PR 24628 at commit [`a7bc4e8`](https://github.com/apache/spark/commit/a7bc4e891ae0862d19403164b223ec3b50a38ff3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver
AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-493826145 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105538/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver
SparkQA commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-493826026 **[Test build #105538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105538/testReport)** for PR 24628 at commit [`a7bc4e8`](https://github.com/apache/spark/commit/a7bc4e891ae0862d19403164b223ec3b50a38ff3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2][test-maven] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
HyukjinKwon commented on issue #24644: [SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2][test-maven] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now URL: https://github.com/apache/spark/pull/24644#issuecomment-493825269 profiles look fine so far. I will switch to SBT's to make sure it doesn't affect our regular builds. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #24644: [SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2][test-maven] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for
HyukjinKwon commented on a change in pull request #24644:
[SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2][test-maven] Exclude
'hive-thriftserver' in modules to test for hadoop3.2 for now
URL: https://github.com/apache/spark/pull/24644#discussion_r285417372
##
File path: dev/run-tests.py
##
@@ -122,9 +123,15 @@ def determine_modules_to_test(changed_modules):
# If we need to run all of the tests, then we should short-circuit and
return 'root'
if modules.root in modules_to_test:
return [modules.root]
-return toposort_flatten(
+changed_modules = toposort_flatten(
{m: set(m.dependencies).intersection(modules_to_test) for m in
modules_to_test}, sort=True)
+# TODO: Skip hive-thriftserver module for hadoop-3.2. remove this once
hadoop-3.2 support it
Review comment:
To be clear, this fix itself is a temp fix too in order to make the affected
scope isolated and minimised. To fix it properly, we should maybe be able to
fix it within `Module` at `modules.py`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
