[GitHub] [spark] huaxingao commented on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output
huaxingao commented on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output URL: https://github.com/apache/spark/pull/26247#issuecomment-546213082 @viirya @zhengruifeng Thanks for the review. I made the changes. It's probably a overkill to have a single column test for each of the multi-column test, but i will keep these if no objections. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #25926: [SPARK-9612][ML] Add instance weight support for GBTs
zhengruifeng commented on issue #25926: [SPARK-9612][ML] Add instance weight support for GBTs URL: https://github.com/apache/spark/pull/25926#issuecomment-546211594 Merged to master, thanks @imatiach-msft @srowen for reviewing! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output
AmplabJenkins commented on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output URL: https://github.com/apache/spark/pull/26247#issuecomment-546211402 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17612/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output
AmplabJenkins removed a comment on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output URL: https://github.com/apache/spark/pull/26247#issuecomment-546211402 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17612/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output
AmplabJenkins removed a comment on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output URL: https://github.com/apache/spark/pull/26247#issuecomment-546211397 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output
AmplabJenkins commented on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output URL: https://github.com/apache/spark/pull/26247#issuecomment-546211397 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output
SparkQA commented on issue #26247: [SPARK-29566][ML] Imputer should support single-column input/output URL: https://github.com/apache/spark/pull/26247#issuecomment-546211029 **[Test build #112652 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112652/testReport)** for PR 26247 at commit [`ca866f0`](https://github.com/apache/spark/commit/ca866f095615dfa263da976da58845fbdc6bba60). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #25926: [SPARK-9612][ML] Add instance weight support for GBTs
zhengruifeng closed pull request #25926: [SPARK-9612][ML] Add instance weight support for GBTs URL: https://github.com/apache/spark/pull/25926 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546209732 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112651/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546209728 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546209728 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546209732 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112651/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
SparkQA commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546209647 **[Test build #112651 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112651/testReport)** for PR 26246 at commit [`17273a0`](https://github.com/apache/spark/commit/17273a033181d175a4e22536b49ba09115b5dd51). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
SparkQA removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546203321 **[Test build #112651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112651/testReport)** for PR 26246 at commit [`17273a0`](https://github.com/apache/spark/commit/17273a033181d175a4e22536b49ba09115b5dd51). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1
HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546204959 If ^ is easy, and can be done in this PR, without a separate branch, 1. we can check if the tests pass 2. we release preview and 3. revert this commit from the master. Sorry if this was discussed already somewhere but we're not going to vote on it and make multiple RCs right (since it's just a preview)? It we're going to vote and make more RCs for preview, it might be best to have a separate branch. Seems like merging script issue can be easily fixed in this case. The latest branch is decided by sorting order so we can name it like `branch-preview-3.0`. JIRA fixed version we will have to fix this line https://github.com/apache/spark/blob/2ec3265ae76fc1e136e44c240c476ce572b679df/dev/merge_spark_pr.py#L244 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26137: [SPARK-29490][SQL] Reset 'WritableColumnVector' in 'RowToColumnarExec'
dongjoon-hyun commented on issue #26137: [SPARK-29490][SQL] Reset 'WritableColumnVector' in 'RowToColumnarExec' URL: https://github.com/apache/spark/pull/26137#issuecomment-546207891 cc @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] iRakson commented on issue #26226: [SPARK-29452][WEBUI] Improve Storage tab tooltip
iRakson commented on issue #26226: [SPARK-29452][WEBUI] Improve Storage tab tooltip URL: https://github.com/apache/spark/pull/26226#issuecomment-546207799 @dongjoon-hyun yes working on this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546206719 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546206723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112648/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546206723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112648/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
SparkQA commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546206631 **[Test build #112648 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112648/testReport)** for PR 26246 at commit [`17273a0`](https://github.com/apache/spark/commit/17273a033181d175a4e22536b49ba09115b5dd51). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546206719 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
SparkQA removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546202019 **[Test build #112648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112648/testReport)** for PR 26246 at commit [`17273a0`](https://github.com/apache/spark/commit/17273a033181d175a4e22536b49ba09115b5dd51). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1
HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546204959 If ^ is easy, and can be done in this PR, without a separate branch, 1. we can test if the tests pass 2. we release preview and 3. revert this commit from the master. Sorry if this was discussed already somewhere but we're not going to vote on it and make multiple RCs right (since it's just a preview)? It we're going to vote and make more RCs for preview, it might be best to have a separate branch. Seems like merging script issue can be easily fixed in this case. The latest branch is decided by sorting order so we can name it like `branch-preview-3.0`. JIRA fixed version we will have to fix this line https://github.com/apache/spark/blob/2ec3265ae76fc1e136e44c240c476ce572b679df/dev/merge_spark_pr.py#L244 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1
HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546204959 If ^ is easy, and can be done in this PR, without a separate branch, 1. we can test if the tests pass 2. we release preview and 3. revert this commit from the master. Sorry if this was discussed already somewhere but we're not going to vote on it and make multiple RCs right? It we're going to vote and make more RCs for preview, it might be best to have a separate branch. Seems like merging script issue can be easily fixed in this case. The latest branch is decided by sorting order so we can name it like `branch-preview-3.0`. JIRA fixed version we will have to fix this line https://github.com/apache/spark/blob/2ec3265ae76fc1e136e44c240c476ce572b679df/dev/merge_spark_pr.py#L244 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1
HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546204959 If ^ is easy, and can be done in this PR, without a separate branch, 1. we can test if the tests pass 2. we release preview and 3. revert this commit from the master. Sorry if this was discussed already somewhere but we're not going to vote on it right? It we're going to vote and make more RC for preview, it might be best to have a separate branch. Seems like merging script issue can be easily fixed in this case. The latest branch is decided by sorting order so we can name it like `branch-preview-3.0`. JIRA fixed version we will have to fix this line https://github.com/apache/spark/blob/2ec3265ae76fc1e136e44c240c476ce572b679df/dev/merge_spark_pr.py#L244 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24566: [SPARK-27667][SQL] support display of current Database in the spark-sql CLI
AmplabJenkins commented on issue #24566: [SPARK-27667][SQL] support display of current Database in the spark-sql CLI URL: https://github.com/apache/spark/pull/24566#issuecomment-546206135 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26137: [SPARK-29490][SQL] Reset 'WritableColumnVector' in 'RowToColumnarExec'
dongjoon-hyun commented on a change in pull request #26137: [SPARK-29490][SQL]
Reset 'WritableColumnVector' in 'RowToColumnarExec'
URL: https://github.com/apache/spark/pull/26137#discussion_r338890512
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
##
@@ -171,6 +172,30 @@ class SparkSessionExtensionSuite extends SparkFunSuite {
}
}
+ test("reset column vectors") {
+val session = SparkSession.builder()
+ .master("local[1]")
+ .config(COLUMN_BATCH_SIZE.key, 2)
+ .withExtensions { extensions =>
+extensions.injectColumnar(session =>
+ MyColumarRule(PreRuleReplaceAddWithBrokenVersion(), MyPostRule())) }
+ .getOrCreate()
+
+try {
+ assert(session.sessionState.columnarRules.contains(
+MyColumarRule(PreRuleReplaceAddWithBrokenVersion(), MyPostRule(
+ import session.sqlContext.implicits._
+
+ val input = Seq((100L), (200L), (300L))
+ val data = input.toDF("vals").repartition(1)
+ val df = data.selectExpr("vals + 1")
+ val result = df.collect()
+ assert(result sameElements input.map(x => Row(x + 2)))
+} finally {
+ stop(session)
+}
+ }
Review comment:
Ur, no~ Not moving this test case. What I meant was a small new UT inside
some test suites in that packages.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1
HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546204959 If ^ is easy, and can be done in this PR, without a separate branch, 1. we can test if the tests pass 2. we release preview and 3. revert this commit from the master. Sorry if this was discussed already somewhere but we're not going to vote on it right? It we're going to vote and make more RC for preview, it might best to have a separate branch. Seems like merging script issue can be easily fixed in this case. The latest branch is decided by sorting order so we can name it like `branch-preview-3.0`. JIRA fixed version we will have to fix this line https://github.com/apache/spark/blob/2ec3265ae76fc1e136e44c240c476ce572b679df/dev/merge_spark_pr.py#L244 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership
dongjoon-hyun closed pull request #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership URL: https://github.com/apache/spark/pull/26248 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1
HyukjinKwon edited a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546204959 If ^ is easy, and can be done in this PR, without a separate branch, 1. we can test if the tests pass 2. we release preview and 3. revert this commit from the master. Sorry if this was discussed already somewhere but we're not going to vote on it right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rongma1997 opened a new pull request #26137: [SPARK-29490][SQL] Reset 'WritableColumnVector' in 'RowToColumnarExec'
rongma1997 opened a new pull request #26137: [SPARK-29490][SQL] Reset 'WritableColumnVector' in 'RowToColumnarExec' URL: https://github.com/apache/spark/pull/26137 ### What changes were proposed in this pull request? Reset the `WritableColumnVector` when getting "next" ColumnarBatch in `RowToColumnarExec` ### Why are the changes needed? When converting `Iterator[InternalRow]` to `Iterator[ColumnarBatch]`, the vectors used to create a new `ColumnarBatch` should be reset in the iterator's "next()" method. ### Does this PR introduce any user-facing change? No ### How was this patch tested? N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rongma1997 closed pull request #26137: [SPARK-29490][SQL] Reset 'WritableColumnVector' in 'RowToColumnarExec'
rongma1997 closed pull request #26137: [SPARK-29490][SQL] Reset 'WritableColumnVector' in 'RowToColumnarExec' URL: https://github.com/apache/spark/pull/26137 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD
dongjoon-hyun commented on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD URL: https://github.com/apache/spark/pull/26251#issuecomment-546205184 cc @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1
HyukjinKwon commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546204959 If ^ is easy, and can be done in this PR, 1. we can test if the tests pass 2. we release preview and 3. revert this commit from the master. Sorry if this was discussed already somewhere but we're not going to vote on it right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership
dongjoon-hyun commented on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership URL: https://github.com/apache/spark/pull/26248#issuecomment-546204751 Merged to branch-2.4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener.
AmplabJenkins commented on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener. URL: https://github.com/apache/spark/pull/26218#issuecomment-546204583 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112633/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener.
AmplabJenkins removed a comment on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener. URL: https://github.com/apache/spark/pull/26218#issuecomment-546204583 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112633/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener.
dongjoon-hyun closed pull request #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener. URL: https://github.com/apache/spark/pull/26218 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener.
AmplabJenkins removed a comment on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener. URL: https://github.com/apache/spark/pull/26218#issuecomment-546204578 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener.
AmplabJenkins commented on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener. URL: https://github.com/apache/spark/pull/26218#issuecomment-546204578 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener.
SparkQA removed a comment on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener. URL: https://github.com/apache/spark/pull/26218#issuecomment-546158003 **[Test build #112633 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112633/testReport)** for PR 26218 at commit [`6377c70`](https://github.com/apache/spark/commit/6377c701620d9c6b344993a8bf4c2c1acd267c00). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener.
SparkQA commented on issue #26218: [SPARK-29562][sql] Speed up and slim down metric aggregation in SQL listener. URL: https://github.com/apache/spark/pull/26218#issuecomment-546204249 **[Test build #112633 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112633/testReport)** for PR 26218 at commit [`6377c70`](https://github.com/apache/spark/commit/6377c701620d9c6b344993a8bf4c2c1acd267c00). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] firestarman edited a comment on issue #25983: [SPARK-29327][MLLIB]Support specifying features via multiple columns
firestarman edited a comment on issue #25983: [SPARK-29327][MLLIB]Support specifying features via multiple columns URL: https://github.com/apache/spark/pull/25983#issuecomment-546179121 @zhengruifeng > `VectorAssembler` only trigger a `first()` job to get the sizes of input vectors. > And I think we can futhuer optimize it so that this `first()` job is not triggered when all columns are double values. > > > Do you mean column-based parallelism used in distributed tree building? Such function is not exposed to end users, and what you need to do is only to set params like `(..., updater=distcol)`. > If some alg will benefit from column-based parallelism, I guess it is better to split the features internally. Thanks for review. Actually this is not only for alg that benefit from column-based parallelism, but also for spark ML application users, with this PR and its implemention in MLLibs, they don't need `VectorAssembler` any more for ML features column things, instead simply and directly set feature column names to `Predictor`, just like what you do with `label` and `prediction`. What's more, the original column names are required for MLLibs to parse the info of the column-based datasets. But currently there is no API to save the original column names in a `Predictor` . > No alg in MLLibs is designed to fit/transform with column-based datasets for now, so I do not prefer to add this feature. I cannot find any related docs in [XGBoost Parameters](https://xgboost.readthedocs.io/en/latest/parameter.html#xgboost-parameters). Could you please provide a link for this? XGBoost should support column-based datasets now. you can find clues from code or PRs, like https://github.com/dmlc/xgboost/pull/4816 https://github.com/dmlc/xgboost/pull/4850 https://github.com/dmlc/xgboost/pull/4902 One more thing, would it be more reasonable to define the APIs firstly in Spakr ML interface for requirements from users, then have them implemented in Spark MLLibs and other ML implementations ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership
AmplabJenkins removed a comment on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership URL: https://github.com/apache/spark/pull/26248#issuecomment-546203589 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112642/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership
AmplabJenkins removed a comment on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership URL: https://github.com/apache/spark/pull/26248#issuecomment-546203586 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership
AmplabJenkins commented on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership URL: https://github.com/apache/spark/pull/26248#issuecomment-546203589 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112642/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership
AmplabJenkins commented on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership URL: https://github.com/apache/spark/pull/26248#issuecomment-546203586 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership
SparkQA removed a comment on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership URL: https://github.com/apache/spark/pull/26248#issuecomment-546179470 **[Test build #112642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112642/testReport)** for PR 26248 at commit [`08929ca`](https://github.com/apache/spark/commit/08929cae4093188e0fc22595906f8bc968c09e72). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-546203187 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112640/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD
AmplabJenkins commented on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD URL: https://github.com/apache/spark/pull/26251#issuecomment-546203538 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17611/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD
AmplabJenkins removed a comment on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD URL: https://github.com/apache/spark/pull/26251#issuecomment-546203532 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD
AmplabJenkins removed a comment on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD URL: https://github.com/apache/spark/pull/26251#issuecomment-546203538 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17611/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD
AmplabJenkins commented on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD URL: https://github.com/apache/spark/pull/26251#issuecomment-546203532 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership
SparkQA commented on issue #26248: [SPARK-29498][SQL][2.4] CatalogTable to HiveTable should not change the table's ownership URL: https://github.com/apache/spark/pull/26248#issuecomment-546203424 **[Test build #112642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112642/testReport)** for PR 26248 at commit [`08929ca`](https://github.com/apache/spark/commit/08929cae4093188e0fc22595906f8bc968c09e72). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-546203184 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD
SparkQA commented on issue #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD URL: https://github.com/apache/spark/pull/26251#issuecomment-546203324 **[Test build #112650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112650/testReport)** for PR 26251 at commit [`4d9d474`](https://github.com/apache/spark/commit/4d9d474dc936fdb3714497ea324a077a4559cf50). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
SparkQA commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546203321 **[Test build #112651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112651/testReport)** for PR 26246 at commit [`17273a0`](https://github.com/apache/spark/commit/17273a033181d175a4e22536b49ba09115b5dd51). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-546203187 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112640/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-546203184 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
SparkQA removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-546174936 **[Test build #112640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112640/testReport)** for PR 25333 at commit [`e698c49`](https://github.com/apache/spark/commit/e698c49d50d723751217f1e635a4f6b9c745ec32). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1
HyukjinKwon commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546203075 Thanks for pointer @dongjoon-hyun. Alright, then, we don't do PyPI too I guess @holdenk? ``` /tmp/tmp.RRgnNxjsCA/3.5/lib/python3.5/site-packages/setuptools/dist.py:398: UserWarning: Normalizing '3.0.0.preview' to '3.0.0rc0' .. Requirement '/home/jenkins/workspace/SparkPullRequestBuilder@2/python/dist/pyspark-3.0.0.preview.tar.gz' looks like a filename, but the file does not exist Processing ./python/dist/pyspark-3.0.0.preview.tar.gz Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/home/jenkins/workspace/SparkPullRequestBuilder@2/python/dist/pyspark-3.0.0.preview.tar.gz' ``` This looks just pip packaging test failure. `setup.py` is only for PIP packaging and we don't probably need it. Can we just completely skip pip packaging or use some workarounds to disable the normalizing the name (https://github.com/pypa/setuptools/issues/308) for this preview? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya opened a new pull request #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD
viirya opened a new pull request #26251: [SPARK-29182][Core][Followup] Cache preferred locations of checkpointed RDD URL: https://github.com/apache/spark/pull/26251 ### What changes were proposed in this pull request? This is a followup to #25856. This fixes the document about the config value of spark.rdd.checkpoint.cachePreferredLocsExpireTime. ### Why are the changes needed? The document is not correct. spark.rdd.checkpoint.cachePreferredLocsExpireTime can not be 0. ### Does this PR introduce any user-facing change? No ### How was this patch tested? This is document only change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
SparkQA commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-546202978 **[Test build #112640 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112640/testReport)** for PR 25333 at commit [`e698c49`](https://github.com/apache/spark/commit/e698c49d50d723751217f1e635a4f6b9c745ec32). * This patch **fails PySpark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary
AmplabJenkins removed a comment on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary URL: https://github.com/apache/spark/pull/25899#issuecomment-546202548 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112635/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary
AmplabJenkins removed a comment on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary URL: https://github.com/apache/spark/pull/25899#issuecomment-546202545 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary
AmplabJenkins commented on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary URL: https://github.com/apache/spark/pull/25899#issuecomment-546202548 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112635/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary
AmplabJenkins commented on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary URL: https://github.com/apache/spark/pull/25899#issuecomment-546202545 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546202260 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins removed a comment on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546202263 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17609/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1
AmplabJenkins removed a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546202298 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1
AmplabJenkins removed a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546202301 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17610/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary
SparkQA removed a comment on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary URL: https://github.com/apache/spark/pull/25899#issuecomment-546159429 **[Test build #112635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112635/testReport)** for PR 25899 at commit [`b2fb28a`](https://github.com/apache/spark/commit/b2fb28afbe2a85e7ad08090428a45054e4e1cc91). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546202263 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17609/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1
AmplabJenkins commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546202298 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1
AmplabJenkins commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546202301 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17610/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary
SparkQA commented on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary URL: https://github.com/apache/spark/pull/25899#issuecomment-546202247 **[Test build #112635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112635/testReport)** for PR 25899 at commit [`b2fb28a`](https://github.com/apache/spark/commit/b2fb28afbe2a85e7ad08090428a45054e4e1cc91). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
AmplabJenkins commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546202260 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25856: [SPARK-29182][Core] Cache preferred locations of checkpointed RDD
viirya commented on a change in pull request #25856: [SPARK-29182][Core] Cache
preferred locations of checkpointed RDD
URL: https://github.com/apache/spark/pull/25856#discussion_r338887135
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -783,6 +783,17 @@ package object config {
.booleanConf
.createWithDefault(false)
+ private[spark] val CACHE_CHECKPOINT_PREFERRED_LOCS_EXPIRE_TIME =
+ConfigBuilder("spark.rdd.checkpoint.cachePreferredLocsExpireTime")
+ .internal()
+ .doc("Expire time in minutes for caching preferred locations of
checkpointed RDD." +
+"Caching preferred locations can relieve query loading to DFS and save
the query " +
+"time. The drawback is that the cached locations can be possibly
outdated and " +
+"lose data locality. If this config is not specified or is 0, it will
not cache.")
Review comment:
yea, good catch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1
AmplabJenkins removed a comment on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546088195 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1
SparkQA commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546202021 **[Test build #112649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112649/testReport)** for PR 26243 at commit [`f3731db`](https://github.com/apache/spark/commit/f3731dbb42e01d43b1f545c182fb91ecf8932a80). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
SparkQA commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546202019 **[Test build #112648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112648/testReport)** for PR 26246 at commit [`17273a0`](https://github.com/apache/spark/commit/17273a033181d175a4e22536b49ba09115b5dd51). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
shahidki31 commented on a change in pull request #26246: [SPARK-29589][WEBUI]
Support pagination for sqlstats session table in JDBC/ODBC Session page
URL: https://github.com/apache/spark/pull/26246#discussion_r338886928
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala
##
@@ -328,11 +320,12 @@ private[ui] class SqlStatsPagedTable(
{if (info.closeTimestamp > 0) formatDate(info.closeTimestamp)}
+
-{UIUtils.formatDuration(executionTime)}
+{formatDurationVerbose(executionTime)}
Review comment:
Yes. For example 5 sec 356 ms, `formatDuration` will give 5.3 s, whereas
`formatDurationVerbose` will give 5 sec 356 ms. I think, we should change this
in other pages too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1
HyukjinKwon commented on issue #26243: Prepare Spark release v3.0.0-preview-rc1 URL: https://github.com/apache/spark/pull/26243#issuecomment-546201731 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page
shahidki31 commented on issue #26246: [SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page URL: https://github.com/apache/spark/pull/26246#issuecomment-546201876 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25856: [SPARK-29182][Core] Cache preferred locations of checkpointed RDD
gatorsmile commented on a change in pull request #25856: [SPARK-29182][Core]
Cache preferred locations of checkpointed RDD
URL: https://github.com/apache/spark/pull/25856#discussion_r338886220
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -783,6 +783,17 @@ package object config {
.booleanConf
.createWithDefault(false)
+ private[spark] val CACHE_CHECKPOINT_PREFERRED_LOCS_EXPIRE_TIME =
+ConfigBuilder("spark.rdd.checkpoint.cachePreferredLocsExpireTime")
+ .internal()
+ .doc("Expire time in minutes for caching preferred locations of
checkpointed RDD." +
+"Caching preferred locations can relieve query loading to DFS and save
the query " +
+"time. The drawback is that the cached locations can be possibly
outdated and " +
+"lose data locality. If this config is not specified or is 0, it will
not cache.")
Review comment:
The following checkValue already disallows 0, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #26247: [SPARK-29566][ML] Imputer should support single-column input/output
zhengruifeng commented on a change in pull request #26247: [SPARK-29566][ML]
Imputer should support single-column input/output
URL: https://github.com/apache/spark/pull/26247#discussion_r338885229
##
File path: python/pyspark/ml/feature.py
##
@@ -1284,11 +1333,11 @@ def __init__(self, strategy="mean",
missingValue=float("nan"), inputCols=None,
@keyword_only
@since("2.2.0")
-def setParams(self, strategy="mean", missingValue=float("nan"),
inputCols=None,
- outputCols=None):
+def setParams(self, strategy="mean", missingValue=float("nan"),
inputCol=None, inputCols=None,
+ outputCol=None, outputCols=None):
"""
-setParams(self, strategy="mean", missingValue=float("nan"),
inputCols=None, \
- outputCols=None)
+setParams(self, strategy="mean", missingValue=float("nan"),
inputCol=None, \
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #26247: [SPARK-29566][ML] Imputer should support single-column input/output
zhengruifeng commented on a change in pull request #26247: [SPARK-29566][ML]
Imputer should support single-column input/output
URL: https://github.com/apache/spark/pull/26247#discussion_r338885206
##
File path: python/pyspark/ml/feature.py
##
@@ -1284,11 +1333,11 @@ def __init__(self, strategy="mean",
missingValue=float("nan"), inputCols=None,
@keyword_only
@since("2.2.0")
-def setParams(self, strategy="mean", missingValue=float("nan"),
inputCols=None,
- outputCols=None):
+def setParams(self, strategy="mean", missingValue=float("nan"),
inputCol=None, inputCols=None,
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #26247: [SPARK-29566][ML] Imputer should support single-column input/output
zhengruifeng commented on a change in pull request #26247: [SPARK-29566][ML]
Imputer should support single-column input/output
URL: https://github.com/apache/spark/pull/26247#discussion_r338885161
##
File path: python/pyspark/ml/feature.py
##
@@ -1270,11 +1319,11 @@ class Imputer(JavaEstimator, _ImputerParams,
JavaMLReadable, JavaMLWritable):
"""
@keyword_only
-def __init__(self, strategy="mean", missingValue=float("nan"),
inputCols=None,
- outputCols=None):
+def __init__(self, strategy="mean", missingValue=float("nan"),
inputCol=None, inputCols=None,
Review comment:
Here we must place newly added params `inputCol` & `outputCol` behind
existing ones
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer
AmplabJenkins removed a comment on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer URL: https://github.com/apache/spark/pull/26109#issuecomment-546199317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112634/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer
AmplabJenkins removed a comment on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer URL: https://github.com/apache/spark/pull/26109#issuecomment-546199309 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #26247: [SPARK-29566][ML] Imputer should support single-column input/output
zhengruifeng commented on a change in pull request #26247: [SPARK-29566][ML]
Imputer should support single-column input/output
URL: https://github.com/apache/spark/pull/26247#discussion_r338885189
##
File path: python/pyspark/ml/feature.py
##
@@ -1270,11 +1319,11 @@ class Imputer(JavaEstimator, _ImputerParams,
JavaMLReadable, JavaMLWritable):
"""
@keyword_only
-def __init__(self, strategy="mean", missingValue=float("nan"),
inputCols=None,
- outputCols=None):
+def __init__(self, strategy="mean", missingValue=float("nan"),
inputCol=None, inputCols=None,
+ outputCol=None, outputCols=None):
"""
-__init__(self, strategy="mean", missingValue=float("nan"),
inputCols=None, \
- outputCols=None):
+__init__(self, strategy="mean", missingValue=float("nan"),
inputCol=None, \
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer
AmplabJenkins commented on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer URL: https://github.com/apache/spark/pull/26109#issuecomment-546199317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112634/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer
AmplabJenkins commented on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer URL: https://github.com/apache/spark/pull/26109#issuecomment-546199309 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer
SparkQA removed a comment on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer URL: https://github.com/apache/spark/pull/26109#issuecomment-546157987 **[Test build #112634 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112634/testReport)** for PR 26109 at commit [`6e908d1`](https://github.com/apache/spark/commit/6e908d172981c48790a7ac6692cdc19652caaf8c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer
SparkQA commented on issue #26109: [SPARK-29461][SQL] Measure the number of records being updated for JDBC writer URL: https://github.com/apache/spark/pull/26109#issuecomment-546199001 **[Test build #112634 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112634/testReport)** for PR 26109 at commit [`6e908d1`](https://github.com/apache/spark/commit/6e908d172981c48790a7ac6692cdc19652caaf8c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26249: [SPARK-29597][DOCS] Deprecate old Java 8 versions prior to 8u92
dongjoon-hyun commented on issue #26249: [SPARK-29597][DOCS] Deprecate old Java 8 versions prior to 8u92 URL: https://github.com/apache/spark/pull/26249#issuecomment-546198799 Thank you, @maropu ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26219: [SPARK-29563][SQL] CREATE TABLE LIKE should look up catalog/table like v2 commands
AmplabJenkins removed a comment on issue #26219: [SPARK-29563][SQL] CREATE TABLE LIKE should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26219#issuecomment-546197477 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26219: [SPARK-29563][SQL] CREATE TABLE LIKE should look up catalog/table like v2 commands
AmplabJenkins removed a comment on issue #26219: [SPARK-29563][SQL] CREATE TABLE LIKE should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26219#issuecomment-546197485 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112632/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26219: [SPARK-29563][SQL] CREATE TABLE LIKE should look up catalog/table like v2 commands
AmplabJenkins commented on issue #26219: [SPARK-29563][SQL] CREATE TABLE LIKE should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26219#issuecomment-546197485 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112632/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
