[GitHub] [spark] SparkQA commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-967796938 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49652/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
SparkQA commented on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-967796776 **[Test build #145184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145184/testReport)** for PR 34579 at commit [`0b683a8`](https://github.com/apache/spark/commit/0b683a8bd99ef443fedc52c803b6e1878ad8755e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
AmplabJenkins removed a comment on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-967796597 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145180/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
AmplabJenkins removed a comment on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-967796596 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145182/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
AmplabJenkins commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-967796597 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145180/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
AmplabJenkins commented on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-967796596 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145182/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] hgs19921112 opened a new pull request #34492: [SPARK-37216][SQL] Add the Hive macro functionality to SparkSQL
hgs19921112 opened a new pull request #34492: URL: https://github.com/apache/spark/pull/34492 ### What changes were proposed in this pull request? Add the Hive macro functionality to SparkSQL ### Why are the changes needed? Some Hive sql can move to SparkSQL Smoothly ### Does this PR introduce _any_ user-facing change? Some new DDL like 'create temparory macro ...' ### How was this patch tested? unit test Authored-by: hgs19921112 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] hgs19921112 closed pull request #34492: [SPARK-37216][SQL] Add the Hive macro functionality to SparkSQL
hgs19921112 closed pull request #34492: URL: https://github.com/apache/spark/pull/34492 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
SparkQA removed a comment on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-96255 **[Test build #145182 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145182/testReport)** for PR 34579 at commit [`0b683a8`](https://github.com/apache/spark/commit/0b683a8bd99ef443fedc52c803b6e1878ad8755e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
SparkQA removed a comment on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-967763140 **[Test build #145180 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145180/testReport)** for PR 33588 at commit [`91331c7`](https://github.com/apache/spark/commit/91331c76a9f211ce1ec91aeb3d95d082de8e6f98). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
SparkQA commented on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-967794722 **[Test build #145182 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145182/testReport)** for PR 34579 at commit [`0b683a8`](https://github.com/apache/spark/commit/0b683a8bd99ef443fedc52c803b6e1878ad8755e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
SparkQA commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-967794238 **[Test build #145180 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145180/testReport)** for PR 33588 at commit [`91331c7`](https://github.com/apache/spark/commit/91331c76a9f211ce1ec91aeb3d95d082de8e6f98). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on pull request #34572: Don't obtain JDBC connection for empty partitions
srowen commented on pull request #34572: URL: https://github.com/apache/spark/pull/34572#issuecomment-967794243 Oops I just forgot to link it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
dongjoon-hyun commented on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-967792488 There are two known failures, but this PR has another failure. Although it looks like a flaky one, I re-trigger tests to make it sure. ``` - Run SparkRemoteFileTest using a remote data file *** FAILED *** ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
dongjoon-hyun commented on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-967792347 Retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
dongjoon-hyun commented on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967792227 Thank you so much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34574: [SPARK-37139][PYTHON][FOLLOWUP] Fix class variable type hints in taskcontext.py
HyukjinKwon closed pull request #34574: URL: https://github.com/apache/spark/pull/34574 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34574: [SPARK-37139][PYTHON][FOLLOWUP] Fix class variable type hints in taskcontext.py
HyukjinKwon commented on pull request #34574: URL: https://github.com/apache/spark/pull/34574#issuecomment-967792124 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34572: Don't obtain JDBC connection for empty partitions
HyukjinKwon commented on pull request #34572: URL: https://github.com/apache/spark/pull/34572#issuecomment-967792085 @srowen, it's sort of minor but I think it might better to create a JIRA (since it's a perf improvement technically). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-967792027 **[Test build #145183 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145183/testReport)** for PR 34575 at commit [`fc043fd`](https://github.com/apache/spark/commit/fc043fd5e2fa65aadae7648ef67851ffa6d379c9). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
AmplabJenkins removed a comment on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-967680792 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
HyukjinKwon commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-967791845 ok to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
sarutak commented on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967790839 Merged to `master`. Thank you @dongjoon-hyun ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak closed pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
sarutak closed pull request #34576: URL: https://github.com/apache/spark/pull/34576 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34555: [SPARK-37288][PYTHON][3.2] Backport since annotation update
HyukjinKwon commented on pull request #34555: URL: https://github.com/apache/spark/pull/34555#issuecomment-967790800 I switched the JIRA to a bug, and described which bug this PR fixes. It wouldn't introduce any user-facing behaviour. cc @dongjoon-hyun. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
AmplabJenkins removed a comment on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-967790143 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49651/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34543: [SPARK-37266][SQL] View text can only be SELECT queries
AmplabJenkins commented on pull request #34543: URL: https://github.com/apache/spark/pull/34543#issuecomment-967790208 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145178/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34543: [SPARK-37266][SQL] View text can only be SELECT queries
AmplabJenkins removed a comment on pull request #34543: URL: https://github.com/apache/spark/pull/34543#issuecomment-967790208 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145178/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
AmplabJenkins commented on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-967790143 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49651/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34543: [SPARK-37266][SQL] View text can only be SELECT queries
SparkQA removed a comment on pull request #34543: URL: https://github.com/apache/spark/pull/34543#issuecomment-967752729 **[Test build #145178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145178/testReport)** for PR 34543 at commit [`0f46831`](https://github.com/apache/spark/commit/0f46831ec573ab90b72ffd917a773ebdd840e17f). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34543: [SPARK-37266][SQL] View text can only be SELECT queries
SparkQA commented on pull request #34543: URL: https://github.com/apache/spark/pull/34543#issuecomment-967789997 **[Test build #145178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145178/testReport)** for PR 34543 at commit [`0f46831`](https://github.com/apache/spark/commit/0f46831ec573ab90b72ffd917a773ebdd840e17f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
dongjoon-hyun commented on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967788291 Could you review this, @sarutak ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
dongjoon-hyun edited a comment on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967788067 Thank you, @viirya and @sarutak ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
dongjoon-hyun commented on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967788067 Thank you, @sarutak ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
SparkQA commented on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-967787532 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49651/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak closed pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
sarutak closed pull request #34577: URL: https://github.com/apache/spark/pull/34577 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
sarutak commented on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967786943 LGTM. Merging to `master`. Thank you @dongjoon-hyun ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34504: [SPARK-37226][SQL] Filter push down through window
AmplabJenkins removed a comment on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-967784061 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49650/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window
AmplabJenkins commented on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-967784061 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49650/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
SparkQA commented on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-967782600 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49651/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window
SparkQA commented on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-967781922 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49650/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34532: [SPARK-37256][SQL] Replace `ScalaObjectMapper` with `ClassTagExtensions` to fix compilation warning
LuciferYang commented on pull request #34532: URL: https://github.com/apache/spark/pull/34532#issuecomment-967778794 Should we merge this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34454: [SPARK-37013][CORE][SQL][FOLLOWUP] Use the new error framework to throw error in `FormatString`
LuciferYang commented on pull request #34454: URL: https://github.com/apache/spark/pull/34454#issuecomment-967778711 Anything else needs to be changed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
SparkQA commented on pull request #34579: URL: https://github.com/apache/spark/pull/34579#issuecomment-96255 **[Test build #145182 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145182/testReport)** for PR 34579 at commit [`0b683a8`](https://github.com/apache/spark/commit/0b683a8bd99ef443fedc52c803b6e1878ad8755e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak opened a new pull request #34579: [SPARK-37314][K8S][BUILD] Upgrade kubernetes-client to 5.10.1
sarutak opened a new pull request #34579: URL: https://github.com/apache/spark/pull/34579 ### What changes were proposed in this pull request? This PR upgrades kubernetes-client from `5.9.0` to `5.10.1`. ### Why are the changes needed? kubernetes-client 5.10.0 and 5.10.1 were released, which include some bug fixes. https://github.com/fabric8io/kubernetes-client/releases/tag/v5.10.0 https://github.com/fabric8io/kubernetes-client/releases/tag/v5.10.1 Especially, the connection leak issue would affect Spark. https://github.com/fabric8io/kubernetes-client/issues/3561 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
AmplabJenkins removed a comment on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967776695 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145179/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33008: [WIP][SPARK-35801][SQL] Support DELETE operations that require rewriting data
AmplabJenkins removed a comment on pull request #33008: URL: https://github.com/apache/spark/pull/33008#issuecomment-967776577 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145176/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
AmplabJenkins commented on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967776695 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145179/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
AmplabJenkins removed a comment on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-967776576 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49649/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
AmplabJenkins commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-967776576 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49649/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33008: [WIP][SPARK-35801][SQL] Support DELETE operations that require rewriting data
AmplabJenkins commented on pull request #33008: URL: https://github.com/apache/spark/pull/33008#issuecomment-967776577 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145176/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
SparkQA removed a comment on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967755244 **[Test build #145179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145179/testReport)** for PR 34577 at commit [`6d0a9a4`](https://github.com/apache/spark/commit/6d0a9a4aee0f0256e921bc037a435f0e99148a78). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
SparkQA commented on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967776464 **[Test build #145179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145179/testReport)** for PR 34577 at commit [`6d0a9a4`](https://github.com/apache/spark/commit/6d0a9a4aee0f0256e921bc037a435f0e99148a78). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33008: [WIP][SPARK-35801][SQL] Support DELETE operations that require rewriting data
SparkQA removed a comment on pull request #33008: URL: https://github.com/apache/spark/pull/33008#issuecomment-967722938 **[Test build #145176 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145176/testReport)** for PR 33008 at commit [`5e95a57`](https://github.com/apache/spark/commit/5e95a57812e0a46205f2b6e2993790b520b07067). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33008: [WIP][SPARK-35801][SQL] Support DELETE operations that require rewriting data
SparkQA commented on pull request #33008: URL: https://github.com/apache/spark/pull/33008#issuecomment-967775164 **[Test build #145176 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145176/testReport)** for PR 33008 at commit [`5e95a57`](https://github.com/apache/spark/commit/5e95a57812e0a46205f2b6e2993790b520b07067). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class InternalRowProjection(schema: StructType, colOrdinals: Seq[Int]) extends InternalRow ` * `trait RewriteRowLevelCommand extends Rule[LogicalPlan] ` * `case class ReplaceData(` * `case class WriteDelta(` * `trait RowLevelCommand extends Command with SupportsSubquery ` * `case class WriteDeltaProjections(` * `case class RowLevelOperationInfoImpl(` * `case class RowLevelOperationTable(` * `case class ReplaceDataExec(` * `case class WriteDeltaExec(` * `trait WritingSparkTask extends Logging with Serializable ` * `case class DeltaWritingSparkTask(projs: WriteDeltaProjections) extends WritingSparkTask ` * `case class DeltaWithMetadataWritingSparkTask(` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window
SparkQA commented on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-967774937 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49650/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
SparkQA commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-967773338 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49649/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window
SparkQA commented on pull request #34504: URL: https://github.com/apache/spark/pull/34504#issuecomment-967770239 **[Test build #145181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145181/testReport)** for PR 34504 at commit [`a3a7648`](https://github.com/apache/spark/commit/a3a764888454c8f0778734ead1309c717e5e6407). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
AmplabJenkins removed a comment on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967770093 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49648/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
AmplabJenkins removed a comment on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967770094 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49646/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34543: [SPARK-37266][SQL] View text can only be SELECT queries
AmplabJenkins removed a comment on pull request #34543: URL: https://github.com/apache/spark/pull/34543#issuecomment-967770095 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49647/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
AmplabJenkins commented on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967770093 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49648/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34543: [SPARK-37266][SQL] View text can only be SELECT queries
AmplabJenkins commented on pull request #34543: URL: https://github.com/apache/spark/pull/34543#issuecomment-967770095 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49647/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
AmplabJenkins commented on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967770094 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49646/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
SparkQA commented on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967769116 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49646/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
SparkQA commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-967768287 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49649/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rmcyang commented on pull request #34461: [SPARK-37023][CORE] Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry
rmcyang commented on pull request #34461: URL: https://github.com/apache/spark/pull/34461#issuecomment-967767497 Filed SPARK-37313 for the issue that @Ngone51 raised. Thanks for the reviews! @mridulm @dongjoon-hyun @Ngone51 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
SparkQA commented on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967767066 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49648/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34543: [SPARK-37266][SQL] View text can only be SELECT queries
SparkQA commented on pull request #34543: URL: https://github.com/apache/spark/pull/34543#issuecomment-967766557 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49647/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sleep1661 commented on pull request #33872: [SPARK-36575][CORE] Should ignore task finished event if its task set is gone in TaskSchedulerImpl.handleSuccessfulTask
sleep1661 commented on pull request #33872: URL: https://github.com/apache/spark/pull/33872#issuecomment-967763573 @Ngone51 @mridulm I had created new PR (https://github.com/apache/spark/pull/34578) for this. BTW sorry about the unit test problems caused by this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
SparkQA commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-967763140 **[Test build #145180 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145180/testReport)** for PR 33588 at commit [`91331c7`](https://github.com/apache/spark/commit/91331c76a9f211ce1ec91aeb3d95d082de8e6f98). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
AmplabJenkins removed a comment on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967762581 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145177/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34578: [SPARK-37300][CORE] TaskSchedulerImpl should ignore task finished eve…
AmplabJenkins commented on pull request #34578: URL: https://github.com/apache/spark/pull/34578#issuecomment-967762640 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
AmplabJenkins commented on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967762581 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145177/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
SparkQA commented on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967761980 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49648/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
SparkQA commented on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967761226 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49646/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
SparkQA removed a comment on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967752700 **[Test build #145177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145177/testReport)** for PR 34576 at commit [`3c7494f`](https://github.com/apache/spark/commit/3c7494fe3b86739abb5ae144dd866dfeac8169a9). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34543: [SPARK-37266][SQL] View text can only be SELECT queries
SparkQA commented on pull request #34543: URL: https://github.com/apache/spark/pull/34543#issuecomment-967760254 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49647/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #34504: [SPARK-37226][SQL] Filter push down through window
wangyum commented on pull request #34504:
URL: https://github.com/apache/spark/pull/34504#issuecomment-967757751
@tanelk I did a benchmark and this implementation is faster than SPARK-37099.
```scala
import org.apache.spark.benchmark.Benchmark
val numRows = 1024 * 1024 * 20
spark.sql(s"CREATE TABLE t1 using parquet AS SELECT id as a, id as b FROM
range(${numRows}L)")
val benchmark = new Benchmark("Benchmark filter push down through window",
numRows, minNumIters = 5)
Seq(1, 1000).foreach { threshold =>
val name = s"Filter push down through window ${if (threshold > 1)
"(Enabled)" else "(Disabled)"}"
benchmark.addCase(name) { _ =>
withSQLConf("spark.sql.execution.topKSortFallbackThreshold" ->
s"$threshold") {
spark.sql("SELECT * FROM (SELECT *, ROW_NUMBER() OVER(ORDER BY a) AS
rn FROM t1) t WHERE rn > 100 and rn <=
200").write.format("noop").mode("Overwrite").save()
}
}
}
benchmark.addCase("[SPARK-37099][SQL] Impl a rank-based filter to optimize
top-k computation") { _ =>
withSQLConf(
"spark.sql.rankLimit.enabled" -> "true",
"spark.sql.execution.topKSortFallbackThreshold" -> "0") {
spark.sql("SELECT * FROM (SELECT *, ROW_NUMBER() OVER(ORDER BY a) AS rn
FROM t1) t WHERE rn > 100 and rn <=
200").write.format("noop").mode("Overwrite").save()
}
}
benchmark.run()
```
```
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark filter push down through window:
Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative
-
Filter push down through window (Disabled)
11289 177091128 1.9 538.3 1.0X
Filter push down through window (Enabled)
1252 1345 114 16.8 59.7 9.0X
[SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation
2542 2666 143 8.3 121.2 4.4X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sleep1661 opened a new pull request #34578: [SPARK-37300][CORE] TaskSchedulerImpl should ignore task finished eve…
sleep1661 opened a new pull request #34578: URL: https://github.com/apache/spark/pull/34578 ### What changes were proposed in this pull request? `TaskSchedulerImpl` handle task finished event at `handleSuccessfulTask` and `handleFailedTask` , but in some case the task was already finished state, which we should ignore task finished event. Case describe: when a executor finished a task of some stage, the driver will receive a StatusUpdate event to handle it. At the same time the driver found the executor heartbeat timed out, so the dirver also need handle ExecutorLost event simultaneously. There was a race condition issues here, which will make TaskSetManager.successful and TaskSetManager.tasksSuccessful wrong result. ### Why are the changes needed? It will cause `TaskSetManager.successful` and `TaskSetManager.tasksSuccessful` wrong result. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Add a new test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
SparkQA commented on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967756785 **[Test build #145177 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145177/testReport)** for PR 34576 at commit [`3c7494f`](https://github.com/apache/spark/commit/3c7494fe3b86739abb5ae144dd866dfeac8169a9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
SparkQA commented on pull request #34577: URL: https://github.com/apache/spark/pull/34577#issuecomment-967755244 **[Test build #145179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145179/testReport)** for PR 34577 at commit [`6d0a9a4`](https://github.com/apache/spark/commit/6d0a9a4aee0f0256e921bc037a435f0e99148a78). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #34577: [SPARK-37312][TESTS] Add `.java-version` to `.gitignore` and `.rat-excludes`
dongjoon-hyun opened a new pull request #34577: URL: https://github.com/apache/spark/pull/34577 ### What changes were proposed in this pull request? To support Java 8/11/17 test more easily, this PR aims to add `.java-version` to `.gitignore` and `.rat-excludes`. ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34543: [SPARK-37266][SQL] View text can only be SELECT queries
SparkQA commented on pull request #34543: URL: https://github.com/apache/spark/pull/34543#issuecomment-967752729 **[Test build #145178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145178/testReport)** for PR 34543 at commit [`0f46831`](https://github.com/apache/spark/commit/0f46831ec573ab90b72ffd917a773ebdd840e17f). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
SparkQA commented on pull request #34576: URL: https://github.com/apache/spark/pull/34576#issuecomment-967752700 **[Test build #145177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145177/testReport)** for PR 34576 at commit [`3c7494f`](https://github.com/apache/spark/commit/3c7494fe3b86739abb5ae144dd866dfeac8169a9). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33008: [WIP][SPARK-35801][SQL] Support DELETE operations that require rewriting data
AmplabJenkins removed a comment on pull request #33008: URL: https://github.com/apache/spark/pull/33008#issuecomment-967752566 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49645/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33008: [WIP][SPARK-35801][SQL] Support DELETE operations that require rewriting data
AmplabJenkins commented on pull request #33008: URL: https://github.com/apache/spark/pull/33008#issuecomment-967752566 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49645/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #34576: [SPARK-37282][TESTS][FOLLOWUP] Mark `YarnShuffleServiceSuite` as ExtendedLevelDBTest
dongjoon-hyun opened a new pull request #34576: URL: https://github.com/apache/spark/pull/34576 ### What changes were proposed in this pull request? This PR is a follow-up of #34548. This is missed due to `-Pyarn` profile. ### Why are the changes needed? This is required to pass `yarn` module on Apple Silicon. ``` $ build/sbt "yarn/test" ... [info] YarnShuffleServiceSuite: [info] org.apache.spark.network.yarn.YarnShuffleServiceSuite *** ABORTED *** (20 milliseconds) [info] java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 ... ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? A manual test on Apple Silicon. ``` $ build/sbt "yarn/test" -Pyarn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33008: [WIP][SPARK-35801][SQL] Support DELETE operations that require rewriting data
SparkQA commented on pull request #33008: URL: https://github.com/apache/spark/pull/33008#issuecomment-967748148 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49645/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34396: [SPARK-37124][SQL] Support RowToColumnarExec with Arrow format
HyukjinKwon commented on a change in pull request #34396:
URL: https://github.com/apache/spark/pull/34396#discussion_r748657399
##
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##
@@ -458,6 +462,34 @@ case class RowToColumnarExec(child: SparkPlan) extends
RowToColumnarTransition {
// This avoids calling `schema` in the RDD closure, so that we don't need
to include the entire
// plan (this) in the closure.
val localSchema = this.schema
+if (enableArrowColumnVector) {
+ val maxRecordsPerBatch = SQLConf.get.arrowMaxRecordsPerBatch
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+ return child.execute().mapPartitionsInternal { rowIterator =>
+val context = TaskContext.get()
+val allocator = ArrowUtils.getDefaultAllocator
+val bytesIterator = ArrowConverters
+ .toBatchIterator(rowIterator, localSchema, maxRecordsPerBatch,
timeZoneId, context)
Review comment:
Oh yeah. I think we shouldn't do the byte conversion step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34396: [SPARK-37124][SQL] Support RowToColumnarExec with Arrow format
HyukjinKwon commented on a change in pull request #34396:
URL: https://github.com/apache/spark/pull/34396#discussion_r748657399
##
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##
@@ -458,6 +462,34 @@ case class RowToColumnarExec(child: SparkPlan) extends
RowToColumnarTransition {
// This avoids calling `schema` in the RDD closure, so that we don't need
to include the entire
// plan (this) in the closure.
val localSchema = this.schema
+if (enableArrowColumnVector) {
+ val maxRecordsPerBatch = SQLConf.get.arrowMaxRecordsPerBatch
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+ return child.execute().mapPartitionsInternal { rowIterator =>
+val context = TaskContext.get()
+val allocator = ArrowUtils.getDefaultAllocator
+val bytesIterator = ArrowConverters
+ .toBatchIterator(rowIterator, localSchema, maxRecordsPerBatch,
timeZoneId, context)
Review comment:
Oh yeah. I think we shouldn't do the conversion step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #33114: [SPARK-35913][SQL] Create hive permanent function with owner name
github-actions[bot] closed pull request #33114: URL: https://github.com/apache/spark/pull/33114 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #27432: [SPARK-28325][SQL] Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
github-actions[bot] commented on pull request #27432: URL: https://github.com/apache/spark/pull/27432#issuecomment-967738019 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #33520: [SPARK-36289][SQL] Rewrite distinct count case when expressions without Expand node
github-actions[bot] closed pull request #33520: URL: https://github.com/apache/spark/pull/33520 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #32979: [SPARK-35828][K8S] Skip retrieving the non-exist driver pod for client mode
github-actions[bot] commented on pull request #32979: URL: https://github.com/apache/spark/pull/32979#issuecomment-967738007 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #33361: [SPARK-36155][SQL] Eliminate outer join base uniqueness
github-actions[bot] closed pull request #33361: URL: https://github.com/apache/spark/pull/33361 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] BryanCutler commented on a change in pull request #34396: [SPARK-37124][SQL] Support RowToColumnarExec with Arrow format
BryanCutler commented on a change in pull request #34396:
URL: https://github.com/apache/spark/pull/34396#discussion_r748648880
##
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##
@@ -458,6 +462,34 @@ case class RowToColumnarExec(child: SparkPlan) extends
RowToColumnarTransition {
// This avoids calling `schema` in the RDD closure, so that we don't need
to include the entire
// plan (this) in the closure.
val localSchema = this.schema
+if (enableArrowColumnVector) {
+ val maxRecordsPerBatch = SQLConf.get.arrowMaxRecordsPerBatch
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+ return child.execute().mapPartitionsInternal { rowIterator =>
+val context = TaskContext.get()
+val allocator = ArrowUtils.getDefaultAllocator
+val bytesIterator = ArrowConverters
+ .toBatchIterator(rowIterator, localSchema, maxRecordsPerBatch,
timeZoneId, context)
Review comment:
Sounds like this is working towards similar things. @xuechendi the
reason I brought up the intermediate conversion to bytes is that it's an
expensive step and not necessary if you are just converting `ArrowRecordBatch`
<-> `ColumnarBatch`. It's done in `ArrowConverters` specifically to send/read
Arrow messages over a socket with Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
brkyvz commented on a change in pull request #34575:
URL: https://github.com/apache/spark/pull/34575#discussion_r748647686
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
##
@@ -276,3 +276,10 @@ object LogicalPlanIntegrity {
checkIfSameExprIdNotReused(plan) && hasUniqueExprIdsForOutput(plan)
}
}
+
+/**
+ * A logical plan node with exposed metadata columns
Review comment:
A logical plan node that can generate metadata columns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
brkyvz commented on a change in pull request #34575:
URL: https://github.com/apache/spark/pull/34575#discussion_r748647591
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
##
@@ -438,3 +438,70 @@ object VirtualColumn {
val groupingIdName: String = "spark_grouping_id"
val groupingIdAttribute: UnresolvedAttribute =
UnresolvedAttribute(groupingIdName)
}
+
+/**
+ * The internal representation of the hidden metadata column
+ */
+class MetadataAttribute(
+override val name: String,
+override val dataType: DataType,
+override val nullable: Boolean = true,
+override val metadata: Metadata = Metadata.empty)(
+override val exprId: ExprId = NamedExpression.newExprId,
+override val qualifier: Seq[String] = Seq.empty[String])
+ extends AttributeReference(name, dataType, nullable, metadata)(exprId,
qualifier) {
Review comment:
Let's not extend `AttributeReference`, otherwise `copy` can cause issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
brkyvz commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-967735222 ok to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
