[GitHub] [spark] SparkQA commented on pull request #32504: [SPARK-35013][CORE] Don't allow to set spark.driver.cores=0
SparkQA commented on pull request #32504: URL: https://github.com/apache/spark/pull/32504#issuecomment-839125056 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32504: [SPARK-35013][CORE] Don't allow to set spark.driver.cores=0
SparkQA commented on pull request #32504: URL: https://github.com/apache/spark/pull/32504#issuecomment-839121292 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42918/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32469: [SPARK-35338][PYTHON] Separate arithmetic operations into data type based structures
SparkQA removed a comment on pull request #32469: URL: https://github.com/apache/spark/pull/32469#issuecomment-838861089 **[Test build #138389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138389/testReport)** for PR 32469 at commit [`a4a6581`](https://github.com/apache/spark/commit/a4a65811791745fa0a473f18253f72662f4e24eb). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32469: [SPARK-35338][PYTHON] Separate arithmetic operations into data type based structures
SparkQA commented on pull request #32469: URL: https://github.com/apache/spark/pull/32469#issuecomment-839088570 **[Test build #138389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138389/testReport)** for PR 32469 at commit [`a4a6581`](https://github.com/apache/spark/commit/a4a65811791745fa0a473f18253f72662f4e24eb). * This patch **fails PySpark pip packaging tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on pull request #32415: [SPARK-35295][ML] Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0
srowen commented on pull request #32415: URL: https://github.com/apache/spark/pull/32415#issuecomment-839063011 OK, in any event `core` is fine, just not sure why it doesn't show up in the transitive dependencies then. Yeah we don't want to depend on `all` except within the profile. I think this is OK. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32436: [SPARK-34764][CORE][K8S][UI] Propagate reason for exec loss to Web UI
SparkQA commented on pull request #32436: URL: https://github.com/apache/spark/pull/32436#issuecomment-839062078 **[Test build #138401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138401/testReport)** for PR 32436 at commit [`3ab5a09`](https://github.com/apache/spark/commit/3ab5a09845acc6f7674e4a43e753fe757b32bfd8). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32505: [SPARK-35373][BUILD] Check Maven artifact checksum in build/mvn
AmplabJenkins commented on pull request #32505: URL: https://github.com/apache/spark/pull/32505#issuecomment-839061303 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138384/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32389: [SPARK-35263] [TEST] Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
AmplabJenkins removed a comment on pull request #32389: URL: https://github.com/apache/spark/pull/32389#issuecomment-839059542 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138386/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32505: [SPARK-35373][BUILD] Check Maven artifact checksum in build/mvn
SparkQA removed a comment on pull request #32505: URL: https://github.com/apache/spark/pull/32505#issuecomment-838775701 **[Test build #138384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138384/testReport)** for PR 32505 at commit [`b0db41e`](https://github.com/apache/spark/commit/b0db41e6ee0154d247bbcd5575e1edcfc7f7f311). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32389: [SPARK-35263] [TEST] Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
AmplabJenkins commented on pull request #32389: URL: https://github.com/apache/spark/pull/32389#issuecomment-839059542 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138386/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32505: [SPARK-35373][BUILD] Check Maven artifact checksum in build/mvn
SparkQA commented on pull request #32505: URL: https://github.com/apache/spark/pull/32505#issuecomment-839059074 **[Test build #138384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138384/testReport)** for PR 32505 at commit [`b0db41e`](https://github.com/apache/spark/commit/b0db41e6ee0154d247bbcd5575e1edcfc7f7f311). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
AmplabJenkins removed a comment on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-839055033 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138388/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
SparkQA removed a comment on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-838860395 **[Test build #138388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138388/testReport)** for PR 32506 at commit [`45a6833`](https://github.com/apache/spark/commit/45a68335201ff7fbc69ec01ca0f2769fa058278d). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] luhenry commented on pull request #32415: [SPARK-35295][ML] Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0
luhenry commented on pull request #32415: URL: https://github.com/apache/spark/pull/32415#issuecomment-839056925 > So the `core` artifact is no longer part of the transitive deps? I would think `all` needs it, still, but, not sure. `core` is still part of the transitive dependencies through `breeze`. If we add `all`, then we made sure that it is on the classpath, and that `breeze` has access to the accelerated implementation in `com.github.fommil.netlib:all`. And `all` itself references `core`, so not referencing `core` isn't an issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32389: [SPARK-35263] [TEST] Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
SparkQA removed a comment on pull request #32389: URL: https://github.com/apache/spark/pull/32389#issuecomment-838776322 **[Test build #138386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138386/testReport)** for PR 32389 at commit [`b2abb87`](https://github.com/apache/spark/commit/b2abb87f3dfafc03ade66d3c3bc333440fde761a). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32494: [SPARK-35362][SQL] Update null count in the column stats for UNION operator stats estimation
SparkQA commented on pull request #32494: URL: https://github.com/apache/spark/pull/32494#issuecomment-839056599 **[Test build #138400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138400/testReport)** for PR 32494 at commit [`a1b1d5f`](https://github.com/apache/spark/commit/a1b1d5f7407efa2b5026139055d87ed63c1f3241). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32389: [SPARK-35263] [TEST] Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
SparkQA commented on pull request #32389: URL: https://github.com/apache/spark/pull/32389#issuecomment-839056329 **[Test build #138386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138386/testReport)** for PR 32389 at commit [`b2abb87`](https://github.com/apache/spark/commit/b2abb87f3dfafc03ade66d3c3bc333440fde761a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
AmplabJenkins commented on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-839055033 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138388/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
SparkQA commented on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-839052868 **[Test build #138388 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138388/testReport)** for PR 32506 at commit [`45a6833`](https://github.com/apache/spark/commit/45a68335201ff7fbc69ec01ca0f2769fa058278d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32498: [SPARK-35368][SQL] Update histogram statistics for RANGE operator for stats estimation
SparkQA commented on pull request #32498: URL: https://github.com/apache/spark/pull/32498#issuecomment-839051298 **[Test build #138399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138399/testReport)** for PR 32498 at commit [`45845e3`](https://github.com/apache/spark/commit/45845e383291a635c7a0dd5113fb5a18679d35ee). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32178: [DONOTMERGE] initial commit for skeleton ansible for jenkins worker config
AmplabJenkins removed a comment on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-839048801 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42917/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32178: [DONOTMERGE] initial commit for skeleton ansible for jenkins worker config
SparkQA commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-839048674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32178: [DONOTMERGE] initial commit for skeleton ansible for jenkins worker config
AmplabJenkins commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-839048801 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42917/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32507: [SPARK-35361][SQL] Improve performance for ApplyFunctionExpression
AmplabJenkins removed a comment on pull request #32507: URL: https://github.com/apache/spark/pull/32507#issuecomment-839048096 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42916/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32507: [SPARK-35361][SQL] Improve performance for ApplyFunctionExpression
AmplabJenkins commented on pull request #32507: URL: https://github.com/apache/spark/pull/32507#issuecomment-839048096 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42916/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32507: [SPARK-35361][SQL] Improve performance for ApplyFunctionExpression
SparkQA commented on pull request #32507: URL: https://github.com/apache/spark/pull/32507#issuecomment-839048025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32504: [SPARK-35013][CORE] Don't allow to set spark.driver.cores=0
SparkQA commented on pull request #32504: URL: https://github.com/apache/spark/pull/32504#issuecomment-839045451 **[Test build #138398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138398/testReport)** for PR 32504 at commit [`a63fcc3`](https://github.com/apache/spark/commit/a63fcc3e9a8d6cd51a1619829a854179e547254d). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32178: [DONOTMERGE] initial commit for skeleton ansible for jenkins worker config
SparkQA commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-839035267 **[Test build #138397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138397/testReport)** for PR 32178 at commit [`e809f2f`](https://github.com/apache/spark/commit/e809f2f717ddb355b375e2c52f56865413add5a2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32504: [SPARK-35013][CORE] Don't allow to set spark.driver.cores=0
SparkQA commented on pull request #32504: URL: https://github.com/apache/spark/pull/32504#issuecomment-839034321 **[Test build #138396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138396/testReport)** for PR 32504 at commit [`89a2796`](https://github.com/apache/spark/commit/89a2796d79d298fb34f94b7fa82c1f99c602f2df). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32292: [WIP][SPARK-35162][SQL] New SQL functions: TRY_ADD/TRY_DIVIDE
AmplabJenkins removed a comment on pull request #32292: URL: https://github.com/apache/spark/pull/32292#issuecomment-839033589 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42915/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32499: [SPARK-29145][SQL][FOLLOWUP] Support sub-queries in join conditions
AmplabJenkins removed a comment on pull request #32499: URL: https://github.com/apache/spark/pull/32499#issuecomment-839033592 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138381/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32503: [WIP] better error message
AmplabJenkins removed a comment on pull request #32503: URL: https://github.com/apache/spark/pull/32503#issuecomment-839033596 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138382/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32292: [WIP][SPARK-35162][SQL] New SQL functions: TRY_ADD/TRY_DIVIDE
AmplabJenkins commented on pull request #32292: URL: https://github.com/apache/spark/pull/32292#issuecomment-839033589 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42915/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32499: [SPARK-29145][SQL][FOLLOWUP] Support sub-queries in join conditions
AmplabJenkins commented on pull request #32499: URL: https://github.com/apache/spark/pull/32499#issuecomment-839033592 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138381/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32503: [WIP] better error message
AmplabJenkins commented on pull request #32503: URL: https://github.com/apache/spark/pull/32503#issuecomment-839033596 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138382/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32503: [WIP] better error message
SparkQA removed a comment on pull request #32503: URL: https://github.com/apache/spark/pull/32503#issuecomment-838590574 **[Test build #138382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138382/testReport)** for PR 32503 at commit [`1a6832e`](https://github.com/apache/spark/commit/1a6832e15302b47f34d9d00b172bf5bb0d18032c). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on pull request #32504: [SPARK-35013][CORE] Don't allow to set spark.driver.cores=0
shahidki31 commented on pull request #32504: URL: https://github.com/apache/spark/pull/32504#issuecomment-839025192 Sure @dongjoon-hyun . Let me go through the doc. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32503: [WIP] better error message
SparkQA commented on pull request #32503: URL: https://github.com/apache/spark/pull/32503#issuecomment-839023948 **[Test build #138382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138382/testReport)** for PR 32503 at commit [`1a6832e`](https://github.com/apache/spark/commit/1a6832e15302b47f34d9d00b172bf5bb0d18032c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32504: [SPARK-35013][CORE] Don't allow to set spark.driver.cores=0
dongjoon-hyun commented on pull request #32504: URL: https://github.com/apache/spark/pull/32504#issuecomment-839021693 Apache Spark depends on your GitHub Repo's GitHub Action jobs. - https://spark.apache.org/developer-tools.html Could you update your master branch and your PR branch once more and enable all GitHub Actions jobs in your repo like the following? - https://github.com/apache/spark/actions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on pull request #32505: [SPARK-35373][BUILD] Check Maven artifact checksum in build/mvn
srowen commented on pull request #32505: URL: https://github.com/apache/spark/pull/32505#issuecomment-839019681 That much I'm not worried about, as this only concerns Maven distributions, and any version we used would be long since published and mirrored, nothing released days ago. (Yes we already face the problem of offline mirrors, but fallback to archive.apache.org anyway) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32292: [WIP][SPARK-35162][SQL] New SQL functions: TRY_ADD/TRY_DIVIDE
SparkQA commented on pull request #32292: URL: https://github.com/apache/spark/pull/32292#issuecomment-839015904 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32499: [SPARK-29145][SQL][FOLLOWUP] Support sub-queries in join conditions
SparkQA removed a comment on pull request #32499: URL: https://github.com/apache/spark/pull/32499#issuecomment-838571340 **[Test build #138381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138381/testReport)** for PR 32499 at commit [`d9cad03`](https://github.com/apache/spark/commit/d9cad035457298ed65a96a0a208747d975a63ef7). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32499: [SPARK-29145][SQL][FOLLOWUP] Support sub-queries in join conditions
SparkQA commented on pull request #32499: URL: https://github.com/apache/spark/pull/32499#issuecomment-839004916 **[Test build #138381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138381/testReport)** for PR 32499 at commit [`d9cad03`](https://github.com/apache/spark/commit/d9cad035457298ed65a96a0a208747d975a63ef7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32292: [WIP][SPARK-35162][SQL] New SQL functions: TRY_ADD/TRY_DIVIDE
AmplabJenkins removed a comment on pull request #32292: URL: https://github.com/apache/spark/pull/32292#issuecomment-838973502 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42914/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32292: [WIP][SPARK-35162][SQL] New SQL functions: TRY_ADD/TRY_DIVIDE
AmplabJenkins commented on pull request #32292: URL: https://github.com/apache/spark/pull/32292#issuecomment-838973502 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42914/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32292: [WIP][SPARK-35162][SQL] New SQL functions: TRY_ADD/TRY_DIVIDE
SparkQA commented on pull request #32292: URL: https://github.com/apache/spark/pull/32292#issuecomment-838973377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32436: [SPARK-34764][CORE][K8S][UI] Propagate reason for exec loss to Web UI
AmplabJenkins removed a comment on pull request #32436: URL: https://github.com/apache/spark/pull/32436#issuecomment-838966502 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42913/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32436: [SPARK-34764][CORE][K8S][UI] Propagate reason for exec loss to Web UI
SparkQA commented on pull request #32436: URL: https://github.com/apache/spark/pull/32436#issuecomment-838966445 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42913/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32436: [SPARK-34764][CORE][K8S][UI] Propagate reason for exec loss to Web UI
AmplabJenkins commented on pull request #32436: URL: https://github.com/apache/spark/pull/32436#issuecomment-838966502 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42913/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32178: [DONOTMERGE] initial commit for skeleton ansible for jenkins worker config
SparkQA commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-838965272 **[Test build #138395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138395/testReport)** for PR 32178 at commit [`89b36c9`](https://github.com/apache/spark/commit/89b36c99462838d8dd9bde1ce4079a5ac3aa142b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32507: [SPARK-35361][SQL] Improve performance for ApplyFunctionExpression
SparkQA commented on pull request #32507: URL: https://github.com/apache/spark/pull/32507#issuecomment-838964138 **[Test build #138394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138394/testReport)** for PR 32507 at commit [`900f34c`](https://github.com/apache/spark/commit/900f34c8f9faf98b3aaf47bd21fe2cbad2a2f659). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sunchao opened a new pull request #32507: [SPARK-35361][SQL] Improve performance for ApplyFunctionExpression
sunchao opened a new pull request #32507: URL: https://github.com/apache/spark/pull/32507 ### What changes were proposed in this pull request? In `ApplyFunctionExpression`, move `zipWithIndex` out of the loop for each input row. ### Why are the changes needed? When the `ScalarFunction` is trivial, `zipWithIndex` could incur significant costs, as shown below: https://user-images.githubusercontent.com/506679/117866421-fb19de80-b24b-11eb-8c94-d5e8c8b1eda9.png;> By removing it out of the loop, I'm seeing sometimes 2X improvement from `V2FunctionBenchmark`. For instance: Before: ``` native_long_add 32437 32896 434 15.4 64.9 1.0X java_long_add_default 85675 97045 NaN 5.8 171.3 0.4X ``` After: ``` native_long_add 30182 30387 279 16.6 60.4 1.0X java_long_add_default 42862 43009 209 11.7 85.7 0.7X ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32178: [DONOTMERGE] initial commit for skeleton ansible for jenkins worker config
SparkQA commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-838959391 **[Test build #138393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138393/testReport)** for PR 32178 at commit [`2c3ebfd`](https://github.com/apache/spark/commit/2c3ebfd459db204772ab0841481f173238a4b7b1). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #32469: [SPARK-35338][PYTHON] Separate arithmetic operations into data type based structures
xinrong-databricks commented on pull request #32469: URL: https://github.com/apache/spark/pull/32469#issuecomment-838957172 CC @ueshin @HyukjinKwon @itholic -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32436: [SPARK-34764][CORE][K8S][UI] Propagate reason for exec loss to Web UI
SparkQA commented on pull request #32436: URL: https://github.com/apache/spark/pull/32436#issuecomment-838956958 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42913/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on pull request #32178: [DONOTMERGE] initial commit for skeleton ansible for jenkins worker config
shaneknapp commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-838952376 > > @shaneknapp Hi, shane, do you have any plan to append Arm node [1] setup related configuration on your patch? Thanks > > [1] https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ > > no, all of that setup was manual :\ ...and all i did IIRC was set up the python venvs. let's revisit this after this pr is merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on pull request #32178: [DONOTMERGE] initial commit for skeleton ansible for jenkins worker config
shaneknapp commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-838950631 > @shaneknapp Hi, shane, do you have any plan to append Arm node [1] setup related configuration on your patch? Thanks > > [1] https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ no, all of that setup was manual :\ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32292: [WIP][SPARK-35162][SQL] New SQL functions: TRY_ADD/TRY_DIVIDE
SparkQA commented on pull request #32292: URL: https://github.com/apache/spark/pull/32292#issuecomment-838947759 **[Test build #138392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138392/testReport)** for PR 32292 at commit [`75dee4a`](https://github.com/apache/spark/commit/75dee4ab6eb56a43ed2ad73f222f299ebd6a890e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
AmplabJenkins removed a comment on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-838944933 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42911/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32452: [SPARK-35243][SQL] Support columnar execution on ANSI interval types
AmplabJenkins removed a comment on pull request #32452: URL: https://github.com/apache/spark/pull/32452#issuecomment-838944934 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138378/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32504: [Minor][SPARK-35013]Don't allow to set spark.driver.cores=0
AmplabJenkins removed a comment on pull request #32504: URL: https://github.com/apache/spark/pull/32504#issuecomment-838944936 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138383/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32469: [WIP][SPARK-35338][PYTHON] Separate arithmetic operations into data type based structures
AmplabJenkins removed a comment on pull request #32469: URL: https://github.com/apache/spark/pull/32469#issuecomment-838944932 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42912/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32452: [SPARK-35243][SQL] Support columnar execution on ANSI interval types
AmplabJenkins commented on pull request #32452: URL: https://github.com/apache/spark/pull/32452#issuecomment-838944934 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138378/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32504: [Minor][SPARK-35013]Don't allow to set spark.driver.cores=0
AmplabJenkins commented on pull request #32504: URL: https://github.com/apache/spark/pull/32504#issuecomment-838944936 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138383/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32469: [WIP][SPARK-35338][PYTHON] Separate arithmetic operations into data type based structures
AmplabJenkins commented on pull request #32469: URL: https://github.com/apache/spark/pull/32469#issuecomment-838944932 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42912/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
AmplabJenkins commented on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-838944933 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42911/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32298: [WIP][SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r630412347
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
> I would pursue (1) in this PR first and maybe (2) in a separate one.
Does this sound acceptable?
Yeah, that sounds great. Thanks a lot, @peter-toth!
> There are 2 aggregates in both subqueries so without dedup both (2) and
this PR could cause regressions.
IIUC, I think it sounds like an existing bug (or missing feature) for struct
subfield pruning, which could be blocking (2) but is orthogonal to (2). For
instance, if I write your example join query manually, I'd expect the struct
subfield pruning to happen to the struct constructor, regardless of the
existence of subqueries.
> I've never seen such transformations in SparkStrategys.
It's not uncommon in exploration Strategies such as index selection, common
subplan dedup etc., when we substitute the subtree of a tree node T with
another subtree (from somewhere else in the plan or a different access path)
that may contain unneeded columns for T. Spark doesn't have those strategies
for now, but I'll not be surprised if some constributors add them down the road.
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *
[GitHub] [spark] sigmod commented on a change in pull request #32298: [WIP][SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r630412347
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
> I would pursue (1) in this PR first and maybe (2) in a separate one.
Does this sound acceptable?
Yeah, that sounds great. Thanks a lot, @peter-toth!
> There are 2 aggregates in both subqueries so without dedup both (2) and
this PR could cause regressions.
IIUC, I think it sounds like an existing bug (or missing feature) for struct
subfield pruning, which could be blocking (2) but is orthogonal to (2). For
instance, if I write your example join query manually, I'd expect the struct
subfield pruning to happen to the struct constructor, regardless of the
existence of subqueries.
> I've never seen such transformations in SparkStrategys.
It's not uncommon in exploration Strategies such as index selection, common
subplan dedup etc., when we substitute the subtree of a tree node T with
another subtree (from somewhere else in the plan or a different access path)
that may contain unneeded columns for T. Spark doesn't have those strategies
for now, but I'll not be surprised if we add them down the road.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32298: [WIP][SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r630412347
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
> I would pursue (1) in this PR first and maybe (2) in a separate one.
Does this sound acceptable?
Yeah, that sounds great. Thanks a lot, @peter-toth!
> There are 2 aggregates in both subqueries so without dedup both (2) and
this PR could cause regressions.
IIUC, I think it sounds like an existing bug (or missing feature) for struct
subfield pruning, which could be blocking (2) but is orthogonal to (2). For
instance, if I write your example join query manually, I'd expect the struct
subfield pruning to happen to the struct constructor, regardless of the
existence of subqueries.
> I've never seen such transformations in SparkStrategys.
It's not uncommon in exploration Strategies such as index selection, common
subplan dedup, when we substitute the subtree of a tree node T with another
subtree (from somewhere else in the plan or a different access path) that may
contain unneeded columns for T. Spark doesn't have those strategies for now,
but I'll not be surprised if we add them down the road.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32298: [WIP][SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r630412347
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
> I would pursue (1) in this PR first and maybe (2) in a separate one.
Does this sound acceptable?
Yeah, that sounds great. Thanks a lot, @peter-toth!
> There are 2 aggregates in both subqueries so without dedup both (2) and
this PR could cause regressions.
IIUC, I think it sounds like an existing bug (or missing feature) for struct
subfield pruning, which could be blocking (2) but is orthogonal to (2). For
instance, if I write your example join query manually, I'd expect the struct
subfield pruning to happen to the struct constructor, regardless of the
existence of subqueries.
> I've never seen such transformations in SparkStrategys.
It's not uncommon in exploration Strategies such as index selection, common
subplan dedup, when we substitute the subtree of a tree node T with another
subtree (from somewhere else in the plan or a different access path) that may
contain unneeded columns for T.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
SparkQA commented on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-838932173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32298: [WIP][SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r630412347
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
> I would pursue (1) in this PR first and maybe (2) in a separate one.
Does this sound acceptable?
Yeah, that sounds great. Thanks a lot, @peter-toth!
> There are 2 aggregates in both subqueries so without dedup both (2) and
this PR could cause regressions.
IIUC, I think it sounds like an existing bug for struct subfield pruning,
which could be blocking (2) but is orthogonal to (2). For instance, if I write
your example join query manually, I'd expect the struct subfield pruning to
happen to the struct constructor, regardless of the existence of subqueries.
> I've never seen such transformations in SparkStrategys.
It's not uncommon in exploration Strategies such as index selection, common
subplan dedup, when we substitute the subtree of a tree node T with another
subtree (from somewhere else in the plan or a different access path) that may
contain unneeded columns for T.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32469: [WIP][SPARK-35338][PYTHON] Separate arithmetic operations into data type based structures
SparkQA commented on pull request #32469: URL: https://github.com/apache/spark/pull/32469#issuecomment-838920549 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42912/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32502: [SPARK-35372][BUILD] Increase stack size for Scala compilation in Maven build
dongjoon-hyun commented on pull request #32502: URL: https://github.com/apache/spark/pull/32502#issuecomment-838915154 +1, LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32504: [Minor][SPARK-35013]Don't allow to set spark.driver.cores=0
SparkQA removed a comment on pull request #32504: URL: https://github.com/apache/spark/pull/32504#issuecomment-838669907 **[Test build #138383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138383/testReport)** for PR 32504 at commit [`b615db6`](https://github.com/apache/spark/commit/b615db6224944986cfe4d29bcb1e87b3d296add3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32504: [Minor][SPARK-35013]Don't allow to set spark.driver.cores=0
SparkQA commented on pull request #32504: URL: https://github.com/apache/spark/pull/32504#issuecomment-838902238 **[Test build #138383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138383/testReport)** for PR 32504 at commit [`b615db6`](https://github.com/apache/spark/commit/b615db6224944986cfe4d29bcb1e87b3d296add3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32452: [SPARK-35243][SQL] Support columnar execution on ANSI interval types
SparkQA removed a comment on pull request #32452: URL: https://github.com/apache/spark/pull/32452#issuecomment-838464756 **[Test build #138378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138378/testReport)** for PR 32452 at commit [`d911e8a`](https://github.com/apache/spark/commit/d911e8a732572b1a86ee044825cf17b0e46563dd). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32452: [SPARK-35243][SQL] Support columnar execution on ANSI interval types
SparkQA commented on pull request #32452: URL: https://github.com/apache/spark/pull/32452#issuecomment-83319 **[Test build #138378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138378/testReport)** for PR 32452 at commit [`d911e8a`](https://github.com/apache/spark/commit/d911e8a732572b1a86ee044825cf17b0e46563dd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32292: [WIP][SPARK-35162][SQL] New SQL functions: TRY_ADD/TRY_DIVIDE
SparkQA commented on pull request #32292: URL: https://github.com/apache/spark/pull/32292#issuecomment-838883496 **[Test build #138391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138391/testReport)** for PR 32292 at commit [`a1d5742`](https://github.com/apache/spark/commit/a1d5742301bcd0ee389241cb93d11ff8d344258e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32411: [SPARK-28551][SQL]CTAS with LOCATION , should not allow to a non-empty directory.
AmplabJenkins removed a comment on pull request #32411: URL: https://github.com/apache/spark/pull/32411#issuecomment-838856363 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42908/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32436: [SPARK-34764][CORE][K8S][UI] Propagate reason for exec loss to Web UI
AmplabJenkins removed a comment on pull request #32436: URL: https://github.com/apache/spark/pull/32436#issuecomment-832864910 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42699/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32389: [SPARK-35263] [TEST] Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
AmplabJenkins removed a comment on pull request #32389: URL: https://github.com/apache/spark/pull/32389#issuecomment-838856369 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42909/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32505: [SPARK-35373][BUILD] Check Maven artifact checksum in build/mvn
AmplabJenkins removed a comment on pull request #32505: URL: https://github.com/apache/spark/pull/32505#issuecomment-838856364 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42907/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32498: [SPARK-35368][SQL] Update histogram statistics for RANGE operator for stats estimation
AmplabJenkins removed a comment on pull request #32498: URL: https://github.com/apache/spark/pull/32498#issuecomment-838856365 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42910/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32397: [SPARK-35084][CORE] Spark 3: supporting "--packages" in k8s cluster mode
AmplabJenkins removed a comment on pull request #32397: URL: https://github.com/apache/spark/pull/32397#issuecomment-838856366 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138380/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32436: [SPARK-34764][CORE][K8S][UI] Propagate reason for exec loss to Web UI
SparkQA commented on pull request #32436: URL: https://github.com/apache/spark/pull/32436#issuecomment-838867786 **[Test build #138390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138390/testReport)** for PR 32436 at commit [`a4263cd`](https://github.com/apache/spark/commit/a4263cd86c8fa8a87ab05435eca59415d4a6b6f6). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak closed pull request #32502: [SPARK-35372][BUILD] Increase stack size for Scala compilation in Maven build
sarutak closed pull request #32502: URL: https://github.com/apache/spark/pull/32502 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32469: [WIP][SPARK-35338][PYTHON] Separate arithmetic operations into data type based structures
SparkQA commented on pull request #32469: URL: https://github.com/apache/spark/pull/32469#issuecomment-838861089 **[Test build #138389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138389/testReport)** for PR 32469 at commit [`a4a6581`](https://github.com/apache/spark/commit/a4a65811791745fa0a473f18253f72662f4e24eb). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
SparkQA commented on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-838860395 **[Test build #138388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138388/testReport)** for PR 32506 at commit [`45a6833`](https://github.com/apache/spark/commit/45a68335201ff7fbc69ec01ca0f2769fa058278d). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #32502: [SPARK-35372][BUILD] Increase stack size for Scala compilation in Maven build
sarutak commented on pull request #32502: URL: https://github.com/apache/spark/pull/32502#issuecomment-838857505 I think the Jenkins failure is not related to this change. Merging to `master` to recover build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32397: [SPARK-35084][CORE] Spark 3: supporting "--packages" in k8s cluster mode
AmplabJenkins commented on pull request #32397: URL: https://github.com/apache/spark/pull/32397#issuecomment-838856366 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138380/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32411: [SPARK-28551][SQL]CTAS with LOCATION , should not allow to a non-empty directory.
AmplabJenkins commented on pull request #32411: URL: https://github.com/apache/spark/pull/32411#issuecomment-838856363 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42908/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32505: [SPARK-35373][BUILD] Check Maven artifact checksum in build/mvn
AmplabJenkins commented on pull request #32505: URL: https://github.com/apache/spark/pull/32505#issuecomment-838856364 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42907/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32389: [SPARK-35263] [TEST] Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
AmplabJenkins commented on pull request #32389: URL: https://github.com/apache/spark/pull/32389#issuecomment-838856369 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42909/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32498: [SPARK-35368][SQL] Update histogram statistics for RANGE operator for stats estimation
AmplabJenkins commented on pull request #32498: URL: https://github.com/apache/spark/pull/32498#issuecomment-838856365 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42910/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32498: [SPARK-35368][SQL] Update histogram statistics for RANGE operator for stats estimation
SparkQA commented on pull request #32498: URL: https://github.com/apache/spark/pull/32498#issuecomment-838852137 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32389: [SPARK-35263] [TEST] Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
SparkQA commented on pull request #32389: URL: https://github.com/apache/spark/pull/32389#issuecomment-838850182 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42909/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32411: [SPARK-28551][SQL]CTAS with LOCATION , should not allow to a non-empty directory.
SparkQA commented on pull request #32411: URL: https://github.com/apache/spark/pull/32411#issuecomment-838844413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32505: [SPARK-35373][BUILD] Check Maven artifact checksum in build/mvn
SparkQA commented on pull request #32505: URL: https://github.com/apache/spark/pull/32505#issuecomment-838842848 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32389: [SPARK-35263] [TEST] Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
SparkQA commented on pull request #32389: URL: https://github.com/apache/spark/pull/32389#issuecomment-838842568 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42909/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
