[GitHub] [spark] dongjoon-hyun commented on pull request #33612: [SPARK-36383][CORE] Avoid NullPointerException during executor shutdown
dongjoon-hyun commented on pull request #33612: URL: https://github.com/apache/spark/pull/33612#issuecomment-891587669 Merged to master. Thank you, @Ngone51 and @attilapiros . BTW, @Ngone51 . I didn't backport this because SPARK-36383 is created as `Improvement` JIRA. If you need this in release branches, please switch to `Bug`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #33612: [SPARK-36383][CORE] Avoid NullPointerException during executor shutdown
dongjoon-hyun closed pull request #33612: URL: https://github.com/apache/spark/pull/33612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
AmplabJenkins removed a comment on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891584488 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46493/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #33556: [SPARK-36324][CORE] Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
LuciferYang edited a comment on pull request #33556: URL: https://github.com/apache/spark/pull/33556#issuecomment-891569743 @mridulm thanks for your explain I test `DeflaterOutputStream`, there will be 8 bytes([120,-100,3,0,0,0,0,1]) at the end of the file even if we don't write anything and I think I basically understand what you mean. However, I have another question：If this scenario is what we need to consider, should we need call `revertPartialWritesAndClose ` when ` objectsWritten > 0` also? otherwise the meta data will also be written to the file ... Maybe a strange result will appear here: 1. No writes at all(`call revertPartialWritesAndClose now`): empty file 2. objectsWritten == 0(`call revertPartialWritesAndClose now`)：only data in the file 3. objectsWritten > 0(`call flush()+close() now`): both data and meta in the file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
AmplabJenkins commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891584488 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46493/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891584455 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46493/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #33556: [SPARK-36324][CORE] Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
LuciferYang edited a comment on pull request #33556: URL: https://github.com/apache/spark/pull/33556#issuecomment-891569743 @mridulm thanks for your explain I test `DeflaterOutputStream`, there will be 8 bytes([120,-100,3,0,0,0,0,1]) at the end of the file even if we don't write anything and I think I basically understand what you mean. However, I have another question：If this scenario is what we need to consider, should we need call `revertPartialWritesAndClose ` when ` objectsWritten > 0` also? otherwise the meta data will also be written to the file ... Otherwise, a strange result will appear here: 1. No writes at all(`call revertPartialWritesAndClose now`): empty file 2. objectsWritten == 0(`call revertPartialWritesAndClose now`)：only data in the file 3. objectsWritten > 0(`call flush()+close() now`): both data and meta in the file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #33556: [SPARK-36324][CORE] Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
LuciferYang edited a comment on pull request #33556: URL: https://github.com/apache/spark/pull/33556#issuecomment-891569743 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] senthh commented on pull request #33577: [SPARK-36327][SQL] Spark sql creates staging dir inside database directory rather than creating inside table directory
senthh commented on pull request #33577: URL: https://github.com/apache/spark/pull/33577#issuecomment-891581907 @dongjoon-hyun , we tested in our internal cluster also with Customer's cluster manually. As this issue occur only when viewfs is configured in cluster, I think test coverage that meets viewfs will not be possible. But we can test this scenario manually if we have configured viewFs in our cluster. And I have added manual test results in the previous post. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33590: [SPARK-36359][SQL] Coalesce drop all expressions after the first non nullable expression
AmplabJenkins removed a comment on pull request #33590: URL: https://github.com/apache/spark/pull/33590#issuecomment-891581613 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141969/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33509: [SPARK-36280][SQL] Remove redundant aliases after RewritePredicateSubquery
AmplabJenkins removed a comment on pull request #33509: URL: https://github.com/apache/spark/pull/33509#issuecomment-891581610 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46491/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
AmplabJenkins removed a comment on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891581609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
AmplabJenkins removed a comment on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891581614 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141977/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
AmplabJenkins commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891581611 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33509: [SPARK-36280][SQL] Remove redundant aliases after RewritePredicateSubquery
AmplabJenkins commented on pull request #33509: URL: https://github.com/apache/spark/pull/33509#issuecomment-891581610 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46491/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33590: [SPARK-36359][SQL] Coalesce drop all expressions after the first non nullable expression
AmplabJenkins commented on pull request #33590: URL: https://github.com/apache/spark/pull/33590#issuecomment-891581613 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141969/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
AmplabJenkins commented on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891581614 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141977/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA removed a comment on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891522923 **[Test build #141979 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141979/testReport)** for PR 33583 at commit [`bad679f`](https://github.com/apache/spark/commit/bad679fa7b9b90073f1cf2a9ef0384d8a82118cf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891576855 **[Test build #141979 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141979/testReport)** for PR 33583 at commit [`bad679f`](https://github.com/apache/spark/commit/bad679fa7b9b90073f1cf2a9ef0384d8a82118cf). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
dongjoon-hyun commented on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891575643 Please let me know if we need this in `branch-3.2`. cc @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
dongjoon-hyun commented on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891575005 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
dongjoon-hyun closed pull request #33616: URL: https://github.com/apache/spark/pull/33616 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33590: [SPARK-36359][SQL] Coalesce drop all expressions after the first non nullable expression
SparkQA removed a comment on pull request #33590: URL: https://github.com/apache/spark/pull/33590#issuecomment-891447174 **[Test build #141969 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141969/testReport)** for PR 33590 at commit [`641fa9e`](https://github.com/apache/spark/commit/641fa9eb4cb173f49efbeeae59e2f1c9f72f0a1e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA removed a comment on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891529512 **[Test build #141982 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141982/testReport)** for PR 33583 at commit [`b57e120`](https://github.com/apache/spark/commit/b57e120e65af4ac157eaf911eb8faada0925b64b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #33556: [SPARK-36324][CORE] Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
LuciferYang edited a comment on pull request #33556: URL: https://github.com/apache/spark/pull/33556#issuecomment-891569743 @mridulm thanks for your explain I test `DeflaterOutputStream`, there will be 8 bytes([120,-100,3,0,0,0,0,1]) at the end of the file even if we don't write anything and I think I basically understand what you mean. However, I have another question：If this scenario is what we need to consider, should we need call `revertPartialWritesAndClose ` when ` objectsWritten > 0` also? otherwise the meta data will also be written to the file ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891571293 **[Test build #141982 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141982/testReport)** for PR 33583 at commit [`b57e120`](https://github.com/apache/spark/commit/b57e120e65af4ac157eaf911eb8faada0925b64b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #33556: [SPARK-36324][CORE] Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
LuciferYang edited a comment on pull request #33556: URL: https://github.com/apache/spark/pull/33556#issuecomment-891569743 @mridulm thanks for your explain I test `DeflaterOutputStream`, there will be 8 bytes([120,-100,3,0,0,0,0,1]) at the end of the file even if we don't write anything and I think I basically understand what you mean. However, I have another question：If this scenario is what we need to consider, should we need call `revertPartialWritesAndClose ` when ` objectsWritten > 0` also? otherwise these 8 bytes will also be written to the file ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #33556: [SPARK-36324][CORE] Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
LuciferYang edited a comment on pull request #33556: URL: https://github.com/apache/spark/pull/33556#issuecomment-891569743 @mridulm thx for your explain I test `DeflaterOutputStream`, there will be 8 bytes([120,-100,3,0,0,0,0,1]) at the end of the file even if we don't write anything and I think I basically understand what you mean. However, I have another question：If this scenario is what we need to consider, should we need call `revertPartialWritesAndClose ` when ` objectsWritten > 0` also? otherwise these 8 bytes will also be written to the file ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33590: [SPARK-36359][SQL] Coalesce drop all expressions after the first non nullable expression
SparkQA commented on pull request #33590: URL: https://github.com/apache/spark/pull/33590#issuecomment-891570088 **[Test build #141969 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141969/testReport)** for PR 33590 at commit [`641fa9e`](https://github.com/apache/spark/commit/641fa9eb4cb173f49efbeeae59e2f1c9f72f0a1e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #33556: [SPARK-36324][CORE] Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
LuciferYang commented on pull request #33556: URL: https://github.com/apache/spark/pull/33556#issuecomment-891569743 @mridulm thx for your explain I test `DeflaterOutputStream`, there will be 8 bytes([120,-100,3,0,0,0,0,1]) here even if we don't write anything and I think I basically understand what you mean. However, I have another question：If this scenario is what we need to consider, should we need call `revertPartialWritesAndClose ` when ` objectsWritten > 0` also? otherwise these 8 bytes will also be written to the file ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #33556: [SPARK-36324][CORE] Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
Ngone51 commented on pull request #33556:
URL: https://github.com/apache/spark/pull/33556#issuecomment-891567046
>I mean closeResources() will set initialized as false and initialized is
always false in finally block, so will "just avoid file >operations when
initialized=false in revertPartialWritesAndClose" become "always not do these
file operations" ?
>
>Utils.tryWithSafeFinally {
if (initialized) {
writeMetrics.decBytesWritten(reportedPosition - committedPosition)
writeMetrics.decRecordsWritten(numRecordsWritten)
streamOpen = false
closeResources() // `closeResources()` will set `initialized` as
false
}
} {
// `initialized` is always false in this block
// do truncate file operations.
}
You're right...I was thinking about the case where there's no data at all..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33509: [SPARK-36280][SQL] Remove redundant aliases after RewritePredicateSubquery
SparkQA commented on pull request #33509: URL: https://github.com/apache/spark/pull/33509#issuecomment-891564184 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46491/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #33618: [SPARK-36381][SQL] Add case sensitive and case insensitive compare for checking column name exist when alter table
Peng-Lei commented on pull request #33618: URL: https://github.com/apache/spark/pull/33618#issuecomment-891563360 > I think it's good to have this check so that it's consistent with `SchemaUtils.checkColumnNameDuplication` behavior. Thank you very much -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
SparkQA removed a comment on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891499166 **[Test build #141977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141977/testReport)** for PR 33616 at commit [`9ddae21`](https://github.com/apache/spark/commit/9ddae214fa68826d15519d67e713a61aae72919e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891563202 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46490/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
SparkQA commented on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891562941 **[Test build #141977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141977/testReport)** for PR 33616 at commit [`9ddae21`](https://github.com/apache/spark/commit/9ddae214fa68826d15519d67e713a61aae72919e). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #33556: [SPARK-36324][CORE] Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
Ngone51 commented on pull request #33556: URL: https://github.com/apache/spark/pull/33556#issuecomment-891562262 > In these, it is possible for (meta-) data to be written out during close even if objectsWritten == 0. @mridulm In that case, shouldn't we add another flag to indicate if there's any metadata written? `objectsWritten` definitely wouldn't work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] eejbyfeldt commented on pull request #33205: [SPARK-20384][SQL] Support value class in nested schema for Dataset
eejbyfeldt commented on pull request #33205: URL: https://github.com/apache/spark/pull/33205#issuecomment-891558552 > Is there any behavior change you can think of that might affect users? Hi Sean, thanks for having a look! This only changes is for case class containing value class. e.g ``` case class IntWrapper(value: Int) extends AnyVal case class DatasetModel(wrappedInt: IntWrapper) ``` Before this patch trying to create a `Dataset` using the `DatasetModel` would result in rumtime error like: ``` 21/08/03 07:50:01 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 85, Column 1: Assignment conversion not possible from type "int" to type "example.IntWrapper" org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 85, Column 1: Assignment conversion not possible from type "int" to type "example.IntWrapper" at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12021) at org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10851) ... ``` But with this patch it will work like expected. Unless someone explicitly depend on having this failure I don't think there should be any behavior change that is noticeable for users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891555119 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46493/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhouyejoe commented on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
zhouyejoe commented on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891554593 +1. LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhuqi-lucas commented on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
zhuqi-lucas commented on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891552583 +1, LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
AmplabJenkins removed a comment on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891550216 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141978/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA removed a comment on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891501800 **[Test build #141978 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141978/testReport)** for PR 33583 at commit [`9cc00c0`](https://github.com/apache/spark/commit/9cc00c067de00b89fa348a857d7c18eee7de7489). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
AmplabJenkins commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891550216 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141978/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891550056 **[Test build #141978 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141978/testReport)** for PR 33583 at commit [`9cc00c0`](https://github.com/apache/spark/commit/9cc00c067de00b89fa348a857d7c18eee7de7489). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ProjectionOverSchema(schema: StructType, attNameMap: Map[String, String]) ` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
AmplabJenkins removed a comment on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891549710 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46489/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32355: [SPARK-35221][SQL] Add join hint build side check
AmplabJenkins removed a comment on pull request #32355: URL: https://github.com/apache/spark/pull/32355#issuecomment-891549708 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46492/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
AmplabJenkins removed a comment on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891549706 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46488/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32355: [SPARK-35221][SQL] Add join hint build side check
AmplabJenkins commented on pull request #32355: URL: https://github.com/apache/spark/pull/32355#issuecomment-891549708 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46492/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
AmplabJenkins commented on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891549706 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46488/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
AmplabJenkins commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891549710 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46489/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
itholic commented on a change in pull request #33581:
URL: https://github.com/apache/spark/pull/33581#discussion_r681451616
##
File path: python/pyspark/pandas/data_type_ops/base.py
##
@@ -314,9 +320,11 @@ def __or__(self, left: IndexOpsLike, right: Any) ->
SeriesOrIndex:
raise TypeError("Bitwise or can not be applied to %s." %
self.pretty_name)
def rand(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
Review comment:
Oh, yeah I got it. Looks good to me as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhuqi-lucas edited a comment on pull request #33617: [SPARK-35548][CORE][SHUFFLE] Handling new attempt has started error m…
zhuqi-lucas edited a comment on pull request #33617: URL: https://github.com/apache/spark/pull/33617#issuecomment-891496800 cc @Ngone51 @zhouyejoe @mridulm @dongjoon-hyun @HyukjinKwon Could you help review this, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #32964: [SPARK-35811][PYTHON] Deprecate DataFrame.to_spark_io
itholic commented on a change in pull request #32964:
URL: https://github.com/apache/spark/pull/32964#discussion_r681449016
##
File path: python/pyspark/pandas/frame.py
##
@@ -4815,6 +4815,13 @@ def to_spark_io(
index_col: Optional[Union[str, List[str]]] = None,
**options
) -> None:
+"""An alias for :func:`DataFrame.spark.to_spark_io`.
+See
:meth:`pyspark.pandas.spark.accessors.SparkFrameMethods.to_spark_io`.
+
+.. deprecated:: 3.2.0
+Use :func:`DataFrame.spark.to_spark_io` instead.
+"""
+warnings.warn("Deprecated in 3.2, Use spark.to_spark_io instead.",
FutureWarning)
Review comment:
Sounds good! I'll address it, thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33509: [SPARK-36280][SQL] Remove redundant aliases after RewritePredicateSubquery
SparkQA commented on pull request #33509: URL: https://github.com/apache/spark/pull/33509#issuecomment-891545086 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46491/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891544065 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46490/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32355: [SPARK-35221][SQL] Add join hint build side check
SparkQA commented on pull request #32355: URL: https://github.com/apache/spark/pull/32355#issuecomment-891543486 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46492/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891537889 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46489/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #33618: [SPARK-36381][SQL] Add case sensitive and case insensitive compare for checking column name exist when alter table
imback82 commented on a change in pull request #33618:
URL: https://github.com/apache/spark/pull/33618#discussion_r681440489
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala
##
@@ -407,6 +408,65 @@ trait AlterTableTests extends SharedSparkSession {
}
}
+ test("SPARK-36381: Alter add column exist check in case sensitive") {
Review comment:
maybe move this to
https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/connector/V2CommandsCaseSensitivitySuite.scala?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
SparkQA commented on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891532825 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46488/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891529512 **[Test build #141982 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141982/testReport)** for PR 33583 at commit [`b57e120`](https://github.com/apache/spark/commit/b57e120e65af4ac157eaf911eb8faada0925b64b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33614: [SPARK-36367][3.2][PYTHON] Partially backport to avoid unexpected error with pandas 1.3
HyukjinKwon closed pull request #33614: URL: https://github.com/apache/spark/pull/33614 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33614: [SPARK-36367][3.2][PYTHON] Partially backport to avoid unexpected error with pandas 1.3
HyukjinKwon edited a comment on pull request #33614: URL: https://github.com/apache/spark/pull/33614#issuecomment-891528194 I think it's fine ... pandas on Spark will work 99.9% fine with pandas 1.3+ ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33614: [SPARK-36367][3.2][PYTHON] Partially backport to avoid unexpected error with pandas 1.3
HyukjinKwon commented on pull request #33614: URL: https://github.com/apache/spark/pull/33614#issuecomment-891528315 let me merge this in first anyway since RC will likely be cut out soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33614: [SPARK-36367][3.2][PYTHON] Partially backport to avoid unexpected error with pandas 1.3
HyukjinKwon commented on pull request #33614: URL: https://github.com/apache/spark/pull/33614#issuecomment-891528194 I think it's fine ... pandas on Spark will work 99% fine with pandas 1.3+ ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33598: [SPARK-36345][SPARK-36367][INFRA][PYTHON] Disable tests failed by the incompatible behavior of pandas 1.3
HyukjinKwon closed pull request #33598: URL: https://github.com/apache/spark/pull/33598 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33598: [SPARK-36345][SPARK-36367][INFRA][PYTHON] Disable tests failed by the incompatible behavior of pandas 1.3
HyukjinKwon commented on pull request #33598: URL: https://github.com/apache/spark/pull/33598#issuecomment-891527673 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33560: [SPARK-36331][CORE] Add standard SQLSTATEs to error guidelines
HyukjinKwon commented on pull request #33560: URL: https://github.com/apache/spark/pull/33560#issuecomment-891526878 Merged to master and branch-3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33560: [SPARK-36331][CORE] Add standard SQLSTATEs to error guidelines
HyukjinKwon closed pull request #33560: URL: https://github.com/apache/spark/pull/33560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32355: [SPARK-35221][SQL] Add join hint build side check
SparkQA commented on pull request #32355: URL: https://github.com/apache/spark/pull/32355#issuecomment-891523414 **[Test build #141981 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141981/testReport)** for PR 32355 at commit [`11c5700`](https://github.com/apache/spark/commit/11c57008d2b61ac3ea258ca4edc0777e9a8fb7cc). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33509: [SPARK-36280][SQL] Remove redundant aliases after RewritePredicateSubquery
SparkQA commented on pull request #33509: URL: https://github.com/apache/spark/pull/33509#issuecomment-891522999 **[Test build #141980 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141980/testReport)** for PR 33509 at commit [`3c89d19`](https://github.com/apache/spark/commit/3c89d1978ccb0231c154c1e4e32116132d68fa80). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891522923 **[Test build #141979 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141979/testReport)** for PR 33583 at commit [`bad679f`](https://github.com/apache/spark/commit/bad679fa7b9b90073f1cf2a9ef0384d8a82118cf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33615: [SPARK-36374][SHUFFLE][DOC] Push-based shuffle high level user documentation
AmplabJenkins removed a comment on pull request #33615: URL: https://github.com/apache/spark/pull/33615#issuecomment-891522011 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46486/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33615: [SPARK-36374][SHUFFLE][DOC] Push-based shuffle high level user documentation
AmplabJenkins commented on pull request #33615: URL: https://github.com/apache/spark/pull/33615#issuecomment-891522011 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46486/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33615: [SPARK-36374][SHUFFLE][DOC] Push-based shuffle high level user documentation
SparkQA commented on pull request #33615: URL: https://github.com/apache/spark/pull/33615#issuecomment-891521982 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46486/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33618: [SPARK-36381][SQL] Add case sensitive and case insensitive compare for checking column name exist when alter table
AmplabJenkins commented on pull request #33618: URL: https://github.com/apache/spark/pull/33618#issuecomment-891521779 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
AmplabJenkins removed a comment on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-891521259 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46483/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33605: [SPARK-32923][FOLLOW-UP] Clean up older shuffleMergeId shuffle files when finalize request for higher shuffleMergeId is receive
AmplabJenkins removed a comment on pull request #33605: URL: https://github.com/apache/spark/pull/33605#issuecomment-891521262 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46487/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33607: [SPARK-36175][SQL][FOLLOWUP] Improve the comments for AvroDeserializer/AvroSerializer
AmplabJenkins removed a comment on pull request #33607: URL: https://github.com/apache/spark/pull/33607#issuecomment-891521264 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46482/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
AmplabJenkins removed a comment on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891521258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
AmplabJenkins commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891521260 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33605: [SPARK-32923][FOLLOW-UP] Clean up older shuffleMergeId shuffle files when finalize request for higher shuffleMergeId is received
AmplabJenkins commented on pull request #33605: URL: https://github.com/apache/spark/pull/33605#issuecomment-891521262 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46487/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33607: [SPARK-36175][SQL][FOLLOWUP] Improve the comments for AvroDeserializer/AvroSerializer
AmplabJenkins commented on pull request #33607: URL: https://github.com/apache/spark/pull/33607#issuecomment-891521264 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46482/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
AmplabJenkins commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-891521259 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46483/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891520713 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46489/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #33509: [SPARK-36280][SQL] Remove redundant aliases after RewritePredicateSubquery
wangyum commented on a change in pull request #33509:
URL: https://github.com/apache/spark/pull/33509#discussion_r681429815
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
##
@@ -1876,4 +1877,29 @@ class SubquerySuite extends QueryTest with
SharedSparkSession with AdaptiveSpark
"ReusedSubqueryExec should reuse an existing subquery")
}
}
+
+ test("SPARK-36280: Remove redundant aliases after RewritePredicateSubquery")
{
+sql("CREATE TABLE t1 USING parquet AS SELECT id AS a, id AS b, id AS c
FROM range(10)")
Review comment:
Fixed it. Sorry. I don't know why I forgot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32355: [SPARK-35221][SQL] Add join hint build side check
ulysses-you commented on pull request #32355: URL: https://github.com/apache/spark/pull/32355#issuecomment-891519254 thank you @cloud-fan for review, added two methods in `JoinSelection`: * `checkHintBuildSide` is to check hint build side * `checkHintNonEquiJoin` is to check hint equi join And also added test for the equi join check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32355: [SPARK-35221][SQL] Add join hint build side check
ulysses-you commented on a change in pull request #32355:
URL: https://github.com/apache/spark/pull/32355#discussion_r681429032
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala
##
@@ -42,6 +45,17 @@ object HintErrorLogger extends HintErrorHandler with Logging
{
logWarning(s"A join hint $hint is specified but it is not part of a join
relation.")
}
+ override def joinBuildSideNotSupported(joinType: JoinType, joinHint:
JoinHint): Unit = {
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA removed a comment on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891474270 **[Test build #141974 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141974/testReport)** for PR 33583 at commit [`571c56a`](https://github.com/apache/spark/commit/571c56ad348ccfbec8aaab83e8978f2e005b62a5). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891518865 **[Test build #141974 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141974/testReport)** for PR 33583 at commit [`571c56a`](https://github.com/apache/spark/commit/571c56ad348ccfbec8aaab83e8978f2e005b62a5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33616: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
SparkQA commented on pull request #33616: URL: https://github.com/apache/spark/pull/33616#issuecomment-891517771 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46488/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33605: [SPARK-32923][FOLLOW-UP] Clean up older shuffleMergeId shuffle files when finalize request for higher shuffleMergeId is received
SparkQA commented on pull request #33605: URL: https://github.com/apache/spark/pull/33605#issuecomment-891517354 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46487/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33607: [SPARK-36175][SQL][FOLLOWUP] Improve the comments for AvroDeserializer/AvroSerializer
SparkQA commented on pull request #33607: URL: https://github.com/apache/spark/pull/33607#issuecomment-891515597 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46482/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
SparkQA commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-891514642 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46483/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891508250 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46485/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #33618: [SPARK-36381][SQL] Add case sensitive and case insensitive compare for checking column name exist when alter table
Peng-Lei commented on pull request #33618: URL: https://github.com/apache/spark/pull/33618#issuecomment-891507272 @imback82 @cloud-fan Could you take a look ? I'm not quite sure if this is needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Peng-Lei opened a new pull request #33618: [SPARK-36381][SQL] Add case sensitive and case insensitive compare for checking column name exist when alter table
Peng-Lei opened a new pull request #33618: URL: https://github.com/apache/spark/pull/33618 ### What changes were proposed in this pull request? Add the Resolver to `checkColumnNotExists` to check name exist in case sensitive. ### Why are the changes needed? At now the resolver is `_ == _` of `findNestedField` called by `checkColumnNotExists` Add `alter.conf.resolver` to it. [SPARK-36381](https://issues.apache.org/jira/browse/SPARK-36381) ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Add ut tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #33560: [SPARK-36331][CORE] Add standard SQLSTATEs to error guidelines
karenfeng commented on a change in pull request #33560: URL: https://github.com/apache/spark/pull/33560#discussion_r681416650 ## File path: core/src/main/resources/error/README.md ## @@ -79,16 +79,177 @@ The message format accepts string parameters via the C-style printf syntax. The quality of the error message should match the [guidelines](https://spark.apache.org/error-message-guidelines.html). -Invariants: + Invariants - Unique ### SQLSTATE SQLSTATE is an optional portable error identifier across SQL engines. For consistency, Spark only sets SQLSTATE as defined in the ANSI/ISO standard. -Spark does not define its own classes or subclasses. +SQLSTATE comprises a 2-character class value followed by a 3-character subclass value. +Spark only uses the standard-defined classes and subclasses, and does not use implementation-defined classes or subclasses. -Invariants: + Invariants - Consistent across releases + + ANSI/ISO standard + +The following SQLSTATEs are from ISO/IEC CD 9075-2. + +|SQLSTATE|Class|Condition |Subclass|Subcondition | +||-|||---| +|07000 |07 |dynamic SQL error |000 |(no subclass) | Review comment: I don't particularly care either way. Given that this is pulled directly from the SQL manual and this is the style they went with, I'm ok with keeping it as is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33583: [WIP][SPARK-36352][SQL] Spark should check result plan's output schema name
SparkQA commented on pull request #33583: URL: https://github.com/apache/spark/pull/33583#issuecomment-891501800 **[Test build #141978 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141978/testReport)** for PR 33583 at commit [`9cc00c0`](https://github.com/apache/spark/commit/9cc00c067de00b89fa348a857d7c18eee7de7489). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33615: [SPARK-36374][SHUFFLE][DOC] Push-based shuffle high level user documentation
SparkQA commented on pull request #33615: URL: https://github.com/apache/spark/pull/33615#issuecomment-891499615 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46486/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
