[GitHub] [spark] SparkQA commented on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

SparkQA commented on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925504422 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48036/ -- This is an automated message from the A

[GitHub] [spark] HyukjinKwon commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options

HyukjinKwon commented on pull request #29535: URL: https://github.com/apache/spark/pull/29535#issuecomment-925504281 this creates a myth that setting `options` will overwrite table properties. see also https://github.com/apache/spark/pull/34072 -- This is an automated message from the Ap

[GitHub] [spark] HyukjinKwon commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options

HyukjinKwon commented on pull request #29535: URL: https://github.com/apache/spark/pull/29535#issuecomment-925504126 So `UnresolvedReleation` is shared for both cases but conditionally use the `UnresolvedReleation.options` only for Scan? that's very confusing. -- This is an automated mes

[GitHub] [spark] cloud-fan commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options

cloud-fan commented on pull request #29535: URL: https://github.com/apache/spark/pull/29535#issuecomment-925503778 it's table properties vs scan options -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] mridulm commented on a change in pull request #34043: [SPARK-36782][CORE] Avoid blocking dispatcher-BlockManagerMaster during UpdateBlockInfo

mridulm commented on a change in pull request #34043: URL: https://github.com/apache/spark/pull/34043#discussion_r714464726 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -117,12 +117,15 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] SparkQA commented on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

SparkQA commented on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925502753 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48035/ -- This is an automated message from the A

[GitHub] [spark] Ngone51 commented on a change in pull request #34043: [SPARK-36782][CORE] Avoid blocking dispatcher-BlockManagerMaster during UpdateBlockInfo

Ngone51 commented on a change in pull request #34043: URL: https://github.com/apache/spark/pull/34043#discussion_r714465842 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -117,12 +117,15 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] HyukjinKwon commented on pull request #29535: [SPARK-32592][SQL] Make DataFrameReader.table take the specified options

HyukjinKwon commented on pull request #29535: URL: https://github.com/apache/spark/pull/29535#issuecomment-925500304 wait, I get confused here. We already defined a table with options. How does it work with the newly set options? are they merged? -- This is an automated message from the

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34072: [SPARK-36680][CATALYST] Supports Dynamic Table Options for Spark SQL

HyukjinKwon commented on a change in pull request #34072: URL: https://github.com/apache/spark/pull/34072#discussion_r714462840 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -1244,15 +1245,21 @@ class AstBuilder extends S

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

AmplabJenkins removed a comment on pull request #34046: URL: https://github.com/apache/spark/pull/34046#issuecomment-925497981 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143530/ -

[GitHub] [spark] SparkQA removed a comment on pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

SparkQA removed a comment on pull request #34046: URL: https://github.com/apache/spark/pull/34046#issuecomment-925493213 **[Test build #143530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143530/testReport)** for PR 34046 at commit [`80b24bd`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

AmplabJenkins commented on pull request #34046: URL: https://github.com/apache/spark/pull/34046#issuecomment-925497981 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143530/ -- This

[GitHub] [spark] SparkQA commented on pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

SparkQA commented on pull request #34046: URL: https://github.com/apache/spark/pull/34046#issuecomment-925497956 **[Test build #143530 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143530/testReport)** for PR 34046 at commit [`80b24bd`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

SparkQA commented on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925497597 **[Test build #143533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143533/testReport)** for PR 34033 at commit [`293daea`](https://github.com

[GitHub] [spark] huaxingao commented on pull request #34030: [SPARK-36790][SQL] Update user-facing catalog to adapt CatalogPlugin

huaxingao commented on pull request #34030: URL: https://github.com/apache/spark/pull/34030#issuecomment-925495648 > Another question is, do we need to add more function overloads with an extra catalog parameter? Agree not to add more function overloading. -- This is an automated

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

AngersZh commented on a change in pull request #34033: URL: https://github.com/apache/spark/pull/34033#discussion_r714460173 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala ## @@ -562,6 +567,8 @@ case class InSet(child:

[GitHub] [spark] SparkQA commented on pull request #34073: [SPARK-36760][SQL][FOLLOWUP] Add interface SupportsPushDownV2Filters

SparkQA commented on pull request #34073: URL: https://github.com/apache/spark/pull/34073#issuecomment-925495331 **[Test build #143532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143532/testReport)** for PR 34073 at commit [`3a0052f`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #33844: [SPARK-36506][PYTHON] Improve test coverage for series.py and indexes/*.py.

SparkQA commented on pull request #33844: URL: https://github.com/apache/spark/pull/33844#issuecomment-925494603 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48037/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33627: [SPARK-36405] Check that SQLSTATEs are valid

SparkQA commented on pull request #33627: URL: https://github.com/apache/spark/pull/33627#issuecomment-925493660 **[Test build #143531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143531/testReport)** for PR 33627 at commit [`1877bc4`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

AmplabJenkins removed a comment on pull request #34046: URL: https://github.com/apache/spark/pull/34046#issuecomment-922637790 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [spark] SparkQA commented on pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

SparkQA commented on pull request #34046: URL: https://github.com/apache/spark/pull/34046#issuecomment-925493213 **[Test build #143530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143530/testReport)** for PR 34046 at commit [`80b24bd`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

AmplabJenkins removed a comment on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-925490991 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48034/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33844: [SPARK-36506][PYTHON] Improve test coverage for series.py and indexes/*.py.

AmplabJenkins removed a comment on pull request #33844: URL: https://github.com/apache/spark/pull/33844#issuecomment-925490993 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143529/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join

AmplabJenkins removed a comment on pull request #34069: URL: https://github.com/apache/spark/pull/34069#issuecomment-925490994 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48032/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

AmplabJenkins removed a comment on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-925490992 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48033/

[GitHub] [spark] AmplabJenkins commented on pull request #33844: [SPARK-36506][PYTHON] Improve test coverage for series.py and indexes/*.py.

AmplabJenkins commented on pull request #33844: URL: https://github.com/apache/spark/pull/33844#issuecomment-925490993 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143529/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

AmplabJenkins commented on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-925490992 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48033/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join

AmplabJenkins commented on pull request #34069: URL: https://github.com/apache/spark/pull/34069#issuecomment-925490994 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48032/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

AmplabJenkins commented on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-925490991 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48034/ -- T

[GitHub] [spark] SparkQA commented on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

SparkQA commented on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925489997 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48036/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

SparkQA commented on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-925489894 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48033/ -- This is an automated message from the A

[GitHub] [spark] SparkQA removed a comment on pull request #33844: [SPARK-36506][PYTHON] Improve test coverage for series.py and indexes/*.py.

SparkQA removed a comment on pull request #33844: URL: https://github.com/apache/spark/pull/33844#issuecomment-925480993 **[Test build #143529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143529/testReport)** for PR 33844 at commit [`90e7ae9`](https://gi

[GitHub] [spark] SparkQA commented on pull request #33844: [SPARK-36506][PYTHON] Improve test coverage for series.py and indexes/*.py.

SparkQA commented on pull request #33844: URL: https://github.com/apache/spark/pull/33844#issuecomment-925488482 **[Test build #143529 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143529/testReport)** for PR 33844 at commit [`90e7ae9`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join

SparkQA commented on pull request #34069: URL: https://github.com/apache/spark/pull/34069#issuecomment-925488226 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48032/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

SparkQA commented on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925488150 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48035/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon commented on pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

HyukjinKwon commented on pull request #34046: URL: https://github.com/apache/spark/pull/34046#issuecomment-925487408 ok to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[GitHub] [spark] HyukjinKwon closed pull request #34031: [SPARK-36791][DOCS] Fix spelling mistakes in running-on-yarn.md file where JHS_POST should be JHS_HOST

HyukjinKwon closed pull request #34031: URL: https://github.com/apache/spark/pull/34031 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-un

[GitHub] [spark] HyukjinKwon commented on pull request #34031: [SPARK-36791][DOCS] Fix spelling mistakes in running-on-yarn.md file where JHS_POST should be JHS_HOST

HyukjinKwon commented on pull request #34031: URL: https://github.com/apache/spark/pull/34031#issuecomment-925486202 Merged to master, branch-3.2, banch-3.1, and branch-3.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

HyukjinKwon commented on a change in pull request #34058: URL: https://github.com/apache/spark/pull/34058#discussion_r714452391 ## File path: python/pyspark/pandas/typedef/typehints.py ## @@ -673,98 +673,146 @@ def create_tuple_for_frame_type(params: Any) -> object: Typing

[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

SparkQA commented on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-925485874 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48034/ -- This is an automated message from the A

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

HyukjinKwon commented on a change in pull request #34058: URL: https://github.com/apache/spark/pull/34058#discussion_r714452107 ## File path: python/pyspark/pandas/typedef/typehints.py ## @@ -673,98 +673,146 @@ def create_tuple_for_frame_type(params: Any) -> object: Typing

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

HyukjinKwon commented on a change in pull request #34058: URL: https://github.com/apache/spark/pull/34058#discussion_r714451936 ## File path: python/pyspark/pandas/typedef/typehints.py ## @@ -673,98 +673,146 @@ def create_tuple_for_frame_type(params: Any) -> object: Typing

[GitHub] [spark] HyukjinKwon commented on pull request #33844: [SPARK-36506][PYTHON] Improve test coverage for series.py and indexes/*.py.

HyukjinKwon commented on pull request #33844: URL: https://github.com/apache/spark/pull/33844#issuecomment-925485282 @itholic mind updating Pr description too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33989: [SPARK-36676][SQL][BUILD] Create shaded Hive module and upgrade Guava version to 30.1.1-jre

HyukjinKwon commented on a change in pull request #33989: URL: https://github.com/apache/spark/pull/33989#discussion_r714448159 ## File path: assembly/pom.xml ## @@ -165,6 +169,13 @@ hive + + + org.apache.spark + spark-hive

[GitHub] [spark] SparkQA commented on pull request #33844: [SPARK-36506][PYTHON] Improve test coverage for series.py and indexes/*.py.

SparkQA commented on pull request #33844: URL: https://github.com/apache/spark/pull/33844#issuecomment-925480993 **[Test build #143529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143529/testReport)** for PR 33844 at commit [`90e7ae9`](https://github.com

[GitHub] [spark] daugraph commented on a change in pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

daugraph commented on a change in pull request #34046: URL: https://github.com/apache/spark/pull/34046#discussion_r714446941 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ## @@ -66,6 +74,15 @@ private[spark] class Client

[GitHub] [spark] cloud-fan commented on a change in pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

cloud-fan commented on a change in pull request #34033: URL: https://github.com/apache/spark/pull/34033#discussion_r714446914 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala ## @@ -562,6 +567,8 @@ case class InSet(child: Exp

[GitHub] [spark] sigmod commented on a change in pull request #34053: [SPARK-36813][SQL][PYTHON] Propose an infrastructure of as-of join and imlement ps.merge_asof

sigmod commented on a change in pull request #34053: URL: https://github.com/apache/spark/pull/34053#discussion_r714446686 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameAsOfJoinSuite.scala ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Found

[GitHub] [spark] sigmod commented on a change in pull request #34053: [SPARK-36813][SQL][PYTHON] Propose an infrastructure of as-of join and imlement ps.merge_asof

sigmod commented on a change in pull request #34053: URL: https://github.com/apache/spark/pull/34053#discussion_r714443723 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -2122,6 +2125,68 @@ object RewriteIntersectAll ext

[GitHub] [spark] SparkQA commented on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

SparkQA commented on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925479253 **[Test build #143528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143528/testReport)** for PR 34033 at commit [`174ac71`](https://github.com

[GitHub] [spark] daugraph commented on a change in pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

daugraph commented on a change in pull request #34046: URL: https://github.com/apache/spark/pull/34046#discussion_r714445692 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ## @@ -66,6 +74,15 @@ private[spark] class Client

[GitHub] [spark] HyukjinKwon commented on pull request #34051: [SPARK-36809][SQL] Remove broadcast for InSubqueryExec used in DPP

HyukjinKwon commented on pull request #34051: URL: https://github.com/apache/spark/pull/34051#issuecomment-925478747 Otherwise, the change looks making sense to me 2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[GitHub] [spark] cloud-fan commented on a change in pull request #34073: [SPARK-36760][SQL][FOLLOWUP] Add interface SupportsPushDownV2Filters

cloud-fan commented on a change in pull request #34073: URL: https://github.com/apache/spark/pull/34073#discussion_r714445044 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownV2Filters.java ## @@ -22,23 +22,26 @@ /** * A mix-in

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34073: [SPARK-36760][SQL][FOLLOWUP] Add interface SupportsPushDownV2Filters

AmplabJenkins removed a comment on pull request #34073: URL: https://github.com/apache/spark/pull/34073#issuecomment-925477621 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143523/ -

[GitHub] [spark] AmplabJenkins commented on pull request #34073: [SPARK-36760][SQL][FOLLOWUP] Add interface SupportsPushDownV2Filters

AmplabJenkins commented on pull request #34073: URL: https://github.com/apache/spark/pull/34073#issuecomment-925477621 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143523/ -- This

[GitHub] [spark] AngersZhuuuu commented on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

AngersZh commented on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925477420 ping @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

AmplabJenkins removed a comment on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925477202 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143527/ -

[GitHub] [spark] SparkQA removed a comment on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

SparkQA removed a comment on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925475024 **[Test build #143527 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143527/testReport)** for PR 34033 at commit [`87df7b0`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

AmplabJenkins commented on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925477202 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143527/ -- This

[GitHub] [spark] SparkQA commented on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

SparkQA commented on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925477170 **[Test build #143527 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143527/testReport)** for PR 34033 at commit [`87df7b0`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #34073: [SPARK-36760][SQL][FOLLOWUP] Add interface SupportsPushDownV2Filters

SparkQA removed a comment on pull request #34073: URL: https://github.com/apache/spark/pull/34073#issuecomment-925377369 **[Test build #143523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143523/testReport)** for PR 34073 at commit [`1014995`](https://gi

[GitHub] [spark] SparkQA commented on pull request #34073: [SPARK-36760][SQL][FOLLOWUP] Add interface SupportsPushDownV2Filters

SparkQA commented on pull request #34073: URL: https://github.com/apache/spark/pull/34073#issuecomment-925476776 **[Test build #143523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143523/testReport)** for PR 34073 at commit [`1014995`](https://github.co

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34051: [SPARK-36809][SQL] Remove broadcast for InSubqueryExec used in DPP

HyukjinKwon commented on a change in pull request #34051: URL: https://github.com/apache/spark/pull/34051#discussion_r714443615 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala ## @@ -157,7 +161,8 @@ case class InSubqueryExec( child = c

[GitHub] [spark] SparkQA commented on pull request #34033: [SPARK-36792][SQL] InSet should handle NaN

SparkQA commented on pull request #34033: URL: https://github.com/apache/spark/pull/34033#issuecomment-925475024 **[Test build #143527 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143527/testReport)** for PR 34033 at commit [`87df7b0`](https://github.com

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34051: [SPARK-36809][SQL] Remove broadcast for InSubqueryExec used in DPP

HyukjinKwon commented on a change in pull request #34051: URL: https://github.com/apache/spark/pull/34051#discussion_r714442193 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala ## @@ -104,17 +104,18 @@ case class ScalarSubquery( } /** - *

[GitHub] [spark] jiaoqingbo commented on pull request #34031: [SPARK-36791][DOCS] Fix spelling mistakes in running-on-yarn.md file where JHS_POST should be JHS_HOST

jiaoqingbo commented on pull request #34031: URL: https://github.com/apache/spark/pull/34031#issuecomment-925473341 @HyukjinKwon https://user-images.githubusercontent.com/14961757/134449136-823c6510-4fed-4518-9b61-eb6cc4629681.png";> https://user-images.githubusercontent.com/14961757/1

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34051: [SPARK-36809][SQL] Remove broadcast for InSubqueryExec used in DPP

HyukjinKwon commented on a change in pull request #34051: URL: https://github.com/apache/spark/pull/34051#discussion_r714441608 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala ## @@ -104,17 +104,18 @@ case class ScalarSubquery( } /** - *

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

AmplabJenkins removed a comment on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-925472750 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143525/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33903: [SPARK-36656][SQL][TEST] CollapseProject should not collapse correlated scalar subqueries

AmplabJenkins removed a comment on pull request #33903: URL: https://github.com/apache/spark/pull/33903#issuecomment-925472748 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143521/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

AmplabJenkins removed a comment on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-925472747 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143520/ -

[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

AmplabJenkins commented on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-925472747 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143520/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33903: [SPARK-36656][SQL][TEST] CollapseProject should not collapse correlated scalar subqueries

AmplabJenkins commented on pull request #33903: URL: https://github.com/apache/spark/pull/33903#issuecomment-925472748 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143521/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

AmplabJenkins commented on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-925472750 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143525/ -- This

[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

SparkQA commented on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-925471709 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48034/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

SparkQA commented on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-925471508 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48033/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34051: [SPARK-36809][SQL] Remove broadcast for InSubqueryExec used in DPP

HyukjinKwon commented on a change in pull request #34051: URL: https://github.com/apache/spark/pull/34051#discussion_r714439033 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala ## @@ -130,14 +131,17 @@ case class InSubqueryExec( } else {

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34051: [SPARK-36809][SQL] Remove broadcast for InSubqueryExec used in DPP

HyukjinKwon commented on a change in pull request #34051: URL: https://github.com/apache/spark/pull/34051#discussion_r714438751 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala ## @@ -130,14 +131,17 @@ case class InSubqueryExec( } else {

[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join

SparkQA commented on pull request #34069: URL: https://github.com/apache/spark/pull/34069#issuecomment-925470007 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48032/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon commented on pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

HyukjinKwon commented on pull request #34046: URL: https://github.com/apache/spark/pull/34046#issuecomment-925469751 Also, please enable GitHub Actions in your forked repository (see also https://github.com/apache/spark/pull/34046/checks?check_run_id=3648181789) -- This is an automated m

[GitHub] [spark] HyukjinKwon commented on pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

HyukjinKwon commented on pull request #34046: URL: https://github.com/apache/spark/pull/34046#issuecomment-925469670 Can you show how you tested this? Showing a self-reproducer with before/after console message would be helpful for users. cc @tgravescs @mridulm FYI -- This is an automat

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34046: [SPARK-36804][YARN] Using the verbose parameter in yarn mode would cause application submission failure

HyukjinKwon commented on a change in pull request #34046: URL: https://github.com/apache/spark/pull/34046#discussion_r714437585 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ## @@ -66,6 +74,15 @@ private[spark] class Cli

[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

SparkQA removed a comment on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-925365774 **[Test build #143520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143520/testReport)** for PR 34038 at commit [`8db8b50`](https://gi

[GitHub] [spark] SparkQA removed a comment on pull request #33903: [SPARK-36656][SQL][TEST] CollapseProject should not collapse correlated scalar subqueries

SparkQA removed a comment on pull request #33903: URL: https://github.com/apache/spark/pull/33903#issuecomment-925369367 **[Test build #143521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143521/testReport)** for PR 33903 at commit [`9bbbdf2`](https://gi

[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

SparkQA commented on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-925468415 **[Test build #143520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143520/testReport)** for PR 34038 at commit [`8db8b50`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #33903: [SPARK-36656][SQL][TEST] CollapseProject should not collapse correlated scalar subqueries

SparkQA commented on pull request #33903: URL: https://github.com/apache/spark/pull/33903#issuecomment-925468420 **[Test build #143521 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143521/testReport)** for PR 33903 at commit [`9bbbdf2`](https://github.co

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34009: [SPARK-34378][SQL][AVRO] Enhance AvroSerializer validation to allow extra nullable Avro fields

HyukjinKwon commented on a change in pull request #34009: URL: https://github.com/apache/spark/pull/34009#discussion_r714436477 ## File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala ## @@ -330,4 +332,9 @@ private[sql] object AvroUtils extends Log

[GitHub] [spark] SparkQA removed a comment on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

SparkQA removed a comment on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-925457249 **[Test build #143525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143525/testReport)** for PR 34058 at commit [`8e4bedc`](https://gi

[GitHub] [spark] SparkQA commented on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

SparkQA commented on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-925465871 **[Test build #143525 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143525/testReport)** for PR 34058 at commit [`8e4bedc`](https://github.co

[GitHub] [spark] Ngone51 commented on a change in pull request #34043: [SPARK-36782][CORE] Avoid blocking dispatcher-BlockManagerMaster during UpdateBlockInfo

Ngone51 commented on a change in pull request #34043: URL: https://github.com/apache/spark/pull/34043#discussion_r714433856 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -117,12 +117,15 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] Ngone51 commented on a change in pull request #34043: [SPARK-36782][CORE] Avoid blocking dispatcher-BlockManagerMaster during UpdateBlockInfo

Ngone51 commented on a change in pull request #34043: URL: https://github.com/apache/spark/pull/34043#discussion_r714433856 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -117,12 +117,15 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] Ngone51 commented on pull request #34043: [SPARK-36782][CORE] Avoid blocking dispatcher-BlockManagerMaster during UpdateBlockInfo

Ngone51 commented on pull request #34043: URL: https://github.com/apache/spark/pull/34043#issuecomment-925463156 > Given there was no MT-safety reason to delegate to DAGScheduler, I was thinking along the lines of what @f-thiele's change : we already have other Future invocations in BlockM

[GitHub] [spark] viirya commented on a change in pull request #34062: [SPARK-36819][SQL] Don't insert redundant filters in case static partition pruning can be done

viirya commented on a change in pull request #34062: URL: https://github.com/apache/spark/pull/34062#discussion_r714431072 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala ## @@ -282,13 +284,13 @@ object PartitionPrunin

[GitHub] [spark] dgd-contributor commented on a change in pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

dgd-contributor commented on a change in pull request #34058: URL: https://github.com/apache/spark/pull/34058#discussion_r714430891 ## File path: python/pyspark/pandas/typedef/typehints.py ## @@ -673,98 +673,146 @@ def create_tuple_for_frame_type(params: Any) -> object: Ty

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33980: [WIP][SPARK-32285][PYTHON] Add PySpark support for nested timestamps with arrow

HyukjinKwon commented on a change in pull request #33980: URL: https://github.com/apache/spark/pull/33980#discussion_r714430344 ## File path: python/pyspark/sql/tests/test_arrow.py ## @@ -513,12 +531,12 @@ def run_test(num_records, num_parts, max_records, use_delay=False):

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33980: [WIP][SPARK-32285][PYTHON] Add PySpark support for nested timestamps with arrow

HyukjinKwon commented on a change in pull request #33980: URL: https://github.com/apache/spark/pull/33980#discussion_r714430165 ## File path: python/pyspark/sql/pandas/types.py ## @@ -190,10 +191,19 @@ def _check_series_localize_timestamps(s, timezone): """ from pyspa

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33980: [WIP][SPARK-32285][PYTHON] Add PySpark support for nested timestamps with arrow

HyukjinKwon commented on a change in pull request #33980: URL: https://github.com/apache/spark/pull/33980#discussion_r714430038 ## File path: python/pyspark/sql/pandas/types.py ## @@ -190,10 +191,19 @@ def _check_series_localize_timestamps(s, timezone): """ from pyspa

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33980: [WIP][SPARK-32285][PYTHON] Add PySpark support for nested timestamps with arrow

HyukjinKwon commented on a change in pull request #33980: URL: https://github.com/apache/spark/pull/33980#discussion_r714429794 ## File path: python/pyspark/sql/pandas/types.py ## @@ -20,6 +20,7 @@ pandas instances during the type conversion. """ + Review comment: L

[GitHub] [spark] dgd-contributor commented on a change in pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

dgd-contributor commented on a change in pull request #34058: URL: https://github.com/apache/spark/pull/34058#discussion_r714427614 ## File path: python/pyspark/pandas/typedef/typehints.py ## @@ -673,98 +673,146 @@ def create_tuple_for_frame_type(params: Any) -> object: Ty

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34036: [SPARK-36795][SQL] Explain Formatted has Duplicate Node IDs

HyukjinKwon commented on a change in pull request #34036: URL: https://github.com/apache/spark/pull/34036#discussion_r714427286 ## File path: sql/core/src/test/scala/org/apache/spark/sql/ExplainSuite.scala ## @@ -704,6 +704,31 @@ class ExplainSuiteAE extends ExplainSuiteHelper

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

AmplabJenkins removed a comment on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-924076359 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

<    1   2   3   4   5   6   >