[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950587579 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49047/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

AmplabJenkins commented on pull request #34234: URL: https://github.com/apache/spark/pull/34234#issuecomment-950587581 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49048/ -- T

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

dongjoon-hyun edited a comment on pull request #32583: URL: https://github.com/apache/spark/pull/32583#issuecomment-950586648 Thank you for updating, @cxzl25 . Pending CIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] dongjoon-hyun commented on pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

dongjoon-hyun commented on pull request #32583: URL: https://github.com/apache/spark/pull/32583#issuecomment-950586648 Pending CIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950580099 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49047/ -- This is an automated message from the A

[GitHub] [spark] LuciferYang edited a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

LuciferYang edited a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950578362 @dongjoon-hyun > I need to ask about that this works with Java8 build and Java17 run too, @LuciferYang . Could you be clear for both cases: Java8 build + Java1

[GitHub] [spark] SparkQA commented on pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

SparkQA commented on pull request #34234: URL: https://github.com/apache/spark/pull/34234#issuecomment-950578600 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49048/ -- This is an automated message from the A

[GitHub] [spark] LuciferYang commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

LuciferYang commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950578362 @dongjoon-hyun > I need to ask about that this works with Java8 build and Java17 run too, @LuciferYang . Could you be clear for both cases: Java8 build + Java17 run a

[GitHub] [spark] cxzl25 commented on pull request #34041: [SPARK-36799][SQL] Pass queryExecution name in CLI when only select query

cxzl25 commented on pull request #34041: URL: https://github.com/apache/spark/pull/34041#issuecomment-950571270 @dongjoon-hyun @viirya Help review when you have time, thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

SparkQA commented on pull request #32583: URL: https://github.com/apache/spark/pull/32583#issuecomment-950567464 **[Test build #144578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144578/testReport)** for PR 32583 at commit [`1e13812`](https://github.com

[GitHub] [spark] cxzl25 commented on a change in pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

cxzl25 commented on a change in pull request #32583: URL: https://github.com/apache/spark/pull/32583#discussion_r735284825 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HivePartitionFilteringSuite.scala ## @@ -31,16 +31,18 @@ import org.apache.spark.sq

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

AmplabJenkins removed a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950560698 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49045/

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950560671 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49045/ -- This is an automated message from the A

[GitHub] [spark] AmplabJenkins commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

AmplabJenkins commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950560698 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49045/ -- T

[GitHub] [spark] LuciferYang commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

LuciferYang commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950556540 @dongjoon-hyun OK ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[GitHub] [spark] LuciferYang commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

LuciferYang commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950555453 > I wonder what happens if we don't clear this field in the closure in this case - seems kind of risky to do this. That said, who knows what behavior differences arise if we

[GitHub] [spark] dongjoon-hyun commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

dongjoon-hyun commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950555378 I need to ask about that this works with Java8 build and Java17 run too, @LuciferYang . Could you be clear for both cases: `Java8 build + Java17 run` and `Java17 build + J

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

AmplabJenkins removed a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950554993 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

AmplabJenkins removed a comment on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950554990 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144571/ -

[GitHub] [spark] AmplabJenkins commented on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

AmplabJenkins commented on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-950554991 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144569/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

AmplabJenkins commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950554994 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

AmplabJenkins removed a comment on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-950554991 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144569/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

AmplabJenkins removed a comment on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950554992 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49046/

[GitHub] [spark] AmplabJenkins commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

AmplabJenkins commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950554990 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144571/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

AmplabJenkins commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950554992 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49046/ -- T

[GitHub] [spark] LuciferYang commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

LuciferYang commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950554323 Yes, I have added this step to the PR description -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[GitHub] [spark] LuciferYang commented on a change in pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

LuciferYang commented on a change in pull request #34368: URL: https://github.com/apache/spark/pull/34368#discussion_r735276108 ## File path: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala ## @@ -394,8 +395,17 @@ private[spark] object ClosureCleaner extends Logg

[GitHub] [spark] dongjoon-hyun commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

dongjoon-hyun commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950553649 No problem. Thank you for your swift answer. Could you revise the PR description according to the procedure? -- This is an automated message from the Apache Git Service.

[GitHub] [spark] SparkQA removed a comment on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

SparkQA removed a comment on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950482146 **[Test build #144571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144571/testReport)** for PR 34372 at commit [`5c8792d`](https://gi

[GitHub] [spark] LuciferYang edited a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

LuciferYang edited a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950552904 @dongjoon-hyun sorry, did you execute `mvn clean install -pl repl -am -DskipTests` ? If not, you can execute it first. I changed description I used zulu17:

[GitHub] [spark] SparkQA removed a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

SparkQA removed a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950490362 **[Test build #144574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144574/testReport)** for PR 34368 at commit [`dcc06a4`](https://gi

[GitHub] [spark] HyukjinKwon edited a comment on pull request #26783: [SPARK-30153][PYTHON][WIP] Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

HyukjinKwon edited a comment on pull request #26783: URL: https://github.com/apache/spark/pull/26783#issuecomment-950552721 @tgravescs and @revans2 FYI. I am thinking about introducing an API like `DataFrame.mapInArrow` like [`DataFrame.mapInPandas`](https://spark.apache.org/docs/latest/ap

[GitHub] [spark] SparkQA commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

SparkQA commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950553295 **[Test build #144571 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144571/testReport)** for PR 34372 at commit [`5c8792d`](https://github.co

[GitHub] [spark] HyukjinKwon commented on pull request #26783: [SPARK-30153][PYTHON][WIP] Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

HyukjinKwon commented on pull request #26783: URL: https://github.com/apache/spark/pull/26783#issuecomment-950553265 The main reason of this API are: - pandas instances don't support nested instances, and it disallows vectorized operations on that (PySpark) - Easier Arrow integration

[GitHub] [spark] LuciferYang edited a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

LuciferYang edited a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950552904 @dongjoon-hyun sorry, did you execute `mvn clean install -pl repl -am -DskipTests` ? If not, you can execute it first. I changed description I used zulu17

[GitHub] [spark] LuciferYang edited a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

LuciferYang edited a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950552904 @dongjoon-hyun sorry, did you execute `mvn clean install -pl repl -am -DskipTests` ? If not, you can execute it first. I changed description I test with zulu17

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950553099 **[Test build #144574 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144574/testReport)** for PR 34368 at commit [`dcc06a4`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

SparkQA removed a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950482167 **[Test build #144572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144572/testReport)** for PR 34368 at commit [`3d1daff`](https://gi

[GitHub] [spark] SparkQA removed a comment on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

SparkQA removed a comment on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-950438237 **[Test build #144569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144569/testReport)** for PR 34356 at commit [`bf3e314`](https://gi

[GitHub] [spark] LuciferYang commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

LuciferYang commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950552904 sorry, did you execute `mvn clean install -pl repl -am -DskipTests` ? If not, you can execute it first. I changed description -- This is an automated m

[GitHub] [spark] HyukjinKwon commented on pull request #26783: [SPARK-30153][PYTHON][WIP] Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

HyukjinKwon commented on pull request #26783: URL: https://github.com/apache/spark/pull/26783#issuecomment-950552721 @tgravescs and @revans2 FYI. I am thinking about introducing an API like `DataFrame.mapInArrow` like [`DataFrame.mapInPandas`](https://spark.apache.org/docs/latest/api/pytho

[GitHub] [spark] SparkQA commented on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

SparkQA commented on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-950552263 **[Test build #144569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144569/testReport)** for PR 34356 at commit [`bf3e314`](https://github.co

[GitHub] [spark] cloud-fan commented on pull request #34358: [SPARK-37087][SQL] Merge three relation resolution rules into one

cloud-fan commented on pull request #34358: URL: https://github.com/apache/spark/pull/34358#issuecomment-950552208 also cc @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[GitHub] [spark] cloud-fan commented on a change in pull request #34358: [SPARK-37087][SQL] Merge three relation resolution rules into one

cloud-fan commented on a change in pull request #34358: URL: https://github.com/apache/spark/pull/34358#discussion_r735274431 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1184,24 +1039,22 @@ class Analyzer(override val

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950551898 **[Test build #144572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144572/testReport)** for PR 34368 at commit [`3d1daff`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

SparkQA commented on pull request #34234: URL: https://github.com/apache/spark/pull/34234#issuecomment-950550704 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49048/ -- This is an automated message from the Apache

[GitHub] [spark] PengleiShi commented on a change in pull request #33914: [SPARK-32268][SQL] Dynamic bloom filter join pruning

PengleiShi commented on a change in pull request #33914: URL: https://github.com/apache/spark/pull/33914#discussion_r733515433 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/DynamicBloomFilterPruning.scala ## @@ -0,0 +1,191 @@ +/* + * Licen

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950549267 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49047/ -- This is an automated message from the Apache

[GitHub] [spark] dongjoon-hyun commented on pull request #34368: [SPARK-37072][CORE] Pass all UTs in `repl` with Java 17

dongjoon-hyun commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950548048 Does this code work for all JVMs, @LuciferYang ? When I tried this on Java 17, it seems to fail. ``` $ java -version openjdk version "17" 2021-09-14 LTS OpenJDK

[GitHub] [spark] SparkQA commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

SparkQA commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950547919 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49046/ -- This is an automated message from the A

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

dongjoon-hyun commented on a change in pull request #34368: URL: https://github.com/apache/spark/pull/34368#discussion_r735268807 ## File path: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala ## @@ -394,8 +395,17 @@ private[spark] object ClosureCleaner extends Lo

[GitHub] [spark] tanelk commented on a change in pull request #33404: [SPARK-36194][SQL] Remove the aggregation from left semi/anti join if the same aggregation has already been done on left side

tanelk commented on a change in pull request #33404: URL: https://github.com/apache/spark/pull/33404#discussion_r735264211 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RemoveRedundantAggregates.scala ## @@ -47,6 +47,10 @@ object RemoveRedun

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950530139 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49045/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

AmplabJenkins removed a comment on pull request #34234: URL: https://github.com/apache/spark/pull/34234#issuecomment-939396991 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [spark] SparkQA commented on pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

SparkQA commented on pull request #34234: URL: https://github.com/apache/spark/pull/34234#issuecomment-950529556 **[Test build #144577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144577/testReport)** for PR 34234 at commit [`4068f58`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950529376 **[Test build #144576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144576/testReport)** for PR 34367 at commit [`3c51865`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

SparkQA commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950528353 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49046/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

AmplabJenkins removed a comment on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950527920 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49043/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

AmplabJenkins removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950527923 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49044/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

AmplabJenkins removed a comment on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950527922 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49042/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

AmplabJenkins removed a comment on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950527921 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144575/ -

[GitHub] [spark] AmplabJenkins commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

AmplabJenkins commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950527922 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49042/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

AmplabJenkins commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950527921 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144575/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950527923 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49044/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

AmplabJenkins commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950527920 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49043/ -- T

[GitHub] [spark] zhengruifeng commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

zhengruifeng commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950527904 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [spark] SparkQA removed a comment on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

SparkQA removed a comment on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950511723 **[Test build #144575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144575/testReport)** for PR 34375 at commit [`e8de4ac`](https://gi

[GitHub] [spark] SparkQA commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

SparkQA commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950525915 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49042/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

SparkQA commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950524514 **[Test build #144575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144575/testReport)** for PR 34375 at commit [`e8de4ac`](https://github.co

[GitHub] [spark] HyukjinKwon closed pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

HyukjinKwon closed pull request #34372: URL: https://github.com/apache/spark/pull/34372 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-un

[GitHub] [spark] HyukjinKwon commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

HyukjinKwon commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950523923 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [spark] HyukjinKwon commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

HyukjinKwon commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950523509 I cherry-picked https://github.com/apache/spark/commit/76a317aaafbf52e8e1d5687e9d72e3a22b0a322e back to the master branch. -- This is an automated message from the Apache

[GitHub] [spark] dongjoon-hyun commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

dongjoon-hyun commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950522783 No problem, @HyukjinKwon ~ We are okay because this is not released and we can fix it. -- This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [spark] dongjoon-hyun commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

dongjoon-hyun commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950522434 Ya, this commit technically reverted `[SPARK-37084][SQL] Set spark.sql.files.openCostInBytes to bytesConf` -- This is an automated message from the Apache Git Service. T

[GitHub] [spark] HyukjinKwon commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

HyukjinKwon commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950522356 sorry it was my bad. let me correct this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [spark] HyukjinKwon edited a comment on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

HyukjinKwon edited a comment on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950522016 This is a bit odds because the PR shows the diff only the one in description https://github.com/apache/spark/pull/34371/files, and I haven't had any problem in such c

[GitHub] [spark] HyukjinKwon commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

HyukjinKwon commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950522016 This is a bit odds because the PR shows the diff only the one in description https://github.com/apache/spark/pull/34371/files -- This is an automated message from the Apac

[GitHub] [spark] HyukjinKwon commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

HyukjinKwon commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950521745 oops -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

dongjoon-hyun edited a comment on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950521492 @HyukjinKwon , the final commit looks still wrong to me. - https://github.com/apache/spark/commit/360897153755e76608a7c067d01f635fca2a5da8 Could you check

[GitHub] [spark] dongjoon-hyun commented on pull request #34371: [SPARK-37091][R] SystemRequirements to include Java < 18

dongjoon-hyun commented on pull request #34371: URL: https://github.com/apache/spark/pull/34371#issuecomment-950521492 @HyukjinKwon , the final commit looks still wrong to me. - https://github.com/apache/spark/commit/360897153755e76608a7c067d01f635fca2a5da8 Could you check the fin

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950518495 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49043/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950517863 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49044/ -- This is an automated message from the A

[GitHub] [spark] dongjoon-hyun commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

dongjoon-hyun commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950515608 Thank you for updating, @LuciferYang . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [spark] dchvn commented on pull request #34235: [SPARK-36968][PYTHON] ps.Series.dot raise "matrices are not aligned" if index is not same

dchvn commented on pull request #34235: URL: https://github.com/apache/spark/pull/34235#issuecomment-950515078 Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[GitHub] [spark] mridulm commented on pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

mridulm commented on pull request #34234: URL: https://github.com/apache/spark/pull/34234#issuecomment-950514370 Ok to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] mridulm commented on a change in pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

mridulm commented on a change in pull request #34234: URL: https://github.com/apache/spark/pull/34234#discussion_r735245433 ## File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ## @@ -255,9 +255,37 @@ private[spark] object HighlyCompressedMapStatus {

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

AmplabJenkins removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950512011 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144570/ -

[GitHub] [spark] AmplabJenkins commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

AmplabJenkins commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950512011 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144570/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

SparkQA removed a comment on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950443311 **[Test build #144570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144570/testReport)** for PR 34367 at commit [`134067f`](https://gi

[GitHub] [spark] SparkQA commented on pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

SparkQA commented on pull request #34375: URL: https://github.com/apache/spark/pull/34375#issuecomment-950511723 **[Test build #144575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144575/testReport)** for PR 34375 at commit [`e8de4ac`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950511716 **[Test build #144570 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144570/testReport)** for PR 34367 at commit [`134067f`](https://github.co

[GitHub] [spark] dchvn opened a new pull request #34375: [SPARK-37107][PYTHON] Inline type hints for python/pyspark/status.py

dchvn opened a new pull request #34375: URL: https://github.com/apache/spark/pull/34375 ### What changes were proposed in this pull request? Inline type hints for python/pyspark/status.py ### Why are the changes needed? We can take advantage of static type checking w

[GitHub] [spark] SparkQA commented on pull request #34372: [SPARK-37101][CORE] In class ShuffleBlockPusher, use config instead of key

SparkQA commented on pull request #34372: URL: https://github.com/apache/spark/pull/34372#issuecomment-950501750 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49042/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34367: [SPARK-37099][SQL] Impl a rank-based filter to optimize top-k computation

SparkQA commented on pull request #34367: URL: https://github.com/apache/spark/pull/34367#issuecomment-950500260 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49044/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34368: [SPARK-37072][CORE][TEST] Pass all UTs in `repl` with Java 17

SparkQA commented on pull request #34368: URL: https://github.com/apache/spark/pull/34368#issuecomment-950499077 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49043/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon closed pull request #34235: [SPARK-36968][PYTHON] ps.Series.dot raise "matrices are not aligned" if index is not same

HyukjinKwon closed pull request #34235: URL: https://github.com/apache/spark/pull/34235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-un

[GitHub] [spark] HyukjinKwon commented on pull request #34235: [SPARK-36968][PYTHON] ps.Series.dot raise "matrices are not aligned" if index is not same

HyukjinKwon commented on pull request #34235: URL: https://github.com/apache/spark/pull/34235#issuecomment-950496167 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

dongjoon-hyun commented on a change in pull request #32583: URL: https://github.com/apache/spark/pull/32583#discussion_r735235386 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -1008,6 +1008,16 @@ object SQLConf { .booleanCon

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

dongjoon-hyun commented on a change in pull request #32583: URL: https://github.com/apache/spark/pull/32583#discussion_r735235184 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -1010,8 +1010,8 @@ object SQLConf { val HIVE_METAS

[GitHub] [spark] dchvn commented on pull request #34235: [SPARK-36968][PYTHON] ps.Series.dot raise "matrices are not aligned" if index is not same

dchvn commented on pull request #34235: URL: https://github.com/apache/spark/pull/34235#issuecomment-950494670 Ping @ueshin @itholic @xinrong-databricks , Could you take a look? Many thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please lo

  1   2   3   >