[GitHub] [spark] cloud-fan commented on a change in pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

cloud-fan commented on a change in pull request #33803: URL: https://github.com/apache/spark/pull/33803#discussion_r704071320 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/filter/In.java ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache

[GitHub] [spark] LuciferYang opened a new pull request #33933: [SPARK-36690][SS] Clean up deprecated api usage after upgrade commons-pool2 to 2.11.1

LuciferYang opened a new pull request #33933: URL: https://github.com/apache/spark/pull/33933 ### What changes were proposed in this pull request? SPARK-36583 upgrade `Apache commons-pool2` from 2.6.2 to 2.11.1 and there are some deprecated API usage related to it that need to be

[GitHub] [spark] LuciferYang commented on pull request #33926: [SPARK-36684][SQL][TESTS] Add Jackson test dependencies to `sql/core` module at `hadoop-2.7` profile

LuciferYang commented on pull request #33926: URL: https://github.com/apache/spark/pull/33926#issuecomment-914941096 thanks @dongjoon-hyun and @sunchao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

HyukjinKwon commented on a change in pull request #33803: URL: https://github.com/apache/spark/pull/33803#discussion_r704070307 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/filter/In.java ## @@ -0,0 +1,69 @@ +/* + * Licensed to the

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

HyukjinKwon commented on a change in pull request #33803: URL: https://github.com/apache/spark/pull/33803#discussion_r704069808 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/filter/AlwaysTrue.java ## @@ -0,0 +1,36 @@ +/* + * Licensed to

[GitHub] [spark] SparkQA commented on pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

SparkQA commented on pull request #33911: URL: https://github.com/apache/spark/pull/33911#issuecomment-914939272 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47568/ -- This is an automated message from the

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

HyukjinKwon commented on a change in pull request #33803: URL: https://github.com/apache/spark/pull/33803#discussion_r704069445 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/filter/EqualNullSafe.java ## @@ -0,0 +1,64 @@ +/* + * Licensed

[GitHub] [spark] SparkQA commented on pull request #33930: [SPARK-36665][SQL] Add more Not operator simplifications

SparkQA commented on pull request #33930: URL: https://github.com/apache/spark/pull/33930#issuecomment-914939020 **[Test build #143068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143068/testReport)** for PR 33930 at commit

[GitHub] [spark] SparkQA commented on pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

SparkQA commented on pull request #33911: URL: https://github.com/apache/spark/pull/33911#issuecomment-914939044 **[Test build #143066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143066/testReport)** for PR 33911 at commit

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

HyukjinKwon commented on a change in pull request #33803: URL: https://github.com/apache/spark/pull/33803#discussion_r704066217 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/V2FiltersSuite.scala ## @@ -0,0 +1,176 @@ +/* + * Licensed to

[GitHub] [spark] Ngone51 commented on pull request #33872: [SPARK-36575][CORE] Should ignore task finished event if its task set is gone in TaskSchedulerImpl.handleSuccessfulTask

Ngone51 commented on pull request #33872: URL: https://github.com/apache/spark/pull/33872#issuecomment-914934081 @mridulm As we discussed at https://github.com/apache/spark/pull/33872#discussion_r703279670, I think me and @sleep1661 have reached the agreement that the hang issue exists in

[GitHub] [spark] HyukjinKwon commented on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame

HyukjinKwon commented on pull request #33929: URL: https://github.com/apache/spark/pull/33929#issuecomment-914933856 cc @ueshin @itholic FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33872: [SPARK-36575][CORE] Should ignore task finished event if its task set is gone in TaskSchedulerImpl.handleSuccessfulTask

AmplabJenkins removed a comment on pull request #33872: URL: https://github.com/apache/spark/pull/33872#issuecomment-910840266 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47435/

[GitHub] [spark] Ngone51 commented on a change in pull request #33872: [SPARK-36575][CORE] Should ignore task finished event if its task set is gone in TaskSchedulerImpl.handleSuccessfulTask

Ngone51 commented on a change in pull request #33872: URL: https://github.com/apache/spark/pull/33872#discussion_r704059869 ## File path: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ## @@ -1995,6 +2000,61 @@ class TaskSchedulerImplSuite extends

[GitHub] [spark] SparkQA commented on pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

SparkQA commented on pull request #33803: URL: https://github.com/apache/spark/pull/33803#issuecomment-914928927 **[Test build #143067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143067/testReport)** for PR 33803 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33931: [SPARK-36689][PYTHON] Cleanup the deprecated APIs and raise proper warning message.

AmplabJenkins removed a comment on pull request #33931: URL: https://github.com/apache/spark/pull/33931#issuecomment-914928467 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

SparkQA commented on pull request #33911: URL: https://github.com/apache/spark/pull/33911#issuecomment-914928745 **[Test build #143066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143066/testReport)** for PR 33911 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33932: [SPARK-33781][SHUFFLE] Improve caching of MergeStatus on the executor side to save memory

AmplabJenkins commented on pull request #33932: URL: https://github.com/apache/spark/pull/33932#issuecomment-914928547 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins commented on pull request #33931: [SPARK-36689][PYTHON] Cleanup the deprecated APIs and raise proper warning message.

AmplabJenkins commented on pull request #33931: URL: https://github.com/apache/spark/pull/33931#issuecomment-914928468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] HyukjinKwon commented on pull request #33928: [SPARK-36686][SQL] Fix SimplifyConditionalsInPredicate to be null-safe

HyukjinKwon commented on pull request #33928: URL: https://github.com/apache/spark/pull/33928#issuecomment-914927592 Also, Apache Spark uses forked repository's resource. Please enable Github Actions in your forked repository (see also

[GitHub] [spark] HyukjinKwon commented on pull request #33928: [SPARK-36686][SQL] Fix SimplifyConditionalsInPredicate to be null-safe

HyukjinKwon commented on pull request #33928: URL: https://github.com/apache/spark/pull/33928#issuecomment-914927440 Can you add a test and show the reproducer with incorrect results in the PR description? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA commented on pull request #33931: [SPARK-36689][PYTHON] Cleanup the deprecated APIs and raise proper warning message.

SparkQA commented on pull request #33931: URL: https://github.com/apache/spark/pull/33931#issuecomment-914927332 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47567/ -- This is an automated message from the

[GitHub] [spark] itholic commented on a change in pull request #33858: [SPARK-36402][PYTHON] Implement Series.combine

itholic commented on a change in pull request #33858: URL: https://github.com/apache/spark/pull/33858#discussion_r704045118 ## File path: python/pyspark/pandas/series.py ## @@ -4475,6 +4477,146 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] HyukjinKwon closed pull request #33925: [SPARK-36688][R] Add cot as an R function

HyukjinKwon closed pull request #33925: URL: https://github.com/apache/spark/pull/33925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] itholic commented on a change in pull request #33858: [SPARK-36402][PYTHON] Implement Series.combine

itholic commented on a change in pull request #33858: URL: https://github.com/apache/spark/pull/33858#discussion_r704045118 ## File path: python/pyspark/pandas/series.py ## @@ -4475,6 +4477,146 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] HyukjinKwon commented on pull request #33925: [SPARK-36688][R] Add cot as an R function

HyukjinKwon commented on pull request #33925: URL: https://github.com/apache/spark/pull/33925#issuecomment-914926027 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA removed a comment on pull request #33931: [SPARK-36689][PYTHON] Cleanup the deprecated APIs and raise proper warning message.

SparkQA removed a comment on pull request #33931: URL: https://github.com/apache/spark/pull/33931#issuecomment-914903403 **[Test build #143064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143064/testReport)** for PR 33931 at commit

[GitHub] [spark] itholic commented on a change in pull request #33858: [SPARK-36402][PYTHON] Implement Series.combine

itholic commented on a change in pull request #33858: URL: https://github.com/apache/spark/pull/33858#discussion_r704045118 ## File path: python/pyspark/pandas/series.py ## @@ -4475,6 +4477,146 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] SparkQA commented on pull request #33931: [SPARK-36689][PYTHON] Cleanup the deprecated APIs and raise proper warning message.

SparkQA commented on pull request #33931: URL: https://github.com/apache/spark/pull/33931#issuecomment-914924134 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47567/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

SparkQA commented on pull request #33911: URL: https://github.com/apache/spark/pull/33911#issuecomment-914923853 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47568/ -- This is an automated message from the Apache

[GitHub] [spark] gengliangwang commented on a change in pull request #33919: [SPARK-36674][SQL] Support ILIKE - case insensitive LIKE

gengliangwang commented on a change in pull request #33919: URL: https://github.com/apache/spark/pull/33919#discussion_r704052590 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala ## @@ -121,70 +187,73 @@ abstract

[GitHub] [spark] itholic commented on a change in pull request #33858: [SPARK-36402][PYTHON] Implement Series.combine

itholic commented on a change in pull request #33858: URL: https://github.com/apache/spark/pull/33858#discussion_r704045118 ## File path: python/pyspark/pandas/series.py ## @@ -4475,6 +4477,146 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] itholic commented on a change in pull request #33858: [SPARK-36402][PYTHON] Implement Series.combine

itholic commented on a change in pull request #33858: URL: https://github.com/apache/spark/pull/33858#discussion_r704045118 ## File path: python/pyspark/pandas/series.py ## @@ -4475,6 +4477,146 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] itholic commented on a change in pull request #33858: [SPARK-36402][PYTHON] Implement Series.combine

itholic commented on a change in pull request #33858: URL: https://github.com/apache/spark/pull/33858#discussion_r704045118 ## File path: python/pyspark/pandas/series.py ## @@ -4475,6 +4477,146 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] MaxGekk commented on a change in pull request #33919: [SPARK-36674][SQL] Support ILIKE - case insensitive LIKE

MaxGekk commented on a change in pull request #33919: URL: https://github.com/apache/spark/pull/33919#discussion_r704049382 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -798,7 +798,7 @@ predicate | NOT? kind=IN '('

[GitHub] [spark] MaxGekk commented on a change in pull request #33919: [SPARK-36674][SQL] Support ILIKE - case insensitive LIKE

MaxGekk commented on a change in pull request #33919: URL: https://github.com/apache/spark/pull/33919#discussion_r704049052 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala ## @@ -121,70 +187,73 @@ abstract class

[GitHub] [spark] cloud-fan commented on a change in pull request #33919: [SPARK-36674][SQL] Support ILIKE - case insensitive LIKE

cloud-fan commented on a change in pull request #33919: URL: https://github.com/apache/spark/pull/33919#discussion_r704048628 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -798,7 +798,7 @@ predicate | NOT? kind=IN '('

[GitHub] [spark] itholic commented on a change in pull request #33858: [SPARK-36402][PYTHON] Implement Series.combine

itholic commented on a change in pull request #33858: URL: https://github.com/apache/spark/pull/33858#discussion_r704046963 ## File path: python/pyspark/pandas/series.py ## @@ -4475,6 +4477,146 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] rmcyang commented on pull request #33932: [Spark-33781][SHUFFLE] Improve caching of MergeStatus on the executor side to save memory

rmcyang commented on pull request #33932: URL: https://github.com/apache/spark/pull/33932#issuecomment-914914201 cc @Ngone51 @dongjoon-hyun @Victsm @otterc @mridulm @venkata91 @zhouyejoe Please take a look. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] itholic commented on a change in pull request #33858: [SPARK-36402][PYTHON] Implement Series.combine

itholic commented on a change in pull request #33858: URL: https://github.com/apache/spark/pull/33858#discussion_r704045118 ## File path: python/pyspark/pandas/series.py ## @@ -4475,6 +4477,146 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] SparkQA commented on pull request #33931: [SPARK-36689][PYTHON] Cleanup the deprecated APIs and raise proper warning message.

SparkQA commented on pull request #33931: URL: https://github.com/apache/spark/pull/33931#issuecomment-914913850 **[Test build #143064 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143064/testReport)** for PR 33931 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

cloud-fan commented on a change in pull request #33803: URL: https://github.com/apache/spark/pull/33803#discussion_r704045703 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/V2FiltersSuite.scala ## @@ -0,0 +1,176 @@ +/* + * Licensed to the

[GitHub] [spark] cloud-fan commented on a change in pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

cloud-fan commented on a change in pull request #33803: URL: https://github.com/apache/spark/pull/33803#discussion_r704045418 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/V2FiltersSuite.scala ## @@ -0,0 +1,176 @@ +/* + * Licensed to the

[GitHub] [spark] itholic commented on a change in pull request #33858: [SPARK-36402][PYTHON] Implement Series.combine

itholic commented on a change in pull request #33858: URL: https://github.com/apache/spark/pull/33858#discussion_r704045118 ## File path: python/pyspark/pandas/series.py ## @@ -4475,6 +4477,146 @@ def replace( return self._with_new_scol(current) # TODO: dtype? +

[GitHub] [spark] dgd-contributor commented on a change in pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

dgd-contributor commented on a change in pull request #33911: URL: https://github.com/apache/spark/pull/33911#discussion_r704017833 ## File path: python/pyspark/pandas/data_type_ops/boolean_ops.py ## @@ -248,6 +249,30 @@ def and_func(left: Column, right: Any) -> Column:

[GitHub] [spark] rmcyang opened a new pull request #33932: [Spark-33781][SHUFFLE] Improve caching of MergeStatus on the executor side to save memory

rmcyang opened a new pull request #33932: URL: https://github.com/apache/spark/pull/33932 ### What changes were proposed in this pull request? This is one of the patches for SPARK-33235: Push-based Shuffle Improvement Tasks. At high level, in `MapOutputTrackerWorker`, if

[GitHub] [spark] dongjoon-hyun commented on pull request #33921: [SPARK-36677][SQL] NestedColumnAliasing should not push down aggregate functions into projections

dongjoon-hyun commented on pull request #33921: URL: https://github.com/apache/spark/pull/33921#issuecomment-914911930 +1, LGTM. Thank you, @vicennial , @HyukjinKwon , @hvanhovell , @viirya . -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] dongjoon-hyun commented on pull request #33903: [SPARK-36656][SQL] CollapseProject should not collapse correlated scalar subqueries

dongjoon-hyun commented on pull request #33903: URL: https://github.com/apache/spark/pull/33903#issuecomment-914911371 Thank you for the PR description updates, @allisonwang-db . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun commented on pull request #33926: [SPARK-36684][SQL][TESTS] Add Jackson test dependencies to `sql/core` module at `hadoop-2.7` profile

dongjoon-hyun commented on pull request #33926: URL: https://github.com/apache/spark/pull/33926#issuecomment-914907058 Merged to master/3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun closed pull request #33926: [SPARK-36684][SQL][TESTS] Add Jackson test dependencies to `sql/core` module at `hadoop-2.7` profile

dongjoon-hyun closed pull request #33926: URL: https://github.com/apache/spark/pull/33926 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

AmplabJenkins removed a comment on pull request #33911: URL: https://github.com/apache/spark/pull/33911#issuecomment-914903936 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143065/

[GitHub] [spark] SparkQA removed a comment on pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

SparkQA removed a comment on pull request #33911: URL: https://github.com/apache/spark/pull/33911#issuecomment-914903508 **[Test build #143065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143065/testReport)** for PR 33911 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

AmplabJenkins commented on pull request #33911: URL: https://github.com/apache/spark/pull/33911#issuecomment-914903936 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143065/ -- This

[GitHub] [spark] SparkQA commented on pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

SparkQA commented on pull request #33911: URL: https://github.com/apache/spark/pull/33911#issuecomment-914903911 **[Test build #143065 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143065/testReport)** for PR 33911 at commit

[GitHub] [spark] SparkQA commented on pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

SparkQA commented on pull request #33911: URL: https://github.com/apache/spark/pull/33911#issuecomment-914903508 **[Test build #143065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143065/testReport)** for PR 33911 at commit

[GitHub] [spark] SparkQA commented on pull request #33931: [SPARK-36689][PYTHON] Cleanup the deprecated APIs and raise proper warning message.

SparkQA commented on pull request #33931: URL: https://github.com/apache/spark/pull/33931#issuecomment-914903403 **[Test build #143064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143064/testReport)** for PR 33931 at commit

[GitHub] [spark] dgd-contributor commented on a change in pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

dgd-contributor commented on a change in pull request #33911: URL: https://github.com/apache/spark/pull/33911#discussion_r704018411 ## File path: python/pyspark/pandas/base.py ## @@ -422,12 +422,18 @@ def __and__(self, other: Any) -> SeriesOrIndex: def __or__(self, other:

[GitHub] [spark] dgd-contributor commented on a change in pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

dgd-contributor commented on a change in pull request #33911: URL: https://github.com/apache/spark/pull/33911#discussion_r704017833 ## File path: python/pyspark/pandas/data_type_ops/boolean_ops.py ## @@ -248,6 +249,30 @@ def and_func(left: Column, right: Any) -> Column:

[GitHub] [spark] itholic opened a new pull request #33931: [SPARK-36689][PYTHON] Cleanup the deprecated APIs and raise proper warning message.

itholic opened a new pull request #33931: URL: https://github.com/apache/spark/pull/33931 ### What changes were proposed in this pull request? This PR proposes cleanup the deprecated APIs in `missing/*.py`, and raise proper warning message for the deprecated APIs such as pandas

[GitHub] [spark] cloud-fan closed pull request #33922: [SPARK-35803][SQL] Support DataSource V2 CreateTempViewUsing

cloud-fan closed pull request #33922: URL: https://github.com/apache/spark/pull/33922 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] cloud-fan commented on pull request #33922: [SPARK-35803][SQL] Support DataSource V2 CreateTempViewUsing

cloud-fan commented on pull request #33922: URL: https://github.com/apache/spark/pull/33922#issuecomment-914899179 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] gengliangwang commented on a change in pull request #33919: [SPARK-36674][SQL] Support ILIKE - case insensitive LIKE

gengliangwang commented on a change in pull request #33919: URL: https://github.com/apache/spark/pull/33919#discussion_r704000135 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala ## @@ -121,70 +187,73 @@ abstract

[GitHub] [spark] gengliangwang commented on a change in pull request #33919: [SPARK-36674][SQL] Support ILIKE - case insensitive LIKE

gengliangwang commented on a change in pull request #33919: URL: https://github.com/apache/spark/pull/33919#discussion_r703999493 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala ## @@ -121,70 +187,73 @@ abstract

[GitHub] [spark] yaooqinn commented on a change in pull request #33888: [SPARK-36634][SQL] Support access and read parquet file by column ordinal

yaooqinn commented on a change in pull request #33888: URL: https://github.com/apache/spark/pull/33888#discussion_r703998755 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala ## @@ -901,6 +901,91 @@ abstract class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33922: [SPARK-35803][SQL] Support DataSource V2 CreateTempViewUsing

AmplabJenkins removed a comment on pull request #33922: URL: https://github.com/apache/spark/pull/33922#issuecomment-914707548 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins commented on pull request #33922: [SPARK-35803][SQL] Support DataSource V2 CreateTempViewUsing

AmplabJenkins commented on pull request #33922: URL: https://github.com/apache/spark/pull/33922#issuecomment-914881229 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143061/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #33922: [SPARK-35803][SQL] Support DataSource V2 CreateTempViewUsing

SparkQA removed a comment on pull request #33922: URL: https://github.com/apache/spark/pull/33922#issuecomment-914669562 **[Test build #143061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143061/testReport)** for PR 33922 at commit

[GitHub] [spark] cloud-fan commented on pull request #33927: [SPARK-34952][SQL][FOLLOWUP] Change column type to be NamedReference

cloud-fan commented on pull request #33927: URL: https://github.com/apache/spark/pull/33927#issuecomment-914866940 > "Fix problem" sounds too general. :) yea, should be something like `FieldReference is a private class` -- This is an automated message from the Apache Git Service.

[GitHub] [spark] SparkQA commented on pull request #33922: [SPARK-35803][SQL] Support DataSource V2 CreateTempViewUsing

SparkQA commented on pull request #33922: URL: https://github.com/apache/spark/pull/33922#issuecomment-914865788 **[Test build #143061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143061/testReport)** for PR 33922 at commit

[GitHub] [spark] sleep1661 commented on a change in pull request #33872: [SPARK-36575][CORE] Should ignore task finished event if its task set is gone in TaskSchedulerImpl.handleSuccessfulTask

sleep1661 commented on a change in pull request #33872: URL: https://github.com/apache/spark/pull/33872#discussion_r703986125 ## File path: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ## @@ -1995,6 +2000,61 @@ class TaskSchedulerImplSuite

[GitHub] [spark] viirya commented on pull request #33927: [SPARK-34952][SQL][FOLLOWUP] Change column type to be NamedReference

viirya commented on pull request #33927: URL: https://github.com/apache/spark/pull/33927#issuecomment-914863184 "Fix problem" sounds too general. :) @huaxingao Maybe add a few words there? -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] yutoacts commented on a change in pull request #33925: [SPARK-36688][R] Add cot as an R function

yutoacts commented on a change in pull request #33925: URL: https://github.com/apache/spark/pull/33925#discussion_r703985509 ## File path: R/pkg/R/functions.R ## @@ -870,6 +870,19 @@ setMethod("cosh", column(jc) }) +#' @details +#' \code{cot}: Returns

[GitHub] [spark] cloud-fan commented on pull request #33924: [SPARK-36682][CORE][TEST] Add Hadoop sequence file test for different Hadoop codecs

cloud-fan commented on pull request #33924: URL: https://github.com/apache/spark/pull/33924#issuecomment-914861488 late LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA removed a comment on pull request #33930: [SPARK-36665][SQL] Add more Not operator simplifications

SparkQA removed a comment on pull request #33930: URL: https://github.com/apache/spark/pull/33930#issuecomment-914726982 **[Test build #143063 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143063/testReport)** for PR 33930 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33930: [SPARK-36665][SQL] Add more Not operator simplifications

AmplabJenkins removed a comment on pull request #33930: URL: https://github.com/apache/spark/pull/33930#issuecomment-914852940 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143063/

[GitHub] [spark] AmplabJenkins commented on pull request #33930: [SPARK-36665][SQL] Add more Not operator simplifications

AmplabJenkins commented on pull request #33930: URL: https://github.com/apache/spark/pull/33930#issuecomment-914852940 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143063/ -- This

[GitHub] [spark] SparkQA commented on pull request #33930: [SPARK-36665][SQL] Add more Not operator simplifications

SparkQA commented on pull request #33930: URL: https://github.com/apache/spark/pull/33930#issuecomment-914852515 **[Test build #143063 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143063/testReport)** for PR 33930 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

AmplabJenkins removed a comment on pull request #33803: URL: https://github.com/apache/spark/pull/33803#issuecomment-914852110 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143060/

[GitHub] [spark] AmplabJenkins commented on pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

AmplabJenkins commented on pull request #33803: URL: https://github.com/apache/spark/pull/33803#issuecomment-914852110 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143060/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

SparkQA removed a comment on pull request #33803: URL: https://github.com/apache/spark/pull/33803#issuecomment-914647983 **[Test build #143060 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143060/testReport)** for PR 33803 at commit

[GitHub] [spark] SparkQA commented on pull request #33803: [SPARK-36556][SQL] Add DSV2 filters

SparkQA commented on pull request #33803: URL: https://github.com/apache/spark/pull/33803#issuecomment-914850617 **[Test build #143060 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143060/testReport)** for PR 33803 at commit

[GitHub] [spark] dgd-contributor commented on a change in pull request #33864: [SPARK-36296][SQL] Refactor seventh set of 20 in QueryExecutionErrors to use error classes

dgd-contributor commented on a change in pull request #33864: URL: https://github.com/apache/spark/pull/33864#discussion_r703973612 ## File path: core/src/main/resources/error/error-classes.json ## @@ -3,9 +3,29 @@ "message" : [ "Field name %s is ambiguous and has %s

[GitHub] [spark] dgd-contributor commented on a change in pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame

dgd-contributor commented on a change in pull request #33929: URL: https://github.com/apache/spark/pull/33929#discussion_r703961645 ## File path: python/pyspark/pandas/tests/test_dataframe.py ## @@ -1287,8 +1278,42 @@ def test_drop(self): pdf = pd.DataFrame({10: [1,

[GitHub] [spark] WeichenXu123 commented on pull request #33853: [SPARK-36642][SQL] Add df.withMetadata: a syntax suger to update the metadata of a dataframe

WeichenXu123 commented on pull request #33853: URL: https://github.com/apache/spark/pull/33853#issuecomment-914792425 merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] WeichenXu123 closed pull request #33853: [SPARK-36642][SQL] Add df.withMetadata: a syntax suger to update the metadata of a dataframe

WeichenXu123 closed pull request #33853: URL: https://github.com/apache/spark/pull/33853 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] itholic commented on a change in pull request #33911: [SPARK-36653][PYTHON] implement Series.__xor__ and Series.__rxor__

itholic commented on a change in pull request #33911: URL: https://github.com/apache/spark/pull/33911#discussion_r703957761 ## File path: python/pyspark/pandas/base.py ## @@ -422,12 +422,18 @@ def __and__(self, other: Any) -> SeriesOrIndex: def __or__(self, other: Any) ->

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame

AmplabJenkins removed a comment on pull request #33929: URL: https://github.com/apache/spark/pull/33929#issuecomment-914777320 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47565/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33930: [SPARK-36665][SQL] Add more Not operator simplifications

AmplabJenkins removed a comment on pull request #33930: URL: https://github.com/apache/spark/pull/33930#issuecomment-914777321 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47566/

[GitHub] [spark] AmplabJenkins commented on pull request #33930: [SPARK-36665][SQL] Add more Not operator simplifications

AmplabJenkins commented on pull request #33930: URL: https://github.com/apache/spark/pull/33930#issuecomment-914777321 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47566/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame

AmplabJenkins commented on pull request #33929: URL: https://github.com/apache/spark/pull/33929#issuecomment-914777320 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47565/ --

[GitHub] [spark] HyukjinKwon closed pull request #33850: [SPARK-36531][SPARK-36515][PYTHON] Improve test coverage for data_type_ops/* and groupby.

HyukjinKwon closed pull request #33850: URL: https://github.com/apache/spark/pull/33850 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon edited a comment on pull request #33850: [SPARK-36531][SPARK-36515][PYTHON] Improve test coverage for data_type_ops/* and groupby.

HyukjinKwon edited a comment on pull request #33850: URL: https://github.com/apache/spark/pull/33850#issuecomment-914771797 Merged to master and branch-3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] HyukjinKwon commented on pull request #33850: [SPARK-36531][SPARK-36515][PYTHON] Improve test coverage for data_type_ops/* and groupby.

HyukjinKwon commented on pull request #33850: URL: https://github.com/apache/spark/pull/33850#issuecomment-914771797 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA commented on pull request #33930: [SPARK-36665][SQL] Add more Not operator simplifications

SparkQA commented on pull request #33930: URL: https://github.com/apache/spark/pull/33930#issuecomment-914770209 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47566/ -- This is an automated message from the

[GitHub] [spark] HyukjinKwon commented on pull request #33924: [SPARK-36682][CORE][TEST] Add Hadoop sequence file test for different Hadoop codecs

HyukjinKwon commented on pull request #33924: URL: https://github.com/apache/spark/pull/33924#issuecomment-914768778 LGTM2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] HyukjinKwon commented on pull request #33910: [SPARK-36666][SQL] Fix regression in AQEShuffleReadExec

HyukjinKwon commented on pull request #33910: URL: https://github.com/apache/spark/pull/33910#issuecomment-914767854 Thanks LGTM2. BTW, it would have been nicer to fix the PR title .. otherwise every regression in AQE would have the same PR title with this. -- This is an

[GitHub] [spark] viirya closed pull request #33921: [SPARK-36677][SQL] NestedColumnAliasing should not push down aggregate functions into projections

viirya closed pull request #33921: URL: https://github.com/apache/spark/pull/33921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] viirya commented on pull request #33921: [SPARK-36677][SQL] NestedColumnAliasing should not push down aggregate functions into projections

viirya commented on pull request #33921: URL: https://github.com/apache/spark/pull/33921#issuecomment-914764690 Thanks for your contribution! Merging to master/3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] SparkQA commented on pull request #33930: [SPARK-36665][SQL] Add more Not operator simplifications

SparkQA commented on pull request #33930: URL: https://github.com/apache/spark/pull/33930#issuecomment-914760492 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47566/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame

SparkQA commented on pull request #33929: URL: https://github.com/apache/spark/pull/33929#issuecomment-914753817 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47565/ -- This is an automated message from the

  1   2   3   4   5   >