[GitHub] [spark] ekoifman commented on pull request #32872: [SPARK-35639][SQL] Make hasCoalescedPartition return true if something was actually coalesced

ekoifman commented on pull request #32872: URL: https://github.com/apache/spark/pull/32872#issuecomment-875248394 @ulysses-you I provided a link to an explanation. I haven't seen any response to it - is the explanation adequate? -- This is an automated message from the Apache Git

[GitHub] [spark] linhongliu-db commented on a change in pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

linhongliu-db commented on a change in pull request #32959: URL: https://github.com/apache/spark/pull/32959#discussion_r665015940 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/AnsiCastSuiteBase.scala ## @@ -392,8 +391,6 @@ abstract class

[GitHub] [spark] linhongliu-db commented on a change in pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

linhongliu-db commented on a change in pull request #32959: URL: https://github.com/apache/spark/pull/32959#discussion_r665015489 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala ## @@ -249,70 +249,107 @@ object DateTimeUtils {

[GitHub] [spark] SparkQA commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875247545 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45237/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects

SparkQA commented on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875245330 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45236/ -- This is an automated message from the

[GitHub] [spark] SparkQA removed a comment on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA removed a comment on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875234125 **[Test build #140728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140728/testReport)** for PR 33237 at commit

[GitHub] [spark] SparkQA commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875245018 **[Test build #140728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140728/testReport)** for PR 33237 at commit

[GitHub] [spark] cloud-fan commented on pull request #33236: [SPARK-32577][SQL][TEST] Fix the config value for shuffled hash join in test in-joins.sql

cloud-fan commented on pull request #33236: URL: https://github.com/apache/spark/pull/33236#issuecomment-875242730 are there more places that need to fix? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] wangyum commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1

wangyum commented on pull request #33100: URL: https://github.com/apache/spark/pull/33100#issuecomment-875241720 I have a PR to fix it: https://github.com/apache/spark/pull/33240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #33236: [SPARK-32577][SQL][TEST] Fix the config value for shuffled hash join in test in-joins.sql

SparkQA removed a comment on pull request #33236: URL: https://github.com/apache/spark/pull/33236#issuecomment-875137761 **[Test build #140720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140720/testReport)** for PR 33236 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33236: [SPARK-32577][SQL][TEST] Fix the config value for shuffled hash join in test in-joins.sql

AmplabJenkins removed a comment on pull request #33236: URL: https://github.com/apache/spark/pull/33236#issuecomment-875239859 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140720/

[GitHub] [spark] SparkQA commented on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits

SparkQA commented on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875240857 **[Test build #140732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140732/testReport)** for PR 33231 at commit

[GitHub] [spark] HyukjinKwon commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1

HyukjinKwon commented on pull request #33100: URL: https://github.com/apache/spark/pull/33100#issuecomment-875240629 Should we revert this, @wangyum and @allisonwang-db ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AmplabJenkins commented on pull request #33236: [SPARK-32577][SQL][TEST] Fix the config value for shuffled hash join in test in-joins.sql

AmplabJenkins commented on pull request #33236: URL: https://github.com/apache/spark/pull/33236#issuecomment-875239859 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140720/ -- This

[GitHub] [spark] SparkQA commented on pull request #33240: [SPARK-35906][SQL][FOLLOWUP] Recursive remove sort if the maximum number of rows less than or equal to 1

SparkQA commented on pull request #33240: URL: https://github.com/apache/spark/pull/33240#issuecomment-875239152 **[Test build #140731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140731/testReport)** for PR 33240 at commit

[GitHub] [spark] SparkQA commented on pull request #33236: [SPARK-32577][SQL][TEST] Fix the config value for shuffled hash join in test in-joins.sql

SparkQA commented on pull request #33236: URL: https://github.com/apache/spark/pull/33236#issuecomment-875238958 **[Test build #140720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140720/testReport)** for PR 33236 at commit

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits

AngersZh commented on a change in pull request #33231: URL: https://github.com/apache/spark/pull/33231#discussion_r665008847 ## File path: sql/core/src/test/resources/sql-tests/results/interval.sql.out ## @@ -1459,3 +1459,83 @@ struct<> -- !query output

[GitHub] [spark] wangyum commented on pull request #33240: [SPARK-35906][SQL] Recursive remove sort if the maximum number of rows less than or equal to 1

wangyum commented on pull request #33240: URL: https://github.com/apache/spark/pull/33240#issuecomment-875238009 cc @allisonwang-db -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits

AngersZh commented on a change in pull request #33231: URL: https://github.com/apache/spark/pull/33231#discussion_r665008483 ## File path: sql/core/src/test/resources/sql-tests/results/interval.sql.out ## @@ -1459,3 +1459,83 @@ struct<> -- !query output

[GitHub] [spark] wangyum commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1

wangyum commented on pull request #33100: URL: https://github.com/apache/spark/pull/33100#issuecomment-875237884 Thank you @allisonwang-db. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits

AngersZh commented on a change in pull request #33231: URL: https://github.com/apache/spark/pull/33231#discussion_r665008483 ## File path: sql/core/src/test/resources/sql-tests/results/interval.sql.out ## @@ -1459,3 +1459,83 @@ struct<> -- !query output

[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-875237620 **[Test build #140730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140730/testReport)** for PR 32959 at commit

[GitHub] [spark] wangyum opened a new pull request #33240: [SPARK-35906][SQL] Recursive remove sort if the maximum number of rows less than or equal to 1

wangyum opened a new pull request #33240: URL: https://github.com/apache/spark/pull/33240 ### What changes were proposed in this pull request? Make it recursive remove sort if the maximum number of rows less than or equal to 1. For example: ```sql select a from (select a from

[GitHub] [spark] SparkQA commented on pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java

SparkQA commented on pull request #33164: URL: https://github.com/apache/spark/pull/33164#issuecomment-875234189 **[Test build #140729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140729/testReport)** for PR 33164 at commit

[GitHub] [spark] SparkQA commented on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path

SparkQA commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875234084 **[Test build #140727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140727/testReport)** for PR 33239 at commit

[GitHub] [spark] SparkQA commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875234125 **[Test build #140728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140728/testReport)** for PR 33237 at commit

[GitHub] [spark] SparkQA commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875233818 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45237/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33162: [SPARK-35615][PYTHON] Make unary and comparison operators data-type-based

AmplabJenkins removed a comment on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-875233295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33235: [SPARK-36028][SQL] Allow Project to host outer references in scalar subqueries

AmplabJenkins removed a comment on pull request #33235: URL: https://github.com/apache/spark/pull/33235#issuecomment-875233291 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140717/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

AmplabJenkins removed a comment on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875233292 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140719/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

AmplabJenkins removed a comment on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875233293 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140726/

[GitHub] [spark] AmplabJenkins commented on pull request #33162: [SPARK-35615][PYTHON] Make unary and comparison operators data-type-based

AmplabJenkins commented on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-875233296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] AmplabJenkins commented on pull request #33235: [SPARK-36028][SQL] Allow Project to host outer references in scalar subqueries

AmplabJenkins commented on pull request #33235: URL: https://github.com/apache/spark/pull/33235#issuecomment-875233291 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140717/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

AmplabJenkins commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875233293 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140726/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

AmplabJenkins commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875233292 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140719/ -- This

[GitHub] [spark] Peng-Lei commented on a change in pull request #33219: [SPARK-36012][SQL] Add null flag in SHOW CREATE TABLE

Peng-Lei commented on a change in pull request #33219: URL: https://github.com/apache/spark/pull/33219#discussion_r665005082 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala ## @@ -654,7 +654,7 @@ class DataFrameSetOperationsSuite

[GitHub] [spark] cfmcgrady commented on a change in pull request #33212: [SPARK-35912][SQL] Fix nullability of `spark.read.json`

cfmcgrady commented on a change in pull request #33212: URL: https://github.com/apache/spark/pull/33212#discussion_r665004974 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala ## @@ -404,6 +407,19 @@ class JacksonParser( }

[GitHub] [spark] SparkQA commented on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects

SparkQA commented on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875231879 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45236/ -- This is an automated message from the Apache

[GitHub] [spark] viirya commented on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path

viirya commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875230207 cc @cloud-fan first for the draft idea. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] viirya opened a new pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path

viirya opened a new pull request #33239: URL: https://github.com/apache/spark/pull/33239 ### What changes were proposed in this pull request? We add the interface for DS v2 metrics in SPARK-34366. It is only added for reading path, though. This patch extends the metrics

[GitHub] [spark] cfmcgrady commented on a change in pull request #33212: [SPARK-35912][SQL] Fix nullability of `spark.read.json`

cfmcgrady commented on a change in pull request #33212: URL: https://github.com/apache/spark/pull/33212#discussion_r665002215 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ## @@ -2924,6 +2924,50 @@ abstract class

[GitHub] [spark] SparkQA commented on pull request #33162: [SPARK-35615][PYTHON] Make unary and comparison operators data-type-based

SparkQA commented on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-875228976 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45235/ --

[GitHub] [spark] SparkQA removed a comment on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA removed a comment on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875217481 **[Test build #140726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140726/testReport)** for PR 33237 at commit

[GitHub] [spark] SparkQA commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875226942 **[Test build #140726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140726/testReport)** for PR 33237 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #33162: [SPARK-35615][PYTHON] Make unary and comparison operators data-type-based

SparkQA removed a comment on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-875215954 **[Test build #140725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140725/testReport)** for PR 33162 at commit

[GitHub] [spark] SparkQA commented on pull request #33162: [SPARK-35615][PYTHON] Make unary and comparison operators data-type-based

SparkQA commented on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-875225667 **[Test build #140725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140725/testReport)** for PR 33162 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

SparkQA removed a comment on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875113228 **[Test build #140719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140719/testReport)** for PR 33214 at commit

[GitHub] [spark] SparkQA commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

SparkQA commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875221244 **[Test build #140719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140719/testReport)** for PR 33214 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #33235: [SPARK-36028][SQL] Allow Project to host outer references in scalar subqueries

SparkQA removed a comment on pull request #33235: URL: https://github.com/apache/spark/pull/33235#issuecomment-875110687 **[Test build #140717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140717/testReport)** for PR 33235 at commit

[GitHub] [spark] SparkQA commented on pull request #33235: [SPARK-36028][SQL] Allow Project to host outer references in scalar subqueries

SparkQA commented on pull request #33235: URL: https://github.com/apache/spark/pull/33235#issuecomment-875218762 **[Test build #140717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140717/testReport)** for PR 33235 at commit

[GitHub] [spark] itholic commented on a change in pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

itholic commented on a change in pull request #33214: URL: https://github.com/apache/spark/pull/33214#discussion_r664993005 ## File path: python/pyspark/sql/types.py ## @@ -1020,14 +1020,22 @@ def _infer_type(obj): return dataType() if isinstance(obj, dict): -

[GitHub] [spark] maryannxue commented on pull request #33140: [SPARK-35881][SQL] Add support for columnar execution of final query stage in AdaptiveSparkPlanExec

maryannxue commented on pull request #33140: URL: https://github.com/apache/spark/pull/33140#issuecomment-875218612 @andygrove We have similar requirement from Databricks. The change we made was passing a flag into the columnar rules specifying the desired format: row or columnar. Would

[GitHub] [spark] SparkQA commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875217481 **[Test build #140726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140726/testReport)** for PR 33237 at commit

[GitHub] [spark] ulysses-you commented on pull request #32872: [SPARK-35639][SQL] Make hasCoalescedPartition return true if something was actually coalesced

ulysses-you commented on pull request #32872: URL: https://github.com/apache/spark/pull/32872#issuecomment-875216687 hi @ekoifman any update ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] SparkQA commented on pull request #33162: [SPARK-35615][PYTHON] Make unary and comparison operators data-type-based

SparkQA commented on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-875215954 **[Test build #140725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140725/testReport)** for PR 33162 at commit

[GitHub] [spark] SparkQA commented on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects

SparkQA commented on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875215847 **[Test build #140724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140724/testReport)** for PR 33238 at commit

[GitHub] [spark] ulysses-you commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

ulysses-you commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-875215416 @cloud-fan created https://github.com/apache/spark/pull/33238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] ulysses-you opened a new pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects

ulysses-you opened a new pull request #33238: URL: https://github.com/apache/spark/pull/33238 backport [#33222](https://github.com/apache/spark/pull/33222) ### What changes were proposed in this pull request? The RemoveRedundantProjects feature can conflict with the AQE

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33162: [SPARK-35615][PYTHON] Make unary and comparison operators data-type-based

AmplabJenkins removed a comment on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-875145885 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45229/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

AmplabJenkins removed a comment on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875209780 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45234/

[GitHub] [spark] AmplabJenkins commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

AmplabJenkins commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875209780 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45234/ --

[GitHub] [spark] SparkQA commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

SparkQA commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875202800 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45234/ --

[GitHub] [spark] HyukjinKwon closed pull request #31998: [SPARK-34859][SQL] parquet vectorized reader - support column index with rowIndexes

HyukjinKwon closed pull request #31998: URL: https://github.com/apache/spark/pull/31998 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] viirya commented on a change in pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

viirya commented on a change in pull request #33214: URL: https://github.com/apache/spark/pull/33214#discussion_r664970252 ## File path: python/pyspark/sql/types.py ## @@ -1020,14 +1020,22 @@ def _infer_type(obj): return dataType() if isinstance(obj, dict): -

[GitHub] [spark] SparkQA commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

SparkQA commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875188042 **[Test build #140723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140723/testReport)** for PR 33214 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

AmplabJenkins removed a comment on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875187262 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45233/

[GitHub] [spark] drakelam commented on a change in pull request #33228: Update README.md

drakelam commented on a change in pull request #33228: URL: https://github.com/apache/spark/pull/33228#discussion_r664966843 ## File path: README.md ## @@ -21,6 +21,8 @@ You can find the latest Spark documentation, including a programming guide, on the [project web

[GitHub] [spark] AmplabJenkins commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

AmplabJenkins commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875187262 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45233/ --

[GitHub] [spark] HyukjinKwon edited a comment on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

HyukjinKwon edited a comment on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875185361 cc @viirya or @ueshin would you mind taking a quick look please? This is ready for a review. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] HyukjinKwon commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

HyukjinKwon commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875185361 cc @viirya or @ueshin would you mind taking a quick look please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

HyukjinKwon commented on a change in pull request #33214: URL: https://github.com/apache/spark/pull/33214#discussion_r664964735 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -3327,6 +3327,13 @@ object SQLConf { .intConf

[GitHub] [spark] sarutak commented on a change in pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types

sarutak commented on a change in pull request #32949: URL: https://github.com/apache/spark/pull/32949#discussion_r664963511 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -2526,10 +2526,10 @@ object DatePart

[GitHub] [spark] sarutak commented on a change in pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types

sarutak commented on a change in pull request #32949: URL: https://github.com/apache/spark/pull/32949#discussion_r664961224 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -2526,10 +2526,10 @@ object DatePart

[GitHub] [spark] SparkQA commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875179117 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45233/ --

[GitHub] [spark] viirya commented on pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre

viirya commented on pull request #29326: URL: https://github.com/apache/spark/pull/29326#issuecomment-875177916 Hmm, I looked at `isSharedClass`, looks like `common-lang3`, orc, etc. are already non-shared classes. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] itholic commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

itholic commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875176124 Thanks! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33236: [SPARK-32577][SQL][TEST] Fix the config value for shuffled hash join in test in-joins.sql

AmplabJenkins removed a comment on pull request #33236: URL: https://github.com/apache/spark/pull/33236#issuecomment-875167975 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45231/

[GitHub] [spark] SparkQA commented on pull request #33236: [SPARK-32577][SQL][TEST] Fix the config value for shuffled hash join in test in-joins.sql

SparkQA commented on pull request #33236: URL: https://github.com/apache/spark/pull/33236#issuecomment-875167952 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45231/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins commented on pull request #33236: [SPARK-32577][SQL][TEST] Fix the config value for shuffled hash join in test in-joins.sql

AmplabJenkins commented on pull request #33236: URL: https://github.com/apache/spark/pull/33236#issuecomment-875167975 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45231/ --

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

HyukjinKwon commented on a change in pull request #33223: URL: https://github.com/apache/spark/pull/33223#discussion_r664951709 ## File path: .github/workflows/build_and_test.yml ## @@ -366,9 +366,7 @@ jobs: # See also

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

HyukjinKwon commented on a change in pull request #33223: URL: https://github.com/apache/spark/pull/33223#discussion_r664951709 ## File path: .github/workflows/build_and_test.yml ## @@ -366,9 +366,7 @@ jobs: # See also

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

HyukjinKwon commented on a change in pull request #33214: URL: https://github.com/apache/spark/pull/33214#discussion_r664951035 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -3327,6 +3327,13 @@ object SQLConf { .intConf

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

HyukjinKwon commented on a change in pull request #33214: URL: https://github.com/apache/spark/pull/33214#discussion_r664950636 ## File path: python/pyspark/sql/tests/test_types.py ## @@ -204,6 +204,21 @@ def test_infer_nested_schema(self): df =

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

AmplabJenkins removed a comment on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875162378 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140722/

[GitHub] [spark] SparkQA removed a comment on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA removed a comment on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875161834 **[Test build #140722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140722/testReport)** for PR 33237 at commit

[GitHub] [spark] SparkQA commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875162363 **[Test build #140722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140722/testReport)** for PR 33237 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

AmplabJenkins commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875162378 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140722/ -- This

[GitHub] [spark] SparkQA commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

SparkQA commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875161834 **[Test build #140722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140722/testReport)** for PR 33237 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

AmplabJenkins removed a comment on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875160930 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45230/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33160: [SPARK-35959][BUILD][test-maven][test-hadoop3.2][test-java11] Add a new Maven profile "no-shaded-hadoop-client" for Hadoop vers

AmplabJenkins removed a comment on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-875160927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

AmplabJenkins commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875160930 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45230/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33160: [SPARK-35959][BUILD][test-maven][test-hadoop3.2][test-java11] Add a new Maven profile "no-shaded-hadoop-client" for Hadoop versions old

AmplabJenkins commented on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-875160927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA commented on pull request #33160: [SPARK-35959][BUILD][test-maven][test-hadoop3.2][test-java11] Add a new Maven profile "no-shaded-hadoop-client" for Hadoop versions older tha

SparkQA commented on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-875156683 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45232/ -- This is an automated message from the

[GitHub] [spark] SparkQA removed a comment on pull request #33160: [SPARK-35959][BUILD][test-maven][test-hadoop3.2][test-java11] Add a new Maven profile "no-shaded-hadoop-client" for Hadoop versions o

SparkQA removed a comment on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-875137896 **[Test build #140721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140721/testReport)** for PR 33160 at commit

[GitHub] [spark] SparkQA commented on pull request #33160: [SPARK-35959][BUILD][test-maven][test-hadoop3.2][test-java11] Add a new Maven profile "no-shaded-hadoop-client" for Hadoop versions older tha

SparkQA commented on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-875156397 **[Test build #140721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140721/testReport)** for PR 33160 at commit

[GitHub] [spark] SparkQA commented on pull request #33160: [SPARK-35959][BUILD][test-maven][test-hadoop3.2][test-java11] Add a new Maven profile "no-shaded-hadoop-client" for Hadoop versions older tha

SparkQA commented on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-875155890 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45232/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33236: [SPARK-32577][SQL][TEST] Fix the config value for shuffled hash join in test in-joins.sql

SparkQA commented on pull request #33236: URL: https://github.com/apache/spark/pull/33236#issuecomment-875155527 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45231/ -- This is an automated message from the Apache

[GitHub] [spark] xinrong-databricks opened a new pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations

xinrong-databricks opened a new pull request #33237: URL: https://github.com/apache/spark/pull/33237 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

[GitHub] [spark] SparkQA commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

SparkQA commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875148688 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45230/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins commented on pull request #33162: [SPARK-35615][PYTHON] Make unary and comparison operators data-type-based

AmplabJenkins commented on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-875145885 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45229/ --

<    1   2   3   4   5   6   7   8   9   10   >