[GitHub] [spark] AmplabJenkins removed a comment on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

AmplabJenkins removed a comment on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948904149 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

AmplabJenkins commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948904609 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144513/ -- This

[GitHub] [spark] SparkQA commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948904380 **[Test build #144513 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144513/testReport)** for PR 34357 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

AmplabJenkins commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948904149 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144512/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA removed a comment on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948861549 **[Test build #144512 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144512/testReport)** for PR 34357 at commit

[GitHub] [spark] SparkQA commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948895206 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48984/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948891854 **[Test build #144512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144512/testReport)** for PR 34357 at commit

[GitHub] [spark] SparkQA commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948876466 **[Test build #144513 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144513/testReport)** for PR 34357 at commit

[GitHub] [spark] SparkQA commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948861549 **[Test build #144512 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144512/testReport)** for PR 34357 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34359: [SPARK-36986] - Improving external schema management flexibility on DataSet and StructType

AmplabJenkins commented on pull request #34359: URL: https://github.com/apache/spark/pull/34359#issuecomment-948861201 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] ankurdave commented on pull request #34245: [SPARK-37088][PYSPARK][SQL] Writer thread must not access input after task completion listener returns

ankurdave commented on pull request #34245: URL: https://github.com/apache/spark/pull/34245#issuecomment-948860753 I think I know why this is happening. The task completion listener that closes the vectorized reader is registered *lazily* in

[GitHub] [spark] JoshRosen commented on a change in pull request #34353: [SPARK-37084][SQL] Set spark.sql.files.openCostInBytes to bytesConf

JoshRosen commented on a change in pull request #34353: URL: https://github.com/apache/spark/pull/34353#discussion_r733910512 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -1415,8 +1415,8 @@ object SQLConf { " bigger files

[GitHub] [spark] sarutak edited a comment on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

sarutak edited a comment on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948850653 Hmm, recently, AppVeyor seems to fail almost every time. https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/history But I'll re-trigger it. --

[GitHub] [spark] sarutak commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

sarutak commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948850653 Hmm, recently, AppVeyor seems to fail almost every time. But I'll re-trigger it. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

dongjoon-hyun commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948848340 Could you check `AppVeyor` failure? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] risinga opened a new pull request #34359: [SPARK-36986] - Improving external schema management flexibility on DataSet and StructType

risinga opened a new pull request #34359: URL: https://github.com/apache/spark/pull/34359 ### What changes were proposed in this pull request? These are the following proposed improvements: 1 - ability to retrieve from StructType, the field's name and schema in one single call,

[GitHub] [spark] sunchao commented on a change in pull request #34308: [SPARK-37035][SQL] Improve error message when use parquet vectorize reader

sunchao commented on a change in pull request #34308: URL: https://github.com/apache/spark/pull/34308#discussion_r733894148 ## File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java ## @@ -350,7 +350,8 @@

[GitHub] [spark] ankurdave commented on pull request #34245: [SPARK-37088][PYSPARK][SQL] Writer thread must not access input after task completion listener returns

ankurdave commented on pull request #34245: URL: https://github.com/apache/spark/pull/34245#issuecomment-948836489 I noticed it occurred on another recent PR as well: https://github.com/apache/spark/pull/34352

[GitHub] [spark] sunchao commented on a change in pull request #34337: [SPARK-37066][SQL] Improve error message to show file path when failed to read next file

sunchao commented on a change in pull request #34337: URL: https://github.com/apache/spark/pull/34337#discussion_r733883879 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala ## @@ -586,10 +586,11 @@ object QueryExecutionErrors {

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

AmplabJenkins removed a comment on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948793368 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48983/

[GitHub] [spark] SparkQA commented on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

SparkQA commented on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948793336 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48983/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

AmplabJenkins removed a comment on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948793183 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144508/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

AmplabJenkins removed a comment on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948793185 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48981/

[GitHub] [spark] AmplabJenkins commented on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

AmplabJenkins commented on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948793368 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48983/ --

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34358: [SPARK-37087][SQL] Merge three relation resolutions into one

AmplabJenkins removed a comment on pull request #34358: URL: https://github.com/apache/spark/pull/34358#issuecomment-948793184 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

AmplabJenkins commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948793183 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144508/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

AmplabJenkins commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948793185 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48981/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34358: [SPARK-37087][SQL] Merge three relation resolutions into one

AmplabJenkins commented on pull request #34358: URL: https://github.com/apache/spark/pull/34358#issuecomment-948793186 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA removed a comment on pull request #34358: [SPARK-37087][SQL] Merge three relation resolutions into one

SparkQA removed a comment on pull request #34358: URL: https://github.com/apache/spark/pull/34358#issuecomment-948698897 **[Test build #144510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144510/testReport)** for PR 34358 at commit

[GitHub] [spark] SparkQA commented on pull request #34358: [SPARK-37087][SQL] Merge three relation resolutions into one

SparkQA commented on pull request #34358: URL: https://github.com/apache/spark/pull/34358#issuecomment-948792060 **[Test build #144510 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144510/testReport)** for PR 34358 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA removed a comment on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948635526 **[Test build #144508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144508/testReport)** for PR 34296 at commit

[GitHub] [spark] SparkQA commented on pull request #34358: [SPARK-37087][SQL] Merge three relation resolutions into one

SparkQA commented on pull request #34358: URL: https://github.com/apache/spark/pull/34358#issuecomment-948779230 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48982/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948776672 **[Test build #144508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144508/testReport)** for PR 34296 at commit

[GitHub] [spark] SparkQA commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948772010 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48981/ -- This is an automated message from the

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34327: [SPARK-37056][CORE] Remove unused code in HistoryServer and MetricsSystem 's unit test

dongjoon-hyun commented on a change in pull request #34327: URL: https://github.com/apache/spark/pull/34327#discussion_r733835205 ## File path: core/src/test/scala/org/apache/spark/metrics/MetricsSystemSuite.scala ## @@ -22,7 +22,7 @@ import

[GitHub] [spark] SparkQA commented on pull request #34358: [SPARK-37087][SQL] Merge three relation resolutions into one

SparkQA commented on pull request #34358: URL: https://github.com/apache/spark/pull/34358#issuecomment-948740808 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48982/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

AmplabJenkins removed a comment on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948740708 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144509/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

AmplabJenkins removed a comment on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948740705 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

SparkQA commented on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948740873 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48983/ -- This is an automated message from the Apache

[GitHub] [spark] linar-jether commented on pull request #29719: [SPARK-32846][SQL][PYTHON] Support createDataFrame from an RDD of pd.DataFrames

linar-jether commented on pull request #29719: URL: https://github.com/apache/spark/pull/29719#issuecomment-948737585 Regarding the issues you've mentioned, i think a simple `RDD[arrow]` -> `spark.DataFrame` public api would make most of these use cases pretty simple to implement.

[GitHub] [spark] linar-jether edited a comment on pull request #29719: [SPARK-32846][SQL][PYTHON] Support createDataFrame from an RDD of pd.DataFrames

linar-jether edited a comment on pull request #29719: URL: https://github.com/apache/spark/pull/29719#issuecomment-948737585 Regarding the issues you've mentioned, i think a simple `RDD[arrow]` -> `spark.DataFrame` public api would make most of these use cases pretty simple to implement.

[GitHub] [spark] SparkQA removed a comment on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA removed a comment on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948691357 **[Test build #144509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144509/testReport)** for PR 34357 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34355: [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17

SparkQA removed a comment on pull request #34355: URL: https://github.com/apache/spark/pull/34355#issuecomment-948579030 **[Test build #144506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144506/testReport)** for PR 34355 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA removed a comment on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948579161 **[Test build #144507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144507/testReport)** for PR 34296 at commit

[GitHub] [spark] dongjoon-hyun closed pull request #34355: [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17

dongjoon-hyun closed pull request #34355: URL: https://github.com/apache/spark/pull/34355 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] SparkQA commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948733783 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48981/ -- This is an automated message from the Apache

[GitHub] [spark] dongjoon-hyun commented on pull request #34337: [SPARK-37066][SQL] Improve error message to show file path when failed to read next file

dongjoon-hyun commented on pull request #34337: URL: https://github.com/apache/spark/pull/34337#issuecomment-948728514 Thank you for generalizing and updating, @AngersZh . Could you check the UT failures? It looks like wrapping exceptions requires us revise the test case asserts.

[GitHub] [spark] SparkQA commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948726316 **[Test build #144509 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144509/testReport)** for PR 34357 at commit

[GitHub] [spark] SparkQA commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948722451 **[Test build #144507 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144507/testReport)** for PR 34296 at commit

[GitHub] [spark] linar-jether commented on pull request #29719: [SPARK-32846][SQL][PYTHON] Support createDataFrame from an RDD of pd.DataFrames

linar-jether commented on pull request #29719: URL: https://github.com/apache/spark/pull/29719#issuecomment-948721250 @HyukjinKwon What do you mean by pseudo codes? My initial snippet for using pandas<->arrow<->spark conversions was done using this:

[GitHub] [spark] SparkQA commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948715595 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48980/ -- This is an automated message from the

[GitHub] [spark] ankurdave edited a comment on pull request #34245: [SPARK-33277][PYSPARK][SQL] Writer thread must not access input after task completion listener returns

ankurdave edited a comment on pull request #34245: URL: https://github.com/apache/spark/pull/34245#issuecomment-948713545 @cloud-fan Thanks! I created https://issues.apache.org/jira/browse/SPARK-37088. I noticed that `test_udf_with_column_vector` failed in `branch-3.2`, seemingly

[GitHub] [spark] ankurdave commented on pull request #34245: [SPARK-33277][PYSPARK][SQL] Writer thread must not access input after task completion listener returns

ankurdave commented on pull request #34245: URL: https://github.com/apache/spark/pull/34245#issuecomment-948713545 @cloud-fan Thanks! I created https://issues.apache.org/jira/browse/SPARK-37088. I noticed that `test_udf_with_column_vector ` failed in `branch-3.2`, seemingly with

[GitHub] [spark] AmplabJenkins commented on pull request #34355: [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17

AmplabJenkins commented on pull request #34355: URL: https://github.com/apache/spark/pull/34355#issuecomment-948710554 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144506/ -- This

[GitHub] [spark] SparkQA commented on pull request #34355: [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17

SparkQA commented on pull request #34355: URL: https://github.com/apache/spark/pull/34355#issuecomment-948708313 **[Test build #144506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144506/testReport)** for PR 34355 at commit

[GitHub] [spark] crflynn commented on pull request #34320: [SPARK-18621][PYTHON] make sql type reprs eval-able

crflynn commented on pull request #34320: URL: https://github.com/apache/spark/pull/34320#issuecomment-948703986 I think I've got everything passing. There were a lot of doctests that needed to be updated. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA commented on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

SparkQA commented on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948702894 **[Test build #144511 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144511/testReport)** for PR 34338 at commit

[GitHub] [spark] SparkQA commented on pull request #34358: [SPARK-37087][SQL] Merge three relation resolutions into one

SparkQA commented on pull request #34358: URL: https://github.com/apache/spark/pull/34358#issuecomment-948698897 **[Test build #144510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144510/testReport)** for PR 34358 at commit

[GitHub] [spark] cloud-fan commented on pull request #34358: [SPARK-37087][SQL] Merge three relation resolutions into one

cloud-fan commented on pull request #34358: URL: https://github.com/apache/spark/pull/34358#issuecomment-948696725 cc @viirya @maropu @imback82 @allisonwang-db -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] cloud-fan opened a new pull request #34358: [SPARK-37087][SQL] Merge three relation resolutions into one

cloud-fan opened a new pull request #34358: URL: https://github.com/apache/spark/pull/34358 ### What changes were proposed in this pull request? Today, Spark has 3 analyzer rules to resolve relations: `ResolveTempViews`, `ResolveTables` and `ResolveRelations`. This leads to

[GitHub] [spark] SparkQA commented on pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

SparkQA commented on pull request #34357: URL: https://github.com/apache/spark/pull/34357#issuecomment-948691357 **[Test build #144509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144509/testReport)** for PR 34357 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34355: [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17

AmplabJenkins removed a comment on pull request #34355: URL: https://github.com/apache/spark/pull/34355#issuecomment-948690522 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48978/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

AmplabJenkins removed a comment on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948690506 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48979/

[GitHub] [spark] AmplabJenkins commented on pull request #34355: [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17

AmplabJenkins commented on pull request #34355: URL: https://github.com/apache/spark/pull/34355#issuecomment-948690522 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48978/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

AmplabJenkins commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948690506 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48979/ --

[GitHub] [spark] SparkQA commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948674345 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48980/ -- This is an automated message from the Apache

[GitHub] [spark] sarutak opened a new pull request #34357: [SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13

sarutak opened a new pull request #34357: URL: https://github.com/apache/spark/pull/34357 ### What changes were proposed in this pull request? This PR fixes an issue that the R test of FPGrowthModel fails with Scala 2.13. Similar to the issue filed in SPARK-37059 (#34330), the R

[GitHub] [spark] SparkQA commented on pull request #34355: [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17

SparkQA commented on pull request #34355: URL: https://github.com/apache/spark/pull/34355#issuecomment-948668350 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48978/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948660449 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48979/ -- This is an automated message from the

[GitHub] [spark] cloud-fan commented on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

cloud-fan commented on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948652872 there are test failures -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

AmplabJenkins removed a comment on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948644567 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144502/

[GitHub] [spark] AmplabJenkins commented on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

AmplabJenkins commented on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948644567 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144502/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

SparkQA removed a comment on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948485617 **[Test build #144502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144502/testReport)** for PR 34338 at commit

[GitHub] [spark] SparkQA commented on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

SparkQA commented on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948643993 **[Test build #144502 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144502/testReport)** for PR 34338 at commit

[GitHub] [spark] SparkQA commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948635526 **[Test build #144508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144508/testReport)** for PR 34296 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

AmplabJenkins commented on pull request #34356: URL: https://github.com/apache/spark/pull/34356#issuecomment-948633161 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

AmplabJenkins removed a comment on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948632459 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144504/

[GitHub] [spark] AmplabJenkins commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

AmplabJenkins commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948632459 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144504/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA removed a comment on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948485764 **[Test build #144504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144504/testReport)** for PR 34296 at commit

[GitHub] [spark] LucaCanali commented on pull request #26783: [SPARK-30153][PYTHON][WIP] Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

LucaCanali commented on pull request #26783: URL: https://github.com/apache/spark/pull/26783#issuecomment-948621085 Thanks @HyukjinKwon for following up on this. We need something that produces pyarrow Arrays or Tables, then Awkward Array can view them in a zero-copy way. What we

[GitHub] [spark] SparkQA commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948621171 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48979/ -- This is an automated message from the Apache

[GitHub] [spark] wankunde commented on a change in pull request #34234: [SPARK-36967][CORE] Report accurate shuffle block size if its skewed

wankunde commented on a change in pull request #34234: URL: https://github.com/apache/spark/pull/34234#discussion_r733679745 ## File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ## @@ -255,9 +255,24 @@ private[spark] object HighlyCompressedMapStatus {

[GitHub] [spark] SparkQA commented on pull request #34355: [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17

SparkQA commented on pull request #34355: URL: https://github.com/apache/spark/pull/34355#issuecomment-948619246 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48978/ -- This is an automated message from the Apache

[GitHub] [spark] cloud-fan closed pull request #34154: [SPARK-37047][SQL] Add lpad and rpad functions for binary strings

cloud-fan closed pull request #34154: URL: https://github.com/apache/spark/pull/34154 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] cloud-fan commented on pull request #34154: [SPARK-37047][SQL] Add lpad and rpad functions for binary strings

cloud-fan commented on pull request #34154: URL: https://github.com/apache/spark/pull/34154#issuecomment-948614742 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948611438 **[Test build #144504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144504/testReport)** for PR 34296 at commit

[GitHub] [spark] nicolasazrak opened a new pull request #34356: [SPARK-36554][PYTHON] Expose make_date expression in functions.scala

nicolasazrak opened a new pull request #34356: URL: https://github.com/apache/spark/pull/34356 ### What changes were proposed in this pull request? This commit solves `SPARK-36554`. It exposes the `make_date` sql expression into the scala functions file to be able to use it in

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34337: [SPARK-37066][SQL] Improve error message to show file path when failed to read next file

AmplabJenkins removed a comment on pull request #34337: URL: https://github.com/apache/spark/pull/34337#issuecomment-948589956 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144503/

[GitHub] [spark] AmplabJenkins commented on pull request #34337: [SPARK-37066][SQL] Improve error message to show file path when failed to read next file

AmplabJenkins commented on pull request #34337: URL: https://github.com/apache/spark/pull/34337#issuecomment-948589956 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144503/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #34337: [SPARK-37066][SQL] Improve error message to show file path when failed to read next file

SparkQA removed a comment on pull request #34337: URL: https://github.com/apache/spark/pull/34337#issuecomment-948485642 **[Test build #144503 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144503/testReport)** for PR 34337 at commit

[GitHub] [spark] SparkQA commented on pull request #34337: [SPARK-37066][SQL] Improve error message to show file path when failed to read next file

SparkQA commented on pull request #34337: URL: https://github.com/apache/spark/pull/34337#issuecomment-948589547 **[Test build #144503 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144503/testReport)** for PR 34337 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34337: [SPARK-37066][SQL] Improve error message to show file path when failed to read next file

AmplabJenkins removed a comment on pull request #34337: URL: https://github.com/apache/spark/pull/34337#issuecomment-948578407 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48975/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34338: [SPARK-37067][SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils

AmplabJenkins removed a comment on pull request #34338: URL: https://github.com/apache/spark/pull/34338#issuecomment-948578408 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48974/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

AmplabJenkins removed a comment on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948578406 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48976/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34241: [SPARK-36975][SQL] Correct the hive client calls‘s metrics in HiveClientImpl

AmplabJenkins removed a comment on pull request #34241: URL: https://github.com/apache/spark/pull/34241#issuecomment-948578409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

SparkQA commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948579161 **[Test build #144507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144507/testReport)** for PR 34296 at commit

[GitHub] [spark] SparkQA commented on pull request #34355: [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17

SparkQA commented on pull request #34355: URL: https://github.com/apache/spark/pull/34355#issuecomment-948579030 **[Test build #144506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144506/testReport)** for PR 34355 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34296: [SPARK-36989][TESTS][PYTHON] Add type hints data tests

AmplabJenkins commented on pull request #34296: URL: https://github.com/apache/spark/pull/34296#issuecomment-948578406 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48976/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34241: [SPARK-36975][SQL] Correct the hive client calls‘s metrics in HiveClientImpl

AmplabJenkins commented on pull request #34241: URL: https://github.com/apache/spark/pull/34241#issuecomment-948578409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] AmplabJenkins commented on pull request #34337: [SPARK-37066][SQL] Improve error message to show file path when failed to read next file

AmplabJenkins commented on pull request #34337: URL: https://github.com/apache/spark/pull/34337#issuecomment-948578407 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48975/ --

<    1   2   3   4   5   >