[GitHub] [spark] cloud-fan closed pull request #31836: [SPARK-34742][SQL] ANSI mode: Abs throws exception if input is out of range

cloud-fan closed pull request #31836: URL: https://github.com/apache/spark/pull/31836 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] cloud-fan commented on pull request #31836: [SPARK-34742][SQL] ANSI mode: Abs throws exception if input is out of range

cloud-fan commented on pull request #31836: URL: https://github.com/apache/spark/pull/31836#issuecomment-800845750 the scala 2.13 failure is unrelated, I'm merging it to master, thanks! This is an automated message from the A

[GitHub] [spark] cloud-fan commented on a change in pull request #31835: [SPARK-34741][SQL] MergeIntoTable should avoid ambiguous reference in UpdateAction

cloud-fan commented on a change in pull request #31835: URL: https://github.com/apache/spark/pull/31835#discussion_r595754057 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseInPredicateSuite.scala ## @@ -456,15 +456,21 @@ c

[GitHub] [spark] AmplabJenkins commented on pull request #31861: Revert "[SPARK-34739][SQL] Support add/subtract of a year-month interval to/from a timestamp"

AmplabJenkins commented on pull request #31861: URL: https://github.com/apache/spark/pull/31861#issuecomment-800841801 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40725/ -

[GitHub] [spark] SparkQA commented on pull request #31861: Revert "[SPARK-34739][SQL] Support add/subtract of a year-month interval to/from a timestamp"

SparkQA commented on pull request #31861: URL: https://github.com/apache/spark/pull/31861#issuecomment-800841782 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40725/ ---

[GitHub] [spark] AmplabJenkins commented on pull request #31861: Revert "[SPARK-34739][SQL] Support add/subtract of a year-month interval to/from a timestamp"

AmplabJenkins commented on pull request #31861: URL: https://github.com/apache/spark/pull/31861#issuecomment-800840243 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136143/ -

[GitHub] [spark] SparkQA commented on pull request #31861: Revert "[SPARK-34739][SQL] Support add/subtract of a year-month interval to/from a timestamp"

SparkQA commented on pull request #31861: URL: https://github.com/apache/spark/pull/31861#issuecomment-800837921 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40725/ -

[GitHub] [spark] Ngone51 commented on pull request #31835: [SPARK-34741][SQL] MergeIntoTable should avoid ambiguous reference in UpdateAction

Ngone51 commented on pull request #31835: URL: https://github.com/apache/spark/pull/31835#issuecomment-800833107 @aokolnychyi @dongjoon-hyun The use case mentioned by @cloud-fan is a good example. However, testing it only at the analysis phase is not enough. Because, IIUC, reference bindin

[GitHub] [spark] wangyum commented on pull request #31857: [SPARK-28220][SQL] Push down the foldable predicate to both sides of Join

wangyum commented on pull request #31857: URL: https://github.com/apache/spark/pull/31857#issuecomment-800830848 cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [spark] cloud-fan commented on pull request #31855: [SPARK-34761][SQL] Support add/subtract of a day-time interval to/from a timestamp

cloud-fan commented on pull request #31855: URL: https://github.com/apache/spark/pull/31855#issuecomment-800828541 I think we should add days and times separately, to be consistent with the legacy interval type which has days as a field. ---

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31858: [SPARK-34768][SQL] Respect the default input buffer size in Univocity

HyukjinKwon commented on a change in pull request #31858: URL: https://github.com/apache/spark/pull/31858#discussion_r595740479 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala ## @@ -166,8 +166,6 @@ class CSVOptions( val quoteA

[GitHub] [spark] cloud-fan commented on a change in pull request #31848: [SPARK-34756][SQL] Fix FileScan equality check

cloud-fan commented on a change in pull request #31848: URL: https://github.com/apache/spark/pull/31848#discussion_r595739904 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala ## @@ -84,11 +85,24 @@ trait FileScan extends Scan

[GitHub] [spark] cloud-fan commented on a change in pull request #31848: [SPARK-34756][SQL] Fix FileScan equality check

cloud-fan commented on a change in pull request #31848: URL: https://github.com/apache/spark/pull/31848#discussion_r595739690 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala ## @@ -84,11 +85,24 @@ trait FileScan extends Scan

[GitHub] [spark] MaxGekk commented on a change in pull request #31858: [SPARK-34768][SQL] Respect the default input buffer size in Univocity

MaxGekk commented on a change in pull request #31858: URL: https://github.com/apache/spark/pull/31858#discussion_r595737885 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala ## @@ -166,8 +166,6 @@ class CSVOptions( val quoteAll =

[GitHub] [spark] MaxGekk commented on a change in pull request #31858: [SPARK-34768][SQL] Respect the default input buffer size in Univocity

MaxGekk commented on a change in pull request #31858: URL: https://github.com/apache/spark/pull/31858#discussion_r595736246 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ## @@ -2452,6 +2452,27 @@ abstract class CSVSuite

[GitHub] [spark] gengliangwang commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

gengliangwang commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-800823159 BTW I will add Spark document for the ANSI Type Coercion after this one. This is an automated message from

[GitHub] [spark] cloud-fan closed pull request #31825: [SPARK-34699][SQL] 'CREATE OR REPLACE TEMP VIEW USING' should uncache correctly

cloud-fan closed pull request #31825: URL: https://github.com/apache/spark/pull/31825 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] cloud-fan commented on pull request #31825: [SPARK-34699][SQL] 'CREATE OR REPLACE TEMP VIEW USING' should uncache correctly

cloud-fan commented on pull request #31825: URL: https://github.com/apache/spark/pull/31825#issuecomment-800821092 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31839: [SPARK-34087][SQL] Fix memory leak of ExecutionListenerBus

AmplabJenkins removed a comment on pull request #31839: URL: https://github.com/apache/spark/pull/31839#issuecomment-800820647 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136141/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31827: [SPARK-34492][DOCS] Add "CSV Files" page for Data Source documents.

AmplabJenkins removed a comment on pull request #31827: URL: https://github.com/apache/spark/pull/31827#issuecomment-800820670 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136142/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31839: [SPARK-34087][SQL] Fix memory leak of ExecutionListenerBus

AmplabJenkins commented on pull request #31839: URL: https://github.com/apache/spark/pull/31839#issuecomment-800820647 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136141/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31860: [SPARK-34770][SQL] InMemoryCatalog.tableExists should not fail if database doesn't exist

AmplabJenkins commented on pull request #31860: URL: https://github.com/apache/spark/pull/31860#issuecomment-800820602 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136140/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31839: [SPARK-34087][SQL] Fix memory leak of ExecutionListenerBus

AmplabJenkins removed a comment on pull request #31839: URL: https://github.com/apache/spark/pull/31839#issuecomment-800819675 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40723/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31827: [SPARK-34492][DOCS] Add "CSV Files" page for Data Source documents.

AmplabJenkins removed a comment on pull request #31827: URL: https://github.com/apache/spark/pull/31827#issuecomment-798822113 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return narrowest convertible type among TypeCollection

AmplabJenkins removed a comment on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-800812580 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40720/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31850: [SPARK-21449][SQL][FOLLOWUP] Avoid log undesirable IllegalStateException when state close

AmplabJenkins removed a comment on pull request #31850: URL: https://github.com/apache/spark/pull/31850#issuecomment-800795695 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #31839: [SPARK-34087][SQL] Fix memory leak of ExecutionListenerBus

AmplabJenkins commented on pull request #31839: URL: https://github.com/apache/spark/pull/31839#issuecomment-800819675 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40723/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31860: [SPARK-34770][SQL] InMemoryCatalog.tableExists should not fail if database doesn't exist

AmplabJenkins commented on pull request #31860: URL: https://github.com/apache/spark/pull/31860#issuecomment-800819618 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40722/ -

[GitHub] [spark] MaxGekk opened a new pull request #31861: Revert "[SPARK-34739][SQL] Support add/subtract of a year-month interval to/from a timestamp"

MaxGekk opened a new pull request #31861: URL: https://github.com/apache/spark/pull/31861 ### What changes were proposed in this pull request? Revert the commit https://github.com/apache/spark/commit/9809a2f1c5187205c81542dbdc84b71db535f6e1 ### Why are the changes needed? It ca

[GitHub] [spark] AmplabJenkins commented on pull request #31850: [SPARK-21449][SQL][FOLLOWUP] Avoid log undesirable IllegalStateException when state close

AmplabJenkins commented on pull request #31850: URL: https://github.com/apache/spark/pull/31850#issuecomment-800812582 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40719/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return narrowest convertible type among TypeCollection

AmplabJenkins commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-800812580 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40720/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31858: [SPARK-34768][SQL] Respect the default input buffer size in Univocity

AmplabJenkins commented on pull request #31858: URL: https://github.com/apache/spark/pull/31858#issuecomment-800812579 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] Ngone51 commented on a change in pull request #31835: [SPARK-34741][SQL] MergeIntoTable should avoid ambiguous reference in UpdateAction

Ngone51 commented on a change in pull request #31835: URL: https://github.com/apache/spark/pull/31835#discussion_r595727193 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseInPredicateSuite.scala ## @@ -456,15 +456,21 @@ cla

[GitHub] [spark] cloud-fan commented on pull request #31860: [SPARK-34770][SQL] InMemoryCatalog.tableExists should not fail if database doesn't exist

cloud-fan commented on pull request #31860: URL: https://github.com/apache/spark/pull/31860#issuecomment-800810882 cc @dongjoon-hyun @maropu This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] cloud-fan commented on a change in pull request #31860: [SPARK-34770][SQL] InMemoryCatalog.tableExists should not fail if database doesn't exist

cloud-fan commented on a change in pull request #31860: URL: https://github.com/apache/spark/pull/31860#discussion_r595726401 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala ## @@ -704,6 +704,9 @@ abstract class Session

[GitHub] [spark] cloud-fan opened a new pull request #31860: [SPARK-34770][SQL] InMemoryCatalog.tableExists should not fail if database doesn't exist

cloud-fan opened a new pull request #31860: URL: https://github.com/apache/spark/pull/31860 ### What changes were proposed in this pull request? This PR updates `InMemoryCatalog.tableExists` to return false if database doesn't exist, instead of failing. The new behavior is co

[GitHub] [spark] Ngone51 commented on a change in pull request #31839: [SPARK-34087][SQL] Fix memory leak of ExecutionListenerBus

Ngone51 commented on a change in pull request #31839: URL: https://github.com/apache/spark/pull/31839#discussion_r595725614 ## File path: sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala ## @@ -76,7 +76,11 @@ trait QueryExecutionListener { class

[GitHub] [spark] HyukjinKwon commented on pull request #31827: [SPARK-34492][DOCS] Add "CSV Files" page for Data Source documents.

HyukjinKwon commented on pull request #31827: URL: https://github.com/apache/spark/pull/31827#issuecomment-800809115 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31827: [SPARK-34492][DOCS] Add "CSV Files" page for Data Source documents.

HyukjinKwon commented on a change in pull request #31827: URL: https://github.com/apache/spark/pull/31827#discussion_r595724792 ## File path: examples/src/main/scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala ## @@ -246,6 +247,47 @@ object SQLDataSourceExample {

[GitHub] [spark] hiboyang commented on pull request #30763: [SPARK-31801][API][SHUFFLE] Register map output metadata

hiboyang commented on pull request #30763: URL: https://github.com/apache/spark/pull/30763#issuecomment-800807782 > We are all agree more abstraction here is really a good idea and reading [#30763 (comment)](https://github.com/apache/spark/pull/30763#issuecomment-792865534) gives me the i

[GitHub] [spark] MaxGekk commented on pull request #31855: [SPARK-34761][SQL] Support add/subtract of a day-time interval to/from a timestamp

MaxGekk commented on pull request #31855: URL: https://github.com/apache/spark/pull/31855#issuecomment-800807837 @cloud-fan @srielau I doubt about semantic of the operation. Should we add the interval as "physical" duration like in this PR by just shifting timestamp offset in micros, or co

[GitHub] [spark] gengliangwang commented on a change in pull request #31836: [SPARK-34742][SQL] ANSI mode: Abs throws exception if input is out of range

gengliangwang commented on a change in pull request #31836: URL: https://github.com/apache/spark/pull/31836#discussion_r595718397 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala ## @@ -145,18 +145,39 @@ case class UnaryPosit

[GitHub] [spark] maropu commented on a change in pull request #31836: [SPARK-34742][SQL] ANSI mode: Abs throws exception if input is out of range

maropu commented on a change in pull request #31836: URL: https://github.com/apache/spark/pull/31836#discussion_r595716452 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala ## @@ -145,18 +145,39 @@ case class UnaryPositive(chi

[GitHub] [spark] AmplabJenkins commented on pull request #31850: [SPARK-21449][SQL][FOLLOWUP] Avoid log undesirable IllegalStateException when state close

AmplabJenkins commented on pull request #31850: URL: https://github.com/apache/spark/pull/31850#issuecomment-800795695 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136137/ -

[GitHub] [spark] SparkQA commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return narrowest convertible type among TypeCollection

SparkQA commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-800795486 **[Test build #136138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136138/testReport)** for PR 31859 at commit [`13f774a`](https://github.com

[GitHub] [spark] gengliangwang opened a new pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return narrowest convertible type among TypeCollection

gengliangwang opened a new pull request #31859: URL: https://github.com/apache/spark/pull/31859 ### What changes were proposed in this pull request? Currently, when implicit casting a data type to a `TypeCollection`, Spark returns the first convertible data type among `TypeCo

[GitHub] [spark] HyukjinKwon commented on pull request #31858: [SPARK-34768 ][SQL] Respect the default input buffer size in Univocity

HyukjinKwon commented on pull request #31858: URL: https://github.com/apache/spark/pull/31858#issuecomment-800794256 cc @MaxGekk can you take a look please? This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] HyukjinKwon opened a new pull request #31858: [SPARK-34768 ][SQL] Respect the default input buffer size in Univocity

HyukjinKwon opened a new pull request #31858: URL: https://github.com/apache/spark/pull/31858 ### What changes were proposed in this pull request? This PR proposes to follow Univocity's input buffer. ### Why are the changes needed? - Firstly, it's best to trust their jud

[GitHub] [spark] attilapiros commented on pull request #30763: [SPARK-31801][API][SHUFFLE] Register map output metadata

attilapiros commented on pull request #30763: URL: https://github.com/apache/spark/pull/30763#issuecomment-800791087 We are all agree more abstraction here is really a good idea and reading https://github.com/apache/spark/pull/30763#issuecomment-792865534 gives me the impression we both wo

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31836: [SPARK-34742][SQL] ANSI mode: Abs throws exception if input is out of range

AmplabJenkins removed a comment on pull request #31836: URL: https://github.com/apache/spark/pull/31836#issuecomment-800776335 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] MaxGekk commented on pull request #31855: [SPARK-34761][SQL] Support add/subtract of a day-time interval to/from a timestamp

MaxGekk commented on pull request #31855: URL: https://github.com/apache/spark/pull/31855#issuecomment-800790381 @cloud-fan @yaooqinn Could you review this PR, please. This is an automated message from the Apache Git Service.

[GitHub] [spark] AmplabJenkins commented on pull request #31857: [SPARK-28220][SQL] Push down the foldable predicate to both sides of Join

AmplabJenkins commented on pull request #31857: URL: https://github.com/apache/spark/pull/31857#issuecomment-800790035 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40717/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31836: [SPARK-34742][SQL] ANSI mode: Abs throws exception if input is out of range

AmplabJenkins commented on pull request #31836: URL: https://github.com/apache/spark/pull/31836#issuecomment-800790037 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40718/ -

[GitHub] [spark] yaooqinn commented on pull request #31850: [SPARK-21449][SQL][FOLLOWUP] Avoid log undesirable IllegalStateException when state close

yaooqinn commented on pull request #31850: URL: https://github.com/apache/spark/pull/31850#issuecomment-800783933 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] AmplabJenkins commented on pull request #31836: [SPARK-34742][SQL] ANSI mode: Abs throws exception if input is out of range

AmplabJenkins commented on pull request #31836: URL: https://github.com/apache/spark/pull/31836#issuecomment-800776335 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136136/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31857: [SPARK-28220][SQL] Push down the foldable predicate to both sides of Join

AmplabJenkins commented on pull request #31857: URL: https://github.com/apache/spark/pull/31857#issuecomment-800776309 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136135/ -

[GitHub] [spark] linhongliu-db commented on pull request #31853: [SPARK-34504][SQL] Avoid unnecessary resolving of SQL temp views for DDL commands

linhongliu-db commented on pull request #31853: URL: https://github.com/apache/spark/pull/31853#issuecomment-800775591 late LGTM This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] sadhen commented on a change in pull request #31735: [SPARK-34600][PYTHON][SQL] Return User-defined types from Pandas UDF

sadhen commented on a change in pull request #31735: URL: https://github.com/apache/spark/pull/31735#discussion_r595695013 ## File path: python/pyspark/sql/pandas/types.py ## @@ -74,6 +74,8 @@ def to_arrow_type(dt): arrow_type = pa.struct(fields) elif type(dt) ==

[GitHub] [spark] eddyxu commented on a change in pull request #31735: [SPARK-34600][PYTHON][SQL] Return User-defined types from Pandas UDF

eddyxu commented on a change in pull request #31735: URL: https://github.com/apache/spark/pull/31735#discussion_r595694168 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowEvalPythonExec.scala ## @@ -89,9 +89,35 @@ case class ArrowEvalPythonExec

[GitHub] [spark] AmplabJenkins commented on pull request #31856: [SPARK-34766][SQL] Do not capture maven config for views

AmplabJenkins commented on pull request #31856: URL: https://github.com/apache/spark/pull/31856#issuecomment-800771398 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] sadhen commented on a change in pull request #31735: [SPARK-34600][PYTHON][SQL] Return User-defined types from Pandas UDF

sadhen commented on a change in pull request #31735: URL: https://github.com/apache/spark/pull/31735#discussion_r595691224 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowEvalPythonExec.scala ## @@ -89,9 +89,35 @@ case class ArrowEvalPythonExec

[GitHub] [spark] wangyum commented on a change in pull request #31857: [SPARK-28220][SQL] Push down the foldable predicate to both sides of Join

wangyum commented on a change in pull request #31857: URL: https://github.com/apache/spark/pull/31857#discussion_r595688798 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala ## @@ -1384,4 +1384,14 @@ class FilterPushdow

[GitHub] [spark] cloud-fan commented on a change in pull request #31835: [SPARK-34741][SQL] MergeIntoTable should avoid ambiguous reference in UpdateAction

cloud-fan commented on a change in pull request #31835: URL: https://github.com/apache/spark/pull/31835#discussion_r595690474 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala ## @@ -671,6 +671,19 @@ class AnalysisSuite extend

[GitHub] [spark] wangyum commented on a change in pull request #31857: [SPARK-28220][SQL] Push down the foldable predicate to both sides of Join

wangyum commented on a change in pull request #31857: URL: https://github.com/apache/spark/pull/31857#discussion_r595688798 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala ## @@ -1384,4 +1384,14 @@ class FilterPushdow

[GitHub] [spark] SparkQA commented on pull request #31856: [SPARK-34766][SQL] Do not capture maven config for views

SparkQA commented on pull request #31856: URL: https://github.com/apache/spark/pull/31856#issuecomment-800768241 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40716/ ---

[GitHub] [spark] wangyum commented on a change in pull request #31857: [SPARK-28220][SQL] Push down the foldable predicate to both sides of Join

wangyum commented on a change in pull request #31857: URL: https://github.com/apache/spark/pull/31857#discussion_r595688798 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala ## @@ -1384,4 +1384,14 @@ class FilterPushdow

[GitHub] [spark] SparkQA commented on pull request #31856: [SPARK-34766][SQL] Do not capture maven config for views

SparkQA commented on pull request #31856: URL: https://github.com/apache/spark/pull/31856#issuecomment-800765092 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40716/ -

[GitHub] [spark] wangyum opened a new pull request #31857: [SPARK-28220][SQL] Push down the foldable predicate to both sides of Join

wangyum opened a new pull request #31857: URL: https://github.com/apache/spark/pull/31857 ### What changes were proposed in this pull request? It will push down the foldable predicate to left side if it is a RIGHT JOIN. But it will not push down the foldable predicate to right side i

[GitHub] [spark] gengliangwang commented on a change in pull request #31836: [SPARK-34742][SQL] ANSI mode: Abs throws exception if input is out of range

gengliangwang commented on a change in pull request #31836: URL: https://github.com/apache/spark/pull/31836#discussion_r595684119 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala ## @@ -145,18 +145,39 @@ case class UnaryPosit

[GitHub] [spark] cloud-fan edited a comment on pull request #31853: [SPARK-34504][SQL] Avoid unnecessary resolving of SQL temp views for DDL commands

cloud-fan edited a comment on pull request #31853: URL: https://github.com/apache/spark/pull/31853#issuecomment-800760869 yea the scala 2.13 issue is not related, I'm merging it to master/3.1, thanks for the review! This is

[GitHub] [spark] cloud-fan closed pull request #31853: [SPARK-34504][SQL] Avoid unnecessary resolving of SQL temp views for DDL commands

cloud-fan closed pull request #31853: URL: https://github.com/apache/spark/pull/31853 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] cloud-fan commented on pull request #31853: [SPARK-34504][SQL] Avoid unnecessary resolving of SQL temp views for DDL commands

cloud-fan commented on pull request #31853: URL: https://github.com/apache/spark/pull/31853#issuecomment-800760869 yea the scala 2.13 issue is not related, I'm merging it, thanks for the review! This is an automated message

[GitHub] [spark] cloud-fan commented on a change in pull request #31839: [SPARK-34087][SQL] Fix memory leak of ExecutionListenerBus

cloud-fan commented on a change in pull request #31839: URL: https://github.com/apache/spark/pull/31839#discussion_r595681661 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SparkSessionBuilderSuite.scala ## @@ -40,6 +45,37 @@ class SparkSessionBuilderSuite extends

[GitHub] [spark] cloud-fan commented on a change in pull request #31839: [SPARK-34087][SQL] Fix memory leak of ExecutionListenerBus

cloud-fan commented on a change in pull request #31839: URL: https://github.com/apache/spark/pull/31839#discussion_r595681418 ## File path: sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala ## @@ -76,7 +76,11 @@ trait QueryExecutionListener { clas

[GitHub] [spark] LuciferYang commented on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

LuciferYang commented on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-800757254 cc @dongjoon-hyun @HyukjinKwon @srowen @maropu can you help to review this pr? thx ~ This is an automated m

[GitHub] [spark] ulysses-you opened a new pull request #31856: [SPARK-34766][SQL] Do not capture maven config for views

ulysses-you opened a new pull request #31856: URL: https://github.com/apache/spark/pull/31856 ### What changes were proposed in this pull request? Skip capture maven repo config for views. ### Why are the changes needed? Due to the bad network, we always use the

[GitHub] [spark] HeartSaVioR commented on pull request #31355: [SPARK-34255][SQL] Support partitioning with static number on required distribution and ordering on V2 write

HeartSaVioR commented on pull request #31355: URL: https://github.com/apache/spark/pull/31355#issuecomment-800730995 I'll just remove the handling of non specific distribution case early next week and ask for review again unless I hear some voice before. --

[GitHub] [spark] Ngone51 commented on a change in pull request #31839: [SPARK-34087][SQL] Fix memory leak of ExecutionListenerBus

Ngone51 commented on a change in pull request #31839: URL: https://github.com/apache/spark/pull/31839#discussion_r595655573 ## File path: sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala ## @@ -76,7 +76,11 @@ trait QueryExecutionListener { class

[GitHub] [spark] Ngone51 commented on pull request #30763: [SPARK-31801][API][SHUFFLE] Register map output metadata

Ngone51 commented on pull request #30763: URL: https://github.com/apache/spark/pull/30763#issuecomment-800730212 I'm waiting for @attilapiros 's feedback. This is an automated message from the Apache Git Service. To respond t

[GitHub] [spark] AmplabJenkins removed a comment on pull request #24559: [SPARK-27658][SQL] Add FunctionCatalog API

AmplabJenkins removed a comment on pull request #24559: URL: https://github.com/apache/spark/pull/24559#issuecomment-800729668 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40714/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31851: [SPARK-34760][EXAMPLES][MINOR] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

AmplabJenkins removed a comment on pull request #31851: URL: https://github.com/apache/spark/pull/31851#issuecomment-800715108 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

AmplabJenkins removed a comment on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-800729622 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31850: [SPARK-21449][SQL][FOLLOWUP] Avoid log undesirable IllegalStateException when state close

AmplabJenkins removed a comment on pull request #31850: URL: https://github.com/apache/spark/pull/31850#issuecomment-800197919 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #31851: [SPARK-34760][EXAMPLES][MINOR] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

AmplabJenkins commented on pull request #31851: URL: https://github.com/apache/spark/pull/31851#issuecomment-800729667 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40712/ -

[GitHub] [spark] AmplabJenkins commented on pull request #24559: [SPARK-27658][SQL] Add FunctionCatalog API

AmplabJenkins commented on pull request #24559: URL: https://github.com/apache/spark/pull/24559#issuecomment-800729668 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40714/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31850: [SPARK-21449][SQL][FOLLOWUP] Avoid log undesirable IllegalStateException when state close

AmplabJenkins commented on pull request #31850: URL: https://github.com/apache/spark/pull/31850#issuecomment-800729670 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40713/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

AmplabJenkins commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-800729669 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136133/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

AmplabJenkins commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-800729622 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40715/ -

[GitHub] [spark] SparkQA commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

SparkQA commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-800729611 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40715/ ---

[GitHub] [spark] SparkQA commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

SparkQA commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-800726771 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40715/ -

[GitHub] [spark] SparkQA commented on pull request #24559: [SPARK-27658][SQL] Add FunctionCatalog API

SparkQA commented on pull request #24559: URL: https://github.com/apache/spark/pull/24559#issuecomment-800721970 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40714/

[GitHub] [spark] AmplabJenkins commented on pull request #31850: [SPARK-21449][SQL][FOLLOWUP] Avoid log undesirable IllegalStateException when state close

AmplabJenkins commented on pull request #31850: URL: https://github.com/apache/spark/pull/31850#issuecomment-800715138 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136131/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31851: [SPARK-34760][EXAMPLES][MINOR] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

AmplabJenkins commented on pull request #31851: URL: https://github.com/apache/spark/pull/31851#issuecomment-800715108 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136130/ -

[GitHub] [spark] zhengruifeng commented on pull request #31693: [SPARK-34448][ML] Binary logistic regression incorrectly computes the intercept and coefficients with small var features

zhengruifeng commented on pull request #31693: URL: https://github.com/apache/spark/pull/31693#issuecomment-800713459 this PR maybe too big, after a offline discussion with weichen, I will split it into serveral prs This is

[GitHub] [spark] AmplabJenkins removed a comment on pull request #24559: [SPARK-27658][SQL] Add FunctionCatalog API

AmplabJenkins removed a comment on pull request #24559: URL: https://github.com/apache/spark/pull/24559#issuecomment-800712542 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136132/ -

[GitHub] [spark] SparkQA removed a comment on pull request #24559: [SPARK-27658][SQL] Add FunctionCatalog API

SparkQA removed a comment on pull request #24559: URL: https://github.com/apache/spark/pull/24559#issuecomment-800710515 **[Test build #136132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136132/testReport)** for PR 24559 at commit [`c0e9209`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #24559: [SPARK-27658][SQL] Add FunctionCatalog API

AmplabJenkins commented on pull request #24559: URL: https://github.com/apache/spark/pull/24559#issuecomment-800712542 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136132/ -

[GitHub] [spark] SparkQA commented on pull request #24559: [SPARK-27658][SQL] Add FunctionCatalog API

SparkQA commented on pull request #24559: URL: https://github.com/apache/spark/pull/24559#issuecomment-800712523 **[Test build #136132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136132/testReport)** for PR 24559 at commit [`c0e9209`](https://github.co

[GitHub] [spark] zhengruifeng commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

zhengruifeng commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-800712077 retest this please This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [spark] SparkQA commented on pull request #24559: [SPARK-27658][SQL] Add FunctionCatalog API

SparkQA commented on pull request #24559: URL: https://github.com/apache/spark/pull/24559#issuecomment-800710515 **[Test build #136132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136132/testReport)** for PR 24559 at commit [`c0e9209`](https://github.com

  1   2   3   4   >