[jira] [Assigned] (SPARK-40051) Make pyspark.sql.catalog examples self-contained

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40051: Assignee: Apache Spark > Make pyspark.sql.catalog examples self-contained >

[jira] [Assigned] (SPARK-40051) Make pyspark.sql.catalog examples self-contained

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40051: Assignee: (was: Apache Spark) > Make pyspark.sql.catalog examples self-contained >

[jira] [Commented] (SPARK-40051) Make pyspark.sql.catalog examples self-contained

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578820#comment-17578820 ] Apache Spark commented on SPARK-40051: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-40056) Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578819#comment-17578819 ] Apache Spark commented on SPARK-40056: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-40056) Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40056: Assignee: (was: Apache Spark) > Upgrade mvn-scalafmt from 1.0.4 to

[jira] [Commented] (SPARK-40056) Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578818#comment-17578818 ] Apache Spark commented on SPARK-40056: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-40056) Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40056: Assignee: Apache Spark > Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9 >

[jira] [Created] (SPARK-40056) Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9

2022-08-11 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-40056: --- Summary: Upgrade mvn-scalafmt from 1.0.4 to 1.1.1640084764.9f463a9 Key: SPARK-40056 URL: https://issues.apache.org/jira/browse/SPARK-40056 Project: Spark

[jira] [Commented] (SPARK-40055) listCatalogs should also return spark_catalog even spark_catalog implementation is defaultSessionCatalog

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578815#comment-17578815 ] Apache Spark commented on SPARK-40055: -- User 'amaliujia' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40055) listCatalogs should also return spark_catalog even spark_catalog implementation is defaultSessionCatalog

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40055: Assignee: Apache Spark > listCatalogs should also return spark_catalog even

[jira] [Assigned] (SPARK-40055) listCatalogs should also return spark_catalog even spark_catalog implementation is defaultSessionCatalog

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40055: Assignee: (was: Apache Spark) > listCatalogs should also return spark_catalog even

[jira] [Commented] (SPARK-40055) listCatalogs should also return spark_catalog even spark_catalog implementation is defaultSessionCatalog

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578813#comment-17578813 ] Apache Spark commented on SPARK-40055: -- User 'amaliujia' has created a pull request for this issue:

[jira] [Commented] (SPARK-40053) HiveExternalCatalogVersionsSuite will test all spark versions and aborted when Python 2.7 is used

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578812#comment-17578812 ] Apache Spark commented on SPARK-40053: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-40053) HiveExternalCatalogVersionsSuite will test all spark versions and aborted when Python 2.7 is used

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40053: Assignee: (was: Apache Spark) > HiveExternalCatalogVersionsSuite will test all spark

[jira] [Commented] (SPARK-40053) HiveExternalCatalogVersionsSuite will test all spark versions and aborted when Python 2.7 is used

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578811#comment-17578811 ] Apache Spark commented on SPARK-40053: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-40053) HiveExternalCatalogVersionsSuite will test all spark versions and aborted when Python 2.7 is used

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40053: Assignee: Apache Spark > HiveExternalCatalogVersionsSuite will test all spark versions

[jira] [Created] (SPARK-40055) listCatalogs should also return spark_catalog even spark_catalog implementation is defaultSessionCatalog

2022-08-11 Thread Rui Wang (Jira)
Rui Wang created SPARK-40055: Summary: listCatalogs should also return spark_catalog even spark_catalog implementation is defaultSessionCatalog Key: SPARK-40055 URL: https://issues.apache.org/jira/browse/SPARK-40055

[jira] [Commented] (SPARK-40054) Restore the error handling syntax of try_cast()

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578800#comment-17578800 ] Apache Spark commented on SPARK-40054: -- User 'gengliangwang' has created a pull request for this

[jira] [Commented] (SPARK-40054) Restore the error handling syntax of try_cast()

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578799#comment-17578799 ] Apache Spark commented on SPARK-40054: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-40054) Restore the error handling syntax of try_cast()

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40054: Assignee: Apache Spark (was: Gengliang Wang) > Restore the error handling syntax of

[jira] [Assigned] (SPARK-40054) Restore the error handling syntax of try_cast()

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40054: Assignee: Gengliang Wang (was: Apache Spark) > Restore the error handling syntax of

[jira] [Created] (SPARK-40054) Restore the error handling syntax of try_cast()

2022-08-11 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-40054: -- Summary: Restore the error handling syntax of try_cast() Key: SPARK-40054 URL: https://issues.apache.org/jira/browse/SPARK-40054 Project: Spark Issue

[jira] [Created] (SPARK-40053) HiveExternalCatalogVersionsSuite will test all spark versions and aborted when Python 2.7 is used

2022-08-11 Thread Yang Jie (Jira)
Yang Jie created SPARK-40053: Summary: HiveExternalCatalogVersionsSuite will test all spark versions and aborted when Python 2.7 is used Key: SPARK-40053 URL: https://issues.apache.org/jira/browse/SPARK-40053

[jira] [Resolved] (SPARK-39635) Custom driver metrics for Datasource v2

2022-08-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-39635. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37205

[jira] [Assigned] (SPARK-39635) Custom driver metrics for Datasource v2

2022-08-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-39635: --- Assignee: Karuppayya > Custom driver metrics for Datasource v2 >

[jira] [Commented] (SPARK-40052) Handle direct byte buffers in VectorizedDeltaBinaryPackedReader

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578784#comment-17578784 ] Apache Spark commented on SPARK-40052: -- User 'sadikovi' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40052) Handle direct byte buffers in VectorizedDeltaBinaryPackedReader

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40052: Assignee: Apache Spark > Handle direct byte buffers in VectorizedDeltaBinaryPackedReader

[jira] [Assigned] (SPARK-40052) Handle direct byte buffers in VectorizedDeltaBinaryPackedReader

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40052: Assignee: (was: Apache Spark) > Handle direct byte buffers in

[jira] [Commented] (SPARK-40052) Handle direct byte buffers in VectorizedDeltaBinaryPackedReader

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578783#comment-17578783 ] Apache Spark commented on SPARK-40052: -- User 'sadikovi' has created a pull request for this issue:

[jira] [Created] (SPARK-40052) Handle direct byte buffers in VectorizedDeltaBinaryPackedReader

2022-08-11 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-40052: Summary: Handle direct byte buffers in VectorizedDeltaBinaryPackedReader Key: SPARK-40052 URL: https://issues.apache.org/jira/browse/SPARK-40052 Project: Spark

[jira] [Created] (SPARK-40051) Make pyspark.sql.catalog examples self-contained

2022-08-11 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-40051: Summary: Make pyspark.sql.catalog examples self-contained Key: SPARK-40051 URL: https://issues.apache.org/jira/browse/SPARK-40051 Project: Spark Issue Type:

[jira] [Updated] (SPARK-40050) Eliminate sort if parent is local limit

2022-08-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40050: Description: It seems we can remove Sort operator: {code:scala} val projectPlan =

[jira] [Created] (SPARK-40050) Eliminate sort if parent is local limit

2022-08-11 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-40050: --- Summary: Eliminate sort if parent is local limit Key: SPARK-40050 URL: https://issues.apache.org/jira/browse/SPARK-40050 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-39985) Test DEFAULT column values with DataFrames

2022-08-11 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-39985: -- Assignee: Daniel > Test DEFAULT column values with DataFrames >

[jira] [Resolved] (SPARK-39985) Test DEFAULT column values with DataFrames

2022-08-11 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-39985. Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37423

[jira] [Resolved] (SPARK-40029) Make pyspark.sql.types examples self-contained

2022-08-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40029. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37465

[jira] [Assigned] (SPARK-40029) Make pyspark.sql.types examples self-contained

2022-08-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40029: Assignee: Yikun Jiang > Make pyspark.sql.types examples self-contained >

[jira] [Created] (SPARK-40049) Add adaptive plan case in ReplaceNullWithFalseInPredicateEndToEndSuite

2022-08-11 Thread Kazuyuki Tanimura (Jira)
Kazuyuki Tanimura created SPARK-40049: - Summary: Add adaptive plan case in ReplaceNullWithFalseInPredicateEndToEndSuite Key: SPARK-40049 URL: https://issues.apache.org/jira/browse/SPARK-40049

[jira] [Commented] (SPARK-40047) Exclude unused xalan transitive dependency from htmlunit

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578705#comment-17578705 ] Dongjoon Hyun commented on SPARK-40047: --- Hi, [~LuciferYang]. Apache Spark is not affected by that

[jira] [Updated] (SPARK-40047) Exclude unused xalan transitive dependency from htmlunit

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40047: -- Issue Type: Task (was: Improvement) > Exclude unused xalan transitive dependency from

[jira] [Resolved] (SPARK-40047) Exclude unused xalan transitive dependency from htmlunit

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-40047. --- Fix Version/s: 3.3.1 3.2.3 3.4.0 Resolution:

[jira] [Assigned] (SPARK-40047) Exclude unused xalan transitive dependency from htmlunit

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-40047: - Assignee: Yang Jie > Exclude unused xalan transitive dependency from htmlunit >

[jira] [Updated] (SPARK-40047) Exclude unused xalan transitive dependency from htmlunit

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40047: -- Component/s: Tests (was: Build) > Exclude unused xalan transitive

[jira] [Updated] (SPARK-40047) Exclude unused xalan from htmlunit

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40047: -- Summary: Exclude unused xalan from htmlunit (was: htmlunit's delivery dependency xalan has a

[jira] [Updated] (SPARK-40047) Exclude unused xalan transitive dependency from htmlunit

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40047: -- Summary: Exclude unused xalan transitive dependency from htmlunit (was: Exclude unused xalan

[jira] [Resolved] (SPARK-39927) Upgrade Avro to version 1.11.1

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39927. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37352

[jira] [Assigned] (SPARK-39927) Upgrade Avro to version 1.11.1

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39927: - Assignee: Ismaël Mejía > Upgrade Avro to version 1.11.1 >

[jira] [Updated] (SPARK-36176) Expose tableExists in pyspark.sql.catalog

2022-08-11 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-36176: --- Fix Version/s: 3.3.0 (was: 3.2.0) > Expose tableExists in

[jira] [Commented] (SPARK-36176) Expose tableExists in pyspark.sql.catalog

2022-08-11 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578592#comment-17578592 ] Josh Rosen commented on SPARK-36176: Changed the Fix Version on JIRA: this landed in 3.3.0, not

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

2022-08-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Kai-Hsun Chen (was: Kai-Hsun Chen) > Improve LaunchTask process

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

2022-08-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Kai-Hsun Chen (was: Mridul Muralidharan) > Improve LaunchTask

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

2022-08-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Mridul Muralidharan > Improve LaunchTask process to avoid Stage

[jira] [Resolved] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

2022-08-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-39955. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-38921) Use k8s-client to create queue resource in Volcano IT

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38921. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36219

[jira] [Assigned] (SPARK-38921) Use k8s-client to create queue resource in Volcano IT

2022-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38921: - Assignee: Yikun Jiang > Use k8s-client to create queue resource in Volcano IT >

[jira] [Resolved] (SPARK-39966) Use V2 Filter in SupportsDelete

2022-08-11 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-39966. Fix Version/s: 3.4.0 Assignee: Huaxin Gao (was: Apache Spark) Resolution: Fixed

[jira] [Commented] (SPARK-40048) Partitions are traversed multiple times invalidating Accumulator consistency

2022-08-11 Thread sam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578447#comment-17578447 ] sam commented on SPARK-40048: - I've found a very dodgy hack around with this: ``` def

[jira] [Updated] (SPARK-40048) Partitions are traversed multiple times invalidating Accumulator consistency

2022-08-11 Thread sam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam updated SPARK-40048: Description: We are trying to use Accumulators to count RDDs without having to force `.count()` on them for

[jira] [Commented] (SPARK-37690) Recursive view `df` detected (cycle: `df` -> `df`)

2022-08-11 Thread Z. S. (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578422#comment-17578422 ] Z. S. commented on SPARK-37690: --- If anyone figures out a workaround please let us know..  > Recursive

[jira] [Assigned] (SPARK-40027) Make pyspark.sql.streaming.readwriter examples self-contained

2022-08-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40027: Assignee: Hyukjin Kwon > Make pyspark.sql.streaming.readwriter examples self-contained >

[jira] [Resolved] (SPARK-40027) Make pyspark.sql.streaming.readwriter examples self-contained

2022-08-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40027. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37461

[jira] [Commented] (SPARK-40014) Support cast of decimals to ANSI intervals

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578392#comment-17578392 ] Apache Spark commented on SPARK-40014: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40014) Support cast of decimals to ANSI intervals

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40014: Assignee: Apache Spark > Support cast of decimals to ANSI intervals >

[jira] [Assigned] (SPARK-40014) Support cast of decimals to ANSI intervals

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40014: Assignee: (was: Apache Spark) > Support cast of decimals to ANSI intervals >

[jira] [Commented] (SPARK-40014) Support cast of decimals to ANSI intervals

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578391#comment-17578391 ] Apache Spark commented on SPARK-40014: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40042) Make pyspark.sql.streaming.query examples self-contained

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40042: Assignee: (was: Apache Spark) > Make pyspark.sql.streaming.query examples

[jira] [Assigned] (SPARK-40042) Make pyspark.sql.streaming.query examples self-contained

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40042: Assignee: Apache Spark > Make pyspark.sql.streaming.query examples self-contained >

[jira] [Commented] (SPARK-40042) Make pyspark.sql.streaming.query examples self-contained

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578373#comment-17578373 ] Apache Spark commented on SPARK-40042: -- User 'dcoliversun' has created a pull request for this

[jira] [Commented] (SPARK-40042) Make pyspark.sql.streaming.query examples self-contained

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578372#comment-17578372 ] Apache Spark commented on SPARK-40042: -- User 'dcoliversun' has created a pull request for this

[jira] [Commented] (SPARK-40048) Partitions are traversed multiple times invalidating Accumulator consistency

2022-08-11 Thread sam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578369#comment-17578369 ] sam commented on SPARK-40048: - Also confirmed no eviction seems to be happening with

[jira] [Updated] (SPARK-40021) `Cancel workflow` can not cancel some jobs

2022-08-11 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-40021: -- Component/s: (was: Tests) > `Cancel workflow` can not cancel some jobs >

[jira] [Updated] (SPARK-40048) Partitions are traversed multiple times invalidating Accumulator consistency

2022-08-11 Thread sam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam updated SPARK-40048: Description: We are trying to use Accumulators to count RDDs without having to force `.count()` on them for

[jira] [Commented] (SPARK-40039) Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

2022-08-11 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578334#comment-17578334 ] Steve Loughran commented on SPARK-40039: doesn't actualy use MPU; if you haven't uploaded any

[jira] [Commented] (SPARK-40047) htmlunit's delivery dependency xalan has a CVE vulnerability

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578333#comment-17578333 ] Apache Spark commented on SPARK-40047: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-40047) htmlunit's delivery dependency xalan has a CVE vulnerability

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578332#comment-17578332 ] Apache Spark commented on SPARK-40047: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-40047) htmlunit's delivery dependency xalan has a CVE vulnerability

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40047: Assignee: (was: Apache Spark) > htmlunit's delivery dependency xalan has a CVE

[jira] [Assigned] (SPARK-40047) htmlunit's delivery dependency xalan has a CVE vulnerability

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40047: Assignee: Apache Spark > htmlunit's delivery dependency xalan has a CVE vulnerability >

[jira] [Created] (SPARK-40048) Partitions are traversed multiple times invalidating Accumulator consistency

2022-08-11 Thread sam (Jira)
sam created SPARK-40048: --- Summary: Partitions are traversed multiple times invalidating Accumulator consistency Key: SPARK-40048 URL: https://issues.apache.org/jira/browse/SPARK-40048 Project: Spark

[jira] [Created] (SPARK-40047) htmlunit's delivery dependency xalan has a CVE vulnerability

2022-08-11 Thread Yang Jie (Jira)
Yang Jie created SPARK-40047: Summary: htmlunit's delivery dependency xalan has a CVE vulnerability Key: SPARK-40047 URL: https://issues.apache.org/jira/browse/SPARK-40047 Project: Spark Issue

[jira] [Assigned] (SPARK-40046) Use Jackson instead of json4s to serialize `RocksDBMetrics`

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40046: Assignee: Apache Spark > Use Jackson instead of json4s to serialize `RocksDBMetrics` >

[jira] [Assigned] (SPARK-40046) Use Jackson instead of json4s to serialize `RocksDBMetrics`

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40046: Assignee: (was: Apache Spark) > Use Jackson instead of json4s to serialize

[jira] [Commented] (SPARK-40046) Use Jackson instead of json4s to serialize `RocksDBMetrics`

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578297#comment-17578297 ] Apache Spark commented on SPARK-40046: -- User 'LuciferYang' has created a pull request for this

[jira] [Updated] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-40045: --- Description: {code:java} select id, data FROM testcat.ns1.ns2.table where id =2 and md5(data) =

[jira] [Created] (SPARK-40046) Use Jackson instead of json4s to serialize `RocksDBMetrics`

2022-08-11 Thread Yang Jie (Jira)
Yang Jie created SPARK-40046: Summary: Use Jackson instead of json4s to serialize `RocksDBMetrics` Key: SPARK-40046 URL: https://issues.apache.org/jira/browse/SPARK-40046 Project: Spark Issue

[jira] [Commented] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578290#comment-17578290 ] Apache Spark commented on SPARK-40045: -- User 'caican00' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40045: Assignee: Apache Spark > The order of filtering predicates is not reasonable >

[jira] [Assigned] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40045: Assignee: (was: Apache Spark) > The order of filtering predicates is not reasonable

[jira] [Commented] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578289#comment-17578289 ] Apache Spark commented on SPARK-40045: -- User 'caican00' has created a pull request for this issue:

[jira] [Updated] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-40045: --- Description: {code:java} select id, data FROM testcat.ns1.ns2.table where id =2 and md5(data) =

[jira] [Updated] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-40045: --- Description: {code:java} select id, data FROM testcat.ns1.ns2.table where id =2 and md5(data) =

[jira] [Commented] (SPARK-40021) `Cancel workflow` can not cancel some jobs

2022-08-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578278#comment-17578278 ] Yikun Jiang commented on SPARK-40021: - FYI: [https://github.com/orgs/community/discussions/24932]  

[jira] [Updated] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-40045: --- Description: {code:java} select id, data FROM testcat.ns1.ns2.table where id =2 and md5(data) =

[jira] [Updated] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-40045: --- Description: {code:java} select id, data FROM testcat.ns1.ns2.table where id =2 and md5(data) =

[jira] [Updated] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-40045: --- Description: {code:java} select id, data FROM testcat.ns1.ns2.table where id =2 and md5(data) =

[jira] [Created] (SPARK-40045) The order of filtering predicates is not reasonable

2022-08-11 Thread caican (Jira)
caican created SPARK-40045: -- Summary: The order of filtering predicates is not reasonable Key: SPARK-40045 URL: https://issues.apache.org/jira/browse/SPARK-40045 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-40043) Document DataStreamWriter.toTable and DataStreamReader.table

2022-08-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40043. -- Fix Version/s: 3.3.1 3.1.4 3.2.3

[jira] [Assigned] (SPARK-40043) Document DataStreamWriter.toTable and DataStreamReader.table

2022-08-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40043: Assignee: Hyukjin Kwon > Document DataStreamWriter.toTable and DataStreamReader.table >