[jira] [Updated] (SPARK-39921) SkewJoin--Stream side skew in BroadcastJoin

2022-07-28 Thread wang-zhun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wang-zhun updated SPARK-39921: -- Description: There is data skew in BroadcastJoin. Since LocalShuffle cannot cover all cases, we can r

[jira] [Updated] (SPARK-39921) SkewJoin--Stream side skew in BroadcastJoin

2022-07-28 Thread wang-zhun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wang-zhun updated SPARK-39921: -- Attachment: 1.png 2.png 3.png 4.png > SkewJoin--Stream

[jira] [Created] (SPARK-39922) Add a microbenchmark for `Utils#getIteratorSize`

2022-07-28 Thread Yang Jie (Jira)
Yang Jie created SPARK-39922: Summary: Add a microbenchmark for `Utils#getIteratorSize` Key: SPARK-39922 URL: https://issues.apache.org/jira/browse/SPARK-39922 Project: Spark Issue Type: Improvem

[jira] [Created] (SPARK-39921) SkewJoin--Stream side skew in BroadcastJoin

2022-07-28 Thread wang-zhun (Jira)
wang-zhun created SPARK-39921: - Summary: SkewJoin--Stream side skew in BroadcastJoin Key: SPARK-39921 URL: https://issues.apache.org/jira/browse/SPARK-39921 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-37210) An error occurred while concurrently writing to different static partitions

2022-07-28 Thread Zhen Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572754#comment-17572754 ] Zhen Wang edited comment on SPARK-37210 at 7/29/22 6:27 AM:

[jira] [Updated] (SPARK-39920) Interrupt on running spark job doesn't kill underneath spark job

2022-07-28 Thread Neha Singla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Singla updated SPARK-39920: Priority: Blocker (was: Major) > Interrupt on running spark job doesn't kill underneath spark job

[jira] [Commented] (SPARK-37210) An error occurred while concurrently writing to different static partitions

2022-07-28 Thread Zhen Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572754#comment-17572754 ] Zhen Wang commented on SPARK-37210: --- > BTW, could you please reopen the PR freshly? I'

[jira] [Updated] (SPARK-39920) Interrupt on running spark job doesn't kill underneath spark job

2022-07-28 Thread Neha Singla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Singla updated SPARK-39920: Attachment: interrupt_spark_job.ipynb > Interrupt on running spark job doesn't kill underneath spa

[jira] [Created] (SPARK-39920) Interrupt on running spark job doesn't kill underneath spark job

2022-07-28 Thread Neha Singla (Jira)
Neha Singla created SPARK-39920: --- Summary: Interrupt on running spark job doesn't kill underneath spark job Key: SPARK-39920 URL: https://issues.apache.org/jira/browse/SPARK-39920 Project: Spark

[jira] [Created] (SPARK-39919) Hitting Interrupt when no cell is running kills jupyter kernel

2022-07-28 Thread Neha Singla (Jira)
Neha Singla created SPARK-39919: --- Summary: Hitting Interrupt when no cell is running kills jupyter kernel Key: SPARK-39919 URL: https://issues.apache.org/jira/browse/SPARK-39919 Project: Spark

[jira] [Assigned] (SPARK-39918) Replace the wording "un-comparable" with "incomparable"

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39918: Assignee: Apache Spark (was: Gengliang Wang) > Replace the wording "un-comparable" with

[jira] [Commented] (SPARK-39918) Replace the wording "un-comparable" with "incomparable"

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572747#comment-17572747 ] Apache Spark commented on SPARK-39918: -- User 'gengliangwang' has created a pull req

[jira] [Assigned] (SPARK-39918) Replace the wording "un-comparable" with "incomparable"

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39918: Assignee: Gengliang Wang (was: Apache Spark) > Replace the wording "un-comparable" with

[jira] [Created] (SPARK-39918) Replace the wording "un-comparable" with "incomparable"

2022-07-28 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-39918: -- Summary: Replace the wording "un-comparable" with "incomparable" Key: SPARK-39918 URL: https://issues.apache.org/jira/browse/SPARK-39918 Project: Spark I

[jira] [Resolved] (SPARK-39905) Remove checkErrorClass()

2022-07-28 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-39905. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37322 [https://github.com

[jira] [Assigned] (SPARK-39917) Use different error classes for numeric/interval arithmetic overflow

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39917: Assignee: Apache Spark (was: Gengliang Wang) > Use different error classes for numeric/i

[jira] [Assigned] (SPARK-39917) Use different error classes for numeric/interval arithmetic overflow

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39917: Assignee: Gengliang Wang (was: Apache Spark) > Use different error classes for numeric/i

[jira] [Commented] (SPARK-39917) Use different error classes for numeric/interval arithmetic overflow

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572743#comment-17572743 ] Apache Spark commented on SPARK-39917: -- User 'gengliangwang' has created a pull req

[jira] [Created] (SPARK-39917) Use different error classes for numeric/interval arithmetic overflow

2022-07-28 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-39917: -- Summary: Use different error classes for numeric/interval arithmetic overflow Key: SPARK-39917 URL: https://issues.apache.org/jira/browse/SPARK-39917 Project: Spa

[jira] [Assigned] (SPARK-39906) Eliminate build warnings - 'sbt 0.13 shell syntax is deprecated; use slash syntax instead'

2022-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-39906: Assignee: BingKun Pan > Eliminate build warnings - 'sbt 0.13 shell syntax is deprecated;

[jira] [Resolved] (SPARK-39906) Eliminate build warnings - 'sbt 0.13 shell syntax is deprecated; use slash syntax instead'

2022-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-39906. -- Resolution: Fixed Issue resolved by pull request 37326 [https://github.com/apache/spark/pull/3

[jira] [Assigned] (SPARK-39916) Merge SchemaUtils from mlib to SQL

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39916: Assignee: (was: Apache Spark) > Merge SchemaUtils from mlib to SQL >

[jira] [Commented] (SPARK-39916) Merge SchemaUtils from mlib to SQL

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572726#comment-17572726 ] Apache Spark commented on SPARK-39916: -- User 'deshanxiao' has created a pull reques

[jira] [Assigned] (SPARK-39916) Merge SchemaUtils from mlib to SQL

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39916: Assignee: Apache Spark > Merge SchemaUtils from mlib to SQL > ---

[jira] [Updated] (SPARK-39916) Merge SchemaUtils from mlib to SQL

2022-07-28 Thread deshanxiao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deshanxiao updated SPARK-39916: --- Description: Today we have two SchemaUtils: SQL SchemaUtils and mllib SchemaUtils. the SchemaUtils o

[jira] [Updated] (SPARK-39916) Merge SchemaUtils from mlib to SQL

2022-07-28 Thread deshanxiao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deshanxiao updated SPARK-39916: --- Description: Today we have two SchemaUtils: SQL SchemaUtils and mllib SchemaUtils. the SchemaUtils o

[jira] [Created] (SPARK-39916) Merge SchemaUtils from mlib to SQL

2022-07-28 Thread deshanxiao (Jira)
deshanxiao created SPARK-39916: -- Summary: Merge SchemaUtils from mlib to SQL Key: SPARK-39916 URL: https://issues.apache.org/jira/browse/SPARK-39916 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-39862) Fix bug in existence DEFAULT value lookups for V2 data sources

2022-07-28 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-39862: -- Assignee: Daniel > Fix bug in existence DEFAULT value lookups for V2 data sources > -

[jira] [Resolved] (SPARK-39862) Fix bug in existence DEFAULT value lookups for V2 data sources

2022-07-28 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-39862. Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37280 [https:

[jira] [Commented] (SPARK-39915) Dataset.repartition(N) may not create N partitions

2022-07-28 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572685#comment-17572685 ] Shixiong Zhu commented on SPARK-39915: -- cc [~cloud_fan]  > Dataset.repartition(N)

[jira] [Created] (SPARK-39915) Dataset.repartition(N) may not create N partitions

2022-07-28 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-39915: Summary: Dataset.repartition(N) may not create N partitions Key: SPARK-39915 URL: https://issues.apache.org/jira/browse/SPARK-39915 Project: Spark Issue Type

[jira] [Commented] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2022-07-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572634#comment-17572634 ] Dongjoon Hyun commented on SPARK-34788: --- No, what I meant is ensuring enough disk

[jira] [Commented] (SPARK-39895) pyspark drop doesn't accept *cols

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572623#comment-17572623 ] Apache Spark commented on SPARK-39895: -- User 'santosh-d3vpl3x' has created a pull r

[jira] [Commented] (SPARK-39887) Expression transform error

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572615#comment-17572615 ] Apache Spark commented on SPARK-39887: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-39887) Expression transform error

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572613#comment-17572613 ] Apache Spark commented on SPARK-39887: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-39895) pyspark drop doesn't accept *cols

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572601#comment-17572601 ] Apache Spark commented on SPARK-39895: -- User 'santosh-d3vpl3x' has created a pull r

[jira] [Assigned] (SPARK-39895) pyspark drop doesn't accept *cols

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39895: Assignee: Apache Spark > pyspark drop doesn't accept *cols > ---

[jira] [Assigned] (SPARK-39895) pyspark drop doesn't accept *cols

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39895: Assignee: (was: Apache Spark) > pyspark drop doesn't accept *cols >

[jira] [Commented] (SPARK-39895) pyspark drop doesn't accept *cols

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572600#comment-17572600 ] Apache Spark commented on SPARK-39895: -- User 'santosh-d3vpl3x' has created a pull r

[jira] [Resolved] (SPARK-39865) Show proper error messages on the overflow errors of table insert

2022-07-28 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-39865. Fix Version/s: 3.3.1 Resolution: Fixed Issue resolved by pull request 37311 [https:

[jira] [Assigned] (SPARK-39914) Add DS V2 Filter to V1 Filter conversion

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39914: Assignee: (was: Apache Spark) > Add DS V2 Filter to V1 Filter conversion > --

[jira] [Assigned] (SPARK-39914) Add DS V2 Filter to V1 Filter conversion

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39914: Assignee: Apache Spark > Add DS V2 Filter to V1 Filter conversion > -

[jira] [Commented] (SPARK-39914) Add DS V2 Filter to V1 Filter conversion

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572581#comment-17572581 ] Apache Spark commented on SPARK-39914: -- User 'huaxingao' has created a pull request

[jira] [Updated] (SPARK-39914) Add DS V2 Filter to V1 Filter conversion

2022-07-28 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao updated SPARK-39914: --- Summary: Add DS V2 Filter to V1 Filter conversion (was: Add DS V2 Filter to V2 Filter conversion)

[jira] [Created] (SPARK-39914) Add DS V2 Filter to V2 Filter conversion

2022-07-28 Thread Huaxin Gao (Jira)
Huaxin Gao created SPARK-39914: -- Summary: Add DS V2 Filter to V2 Filter conversion Key: SPARK-39914 URL: https://issues.apache.org/jira/browse/SPARK-39914 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-39743) Unable to set zstd compression level while writing parquet files

2022-07-28 Thread zhiming she (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572551#comment-17572551 ] zhiming she commented on SPARK-39743: - [~hyukjin.kwon]  As [~yeachan153]  said, is

[jira] [Commented] (SPARK-39743) Unable to set zstd compression level while writing parquet files

2022-07-28 Thread zhiming she (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572546#comment-17572546 ] zhiming she commented on SPARK-39743: - [~yeachan153]  Only some more important para

[jira] [Commented] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2022-07-28 Thread wenweijian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572536#comment-17572536 ] wenweijian commented on SPARK-34788: I can reproduce it.  If the performance impact

[jira] [Assigned] (SPARK-39913) Upgrade Arrow to 9.0.0

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39913: Assignee: (was: Apache Spark) > Upgrade Arrow to 9.0.0 > -- > >

[jira] [Commented] (SPARK-39913) Upgrade Arrow to 9.0.0

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572520#comment-17572520 ] Apache Spark commented on SPARK-39913: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-39913) Upgrade Arrow to 9.0.0

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39913: Assignee: Apache Spark > Upgrade Arrow to 9.0.0 > -- > >

[jira] [Created] (SPARK-39913) Upgrade Arrow to 9.0.0

2022-07-28 Thread Yang Jie (Jira)
Yang Jie created SPARK-39913: Summary: Upgrade Arrow to 9.0.0 Key: SPARK-39913 URL: https://issues.apache.org/jira/browse/SPARK-39913 Project: Spark Issue Type: Improvement Components:

[jira] [Assigned] (SPARK-39912) Refine CatalogImpl

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39912: Assignee: Apache Spark > Refine CatalogImpl > -- > > Key:

[jira] [Assigned] (SPARK-39912) Refine CatalogImpl

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39912: Assignee: (was: Apache Spark) > Refine CatalogImpl > -- > >

[jira] [Commented] (SPARK-39912) Refine CatalogImpl

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572508#comment-17572508 ] Apache Spark commented on SPARK-39912: -- User 'cloud-fan' has created a pull request

[jira] [Created] (SPARK-39912) Refine CatalogImpl

2022-07-28 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-39912: --- Summary: Refine CatalogImpl Key: SPARK-39912 URL: https://issues.apache.org/jira/browse/SPARK-39912 Project: Spark Issue Type: Improvement Components

[jira] [Commented] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2022-07-28 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572506#comment-17572506 ] wuyi commented on SPARK-34788: -- > Why don't you ensure enough disks from the beginning? On

[jira] [Commented] (SPARK-39901) Reconsider design of ignoreCorruptFiles feature

2022-07-28 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572502#comment-17572502 ] Wei Guo commented on SPARK-39901: - The `ignoreCorruptFiles` features in SQL(spark.sql.fi

[jira] [Commented] (SPARK-37210) An error occurred while concurrently writing to different static partitions

2022-07-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572497#comment-17572497 ] Dongjoon Hyun commented on SPARK-37210: --- BTW, could you please reopen the PR fresh

[jira] [Commented] (SPARK-37210) An error occurred while concurrently writing to different static partitions

2022-07-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572495#comment-17572495 ] Dongjoon Hyun commented on SPARK-37210: --- I'm not sure if the concurrent static INS

[jira] [Commented] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2022-07-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572493#comment-17572493 ] Dongjoon Hyun commented on SPARK-34788: --- Why don't you ensure enough disks from th

[jira] [Commented] (SPARK-39911) Optimize global Sort to RepartitionByExpression

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572488#comment-17572488 ] Apache Spark commented on SPARK-39911: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-10000) Consolidate storage and execution memory management

2022-07-28 Thread sam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572486#comment-17572486 ] sam commented on SPARK-1: - Is there any way to disable Spark from evicting rdd partition

[jira] [Assigned] (SPARK-39911) Optimize global Sort to RepartitionByExpression

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39911: Assignee: (was: Apache Spark) > Optimize global Sort to RepartitionByExpression > ---

[jira] [Assigned] (SPARK-39911) Optimize global Sort to RepartitionByExpression

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39911: Assignee: Apache Spark > Optimize global Sort to RepartitionByExpression > --

[jira] [Commented] (SPARK-39911) Optimize global Sort to RepartitionByExpression

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572487#comment-17572487 ] Apache Spark commented on SPARK-39911: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-14289) Support multiple eviction strategies for cached RDD partitions

2022-07-28 Thread sam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-14289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572485#comment-17572485 ] sam commented on SPARK-14289: - Is there any way to disable Spark from evicting rdd partition

[jira] [Commented] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2022-07-28 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572478#comment-17572478 ] wuyi commented on SPARK-34788: -- [~leo wen] are you able to reproduce in your environment? I

[jira] [Created] (SPARK-39911) Optimize global Sort to RepartitionByExpression

2022-07-28 Thread XiDuo You (Jira)
XiDuo You created SPARK-39911: - Summary: Optimize global Sort to RepartitionByExpression Key: SPARK-39911 URL: https://issues.apache.org/jira/browse/SPARK-39911 Project: Spark Issue Type: Improve

[jira] [Assigned] (SPARK-39528) Use V2 Filter in SupportsRuntimeFiltering

2022-07-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-39528: --- Assignee: Huaxin Gao > Use V2 Filter in SupportsRuntimeFiltering >

[jira] [Resolved] (SPARK-39528) Use V2 Filter in SupportsRuntimeFiltering

2022-07-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-39528. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36918 [https://gith

[jira] [Commented] (SPARK-39910) DataFrameReader API cannot read files from hadoop archives (.har)

2022-07-28 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-39910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572454#comment-17572454 ] Christophe Préaud commented on SPARK-39910: --- This is just a straightforward ex

[jira] [Updated] (SPARK-39910) DataFrameReader API cannot read files from hadoop archives (.har)

2022-07-28 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-39910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christophe Préaud updated SPARK-39910: -- Summary: DataFrameReader API cannot read files from hadoop archives (.har) (was: Data

[jira] [Updated] (SPARK-39910) DataFrameReader API cannot read files from hadoop archives(.har)

2022-07-28 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-39910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christophe Préaud updated SPARK-39910: -- Description: Reading a file from an hadoop archive using the DataFrameReader API retur

[jira] [Created] (SPARK-39910) DataFrameReader API cannot read files from hadoop archives(.har)

2022-07-28 Thread Jira
Christophe Préaud created SPARK-39910: - Summary: DataFrameReader API cannot read files from hadoop archives(.har) Key: SPARK-39910 URL: https://issues.apache.org/jira/browse/SPARK-39910 Project: S

[jira] [Commented] (SPARK-39832) regexp_replace should support column arguments

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572433#comment-17572433 ] Apache Spark commented on SPARK-39832: -- User 'physinet' has created a pull request

[jira] [Assigned] (SPARK-39832) regexp_replace should support column arguments

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39832: Assignee: (was: Apache Spark) > regexp_replace should support column arguments >

[jira] [Commented] (SPARK-39832) regexp_replace should support column arguments

2022-07-28 Thread Brian Schaefer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572432#comment-17572432 ] Brian Schaefer commented on SPARK-39832: Yes, I've started a draft PR here: http

[jira] [Assigned] (SPARK-39832) regexp_replace should support column arguments

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39832: Assignee: Apache Spark > regexp_replace should support column arguments > ---

[jira] [Updated] (SPARK-37210) An error occurred while concurrently writing to different static partitions

2022-07-28 Thread Zhen Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhen Wang updated SPARK-37210: -- Description: An error occurred while concurrently writing to different static partitions. For writing

[jira] [Commented] (SPARK-37210) An error occurred while concurrently writing to different static partitions

2022-07-28 Thread Zhen Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572408#comment-17572408 ] Zhen Wang commented on SPARK-37210: --- The concurrent writes in INSERT INTO mode seems t

[jira] [Commented] (SPARK-39909) Organize the check of push down information for JDBCV2Suite

2022-07-28 Thread Mars (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572362#comment-17572362 ] Mars commented on SPARK-39909: -- assign [~miracle] > Organize the check of push down infor

[jira] [Updated] (SPARK-39909) Organize the check of push down information for JDBCV2Suite

2022-07-28 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-39909: --- Description: Currently, JDBCV2Suite have many test cases check the push-down information looks not

[jira] [Created] (SPARK-39909) Organize the check of push down information for JDBCV2Suite

2022-07-28 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-39909: -- Summary: Organize the check of push down information for JDBCV2Suite Key: SPARK-39909 URL: https://issues.apache.org/jira/browse/SPARK-39909 Project: Spark Issu

[jira] [Created] (SPARK-39908) Translate common used aggregate functions for pushdown (non ANSI)

2022-07-28 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-39908: -- Summary: Translate common used aggregate functions for pushdown (non ANSI) Key: SPARK-39908 URL: https://issues.apache.org/jira/browse/SPARK-39908 Project: Spark

[jira] [Comment Edited] (SPARK-39743) Unable to set zstd compression level while writing parquet files

2022-07-28 Thread Yeachan Park (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572308#comment-17572308 ] Yeachan Park edited comment on SPARK-39743 at 7/28/22 8:29 AM: ---

[jira] [Comment Edited] (SPARK-39743) Unable to set zstd compression level while writing parquet files

2022-07-28 Thread Yeachan Park (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572308#comment-17572308 ] Yeachan Park edited comment on SPARK-39743 at 7/28/22 8:28 AM: ---

[jira] [Commented] (SPARK-39743) Unable to set zstd compression level while writing parquet files

2022-07-28 Thread Yeachan Park (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572308#comment-17572308 ] Yeachan Park commented on SPARK-39743: -- OK, thanks for the clarification. Should we

[jira] [Assigned] (SPARK-39907) Implement axis and skipna of Series.argmin

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39907: Assignee: (was: Apache Spark) > Implement axis and skipna of Series.argmin >

[jira] [Commented] (SPARK-39907) Implement axis and skipna of Series.argmin

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572293#comment-17572293 ] Apache Spark commented on SPARK-39907: -- User 'zhengruifeng' has created a pull requ

[jira] [Assigned] (SPARK-39907) Implement axis and skipna of Series.argmin

2022-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39907: Assignee: Apache Spark > Implement axis and skipna of Series.argmin > ---

[jira] [Created] (SPARK-39907) Implement axis and skipna of Series.argmin

2022-07-28 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-39907: - Summary: Implement axis and skipna of Series.argmin Key: SPARK-39907 URL: https://issues.apache.org/jira/browse/SPARK-39907 Project: Spark Issue Type: Sub-