[jira] [Commented] (SPARK-38911) 'test 1 resource profile' throws exception when running it in IDEA separately

2022-04-14 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522646#comment-17522646 ] Bobby Wang commented on SPARK-38911: [~tgraves]  Could you help to check that > 'test 1 resource

[jira] [Commented] (SPARK-38911) 'test 1 resource profile' throws exception when running it in IDEA separately

2022-04-14 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522645#comment-17522645 ] Bobby Wang commented on SPARK-38911: Just submited an PR for this issue

[jira] [Commented] (SPARK-38911) 'test 1 resource profile' throws exception when running it in IDEA separately

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522644#comment-17522644 ] Apache Spark commented on SPARK-38911: -- User 'wbo4958' has created a pull request for this issue:

[jira] [Commented] (SPARK-38911) 'test 1 resource profile' throws exception when running it in IDEA separately

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522643#comment-17522643 ] Apache Spark commented on SPARK-38911: -- User 'wbo4958' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38911) 'test 1 resource profile' throws exception when running it in IDEA separately

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38911: Assignee: Apache Spark > 'test 1 resource profile' throws exception when running it in

[jira] [Assigned] (SPARK-38911) 'test 1 resource profile' throws exception when running it in IDEA separately

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38911: Assignee: (was: Apache Spark) > 'test 1 resource profile' throws exception when

[jira] [Created] (SPARK-38911) 'test 1 resource profile' throws exception when running it in IDEA separately

2022-04-14 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-38911: -- Summary: 'test 1 resource profile' throws exception when running it in IDEA separately Key: SPARK-38911 URL: https://issues.apache.org/jira/browse/SPARK-38911 Project:

[jira] [Assigned] (SPARK-38910) Clean sparkStaging dir when WAIT_FOR_APP_COMPLETION is false too

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38910: Assignee: (was: Apache Spark) > Clean sparkStaging dir when WAIT_FOR_APP_COMPLETION

[jira] [Commented] (SPARK-38910) Clean sparkStaging dir when WAIT_FOR_APP_COMPLETION is false too

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522638#comment-17522638 ] Apache Spark commented on SPARK-38910: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-38910) Clean sparkStaging dir when WAIT_FOR_APP_COMPLETION is false too

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38910: Assignee: Apache Spark > Clean sparkStaging dir when WAIT_FOR_APP_COMPLETION is false

[jira] [Created] (SPARK-38910) Clean sparkStaging dir when WAIT_FOR_APP_COMPLETION is false too

2022-04-14 Thread angerszhu (Jira)
angerszhu created SPARK-38910: - Summary: Clean sparkStaging dir when WAIT_FOR_APP_COMPLETION is false too Key: SPARK-38910 URL: https://issues.apache.org/jira/browse/SPARK-38910 Project: Spark

[jira] [Comment Edited] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522630#comment-17522630 ] Dongjoon Hyun edited comment on SPARK-37814 at 4/15/22 3:07 AM: BTW, one

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522630#comment-17522630 ] Dongjoon Hyun commented on SPARK-37814: --- BTW, one more tip to you, [~brentwritescode]. This JIRA

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522627#comment-17522627 ] Dongjoon Hyun commented on SPARK-37814: --- Hi, [~brentwritescode]. Thank you for the suggestion but

[jira] [Assigned] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38909: Assignee: Apache Spark > Encapsulate LevelDB used by ExternalShuffleBlockResolver and >

[jira] [Updated] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

2022-04-14 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-38909: - Description: {{ExternalShuffleBlockResolver}} and {{YarnShuffleService}}  use {{{}LevelDB directly{}}},

[jira] [Commented] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522622#comment-17522622 ] Apache Spark commented on SPARK-38909: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38909: Assignee: (was: Apache Spark) > Encapsulate LevelDB used by

[jira] [Created] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

2022-04-14 Thread Yang Jie (Jira)
Yang Jie created SPARK-38909: Summary: Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB Key: SPARK-38909 URL: https://issues.apache.org/jira/browse/SPARK-38909

[jira] [Commented] (SPARK-38908) Provide query context in runtime error of Casting from String to Number/Date/Timestamp/Boolean

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522613#comment-17522613 ] Apache Spark commented on SPARK-38908: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-38908) Provide query context in runtime error of Casting from String to Number/Date/Timestamp/Boolean

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38908: Assignee: Gengliang Wang (was: Apache Spark) > Provide query context in runtime error

[jira] [Assigned] (SPARK-38908) Provide query context in runtime error of Casting from String to Number/Date/Timestamp/Boolean

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38908: Assignee: Apache Spark (was: Gengliang Wang) > Provide query context in runtime error

[jira] [Created] (SPARK-38908) Provide query context in runtime error of Casting from String to Number/Date/Timestamp/Boolean

2022-04-14 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-38908: -- Summary: Provide query context in runtime error of Casting from String to Number/Date/Timestamp/Boolean Key: SPARK-38908 URL:

[jira] [Commented] (SPARK-38904) Low cost DataFrame schema swap util

2022-04-14 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522610#comment-17522610 ] Rafal Wojdyla commented on SPARK-38904: --- [~hyukjin.kwon] thanks for the comment, sounds good to

[jira] [Assigned] (SPARK-38907) Impl DataFrame.corrwith

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38907: Assignee: Apache Spark > Impl DataFrame.corrwith > --- > >

[jira] [Assigned] (SPARK-38907) Impl DataFrame.corrwith

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38907: Assignee: (was: Apache Spark) > Impl DataFrame.corrwith > --- >

[jira] [Commented] (SPARK-38907) Impl DataFrame.corrwith

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522608#comment-17522608 ] Apache Spark commented on SPARK-38907: -- User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-38907) Impl DataFrame.corrwith

2022-04-14 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-38907: Summary: Impl DataFrame.corrwith Key: SPARK-38907 URL: https://issues.apache.org/jira/browse/SPARK-38907 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-38902) cast as char/varchar result is string, not expect data type

2022-04-14 Thread YuanGuanhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YuanGuanhu updated SPARK-38902: --- Issue Type: Improvement (was: Bug) > cast as char/varchar result is string, not expect data type >

[jira] [Commented] (SPARK-38904) Low cost DataFrame schema swap util

2022-04-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522599#comment-17522599 ] Hyukjin Kwon commented on SPARK-38904: -- I think we should have an API like

[jira] [Assigned] (SPARK-38905) Upgrade ORC to 1.6.14

2022-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38905: - Assignee: Dongjoon Hyun > Upgrade ORC to 1.6.14 > - > >

[jira] [Resolved] (SPARK-38905) Upgrade ORC to 1.6.14

2022-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38905. --- Fix Version/s: 3.2.2 Resolution: Fixed Issue resolved by pull request 36204

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-04-14 Thread Brent (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522589#comment-17522589 ] Brent commented on SPARK-37814: --- [~kabhwan] [~dongjoon] I happened to notice your conversation about

[jira] [Resolved] (SPARK-38823) Incorrect result of dataset reduceGroups in java

2022-04-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38823. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 36183

[jira] [Assigned] (SPARK-38823) Incorrect result of dataset reduceGroups in java

2022-04-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38823: Assignee: Bruce Robbins > Incorrect result of dataset reduceGroups in java >

[jira] [Assigned] (SPARK-38898) Failed to build python docker images due to .cache not found

2022-04-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38898: Assignee: Yikun Jiang > Failed to build python docker images due to .cache not found >

[jira] [Resolved] (SPARK-38898) Failed to build python docker images due to .cache not found

2022-04-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38898. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36198

[jira] [Updated] (SPARK-36664) Log time spent waiting for cluster resources

2022-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36664: -- Affects Version/s: 3.4.0 (was: 3.2.0)

[jira] [Updated] (SPARK-36664) Log time spent waiting for cluster resources

2022-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36664: -- Target Version/s: (was: 3.3.0) > Log time spent waiting for cluster resources >

[jira] [Created] (SPARK-38906) Support joinWith() with watermarks

2022-04-14 Thread John P (Jira)
John P created SPARK-38906: -- Summary: Support joinWith() with watermarks Key: SPARK-38906 URL: https://issues.apache.org/jira/browse/SPARK-38906 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-37395) Inline type hint files for files in python/pyspark/ml

2022-04-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37395. Fix Version/s: 3.3.0 Resolution: Fixed > Inline type hint files for files

[jira] [Assigned] (SPARK-38905) Upgrade ORC to 1.6.14

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38905: Assignee: Apache Spark > Upgrade ORC to 1.6.14 > - > >

[jira] [Assigned] (SPARK-38905) Upgrade ORC to 1.6.14

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38905: Assignee: (was: Apache Spark) > Upgrade ORC to 1.6.14 > - > >

[jira] [Commented] (SPARK-38905) Upgrade ORC to 1.6.14

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522511#comment-17522511 ] Apache Spark commented on SPARK-38905: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-38905) Upgrade ORC to 1.6.14

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522512#comment-17522512 ] Apache Spark commented on SPARK-38905: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Created] (SPARK-38905) Upgrade ORC to 1.6.14

2022-04-14 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-38905: - Summary: Upgrade ORC to 1.6.14 Key: SPARK-38905 URL: https://issues.apache.org/jira/browse/SPARK-38905 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-37405) Inline type hints for python/pyspark/ml/feature.py

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522509#comment-17522509 ] Apache Spark commented on SPARK-37405: -- User 'zero323' has created a pull request for this issue:

[jira] [Resolved] (SPARK-37405) Inline type hints for python/pyspark/ml/feature.py

2022-04-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37405. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Commented] (SPARK-38667) Optimizer generates error when using inner join along with sequence

2022-04-14 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522500#comment-17522500 ] Bruce Robbins commented on SPARK-38667: --- Strangely, I cannot reproduce on 3.2.1 (or master). Maybe

[jira] [Updated] (SPARK-38904) Low cost DataFrame schema swap util

2022-04-14 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rafal Wojdyla updated SPARK-38904: -- Description: This question is related to [https://stackoverflow.com/a/37090151/1661491].

[jira] [Updated] (SPARK-38904) Low cost DataFrame schema swap util

2022-04-14 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rafal Wojdyla updated SPARK-38904: -- Description: This question is related to [https://stackoverflow.com/a/37090151/1661491].

[jira] [Updated] (SPARK-38904) Low cost DataFrame schema swap util

2022-04-14 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rafal Wojdyla updated SPARK-38904: -- Description: This question is related to [https://stackoverflow.com/a/37090151/1661491].

[jira] [Updated] (SPARK-38904) Low cost DataFrame schema swap util

2022-04-14 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rafal Wojdyla updated SPARK-38904: -- Description: This question is related to [https://stackoverflow.com/a/37090151/1661491].

[jira] [Updated] (SPARK-38904) Low cost DataFrame schema swap util

2022-04-14 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rafal Wojdyla updated SPARK-38904: -- Description: This question is related to [https://stackoverflow.com/a/37090151/1661491].

[jira] [Created] (SPARK-38904) Low cost DataFrame schema swap util

2022-04-14 Thread Rafal Wojdyla (Jira)
Rafal Wojdyla created SPARK-38904: - Summary: Low cost DataFrame schema swap util Key: SPARK-38904 URL: https://issues.apache.org/jira/browse/SPARK-38904 Project: Spark Issue Type: New

[jira] [Assigned] (SPARK-38891) Skipping allocating vector for repetition & definition levels when possible

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38891: Assignee: (was: Apache Spark) > Skipping allocating vector for repetition &

[jira] [Commented] (SPARK-38891) Skipping allocating vector for repetition & definition levels when possible

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522458#comment-17522458 ] Apache Spark commented on SPARK-38891: -- User 'sunchao' has created a pull request for this issue:

[jira] [Commented] (SPARK-38891) Skipping allocating vector for repetition & definition levels when possible

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522460#comment-17522460 ] Apache Spark commented on SPARK-38891: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38891) Skipping allocating vector for repetition & definition levels when possible

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38891: Assignee: Apache Spark > Skipping allocating vector for repetition & definition levels

[jira] [Commented] (SPARK-38903) Implement `ignore_index` of `Series.sort_values` and `Series.sort_index`

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522422#comment-17522422 ] Apache Spark commented on SPARK-38903: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-38903) Implement `ignore_index` of `Series.sort_values` and `Series.sort_index`

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38903: Assignee: (was: Apache Spark) > Implement `ignore_index` of `Series.sort_values` and

[jira] [Commented] (SPARK-38903) Implement `ignore_index` of `Series.sort_values` and `Series.sort_index`

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522421#comment-17522421 ] Apache Spark commented on SPARK-38903: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-38903) Implement `ignore_index` of `Series.sort_values` and `Series.sort_index`

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38903: Assignee: Apache Spark > Implement `ignore_index` of `Series.sort_values` and

[jira] [Created] (SPARK-38903) Implement `ignore_index` of `Series.sort_values` and `Series.sort_index`

2022-04-14 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38903: Summary: Implement `ignore_index` of `Series.sort_values` and `Series.sort_index` Key: SPARK-38903 URL: https://issues.apache.org/jira/browse/SPARK-38903 Project:

[jira] [Commented] (SPARK-38462) Use error classes in org.apache.spark.executor

2022-04-14 Thread huangtengfei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522331#comment-17522331 ] huangtengfei commented on SPARK-38462: -- I am working on this. Thanks [~bozhang] > Use error

[jira] [Updated] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

2022-04-14 Thread Mark Khaitman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Khaitman updated SPARK-38881: -- Description: This relates to https://issues.apache.org/jira/browse/SPARK-27420 which was

[jira] [Commented] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522308#comment-17522308 ] Apache Spark commented on SPARK-38881: -- User 'mkman84' has created a pull request for this issue:

[jira] [Commented] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522307#comment-17522307 ] Apache Spark commented on SPARK-38881: -- User 'mkman84' has created a pull request for this issue:

[jira] [Commented] (SPARK-38888) Add `RocksDBProvider` similar to `LevelDBProvider`

2022-04-14 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522294#comment-17522294 ] Yang Jie commented on SPARK-3: -- I provide a [draft pr

[jira] [Resolved] (SPARK-38753) Move the tests for `WRITING_JOB_ABORTED` to QueryExecutionErrorsSuite

2022-04-14 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-38753. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36196

[jira] [Assigned] (SPARK-38753) Move the tests for `WRITING_JOB_ABORTED` to QueryExecutionErrorsSuite

2022-04-14 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-38753: Assignee: Max Gekk > Move the tests for `WRITING_JOB_ABORTED` to QueryExecutionErrorsSuite >

[jira] [Commented] (SPARK-38722) Test the error class: CAST_CAUSES_OVERFLOW

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522263#comment-17522263 ] Apache Spark commented on SPARK-38722: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-38722) Test the error class: CAST_CAUSES_OVERFLOW

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38722: Assignee: Apache Spark > Test the error class: CAST_CAUSES_OVERFLOW >

[jira] [Assigned] (SPARK-38722) Test the error class: CAST_CAUSES_OVERFLOW

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38722: Assignee: (was: Apache Spark) > Test the error class: CAST_CAUSES_OVERFLOW >

[jira] [Created] (SPARK-38902) cast as char/varchar result is string, not expect data type

2022-04-14 Thread YuanGuanhu (Jira)
YuanGuanhu created SPARK-38902: -- Summary: cast as char/varchar result is string, not expect data type Key: SPARK-38902 URL: https://issues.apache.org/jira/browse/SPARK-38902 Project: Spark

[jira] [Created] (SPARK-38901) Support for more misc functions to push down to data source

2022-04-14 Thread Zhixiong Chen (Jira)
Zhixiong Chen created SPARK-38901: - Summary: Support for more misc functions to push down to data source Key: SPARK-38901 URL: https://issues.apache.org/jira/browse/SPARK-38901 Project: Spark

[jira] [Created] (SPARK-38900) Support for more collection functions to push down to data source

2022-04-14 Thread Zhixiong Chen (Jira)
Zhixiong Chen created SPARK-38900: - Summary: Support for more collection functions to push down to data source Key: SPARK-38900 URL: https://issues.apache.org/jira/browse/SPARK-38900 Project: Spark

[jira] [Created] (SPARK-38899) Support for more datetime functions to push down to data source

2022-04-14 Thread Zhixiong Chen (Jira)
Zhixiong Chen created SPARK-38899: - Summary: Support for more datetime functions to push down to data source Key: SPARK-38899 URL: https://issues.apache.org/jira/browse/SPARK-38899 Project: Spark

[jira] [Assigned] (SPARK-38898) Failed to build python docker images due to .cache not found

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38898: Assignee: Apache Spark > Failed to build python docker images due to .cache not found >

[jira] [Commented] (SPARK-38898) Failed to build python docker images due to .cache not found

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522203#comment-17522203 ] Apache Spark commented on SPARK-38898: -- User 'Yikun' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38898) Failed to build python docker images due to .cache not found

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38898: Assignee: (was: Apache Spark) > Failed to build python docker images due to .cache

[jira] [Commented] (SPARK-38898) Failed to build python docker images due to .cache not found

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522201#comment-17522201 ] Apache Spark commented on SPARK-38898: -- User 'Yikun' has created a pull request for this issue:

[jira] [Created] (SPARK-38898) Failed to build python docker images due to .cache not found

2022-04-14 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38898: --- Summary: Failed to build python docker images due to .cache not found Key: SPARK-38898 URL: https://issues.apache.org/jira/browse/SPARK-38898 Project: Spark

[jira] [Updated] (SPARK-38897) Support for more string functions to push down to data source

2022-04-14 Thread Zhixiong Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhixiong Chen updated SPARK-38897: -- Parent: SPARK-38852 Issue Type: Sub-task (was: Improvement) > Support for more

[jira] [Updated] (SPARK-38897) Support for more string functions to push down to data source

2022-04-14 Thread Zhixiong Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhixiong Chen updated SPARK-38897: -- Summary: Support for more string functions to push down to data source (was: More string

[jira] [Created] (SPARK-38897) More string functions support to pushdown to data source

2022-04-14 Thread Zhixiong Chen (Jira)
Zhixiong Chen created SPARK-38897: - Summary: More string functions support to pushdown to data source Key: SPARK-38897 URL: https://issues.apache.org/jira/browse/SPARK-38897 Project: Spark

[jira] [Commented] (SPARK-38438) Can't update spark.jars.packages on existing global/default context

2022-04-14 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522176#comment-17522176 ] Rafal Wojdyla commented on SPARK-38438: --- For posterity see the context in

[jira] [Assigned] (SPARK-38747) Move the tests for `PARSE_SYNTAX_ERROR` to QueryParsingErrorsSuite

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38747: Assignee: Apache Spark > Move the tests for `PARSE_SYNTAX_ERROR` to

[jira] [Assigned] (SPARK-38747) Move the tests for `PARSE_SYNTAX_ERROR` to QueryParsingErrorsSuite

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38747: Assignee: (was: Apache Spark) > Move the tests for `PARSE_SYNTAX_ERROR` to

[jira] [Commented] (SPARK-38747) Move the tests for `PARSE_SYNTAX_ERROR` to QueryParsingErrorsSuite

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522169#comment-17522169 ] Apache Spark commented on SPARK-38747: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-38894) Exclude pyspark.cloudpickle in test coverage report

2022-04-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38894: Assignee: Hyukjin Kwon > Exclude pyspark.cloudpickle in test coverage report >

[jira] [Resolved] (SPARK-38894) Exclude pyspark.cloudpickle in test coverage report

2022-04-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38894. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36191

[jira] [Assigned] (SPARK-38893) Test SourceProgress in PySpark

2022-04-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38893: Assignee: Hyukjin Kwon > Test SourceProgress in PySpark > --

[jira] [Resolved] (SPARK-38893) Test SourceProgress in PySpark

2022-04-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38893. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36190

[jira] [Commented] (SPARK-38753) Move the tests for `WRITING_JOB_ABORTED` to QueryExecutionErrorsSuite

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522145#comment-17522145 ] Apache Spark commented on SPARK-38753: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-38753) Move the tests for `WRITING_JOB_ABORTED` to QueryExecutionErrorsSuite

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522144#comment-17522144 ] Apache Spark commented on SPARK-38753: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38753) Move the tests for `WRITING_JOB_ABORTED` to QueryExecutionErrorsSuite

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38753: Assignee: Apache Spark > Move the tests for `WRITING_JOB_ABORTED` to

[jira] [Assigned] (SPARK-38753) Move the tests for `WRITING_JOB_ABORTED` to QueryExecutionErrorsSuite

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38753: Assignee: (was: Apache Spark) > Move the tests for `WRITING_JOB_ABORTED` to

[jira] [Assigned] (SPARK-38896) Use tryWithResource to recycling KVStoreIterator and remove finalize() from LevelDB/RocksDB

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38896: Assignee: Apache Spark > Use tryWithResource to recycling KVStoreIterator and remove

[jira] [Commented] (SPARK-38896) Use tryWithResource to recycling KVStoreIterator and remove finalize() from LevelDB/RocksDB

2022-04-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522123#comment-17522123 ] Apache Spark commented on SPARK-38896: -- User 'LuciferYang' has created a pull request for this

  1   2   >