[jira] [Resolved] (SPARK-40044) Incorrect target interval type in cast overflow errors

2022-08-10 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40044. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37470

[jira] [Assigned] (SPARK-40044) Incorrect target interval type in cast overflow errors

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40044: Assignee: Apache Spark (was: Max Gekk) > Incorrect target interval type in cast

[jira] [Assigned] (SPARK-40044) Incorrect target interval type in cast overflow errors

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40044: Assignee: Max Gekk (was: Apache Spark) > Incorrect target interval type in cast

[jira] [Commented] (SPARK-40044) Incorrect target interval type in cast overflow errors

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578261#comment-17578261 ] Apache Spark commented on SPARK-40044: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Created] (SPARK-40044) Incorrect target interval type in cast overflow errors

2022-08-10 Thread Max Gekk (Jira)
Max Gekk created SPARK-40044: Summary: Incorrect target interval type in cast overflow errors Key: SPARK-40044 URL: https://issues.apache.org/jira/browse/SPARK-40044 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-40041) Add Document Parameters for pyspark.sql.window

2022-08-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40041: Assignee: Qian Sun > Add Document Parameters for pyspark.sql.window >

[jira] [Resolved] (SPARK-40041) Add Document Parameters for pyspark.sql.window

2022-08-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40041. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37476

[jira] [Assigned] (SPARK-40020) centralize the code of qualifying identifiers in SessionCatalog

2022-08-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40020: --- Assignee: Wenchen Fan > centralize the code of qualifying identifiers in SessionCatalog >

[jira] [Resolved] (SPARK-40020) centralize the code of qualifying identifiers in SessionCatalog

2022-08-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40020. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37415

[jira] [Updated] (SPARK-40026) spark-shell can not instantiate object whose class is defined in REPL dynamically

2022-08-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40026: - Priority: Major (was: Blocker) > spark-shell can not instantiate object whose class is defined

[jira] [Updated] (SPARK-40026) spark-shell can not instantiate object whose class is defined in REPL dynamically

2022-08-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40026: - Issue Type: Bug (was: Question) > spark-shell can not instantiate object whose class is

[jira] [Commented] (SPARK-40043) Document DataStreamWriter.toTable and DataStreamReader.table

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578234#comment-17578234 ] Apache Spark commented on SPARK-40043: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-40043) Document DataStreamWriter.toTable and DataStreamReader.table

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578233#comment-17578233 ] Apache Spark commented on SPARK-40043: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-40043) Document DataStreamWriter.toTable and DataStreamReader.table

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40043: Assignee: Apache Spark > Document DataStreamWriter.toTable and DataStreamReader.table >

[jira] [Assigned] (SPARK-40043) Document DataStreamWriter.toTable and DataStreamReader.table

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40043: Assignee: (was: Apache Spark) > Document DataStreamWriter.toTable and

[jira] [Created] (SPARK-40043) Document DataStreamWriter.toTable and DataStreamReader.table

2022-08-10 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-40043: Summary: Document DataStreamWriter.toTable and DataStreamReader.table Key: SPARK-40043 URL: https://issues.apache.org/jira/browse/SPARK-40043 Project: Spark

[jira] [Commented] (SPARK-40041) Add Document Parameters for pyspark.sql.window

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578230#comment-17578230 ] Apache Spark commented on SPARK-40041: -- User 'dcoliversun' has created a pull request for this

[jira] [Assigned] (SPARK-40041) Add Document Parameters for pyspark.sql.window

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40041: Assignee: Apache Spark > Add Document Parameters for pyspark.sql.window >

[jira] [Assigned] (SPARK-40041) Add Document Parameters for pyspark.sql.window

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40041: Assignee: (was: Apache Spark) > Add Document Parameters for pyspark.sql.window >

[jira] [Commented] (SPARK-40041) Add Document Parameters for pyspark.sql.window

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578229#comment-17578229 ] Apache Spark commented on SPARK-40041: -- User 'dcoliversun' has created a pull request for this

[jira] [Created] (SPARK-40042) Make pyspark.sql.streaming.query examples self-contained

2022-08-10 Thread Qian Sun (Jira)
Qian Sun created SPARK-40042: Summary: Make pyspark.sql.streaming.query examples self-contained Key: SPARK-40042 URL: https://issues.apache.org/jira/browse/SPARK-40042 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-39895) pyspark drop doesn't accept *cols

2022-08-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-39895: Assignee: Santosh Pingale > pyspark drop doesn't accept *cols >

[jira] [Resolved] (SPARK-39895) pyspark drop doesn't accept *cols

2022-08-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-39895. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37335

[jira] [Created] (SPARK-40041) Add Document Parameters for pyspark.sql.window

2022-08-10 Thread Qian Sun (Jira)
Qian Sun created SPARK-40041: Summary: Add Document Parameters for pyspark.sql.window Key: SPARK-40041 URL: https://issues.apache.org/jira/browse/SPARK-40041 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-39708) ALS Model Loading

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578220#comment-17578220 ] Ruifeng Zheng commented on SPARK-39708: --- if you are using `pyspark.mllib.recommendation`, try to

[jira] [Commented] (SPARK-39708) ALS Model Loading

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578219#comment-17578219 ] Ruifeng Zheng commented on SPARK-39708: --- ??File /usr/local/spark/python/pyspark/ml/util.py:332, in

[jira] [Updated] (SPARK-40021) `Cancel workflow` can not cancel some jobs

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-40021: -- Attachment: Screen Shot 2022-08-11 at 10.07.24.png > `Cancel workflow` can not cancel some

[jira] [Comment Edited] (SPARK-40021) `Cancel workflow` can not cancel some jobs

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578212#comment-17578212 ] Ruifeng Zheng edited comment on SPARK-40021 at 8/11/22 2:08 AM: ??Did

[jira] [Assigned] (SPARK-40040) Push local limit to both sides if join condition is empty

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40040: Assignee: Apache Spark > Push local limit to both sides if join condition is empty >

[jira] [Assigned] (SPARK-40040) Push local limit to both sides if join condition is empty

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40040: Assignee: (was: Apache Spark) > Push local limit to both sides if join condition is

[jira] [Commented] (SPARK-40021) `Cancel workflow` can not cancel some jobs

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578212#comment-17578212 ] Ruifeng Zheng commented on SPARK-40021: --- Did you try to kill those Python jobs individually? >

[jira] [Commented] (SPARK-40040) Push local limit to both sides if join condition is empty

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578211#comment-17578211 ] Apache Spark commented on SPARK-40040: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-40040) Push local limit to both sides if join condition is empty

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578210#comment-17578210 ] Apache Spark commented on SPARK-40040: -- User 'wangyum' has created a pull request for this issue:

[jira] [Updated] (SPARK-40040) Push local limit to both sides if join condition is empty

2022-08-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40040: Summary: Push local limit to both sides if join condition is empty (was: Push local limit

[jira] [Created] (SPARK-40040) Push local limit through outer join if join condition is empty

2022-08-10 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-40040: --- Summary: Push local limit through outer join if join condition is empty Key: SPARK-40040 URL: https://issues.apache.org/jira/browse/SPARK-40040 Project: Spark

[jira] [Commented] (SPARK-40039) Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578195#comment-17578195 ] Jungtaek Lim commented on SPARK-40039: -- Very interesting one to see! (Disclaimer: Abortable was

[jira] [Commented] (SPARK-40039) Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578193#comment-17578193 ] Apache Spark commented on SPARK-40039: -- User 'attilapiros' has created a pull request for this

[jira] [Assigned] (SPARK-40039) Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40039: Assignee: Attila Zsolt Piros (was: Apache Spark) > Introducing a streaming checkpoint

[jira] [Commented] (SPARK-40039) Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578192#comment-17578192 ] Apache Spark commented on SPARK-40039: -- User 'attilapiros' has created a pull request for this

[jira] [Assigned] (SPARK-40039) Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40039: Assignee: Apache Spark (was: Attila Zsolt Piros) > Introducing a streaming checkpoint

[jira] [Updated] (SPARK-40039) Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40039: --- Summary: Introducing a streaming checkpoint file manager based on Hadoop's

[jira] [Commented] (SPARK-40039) Introducing checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578189#comment-17578189 ] Attila Zsolt Piros commented on SPARK-40039: I am working on this. > Introducing checkpoint

[jira] [Created] (SPARK-40039) Introducing checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-40039: -- Summary: Introducing checkpoint file manager based on Hadoop's Abortable interface Key: SPARK-40039 URL: https://issues.apache.org/jira/browse/SPARK-40039

[jira] [Resolved] (SPARK-39983) Should not cache unserialized broadcast relations on the driver

2022-08-10 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-39983. Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37413

[jira] [Assigned] (SPARK-39983) Should not cache unserialized broadcast relations on the driver

2022-08-10 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-39983: -- Assignee: Alex Balikov > Should not cache unserialized broadcast relations on the driver >

[jira] [Commented] (SPARK-40037) Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578168#comment-17578168 ] Apache Spark commented on SPARK-40037: -- User 'bjornjorgensen' has created a pull request for this

[jira] [Commented] (SPARK-40038) spark.sql.files.maxPartitionBytes does not observe on-disk compression

2022-08-10 Thread RJ Marcus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578169#comment-17578169 ] RJ Marcus commented on SPARK-40038: --- the screenshot with spillage didn't have the stage fully

[jira] [Commented] (SPARK-40037) Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578167#comment-17578167 ] Apache Spark commented on SPARK-40037: -- User 'bjornjorgensen' has created a pull request for this

[jira] [Updated] (SPARK-40038) spark.sql.files.maxPartitionBytes does not observe on-disk compression

2022-08-10 Thread RJ Marcus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RJ Marcus updated SPARK-40038: -- Attachment: Screenshot from 2022-08-10 16-59-56.png Screenshot from 2022-08-10

[jira] [Assigned] (SPARK-40037) Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40037: Assignee: Apache Spark > Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0 >

[jira] [Assigned] (SPARK-40037) Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40037: Assignee: (was: Apache Spark) > Upgrade com.google.crypto.tink:tink from 1.6.1 to

[jira] [Created] (SPARK-40038) spark.sql.files.maxPartitionBytes does not observe on-disk compression

2022-08-10 Thread RJ Marcus (Jira)
RJ Marcus created SPARK-40038: - Summary: spark.sql.files.maxPartitionBytes does not observe on-disk compression Key: SPARK-40038 URL: https://issues.apache.org/jira/browse/SPARK-40038 Project: Spark

[jira] [Updated] (SPARK-40025) Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark

2022-08-10 Thread Boyang Jerry Peng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated SPARK-40025: -- Description: Project Lightspeed is an umbrella project aimed at improving a couple of

[jira] [Updated] (SPARK-40025) Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark

2022-08-10 Thread Boyang Jerry Peng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated SPARK-40025: -- Description: Project Lightspeed is an umbrella project aimed at improving a couple of

[jira] [Updated] (SPARK-40025) Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark

2022-08-10 Thread Boyang Jerry Peng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated SPARK-40025: -- Summary: Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark

[jira] [Resolved] (SPARK-39743) Unable to set zstd compression level while writing parquet files

2022-08-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39743. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37416

[jira] [Assigned] (SPARK-39743) Unable to set zstd compression level while writing parquet files

2022-08-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39743: - Assignee: zhiming she > Unable to set zstd compression level while writing parquet

[jira] [Commented] (SPARK-39887) Expression transform error

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578094#comment-17578094 ] Apache Spark commented on SPARK-39887: -- User 'peter-toth' has created a pull request for this

[jira] [Resolved] (SPARK-38503) Add warn for getAdditionalPreKubernetesResources in executor side

2022-08-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38503. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35786

[jira] [Assigned] (SPARK-38503) Add warn for getAdditionalPreKubernetesResources in executor side

2022-08-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38503: - Assignee: Yikun Jiang > Add warn for getAdditionalPreKubernetesResources in executor

[jira] [Updated] (SPARK-40037) Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0

2022-08-10 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bjørn Jørgensen updated SPARK-40037: Description: [CVE-2022-25647|https://www.cve.org/CVERecord?id=CVE-2022-25647] [Info at

[jira] [Updated] (SPARK-40037) Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0

2022-08-10 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bjørn Jørgensen updated SPARK-40037: Description: [https://www.cve.org/CVERecord?id=CVE-2022-25647 | ]

[jira] [Updated] (SPARK-40037) Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0

2022-08-10 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bjørn Jørgensen updated SPARK-40037: Description: [https://www.cve.org/CVERecord?id=CVE-2022-25647 | CVE-2022-25647]

[jira] [Updated] (SPARK-40037) Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0

2022-08-10 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-40037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bjørn Jørgensen updated SPARK-40037: Description: [https://www.cve.org/CVERecord?id=CVE-2022-25647| CVE-2022-25647]

[jira] [Created] (SPARK-40037) Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0

2022-08-10 Thread Jira
Bjørn Jørgensen created SPARK-40037: --- Summary: Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0 Key: SPARK-40037 URL: https://issues.apache.org/jira/browse/SPARK-40037 Project: Spark

[jira] [Assigned] (SPARK-40036) LevelDB/RocksDBIterator.next should return false after iterator or db close

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40036: Assignee: (was: Apache Spark) > LevelDB/RocksDBIterator.next should return false

[jira] [Assigned] (SPARK-40036) LevelDB/RocksDBIterator.next should return false after iterator or db close

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40036: Assignee: Apache Spark > LevelDB/RocksDBIterator.next should return false after iterator

[jira] [Commented] (SPARK-40036) LevelDB/RocksDBIterator.next should return false after iterator or db close

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578066#comment-17578066 ] Apache Spark commented on SPARK-40036: -- User 'LuciferYang' has created a pull request for this

[jira] [Created] (SPARK-40036) LevelDB/RocksDBIterator.next should return false after iterator or db close

2022-08-10 Thread Yang Jie (Jira)
Yang Jie created SPARK-40036: Summary: LevelDB/RocksDBIterator.next should return false after iterator or db close Key: SPARK-40036 URL: https://issues.apache.org/jira/browse/SPARK-40036 Project: Spark

[jira] [Commented] (SPARK-38899) DS V2 supports push down datetime functions

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578052#comment-17578052 ] Apache Spark commented on SPARK-38899: -- User 'chenzhx' has created a pull request for this issue:

[jira] [Commented] (SPARK-38899) DS V2 supports push down datetime functions

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578051#comment-17578051 ] Apache Spark commented on SPARK-38899: -- User 'chenzhx' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40034: Assignee: (was: Apache Spark) > PathOutputCommitters to work with dynamic partition

[jira] [Commented] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578014#comment-17578014 ] Apache Spark commented on SPARK-40034: -- User 'steveloughran' has created a pull request for this

[jira] [Assigned] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40034: Assignee: Apache Spark > PathOutputCommitters to work with dynamic partition overwrite >

[jira] [Assigned] (SPARK-40035) Avoid apply filter twice when listing files

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40035: Assignee: (was: Apache Spark) > Avoid apply filter twice when listing files >

[jira] [Commented] (SPARK-40035) Avoid apply filter twice when listing files

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578008#comment-17578008 ] Apache Spark commented on SPARK-40035: -- User 'WangGuangxin' has created a pull request for this

[jira] [Commented] (SPARK-40035) Avoid apply filter twice when listing files

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578009#comment-17578009 ] Apache Spark commented on SPARK-40035: -- User 'WangGuangxin' has created a pull request for this

[jira] [Assigned] (SPARK-40035) Avoid apply filter twice when listing files

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40035: Assignee: Apache Spark > Avoid apply filter twice when listing files >

[jira] [Updated] (SPARK-38910) Clean sparkStaging dir should before unregister()

2022-08-10 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38910: -- Fix Version/s: 3.4.0 > Clean sparkStaging dir should before unregister() >

[jira] [Created] (SPARK-40035) Avoid apply filter twice when listing files

2022-08-10 Thread EdisonWang (Jira)
EdisonWang created SPARK-40035: -- Summary: Avoid apply filter twice when listing files Key: SPARK-40035 URL: https://issues.apache.org/jira/browse/SPARK-40035 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-38910) Clean sparkStaging dir should before unregister()

2022-08-10 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-38910. --- Resolution: Fixed > Clean sparkStaging dir should before unregister() >

[jira] [Assigned] (SPARK-38910) Clean sparkStaging dir should before unregister()

2022-08-10 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-38910: - Assignee: angerszhu > Clean sparkStaging dir should before unregister() >

[jira] [Updated] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite

2022-08-10 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-40034: --- Summary: PathOutputCommitters to work with dynamic partition overwrite (was:

[jira] [Created] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite -if they support it

2022-08-10 Thread Steve Loughran (Jira)
Steve Loughran created SPARK-40034: -- Summary: PathOutputCommitters to work with dynamic partition overwrite -if they support it Key: SPARK-40034 URL: https://issues.apache.org/jira/browse/SPARK-40034

[jira] [Assigned] (SPARK-39734) Add call_udf to pyspark.sql.functions

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-39734: - Assignee: Ruifeng Zheng (was: Andrew Ray) > Add call_udf to pyspark.sql.functions >

[jira] [Assigned] (SPARK-36259) Expose localtimestamp in pyspark.sql.functions

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-36259: - Assignee: Ruifeng Zheng > Expose localtimestamp in pyspark.sql.functions >

[jira] [Assigned] (SPARK-39733) Add map_contains_key to pyspark.sql.functions

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-39733: - Assignee: Ruifeng Zheng > Add map_contains_key to pyspark.sql.functions >

[jira] [Resolved] (SPARK-37348) PySpark pmod function

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-37348. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37449

[jira] [Assigned] (SPARK-37348) PySpark pmod function

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-37348: - Assignee: Ruifeng Zheng > PySpark pmod function > - > >

[jira] [Resolved] (SPARK-39733) Add map_contains_key to pyspark.sql.functions

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-39733. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37449

[jira] [Resolved] (SPARK-36259) Expose localtimestamp in pyspark.sql.functions

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-36259. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37449

[jira] [Assigned] (SPARK-39734) Add call_udf to pyspark.sql.functions

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-39734: - Assignee: Andrew Ray > Add call_udf to pyspark.sql.functions >

[jira] [Resolved] (SPARK-39734) Add call_udf to pyspark.sql.functions

2022-08-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-39734. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37449

[jira] [Commented] (SPARK-40029) Make pyspark.sql.types examples self-contained

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577946#comment-17577946 ] Apache Spark commented on SPARK-40029: -- User 'Yikun' has created a pull request for this issue:

[jira] [Commented] (SPARK-40029) Make pyspark.sql.types examples self-contained

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577945#comment-17577945 ] Apache Spark commented on SPARK-40029: -- User 'Yikun' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40029) Make pyspark.sql.types examples self-contained

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40029: Assignee: Apache Spark > Make pyspark.sql.types examples self-contained >

[jira] [Assigned] (SPARK-40029) Make pyspark.sql.types examples self-contained

2022-08-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40029: Assignee: (was: Apache Spark) > Make pyspark.sql.types examples self-contained >

[jira] [Commented] (SPARK-40032) Support Decimal128 type

2022-08-10 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577942#comment-17577942 ] jiaan.geng commented on SPARK-40032: We are editing the design doc. > Support Decimal128 type >

[jira] [Updated] (SPARK-40032) Support Decimal128 type

2022-08-10 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-40032: --- Attachment: Performance comparison between decimal128 and spark decimal benchmark.pdf > Support

[jira] [Updated] (SPARK-40032) Support Decimal128 type

2022-08-10 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-40032: --- Attachment: (was: Performance comparison between decimal128 and spark decimal benchmark.pdf) >

  1   2   >