[jira] [Created] (SPARK-43317) Support combine adjacent aggregation

2023-04-27 Thread XiDuo You (Jira)
XiDuo You created SPARK-43317: - Summary: Support combine adjacent aggregation Key: SPARK-43317 URL: https://issues.apache.org/jira/browse/SPARK-43317 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-43316) Add more CTE SQL tests

2023-04-27 Thread Runyao.Chen (Jira)
Runyao.Chen created SPARK-43316: --- Summary: Add more CTE SQL tests Key: SPARK-43316 URL: https://issues.apache.org/jira/browse/SPARK-43316 Project: Spark Issue Type: Test Components: S

[jira] [Comment Edited] (SPARK-43106) Data lost from the table if the INSERT OVERWRITE query fails

2023-04-27 Thread kalyan s (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717461#comment-17717461 ] kalyan s edited comment on SPARK-43106 at 4/28/23 5:03 AM: --- [~

[jira] [Commented] (SPARK-43106) Data lost from the table if the INSERT OVERWRITE query fails

2023-04-27 Thread kalyan s (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717461#comment-17717461 ] kalyan s commented on SPARK-43106: -- [~cloud_fan] , [~dongjoon]  , [~gurwls223]  Input

[jira] [Assigned] (SPARK-43302) Make Python UDAF an AggregateFunction

2023-04-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43302: --- Assignee: Wenchen Fan > Make Python UDAF an AggregateFunction > ---

[jira] [Resolved] (SPARK-38461) Use error classes in org.apache.spark.broadcast

2023-04-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38461. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40978 [https://gith

[jira] [Resolved] (SPARK-43302) Make Python UDAF an AggregateFunction

2023-04-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43302. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40739 [https://gith

[jira] [Resolved] (SPARK-43309) Extend INTERNAL_ERROR with category

2023-04-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43309. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40978 [https://gith

[jira] [Assigned] (SPARK-43309) Extend INTERNAL_ERROR with category

2023-04-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43309: --- Assignee: Bo Zhang > Extend INTERNAL_ERROR with category >

[jira] [Assigned] (SPARK-38461) Use error classes in org.apache.spark.broadcast

2023-04-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38461: --- Assignee: Bo Zhang > Use error classes in org.apache.spark.broadcast >

[jira] [Updated] (SPARK-43315) Migrate remaining errors from DataFrame(Reader|Writer) into error class

2023-04-27 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-43315: Summary: Migrate remaining errors from DataFrame(Reader|Writer) into error class (was: Migrate er

[jira] [Created] (SPARK-43315) Migrate errors from DataFrame(Reader|Writer) into error class

2023-04-27 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-43315: --- Summary: Migrate errors from DataFrame(Reader|Writer) into error class Key: SPARK-43315 URL: https://issues.apache.org/jira/browse/SPARK-43315 Project: Spark

[jira] [Created] (SPARK-43314) Migrate Spark Connect client errors into error class

2023-04-27 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-43314: --- Summary: Migrate Spark Connect client errors into error class Key: SPARK-43314 URL: https://issues.apache.org/jira/browse/SPARK-43314 Project: Spark Issue Type

[jira] [Commented] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

2023-04-27 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717418#comment-17717418 ] Jungtaek Lim commented on SPARK-43244: -- [~kimahriman] Great to hear that [~anishsh

[jira] [Commented] (SPARK-43253) Assign a name to the error class _LEGACY_ERROR_TEMP_2017

2023-04-27 Thread LUIZ FERNANDO NEVES DE ARAUJO (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717412#comment-17717412 ] LUIZ FERNANDO NEVES DE ARAUJO commented on SPARK-43253: --- I created

[jira] [Commented] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

2023-04-27 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717407#comment-17717407 ] Anish Shrigondekar commented on SPARK-43244: Yes correct. All the usage will

[jira] [Created] (SPARK-43313) Adding missing default values for MERGE INSERT actions

2023-04-27 Thread Daniel (Jira)
Daniel created SPARK-43313: -- Summary: Adding missing default values for MERGE INSERT actions Key: SPARK-43313 URL: https://issues.apache.org/jira/browse/SPARK-43313 Project: Spark Issue Type: Sub-ta

[jira] [Comment Edited] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

2023-04-27 Thread Adam Binford (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717394#comment-17717394 ] Adam Binford edited comment on SPARK-43244 at 4/27/23 11:08 PM: --

[jira] [Commented] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

2023-04-27 Thread Adam Binford (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717394#comment-17717394 ] Adam Binford commented on SPARK-43244: -- Yeah I just saw that PR and was looking thr

[jira] [Commented] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

2023-04-27 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717393#comment-17717393 ] Anish Shrigondekar commented on SPARK-43244: [~kimahriman] - we did some inv

[jira] [Updated] (SPARK-43312) Protobuf: Allow converting Any fields to JSON

2023-04-27 Thread Raghu Angadi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghu Angadi updated SPARK-43312: - Summary: Protobuf: Allow converting Any fields to JSON (was: Protobfu: Allow converting Any fie

[jira] [Created] (SPARK-43312) Protobfu: Allow converting Any fields to JSON

2023-04-27 Thread Raghu Angadi (Jira)
Raghu Angadi created SPARK-43312: Summary: Protobfu: Allow converting Any fields to JSON Key: SPARK-43312 URL: https://issues.apache.org/jira/browse/SPARK-43312 Project: Spark Issue Type: Tas

[jira] [Commented] (SPARK-43311) RocksDB state store provider memory management enhancements

2023-04-27 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717359#comment-17717359 ] Anish Shrigondekar commented on SPARK-43311: PR here: https://github.com/apa

[jira] [Updated] (SPARK-43310) Dataset.observe is ignored when writing to Kafka with batch query

2023-04-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43310: - Component/s: Structured Streaming > Dataset.observe is ignored when writing to Kafka with batch

[jira] [Commented] (SPARK-43132) Add foreach streaming API in Python

2023-04-27 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717347#comment-17717347 ] Wei Liu commented on SPARK-43132: - i'm working on this   > Add foreach streaming API i

[jira] [Resolved] (SPARK-43054) Support foreach() in streaming spark connect

2023-04-27 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-43054. - Resolution: Duplicate duplicate with 43132 and 43133   > Support foreach() in streaming spark connect

[jira] [Resolved] (SPARK-43298) predict_batch_udf with scalar input fails when batch size consists of a single value

2023-04-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43298. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40967 [https://gi

[jira] [Assigned] (SPARK-43298) predict_batch_udf with scalar input fails when batch size consists of a single value

2023-04-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43298: Assignee: Lee Yang > predict_batch_udf with scalar input fails when batch size consists o

[jira] [Updated] (SPARK-43311) RocksDB state store provider memory management enhancements

2023-04-27 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anish Shrigondekar updated SPARK-43311: --- Description: Today when RocksDB is used as a State Store provider, memory usage when

[jira] [Commented] (SPARK-43311) RocksDB state store provider memory management enhancements

2023-04-27 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717331#comment-17717331 ] Anish Shrigondekar commented on SPARK-43311: [~kabhwan] - will send the PR f

[jira] [Created] (SPARK-43311) RocksDB state store provider memory management enhancements

2023-04-27 Thread Anish Shrigondekar (Jira)
Anish Shrigondekar created SPARK-43311: -- Summary: RocksDB state store provider memory management enhancements Key: SPARK-43311 URL: https://issues.apache.org/jira/browse/SPARK-43311 Project: Spar

[jira] [Created] (SPARK-43310) Dataset.observe is ignored when writing to Kafka with batch query

2023-04-27 Thread David Deuber (Jira)
David Deuber created SPARK-43310: Summary: Dataset.observe is ignored when writing to Kafka with batch query Key: SPARK-43310 URL: https://issues.apache.org/jira/browse/SPARK-43310 Project: Spark

[jira] [Updated] (SPARK-43308) Improve scalar subquery logic plan when result are literal

2023-04-27 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-43308: Description: When use scalar subquery, sometimes we can get result before physical plan execute.  Like `

[jira] [Created] (SPARK-43309) Extend INTERNAL_ERROR with category

2023-04-27 Thread Bo Zhang (Jira)
Bo Zhang created SPARK-43309: Summary: Extend INTERNAL_ERROR with category Key: SPARK-43309 URL: https://issues.apache.org/jira/browse/SPARK-43309 Project: Spark Issue Type: Sub-task Co

[jira] [Updated] (SPARK-43308) Improve scalar subquery logic plan when result are literal

2023-04-27 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-43308: Description: When use scalar subquery, sometimes we can get result before physical plan execute.  Like `

[jira] [Updated] (SPARK-43308) Improve scalar subquery logic plan when result are literal

2023-04-27 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-43308: Parent: SPARK-35553 Issue Type: Sub-task (was: Improvement) > Improve scalar subquery logic plan

[jira] [Created] (SPARK-43308) Improve scalar subquery logic plan when result are literal

2023-04-27 Thread Jia Fan (Jira)
Jia Fan created SPARK-43308: --- Summary: Improve scalar subquery logic plan when result are literal Key: SPARK-43308 URL: https://issues.apache.org/jira/browse/SPARK-43308 Project: Spark Issue Type:

[jira] [Commented] (SPARK-43156) Correctness COUNT bug in correlated scalar subselect with `COUNT(*) is null`

2023-04-27 Thread Nikita Awasthi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717154#comment-17717154 ] Nikita Awasthi commented on SPARK-43156: User 'Hisoka-X' has created a pull requ

[jira] [Resolved] (SPARK-43257) Assign a name to the error class _LEGACY_ERROR_TEMP_2022

2023-04-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-43257. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40957 [https://github.com

[jira] [Assigned] (SPARK-43257) Assign a name to the error class _LEGACY_ERROR_TEMP_2022

2023-04-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-43257: Assignee: Jin Helin > Assign a name to the error class _LEGACY_ERROR_TEMP_2022 >

[jira] [Commented] (SPARK-43051) Allow materializing zero values when deserializing protobuf messages

2023-04-27 Thread Nikita Awasthi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717147#comment-17717147 ] Nikita Awasthi commented on SPARK-43051: User 'justaparth' has created a pull re

[jira] [Commented] (SPARK-43257) Assign a name to the error class _LEGACY_ERROR_TEMP_2022

2023-04-27 Thread Jin Helin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717141#comment-17717141 ] Jin Helin commented on SPARK-43257: --- I'd like to work on this. > Assign a name to the

[jira] [Created] (SPARK-43307) Migrate PandasUDF value errors into error class

2023-04-27 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-43307: --- Summary: Migrate PandasUDF value errors into error class Key: SPARK-43307 URL: https://issues.apache.org/jira/browse/SPARK-43307 Project: Spark Issue Type: Sub

[jira] [Created] (SPARK-43306) Migrate `ValueError` from Spark SQL types into error class

2023-04-27 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-43306: --- Summary: Migrate `ValueError` from Spark SQL types into error class Key: SPARK-43306 URL: https://issues.apache.org/jira/browse/SPARK-43306 Project: Spark Issu

[jira] [Created] (SPARK-43305) Add Java17 dockerfiles for 3.4.0

2023-04-27 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-43305: --- Summary: Add Java17 dockerfiles for 3.4.0 Key: SPARK-43305 URL: https://issues.apache.org/jira/browse/SPARK-43305 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-43304) Enable test_to_latex by supporting jinja2>=3.0.0

2023-04-27 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-43304: --- Summary: Enable test_to_latex by supporting jinja2>=3.0.0 Key: SPARK-43304 URL: https://issues.apache.org/jira/browse/SPARK-43304 Project: Spark Issue Type: Su

[jira] [Resolved] (SPARK-43261) Migrate `TypeError` from Spark SQL types into error class

2023-04-27 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43261. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40926 [https://

[jira] [Assigned] (SPARK-43261) Migrate `TypeError` from Spark SQL types into error class

2023-04-27 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43261: - Assignee: Haejoon Lee > Migrate `TypeError` from Spark SQL types into error class > ---

[jira] [Resolved] (SPARK-43219) Website can't find INSERT INTO REPLACE Statement

2023-04-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43219. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40890 [https://gith

[jira] [Assigned] (SPARK-43219) Website can't find INSERT INTO REPLACE Statement

2023-04-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43219: --- Assignee: Jia Fan > Website can't find INSERT INTO REPLACE Statement >

[jira] [Created] (SPARK-43303) Migrate NotImplementedError into PySparkNotImplementedError

2023-04-27 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-43303: --- Summary: Migrate NotImplementedError into PySparkNotImplementedError Key: SPARK-43303 URL: https://issues.apache.org/jira/browse/SPARK-43303 Project: Spark Is