[jira] [Created] (SPARK-37660) Spark-3.2.0 Fetch Hbase Data not working

2021-12-15 Thread Bhavya Raj Sharma (Jira)
Bhavya Raj Sharma created SPARK-37660: - Summary: Spark-3.2.0 Fetch Hbase Data not working Key: SPARK-37660 URL: https://issues.apache.org/jira/browse/SPARK-37660 Project: Spark Issue

[jira] [Updated] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Issue Type: Improvement (was: Bug) > Change emptyValueInRead's effect to that any fields matching this

[jira] [Updated] (SPARK-37060) Report driver status does not handle response from backup masters

2021-12-15 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-37060: - Fix Version/s: 3.1.3 3.2.1 > Report driver status does not handle response from backup

[jira] [Commented] (SPARK-37659) Fix FsHistoryProvider race condition between list and delet log info

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460448#comment-17460448 ] Apache Spark commented on SPARK-37659: -- User 'ulysses-you' has created a pull request for this

[jira] [Assigned] (SPARK-37659) Fix FsHistoryProvider race condition between list and delet log info

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37659: Assignee: (was: Apache Spark) > Fix FsHistoryProvider race condition between list

[jira] [Assigned] (SPARK-37659) Fix FsHistoryProvider race condition between list and delet log info

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37659: Assignee: Apache Spark > Fix FsHistoryProvider race condition between list and delet log

[jira] [Updated] (SPARK-37656) Upgrade SBT to 1.5.7

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37656: - Fix Version/s: 3.2.1 > Upgrade SBT to 1.5.7 > > > Key:

[jira] [Resolved] (SPARK-37658) Skip PIP packaging test if Python version is lower than 3.7

2021-12-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37658. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34917

[jira] [Assigned] (SPARK-37658) Skip PIP packaging test if Python version is lower than 3.7

2021-12-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37658: - Assignee: Hyukjin Kwon > Skip PIP packaging test if Python version is lower than 3.7 >

[jira] [Updated] (SPARK-32294) GroupedData Pandas UDF 2Gb limit

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32294: - Fix Version/s: 3.2.0 (was: 3.3.0) > GroupedData Pandas UDF 2Gb limit >

[jira] [Created] (SPARK-37659) Fix FsHistoryProvider race condition between list and delet log info

2021-12-15 Thread XiDuo You (Jira)
XiDuo You created SPARK-37659: - Summary: Fix FsHistoryProvider race condition between list and delet log info Key: SPARK-37659 URL: https://issues.apache.org/jira/browse/SPARK-37659 Project: Spark

[jira] [Resolved] (SPARK-34521) spark.createDataFrame does not support Pandas StringDtype extension type

2021-12-15 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-34521. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34509

[jira] [Commented] (SPARK-37483) Support push down top N to JDBC data source V2

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460427#comment-17460427 ] Apache Spark commented on SPARK-37483: -- User 'beliefer' has created a pull request for this issue:

[jira] [Updated] (SPARK-37657) Fix the bug in ps.(Series|DataFrame).describe()

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37657: - Description: Initialized in Koalas issue: [https://github.com/databricks/koalas/issues/1888]  

[jira] [Updated] (SPARK-37657) Fix the bug in ps.(Series|DataFrame).describe()

2021-12-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37657: Summary: Fix the bug in ps.(Series|DataFrame).describe() (was: Fix the bug in

[jira] [Updated] (SPARK-37657) Fix the bug in ps.(Series|DataFrame).describe()

2021-12-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37657: Description: Initialized in Koalas issue: [https://github.com/databricks/koalas/issues/1888.]  

[jira] [Resolved] (SPARK-37656) Upgrade SBT to 1.5.7

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37656. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34915

[jira] [Assigned] (SPARK-37651) Use existing active Spark session in all places of pandas API on Spark

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37651: Assignee: Xinrong Meng > Use existing active Spark session in all places of pandas API

[jira] [Resolved] (SPARK-37651) Use existing active Spark session in all places of pandas API on Spark

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37651. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34906

[jira] [Assigned] (SPARK-37658) Skip PIP packaging test if Python version is lower than 3.7

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37658: Assignee: Apache Spark > Skip PIP packaging test if Python version is lower than 3.7 >

[jira] [Assigned] (SPARK-37658) Skip PIP packaging test if Python version is lower than 3.7

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37658: Assignee: (was: Apache Spark) > Skip PIP packaging test if Python version is lower

[jira] [Commented] (SPARK-37658) Skip PIP packaging test if Python version is lower than 3.7

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460397#comment-17460397 ] Apache Spark commented on SPARK-37658: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Created] (SPARK-37658) Skip PIP packaging test if Python version is lower than 3.7

2021-12-15 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-37658: Summary: Skip PIP packaging test if Python version is lower than 3.7 Key: SPARK-37658 URL: https://issues.apache.org/jira/browse/SPARK-37658 Project: Spark

[jira] [Resolved] (SPARK-37632) Drop code targetting Python < 3.7

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37632. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34886

[jira] [Commented] (SPARK-35355) improve execution performance in insert...select...limit case

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460375#comment-17460375 ] Apache Spark commented on SPARK-35355: -- User 'yikf' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37632) Drop code targetting Python < 3.7

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37632: Assignee: Maciej Szymkiewicz > Drop code targetting Python < 3.7 >

[jira] [Updated] (SPARK-37657) Fix the bug in ps.DataFrame.describe()

2021-12-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37657: Description: Initialized in Koalas issue: [https://github.com/databricks/koalas/issues/1888.]  

[jira] [Updated] (SPARK-37657) Fix the bug in ps.DataFrame.describe()

2021-12-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37657: Description: Initialized in Koalas issue: [https://github.com/databricks/koalas/issues/1888.]  

[jira] [Updated] (SPARK-37657) Fix the bug in ps.DataFrame.describe()

2021-12-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37657: Description: Initialized in Koalas issue: [https://github.com/databricks/koalas/issues/1888.]  

[jira] [Commented] (SPARK-37657) Fix the bug in ps.DataFrame.describe()

2021-12-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460366#comment-17460366 ] Haejoon Lee commented on SPARK-37657: - I'm working on this > Fix the bug in ps.DataFrame.describe()

[jira] [Updated] (SPARK-37657) Fix the bug in ps.DataFrame.describe()

2021-12-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37657: Description: Initialized in Koalas issue: [https://github.com/databricks/koalas/issues/1888.]  

[jira] [Created] (SPARK-37657) Fix the bug in ps.DataFrame.describe()

2021-12-15 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-37657: --- Summary: Fix the bug in ps.DataFrame.describe() Key: SPARK-37657 URL: https://issues.apache.org/jira/browse/SPARK-37657 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-37656) Upgrade SBT to 1.5.7

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37656: Assignee: Apache Spark (was: Kousuke Saruta) > Upgrade SBT to 1.5.7 >

[jira] [Assigned] (SPARK-37656) Upgrade SBT to 1.5.7

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37656: Assignee: Kousuke Saruta (was: Apache Spark) > Upgrade SBT to 1.5.7 >

[jira] [Commented] (SPARK-37656) Upgrade SBT to 1.5.7

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460357#comment-17460357 ] Apache Spark commented on SPARK-37656: -- User 'sarutak' has created a pull request for this issue:

[jira] [Commented] (SPARK-37656) Upgrade SBT to 1.5.7

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460358#comment-17460358 ] Apache Spark commented on SPARK-37656: -- User 'sarutak' has created a pull request for this issue:

[jira] [Created] (SPARK-37656) Upgrade SBT to 1.5.7

2021-12-15 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-37656: -- Summary: Upgrade SBT to 1.5.7 Key: SPARK-37656 URL: https://issues.apache.org/jira/browse/SPARK-37656 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-34198) Add RocksDB StateStore implementation

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34198: - Shepherd: L. C. Hsieh > Add RocksDB StateStore implementation >

[jira] [Assigned] (SPARK-34198) Add RocksDB StateStore implementation

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-34198: Assignee: Yuanjian Li (was: Apache Spark) > Add RocksDB StateStore implementation >

[jira] [Assigned] (SPARK-37483) Support push down top N to JDBC data source V2

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37483: Assignee: (was: Apache Spark) > Support push down top N to JDBC data source V2 >

[jira] [Assigned] (SPARK-37483) Support push down top N to JDBC data source V2

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37483: Assignee: Apache Spark > Support push down top N to JDBC data source V2 >

[jira] [Updated] (SPARK-37483) Support push down top N to JDBC data source V2

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37483: - Fix Version/s: (was: 3.3.0) > Support push down top N to JDBC data source V2 >

[jira] [Reopened] (SPARK-37483) Support push down top N to JDBC data source V2

2021-12-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-37483: -- Assignee: (was: jiaan.geng) Reverted at

[jira] [Commented] (SPARK-37627) Add sorted column in BucketTransform

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460331#comment-17460331 ] Apache Spark commented on SPARK-37627: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Commented] (SPARK-37627) Add sorted column in BucketTransform

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460329#comment-17460329 ] Apache Spark commented on SPARK-37627: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Commented] (SPARK-37630) Security issue from Log4j 1.X exploit

2021-12-15 Thread Samved Chandrakant Divekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460321#comment-17460321 ] Samved Chandrakant Divekar commented on SPARK-37630: New CVE which specifically

[jira] [Updated] (SPARK-37655) Add RocksDB Implementation for KVStore

2021-12-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37655: -- Parent: SPARK-33772 Issue Type: Sub-task (was: Improvement) > Add RocksDB

[jira] [Assigned] (SPARK-37655) Add RocksDB Implementation for KVStore

2021-12-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37655: - Assignee: Dongjoon Hyun (was: Apache Spark) > Add RocksDB Implementation for KVStore

[jira] [Updated] (SPARK-37653) Upgrade RoaringBitmap to 0.9.23

2021-12-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37653: -- Parent: SPARK-33772 Issue Type: Sub-task (was: Improvement) > Upgrade RoaringBitmap

[jira] [Assigned] (SPARK-37653) Upgrade RoaringBitmap to 0.9.23

2021-12-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37653: - Assignee: Yang Jie > Upgrade RoaringBitmap to 0.9.23 > ---

[jira] [Resolved] (SPARK-37653) Upgrade RoaringBitmap to 0.9.23

2021-12-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37653. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34909

[jira] [Commented] (SPARK-37655) Add RocksDB Implementation for KVStore

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460284#comment-17460284 ] Apache Spark commented on SPARK-37655: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-37655) Add RocksDB Implementation for KVStore

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37655: Assignee: Apache Spark > Add RocksDB Implementation for KVStore >

[jira] [Assigned] (SPARK-37655) Add RocksDB Implementation for KVStore

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37655: Assignee: (was: Apache Spark) > Add RocksDB Implementation for KVStore >

[jira] [Assigned] (SPARK-37655) Add RocksDB Implementation for KVStore

2021-12-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37655: Assignee: Apache Spark > Add RocksDB Implementation for KVStore >

[jira] [Created] (SPARK-37655) Add RocksDB Implementation for KVStore

2021-12-15 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-37655: - Summary: Add RocksDB Implementation for KVStore Key: SPARK-37655 URL: https://issues.apache.org/jira/browse/SPARK-37655 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37154) Inline type hints for python/pyspark/rdd.py

2021-12-15 Thread Byron Hsu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460243#comment-17460243 ] Byron Hsu commented on SPARK-37154: --- I am looking into this one > Inline type hints for

[jira] [Updated] (SPARK-37633) Unwrap cast should skip if downcast failed with ansi enabled

2021-12-15 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-37633: - Affects Version/s: (was: 3.0.3) > Unwrap cast should skip if downcast failed with ansi enabled >

[jira] [Assigned] (SPARK-37633) Unwrap cast should skip if downcast failed with ansi enabled

2021-12-15 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned SPARK-37633: Assignee: Manu Zhang > Unwrap cast should skip if downcast failed with ansi enabled >

[jira] [Resolved] (SPARK-37633) Unwrap cast should skip if downcast failed with ansi enabled

2021-12-15 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-37633. -- Fix Version/s: 3.3.0 3.2.1 Resolution: Fixed Issue resolved by pull request

[jira] [Comment Edited] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460144#comment-17460144 ] Wei Guo edited comment on SPARK-37604 at 12/15/21, 6:05 PM: For codes:

[jira] [Comment Edited] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460144#comment-17460144 ] Wei Guo edited comment on SPARK-37604 at 12/15/21, 6:05 PM: For codes:

[jira] [Updated] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Attachment: (was: image-2021-12-16-01-57-55-864.png) > Change emptyValueInRead's effect to that any

[jira] [Comment Edited] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460144#comment-17460144 ] Wei Guo edited comment on SPARK-37604 at 12/15/21, 6:04 PM: For codes:

[jira] [Comment Edited] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460144#comment-17460144 ] Wei Guo edited comment on SPARK-37604 at 12/15/21, 6:03 PM: For codes:

[jira] [Comment Edited] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460144#comment-17460144 ] Wei Guo edited comment on SPARK-37604 at 12/15/21, 6:03 PM: For codes:

[jira] [Updated] (SPARK-37654) Regression - NullPointerException in Row.getSeq when field null

2021-12-15 Thread Brandon Dahler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Dahler updated SPARK-37654: --- Description: h2. Description A NullPointerException occurs in 

[jira] [Updated] (SPARK-37654) Regression - NullPointerException in Row.getSeq when field null

2021-12-15 Thread Brandon Dahler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Dahler updated SPARK-37654: --- Description: h2. Description A NullPointerException occurs in 

[jira] [Updated] (SPARK-37654) Regression - NullPointerException in Row.getSeq when field null

2021-12-15 Thread Brandon Dahler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Dahler updated SPARK-37654: --- Environment: (was: Tested against the following releases using the provided

[jira] [Created] (SPARK-37654) Regression - NullPointerException in Row.getSeq when field null

2021-12-15 Thread Brandon Dahler (Jira)
Brandon Dahler created SPARK-37654: -- Summary: Regression - NullPointerException in Row.getSeq when field null Key: SPARK-37654 URL: https://issues.apache.org/jira/browse/SPARK-37654 Project: Spark

[jira] [Commented] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460144#comment-17460144 ] Wei Guo commented on SPARK-37604: - For codes: {code:scala} val data = Seq(("Tesla", "")).toDF("make",

[jira] [Updated] (SPARK-37572) Flexible ways of launching executors

2021-12-15 Thread Dagang Wei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei updated SPARK-37572: --- Affects Version/s: 3.3.0 (was: 3.2.0) > Flexible ways of launching

[jira] [Updated] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Attachment: empty_test.png > Change emptyValueInRead's effect to that any fields matching this string

[jira] [Updated] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Attachment: image-2021-12-16-01-57-55-864.png > Change emptyValueInRead's effect to that any fields

[jira] [Updated] (SPARK-37572) Flexible ways of launching executors

2021-12-15 Thread Dagang Wei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei updated SPARK-37572: --- Description: Currently Spark launches executor processes by constructing and running commands [1],

[jira] [Updated] (SPARK-37604) Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Summary: Change emptyValueInRead's effect to that any fields matching this string will be set as "" when

[jira] [Commented] (SPARK-37630) Security issue from Log4j 1.X exploit

2021-12-15 Thread Ankur Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460117#comment-17460117 ] Ankur Jain commented on SPARK-37630: Log4j 2.15.0 also has a vulnerability, good if we can upgrade

[jira] [Commented] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460111#comment-17460111 ] Max Gekk commented on SPARK-37604: -- > but "EMPTY" strings in csv files can not be parsed as empty

[jira] [Comment Edited] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460091#comment-17460091 ] Wei Guo edited comment on SPARK-37604 at 12/15/21, 4:48 PM: [~hyukjin.kwon],

[jira] [Commented] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460091#comment-17460091 ] Wei Guo commented on SPARK-37604: - [~hyukjin.kwon][~maxgekk] Shall we have a simple discussion about it

[jira] [Commented] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460084#comment-17460084 ] Wei Guo commented on SPARK-37604: - Maybe this issue is not a notable bug or promotion, but, for users'

[jira] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604 ] Wei Guo deleted comment on SPARK-37604: - was (Author: wayne guo): The current behavior of emptyValueInRead is more like the function for null values in Dataset:  {code:scala}

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Resolved] (SPARK-37483) Support push down top N to JDBC data source V2

2021-12-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37483. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34738

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Assigned] (SPARK-37483) Support push down top N to JDBC data source V2

2021-12-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37483: --- Assignee: jiaan.geng > Support push down top N to JDBC data source V2 >

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

[jira] [Updated] (SPARK-37604) The option emptyValueInRead(in CSVOptions) is suggested to be designed as that any fields matching this string will be set as empty values "" when reading

2021-12-15 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-37604: Description: The csv data format is imported from databricks

  1   2   >