[jira] [Assigned] (SPARK-37202) Temp view didn't collect temp function that registered with catalog API

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37202: Assignee: (was: Apache Spark) > Temp view didn't collect temp function that

[jira] [Assigned] (SPARK-37202) Temp view didn't collect temp function that registered with catalog API

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37202: Assignee: Apache Spark > Temp view didn't collect temp function that registered with

[jira] [Commented] (SPARK-37202) Temp view didn't collect temp function that registered with catalog API

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437730#comment-17437730 ] Apache Spark commented on SPARK-37202: -- User 'linhongliu-db' has created a pull request for this

[jira] [Created] (SPARK-37202) Temp view didn't collect temp function that registered with catalog API

2021-11-02 Thread Linhong Liu (Jira)
Linhong Liu created SPARK-37202: --- Summary: Temp view didn't collect temp function that registered with catalog API Key: SPARK-37202 URL: https://issues.apache.org/jira/browse/SPARK-37202 Project: Spark

[jira] [Commented] (SPARK-24774) support reading AVRO logical types - Decimal

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437714#comment-17437714 ] Apache Spark commented on SPARK-24774: -- User 'gengliangwang' has created a pull request for this

[jira] [Commented] (SPARK-24774) support reading AVRO logical types - Decimal

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437713#comment-17437713 ] Apache Spark commented on SPARK-24774: -- User 'gengliangwang' has created a pull request for this

[jira] [Resolved] (SPARK-37191) Allow merging DecimalTypes with different precision values

2021-11-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37191. - Resolution: Fixed Issue resolved by pull request 34462

[jira] [Assigned] (SPARK-37191) Allow merging DecimalTypes with different precision values

2021-11-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37191: --- Assignee: Ivan > Allow merging DecimalTypes with different precision values >

[jira] [Resolved] (SPARK-32567) Code-gen for full outer shuffled hash join

2021-11-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32567. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 3

[jira] [Assigned] (SPARK-32567) Code-gen for full outer shuffled hash join

2021-11-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32567: --- Assignee: Cheng Su > Code-gen for full outer shuffled hash join >

[jira] [Updated] (SPARK-37201) Spark SQL reads unnecessary nested fields (filter after explode)

2021-11-02 Thread Sergey Kotlov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Kotlov updated SPARK-37201: -- Summary: Spark SQL reads unnecessary nested fields (filter after explode) (was: Spark SQL

[jira] [Updated] (SPARK-37201) Spark SQL reads unnecсessary nested fields (filter after explode)

2021-11-02 Thread Sergey Kotlov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Kotlov updated SPARK-37201: -- Summary: Spark SQL reads unnecсessary nested fields (filter after explode) (was: Spark SQL

[jira] [Created] (SPARK-37201) Spark SQL reads unnecessary nested fields (filter after explode)

2021-11-02 Thread Sergey Kotlov (Jira)
Sergey Kotlov created SPARK-37201: - Summary: Spark SQL reads unnecessary nested fields (filter after explode) Key: SPARK-37201 URL: https://issues.apache.org/jira/browse/SPARK-37201 Project: Spark

[jira] [Resolved] (SPARK-36935) Enhance ParquetSchemaConverter to capture Parquet repetition & definition level

2021-11-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36935. --- Fix Version/s: 3.3.0 Assignee: Chao Sun Resolution: Fixed This is resolved

[jira] [Assigned] (SPARK-36879) Support Parquet v2 data page encodings for the vectorized path

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36879: Assignee: (was: Apache Spark) > Support Parquet v2 data page encodings for the

[jira] [Commented] (SPARK-36879) Support Parquet v2 data page encodings for the vectorized path

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437628#comment-17437628 ] Apache Spark commented on SPARK-36879: -- User 'parthchandra' has created a pull request for this

[jira] [Assigned] (SPARK-36879) Support Parquet v2 data page encodings for the vectorized path

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36879: Assignee: Apache Spark > Support Parquet v2 data page encodings for the vectorized path

[jira] [Assigned] (SPARK-37200) Drop index support

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37200: Assignee: (was: Apache Spark) > Drop index support > -- > >

[jira] [Commented] (SPARK-37200) Drop index support

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437597#comment-17437597 ] Apache Spark commented on SPARK-37200: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37200) Drop index support

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37200: Assignee: Apache Spark > Drop index support > -- > >

[jira] [Created] (SPARK-37200) Drop index support

2021-11-02 Thread Huaxin Gao (Jira)
Huaxin Gao created SPARK-37200: -- Summary: Drop index support Key: SPARK-37200 URL: https://issues.apache.org/jira/browse/SPARK-37200 Project: Spark Issue Type: Sub-task Components:

[jira] [Commented] (SPARK-37185) DataFrame.take() only uses one worker

2021-11-02 Thread mathieu longtin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437585#comment-17437585 ] mathieu longtin commented on SPARK-37185: - It seems to try to optimize for a simple query, but

[jira] [Commented] (SPARK-37199) Add a deterministic field to QueryPlan

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437527#comment-17437527 ] Apache Spark commented on SPARK-37199: -- User 'somani' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37199) Add a deterministic field to QueryPlan

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37199: Assignee: Apache Spark > Add a deterministic field to QueryPlan >

[jira] [Assigned] (SPARK-37199) Add a deterministic field to QueryPlan

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37199: Assignee: (was: Apache Spark) > Add a deterministic field to QueryPlan >

[jira] [Updated] (SPARK-37199) Add a deterministic field to QueryPlan

2021-11-02 Thread Abhishek Somani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Somani updated SPARK-37199: Description: We have a _deterministic_ field in

[jira] [Updated] (SPARK-37197) PySpark pandas recent issues from chconnell

2021-11-02 Thread Chuck Connell (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuck Connell updated SPARK-37197: -- Description: SPARK-37180  PySpark.pandas should support __version__ SPARK-37181  

[jira] [Commented] (SPARK-37199) Add a deterministic field to QueryPlan

2021-11-02 Thread Abhishek Somani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437514#comment-17437514 ] Abhishek Somani commented on SPARK-37199: - Will add a PR soon. > Add a deterministic field to

[jira] [Created] (SPARK-37198) pyspark.pandas read_csv() and to_csv() should handle local files

2021-11-02 Thread Chuck Connell (Jira)
Chuck Connell created SPARK-37198: - Summary: pyspark.pandas read_csv() and to_csv() should handle local files Key: SPARK-37198 URL: https://issues.apache.org/jira/browse/SPARK-37198 Project: Spark

[jira] [Created] (SPARK-37199) Add a deterministic field to QueryPlan

2021-11-02 Thread Abhishek Somani (Jira)
Abhishek Somani created SPARK-37199: --- Summary: Add a deterministic field to QueryPlan Key: SPARK-37199 URL: https://issues.apache.org/jira/browse/SPARK-37199 Project: Spark Issue Type:

[jira] [Updated] (SPARK-37199) Add a deterministic field to QueryPlan

2021-11-02 Thread Abhishek Somani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Somani updated SPARK-37199: Description: We have a _deterministic_ field in

[jira] [Updated] (SPARK-35496) Upgrade Scala 2.13 to 2.13.7

2021-11-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35496: -- Affects Version/s: (was: 3.2.0) > Upgrade Scala 2.13 to 2.13.7 >

[jira] [Assigned] (SPARK-37066) Improve ORC RecordReader's error message

2021-11-02 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-37066: Assignee: angerszhu > Improve ORC RecordReader's error message >

[jira] [Resolved] (SPARK-37066) Improve ORC RecordReader's error message

2021-11-02 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-37066. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34337

[jira] [Updated] (SPARK-37066) Improve ORC RecordReader's error message

2021-11-02 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-37066: - Priority: Minor (was: Major) > Improve ORC RecordReader's error message >

[jira] [Comment Edited] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437394#comment-17437394 ] Nicholas Chammas edited comment on SPARK-24853 at 11/2/21, 2:41 PM:

[jira] [Commented] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437396#comment-17437396 ] Nicholas Chammas commented on SPARK-24853: -- The [contributing

[jira] [Updated] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-24853: - Priority: Minor (was: Major) > Support Column type for withColumn and

[jira] [Reopened] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas reopened SPARK-24853: -- > Support Column type for withColumn and withColumnRenamed apis >

[jira] [Commented] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437394#comment-17437394 ] Nicholas Chammas commented on SPARK-24853: -- [~hyukjin.kwon] - It's not just for consistency. As

[jira] [Updated] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-24853: - Affects Version/s: 3.2.0 > Support Column type for withColumn and withColumnRenamed

[jira] [Comment Edited] (SPARK-37189) pyspark.pandas histogram accepts the range option but does not use it

2021-11-02 Thread Chuck Connell (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437329#comment-17437329 ] Chuck Connell edited comment on SPARK-37189 at 11/2/21, 1:18 PM: - Ok,

[jira] [Created] (SPARK-37197) PySpark pandas recent issues from chconnell

2021-11-02 Thread Chuck Connell (Jira)
Chuck Connell created SPARK-37197: - Summary: PySpark pandas recent issues from chconnell Key: SPARK-37197 URL: https://issues.apache.org/jira/browse/SPARK-37197 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37189) pyspark.pandas histogram accepts the range option but does not use it

2021-11-02 Thread Chuck Connell (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437329#comment-17437329 ] Chuck Connell commented on SPARK-37189: --- Ok, will do. FYI, getting covid shot today, so I may be

[jira] [Commented] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

2021-11-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437259#comment-17437259 ] Steve Loughran commented on SPARK-23977: [~gumartinm] can I draw your attention to Apache

[jira] [Updated] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-02 Thread Sergey Yavorski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Yavorski updated SPARK-37196: Description: Still reproducible in Spark 3.0.1. +*How to reproduce:*+ *SHELL* >

[jira] [Created] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-02 Thread Sergey Yavorski (Jira)
Sergey Yavorski created SPARK-37196: --- Summary: NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106) Key: SPARK-37196 URL: https://issues.apache.org/jira/browse/SPARK-37196

[jira] [Resolved] (SPARK-37176) JsonSource's infer should have the same exception handle logic as JacksonParser's parse logic

2021-11-02 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-37176. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34455

[jira] [Assigned] (SPARK-37176) JsonSource's infer should have the same exception handle logic as JacksonParser's parse logic

2021-11-02 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-37176: Assignee: Xianjin YE > JsonSource's infer should have the same exception handle logic as >

[jira] [Assigned] (SPARK-37194) Avoid unnecessary sort in FileFormatWriter if it's not dynamic partition

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37194: Assignee: (was: Apache Spark) > Avoid unnecessary sort in FileFormatWriter if it's

[jira] [Assigned] (SPARK-37194) Avoid unnecessary sort in FileFormatWriter if it's not dynamic partition

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37194: Assignee: Apache Spark > Avoid unnecessary sort in FileFormatWriter if it's not dynamic

[jira] [Commented] (SPARK-37194) Avoid unnecessary sort in FileFormatWriter if it's not dynamic partition

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437226#comment-17437226 ] Apache Spark commented on SPARK-37194: -- User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-36895) Add Create Index syntax support

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437216#comment-17437216 ] Apache Spark commented on SPARK-36895: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-37034) What's the progress of vectorized execution for spark?

2021-11-02 Thread xiaoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437213#comment-17437213 ] xiaoli commented on SPARK-37034: [~cloud_fan] Thanks very much. The hint in your answer helps me find

[jira] [Created] (SPARK-37195) Unify v1 and v2 SHOW TBLPROPERTIES tests

2021-11-02 Thread PengLei (Jira)
PengLei created SPARK-37195: --- Summary: Unify v1 and v2 SHOW TBLPROPERTIES tests Key: SPARK-37195 URL: https://issues.apache.org/jira/browse/SPARK-37195 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-36924) CAST between ANSI intervals and numerics

2021-11-02 Thread PengLei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437198#comment-17437198 ] PengLei commented on SPARK-36924: - woking on this later > CAST between ANSI intervals and numerics >

[jira] [Updated] (SPARK-37194) Avoid unnecessary sort in FileFormatWriter if it's not dynamic partition

2021-11-02 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37194: -- Description: `FileFormatWriter.write` will sort the partition and bucket column before writing. I

[jira] [Commented] (SPARK-26365) spark-submit for k8s cluster doesn't propagate exit code

2021-11-02 Thread Oscar Bonilla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437197#comment-17437197 ] Oscar Bonilla commented on SPARK-26365: --- Hi [~Gangishetty], unfortunately I'm not a contributor to

[jira] [Updated] (SPARK-37194) Avoid unnecessary sort in FileFormatWriter if it's not dynamic partition

2021-11-02 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37194: -- Description: `FileFormatWriter.write` will sort the partition and bucket column before writing. I

[jira] [Created] (SPARK-37194) Avoid unnecessary sort in FileFormatWriter if it's not dynamic partition

2021-11-02 Thread XiDuo You (Jira)
XiDuo You created SPARK-37194: - Summary: Avoid unnecessary sort in FileFormatWriter if it's not dynamic partition Key: SPARK-37194 URL: https://issues.apache.org/jira/browse/SPARK-37194 Project: Spark

[jira] [Comment Edited] (SPARK-37174) WARN WindowExec: No Partition Defined is being printed 4 times.

2021-11-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437184#comment-17437184 ] Hyukjin Kwon edited comment on SPARK-37174 at 11/2/21, 7:57 AM: This is

[jira] [Commented] (SPARK-37174) WARN WindowExec: No Partition Defined is being printed 4 times.

2021-11-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437184#comment-17437184 ] Hyukjin Kwon commented on SPARK-37174: -- This is related to default index, see also

[jira] [Updated] (SPARK-37177) Support LONG argument to the Spark SQL LIMIT clause

2021-11-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37177: - Component/s: (was: Spark Core) SQL > Support LONG argument to the Spark

[jira] [Commented] (SPARK-37185) DataFrame.take() only uses one worker

2021-11-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437182#comment-17437182 ] Hyukjin Kwon commented on SPARK-37185: -- can you show the perf diff between both codes? >

[jira] [Commented] (SPARK-37185) DataFrame.take() only uses one worker

2021-11-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437180#comment-17437180 ] Hyukjin Kwon commented on SPARK-37185: -- isn't it more optimized to use only one partition on one

[jira] [Commented] (SPARK-37189) pyspark.pandas histogram accepts the range option but does not use it

2021-11-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437179#comment-17437179 ] Hyukjin Kwon commented on SPARK-37189: -- Hey, thanks for reporting issues. [~chconnell]would you

[jira] [Commented] (SPARK-37030) Maven build failed in windows!

2021-11-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437178#comment-17437178 ] Hyukjin Kwon commented on SPARK-37030: -- You need {{bash}} on the path on Windows, either from

Subscribe

2021-11-02 Thread XING JIN
- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-37156) Inline type hints for python/pyspark/storagelevel.py

2021-11-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37156. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34437

[jira] [Assigned] (SPARK-37156) Inline type hints for python/pyspark/storagelevel.py

2021-11-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37156: Assignee: Apache Spark > Inline type hints for python/pyspark/storagelevel.py >

[jira] [Commented] (SPARK-37030) Maven build failed in windows!

2021-11-02 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437168#comment-17437168 ] Shockang commented on SPARK-37030: -- [~hyukjin.kwon] Even if the community does not support Spark in

[jira] [Commented] (SPARK-37152) Inline type hints for python/pyspark/context.py

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437161#comment-17437161 ] Apache Spark commented on SPARK-37152: -- User 'ByronHsu' has created a pull request for this issue:

[jira] [Commented] (SPARK-37152) Inline type hints for python/pyspark/context.py

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437159#comment-17437159 ] Apache Spark commented on SPARK-37152: -- User 'ByronHsu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37152) Inline type hints for python/pyspark/context.py

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37152: Assignee: Apache Spark > Inline type hints for python/pyspark/context.py >

[jira] [Assigned] (SPARK-37152) Inline type hints for python/pyspark/context.py

2021-11-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37152: Assignee: (was: Apache Spark) > Inline type hints for python/pyspark/context.py >

[jira] [Assigned] (SPARK-37168) Improve error messages for SQL functions and operators under ANSI mode

2021-11-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37168: --- Assignee: Allison Wang > Improve error messages for SQL functions and operators under ANSI

[jira] [Resolved] (SPARK-37168) Improve error messages for SQL functions and operators under ANSI mode

2021-11-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37168. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34443

[jira] [Assigned] (SPARK-37164) Add ExpressionBuilder for functions with complex overloads

2021-11-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37164: --- Assignee: Wenchen Fan > Add ExpressionBuilder for functions with complex overloads >

[jira] [Resolved] (SPARK-37164) Add ExpressionBuilder for functions with complex overloads

2021-11-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37164. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34441