[jira] [Resolved] (SPARK-36626) Support TimestampNTZ in createDataFrame/toPandas and Python UDFs

2021-09-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36626. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33876

[jira] [Assigned] (SPARK-36626) Support TimestampNTZ in createDataFrame/toPandas and Python UDFs

2021-09-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36626: Assignee: Hyukjin Kwon > Support TimestampNTZ in createDataFrame/toPandas and Python

[jira] [Resolved] (SPARK-36631) Ask users if they want to download and install SparkR in non Spark scripts

2021-09-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36631. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33887

[jira] [Assigned] (SPARK-36631) Ask users if they want to download and install SparkR in non Spark scripts

2021-09-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36631: Assignee: Hyukjin Kwon > Ask users if they want to download and install SparkR in non

[jira] [Updated] (SPARK-36635) spark-sql do NOT support that select name expression as string type now

2021-09-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36635: - Component/s: (was: Block Manager) SQL > spark-sql do NOT support that

[jira] [Updated] (SPARK-36635) spark-sql do NOT support that select name expression as string type now

2021-09-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36635: - Description: The follow statement would throw an exception. {code:java} sql("SELECT age as

[jira] [Commented] (SPARK-36635) spark-sql do NOT support that select name expression as string type now

2021-09-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408530#comment-17408530 ] Hyukjin Kwon commented on SPARK-36635: -- did it work before? > spark-sql do NOT support that

[jira] [Updated] (SPARK-36575) Executor lost may cause spark stage to hang

2021-09-01 Thread hujiahua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hujiahua updated SPARK-36575: - Description: When a executor finished a task of some stage, the driver will receive a `StatusUpdate`

[jira] [Updated] (SPARK-36575) Executor lost may cause spark stage to hang

2021-09-01 Thread hujiahua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hujiahua updated SPARK-36575: - Description: When a executor finished a task of some stage, the driver will receive a `StatusUpdate`

[jira] [Resolved] (SPARK-36621) Add Apache license header to Pandas API on Spark userguides

2021-09-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36621. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33871

[jira] [Assigned] (SPARK-36621) Add Apache license header to Pandas API on Spark userguides

2021-09-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36621: Assignee: Leona Yoda > Add Apache license header to Pandas API on Spark userguides >

[jira] [Commented] (SPARK-36438) Support list-like Python objects for Series comparison

2021-09-01 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408505#comment-17408505 ] Haejoon Lee commented on SPARK-36438: - I'm working on this > Support list-like Python objects for

[jira] [Updated] (SPARK-36575) Executor lost may cause spark stage to hang

2021-09-01 Thread hujiahua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hujiahua updated SPARK-36575: - Description: When a executor finished a task of some stage, the driver will receive a `StatusUpdate`

[jira] [Updated] (SPARK-36575) Executor lost may cause spark stage to hang

2021-09-01 Thread hujiahua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hujiahua updated SPARK-36575: - Affects Version/s: (was: 3.1.2) 2.3.3 > Executor lost may cause spark stage

[jira] [Created] (SPARK-36648) Implement __setitem__ of label-based MultiIndex

2021-09-01 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36648: Summary: Implement __setitem__ of label-based MultiIndex Key: SPARK-36648 URL: https://issues.apache.org/jira/browse/SPARK-36648 Project: Spark Issue Type:

[jira] [Updated] (SPARK-36628) Implement __getitem__ of label-based MultiIndex

2021-09-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36628: - Summary: Implement __getitem__ of label-based MultiIndex (was: Implement label-based

[jira] [Created] (SPARK-36647) Push down filter by partition column for Aggregate (Min/Max/Count) for Parquet

2021-09-01 Thread Huaxin Gao (Jira)
Huaxin Gao created SPARK-36647: -- Summary: Push down filter by partition column for Aggregate (Min/Max/Count) for Parquet Key: SPARK-36647 URL: https://issues.apache.org/jira/browse/SPARK-36647 Project:

[jira] [Created] (SPARK-36646) push down group by partition column for Aggregate (Min/Max/Count) for Parquet

2021-09-01 Thread Huaxin Gao (Jira)
Huaxin Gao created SPARK-36646: -- Summary: push down group by partition column for Aggregate (Min/Max/Count) for Parquet Key: SPARK-36646 URL: https://issues.apache.org/jira/browse/SPARK-36646 Project:

[jira] [Assigned] (SPARK-36645) Aggregate (Min/Max/Count) push down for Parquet

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36645: Assignee: (was: Apache Spark) > Aggregate (Min/Max/Count) push down for Parquet >

[jira] [Commented] (SPARK-36645) Aggregate (Min/Max/Count) push down for Parquet

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408359#comment-17408359 ] Apache Spark commented on SPARK-36645: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36645) Aggregate (Min/Max/Count) push down for Parquet

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36645: Assignee: Apache Spark > Aggregate (Min/Max/Count) push down for Parquet >

[jira] [Created] (SPARK-36645) Aggregate (Min/Max/Count) push down for Parquet

2021-09-01 Thread Huaxin Gao (Jira)
Huaxin Gao created SPARK-36645: -- Summary: Aggregate (Min/Max/Count) push down for Parquet Key: SPARK-36645 URL: https://issues.apache.org/jira/browse/SPARK-36645 Project: Spark Issue Type:

[jira] [Created] (SPARK-36644) Push down boolean column filter

2021-09-01 Thread Kazuyuki Tanimura (Jira)
Kazuyuki Tanimura created SPARK-36644: - Summary: Push down boolean column filter Key: SPARK-36644 URL: https://issues.apache.org/jira/browse/SPARK-36644 Project: Spark Issue Type:

[jira] [Commented] (SPARK-36644) Push down boolean column filter

2021-09-01 Thread Kazuyuki Tanimura (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408348#comment-17408348 ] Kazuyuki Tanimura commented on SPARK-36644: --- I am working on this issue > Push down boolean

[jira] [Commented] (SPARK-36622) spark.history.kerberos.principal doesn't take value _HOST

2021-09-01 Thread Thejdeep Gudivada (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408341#comment-17408341 ] Thejdeep Gudivada commented on SPARK-36622: --- Is it possible to just use bash eval of

[jira] [Commented] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408339#comment-17408339 ] Apache Spark commented on SPARK-36639: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36639: Assignee: (was: Apache Spark) > SQL sequence function with interval returns

[jira] [Assigned] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36639: Assignee: Apache Spark > SQL sequence function with interval returns unexpected error in

[jira] [Commented] (SPARK-34276) Check the unreleased/unresolved JIRAs/PRs of Parquet 1.11 and 1.12

2021-09-01 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408324#comment-17408324 ] Chao Sun commented on SPARK-34276: -- I did some study on the code and it seems this will only affect

[jira] [Updated] (SPARK-36643) Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set

2021-09-01 Thread Senthil Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Senthil Kumar updated SPARK-36643: -- Component/s: SQL > Add more information in ERROR log while SparkConf is modified when >

[jira] [Commented] (SPARK-36643) Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408292#comment-17408292 ] Apache Spark commented on SPARK-36643: -- User 'senthh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36643) Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36643: Assignee: Apache Spark > Add more information in ERROR log while SparkConf is modified

[jira] [Commented] (SPARK-36643) Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408290#comment-17408290 ] Apache Spark commented on SPARK-36643: -- User 'senthh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36643) Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36643: Assignee: (was: Apache Spark) > Add more information in ERROR log while SparkConf is

[jira] [Comment Edited] (SPARK-36643) Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set

2021-09-01 Thread Senthil Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408282#comment-17408282 ] Senthil Kumar edited comment on SPARK-36643 at 9/1/21, 5:20 PM: New

[jira] [Commented] (SPARK-36643) Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set

2021-09-01 Thread Senthil Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408282#comment-17408282 ] Senthil Kumar commented on SPARK-36643: --- New ERROR message will be as below,   {code:java} scala>

[jira] [Commented] (SPARK-36643) Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set

2021-09-01 Thread Senthil Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408274#comment-17408274 ] Senthil Kumar commented on SPARK-36643: --- Creating PR for this > Add more information in ERROR log

[jira] [Created] (SPARK-36643) Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set

2021-09-01 Thread Senthil Kumar (Jira)
Senthil Kumar created SPARK-36643: - Summary: Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set Key: SPARK-36643 URL:

[jira] [Commented] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-01 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408267#comment-17408267 ] Kousuke Saruta commented on SPARK-36639: [~amstel8] Thank you for reporting. SPARK-31980 seems

[jira] [Commented] (SPARK-36642) Add df.withMetadata: a syntax suger to update the metadata of a dataframe

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408255#comment-17408255 ] Apache Spark commented on SPARK-36642: -- User 'liangz1' has created a pull request for this issue:

[jira] [Commented] (SPARK-36642) Add df.withMetadata: a syntax suger to update the metadata of a dataframe

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408254#comment-17408254 ] Apache Spark commented on SPARK-36642: -- User 'liangz1' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36642) Add df.withMetadata: a syntax suger to update the metadata of a dataframe

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36642: Assignee: Apache Spark > Add df.withMetadata: a syntax suger to update the metadata of a

[jira] [Assigned] (SPARK-36642) Add df.withMetadata: a syntax suger to update the metadata of a dataframe

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36642: Assignee: (was: Apache Spark) > Add df.withMetadata: a syntax suger to update the

[jira] [Updated] (SPARK-36641) Support using SQL string as a general Data Source V2 Expression

2021-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-36641: Description: In Data Source V2 APIs, we kind of need to fork the catalyst Expressions and create

[jira] [Created] (SPARK-36642) Add df.withMetadata: a syntax suger to update the metadata of a dataframe

2021-09-01 Thread Liang Zhang (Jira)
Liang Zhang created SPARK-36642: --- Summary: Add df.withMetadata: a syntax suger to update the metadata of a dataframe Key: SPARK-36642 URL: https://issues.apache.org/jira/browse/SPARK-36642 Project:

[jira] [Created] (SPARK-36641) Support using SQL string as a general Data Source V2 Expression

2021-09-01 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-36641: --- Summary: Support using SQL string as a general Data Source V2 Expression Key: SPARK-36641 URL: https://issues.apache.org/jira/browse/SPARK-36641 Project: Spark

[jira] [Updated] (SPARK-36640) More operator pushdown API in Data Source V2

2021-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-36640: Description: As of Spark 3.2, we have DS v2 pushdown APIs for Filter, Aggregate, and column

[jira] [Created] (SPARK-36640) More operator pushdown API in Data Source V2

2021-09-01 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-36640: --- Summary: More operator pushdown API in Data Source V2 Key: SPARK-36640 URL: https://issues.apache.org/jira/browse/SPARK-36640 Project: Spark Issue Type:

[jira] [Updated] (SPARK-34651) Improve ZSTD support

2021-09-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34651: -- Labels: releasenotes (was: ) > Improve ZSTD support > > >

[jira] [Resolved] (SPARK-34651) Improve ZSTD support

2021-09-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34651. --- Fix Version/s: 3.2.0 Resolution: Fixed > Improve ZSTD support >

[jira] [Updated] (SPARK-36629) Upgrade aircompressor to 1.21

2021-09-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36629: -- Parent: SPARK-34651 Issue Type: Sub-task (was: Bug) > Upgrade aircompressor to 1.21

[jira] [Updated] (SPARK-35181) Use zstd for spark.io.compression.codec by default

2021-09-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35181: -- Parent: (was: SPARK-34651) Issue Type: Improvement (was: Sub-task) > Use zstd

[jira] [Assigned] (SPARK-36638) Generalize OptimizeSkewedJoin

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36638: Assignee: Apache Spark > Generalize OptimizeSkewedJoin > - >

[jira] [Assigned] (SPARK-36638) Generalize OptimizeSkewedJoin

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36638: Assignee: (was: Apache Spark) > Generalize OptimizeSkewedJoin >

[jira] [Commented] (SPARK-36638) Generalize OptimizeSkewedJoin

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408141#comment-17408141 ] Apache Spark commented on SPARK-36638: -- User 'zhengruifeng' has created a pull request for this

[jira] [Commented] (SPARK-36638) Generalize OptimizeSkewedJoin

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408140#comment-17408140 ] Apache Spark commented on SPARK-36638: -- User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-01 Thread Ignatiy Vdovichenko (Jira)
Ignatiy Vdovichenko created SPARK-36639: --- Summary: SQL sequence function with interval returns unexpected error in latest versions Key: SPARK-36639 URL: https://issues.apache.org/jira/browse/SPARK-36639

[jira] [Created] (SPARK-36638) Generalize OptimizeSkewedJoin

2021-09-01 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-36638: Summary: Generalize OptimizeSkewedJoin Key: SPARK-36638 URL: https://issues.apache.org/jira/browse/SPARK-36638 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-36637) Bad error message when using non-existing named window

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408099#comment-17408099 ] Apache Spark commented on SPARK-36637: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-36637) Bad error message when using non-existing named window

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408098#comment-17408098 ] Apache Spark commented on SPARK-36637: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-36637) Bad error message when using non-existing named window

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36637: Assignee: Apache Spark > Bad error message when using non-existing named window >

[jira] [Assigned] (SPARK-36637) Bad error message when using non-existing named window

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36637: Assignee: (was: Apache Spark) > Bad error message when using non-existing named

[jira] [Issue Comment Deleted] (SPARK-36633) DivideDTInterval should consider ansi mode.

2021-09-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-36633: --- Comment: was deleted (was: I'm working on.) > DivideDTInterval should consider ansi mode. >

[jira] [Commented] (SPARK-36633) DivideDTInterval should consider ansi mode.

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408075#comment-17408075 ] Apache Spark commented on SPARK-36633: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36633) DivideDTInterval should consider ansi mode.

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36633: Assignee: (was: Apache Spark) > DivideDTInterval should consider ansi mode. >

[jira] [Commented] (SPARK-36633) DivideDTInterval should consider ansi mode.

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408074#comment-17408074 ] Apache Spark commented on SPARK-36633: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36633) DivideDTInterval should consider ansi mode.

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36633: Assignee: Apache Spark > DivideDTInterval should consider ansi mode. >

[jira] [Issue Comment Deleted] (SPARK-36632) DivideYMInterval should consider ansi mode.

2021-09-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-36632: --- Comment: was deleted (was: I'm working on.) > DivideYMInterval should consider ansi mode. >

[jira] [Assigned] (SPARK-36442) Do not reference the deprecated UserDefinedAggregateFunction

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36442: Assignee: (was: Apache Spark) > Do not reference the deprecated

[jira] [Commented] (SPARK-36442) Do not reference the deprecated UserDefinedAggregateFunction

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408034#comment-17408034 ] Apache Spark commented on SPARK-36442: -- User 'yutoacts' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36442) Do not reference the deprecated UserDefinedAggregateFunction

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36442: Assignee: Apache Spark > Do not reference the deprecated UserDefinedAggregateFunction >

[jira] [Commented] (SPARK-36632) DivideYMInterval should consider ansi mode.

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407982#comment-17407982 ] Apache Spark commented on SPARK-36632: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36632) DivideYMInterval should consider ansi mode.

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36632: Assignee: (was: Apache Spark) > DivideYMInterval should consider ansi mode. >

[jira] [Assigned] (SPARK-36632) DivideYMInterval should consider ansi mode.

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36632: Assignee: Apache Spark > DivideYMInterval should consider ansi mode. >

[jira] [Created] (SPARK-36637) Bad error message when using non-existing named window

2021-09-01 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-36637: --- Summary: Bad error message when using non-existing named window Key: SPARK-36637 URL: https://issues.apache.org/jira/browse/SPARK-36637 Project: Spark Issue

[jira] [Commented] (SPARK-34276) Check the unreleased/unresolved JIRAs/PRs of Parquet 1.11 and 1.12

2021-09-01 Thread Gabor Szadovszky (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407960#comment-17407960 ] Gabor Szadovszky commented on SPARK-34276: -- [~csun], any application is using parquet-mr 1.12.0

[jira] [Commented] (SPARK-36634) Support access and read parquet file by column index

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407957#comment-17407957 ] Apache Spark commented on SPARK-36634: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36634) Support access and read parquet file by column index

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36634: Assignee: (was: Apache Spark) > Support access and read parquet file by column index

[jira] [Assigned] (SPARK-36634) Support access and read parquet file by column index

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36634: Assignee: Apache Spark > Support access and read parquet file by column index >

[jira] [Created] (SPARK-36636) SparkContextSuite random failure in Scala 2.13

2021-09-01 Thread Yang Jie (Jira)
Yang Jie created SPARK-36636: Summary: SparkContextSuite random failure in Scala 2.13 Key: SPARK-36636 URL: https://issues.apache.org/jira/browse/SPARK-36636 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-36635) spark-sql do NOT support that select name expression as string type now

2021-09-01 Thread weixiuli (Jira)
weixiuli created SPARK-36635: Summary: spark-sql do NOT support that select name expression as string type now Key: SPARK-36635 URL: https://issues.apache.org/jira/browse/SPARK-36635 Project: Spark

[jira] [Created] (SPARK-36634) Support access and read parquet file by column index

2021-09-01 Thread Kent Yao (Jira)
Kent Yao created SPARK-36634: Summary: Support access and read parquet file by column index Key: SPARK-36634 URL: https://issues.apache.org/jira/browse/SPARK-36634 Project: Spark Issue Type:

[jira] [Created] (SPARK-36633) DivideDTInterval should consider ansi mode.

2021-09-01 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-36633: -- Summary: DivideDTInterval should consider ansi mode. Key: SPARK-36633 URL: https://issues.apache.org/jira/browse/SPARK-36633 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-36632) DivideYMInterval should consider ansi mode.

2021-09-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-36632: --- Description: DivideYMInterval not consider the ansi mode, we should support it. (was:

[jira] [Commented] (SPARK-36633) DivideDTInterval should consider ansi mode.

2021-09-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407947#comment-17407947 ] jiaan.geng commented on SPARK-36633: I'm working on. > DivideDTInterval should consider ansi mode.

[jira] [Updated] (SPARK-36632) DivideYMInterval should consider ansi mode.

2021-09-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-36632: --- Summary: DivideYMInterval should consider ansi mode. (was: DivideInterval should consider ansi

[jira] [Commented] (SPARK-36631) Ask users if they want to download and install SparkR in non Spark scripts

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407943#comment-17407943 ] Apache Spark commented on SPARK-36631: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-36631) Ask users if they want to download and install SparkR in non Spark scripts

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36631: Assignee: (was: Apache Spark) > Ask users if they want to download and install

[jira] [Commented] (SPARK-36631) Ask users if they want to download and install SparkR in non Spark scripts

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407941#comment-17407941 ] Apache Spark commented on SPARK-36631: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-36631) Ask users if they want to download and install SparkR in non Spark scripts

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36631: Assignee: Apache Spark > Ask users if they want to download and install SparkR in non

[jira] [Updated] (SPARK-36630) Add the option to use physical statistics to avoid large tables being broadcast

2021-09-01 Thread gaoyajun02 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaoyajun02 updated SPARK-36630: --- Description: Currently when AQE's queryStage is not materialized, it uses the stats of the logical

[jira] [Assigned] (SPARK-36604) timestamp type column analyze result is wrong

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36604: Assignee: Apache Spark > timestamp type column analyze result is wrong >

[jira] [Commented] (SPARK-36604) timestamp type column analyze result is wrong

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407918#comment-17407918 ] Apache Spark commented on SPARK-36604: -- User 'fhygh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36604) timestamp type column analyze result is wrong

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36604: Assignee: (was: Apache Spark) > timestamp type column analyze result is wrong >

[jira] [Commented] (SPARK-36400) Redact sensitive information in Spark Thrift Server UI

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407915#comment-17407915 ] Apache Spark commented on SPARK-36400: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36575) Executor lost may cause spark stage to hang

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36575: Assignee: (was: Apache Spark) > Executor lost may cause spark stage to hang >

[jira] [Assigned] (SPARK-36575) Executor lost may cause spark stage to hang

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36575: Assignee: Apache Spark > Executor lost may cause spark stage to hang >

[jira] [Commented] (SPARK-36575) Executor lost may cause spark stage to hang

2021-09-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407894#comment-17407894 ] Apache Spark commented on SPARK-36575: -- User 'sleep1661' has created a pull request for this issue:

[jira] [Commented] (SPARK-36632) DivideInterval should consider ansi mode.

2021-09-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407885#comment-17407885 ] jiaan.geng commented on SPARK-36632: I'm working on. > DivideInterval should consider ansi mode. >

[jira] [Created] (SPARK-36632) DivideInterval should consider ansi mode.

2021-09-01 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-36632: -- Summary: DivideInterval should consider ansi mode. Key: SPARK-36632 URL: https://issues.apache.org/jira/browse/SPARK-36632 Project: Spark Issue Type: Sub-task

  1   2   >