[jira] [Updated] (SPARK-36329) show api of Dataset should get as input the output method

2021-07-28 Thread Izek Greenfield (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Izek Greenfield updated SPARK-36329: Description: For now show is: {code:scala} def show(numRows: Int, truncate: Boolean):

[jira] [Resolved] (SPARK-36323) Support ANSI interval literals for TimeWindow

2021-07-28 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36323. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33551

[jira] [Commented] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389271#comment-17389271 ] Apache Spark commented on SPARK-35806: -- User 'itholic' has created a pull request for this issue:

[jira] [Commented] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389270#comment-17389270 ] Apache Spark commented on SPARK-35806: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36335) Document local-cluster mode option in developper tools doc

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36335: Assignee: Apache Spark > Document local-cluster mode option in developper tools doc >

[jira] [Assigned] (SPARK-36335) Document local-cluster mode option in developper tools doc

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36335: Assignee: (was: Apache Spark) > Document local-cluster mode option in developper

[jira] [Commented] (SPARK-36335) Document local-cluster mode option in developper tools doc

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389269#comment-17389269 ] Apache Spark commented on SPARK-36335: -- User 'yutoacts' has created a pull request for this issue:

[jira] [Updated] (SPARK-36338) Move distributed-sequence implementation to Scala side

2021-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36338: - Summary: Move distributed-sequence implementation to Scala side (was: Move

[jira] [Updated] (SPARK-36339) aggsBuffer should collect AggregateExpression in the map range

2021-07-28 Thread gaoyajun02 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaoyajun02 updated SPARK-36339: --- Description: show demo for this ISSUE: {code:java} // SQL without error SELECT name, count(name) c

[jira] [Created] (SPARK-36339) aggsBuffer should collect AggregateExpression in the map range

2021-07-28 Thread gaoyajun02 (Jira)
gaoyajun02 created SPARK-36339: -- Summary: aggsBuffer should collect AggregateExpression in the map range Key: SPARK-36339 URL: https://issues.apache.org/jira/browse/SPARK-36339 Project: Spark

[jira] [Created] (SPARK-36338) Move distributed-sequence implementation to Scala side for simplicity

2021-07-28 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-36338: Summary: Move distributed-sequence implementation to Scala side for simplicity Key: SPARK-36338 URL: https://issues.apache.org/jira/browse/SPARK-36338 Project: Spark

[jira] [Assigned] (SPARK-36254) Install mlflow in Github Actions CI

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36254: Assignee: (was: Apache Spark) > Install mlflow in Github Actions CI >

[jira] [Commented] (SPARK-36254) Install mlflow in Github Actions CI

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389255#comment-17389255 ] Apache Spark commented on SPARK-36254: -- User 'itholic' has created a pull request for this issue:

[jira] [Commented] (SPARK-36254) Install mlflow in Github Actions CI

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389256#comment-17389256 ] Apache Spark commented on SPARK-36254: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36254) Install mlflow in Github Actions CI

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36254: Assignee: Apache Spark > Install mlflow in Github Actions CI >

[jira] [Updated] (SPARK-36254) Install mlflow in Github Actions CI

2021-07-28 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-36254: Description: Since the pandas-on-Spark includes the mlflow features and related tests, we should

[jira] [Comment Edited] (SPARK-36000) Support creation and operations of ps.Series/Index with Decimal('NaN')

2021-07-28 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389243#comment-17389243 ] Yikun Jiang edited comment on SPARK-36000 at 7/29/21, 4:15 AM: ---

[jira] [Updated] (SPARK-36254) Install mlflow in Github Actions CI

2021-07-28 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-36254: Description: Since the pandas-on-Spark includes the mlflow features and related tests, we should

[jira] [Updated] (SPARK-36254) Install mlflow in Github Actions CI

2021-07-28 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-36254: Summary: Install mlflow in Github Actions CI (was: Install mlflow and delta in Github Actions

[jira] [Updated] (SPARK-36337) decimal('Nan') is unsupported in net.razorvine.pickle

2021-07-28 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36337: Description: Decimal('NaN') is not supported by net.razorvine.pickle now. In Python {code:java}

[jira] [Comment Edited] (SPARK-36000) Support creation and operations of ps.Series/Index with Decimal('NaN')

2021-07-28 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389243#comment-17389243 ] Yikun Jiang edited comment on SPARK-36000 at 7/29/21, 4:02 AM: ---

[jira] [Created] (SPARK-36337) decimal('Nan') is unsupported in net.razorvine.pickle

2021-07-28 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-36337: --- Summary: decimal('Nan') is unsupported in net.razorvine.pickle Key: SPARK-36337 URL: https://issues.apache.org/jira/browse/SPARK-36337 Project: Spark Issue

[jira] [Commented] (SPARK-36000) Support creation and operations of ps.Series/Index with Decimal('NaN')

2021-07-28 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389243#comment-17389243 ] Yikun Jiang commented on SPARK-36000: - [~XinrongM] I did some investigation on this, I found the

[jira] [Commented] (SPARK-36336) Define the new exception that mix SparkThrowable for all base exe in QueryExecutionErrors

2021-07-28 Thread PengLei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389236#comment-17389236 ] PengLei commented on SPARK-36336: - I am woking on this > Define the new exception that mix

[jira] [Commented] (SPARK-36094) Group SQL component error messages in Spark error class JSON file

2021-07-28 Thread PengLei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389235#comment-17389235 ] PengLei commented on SPARK-36094: - [~karenfeng] Before refactor all query execution, I want to define

[jira] [Updated] (SPARK-36336) Define the new exception that mix SparkThrowable for all base exe in QueryExecutionErrors

2021-07-28 Thread PengLei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PengLei updated SPARK-36336: Description: The Throwable should extend 

[jira] [Created] (SPARK-36336) Define the new exception that mix SparkThrowable for all base exe in QueryExecutionErrors

2021-07-28 Thread PengLei (Jira)
PengLei created SPARK-36336: --- Summary: Define the new exception that mix SparkThrowable for all base exe in QueryExecutionErrors Key: SPARK-36336 URL: https://issues.apache.org/jira/browse/SPARK-36336

[jira] [Commented] (SPARK-36271) Hive SerDe and V1 Insert data to parquet/orc/avro need to check schema too

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389226#comment-17389226 ] Apache Spark commented on SPARK-36271: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-36271) Hive SerDe and V1 Insert data to parquet/orc/avro need to check schema too

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36271: Assignee: Apache Spark > Hive SerDe and V1 Insert data to parquet/orc/avro need to check

[jira] [Assigned] (SPARK-36271) Hive SerDe and V1 Insert data to parquet/orc/avro need to check schema too

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36271: Assignee: (was: Apache Spark) > Hive SerDe and V1 Insert data to parquet/orc/avro

[jira] [Updated] (SPARK-36271) Hive SerDe and V1 Insert data to parquet/orc/avro need to check schema too

2021-07-28 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-36271: -- Summary: Hive SerDe and V1 Insert data to parquet/orc/avro need to check schema too (was: Insert

[jira] [Resolved] (SPARK-36272) Re-enable SPARK-32629 ShuffledHashJoin(full outer) metrics

2021-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36272. -- Fix Version/s: 3.1.3 3.2.0 Assignee: Cheng Su Resolution:

[jira] [Created] (SPARK-36335) Document local-cluster mode option in developper tools doc

2021-07-28 Thread Yuto Akutsu (Jira)
Yuto Akutsu created SPARK-36335: --- Summary: Document local-cluster mode option in developper tools doc Key: SPARK-36335 URL: https://issues.apache.org/jira/browse/SPARK-36335 Project: Spark

[jira] [Resolved] (SPARK-36286) Block some invalid datetime string

2021-07-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36286. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33490

[jira] [Assigned] (SPARK-36286) Block some invalid datetime string

2021-07-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36286: --- Assignee: Linhong Liu > Block some invalid datetime string >

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-07-28 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389144#comment-17389144 ] L. C. Hsieh commented on SPARK-18105: - Looked at lz4 codebase and the reported failures. I suspect

[jira] [Commented] (SPARK-33298) Introduce new API to FileCommitProtocol allow flexible file naming

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389141#comment-17389141 ] Apache Spark commented on SPARK-33298: -- User 'c21' has created a pull request for this issue:

[jira] [Commented] (SPARK-36136) Move PruneFileSourcePartitionsSuite out of org.apache.spark.sql.hive

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389129#comment-17389129 ] Apache Spark commented on SPARK-36136: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36334) Add a new conf to allow K8s API server-side cache for pod listing

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36334: Assignee: (was: Apache Spark) > Add a new conf to allow K8s API server-side cache

[jira] [Updated] (SPARK-36334) Add a new conf to allow K8s API server-side cache for pod listing

2021-07-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36334: -- Description: -

[jira] [Assigned] (SPARK-36334) Add a new conf to allow K8s API server-side cache for pod listing

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36334: Assignee: Apache Spark > Add a new conf to allow K8s API server-side cache for pod

[jira] [Commented] (SPARK-36334) Add a new conf to allow K8s API server-side cache for pod listing

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389125#comment-17389125 ] Apache Spark commented on SPARK-36334: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Created] (SPARK-36334) Add a new conf to allow K8s API server-side cache for pod listing

2021-07-28 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-36334: - Summary: Add a new conf to allow K8s API server-side cache for pod listing Key: SPARK-36334 URL: https://issues.apache.org/jira/browse/SPARK-36334 Project: Spark

[jira] [Commented] (SPARK-36333) Reuse isnull where the null check is needed.

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389124#comment-17389124 ] Apache Spark commented on SPARK-36333: -- User 'ueshin' has created a pull request for this issue:

[jira] [Commented] (SPARK-36333) Reuse isnull where the null check is needed.

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389123#comment-17389123 ] Apache Spark commented on SPARK-36333: -- User 'ueshin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36333) Reuse isnull where the null check is needed.

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36333: Assignee: (was: Apache Spark) > Reuse isnull where the null check is needed. >

[jira] [Assigned] (SPARK-36333) Reuse isnull where the null check is needed.

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36333: Assignee: Apache Spark > Reuse isnull where the null check is needed. >

[jira] [Updated] (SPARK-36333) Reuse isnull where the null check is needed.

2021-07-28 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-36333: -- Summary: Reuse isnull where the null check is needed. (was: Reuse isnull where the

[jira] [Resolved] (SPARK-22390) Aggregate push down

2021-07-28 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-22390. Resolution: Duplicate > Aggregate push down > --- > > Key:

[jira] [Resolved] (SPARK-32833) JDBC V2 Datasource aggregate push down

2021-07-28 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-32833. Resolution: Duplicate > JDBC V2 Datasource aggregate push down >

[jira] [Created] (SPARK-36333) Reuse isnull where the nullability check is needed.

2021-07-28 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-36333: - Summary: Reuse isnull where the nullability check is needed. Key: SPARK-36333 URL: https://issues.apache.org/jira/browse/SPARK-36333 Project: Spark Issue

[jira] [Resolved] (SPARK-36190) Improve the rest of DataTypeOps tests by avoiding joins

2021-07-28 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36190. --- Fix Version/s: 3.2.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Assigned] (SPARK-36332) Cleanup RemoteBlockPushResolver log messages

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36332: Assignee: (was: Apache Spark) > Cleanup RemoteBlockPushResolver log messages >

[jira] [Commented] (SPARK-36332) Cleanup RemoteBlockPushResolver log messages

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389030#comment-17389030 ] Apache Spark commented on SPARK-36332: -- User 'venkata91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36332) Cleanup RemoteBlockPushResolver log messages

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36332: Assignee: Apache Spark > Cleanup RemoteBlockPushResolver log messages >

[jira] [Updated] (SPARK-36094) Group SQL component error messages in Spark error class JSON file

2021-07-28 Thread Karen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Feng updated SPARK-36094: --- Description: To improve auditing, reduce duplication, and improve quality of error messages thrown

[jira] [Created] (SPARK-36332) Cleanup RemoteBlockPushResolver log messages

2021-07-28 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-36332: --- Summary: Cleanup RemoteBlockPushResolver log messages Key: SPARK-36332 URL: https://issues.apache.org/jira/browse/SPARK-36332 Project:

[jira] [Resolved] (SPARK-36236) RocksDB state store: Add additional metrics for better observability into state store operations

2021-07-28 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-36236. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33455

[jira] [Assigned] (SPARK-36236) RocksDB state store: Add additional metrics for better observability into state store operations

2021-07-28 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-36236: --- Assignee: Venki Korukanti > RocksDB state store: Add additional metrics for better

[jira] [Commented] (SPARK-36331) Add SQLSTATE guideline

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389005#comment-17389005 ] Apache Spark commented on SPARK-36331: -- User 'karenfeng' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36331) Add SQLSTATE guideline

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36331: Assignee: (was: Apache Spark) > Add SQLSTATE guideline > -- > >

[jira] [Assigned] (SPARK-36331) Add SQLSTATE guideline

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36331: Assignee: Apache Spark > Add SQLSTATE guideline > -- > >

[jira] [Created] (SPARK-36331) Add SQLSTATE guideline

2021-07-28 Thread Karen Feng (Jira)
Karen Feng created SPARK-36331: -- Summary: Add SQLSTATE guideline Key: SPARK-36331 URL: https://issues.apache.org/jira/browse/SPARK-36331 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-36330) Use atomic renames for diskstore block writes & do "fall-through" reads

2021-07-28 Thread Holden Karau (Jira)
Holden Karau created SPARK-36330: Summary: Use atomic renames for diskstore block writes & do "fall-through" reads Key: SPARK-36330 URL: https://issues.apache.org/jira/browse/SPARK-36330 Project:

[jira] [Commented] (SPARK-36024) Switch the datasource example due to the depreciation of the dataset

2021-07-28 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388969#comment-17388969 ] Steve Loughran commented on SPARK-36024: yes, you can change the example. For hadoop we're

[jira] [Created] (SPARK-36329) show api of Dataset should get as input the output method

2021-07-28 Thread Izek Greenfield (Jira)
Izek Greenfield created SPARK-36329: --- Summary: show api of Dataset should get as input the output method Key: SPARK-36329 URL: https://issues.apache.org/jira/browse/SPARK-36329 Project: Spark

[jira] [Resolved] (SPARK-36143) Adjust `astype` of fractional Series with missing values to follow pandas

2021-07-28 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36143. --- Fix Version/s: 3.2.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Assigned] (SPARK-36229) conv() inconsistently handles invalid strings with > 64 invalid characters

2021-07-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36229: --- Assignee: dgd_contributor > conv() inconsistently handles invalid strings with > 64

[jira] [Resolved] (SPARK-36229) conv() inconsistently handles invalid strings with > 64 invalid characters

2021-07-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36229. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33459

[jira] [Updated] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

2021-07-28 Thread Eugene Koifman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated SPARK-35639: --- Description: {{CustomShuffleReaderExec}} reports "number of skewed partitions" and "number

[jira] [Assigned] (SPARK-36095) Group exception messages in core/rdd

2021-07-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36095: --- Assignee: dgd_contributor > Group exception messages in core/rdd >

[jira] [Resolved] (SPARK-36095) Group exception messages in core/rdd

2021-07-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36095. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33317

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-07-28 Thread Arghya Saha (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388797#comment-17388797 ] Arghya Saha commented on SPARK-18105: - [~dongjoon] Can we please address this before next release,

[jira] [Comment Edited] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-07-28 Thread Arghya Saha (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388794#comment-17388794 ] Arghya Saha edited comment on SPARK-18105 at 7/28/21, 1:55 PM: --- I am also

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-07-28 Thread Arghya Saha (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388794#comment-17388794 ] Arghya Saha commented on SPARK-18105: - I am also facing the same error, I have raised a duplicate

[jira] [Updated] (SPARK-36196) Spark FetchFailedException Stream is corrupted Error

2021-07-28 Thread Arghya Saha (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arghya Saha updated SPARK-36196: Affects Version/s: (was: 3.1.2) > Spark FetchFailedException Stream is corrupted Error >

[jira] [Commented] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388739#comment-17388739 ] Apache Spark commented on SPARK-34265: -- User 'LucaCanali' has created a pull request for this

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2021-07-28 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Attachment: Python_UDF_instrumentation_lite_BatchEvalPython.png > Instrument Python UDF execution

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2021-07-28 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Attachment: Python_UDF_instrumentation_lite_ArrowEvalPython.png > Instrument Python UDF execution

[jira] [Updated] (SPARK-36328) HadoopRDD#getPartitions fetches FileSystem Delegation Token for every partition

2021-07-28 Thread Prabhu Joseph (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated SPARK-36328: -- Description: Spark Job creates a separate JobConf for every RDD (every hive table partition)

[jira] [Updated] (SPARK-36328) HadoopRDD#getPartitions fetches FileSystem Delegation Token for every partition

2021-07-28 Thread Prabhu Joseph (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated SPARK-36328: -- Summary: HadoopRDD#getPartitions fetches FileSystem Delegation Token for every partition

[jira] [Created] (SPARK-36328) HadoopRDD#getPartitions fetches FileSystem Delegation Token for evert partition

2021-07-28 Thread Prabhu Joseph (Jira)
Prabhu Joseph created SPARK-36328: - Summary: HadoopRDD#getPartitions fetches FileSystem Delegation Token for evert partition Key: SPARK-36328 URL: https://issues.apache.org/jira/browse/SPARK-36328

[jira] [Assigned] (SPARK-36320) Fix Series/Index.copy() to drop extra columns.

2021-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36320: Assignee: Takuya Ueshin > Fix Series/Index.copy() to drop extra columns. >

[jira] [Resolved] (SPARK-36320) Fix Series/Index.copy() to drop extra columns.

2021-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36320. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33549

[jira] [Assigned] (SPARK-36326) Use Map.computeIfAbsent to simplify the process of HeapMemoryAllocator.bufferPoolsBySize init new item

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36326: Assignee: Apache Spark > Use Map.computeIfAbsent to simplify the process of >

[jira] [Commented] (SPARK-36326) Use Map.computeIfAbsent to simplify the process of HeapMemoryAllocator.bufferPoolsBySize init new item

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388605#comment-17388605 ] Apache Spark commented on SPARK-36326: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-36326) Use Map.computeIfAbsent to simplify the process of HeapMemoryAllocator.bufferPoolsBySize init new item

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36326: Assignee: (was: Apache Spark) > Use Map.computeIfAbsent to simplify the process of

[jira] [Commented] (SPARK-36327) Spark sql creates staging dir inside database directory rather than creating inside table directory

2021-07-28 Thread Senthil Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388597#comment-17388597 ] Senthil Kumar commented on SPARK-36327: --- Shall I work on this Jira to fix this issue? > Spark sql

[jira] [Created] (SPARK-36327) Spark sql creates staging dir inside database directory rather than creating inside table directory

2021-07-28 Thread Senthil Kumar (Jira)
Senthil Kumar created SPARK-36327: - Summary: Spark sql creates staging dir inside database directory rather than creating inside table directory Key: SPARK-36327 URL:

[jira] [Updated] (SPARK-36326) Use Map.computeIfAbsent to simplify the process of HeapMemoryAllocator.bufferPoolsBySize init new item

2021-07-28 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-36326: - Priority: Trivial (was: Minor) > Use Map.computeIfAbsent to simplify the process of >

[jira] [Created] (SPARK-36326) Use Map.computeIfAbsent to simplify the process of HeapMemoryAllocator.bufferPoolsBySize init new item

2021-07-28 Thread Yang Jie (Jira)
Yang Jie created SPARK-36326: Summary: Use Map.computeIfAbsent to simplify the process of HeapMemoryAllocator.bufferPoolsBySize init new item Key: SPARK-36326 URL: https://issues.apache.org/jira/browse/SPARK-36326

[jira] [Created] (SPARK-36325) Writing to hiveserver throught jdbc throws ParseException

2021-07-28 Thread Jira
Jesús Ricardo Ballesteros Molina created SPARK-36325: Summary: Writing to hiveserver throught jdbc throws ParseException Key: SPARK-36325 URL: https://issues.apache.org/jira/browse/SPARK-36325

[jira] [Commented] (SPARK-36270) Change memory settings for enabling GA

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388583#comment-17388583 ] Apache Spark commented on SPARK-36270: -- User 'linhongliu-db' has created a pull request for this

[jira] [Commented] (SPARK-36270) Change memory settings for enabling GA

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388581#comment-17388581 ] Apache Spark commented on SPARK-36270: -- User 'linhongliu-db' has created a pull request for this

[jira] [Assigned] (SPARK-29285) Temporary shuffle and local block should be able to handle disk failures

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-29285: Assignee: Apache Spark > Temporary shuffle and local block should be able to handle disk

[jira] [Assigned] (SPARK-29285) Temporary shuffle and local block should be able to handle disk failures

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-29285: Assignee: (was: Apache Spark) > Temporary shuffle and local block should be able to

[jira] [Commented] (SPARK-36324) Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388540#comment-17388540 ] Apache Spark commented on SPARK-36324: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-36324) Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388539#comment-17388539 ] Apache Spark commented on SPARK-36324: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-36324) Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36324: Assignee: (was: Apache Spark) > Replace revertPartialWritesAndClose with close in

[jira] [Assigned] (SPARK-36324) Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill

2021-07-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36324: Assignee: Apache Spark > Replace revertPartialWritesAndClose with close in

  1   2   >