[jira] [Created] (SPARK-39344) Only disable bucketing when autoBucketedScan is enabled if bucket columns are not in scan output
Manu Zhang created SPARK-39344: -- Summary: Only disable bucketing when autoBucketedScan is enabled if bucket columns are not in scan output Key: SPARK-39344 URL: https://issues.apache.org/jira/browse/SPARK-39344 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.3.0 Reporter: Manu Zhang Currently, bucketing was disabled when bucket columns are not in scan output after https://github.com/apache/spark/pull/27924. It break existing applications whose input size is huge by creating too many FilePartitions and causing driver hang. And it cannot be switched off. This is to propose merging the rule into DisableUnnecessaryBucketedScan. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39343) DescribeTableExec should redact properties
[ https://issues.apache.org/jira/browse/SPARK-39343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544131#comment-17544131 ] Apache Spark commented on SPARK-39343: -- User 'AngersZh' has created a pull request for this issue: https://github.com/apache/spark/pull/36731 > DescribeTableExec should redact properties > -- > > Key: SPARK-39343 > URL: https://issues.apache.org/jira/browse/SPARK-39343 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: angerszhu >Priority: Major > > DescribeTableExec should redact properties -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39343) DescribeTableExec should redact properties
[ https://issues.apache.org/jira/browse/SPARK-39343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39343: Assignee: (was: Apache Spark) > DescribeTableExec should redact properties > -- > > Key: SPARK-39343 > URL: https://issues.apache.org/jira/browse/SPARK-39343 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: angerszhu >Priority: Major > > DescribeTableExec should redact properties -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39343) DescribeTableExec should redact properties
[ https://issues.apache.org/jira/browse/SPARK-39343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39343: Assignee: Apache Spark > DescribeTableExec should redact properties > -- > > Key: SPARK-39343 > URL: https://issues.apache.org/jira/browse/SPARK-39343 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: angerszhu >Assignee: Apache Spark >Priority: Major > > DescribeTableExec should redact properties -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39343) DescribeTableExec should redact properties
[ https://issues.apache.org/jira/browse/SPARK-39343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544130#comment-17544130 ] Apache Spark commented on SPARK-39343: -- User 'AngersZh' has created a pull request for this issue: https://github.com/apache/spark/pull/36731 > DescribeTableExec should redact properties > -- > > Key: SPARK-39343 > URL: https://issues.apache.org/jira/browse/SPARK-39343 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: angerszhu >Priority: Major > > DescribeTableExec should redact properties -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-39343) DescribeTableExec should redact properties
angerszhu created SPARK-39343: - Summary: DescribeTableExec should redact properties Key: SPARK-39343 URL: https://issues.apache.org/jira/browse/SPARK-39343 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.3.0 Reporter: angerszhu DescribeTableExec should redact properties -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39342) ShowTablePropertiesCommand/ShowTablePropertiesExec should redact properties.
[ https://issues.apache.org/jira/browse/SPARK-39342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544128#comment-17544128 ] Apache Spark commented on SPARK-39342: -- User 'AngersZh' has created a pull request for this issue: https://github.com/apache/spark/pull/36730 > ShowTablePropertiesCommand/ShowTablePropertiesExec should redact properties. > > > Key: SPARK-39342 > URL: https://issues.apache.org/jira/browse/SPARK-39342 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: angerszhu >Priority: Major > > ShowTablePropertiesCommand should redact properties. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39342) ShowTablePropertiesCommand/ShowTablePropertiesExec should redact properties.
[ https://issues.apache.org/jira/browse/SPARK-39342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39342: Assignee: (was: Apache Spark) > ShowTablePropertiesCommand/ShowTablePropertiesExec should redact properties. > > > Key: SPARK-39342 > URL: https://issues.apache.org/jira/browse/SPARK-39342 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: angerszhu >Priority: Major > > ShowTablePropertiesCommand should redact properties. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39342) ShowTablePropertiesCommand/ShowTablePropertiesExec should redact properties.
[ https://issues.apache.org/jira/browse/SPARK-39342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39342: Assignee: Apache Spark > ShowTablePropertiesCommand/ShowTablePropertiesExec should redact properties. > > > Key: SPARK-39342 > URL: https://issues.apache.org/jira/browse/SPARK-39342 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: angerszhu >Assignee: Apache Spark >Priority: Major > > ShowTablePropertiesCommand should redact properties. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39342) ShowTablePropertiesCommand/ShowTablePropertiesExec should redact properties.
[ https://issues.apache.org/jira/browse/SPARK-39342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-39342: -- Summary: ShowTablePropertiesCommand/ShowTablePropertiesExec should redact properties. (was: ShowTablePropertiesCommand should redact properties.) > ShowTablePropertiesCommand/ShowTablePropertiesExec should redact properties. > > > Key: SPARK-39342 > URL: https://issues.apache.org/jira/browse/SPARK-39342 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: angerszhu >Priority: Major > > ShowTablePropertiesCommand should redact properties. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-39342) ShowTablePropertiesCommand should redact properties.
angerszhu created SPARK-39342: - Summary: ShowTablePropertiesCommand should redact properties. Key: SPARK-39342 URL: https://issues.apache.org/jira/browse/SPARK-39342 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.3.0 Reporter: angerszhu ShowTablePropertiesCommand should redact properties. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39295) Improve documentation of pandas API support list.
[ https://issues.apache.org/jira/browse/SPARK-39295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39295: Assignee: (was: Apache Spark) > Improve documentation of pandas API support list. > - > > Key: SPARK-39295 > URL: https://issues.apache.org/jira/browse/SPARK-39295 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Hyunwoo Park >Priority: Major > > The description provided in the supported pandas API list document or the > code comment needs improvement. Also, there are cases where the link of the > function property provided in the document is not connected, so it needs to > be corrected. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39295) Improve documentation of pandas API support list.
[ https://issues.apache.org/jira/browse/SPARK-39295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544108#comment-17544108 ] Apache Spark commented on SPARK-39295: -- User 'beobest2' has created a pull request for this issue: https://github.com/apache/spark/pull/36729 > Improve documentation of pandas API support list. > - > > Key: SPARK-39295 > URL: https://issues.apache.org/jira/browse/SPARK-39295 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Hyunwoo Park >Priority: Major > > The description provided in the supported pandas API list document or the > code comment needs improvement. Also, there are cases where the link of the > function property provided in the document is not connected, so it needs to > be corrected. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39295) Improve documentation of pandas API support list.
[ https://issues.apache.org/jira/browse/SPARK-39295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39295: Assignee: Apache Spark > Improve documentation of pandas API support list. > - > > Key: SPARK-39295 > URL: https://issues.apache.org/jira/browse/SPARK-39295 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Hyunwoo Park >Assignee: Apache Spark >Priority: Major > > The description provided in the supported pandas API list document or the > code comment needs improvement. Also, there are cases where the link of the > function property provided in the document is not connected, so it needs to > be corrected. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-39335) DescribeTableCommand should redact properties
[ https://issues.apache.org/jira/browse/SPARK-39335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-39335. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36722 [https://github.com/apache/spark/pull/36722] > DescribeTableCommand should redact properties > - > > Key: SPARK-39335 > URL: https://issues.apache.org/jira/browse/SPARK-39335 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: angerszhu >Assignee: angerszhu >Priority: Major > Fix For: 3.4.0 > > > Now we only redact storage properties when desc table, for normal properties > should redact too. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39335) DescribeTableCommand should redact properties
[ https://issues.apache.org/jira/browse/SPARK-39335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-39335: --- Assignee: angerszhu > DescribeTableCommand should redact properties > - > > Key: SPARK-39335 > URL: https://issues.apache.org/jira/browse/SPARK-39335 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: angerszhu >Assignee: angerszhu >Priority: Major > > Now we only redact storage properties when desc table, for normal properties > should redact too. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-39262) Correct the behavior of creating DataFrame from an RDD
[ https://issues.apache.org/jira/browse/SPARK-39262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-39262. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36640 [https://github.com/apache/spark/pull/36640] > Correct the behavior of creating DataFrame from an RDD > -- > > Key: SPARK-39262 > URL: https://issues.apache.org/jira/browse/SPARK-39262 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Xinrong Meng >Assignee: Xinrong Meng >Priority: Major > Fix For: 3.4.0 > > > Correct the behavior of creating DataFrame from an RDD **with `0` or an empty > list as the first element**. > > Before: > ```py > >>> spark.createDataFrame(spark._sc.parallelize([0, 1])) > Traceback (most recent call last): > ... > ValueError: The first row in RDD is empty, can not infer schema > >>> spark.createDataFrame(spark._sc.parallelize([[], []])) > Traceback (most recent call last): > ... > ValueError: The first row in RDD is empty, can not infer schema > ``` > After: > ```py > >>> spark.createDataFrame(spark._sc.parallelize([0, 1])) > Traceback (most recent call last): > > ... > TypeError: Can not infer schema for type: > >>> spark.createDataFrame(spark._sc.parallelize([[], []])) > DataFrame[] > > >>> spark.createDataFrame(spark._sc.parallelize([[], []])).show() > ++ > || > ++ > || > || > ++ > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39262) Correct the behavior of creating DataFrame from an RDD
[ https://issues.apache.org/jira/browse/SPARK-39262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-39262: Assignee: Xinrong Meng > Correct the behavior of creating DataFrame from an RDD > -- > > Key: SPARK-39262 > URL: https://issues.apache.org/jira/browse/SPARK-39262 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Xinrong Meng >Assignee: Xinrong Meng >Priority: Major > > Correct the behavior of creating DataFrame from an RDD **with `0` or an empty > list as the first element**. > > Before: > ```py > >>> spark.createDataFrame(spark._sc.parallelize([0, 1])) > Traceback (most recent call last): > ... > ValueError: The first row in RDD is empty, can not infer schema > >>> spark.createDataFrame(spark._sc.parallelize([[], []])) > Traceback (most recent call last): > ... > ValueError: The first row in RDD is empty, can not infer schema > ``` > After: > ```py > >>> spark.createDataFrame(spark._sc.parallelize([0, 1])) > Traceback (most recent call last): > > ... > TypeError: Can not infer schema for type: > >>> spark.createDataFrame(spark._sc.parallelize([[], []])) > DataFrame[] > > >>> spark.createDataFrame(spark._sc.parallelize([[], []])).show() > ++ > || > ++ > || > || > ++ > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39341) KubernetesExecutorBackend should allow IPv6 pod IP
[ https://issues.apache.org/jira/browse/SPARK-39341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39341: - Assignee: William Hyun > KubernetesExecutorBackend should allow IPv6 pod IP > -- > > Key: SPARK-39341 > URL: https://issues.apache.org/jira/browse/SPARK-39341 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: William Hyun >Assignee: William Hyun >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-39341) KubernetesExecutorBackend should allow IPv6 pod IP
[ https://issues.apache.org/jira/browse/SPARK-39341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39341. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 36728 [https://github.com/apache/spark/pull/36728] > KubernetesExecutorBackend should allow IPv6 pod IP > -- > > Key: SPARK-39341 > URL: https://issues.apache.org/jira/browse/SPARK-39341 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: William Hyun >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39341) KubernetesExecutorBackend should allow IPv6 pod IP
[ https://issues.apache.org/jira/browse/SPARK-39341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544036#comment-17544036 ] Apache Spark commented on SPARK-39341: -- User 'williamhyun' has created a pull request for this issue: https://github.com/apache/spark/pull/36728 > KubernetesExecutorBackend should allow IPv6 pod IP > -- > > Key: SPARK-39341 > URL: https://issues.apache.org/jira/browse/SPARK-39341 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: William Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39341) KubernetesExecutorBackend should allow IPv6 pod IP
[ https://issues.apache.org/jira/browse/SPARK-39341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39341: Assignee: (was: Apache Spark) > KubernetesExecutorBackend should allow IPv6 pod IP > -- > > Key: SPARK-39341 > URL: https://issues.apache.org/jira/browse/SPARK-39341 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: William Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39341) KubernetesExecutorBackend should allow IPv6 pod IP
[ https://issues.apache.org/jira/browse/SPARK-39341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39341: Assignee: Apache Spark > KubernetesExecutorBackend should allow IPv6 pod IP > -- > > Key: SPARK-39341 > URL: https://issues.apache.org/jira/browse/SPARK-39341 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: William Hyun >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-39341) KubernetesExecutorBackend should allow IPv6 pod IP
William Hyun created SPARK-39341: Summary: KubernetesExecutorBackend should allow IPv6 pod IP Key: SPARK-39341 URL: https://issues.apache.org/jira/browse/SPARK-39341 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 3.3.0 Reporter: William Hyun -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39340) DS v2 agg pushdown should allow dots in the name of top-level columns
[ https://issues.apache.org/jira/browse/SPARK-39340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544002#comment-17544002 ] Apache Spark commented on SPARK-39340: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/36727 > DS v2 agg pushdown should allow dots in the name of top-level columns > - > > Key: SPARK-39340 > URL: https://issues.apache.org/jira/browse/SPARK-39340 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.2.0 >Reporter: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39340) DS v2 agg pushdown should allow dots in the name of top-level columns
[ https://issues.apache.org/jira/browse/SPARK-39340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39340: Assignee: (was: Apache Spark) > DS v2 agg pushdown should allow dots in the name of top-level columns > - > > Key: SPARK-39340 > URL: https://issues.apache.org/jira/browse/SPARK-39340 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.2.0 >Reporter: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39340) DS v2 agg pushdown should allow dots in the name of top-level columns
[ https://issues.apache.org/jira/browse/SPARK-39340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544001#comment-17544001 ] Apache Spark commented on SPARK-39340: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/36727 > DS v2 agg pushdown should allow dots in the name of top-level columns > - > > Key: SPARK-39340 > URL: https://issues.apache.org/jira/browse/SPARK-39340 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.2.0 >Reporter: Wenchen Fan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39340) DS v2 agg pushdown should allow dots in the name of top-level columns
[ https://issues.apache.org/jira/browse/SPARK-39340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39340: Assignee: Apache Spark > DS v2 agg pushdown should allow dots in the name of top-level columns > - > > Key: SPARK-39340 > URL: https://issues.apache.org/jira/browse/SPARK-39340 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.2.0 >Reporter: Wenchen Fan >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-39340) DS v2 agg pushdown should allow dots in the name of top-level columns
Wenchen Fan created SPARK-39340: --- Summary: DS v2 agg pushdown should allow dots in the name of top-level columns Key: SPARK-39340 URL: https://issues.apache.org/jira/browse/SPARK-39340 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.2.0 Reporter: Wenchen Fan -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39338) Remove dynamic pruning subquery if pruningKey's references is empty
[ https://issues.apache.org/jira/browse/SPARK-39338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-39338: Issue Type: Bug (was: Improvement) > Remove dynamic pruning subquery if pruningKey's references is empty > --- > > Key: SPARK-39338 > URL: https://issues.apache.org/jira/browse/SPARK-39338 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.4.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39338) Remove dynamic pruning subquery if pruningKey's references is empty
[ https://issues.apache.org/jira/browse/SPARK-39338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-39338: Affects Version/s: 3.3.0 > Remove dynamic pruning subquery if pruningKey's references is empty > --- > > Key: SPARK-39338 > URL: https://issues.apache.org/jira/browse/SPARK-39338 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.3.0, 3.4.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39332) Upgrade RoaringBitmap to 0.9.28
[ https://issues.apache.org/jira/browse/SPARK-39332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39332: - Assignee: Yang Jie > Upgrade RoaringBitmap to 0.9.28 > --- > > Key: SPARK-39332 > URL: https://issues.apache.org/jira/browse/SPARK-39332 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-39332) Upgrade RoaringBitmap to 0.9.28
[ https://issues.apache.org/jira/browse/SPARK-39332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39332. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36719 [https://github.com/apache/spark/pull/36719] > Upgrade RoaringBitmap to 0.9.28 > --- > > Key: SPARK-39332 > URL: https://issues.apache.org/jira/browse/SPARK-39332 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39339) Support TimestampNTZ in JDBC data source
[ https://issues.apache.org/jira/browse/SPARK-39339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39339: Assignee: Apache Spark > Support TimestampNTZ in JDBC data source > > > Key: SPARK-39339 > URL: https://issues.apache.org/jira/browse/SPARK-39339 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Ivan Sadikov >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39339) Support TimestampNTZ in JDBC data source
[ https://issues.apache.org/jira/browse/SPARK-39339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543830#comment-17543830 ] Apache Spark commented on SPARK-39339: -- User 'sadikovi' has created a pull request for this issue: https://github.com/apache/spark/pull/36726 > Support TimestampNTZ in JDBC data source > > > Key: SPARK-39339 > URL: https://issues.apache.org/jira/browse/SPARK-39339 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Ivan Sadikov >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39339) Support TimestampNTZ in JDBC data source
[ https://issues.apache.org/jira/browse/SPARK-39339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39339: Assignee: (was: Apache Spark) > Support TimestampNTZ in JDBC data source > > > Key: SPARK-39339 > URL: https://issues.apache.org/jira/browse/SPARK-39339 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Ivan Sadikov >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39334) Change to exclude `slf4j-reload4j` for `hadoop-minikdc`
[ https://issues.apache.org/jira/browse/SPARK-39334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39334: -- Issue Type: Test (was: Improvement) > Change to exclude `slf4j-reload4j` for `hadoop-minikdc` > --- > > Key: SPARK-39334 > URL: https://issues.apache.org/jira/browse/SPARK-39334 > Project: Spark > Issue Type: Test > Components: Build, Tests >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Fix For: 3.3.0 > > > HADOOP-18088 Replace log4j 1.x with reload4j in Hadoop 3.3.3 and there will > be waring log when in Spark when test > ``` > Testing started at 10:11 ... > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.2/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39334) Change to exclude `slf4j-reload4j` for `hadoop-minikdc`
[ https://issues.apache.org/jira/browse/SPARK-39334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39334: -- Component/s: Tests > Change to exclude `slf4j-reload4j` for `hadoop-minikdc` > --- > > Key: SPARK-39334 > URL: https://issues.apache.org/jira/browse/SPARK-39334 > Project: Spark > Issue Type: Improvement > Components: Build, Tests >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Fix For: 3.3.0 > > > HADOOP-18088 Replace log4j 1.x with reload4j in Hadoop 3.3.3 and there will > be waring log when in Spark when test > ``` > Testing started at 10:11 ... > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.2/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39334) Change to exclude `slf4j-reload4j` for `hadoop-minikdc`
[ https://issues.apache.org/jira/browse/SPARK-39334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39334: -- Priority: Minor (was: Major) > Change to exclude `slf4j-reload4j` for `hadoop-minikdc` > --- > > Key: SPARK-39334 > URL: https://issues.apache.org/jira/browse/SPARK-39334 > Project: Spark > Issue Type: Improvement > Components: Build, Tests >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Fix For: 3.3.0 > > > HADOOP-18088 Replace log4j 1.x with reload4j in Hadoop 3.3.3 and there will > be waring log when in Spark when test > ``` > Testing started at 10:11 ... > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.2/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39334) Change to exclude `slf4j-reload4j` for `hadoop-minikdc`
[ https://issues.apache.org/jira/browse/SPARK-39334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39334: -- Summary: Change to exclude `slf4j-reload4j` for `hadoop-minikdc` (was: Change to exclude `slf4j-reload4j` for `hadoop-minikdc` after upgrade hadoop to 3.3.3) > Change to exclude `slf4j-reload4j` for `hadoop-minikdc` > --- > > Key: SPARK-39334 > URL: https://issues.apache.org/jira/browse/SPARK-39334 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Fix For: 3.3.0 > > > HADOOP-18088 Replace log4j 1.x with reload4j in Hadoop 3.3.3 and there will > be waring log when in Spark when test > ``` > Testing started at 10:11 ... > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.2/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-39334) Change to exclude `slf4j-reload4j` for `hadoop-minikdc` after upgrade hadoop to 3.3.3
[ https://issues.apache.org/jira/browse/SPARK-39334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39334. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 36721 [https://github.com/apache/spark/pull/36721] > Change to exclude `slf4j-reload4j` for `hadoop-minikdc` after upgrade hadoop > to 3.3.3 > - > > Key: SPARK-39334 > URL: https://issues.apache.org/jira/browse/SPARK-39334 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Fix For: 3.3.0 > > > HADOOP-18088 Replace log4j 1.x with reload4j in Hadoop 3.3.3 and there will > be waring log when in Spark when test > ``` > Testing started at 10:11 ... > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.2/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39334) Change to exclude `slf4j-reload4j` for `hadoop-minikdc` after upgrade hadoop to 3.3.3
[ https://issues.apache.org/jira/browse/SPARK-39334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39334: - Assignee: Yang Jie > Change to exclude `slf4j-reload4j` for `hadoop-minikdc` after upgrade hadoop > to 3.3.3 > - > > Key: SPARK-39334 > URL: https://issues.apache.org/jira/browse/SPARK-39334 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > > HADOOP-18088 Replace log4j 1.x with reload4j in Hadoop 3.3.3 and there will > be waring log when in Spark when test > ``` > Testing started at 10:11 ... > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.2/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/Users/yangjie01/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-39339) Support TimestampNTZ in JDBC data source
Ivan Sadikov created SPARK-39339: Summary: Support TimestampNTZ in JDBC data source Key: SPARK-39339 URL: https://issues.apache.org/jira/browse/SPARK-39339 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.3.0 Reporter: Ivan Sadikov -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39322) Remove `Experimental` from `spark.dynamicAllocation.shuffleTracking.enabled`
[ https://issues.apache.org/jira/browse/SPARK-39322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543815#comment-17543815 ] Apache Spark commented on SPARK-39322: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/36725 > Remove `Experimental` from `spark.dynamicAllocation.shuffleTracking.enabled` > > > Key: SPARK-39322 > URL: https://issues.apache.org/jira/browse/SPARK-39322 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.3.1 > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39337) Refactor DescribeTableExec
[
https://issues.apache.org/jira/browse/SPARK-39337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543813#comment-17543813
]
Apache Spark commented on SPARK-39337:
--
User 'AngersZh' has created a pull request for this issue:
https://github.com/apache/spark/pull/36723
> Refactor DescribeTableExec
> --
>
> Key: SPARK-39337
> URL: https://issues.apache.org/jira/browse/SPARK-39337
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: angerszhu
>Priority: Major
>
> Repeated code, refactor the code.
> {code:java}
> private def addTableDetails(rows: ArrayBuffer[InternalRow]): Unit = {
> rows += emptyRow()
> rows += toCatalystRow("# Detailed Table Information", "", "")
> rows += toCatalystRow("Name", table.name(), "")
> CatalogV2Util.TABLE_RESERVED_PROPERTIES.foreach(propKey => {
> if (table.properties.containsKey(propKey)) {
> rows += toCatalystRow(propKey.capitalize,
> table.properties.get(propKey), "")
> }
> })
> val properties =
> table.properties.asScala.toList
> .filter(kv =>
> !CatalogV2Util.TABLE_RESERVED_PROPERTIES.contains(kv._1))
> .sortBy(_._1).map {
> case (key, value) => key + "=" + value
> }.mkString("[", ",", "]")
> rows += toCatalystRow("Table Properties", properties, "")
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39322) Remove `Experimental` from `spark.dynamicAllocation.shuffleTracking.enabled`
[ https://issues.apache.org/jira/browse/SPARK-39322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543814#comment-17543814 ] Apache Spark commented on SPARK-39322: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/36725 > Remove `Experimental` from `spark.dynamicAllocation.shuffleTracking.enabled` > > > Key: SPARK-39322 > URL: https://issues.apache.org/jira/browse/SPARK-39322 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.3.1 > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39337) Refactor DescribeTableExec
[
https://issues.apache.org/jira/browse/SPARK-39337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-39337:
Assignee: (was: Apache Spark)
> Refactor DescribeTableExec
> --
>
> Key: SPARK-39337
> URL: https://issues.apache.org/jira/browse/SPARK-39337
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: angerszhu
>Priority: Major
>
> Repeated code, refactor the code.
> {code:java}
> private def addTableDetails(rows: ArrayBuffer[InternalRow]): Unit = {
> rows += emptyRow()
> rows += toCatalystRow("# Detailed Table Information", "", "")
> rows += toCatalystRow("Name", table.name(), "")
> CatalogV2Util.TABLE_RESERVED_PROPERTIES.foreach(propKey => {
> if (table.properties.containsKey(propKey)) {
> rows += toCatalystRow(propKey.capitalize,
> table.properties.get(propKey), "")
> }
> })
> val properties =
> table.properties.asScala.toList
> .filter(kv =>
> !CatalogV2Util.TABLE_RESERVED_PROPERTIES.contains(kv._1))
> .sortBy(_._1).map {
> case (key, value) => key + "=" + value
> }.mkString("[", ",", "]")
> rows += toCatalystRow("Table Properties", properties, "")
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39337) Refactor DescribeTableExec
[
https://issues.apache.org/jira/browse/SPARK-39337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-39337:
Assignee: Apache Spark
> Refactor DescribeTableExec
> --
>
> Key: SPARK-39337
> URL: https://issues.apache.org/jira/browse/SPARK-39337
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: angerszhu
>Assignee: Apache Spark
>Priority: Major
>
> Repeated code, refactor the code.
> {code:java}
> private def addTableDetails(rows: ArrayBuffer[InternalRow]): Unit = {
> rows += emptyRow()
> rows += toCatalystRow("# Detailed Table Information", "", "")
> rows += toCatalystRow("Name", table.name(), "")
> CatalogV2Util.TABLE_RESERVED_PROPERTIES.foreach(propKey => {
> if (table.properties.containsKey(propKey)) {
> rows += toCatalystRow(propKey.capitalize,
> table.properties.get(propKey), "")
> }
> })
> val properties =
> table.properties.asScala.toList
> .filter(kv =>
> !CatalogV2Util.TABLE_RESERVED_PROPERTIES.contains(kv._1))
> .sortBy(_._1).map {
> case (key, value) => key + "=" + value
> }.mkString("[", ",", "]")
> rows += toCatalystRow("Table Properties", properties, "")
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39337) Refactor DescribeTableExec
[
https://issues.apache.org/jira/browse/SPARK-39337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543812#comment-17543812
]
Apache Spark commented on SPARK-39337:
--
User 'AngersZh' has created a pull request for this issue:
https://github.com/apache/spark/pull/36723
> Refactor DescribeTableExec
> --
>
> Key: SPARK-39337
> URL: https://issues.apache.org/jira/browse/SPARK-39337
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: angerszhu
>Priority: Major
>
> Repeated code, refactor the code.
> {code:java}
> private def addTableDetails(rows: ArrayBuffer[InternalRow]): Unit = {
> rows += emptyRow()
> rows += toCatalystRow("# Detailed Table Information", "", "")
> rows += toCatalystRow("Name", table.name(), "")
> CatalogV2Util.TABLE_RESERVED_PROPERTIES.foreach(propKey => {
> if (table.properties.containsKey(propKey)) {
> rows += toCatalystRow(propKey.capitalize,
> table.properties.get(propKey), "")
> }
> })
> val properties =
> table.properties.asScala.toList
> .filter(kv =>
> !CatalogV2Util.TABLE_RESERVED_PROPERTIES.contains(kv._1))
> .sortBy(_._1).map {
> case (key, value) => key + "=" + value
> }.mkString("[", ",", "]")
> rows += toCatalystRow("Table Properties", properties, "")
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39338) Remove dynamic pruning subquery if pruningKey's references is empty
[ https://issues.apache.org/jira/browse/SPARK-39338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39338: Assignee: Apache Spark > Remove dynamic pruning subquery if pruningKey's references is empty > --- > > Key: SPARK-39338 > URL: https://issues.apache.org/jira/browse/SPARK-39338 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Yuming Wang >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39338) Remove dynamic pruning subquery if pruningKey's references is empty
[ https://issues.apache.org/jira/browse/SPARK-39338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39338: Assignee: (was: Apache Spark) > Remove dynamic pruning subquery if pruningKey's references is empty > --- > > Key: SPARK-39338 > URL: https://issues.apache.org/jira/browse/SPARK-39338 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39338) Remove dynamic pruning subquery if pruningKey's references is empty
[ https://issues.apache.org/jira/browse/SPARK-39338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543802#comment-17543802 ] Apache Spark commented on SPARK-39338: -- User 'wangyum' has created a pull request for this issue: https://github.com/apache/spark/pull/36724 > Remove dynamic pruning subquery if pruningKey's references is empty > --- > > Key: SPARK-39338 > URL: https://issues.apache.org/jira/browse/SPARK-39338 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-39338) Remove dynamic pruning subquery if pruningKey's references is empty
Yuming Wang created SPARK-39338: --- Summary: Remove dynamic pruning subquery if pruningKey's references is empty Key: SPARK-39338 URL: https://issues.apache.org/jira/browse/SPARK-39338 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.4.0 Reporter: Yuming Wang -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37523) Support optimize skewed partitions in Distribution and Ordering if numPartitions is not specified
[ https://issues.apache.org/jira/browse/SPARK-37523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37523. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36644 [https://github.com/apache/spark/pull/36644] > Support optimize skewed partitions in Distribution and Ordering if > numPartitions is not specified > - > > Key: SPARK-37523 > URL: https://issues.apache.org/jira/browse/SPARK-37523 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.2.1, 3.3.0 >Reporter: Huaxin Gao >Assignee: Huaxin Gao >Priority: Major > Fix For: 3.4.0 > > > When doing repartition in distribution and sort, if data source requests for > a specific number of partitions, we should not optimize repartition. However, > if data source does not request for a specific number of partitions, Spark > should optimize repartition and split the skewed partitions if necessary. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37523) Support optimize skewed partitions in Distribution and Ordering if numPartitions is not specified
[ https://issues.apache.org/jira/browse/SPARK-37523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37523: --- Assignee: Huaxin Gao > Support optimize skewed partitions in Distribution and Ordering if > numPartitions is not specified > - > > Key: SPARK-37523 > URL: https://issues.apache.org/jira/browse/SPARK-37523 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.2.1, 3.3.0 >Reporter: Huaxin Gao >Assignee: Huaxin Gao >Priority: Major > > When doing repartition in distribution and sort, if data source requests for > a specific number of partitions, we should not optimize repartition. However, > if data source does not request for a specific number of partitions, Spark > should optimize repartition and split the skewed partitions if necessary. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
