[jira] [Assigned] (SPARK-48602) Make csv generator support different output style with spark.sql.binaryOutputStyle
[ https://issues.apache.org/jira/browse/SPARK-48602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-48602: Assignee: Kent Yao > Make csv generator support different output style with > spark.sql.binaryOutputStyle > -- > > Key: SPARK-48602 > URL: https://issues.apache.org/jira/browse/SPARK-48602 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48602) Make csv generator support different output style with spark.sql.binaryOutputStyle
[ https://issues.apache.org/jira/browse/SPARK-48602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48602. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46956 [https://github.com/apache/spark/pull/46956] > Make csv generator support different output style with > spark.sql.binaryOutputStyle > -- > > Key: SPARK-48602 > URL: https://issues.apache.org/jira/browse/SPARK-48602 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48583) Replace deprecated classes and methods of commons-io called in Spark
[ https://issues.apache.org/jira/browse/SPARK-48583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-48583: Assignee: Wei Guo > Replace deprecated classes and methods of commons-io called in Spark > > > Key: SPARK-48583 > URL: https://issues.apache.org/jira/browse/SPARK-48583 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wei Guo >Assignee: Wei Guo >Priority: Major > Labels: pull-request-available > > There are some deprecated classes and methods in commons-io called in Spark, > we need to replace them: > * writeStringToFile(final File file, final String data) > * CountingInputStream -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48583) Replace deprecated classes and methods of commons-io called in Spark
[ https://issues.apache.org/jira/browse/SPARK-48583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-48583. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46935 [https://github.com/apache/spark/pull/46935] > Replace deprecated classes and methods of commons-io called in Spark > > > Key: SPARK-48583 > URL: https://issues.apache.org/jira/browse/SPARK-48583 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wei Guo >Assignee: Wei Guo >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > There are some deprecated classes and methods in commons-io called in Spark, > we need to replace them: > * writeStringToFile(final File file, final String data) > * CountingInputStream -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48610) Remove ExplainUtils.processPlan synchronize
[ https://issues.apache.org/jira/browse/SPARK-48610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48610: --- Labels: pull-request-available (was: ) > Remove ExplainUtils.processPlan synchronize > --- > > Key: SPARK-48610 > URL: https://issues.apache.org/jira/browse/SPARK-48610 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Ziqi Liu >Priority: Major > Labels: pull-request-available > > [https://github.com/apache/spark/pull/45282] introduced synchronize to > `ExplainUtils.processPlan` to avoid race condition when multiple queries > refers to same cached plan. > The granularity of lock is too large. We can try to fix the root cause of > this concurrency issue by refactoring the usage of mutable OP_ID_TAG, which > is not a good practice in terms of immutable nature of SparkPlan. Instead, > we can use an auxiliary id map, with object identity as the key. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48610) Remove ExplainUtils.processPlan synchronize
Ziqi Liu created SPARK-48610: Summary: Remove ExplainUtils.processPlan synchronize Key: SPARK-48610 URL: https://issues.apache.org/jira/browse/SPARK-48610 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Ziqi Liu [https://github.com/apache/spark/pull/45282] introduced synchronize to `ExplainUtils.processPlan` to avoid race condition when multiple queries refers to same cached plan. The granularity of lock is too large. We can try to fix the root cause of this concurrency issue by refactoring the usage of mutable OP_ID_TAG, which is not a good practice in terms of immutable nature of SparkPlan. Instead, we can use an auxiliary id map, with object identity as the key. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48593) Fix the string representation of lambda function
[ https://issues.apache.org/jira/browse/SPARK-48593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48593. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46948 [https://github.com/apache/spark/pull/46948] > Fix the string representation of lambda function > > > Key: SPARK-48593 > URL: https://issues.apache.org/jira/browse/SPARK-48593 > Project: Spark > Issue Type: Bug > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48593) Fix the string representation of lambda function
[ https://issues.apache.org/jira/browse/SPARK-48593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48593: Assignee: Ruifeng Zheng > Fix the string representation of lambda function > > > Key: SPARK-48593 > URL: https://issues.apache.org/jira/browse/SPARK-48593 > Project: Spark > Issue Type: Bug > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48421) SPJ: Add documentation
[ https://issues.apache.org/jira/browse/SPARK-48421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48421: Assignee: Szehon Ho > SPJ: Add documentation > -- > > Key: SPARK-48421 > URL: https://issues.apache.org/jira/browse/SPARK-48421 > Project: Spark > Issue Type: Documentation > Components: SQL >Affects Versions: 4.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > > As part of SPARK-48329, we mentioned "Storage Partition Join" but noticed > there is no documentation describing the same. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48421) SPJ: Add documentation
[ https://issues.apache.org/jira/browse/SPARK-48421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48421. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46745 [https://github.com/apache/spark/pull/46745] > SPJ: Add documentation > -- > > Key: SPARK-48421 > URL: https://issues.apache.org/jira/browse/SPARK-48421 > Project: Spark > Issue Type: Documentation > Components: SQL >Affects Versions: 4.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > As part of SPARK-48329, we mentioned "Storage Partition Join" but noticed > there is no documentation describing the same. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48591) Simplify the if-else branches with `F.lit`
[ https://issues.apache.org/jira/browse/SPARK-48591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48591. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46946 [https://github.com/apache/spark/pull/46946] > Simplify the if-else branches with `F.lit` > -- > > Key: SPARK-48591 > URL: https://issues.apache.org/jira/browse/SPARK-48591 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48591) Simplify the if-else branches with `F.lit`
[ https://issues.apache.org/jira/browse/SPARK-48591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48591: Assignee: Ruifeng Zheng > Simplify the if-else branches with `F.lit` > -- > > Key: SPARK-48591 > URL: https://issues.apache.org/jira/browse/SPARK-48591 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48598) Propagate cached schema in dataframe operations
[ https://issues.apache.org/jira/browse/SPARK-48598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48598: Assignee: Ruifeng Zheng > Propagate cached schema in dataframe operations > --- > > Key: SPARK-48598 > URL: https://issues.apache.org/jira/browse/SPARK-48598 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48598) Propagate cached schema in dataframe operations
[ https://issues.apache.org/jira/browse/SPARK-48598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48598. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46954 [https://github.com/apache/spark/pull/46954] > Propagate cached schema in dataframe operations > --- > > Key: SPARK-48598 > URL: https://issues.apache.org/jira/browse/SPARK-48598 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48609) Upgrade `scala-xml` to 2.3
[ https://issues.apache.org/jira/browse/SPARK-48609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48609: --- Labels: pull-request-available (was: ) > Upgrade `scala-xml` to 2.3 > -- > > Key: SPARK-48609 > URL: https://issues.apache.org/jira/browse/SPARK-48609 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48609) Upgrade `scala-xml` to 2.3
BingKun Pan created SPARK-48609: --- Summary: Upgrade `scala-xml` to 2.3 Key: SPARK-48609 URL: https://issues.apache.org/jira/browse/SPARK-48609 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: BingKun Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48589) Add option snapshotStartBatchId and snapshotPartitionId to state data source
[ https://issues.apache.org/jira/browse/SPARK-48589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48589: --- Labels: pull-request-available (was: ) > Add option snapshotStartBatchId and snapshotPartitionId to state data source > > > Key: SPARK-48589 > URL: https://issues.apache.org/jira/browse/SPARK-48589 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Yuchen Liu >Priority: Major > Labels: pull-request-available > > Define two new options, _snapshotStartBatchId_ and _snapshotPartitionId_, for > the existing state reader. Both of them should be provided at the same time. > # When there is no snapshot file at that batch (note there is an off-by-one > issue between version and batch Id), throw an exception. > # Otherwise, the reader should continue to rebuild the state by reading > delta files only, and ignore all snapshot files afterwards. > # Note that if a batchId option is already specified. That batchId is the > ending batchId, we should then end at that batchId. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48607) Support more types of expressions in VALUES lists
[ https://issues.apache.org/jira/browse/SPARK-48607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel resolved SPARK-48607. Resolution: Duplicate > Support more types of expressions in VALUES lists > - > > Key: SPARK-48607 > URL: https://issues.apache.org/jira/browse/SPARK-48607 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 4.0.0 >Reporter: Daniel >Priority: Major > > Currently we return the > `INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE` error class > [1] if SQL VALUES lists contain non-literal expressions such as > CURRENT_DATE(). In this Jira we propose to relax that restriction to allow > more types of expressions. > [1] > https://github.com/apache/spark/blob/b5e1b7988031044d3cbdb277668b775c08db1a74/common/utils/src/main/resources/error/error-conditions.json#L2247 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48608) Spark 3.5: fails to build with value defaultValueNotConstantError is not a member of object org.apache.spark.sql.errors.QueryCompilationErrors
[ https://issues.apache.org/jira/browse/SPARK-48608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-48608: -- Priority: Blocker (was: Major) > Spark 3.5: fails to build with value defaultValueNotConstantError is not a > member of object org.apache.spark.sql.errors.QueryCompilationErrors > --- > > Key: SPARK-48608 > URL: https://issues.apache.org/jira/browse/SPARK-48608 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.5.2 >Reporter: Thomas Graves >Priority: Blocker > > PR [https://github.com/apache/spark/pull/46594] seems to have broken the > Spark 3.5 build. > [ERROR] [Error] > ...sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala:299: > value defaultValueNotConstantError is not a member of object > org.apache.spark.sql.errors.QueryCompilationErrors > I don't see that definition defined on the 3.5 branch - > [https://github.com/apache/spark/blob/branch-3.5/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala] > I see it defined on master by > https://issues.apache.org/jira/browse/SPARK-46905 which only went into 4.0 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48608) Spark 3.5: fails to build with value defaultValueNotConstantError is not a member of object org.apache.spark.sql.errors.QueryCompilationErrors
Thomas Graves created SPARK-48608: - Summary: Spark 3.5: fails to build with value defaultValueNotConstantError is not a member of object org.apache.spark.sql.errors.QueryCompilationErrors Key: SPARK-48608 URL: https://issues.apache.org/jira/browse/SPARK-48608 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.5.2 Reporter: Thomas Graves PR [https://github.com/apache/spark/pull/46594] seems to have broken the Spark 3.5 build. [ERROR] [Error] ...sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala:299: value defaultValueNotConstantError is not a member of object org.apache.spark.sql.errors.QueryCompilationErrors I don't see that definition defined on the 3.5 branch - [https://github.com/apache/spark/blob/branch-3.5/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala] I see it defined on master by https://issues.apache.org/jira/browse/SPARK-46905 which only went into 4.0 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48607) Support more types of expressions in VALUES lists
Daniel created SPARK-48607: -- Summary: Support more types of expressions in VALUES lists Key: SPARK-48607 URL: https://issues.apache.org/jira/browse/SPARK-48607 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 4.0.0 Reporter: Daniel Currently we return the `INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE` error class [1] if SQL VALUES lists contain non-literal expressions such as CURRENT_DATE(). In this Jira we propose to relax that restriction to allow more types of expressions. [1] https://github.com/apache/spark/blob/b5e1b7988031044d3cbdb277668b775c08db1a74/common/utils/src/main/resources/error/error-conditions.json#L2247 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48606) Upgrade `google-java-format` to 1.22.0
[
https://issues.apache.org/jira/browse/SPARK-48606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-48606.
---
Fix Version/s: kubernetes-operator-0.1.0
Resolution: Fixed
Issue resolved by pull request 15
[https://github.com/apache/spark-kubernetes-operator/pull/15]
> Upgrade `google-java-format` to 1.22.0
> --
>
> Key: SPARK-48606
> URL: https://issues.apache.org/jira/browse/SPARK-48606
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: kubernetes-operator-0.1.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Labels: pull-request-available
> Fix For: kubernetes-operator-0.1.0
>
>
> This issue aims to upgrade `google-java-format` plugin of Spark Kubernetes
> Operator repository to bring the latest bug fixes like the following. The
> latest version is recommended.
> {code}
> java.lang.Exception: google-java-format 1.17.0 is currently being used, but
> outdated.
> google-java-format 1.19.2 is the recommended version, which may have fixed
> this problem.
> google-java-format 1.19.2 requires JVM 11+.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48606) Upgrade `google-java-format` to 1.22.0
[
https://issues.apache.org/jira/browse/SPARK-48606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-48606:
-
Assignee: Dongjoon Hyun
> Upgrade `google-java-format` to 1.22.0
> --
>
> Key: SPARK-48606
> URL: https://issues.apache.org/jira/browse/SPARK-48606
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: kubernetes-operator-0.1.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Labels: pull-request-available
>
> This issue aims to upgrade `google-java-format` plugin of Spark Kubernetes
> Operator repository to bring the latest bug fixes like the following. The
> latest version is recommended.
> {code}
> java.lang.Exception: google-java-format 1.17.0 is currently being used, but
> outdated.
> google-java-format 1.19.2 is the recommended version, which may have fixed
> this problem.
> google-java-format 1.19.2 requires JVM 11+.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48606) Upgrade `google-java-format` to 1.22.0
[
https://issues.apache.org/jira/browse/SPARK-48606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-48606:
---
Labels: pull-request-available (was: )
> Upgrade `google-java-format` to 1.22.0
> --
>
> Key: SPARK-48606
> URL: https://issues.apache.org/jira/browse/SPARK-48606
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: kubernetes-operator-0.1.0
>Reporter: Dongjoon Hyun
>Priority: Major
> Labels: pull-request-available
>
> This issue aims to upgrade `google-java-format` plugin of Spark Kubernetes
> Operator repository to bring the latest bug fixes like the following. The
> latest version is recommended.
> {code}
> java.lang.Exception: google-java-format 1.17.0 is currently being used, but
> outdated.
> google-java-format 1.19.2 is the recommended version, which may have fixed
> this problem.
> google-java-format 1.19.2 requires JVM 11+.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48606) Upgrade `google-java-format` to 1.22.0
Dongjoon Hyun created SPARK-48606:
-
Summary: Upgrade `google-java-format` to 1.22.0
Key: SPARK-48606
URL: https://issues.apache.org/jira/browse/SPARK-48606
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: kubernetes-operator-0.1.0
Reporter: Dongjoon Hyun
This issue aims to upgrade `google-java-format` plugin of Spark Kubernetes
Operator repository to bring the latest bug fixes like the following. The
latest version is recommended.
{code}
java.lang.Exception: google-java-format 1.17.0 is currently being used, but
outdated.
google-java-format 1.19.2 is the recommended version, which may have fixed this
problem.
google-java-format 1.19.2 requires JVM 11+.
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48605) UI display driver thread first on the thread dump page
[ https://issues.apache.org/jira/browse/SPARK-48605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48605: --- Labels: pull-request-available (was: ) > UI display driver thread first on the thread dump page > -- > > Key: SPARK-48605 > URL: https://issues.apache.org/jira/browse/SPARK-48605 > Project: Spark > Issue Type: Improvement > Components: UI >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48605) UI display driver thread first on the thread dump page
Cheng Pan created SPARK-48605: - Summary: UI display driver thread first on the thread dump page Key: SPARK-48605 URL: https://issues.apache.org/jira/browse/SPARK-48605 Project: Spark Issue Type: Improvement Components: UI Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48604) Replace deprecated classes and methods of arrow-vector called in Spark
Wei Guo created SPARK-48604: --- Summary: Replace deprecated classes and methods of arrow-vector called in Spark Key: SPARK-48604 URL: https://issues.apache.org/jira/browse/SPARK-48604 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Wei Guo There are some deprecated classes and methods in commons-io called in Spark, we need to replace them: * writeStringToFile(final File file, final String data) * CountingInputStream -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48604) Replace deprecated classes and methods of arrow-vector called in Spark
[ https://issues.apache.org/jira/browse/SPARK-48604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-48604: Description: There are some deprecated classes and methods in arrow-vector called in Spark, we need to replace them: * ArrowType.Decimal(precision, scale) was: There are some deprecated classes and methods in commons-io called in Spark, we need to replace them: * writeStringToFile(final File file, final String data) * CountingInputStream > Replace deprecated classes and methods of arrow-vector called in Spark > -- > > Key: SPARK-48604 > URL: https://issues.apache.org/jira/browse/SPARK-48604 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wei Guo >Priority: Major > Labels: pull-request-available > > There are some deprecated classes and methods in arrow-vector called in > Spark, we need to replace them: > * ArrowType.Decimal(precision, scale) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48603) Update *ParquetReadSchemaSuite to cover type widen capability
[ https://issues.apache.org/jira/browse/SPARK-48603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48603: --- Labels: pull-request-available (was: ) > Update *ParquetReadSchemaSuite to cover type widen capability > - > > Key: SPARK-48603 > URL: https://issues.apache.org/jira/browse/SPARK-48603 > Project: Spark > Issue Type: Test > Components: Tests >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48603) Update *ParquetReadSchemaSuite to cover type widen capability
Cheng Pan created SPARK-48603: - Summary: Update *ParquetReadSchemaSuite to cover type widen capability Key: SPARK-48603 URL: https://issues.apache.org/jira/browse/SPARK-48603 Project: Spark Issue Type: Test Components: Tests Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48596) Perf improvement for calculating hex string for long
[ https://issues.apache.org/jira/browse/SPARK-48596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48596. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46952 [https://github.com/apache/spark/pull/46952] > Perf improvement for calculating hex string for long > > > Key: SPARK-48596 > URL: https://issues.apache.org/jira/browse/SPARK-48596 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48596) Perf improvement for calculating hex string for long
[ https://issues.apache.org/jira/browse/SPARK-48596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-48596: Assignee: Kent Yao > Perf improvement for calculating hex string for long > > > Key: SPARK-48596 > URL: https://issues.apache.org/jira/browse/SPARK-48596 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48602) Make csv generator support different output style with spark.sql.binaryOutputStyle
[ https://issues.apache.org/jira/browse/SPARK-48602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48602: --- Labels: pull-request-available (was: ) > Make csv generator support different output style with > spark.sql.binaryOutputStyle > -- > > Key: SPARK-48602 > URL: https://issues.apache.org/jira/browse/SPARK-48602 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Kent Yao >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48602) Make csv generator support different output style with spark.sql.binaryOutputStyle
Kent Yao created SPARK-48602: Summary: Make csv generator support different output style with spark.sql.binaryOutputStyle Key: SPARK-48602 URL: https://issues.apache.org/jira/browse/SPARK-48602 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Kent Yao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48600) Fix Lag expression implicit casting
[ https://issues.apache.org/jira/browse/SPARK-48600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihailo Milosevic updated SPARK-48600: -- Description: Expression walker found the error of Lag expression not behaving properly when default value is passed with different collation. > Fix Lag expression implicit casting > --- > > Key: SPARK-48600 > URL: https://issues.apache.org/jira/browse/SPARK-48600 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Mihailo Milosevic >Priority: Major > > Expression walker found the error of Lag expression not behaving properly > when default value is passed with different collation. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48600) Fix Lag expression implicit casting
Mihailo Milosevic created SPARK-48600: - Summary: Fix Lag expression implicit casting Key: SPARK-48600 URL: https://issues.apache.org/jira/browse/SPARK-48600 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 4.0.0 Reporter: Mihailo Milosevic -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48599) OIDC support for Spark
Yeachan Park created SPARK-48599: Summary: OIDC support for Spark Key: SPARK-48599 URL: https://issues.apache.org/jira/browse/SPARK-48599 Project: Spark Issue Type: New Feature Components: Spark Core Affects Versions: 3.4.3 Reporter: Yeachan Park Add OIDC support to authenticate to Spark. To be updated, WIP -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48342) [M0] Parser support
[ https://issues.apache.org/jira/browse/SPARK-48342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48342: -- Assignee: (was: Apache Spark) > [M0] Parser support > --- > > Key: SPARK-48342 > URL: https://issues.apache.org/jira/browse/SPARK-48342 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: David Milicevic >Priority: Major > Labels: pull-request-available > > Implement parse for SQL scripting with all supporting changes for upcoming > interpreter implementation and future extensions of the parser: > * Parser - support only compound statements > * Parser testing > > For more details, design doc can be found in parent Jira item. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48342) [M0] Parser support
[ https://issues.apache.org/jira/browse/SPARK-48342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48342: -- Assignee: Apache Spark > [M0] Parser support > --- > > Key: SPARK-48342 > URL: https://issues.apache.org/jira/browse/SPARK-48342 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: David Milicevic >Assignee: Apache Spark >Priority: Major > Labels: pull-request-available > > Implement parse for SQL scripting with all supporting changes for upcoming > interpreter implementation and future extensions of the parser: > * Parser - support only compound statements > * Parser testing > > For more details, design doc can be found in parent Jira item. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48595) Cleanup deprecated api usage related to commons-compress
[ https://issues.apache.org/jira/browse/SPARK-48595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48595. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46950 [https://github.com/apache/spark/pull/46950] > Cleanup deprecated api usage related to commons-compress > > > Key: SPARK-48595 > URL: https://issues.apache.org/jira/browse/SPARK-48595 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48595) Cleanup deprecated api usage related to commons-compress
[ https://issues.apache.org/jira/browse/SPARK-48595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-48595: Assignee: Yang Jie > Cleanup deprecated api usage related to commons-compress > > > Key: SPARK-48595 > URL: https://issues.apache.org/jira/browse/SPARK-48595 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48584) Perf improvement for unescapePathName
[ https://issues.apache.org/jira/browse/SPARK-48584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48584. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46938 [https://github.com/apache/spark/pull/46938] > Perf improvement for unescapePathName > - > > Key: SPARK-48584 > URL: https://issues.apache.org/jira/browse/SPARK-48584 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48583) Replace deprecated classes and methods of commons-io called in Spark
[ https://issues.apache.org/jira/browse/SPARK-48583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-48583: Summary: Replace deprecated classes and methods of commons-io called in Spark (was: Replace deprecated classes and methods of `commons-io` called in Spark) > Replace deprecated classes and methods of commons-io called in Spark > > > Key: SPARK-48583 > URL: https://issues.apache.org/jira/browse/SPARK-48583 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wei Guo >Priority: Major > Labels: pull-request-available > > There are some deprecated classes and methods in commons-io called in Spark, > we need to replace them: > * writeStringToFile(final File file, final String data) > * CountingInputStream -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48583) Replace deprecated classes and methods of `commons-io` called in Spark
[ https://issues.apache.org/jira/browse/SPARK-48583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-48583: Description: There are some deprecated classes and methods in commons-io called in Spark, we need to replace them: * writeStringToFile(final File file, final String data) * CountingInputStream was: There are some deprecated classes and methods in `commons-io` called in Spark, we need to replace them: * `writeStringToFile(final File file, final String data)； * `CountingInputStream` > Replace deprecated classes and methods of `commons-io` called in Spark > -- > > Key: SPARK-48583 > URL: https://issues.apache.org/jira/browse/SPARK-48583 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wei Guo >Priority: Major > Labels: pull-request-available > > There are some deprecated classes and methods in commons-io called in Spark, > we need to replace them: > * writeStringToFile(final File file, final String data) > * CountingInputStream -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48583) Replace deprecated classes and methods of `commons-io` called in Spark
[ https://issues.apache.org/jira/browse/SPARK-48583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-48583: Description: There are some deprecated classes and methods in `commons-io` called in Spark, we need to replace them: * `writeStringToFile(final File file, final String data)； * `CountingInputStream` was:Method `writeStringToFile(final File file, final String data)` in class `FileUtils` is deprecated, use `writeStringToFile(final File file, final String data, final Charset charset)` instead in UDFXPathUtilSuite. > Replace deprecated classes and methods of `commons-io` called in Spark > -- > > Key: SPARK-48583 > URL: https://issues.apache.org/jira/browse/SPARK-48583 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wei Guo >Priority: Major > Labels: pull-request-available > > There are some deprecated classes and methods in `commons-io` called in > Spark, we need to replace them: > * `writeStringToFile(final File file, final String data)； > * `CountingInputStream` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48583) Replace deprecated classes and methods of `commons-io` called in Spark
[ https://issues.apache.org/jira/browse/SPARK-48583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-48583: Description: There are some deprecated classes and methods in `commons-io` called in Spark, we need to replace them: * `writeStringToFile(final File file, final String data)； * `CountingInputStream` was: There are some deprecated classes and methods in `commons-io` called in Spark, we need to replace them: * `writeStringToFile(final File file, final String data)； * `CountingInputStream` > Replace deprecated classes and methods of `commons-io` called in Spark > -- > > Key: SPARK-48583 > URL: https://issues.apache.org/jira/browse/SPARK-48583 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wei Guo >Priority: Major > Labels: pull-request-available > > There are some deprecated classes and methods in `commons-io` called in > Spark, we need to replace them: > * `writeStringToFile(final File file, final String data)； > * `CountingInputStream` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48583) Replace deprecated classes and methods of `commons-io` called in Spark
[ https://issues.apache.org/jira/browse/SPARK-48583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-48583: Summary: Replace deprecated classes and methods of `commons-io` called in Spark (was: Replace deprecated `FileUtils#writeStringToFile` ) > Replace deprecated classes and methods of `commons-io` called in Spark > -- > > Key: SPARK-48583 > URL: https://issues.apache.org/jira/browse/SPARK-48583 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wei Guo >Priority: Major > Labels: pull-request-available > > Method `writeStringToFile(final File file, final String data)` in class > `FileUtils` is deprecated, use `writeStringToFile(final File file, final > String data, final Charset charset)` instead in UDFXPathUtilSuite. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48598) Propagate cached schema in dataframe operations
[ https://issues.apache.org/jira/browse/SPARK-48598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48598: --- Labels: pull-request-available (was: ) > Propagate cached schema in dataframe operations > --- > > Key: SPARK-48598 > URL: https://issues.apache.org/jira/browse/SPARK-48598 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48597) Distinguish the streaming nodes from the text representation of logical plan
[ https://issues.apache.org/jira/browse/SPARK-48597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48597: --- Labels: pull-request-available (was: ) > Distinguish the streaming nodes from the text representation of logical plan > > > Key: SPARK-48597 > URL: https://issues.apache.org/jira/browse/SPARK-48597 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Jungtaek Lim >Priority: Major > Labels: pull-request-available > > We had a hard time to figure out whether the nodes are streaming or not, when > we debugged the issue https://issues.apache.org/jira/browse/SPARK-47305 . > Plan text for logical plan does not show the property of isStreaming, hence > we had to speculate the value based on other context. In addition, even > though the type of leaf node is explicitly meant to be streaming which > enables us to track down the isStreaming for certain subtree, the plan could > be very long and it’s a non-trivial effort to trace down to the leaf nodes. > Also, if the leaf nodes are skipped on the representation due to the size, > there is no way to get the information of isStreaming. > We propose to introduce a marker of the representation for streaming, which > will be shown in the text logical plan. There is no concept of "isStreaming" > in physical plan, so the change only needs to happen in logical plan. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48598) Propagate cached schema in dataframe operations
Ruifeng Zheng created SPARK-48598: - Summary: Propagate cached schema in dataframe operations Key: SPARK-48598 URL: https://issues.apache.org/jira/browse/SPARK-48598 Project: Spark Issue Type: Sub-task Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48581) Upgrade dropwizard metrics to 4.2.26
[ https://issues.apache.org/jira/browse/SPARK-48581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48581. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46932 [https://github.com/apache/spark/pull/46932] > Upgrade dropwizard metrics to 4.2.26 > > > Key: SPARK-48581 > URL: https://issues.apache.org/jira/browse/SPARK-48581 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Wei Guo >Assignee: Wei Guo >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48597) Distinguish the streaming nodes from the text representation of logical plan
Jungtaek Lim created SPARK-48597: Summary: Distinguish the streaming nodes from the text representation of logical plan Key: SPARK-48597 URL: https://issues.apache.org/jira/browse/SPARK-48597 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Jungtaek Lim We had a hard time to figure out whether the nodes are streaming or not, when we debugged the issue https://issues.apache.org/jira/browse/SPARK-47305 . Plan text for logical plan does not show the property of isStreaming, hence we had to speculate the value based on other context. In addition, even though the type of leaf node is explicitly meant to be streaming which enables us to track down the isStreaming for certain subtree, the plan could be very long and it’s a non-trivial effort to trace down to the leaf nodes. Also, if the leaf nodes are skipped on the representation due to the size, there is no way to get the information of isStreaming. We propose to introduce a marker of the representation for streaming, which will be shown in the text logical plan. There is no concept of "isStreaming" in physical plan, so the change only needs to happen in logical plan. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48597) Distinguish the streaming nodes from the text representation of logical plan
[ https://issues.apache.org/jira/browse/SPARK-48597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17854291#comment-17854291 ] Jungtaek Lim commented on SPARK-48597: -- Will submit a PR sooner. > Distinguish the streaming nodes from the text representation of logical plan > > > Key: SPARK-48597 > URL: https://issues.apache.org/jira/browse/SPARK-48597 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Jungtaek Lim >Priority: Major > > We had a hard time to figure out whether the nodes are streaming or not, when > we debugged the issue https://issues.apache.org/jira/browse/SPARK-47305 . > Plan text for logical plan does not show the property of isStreaming, hence > we had to speculate the value based on other context. In addition, even > though the type of leaf node is explicitly meant to be streaming which > enables us to track down the isStreaming for certain subtree, the plan could > be very long and it’s a non-trivial effort to trace down to the leaf nodes. > Also, if the leaf nodes are skipped on the representation due to the size, > there is no way to get the information of isStreaming. > We propose to introduce a marker of the representation for streaming, which > will be shown in the text logical plan. There is no concept of "isStreaming" > in physical plan, so the change only needs to happen in logical plan. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48596) Perf improvement for calculating hex string for long
[ https://issues.apache.org/jira/browse/SPARK-48596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48596: --- Labels: pull-request-available (was: ) > Perf improvement for calculating hex string for long > > > Key: SPARK-48596 > URL: https://issues.apache.org/jira/browse/SPARK-48596 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Kent Yao >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48596) Perf improvement for calculating hex string for long
Kent Yao created SPARK-48596: Summary: Perf improvement for calculating hex string for long Key: SPARK-48596 URL: https://issues.apache.org/jira/browse/SPARK-48596 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Kent Yao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
