[jira] [Resolved] (SPARK-47085) Preformance issue on thrift API
[
https://issues.apache.org/jira/browse/SPARK-47085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kent Yao resolved SPARK-47085.
--
Fix Version/s: 4.0.0
Resolution: Fixed
resolved by https://github.com/apache/spark/pull/45155
> Preformance issue on thrift API
> ---
>
> Key: SPARK-47085
> URL: https://issues.apache.org/jira/browse/SPARK-47085
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1, 3.5.0
>Reporter: Izek Greenfield
>Assignee: Izek Greenfield
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> in class `RowSetUtils` there is a loop that has O(n^2) complexity:
> {code:scala}
> ...
> while (i < rowSize) {
> val row = rows(I)
> ...
> {code}
> It can be easily converted into O( n ) complexity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47085) Preformance issue on thrift API
[
https://issues.apache.org/jira/browse/SPARK-47085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kent Yao reassigned SPARK-47085:
Assignee: Izek Greenfield
> Preformance issue on thrift API
> ---
>
> Key: SPARK-47085
> URL: https://issues.apache.org/jira/browse/SPARK-47085
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1, 3.5.0
>Reporter: Izek Greenfield
>Assignee: Izek Greenfield
>Priority: Major
> Labels: pull-request-available
>
> in class `RowSetUtils` there is a loop that has O(n^2) complexity:
> {code:scala}
> ...
> while (i < rowSize) {
> val row = rows(I)
> ...
> {code}
> It can be easily converted into O( n ) complexity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46812) Make `mapInPandas` / mapInArrow` support ResourceProfile (Stage-Level scheduling)
[ https://issues.apache.org/jira/browse/SPARK-46812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu resolved SPARK-46812. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44852 [https://github.com/apache/spark/pull/44852] > Make `mapInPandas` / mapInArrow` support ResourceProfile (Stage-Level > scheduling) > - > > Key: SPARK-46812 > URL: https://issues.apache.org/jira/browse/SPARK-46812 > Project: Spark > Issue Type: New Feature > Components: PySpark, SQL >Affects Versions: 4.0.0, 3.5.2 >Reporter: Bobby Wang >Assignee: Bobby Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Supporting stage-level scheduling for DataFrame API mapInPandas and > mapInArrow is quite needed for Spark ML use cases. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46812) Make `mapInPandas` / mapInArrow` support ResourceProfile (Stage-Level scheduling)
[ https://issues.apache.org/jira/browse/SPARK-46812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu reassigned SPARK-46812: -- Assignee: Bobby Wang > Make `mapInPandas` / mapInArrow` support ResourceProfile (Stage-Level > scheduling) > - > > Key: SPARK-46812 > URL: https://issues.apache.org/jira/browse/SPARK-46812 > Project: Spark > Issue Type: New Feature > Components: PySpark, SQL >Affects Versions: 4.0.0, 3.5.2 >Reporter: Bobby Wang >Assignee: Bobby Wang >Priority: Major > Labels: pull-request-available > > Supporting stage-level scheduling for DataFrame API mapInPandas and > mapInArrow is quite needed for Spark ML use cases. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47072) Wrong error message for incorrect ANSI intervals
[
https://issues.apache.org/jira/browse/SPARK-47072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47072:
-
Fix Version/s: 3.5.2
> Wrong error message for incorrect ANSI intervals
>
>
> Key: SPARK-47072
> URL: https://issues.apache.org/jira/browse/SPARK-47072
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.2, 3.5.0, 4.0.0
>Reporter: Max Gekk
>Assignee: Max Gekk
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0, 3.5.2, 3.4.3
>
>
> When Spark SQL cannot recognise ANSI interval, it outputs wrong pattern for
> particular ANSI interval. For example, it cannot recognise year-month
> interval, but says about day-time interval:
> {code:sql}
> spark-sql (default)> select interval '-\t2-2\t' year to month;
> Interval string does not match year-month format of `[+|-]d h`, `INTERVAL
> [+|-]'[+|-]d h' DAY TO HOUR` when cast to interval year to month: - 2-2 .
> (line 1, pos 16)
> == SQL ==
> select interval '-\t2-2\t' year to month
> ^^^
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47072) Wrong error message for incorrect ANSI intervals
[
https://issues.apache.org/jira/browse/SPARK-47072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47072:
-
Fix Version/s: (was: 3.5.1)
> Wrong error message for incorrect ANSI intervals
>
>
> Key: SPARK-47072
> URL: https://issues.apache.org/jira/browse/SPARK-47072
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.2, 3.5.0, 4.0.0
>Reporter: Max Gekk
>Assignee: Max Gekk
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0, 3.4.3
>
>
> When Spark SQL cannot recognise ANSI interval, it outputs wrong pattern for
> particular ANSI interval. For example, it cannot recognise year-month
> interval, but says about day-time interval:
> {code:sql}
> spark-sql (default)> select interval '-\t2-2\t' year to month;
> Interval string does not match year-month format of `[+|-]d h`, `INTERVAL
> [+|-]'[+|-]d h' DAY TO HOUR` when cast to interval year to month: - 2-2 .
> (line 1, pos 16)
> == SQL ==
> select interval '-\t2-2\t' year to month
> ^^^
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47072) Wrong error message for incorrect ANSI intervals
[
https://issues.apache.org/jira/browse/SPARK-47072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47072:
-
Fix Version/s: 3.5.1
> Wrong error message for incorrect ANSI intervals
>
>
> Key: SPARK-47072
> URL: https://issues.apache.org/jira/browse/SPARK-47072
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.2, 3.5.0, 4.0.0
>Reporter: Max Gekk
>Assignee: Max Gekk
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0, 3.5.1, 3.4.3
>
>
> When Spark SQL cannot recognise ANSI interval, it outputs wrong pattern for
> particular ANSI interval. For example, it cannot recognise year-month
> interval, but says about day-time interval:
> {code:sql}
> spark-sql (default)> select interval '-\t2-2\t' year to month;
> Interval string does not match year-month format of `[+|-]d h`, `INTERVAL
> [+|-]'[+|-]d h' DAY TO HOUR` when cast to interval year to month: - 2-2 .
> (line 1, pos 16)
> == SQL ==
> select interval '-\t2-2\t' year to month
> ^^^
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47072) Wrong error message for incorrect ANSI intervals
[
https://issues.apache.org/jira/browse/SPARK-47072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47072:
-
Fix Version/s: 3.4.3
> Wrong error message for incorrect ANSI intervals
>
>
> Key: SPARK-47072
> URL: https://issues.apache.org/jira/browse/SPARK-47072
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.2, 3.5.0, 4.0.0
>Reporter: Max Gekk
>Assignee: Max Gekk
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0, 3.4.3
>
>
> When Spark SQL cannot recognise ANSI interval, it outputs wrong pattern for
> particular ANSI interval. For example, it cannot recognise year-month
> interval, but says about day-time interval:
> {code:sql}
> spark-sql (default)> select interval '-\t2-2\t' year to month;
> Interval string does not match year-month format of `[+|-]d h`, `INTERVAL
> [+|-]'[+|-]d h' DAY TO HOUR` when cast to interval year to month: - 2-2 .
> (line 1, pos 16)
> == SQL ==
> select interval '-\t2-2\t' year to month
> ^^^
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47081) Support Query Execution Progress Messages
[ https://issues.apache.org/jira/browse/SPARK-47081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47081: --- Labels: pull-request-available (was: ) > Support Query Execution Progress Messages > - > > Key: SPARK-47081 > URL: https://issues.apache.org/jira/browse/SPARK-47081 > Project: Spark > Issue Type: Improvement > Components: Connect >Affects Versions: 3.5.0 >Reporter: Martin Grund >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Spark Connect should support reporting basic query progress to the client. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-45881) Use Higher Order aggregate functions from SQL
[
https://issues.apache.org/jira/browse/SPARK-45881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818356#comment-17818356
]
Steven Aerts commented on SPARK-45881:
--
The pull request just got closed because of inactivity, while I think it is
still relevant.
Can someone take a look?
> Use Higher Order aggregate functions from SQL
> -
>
> Key: SPARK-45881
> URL: https://issues.apache.org/jira/browse/SPARK-45881
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Steven Aerts
>Priority: Major
> Labels: pull-request-available
>
> Higher order aggregate funtions are aggregation function which take a lambda
> function as a parameter.
> An example for this from presto is the runction
> {{[reduce_agg|https://prestodb.io/docs/current/functions/aggregate.html#reduce_agg]}}
> which has the signature {{reduce_agg(inputValue T, initialState S,
> inputFunction(S, T, S), combineFunction(S, S, S))}} and it works like this:
> {code:java}
> SELECT id, reduce_agg(value, 0, (a, b) -> a + b, (a, b) -> a + b)
> FROM (VALUES (1, 2), (1, 3), (1, 4), (2, 20), (2, 30), (2, 40)) AS t(id,
> value)
> GROUP BY id;
> -- (1, 9)
> -- (2, 90)
> {code}
> In Spark you can today define, implement and use such a custom function from
> the scala API by implementing a case class which extends from
> {{TypedImperativeAggregate}} and add the {{HigherOrderFunction}} trait.
> However if you try to use this function from the sql api, you get:
> {code:java}
> org.apache.spark.sql.AnalysisException: A lambda function should only be used
> in a higher order function. However, its class is
> org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression,
> which is not a higher order function.; line 2 pos 2
> at
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:52)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$22$$anonfun$applyOrElse$153.$anonfun$applyOrElse$155(Analyzer.scala:2142)
> at scala.Option.map(Option.scala:230)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$22$$anonfun$applyOrElse$153.$anonfun$applyOrElse$154(Analyzer.scala:2135)
> at
> org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:100)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$22$$anonfun$applyOrElse$153.applyOrElse(Analyzer.scala:2143)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$22$$anonfun$applyOrElse$153.applyOrElse(Analyzer.scala:2132)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
> {code}
> There is just a small thing missing in the {{Analyzer}} to get all of this
> working, we will provide a fix, unblocking higher order aggregate functions
> in spark sql.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47087) Raise Spark's exception with an error class in config value check
[ https://issues.apache.org/jira/browse/SPARK-47087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47087: --- Labels: pull-request-available (was: ) > Raise Spark's exception with an error class in config value check > - > > Key: SPARK-47087 > URL: https://issues.apache.org/jira/browse/SPARK-47087 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Max Gekk >Assignee: Max Gekk >Priority: Major > Labels: pull-request-available > > Currently, Spark throws *IllegalArgumentException* in `checkValue` of > ConfigBuilder. Need to overload `checkValue` to throw > `SparkIllegalArgumentException` with an error class. This should improve user > experience with Spark SQL, and impressions of Spark's errors. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-47089) Migrate mockito 4 to mockito5
[ https://issues.apache.org/jira/browse/SPARK-47089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818338#comment-17818338 ] Yang Jie commented on SPARK-47089: -- Thanks [~panbingkun] > Migrate mockito 4 to mockito5 > - > > Key: SPARK-47089 > URL: https://issues.apache.org/jira/browse/SPARK-47089 > Project: Spark > Issue Type: Improvement > Components: Build, Tests >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47085) Preformance issue on thrift API
[
https://issues.apache.org/jira/browse/SPARK-47085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-47085:
---
Labels: pull-request-available (was: )
> Preformance issue on thrift API
> ---
>
> Key: SPARK-47085
> URL: https://issues.apache.org/jira/browse/SPARK-47085
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1, 3.5.0
>Reporter: Izek Greenfield
>Priority: Major
> Labels: pull-request-available
>
> in class `RowSetUtils` there is a loop that has O(n^2) complexity:
> {code:scala}
> ...
> while (i < rowSize) {
> val row = rows(I)
> ...
> {code}
> It can be easily converted into O( n ) complexity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47090) Skip JDK 17/21 Maven build in branch-3.4 scheduled job
[ https://issues.apache.org/jira/browse/SPARK-47090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47090: --- Labels: pull-request-available (was: ) > Skip JDK 17/21 Maven build in branch-3.4 scheduled job > -- > > Key: SPARK-47090 > URL: https://issues.apache.org/jira/browse/SPARK-47090 > Project: Spark > Issue Type: Improvement > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > > https://github.com/apache/spark/actions/runs/7928294496/job/21664443573 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47089) Migrate mockito 4 to mockito5
[ https://issues.apache.org/jira/browse/SPARK-47089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47089: --- Labels: pull-request-available (was: ) > Migrate mockito 4 to mockito5 > - > > Key: SPARK-47089 > URL: https://issues.apache.org/jira/browse/SPARK-47089 > Project: Spark > Issue Type: Improvement > Components: Build, Tests >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47090) Skip JDK 17/21 Maven build in branch-3.4 scheduled job
[ https://issues.apache.org/jira/browse/SPARK-47090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-47090: - Summary: Skip JDK 17/21 Maven build in branch-3.4 scheduled job (was: Skip JDK 21 build in branch-3.4 scheduled job) > Skip JDK 17/21 Maven build in branch-3.4 scheduled job > -- > > Key: SPARK-47090 > URL: https://issues.apache.org/jira/browse/SPARK-47090 > Project: Spark > Issue Type: Improvement > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Priority: Major > > https://github.com/apache/spark/actions/runs/7928294496/job/21664443573 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47090) Skip JDK 21 build in branch-3.4 scheduled job
Hyukjin Kwon created SPARK-47090: Summary: Skip JDK 21 build in branch-3.4 scheduled job Key: SPARK-47090 URL: https://issues.apache.org/jira/browse/SPARK-47090 Project: Spark Issue Type: Improvement Components: Project Infra Affects Versions: 4.0.0 Reporter: Hyukjin Kwon https://github.com/apache/spark/actions/runs/7928294496/job/21664443573 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47089) Migrate mockito 4 to mockito5
Yang Jie created SPARK-47089: Summary: Migrate mockito 4 to mockito5 Key: SPARK-47089 URL: https://issues.apache.org/jira/browse/SPARK-47089 Project: Spark Issue Type: Improvement Components: Build, Tests Affects Versions: 4.0.0 Reporter: Yang Jie -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46938) Migrate jetty 10 to jetty 11
[ https://issues.apache.org/jira/browse/SPARK-46938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46938: --- Labels: pull-request-available (was: ) > Migrate jetty 10 to jetty 11 > > > Key: SPARK-46938 > URL: https://issues.apache.org/jira/browse/SPARK-46938 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47083) Upgrade `commons-codec` to 1.16.1
[ https://issues.apache.org/jira/browse/SPARK-47083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47083. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45152 [https://github.com/apache/spark/pull/45152] > Upgrade `commons-codec` to 1.16.1 > - > > Key: SPARK-47083 > URL: https://issues.apache.org/jira/browse/SPARK-47083 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47083) Upgrade `commons-codec` to 1.16.1
[ https://issues.apache.org/jira/browse/SPARK-47083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47083: - Assignee: BingKun Pan > Upgrade `commons-codec` to 1.16.1 > - > > Key: SPARK-47083 > URL: https://issues.apache.org/jira/browse/SPARK-47083 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47083) Upgrade `commons-codec` to 1.16.1
[ https://issues.apache.org/jira/browse/SPARK-47083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47083: --- Labels: pull-request-available (was: ) > Upgrade `commons-codec` to 1.16.1 > - > > Key: SPARK-47083 > URL: https://issues.apache.org/jira/browse/SPARK-47083 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47084) Upgrade joda-time to 2.12.7
[ https://issues.apache.org/jira/browse/SPARK-47084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-47084. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45153 [https://github.com/apache/spark/pull/45153] > Upgrade joda-time to 2.12.7 > > > Key: SPARK-47084 > URL: https://issues.apache.org/jira/browse/SPARK-47084 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47088) Utilize BigDecimal to calculate the GPU resource
Bobby Wang created SPARK-47088: -- Summary: Utilize BigDecimal to calculate the GPU resource Key: SPARK-47088 URL: https://issues.apache.org/jira/browse/SPARK-47088 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 4.0.0 Reporter: Bobby Wang To prevent precision errors, the current method of calculating GPU resources involves multiplying by 1E16 to convert doubles to Longs. If needed, it will also convert Longs back to doubles. This approach introduces redundancy in the code, especially for test code. More details can be found at https://github.com/apache/spark/pull/44690#discussion_r1482301112 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47087) Raise Spark's exception with an error class in config value check
Max Gekk created SPARK-47087: Summary: Raise Spark's exception with an error class in config value check Key: SPARK-47087 URL: https://issues.apache.org/jira/browse/SPARK-47087 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 4.0.0 Reporter: Max Gekk Assignee: Max Gekk Currently, Spark throws *IllegalArgumentException* in `checkValue` of ConfigBuilder. Need to overload `checkValue` to throw `SparkIllegalArgumentException` with an error class. This should improve user experience with Spark SQL, and impressions of Spark's errors. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47086) Migrate jetty 11 to jetty 12
HiuFung Kwok created SPARK-47086: Summary: Migrate jetty 11 to jetty 12 Key: SPARK-47086 URL: https://issues.apache.org/jira/browse/SPARK-47086 Project: Spark Issue Type: Sub-task Components: Build Affects Versions: 4.0.0 Reporter: HiuFung Kwok -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-47085) Preformance issue on thrift API
[
https://issues.apache.org/jira/browse/SPARK-47085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818256#comment-17818256
]
Izek Greenfield commented on SPARK-47085:
-
https://github.com/apache/spark/pull/45155
> Preformance issue on thrift API
> ---
>
> Key: SPARK-47085
> URL: https://issues.apache.org/jira/browse/SPARK-47085
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.4.1, 3.5.0
>Reporter: Izek Greenfield
>Priority: Major
>
> in class `RowSetUtils` there is a loop that has O(n^2) complexity:
> {code:scala}
> ...
> while (i < rowSize) {
> val row = rows(I)
> ...
> {code}
> It can be easily converted into O( n ) complexity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47085) Preformance issue on thrift API
Izek Greenfield created SPARK-47085:
---
Summary: Preformance issue on thrift API
Key: SPARK-47085
URL: https://issues.apache.org/jira/browse/SPARK-47085
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.5.0, 3.4.1
Reporter: Izek Greenfield
in class `RowSetUtils` there is a loop that has O(n^2) complexity:
{code:scala}
...
while (i < rowSize) {
val row = rows(I)
...
{code}
It can be easily converted into O( n ) complexity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
