[jira] [Created] (SPARK-45368) Remove scala2.12 compatibility logic for DoubleType, FloatType, Decimal
BingKun Pan created SPARK-45368: --- Summary: Remove scala2.12 compatibility logic for DoubleType, FloatType, Decimal Key: SPARK-45368 URL: https://issues.apache.org/jira/browse/SPARK-45368 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 4.0.0 Reporter: BingKun Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45366) Remove productHash from TreeNode
[ https://issues.apache.org/jira/browse/SPARK-45366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45366: --- Labels: pull-request-available (was: ) > Remove productHash from TreeNode > > > Key: SPARK-45366 > URL: https://issues.apache.org/jira/browse/SPARK-45366 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45367) Add errorclass and sqlstate for: _LEGACY_ERROR_TEMP_1273
Serge Rielau created SPARK-45367: Summary: Add errorclass and sqlstate for: _LEGACY_ERROR_TEMP_1273 Key: SPARK-45367 URL: https://issues.apache.org/jira/browse/SPARK-45367 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.5.0 Reporter: Serge Rielau This seems to be a very common error -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-45282) Join loses records for cached datasets
[
https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769893#comment-17769893
]
koert kuipers edited comment on SPARK-45282 at 9/28/23 4:07 AM:
yes i can reproduce it.
master branch on commit:
{code:java}
commit 7e8aafd2c0f1f6fcd03a69afe2b85fd3fda95d20 (HEAD -> master,
upstream/master)
Author: lanmengran1
Date: Tue Sep 26 21:01:02 2023 -0500 [SPARK-45334][SQL] Remove misleading
comment in parquetSchemaConverter {code}
i build spark for k8s using:
{code:java}
$ dev/make-distribution.sh --name kubernetes --tgz -Pkubernetes -Phadoop-cloud
{code}
created docker container using Dockerfile provided in
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
launch pod and shell inside:
{code:java}
185@proxy:~/work-dir$ export SPARK_LOCAL_HOSTNAME=$(hostname -i
185@proxy:~/work-dir$ export SPARK_PUBLIC_DNS=$(hostname -i)
185@proxy:~/work-dir$ /opt/spark/bin/spark-shell --master
k8s://https://kubernetes.default:443 --deploy-mode client --num-executors 4
--executor-memory 2G --conf
spark.serializer=org.apache.spark.serializer.KryoSerializer --conf
spark.kubernetes.namespace=default --conf
spark.sql.adaptive.coalescePartitions.parallelismFirst=false --conf
spark.sql.adaptive.enabled=true --conf
spark.sql.adaptive.advisoryPartitionSizeInBytes=33554432 --conf
spark.sql.optimizer.canChangeCachedPlanOutputPartitioning=true --conf
spark.kubernetes.container.image=.dkr.ecr.us-east-1.amazonaws.com/spark:4.0.0-SNAPSHOT
23/09/28 03:44:57 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
/_/
Using Scala version 2.13.11 (OpenJDK 64-Bit Server VM, Java 21)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context Web UI available at http://10.177.71.94:4040
Spark context available as 'sc' (master = k8s://https://kubernetes.default:443,
app id = spark-5ab0957571944828866a2f23068ff180).
Spark session available as 'spark'.scala> :paste
// Entering paste mode (ctrl-D to finish)import java.util.UUID
import org.apache.spark.sql.functions.col
import spark.implicits._
val data = (1 to 100).toDS().map(i => UUID.randomUUID().toString).persist()
val left = data.map(k => (k, 1))
val right = data.map(k => (k, k)) // if i change this to k => (k, 1) it works!
println("number of left " + left.count())
println("number of right " + right.count())
println("number of (left join right) " +
left.toDF("key", "vertex").join(right.toDF("key", "state"), "key").count()
)
val left1 = left
.toDF("key", "vertex")
.repartition(col("key")) // comment out this line to make it work
.persist()
println("number of left1 " + left1.count())
val right1 = right
.toDF("key", "state")
.repartition(col("key")) // comment out this line to make it work
.persist()
println("number of right1 " + right1.count())
println("number of (left1 join right1) " + left1.join(right1, "key").count())
// this gives incorrect result
// Exiting paste mode, now interpreting.
23/09/28 03:45:30 WARN TaskSetManager: Stage 0 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
23/09/28 03:45:34 WARN TaskSetManager: Stage 1 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of left 100
23/09/28 03:45:36 WARN TaskSetManager: Stage 4 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of right 100
23/09/28 03:45:39 WARN TaskSetManager: Stage 7 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
23/09/28 03:45:40 WARN TaskSetManager: Stage 8 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of (left join right) 100
23/09/28 03:45:45 WARN TaskSetManager: Stage 16 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of left1 100
23/09/28 03:45:48 WARN TaskSetManager: Stage 24 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of right1 100
number of (left1 join right1) 850735
import
[jira] [Comment Edited] (SPARK-45282) Join loses records for cached datasets
[
https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769893#comment-17769893
]
koert kuipers edited comment on SPARK-45282 at 9/28/23 4:06 AM:
yes i can reproduce it.
master branch on commit:
{code:java}
commit 7e8aafd2c0f1f6fcd03a69afe2b85fd3fda95d20 (HEAD -> master,
upstream/master)
Author: lanmengran1
Date: Tue Sep 26 21:01:02 2023 -0500 [SPARK-45334][SQL] Remove misleading
comment in parquetSchemaConverter {code}
i build spark for k8s using:
{code:java}
$ dev/make-distribution.sh --name kubernetes --tgz -Pkubernetes -Phadoop-cloud
{code}
created docker container using Dockerfile provided in
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
launch pod and shell inside:
{code:java}
185@proxy:~/work-dir$ export SPARK_LOCAL_HOSTNAME=$(hostname -i
185@proxy:~/work-dir$ export SPARK_PUBLIC_DNS=$(hostname -i)
185@proxy:~/work-dir$ /opt/spark/bin/spark-shell --master
k8s://https://kubernetes.default:443 --deploy-mode client --num-executors 4
--executor-memory 2G --conf
spark.serializer=org.apache.spark.serializer.KryoSerializer --conf
spark.kubernetes.namespace=default --conf
spark.sql.adaptive.coalescePartitions.parallelismFirst=false --conf
spark.sql.adaptive.enabled=true --conf
spark.sql.adaptive.advisoryPartitionSizeInBytes=33554432 --conf
spark.sql.optimizer.canChangeCachedPlanOutputPartitioning=true --conf
spark.kubernetes.container.image=.dkr.ecr.us-east-1.amazonaws.com/spark:4.0.0-SNAPSHOT
23/09/28 03:44:57 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
/_/
Using Scala version 2.13.11 (OpenJDK 64-Bit Server VM, Java 21)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context Web UI available at http://10.177.71.94:4040
Spark context available as 'sc' (master = k8s://https://kubernetes.default:443,
app id = spark-5ab0957571944828866a2f23068ff180).
Spark session available as 'spark'.scala> :paste
// Entering paste mode (ctrl-D to finish)import java.util.UUID
import org.apache.spark.sql.functions.col
import spark.implicits._
val data = (1 to 100).toDS().map(i => UUID.randomUUID().toString).persist()
val left = data.map(k => (k, 1))
val right = data.map(k => (k, k)) // if i change this to k => (k, 1) it works!
println("number of left " + left.count())
println("number of right " + right.count())
println("number of (left join right) " +
left.toDF("key", "vertex").join(right.toDF("key", "state"), "key").count()
)
val left1 = left
.toDF("key", "vertex")
.repartition(col("key")) // comment out this line to make it work
.persist()
println("number of left1 " + left1.count())
val right1 = right
.toDF("key", "state")
.repartition(col("key")) // comment out this line to make it work
.persist()
println("number of right1 " + right1.count())
println("number of (left1 join right1) " + left1.join(right1, "key").count())
// this gives incorrect result
// Exiting paste mode, now interpreting.
23/09/28 03:45:30 WARN TaskSetManager: Stage 0 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
23/09/28 03:45:34 WARN TaskSetManager: Stage 1 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of left 100
23/09/28 03:45:36 WARN TaskSetManager: Stage 4 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of right 100
23/09/28 03:45:39 WARN TaskSetManager: Stage 7 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
23/09/28 03:45:40 WARN TaskSetManager: Stage 8 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of (left join right) 100
23/09/28 03:45:45 WARN TaskSetManager: Stage 16 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of left1 100
23/09/28 03:45:48 WARN TaskSetManager: Stage 24 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of right1 100
number of (left1 join right1) 850735
import
[jira] [Commented] (SPARK-45282) Join loses records for cached datasets
[
https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769893#comment-17769893
]
koert kuipers commented on SPARK-45282:
---
yes i can reproduce it.
master branch on commit:
{code:java}
commit 7e8aafd2c0f1f6fcd03a69afe2b85fd3fda95d20 (HEAD -> master,
upstream/master)
Author: lanmengran1
Date: Tue Sep 26 21:01:02 2023 -0500 [SPARK-45334][SQL] Remove misleading
comment in parquetSchemaConverter {code}
i build spark for k8s using:
{code:java}
$ dev/make-distribution.sh --name kubernetes --tgz -Pkubernetes -Phadoop-cloud
{code}
created docker container using Dockerfile provided in
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
launch pod and shell inside:
{code:java}
185@proxy:~/work-dir$ export SPARK_LOCAL_HOSTNAME=$(hostname -i
185@proxy:~/work-dir$ export SPARK_PUBLIC_DNS=$(hostname -i)
185@proxy:~/work-dir$ /opt/spark/bin/spark-shell --master
k8s://https://kubernetes.default:443 --deploy-mode client --num-executors 4
--executor-memory 2G --conf
spark.serializer=org.apache.spark.serializer.KryoSerializer --conf
spark.kubernetes.namespace=default --conf
spark.sql.adaptive.coalescePartitions.parallelismFirst=false --conf
spark.sql.adaptive.enabled=true --conf
spark.sql.adaptive.advisoryPartitionSizeInBytes=33554432 --conf
spark.sql.optimizer.canChangeCachedPlanOutputPartitioning=true --conf
spark.kubernetes.container.image=.dkr.ecr.us-east-1.amazonaws.com/spark:4.0.0-SNAPSHOT
23/09/28 03:44:57 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
/_/
Using Scala version 2.13.11 (OpenJDK 64-Bit Server VM, Java 21)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context Web UI available at http://10.177.71.94:4040
Spark context available as 'sc' (master = k8s://https://kubernetes.default:443,
app id = spark-5ab0957571944828866a2f23068ff180).
Spark session available as 'spark'.scala> :paste
// Entering paste mode (ctrl-D to finish)import java.util.UUID
import org.apache.spark.sql.functions.col
import spark.implicits._
val data = (1 to 100).toDS().map(i => UUID.randomUUID().toString).persist()
val left = data.map(k => (k, 1))
val right = data.map(k => (k, k)) // if i change this to k => (k, 1) it works!
println("number of left " + left.count())
println("number of right " + right.count())
println("number of (left join right) " +
left.toDF("key", "vertex").join(right.toDF("key", "state"), "key").count()
)
val left1 = left
.toDF("key", "vertex")
.repartition(col("key")) // comment out this line to make it work
.persist()
println("number of left1 " + left1.count())
val right1 = right
.toDF("key", "state")
.repartition(col("key")) // comment out this line to make it work
.persist()
println("number of right1 " + right1.count())
println("number of (left1 join right1) " + left1.join(right1, "key").count())
// this gives incorrect result
// Exiting paste mode, now interpreting.
23/09/28 03:45:30 WARN TaskSetManager: Stage 0 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
23/09/28 03:45:34 WARN TaskSetManager: Stage 1 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of left 100
23/09/28 03:45:36 WARN TaskSetManager: Stage 4 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of right 100
23/09/28 03:45:39 WARN TaskSetManager: Stage 7 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
23/09/28 03:45:40 WARN TaskSetManager: Stage 8 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of (left join right) 100
23/09/28 03:45:45 WARN TaskSetManager: Stage 16 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of left1 100
23/09/28 03:45:48 WARN TaskSetManager: Stage 24 contains a task of very large
size (6631 KiB). The maximum recommended task size is 1000 KiB.
number of right1 100
number of (left1 join right1) 850735
import java.util.UUID
import
[jira] [Updated] (SPARK-45361) Describe characters unescaping in string literals
[ https://issues.apache.org/jira/browse/SPARK-45361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45361: --- Labels: pull-request-available (was: ) > Describe characters unescaping in string literals > - > > Key: SPARK-45361 > URL: https://issues.apache.org/jira/browse/SPARK-45361 > Project: Spark > Issue Type: Documentation > Components: SQL >Affects Versions: 4.0.0 >Reporter: Max Gekk >Assignee: Max Gekk >Priority: Major > Labels: pull-request-available > > Update the page > https://spark.apache.org/docs/latest/sql-ref-literals.html#string-literal and > describe the escaping implemented at > https://github.com/apache/spark/blob/9109d7037f44158e72d14019eb33f9c7b8838868/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala#L38 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33458) Hive partition pruning support Contains, StartsWith and EndsWith predicate
[ https://issues.apache.org/jira/browse/SPARK-33458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769889#comment-17769889 ] dzcxzl commented on SPARK-33458: After [HIVE-22900|https://issues.apache.org/jira/browse/HIVE-22900] (HMS 4.0), like filter partition supports direct sql. Now Spark uses .* method, which may cause incorrect results. Because .* is the way to write JDO query, direct sql must use %. > Hive partition pruning support Contains, StartsWith and EndsWith predicate > -- > > Key: SPARK-33458 > URL: https://issues.apache.org/jira/browse/SPARK-33458 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Fix For: 3.1.0 > > > Hive partition pruning can support Contains, StartsWith and EndsWith > predicate: > https://github.com/apache/hive/blob/0c2c8a7f57330880f156466526bc0fdc94681035/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L1074-L1075 > https://github.com/apache/hive/commit/0c2c8a7f57330880f156466526bc0fdc94681035#diff-b1200d4259fafd48d7bbd0050e89772218813178f68461a2e82551c52319b282 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45344) Remove all scala version string check
[ https://issues.apache.org/jira/browse/SPARK-45344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45344. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43133 [https://github.com/apache/spark/pull/43133] > Remove all scala version string check > - > > Key: SPARK-45344 > URL: https://issues.apache.org/jira/browse/SPARK-45344 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45344) Remove all scala version string check
[ https://issues.apache.org/jira/browse/SPARK-45344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45344: Assignee: Yang Jie > Remove all scala version string check > - > > Key: SPARK-45344 > URL: https://issues.apache.org/jira/browse/SPARK-45344 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45353) Refine docstring of `create_map/slice/array_join`
[ https://issues.apache.org/jira/browse/SPARK-45353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45353. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43145 [https://github.com/apache/spark/pull/43145] > Refine docstring of `create_map/slice/array_join` > - > > Key: SPARK-45353 > URL: https://issues.apache.org/jira/browse/SPARK-45353 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45353) Refine docstring of `create_map/slice/array_join`
[ https://issues.apache.org/jira/browse/SPARK-45353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45353: Assignee: Yang Jie > Refine docstring of `create_map/slice/array_join` > - > > Key: SPARK-45353 > URL: https://issues.apache.org/jira/browse/SPARK-45353 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45342) Remove the scala doc compilation option specific to Scala 2.12.
[ https://issues.apache.org/jira/browse/SPARK-45342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45342. -- Resolution: Duplicate > Remove the scala doc compilation option specific to Scala 2.12. > --- > > Key: SPARK-45342 > URL: https://issues.apache.org/jira/browse/SPARK-45342 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45364) Clean up the unnecessary Scala 2.12 logical in SparkBuild
[ https://issues.apache.org/jira/browse/SPARK-45364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-45364: - Parent: SPARK-45314 Issue Type: Sub-task (was: Improvement) > Clean up the unnecessary Scala 2.12 logical in SparkBuild > - > > Key: SPARK-45364 > URL: https://issues.apache.org/jira/browse/SPARK-45364 > Project: Spark > Issue Type: Sub-task > Components: Build, Project Infra >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45342) Remove the scala doc compilation option specific to Scala 2.12.
[ https://issues.apache.org/jira/browse/SPARK-45342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45342: --- Labels: pull-request-available (was: ) > Remove the scala doc compilation option specific to Scala 2.12. > --- > > Key: SPARK-45342 > URL: https://issues.apache.org/jira/browse/SPARK-45342 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45366) Remove productHash from TreeNode
BingKun Pan created SPARK-45366: --- Summary: Remove productHash from TreeNode Key: SPARK-45366 URL: https://issues.apache.org/jira/browse/SPARK-45366 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 4.0.0 Reporter: BingKun Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45365) Allow the daily tests of branch-3.4 to use the new test group tags
[ https://issues.apache.org/jira/browse/SPARK-45365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45365: --- Labels: pull-request-available (was: ) > Allow the daily tests of branch-3.4 to use the new test group tags > -- > > Key: SPARK-45365 > URL: https://issues.apache.org/jira/browse/SPARK-45365 > Project: Spark > Issue Type: Improvement > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-44442) Drop mesos support
[
https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-2:
Assignee: Sean R. Owen
> Drop mesos support
> --
>
> Key: SPARK-2
> URL: https://issues.apache.org/jira/browse/SPARK-2
> Project: Spark
> Issue Type: Sub-task
> Components: Mesos
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Sean R. Owen
>Priority: Major
> Labels: pull-request-available
>
> [https://spark.apache.org/docs/latest/running-on-mesos.html]
>
> {_}Note{_}: Apache Mesos support is deprecated as of Apache Spark 3.2.0. It
> will be removed in a future version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45364) Clean up the unnecessary Scala 2.12 logical in SparkBuild
[ https://issues.apache.org/jira/browse/SPARK-45364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45364: --- Labels: pull-request-available (was: ) > Clean up the unnecessary Scala 2.12 logical in SparkBuild > - > > Key: SPARK-45364 > URL: https://issues.apache.org/jira/browse/SPARK-45364 > Project: Spark > Issue Type: Improvement > Components: Build, Project Infra >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44442) Drop mesos support
[
https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-2.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43135
[https://github.com/apache/spark/pull/43135]
> Drop mesos support
> --
>
> Key: SPARK-2
> URL: https://issues.apache.org/jira/browse/SPARK-2
> Project: Spark
> Issue Type: Sub-task
> Components: Mesos
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Sean R. Owen
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> [https://spark.apache.org/docs/latest/running-on-mesos.html]
>
> {_}Note{_}: Apache Mesos support is deprecated as of Apache Spark 3.2.0. It
> will be removed in a future version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45365) Allow the daily tests of branch-3.4 to use the new test group tags
Yang Jie created SPARK-45365: Summary: Allow the daily tests of branch-3.4 to use the new test group tags Key: SPARK-45365 URL: https://issues.apache.org/jira/browse/SPARK-45365 Project: Spark Issue Type: Improvement Components: Project Infra Affects Versions: 4.0.0 Reporter: Yang Jie -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44074) `Logging plan changes for execution` test failed
[
https://issues.apache.org/jira/browse/SPARK-44074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-44074:
---
Labels: pull-request-available (was: )
> `Logging plan changes for execution` test failed
>
>
> Key: SPARK-44074
> URL: https://issues.apache.org/jira/browse/SPARK-44074
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 3.5.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> run {{build/sbt clean "sql/test"
> -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest}}
> {{}}
> {code:java}
> 2023-06-15T19:58:34.4105460Z �[0m[�[0m�[0minfo�[0m]
> �[0m�[0m�[32mQueryExecutionSuite:�[0m�[0m
> 2023-06-15T19:58:34.5395268Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- dumping
> query execution info to a file (77 milliseconds)�[0m�[0m
> 2023-06-15T19:58:34.5856902Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- dumping
> query execution info to an existing file (49 milliseconds)�[0m�[0m
> 2023-06-15T19:58:34.6099849Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- dumping
> query execution info to non-existing folder (25 milliseconds)�[0m�[0m
> 2023-06-15T19:58:34.6136467Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- dumping
> query execution info by invalid path (4 milliseconds)�[0m�[0m
> 2023-06-15T19:58:34.6425071Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- dumping
> query execution info to a file - explainMode=formatted (28
> milliseconds)�[0m�[0m
> 2023-06-15T19:58:34.7084916Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- limit
> number of fields by sql config (66 milliseconds)�[0m�[0m
> 2023-06-15T19:58:34.7432299Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- check
> maximum fields restriction (34 milliseconds)�[0m�[0m
> 2023-06-15T19:58:34.7554546Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- toString()
> exception/error handling (11 milliseconds)�[0m�[0m
> 2023-06-15T19:58:34.7621424Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m-
> SPARK-28346: clone the query plan between different stages (6
> milliseconds)�[0m�[0m
> 2023-06-15T19:58:34.8001412Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m- Logging
> plan changes for execution *** FAILED *** (12 milliseconds)�[0m�[0m
> 2023-06-15T19:58:34.8007977Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m
> testAppender.loggingEvents.exists(((x$10:
> org.apache.logging.log4j.core.LogEvent) =>
> x$10.getMessage().getFormattedMessage().contains(expectedMsg))) was false
> (QueryExecutionSuite.scala:232)�[0m�[0m
> {code}
>
> but run {{build/sbt "sql/testOnly *QueryExecutionSuite"}} not this issue,
> need to investigate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45364) Clean up the unnecessary Scala 2.12 logical in SparkBuild
BingKun Pan created SPARK-45364: --- Summary: Clean up the unnecessary Scala 2.12 logical in SparkBuild Key: SPARK-45364 URL: https://issues.apache.org/jira/browse/SPARK-45364 Project: Spark Issue Type: Improvement Components: Build, Project Infra Affects Versions: 4.0.0 Reporter: BingKun Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43801) Support unwrap date type to string type in UnwrapCastInBinaryComparison
[ https://issues.apache.org/jira/browse/SPARK-43801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-43801: --- Labels: pull-request-available (was: ) > Support unwrap date type to string type in UnwrapCastInBinaryComparison > --- > > Key: SPARK-43801 > URL: https://issues.apache.org/jira/browse/SPARK-43801 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.5.0 >Reporter: Pucheng Yang >Priority: Major > Labels: pull-request-available > > Similar to https://issues.apache.org/jira/browse/SPARK-42597 and others, add > support to > UnwrapCastInBinaryComparison such that it can unwrap date type to string type. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-38230) InsertIntoHadoopFsRelationCommand unnecessarily fetches details of partitions in most cases
[ https://issues.apache.org/jira/browse/SPARK-38230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-38230: --- Labels: pull-request-available (was: ) > InsertIntoHadoopFsRelationCommand unnecessarily fetches details of partitions > in most cases > --- > > Key: SPARK-38230 > URL: https://issues.apache.org/jira/browse/SPARK-38230 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.2, 3.3.0, 3.4.0, 3.5.0 >Reporter: Coal Chan >Priority: Major > Labels: pull-request-available > > In > `org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand`, > `sparkSession.sessionState.catalog.listPartitions` will call method > `org.apache.hadoop.hive.metastore.listPartitionsPsWithAuth` of hive metastore > client, this method will produce multiple queries per partition on hive > metastore db. So when you insert into a table which has too many > partitions(ie: 10k), it will produce too many queries on hive metastore > db(ie: n * 10k = 10nk), it puts a lot of strain on the database. > In fact, it calls method `listPartitions` in order to get locations of > partitions and get `customPartitionLocations`. But in most cases, we do not > have custom partitions, we can just get partition names, so we can call > method listPartitionNames. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45363) Avro connector should use new error class framework
Neil Ramaswamy created SPARK-45363: -- Summary: Avro connector should use new error class framework Key: SPARK-45363 URL: https://issues.apache.org/jira/browse/SPARK-45363 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Neil Ramaswamy In our connector for Avro (AvroDataToCatalyst), if we detect a malformed record, we currently throw a raw SparkException. We'd like to use our new error class framework for consistency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45354) Resolve functions bottom-up
[
https://issues.apache.org/jira/browse/SPARK-45354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth resolved SPARK-45354.
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43146
[https://github.com/apache/spark/pull/43146]
> Resolve functions bottom-up
> ---
>
> Key: SPARK-45354
> URL: https://issues.apache.org/jira/browse/SPARK-45354
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Peter Toth
>Assignee: Peter Toth
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> This PR proposes bottum-up resolution in {{{}ResolveFunctions{}}}, which is
> much faster if we have deeply nested {{{}UnresolvedFunctions{}}}. These
> structures are more likely to occur after
> [#42864|https://github.com/apache/spark/pull/42864].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45354) Resolve functions bottom-up
[
https://issues.apache.org/jira/browse/SPARK-45354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth reassigned SPARK-45354:
--
Assignee: Peter Toth
> Resolve functions bottom-up
> ---
>
> Key: SPARK-45354
> URL: https://issues.apache.org/jira/browse/SPARK-45354
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Peter Toth
>Assignee: Peter Toth
>Priority: Major
> Labels: pull-request-available
>
> This PR proposes bottum-up resolution in {{{}ResolveFunctions{}}}, which is
> much faster if we have deeply nested {{{}UnresolvedFunctions{}}}. These
> structures are more likely to occur after
> [#42864|https://github.com/apache/spark/pull/42864].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45362) Project out PARTITION BY expressions before 'eval' method consumes input rows
[ https://issues.apache.org/jira/browse/SPARK-45362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45362: --- Labels: pull-request-available (was: ) > Project out PARTITION BY expressions before 'eval' method consumes input rows > - > > Key: SPARK-45362 > URL: https://issues.apache.org/jira/browse/SPARK-45362 > Project: Spark > Issue Type: Sub-task > Components: PySpark, SQL >Affects Versions: 4.0.0 >Reporter: Daniel >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45362) Project out PARTITION BY expressions before 'eval' method consumes input rows
Daniel created SPARK-45362: -- Summary: Project out PARTITION BY expressions before 'eval' method consumes input rows Key: SPARK-45362 URL: https://issues.apache.org/jira/browse/SPARK-45362 Project: Spark Issue Type: Sub-task Components: PySpark, SQL Affects Versions: 4.0.0 Reporter: Daniel -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45361) Describe characters unescaping in string literals
Max Gekk created SPARK-45361: Summary: Describe characters unescaping in string literals Key: SPARK-45361 URL: https://issues.apache.org/jira/browse/SPARK-45361 Project: Spark Issue Type: Documentation Components: SQL Affects Versions: 4.0.0 Reporter: Max Gekk Assignee: Max Gekk Update the page https://spark.apache.org/docs/latest/sql-ref-literals.html#string-literal and describe the escaping implemented at https://github.com/apache/spark/blob/9109d7037f44158e72d14019eb33f9c7b8838868/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala#L38 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45357) Maven test `SparkConnectProtoSuite` failed
[
https://issues.apache.org/jira/browse/SPARK-45357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-45357:
---
Labels: pull-request-available (was: )
> Maven test `SparkConnectProtoSuite` failed
> --
>
> Key: SPARK-45357
> URL: https://issues.apache.org/jira/browse/SPARK-45357
> Project: Spark
> Issue Type: Bug
> Components: Connect
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Priority: Major
> Labels: pull-request-available
>
>
> build/mvn clean install -pl connector/connect/server -am -DskipTests
> mvn test -pl connector/connect/server
>
> {code:java}
> - Test observe *** FAILED ***
> == FAIL: Plans do not match ===
> !CollectMetrics my_metric, [min(id#0) AS min_val#0, max(id#0) AS max_val#0,
> sum(id#0) AS sum(id)#0L], 0 CollectMetrics my_metric, [min(id#0) AS
> min_val#0, max(id#0) AS max_val#0, sum(id#0) AS sum(id)#0L], 53
> +- LocalRelation , [id#0, name#0]
> +- LocalRelation , [id#0, name#0]
> (PlanTest.scala:179) {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-44838) Enhance raise_error() to exploit the new error framework
[ https://issues.apache.org/jira/browse/SPARK-44838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-44838: -- Assignee: Serge Rielau > Enhance raise_error() to exploit the new error framework > > > Key: SPARK-44838 > URL: https://issues.apache.org/jira/browse/SPARK-44838 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > > raise_error() and assert_true() do not presently utilize the new error > framework. > We want to generalize raise_error() to take an error class, sqlstate and > message parameters as arguments to compose a well-formed error condition. > The existing assert_true(0 and raise_error() versions should return an error > class -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44838) Enhance raise_error() to exploit the new error framework
[ https://issues.apache.org/jira/browse/SPARK-44838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-44838. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42985 [https://github.com/apache/spark/pull/42985] > Enhance raise_error() to exploit the new error framework > > > Key: SPARK-44838 > URL: https://issues.apache.org/jira/browse/SPARK-44838 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > raise_error() and assert_true() do not presently utilize the new error > framework. > We want to generalize raise_error() to take an error class, sqlstate and > message parameters as arguments to compose a well-formed error condition. > The existing assert_true(0 and raise_error() versions should return an error > class -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44539) Upgrade RoaringBitmap to 1.0.0
[ https://issues.apache.org/jira/browse/SPARK-44539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-44539: - Priority: Minor (was: Trivial) > Upgrade RoaringBitmap to 1.0.0 > --- > > Key: SPARK-44539 > URL: https://issues.apache.org/jira/browse/SPARK-44539 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-44539) Upgrade RoaringBitmap to 1.0.0
[ https://issues.apache.org/jira/browse/SPARK-44539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-44539: Assignee: BingKun Pan > Upgrade RoaringBitmap to 1.0.0 > --- > > Key: SPARK-44539 > URL: https://issues.apache.org/jira/browse/SPARK-44539 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Trivial > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44539) Upgrade RoaringBitmap to 1.0.0
[ https://issues.apache.org/jira/browse/SPARK-44539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44539. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42143 [https://github.com/apache/spark/pull/42143] > Upgrade RoaringBitmap to 1.0.0 > --- > > Key: SPARK-44539 > URL: https://issues.apache.org/jira/browse/SPARK-44539 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45360) Initialize spark session builder configuration from SPARK_REMOTE
[ https://issues.apache.org/jira/browse/SPARK-45360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45360: --- Labels: pull-request-available (was: ) > Initialize spark session builder configuration from SPARK_REMOTE > > > Key: SPARK-45360 > URL: https://issues.apache.org/jira/browse/SPARK-45360 > Project: Spark > Issue Type: New Feature > Components: Connect >Affects Versions: 3.5.0, 4.0.0 >Reporter: Yihong He >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45360) Initialize spark session builder configuration from SPARK_REMOTE
Yihong He created SPARK-45360: - Summary: Initialize spark session builder configuration from SPARK_REMOTE Key: SPARK-45360 URL: https://issues.apache.org/jira/browse/SPARK-45360 Project: Spark Issue Type: New Feature Components: Connect Affects Versions: 3.5.0, 4.0.0 Reporter: Yihong He -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45242) Use DataFrame ID to semantically validate CollectMetrics
[ https://issues.apache.org/jira/browse/SPARK-45242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45242: --- Labels: pull-request-available (was: ) > Use DataFrame ID to semantically validate CollectMetrics > - > > Key: SPARK-45242 > URL: https://issues.apache.org/jira/browse/SPARK-45242 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45357) Maven test `SparkConnectProtoSuite` failed
[
https://issues.apache.org/jira/browse/SPARK-45357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-45357:
-
Description:
build/mvn clean install -pl connector/connect/server -am -DskipTests
mvn test -pl connector/connect/server
{code:java}
- Test observe *** FAILED ***
== FAIL: Plans do not match ===
!CollectMetrics my_metric, [min(id#0) AS min_val#0, max(id#0) AS max_val#0,
sum(id#0) AS sum(id)#0L], 0 CollectMetrics my_metric, [min(id#0) AS
min_val#0, max(id#0) AS max_val#0, sum(id#0) AS sum(id)#0L], 53
+- LocalRelation , [id#0, name#0]
+- LocalRelation , [id#0, name#0]
(PlanTest.scala:179) {code}
was:
{code:java}
- Test observe *** FAILED ***
== FAIL: Plans do not match ===
!CollectMetrics my_metric, [min(id#0) AS min_val#0, max(id#0) AS max_val#0,
sum(id#0) AS sum(id)#0L], 0 CollectMetrics my_metric, [min(id#0) AS
min_val#0, max(id#0) AS max_val#0, sum(id#0) AS sum(id)#0L], 53
+- LocalRelation , [id#0, name#0]
+- LocalRelation , [id#0, name#0]
(PlanTest.scala:179) {code}
> Maven test `SparkConnectProtoSuite` failed
> --
>
> Key: SPARK-45357
> URL: https://issues.apache.org/jira/browse/SPARK-45357
> Project: Spark
> Issue Type: Bug
> Components: Connect
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Priority: Major
>
>
> build/mvn clean install -pl connector/connect/server -am -DskipTests
> mvn test -pl connector/connect/server
>
> {code:java}
> - Test observe *** FAILED ***
> == FAIL: Plans do not match ===
> !CollectMetrics my_metric, [min(id#0) AS min_val#0, max(id#0) AS max_val#0,
> sum(id#0) AS sum(id)#0L], 0 CollectMetrics my_metric, [min(id#0) AS
> min_val#0, max(id#0) AS max_val#0, sum(id#0) AS sum(id)#0L], 53
> +- LocalRelation , [id#0, name#0]
> +- LocalRelation , [id#0, name#0]
> (PlanTest.scala:179) {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45358) Remove shim classes for Hive prior 2.0.0
[ https://issues.apache.org/jira/browse/SPARK-45358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45358: --- Labels: pull-request-available (was: ) > Remove shim classes for Hive prior 2.0.0 > > > Key: SPARK-45358 > URL: https://issues.apache.org/jira/browse/SPARK-45358 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45354) Resolve functions bottom-up
[
https://issues.apache.org/jira/browse/SPARK-45354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth updated SPARK-45354:
---
Description: This PR proposes bottum-up resolution in
{{{}ResolveFunctions{}}}, which is much faster if we have deeply nested
{{{}UnresolvedFunctions{}}}. These structures are more likely to occur after
[#42864|https://github.com/apache/spark/pull/42864].
> Resolve functions bottom-up
> ---
>
> Key: SPARK-45354
> URL: https://issues.apache.org/jira/browse/SPARK-45354
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Peter Toth
>Priority: Major
> Labels: pull-request-available
>
> This PR proposes bottum-up resolution in {{{}ResolveFunctions{}}}, which is
> much faster if we have deeply nested {{{}UnresolvedFunctions{}}}. These
> structures are more likely to occur after
> [#42864|https://github.com/apache/spark/pull/42864].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45343) CSV multiLine documentation is confusing
[ https://issues.apache.org/jira/browse/SPARK-45343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45343: Assignee: Bill Schneider > CSV multiLine documentation is confusing > > > Key: SPARK-45343 > URL: https://issues.apache.org/jira/browse/SPARK-45343 > Project: Spark > Issue Type: Documentation > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: Bill Schneider >Assignee: Bill Schneider >Priority: Trivial > Labels: pull-request-available > > This is confusing, maybe copy-paste from JSON: > |Parse one record, which may span multiple lines, per file. CSV built-in > functions ignore this option.| > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45343) CSV multiLine documentation is confusing
[ https://issues.apache.org/jira/browse/SPARK-45343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45343. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43132 [https://github.com/apache/spark/pull/43132] > CSV multiLine documentation is confusing > > > Key: SPARK-45343 > URL: https://issues.apache.org/jira/browse/SPARK-45343 > Project: Spark > Issue Type: Documentation > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: Bill Schneider >Assignee: Bill Schneider >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > > This is confusing, maybe copy-paste from JSON: > |Parse one record, which may span multiple lines, per file. CSV built-in > functions ignore this option.| > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-44170) Migrating Junit4 to Junit5
[ https://issues.apache.org/jira/browse/SPARK-44170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-44170: Assignee: Yang Jie > Migrating Junit4 to Junit5 > -- > > Key: SPARK-44170 > URL: https://issues.apache.org/jira/browse/SPARK-44170 > Project: Spark > Issue Type: Sub-task > Components: Tests >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > > JUnit5 is a powerful and flexible update to the JUnit framework, and it > provides a variety of improvements and new features to organize and > describe test cases, as well as help in understanding test results： > # JUnit 5 leverages features from Java 8 or later, such as lambda functions, > making tests more powerful and easier to maintain, but Junit 4 still a Java 7 > compatible version > # JUnit 5 has added some useful new features for describing, organizing, and > executing tests. For examples: [Parameterized > Tests|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests] > and [Conditional Test > Execution|https://junit.org/junit5/docs/current/user-guide/#extensions-conditions] > may make our test code look simpler, [Parallel > Execution|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution] > may make our test faster > > More importantly, Junit4 is currently an inactive project, which has not > released a new version for more than two years > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44170) Migrating Junit4 to Junit5
[ https://issues.apache.org/jira/browse/SPARK-44170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44170. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43074 [https://github.com/apache/spark/pull/43074] > Migrating Junit4 to Junit5 > -- > > Key: SPARK-44170 > URL: https://issues.apache.org/jira/browse/SPARK-44170 > Project: Spark > Issue Type: Sub-task > Components: Tests >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > JUnit5 is a powerful and flexible update to the JUnit framework, and it > provides a variety of improvements and new features to organize and > describe test cases, as well as help in understanding test results： > # JUnit 5 leverages features from Java 8 or later, such as lambda functions, > making tests more powerful and easier to maintain, but Junit 4 still a Java 7 > compatible version > # JUnit 5 has added some useful new features for describing, organizing, and > executing tests. For examples: [Parameterized > Tests|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests] > and [Conditional Test > Execution|https://junit.org/junit5/docs/current/user-guide/#extensions-conditions] > may make our test code look simpler, [Parallel > Execution|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution] > may make our test faster > > More importantly, Junit4 is currently an inactive project, which has not > released a new version for more than two years > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45338: Assignee: Jia Fan > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Assignee: Jia Fan >Priority: Major > Labels: pull-request-available > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45338. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43126 [https://github.com/apache/spark/pull/43126] > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Assignee: Jia Fan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45341) Make the sbt doc command execute successfully with Java 17
[
https://issues.apache.org/jira/browse/SPARK-45341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-45341:
-
Priority: Minor (was: Major)
> Make the sbt doc command execute successfully with Java 17
> --
>
> Key: SPARK-45341
> URL: https://issues.apache.org/jira/browse/SPARK-45341
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> [error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/Picked up
> JAVA_TOOL_OPTIONS:-Duser.language=en
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/ArrayWrappers.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVIndex.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/UnsupportedStoreVersionException.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreView.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreSerializer.java...
> [error] Constructing Javadoc information...
> [error] Building index for all the packages and classes...
> [error] Standard Doclet version 17.0.8+7-LTS
> [error] Building tree for all the packages and classes...
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java:32:1:
> error: heading used out of sequence: , compared to implicit preceding
> heading:
> [error] * Serialization
> [error] ^Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/InMemoryStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVIndex.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreIterator.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreSerializer.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreView.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVTypeInfo.html...
> [error] Generating
>
[jira] [Resolved] (SPARK-45341) Make the sbt doc command execute successfully with Java 17
[
https://issues.apache.org/jira/browse/SPARK-45341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-45341.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43131
[https://github.com/apache/spark/pull/43131]
> Make the sbt doc command execute successfully with Java 17
> --
>
> Key: SPARK-45341
> URL: https://issues.apache.org/jira/browse/SPARK-45341
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> [error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/Picked up
> JAVA_TOOL_OPTIONS:-Duser.language=en
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/ArrayWrappers.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVIndex.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/UnsupportedStoreVersionException.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreView.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreSerializer.java...
> [error] Constructing Javadoc information...
> [error] Building index for all the packages and classes...
> [error] Standard Doclet version 17.0.8+7-LTS
> [error] Building tree for all the packages and classes...
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java:32:1:
> error: heading used out of sequence: , compared to implicit preceding
> heading:
> [error] * Serialization
> [error] ^Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/InMemoryStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVIndex.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreIterator.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreSerializer.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreView.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVTypeInfo.html...
> [error] Generating
[jira] [Assigned] (SPARK-45341) Make the sbt doc command execute successfully with Java 17
[
https://issues.apache.org/jira/browse/SPARK-45341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-45341:
Assignee: Yang Jie
> Make the sbt doc command execute successfully with Java 17
> --
>
> Key: SPARK-45341
> URL: https://issues.apache.org/jira/browse/SPARK-45341
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/Picked up
> JAVA_TOOL_OPTIONS:-Duser.language=en
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/ArrayWrappers.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVIndex.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/UnsupportedStoreVersionException.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreView.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreSerializer.java...
> [error] Constructing Javadoc information...
> [error] Building index for all the packages and classes...
> [error] Standard Doclet version 17.0.8+7-LTS
> [error] Building tree for all the packages and classes...
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java:32:1:
> error: heading used out of sequence: , compared to implicit preceding
> heading:
> [error] * Serialization
> [error] ^Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/InMemoryStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVIndex.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreIterator.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreSerializer.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreView.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVTypeInfo.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/LevelDB.html...
>
[jira] [Updated] (SPARK-45359) DataFrame.{columns, colRegex, explain} should raise exceptions when plan is invalid
[
https://issues.apache.org/jira/browse/SPARK-45359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-45359:
---
Labels: pull-request-available (was: )
> DataFrame.{columns, colRegex, explain} should raise exceptions when plan is
> invalid
> ---
>
> Key: SPARK-45359
> URL: https://issues.apache.org/jira/browse/SPARK-45359
> Project: Spark
> Issue Type: Improvement
> Components: Connect, PySpark
>Affects Versions: 4.0.0
>Reporter: Ruifeng Zheng
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45359) DataFrame.{columns, colRegex, explain} should raise exceptions when plan is invalid
Ruifeng Zheng created SPARK-45359:
-
Summary: DataFrame.{columns, colRegex, explain} should raise
exceptions when plan is invalid
Key: SPARK-45359
URL: https://issues.apache.org/jira/browse/SPARK-45359
Project: Spark
Issue Type: Improvement
Components: Connect, PySpark
Affects Versions: 4.0.0
Reporter: Ruifeng Zheng
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45358) Remove shim classes for Hive prior 2.0.0
Cheng Pan created SPARK-45358: - Summary: Remove shim classes for Hive prior 2.0.0 Key: SPARK-45358 URL: https://issues.apache.org/jira/browse/SPARK-45358 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45357) Maven test `SparkConnectProtoSuite` failed
Yang Jie created SPARK-45357:
Summary: Maven test `SparkConnectProtoSuite` failed
Key: SPARK-45357
URL: https://issues.apache.org/jira/browse/SPARK-45357
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 4.0.0
Reporter: Yang Jie
{code:java}
- Test observe *** FAILED ***
== FAIL: Plans do not match ===
!CollectMetrics my_metric, [min(id#0) AS min_val#0, max(id#0) AS max_val#0,
sum(id#0) AS sum(id)#0L], 0 CollectMetrics my_metric, [min(id#0) AS
min_val#0, max(id#0) AS max_val#0, sum(id#0) AS sum(id)#0L], 53
+- LocalRelation , [id#0, name#0]
+- LocalRelation , [id#0, name#0]
(PlanTest.scala:179) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45356) Optimize the Maven daily test configuration
[ https://issues.apache.org/jira/browse/SPARK-45356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-45356: - Summary: Optimize the Maven daily test configuration (was: Sync Python dependencies installation for Maven daily test) > Optimize the Maven daily test configuration > --- > > Key: SPARK-45356 > URL: https://issues.apache.org/jira/browse/SPARK-45356 > Project: Spark > Issue Type: Improvement > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31622) Test-jar in the Spark distribution
[
https://issues.apache.org/jira/browse/SPARK-31622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769558#comment-17769558
]
Arseniy Tashoyan commented on SPARK-31622:
--
In Spark 3.4.1 there are no test jars in the distribution. Is this issue fixed?
> Test-jar in the Spark distribution
> --
>
> Key: SPARK-31622
> URL: https://issues.apache.org/jira/browse/SPARK-31622
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 3.0.0
>Reporter: Arseniy Tashoyan
>Priority: Minor
>
> The jar with classifier *tests* is delivered in the Spark distribution:
> {code:java}
> ls -1 spark-3.0.0-preview2-bin-hadoop2.7/jars/ | grep tests
> spark-tags_2.12-3.0.0-preview2-tests.jar
> {code}
> Normally, test-jars should not be used for production.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45356) Sync Python dependencies installation for Maven daily test
[ https://issues.apache.org/jira/browse/SPARK-45356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45356: --- Labels: pull-request-available (was: ) > Sync Python dependencies installation for Maven daily test > -- > > Key: SPARK-45356 > URL: https://issues.apache.org/jira/browse/SPARK-45356 > Project: Spark > Issue Type: Improvement > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45356) Sync Python dependencies installation for Maven daily test
Yang Jie created SPARK-45356: Summary: Sync Python dependencies installation for Maven daily test Key: SPARK-45356 URL: https://issues.apache.org/jira/browse/SPARK-45356 Project: Spark Issue Type: Improvement Components: Project Infra Affects Versions: 4.0.0 Reporter: Yang Jie -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45355) Fix function groups in Scala Doc
Ruifeng Zheng created SPARK-45355: - Summary: Fix function groups in Scala Doc Key: SPARK-45355 URL: https://issues.apache.org/jira/browse/SPARK-45355 Project: Spark Issue Type: Improvement Components: Documentation, SQL Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45337) Refactor `AbstractCommandBuilder#getScalaVersion` to remove the check for Scala 2.12.
[ https://issues.apache.org/jira/browse/SPARK-45337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45337. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43125 [https://github.com/apache/spark/pull/43125] > Refactor `AbstractCommandBuilder#getScalaVersion` to remove the check for > Scala 2.12. > -- > > Key: SPARK-45337 > URL: https://issues.apache.org/jira/browse/SPARK-45337 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45337) Refactor `AbstractCommandBuilder#getScalaVersion` to remove the check for Scala 2.12.
[ https://issues.apache.org/jira/browse/SPARK-45337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45337: Assignee: Yang Jie > Refactor `AbstractCommandBuilder#getScalaVersion` to remove the check for > Scala 2.12. > -- > > Key: SPARK-45337 > URL: https://issues.apache.org/jira/browse/SPARK-45337 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45354) Resolve functions bottom-up
[ https://issues.apache.org/jira/browse/SPARK-45354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45354: --- Labels: pull-request-available (was: ) > Resolve functions bottom-up > --- > > Key: SPARK-45354 > URL: https://issues.apache.org/jira/browse/SPARK-45354 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Peter Toth >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45354) Resolve functions bottom-up
Peter Toth created SPARK-45354: -- Summary: Resolve functions bottom-up Key: SPARK-45354 URL: https://issues.apache.org/jira/browse/SPARK-45354 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Peter Toth -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45353) Refine docstring of `create_map/slice/array_join`
[ https://issues.apache.org/jira/browse/SPARK-45353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45353: --- Labels: pull-request-available (was: ) > Refine docstring of `create_map/slice/array_join` > - > > Key: SPARK-45353 > URL: https://issues.apache.org/jira/browse/SPARK-45353 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45353) Refine docstring of `create_map/slice/array_join`
Yang Jie created SPARK-45353: Summary: Refine docstring of `create_map/slice/array_join` Key: SPARK-45353 URL: https://issues.apache.org/jira/browse/SPARK-45353 Project: Spark Issue Type: Sub-task Components: Documentation, PySpark Affects Versions: 4.0.0 Reporter: Yang Jie -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45338: -- Assignee: Apache Spark > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Assignee: Apache Spark >Priority: Major > Labels: pull-request-available > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45338: -- Assignee: (was: Apache Spark) > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Priority: Major > Labels: pull-request-available > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44101) Support pandas 2
[ https://issues.apache.org/jira/browse/SPARK-44101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-44101. - Resolution: Fixed > Support pandas 2 > > > Key: SPARK-44101 > URL: https://issues.apache.org/jira/browse/SPARK-44101 > Project: Spark > Issue Type: Umbrella > Components: Pandas API on Spark, PySpark >Affects Versions: 4.0.0 >Reporter: Haejoon Lee >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45338: -- Assignee: (was: Apache Spark) > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Priority: Major > Labels: pull-request-available > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45338: -- Assignee: Apache Spark > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Assignee: Apache Spark >Priority: Major > Labels: pull-request-available > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45338: -- Assignee: Apache Spark > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Assignee: Apache Spark >Priority: Major > Labels: pull-request-available > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45338: -- Assignee: (was: Apache Spark) > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Priority: Major > Labels: pull-request-available > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45338: -- Assignee: Apache Spark > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Assignee: Apache Spark >Priority: Major > Labels: pull-request-available > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45338: -- Assignee: (was: Apache Spark) > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Priority: Major > Labels: pull-request-available > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45351) Change RocksDB as default shuffle service db backend
[ https://issues.apache.org/jira/browse/SPARK-45351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45351: -- Assignee: (was: Apache Spark) > Change RocksDB as default shuffle service db backend > > > Key: SPARK-45351 > URL: https://issues.apache.org/jira/browse/SPARK-45351 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Jia Fan >Priority: Major > Labels: pull-request-available > > Change RocksDB as default shuffle service db backend, because we will remove > leveldb in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45351) Change RocksDB as default shuffle service db backend
[ https://issues.apache.org/jira/browse/SPARK-45351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45351: -- Assignee: Apache Spark > Change RocksDB as default shuffle service db backend > > > Key: SPARK-45351 > URL: https://issues.apache.org/jira/browse/SPARK-45351 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Jia Fan >Assignee: Apache Spark >Priority: Major > Labels: pull-request-available > > Change RocksDB as default shuffle service db backend, because we will remove > leveldb in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45352) Remove window partition if partition expression are foldable
[
https://issues.apache.org/jira/browse/SPARK-45352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuml updated SPARK-45352:
--
Priority: Major (was: Minor)
> Remove window partition if partition expression are foldable
>
>
> Key: SPARK-45352
> URL: https://issues.apache.org/jira/browse/SPARK-45352
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: zhuml
>Priority: Major
> Labels: pull-request-available
>
> Foldable partition is redundant, remove it not only can simplify plan, but
> some rules can also take effect when the partitions are all foldable, such as
> `LimitPushDownThroughWindow{{{}`{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45352) Remove window partition if partition expression are foldable
[
https://issues.apache.org/jira/browse/SPARK-45352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuml updated SPARK-45352:
--
Description: Foldable partition is redundant, remove it not only can
simplify plan, but some rules can also take effect when the partitions are all
foldable, such as `LimitPushDownThroughWindow{{{}`{}}}. (was: Foldable
partition is redundant, remove it not only can simplify plan, but some rules
can also take effect when the partitions are all foldable, such as
`{{{}InferWindowGroupLimit`{}}}.)
> Remove window partition if partition expression are foldable
>
>
> Key: SPARK-45352
> URL: https://issues.apache.org/jira/browse/SPARK-45352
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: zhuml
>Priority: Minor
> Labels: pull-request-available
>
> Foldable partition is redundant, remove it not only can simplify plan, but
> some rules can also take effect when the partitions are all foldable, such as
> `LimitPushDownThroughWindow{{{}`{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45341) Make the sbt doc command execute successfully with Java 17
[
https://issues.apache.org/jira/browse/SPARK-45341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot reassigned SPARK-45341:
--
Assignee: Apache Spark
> Make the sbt doc command execute successfully with Java 17
> --
>
> Key: SPARK-45341
> URL: https://issues.apache.org/jira/browse/SPARK-45341
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Apache Spark
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/Picked up
> JAVA_TOOL_OPTIONS:-Duser.language=en
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/ArrayWrappers.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVIndex.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/UnsupportedStoreVersionException.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreView.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreSerializer.java...
> [error] Constructing Javadoc information...
> [error] Building index for all the packages and classes...
> [error] Standard Doclet version 17.0.8+7-LTS
> [error] Building tree for all the packages and classes...
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java:32:1:
> error: heading used out of sequence: , compared to implicit preceding
> heading:
> [error] * Serialization
> [error] ^Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/InMemoryStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVIndex.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreIterator.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreSerializer.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreView.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVTypeInfo.html...
> [error] Generating
>
[jira] [Assigned] (SPARK-45341) Make the sbt doc command execute successfully with Java 17
[
https://issues.apache.org/jira/browse/SPARK-45341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot reassigned SPARK-45341:
--
Assignee: (was: Apache Spark)
> Make the sbt doc command execute successfully with Java 17
> --
>
> Key: SPARK-45341
> URL: https://issues.apache.org/jira/browse/SPARK-45341
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/Picked up
> JAVA_TOOL_OPTIONS:-Duser.language=en
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/ArrayWrappers.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVIndex.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/UnsupportedStoreVersionException.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreView.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreSerializer.java...
> [error] Constructing Javadoc information...
> [error] Building index for all the packages and classes...
> [error] Standard Doclet version 17.0.8+7-LTS
> [error] Building tree for all the packages and classes...
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java:32:1:
> error: heading used out of sequence: , compared to implicit preceding
> heading:
> [error] * Serialization
> [error] ^Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/InMemoryStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVIndex.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreIterator.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreSerializer.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreView.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVTypeInfo.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/LevelDB.html...
> [error]
[jira] [Assigned] (SPARK-45352) Remove window partition if partition expression are foldable
[
https://issues.apache.org/jira/browse/SPARK-45352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot reassigned SPARK-45352:
--
Assignee: Apache Spark
> Remove window partition if partition expression are foldable
>
>
> Key: SPARK-45352
> URL: https://issues.apache.org/jira/browse/SPARK-45352
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: zhuml
>Assignee: Apache Spark
>Priority: Minor
> Labels: pull-request-available
>
> Foldable partition is redundant, remove it not only can simplify plan, but
> some rules can also take effect when the partitions are all foldable, such as
> `{{{}InferWindowGroupLimit`{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45352) Remove window partition if partition expression are foldable
[
https://issues.apache.org/jira/browse/SPARK-45352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot reassigned SPARK-45352:
--
Assignee: (was: Apache Spark)
> Remove window partition if partition expression are foldable
>
>
> Key: SPARK-45352
> URL: https://issues.apache.org/jira/browse/SPARK-45352
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: zhuml
>Priority: Minor
> Labels: pull-request-available
>
> Foldable partition is redundant, remove it not only can simplify plan, but
> some rules can also take effect when the partitions are all foldable, such as
> `{{{}InferWindowGroupLimit`{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45268) python function categories should be consistent with SQL function groups
[ https://issues.apache.org/jira/browse/SPARK-45268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-45268: - Assignee: Ruifeng Zheng > python function categories should be consistent with SQL function groups > > > Key: SPARK-45268 > URL: https://issues.apache.org/jira/browse/SPARK-45268 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45268) python function categories should be consistent with SQL function groups
[ https://issues.apache.org/jira/browse/SPARK-45268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-45268. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43045 [https://github.com/apache/spark/pull/43045] > python function categories should be consistent with SQL function groups > > > Key: SPARK-45268 > URL: https://issues.apache.org/jira/browse/SPARK-45268 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45352) Remove window partition if partition expression are foldable
[
https://issues.apache.org/jira/browse/SPARK-45352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuml updated SPARK-45352:
--
Description: Foldable partition is redundant, remove it not only can
simplify plan, but some rules can also take effect when the partitions are all
foldable, such as {{{}InferWindowGroupLimit{}}}.
> Remove window partition if partition expression are foldable
>
>
> Key: SPARK-45352
> URL: https://issues.apache.org/jira/browse/SPARK-45352
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: zhuml
>Priority: Minor
> Labels: pull-request-available
>
> Foldable partition is redundant, remove it not only can simplify plan, but
> some rules can also take effect when the partitions are all foldable, such as
> {{{}InferWindowGroupLimit{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45352) Remove window partition if partition expression are foldable
[
https://issues.apache.org/jira/browse/SPARK-45352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuml updated SPARK-45352:
--
Description: Foldable partition is redundant, remove it not only can
simplify plan, but some rules can also take effect when the partitions are all
foldable, such as `{{{}InferWindowGroupLimit`{}}}. (was: Foldable partition is
redundant, remove it not only can simplify plan, but some rules can also take
effect when the partitions are all foldable, such as
{{{}InferWindowGroupLimit{}}}.)
> Remove window partition if partition expression are foldable
>
>
> Key: SPARK-45352
> URL: https://issues.apache.org/jira/browse/SPARK-45352
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: zhuml
>Priority: Minor
> Labels: pull-request-available
>
> Foldable partition is redundant, remove it not only can simplify plan, but
> some rules can also take effect when the partitions are all foldable, such as
> `{{{}InferWindowGroupLimit`{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45352) Remove window partition if partition expression are foldable
[ https://issues.apache.org/jira/browse/SPARK-45352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45352: --- Labels: pull-request-available (was: ) > Remove window partition if partition expression are foldable > > > Key: SPARK-45352 > URL: https://issues.apache.org/jira/browse/SPARK-45352 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.5.0 >Reporter: zhuml >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45352) Remove window partition if partition expression are foldable
[ https://issues.apache.org/jira/browse/SPARK-45352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuml updated SPARK-45352: -- Priority: Minor (was: Major) > Remove window partition if partition expression are foldable > > > Key: SPARK-45352 > URL: https://issues.apache.org/jira/browse/SPARK-45352 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.5.0 >Reporter: zhuml >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45352) Remove window partition if partition expression are foldable
zhuml created SPARK-45352: - Summary: Remove window partition if partition expression are foldable Key: SPARK-45352 URL: https://issues.apache.org/jira/browse/SPARK-45352 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: zhuml -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45346) Parquet schema inference should respect case sensitive flag when merging schema
[ https://issues.apache.org/jira/browse/SPARK-45346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45346. - Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull request 43134 [https://github.com/apache/spark/pull/43134] > Parquet schema inference should respect case sensitive flag when merging > schema > --- > > Key: SPARK-45346 > URL: https://issues.apache.org/jira/browse/SPARK-45346 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.4.0, 3.5.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.5.1, 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45346) Parquet schema inference should respect case sensitive flag when merging schema
[ https://issues.apache.org/jira/browse/SPARK-45346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45346: --- Assignee: Wenchen Fan > Parquet schema inference should respect case sensitive flag when merging > schema > --- > > Key: SPARK-45346 > URL: https://issues.apache.org/jira/browse/SPARK-45346 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.4.0, 3.5.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45350) Rename the imported Java Boolean to JBoolean
[ https://issues.apache.org/jira/browse/SPARK-45350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45350: --- Labels: pull-request-available (was: ) > Rename the imported Java Boolean to JBoolean > > > Key: SPARK-45350 > URL: https://issues.apache.org/jira/browse/SPARK-45350 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Minor > Labels: pull-request-available > > Some places have used `import java.lang.Boolean` for the import of Java > Boolean type, which can easily cause ambiguity, it should be renamed to > JBoolean. > > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-43662) Enable ReshapeParityTests.test_merge_asof
[ https://issues.apache.org/jira/browse/SPARK-43662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43662. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43137 [https://github.com/apache/spark/pull/43137] > Enable ReshapeParityTests.test_merge_asof > - > > Key: SPARK-43662 > URL: https://issues.apache.org/jira/browse/SPARK-43662 > Project: Spark > Issue Type: Sub-task > Components: Connect, Pandas API on Spark >Affects Versions: 3.5.0 >Reporter: Haejoon Lee >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Enable ReshapeParityTests.test_merge_asof -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-43662) Enable ReshapeParityTests.test_merge_asof
[ https://issues.apache.org/jira/browse/SPARK-43662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43662: - Assignee: Takuya Ueshin > Enable ReshapeParityTests.test_merge_asof > - > > Key: SPARK-43662 > URL: https://issues.apache.org/jira/browse/SPARK-43662 > Project: Spark > Issue Type: Sub-task > Components: Connect, Pandas API on Spark >Affects Versions: 3.5.0 >Reporter: Haejoon Lee >Assignee: Takuya Ueshin >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Enable ReshapeParityTests.test_merge_asof -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45308) Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0.
[ https://issues.apache.org/jira/browse/SPARK-45308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-45308. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43096 [https://github.com/apache/spark/pull/43096] > Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas > 2.0.0. > > > Key: SPARK-45308 > URL: https://issues.apache.org/jira/browse/SPARK-45308 > Project: Spark > Issue Type: Sub-task > Components: Pandas API on Spark, Tests >Affects Versions: 4.0.0 >Reporter: Haejoon Lee >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45308) Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0.
[ https://issues.apache.org/jira/browse/SPARK-45308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-45308: - Assignee: Haejoon Lee > Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas > 2.0.0. > > > Key: SPARK-45308 > URL: https://issues.apache.org/jira/browse/SPARK-45308 > Project: Spark > Issue Type: Sub-task > Components: Pandas API on Spark, Tests >Affects Versions: 4.0.0 >Reporter: Haejoon Lee >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-45057) Deadlock caused by rdd replication level of 2
[ https://issues.apache.org/jira/browse/SPARK-45057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769443#comment-17769443 ] wuyi commented on SPARK-45057: -- In the case of "Received UploadBlock request from T1 (blocked by T4)", shouldn't it be blocked by T3? > Deadlock caused by rdd replication level of 2 > - > > Key: SPARK-45057 > URL: https://issues.apache.org/jira/browse/SPARK-45057 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.4.1 >Reporter: Zhongwei Zhu >Priority: Major > Labels: pull-request-available > > > When 2 tasks try to compute same rdd with replication level of 2 and running > on only 2 executors. Deadlock will happen. > Task only release lock after writing into local machine and replicate to > remote executor. > > ||Time||Exe 1 (Task Thread T1)||Exe 1 (Shuffle Server Thread T2)||Exe 2 (Task > Thread T3)||Exe 2 (Shuffle Server Thread T4)|| > |T0|write lock of rdd| | | | > |T1| | |write lock of rdd| | > |T2|replicate -> UploadBlockSync (blocked by T4)| | | | > |T3| | | |Received UploadBlock request from T1 (blocked by T4)| > |T4| | |replicate -> UploadBlockSync (blocked by T2)| | > |T5| |Received UploadBlock request from T3 (blocked by T1)| | | > |T6|Deadlock|Deadlock|Deadlock|Deadlock| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45267) Change the default value for `numeric_only`.
[ https://issues.apache.org/jira/browse/SPARK-45267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-45267: - Assignee: Haejoon Lee > Change the default value for `numeric_only`. > > > Key: SPARK-45267 > URL: https://issues.apache.org/jira/browse/SPARK-45267 > Project: Spark > Issue Type: Sub-task > Components: Pandas API on Spark >Affects Versions: 4.0.0 >Reporter: Haejoon Lee >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > > To follow the Pandas 2.0.0 and above. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
