[jira] [Deleted] (SPARK-30709) Spark 2.3 to Spark 2.4 Upgrade. Problems reading HIVE partitioned tables.
[ https://issues.apache.org/jira/browse/SPARK-30709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen deleted SPARK-30709: - > Spark 2.3 to Spark 2.4 Upgrade. Problems reading HIVE partitioned tables. > - > > Key: SPARK-30709 > URL: https://issues.apache.org/jira/browse/SPARK-30709 > Project: Spark > Issue Type: Question > Environment: PRE- Production >Reporter: Carlos Mario >Priority: Major > Labels: SQL, Spark > > Hello > We recently updated our preproduction environment from Spark 2.3 to Spark > 2.4.0 > Along time we have created a big amount of tables in Hive Metastore, > partitioned by 2 fields one of them String and the other one BigInt. > We were reading this tables with Spark 2.3 with no problem, but after > upgrading to Spark 2.4 we get the following log every time we run our SW: > > log_filterBIGINT.out: > Caused by: MetaException(message:Filtering is supported only on partition > keys of type string) Caused by: MetaException(message:Filtering is supported > only on partition keys of type string) Caused by: > MetaException(message:Filtering is supported only on partition keys of type > string) > > hadoop-cmf-hive-HIVEMETASTORE-isblcsmsttc0001.scisb.isban.corp.log.out.1: > > 2020-01-10 09:36:05,781 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-5-thread-138]: > MetaException(message:Filtering is supported only on partition keys of type > string) > 2020-01-10 11:19:19,208 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-5-thread-187]: > MetaException(message:Filtering is supported only on partition keys of type > string) > 2020-01-10 11:19:54,780 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-5-thread-167]: > MetaException(message:Filtering is supported only on partition keys of type > string) > > > We know the best practice from Spark point of view is to use 'STRING' type > for partition columns, but we need to explore a solution we'll be able to > deploy with ease, due to the big amount of tables created with a bigiint type > column partition. > > As a first solution we tried to set the > spark.sql.hive.manageFilesourcePartitions parameter to false in the Spark > Submmit, but after reruning the SW the error stood still. > > Is there anyone in the community who experienced the same problem? What was > the solution for it? > > Kind Regards and thanks in advance. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47172) Upgrade Transport block cipher mode to GCM
[ https://issues.apache.org/jira/browse/SPARK-47172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-47172: - Shepherd: (was: Sean R. Owen) > Upgrade Transport block cipher mode to GCM > -- > > Key: SPARK-47172 > URL: https://issues.apache.org/jira/browse/SPARK-47172 > Project: Spark > Issue Type: Improvement > Components: Security >Affects Versions: 3.4.2, 3.5.0 >Reporter: Steve Weis >Priority: Minor > > The cipher transformation currently used for encrypting RPC calls is an > unauthenticated mode (AES/CTR/NoPadding). This needs to be upgraded to an > authenticated mode (AES/GCM/NoPadding) to prevent ciphertext from being > modified in transit. > The relevant line is here: > [https://github.com/apache/spark/blob/a939a7d0fd9c6b23c879cbee05275c6fbc939e38/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java#L220] > GCM is relatively more computationally expensive than CTR and adds a 16-byte > block of authentication tag data to each payload. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46760) Make the document of spark.sql.adaptive.coalescePartitions.parallelismFirst clearer
[ https://issues.apache.org/jira/browse/SPARK-46760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-46760. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44787 [https://github.com/apache/spark/pull/44787] > Make the document of spark.sql.adaptive.coalescePartitions.parallelismFirst > clearer > --- > > Key: SPARK-46760 > URL: https://issues.apache.org/jira/browse/SPARK-46760 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46760) Make the document of spark.sql.adaptive.coalescePartitions.parallelismFirst clearer
[ https://issues.apache.org/jira/browse/SPARK-46760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-46760: - Priority: Trivial (was: Major) > Make the document of spark.sql.adaptive.coalescePartitions.parallelismFirst > clearer > --- > > Key: SPARK-46760 > URL: https://issues.apache.org/jira/browse/SPARK-46760 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Trivial > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45110) Upgrade rocksdbjni to 8.8.1
[ https://issues.apache.org/jira/browse/SPARK-45110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45110: - Summary: Upgrade rocksdbjni to 8.8.1 (was: Upgrade rocksdbjni to 8.6.7) > Upgrade rocksdbjni to 8.8.1 > --- > > Key: SPARK-45110 > URL: https://issues.apache.org/jira/browse/SPARK-45110 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.5.0, 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45110) Upgrade rocksdbjni to 8.6.7
[ https://issues.apache.org/jira/browse/SPARK-45110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45110. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43924 [https://github.com/apache/spark/pull/43924] > Upgrade rocksdbjni to 8.6.7 > --- > > Key: SPARK-45110 > URL: https://issues.apache.org/jira/browse/SPARK-45110 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.5.0, 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45110) Upgrade rocksdbjni to 8.8.1
[ https://issues.apache.org/jira/browse/SPARK-45110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45110: - Issue Type: Task (was: Bug) > Upgrade rocksdbjni to 8.8.1 > --- > > Key: SPARK-45110 > URL: https://issues.apache.org/jira/browse/SPARK-45110 > Project: Spark > Issue Type: Task > Components: Build >Affects Versions: 3.5.0, 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46473) Reuse `getPartitionedFile` method
[ https://issues.apache.org/jira/browse/SPARK-46473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-46473. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44437 [https://github.com/apache/spark/pull/44437] > Reuse `getPartitionedFile` method > - > > Key: SPARK-46473 > URL: https://issues.apache.org/jira/browse/SPARK-46473 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.5.0 >Reporter: xiaoping.huang >Assignee: xiaoping.huang >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46473) Reuse `getPartitionedFile` method
[ https://issues.apache.org/jira/browse/SPARK-46473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-46473: Assignee: xiaoping.huang > Reuse `getPartitionedFile` method > - > > Key: SPARK-46473 > URL: https://issues.apache.org/jira/browse/SPARK-46473 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.5.0 >Reporter: xiaoping.huang >Assignee: xiaoping.huang >Priority: Trivial > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46929) Use ThreadUtils.shutdown to close thread pools
[ https://issues.apache.org/jira/browse/SPARK-46929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-46929. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44962 [https://github.com/apache/spark/pull/44962] > Use ThreadUtils.shutdown to close thread pools > -- > > Key: SPARK-46929 > URL: https://issues.apache.org/jira/browse/SPARK-46929 > Project: Spark > Issue Type: Improvement > Components: Connect, Spark Core, SS >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46929) Use ThreadUtils.shutdown to close thread pools
[ https://issues.apache.org/jira/browse/SPARK-46929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-46929: - Priority: Trivial (was: Major) > Use ThreadUtils.shutdown to close thread pools > -- > > Key: SPARK-46929 > URL: https://issues.apache.org/jira/browse/SPARK-46929 > Project: Spark > Issue Type: Improvement > Components: Connect, Spark Core, SS >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Trivial > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46400) When there are corrupted files in the local maven repo, retry to skip this cache
[ https://issues.apache.org/jira/browse/SPARK-46400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-46400. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44343 [https://github.com/apache/spark/pull/44343] > When there are corrupted files in the local maven repo, retry to skip this > cache > > > Key: SPARK-46400 > URL: https://issues.apache.org/jira/browse/SPARK-46400 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46400) When there are corrupted files in the local maven repo, retry to skip this cache
[ https://issues.apache.org/jira/browse/SPARK-46400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-46400: Assignee: BingKun Pan > When there are corrupted files in the local maven repo, retry to skip this > cache > > > Key: SPARK-46400 > URL: https://issues.apache.org/jira/browse/SPARK-46400 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45522) Migrate jetty 9 to jetty 12
[ https://issues.apache.org/jira/browse/SPARK-45522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45522: Assignee: HiuFung > Migrate jetty 9 to jetty 12 > --- > > Key: SPARK-45522 > URL: https://issues.apache.org/jira/browse/SPARK-45522 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: HiuFung >Priority: Minor > Labels: pull-request-available > > Jetty 12 supports JakartaEE 8/JakartaEE 9/JakartaEE 10 simultaneously. But > the version span is quite large, need to read the documentation in detail, > not sure if it can be completed within the 4.0 cycle, so it's set to low > priority. > > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45522) Migrate jetty 9 to jetty 12
[ https://issues.apache.org/jira/browse/SPARK-45522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45522. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43765 [https://github.com/apache/spark/pull/43765] > Migrate jetty 9 to jetty 12 > --- > > Key: SPARK-45522 > URL: https://issues.apache.org/jira/browse/SPARK-45522 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: HiuFung >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > > Jetty 12 supports JakartaEE 8/JakartaEE 9/JakartaEE 10 simultaneously. But > the version span is quite large, need to read the documentation in detail, > not sure if it can be completed within the 4.0 cycle, so it's set to low > priority. > > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46473) Reuse `getPartitionedFile` method
[ https://issues.apache.org/jira/browse/SPARK-46473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-46473: - Priority: Trivial (was: Minor) > Reuse `getPartitionedFile` method > - > > Key: SPARK-46473 > URL: https://issues.apache.org/jira/browse/SPARK-46473 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.5.0 >Reporter: xiaoping.huang >Priority: Trivial > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31794) Incorrect distribution with repartitionByRange and repartition column expression
[
https://issues.apache.org/jira/browse/SPARK-31794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810476#comment-17810476
]
Sean R. Owen commented on SPARK-31794:
--
Not that it helps, but I observe the same behavior, and it must be a bug. The
result is exactly as expected, except that the first partition is 2x the size
of the others, and the last partition is empty. I tried tweaks to identify the
problem (i.e. what if I increased the desired partition count by 1? added 1 to
my indices?) but didn't help.
I don't have any fix or further insight, just adding that yeah this seems to be
a problem
> Incorrect distribution with repartitionByRange and repartition column
> expression
>
>
> Key: SPARK-31794
> URL: https://issues.apache.org/jira/browse/SPARK-31794
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.2, 2.4.5, 3.0.1
> Environment: Sample code for obtaining the above test results.
> import java.io.File
> import java.io.PrintWriter
> val logfile="/tmp/sparkdftest.log"
> val writer = new PrintWriter(logfile)
> writer.println("Spark Version " + sc.version)
> val df= Range(1, 1002).toDF("val")
> writer.println("Default Partition Length:" + df.rdd.partitions.length)
> writer.println("Default Partition getNumPartitions:" +
> df.rdd.getNumPartitions)
> writer.println("Default Partition groupBy spark_partition_id:" +
> df.groupBy(spark_partition_id).count().rdd.partitions.length)
> val dfcount=df.mapPartitions\{part => Iterator(part.size)}
> writer.println("Default Partition:" + dfcount.collect().toList)
> val numparts=24
> val dfparts_range=df.withColumn("partid", $"val" %
> numparts).repartitionByRange(numparts, $"partid")
> writer.println("repartitionByRange Length:" +
> dfparts_range.rdd.partitions.length)
> writer.println("repartitionByRange getNumPartitions:" +
> dfparts_range.rdd.getNumPartitions)
> writer.println("repartitionByRange groupBy spark_partition_id:" +
> dfparts_range.groupBy(spark_partition_id).count().rdd.partitions.length)
> val dfpartscount=dfparts_range.mapPartitions\{part => Iterator(part.size)}
> writer.println("repartitionByRange: " + dfpartscount.collect().toList)
> val dfparts_expr=df.withColumn("partid", $"val" %
> numparts).repartition(numparts, $"partid")
> writer.println("repartition by column expr Length:" +
> dfparts_expr.rdd.partitions.length)
> writer.println("repartition by column expr getNumPartitions:" +
> dfparts_expr.rdd.getNumPartitions)
> writer.println("repartition by column expr groupBy spark_partitoin_id:" +
> dfparts_expr.groupBy(spark_partition_id).count().rdd.partitions.length)
> val dfpartscount=dfparts_expr.mapPartitions\{part => Iterator(part.size)}
> writer.println("repartition by column expr:" + dfpartscount.collect().toList)
> writer.close()
>Reporter: Ramesha Bhatta
>Priority: Major
> Labels: performance
>
> Both repartitionByRange and repartition(, ) resulting in wrong
> distribution within the resulting partition.
>
> In the Range partition one of the partition has 2x volume and last one with
> zero. In repartition this is more problematic with some partition with 4x,
> 2x the avg and many partitions with zero volume.
>
> This distribution imbalance can cause performance problem in a concurrent
> environment.
> Details from testing in 3 different versions.
> |Verion 2.3.2|Version 2.4.5|Versoin 3.0 Preview2|
> |Spark Version 2.3.2.3.1.4.0-315|Spark Version 2.4.5|Spark Version
> 3.0.0-preview2|
> |Default Partition Length:2|Default Partition Length:2|Default Partition
> Length:80|
> |Default Partition getNumPartitions:2|Default Partition
> getNumPartitions:2|Default Partition getNumPartitions:80|
> |Default Partition groupBy spark_partition_id:200|Default Partition groupBy
> spark_partition_id:200|Default Partition groupBy spark_partition_id:200|
> |repartitionByRange Length:24|repartitionByRange Length:24|repartitionByRange
> Length:24|
> |repartitionByRange getNumPartitions:24|repartitionByRange
> getNumPartitions:24|repartitionByRange getNumPartitions:24|
> |repartitionByRange groupBy spark_partition_id:200|repartitionByRange groupBy
> spark_partition_id:200|repartitionByRange groupBy spark_partition_id:200|
> |repartitionByRange: List(83, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42,
> 42, 42, 42, 42, 41, 41, 41, 41, 41, 41, 0)|repartitionByRange: List(83, 42,
> 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 41, 41, 41, 41,
> 41, 41, 0)|repartitionByRange: List(83, 42, 42, 42, 42, 42, 42, 42, 42, 42,
> 42, 42, 42, 42, 42, 42, 42, 41, 41, 41, 41, 41, 41, 0)|
> |repartition by column expr Length:24|repartition by column expr
> Length:24|repartition by column expr Length:24|
> |repartition by column
[jira] [Updated] (SPARK-46637) Enhancing the Visual Appeal of Spark doc website
[ https://issues.apache.org/jira/browse/SPARK-46637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-46637: - Priority: Minor (was: Major) > Enhancing the Visual Appeal of Spark doc website > > > Key: SPARK-46637 > URL: https://issues.apache.org/jira/browse/SPARK-46637 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 4.0.0, 3.5.1 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46124) Replace explicit `ArrayOps#toSeq` with `s.c.immutable.ArraySeq.unsafeWrapArray`
[ https://issues.apache.org/jira/browse/SPARK-46124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-46124. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44041 [https://github.com/apache/spark/pull/44041] > Replace explicit `ArrayOps#toSeq` with > `s.c.immutable.ArraySeq.unsafeWrapArray` > > > Key: SPARK-46124 > URL: https://issues.apache.org/jira/browse/SPARK-46124 > Project: Spark > Issue Type: Sub-task > Components: DStreams, Kubernetes, ML, MLlib, Spark Core, SQL, > Structured Streaming, YARN >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > There is a behavioral difference between Scala 2.13 and 2.12 for explicit > `ArrayOps.toSeq` calls, similar to the implicit conversion from `Array` to > `Seq`. > In Scala 2.12, it returns a `mutable.WrappedArray`, which does not involve a > collection copy. > ```scala > Welcome to Scala 2.12.18 (OpenJDK 64-Bit Server VM, Java 17.0.9). > Type in expressions for evaluation. Or try :help. > scala> Array(1,2,3).toSeq > res0: Seq[Int] = WrappedArray(1, 2, 3) > ``` > However, in Scala 2.13, it returns an `immutable.ArraySeq` that with > collection copy. > Since we have always used the non-collection copy behavior for this explicit > conversion in the era of Scala 2.12, it is safe to assume that no collection > copy is needed for Scala 2.13. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46098) Reduce stack depth by replace (string|array).size with (string|array).length
[ https://issues.apache.org/jira/browse/SPARK-46098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-46098: - Priority: Minor (was: Major) > Reduce stack depth by replace (string|array).size with (string|array).length > > > Key: SPARK-46098 > URL: https://issues.apache.org/jira/browse/SPARK-46098 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Minor > Fix For: 4.0.0 > > > There are a lot of (string|array).size called. > In fact, the size calls the underlying length, this behavior increase the > stack length. > We should call (string|array).length directly. > We also get the compile waring Replace .size with .length on arrays and > strings -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46098) Reduce stack depth by replace (string|array).size with (string|array).length
[ https://issues.apache.org/jira/browse/SPARK-46098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-46098. -- Fix Version/s: 4.0.0 Resolution: Fixed > Reduce stack depth by replace (string|array).size with (string|array).length > > > Key: SPARK-46098 > URL: https://issues.apache.org/jira/browse/SPARK-46098 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Major > Fix For: 4.0.0 > > > There are a lot of (string|array).size called. > In fact, the size calls the underlying length, this behavior increase the > stack length. > We should call (string|array).length directly. > We also get the compile waring Replace .size with .length on arrays and > strings -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46101) Replace (string|array).size with (string|array).length in all the modules
[ https://issues.apache.org/jira/browse/SPARK-46101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-46101. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44016 [https://github.com/apache/spark/pull/44016] > Replace (string|array).size with (string|array).length in all the modules > - > > Key: SPARK-46101 > URL: https://issues.apache.org/jira/browse/SPARK-46101 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46100) Replace (string|array).size with (string|array).length in module core
[ https://issues.apache.org/jira/browse/SPARK-46100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-46100. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44011 [https://github.com/apache/spark/pull/44011] > Replace (string|array).size with (string|array).length in module core > - > > Key: SPARK-46100 > URL: https://issues.apache.org/jira/browse/SPARK-46100 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46101) Replace (string|array).size with (string|array).length in module SQL
[ https://issues.apache.org/jira/browse/SPARK-46101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-46101: - Priority: Minor (was: Major) Summary: Replace (string|array).size with (string|array).length in module SQL (was: Fix these issue in module sql) > Replace (string|array).size with (string|array).length in module SQL > > > Key: SPARK-46101 > URL: https://issues.apache.org/jira/browse/SPARK-46101 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46100) Replace (string|array).size with (string|array).length in module core
[ https://issues.apache.org/jira/browse/SPARK-46100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-46100: - Summary: Replace (string|array).size with (string|array).length in module core (was: Fix these issue in module core) > Replace (string|array).size with (string|array).length in module core > - > > Key: SPARK-46100 > URL: https://issues.apache.org/jira/browse/SPARK-46100 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46100) Replace (string|array).size with (string|array).length in module core
[ https://issues.apache.org/jira/browse/SPARK-46100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-46100: - Priority: Minor (was: Major) > Replace (string|array).size with (string|array).length in module core > - > > Key: SPARK-46100 > URL: https://issues.apache.org/jira/browse/SPARK-46100 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Jiaan Geng >Assignee: Jiaan Geng >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45686) Fix `method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is deprecated`
[
https://issues.apache.org/jira/browse/SPARK-45686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-45686:
Assignee: Yang Jie
> Fix `method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is
> deprecated`
>
>
> Key: SPARK-45686
> URL: https://issues.apache.org/jira/browse/SPARK-45686
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:57:31:
> method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is
> deprecated (since 2.13.0): implicit conversions from Array to
> immutable.IndexedSeq are implemented by copying; use `toIndexedSeq`
> explicitly if you want to copy, or use the more efficient non-copying
> ArraySeq.unsafeWrapArray
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=deprecation,
> site=org.apache.spark.ml.linalg.Vector.equals,
> origin=scala.LowPriorityImplicits2.copyArrayToImmutableIndexedSeq,
> version=2.13.0
> [error] Vectors.equals(s1.indices, s1.values, s2.indices,
> s2.values)
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:57:54:
> method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is
> deprecated (since 2.13.0): implicit conversions from Array to
> immutable.IndexedSeq are implemented by copying; use `toIndexedSeq`
> explicitly if you want to copy, or use the more efficient non-copying
> ArraySeq.unsafeWrapArray
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=deprecation,
> site=org.apache.spark.ml.linalg.Vector.equals,
> origin=scala.LowPriorityImplicits2.copyArrayToImmutableIndexedSeq,
> version=2.13.0
> [error] Vectors.equals(s1.indices, s1.values, s2.indices,
> s2.values)
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:59:31:
> method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is
> deprecated (since 2.13.0): implicit conversions from Array to
> immutable.IndexedSeq are implemented by copying; use `toIndexedSeq`
> explicitly if you want to copy, or use the more efficient non-copying
> ArraySeq.unsafeWrapArray
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=deprecation,
> site=org.apache.spark.ml.linalg.Vector.equals,
> origin=scala.LowPriorityImplicits2.copyArrayToImmutableIndexedSeq,
> version=2.13.0
> [error] Vectors.equals(s1.indices, s1.values, 0 until d1.size,
> d1.values)
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:61:59:
> method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is
> deprecated (since 2.13.0): implicit conversions from Array to
> immutable.IndexedSeq are implemented by copying; use `toIndexedSeq`
> explicitly if you want to copy, or use the more efficient non-copying
> ArraySeq.unsafeWrapArray
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=deprecation,
> site=org.apache.spark.ml.linalg.Vector.equals,
> origin=scala.LowPriorityImplicits2.copyArrayToImmutableIndexedSeq,
> version=2.13.0
> [error] Vectors.equals(0 until d1.size, d1.values, s1.indices,
> s1.values)
> [error] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45686) Fix `method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is deprecated`
[
https://issues.apache.org/jira/browse/SPARK-45686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-45686.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43670
[https://github.com/apache/spark/pull/43670]
> Fix `method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is
> deprecated`
>
>
> Key: SPARK-45686
> URL: https://issues.apache.org/jira/browse/SPARK-45686
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:57:31:
> method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is
> deprecated (since 2.13.0): implicit conversions from Array to
> immutable.IndexedSeq are implemented by copying; use `toIndexedSeq`
> explicitly if you want to copy, or use the more efficient non-copying
> ArraySeq.unsafeWrapArray
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=deprecation,
> site=org.apache.spark.ml.linalg.Vector.equals,
> origin=scala.LowPriorityImplicits2.copyArrayToImmutableIndexedSeq,
> version=2.13.0
> [error] Vectors.equals(s1.indices, s1.values, s2.indices,
> s2.values)
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:57:54:
> method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is
> deprecated (since 2.13.0): implicit conversions from Array to
> immutable.IndexedSeq are implemented by copying; use `toIndexedSeq`
> explicitly if you want to copy, or use the more efficient non-copying
> ArraySeq.unsafeWrapArray
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=deprecation,
> site=org.apache.spark.ml.linalg.Vector.equals,
> origin=scala.LowPriorityImplicits2.copyArrayToImmutableIndexedSeq,
> version=2.13.0
> [error] Vectors.equals(s1.indices, s1.values, s2.indices,
> s2.values)
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:59:31:
> method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is
> deprecated (since 2.13.0): implicit conversions from Array to
> immutable.IndexedSeq are implemented by copying; use `toIndexedSeq`
> explicitly if you want to copy, or use the more efficient non-copying
> ArraySeq.unsafeWrapArray
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=deprecation,
> site=org.apache.spark.ml.linalg.Vector.equals,
> origin=scala.LowPriorityImplicits2.copyArrayToImmutableIndexedSeq,
> version=2.13.0
> [error] Vectors.equals(s1.indices, s1.values, 0 until d1.size,
> d1.values)
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:61:59:
> method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is
> deprecated (since 2.13.0): implicit conversions from Array to
> immutable.IndexedSeq are implemented by copying; use `toIndexedSeq`
> explicitly if you want to copy, or use the more efficient non-copying
> ArraySeq.unsafeWrapArray
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=deprecation,
> site=org.apache.spark.ml.linalg.Vector.equals,
> origin=scala.LowPriorityImplicits2.copyArrayToImmutableIndexedSeq,
> version=2.13.0
> [error] Vectors.equals(0 until d1.size, d1.values, s1.indices,
> s1.values)
> [error] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45687) Fix `Passing an explicit array value to a Scala varargs method is deprecated`
[
https://issues.apache.org/jira/browse/SPARK-45687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-45687.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43642
[https://github.com/apache/spark/pull/43642]
> Fix `Passing an explicit array value to a Scala varargs method is deprecated`
> -
>
> Key: SPARK-45687
> URL: https://issues.apache.org/jira/browse/SPARK-45687
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Tengfei Huang
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Passing an explicit array value to a Scala varargs method is deprecated
> (since 2.13.0) and will result in a defensive copy; Use the more efficient
> non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
>
> {code:java}
> [warn]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala:945:21:
> Passing an explicit array value to a Scala varargs method is deprecated
> (since 2.13.0) and will result in a defensive copy; Use the more efficient
> non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
> [warn] Applicable -Wconf / @nowarn filters for this warning: msg= message>, cat=deprecation,
> site=org.apache.spark.sql.hive.execution.AggregationQuerySuite, version=2.13.0
> [warn] df.agg(udaf(allColumns: _*)),
> [warn] ^
> [warn]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ObjectHashAggregateSuite.scala:156:48:
> Passing an explicit array value to a Scala varargs method is deprecated
> (since 2.13.0) and will result in a defensive copy; Use the more efficient
> non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
> [warn] Applicable -Wconf / @nowarn filters for this warning: msg= message>, cat=deprecation,
> site=org.apache.spark.sql.hive.execution.ObjectHashAggregateSuite,
> version=2.13.0
> [warn] df.agg(aggFunctions.head, aggFunctions.tail: _*),
> [warn] ^
> [warn]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ObjectHashAggregateSuite.scala:161:76:
> Passing an explicit array value to a Scala varargs method is deprecated
> (since 2.13.0) and will result in a defensive copy; Use the more efficient
> non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
> [warn] Applicable -Wconf / @nowarn filters for this warning: msg= message>, cat=deprecation,
> site=org.apache.spark.sql.hive.execution.ObjectHashAggregateSuite,
> version=2.13.0
> [warn] df.groupBy($"id" % 4 as "mod").agg(aggFunctions.head,
> aggFunctions.tail: _*),
> [warn]
> ^
> [warn]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ObjectHashAggregateSuite.scala:171:50:
> Passing an explicit array value to a Scala varargs method is deprecated
> (since 2.13.0) and will result in a defensive copy; Use the more efficient
> non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
> [warn] Applicable -Wconf / @nowarn filters for this warning: msg= message>, cat=deprecation,
> site=org.apache.spark.sql.hive.execution.ObjectHashAggregateSuite,
> version=2.13.0
> [warn] df.agg(aggFunctions.head, aggFunctions.tail: _*),
> [warn] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45687) Fix `Passing an explicit array value to a Scala varargs method is deprecated`
[
https://issues.apache.org/jira/browse/SPARK-45687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-45687:
Assignee: Tengfei Huang
> Fix `Passing an explicit array value to a Scala varargs method is deprecated`
> -
>
> Key: SPARK-45687
> URL: https://issues.apache.org/jira/browse/SPARK-45687
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Tengfei Huang
>Priority: Major
> Labels: pull-request-available
>
> Passing an explicit array value to a Scala varargs method is deprecated
> (since 2.13.0) and will result in a defensive copy; Use the more efficient
> non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
>
> {code:java}
> [warn]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala:945:21:
> Passing an explicit array value to a Scala varargs method is deprecated
> (since 2.13.0) and will result in a defensive copy; Use the more efficient
> non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
> [warn] Applicable -Wconf / @nowarn filters for this warning: msg= message>, cat=deprecation,
> site=org.apache.spark.sql.hive.execution.AggregationQuerySuite, version=2.13.0
> [warn] df.agg(udaf(allColumns: _*)),
> [warn] ^
> [warn]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ObjectHashAggregateSuite.scala:156:48:
> Passing an explicit array value to a Scala varargs method is deprecated
> (since 2.13.0) and will result in a defensive copy; Use the more efficient
> non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
> [warn] Applicable -Wconf / @nowarn filters for this warning: msg= message>, cat=deprecation,
> site=org.apache.spark.sql.hive.execution.ObjectHashAggregateSuite,
> version=2.13.0
> [warn] df.agg(aggFunctions.head, aggFunctions.tail: _*),
> [warn] ^
> [warn]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ObjectHashAggregateSuite.scala:161:76:
> Passing an explicit array value to a Scala varargs method is deprecated
> (since 2.13.0) and will result in a defensive copy; Use the more efficient
> non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
> [warn] Applicable -Wconf / @nowarn filters for this warning: msg= message>, cat=deprecation,
> site=org.apache.spark.sql.hive.execution.ObjectHashAggregateSuite,
> version=2.13.0
> [warn] df.groupBy($"id" % 4 as "mod").agg(aggFunctions.head,
> aggFunctions.tail: _*),
> [warn]
> ^
> [warn]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ObjectHashAggregateSuite.scala:171:50:
> Passing an explicit array value to a Scala varargs method is deprecated
> (since 2.13.0) and will result in a defensive copy; Use the more efficient
> non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
> [warn] Applicable -Wconf / @nowarn filters for this warning: msg= message>, cat=deprecation,
> site=org.apache.spark.sql.hive.execution.ObjectHashAggregateSuite,
> version=2.13.0
> [warn] df.agg(aggFunctions.head, aggFunctions.tail: _*),
> [warn] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-42821) Remove unused parameters in splitFiles methods
[ https://issues.apache.org/jira/browse/SPARK-42821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-42821. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 40454 [https://github.com/apache/spark/pull/40454] > Remove unused parameters in splitFiles methods > -- > > Key: SPARK-42821 > URL: https://issues.apache.org/jira/browse/SPARK-42821 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42821) Remove unused parameters in splitFiles methods
[ https://issues.apache.org/jira/browse/SPARK-42821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-42821: Assignee: BingKun Pan > Remove unused parameters in splitFiles methods > -- > > Key: SPARK-42821 > URL: https://issues.apache.org/jira/browse/SPARK-42821 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45368) Remove scala2.12 compatibility logic for DoubleType, FloatType, Decimal
[ https://issues.apache.org/jira/browse/SPARK-45368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45368. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43456 [https://github.com/apache/spark/pull/43456] > Remove scala2.12 compatibility logic for DoubleType, FloatType, Decimal > --- > > Key: SPARK-45368 > URL: https://issues.apache.org/jira/browse/SPARK-45368 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: tangjiafu >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45368) Remove scala2.12 compatibility logic for DoubleType, FloatType, Decimal
[ https://issues.apache.org/jira/browse/SPARK-45368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45368: Assignee: tangjiafu > Remove scala2.12 compatibility logic for DoubleType, FloatType, Decimal > --- > > Key: SPARK-45368 > URL: https://issues.apache.org/jira/browse/SPARK-45368 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: tangjiafu >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45605) Replace `s.c.MapOps.mapValues` with `s.c.MapOps.view.mapValues`
[
https://issues.apache.org/jira/browse/SPARK-45605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-45605:
-
Priority: Minor (was: Major)
>Replace `s.c.MapOps.mapValues` with `s.c.MapOps.view.mapValues`
> --
>
> Key: SPARK-45605
> URL: https://issues.apache.org/jira/browse/SPARK-45605
> Project: Spark
> Issue Type: Sub-task
> Components: Connect, DStreams, Examples, MLlib, Spark Core, SS
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> @deprecated("Use .view.mapValues(f). A future version will include a strict
> version of this method (for now, .view.mapValues(f).toMap).", "2.13.0")
> def mapValues[W](f: V => W): MapView[K, W] = new MapView.MapValues(this, f)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45605) Replace `s.c.MapOps.mapValues` with `s.c.MapOps.view.mapValues`
[
https://issues.apache.org/jira/browse/SPARK-45605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-45605.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43448
[https://github.com/apache/spark/pull/43448]
>Replace `s.c.MapOps.mapValues` with `s.c.MapOps.view.mapValues`
> --
>
> Key: SPARK-45605
> URL: https://issues.apache.org/jira/browse/SPARK-45605
> Project: Spark
> Issue Type: Sub-task
> Components: Connect, DStreams, Examples, MLlib, Spark Core, SS
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> @deprecated("Use .view.mapValues(f). A future version will include a strict
> version of this method (for now, .view.mapValues(f).toMap).", "2.13.0")
> def mapValues[W](f: V => W): MapView[K, W] = new MapView.MapValues(this, f)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45636) Upgrade jersey to 2.41
[ https://issues.apache.org/jira/browse/SPARK-45636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45636. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43490 [https://github.com/apache/spark/pull/43490] > Upgrade jersey to 2.41 > -- > > Key: SPARK-45636 > URL: https://issues.apache.org/jira/browse/SPARK-45636 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > https://github.com/eclipse-ee4j/jersey/releases/tag/2.41 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45636) Upgrade jersey to 2.41
[ https://issues.apache.org/jira/browse/SPARK-45636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45636: Assignee: Yang Jie > Upgrade jersey to 2.41 > -- > > Key: SPARK-45636 > URL: https://issues.apache.org/jira/browse/SPARK-45636 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > > https://github.com/eclipse-ee4j/jersey/releases/tag/2.41 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45636) Upgrade jersey to 2.41
[ https://issues.apache.org/jira/browse/SPARK-45636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45636: - Priority: Minor (was: Major) > Upgrade jersey to 2.41 > -- > > Key: SPARK-45636 > URL: https://issues.apache.org/jira/browse/SPARK-45636 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > > https://github.com/eclipse-ee4j/jersey/releases/tag/2.41 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-40154) PySpark: DataFrame.cache docstring gives wrong storage level
[
https://issues.apache.org/jira/browse/SPARK-40154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-40154:
-
Priority: Trivial (was: Minor)
> PySpark: DataFrame.cache docstring gives wrong storage level
>
>
> Key: SPARK-40154
> URL: https://issues.apache.org/jira/browse/SPARK-40154
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Paul Staab
>Assignee: Paul Staab
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.4.2, 4.0.0, 3.5.1
>
>
> The docstring of the `DataFrame.cache()` method currently states that it uses
> a serialized storage level
> {code:java}
> Persists the :class:`DataFrame` with the default storage level
> (`MEMORY_AND_DISK`).
> [...]
> -The default storage level has changed to `MEMORY_AND_DISK` to match
> Scala in 2.0.{code}
> while `DataFrame.persist()` states that it uses a deserialized storage level
> {code:java}
> If no storage level is specified defaults to (`MEMORY_AND_DISK_DESER`)
> [...]
> The default storage level has changed to `MEMORY_AND_DISK_DESER` to match
> Scala in 3.0.{code}
>
> However, in practice both `.cache()` and `.persist()` use deserialized
> storage levels:
> {code:java}
> import pyspark
> from pyspark.sql import SparkSession
> from pyspark import StorageLevel
> print(pyspark.__version__)
> # 3.3.0
> spark = SparkSession.builder.master("local[2]").getOrCreate()
> df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a",
> "col_b"])
> df = df.cache()
> df.count()
> # Storage level in Spark UI: "Disk Memory Deserialized 1x Replicated"
> df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a",
> "col_b"])
> df = df.persist()
> df.count()
> # Storage level in Spark UI: "Disk Memory Deserialized 1x Replicated"
> df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a",
> "col_b"])
> df = df.persist(StorageLevel.MEMORY_AND_DISK)
> df.count()
> # Storage level in Spark UI: "Disk Memory Serialized 1x Replicated"{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-40154) PySpark: DataFrame.cache docstring gives wrong storage level
[
https://issues.apache.org/jira/browse/SPARK-40154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-40154.
--
Fix Version/s: 3.5.1
4.0.0
3.4.2
Resolution: Fixed
Issue resolved by pull request 43229
[https://github.com/apache/spark/pull/43229]
> PySpark: DataFrame.cache docstring gives wrong storage level
>
>
> Key: SPARK-40154
> URL: https://issues.apache.org/jira/browse/SPARK-40154
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Paul Staab
>Assignee: Paul Staab
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.5.1, 4.0.0, 3.4.2
>
>
> The docstring of the `DataFrame.cache()` method currently states that it uses
> a serialized storage level
> {code:java}
> Persists the :class:`DataFrame` with the default storage level
> (`MEMORY_AND_DISK`).
> [...]
> -The default storage level has changed to `MEMORY_AND_DISK` to match
> Scala in 2.0.{code}
> while `DataFrame.persist()` states that it uses a deserialized storage level
> {code:java}
> If no storage level is specified defaults to (`MEMORY_AND_DISK_DESER`)
> [...]
> The default storage level has changed to `MEMORY_AND_DISK_DESER` to match
> Scala in 3.0.{code}
>
> However, in practice both `.cache()` and `.persist()` use deserialized
> storage levels:
> {code:java}
> import pyspark
> from pyspark.sql import SparkSession
> from pyspark import StorageLevel
> print(pyspark.__version__)
> # 3.3.0
> spark = SparkSession.builder.master("local[2]").getOrCreate()
> df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a",
> "col_b"])
> df = df.cache()
> df.count()
> # Storage level in Spark UI: "Disk Memory Deserialized 1x Replicated"
> df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a",
> "col_b"])
> df = df.persist()
> df.count()
> # Storage level in Spark UI: "Disk Memory Deserialized 1x Replicated"
> df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a",
> "col_b"])
> df = df.persist(StorageLevel.MEMORY_AND_DISK)
> df.count()
> # Storage level in Spark UI: "Disk Memory Serialized 1x Replicated"{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40154) PySpark: DataFrame.cache docstring gives wrong storage level
[
https://issues.apache.org/jira/browse/SPARK-40154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-40154:
Assignee: Paul Staab
> PySpark: DataFrame.cache docstring gives wrong storage level
>
>
> Key: SPARK-40154
> URL: https://issues.apache.org/jira/browse/SPARK-40154
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Paul Staab
>Assignee: Paul Staab
>Priority: Minor
> Labels: pull-request-available
>
> The docstring of the `DataFrame.cache()` method currently states that it uses
> a serialized storage level
> {code:java}
> Persists the :class:`DataFrame` with the default storage level
> (`MEMORY_AND_DISK`).
> [...]
> -The default storage level has changed to `MEMORY_AND_DISK` to match
> Scala in 2.0.{code}
> while `DataFrame.persist()` states that it uses a deserialized storage level
> {code:java}
> If no storage level is specified defaults to (`MEMORY_AND_DISK_DESER`)
> [...]
> The default storage level has changed to `MEMORY_AND_DISK_DESER` to match
> Scala in 3.0.{code}
>
> However, in practice both `.cache()` and `.persist()` use deserialized
> storage levels:
> {code:java}
> import pyspark
> from pyspark.sql import SparkSession
> from pyspark import StorageLevel
> print(pyspark.__version__)
> # 3.3.0
> spark = SparkSession.builder.master("local[2]").getOrCreate()
> df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a",
> "col_b"])
> df = df.cache()
> df.count()
> # Storage level in Spark UI: "Disk Memory Deserialized 1x Replicated"
> df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a",
> "col_b"])
> df = df.persist()
> df.count()
> # Storage level in Spark UI: "Disk Memory Deserialized 1x Replicated"
> df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a",
> "col_b"])
> df = df.persist(StorageLevel.MEMORY_AND_DISK)
> df.count()
> # Storage level in Spark UI: "Disk Memory Serialized 1x Replicated"{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45610) Fix "Auto-application to `()` is deprecated."
[
https://issues.apache.org/jira/browse/SPARK-45610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-45610.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43472
[https://github.com/apache/spark/pull/43472]
> Fix "Auto-application to `()` is deprecated."
> -
>
> Key: SPARK-45610
> URL: https://issues.apache.org/jira/browse/SPARK-45610
> Project: Spark
> Issue Type: Sub-task
> Components: GraphX, MLlib, Spark Core, SQL, Structured Streaming
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> For the following case, a compile warning will be issued in Scala 2.13：
>
> {code:java}
> Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.8).
> Type in expressions for evaluation. Or try :help.
> scala> class Foo {
> | def isEmpty(): Boolean = true
> | def isTrue(x: Boolean): Boolean = x
> | }
> class Foo
> scala> val foo = new Foo
> val foo: Foo = Foo@7061622
> scala> val ret = foo.isEmpty
> ^
> warning: Auto-application to `()` is deprecated. Supply the empty
> argument list `()` explicitly to invoke method isEmpty,
> or remove the empty argument list from its definition (Java-defined
> methods are exempt).
> In Scala 3, an unapplied method like this will be eta-expanded into a
> function. [quickfixable]
> val ret: Boolean = true {code}
> But for Scala 3, it is a compile error:
> {code:java}
> Welcome to Scala 3.3.1 (17.0.8, Java OpenJDK 64-Bit Server VM).
> Type in expressions for evaluation. Or try :help.
>
>
>
>
> scala> class Foo {
> | def isEmpty(): Boolean = true
> | def isTrue(x: Boolean): Boolean = x
> | }
> // defined class Foo
>
>
>
>
> scala> val foo = new Foo
> val foo: Foo = Foo@591f6f83
>
>
>
>
> scala> val ret = foo.isEmpty
> -- [E100] Syntax Error:
>
> 1 |val ret = foo.isEmpty
> | ^^^
> | method isEmpty in class Foo must be called with () argument
> |
> | longer explanation available when compiling with `-explain`
> 1 error found {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45610) Fix "Auto-application to `()` is deprecated."
[
https://issues.apache.org/jira/browse/SPARK-45610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-45610:
Assignee: Yang Jie
> Fix "Auto-application to `()` is deprecated."
> -
>
> Key: SPARK-45610
> URL: https://issues.apache.org/jira/browse/SPARK-45610
> Project: Spark
> Issue Type: Sub-task
> Components: GraphX, MLlib, Spark Core, SQL, Structured Streaming
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
>
> For the following case, a compile warning will be issued in Scala 2.13：
>
> {code:java}
> Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.8).
> Type in expressions for evaluation. Or try :help.
> scala> class Foo {
> | def isEmpty(): Boolean = true
> | def isTrue(x: Boolean): Boolean = x
> | }
> class Foo
> scala> val foo = new Foo
> val foo: Foo = Foo@7061622
> scala> val ret = foo.isEmpty
> ^
> warning: Auto-application to `()` is deprecated. Supply the empty
> argument list `()` explicitly to invoke method isEmpty,
> or remove the empty argument list from its definition (Java-defined
> methods are exempt).
> In Scala 3, an unapplied method like this will be eta-expanded into a
> function. [quickfixable]
> val ret: Boolean = true {code}
> But for Scala 3, it is a compile error:
> {code:java}
> Welcome to Scala 3.3.1 (17.0.8, Java OpenJDK 64-Bit Server VM).
> Type in expressions for evaluation. Or try :help.
>
>
>
>
> scala> class Foo {
> | def isEmpty(): Boolean = true
> | def isTrue(x: Boolean): Boolean = x
> | }
> // defined class Foo
>
>
>
>
> scala> val foo = new Foo
> val foo: Foo = Foo@591f6f83
>
>
>
>
> scala> val ret = foo.isEmpty
> -- [E100] Syntax Error:
>
> 1 |val ret = foo.isEmpty
> | ^^^
> | method isEmpty in class Foo must be called with () argument
> |
> | longer explanation available when compiling with `-explain`
> 1 error found {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-45610) Handle "Auto-application to `()` is deprecated."
[
https://issues.apache.org/jira/browse/SPARK-45610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1405#comment-1405
]
Sean R. Owen commented on SPARK-45610:
--
I think it's better to make big changes at major version boundaries. I'd expect
we support Scala 3 at some point for Spark 4.x. Therefore I think it'd be OK to
proceed with these changes now for 4.0.
> Handle "Auto-application to `()` is deprecated."
>
>
> Key: SPARK-45610
> URL: https://issues.apache.org/jira/browse/SPARK-45610
> Project: Spark
> Issue Type: Sub-task
> Components: GraphX, MLlib, Spark Core, SQL, Structured Streaming
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Priority: Major
>
> For the following case, a compile warning will be issued in Scala 2.13：
>
> {code:java}
> Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.8).
> Type in expressions for evaluation. Or try :help.
> scala> class Foo {
> | def isEmpty(): Boolean = true
> | def isTrue(x: Boolean): Boolean = x
> | }
> class Foo
> scala> val foo = new Foo
> val foo: Foo = Foo@7061622
> scala> val ret = foo.isEmpty
> ^
> warning: Auto-application to `()` is deprecated. Supply the empty
> argument list `()` explicitly to invoke method isEmpty,
> or remove the empty argument list from its definition (Java-defined
> methods are exempt).
> In Scala 3, an unapplied method like this will be eta-expanded into a
> function. [quickfixable]
> val ret: Boolean = true {code}
> But for Scala 3, it is a compile error:
> {code:java}
> Welcome to Scala 3.3.1 (17.0.8, Java OpenJDK 64-Bit Server VM).
> Type in expressions for evaluation. Or try :help.
>
>
>
>
> scala> class Foo {
> | def isEmpty(): Boolean = true
> | def isTrue(x: Boolean): Boolean = x
> | }
> // defined class Foo
>
>
>
>
> scala> val foo = new Foo
> val foo: Foo = Foo@591f6f83
>
>
>
>
> scala> val ret = foo.isEmpty
> -- [E100] Syntax Error:
>
> 1 |val ret = foo.isEmpty
> | ^^^
> | method isEmpty in class Foo must be called with () argument
> |
> | longer explanation available when compiling with `-explain`
> 1 error found {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45588) Minor scaladoc improvement in StreamingForeachBatchHelper
[ https://issues.apache.org/jira/browse/SPARK-45588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45588: - Issue Type: Improvement (was: Bug) Priority: Trivial (was: Major) > Minor scaladoc improvement in StreamingForeachBatchHelper > - > > Key: SPARK-45588 > URL: https://issues.apache.org/jira/browse/SPARK-45588 > Project: Spark > Issue Type: Improvement > Components: Connect, Structured Streaming >Affects Versions: 4.0.0 >Reporter: Raghu Angadi >Priority: Trivial > Labels: pull-request-available > > Document RunnerCleaner in StreamingForeachBatchHelper. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45549) Remove unused `numExistingExecutors` in `CoarseGrainedSchedulerBackend`
[ https://issues.apache.org/jira/browse/SPARK-45549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45549: - Priority: Trivial (was: Minor) > Remove unused `numExistingExecutors` in `CoarseGrainedSchedulerBackend` > --- > > Key: SPARK-45549 > URL: https://issues.apache.org/jira/browse/SPARK-45549 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: xiaoping.huang >Priority: Trivial > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45564) Simplify 'DataFrameStatFunctions.bloomFilter' function with 'BloomFilterAggregate' expression
[ https://issues.apache.org/jira/browse/SPARK-45564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45564: - Priority: Minor (was: Major) > Simplify 'DataFrameStatFunctions.bloomFilter' function with > 'BloomFilterAggregate' expression > - > > Key: SPARK-45564 > URL: https://issues.apache.org/jira/browse/SPARK-45564 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45564) Simplify 'DataFrameStatFunctions.bloomFilter' function with 'BloomFilterAggregate' expression
[ https://issues.apache.org/jira/browse/SPARK-45564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45564. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43391 [https://github.com/apache/spark/pull/43391] > Simplify 'DataFrameStatFunctions.bloomFilter' function with > 'BloomFilterAggregate' expression > - > > Key: SPARK-45564 > URL: https://issues.apache.org/jira/browse/SPARK-45564 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45564) Simplify 'DataFrameStatFunctions.bloomFilter' function with 'BloomFilterAggregate' expression
[ https://issues.apache.org/jira/browse/SPARK-45564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45564: Assignee: Ruifeng Zheng > Simplify 'DataFrameStatFunctions.bloomFilter' function with > 'BloomFilterAggregate' expression > - > > Key: SPARK-45564 > URL: https://issues.apache.org/jira/browse/SPARK-45564 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45512) Fix compilation warnings related to other-nullary-override
[
https://issues.apache.org/jira/browse/SPARK-45512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-45512.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43332
[https://github.com/apache/spark/pull/43332]
> Fix compilation warnings related to other-nullary-override
> --
>
> Key: SPARK-45512
> URL: https://issues.apache.org/jira/browse/SPARK-45512
> Project: Spark
> Issue Type: Sub-task
> Components: DStreams, Spark Core, SQL
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/CloseableIterator.scala:36:16:
> method with a single empty parameter list overrides method hasNext in trait
> Iterator defined without a parameter list [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.WrappedCloseableIterator
> [error] override def hasNext(): Boolean = innerIterator.hasNext
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ExecutePlanResponseReattachableIterator.scala:136:16:
> method without a parameter list overrides method hasNext in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.ExecutePlanResponseReattachableIterator
> [error] override def hasNext: Boolean = synchronized {
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:73:20:
> method without a parameter list overrides method hasNext in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.GrpcExceptionConverter.convertIterator
> [error] override def hasNext: Boolean = {
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala:77:18:
> method without a parameter list overrides method next in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.GrpcRetryHandler.RetryIterator
> [error] override def next: U = {
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala:81:18:
> method without a parameter list overrides method hasNext in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.GrpcRetryHandler.RetryIterator
> [error] override def hasNext: Boolean = {
> [error] ^
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45512) Fix compilation warnings related to other-nullary-override
[
https://issues.apache.org/jira/browse/SPARK-45512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-45512:
-
Priority: Minor (was: Major)
> Fix compilation warnings related to other-nullary-override
> --
>
> Key: SPARK-45512
> URL: https://issues.apache.org/jira/browse/SPARK-45512
> Project: Spark
> Issue Type: Sub-task
> Components: DStreams, Spark Core, SQL
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/CloseableIterator.scala:36:16:
> method with a single empty parameter list overrides method hasNext in trait
> Iterator defined without a parameter list [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.WrappedCloseableIterator
> [error] override def hasNext(): Boolean = innerIterator.hasNext
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ExecutePlanResponseReattachableIterator.scala:136:16:
> method without a parameter list overrides method hasNext in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.ExecutePlanResponseReattachableIterator
> [error] override def hasNext: Boolean = synchronized {
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:73:20:
> method without a parameter list overrides method hasNext in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.GrpcExceptionConverter.convertIterator
> [error] override def hasNext: Boolean = {
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala:77:18:
> method without a parameter list overrides method next in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.GrpcRetryHandler.RetryIterator
> [error] override def next: U = {
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala:81:18:
> method without a parameter list overrides method hasNext in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.GrpcRetryHandler.RetryIterator
> [error] override def hasNext: Boolean = {
> [error] ^
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45512) Fix compilation warnings related to other-nullary-override
[
https://issues.apache.org/jira/browse/SPARK-45512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-45512:
Assignee: Yang Jie
> Fix compilation warnings related to other-nullary-override
> --
>
> Key: SPARK-45512
> URL: https://issues.apache.org/jira/browse/SPARK-45512
> Project: Spark
> Issue Type: Sub-task
> Components: DStreams, Spark Core, SQL
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/CloseableIterator.scala:36:16:
> method with a single empty parameter list overrides method hasNext in trait
> Iterator defined without a parameter list [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.WrappedCloseableIterator
> [error] override def hasNext(): Boolean = innerIterator.hasNext
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ExecutePlanResponseReattachableIterator.scala:136:16:
> method without a parameter list overrides method hasNext in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.ExecutePlanResponseReattachableIterator
> [error] override def hasNext: Boolean = synchronized {
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:73:20:
> method without a parameter list overrides method hasNext in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.GrpcExceptionConverter.convertIterator
> [error] override def hasNext: Boolean = {
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala:77:18:
> method without a parameter list overrides method next in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.GrpcRetryHandler.RetryIterator
> [error] override def next: U = {
> [error] ^
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala:81:18:
> method without a parameter list overrides method hasNext in class
> WrappedCloseableIterator defined with a single empty parameter list
> [quickfixable]
> [error] Applicable -Wconf / @nowarn filters for this fatal warning: msg= of the message>, cat=other-nullary-override,
> site=org.apache.spark.sql.connect.client.GrpcRetryHandler.RetryIterator
> [error] override def hasNext: Boolean = {
> [error] ^
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45273) Http header Attack【HttpSecurityFilter】
[ https://issues.apache.org/jira/browse/SPARK-45273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45273. -- Resolution: Invalid > Http header Attack【HttpSecurityFilter】 > -- > > Key: SPARK-45273 > URL: https://issues.apache.org/jira/browse/SPARK-45273 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: chenyu >Priority: Major > Labels: pull-request-available > > There is an HTTP host header attack vulnerability in the target URL -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45467) Replace `Proxy.getProxyClass()` with `Proxy.newProxyInstance().getClass`
[
https://issues.apache.org/jira/browse/SPARK-45467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-45467.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43291
[https://github.com/apache/spark/pull/43291]
> Replace `Proxy.getProxyClass()` with `Proxy.newProxyInstance().getClass`
>
>
> Key: SPARK-45467
> URL: https://issues.apache.org/jira/browse/SPARK-45467
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> * @deprecated Proxy classes generated in a named module are encapsulated
> * and not accessible to code outside its module.
> * {@link Constructor#newInstance(Object...) Constructor.newInstance}
> * will throw {@code IllegalAccessException} when it is called on
> * an inaccessible proxy class.
> * Use {@link #newProxyInstance(ClassLoader, Class[], InvocationHandler)}
> * to create a proxy instance instead.
> *
> * @see Package and Module Membership of Proxy Class
> * @revised 9
> */
> @Deprecated
> @CallerSensitive
> public static Class getProxyClass(ClassLoader loader,
> Class... interfaces)
> throws IllegalArgumentException {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45467) Replace `Proxy.getProxyClass()` with `Proxy.newProxyInstance().getClass`
[
https://issues.apache.org/jira/browse/SPARK-45467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-45467:
Assignee: Yang Jie
> Replace `Proxy.getProxyClass()` with `Proxy.newProxyInstance().getClass`
>
>
> Key: SPARK-45467
> URL: https://issues.apache.org/jira/browse/SPARK-45467
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> * @deprecated Proxy classes generated in a named module are encapsulated
> * and not accessible to code outside its module.
> * {@link Constructor#newInstance(Object...) Constructor.newInstance}
> * will throw {@code IllegalAccessException} when it is called on
> * an inaccessible proxy class.
> * Use {@link #newProxyInstance(ClassLoader, Class[], InvocationHandler)}
> * to create a proxy instance instead.
> *
> * @see Package and Module Membership of Proxy Class
> * @revised 9
> */
> @Deprecated
> @CallerSensitive
> public static Class getProxyClass(ClassLoader loader,
> Class... interfaces)
> throws IllegalArgumentException {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45467) Replace `Proxy.getProxyClass()` with `Proxy.newProxyInstance().getClass`
[
https://issues.apache.org/jira/browse/SPARK-45467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-45467:
-
Priority: Minor (was: Major)
> Replace `Proxy.getProxyClass()` with `Proxy.newProxyInstance().getClass`
>
>
> Key: SPARK-45467
> URL: https://issues.apache.org/jira/browse/SPARK-45467
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> * @deprecated Proxy classes generated in a named module are encapsulated
> * and not accessible to code outside its module.
> * {@link Constructor#newInstance(Object...) Constructor.newInstance}
> * will throw {@code IllegalAccessException} when it is called on
> * an inaccessible proxy class.
> * Use {@link #newProxyInstance(ClassLoader, Class[], InvocationHandler)}
> * to create a proxy instance instead.
> *
> * @see Package and Module Membership of Proxy Class
> * @revised 9
> */
> @Deprecated
> @CallerSensitive
> public static Class getProxyClass(ClassLoader loader,
> Class... interfaces)
> throws IllegalArgumentException {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45496) Fix the compilation warning related to other-pure-statement
[
https://issues.apache.org/jira/browse/SPARK-45496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-45496.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43312
[https://github.com/apache/spark/pull/43312]
> Fix the compilation warning related to other-pure-statement
> ---
>
> Key: SPARK-45496
> URL: https://issues.apache.org/jira/browse/SPARK-45496
> Project: Spark
> Issue Type: Sub-task
> Components: DStreams, Spark Core
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> "-Wconf:cat=other-match-analysis=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv",
> "-Wconf:cat=other-pure-statement=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv",
> "-Wconf:cat=other-pure-statement=org.apache.spark.scheduler.OutputCommitCoordinatorSuite:wv",
> "-Wconf:cat=other-pure-statement=org.apache.spark.sql.streaming.sources.StreamingDataSourceV2Suite.testPositiveCase.\\$anonfun:wv",
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45496) Fix the compilation warning related to other-pure-statement
[
https://issues.apache.org/jira/browse/SPARK-45496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-45496:
-
Priority: Minor (was: Major)
> Fix the compilation warning related to other-pure-statement
> ---
>
> Key: SPARK-45496
> URL: https://issues.apache.org/jira/browse/SPARK-45496
> Project: Spark
> Issue Type: Sub-task
> Components: DStreams, Spark Core
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Minor
> Labels: pull-request-available
>
> {code:java}
> "-Wconf:cat=other-match-analysis=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv",
> "-Wconf:cat=other-pure-statement=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv",
> "-Wconf:cat=other-pure-statement=org.apache.spark.scheduler.OutputCommitCoordinatorSuite:wv",
> "-Wconf:cat=other-pure-statement=org.apache.spark.sql.streaming.sources.StreamingDataSourceV2Suite.testPositiveCase.\\$anonfun:wv",
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-43828) Add config to control whether close idle connection
[ https://issues.apache.org/jira/browse/SPARK-43828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-43828. -- Resolution: Won't Fix > Add config to control whether close idle connection > --- > > Key: SPARK-43828 > URL: https://issues.apache.org/jira/browse/SPARK-43828 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Zhongwei Zhu >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45496) Fix the compilation warning related to other-pure-statement
[
https://issues.apache.org/jira/browse/SPARK-45496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-45496:
Assignee: Yang Jie
> Fix the compilation warning related to other-pure-statement
> ---
>
> Key: SPARK-45496
> URL: https://issues.apache.org/jira/browse/SPARK-45496
> Project: Spark
> Issue Type: Sub-task
> Components: DStreams, Spark Core
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> "-Wconf:cat=other-match-analysis=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv",
> "-Wconf:cat=other-pure-statement=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv",
> "-Wconf:cat=other-pure-statement=org.apache.spark.scheduler.OutputCommitCoordinatorSuite:wv",
> "-Wconf:cat=other-pure-statement=org.apache.spark.sql.streaming.sources.StreamingDataSourceV2Suite.testPositiveCase.\\$anonfun:wv",
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45377) [CORE] Handle InputStream in NettyLogger
[ https://issues.apache.org/jira/browse/SPARK-45377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45377: - Priority: Minor (was: Major) > [CORE] Handle InputStream in NettyLogger > > > Key: SPARK-45377 > URL: https://issues.apache.org/jira/browse/SPARK-45377 > Project: Spark > Issue Type: Task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Hasnain Lakhani >Assignee: Hasnain Lakhani >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > > Allow NettyLogger to also print the size of InputStreams which aids debugging > for SSL functionality -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45377) [CORE] Handle InputStream in NettyLogger
[ https://issues.apache.org/jira/browse/SPARK-45377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45377. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43165 [https://github.com/apache/spark/pull/43165] > [CORE] Handle InputStream in NettyLogger > > > Key: SPARK-45377 > URL: https://issues.apache.org/jira/browse/SPARK-45377 > Project: Spark > Issue Type: Task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Hasnain Lakhani >Assignee: Hasnain Lakhani >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Allow NettyLogger to also print the size of InputStreams which aids debugging > for SSL functionality -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45377) [CORE] Handle InputStream in NettyLogger
[ https://issues.apache.org/jira/browse/SPARK-45377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45377: Assignee: Hasnain Lakhani > [CORE] Handle InputStream in NettyLogger > > > Key: SPARK-45377 > URL: https://issues.apache.org/jira/browse/SPARK-45377 > Project: Spark > Issue Type: Task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Hasnain Lakhani >Assignee: Hasnain Lakhani >Priority: Major > Labels: pull-request-available > > Allow NettyLogger to also print the size of InputStreams which aids debugging > for SSL functionality -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45389) Correct MetaException matching rule on getting partition metadata
[ https://issues.apache.org/jira/browse/SPARK-45389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45389. -- Fix Version/s: 4.0.0 Assignee: Cheng Pan Resolution: Fixed https://github.com/apache/spark/pull/43191 > Correct MetaException matching rule on getting partition metadata > - > > Key: SPARK-45389 > URL: https://issues.apache.org/jira/browse/SPARK-45389 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.3.3 >Reporter: Cheng Pan >Assignee: Cheng Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44895) Considering 'daemon', 'priority' from higher JDKs for ThreadStackTrace class
[
https://issues.apache.org/jira/browse/SPARK-44895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-44895:
-
Priority: Minor (was: Major)
> Considering 'daemon', 'priority' from higher JDKs for ThreadStackTrace class
>
>
> Key: SPARK-44895
> URL: https://issues.apache.org/jira/browse/SPARK-44895
> Project: Spark
> Issue Type: Sub-task
> Components: Web UI
>Affects Versions: 4.0.0
>Reporter: Kent Yao
>Assignee: Kent Yao
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> jshell> var t = java.lang.management.ManagementFactory.getThreadMXBean()t ==>
> com.sun.management.internal.HotSpotThreadImpl@7daf6ecc
> jshell> var tt = t.dumpAllThreads(true, true)tt ==> ThreadInfo[10] { "main"
> prio=5 Id=1 RUNNABLE at ... k$NonfairSync@27fa135a
> }
> jshell> for (java.lang.management.ThreadInfo t1: tt)
> {System.out.println(t1.toString());}"main" prio=5 Id=1 RUNNABLE at
> java.management@20.0.1/sun.management.ThreadImpl.dumpThreads0(Native Method)
> at
> java.management@20.0.1/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:540)
> at
> java.management@20.0.1/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:527)
> at REPL.$JShell$12.do_it$Aux($JShell$12.java:7) at
> REPL.$JShell$12.do_it$($JShell$12.java:11) at
> java.base@20.0.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
> at
> java.base@20.0.1/java.lang.invoke.LambdaForm$MH/0x007001008c00.invoke(LambdaForm$MH)
> at
> java.base@20.0.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
> ...
> "Reference Handler" daemon prio=10 Id=8 RUNNABLE at
> java.base@20.0.1/java.lang.ref.Reference.waitForReferencePendingList(Native
> Method) at
> java.base@20.0.1/java.lang.ref.Reference.processPendingReferences(Reference.java:246)
> at
> java.base@20.0.1/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:208)
> {code}
> the `daemon prio=10` is not available for ThreadInfo of jdk8
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44895) Considering 'daemon', 'priority' from higher JDKs for ThreadStackTrace class
[
https://issues.apache.org/jira/browse/SPARK-44895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-44895.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43095
[https://github.com/apache/spark/pull/43095]
> Considering 'daemon', 'priority' from higher JDKs for ThreadStackTrace class
>
>
> Key: SPARK-44895
> URL: https://issues.apache.org/jira/browse/SPARK-44895
> Project: Spark
> Issue Type: Sub-task
> Components: Web UI
>Affects Versions: 4.0.0
>Reporter: Kent Yao
>Assignee: Kent Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> jshell> var t = java.lang.management.ManagementFactory.getThreadMXBean()t ==>
> com.sun.management.internal.HotSpotThreadImpl@7daf6ecc
> jshell> var tt = t.dumpAllThreads(true, true)tt ==> ThreadInfo[10] { "main"
> prio=5 Id=1 RUNNABLE at ... k$NonfairSync@27fa135a
> }
> jshell> for (java.lang.management.ThreadInfo t1: tt)
> {System.out.println(t1.toString());}"main" prio=5 Id=1 RUNNABLE at
> java.management@20.0.1/sun.management.ThreadImpl.dumpThreads0(Native Method)
> at
> java.management@20.0.1/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:540)
> at
> java.management@20.0.1/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:527)
> at REPL.$JShell$12.do_it$Aux($JShell$12.java:7) at
> REPL.$JShell$12.do_it$($JShell$12.java:11) at
> java.base@20.0.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
> at
> java.base@20.0.1/java.lang.invoke.LambdaForm$MH/0x007001008c00.invoke(LambdaForm$MH)
> at
> java.base@20.0.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
> ...
> "Reference Handler" daemon prio=10 Id=8 RUNNABLE at
> java.base@20.0.1/java.lang.ref.Reference.waitForReferencePendingList(Native
> Method) at
> java.base@20.0.1/java.lang.ref.Reference.processPendingReferences(Reference.java:246)
> at
> java.base@20.0.1/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:208)
> {code}
> the `daemon prio=10` is not available for ThreadInfo of jdk8
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-44895) Considering 'daemon', 'priority' from higher JDKs for ThreadStackTrace class
[
https://issues.apache.org/jira/browse/SPARK-44895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-44895:
Assignee: Kent Yao
> Considering 'daemon', 'priority' from higher JDKs for ThreadStackTrace class
>
>
> Key: SPARK-44895
> URL: https://issues.apache.org/jira/browse/SPARK-44895
> Project: Spark
> Issue Type: Sub-task
> Components: Web UI
>Affects Versions: 4.0.0
>Reporter: Kent Yao
>Assignee: Kent Yao
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> jshell> var t = java.lang.management.ManagementFactory.getThreadMXBean()t ==>
> com.sun.management.internal.HotSpotThreadImpl@7daf6ecc
> jshell> var tt = t.dumpAllThreads(true, true)tt ==> ThreadInfo[10] { "main"
> prio=5 Id=1 RUNNABLE at ... k$NonfairSync@27fa135a
> }
> jshell> for (java.lang.management.ThreadInfo t1: tt)
> {System.out.println(t1.toString());}"main" prio=5 Id=1 RUNNABLE at
> java.management@20.0.1/sun.management.ThreadImpl.dumpThreads0(Native Method)
> at
> java.management@20.0.1/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:540)
> at
> java.management@20.0.1/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:527)
> at REPL.$JShell$12.do_it$Aux($JShell$12.java:7) at
> REPL.$JShell$12.do_it$($JShell$12.java:11) at
> java.base@20.0.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
> at
> java.base@20.0.1/java.lang.invoke.LambdaForm$MH/0x007001008c00.invoke(LambdaForm$MH)
> at
> java.base@20.0.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
> ...
> "Reference Handler" daemon prio=10 Id=8 RUNNABLE at
> java.base@20.0.1/java.lang.ref.Reference.waitForReferencePendingList(Native
> Method) at
> java.base@20.0.1/java.lang.ref.Reference.processPendingReferences(Reference.java:246)
> at
> java.base@20.0.1/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:208)
> {code}
> the `daemon prio=10` is not available for ThreadInfo of jdk8
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45364) Clean up the unnecessary Scala 2.12 logical in SparkBuild
[ https://issues.apache.org/jira/browse/SPARK-45364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45364: Assignee: BingKun Pan > Clean up the unnecessary Scala 2.12 logical in SparkBuild > - > > Key: SPARK-45364 > URL: https://issues.apache.org/jira/browse/SPARK-45364 > Project: Spark > Issue Type: Sub-task > Components: Build, Project Infra >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45364) Clean up the unnecessary Scala 2.12 logical in SparkBuild
[ https://issues.apache.org/jira/browse/SPARK-45364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45364: - Priority: Trivial (was: Minor) > Clean up the unnecessary Scala 2.12 logical in SparkBuild > - > > Key: SPARK-45364 > URL: https://issues.apache.org/jira/browse/SPARK-45364 > Project: Spark > Issue Type: Sub-task > Components: Build, Project Infra >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45364) Clean up the unnecessary Scala 2.12 logical in SparkBuild
[ https://issues.apache.org/jira/browse/SPARK-45364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45364. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43158 [https://github.com/apache/spark/pull/43158] > Clean up the unnecessary Scala 2.12 logical in SparkBuild > - > > Key: SPARK-45364 > URL: https://issues.apache.org/jira/browse/SPARK-45364 > Project: Spark > Issue Type: Sub-task > Components: Build, Project Infra >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-44442) Drop mesos support
[
https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-2:
Assignee: Sean R. Owen
> Drop mesos support
> --
>
> Key: SPARK-2
> URL: https://issues.apache.org/jira/browse/SPARK-2
> Project: Spark
> Issue Type: Sub-task
> Components: Mesos
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Sean R. Owen
>Priority: Major
> Labels: pull-request-available
>
> [https://spark.apache.org/docs/latest/running-on-mesos.html]
>
> {_}Note{_}: Apache Mesos support is deprecated as of Apache Spark 3.2.0. It
> will be removed in a future version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44442) Drop mesos support
[
https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-2.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43135
[https://github.com/apache/spark/pull/43135]
> Drop mesos support
> --
>
> Key: SPARK-2
> URL: https://issues.apache.org/jira/browse/SPARK-2
> Project: Spark
> Issue Type: Sub-task
> Components: Mesos
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Sean R. Owen
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> [https://spark.apache.org/docs/latest/running-on-mesos.html]
>
> {_}Note{_}: Apache Mesos support is deprecated as of Apache Spark 3.2.0. It
> will be removed in a future version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44539) Upgrade RoaringBitmap to 1.0.0
[ https://issues.apache.org/jira/browse/SPARK-44539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-44539: - Priority: Minor (was: Trivial) > Upgrade RoaringBitmap to 1.0.0 > --- > > Key: SPARK-44539 > URL: https://issues.apache.org/jira/browse/SPARK-44539 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-44539) Upgrade RoaringBitmap to 1.0.0
[ https://issues.apache.org/jira/browse/SPARK-44539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-44539: Assignee: BingKun Pan > Upgrade RoaringBitmap to 1.0.0 > --- > > Key: SPARK-44539 > URL: https://issues.apache.org/jira/browse/SPARK-44539 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Trivial > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44539) Upgrade RoaringBitmap to 1.0.0
[ https://issues.apache.org/jira/browse/SPARK-44539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44539. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42143 [https://github.com/apache/spark/pull/42143] > Upgrade RoaringBitmap to 1.0.0 > --- > > Key: SPARK-44539 > URL: https://issues.apache.org/jira/browse/SPARK-44539 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45343) CSV multiLine documentation is confusing
[ https://issues.apache.org/jira/browse/SPARK-45343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45343: Assignee: Bill Schneider > CSV multiLine documentation is confusing > > > Key: SPARK-45343 > URL: https://issues.apache.org/jira/browse/SPARK-45343 > Project: Spark > Issue Type: Documentation > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: Bill Schneider >Assignee: Bill Schneider >Priority: Trivial > Labels: pull-request-available > > This is confusing, maybe copy-paste from JSON: > |Parse one record, which may span multiple lines, per file. CSV built-in > functions ignore this option.| > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45343) CSV multiLine documentation is confusing
[ https://issues.apache.org/jira/browse/SPARK-45343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45343. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43132 [https://github.com/apache/spark/pull/43132] > CSV multiLine documentation is confusing > > > Key: SPARK-45343 > URL: https://issues.apache.org/jira/browse/SPARK-45343 > Project: Spark > Issue Type: Documentation > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: Bill Schneider >Assignee: Bill Schneider >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > > This is confusing, maybe copy-paste from JSON: > |Parse one record, which may span multiple lines, per file. CSV built-in > functions ignore this option.| > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-44170) Migrating Junit4 to Junit5
[ https://issues.apache.org/jira/browse/SPARK-44170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-44170: Assignee: Yang Jie > Migrating Junit4 to Junit5 > -- > > Key: SPARK-44170 > URL: https://issues.apache.org/jira/browse/SPARK-44170 > Project: Spark > Issue Type: Sub-task > Components: Tests >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > > JUnit5 is a powerful and flexible update to the JUnit framework, and it > provides a variety of improvements and new features to organize and > describe test cases, as well as help in understanding test results： > # JUnit 5 leverages features from Java 8 or later, such as lambda functions, > making tests more powerful and easier to maintain, but Junit 4 still a Java 7 > compatible version > # JUnit 5 has added some useful new features for describing, organizing, and > executing tests. For examples: [Parameterized > Tests|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests] > and [Conditional Test > Execution|https://junit.org/junit5/docs/current/user-guide/#extensions-conditions] > may make our test code look simpler, [Parallel > Execution|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution] > may make our test faster > > More importantly, Junit4 is currently an inactive project, which has not > released a new version for more than two years > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44170) Migrating Junit4 to Junit5
[ https://issues.apache.org/jira/browse/SPARK-44170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44170. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43074 [https://github.com/apache/spark/pull/43074] > Migrating Junit4 to Junit5 > -- > > Key: SPARK-44170 > URL: https://issues.apache.org/jira/browse/SPARK-44170 > Project: Spark > Issue Type: Sub-task > Components: Tests >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > JUnit5 is a powerful and flexible update to the JUnit framework, and it > provides a variety of improvements and new features to organize and > describe test cases, as well as help in understanding test results： > # JUnit 5 leverages features from Java 8 or later, such as lambda functions, > making tests more powerful and easier to maintain, but Junit 4 still a Java 7 > compatible version > # JUnit 5 has added some useful new features for describing, organizing, and > executing tests. For examples: [Parameterized > Tests|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests] > and [Conditional Test > Execution|https://junit.org/junit5/docs/current/user-guide/#extensions-conditions] > may make our test code look simpler, [Parallel > Execution|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution] > may make our test faster > > More importantly, Junit4 is currently an inactive project, which has not > released a new version for more than two years > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45338: Assignee: Jia Fan > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Assignee: Jia Fan >Priority: Major > Labels: pull-request-available > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45338) Remove scala.collection.JavaConverters
[ https://issues.apache.org/jira/browse/SPARK-45338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45338. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43126 [https://github.com/apache/spark/pull/43126] > Remove scala.collection.JavaConverters > -- > > Key: SPARK-45338 > URL: https://issues.apache.org/jira/browse/SPARK-45338 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Jia Fan >Assignee: Jia Fan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Remove deprecated scala.collection.JavaConverters, replaced by > scala.jdk.CollectionConverters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45341) Make the sbt doc command execute successfully with Java 17
[
https://issues.apache.org/jira/browse/SPARK-45341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-45341:
-
Priority: Minor (was: Major)
> Make the sbt doc command execute successfully with Java 17
> --
>
> Key: SPARK-45341
> URL: https://issues.apache.org/jira/browse/SPARK-45341
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> [error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/Picked up
> JAVA_TOOL_OPTIONS:-Duser.language=en
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/ArrayWrappers.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVIndex.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/UnsupportedStoreVersionException.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreView.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreSerializer.java...
> [error] Constructing Javadoc information...
> [error] Building index for all the packages and classes...
> [error] Standard Doclet version 17.0.8+7-LTS
> [error] Building tree for all the packages and classes...
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java:32:1:
> error: heading used out of sequence: , compared to implicit preceding
> heading:
> [error] * Serialization
> [error] ^Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/InMemoryStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVIndex.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreIterator.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreSerializer.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreView.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVTypeInfo.html...
> [error] Generating
>
[jira] [Resolved] (SPARK-45341) Make the sbt doc command execute successfully with Java 17
[
https://issues.apache.org/jira/browse/SPARK-45341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-45341.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43131
[https://github.com/apache/spark/pull/43131]
> Make the sbt doc command execute successfully with Java 17
> --
>
> Key: SPARK-45341
> URL: https://issues.apache.org/jira/browse/SPARK-45341
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> [error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/Picked up
> JAVA_TOOL_OPTIONS:-Duser.language=en
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/ArrayWrappers.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVIndex.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/UnsupportedStoreVersionException.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreView.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreSerializer.java...
> [error] Constructing Javadoc information...
> [error] Building index for all the packages and classes...
> [error] Standard Doclet version 17.0.8+7-LTS
> [error] Building tree for all the packages and classes...
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java:32:1:
> error: heading used out of sequence: , compared to implicit preceding
> heading:
> [error] * Serialization
> [error] ^Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/InMemoryStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVIndex.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreIterator.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreSerializer.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreView.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVTypeInfo.html...
> [error] Generating
[jira] [Assigned] (SPARK-45341) Make the sbt doc command execute successfully with Java 17
[
https://issues.apache.org/jira/browse/SPARK-45341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-45341:
Assignee: Yang Jie
> Make the sbt doc command execute successfully with Java 17
> --
>
> Key: SPARK-45341
> URL: https://issues.apache.org/jira/browse/SPARK-45341
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/Picked up
> JAVA_TOOL_OPTIONS:-Duser.language=en
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/ArrayWrappers.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVIndex.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/UnsupportedStoreVersionException.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreView.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVTypeInfo.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java...
> [error] Loading source file
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreSerializer.java...
> [error] Constructing Javadoc information...
> [error] Building index for all the packages and classes...
> [error] Standard Doclet version 17.0.8+7-LTS
> [error] Building tree for all the packages and classes...
> [error]
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java:32:1:
> error: heading used out of sequence: , compared to implicit preceding
> heading:
> [error] * Serialization
> [error] ^Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/InMemoryStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVIndex.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStore.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreIterator.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreSerializer.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVStoreView.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/KVTypeInfo.html...
> [error] Generating
> /Users/yangjie01/SourceCode/git/spark-mine-sbt/common/kvstore/target/scala-2.13/api/org/apache/spark/util/kvstore/LevelDB.html...
>
[jira] [Assigned] (SPARK-45334) Remove misleading comment in parquetSchemaConverter
[
https://issues.apache.org/jira/browse/SPARK-45334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-45334:
Assignee: Mengran Lan
> Remove misleading comment in parquetSchemaConverter
> ---
>
> Key: SPARK-45334
> URL: https://issues.apache.org/jira/browse/SPARK-45334
> Project: Spark
> Issue Type: Documentation
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Mengran Lan
>Assignee: Mengran Lan
>Priority: Trivial
> Labels: pull-request-available
>
> I'm debugging a parquet issue and reading spark code as references. Happened
> to find a misleading comment which remains in the latest version as well.
> {code:java}
> Types
> .buildGroup(repetition).as(LogicalTypeAnnotation.listType())
> .addField(Types
> .buildGroup(REPEATED)
> // "array" is the name chosen by parquet-hive (1.7.0 and prior version)
> .addField(convertField(StructField("array", elementType, nullable)))
> .named("bag"))
> .named(field.name) {code}
> the comment above is misleading since Hive always uses "array_element" as the
> name.
> It is imported by this PR [https://github.com/apache/spark/pull/14399] and
> relates to this issue https://issues.apache.org/jira/browse/SPARK-16777
> Furthermore, the parquet-hive module has been removed from the parquet-mr
> project https://issues.apache.org/jira/browse/PARQUET-1676
> I suggest removing this piece of comment and will submit a PR later.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45334) Remove misleading comment in parquetSchemaConverter
[
https://issues.apache.org/jira/browse/SPARK-45334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-45334.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 43119
[https://github.com/apache/spark/pull/43119]
> Remove misleading comment in parquetSchemaConverter
> ---
>
> Key: SPARK-45334
> URL: https://issues.apache.org/jira/browse/SPARK-45334
> Project: Spark
> Issue Type: Documentation
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Mengran Lan
>Assignee: Mengran Lan
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> I'm debugging a parquet issue and reading spark code as references. Happened
> to find a misleading comment which remains in the latest version as well.
> {code:java}
> Types
> .buildGroup(repetition).as(LogicalTypeAnnotation.listType())
> .addField(Types
> .buildGroup(REPEATED)
> // "array" is the name chosen by parquet-hive (1.7.0 and prior version)
> .addField(convertField(StructField("array", elementType, nullable)))
> .named("bag"))
> .named(field.name) {code}
> the comment above is misleading since Hive always uses "array_element" as the
> name.
> It is imported by this PR [https://github.com/apache/spark/pull/14399] and
> relates to this issue https://issues.apache.org/jira/browse/SPARK-16777
> Furthermore, the parquet-hive module has been removed from the parquet-mr
> project https://issues.apache.org/jira/browse/PARQUET-1676
> I suggest removing this piece of comment and will submit a PR later.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44366) Migrate antlr4 from 4.9 to 4.10+
[ https://issues.apache.org/jira/browse/SPARK-44366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44366. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43075 [https://github.com/apache/spark/pull/43075] > Migrate antlr4 from 4.9 to 4.10+ > > > Key: SPARK-44366 > URL: https://issues.apache.org/jira/browse/SPARK-44366 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: Yang Jie >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-44366) Migrate antlr4 from 4.9 to 4.10+
[ https://issues.apache.org/jira/browse/SPARK-44366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-44366: Assignee: Yang Jie > Migrate antlr4 from 4.9 to 4.10+ > > > Key: SPARK-44366 > URL: https://issues.apache.org/jira/browse/SPARK-44366 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: Yang Jie >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45343) CSV multiLine documentation is confusing
[ https://issues.apache.org/jira/browse/SPARK-45343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45343: - Priority: Trivial (was: Major) > CSV multiLine documentation is confusing > > > Key: SPARK-45343 > URL: https://issues.apache.org/jira/browse/SPARK-45343 > Project: Spark > Issue Type: Documentation > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: Bill Schneider >Priority: Trivial > Labels: pull-request-available > > This is confusing, maybe copy-paste from JSON: > |Parse one record, which may span multiple lines, per file. CSV built-in > functions ignore this option.| > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45248) Slow HTTP Denial of Service Attack【jetty server in spark】
[ https://issues.apache.org/jira/browse/SPARK-45248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45248: - Issue Type: Improvement (was: Bug) Priority: Minor (was: Major) > Slow HTTP Denial of Service Attack【jetty server in spark】 > - > > Key: SPARK-45248 > URL: https://issues.apache.org/jira/browse/SPARK-45248 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: chenyu >Assignee: chenyu >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > > The Jetty server may be slow HTTP denial of service attacks in spark ui. > The default value of connector IdleTimeout is 3. > This could trigger such an attack -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45248) Slow HTTP Denial of Service Attack【jetty server in spark】
[ https://issues.apache.org/jira/browse/SPARK-45248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45248. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43078 [https://github.com/apache/spark/pull/43078] > Slow HTTP Denial of Service Attack【jetty server in spark】 > - > > Key: SPARK-45248 > URL: https://issues.apache.org/jira/browse/SPARK-45248 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: chenyu >Assignee: chenyu >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > The Jetty server may be slow HTTP denial of service attacks in spark ui. > The default value of connector IdleTimeout is 3. > This could trigger such an attack -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45248) Slow HTTP Denial of Service Attack【jetty server in spark】
[ https://issues.apache.org/jira/browse/SPARK-45248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45248: Assignee: chenyu > Slow HTTP Denial of Service Attack【jetty server in spark】 > - > > Key: SPARK-45248 > URL: https://issues.apache.org/jira/browse/SPARK-45248 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: chenyu >Assignee: chenyu >Priority: Major > Labels: pull-request-available > > The Jetty server may be slow HTTP denial of service attacks in spark ui. > The default value of connector IdleTimeout is 3. > This could trigger such an attack -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45286) Add back Matomo analytics to release docs
[ https://issues.apache.org/jira/browse/SPARK-45286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45286. -- Fix Version/s: 3.3.4 3.5.1 4.0.0 3.4.2 Resolution: Fixed Issue resolved by pull request 43063 [https://github.com/apache/spark/pull/43063] > Add back Matomo analytics to release docs > - > > Key: SPARK-45286 > URL: https://issues.apache.org/jira/browse/SPARK-45286 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Sean R. Owen >Assignee: Sean R. Owen >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.4, 3.5.1, 4.0.0, 3.4.2 > > > We had previously removed Google Analytics from the website and release docs, > per ASF policy: https://github.com/apache/spark/pull/36310 > We just restored analytics using the ASF-hosted Matomo service on the website: > https://github.com/apache/spark-website/commit/a1548627b48a62c2e51870d1488ca3e09397bd30 > This change would put the same new tracking code back into the release docs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45286) Add back Matomo analytics to release docs
[ https://issues.apache.org/jira/browse/SPARK-45286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45286: - Target Version/s: 3.4.2, 4.0.0, 3.5.1 (was: 4.0.0) > Add back Matomo analytics to release docs > - > > Key: SPARK-45286 > URL: https://issues.apache.org/jira/browse/SPARK-45286 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Sean R. Owen >Assignee: Sean R. Owen >Priority: Minor > Labels: pull-request-available > > We had previously removed Google Analytics from the website and release docs, > per ASF policy: https://github.com/apache/spark/pull/36310 > We just restored analytics using the ASF-hosted Matomo service on the website: > https://github.com/apache/spark-website/commit/a1548627b48a62c2e51870d1488ca3e09397bd30 > This change would put the same new tracking code back into the release docs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-45273) Http header Attack【HttpSecurityFilter】
[ https://issues.apache.org/jira/browse/SPARK-45273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768144#comment-17768144 ] Sean R. Owen commented on SPARK-45273: -- Yep we typically evaluate security reports on priv...@spark.apache.org first, not here > Http header Attack【HttpSecurityFilter】 > -- > > Key: SPARK-45273 > URL: https://issues.apache.org/jira/browse/SPARK-45273 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: chenyu >Priority: Major > > There is an HTTP host header attack vulnerability in the target URL -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45286) Add back Matomo analytics to release docs
Sean R. Owen created SPARK-45286: Summary: Add back Matomo analytics to release docs Key: SPARK-45286 URL: https://issues.apache.org/jira/browse/SPARK-45286 Project: Spark Issue Type: Improvement Components: Documentation Affects Versions: 4.0.0 Reporter: Sean R. Owen Assignee: Sean R. Owen We had previously removed Google Analytics from the website and release docs, per ASF policy: https://github.com/apache/spark/pull/36310 We just restored analytics using the ASF-hosted Matomo service on the website: https://github.com/apache/spark-website/commit/a1548627b48a62c2e51870d1488ca3e09397bd30 This change would put the same new tracking code back into the release docs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-45148) Upgrade ScalaTest to the 3.2.17 series.
[ https://issues.apache.org/jira/browse/SPARK-45148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45148: Assignee: Yang Jie > Upgrade ScalaTest to the 3.2.17 series. > --- > > Key: SPARK-45148 > URL: https://issues.apache.org/jira/browse/SPARK-45148 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-45148) Upgrade ScalaTest to the 3.2.17 series.
[ https://issues.apache.org/jira/browse/SPARK-45148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45148. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42906 [https://github.com/apache/spark/pull/42906] > Upgrade ScalaTest to the 3.2.17 series. > --- > > Key: SPARK-45148 > URL: https://issues.apache.org/jira/browse/SPARK-45148 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45195) Update Example with docker official image
[ https://issues.apache.org/jira/browse/SPARK-45195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45195: - Priority: Minor (was: Major) > Update Example with docker official image > - > > Key: SPARK-45195 > URL: https://issues.apache.org/jira/browse/SPARK-45195 > Project: Spark > Issue Type: Sub-task > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Minor > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
