[jira] [Created] (SPARK-49752) Remove workaround for YARN-3350
Cheng Pan created SPARK-49752: - Summary: Remove workaround for YARN-3350 Key: SPARK-49752 URL: https://issues.apache.org/jira/browse/SPARK-49752 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49751) Fix deserialization of SparkListenerConnectServiceStarted event
Cheng Pan created SPARK-49751: - Summary: Fix deserialization of SparkListenerConnectServiceStarted event Key: SPARK-49751 URL: https://issues.apache.org/jira/browse/SPARK-49751 Project: Spark Issue Type: Bug Components: Connect Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49750) Update docs to mention delegation token support on K8s mode
Cheng Pan created SPARK-49750: - Summary: Update docs to mention delegation token support on K8s mode Key: SPARK-49750 URL: https://issues.apache.org/jira/browse/SPARK-49750 Project: Spark Issue Type: Documentation Components: Documentation Affects Versions: 3.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49732) Spark deamons should respect spark.log.structuredLogging.enabled conf
Cheng Pan created SPARK-49732: - Summary: Spark deamons should respect spark.log.structuredLogging.enabled conf Key: SPARK-49732 URL: https://issues.apache.org/jira/browse/SPARK-49732 Project: Spark Issue Type: Bug Components: Kubernetes, Spark Core Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49457) Remove uncommon curl option --retry-all-errors
Cheng Pan created SPARK-49457: - Summary: Remove uncommon curl option --retry-all-errors Key: SPARK-49457 URL: https://issues.apache.org/jira/browse/SPARK-49457 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49370) maven.scaladoc.skip should not affect test code compilation
Cheng Pan created SPARK-49370: - Summary: maven.scaladoc.skip should not affect test code compilation Key: SPARK-49370 URL: https://issues.apache.org/jira/browse/SPARK-49370 Project: Spark Issue Type: Bug Components: Build Affects Versions: 3.5.0, 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49368) Avoid accessing protobuf lite classes directly
Cheng Pan created SPARK-49368: - Summary: Avoid accessing protobuf lite classes directly Key: SPARK-49368 URL: https://issues.apache.org/jira/browse/SPARK-49368 Project: Spark Issue Type: Improvement Components: Connect Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49199) `build/mvn` should check project local install Maven ahead
Cheng Pan created SPARK-49199: - Summary: `build/mvn` should check project local install Maven ahead Key: SPARK-49199 URL: https://issues.apache.org/jira/browse/SPARK-49199 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.4.3 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49136) Eagerly initialize DriverManager on Driver
Cheng Pan created SPARK-49136: - Summary: Eagerly initialize DriverManager on Driver Key: SPARK-49136 URL: https://issues.apache.org/jira/browse/SPARK-49136 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.4.3 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49120) Bump Gson 2.11.0
Cheng Pan created SPARK-49120: - Summary: Bump Gson 2.11.0 Key: SPARK-49120 URL: https://issues.apache.org/jira/browse/SPARK-49120 Project: Spark Issue Type: Dependency upgrade Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49015) Connect Server should respect spark.log.structuredLogging.enabled
Cheng Pan created SPARK-49015: - Summary: Connect Server should respect spark.log.structuredLogging.enabled Key: SPARK-49015 URL: https://issues.apache.org/jira/browse/SPARK-49015 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863386#comment-17863386 ] Cheng Pan edited comment on SPARK-18105 at 7/5/24 4:36 PM: --- there is an upstream XFS kernel bug identified by the Bilibili team that affects the spark shuffle use case, see details at [https://lore.kernel.org/linux-xfs/20220817093627.gz3600...@dread.disaster.area/], disabling `spark.file.transferTo` could also workaround this issue. was (Author: chengpan): there is an XFS kernel bug identified by the Bilibili team that affects the spark shuffle use case, see details at [https://lore.kernel.org/linux-xfs/20220817093627.gz3600...@dread.disaster.area/], disabling `spark.file.transferTo` could also workaround this issue. > LZ4 failed to decompress a stream of shuffled data > -- > > Key: SPARK-18105 > URL: https://issues.apache.org/jira/browse/SPARK-18105 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.1, 3.1.1 >Reporter: Davies Liu >Priority: Major > Attachments: TestWeightedGraph.java > > > When lz4 is used to compress the shuffle files, it may fail to decompress it > as "stream is corrupt" > {code} > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task 92 in stage 5.0 failed 4 times, most recent failure: Lost task 92.3 in > stage 5.0 (TID 16616, 10.0.27.18): java.io.IOException: Stream is corrupted > at > org.apache.spark.io.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:220) > at > org.apache.spark.io.LZ4BlockInputStream.available(LZ4BlockInputStream.java:109) > at java.io.BufferedInputStream.read(BufferedInputStream.java:353) > at java.io.DataInputStream.read(DataInputStream.java:149) > at com.google.common.io.ByteStreams.read(ByteStreams.java:828) > at com.google.common.io.ByteStreams.readFully(ByteStreams.java:695) > at > org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$3$$anon$1.next(UnsafeRowSerializer.scala:127) > at > org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$3$$anon$1.next(UnsafeRowSerializer.scala:110) > at scala.collection.Iterator$$anon$13.next(Iterator.scala:372) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at > org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) > at > org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown > Source) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown > Source) > at > org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) > at > org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370) > at > org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:397) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > https://github.com/jpountz/lz4-java/issues/89 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863386#comment-17863386 ] Cheng Pan commented on SPARK-18105: --- there is an XFS kernel bug identified by the Bilibili team that affects the spark shuffle use case, see details at [https://lore.kernel.org/linux-xfs/20220817093627.gz3600...@dread.disaster.area/], disabling `spark.file.transferTo` could also workaround this issue. > LZ4 failed to decompress a stream of shuffled data > -- > > Key: SPARK-18105 > URL: https://issues.apache.org/jira/browse/SPARK-18105 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.1, 3.1.1 >Reporter: Davies Liu >Priority: Major > Attachments: TestWeightedGraph.java > > > When lz4 is used to compress the shuffle files, it may fail to decompress it > as "stream is corrupt" > {code} > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task 92 in stage 5.0 failed 4 times, most recent failure: Lost task 92.3 in > stage 5.0 (TID 16616, 10.0.27.18): java.io.IOException: Stream is corrupted > at > org.apache.spark.io.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:220) > at > org.apache.spark.io.LZ4BlockInputStream.available(LZ4BlockInputStream.java:109) > at java.io.BufferedInputStream.read(BufferedInputStream.java:353) > at java.io.DataInputStream.read(DataInputStream.java:149) > at com.google.common.io.ByteStreams.read(ByteStreams.java:828) > at com.google.common.io.ByteStreams.readFully(ByteStreams.java:695) > at > org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$3$$anon$1.next(UnsafeRowSerializer.scala:127) > at > org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$3$$anon$1.next(UnsafeRowSerializer.scala:110) > at scala.collection.Iterator$$anon$13.next(Iterator.scala:372) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at > org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) > at > org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown > Source) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown > Source) > at > org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) > at > org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370) > at > org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:397) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > https://github.com/jpountz/lz4-java/issues/89 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48775) Replace SQLContext with SparkSession in STS
[ https://issues.apache.org/jira/browse/SPARK-48775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-48775: -- Summary: Replace SQLContext with SparkSession in STS (was: Replace SQLContext with SparkSession in new introduced API) > Replace SQLContext with SparkSession in STS > --- > > Key: SPARK-48775 > URL: https://issues.apache.org/jira/browse/SPARK-48775 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48765) Enhance default value evaluation for SPARK_IDENT_STRING
Cheng Pan created SPARK-48765: - Summary: Enhance default value evaluation for SPARK_IDENT_STRING Key: SPARK-48765 URL: https://issues.apache.org/jira/browse/SPARK-48765 Project: Spark Issue Type: Improvement Components: Deploy Affects Versions: 3.4.3 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48672) Update Jakarta Servlet reference in security page
Cheng Pan created SPARK-48672: - Summary: Update Jakarta Servlet reference in security page Key: SPARK-48672 URL: https://issues.apache.org/jira/browse/SPARK-48672 Project: Spark Issue Type: Documentation Components: Documentation Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48651) Document configuring different JDK for Spark on YARN
Cheng Pan created SPARK-48651: - Summary: Document configuring different JDK for Spark on YARN Key: SPARK-48651 URL: https://issues.apache.org/jira/browse/SPARK-48651 Project: Spark Issue Type: Documentation Components: Documentation Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48642) False reported SparkOutOfMemoryError caused by killing task on spilling
Cheng Pan created SPARK-48642: - Summary: False reported SparkOutOfMemoryError caused by killing task on spilling Key: SPARK-48642 URL: https://issues.apache.org/jira/browse/SPARK-48642 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.3.1 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48611) Log TID for input split in HadoopRDD and NewHadoopRDD
Cheng Pan created SPARK-48611: - Summary: Log TID for input split in HadoopRDD and NewHadoopRDD Key: SPARK-48611 URL: https://issues.apache.org/jira/browse/SPARK-48611 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48605) UI display driver thread first on the thread dump page
Cheng Pan created SPARK-48605: - Summary: UI display driver thread first on the thread dump page Key: SPARK-48605 URL: https://issues.apache.org/jira/browse/SPARK-48605 Project: Spark Issue Type: Improvement Components: UI Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48603) Update *ParquetReadSchemaSuite to cover type widen capability
Cheng Pan created SPARK-48603: - Summary: Update *ParquetReadSchemaSuite to cover type widen capability Key: SPARK-48603 URL: https://issues.apache.org/jira/browse/SPARK-48603 Project: Spark Issue Type: Test Components: Tests Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48565) Fix thread dump display in UI
Cheng Pan created SPARK-48565: - Summary: Fix thread dump display in UI Key: SPARK-48565 URL: https://issues.apache.org/jira/browse/SPARK-48565 Project: Spark Issue Type: Bug Components: UI Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
[ https://issues.apache.org/jira/browse/SPARK-48238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846913#comment-17846913 ] Cheng Pan commented on SPARK-48238: --- [~dongjoon] [~HF] I opened [https://github.com/apache/spark/pull/46611] to address the YARN incompatible issue by re-implementing a functionally equivalent Filter, please let me know what you think about this approach. > Spark fail to start due to class > o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter > --- > > Key: SPARK-48238 > URL: https://issues.apache.org/jira/browse/SPARK-48238 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Priority: Blocker > Labels: pull-request-available > > I tested the latest master branch, it failed to start on YARN mode > {code:java} > dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code} > > {code:java} > $ bin/spark-sql --master yarn > WARNING: Using incubator modules: jdk.incubator.vector > Setting default log level to "WARN". > To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use > setLogLevel(newLevel). > 2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor > spark.yarn.archive} is set, falling back to uploading libraries under > SPARK_HOME. > 2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext. > org.sparkproject.jetty.util.MultiException: Multiple exceptions > at > org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619) > ~[scala-library-2.13.13.jar:?] > at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617) > ~[scala-library-2.13.13.jar:?] > at scala.collection.AbstractIterable.foreach(Iterable.scala:935) > ~[scala-library-2.13.13.jar:?] > at > org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?] > at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?] > at org.apache.spark.SparkContext.(SparkContext.scala:690) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118) > ~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at scala.Option.getOrElse(Option.scala:201) [scala-library-2.13.13.jar:?] > at > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.
[jira] [Created] (SPARK-48257) Polish POM for Hive dependencies
Cheng Pan created SPARK-48257: - Summary: Polish POM for Hive dependencies Key: SPARK-48257 URL: https://issues.apache.org/jira/browse/SPARK-48257 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48255) Guava should not respect hadoop.deps.scope
Cheng Pan created SPARK-48255: - Summary: Guava should not respect hadoop.deps.scope Key: SPARK-48255 URL: https://issues.apache.org/jira/browse/SPARK-48255 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48254) Enhance Guava version extraction rule in dev/test-dependencies.sh
Cheng Pan created SPARK-48254: - Summary: Enhance Guava version extraction rule in dev/test-dependencies.sh Key: SPARK-48254 URL: https://issues.apache.org/jira/browse/SPARK-48254 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
[ https://issues.apache.org/jira/browse/SPARK-48238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845781#comment-17845781 ] Cheng Pan commented on SPARK-48238: --- [~HF] javax => jakarta migration would be complex on the Hadoop side, as Jetty version couples with Jersey, there is a long standing issue of Jersey upgrading tracked in HADOOP-15984. For reverting, I would suggest reverting to 9.4 instead of 10, see https://github.com/apache/spark/pull/43765#issuecomment-2106600268 > Spark fail to start due to class > o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter > --- > > Key: SPARK-48238 > URL: https://issues.apache.org/jira/browse/SPARK-48238 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Priority: Blocker > > I tested the latest master branch, it failed to start on YARN mode > {code:java} > dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code} > > {code:java} > $ bin/spark-sql --master yarn > WARNING: Using incubator modules: jdk.incubator.vector > Setting default log level to "WARN". > To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use > setLogLevel(newLevel). > 2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor > spark.yarn.archive} is set, falling back to uploading libraries under > SPARK_HOME. > 2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext. > org.sparkproject.jetty.util.MultiException: Multiple exceptions > at > org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619) > ~[scala-library-2.13.13.jar:?] > at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617) > ~[scala-library-2.13.13.jar:?] > at scala.collection.AbstractIterable.foreach(Iterable.scala:935) > ~[scala-library-2.13.13.jar:?] > at > org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?] > at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?] > at org.apache.spark.SparkContext.(SparkContext.scala:690) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963) > ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118) > ~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at scala.Option.getOrElse(Option.scala:201) [scala-library-2.13.13.jar:?] > at > org.apache.s
[jira] [Created] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
Cheng Pan created SPARK-48238: - Summary: Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter Key: SPARK-48238 URL: https://issues.apache.org/jira/browse/SPARK-48238 Project: Spark Issue Type: Bug Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan I tested the latest master branch, it failed to start on YARN mode {code:java} dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code} {code:java} $ bin/spark-sql --master yarn WARNING: Using incubator modules: jdk.incubator.vector Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive} is set, falling back to uploading libraries under SPARK_HOME. 2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext. org.sparkproject.jetty.util.MultiException: Multiple exceptions at org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619) ~[scala-library-2.13.13.jar:?] at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617) ~[scala-library-2.13.13.jar:?] at scala.collection.AbstractIterable.foreach(Iterable.scala:935) ~[scala-library-2.13.13.jar:?] at org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?] at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?] at org.apache.spark.SparkContext.(SparkContext.scala:690) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963) ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118) ~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at scala.Option.getOrElse(Option.scala:201) [scala-library-2.13.13.jar:?] at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1112) [spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:64) [spark-hive-thriftserver_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:405) [spark-hive-thriftserver_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:162) [spark-hive-thriftserver_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) [spark-hive-thriftserver_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at java.base/jdk.internal.reflec
[jira] [Created] (SPARK-48231) Remove unused CodeHaus Jackson dependencies
Cheng Pan created SPARK-48231: - Summary: Remove unused CodeHaus Jackson dependencies Key: SPARK-48231 URL: https://issues.apache.org/jira/browse/SPARK-48231 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48230) Remove unused jodd-core
Cheng Pan created SPARK-48230: - Summary: Remove unused jodd-core Key: SPARK-48230 URL: https://issues.apache.org/jira/browse/SPARK-48230 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48038) Promote driverServiceName to KubernetesDriverConf
[ https://issues.apache.org/jira/browse/SPARK-48038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-48038: -- Summary: Promote driverServiceName to KubernetesDriverConf (was: Promote driverServiceName to DriverServiceFeatureStep) > Promote driverServiceName to KubernetesDriverConf > - > > Key: SPARK-48038 > URL: https://issues.apache.org/jira/browse/SPARK-48038 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48038) Promote driverServiceName to DriverServiceFeatureStep
Cheng Pan created SPARK-48038: - Summary: Promote driverServiceName to DriverServiceFeatureStep Key: SPARK-48038 URL: https://issues.apache.org/jira/browse/SPARK-48038 Project: Spark Issue Type: Improvement Components: Kubernetes Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47974) Remove install_scala from build/mvn
Cheng Pan created SPARK-47974: - Summary: Remove install_scala from build/mvn Key: SPARK-47974 URL: https://issues.apache.org/jira/browse/SPARK-47974 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47954) Support creating ingress entry for external UI access
Cheng Pan created SPARK-47954: - Summary: Support creating ingress entry for external UI access Key: SPARK-47954 URL: https://issues.apache.org/jira/browse/SPARK-47954 Project: Spark Issue Type: Test Components: Kubernetes Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-47869) Upgrade built in hive to Hive-4.0
[ https://issues.apache.org/jira/browse/SPARK-47869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17839669#comment-17839669 ] Cheng Pan commented on SPARK-47869: --- cross link SPARK-44114 > Upgrade built in hive to Hive-4.0 > - > > Key: SPARK-47869 > URL: https://issues.apache.org/jira/browse/SPARK-47869 > Project: Spark > Issue Type: Task > Components: Spark Core >Affects Versions: 3.5.1 >Reporter: Simhadri Govindappa >Priority: Major > > Hive 4.0 has been released. It brings in a lot of new features, bug fixes and > performance improvements. > We would like to update the version of hive used in spark to hive-4.0 > [https://lists.apache.org/thread/2jqpvsx8n801zb5pmlhb8f4zloq27p82] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47928) Speed up test "Add jar support Ivy URI in SQL"
[ https://issues.apache.org/jira/browse/SPARK-47928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-47928: -- Affects Version/s: 3.2.0 (was: 4.0.0) > Speed up test "Add jar support Ivy URI in SQL" > -- > > Key: SPARK-47928 > URL: https://issues.apache.org/jira/browse/SPARK-47928 > Project: Spark > Issue Type: Test > Components: SQL >Affects Versions: 3.2.0 >Reporter: Cheng Pan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47928) Speed up test "Add jar support Ivy URI in SQL"
Cheng Pan created SPARK-47928: - Summary: Speed up test "Add jar support Ivy URI in SQL" Key: SPARK-47928 URL: https://issues.apache.org/jira/browse/SPARK-47928 Project: Spark Issue Type: Test Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47850) Support converting insert for unpartitioned Hive table
Cheng Pan created SPARK-47850: - Summary: Support converting insert for unpartitioned Hive table Key: SPARK-47850 URL: https://issues.apache.org/jira/browse/SPARK-47850 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47847) Deprecate spark.network.remoteReadNioBufferConversion
Cheng Pan created SPARK-47847: - Summary: Deprecate spark.network.remoteReadNioBufferConversion Key: SPARK-47847 URL: https://issues.apache.org/jira/browse/SPARK-47847 Project: Spark Issue Type: Improvement Components: Shuffle, Spark Core Affects Versions: 3.5.2 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47835) Remove switch for remoteReadNioBufferConversion
Cheng Pan created SPARK-47835: - Summary: Remove switch for remoteReadNioBufferConversion Key: SPARK-47835 URL: https://issues.apache.org/jira/browse/SPARK-47835 Project: Spark Issue Type: Improvement Components: Shuffle Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47833) Supply caller stackstrace for checkAndGlobPathIfNecessary AnalysisException
Cheng Pan created SPARK-47833: - Summary: Supply caller stackstrace for checkAndGlobPathIfNecessary AnalysisException Key: SPARK-47833 URL: https://issues.apache.org/jira/browse/SPARK-47833 Project: Spark Issue Type: Improvement Components: Spark Core, SQL Affects Versions: 3.1.3 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47778) Promote `--wait` option to all services
Cheng Pan created SPARK-47778: - Summary: Promote `--wait` option to all services Key: SPARK-47778 URL: https://issues.apache.org/jira/browse/SPARK-47778 Project: Spark Issue Type: New Feature Components: Deploy Affects Versions: 4.0.0 Reporter: Cheng Pan SPARK-47040 add `--wait` support to `start-connect-server.sh` ./sbin/start-connect-server.sh [--wait] [options] In [https://github.com/apache/spark/pull/45852,] we discussed and reached a consensus to promote `–wait` to all service for consistency -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47610) Always set io.netty.tryReflectionSetAccessible=true
Cheng Pan created SPARK-47610: - Summary: Always set io.netty.tryReflectionSetAccessible=true Key: SPARK-47610 URL: https://issues.apache.org/jira/browse/SPARK-47610 Project: Spark Issue Type: Improvement Components: Build, Spark Core Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47103) Make the default storage level of intermediate datasets for MLlib configurable
Cheng Pan created SPARK-47103: - Summary: Make the default storage level of intermediate datasets for MLlib configurable Key: SPARK-47103 URL: https://issues.apache.org/jira/browse/SPARK-47103 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46977) A failed request to obtain a token from one NameNode should not block subsequent token requests
Cheng Pan created SPARK-46977: - Summary: A failed request to obtain a token from one NameNode should not block subsequent token requests Key: SPARK-46977 URL: https://issues.apache.org/jira/browse/SPARK-46977 Project: Spark Issue Type: Improvement Components: Security, Spark Core Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46918) Replace self-defined variables with Hadoop ContainerExitStatus
Cheng Pan created SPARK-46918: - Summary: Replace self-defined variables with Hadoop ContainerExitStatus Key: SPARK-46918 URL: https://issues.apache.org/jira/browse/SPARK-46918 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46695) Always setting hive.execution.engine to mr
Cheng Pan created SPARK-46695: - Summary: Always setting hive.execution.engine to mr Key: SPARK-46695 URL: https://issues.apache.org/jira/browse/SPARK-46695 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46675) Remove unused inferTimestampNTZ in ParquetReadSupport
Cheng Pan created SPARK-46675: - Summary: Remove unused inferTimestampNTZ in ParquetReadSupport Key: SPARK-46675 URL: https://issues.apache.org/jira/browse/SPARK-46675 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46669) Bump Kubernetes Client 6.10.0
[ https://issues.apache.org/jira/browse/SPARK-46669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan resolved SPARK-46669. --- Resolution: Duplicate > Bump Kubernetes Client 6.10.0 > - > > Key: SPARK-46669 > URL: https://issues.apache.org/jira/browse/SPARK-46669 > Project: Spark > Issue Type: Dependency upgrade > Components: k8s >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46669) Bump Kubernetes Client 6.10.0
Cheng Pan created SPARK-46669: - Summary: Bump Kubernetes Client 6.10.0 Key: SPARK-46669 URL: https://issues.apache.org/jira/browse/SPARK-46669 Project: Spark Issue Type: Dependency upgrade Components: k8s Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46499) Bump sbt-eclipse 6.2.0
Cheng Pan created SPARK-46499: - Summary: Bump sbt-eclipse 6.2.0 Key: SPARK-46499 URL: https://issues.apache.org/jira/browse/SPARK-46499 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46420) Remove unused transport from SparkSQLCLIDriver
Cheng Pan created SPARK-46420: - Summary: Remove unused transport from SparkSQLCLIDriver Key: SPARK-46420 URL: https://issues.apache.org/jira/browse/SPARK-46420 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-45937) Fix documentation of spark.executor.maxNumFailures
[ https://issues.apache.org/jira/browse/SPARK-45937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787926#comment-17787926 ] Cheng Pan commented on SPARK-45937: --- [~tgraves] sorry, I missed this ticket and opened a new one SPARK-45969, PR is ready for review https://github.com/apache/spark/pull/43863 > Fix documentation of spark.executor.maxNumFailures > -- > > Key: SPARK-45937 > URL: https://issues.apache.org/jira/browse/SPARK-45937 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.5.0 >Reporter: Thomas Graves >Priority: Critical > > https://issues.apache.org/jira/browse/SPARK-41210 added support for > spark.executor.maxNumFailures on Kubernetes, it made this config generic and > deprecated the yarn version. This config isn't documented and defaults are > not documented. > > [https://github.com/apache/spark/commit/40872e9a094f8459b0b6f626937ced48a8d98efb] > \ > It also added {color:#0a3069}spark.executor.failuresValidityInterval.{color} > > {color:#0a3069}Both need to have default values specified for yarn and k8s, > it also needs to remove the yarn documentation for equivalent configs > spark.yarn.max.executor.failures configuration{color} > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45969) Document configuration change of executor failure tracker
Cheng Pan created SPARK-45969: - Summary: Document configuration change of executor failure tracker Key: SPARK-45969 URL: https://issues.apache.org/jira/browse/SPARK-45969 Project: Spark Issue Type: Documentation Components: Documentation Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45906) Fix error message extraction from ResourceNotFoundException
Cheng Pan created SPARK-45906: - Summary: Fix error message extraction from ResourceNotFoundException Key: SPARK-45906 URL: https://issues.apache.org/jira/browse/SPARK-45906 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45670) SparkSubmit does not support --total-executor-cores when deploying on K8s
Cheng Pan created SPARK-45670: - Summary: SparkSubmit does not support --total-executor-cores when deploying on K8s Key: SPARK-45670 URL: https://issues.apache.org/jira/browse/SPARK-45670 Project: Spark Issue Type: Bug Components: Spark Submit Affects Versions: 3.5.0, 3.4.1, 3.3.3 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45646) Remove hardcoding time variables prior to Hive 2.0
Cheng Pan created SPARK-45646: - Summary: Remove hardcoding time variables prior to Hive 2.0 Key: SPARK-45646 URL: https://issues.apache.org/jira/browse/SPARK-45646 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45389) Correct MetaException matching rule on getting partition metadata
Cheng Pan created SPARK-45389: - Summary: Correct MetaException matching rule on getting partition metadata Key: SPARK-45389 URL: https://issues.apache.org/jira/browse/SPARK-45389 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.3.3 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45388) Eliminate unnecessary reflection invocation in Hive shim classes
Cheng Pan created SPARK-45388: - Summary: Eliminate unnecessary reflection invocation in Hive shim classes Key: SPARK-45388 URL: https://issues.apache.org/jira/browse/SPARK-45388 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45358) Remove shim classes for Hive prior 2.0.0
Cheng Pan created SPARK-45358: - Summary: Remove shim classes for Hive prior 2.0.0 Key: SPARK-45358 URL: https://issues.apache.org/jira/browse/SPARK-45358 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-45004) Adding extension for Spark SQL authorization with Ranger-Hive policies
[ https://issues.apache.org/jira/browse/SPARK-45004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768581#comment-17768581 ] Cheng Pan commented on SPARK-45004: --- [~Ahmadkhani] The Apache Kyuubi community maintains such an extension https://github.com/apache/kyuubi/tree/master/extensions/spark/kyuubi-spark-authz > Adding extension for Spark SQL authorization with Ranger-Hive policies > -- > > Key: SPARK-45004 > URL: https://issues.apache.org/jira/browse/SPARK-45004 > Project: Spark > Issue Type: New Feature > Components: Spark Core, SQL >Affects Versions: 3.4.2 >Reporter: Hasan >Priority: Major > Original Estimate: 24h > Remaining Estimate: 24h > > Apache Ranger have not plugin for Spark SQL authorization and due to this > limitation it's required some extra work and using extensions to manage Spark > authorizations. > > Spark HWC have some performance issues and Cloudera (CDP) solution is complex > and required extra copy of data (big data!). > > Seems natively adding plugins like "Submarine Spark Security Plugin" in Spark > will help to implementation of standard and high performance solution. > Reference: > [https://submarine.apache.org/zh-cn/docs/0.6.0/userDocs/submarine-security/spark-security/] > > > This plugin works fine with table/column/row level and masking options. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45292) Remove Guava from shared classes from IsolatedClientLoader
Cheng Pan created SPARK-45292: - Summary: Remove Guava from shared classes from IsolatedClientLoader Key: SPARK-45292 URL: https://issues.apache.org/jira/browse/SPARK-45292 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45089) Remove obsolete repo of DB2 JDBC driver
Cheng Pan created SPARK-45089: - Summary: Remove obsolete repo of DB2 JDBC driver Key: SPARK-45089 URL: https://issues.apache.org/jira/browse/SPARK-45089 Project: Spark Issue Type: Test Components: Build, Tests Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44938) Change default value of spark.sql.maxSinglePartitionBytes to 128m
Cheng Pan created SPARK-44938: - Summary: Change default value of spark.sql.maxSinglePartitionBytes to 128m Key: SPARK-44938 URL: https://issues.apache.org/jira/browse/SPARK-44938 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44811) Upgrade Guava to 32+
Cheng Pan created SPARK-44811: - Summary: Upgrade Guava to 32+ Key: SPARK-44811 URL: https://issues.apache.org/jira/browse/SPARK-44811 Project: Spark Issue Type: Dependency upgrade Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan It's a long-standing issue, and was reported many times, e.g. SPARK-38262 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44727) Improve the error message for dynamic allocation conditions
Cheng Pan created SPARK-44727: - Summary: Improve the error message for dynamic allocation conditions Key: SPARK-44727 URL: https://issues.apache.org/jira/browse/SPARK-44727 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44677) Drop legacy Hive-based ORC file format
Cheng Pan created SPARK-44677: - Summary: Drop legacy Hive-based ORC file format Key: SPARK-44677 URL: https://issues.apache.org/jira/browse/SPARK-44677 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan Currently, Spark allows to use spark.sql.orc.impl=native/hive to switch the ORC FileFormat implementation. SPARK-23456(2.4) switched the default value of spark.sql.orc.impl from "hive" to "native". and prepared to drop the "hive" implementation in the future. > ... eventually, Apache Spark will drop old Hive-based ORC code. The native implementation works well during the whole Spark 3.x period, so it's a good time to consider dropping the "hive" one in Spark 4.0. Also, we should take care about the backward-compatibility during change. > BTW, IIRC, there was a different at Hive ORC CHAR implementation before. So, > we couldn't remove it for backward-compatibility issues. Since Spark > implements many CHAR features, we need to re-verify that {{native}} > implementation has all legacy Hive-based ORC features -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44669) Parquet/ORC files written using Hive Serde should has file extension
Cheng Pan created SPARK-44669: - Summary: Parquet/ORC files written using Hive Serde should has file extension Key: SPARK-44669 URL: https://issues.apache.org/jira/browse/SPARK-44669 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.3.2 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44525) Improve error message when Invoke method is not found
Cheng Pan created SPARK-44525: - Summary: Improve error message when Invoke method is not found Key: SPARK-44525 URL: https://issues.apache.org/jira/browse/SPARK-44525 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.3.2 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44489) Make InsertIntoDataSourceDirCommand extends DataWritingCommand
[ https://issues.apache.org/jira/browse/SPARK-44489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan resolved SPARK-44489. --- Resolution: Not A Problem > Make InsertIntoDataSourceDirCommand extends DataWritingCommand > -- > > Key: SPARK-44489 > URL: https://issues.apache.org/jira/browse/SPARK-44489 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.5.0 >Reporter: Cheng Pan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44489) Make InsertIntoDataSourceDirCommand extends DataWritingCommand
Cheng Pan created SPARK-44489: - Summary: Make InsertIntoDataSourceDirCommand extends DataWritingCommand Key: SPARK-44489 URL: https://issues.apache.org/jira/browse/SPARK-44489 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44204) Add missing recordHiveCall for getPartitionNames
Cheng Pan created SPARK-44204: - Summary: Add missing recordHiveCall for getPartitionNames Key: SPARK-44204 URL: https://issues.apache.org/jira/browse/SPARK-44204 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.3.2 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44180) DistributionAndOrderingUtils should apply ResolveTimeZone
Cheng Pan created SPARK-44180: - Summary: DistributionAndOrderingUtils should apply ResolveTimeZone Key: SPARK-44180 URL: https://issues.apache.org/jira/browse/SPARK-44180 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.4.1 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44070) Bump snappy-java 1.1.10.1
Cheng Pan created SPARK-44070: - Summary: Bump snappy-java 1.1.10.1 Key: SPARK-44070 URL: https://issues.apache.org/jira/browse/SPARK-44070 Project: Spark Issue Type: Dependency upgrade Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43880) Organize hadoop-cloud in standard maven project structure
[ https://issues.apache.org/jira/browse/SPARK-43880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43880: -- Parent: SPARK-43197 Issue Type: Sub-task (was: Improvement) > Organize hadoop-cloud in standard maven project structure > - > > Key: SPARK-43880 > URL: https://issues.apache.org/jira/browse/SPARK-43880 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.5.0 >Reporter: Cheng Pan >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43880) Organize hadoop-cloud in standard maven project structure
Cheng Pan created SPARK-43880: - Summary: Organize hadoop-cloud in standard maven project structure Key: SPARK-43880 URL: https://issues.apache.org/jira/browse/SPARK-43880 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43751) Document for unbase64 behavior change
Cheng Pan created SPARK-43751: - Summary: Document for unbase64 behavior change Key: SPARK-43751 URL: https://issues.apache.org/jira/browse/SPARK-43751 Project: Spark Issue Type: Documentation Components: Documentation, SQL Affects Versions: 3.3.3, 3.4.1, 3.5.0 Reporter: Cheng Pan Document behavior change caused by https://issues.apache.org/jira/browse/SPARK-37820 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43575) Exclude duplicated classes from kafka assembly jar
Cheng Pan created SPARK-43575: - Summary: Exclude duplicated classes from kafka assembly jar Key: SPARK-43575 URL: https://issues.apache.org/jira/browse/SPARK-43575 Project: Spark Issue Type: Improvement Components: Build, Structured Streaming Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43519) Bump Parquet 1.13.1
Cheng Pan created SPARK-43519: - Summary: Bump Parquet 1.13.1 Key: SPARK-43519 URL: https://issues.apache.org/jira/browse/SPARK-43519 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43489) Remove protobuf 2.5.0
Cheng Pan created SPARK-43489: - Summary: Remove protobuf 2.5.0 Key: SPARK-43489 URL: https://issues.apache.org/jira/browse/SPARK-43489 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43484) Kafka/Kinesis Assembly should not package hadoop-client-runtime
[ https://issues.apache.org/jira/browse/SPARK-43484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43484: -- Summary: Kafka/Kinesis Assembly should not package hadoop-client-runtime (was: Kafka Assembly should not package hadoop-client-runtime) > Kafka/Kinesis Assembly should not package hadoop-client-runtime > --- > > Key: SPARK-43484 > URL: https://issues.apache.org/jira/browse/SPARK-43484 > Project: Spark > Issue Type: Bug > Components: Build, Structured Streaming >Affects Versions: 3.2.4, 3.3.2, 3.4.0, 3.5.0 >Reporter: Cheng Pan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43484) Kafka Assembly should not package hadoop-client-runtime
Cheng Pan created SPARK-43484: - Summary: Kafka Assembly should not package hadoop-client-runtime Key: SPARK-43484 URL: https://issues.apache.org/jira/browse/SPARK-43484 Project: Spark Issue Type: Bug Components: Build, Structured Streaming Affects Versions: 3.4.0, 3.3.2, 3.2.4, 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43455) Bump kubernetes-client 6.6.1
Cheng Pan created SPARK-43455: - Summary: Bump kubernetes-client 6.6.1 Key: SPARK-43455 URL: https://issues.apache.org/jira/browse/SPARK-43455 Project: Spark Issue Type: Improvement Components: Build, Kubernetes Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43448) Remove dummy hadoop-openstack
Cheng Pan created SPARK-43448: - Summary: Remove dummy hadoop-openstack Key: SPARK-43448 URL: https://issues.apache.org/jira/browse/SPARK-43448 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43395) Excluding macOS tar extended metadata in make-distribution.sh
Cheng Pan created SPARK-43395: - Summary: Excluding macOS tar extended metadata in make-distribution.sh Key: SPARK-43395 URL: https://issues.apache.org/jira/browse/SPARK-43395 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.4.0, 3.3.2, 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43395) Exclude macOS tar extended metadata in make-distribution.sh
[ https://issues.apache.org/jira/browse/SPARK-43395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43395: -- Summary: Exclude macOS tar extended metadata in make-distribution.sh (was: Excluding macOS tar extended metadata in make-distribution.sh) > Exclude macOS tar extended metadata in make-distribution.sh > --- > > Key: SPARK-43395 > URL: https://issues.apache.org/jira/browse/SPARK-43395 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.3.2, 3.4.0, 3.5.0 >Reporter: Cheng Pan >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43394) Upgrade maven to 3.8.8
[ https://issues.apache.org/jira/browse/SPARK-43394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43394: -- Priority: Trivial (was: Major) > Upgrade maven to 3.8.8 > -- > > Key: SPARK-43394 > URL: https://issues.apache.org/jira/browse/SPARK-43394 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.5.0 >Reporter: Cheng Pan >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43394) Upgrade maven to 3.8.8
Cheng Pan created SPARK-43394: - Summary: Upgrade maven to 3.8.8 Key: SPARK-43394 URL: https://issues.apache.org/jira/browse/SPARK-43394 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43390) DSv2 allows CTAS/RTAS to reserve schema nullability
Cheng Pan created SPARK-43390: - Summary: DSv2 allows CTAS/RTAS to reserve schema nullability Key: SPARK-43390 URL: https://issues.apache.org/jira/browse/SPARK-43390 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43356) Migrate deprecated createOrReplace to serverSideApply
[ https://issues.apache.org/jira/browse/SPARK-43356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43356: -- Description: {{public interface CreateOrReplaceable extends Replaceable {}} {{ /**}} {{ * Creates a provided resource in a Kubernetes Cluster. If creation}} {{ * fails with a HTTP_CONFLICT, it tries to replace resource.}} {{ *}} {{ * @return created item returned in kubernetes api response}} {{ *}} {{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()} or attempt a create and edit/patch operation.}} {{ */}} {{ @Deprecated}} {{ T createOrReplace();}} {{ /**}} {{ * Creates an item}} {{ *}} {{ * @return the item from the api server}} {{ */}} {{ T create();}} {{}}} was: {{public interface CreateOrReplaceable extends Replaceable {}}{{ /**}} {{ * Creates a provided resource in a Kubernetes Cluster. If creation}} {{ * fails with a HTTP_CONFLICT, it tries to replace resource.}} {{ *}} {{ * @return created item returned in kubernetes api response}} {{ *}} {{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()} or attempt a create and edit/patch operation.}} {{ */}} {{ @Deprecated}} {{ T createOrReplace();}} {{ /**}} {{ * Creates an item}} {{ *}} {{ * @return the item from the api server}} {{ */}} {{ T create();}} {{}}} > Migrate deprecated createOrReplace to serverSideApply > - > > Key: SPARK-43356 > URL: https://issues.apache.org/jira/browse/SPARK-43356 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.5.0 >Reporter: Cheng Pan >Priority: Major > > > > {{public interface CreateOrReplaceable extends Replaceable {}} > {{ /**}} > {{ * Creates a provided resource in a Kubernetes Cluster. If creation}} > {{ * fails with a HTTP_CONFLICT, it tries to replace resource.}} > {{ *}} > {{ * @return created item returned in kubernetes api response}} > {{ *}} > {{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()} > or attempt a create and edit/patch operation.}} > {{ */}} > {{ @Deprecated}} > {{ T createOrReplace();}} > > {{ /**}} > {{ * Creates an item}} > {{ *}} > {{ * @return the item from the api server}} > {{ */}} > {{ T create();}} > {{}}} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43356) Migrate deprecated createOrReplace to serverSideApply
[ https://issues.apache.org/jira/browse/SPARK-43356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43356: -- Description: {{public interface CreateOrReplaceable extends Replaceable {}}{{ /**}} {{ * Creates a provided resource in a Kubernetes Cluster. If creation}} {{ * fails with a HTTP_CONFLICT, it tries to replace resource.}} {{ *}} {{ * @return created item returned in kubernetes api response}} {{ *}} {{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()} or attempt a create and edit/patch operation.}} {{ */}} {{ @Deprecated}} {{ T createOrReplace();}} {{ /**}} {{ * Creates an item}} {{ *}} {{ * @return the item from the api server}} {{ */}} {{ T create();}} {{}}} was: {{public interface CreateOrReplaceable extends Replaceable {}}{{ /**}} {{ * Creates a provided resource in a Kubernetes Cluster. If creation}} {{ * fails with a HTTP_CONFLICT, it tries to replace resource.}} {{ *}} {{ * @return created item returned in kubernetes api response}} {{ *}} {{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()} or attempt a create and edit/patch operation.}} {{ */}} {{ @Deprecated}} {{ T createOrReplace();}}{{ /**}} {{ * Creates an item}} {{ *}} {{ * @return the item from the api server}} {{ */}} {{ T create();}} {{}}} > Migrate deprecated createOrReplace to serverSideApply > - > > Key: SPARK-43356 > URL: https://issues.apache.org/jira/browse/SPARK-43356 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.5.0 >Reporter: Cheng Pan >Priority: Major > > > > {{public interface CreateOrReplaceable extends Replaceable {}}{{ /**}} > {{ * Creates a provided resource in a Kubernetes Cluster. If creation}} > {{ * fails with a HTTP_CONFLICT, it tries to replace resource.}} > {{ *}} > {{ * @return created item returned in kubernetes api response}} > {{ *}} > {{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()} > or attempt a create and edit/patch operation.}} > {{ */}} > {{ @Deprecated}} > {{ T createOrReplace();}} > > {{ /**}} > {{ * Creates an item}} > {{ *}} > {{ * @return the item from the api server}} > {{ */}} > {{ T create();}} > {{}}} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43356) Migrate deprecated createOrReplace to serverSideApply
Cheng Pan created SPARK-43356: - Summary: Migrate deprecated createOrReplace to serverSideApply Key: SPARK-43356 URL: https://issues.apache.org/jira/browse/SPARK-43356 Project: Spark Issue Type: Improvement Components: Kubernetes Affects Versions: 3.5.0 Reporter: Cheng Pan {{public interface CreateOrReplaceable extends Replaceable {}}{{ /**}} {{ * Creates a provided resource in a Kubernetes Cluster. If creation}} {{ * fails with a HTTP_CONFLICT, it tries to replace resource.}} {{ *}} {{ * @return created item returned in kubernetes api response}} {{ *}} {{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()} or attempt a create and edit/patch operation.}} {{ */}} {{ @Deprecated}} {{ T createOrReplace();}}{{ /**}} {{ * Creates an item}} {{ *}} {{ * @return the item from the api server}} {{ */}} {{ T create();}} {{}}} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43355) Upgrade fabric8io/kubernetes-client 6.6.0
Cheng Pan created SPARK-43355: - Summary: Upgrade fabric8io/kubernetes-client 6.6.0 Key: SPARK-43355 URL: https://issues.apache.org/jira/browse/SPARK-43355 Project: Spark Issue Type: Dependency upgrade Components: Build, Kubernetes Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43352) `setPythonSparkConfProperties` should use conf instead of sparkAppConf
Cheng Pan created SPARK-43352: - Summary: `setPythonSparkConfProperties` should use conf instead of sparkAppConf Key: SPARK-43352 URL: https://issues.apache.org/jira/browse/SPARK-43352 Project: Spark Issue Type: Test Components: Kubernetes, Tests Affects Versions: 3.2.4 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43320) Directly call Hive 2.3.9 API
Cheng Pan created SPARK-43320: - Summary: Directly call Hive 2.3.9 API Key: SPARK-43320 URL: https://issues.apache.org/jira/browse/SPARK-43320 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43319) Remove usage of deprecated DefaultKubernetesClient
Cheng Pan created SPARK-43319: - Summary: Remove usage of deprecated DefaultKubernetesClient Key: SPARK-43319 URL: https://issues.apache.org/jira/browse/SPARK-43319 Project: Spark Issue Type: Test Components: Kubernetes, Tests Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43248) Unnecessary serialize/deserialize of Path on parallel gather partition stats
Cheng Pan created SPARK-43248: - Summary: Unnecessary serialize/deserialize of Path on parallel gather partition stats Key: SPARK-43248 URL: https://issues.apache.org/jira/browse/SPARK-43248 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43211) Remove Hadoop2 support in IsolatedClientLoader
Cheng Pan created SPARK-43211: - Summary: Remove Hadoop2 support in IsolatedClientLoader Key: SPARK-43211 URL: https://issues.apache.org/jira/browse/SPARK-43211 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43208) IsolatedClassLoader should close barrier class InputStream after reading
Cheng Pan created SPARK-43208: - Summary: IsolatedClassLoader should close barrier class InputStream after reading Key: SPARK-43208 URL: https://issues.apache.org/jira/browse/SPARK-43208 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.3.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43202) Replace reflection w/ direct calling for YARN Resource API
Cheng Pan created SPARK-43202: - Summary: Replace reflection w/ direct calling for YARN Resource API Key: SPARK-43202 URL: https://issues.apache.org/jira/browse/SPARK-43202 Project: Spark Issue Type: Sub-task Components: YARN Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org