Re: maven build failing in spark sql w/BouncyCastleProvider CNFE

2022-12-05 Thread Yang,Jie(INF) 
Steve, did compile failed happen when mvn build Spark master with hadoop 
3.4.0-SNAPSHOT?

发件人: Hyukjin Kwon 
日期: 2022年12月6日 星期二 10:27
抄送: Apache Spark Dev 
主题: Re: maven build failing in spark sql w/BouncyCastleProvider CNFE

Steve, does the lower version of scala plugin work for you? If that solves, we 
could temporary downgrade for now.

On Mon, 5 Dec 2022 at 22:23, Steve Loughran  wrote:
 trying to build spark master w/ hadoop trunk and the maven sbt plugin is 
failing. This doesn't happen with the 3.3.5 RC0;

I note that the only mention of this anywhere was me in march.

clearly something in hadoop trunk has changed in a way which is incompatible.

Has anyone else tried such a build/seen this problem? any suggestions of a fix?

Created SPARK-41392 to cover this...

[INFO] 
[ERROR] Failed to execute goal 
net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile 
(scala-test-compile-first) on project spark-sql_2.12: Execution 
scala-test-compile-first of goal 
net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile failed: A required 
class was missing while executing 
net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile: 
org/bouncycastle/jce/provider/BouncyCastleProvider
[ERROR] -
[ERROR] realm =plugin>net.alchim31.maven:scala-maven-plugin:4.7.2
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = 
file:/Users/stevel/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.7.2/scala-maven-plugin-4.7.2.jar
[ERROR] urls[1] = 
file:/Users/stevel/.m2/repository/org/apache/maven/shared/maven-dependency-tree/3.2.0/maven-dependency-tree-3.2.0.jar
[ERROR] urls[2] = 
file:/Users/stevel/.m2/repository/org/eclipse/aether/aether-util/1.0.0.v20140518/aether-util-1.0.0.v20140518.jar
[ERROR] urls[3] = 
file:/Users/stevel/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.1.1/maven-reporting-api-3.1.1.jar
[ERROR] urls[4] = 
file:/Users/stevel/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.11.1/doxia-sink-api-1.11.1.jar
[ERROR] urls[5] = 
file:/Users/stevel/.m2/repository/org/apache/maven/doxia/doxia-logging-api/1.11.1/doxia-logging-api-1.11.1.jar
[ERROR] urls[6] = 
file:/Users/stevel/.m2/repository/org/apache/maven/maven-archiver/3.6.0/maven-archiver-3.6.0.jar
[ERROR] urls[7] = 
file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-io/3.4.0/plexus-io-3.4.0.jar
[ERROR] urls[8] = 
file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.jar
[ERROR] urls[9] = 
file:/Users/stevel/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar
[ERROR] urls[10] = 
file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-utils/3.4.2/plexus-utils-3.4.2.jar
[ERROR] urls[11] = 
file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-archiver/4.5.0/plexus-archiver-4.5.0.jar
[ERROR] urls[12] = 
file:/Users/stevel/.m2/repository/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar
[ERROR] urls[13] = 
file:/Users/stevel/.m2/repository/org/apache/commons/commons-compress/1.21/commons-compress-1.21.jar
[ERROR] urls[14] = 
file:/Users/stevel/.m2/repository/org/iq80/snappy/snappy/0.4/snappy-0.4.jar
[ERROR] urls[15] = 
file:/Users/stevel/.m2/repository/org/tukaani/xz/1.9/xz-1.9.jar
[ERROR] urls[16] = 
file:/Users/stevel/.m2/repository/com/github/luben/zstd-jni/1.5.2-4/zstd-jni-1.5.2-4.jar
[ERROR] urls[17] = 
file:/Users/stevel/.m2/repository/org/scala-sbt/zinc_2.13/1.7.1/zinc_2.13-1.7.1.jar
[ERROR] urls[18] = 
file:/Users/stevel/.m2/repository/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar
[ERROR] urls[19] = 
file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-core_2.13/1.7.1/zinc-core_2.13-1.7.1.jar
[ERROR] urls[20] = 
file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-apiinfo_2.13/1.7.1/zinc-apiinfo_2.13-1.7.1.jar
[ERROR] urls[21] = 
file:/Users/stevel/.m2/repository/org/scala-sbt/compiler-bridge_2.13/1.7.1/compiler-bridge_2.13-1.7.1.jar
[ERROR] urls[22] = 
file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-classpath_2.13/1.7.1/zinc-classpath_2.13-1.7.1.jar
[ERROR] urls[23] = 
file:/Users/stevel/.m2/repository/org/scala-lang/scala-compiler/2.13.8/scala-compiler-2.13.8.jar
[ERROR] urls[24] = 
file:/Users/stevel/.m2/repository/org/scala-sbt/compiler-interface/1.7.1/compiler-interface-1.7.1.jar
[ERROR] urls[25] = 
file:/Users/stevel/.m2/repository/org/scala-sbt/util-interface/1.7.0/util-interface-1.7.0.jar
[ERROR] urls[26] = 
file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-persist-core-assembly/1.7.1/zinc-persist-core-assembly-1.7.1.jar
[ERROR] urls[27] = 
file:/Users/stevel/.m2/repository/org/scala-lang/modules/scala-parallel-collections_2.13/0.2.0/scala-parallel-collections_2.13-0.2.0.jar
[ERROR] urls[28] = 
file:/Users/stevel/.m2/repository/org/scala-sbt/io_2.13/1.7.0/io_2.13-1.7.0.jar
[ERROR] urls[29] = 
file:/Users/stevel/.m2/repository/com/swoval/f

Re: maven build failing in spark sql w/BouncyCastleProvider CNFE

2022-12-05 Thread Hyukjin Kwon 
Steve, does the lower version of scala plugin work for you? If that solves,
we could temporary downgrade for now.

On Mon, 5 Dec 2022 at 22:23, Steve Loughran 
wrote:

>  trying to build spark master w/ hadoop trunk and the maven sbt plugin is
> failing. This doesn't happen with the 3.3.5 RC0;
>
> I note that the only mention of this anywhere was me in march.
>
> clearly something in hadoop trunk has changed in a way which is
> incompatible.
>
> Has anyone else tried such a build/seen this problem? any suggestions of a
> fix?
>
> Created SPARK-41392 to cover this...
>
> [INFO]
> 
> [ERROR] Failed to execute goal
> net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile
> (scala-test-compile-first) on project spark-sql_2.12: Execution
> scala-test-compile-first of goal
> net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile failed: A required
> class was missing while executing
> net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile:
> org/bouncycastle/jce/provider/BouncyCastleProvider
> [ERROR] -
> [ERROR] realm =plugin>net.alchim31.maven:scala-maven-plugin:4.7.2
> [ERROR] strategy =
> org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
> [ERROR] urls[0] =
> file:/Users/stevel/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.7.2/scala-maven-plugin-4.7.2.jar
> [ERROR] urls[1] =
> file:/Users/stevel/.m2/repository/org/apache/maven/shared/maven-dependency-tree/3.2.0/maven-dependency-tree-3.2.0.jar
> [ERROR] urls[2] =
> file:/Users/stevel/.m2/repository/org/eclipse/aether/aether-util/1.0.0.v20140518/aether-util-1.0.0.v20140518.jar
> [ERROR] urls[3] =
> file:/Users/stevel/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.1.1/maven-reporting-api-3.1.1.jar
> [ERROR] urls[4] =
> file:/Users/stevel/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.11.1/doxia-sink-api-1.11.1.jar
> [ERROR] urls[5] =
> file:/Users/stevel/.m2/repository/org/apache/maven/doxia/doxia-logging-api/1.11.1/doxia-logging-api-1.11.1.jar
> [ERROR] urls[6] =
> file:/Users/stevel/.m2/repository/org/apache/maven/maven-archiver/3.6.0/maven-archiver-3.6.0.jar
> [ERROR] urls[7] =
> file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-io/3.4.0/plexus-io-3.4.0.jar
> [ERROR] urls[8] =
> file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.jar
> [ERROR] urls[9] =
> file:/Users/stevel/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar
> [ERROR] urls[10] =
> file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-utils/3.4.2/plexus-utils-3.4.2.jar
> [ERROR] urls[11] =
> file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-archiver/4.5.0/plexus-archiver-4.5.0.jar
> [ERROR] urls[12] =
> file:/Users/stevel/.m2/repository/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar
> [ERROR] urls[13] =
> file:/Users/stevel/.m2/repository/org/apache/commons/commons-compress/1.21/commons-compress-1.21.jar
> [ERROR] urls[14] =
> file:/Users/stevel/.m2/repository/org/iq80/snappy/snappy/0.4/snappy-0.4.jar
> [ERROR] urls[15] =
> file:/Users/stevel/.m2/repository/org/tukaani/xz/1.9/xz-1.9.jar
> [ERROR] urls[16] =
> file:/Users/stevel/.m2/repository/com/github/luben/zstd-jni/1.5.2-4/zstd-jni-1.5.2-4.jar
> [ERROR] urls[17] =
> file:/Users/stevel/.m2/repository/org/scala-sbt/zinc_2.13/1.7.1/zinc_2.13-1.7.1.jar
> [ERROR] urls[18] =
> file:/Users/stevel/.m2/repository/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar
> [ERROR] urls[19] =
> file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-core_2.13/1.7.1/zinc-core_2.13-1.7.1.jar
> [ERROR] urls[20] =
> file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-apiinfo_2.13/1.7.1/zinc-apiinfo_2.13-1.7.1.jar
> [ERROR] urls[21] =
> file:/Users/stevel/.m2/repository/org/scala-sbt/compiler-bridge_2.13/1.7.1/compiler-bridge_2.13-1.7.1.jar
> [ERROR] urls[22] =
> file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-classpath_2.13/1.7.1/zinc-classpath_2.13-1.7.1.jar
> [ERROR] urls[23] =
> file:/Users/stevel/.m2/repository/org/scala-lang/scala-compiler/2.13.8/scala-compiler-2.13.8.jar
> [ERROR] urls[24] =
> file:/Users/stevel/.m2/repository/org/scala-sbt/compiler-interface/1.7.1/compiler-interface-1.7.1.jar
> [ERROR] urls[25] =
> file:/Users/stevel/.m2/repository/org/scala-sbt/util-interface/1.7.0/util-interface-1.7.0.jar
> [ERROR] urls[26] =
> file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-persist-core-assembly/1.7.1/zinc-persist-core-assembly-1.7.1.jar
> [ERROR] urls[27] =
> file:/Users/stevel/.m2/repository/org/scala-lang/modules/scala-parallel-collections_2.13/0.2.0/scala-parallel-collections_2.13-0.2.0.jar
> [ERROR] urls[28] =
> file:/Users/stevel/.m2/repository/org/scala-sbt/io_2.13/1.7.0/io_2.13-1.7.0.jar
> [ERROR] urls[29] =
> file:/Users/stevel/.m2/repository/com/swoval/file-tree-views/2.1.9/file-tree-views-2.1.9.jar
> [ERROR] urls[30] =
> file:/Users/stevel/.m2/rep

Re: [VOTE][RESULT][SPIP] Asynchronous Offset Management in Structured Streaming

2022-12-05 Thread Jerry Peng 
Thanks Jungtaek for shepherding this effort!

On Sun, Dec 4, 2022 at 6:25 PM Jungtaek Lim 
wrote:

> The vote passes with 7 +1s (5 binding +1s).
> Thanks to all who reviews the SPIP doc and votes!
>
> (* = binding)
> +1:
> - Jungtaek Lim
> - Xingbo Jiang
> - Mridul Muralidharan (*)
> - Hyukjin Kwon (*)
> - Shixiong Zhu (*)
> - Wenchen Fan (*)
> - Dongjoon Hyun (*)
>
> +0: None
>
> -1: None
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>

maven build failing in spark sql w/BouncyCastleProvider CNFE

2022-12-05 Thread Steve Loughran 
 trying to build spark master w/ hadoop trunk and the maven sbt plugin is
failing. This doesn't happen with the 3.3.5 RC0;

I note that the only mention of this anywhere was me in march.

clearly something in hadoop trunk has changed in a way which is
incompatible.

Has anyone else tried such a build/seen this problem? any suggestions of a
fix?

Created SPARK-41392 to cover this...

[INFO]

[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile
(scala-test-compile-first) on project spark-sql_2.12: Execution
scala-test-compile-first of goal
net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile failed: A required
class was missing while executing
net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile:
org/bouncycastle/jce/provider/BouncyCastleProvider
[ERROR] -
[ERROR] realm =plugin>net.alchim31.maven:scala-maven-plugin:4.7.2
[ERROR] strategy =
org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] =
file:/Users/stevel/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.7.2/scala-maven-plugin-4.7.2.jar
[ERROR] urls[1] =
file:/Users/stevel/.m2/repository/org/apache/maven/shared/maven-dependency-tree/3.2.0/maven-dependency-tree-3.2.0.jar
[ERROR] urls[2] =
file:/Users/stevel/.m2/repository/org/eclipse/aether/aether-util/1.0.0.v20140518/aether-util-1.0.0.v20140518.jar
[ERROR] urls[3] =
file:/Users/stevel/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.1.1/maven-reporting-api-3.1.1.jar
[ERROR] urls[4] =
file:/Users/stevel/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.11.1/doxia-sink-api-1.11.1.jar
[ERROR] urls[5] =
file:/Users/stevel/.m2/repository/org/apache/maven/doxia/doxia-logging-api/1.11.1/doxia-logging-api-1.11.1.jar
[ERROR] urls[6] =
file:/Users/stevel/.m2/repository/org/apache/maven/maven-archiver/3.6.0/maven-archiver-3.6.0.jar
[ERROR] urls[7] =
file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-io/3.4.0/plexus-io-3.4.0.jar
[ERROR] urls[8] =
file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.jar
[ERROR] urls[9] =
file:/Users/stevel/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar
[ERROR] urls[10] =
file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-utils/3.4.2/plexus-utils-3.4.2.jar
[ERROR] urls[11] =
file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-archiver/4.5.0/plexus-archiver-4.5.0.jar
[ERROR] urls[12] =
file:/Users/stevel/.m2/repository/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar
[ERROR] urls[13] =
file:/Users/stevel/.m2/repository/org/apache/commons/commons-compress/1.21/commons-compress-1.21.jar
[ERROR] urls[14] =
file:/Users/stevel/.m2/repository/org/iq80/snappy/snappy/0.4/snappy-0.4.jar
[ERROR] urls[15] =
file:/Users/stevel/.m2/repository/org/tukaani/xz/1.9/xz-1.9.jar
[ERROR] urls[16] =
file:/Users/stevel/.m2/repository/com/github/luben/zstd-jni/1.5.2-4/zstd-jni-1.5.2-4.jar
[ERROR] urls[17] =
file:/Users/stevel/.m2/repository/org/scala-sbt/zinc_2.13/1.7.1/zinc_2.13-1.7.1.jar
[ERROR] urls[18] =
file:/Users/stevel/.m2/repository/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar
[ERROR] urls[19] =
file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-core_2.13/1.7.1/zinc-core_2.13-1.7.1.jar
[ERROR] urls[20] =
file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-apiinfo_2.13/1.7.1/zinc-apiinfo_2.13-1.7.1.jar
[ERROR] urls[21] =
file:/Users/stevel/.m2/repository/org/scala-sbt/compiler-bridge_2.13/1.7.1/compiler-bridge_2.13-1.7.1.jar
[ERROR] urls[22] =
file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-classpath_2.13/1.7.1/zinc-classpath_2.13-1.7.1.jar
[ERROR] urls[23] =
file:/Users/stevel/.m2/repository/org/scala-lang/scala-compiler/2.13.8/scala-compiler-2.13.8.jar
[ERROR] urls[24] =
file:/Users/stevel/.m2/repository/org/scala-sbt/compiler-interface/1.7.1/compiler-interface-1.7.1.jar
[ERROR] urls[25] =
file:/Users/stevel/.m2/repository/org/scala-sbt/util-interface/1.7.0/util-interface-1.7.0.jar
[ERROR] urls[26] =
file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-persist-core-assembly/1.7.1/zinc-persist-core-assembly-1.7.1.jar
[ERROR] urls[27] =
file:/Users/stevel/.m2/repository/org/scala-lang/modules/scala-parallel-collections_2.13/0.2.0/scala-parallel-collections_2.13-0.2.0.jar
[ERROR] urls[28] =
file:/Users/stevel/.m2/repository/org/scala-sbt/io_2.13/1.7.0/io_2.13-1.7.0.jar
[ERROR] urls[29] =
file:/Users/stevel/.m2/repository/com/swoval/file-tree-views/2.1.9/file-tree-views-2.1.9.jar
[ERROR] urls[30] =
file:/Users/stevel/.m2/repository/net/java/dev/jna/jna/5.12.0/jna-5.12.0.jar
[ERROR] urls[31] =
file:/Users/stevel/.m2/repository/net/java/dev/jna/jna-platform/5.12.0/jna-platform-5.12.0.jar
[ERROR] urls[32] =
file:/Users/stevel/.m2/repository/org/scala-sbt/util-logging_2.13/1.7.0/util-logging_2.13-1.7.0.jar
[ERROR] urls[33] =
file:/Users/stevel/.m2/repository/org/