Re: Problem building spark-catalyst_2.12 with Maven
I've found the problem! It was indeed a local thingy! $ cat ~/.mavenrc MAVEN_OPTS='-XX:+TieredCompilation -XX:TieredStopAtLevel=1' I've added this some time ago. It optimizes the build time. But it seems it also overrides the env var MAVEN_OPTS... Now it fails with: [INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-catalyst_2.12 --- [INFO] Using incremental compilation using Mixed compile order [INFO] Compiler bridge file: /home/martin/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.15__52.0-1.3.1_20191012T045515.jar [INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.15,1.7.6,null) [INFO] Compiling 372 Scala sources and 171 Java sources to /home/martin/git/apache/spark/sql/catalyst/target/scala-2.12/classes ... [ERROR] [Error] : error writing /home/martin/git/apache/spark/sql/catalyst/target/scala-2.12/classes/org/apache/spark/sql/catalyst/analysis/Analyzer$ResolveGroupingAnalytics$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveGroupingAnalytics$$replaceGroupingFunc$1.class: java.nio.file.FileSystemException /home/martin/git/apache/spark/sql/catalyst/target/scala-2.12/classes/org/apache/spark/sql/catalyst/analysis/Analyzer$ResolveGroupingAnalytics$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveGroupingAnalytics$$replaceGroupingFunc$1.class: File name too long but this is well documented: https://spark.apache.org/docs/latest/building-spark.html#encrypted-filesystems All works now! Thank you, Sean! On Thu, Feb 10, 2022 at 10:13 PM Sean Owen wrote: > I think it's another occurrence that I had to change or had to set > MAVEN_OPTS. I think this occurs in a way that this setting doesn't affect, > though I don't quite understand it. Try the stack size in test runner > configs > > On Thu, Feb 10, 2022, 2:02 PM Martin Grigorov > wrote: > >> Hi Sean, >> >> On Thu, Feb 10, 2022 at 5:37 PM Sean Owen wrote: >> >>> Yes I've seen this; the JVM stack size needs to be increased. I'm not >>> sure if it's env specific (though you and I at least have hit it, I think >>> others), or whether we need to change our build script. >>> In the pom.xml file, find "-Xss..." settings and make them something >>> like "-Xss4m", see if that works. >>> >> >> It is already a much bigger value - 128m ( >> https://github.com/apache/spark/blob/50256bde9bdf217413545a6d2945d6c61bf4cfff/pom.xml#L2845 >> ) >> I've tried smaller and bigger values for all jvmArgs next to this one. >> None helped! >> I also have the feeling it is something in my environment that overrides >> these values but so far I cannot identify anything. >> >> >> >>> >>> On Thu, Feb 10, 2022 at 8:54 AM Martin Grigorov >>> wrote: >>> Hi, I am not able to build Spark due to the following error : ERROR] ## Exception when compiling 543 sources to /home/martin/git/apache/spark/sql/catalyst/target/scala-2.12/classes java.lang.BootstrapMethodError: call site initialization exception java.lang.invoke.CallSite.makeSite(CallSite.java:341) java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307) java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297) scala.tools.nsc.typechecker.Typers$Typer.typedBlock(Typers.scala:2504) scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$103(Typers.scala:5711) scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1(Typers.scala:500) scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5746) scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5781) ... Caused by: java.lang.StackOverflowError at java.lang.ref.Reference. (Reference.java:303) at java.lang.ref.WeakReference. (WeakReference.java:57) at java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry. (MethodType.java:1269) at java.lang.invoke.MethodType$ConcurrentWeakInternSet.get (MethodType.java:1216) at java.lang.invoke.MethodType.makeImpl (MethodType.java:302) at java.lang.invoke.MethodType.dropParameterTypes (MethodType.java:573) at java.lang.invoke.MethodType.replaceParameterTypes (MethodType.java:467) at java.lang.invoke.MethodHandle.asSpreader (MethodHandle.java:875) at java.lang.invoke.Invokers.spreadInvoker (Invokers.java:158) at java.lang.invoke.CallSite.makeSite (CallSite.java:324) at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl (MethodHandleNatives.java:307) at java.lang.invoke.MethodHandleNatives.linkCallSite (MethodHandleNatives.java:297) at scala.tools.nsc.typechecker.Typers$Typer.typedBlock (Typers.scala:2504) at scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$103 (Typers.scala:5711) at scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1 (Typers.scala:500)
Re: Problem building spark-catalyst_2.12 with Maven
I think it's another occurrence that I had to change or had to set MAVEN_OPTS. I think this occurs in a way that this setting doesn't affect, though I don't quite understand it. Try the stack size in test runner configs On Thu, Feb 10, 2022, 2:02 PM Martin Grigorov wrote: > Hi Sean, > > On Thu, Feb 10, 2022 at 5:37 PM Sean Owen wrote: > >> Yes I've seen this; the JVM stack size needs to be increased. I'm not >> sure if it's env specific (though you and I at least have hit it, I think >> others), or whether we need to change our build script. >> In the pom.xml file, find "-Xss..." settings and make them something like >> "-Xss4m", see if that works. >> > > It is already a much bigger value - 128m ( > https://github.com/apache/spark/blob/50256bde9bdf217413545a6d2945d6c61bf4cfff/pom.xml#L2845 > ) > I've tried smaller and bigger values for all jvmArgs next to this one. > None helped! > I also have the feeling it is something in my environment that overrides > these values but so far I cannot identify anything. > > > >> >> On Thu, Feb 10, 2022 at 8:54 AM Martin Grigorov >> wrote: >> >>> Hi, >>> >>> I am not able to build Spark due to the following error : >>> >>> ERROR] ## Exception when compiling 543 sources to >>> /home/martin/git/apache/spark/sql/catalyst/target/scala-2.12/classes >>> java.lang.BootstrapMethodError: call site initialization exception >>> java.lang.invoke.CallSite.makeSite(CallSite.java:341) >>> >>> java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307) >>> >>> java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297) >>> scala.tools.nsc.typechecker.Typers$Typer.typedBlock(Typers.scala:2504) >>> >>> scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$103(Typers.scala:5711) >>> >>> scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1(Typers.scala:500) >>> scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5746) >>> scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5781) >>> ... >>> Caused by: java.lang.StackOverflowError >>> at java.lang.ref.Reference. (Reference.java:303) >>> at java.lang.ref.WeakReference. (WeakReference.java:57) >>> at >>> java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry. >>> (MethodType.java:1269) >>> at java.lang.invoke.MethodType$ConcurrentWeakInternSet.get >>> (MethodType.java:1216) >>> at java.lang.invoke.MethodType.makeImpl (MethodType.java:302) >>> at java.lang.invoke.MethodType.dropParameterTypes >>> (MethodType.java:573) >>> at java.lang.invoke.MethodType.replaceParameterTypes >>> (MethodType.java:467) >>> at java.lang.invoke.MethodHandle.asSpreader (MethodHandle.java:875) >>> at java.lang.invoke.Invokers.spreadInvoker (Invokers.java:158) >>> at java.lang.invoke.CallSite.makeSite (CallSite.java:324) >>> at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl >>> (MethodHandleNatives.java:307) >>> at java.lang.invoke.MethodHandleNatives.linkCallSite >>> (MethodHandleNatives.java:297) >>> at scala.tools.nsc.typechecker.Typers$Typer.typedBlock >>> (Typers.scala:2504) >>> at scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$103 >>> (Typers.scala:5711) >>> at >>> scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1 >>> (Typers.scala:500) >>> at scala.tools.nsc.typechecker.Typers$Typer.typed1 >>> (Typers.scala:5746) >>> at scala.tools.nsc.typechecker.Typers$Typer.typed (Typers.scala:5781) >>> >>> I have played a lot with the scala-maven-plugin jvmArg settings at [1] >>> but so far nothing helps. >>> Same error for Scala 2.12 and 2.13. >>> >>> The command I use is: ./build/mvn install -Pkubernetes -DskipTests >>> >>> I need to create a distribution from master branch. >>> >>> Java: 1.8.0_312 >>> Maven: 3.8.4 >>> OS: Ubuntu 21.10 >>> >>> Any hints ? >>> Thank you! >>> >>> 1. >>> https://github.com/apache/spark/blob/50256bde9bdf217413545a6d2945d6c61bf4cfff/pom.xml#L2845-L2849 >>> >>
Re: Problem building spark-catalyst_2.12 with Maven
Hi Sean, On Thu, Feb 10, 2022 at 5:37 PM Sean Owen wrote: > Yes I've seen this; the JVM stack size needs to be increased. I'm not sure > if it's env specific (though you and I at least have hit it, I think > others), or whether we need to change our build script. > In the pom.xml file, find "-Xss..." settings and make them something like > "-Xss4m", see if that works. > It is already a much bigger value - 128m ( https://github.com/apache/spark/blob/50256bde9bdf217413545a6d2945d6c61bf4cfff/pom.xml#L2845 ) I've tried smaller and bigger values for all jvmArgs next to this one. None helped! I also have the feeling it is something in my environment that overrides these values but so far I cannot identify anything. > > On Thu, Feb 10, 2022 at 8:54 AM Martin Grigorov > wrote: > >> Hi, >> >> I am not able to build Spark due to the following error : >> >> ERROR] ## Exception when compiling 543 sources to >> /home/martin/git/apache/spark/sql/catalyst/target/scala-2.12/classes >> java.lang.BootstrapMethodError: call site initialization exception >> java.lang.invoke.CallSite.makeSite(CallSite.java:341) >> >> java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307) >> >> java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297) >> scala.tools.nsc.typechecker.Typers$Typer.typedBlock(Typers.scala:2504) >> >> scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$103(Typers.scala:5711) >> >> scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1(Typers.scala:500) >> scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5746) >> scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5781) >> ... >> Caused by: java.lang.StackOverflowError >> at java.lang.ref.Reference. (Reference.java:303) >> at java.lang.ref.WeakReference. (WeakReference.java:57) >> at >> java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry. >> (MethodType.java:1269) >> at java.lang.invoke.MethodType$ConcurrentWeakInternSet.get >> (MethodType.java:1216) >> at java.lang.invoke.MethodType.makeImpl (MethodType.java:302) >> at java.lang.invoke.MethodType.dropParameterTypes >> (MethodType.java:573) >> at java.lang.invoke.MethodType.replaceParameterTypes >> (MethodType.java:467) >> at java.lang.invoke.MethodHandle.asSpreader (MethodHandle.java:875) >> at java.lang.invoke.Invokers.spreadInvoker (Invokers.java:158) >> at java.lang.invoke.CallSite.makeSite (CallSite.java:324) >> at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl >> (MethodHandleNatives.java:307) >> at java.lang.invoke.MethodHandleNatives.linkCallSite >> (MethodHandleNatives.java:297) >> at scala.tools.nsc.typechecker.Typers$Typer.typedBlock >> (Typers.scala:2504) >> at scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$103 >> (Typers.scala:5711) >> at scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1 >> (Typers.scala:500) >> at scala.tools.nsc.typechecker.Typers$Typer.typed1 (Typers.scala:5746) >> at scala.tools.nsc.typechecker.Typers$Typer.typed (Typers.scala:5781) >> >> I have played a lot with the scala-maven-plugin jvmArg settings at [1] >> but so far nothing helps. >> Same error for Scala 2.12 and 2.13. >> >> The command I use is: ./build/mvn install -Pkubernetes -DskipTests >> >> I need to create a distribution from master branch. >> >> Java: 1.8.0_312 >> Maven: 3.8.4 >> OS: Ubuntu 21.10 >> >> Any hints ? >> Thank you! >> >> 1. >> https://github.com/apache/spark/blob/50256bde9bdf217413545a6d2945d6c61bf4cfff/pom.xml#L2845-L2849 >> >
Re: Problem building spark-catalyst_2.12 with Maven
Yes I've seen this; the JVM stack size needs to be increased. I'm not sure if it's env specific (though you and I at least have hit it, I think others), or whether we need to change our build script. In the pom.xml file, find "-Xss..." settings and make them something like "-Xss4m", see if that works. On Thu, Feb 10, 2022 at 8:54 AM Martin Grigorov wrote: > Hi, > > I am not able to build Spark due to the following error : > > ERROR] ## Exception when compiling 543 sources to > /home/martin/git/apache/spark/sql/catalyst/target/scala-2.12/classes > java.lang.BootstrapMethodError: call site initialization exception > java.lang.invoke.CallSite.makeSite(CallSite.java:341) > > java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307) > > java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297) > scala.tools.nsc.typechecker.Typers$Typer.typedBlock(Typers.scala:2504) > > scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$103(Typers.scala:5711) > > scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1(Typers.scala:500) > scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5746) > scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5781) > ... > Caused by: java.lang.StackOverflowError > at java.lang.ref.Reference. (Reference.java:303) > at java.lang.ref.WeakReference. (WeakReference.java:57) > at > java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry. > (MethodType.java:1269) > at java.lang.invoke.MethodType$ConcurrentWeakInternSet.get > (MethodType.java:1216) > at java.lang.invoke.MethodType.makeImpl (MethodType.java:302) > at java.lang.invoke.MethodType.dropParameterTypes (MethodType.java:573) > at java.lang.invoke.MethodType.replaceParameterTypes > (MethodType.java:467) > at java.lang.invoke.MethodHandle.asSpreader (MethodHandle.java:875) > at java.lang.invoke.Invokers.spreadInvoker (Invokers.java:158) > at java.lang.invoke.CallSite.makeSite (CallSite.java:324) > at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl > (MethodHandleNatives.java:307) > at java.lang.invoke.MethodHandleNatives.linkCallSite > (MethodHandleNatives.java:297) > at scala.tools.nsc.typechecker.Typers$Typer.typedBlock > (Typers.scala:2504) > at scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$103 > (Typers.scala:5711) > at scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1 > (Typers.scala:500) > at scala.tools.nsc.typechecker.Typers$Typer.typed1 (Typers.scala:5746) > at scala.tools.nsc.typechecker.Typers$Typer.typed (Typers.scala:5781) > > I have played a lot with the scala-maven-plugin jvmArg settings at [1] but > so far nothing helps. > Same error for Scala 2.12 and 2.13. > > The command I use is: ./build/mvn install -Pkubernetes -DskipTests > > I need to create a distribution from master branch. > > Java: 1.8.0_312 > Maven: 3.8.4 > OS: Ubuntu 21.10 > > Any hints ? > Thank you! > > 1. > https://github.com/apache/spark/blob/50256bde9bdf217413545a6d2945d6c61bf4cfff/pom.xml#L2845-L2849 >
Problem building spark-catalyst_2.12 with Maven
Hi, I am not able to build Spark due to the following error : ERROR] ## Exception when compiling 543 sources to /home/martin/git/apache/spark/sql/catalyst/target/scala-2.12/classes java.lang.BootstrapMethodError: call site initialization exception java.lang.invoke.CallSite.makeSite(CallSite.java:341) java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307) java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297) scala.tools.nsc.typechecker.Typers$Typer.typedBlock(Typers.scala:2504) scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$103(Typers.scala:5711) scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1(Typers.scala:500) scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5746) scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5781) ... Caused by: java.lang.StackOverflowError at java.lang.ref.Reference. (Reference.java:303) at java.lang.ref.WeakReference. (WeakReference.java:57) at java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry. (MethodType.java:1269) at java.lang.invoke.MethodType$ConcurrentWeakInternSet.get (MethodType.java:1216) at java.lang.invoke.MethodType.makeImpl (MethodType.java:302) at java.lang.invoke.MethodType.dropParameterTypes (MethodType.java:573) at java.lang.invoke.MethodType.replaceParameterTypes (MethodType.java:467) at java.lang.invoke.MethodHandle.asSpreader (MethodHandle.java:875) at java.lang.invoke.Invokers.spreadInvoker (Invokers.java:158) at java.lang.invoke.CallSite.makeSite (CallSite.java:324) at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl (MethodHandleNatives.java:307) at java.lang.invoke.MethodHandleNatives.linkCallSite (MethodHandleNatives.java:297) at scala.tools.nsc.typechecker.Typers$Typer.typedBlock (Typers.scala:2504) at scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$103 (Typers.scala:5711) at scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1 (Typers.scala:500) at scala.tools.nsc.typechecker.Typers$Typer.typed1 (Typers.scala:5746) at scala.tools.nsc.typechecker.Typers$Typer.typed (Typers.scala:5781) I have played a lot with the scala-maven-plugin jvmArg settings at [1] but so far nothing helps. Same error for Scala 2.12 and 2.13. The command I use is: ./build/mvn install -Pkubernetes -DskipTests I need to create a distribution from master branch. Java: 1.8.0_312 Maven: 3.8.4 OS: Ubuntu 21.10 Any hints ? Thank you! 1. https://github.com/apache/spark/blob/50256bde9bdf217413545a6d2945d6c61bf4cfff/pom.xml#L2845-L2849