Re: Does anyone know how to build spark with scala12.4?
No, you have to run ./dev/change-scala-version.sh 2.12 before building for 2.12. That makes all the necessary POM changes. On Wed, Nov 29, 2017 at 8:11 PM Zhang, Liyun wrote: > Hi Sean: > > I have tried to use following script to build package but have problem( > I am building a spark package for Hive on Spark, so use > hadoop2-without-hive) > > > > ./dev/make-distribution.sh --name hadoop2-without-hive --tgz -Pscala-2.12 > -Phadoop-2.7 -Pyarn -Pparquet-provided -Dhadoop.version=2.7.3 > > > >
RE: Does anyone know how to build spark with scala12.4?
Hi Sean: I have tried to use following script to build package but have problem( I am building a spark package for Hive on Spark, so use hadoop2-without-hive) ./dev/make-distribution.sh --name hadoop2-without-hive --tgz -Pscala-2.12 -Phadoop-2.7 -Pyarn -Pparquet-provided -Dhadoop.version=2.7.3 The problem is org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-sketch_2.11: Could not resolve dependencies for project org.apache.spa rk:spark-sketch_2.11:jar:2.3.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-tags_2.12:jar:2.3.0-SNAPSHOT, org.apache.spark:spark-ta gs_2.12:jar:tests:2.3.0-SNAPSHOT: Failure to find org.apache.spark:spark-tags_2.12:jar:2.3.0-SNAPSHOT in https://repository.apache.org/snapshots was cached in the loc al repository, After then I found that the artifactId in $SPARK_SOURCE/common/tags/pom.xml is spark-tags_2.11. I change the $SPARK_SOURCE/common/tags/pom.xml as following, the problem about {{.spark:spark-tags_2.12:jar:2.3.0-SNAPSHOT}} seems not exist git diff common/tags/pom.xml diff --git a/common/tags/pom.xml b/common/tags/pom.xml index f7e586e..5f48105 100644 --- a/common/tags/pom.xml +++ b/common/tags/pom.xml @@ -26,7 +26,8 @@ ../../pom.xml - spark-tags_2.11 + +spark-tags_2.12 jar Spark Project Tags http://spark.apache.org/http://spark.apache.org/%3c/url>> My question is 1. Should I change the $SPARK_SOURCE/common/tags/pom.xml manually to spark-tags_2.12, if need, I guess I need to change {{spark-streaming}} Appreciate to get some feedback from you. Best Regards Kelly Zhang/Zhang,Liyun From: Sean Owen [mailto:so...@cloudera.com] Sent: Tuesday, November 28, 2017 9:52 PM To: Ofir Manor Cc: Zhang, Liyun ; dev Subject: Re: Does anyone know how to build spark with scala12.4? The Scala 2.12 profile mostly works, but not all tests pass. Use -Pscala-2.12 on the command line to build. On Tue, Nov 28, 2017 at 5:36 AM Ofir Manor mailto:ofir.ma...@equalum.io>> wrote: Hi, as far as I know, Spark does not support Scala 2.12. There is on-going work to make refactor / fix Spark source code to support Scala 2.12 - look for multiple emails on this list in the last months from Sean Owen on his progress. Once Spark supports Scala 2.12, I think the next target would be JDK 9 support. Ofir Manor Co-Founder & CTO | Equalum Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io<mailto:ofir.ma...@equalum.io> On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun mailto:liyun.zh...@intel.com>> wrote: Hi all: Does anyone know how to build spark with scala12.4? I want to test whether spark can work on jdk9 or not. Scala12.4 supports jdk9. Does anyone try to build spark with scala 12.4 or compile successfully with jdk9.Appreciate to get some feedback from you. Best Regards Kelly Zhang/Zhang,Liyun
Re: Does anyone know how to build spark with scala12.4?
I see, thanks for your quick response. Best regards, Jerry 2017-11-29 10:45 GMT+08:00 Sean Owen : > Until the 2.12 build passes tests, no. There is still a real outstanding > issue with the closure cleaner and serialization of closures as Java 8 > lambdas. I haven't cracked it, and don't think it's simple, but not > insurmountable. > > The funny thing is most stuff appears to just work without cleaning said > lambdas, because they don't generally capture references in the same way. > So it may be reasonable to advertise 2.12 support as experimental and for > people willing to make their own build. That's why I wanted it in good > enough shape that the scala-2.12 profile produces something basically > functional. > > On Tue, Nov 28, 2017 at 8:43 PM Saisai Shao > wrote: > >> Hi Sean, >> >> Two questions about Scala 2.12 for release artifacts. >> >> Are we planning to ship 2.12 artifacts for Spark 2.3 release? If not, >> will we only ship 2.11 artifacts? >> >> Thanks >> Jerry >> >> 2017-11-28 21:51 GMT+08:00 Sean Owen : >> >>> The Scala 2.12 profile mostly works, but not all tests pass. Use >>> -Pscala-2.12 on the command line to build. >>> >>> On Tue, Nov 28, 2017 at 5:36 AM Ofir Manor >>> wrote: >>> Hi, as far as I know, Spark does not support Scala 2.12. There is on-going work to make refactor / fix Spark source code to support Scala 2.12 - look for multiple emails on this list in the last months from Sean Owen on his progress. Once Spark supports Scala 2.12, I think the next target would be JDK 9 support. Ofir Manor Co-Founder & CTO | Equalum Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun wrote: > Hi all: > > Does anyone know how to build spark with scala12.4? I want to test > whether spark can work on jdk9 or not. Scala12.4 supports jdk9. Does > anyone try to build spark with scala 12.4 or compile successfully with > jdk9.Appreciate to get some feedback from you. > > > > > > Best Regards > > Kelly Zhang/Zhang,Liyun > > > >>
Re: Does anyone know how to build spark with scala12.4?
Until the 2.12 build passes tests, no. There is still a real outstanding issue with the closure cleaner and serialization of closures as Java 8 lambdas. I haven't cracked it, and don't think it's simple, but not insurmountable. The funny thing is most stuff appears to just work without cleaning said lambdas, because they don't generally capture references in the same way. So it may be reasonable to advertise 2.12 support as experimental and for people willing to make their own build. That's why I wanted it in good enough shape that the scala-2.12 profile produces something basically functional. On Tue, Nov 28, 2017 at 8:43 PM Saisai Shao wrote: > Hi Sean, > > Two questions about Scala 2.12 for release artifacts. > > Are we planning to ship 2.12 artifacts for Spark 2.3 release? If not, will > we only ship 2.11 artifacts? > > Thanks > Jerry > > 2017-11-28 21:51 GMT+08:00 Sean Owen : > >> The Scala 2.12 profile mostly works, but not all tests pass. Use >> -Pscala-2.12 on the command line to build. >> >> On Tue, Nov 28, 2017 at 5:36 AM Ofir Manor wrote: >> >>> Hi, >>> as far as I know, Spark does not support Scala 2.12. >>> There is on-going work to make refactor / fix Spark source code to >>> support Scala 2.12 - look for multiple emails on this list in the last >>> months from Sean Owen on his progress. >>> Once Spark supports Scala 2.12, I think the next target would be JDK 9 >>> support. >>> >>> Ofir Manor >>> >>> Co-Founder & CTO | Equalum >>> >>> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io >>> >>> On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun >>> wrote: >>> Hi all: Does anyone know how to build spark with scala12.4? I want to test whether spark can work on jdk9 or not. Scala12.4 supports jdk9. Does anyone try to build spark with scala 12.4 or compile successfully with jdk9.Appreciate to get some feedback from you. Best Regards Kelly Zhang/Zhang,Liyun >>> >>> >
Re: Does anyone know how to build spark with scala12.4?
Hi Sean, Two questions about Scala 2.12 for release artifacts. Are we planning to ship 2.12 artifacts for Spark 2.3 release? If not, will we only ship 2.11 artifacts? Thanks Jerry 2017-11-28 21:51 GMT+08:00 Sean Owen : > The Scala 2.12 profile mostly works, but not all tests pass. Use > -Pscala-2.12 on the command line to build. > > On Tue, Nov 28, 2017 at 5:36 AM Ofir Manor wrote: > >> Hi, >> as far as I know, Spark does not support Scala 2.12. >> There is on-going work to make refactor / fix Spark source code to >> support Scala 2.12 - look for multiple emails on this list in the last >> months from Sean Owen on his progress. >> Once Spark supports Scala 2.12, I think the next target would be JDK 9 >> support. >> >> Ofir Manor >> >> Co-Founder & CTO | Equalum >> >> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io >> >> On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun >> wrote: >> >>> Hi all: >>> >>> Does anyone know how to build spark with scala12.4? I want to test >>> whether spark can work on jdk9 or not. Scala12.4 supports jdk9. Does >>> anyone try to build spark with scala 12.4 or compile successfully with >>> jdk9.Appreciate to get some feedback from you. >>> >>> >>> >>> >>> >>> Best Regards >>> >>> Kelly Zhang/Zhang,Liyun >>> >>> >>> >> >>
Re: Does anyone know how to build spark with scala12.4?
The Scala 2.12 profile mostly works, but not all tests pass. Use -Pscala-2.12 on the command line to build. On Tue, Nov 28, 2017 at 5:36 AM Ofir Manor wrote: > Hi, > as far as I know, Spark does not support Scala 2.12. > There is on-going work to make refactor / fix Spark source code to support > Scala 2.12 - look for multiple emails on this list in the last months from > Sean Owen on his progress. > Once Spark supports Scala 2.12, I think the next target would be JDK 9 > support. > > Ofir Manor > > Co-Founder & CTO | Equalum > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io > > On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun > wrote: > >> Hi all: >> >> Does anyone know how to build spark with scala12.4? I want to test >> whether spark can work on jdk9 or not. Scala12.4 supports jdk9. Does >> anyone try to build spark with scala 12.4 or compile successfully with >> jdk9.Appreciate to get some feedback from you. >> >> >> >> >> >> Best Regards >> >> Kelly Zhang/Zhang,Liyun >> >> >> > >
Re: Does anyone know how to build spark with scala12.4?
Hi, as far as I know, Spark does not support Scala 2.12. There is on-going work to make refactor / fix Spark source code to support Scala 2.12 - look for multiple emails on this list in the last months from Sean Owen on his progress. Once Spark supports Scala 2.12, I think the next target would be JDK 9 support. Ofir Manor Co-Founder & CTO | Equalum Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun wrote: > Hi all: > > Does anyone know how to build spark with scala12.4? I want to test > whether spark can work on jdk9 or not. Scala12.4 supports jdk9. Does > anyone try to build spark with scala 12.4 or compile successfully with > jdk9.Appreciate to get some feedback from you. > > > > > > Best Regards > > Kelly Zhang/Zhang,Liyun > > >