Re: Does anyone know how to build spark with scala12.4?

[Sean Owen]

No, you have to run ./dev/change-scala-version.sh 2.12 before building for
2.12. That makes all the necessary POM changes.

On Wed, Nov 29, 2017 at 8:11 PM Zhang, Liyun  wrote:

> Hi Sean:
>
>   I have tried to use following script to build package but have problem(
> I am building a spark package for Hive on Spark, so use
> hadoop2-without-hive)
>
>
>
> ./dev/make-distribution.sh  --name hadoop2-without-hive --tgz -Pscala-2.12
> -Phadoop-2.7  -Pyarn -Pparquet-provided -Dhadoop.version=2.7.3
>
>
>
>

RE: Does anyone know how to build spark with scala12.4?

[Zhang, Liyun]

Hi Sean:
  I have tried to use following script to build package but have problem( I am 
building a spark package for Hive on Spark, so use hadoop2-without-hive)

./dev/make-distribution.sh  --name hadoop2-without-hive --tgz -Pscala-2.12 
-Phadoop-2.7  -Pyarn -Pparquet-provided -Dhadoop.version=2.7.3


The problem is

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
on project spark-sketch_2.11: Could not resolve dependencies for project 
org.apache.spa  rk:spark-sketch_2.11:jar:2.3.0-SNAPSHOT: The following 
artifacts could not be resolved: 
org.apache.spark:spark-tags_2.12:jar:2.3.0-SNAPSHOT, org.apache.spark:spark-ta  
gs_2.12:jar:tests:2.3.0-SNAPSHOT: Failure to find 
org.apache.spark:spark-tags_2.12:jar:2.3.0-SNAPSHOT in 
https://repository.apache.org/snapshots was cached in the loc  al 
repository,



After then I found that the artifactId in $SPARK_SOURCE/common/tags/pom.xml  is 
spark-tags_2.11. I change the $SPARK_SOURCE/common/tags/pom.xml  as following, 
the problem about {{.spark:spark-tags_2.12:jar:2.3.0-SNAPSHOT}}  seems not exist

git diff common/tags/pom.xml
diff --git a/common/tags/pom.xml b/common/tags/pom.xml
index f7e586e..5f48105 100644
--- a/common/tags/pom.xml
+++ b/common/tags/pom.xml
@@ -26,7 +26,8 @@
 ../../pom.xml
   
-  spark-tags_2.11
+  
+spark-tags_2.12
   jar
   Spark Project Tags
   http://spark.apache.org/http://spark.apache.org/%3c/url>>


My question is

1.   Should I change the $SPARK_SOURCE/common/tags/pom.xml  manually to 
spark-tags_2.12, if need, I guess I need to change {{spark-streaming}}





Appreciate to get some feedback from you.




Best Regards
Kelly Zhang/Zhang,Liyun







From: Sean Owen [mailto:so...@cloudera.com]
Sent: Tuesday, November 28, 2017 9:52 PM
To: Ofir Manor <ofir.ma...@equalum.io>
Cc: Zhang, Liyun <liyun.zh...@intel.com>; dev <dev@spark.apache.org>
Subject: Re: Does anyone know how to build spark with scala12.4?

The Scala 2.12 profile mostly works, but not all tests pass. Use -Pscala-2.12 
on the command line to build.

On Tue, Nov 28, 2017 at 5:36 AM Ofir Manor 
<ofir.ma...@equalum.io<mailto:ofir.ma...@equalum.io>> wrote:
Hi,
as far as I know, Spark does not support Scala 2.12.
There is on-going work to make refactor / fix Spark source code to support 
Scala 2.12 - look for multiple emails on this list in the last months from Sean 
Owen on his progress.
Once Spark supports Scala 2.12, I think the next target would be JDK 9 support.


Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286<tel:%2B972-54-7801286> | Email: 
ofir.ma...@equalum.io<mailto:ofir.ma...@equalum.io>

On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun 
<liyun.zh...@intel.com<mailto:liyun.zh...@intel.com>> wrote:
Hi all:
  Does anyone know how to build spark with scala12.4? I want to test whether 
spark can work on jdk9 or not.  Scala12.4 supports jdk9.  Does anyone try to 
build spark with scala 12.4 or compile successfully with jdk9.Appreciate to get 
some feedback from you.


Best Regards
Kelly Zhang/Zhang,Liyun

Re: Does anyone know how to build spark with scala12.4?

[Saisai Shao]

I see, thanks for your quick response.

Best regards,
Jerry

2017-11-29 10:45 GMT+08:00 Sean Owen :

> Until the 2.12 build passes tests, no. There is still a real outstanding
> issue with the closure cleaner and serialization of closures as Java 8
> lambdas. I haven't cracked it, and don't think it's simple, but not
> insurmountable.
>
> The funny thing is most stuff appears to just work without cleaning said
> lambdas, because they don't generally capture references in the same way.
> So it may be reasonable to advertise 2.12 support as experimental and for
> people willing to make their own build. That's why I wanted it in good
> enough shape that the scala-2.12 profile produces something basically
> functional.
>
> On Tue, Nov 28, 2017 at 8:43 PM Saisai Shao 
> wrote:
>
>> Hi Sean,
>>
>> Two questions about Scala 2.12 for release artifacts.
>>
>> Are we planning to ship 2.12 artifacts for Spark 2.3 release? If not,
>> will we only ship 2.11 artifacts?
>>
>> Thanks
>> Jerry
>>
>> 2017-11-28 21:51 GMT+08:00 Sean Owen :
>>
>>> The Scala 2.12 profile mostly works, but not all tests pass. Use
>>> -Pscala-2.12 on the command line to build.
>>>
>>> On Tue, Nov 28, 2017 at 5:36 AM Ofir Manor 
>>> wrote:
>>>
 Hi,
 as far as I know, Spark does not support Scala 2.12.
 There is on-going work to make refactor / fix Spark source code to
 support Scala 2.12 - look for multiple emails on this list in the last
 months from Sean Owen on his progress.
 Once Spark supports Scala 2.12, I think the next target would be JDK 9
 support.

 Ofir Manor

 Co-Founder & CTO | Equalum

 Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

 On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun 
 wrote:

> Hi all:
>
>   Does anyone know how to build spark with scala12.4? I want to test
> whether spark can work on jdk9 or not.  Scala12.4 supports jdk9.  Does
> anyone try to build spark with scala 12.4 or compile successfully with
> jdk9.Appreciate to get some feedback from you.
>
>
>
>
>
> Best Regards
>
> Kelly Zhang/Zhang,Liyun
>
>
>


>>

Re: Does anyone know how to build spark with scala12.4?

[Sean Owen]

Until the 2.12 build passes tests, no. There is still a real outstanding
issue with the closure cleaner and serialization of closures as Java 8
lambdas. I haven't cracked it, and don't think it's simple, but not
insurmountable.

The funny thing is most stuff appears to just work without cleaning said
lambdas, because they don't generally capture references in the same way.
So it may be reasonable to advertise 2.12 support as experimental and for
people willing to make their own build. That's why I wanted it in good
enough shape that the scala-2.12 profile produces something basically
functional.

On Tue, Nov 28, 2017 at 8:43 PM Saisai Shao  wrote:

> Hi Sean,
>
> Two questions about Scala 2.12 for release artifacts.
>
> Are we planning to ship 2.12 artifacts for Spark 2.3 release? If not, will
> we only ship 2.11 artifacts?
>
> Thanks
> Jerry
>
> 2017-11-28 21:51 GMT+08:00 Sean Owen :
>
>> The Scala 2.12 profile mostly works, but not all tests pass. Use
>> -Pscala-2.12 on the command line to build.
>>
>> On Tue, Nov 28, 2017 at 5:36 AM Ofir Manor  wrote:
>>
>>> Hi,
>>> as far as I know, Spark does not support Scala 2.12.
>>> There is on-going work to make refactor / fix Spark source code to
>>> support Scala 2.12 - look for multiple emails on this list in the last
>>> months from Sean Owen on his progress.
>>> Once Spark supports Scala 2.12, I think the next target would be JDK 9
>>> support.
>>>
>>> Ofir Manor
>>>
>>> Co-Founder & CTO | Equalum
>>>
>>> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>>>
>>> On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun 
>>> wrote:
>>>
 Hi all:

   Does anyone know how to build spark with scala12.4? I want to test
 whether spark can work on jdk9 or not.  Scala12.4 supports jdk9.  Does
 anyone try to build spark with scala 12.4 or compile successfully with
 jdk9.Appreciate to get some feedback from you.





 Best Regards

 Kelly Zhang/Zhang,Liyun



>>>
>>>
>

Re: Does anyone know how to build spark with scala12.4?

[Saisai Shao]

Hi Sean,

Two questions about Scala 2.12 for release artifacts.

Are we planning to ship 2.12 artifacts for Spark 2.3 release? If not, will
we only ship 2.11 artifacts?

Thanks
Jerry

2017-11-28 21:51 GMT+08:00 Sean Owen :

> The Scala 2.12 profile mostly works, but not all tests pass. Use
> -Pscala-2.12 on the command line to build.
>
> On Tue, Nov 28, 2017 at 5:36 AM Ofir Manor  wrote:
>
>> Hi,
>> as far as I know, Spark does not support Scala 2.12.
>> There is on-going work to make refactor / fix Spark source code to
>> support Scala 2.12 - look for multiple emails on this list in the last
>> months from Sean Owen on his progress.
>> Once Spark supports Scala 2.12, I think the next target would be JDK 9
>> support.
>>
>> Ofir Manor
>>
>> Co-Founder & CTO | Equalum
>>
>> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>>
>> On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun 
>> wrote:
>>
>>> Hi all:
>>>
>>>   Does anyone know how to build spark with scala12.4? I want to test
>>> whether spark can work on jdk9 or not.  Scala12.4 supports jdk9.  Does
>>> anyone try to build spark with scala 12.4 or compile successfully with
>>> jdk9.Appreciate to get some feedback from you.
>>>
>>>
>>>
>>>
>>>
>>> Best Regards
>>>
>>> Kelly Zhang/Zhang,Liyun
>>>
>>>
>>>
>>
>>

Re: Does anyone know how to build spark with scala12.4?

[Sean Owen]

The Scala 2.12 profile mostly works, but not all tests pass. Use
-Pscala-2.12 on the command line to build.

On Tue, Nov 28, 2017 at 5:36 AM Ofir Manor  wrote:

> Hi,
> as far as I know, Spark does not support Scala 2.12.
> There is on-going work to make refactor / fix Spark source code to support
> Scala 2.12 - look for multiple emails on this list in the last months from
> Sean Owen on his progress.
> Once Spark supports Scala 2.12, I think the next target would be JDK 9
> support.
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
> On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun 
> wrote:
>
>> Hi all:
>>
>>   Does anyone know how to build spark with scala12.4? I want to test
>> whether spark can work on jdk9 or not.  Scala12.4 supports jdk9.  Does
>> anyone try to build spark with scala 12.4 or compile successfully with
>> jdk9.Appreciate to get some feedback from you.
>>
>>
>>
>>
>>
>> Best Regards
>>
>> Kelly Zhang/Zhang,Liyun
>>
>>
>>
>
>

Re: Does anyone know how to build spark with scala12.4?

[Ofir Manor]

Hi,
as far as I know, Spark does not support Scala 2.12.
There is on-going work to make refactor / fix Spark source code to support
Scala 2.12 - look for multiple emails on this list in the last months from
Sean Owen on his progress.
Once Spark supports Scala 2.12, I think the next target would be JDK 9
support.

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Tue, Nov 28, 2017 at 9:20 AM, Zhang, Liyun  wrote:

> Hi all:
>
>   Does anyone know how to build spark with scala12.4? I want to test
> whether spark can work on jdk9 or not.  Scala12.4 supports jdk9.  Does
> anyone try to build spark with scala 12.4 or compile successfully with
> jdk9.Appreciate to get some feedback from you.
>
>
>
>
>
> Best Regards
>
> Kelly Zhang/Zhang,Liyun
>
>
>