Re: Is it safe to use Scala 2.11 for Spark build?

2014-11-18 Thread Jianshi Huang 
Ok, I'll wait until -Pscala-2.11 is more stable and used by more people.

Thanks for the help!

Jianshi

On Tue, Nov 18, 2014 at 3:49 PM, Ye Xianjin advance...@gmail.com wrote:

 Hi Prashant Sharma,

 It's not even ok to build with scala-2.11 profile on my machine.

 Just check out the master(c6e0c2ab1c29c184a9302d23ad75e4ccd8060242)
 run sbt/sbt -Pscala-2.11 clean assembly:

 .. skip the normal part
 info] Resolving org.scalamacros#quasiquotes_2.11;2.0.1 ...
 [warn] module not found: org.scalamacros#quasiquotes_2.11;2.0.1
 [warn]  local: tried
 [warn]
 /Users/yexianjin/.ivy2/local/org.scalamacros/quasiquotes_2.11/2.0.1/ivys/ivy.xml
 [warn]  public: tried
 [warn]
 https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  central: tried
 [warn]
 https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  apache-repo: tried
 [warn]
 https://repository.apache.org/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  jboss-repo: tried
 [warn]
 https://repository.jboss.org/nexus/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  mqtt-repo: tried
 [warn]
 https://repo.eclipse.org/content/repositories/paho-releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  cloudera-repo: tried
 [warn]
 https://repository.cloudera.com/artifactory/cloudera-repos/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  mapr-repo: tried
 [warn]
 http://repository.mapr.com/maven/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  spring-releases: tried
 [warn]
 https://repo.spring.io/libs-release/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  spark-staging: tried
 [warn]
 https://oss.sonatype.org/content/repositories/orgspark-project-1085/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  spark-staging-hive13: tried
 [warn]
 https://oss.sonatype.org/content/repositories/orgspark-project-1089/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  apache.snapshots: tried
 [warn]
 http://repository.apache.org/snapshots/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [warn]  Maven2 Local: tried
 [warn]
 file:/Users/yexianjin/.m2/repository/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
 [info] Resolving jline#jline;2.12 ...
 [warn] ::
 [warn] ::  UNRESOLVED DEPENDENCIES ::
 [warn] ::
 [warn] :: org.scalamacros#quasiquotes_2.11;2.0.1: not found
 [warn] ::
 [info] Resolving org.scala-lang#scala-library;2.11.2 ...
 [warn]
 [warn] Note: Unresolved dependencies path:
 [warn] org.scalamacros:quasiquotes_2.11:2.0.1
 ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
 [warn]   +- org.apache.spark:spark-catalyst_2.11:1.2.0-SNAPSHOT
 [info] Resolving jline#jline;2.12 ...
 [info] Done updating.
 [info] Updating {file:/Users/yexianjin/spark/}streaming-twitter...
 [info] Updating {file:/Users/yexianjin/spark/}streaming-zeromq...
 [info] Updating {file:/Users/yexianjin/spark/}streaming-flume...
 [info] Updating {file:/Users/yexianjin/spark/}streaming-mqtt...
 [info] Resolving jline#jline;2.12 ...
 [info] Done updating.
 [info] Resolving com.esotericsoftware.minlog#minlog;1.2 ...
 [info] Updating {file:/Users/yexianjin/spark/}streaming-kafka...
 [info] Resolving jline#jline;2.12 ...
 [info] Done updating.
 [info] Resolving jline#jline;2.12 ...
 [info] Done updating.
 [info] Resolving jline#jline;2.12 ...
 [info] Done updating.
 [info] Resolving org.apache.kafka#kafka_2.11;0.8.0 ...
 [warn] module not found: org.apache.kafka#kafka_2.11;0.8.0
 [warn]  local: tried
 [warn]
 /Users/yexianjin/.ivy2/local/org.apache.kafka/kafka_2.11/0.8.0/ivys/ivy.xml
 [warn]  public: tried
 [warn]
 https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
 [warn]  central: tried
 [warn]
 https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
 [warn]  apache-repo: tried
 [warn]
 https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
 [warn]  jboss-repo: tried
 [warn]
 https://repository.jboss.org/nexus/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
 [warn]  mqtt-repo: tried
 [warn]
 https://repo.eclipse.org/content/repositories/paho-releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
 [warn]  cloudera-repo: tried
 [warn]
 https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
 [warn]  mapr-repo: tried
 [warn]

Re: Is it safe to use Scala 2.11 for Spark build?

2014-11-17 Thread Prashant Sharma 
It is safe in the sense we would help you with the fix if you run into
issues. I have used it, but since I worked on the patch the opinion can be
biased. I am using scala 2.11 for day to day development. You should
checkout the build instructions here :
https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md

Prashant Sharma



On Tue, Nov 18, 2014 at 12:19 PM, Jianshi Huang jianshi.hu...@gmail.com
wrote:

 Any notable issues for using Scala 2.11? Is it stable now?

 Or can I use Scala 2.11 in my spark application and use Spark dist build
 with 2.10 ?

 I'm looking forward to migrate to 2.11 for some quasiquote features.
 Couldn't make it run in 2.10...

 Cheers,
 --
 Jianshi Huang

 LinkedIn: jianshi
 Twitter: @jshuang
 Github  Blog: http://huangjs.github.com/

Re: Is it safe to use Scala 2.11 for Spark build?

2014-11-17 Thread Prashant Sharma 
Looks like sbt/sbt -Pscala-2.11 is broken by a recent patch for improving
maven build.

Prashant Sharma



On Tue, Nov 18, 2014 at 12:57 PM, Prashant Sharma scrapco...@gmail.com
wrote:

 It is safe in the sense we would help you with the fix if you run into
 issues. I have used it, but since I worked on the patch the opinion can be
 biased. I am using scala 2.11 for day to day development. You should
 checkout the build instructions here :
 https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md

 Prashant Sharma



 On Tue, Nov 18, 2014 at 12:19 PM, Jianshi Huang jianshi.hu...@gmail.com
 wrote:

 Any notable issues for using Scala 2.11? Is it stable now?

 Or can I use Scala 2.11 in my spark application and use Spark dist build
 with 2.10 ?

 I'm looking forward to migrate to 2.11 for some quasiquote features.
 Couldn't make it run in 2.10...

 Cheers,
 --
 Jianshi Huang

 LinkedIn: jianshi
 Twitter: @jshuang
 Github  Blog: http://huangjs.github.com/

Re: Is it safe to use Scala 2.11 for Spark build?

2014-11-17 Thread Ye Xianjin 
Hi Prashant Sharma, 

It's not even ok to build with scala-2.11 profile on my machine.

Just check out the master(c6e0c2ab1c29c184a9302d23ad75e4ccd8060242)
run sbt/sbt -Pscala-2.11 clean assembly:

.. skip the normal part
info] Resolving org.scalamacros#quasiquotes_2.11;2.0.1 ...
[warn] module not found: org.scalamacros#quasiquotes_2.11;2.0.1
[warn]  local: tried
[warn]   
/Users/yexianjin/.ivy2/local/org.scalamacros/quasiquotes_2.11/2.0.1/ivys/ivy.xml
[warn]  public: tried
[warn]   
https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  central: tried
[warn]   
https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  apache-repo: tried
[warn]   
https://repository.apache.org/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  jboss-repo: tried
[warn]   
https://repository.jboss.org/nexus/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  mqtt-repo: tried
[warn]   
https://repo.eclipse.org/content/repositories/paho-releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  cloudera-repo: tried
[warn]   
https://repository.cloudera.com/artifactory/cloudera-repos/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  mapr-repo: tried
[warn]   
http://repository.mapr.com/maven/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  spring-releases: tried
[warn]   
https://repo.spring.io/libs-release/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  spark-staging: tried
[warn]   
https://oss.sonatype.org/content/repositories/orgspark-project-1085/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  spark-staging-hive13: tried
[warn]   
https://oss.sonatype.org/content/repositories/orgspark-project-1089/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  apache.snapshots: tried
[warn]   
http://repository.apache.org/snapshots/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[warn]  Maven2 Local: tried
[warn]   
file:/Users/yexianjin/.m2/repository/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom
[info] Resolving jline#jline;2.12 ...
[warn] ::
[warn] ::  UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: org.scalamacros#quasiquotes_2.11;2.0.1: not found
[warn] ::
[info] Resolving org.scala-lang#scala-library;2.11.2 ...
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.scalamacros:quasiquotes_2.11:2.0.1 
((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
[warn]  +- org.apache.spark:spark-catalyst_2.11:1.2.0-SNAPSHOT
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Updating {file:/Users/yexianjin/spark/}streaming-twitter...
[info] Updating {file:/Users/yexianjin/spark/}streaming-zeromq...
[info] Updating {file:/Users/yexianjin/spark/}streaming-flume...
[info] Updating {file:/Users/yexianjin/spark/}streaming-mqtt...
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Resolving com.esotericsoftware.minlog#minlog;1.2 ...
[info] Updating {file:/Users/yexianjin/spark/}streaming-kafka...
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Resolving org.apache.kafka#kafka_2.11;0.8.0 ...
[warn] module not found: org.apache.kafka#kafka_2.11;0.8.0
[warn]  local: tried
[warn]   
/Users/yexianjin/.ivy2/local/org.apache.kafka/kafka_2.11/0.8.0/ivys/ivy.xml
[warn]  public: tried
[warn]   
https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn]  central: tried
[warn]   
https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn]  apache-repo: tried
[warn]   
https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn]  jboss-repo: tried
[warn]   
https://repository.jboss.org/nexus/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn]  mqtt-repo: tried
[warn]   
https://repo.eclipse.org/content/repositories/paho-releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn]  cloudera-repo: tried
[warn]   
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn]  mapr-repo: tried
[warn]   
http://repository.mapr.com/maven/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn]  spring-releases: tried
[warn]   
https://repo.spring.io/libs-release/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom
[warn]