Re: Is it safe to use Scala 2.11 for Spark build?
Ok, I'll wait until -Pscala-2.11 is more stable and used by more people. Thanks for the help! Jianshi On Tue, Nov 18, 2014 at 3:49 PM, Ye Xianjin advance...@gmail.com wrote: Hi Prashant Sharma, It's not even ok to build with scala-2.11 profile on my machine. Just check out the master(c6e0c2ab1c29c184a9302d23ad75e4ccd8060242) run sbt/sbt -Pscala-2.11 clean assembly: .. skip the normal part info] Resolving org.scalamacros#quasiquotes_2.11;2.0.1 ... [warn] module not found: org.scalamacros#quasiquotes_2.11;2.0.1 [warn] local: tried [warn] /Users/yexianjin/.ivy2/local/org.scalamacros/quasiquotes_2.11/2.0.1/ivys/ivy.xml [warn] public: tried [warn] https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] central: tried [warn] https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] apache-repo: tried [warn] https://repository.apache.org/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] jboss-repo: tried [warn] https://repository.jboss.org/nexus/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] mqtt-repo: tried [warn] https://repo.eclipse.org/content/repositories/paho-releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] cloudera-repo: tried [warn] https://repository.cloudera.com/artifactory/cloudera-repos/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] mapr-repo: tried [warn] http://repository.mapr.com/maven/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] spring-releases: tried [warn] https://repo.spring.io/libs-release/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] spark-staging: tried [warn] https://oss.sonatype.org/content/repositories/orgspark-project-1085/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] spark-staging-hive13: tried [warn] https://oss.sonatype.org/content/repositories/orgspark-project-1089/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] apache.snapshots: tried [warn] http://repository.apache.org/snapshots/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] Maven2 Local: tried [warn] file:/Users/yexianjin/.m2/repository/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [info] Resolving jline#jline;2.12 ... [warn] :: [warn] :: UNRESOLVED DEPENDENCIES :: [warn] :: [warn] :: org.scalamacros#quasiquotes_2.11;2.0.1: not found [warn] :: [info] Resolving org.scala-lang#scala-library;2.11.2 ... [warn] [warn] Note: Unresolved dependencies path: [warn] org.scalamacros:quasiquotes_2.11:2.0.1 ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76) [warn] +- org.apache.spark:spark-catalyst_2.11:1.2.0-SNAPSHOT [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Updating {file:/Users/yexianjin/spark/}streaming-twitter... [info] Updating {file:/Users/yexianjin/spark/}streaming-zeromq... [info] Updating {file:/Users/yexianjin/spark/}streaming-flume... [info] Updating {file:/Users/yexianjin/spark/}streaming-mqtt... [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Resolving com.esotericsoftware.minlog#minlog;1.2 ... [info] Updating {file:/Users/yexianjin/spark/}streaming-kafka... [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Resolving org.apache.kafka#kafka_2.11;0.8.0 ... [warn] module not found: org.apache.kafka#kafka_2.11;0.8.0 [warn] local: tried [warn] /Users/yexianjin/.ivy2/local/org.apache.kafka/kafka_2.11/0.8.0/ivys/ivy.xml [warn] public: tried [warn] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] central: tried [warn] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] apache-repo: tried [warn] https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] jboss-repo: tried [warn] https://repository.jboss.org/nexus/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] mqtt-repo: tried [warn] https://repo.eclipse.org/content/repositories/paho-releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] cloudera-repo: tried [warn] https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] mapr-repo: tried [warn]
Re: Is it safe to use Scala 2.11 for Spark build?
It is safe in the sense we would help you with the fix if you run into issues. I have used it, but since I worked on the patch the opinion can be biased. I am using scala 2.11 for day to day development. You should checkout the build instructions here : https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md Prashant Sharma On Tue, Nov 18, 2014 at 12:19 PM, Jianshi Huang jianshi.hu...@gmail.com wrote: Any notable issues for using Scala 2.11? Is it stable now? Or can I use Scala 2.11 in my spark application and use Spark dist build with 2.10 ? I'm looking forward to migrate to 2.11 for some quasiquote features. Couldn't make it run in 2.10... Cheers, -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github Blog: http://huangjs.github.com/
Re: Is it safe to use Scala 2.11 for Spark build?
Looks like sbt/sbt -Pscala-2.11 is broken by a recent patch for improving maven build. Prashant Sharma On Tue, Nov 18, 2014 at 12:57 PM, Prashant Sharma scrapco...@gmail.com wrote: It is safe in the sense we would help you with the fix if you run into issues. I have used it, but since I worked on the patch the opinion can be biased. I am using scala 2.11 for day to day development. You should checkout the build instructions here : https://github.com/ScrapCodes/spark-1/blob/patch-3/docs/building-spark.md Prashant Sharma On Tue, Nov 18, 2014 at 12:19 PM, Jianshi Huang jianshi.hu...@gmail.com wrote: Any notable issues for using Scala 2.11? Is it stable now? Or can I use Scala 2.11 in my spark application and use Spark dist build with 2.10 ? I'm looking forward to migrate to 2.11 for some quasiquote features. Couldn't make it run in 2.10... Cheers, -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github Blog: http://huangjs.github.com/
Re: Is it safe to use Scala 2.11 for Spark build?
Hi Prashant Sharma, It's not even ok to build with scala-2.11 profile on my machine. Just check out the master(c6e0c2ab1c29c184a9302d23ad75e4ccd8060242) run sbt/sbt -Pscala-2.11 clean assembly: .. skip the normal part info] Resolving org.scalamacros#quasiquotes_2.11;2.0.1 ... [warn] module not found: org.scalamacros#quasiquotes_2.11;2.0.1 [warn] local: tried [warn] /Users/yexianjin/.ivy2/local/org.scalamacros/quasiquotes_2.11/2.0.1/ivys/ivy.xml [warn] public: tried [warn] https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] central: tried [warn] https://repo1.maven.org/maven2/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] apache-repo: tried [warn] https://repository.apache.org/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] jboss-repo: tried [warn] https://repository.jboss.org/nexus/content/repositories/releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] mqtt-repo: tried [warn] https://repo.eclipse.org/content/repositories/paho-releases/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] cloudera-repo: tried [warn] https://repository.cloudera.com/artifactory/cloudera-repos/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] mapr-repo: tried [warn] http://repository.mapr.com/maven/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] spring-releases: tried [warn] https://repo.spring.io/libs-release/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] spark-staging: tried [warn] https://oss.sonatype.org/content/repositories/orgspark-project-1085/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] spark-staging-hive13: tried [warn] https://oss.sonatype.org/content/repositories/orgspark-project-1089/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] apache.snapshots: tried [warn] http://repository.apache.org/snapshots/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [warn] Maven2 Local: tried [warn] file:/Users/yexianjin/.m2/repository/org/scalamacros/quasiquotes_2.11/2.0.1/quasiquotes_2.11-2.0.1.pom [info] Resolving jline#jline;2.12 ... [warn] :: [warn] :: UNRESOLVED DEPENDENCIES :: [warn] :: [warn] :: org.scalamacros#quasiquotes_2.11;2.0.1: not found [warn] :: [info] Resolving org.scala-lang#scala-library;2.11.2 ... [warn] [warn] Note: Unresolved dependencies path: [warn] org.scalamacros:quasiquotes_2.11:2.0.1 ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76) [warn] +- org.apache.spark:spark-catalyst_2.11:1.2.0-SNAPSHOT [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Updating {file:/Users/yexianjin/spark/}streaming-twitter... [info] Updating {file:/Users/yexianjin/spark/}streaming-zeromq... [info] Updating {file:/Users/yexianjin/spark/}streaming-flume... [info] Updating {file:/Users/yexianjin/spark/}streaming-mqtt... [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Resolving com.esotericsoftware.minlog#minlog;1.2 ... [info] Updating {file:/Users/yexianjin/spark/}streaming-kafka... [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Resolving org.apache.kafka#kafka_2.11;0.8.0 ... [warn] module not found: org.apache.kafka#kafka_2.11;0.8.0 [warn] local: tried [warn] /Users/yexianjin/.ivy2/local/org.apache.kafka/kafka_2.11/0.8.0/ivys/ivy.xml [warn] public: tried [warn] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] central: tried [warn] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] apache-repo: tried [warn] https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] jboss-repo: tried [warn] https://repository.jboss.org/nexus/content/repositories/releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] mqtt-repo: tried [warn] https://repo.eclipse.org/content/repositories/paho-releases/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] cloudera-repo: tried [warn] https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] mapr-repo: tried [warn] http://repository.mapr.com/maven/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn] spring-releases: tried [warn] https://repo.spring.io/libs-release/org/apache/kafka/kafka_2.11/0.8.0/kafka_2.11-0.8.0.pom [warn]