Re: libraryDependencies
Hi,
How did you reference "sparksample"? If it ended up in
/Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample I
believe it was referenced as a git-based project in sbt. Is that
correct?
Also, when you "provided" Spark libs you won't be able to run Spark
apps in sbt. See
https://github.com/sbt/sbt-assembly#-provided-configuration. The trick
is to create a test app that executes main of your standalone app.
Pozdrawiam,
Jacek Laskowski
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Tue, Jul 26, 2016 at 9:18 PM, Martin Somers wrote:
>
> my build file looks like
>
> libraryDependencies ++= Seq(
> // other dependencies here
> "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
> "org.apache.spark" %% "spark-mllib_2.11" % "1.6.0",
> "org.scalanlp" % "breeze_2.11" % "0.7",
> // native libraries are not included by default. add this if
> you want them (as of 0.7)
> // native libraries greatly improve performance, but increase
> jar sizes.
> "org.scalanlp" % "breeze-natives_2.11" % "0.7",
> )
>
> not 100% sure on the version numbers if they are indeed correct
> getting an error of
>
> [info] Resolving jline#jline;2.12.1 ...
> [info] Done updating.
> [info] Compiling 1 Scala source to
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/target/scala-2.11/classes...
> [error]
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:2:
> object mllib is not a member of package org.apache.spark
> [error] import org.apache.spark.mllib.linalg.distributed.RowMatrix
>
> ...
>
>
> Im trying to import in
>
> import org.apache.spark.mllib.linalg.distributed.RowMatrix
> import org.apache.spark.mllib.linalg.SingularValueDecomposition
>
> import org.apache.spark.mllib.linalg.{Vector, Vectors}
>
>
> import breeze.linalg._
> import breeze.linalg.{ Matrix => B_Matrix }
> import breeze.linalg.{ Vector => B_Matrix }
> import breeze.linalg.DenseMatrix
>
> object MyApp {
> def main(args: Array[String]): Unit = {
> //code here
> }
>
>
> It might not be the correct way of doing this
>
> Anyone got any suggestion
> tks
> M
>
>
>
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: libraryDependencies
libraryDependencies ++= Seq(
// other dependencies here
"org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
"org.apache.spark" %% "spark-mllib" % "1.6.2" % "provided",
"org.scalanlp" %% "breeze" % "0.12",
// native libraries are not included by default. add this if
you want them (as of 0.7)
// native libraries greatly improve performance, but increase
jar sizes.
"org.scalanlp" %% "breeze-natives" % "0.12",
)
On Tue, Jul 26, 2016 at 12:49 PM, Martin Somers wrote:
> cheers - I updated
>
> libraryDependencies ++= Seq(
> // other dependencies here
> "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
> "org.apache.spark" %% "spark-mllib_2.10" % "1.6.2",
> "org.scalanlp" %% "breeze" % "0.12",
> // native libraries are not included by default. add this if
> you want them (as of 0.7)
> // native libraries greatly improve performance, but
> increase jar sizes.
> "org.scalanlp" %% "breeze-natives" % "0.12",
> )
>
> and getting similar error
>
> Compiling 1 Scala source to
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/target/scala-2.11/classes...
> [error]
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:2:
> object mllib is not a member of package org.apache.spark
> [error] import org.apache.spark.mllib.linalg.distributed.RowMatrix
> [error] ^
> [error]
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:3:
> object mllib is not a member of package org.apache.spark
> [error] import org.apache.spark.mllib.linalg.SingularValueDecomposition
> [error] ^
> [error]
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:5:
> object mllib is not a member of package org.apache.spark
> [error] import org.apache.spark.mllib.linalg.{Vector, Vectors}
> [error] ^
> [error]
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:8:
> not found: object breeze
>
> On Tue, Jul 26, 2016 at 8:36 PM, Michael Armbrust
> wrote:
>
>> Also, you'll want all of the various spark versions to be the same.
>>
>> On Tue, Jul 26, 2016 at 12:34 PM, Michael Armbrust <
>> mich...@databricks.com> wrote:
>>
>>> If you are using %% (double) then you do not need _2.11.
>>>
>>> On Tue, Jul 26, 2016 at 12:18 PM, Martin Somers
>>> wrote:
>>>
>>>>
>>>> my build file looks like
>>>>
>>>> libraryDependencies ++= Seq(
>>>> // other dependencies here
>>>> "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
>>>> "org.apache.spark" %% "spark-mllib_2.11" % "1.6.0",
>>>> "org.scalanlp" % "breeze_2.11" % "0.7",
>>>> // native libraries are not included by default. add this
>>>> if you want them (as of 0.7)
>>>> // native libraries greatly improve performance, but
>>>> increase jar sizes.
>>>> "org.scalanlp" % "breeze-natives_2.11" % "0.7",
>>>> )
>>>>
>>>> not 100% sure on the version numbers if they are indeed correct
>>>> getting an error of
>>>>
>>>> [info] Resolving jline#jline;2.12.1 ...
>>>> [info] Done updating.
>>>> [info] Compiling 1 Scala source to
>>>> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/target/scala-2.11/classes...
>>>> [error]
>>>> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:2:
>>>> object mllib is not a member of package org.apache.spark
>>>> [error] import org.apache.spark.mllib.linalg.distributed.RowMatrix
>>>>
>>>> ...
>>>>
>>>>
>>>> Im trying to import in
>>>>
>>>> import org.apache.spark.mllib.linalg.distributed.RowMatrix
>>>> import org.apache.spark.mllib.linalg.SingularValueDecomposition
>>>>
>>>> import org.apache.spark.mllib.linalg.{Vector, Vectors}
>>>>
>>>>
>>>> import breeze.linalg._
>>>> import breeze.linalg.{ Matrix => B_Matrix }
>>>> import breeze.linalg.{ Vector => B_Matrix }
>>>> import breeze.linalg.DenseMatrix
>>>>
>>>> object MyApp {
>>>> def main(args: Array[String]): Unit = {
>>>> //code here
>>>> }
>>>>
>>>>
>>>> It might not be the correct way of doing this
>>>>
>>>> Anyone got any suggestion
>>>> tks
>>>> M
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> M
>
Re: libraryDependencies
cheers - I updated
libraryDependencies ++= Seq(
// other dependencies here
"org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
"org.apache.spark" %% "spark-mllib_2.10" % "1.6.2",
"org.scalanlp" %% "breeze" % "0.12",
// native libraries are not included by default. add this if
you want them (as of 0.7)
// native libraries greatly improve performance, but increase
jar sizes.
"org.scalanlp" %% "breeze-natives" % "0.12",
)
and getting similar error
Compiling 1 Scala source to
/Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/target/scala-2.11/classes...
[error]
/Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:2:
object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.linalg.distributed.RowMatrix
[error] ^
[error]
/Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:3:
object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.linalg.SingularValueDecomposition
[error] ^
[error]
/Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:5:
object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.linalg.{Vector, Vectors}
[error] ^
[error]
/Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:8:
not found: object breeze
On Tue, Jul 26, 2016 at 8:36 PM, Michael Armbrust
wrote:
> Also, you'll want all of the various spark versions to be the same.
>
> On Tue, Jul 26, 2016 at 12:34 PM, Michael Armbrust > wrote:
>
>> If you are using %% (double) then you do not need _2.11.
>>
>> On Tue, Jul 26, 2016 at 12:18 PM, Martin Somers
>> wrote:
>>
>>>
>>> my build file looks like
>>>
>>> libraryDependencies ++= Seq(
>>> // other dependencies here
>>> "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
>>> "org.apache.spark" %% "spark-mllib_2.11" % "1.6.0",
>>> "org.scalanlp" % "breeze_2.11" % "0.7",
>>> // native libraries are not included by default. add this
>>> if you want them (as of 0.7)
>>> // native libraries greatly improve performance, but
>>> increase jar sizes.
>>> "org.scalanlp" % "breeze-natives_2.11" % "0.7",
>>> )
>>>
>>> not 100% sure on the version numbers if they are indeed correct
>>> getting an error of
>>>
>>> [info] Resolving jline#jline;2.12.1 ...
>>> [info] Done updating.
>>> [info] Compiling 1 Scala source to
>>> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/target/scala-2.11/classes...
>>> [error]
>>> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:2:
>>> object mllib is not a member of package org.apache.spark
>>> [error] import org.apache.spark.mllib.linalg.distributed.RowMatrix
>>>
>>> ...
>>>
>>>
>>> Im trying to import in
>>>
>>> import org.apache.spark.mllib.linalg.distributed.RowMatrix
>>> import org.apache.spark.mllib.linalg.SingularValueDecomposition
>>>
>>> import org.apache.spark.mllib.linalg.{Vector, Vectors}
>>>
>>>
>>> import breeze.linalg._
>>> import breeze.linalg.{ Matrix => B_Matrix }
>>> import breeze.linalg.{ Vector => B_Matrix }
>>> import breeze.linalg.DenseMatrix
>>>
>>> object MyApp {
>>> def main(args: Array[String]): Unit = {
>>> //code here
>>> }
>>>
>>>
>>> It might not be the correct way of doing this
>>>
>>> Anyone got any suggestion
>>> tks
>>> M
>>>
>>>
>>>
>>>
>>
>
--
M
Re: libraryDependencies
Also, you'll want all of the various spark versions to be the same.
On Tue, Jul 26, 2016 at 12:34 PM, Michael Armbrust
wrote:
> If you are using %% (double) then you do not need _2.11.
>
> On Tue, Jul 26, 2016 at 12:18 PM, Martin Somers wrote:
>
>>
>> my build file looks like
>>
>> libraryDependencies ++= Seq(
>> // other dependencies here
>> "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
>> "org.apache.spark" %% "spark-mllib_2.11" % "1.6.0",
>> "org.scalanlp" % "breeze_2.11" % "0.7",
>> // native libraries are not included by default. add this
>> if you want them (as of 0.7)
>> // native libraries greatly improve performance, but
>> increase jar sizes.
>> "org.scalanlp" % "breeze-natives_2.11" % "0.7",
>> )
>>
>> not 100% sure on the version numbers if they are indeed correct
>> getting an error of
>>
>> [info] Resolving jline#jline;2.12.1 ...
>> [info] Done updating.
>> [info] Compiling 1 Scala source to
>> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/target/scala-2.11/classes...
>> [error]
>> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:2:
>> object mllib is not a member of package org.apache.spark
>> [error] import org.apache.spark.mllib.linalg.distributed.RowMatrix
>>
>> ...
>>
>>
>> Im trying to import in
>>
>> import org.apache.spark.mllib.linalg.distributed.RowMatrix
>> import org.apache.spark.mllib.linalg.SingularValueDecomposition
>>
>> import org.apache.spark.mllib.linalg.{Vector, Vectors}
>>
>>
>> import breeze.linalg._
>> import breeze.linalg.{ Matrix => B_Matrix }
>> import breeze.linalg.{ Vector => B_Matrix }
>> import breeze.linalg.DenseMatrix
>>
>> object MyApp {
>> def main(args: Array[String]): Unit = {
>> //code here
>> }
>>
>>
>> It might not be the correct way of doing this
>>
>> Anyone got any suggestion
>> tks
>> M
>>
>>
>>
>>
>
Re: libraryDependencies
If you are using %% (double) then you do not need _2.11.
On Tue, Jul 26, 2016 at 12:18 PM, Martin Somers wrote:
>
> my build file looks like
>
> libraryDependencies ++= Seq(
> // other dependencies here
> "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
> "org.apache.spark" %% "spark-mllib_2.11" % "1.6.0",
> "org.scalanlp" % "breeze_2.11" % "0.7",
> // native libraries are not included by default. add this if
> you want them (as of 0.7)
> // native libraries greatly improve performance, but
> increase jar sizes.
> "org.scalanlp" % "breeze-natives_2.11" % "0.7",
> )
>
> not 100% sure on the version numbers if they are indeed correct
> getting an error of
>
> [info] Resolving jline#jline;2.12.1 ...
> [info] Done updating.
> [info] Compiling 1 Scala source to
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/target/scala-2.11/classes...
> [error]
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:2:
> object mllib is not a member of package org.apache.spark
> [error] import org.apache.spark.mllib.linalg.distributed.RowMatrix
>
> ...
>
>
> Im trying to import in
>
> import org.apache.spark.mllib.linalg.distributed.RowMatrix
> import org.apache.spark.mllib.linalg.SingularValueDecomposition
>
> import org.apache.spark.mllib.linalg.{Vector, Vectors}
>
>
> import breeze.linalg._
> import breeze.linalg.{ Matrix => B_Matrix }
> import breeze.linalg.{ Vector => B_Matrix }
> import breeze.linalg.DenseMatrix
>
> object MyApp {
> def main(args: Array[String]): Unit = {
> //code here
> }
>
>
> It might not be the correct way of doing this
>
> Anyone got any suggestion
> tks
> M
>
>
>
>
libraryDependencies
my build file looks like
libraryDependencies ++= Seq(
// other dependencies here
"org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
"org.apache.spark" %% "spark-mllib_2.11" % "1.6.0",
"org.scalanlp" % "breeze_2.11" % "0.7",
// native libraries are not included by default. add this if
you want them (as of 0.7)
// native libraries greatly improve performance, but increase
jar sizes.
"org.scalanlp" % "breeze-natives_2.11" % "0.7",
)
not 100% sure on the version numbers if they are indeed correct
getting an error of
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 1 Scala source to
/Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/target/scala-2.11/classes...
[error]
/Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:2:
object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.linalg.distributed.RowMatrix
...
Im trying to import in
import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.mllib.linalg.SingularValueDecomposition
import org.apache.spark.mllib.linalg.{Vector, Vectors}
import breeze.linalg._
import breeze.linalg.{ Matrix => B_Matrix }
import breeze.linalg.{ Vector => B_Matrix }
import breeze.linalg.DenseMatrix
object MyApp {
def main(args: Array[String]): Unit = {
//code here
}
It might not be the correct way of doing this
Anyone got any suggestion
tks
M
Re: sbt package failed: wrong libraryDependencies for spark-streaming?
Hi Tathagata, I was using the "raw" tag in the web-editor. Seems like this doesn't make it into the mail. Here's the message again, this time without those tags: I've added the following to my spark-env.sh: SPARK_CLASSPATH="/disk.b/spark-master-2014-07-28/external/twitter/target/spark-streaming-twitter_2.10-1.1.0-SNAPSHOT.jar" I can now execute import org.apache.spark.streaming.twitter._ import org.apache.spark.streaming.StreamingContext._ without an error in the shell. However, I will get an error when doing this: scala> val ssc = new StreamingContext(sc, Seconds(1)) ssc: org.apache.spark.streaming.StreamingContext = org.apache.spark.streaming.StreamingContext@6e78177b scala> val tweets = TwitterUtils.createStream(ssc, "twitter.txt") error: bad symbolic reference. A signature in TwitterUtils.class refers to term twitter4j in package which is not available. It may be completely missing from the current classpath, or the version on the classpath might be incompatible with the version used when compiling TwitterUtils.class. What am I missing? Do I have to import another jar? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103p6.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: sbt package failed: wrong libraryDependencies for spark-streaming?
Hey Simon, The stuff you are trying to show - logs, contents of spark-env.sh, etc. are missing from the email. At least I am not able to see it (viewing through gmail). Are you pasting screenshots? Those might get blocked out somehow! TD On Thu, Jul 31, 2014 at 6:55 PM, durin wrote: > I've added the following to my spark-env.sh: > > > I can now execute > without an error in the shell. However, I will get an error when doing this: > > > > > What am I missing? Do I have to import another jar? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103p11108.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: sbt package failed: wrong libraryDependencies for spark-streaming?
I've added the following to my spark-env.sh: I can now execute without an error in the shell. However, I will get an error when doing this: What am I missing? Do I have to import another jar? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103p11108.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: sbt package failed: wrong libraryDependencies for spark-streaming?
Hi Tathagata, I didn't mean to say this was an error. According to the other thread I linked, right now there shouldn't be any conflicts, so I wanted to use streaming in the shell for easy testing. I thought I had to create my own project in which I'd add streaming as a dependency, but if I can add it into the config that' even simpler and gets rid of my sbt problem. I'll try that. Simon -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103p11106.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: sbt package failed: wrong libraryDependencies for spark-streaming?
I dont see the error. The twitter stuff (as well as kafka and flume stuff) are treated as "external" projects and are not included in the spark shell. This is because we dont want the dependencies of such non-core functionalities to cause random conflicts with that of core spark. Hence its not possible to use twitter spark streaming in the spark shell. Unless you explicitly add the jar generated in external/twitter/target/scala... To the spark shells extra class path (see spark config page) TD On Thu, Jul 31, 2014 at 5:48 PM, durin wrote: > As suggested here > < http://apache-spark-user-list.1001560.n3.nabble.com/import-org-apache-spark-streaming-twitter-in-Shell-tp9665p9743.html > > , I want to create a minimal project using sbt to be able to use > org.apache.spark.streaming.twitter in the shell. My Spark version is the > latest Master branch compiled with maven3. > > I was following this > < http://spark.apache.org/docs/latest/quick-start.html#standalone-applications > > tutorial to create a small standalone application. > > My simple.sbt file looks like this: > > > > > However, I will get this error when using sbt package: > > > > > I don't quite understand the whole standalone thing yet. sbt is installed on > a different machine than my Spark Cluster, but the tutorial never mentions > any variables that would have to be set. So I did assume sbt didn't actually > need a Spark installation on the machine. > Also, I could imagine that 1.0.1 isn't the correct version when I compiled > it myself. > > > Can someone help me out? In the end, all I want for now is to be able to use > twitterUtils in the spark-shell. > > > Simon > > > > -- > View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
sbt package failed: wrong libraryDependencies for spark-streaming?
As suggested here <http://apache-spark-user-list.1001560.n3.nabble.com/import-org-apache-spark-streaming-twitter-in-Shell-tp9665p9743.html> , I want to create a minimal project using sbt to be able to use org.apache.spark.streaming.twitter in the shell. My Spark version is the latest Master branch compiled with maven3. I was following this <http://spark.apache.org/docs/latest/quick-start.html#standalone-applications> tutorial to create a small standalone application. My simple.sbt file looks like this: However, I will get this error when using sbt package: I don't quite understand the whole standalone thing yet. sbt is installed on a different machine than my Spark Cluster, but the tutorial never mentions any variables that would have to be set. So I did assume sbt didn't actually need a Spark installation on the machine. Also, I could imagine that 1.0.1 isn't the correct version when I compiled it myself. Can someone help me out? In the end, all I want for now is to be able to use twitterUtils in the spark-shell. Simon -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
