Nicholas,
Indeed, I was trying to use sbt to speed up the build. My initial
experiments with the maven process took over 50 minutes, which on a 4-core
2014 MacBookPro seems obscene. Then again, after the failed attempt with
sbt, mvn clean package took only 13 minutes, leading me to think that most
of the time was somehow being spent in downloading and building
dependencies.
Anyway, if sbt is supported it would be great to add docs about somewhere,
especially since, as you point out, most devs are using it.
Thanks for your help.
Alex
On Tue, Nov 4, 2014 at 5:42 PM, Nicholas Chammas <nicholas.cham...@gmail.com
> wrote:
> Zinc, I believe, is something you can install and run to speed up your
> Maven builds. It's not required.
>
> I get a bunch of warnings when compiling with Maven, too. Dunno if they
> are expected or not, but things work fine from there on.
>
> Many people do indeed use sbt. I don't know where we have documentation on
> how to use sbt (we recently removed it from the README), but sbt/sbt clean
> followed by sbt/sbt assembly should work fine.
>
> Maven is indeed the "proper" way to build Spark, but building with sbt is
> supported too and most Spark devs I believe use it because it's faster than
> Maven.
>
> Nick
>
> On Tue, Nov 4, 2014 at 8:03 PM, Alessandro Baretta <alexbare...@gmail.com>
> wrote:
>
>> Nicholas,
>>
>> Yes, I saw them, but they refer to maven, and I'm under the impression
>> that sbt is the preferred way of building spark. Is indeed maven the "right
>> way"? Anyway, as per your advice I ctrl-d'ed my sbt shell and have ran `mvn
>> -DskipTests clean package`, which completed successfully. So, indeed, in
>> trying to use sbt I was on a wild goose chase.
>>
>> Here's a couple of glitches I'm seeing. First of all many warnings such
>> as the following:
>>
>> [WARNING] assert(windowedStream2.generatedRDDs.contains(Time(10000)))
>> [WARNING] ^
>> [WARNING]
>> /home/alex/git/spark/streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala:454:
>> inferred existential type
>> scala.collection.mutable.HashMap[org.apache.spark.streaming.Time,org.apache.spark.rdd.RDD[_$2]]
>> forSome { type _$2 }, which cannot be expressed by wildcards, should be
>> enabled
>> by making the implicit value scala.language.existentials visible.
>>
>> [WARNING]
>> /home/alex/git/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/parquet/FakeParquetSerDe.scala:34:
>> @deprecated now takes two arguments; see the scaladoc.
>> [WARNING] @deprecated("No code should depend on FakeParquetHiveSerDe as
>> it is only intended as a " +
>> [WARNING] ^
>>
>> [WARNING]
>> /home/alex/git/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala:435:
>> trait Deserializer in package serde2 is deprecated: see corresponding
>> Javadoc for more information.
>> [WARNING]
>> Utils.getContextOrSparkClassLoader).asInstanceOf[Class[Deserializer]],
>> [WARNING] ^
>>
>> [WARNING]
>> /home/alex/git/spark/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingKMeans.scala:22:
>> imported `StreamingKMeans' is permanently hidden by definition of object
>> StreamingKMeans in package mllib
>> [WARNING] import org.apache.spark.mllib.clustering.StreamingKMeans
>> Are they expected?
>>
>> Also, mvn complains about not having zinc. Is this a problem?
>>
>> [WARNING] Zinc server is not available at port 3030 - reverting to normal
>> incremental compile
>>
>> Alex
>>
>> On Tue, Nov 4, 2014 at 3:11 PM, Nicholas Chammas <
>> nicholas.cham...@gmail.com> wrote:
>>
>>> FWIW, the "official" build instructions are here:
>>> https://github.com/apache/spark#building-spark
>>>
>>> On Tue, Nov 4, 2014 at 5:11 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>>
>>>> I built based on this commit today and the build was successful.
>>>>
>>>> What command did you use ?
>>>>
>>>> Cheers
>>>>
>>>> On Tue, Nov 4, 2014 at 2:08 PM, Alessandro Baretta <
>>>> alexbare...@gmail.com>
>>>> wrote:
>>>>
>>>> > Fellow Sparkers,
>>>> >
>>>> > I am new here and still trying to learn to crawl. Please, bear with
>>>> me.
>>>> >
>>>> > I just pulled f90ad5d from https://github.com/apache/spark.git and am
>>>> > running the compile command in the sbt shell. This is the error I'm
>>>> seeing:
>>>> >
>>>> > [error]
>>>> >
>>>> >
>>>> /home/alex/git/spark/mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:32:
>>>> > object sql is not a member of package org.apache.spark
>>>> > [error] import org.apache.spark.sql.catalyst.types._
>>>> > [error] ^
>>>> >
>>>> > Am I doing something obscenely stupid is the build genuinely broken?
>>>> >
>>>> > Alex
>>>> >
>>>>
>>>
>>>
>>
>
