Updating docs for running on Mesos
The docs for how to run Spark on Mesos have changed very little since 0.6.0, but setting it up is much easier now than then. Does it make sense to revamp with the below changes? You no longer need to build mesos yourself as pre-built versions are available from Mesosphere: http://mesosphere.io/downloads/ And the instructions guide you towards compiling your own distribution of Spark, when you can use the prebuilt versions of Spark as well. I'd like to split that portion of the documentation into two sections, a build-from-scratch section and a use-prebuilt section. The new outline would look something like this: *Running Spark on Mesos* Installing Mesos - using prebuilt (recommended) - pointer to mesosphere's packages - from scratch - (similar to current) Connecting Spark to Mesos - loading distribution into an accessible location - Spark settings Mesos Run Modes - (same as current) Running Alongside Hadoop - (trim this down) Does that work for people? Thanks! Andrew PS Basically all the same: http://spark.apache.org/docs/0.6.0/running-on-mesos.html http://spark.apache.org/docs/0.6.2/running-on-mesos.html http://spark.apache.org/docs/0.7.3/running-on-mesos.html http://spark.apache.org/docs/0.8.1/running-on-mesos.html http://spark.apache.org/docs/0.9.1/running-on-mesos.html https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
Re: [VOTE] Release Apache Spark 1.0.0 (rc5)
SHA-1 is being end-of-lived so I’d actually say switch to 512 for all of them instead. On May 13, 2014, at 6:49 AM, Sean Owen wrote: > On Tue, May 13, 2014 at 9:36 AM, Patrick Wendell wrote: >> The release files, including signatures, digests, etc. can be found at: >> http://people.apache.org/~pwendell/spark-1.0.0-rc5/ > > Good news is that the sigs, MD5 and SHA are all correct. > > Tiny note: the Maven artifacts use SHA1, while the binary artifacts > use SHA512, which took me a bit of head-scratching to figure out. > > If another RC comes out, I might suggest making it SHA1 everywhere? > But there is nothing wrong with these signatures and checksums. > > Now to look at the contents...
Re: Bug is KryoSerializer under Mesos [work-around included]
Hi Matei, Yes, I'm 100% positive the jar on the executors is the same version. I am building everything and deploying myself. Additionally, while debugging the issue, I forked spark's git repo and added additional logging, which I could see in the driver and executors. These debugging jars exhibited the same behaviour. I agree that the code being called in the executors and the driver *should* be the same, but I found that not to be the case. I forgot to mention that this issue only exhibits it's self under mesos; running in local mode or a standalone cluster (with a single worker, all processes running on my laptop) I wasn't able to reproduce the issue. The classes in question should get registered by chill-scala's AllScalaRegistrar here: https://github.com/twitter/chill/blob/0.3.6/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L166 but, for a reason I haven't tracked down, they don't in my mesos executor. I don't really have a way to test if this is an issue specific only to my mesos cluster, or if it exhibits in all mesos clusters. fwiw, I am running spark-0.9.1 with hadoop 2.0.0-mr1-cdh4.6.0 under mesos 0.18.1 On Mon, May 12, 2014 at 12:02 PM, Matei Zaharia wrote: > Hey Soren, are you sure that the JAR you used on the executors is for the > right version of Spark? Maybe they’re running an older version. The Kryo > serializer should be initialized the same way on both. > > Matei > > On May 12, 2014, at 10:39 AM, Soren Macbeth wrote: > > > I finally managed to track down the source of the kryo issues that I was > > having under mesos. > > > > What happens is the for a reason that I haven't tracked down yet, a > handful > > of the scala collection classes from chill-scala down get registered by > the > > mesos executors, but they do all get registered in the driver process. > > > > This led to scala.Some classes which were serialized by the executors > being > > incorrectly deserialized as scala.collections.Wrappers$SeqWrapper in > driver > > during task deserialization, causing a KryoException. > > > > I resolved this issue in my spark job by explicitly registering the > classes > > in my Registrator like so: > > > > > > kryo.register(scala.collection.convert.Wrappers.IteratorWrapper.class); > > kryo.register(scala.collection.convert.Wrappers.SeqWrapper.class); > > kryo.register(scala.collection.convert.Wrappers.MapWrapper.class); > > kryo.register(scala.collection.convert.Wrappers.JListWrapper.class); > > kryo.register(scala.collection.convert.Wrappers.JMapWrapper.class); > > > > Again, I'm not sure why they don't get registered in the mesos executors, > > but I wanted to report wht I found as well as a workaround in case anyone > > else hit this (extraordinarily frustrating) issue again. > > > > Some interactive debugging note are available in this gist: > > > > https://gist.github.com/sorenmacbeth/28707a7a973f7a1982dc > > > > Cheers, > > Soren > >
Re: [jira] [Created] (SPARK-1767) Prefer HDFS-cached replicas when scheduling data-local tasks
Hi Sandy, I assume you are referring to caching added to datanodes via new caching api via NN ? (To preemptively mmap blocks). I have not looked in detail, but does NN tell us about this in block locations? If yes, we can simply make those process local instead of node local for executors on that node. This would simply be a change to hadoop based rdd partitioning (what makes it tricky is to expose currently 'alive' executors to partition) Thanks Mridul On 15-May-2014 3:49 am, "Sandy Ryza (JIRA)" wrote: > Sandy Ryza created SPARK-1767: > - > > Summary: Prefer HDFS-cached replicas when scheduling > data-local tasks > Key: SPARK-1767 > URL: https://issues.apache.org/jira/browse/SPARK-1767 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.0.0 > Reporter: Sandy Ryza > > > > > > > -- > This message was sent by Atlassian JIRA > (v6.2#6252) >
[VOTE] Release Apache Spark 1.0.0 (rc6)
Please vote on releasing the following candidate as Apache Spark version 1.0.0! This patch has a few minor fixes on top of rc5. I've also built the binary artifacts with Hive support enabled so people can test this configuration. When we release 1.0 we might just release both vanilla and Hive-enabled binaries. The tag to be voted on is v1.0.0-rc6 (commit 54133a): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=54133abdce0246f6643a1112a5204afb2c4caa82 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc6/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachestratos-1011 The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc6-docs/ Please vote on releasing this package as Apache Spark 1.0.0! The vote is open until Saturday, May 17, at 20:58 UTC and passes if amajority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ == API Changes == We welcome users to compile Spark applications against 1.0. There are a few API changes in this release. Here are links to the associated upgrade guides - user facing changes have been kept as small as possible. changes to ML vector specification: http://people.apache.org/~pwendell/spark-1.0.0-rc5-docs/mllib-guide.html#from-09-to-10 changes to the Java API: http://people.apache.org/~pwendell/spark-1.0.0-rc5-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark changes to the streaming API: http://people.apache.org/~pwendell/spark-1.0.0-rc5-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x changes to the GraphX API: http://people.apache.org/~pwendell/spark-1.0.0-rc5-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091 coGroup and related functions now return Iterable[T] instead of Seq[T] ==> Call toSeq on the result to restore the old behavior SparkContext.jarOfClass returns Option[String] instead of Seq[String] ==> Call toSeq on the result to restore old behavior
map() + lookup() exception
When using map() and lookup() in conjunction, I get an exception (each independently works fine). I'm using Spark 0.9.0/Scala 2.10.3 val a = sc.parallelize(Array(11)) val m = sc.parallelize(Array((11,21))) a.map(m.lookup(_)(0)).collect 14/05/14 15:03:35 ERROR Executor: Exception in task ID 23 scala.MatchError: null at org.apache.spark.rdd.PairRDDFunctions.lookup(PairRDDFunctions.scala:551) I'm able to work around it with: a.map((_,0)).join(m).map(_._2._2).collect
[RESULT] [VOTE] Release Apache Spark 1.0.0 (rc5)
This vote is cancelled in favor of rc6. On Wed, May 14, 2014 at 1:04 PM, Patrick Wendell wrote: > I'm cancelling this vote in favor of rc6. > > On Tue, May 13, 2014 at 8:01 AM, Sean Owen wrote: >> On Tue, May 13, 2014 at 2:49 PM, Sean Owen wrote: >>> On Tue, May 13, 2014 at 9:36 AM, Patrick Wendell wrote: The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc5/ >>> >>> Good news is that the sigs, MD5 and SHA are all correct. >>> >>> Tiny note: the Maven artifacts use SHA1, while the binary artifacts >>> use SHA512, which took me a bit of head-scratching to figure out. >>> >>> If another RC comes out, I might suggest making it SHA1 everywhere? >>> But there is nothing wrong with these signatures and checksums. >>> >>> Now to look at the contents... >> >> This is a bit of drudgery that probably needs to be done too: a review >> of the LICENSE and NOTICE file. Having dumped the licenses of >> dependencies, I don't believe these reflect all of the software that's >> going to be distributed in 1.0. >> >> (Good news is there's no forbidden license stuff included AFAICT.) >> >> And good news is that NOTICE can be auto-generated, largely, with a >> Maven plugin. This can be done manually for now. >> >> And there is a license plugin that will list all known licenses of >> transitive dependencies so that LICENSE can be filled out fairly >> easily. >> >> What say? want a JIRA with details?
Re: [VOTE] Release Apache Spark 1.0.0 (rc5)
I'm cancelling this vote in favor of rc6. On Tue, May 13, 2014 at 8:01 AM, Sean Owen wrote: > On Tue, May 13, 2014 at 2:49 PM, Sean Owen wrote: >> On Tue, May 13, 2014 at 9:36 AM, Patrick Wendell wrote: >>> The release files, including signatures, digests, etc. can be found at: >>> http://people.apache.org/~pwendell/spark-1.0.0-rc5/ >> >> Good news is that the sigs, MD5 and SHA are all correct. >> >> Tiny note: the Maven artifacts use SHA1, while the binary artifacts >> use SHA512, which took me a bit of head-scratching to figure out. >> >> If another RC comes out, I might suggest making it SHA1 everywhere? >> But there is nothing wrong with these signatures and checksums. >> >> Now to look at the contents... > > This is a bit of drudgery that probably needs to be done too: a review > of the LICENSE and NOTICE file. Having dumped the licenses of > dependencies, I don't believe these reflect all of the software that's > going to be distributed in 1.0. > > (Good news is there's no forbidden license stuff included AFAICT.) > > And good news is that NOTICE can be auto-generated, largely, with a > Maven plugin. This can be done manually for now. > > And there is a license plugin that will list all known licenses of > transitive dependencies so that LICENSE can be filled out fairly > easily. > > What say? want a JIRA with details?
Re: [VOTE] Release Apache Spark 1.0.0 (rc5)
+1 (non-binding) * Built the release from source. * Compiled Java and Scala apps that interact with HDFS against it. * Ran them in local mode. * Ran them against a pseudo-distributed YARN cluster in both yarn-client mode and yarn-cluster mode. On Tue, May 13, 2014 at 9:09 PM, witgo wrote: > You need to set: > spark.akka.frameSize 5 > spark.default.parallelism1 > > > > > > -- Original -- > From: "Madhu";; > Date: Wed, May 14, 2014 09:15 AM > To: "dev"; > > Subject: Re: [VOTE] Release Apache Spark 1.0.0 (rc5) > > > > I just built rc5 on Windows 7 and tried to reproduce the problem described > in > > https://issues.apache.org/jira/browse/SPARK-1712 > > It works on my machine: > > 14/05/13 21:06:47 INFO DAGScheduler: Stage 1 (sum at :17) finished > in 4.548 s > 14/05/13 21:06:47 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks > have all completed, from pool > 14/05/13 21:06:47 INFO SparkContext: Job finished: sum at :17, > took > 4.814991993 s > res1: Double = 5.05E11 > > I used all defaults, no config files were changed. > Not sure if that makes a difference... > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-0-0-rc5-tp6542p6560.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > .
