On Fri, Feb 20, 2015 at 02:07PM, jay vyas wrote: > yes, i think thats what he means, b/c when running on yarn, you read in the > conf from hadoop_conf, and you manually send jars like spark-examples.jar > (which would, otherwise, be available to workers if you had spark instaled > on all nodes). > > im okay w/ either (standalone, yarn, mesos, whatever) spark deplopyment, > but we should probably pick one :) > > for now, at a minimum, we want to make sure we are able to at least > leverage HDFS properly, even if we just run standalone spark
I don't think it is an issue, really. Or at least it wasn't when we did Spark initially. Would be great if someone is willing to verify - I have no cycles for Spark anymore, honestly. Cos > On Fri, Feb 20, 2015 at 1:58 PM, Konstantin Boudnik <[email protected]> wrote: > > > On Fri, Feb 20, 2015 at 02:21PM, Evans Ye wrote: > > > I don't have spark expertise, but here're some points I'm thinking about. > > > IIRC, spark standalone do not support Kerberos. And the benefit of > > > deploying spark on yarn should be that you don't need to maintain > > packages > > > by your own on hundreds of node cluster. > > > > Could you clarify what you mean by this? Are you referring that you won't > > need > > to install spark-worker on the cluster's nodes? > > > > Cos > > > > > Not sure if there're downsides. Just want to add some points:) > > > > > > 2015-02-20 9:49 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > > > They way we're deploying spark is in the standalone mode - I never > > seen any > > > > value in using YARN for that, but I guess it's just me. > > > > > > > > HDFS use comes with no hassle, AFAIR, the way we setup it up. But my > > > > knowledge > > > > might be a bit outdated... > > > > > > > > Cos > > > > > > > > On Thu, Feb 19, 2015 at 08:45PM, jay vyas wrote: > > > > > hi folks. > > > > > > > > > > is anyone planning to use spark on yarn or spark w/ hdfs in bigtop? > > I > > > > > havent tried either... > > > > > > > > > > - anyone using spark <-> HDFS in bigtop ? Do we need to update any > > spark > > > > > configs to do so ? > > > > > - we want spark to run on yarn ? standalone ? > > > > > > > > > > im spinning some VMs up now, ill let folks know if it works. > > > > > -- > > > > > jay vyas > > > > > > > > > > -- > jay vyas
