no prob im verifying ! On Fri, Feb 20, 2015 at 7:17 PM, Konstantin Boudnik <[email protected]> wrote:
> On Fri, Feb 20, 2015 at 02:07PM, jay vyas wrote: > > yes, i think thats what he means, b/c when running on yarn, you read in > the > > conf from hadoop_conf, and you manually send jars like spark-examples.jar > > (which would, otherwise, be available to workers if you had spark > instaled > > on all nodes). > > > > im okay w/ either (standalone, yarn, mesos, whatever) spark deplopyment, > > but we should probably pick one :) > > > > for now, at a minimum, we want to make sure we are able to at least > > leverage HDFS properly, even if we just run standalone spark > > I don't think it is an issue, really. Or at least it wasn't when we did > Spark > initially. Would be great if someone is willing to verify - I have no > cycles > for Spark anymore, honestly. > > Cos > > > On Fri, Feb 20, 2015 at 1:58 PM, Konstantin Boudnik <[email protected]> > wrote: > > > > > On Fri, Feb 20, 2015 at 02:21PM, Evans Ye wrote: > > > > I don't have spark expertise, but here're some points I'm thinking > about. > > > > IIRC, spark standalone do not support Kerberos. And the benefit of > > > > deploying spark on yarn should be that you don't need to maintain > > > packages > > > > by your own on hundreds of node cluster. > > > > > > Could you clarify what you mean by this? Are you referring that you > won't > > > need > > > to install spark-worker on the cluster's nodes? > > > > > > Cos > > > > > > > Not sure if there're downsides. Just want to add some points:) > > > > > > > > 2015-02-20 9:49 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > > > > > They way we're deploying spark is in the standalone mode - I never > > > seen any > > > > > value in using YARN for that, but I guess it's just me. > > > > > > > > > > HDFS use comes with no hassle, AFAIR, the way we setup it up. But > my > > > > > knowledge > > > > > might be a bit outdated... > > > > > > > > > > Cos > > > > > > > > > > On Thu, Feb 19, 2015 at 08:45PM, jay vyas wrote: > > > > > > hi folks. > > > > > > > > > > > > is anyone planning to use spark on yarn or spark w/ hdfs in > bigtop? > > > I > > > > > > havent tried either... > > > > > > > > > > > > - anyone using spark <-> HDFS in bigtop ? Do we need to update > any > > > spark > > > > > > configs to do so ? > > > > > > - we want spark to run on yarn ? standalone ? > > > > > > > > > > > > im spinning some VMs up now, ill let folks know if it works. > > > > > > -- > > > > > > jay vyas > > > > > > > > > > > > > > > > -- > > jay vyas > -- jay vyas
