Re: Do we really need mesos or yarn? or is standalone sufficent?

[Anant Chintamaneni]

+1

Sent from my iPhone

> On Dec 16, 2016, at 10:45 AM,  
>  wrote:
> 
> In my experience, Standalone works very well in small cluster where there 
> isn’t anything else running.
>  
> Bigger cluster or shared resources, YARN takes a win, surpassing the overhead 
> of spawning containers as opposed to a background running worker.
>  
> Best is if you try both, if standalone is good enough keep it till you need 
> more. Otherwise, try YARN or MESOS depending on the rest of your components.
>  
> 2cents
>  
> Saif
>  
> From: kant kodali [mailto:kanth...@gmail.com] 
> Sent: Friday, December 16, 2016 3:14 AM
> To: user @spark
> Subject: Do we really need mesos or yarn? or is standalone sufficent?
>  
> Do we really need mesos or yarn? or is standalone sufficient for production 
> systems? I understand the difference but I don't know the capabilities of 
> standalone cluster. does anyone have experience deploying standalone in the 
> production?
>  
>  

Re: Error reading HDFS file using spark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1

[anant]

I've received the same error with Spark built using Maven. It turns out that
mesos-0.13.0 depends on protobuf-2.4.1 which is causing the clash at
runtime. Protobuf included by Akka is shaded and doesn't cause any problems.

The solution is to update the mesos dependency to 0.18.0 in spark's pom.xml.
Rebuilding the JAR with this configuration solves the issue.

-Anant



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Error-reading-HDFS-file-using-spark-0-9-0-hadoop-2-2-0-incompatible-protobuf-2-5-and-2-4-1-tp2158p4286.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.