Re: Cross-compatibility of YARN shuffle service

2015-03-26 Thread Sandy Ryza 
Hi Matt,

I'm not sure whether we have documented compatibility guidelines here.
However, a strong goal is to keep the external shuffle service compatible
so that many versions of Spark can run against the same shuffle service.

-Sandy

On Wed, Mar 25, 2015 at 6:44 PM, Matt Cheah mch...@palantir.com wrote:

 Hi everyone,

 I am considering moving from Spark-Standalone to YARN. The context is that
 there are multiple Spark applications that are using different versions of
 Spark that all want to use the same YARN cluster.

 My question is: if I use a single Spark YARN shuffle service jar on the
 Node Manager, will the service work properly with all of the Spark
 applications, regardless of the specific versions of the applications? Or,
 is it it the case that, if I want to use the external shuffle service, I
 need to have all of my applications using the same version of Spark?

 Thanks,

 -Matt Cheah

Cross-compatibility of YARN shuffle service

2015-03-25 Thread Matt Cheah 
Hi everyone,

I am considering moving from Spark-Standalone to YARN. The context is that
there are multiple Spark applications that are using different versions of
Spark that all want to use the same YARN cluster.

My question is: if I use a single Spark YARN shuffle service jar on the Node
Manager, will the service work properly with all of the Spark applications,
regardless of the specific versions of the applications? Or, is it it the
case that, if I want to use the external shuffle service, I need to have all
of my applications using the same version of Spark?

Thanks,

-Matt Cheah




smime.p7s
Description: S/MIME cryptographic signature