RE: Spark cluster set up on EC2 customization

[Sameer Tilak]

Thanks!

Date: Thu, 26 Feb 2015 12:51:21 +0530
Subject: Re: Spark cluster set up on EC2 customization
From: ak...@sigmoidanalytics.com
To: ssti...@live.com
CC: user@spark.apache.org

You can easily add a function (say setup_pig) inside the function setup_cluster 
in this scriptThanksBest Regards

On Thu, Feb 26, 2015 at 7:08 AM, Sameer Tilak ssti...@live.com wrote:






Hi,
I was looking at the documentation for deploying Spark cluster on EC2. 
http://spark.apache.org/docs/latest/ec2-scripts.html
We are using Pig to build the data pipeline and then use MLLib for analytics. I 
was wondering if someone has any experience to include additional 
tools/services such as Pig/Hadoop in the above deployment script?  


  

  

Re: Spark cluster set up on EC2 customization

[Akhil Das]

You can easily add a function (say setup_pig) inside the function
setup_cluster in this script
https://github.com/apache/spark/blob/master/ec2/spark_ec2.py#L649

Thanks
Best Regards

On Thu, Feb 26, 2015 at 7:08 AM, Sameer Tilak ssti...@live.com wrote:

  Hi,

 I was looking at the documentation for deploying Spark cluster on EC2.
 http://spark.apache.org/docs/latest/ec2-scripts.html

 We are using Pig to build the data pipeline and then use MLLib for
 analytics. I was wondering if someone has any experience to include
 additional tools/services such as Pig/Hadoop in the above deployment
 script?

Spark cluster set up on EC2 customization

[Sameer Tilak]

Hi,
I was looking at the documentation for deploying Spark cluster on EC2. 
http://spark.apache.org/docs/latest/ec2-scripts.html
We are using Pig to build the data pipeline and then use MLLib for analytics. I 
was wondering if someone has any experience to include additional 
tools/services such as Pig/Hadoop in the above deployment script?