that's not a bad idea. it would also break the circular dep in versions
that results in spark X's ec2 script installing spark X-1 by default.
best,
matt
On 09/03/2014 01:17 PM, Shivaram Venkataraman wrote:
The spark-ec2 repository isn't a part of Mesos. Back in the days, Spark
used to be hosted in the Mesos github organization as well and so we put
scripts that were used by Spark under the same organization.
FWIW I don't think these scripts belong in the Spark repository. They are
helper scripts that setup EC2 clusters with different components like HDFS,
Spark, Tachyon etc. Also one of the motivations for creating this
repository was the ability to change these scripts without requiring a new
Spark release or a new AMI etc.
We can move the repository to a different github organization like AMPLab
if that'll make sense.
Thanks
Shivaram
On Wed, Sep 3, 2014 at 10:06 AM, Nicholas Chammas <
nicholas.cham...@gmail.com> wrote:
Spawned by this discussion
<https://github.com/apache/spark/pull/1120#issuecomment-54305831>.
See these 2 lines in spark_ec2.py:
- spark_ec2 L42
<
https://github.com/apache/spark/blob/6a72a36940311fcb3429bd34c8818bc7d513115c/ec2/spark_ec2.py#L42
- spark_ec2 L566
<
https://github.com/apache/spark/blob/6a72a36940311fcb3429bd34c8818bc7d513115c/ec2/spark_ec2.py#L566
Why does the spark-ec2 script depend on stuff in the Mesos repo? Should
they be moved to the Spark repo?
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
