tes probably after next week. We are
>> > planning
>> > > on
>> > > >>> taking our client work around hive/spark, plus taking over the
>> bigtop
>> > > >>> automation work to modernize and get that fit for human
>>
gt; >>> or org. All our work and puppet modules will be open sourced,
> > > documented,
> > > >>> hopefully start to rally some other folks around effort that find
> it
> > > useful
> > > >>>
> > > >>> Side note, anothe
for human consumption
>> > outside
>> > >>> or org. All our work and puppet modules will be open sourced,
>> > documented,
>> > >>> hopefully start to rally some other folks around effort that find it
>> > useful
>> > >>>
> We have been leveraging serverspec for some basic infrastructure
> > tests, but
> > >>> with bigtop switching over to gradle builds/testing setup in 0.8 we
> > want to
> > >>> include support for that in our own efforts, probably some stuff that
> >
ed and leveraged in spark world for repeatable/tested
> infrastructure
> >>>
> >>> If anyone has any specific automation questions to your environment you
> >>> can drop me a line directly.., will try to help out best I can. Else
> will
> >>> post
ion questions to your environment you
>>> can drop me a line directly.., will try to help out best I can. Else will
>>> post update to dev list once we get on top of our own product release and
>>> the bigtop work
>>>
>>> Nate
>>>
>>>
>
r own product release and
>> the bigtop work
>>
>> Nate
>>
>>
>> -----Original Message-
>> From: David Rowe [mailto:davidr...@gmail.com]
>> Sent: Thursday, October 02, 2014 4:44 PM
>> To: Nicholas Chammas
>> Cc: dev; Shivaram Venkataram
ev list once we get on top of our own product release and
> the bigtop work
>
> Nate
>
>
> -Original Message-
> From: David Rowe [mailto:davidr...@gmail.com]
> Sent: Thursday, October 02, 2014 4:44 PM
> To: Nicholas Chammas
> Cc: dev; Shivaram Venkataraman
>
our own product release and the bigtop
work
Nate
-Original Message-
From: David Rowe [mailto:davidr...@gmail.com]
Sent: Thursday, October 02, 2014 4:44 PM
To: Nicholas Chammas
Cc: dev; Shivaram Venkataraman
Subject: Re: EC2 clusters ready in launch time + 30 seconds
I think this is
I think this is exactly what packer is for. See e.g.
http://www.packer.io/intro/getting-started/build-image.html
On a related note, the current AMI for hvm systems (e.g. m3.*, r3.*) has a
bad package for httpd, whcih causes ganglia not to start. For some reason I
can't get access to the raw AMI to
Is there perhaps a way to define an AMI programmatically? Like, a
collection of base AMI id + list of required stuff to be installed + list
of required configuration changes. I’m guessing that’s what people use
things like Puppet, Ansible, or maybe also AWS CloudFormation for, right?
If we could d
It should be possible to improve cluster launch time if we are careful
about what commands we run during setup. One way to do this would be to
walk down the list of things we do for cluster initialization and see if
there is anything we can do make things faster. Unfortunately this might be
pretty
On Thu, Jul 10, 2014 at 8:10 PM, Nate D'Amico wrote:
> Starting to work through some automation/config stuff for spark stack on
> EC2 with a project, will be focusing the work through the apache bigtop
> effort to start, can then share with spark community directly as things
> progress if people
You are partially correct.
It's not terribly complex, but also not easy to accomplish. Sounds like you
want to manage some partially/fully baked AMI's with the core spark libs and
dependencies already on the image. Main issues that crop up are:
1) image sprawl, as libs/config/defaults/etc cha
14 matches
Mail list logo
