Re: [Openstack] GlusterFS project proposal online

[Joshua McKenty]

Shehjar,

While I'll be able to provide more detailed feedback on the proposal itself 
after I've had a chance to digest it, can I please ask that you relocate it on 
the wiki? The Governance/Proposed section is for proposals for policy to be 
reviewed by the PPB, and not intended for blueprints.

Thanks!

Joshua McKenty
Piston Cloud Computing, Inc.
(650) 283-6846
jos...@piston.cc



On 2011-06-07, at 1:42 PM, Shehjar Tikoo wrote:

> For those who havent seen this already, GlusterFS proposal is now available 
> at:
> 
> http://wiki.openstack.org/Governance/Proposed/OpenStack%20Cloud%20Storage%20(Gluster)
> 
> Inputs welcome. Please do CC openst...@gluster.com
> 
> Thanks
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Problems with re-bundle

[Muhammad Atif]

Hi Openstack gurus,

We have recently shifted from eucalyptus to openstack and so far, finding it 
quite stable. However, we are having terrible time re-bundling an existing 
image. 

The re-bundled image is able to complete the boot process upto mountall: but 
the 
cloud-init never fires up.

Here is what we have done so far, please let us know what exactly is wrong on 
our side.

1- Downloaded the ubuntu1010 image from openstack site as follows: 
wget 
http://c0179148.cdn1.cloudfiles.rackspacecloud.com/ubuntu1010-UEC-localuser-image.tar.gz

2- publish this image:
uec-publish-tarball ubuntu1010-UEC-localuser-image.tar.gz ubuntu x86_64
3- Now we run the image using euca-run-instance and it boots up fine, we are 
also able to login using the private key and able to install software.

4- When we want to save the instace into an image (rebundle), following 
sequence 
is performed

-   rm -r /etc/udev/rules.d/70-*

-   copy the nove certificates and source .novarc

-   euca-bundle-vol -c ${EC2_CERT} -k ${EC2_PRIVATE_KEY} -u ${EC2_USER_ID} 
--ec2cert ${NOVA_CERT} --no-inherit --kernel aki-0XXX -d /mnt -r x86_64 -p 
ubuntu-1010-rebundle -s 2048 -e /var/lib/dhcp3 


-  euca-upload-bundle -b ubuntu -m  /mnt/ubuntu-1010-rebundle.manifest.xml

   -  euca-register ubuntu/ubuntu-1010-rebundle.manifest.xml

5- Now we fire up this image once it has been successfully uploaded; we get 
stuck after plymouth-splash. Comapred to the original image which started 
cloud-init after this point. The image is replying to pings but refusing ssh. 
It 
can be some odd ssh key issue?

[0.384064] md: autorun ...
[0.384435] md: ... autorun DONE.
[0.386063] EXT3-fs: barriers not enabled
[0.386946] kjournald starting.  Commit interval 5 seconds
[0.387731] EXT3-fs (vda): mounted filesystem with ordered data mode
[0.388644] VFS: Mounted root (ext3 filesystem) readonly on device 252:0.
[0.389738] devtmpfs: mounted
[0.390218] Freeing unused kernel memory: 828k freed
[0.391214] Write protecting the kernel read-only data: 10240k
[0.392153] Freeing unused kernel memory: 308k freed
[0.393078] Freeing unused kernel memory: 1612k freed
mountall: Disconnected from Plymouth
init: plymouth main process (48) killed by SEGV signal
init: plymouth-splash main process (323) terminated with status 2


Any help is greatly appreciated. I am willing to provide you with other logs if 
requested.  



Regards,
Atif
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] GlusterFS project proposal online

[Shehjar Tikoo]

Hi Joshua, this in fact is a proposal albeit not a policy proposal but a 
project proposal(not a blueprint either, which is WIP). Is there a more 
appropriate place for project proposals?


Regards
-Shehjar

Joshua McKenty wrote:

Shehjar,

While I'll be able to provide more detailed feedback on the proposal
itself after I've had a chance to digest it, can I please ask that you
relocate it on the wiki? The Governance/Proposed section is for
proposals for policy to be reviewed by the PPB, and not intended for
blueprints.

Thanks!

Joshua McKenty Piston Cloud Computing, Inc. (650) 283-6846 
jos...@piston.cc




On 2011-06-07, at 1:42 PM, Shehjar Tikoo wrote:


For those who havent seen this already, GlusterFS proposal is now
available at:

http://wiki.openstack.org/Governance/Proposed/OpenStack%20Cloud%20Storage%20(Gluster)


Inputs welcome. Please do CC openst...@gluster.com

Thanks

___ Mailing list:
https://launchpad.net/~openstack Post to :
openstack@lists.launchpad.net Unsubscribe :
https://launchpad.net/~openstack More help   :
https://help.launchpad.net/ListHelp





___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] GlusterFS project proposal online

[Joshua McKenty]

If it's a proposal for work to be done within an existing openstack project 
(e.g., nova-volume), then it would be a blueprint regardless. If it's actually 
a proposal for a new openstack subproject, then this would be the appropriate 
spot, but I would ask that you use the 
http://wiki.openstack.org/Governance/Approved/NewProjectProcess, which involves 
publishing source code prior to new project proposal. In that case, I would 
still suggest you use the wiki for discussion of the proposed project aims and 
technical decisions - just make it a top-level page.


Joshua McKenty
Piston Cloud Computing, Inc.
(650) 283-6846
jos...@piston.cc



On 2011-06-07, at 5:24 PM, Shehjar Tikoo wrote:

> Hi Joshua, this in fact is a proposal albeit not a policy proposal but a 
> project proposal(not a blueprint either, which is WIP). Is there a more 
> appropriate place for project proposals?
> 
> Regards
> -Shehjar
> 
> Joshua McKenty wrote:
>> Shehjar,
>> While I'll be able to provide more detailed feedback on the proposal
>> itself after I've had a chance to digest it, can I please ask that you
>> relocate it on the wiki? The Governance/Proposed section is for
>> proposals for policy to be reviewed by the PPB, and not intended for
>> blueprints.
>> Thanks!
>> Joshua McKenty Piston Cloud Computing, Inc. (650) 283-6846 jos...@piston.cc
>> On 2011-06-07, at 1:42 PM, Shehjar Tikoo wrote:
>>> For those who havent seen this already, GlusterFS proposal is now
>>> available at:
>>> http://wiki.openstack.org/Governance/Proposed/OpenStack%20Cloud%20Storage%20(Gluster)
>>> Inputs welcome. Please do CC openst...@gluster.com
>>> Thanks
>>> ___ Mailing list:
>>> https://launchpad.net/~openstack Post to :
>>> openstack@lists.launchpad.net Unsubscribe :
>>> https://launchpad.net/~openstack More help   :
>>> https://help.launchpad.net/ListHelp
> 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Problems with re-bundle

[Thierry Carrez]

Muhammad Atif wrote:
> 1- Downloaded the ubuntu1010 image from openstack site as follows: 
> 
> wget 
> http://c0179148.cdn1.cloudfiles.rackspacecloud.com/ubuntu1010-UEC-localuser-image.tar.gz

Any chance you could reproduce with an official Ubuntu image (downloaded
from http://uec-images.ubuntu.com/) and file a bug on Launchpad ? This
looks like a guest issue and with official images we  can get help from
Ubuntu cloud image developers if need be...

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] GlusterFS project proposal online

[John Purrier]

On initial reading this is great stuff. As Josh McKenty pointed out you
should relocate the wiki page and then create a blueprint so we can track
the project.

I will have some specific responses and a couple of questions after spending
some time grokking the proposal.

John (johnpur)

-Original Message-
From: openstack-bounces+john=openstack@lists.launchpad.net
[mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf
Of Shehjar Tikoo
Sent: Tuesday, June 07, 2011 12:43 AM
To: openstack@lists.launchpad.net
Subject: [Openstack] GlusterFS project proposal online

For those who havent seen this already, GlusterFS proposal is now available
at:

http://wiki.openstack.org/Governance/Proposed/OpenStack%20Cloud%20Storage%20
(Gluster)

Inputs welcome. Please do CC openst...@gluster.com

Thanks

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Coalescing and Coordinating Continuous Integration, Developer Tools, Function and Smoke Testings Efforts

[Monty Taylor]

Hey all!

It's time to both expand and contract some things around automation and
testing. Thusfar all of the work on Jenkins and Tarmac and other
'official' build and testing automation has been done exclusively by
Soren and me, In the mean team, several people have been hacking on
projects that range from helpful to spectacular, and because of lack of
communication, only some of them are directly useful to the project's
infrastructure. I'd like to change this. Specificaly, I'd like to make
sure we can expand how we're working so that there is greater visibility
in to what/how we doing things in Jenkins land, and that the barrier to
entry to helping out isn't terrible. And I'd like to contract the number
of unrelated efforts in to something resembling a plan.

a) I'm working on geting relevant bits from our jenkins into to VC
somewhere so that it's easier for us all to collaborate. I'm using the
launchpad.net/openstack-ci project for that.

b) I configured the OpenStack Jenkins to use Launchpad's OpenID provider
as a SSO source - so now I can use Launchpad teams to manage who can do
what on Jenkins - which means better federation of that management.

c) I want to set up a regular (say weekly) IRC meeting for everyone
interested in CI, dev tools and functional/smoke testing. (although for
sake of sanity, lets keep "interested" to be "interested in working on"
I was thinking perhaps the hour right before the normal openstack one...
anybody have strong thoughts? Of the topics/actions to be discussed are
managing/moving forward with the current work on the testing
infrastructure, and having discussions about how/if various work by
folks can be leveraged by the Jenkins that is the gatekeeper over
openstack branches.

d) Make sure people know what we're doing over here in OpenStack
CI/Tools land, so that they are aware of the tools already present so
that they don't duplicate effort, and also of the current todo list in
case anyone wants to help. Our Jenkins already does several things that
it turns out nobody knows about. That's gotta get fixed. (Speaking of-
if there's any Java folks laying around out there sick of all the
python, there are a couple of tasks that can use love - more to come on
that)

The meeting should be simple this time around - I propose meeting the
hour before the openstack meeting, starting next week. I don't think we
really need a separate mailing list, but I'm open to opinions... Also,
in the mean time, ping me back and let me know if you're wanting to be
part of this.

Thanks!
Monty

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Coalescing and Coordinating Continuous Integration, Developer Tools, Function and Smoke Testings Efforts

[John Purrier]

Monty, this is great stuff, glad to see significant movement on the
CI/testing front.

The OpenStack PPB has claimed the hour prior to the OpenStack weekly
meeting, can you go 2 hours earlier?

Thanks,

John

-Original Message-
From: openstack-bounces+john=openstack@lists.launchpad.net
[mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf
Of Monty Taylor
Sent: Tuesday, June 07, 2011 10:44 AM
To: openstack@lists.launchpad.net
Subject: [Openstack] Coalescing and Coordinating Continuous Integration,
Developer Tools, Function and Smoke Testings Efforts

Hey all!

It's time to both expand and contract some things around automation and
testing. Thusfar all of the work on Jenkins and Tarmac and other
'official' build and testing automation has been done exclusively by
Soren and me, In the mean team, several people have been hacking on
projects that range from helpful to spectacular, and because of lack of
communication, only some of them are directly useful to the project's
infrastructure. I'd like to change this. Specificaly, I'd like to make
sure we can expand how we're working so that there is greater visibility
in to what/how we doing things in Jenkins land, and that the barrier to
entry to helping out isn't terrible. And I'd like to contract the number
of unrelated efforts in to something resembling a plan.

a) I'm working on geting relevant bits from our jenkins into to VC
somewhere so that it's easier for us all to collaborate. I'm using the
launchpad.net/openstack-ci project for that.

b) I configured the OpenStack Jenkins to use Launchpad's OpenID provider
as a SSO source - so now I can use Launchpad teams to manage who can do
what on Jenkins - which means better federation of that management.

c) I want to set up a regular (say weekly) IRC meeting for everyone
interested in CI, dev tools and functional/smoke testing. (although for
sake of sanity, lets keep "interested" to be "interested in working on"
I was thinking perhaps the hour right before the normal openstack one...
anybody have strong thoughts? Of the topics/actions to be discussed are
managing/moving forward with the current work on the testing
infrastructure, and having discussions about how/if various work by
folks can be leveraged by the Jenkins that is the gatekeeper over
openstack branches.

d) Make sure people know what we're doing over here in OpenStack
CI/Tools land, so that they are aware of the tools already present so
that they don't duplicate effort, and also of the current todo list in
case anyone wants to help. Our Jenkins already does several things that
it turns out nobody knows about. That's gotta get fixed. (Speaking of-
if there's any Java folks laying around out there sick of all the
python, there are a couple of tasks that can use love - more to come on
that)

The meeting should be simple this time around - I propose meeting the
hour before the openstack meeting, starting next week. I don't think we
really need a separate mailing list, but I'm open to opinions... Also,
in the mean time, ping me back and let me know if you're wanting to be
part of this.

Thanks!
Monty

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Overview of CI/Testing

[Monty Taylor]

Hey everybody,

Here's a quick write up on what we have at the moment and what the plan
is moving forward, as it stands today. In case you weren't aware, the
Jenkins instance sits at:

http://jenkins.openstack.org

If you think you want to help administer/hack on our Jenkins, that is
managed through the openstack-ci-admins team on Launchpad.

As most of you are aware, the main trunk branches for nova, swift,
glance and burrow are all managed by Jenkins and Tarmac. What this means
in short is that jenkins/tarmac finds approved merge proposals, runs the
in-tree unit tests on them, and if successful, pushes back to trunk.
Additionally, we have jenkins jobs which run pep8, pylint and code
coverage jobs and produce reports. For instance:

http://jenkins.openstack.org/job/nova-coverage/815/cobertura/?

Once the code has been merged, we have jenkins jobs which produce
tarballs, and debian source packages which are pushed up to launchpad
PPAs so that effectively every trunk commit winds up having an
associated package.

There is also a feature that it seems no one knew about which allows
developers to submit branch URLs to jenkins to have it run its tests on
that branch. For each project, this is ${project_name}-param, so for
instance:

http://jenkins.openstack.org/job/nova-param/

Will allow anyone in the ~nova team to submit a branch and have jenkins
pull it and run on it what it would do via tarmac - helpful when fixing
something tarmac complained about.

We are planning to get rid of tarmac and get that integrated in as a
Jenkins plugin, because otherwise supporting things like NTT's wish for
stricter coverage metrics as branch gatekeepers, or smoketesting is
preventing branches from getting in become really baroque to support.

** If you like hacking on Java - this should be a fun little project.
Send me a note and I can outline what I was going to do and if you want
it it's all yours. **

Moving forward, the big ticket item is testing not only in-tree unit
tests, but installing things and testing that.

Currently, we have jenkins spinning up a cloud server, copying the most
recently built debs from the last successful build to that server,
installing them, starting things up and then running the smoketests
against the installed code. Currently the install/setup is being done by
hand rather than via chef purely so that I could walk through what's
actually needed to get a single-node minimal install working and collect
information on bugs/workarounds that have to happen.

http://jenkins.openstack.org/job/nova-smoketests

The test failures here are due to config issues which should go away
once I migrate that job to using chef for the node code - so I expect
that to go green soon.

The next step then is to replace the use of shell commands over ssh with
the chef recipes. (now that I have a good handle on what's going
on/needed there, following the existing chef automation work is much
more fruitful)

After that, we'll move from launching cloud servers using the one-off
libcloud-based python script I wrote to using the jenkins jclouds
plugin. (if for no other reason that to make sure that all of the moving
parts of this of any complexity are sensibly checked in and have
lifecycles that people can hack on.)

** If you like hacking on Java - this is a project that's both helpful
for us as consumers in OpenStack - but also is something that
effectively will allow other people to use OpenStack to manage Jenkins
build slaves... SO - come and hack on/improviding the jenkins
jclouds-plugin. It's at https://github.com/jenkinsci/jclouds-plugin **

And then we need to apply this to swift/glance as well.


That's just testing API in a VM though, and doesn't get us to testing
actual bare-metal deployment or integration testing. At Rackspace, we
have some machines set aside at the moment, and have had others offer
chunks of machines to test various combinations of things. At its heart,
the abstract version of this looks fairly identical to the smoketests
job - pxe boot machines, shove version to be tested on them, run tests.
However, there are several moving bits on the best way to actually do
the how. At the moment, the fine folks at rPath have a Jenkins
installing and testing rPath OpenStack images, so Mihai and I are going
to look at getting that setup ported to our Jenkins. However, although
that will be an excellent test of code, as our main target platform is
Ubuntu, we're also looking at doing a straight-up cobbler install using
generated .debs. In any case, this is the bit which is still in the
planning and discussion phase, but so far all of the conversations I've
had with folks have been great - and I'd love to get more folks involved
in that (thus this email)

However- latent goal here is that whatever mechanism we're having
Jenkins use to deploy OpenStack onto real hardware should be consumable
and one that actual people might actually use - otherwise what the heck
are we testing?

Additionally, as you may have s

Re: [Openstack] Coalescing and Coordinating Continuous Integration, Developer Tools, Function and Smoke Testings Efforts

[Monty Taylor]

It has been pointed out that the hour before the openstack meeting
belongs to PPB - so let's make that a proposal for 2 hours before
#openstack-meeting.

On 06/07/2011 10:44 AM, Monty Taylor wrote:
> Hey all!
> 
> It's time to both expand and contract some things around automation and
> testing. Thusfar all of the work on Jenkins and Tarmac and other
> 'official' build and testing automation has been done exclusively by
> Soren and me, In the mean team, several people have been hacking on
> projects that range from helpful to spectacular, and because of lack of
> communication, only some of them are directly useful to the project's
> infrastructure. I'd like to change this. Specificaly, I'd like to make
> sure we can expand how we're working so that there is greater visibility
> in to what/how we doing things in Jenkins land, and that the barrier to
> entry to helping out isn't terrible. And I'd like to contract the number
> of unrelated efforts in to something resembling a plan.
> 
> a) I'm working on geting relevant bits from our jenkins into to VC
> somewhere so that it's easier for us all to collaborate. I'm using the
> launchpad.net/openstack-ci project for that.
> 
> b) I configured the OpenStack Jenkins to use Launchpad's OpenID provider
> as a SSO source - so now I can use Launchpad teams to manage who can do
> what on Jenkins - which means better federation of that management.
> 
> c) I want to set up a regular (say weekly) IRC meeting for everyone
> interested in CI, dev tools and functional/smoke testing. (although for
> sake of sanity, lets keep "interested" to be "interested in working on"
> I was thinking perhaps the hour right before the normal openstack one...
> anybody have strong thoughts? Of the topics/actions to be discussed are
> managing/moving forward with the current work on the testing
> infrastructure, and having discussions about how/if various work by
> folks can be leveraged by the Jenkins that is the gatekeeper over
> openstack branches.
> 
> d) Make sure people know what we're doing over here in OpenStack
> CI/Tools land, so that they are aware of the tools already present so
> that they don't duplicate effort, and also of the current todo list in
> case anyone wants to help. Our Jenkins already does several things that
> it turns out nobody knows about. That's gotta get fixed. (Speaking of-
> if there's any Java folks laying around out there sick of all the
> python, there are a couple of tasks that can use love - more to come on
> that)
> 
> The meeting should be simple this time around - I propose meeting the
> hour before the openstack meeting, starting next week. I don't think we
> really need a separate mailing list, but I'm open to opinions... Also,
> in the mean time, ping me back and let me know if you're wanting to be
> part of this.
> 
> Thanks!
> Monty
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Coalescing and Coordinating Continuous Integration, Developer Tools, Function and Smoke Testings Efforts

[Devin Carlen]

Hey Monty,

+1 to all of this.  We appreciate all the work you guys have done keeping this 
stuff going.  

On Jun 7, 2011, at 8:44 AM, Monty Taylor wrote:

> Hey all!
> 
> It's time to both expand and contract some things around automation and
> testing. Thusfar all of the work on Jenkins and Tarmac and other
> 'official' build and testing automation has been done exclusively by
> Soren and me, In the mean team, several people have been hacking on
> projects that range from helpful to spectacular, and because of lack of
> communication, only some of them are directly useful to the project's
> infrastructure. I'd like to change this. Specificaly, I'd like to make
> sure we can expand how we're working so that there is greater visibility
> in to what/how we doing things in Jenkins land, and that the barrier to
> entry to helping out isn't terrible. And I'd like to contract the number
> of unrelated efforts in to something resembling a plan.
> 
> a) I'm working on geting relevant bits from our jenkins into to VC
> somewhere so that it's easier for us all to collaborate. I'm using the
> launchpad.net/openstack-ci project for that.
> 
> b) I configured the OpenStack Jenkins to use Launchpad's OpenID provider
> as a SSO source - so now I can use Launchpad teams to manage who can do
> what on Jenkins - which means better federation of that management.
> 
> c) I want to set up a regular (say weekly) IRC meeting for everyone
> interested in CI, dev tools and functional/smoke testing. (although for
> sake of sanity, lets keep "interested" to be "interested in working on"
> I was thinking perhaps the hour right before the normal openstack one...
> anybody have strong thoughts? Of the topics/actions to be discussed are
> managing/moving forward with the current work on the testing
> infrastructure, and having discussions about how/if various work by
> folks can be leveraged by the Jenkins that is the gatekeeper over
> openstack branches.
> 
> d) Make sure people know what we're doing over here in OpenStack
> CI/Tools land, so that they are aware of the tools already present so
> that they don't duplicate effort, and also of the current todo list in
> case anyone wants to help. Our Jenkins already does several things that
> it turns out nobody knows about. That's gotta get fixed. (Speaking of-
> if there's any Java folks laying around out there sick of all the
> python, there are a couple of tasks that can use love - more to come on
> that)
> 
> The meeting should be simple this time around - I propose meeting the
> hour before the openstack meeting, starting next week. I don't think we
> really need a separate mailing list, but I'm open to opinions... Also,
> in the mean time, ping me back and let me know if you're wanting to be
> part of this.
> 
> Thanks!
> Monty
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Coalescing and Coordinating Continuous Integration, Developer Tools, Function and Smoke Testings Efforts

[Anne Gentle]

+1 from me too, definitely a crucial behind-the-scenes job. I can haz
automated doc builds now thanks to you and CI, and I do appreciate it.

*Anne Gentle*
a...@openstack.org
 my blog  | my
book|
LinkedIn  |
Delicious|
Twitter 
On Tue, Jun 7, 2011 at 1:05 PM, Devin Carlen  wrote:

> Hey Monty,
>
> +1 to all of this.  We appreciate all the work you guys have done keeping
> this stuff going.
>
> On Jun 7, 2011, at 8:44 AM, Monty Taylor wrote:
>
> > Hey all!
> >
> > It's time to both expand and contract some things around automation and
> > testing. Thusfar all of the work on Jenkins and Tarmac and other
> > 'official' build and testing automation has been done exclusively by
> > Soren and me, In the mean team, several people have been hacking on
> > projects that range from helpful to spectacular, and because of lack of
> > communication, only some of them are directly useful to the project's
> > infrastructure. I'd like to change this. Specificaly, I'd like to make
> > sure we can expand how we're working so that there is greater visibility
> > in to what/how we doing things in Jenkins land, and that the barrier to
> > entry to helping out isn't terrible. And I'd like to contract the number
> > of unrelated efforts in to something resembling a plan.
> >
> > a) I'm working on geting relevant bits from our jenkins into to VC
> > somewhere so that it's easier for us all to collaborate. I'm using the
> > launchpad.net/openstack-ci project for that.
> >
> > b) I configured the OpenStack Jenkins to use Launchpad's OpenID provider
> > as a SSO source - so now I can use Launchpad teams to manage who can do
> > what on Jenkins - which means better federation of that management.
> >
> > c) I want to set up a regular (say weekly) IRC meeting for everyone
> > interested in CI, dev tools and functional/smoke testing. (although for
> > sake of sanity, lets keep "interested" to be "interested in working on"
> > I was thinking perhaps the hour right before the normal openstack one...
> > anybody have strong thoughts? Of the topics/actions to be discussed are
> > managing/moving forward with the current work on the testing
> > infrastructure, and having discussions about how/if various work by
> > folks can be leveraged by the Jenkins that is the gatekeeper over
> > openstack branches.
> >
> > d) Make sure people know what we're doing over here in OpenStack
> > CI/Tools land, so that they are aware of the tools already present so
> > that they don't duplicate effort, and also of the current todo list in
> > case anyone wants to help. Our Jenkins already does several things that
> > it turns out nobody knows about. That's gotta get fixed. (Speaking of-
> > if there's any Java folks laying around out there sick of all the
> > python, there are a couple of tasks that can use love - more to come on
> > that)
> >
> > The meeting should be simple this time around - I propose meeting the
> > hour before the openstack meeting, starting next week. I don't think we
> > really need a separate mailing list, but I'm open to opinions... Also,
> > in the mean time, ping me back and let me know if you're wanting to be
> > part of this.
> >
> > Thanks!
> > Monty
> >
> > ___
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Coalescing and Coordinating Continuous Integration, Developer Tools, Function and Smoke Testings Efforts

[Joseph Heck]

love it! Im good with the updated proposed meeting time - will miss it this 
week, but am looking forward to jumping in. 

- joe

On Jun 7, 2011, at 9:33 AM, Monty Taylor  wrote:

> It has been pointed out that the hour before the openstack meeting
> belongs to PPB - so let's make that a proposal for 2 hours before
> #openstack-meeting.
> 
> On 06/07/2011 10:44 AM, Monty Taylor wrote:
>> Hey all!
>> 
>> It's time to both expand and contract some things around automation and
>> testing. Thusfar all of the work on Jenkins and Tarmac and other
>> 'official' build and testing automation has been done exclusively by
>> Soren and me, In the mean team, several people have been hacking on
>> projects that range from helpful to spectacular, and because of lack of
>> communication, only some of them are directly useful to the project's
>> infrastructure. I'd like to change this. Specificaly, I'd like to make
>> sure we can expand how we're working so that there is greater visibility
>> in to what/how we doing things in Jenkins land, and that the barrier to
>> entry to helping out isn't terrible. And I'd like to contract the number
>> of unrelated efforts in to something resembling a plan.
>> 
>> a) I'm working on geting relevant bits from our jenkins into to VC
>> somewhere so that it's easier for us all to collaborate. I'm using the
>> launchpad.net/openstack-ci project for that.
>> 
>> b) I configured the OpenStack Jenkins to use Launchpad's OpenID provider
>> as a SSO source - so now I can use Launchpad teams to manage who can do
>> what on Jenkins - which means better federation of that management.
>> 
>> c) I want to set up a regular (say weekly) IRC meeting for everyone
>> interested in CI, dev tools and functional/smoke testing. (although for
>> sake of sanity, lets keep "interested" to be "interested in working on"
>> I was thinking perhaps the hour right before the normal openstack one...
>> anybody have strong thoughts? Of the topics/actions to be discussed are
>> managing/moving forward with the current work on the testing
>> infrastructure, and having discussions about how/if various work by
>> folks can be leveraged by the Jenkins that is the gatekeeper over
>> openstack branches.
>> 
>> d) Make sure people know what we're doing over here in OpenStack
>> CI/Tools land, so that they are aware of the tools already present so
>> that they don't duplicate effort, and also of the current todo list in
>> case anyone wants to help. Our Jenkins already does several things that
>> it turns out nobody knows about. That's gotta get fixed. (Speaking of-
>> if there's any Java folks laying around out there sick of all the
>> python, there are a couple of tasks that can use love - more to come on
>> that)
>> 
>> The meeting should be simple this time around - I propose meeting the
>> hour before the openstack meeting, starting next week. I don't think we
>> really need a separate mailing list, but I'm open to opinions... Also,
>> in the mean time, ping me back and let me know if you're wanting to be
>> part of this.
>> 
>> Thanks!
>> Monty
>> 
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] OpenStack CLA Information

[Stephen Spector]

Developers:

I want to make everyone aware of an email that is coming to many developers
in the OpenStack community this afternoon.  I am doing a review of the
existing Individual and Corporate CLA agreements and have found 72 Corporate
CLAs and 336 Individual CLAs. The OpenStack Wiki requests that all
Individuals completing the CLA are also required to have someone at their
company complete the Corporate CLA. Of course, all developers not working on
behalf of an organization are not required to have a Corporate CLA. Here is
the Wiki text:
* Sign the CLA  : Every developer needs to
sign the Individual Contributor License agreement
 .  If
you are contributing on behalf of a company, someone at your company needs
to sign the Corporate Contributor License Agreement


The email is coming from Summer Fouche who is the Community Manager Intern
and assisting me in contacting all Individual CLA developers whose company
or organization we could not determine. We are requesting your company or
organization name so we can properly meet the requirement of having all
Corporate CLAs completed by companies with developers working on OpenStack.

If you have any questions about this activity or want to discuss further
please contact me privately so I can provide you with more details. This
initiative is being conducted to ensure that the OpenStack community is
properly managing the IP being submitted by various developers around the
world. 

Thank you,

- - - 
Stephen Spector, Rackspace
OpenStack Community Manager
stephen.spec...@openstack.org
OpenStack Blog   | @opnstk_
 com 
_mgr 
Office  +1 (512) 539-1162 | Mobile +1 (210) 415-0930


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Overview of CI/Testing

[Dan Prince]

Hi Monty,

I spent a few moments to put together some more detailed notes on SmokeStack:

  http://wiki.openstack.org/smokestack

Screenshots and a link to the live server are on that wiki page.

At this point the ability to specify custom nova and glance branches is 
supported. Each job runs a set of smoke tests that cover both the OS and EC2 
APIs.

--

We've been torpedoing quite a few branches with SmokeStack lately. Its proving 
very useful in doing reviews and helping make sure we get things right before 
they land in trunk.

Dan

-Original Message-
From: "Monty Taylor" 
Sent: Tuesday, June 7, 2011 12:30pm
To: "openstack@lists.launchpad.net" 
Cc: "Dan Prince" , "Mihai Ibanescu" , 
"Peter J. Pouliot" 
Subject: Overview of CI/Testing

Hey everybody,

Here's a quick write up on what we have at the moment and what the plan
is moving forward, as it stands today. In case you weren't aware, the
Jenkins instance sits at:

http://jenkins.openstack.org

If you think you want to help administer/hack on our Jenkins, that is
managed through the openstack-ci-admins team on Launchpad.

As most of you are aware, the main trunk branches for nova, swift,
glance and burrow are all managed by Jenkins and Tarmac. What this means
in short is that jenkins/tarmac finds approved merge proposals, runs the
in-tree unit tests on them, and if successful, pushes back to trunk.
Additionally, we have jenkins jobs which run pep8, pylint and code
coverage jobs and produce reports. For instance:

http://jenkins.openstack.org/job/nova-coverage/815/cobertura/?

Once the code has been merged, we have jenkins jobs which produce
tarballs, and debian source packages which are pushed up to launchpad
PPAs so that effectively every trunk commit winds up having an
associated package.

There is also a feature that it seems no one knew about which allows
developers to submit branch URLs to jenkins to have it run its tests on
that branch. For each project, this is ${project_name}-param, so for
instance:

http://jenkins.openstack.org/job/nova-param/

Will allow anyone in the ~nova team to submit a branch and have jenkins
pull it and run on it what it would do via tarmac - helpful when fixing
something tarmac complained about.

We are planning to get rid of tarmac and get that integrated in as a
Jenkins plugin, because otherwise supporting things like NTT's wish for
stricter coverage metrics as branch gatekeepers, or smoketesting is
preventing branches from getting in become really baroque to support.

** If you like hacking on Java - this should be a fun little project.
Send me a note and I can outline what I was going to do and if you want
it it's all yours. **

Moving forward, the big ticket item is testing not only in-tree unit
tests, but installing things and testing that.

Currently, we have jenkins spinning up a cloud server, copying the most
recently built debs from the last successful build to that server,
installing them, starting things up and then running the smoketests
against the installed code. Currently the install/setup is being done by
hand rather than via chef purely so that I could walk through what's
actually needed to get a single-node minimal install working and collect
information on bugs/workarounds that have to happen.

http://jenkins.openstack.org/job/nova-smoketests

The test failures here are due to config issues which should go away
once I migrate that job to using chef for the node code - so I expect
that to go green soon.

The next step then is to replace the use of shell commands over ssh with
the chef recipes. (now that I have a good handle on what's going
on/needed there, following the existing chef automation work is much
more fruitful)

After that, we'll move from launching cloud servers using the one-off
libcloud-based python script I wrote to using the jenkins jclouds
plugin. (if for no other reason that to make sure that all of the moving
parts of this of any complexity are sensibly checked in and have
lifecycles that people can hack on.)

** If you like hacking on Java - this is a project that's both helpful
for us as consumers in OpenStack - but also is something that
effectively will allow other people to use OpenStack to manage Jenkins
build slaves... SO - come and hack on/improviding the jenkins
jclouds-plugin. It's at https://github.com/jenkinsci/jclouds-plugin **

And then we need to apply this to swift/glance as well.


That's just testing API in a VM though, and doesn't get us to testing
actual bare-metal deployment or integration testing. At Rackspace, we
have some machines set aside at the moment, and have had others offer
chunks of machines to test various combinations of things. At its heart,
the abstract version of this looks fairly identical to the smoketests
job - pxe boot machines, shove version to be tested on them, run tests.
However, there are several moving bits on the best way to actually do
the how. At the moment, the fine folks at rPath have a Jenkins
installing and testing rPat

Re: [Openstack] Overview of CI/Testing

[Andy Smith]

Thanks for the update Monty :)


>  That's just testing API in a VM though, and doesn't get us to testing
> actual bare-metal deployment or integration testing. At Rackspace, we
> have some machines set aside at the moment, and have had others offer
> chunks of machines to test various combinations of things. At its heart,
> the abstract version of this looks fairly identical to the smoketests
> job - pxe boot machines, shove version to be tested on them, run tests.
> However, there are several moving bits on the best way to actually do
> the how. At the moment, the fine folks at rPath have a Jenkins
> installing and testing rPath OpenStack images, so Mihai and I are going
> to look at getting that setup ported to our Jenkins. However, although
> that will be an excellent test of code, as our main target platform is
> Ubuntu, we're also looking at doing a straight-up cobbler install using
> generated .debs.


Jesse and I had already gotten quite far along using chef to do the
provisioning of baremetal boxes once we'd pxe booted them into ubuntu, it
seems like chef or puppet (our current preference is chef) should be used
there as well instead of generated .debs.

At the moment the two closest things to being "official" installations for
us (me? are the chef recipes and the nova.sh script (the nova.sh script
obviously being only targeted at testing and dev though), those are what we
use to verify that the system is functional and I think we'd like to use
chef or puppet for baremetal deployments as well.

TL;DR: Can we focus on the chef recipes instead of on .debs?



> In any case, this is the bit which is still in the
> planning and discussion phase, but so far all of the conversations I've
> had with folks have been great - and I'd love to get more folks involved
> in that (thus this email)
>
> However- latent goal here is that whatever mechanism we're having
> Jenkins use to deploy OpenStack onto real hardware should be consumable
> and one that actual people might actually use - otherwise what the heck
> are we testing?
>
> Additionally, as you may have surmised, it is also a goal to run as much
> of this as possible from the OpenStack Jenkins, because that way we can
> as a project choose to incorporate as much of the feedback/results of
> various forms of testing directly in to branch testing/approval as we
> want. For some things (spinning up 20 node OpenStack clusters) doing it
> on every merge proposal or giving all devs the ability to click a button
> and have it run on their branch will likely be overkill - but if it
> turns out not to be, it would be great to have the ability to do it.
>
> End goal is to have:
>  - publicly accessible and usable system for testing and build automation
>  - resources that it uses to spin up clouds in order to test them are
> themselves usable by people to spin up clouds
>  - tooling around this is done in a manner that makes us of and
> contributes back to existing projects (jenkins plugins, patches back to
> cobbler/orchestra/whatever)
>
> If you didn't read my _other_ long email from a few moments ago, actual
> discussion of getting this done - and figuring out other people's
> needs/tools and how to integrate them - is hopefully happening next week
> right before the regular openstack-meeting. In the mean time, please
> either flame on right here in list, or ping me back personally.
>
> Thanks everyone!
> Monty
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Overview of CI/Testing

[Monty Taylor]

On 06/07/2011 02:36 PM, Dan Prince wrote:
> Hi Monty,
> 
> I spent a few moments to put together some more detailed notes on
> SmokeStack:
> 
> http://wiki.openstack.org/smokestack
> 
> Screenshots and a link to the live server are on that wiki page.
> 
> At this point the ability to specify custom nova and glance branches
> is supported. Each job runs a set of smoke tests that cover both the
> OS and EC2 APIs.

Excellent. I'm looking forward to working with you to figure out how we
can integrate this in a way which is usable/consumable by the Jenkins
system.

> We've been torpedoing quite a few branches with SmokeStack lately.
> Its proving very useful in doing reviews and helping make sure we get
> things right before they land in trunk.

Yup. I couldn't agree more - the more pre-merge testing that can happen
the better!

Monty

> -Original Message- From: "Monty Taylor"
>  Sent: Tuesday, June 7, 2011 12:30pm To:
> "openstack@lists.launchpad.net"  Cc:
> "Dan Prince" , "Mihai Ibanescu"
> , "Peter J. Pouliot"  Subject:
> Overview of CI/Testing
> 
> Hey everybody,
> 
> Here's a quick write up on what we have at the moment and what the
> plan is moving forward, as it stands today. In case you weren't
> aware, the Jenkins instance sits at:
> 
> http://jenkins.openstack.org
> 
> If you think you want to help administer/hack on our Jenkins, that
> is managed through the openstack-ci-admins team on Launchpad..
> 
> As most of you are aware, the main trunk branches for nova, swift, 
> glance and burrow are all managed by Jenkins and Tarmac. What this
> means in short is that jenkins/tarmac finds approved merge proposals,
> runs the in-tree unit tests on them, and if successful, pushes back
> to trunk. Additionally, we have jenkins jobs which run pep8, pylint
> and code coverage jobs and produce reports. For instance:
> 
> http://jenkins.openstack.org/job/nova-coverage/815/cobertura/?
> 
> Once the code has been merged, we have jenkins jobs which produce 
> tarballs, and debian source packages which are pushed up to
> launchpad PPAs so that effectively every trunk commit winds up having
> an associated package.
> 
> There is also a feature that it seems no one knew about which allows 
> developers to submit branch URLs to jenkins to have it run its tests
> on that branch. For each project, this is ${project_name}-param, so
> for instance:
> 
> http://jenkins.openstack..org/job/nova-param/
> 
> Will allow anyone in the ~nova team to submit a branch and have
> jenkins pull it and run on it what it would do via tarmac - helpful
> when fixing something tarmac complained about.
> 
> We are planning to get rid of tarmac and get that integrated in as a 
> Jenkins plugin, because otherwise supporting things like NTT's wish
> for stricter coverage metrics as branch gatekeepers, or smoketesting
> is preventing branches from getting in become really baroque to
> support.
> 
> ** If you like hacking on Java - this should be a fun little
> project. Send me a note and I can outline what I was going to do and
> if you want it it's all yours. **
> 
> Moving forward, the big ticket item is testing not only in-tree unit 
> tests, but installing things and testing that.
> 
> Currently, we have jenkins spinning up a cloud server, copying the
> most recently built debs from the last successful build to that
> server, installing them, starting things up and then running the
> smoketests against the installed code. Currently the install/setup is
> being done by hand rather than via chef purely so that I could walk
> through what's actually needed to get a single-node minimal install
> working and collect information on bugs/workarounds that have to
> happen.
> 
> http://jenkins.openstack.org/job/nova-smoketests
> 
> The test failures here are due to config issues which should go away 
> once I migrate that job to using chef for the node code - so I
> expect that to go green soon.
> 
> The next step then is to replace the use of shell commands over ssh
> with the chef recipes. (now that I have a good handle on what's
> going on/needed there, following the existing chef automation work is
> much more fruitful)
> 
> After that, we'll move from launching cloud servers using the
> one-off libcloud-based python script I wrote to using the jenkins
> jclouds plugin. (if for no other reason that to make sure that all of
> the moving parts of this of any complexity are sensibly checked in
> and have lifecycles that people can hack on.)
> 
> ** If you like hacking on Java - this is a project that's both
> helpful for us as consumers in OpenStack - but also is something
> that effectively will allow other people to use OpenStack to manage
> Jenkins build slaves... SO - come and hack on/improviding the
> jenkins jclouds-plugin. It's at
> https://github.com/jenkinsci/jclouds-plugin **
> 
> And then we need to apply this to swift/glance as well.
> 
> 
> That's just testing API in a VM though, and doesn't get us to
> testing actual bare-metal 

Re: [Openstack] Overview of CI/Testing

[Devin Carlen]

+1 for chef over debs!


On Jun 7, 2011, at 12:38 PM, Andy Smith wrote:

> Thanks for the update Monty :)
>  
>  That's just testing API in a VM though, and doesn't get us to testing
> actual bare-metal deployment or integration testing. At Rackspace, we
> have some machines set aside at the moment, and have had others offer
> chunks of machines to test various combinations of things. At its heart,
> the abstract version of this looks fairly identical to the smoketests
> job - pxe boot machines, shove version to be tested on them, run tests.
> However, there are several moving bits on the best way to actually do
> the how. At the moment, the fine folks at rPath have a Jenkins
> installing and testing rPath OpenStack images, so Mihai and I are going
> to look at getting that setup ported to our Jenkins. However, although
> that will be an excellent test of code, as our main target platform is
> Ubuntu, we're also looking at doing a straight-up cobbler install using
> generated .debs.
> 
> Jesse and I had already gotten quite far along using chef to do the 
> provisioning of baremetal boxes once we'd pxe booted them into ubuntu, it 
> seems like chef or puppet (our current preference is chef) should be used 
> there as well instead of generated .debs.
> 
> At the moment the two closest things to being "official" installations for us 
> (me? are the chef recipes and the nova.sh script (the nova.sh script 
> obviously being only targeted at testing and dev though), those are what we 
> use to verify that the system is functional and I think we'd like to use chef 
> or puppet for baremetal deployments as well.
> 
> TL;DR: Can we focus on the chef recipes instead of on .debs?
> 
>  
> In any case, this is the bit which is still in the
> planning and discussion phase, but so far all of the conversations I've
> had with folks have been great - and I'd love to get more folks involved
> in that (thus this email)
> 
> However- latent goal here is that whatever mechanism we're having
> Jenkins use to deploy OpenStack onto real hardware should be consumable
> and one that actual people might actually use - otherwise what the heck
> are we testing?
> 
> Additionally, as you may have surmised, it is also a goal to run as much
> of this as possible from the OpenStack Jenkins, because that way we can
> as a project choose to incorporate as much of the feedback/results of
> various forms of testing directly in to branch testing/approval as we
> want. For some things (spinning up 20 node OpenStack clusters) doing it
> on every merge proposal or giving all devs the ability to click a button
> and have it run on their branch will likely be overkill - but if it
> turns out not to be, it would be great to have the ability to do it.
> 
> End goal is to have:
>  - publicly accessible and usable system for testing and build automation
>  - resources that it uses to spin up clouds in order to test them are
> themselves usable by people to spin up clouds
>  - tooling around this is done in a manner that makes us of and
> contributes back to existing projects (jenkins plugins, patches back to
> cobbler/orchestra/whatever)
> 
> If you didn't read my _other_ long email from a few moments ago, actual
> discussion of getting this done - and figuring out other people's
> needs/tools and how to integrate them - is hopefully happening next week
> right before the regular openstack-meeting. In the mean time, please
> either flame on right here in list, or ping me back personally.
> 
> Thanks everyone!
> Monty
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Coalescing and Coordinating Continuous Integration, Developer Tools, Function and Smoke Testings Efforts

[Andy Smith]

On Tue, Jun 7, 2011 at 8:44 AM, Monty Taylor  wrote:

> Hey all!
>
> It's time to both expand and contract some things around automation and
> testing. Thusfar all of the work on Jenkins and Tarmac and other
> 'official' build and testing automation has been done exclusively by
> Soren and me, In the mean team, several people have been hacking on
> projects that range from helpful to spectacular, and because of lack of
> communication, only some of them are directly useful to the project's
> infrastructure. I'd like to change this. Specificaly, I'd like to make
> sure we can expand how we're working so that there is greater visibility
> in to what/how we doing things in Jenkins land, and that the barrier to
> entry to helping out isn't terrible. And I'd like to contract the number
> of unrelated efforts in to something resembling a plan.
>
> a) I'm working on geting relevant bits from our jenkins into to VC
> somewhere so that it's easier for us all to collaborate. I'm using the
> launchpad.net/openstack-ci project for that.


Would you like to check in the various configs I sent you from the anso
setups, should I propose that merge, or do you think that this is not the
proper place for jenkins configs + testing scripts?


>
> b) I configured the OpenStack Jenkins to use Launchpad's OpenID provider
> as a SSO source - so now I can use Launchpad teams to manage who can do
> what on Jenkins - which means better federation of that management.
>

Yay for no longer having manual config there :)


>
> c) I want to set up a regular (say weekly) IRC meeting for everyone
> interested in CI, dev tools and functional/smoke testing. (although for
> sake of sanity, lets keep "interested" to be "interested in working on"
> I was thinking perhaps the hour right before the normal openstack one...
> anybody have strong thoughts? Of the topics/actions to be discussed are
> managing/moving forward with the current work on the testing
> infrastructure, and having discussions about how/if various work by
> folks can be leveraged by the Jenkins that is the gatekeeper over
> openstack branches.
>

I suppose I should be on that list.


>
> d) Make sure people know what we're doing over here in OpenStack
> CI/Tools land, so that they are aware of the tools already present so
> that they don't duplicate effort, and also of the current todo list in
> case anyone wants to help. Our Jenkins already does several things that
> it turns out nobody knows about. That's gotta get fixed. (Speaking of-
> if there's any Java folks laying around out there sick of all the
> python, there are a couple of tasks that can use love - more to come on
> that)
>
> The meeting should be simple this time around - I propose meeting the
> hour before the openstack meeting, starting next week. I don't think we
> really need a separate mailing list, but I'm open to opinions... Also,
> in the mean time, ping me back and let me know if you're wanting to be
> part of this.
>
> Thanks!
> Monty
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Overview of CI/Testing

[Monty Taylor]

On 06/07/2011 02:38 PM, Andy Smith wrote:
> Thanks for the update Monty :)

My pleasure as always. :)

>  That's just testing API in a VM though, and doesn't get us to testing
> actual bare-metal deployment or integration testing. At Rackspace, we
> have some machines set aside at the moment, and have had others offer
> chunks of machines to test various combinations of things. At its heart,
> the abstract version of this looks fairly identical to the smoketests
> job - pxe boot machines, shove version to be tested on them, run tests.
> However, there are several moving bits on the best way to actually do
> the how. At the moment, the fine folks at rPath have a Jenkins
> installing and testing rPath OpenStack images, so Mihai and I are going
> to look at getting that setup ported to our Jenkins. However, although
> that will be an excellent test of code, as our main target platform is
> Ubuntu, we're also looking at doing a straight-up cobbler install using
> generated .debs.
> 
> 
> Jesse and I had already gotten quite far along using chef to do the
> provisioning of baremetal boxes once we'd pxe booted them into ubuntu,
> it seems like chef or puppet (our current preference is chef) should be
> used there as well instead of generated .debs.

I have every intention of moving the current work that is running to be
based on the chef work you did... but I do not view chef and .debs to be
mutually exclusive options. The goal here is to be able to use chef to
install and configure the official debs. If this is not possible, then
there are fundamental flaws that must be fixed.

> At the moment the two closest things to being "official" installations
> for us (me? are the chef recipes and the nova.sh script (the nova.sh
> script obviously being only targeted at testing and dev though), those
> are what we use to verify that the system is functional and I think we'd
> like to use chef or puppet for baremetal deployments as well.
>
> TL;DR: Can we focus on the chef recipes instead of on .debs?

nova.sh is great for devs, but isn't really something I'd imagine would
be used as the basis of a production deployment (which is kind of the
point of doing post-install smoke testing) And again, chef can happily
install the software from the produced debs.

It's not really just about debs - for the rPath based testing backend,
we'll obviously be testing RPMs. But in general, testing the packaged
software that we ship is kind of important.

To sum up: yes to using the chef recipes, no to "instead of".

Monty

> In any case, this is the bit which is still in the
> planning and discussion phase, but so far all of the conversations I've
> had with folks have been great - and I'd love to get more folks involved
> in that (thus this email)
> 
> However- latent goal here is that whatever mechanism we're having
> Jenkins use to deploy OpenStack onto real hardware should be consumable
> and one that actual people might actually use - otherwise what the heck
> are we testing?
> 
> Additionally, as you may have surmised, it is also a goal to run as much
> of this as possible from the OpenStack Jenkins, because that way we can
> as a project choose to incorporate as much of the feedback/results of
> various forms of testing directly in to branch testing/approval as we
> want. For some things (spinning up 20 node OpenStack clusters) doing it
> on every merge proposal or giving all devs the ability to click a button
> and have it run on their branch will likely be overkill - but if it
> turns out not to be, it would be great to have the ability to do it.
> 
> End goal is to have:
>  - publicly accessible and usable system for testing and build
> automation
>  - resources that it uses to spin up clouds in order to test them are
> themselves usable by people to spin up clouds
>  - tooling around this is done in a manner that makes us of and
> contributes back to existing projects (jenkins plugins, patches back to
> cobbler/orchestra/whatever)
> 
> If you didn't read my _other_ long email from a few moments ago, actual
> discussion of getting this done - and figuring out other people's
> needs/tools and how to integrate them - is hopefully happening next week
> right before the regular openstack-meeting. In the mean time, please
> either flame on right here in list, or ping me back personally.
> 
> Thanks everyone!
> Monty
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> 
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 
> 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launch

Re: [Openstack] Overview of CI/Testing

[Andy Smith]

On Tue, Jun 7, 2011 at 12:59 PM, Monty Taylor  wrote:

> On 06/07/2011 02:38 PM, Andy Smith wrote:
> > Thanks for the update Monty :)
>
> My pleasure as always. :)
>
> >  That's just testing API in a VM though, and doesn't get us to
> testing
> > actual bare-metal deployment or integration testing. At Rackspace, we
> > have some machines set aside at the moment, and have had others offer
> > chunks of machines to test various combinations of things. At its
> heart,
> > the abstract version of this looks fairly identical to the smoketests
> > job - pxe boot machines, shove version to be tested on them, run
> tests.
> > However, there are several moving bits on the best way to actually do
> > the how. At the moment, the fine folks at rPath have a Jenkins
> > installing and testing rPath OpenStack images, so Mihai and I are
> going
> > to look at getting that setup ported to our Jenkins. However,
> although
> > that will be an excellent test of code, as our main target platform
> is
> > Ubuntu, we're also looking at doing a straight-up cobbler install
> using
> > generated .debs.
> >
> >
> > Jesse and I had already gotten quite far along using chef to do the
> > provisioning of baremetal boxes once we'd pxe booted them into ubuntu,
> > it seems like chef or puppet (our current preference is chef) should be
> > used there as well instead of generated .debs.
>
> I have every intention of moving the current work that is running to be
> based on the chef work you did... but I do not view chef and .debs to be
> mutually exclusive options. The goal here is to be able to use chef to
> install and configure the official debs. If this is not possible, then
> there are fundamental flaws that must be fixed.
>
> > At the moment the two closest things to being "official" installations
> > for us (me? are the chef recipes and the nova.sh script (the nova.sh
> > script obviously being only targeted at testing and dev though), those
> > are what we use to verify that the system is functional and I think we'd
> > like to use chef or puppet for baremetal deployments as well.
> >
> > TL;DR: Can we focus on the chef recipes instead of on .debs?
>
> nova.sh is great for devs, but isn't really something I'd imagine would
> be used as the basis of a production deployment (which is kind of the
> point of doing post-install smoke testing)


(I'm pretty sure that is what I said above)


> And again, chef can happily
> install the software from the produced debs.
>
>
Agreed, I think, maybe we're just talking past each other, it sounded form
your email that you would be generating additional debs to handle the
install rather than continuing to use the existing debs (and related deb
generation process). If that is not the case and you instead to use the
packages mostly as they exist today then I think we're already agreeing.


> It's not really just about debs - for the rPath based testing backend,
> we'll obviously be testing RPMs. But in general, testing the packaged
> software that we ship is kind of important.
>
> To sum up: yes to using the chef recipes, no to "instead of".
>
> Monty
>
> > In any case, this is the bit which is still in the
> > planning and discussion phase, but so far all of the conversations
> I've
> > had with folks have been great - and I'd love to get more folks
> involved
> > in that (thus this email)
> >
> > However- latent goal here is that whatever mechanism we're having
> > Jenkins use to deploy OpenStack onto real hardware should be
> consumable
> > and one that actual people might actually use - otherwise what the
> heck
> > are we testing?
> >
> > Additionally, as you may have surmised, it is also a goal to run as
> much
> > of this as possible from the OpenStack Jenkins, because that way we
> can
> > as a project choose to incorporate as much of the feedback/results of
> > various forms of testing directly in to branch testing/approval as we
> > want. For some things (spinning up 20 node OpenStack clusters) doing
> it
> > on every merge proposal or giving all devs the ability to click a
> button
> > and have it run on their branch will likely be overkill - but if it
> > turns out not to be, it would be great to have the ability to do it.
> >
> > End goal is to have:
> >  - publicly accessible and usable system for testing and build
> > automation
> >  - resources that it uses to spin up clouds in order to test them are
> > themselves usable by people to spin up clouds
> >  - tooling around this is done in a manner that makes us of and
> > contributes back to existing projects (jenkins plugins, patches back
> to
> > cobbler/orchestra/whatever)
> >
> > If you didn't read my _other_ long email from a few moments ago,
> actual
> > discussion of getting this done - and figuring out other people's
> > needs/tools and how to integrate them - is hopefull

Re: [Openstack] Coalescing and Coordinating Continuous Integration, Developer Tools, Function and Smoke Testings Efforts

[Monty Taylor]

On 06/07/2011 02:52 PM, Andy Smith wrote:
> 
> 
> On Tue, Jun 7, 2011 at 8:44 AM, Monty Taylor  > wrote:
> 
> Hey all!
> 
> It's time to both expand and contract some things around automation and
> testing. Thusfar all of the work on Jenkins and Tarmac and other
> 'official' build and testing automation has been done exclusively by
> Soren and me, In the mean team, several people have been hacking on
> projects that range from helpful to spectacular, and because of lack of
> communication, only some of them are directly useful to the project's
> infrastructure. I'd like to change this. Specificaly, I'd like to make
> sure we can expand how we're working so that there is greater visibility
> in to what/how we doing things in Jenkins land, and that the barrier to
> entry to helping out isn't terrible. And I'd like to contract the number
> of unrelated efforts in to something resembling a plan.
> 
> a) I'm working on geting relevant bits from our jenkins into to VC
> somewhere so that it's easier for us all to collaborate. I'm using the
> launchpad.net/openstack-ci 
> project for that.
> 
> 
> Would you like to check in the various configs I sent you from the anso
> setups, should I propose that merge, or do you think that this is not
> the proper place for jenkins configs + testing scripts? 

I can stick them in or you can. I'm not particularly thrilled with the
jenkins config.xml + version control story at the moment. (although I
did add the config history plugin so that we're not just constantly
overwriting that stuff through the web) (if you stick them in, shove
them in a subfolder or something- I've currently checking the
openstack-ci branch out straight in to the jenkins root.

> 
> b) I configured the OpenStack Jenkins to use Launchpad's OpenID provider
> as a SSO source - so now I can use Launchpad teams to manage who can do
> what on Jenkins - which means better federation of that management.
> 
> 
> Yay for no longer having manual config there :)

zomg. you're not kidding.

> c) I want to set up a regular (say weekly) IRC meeting for everyone
> interested in CI, dev tools and functional/smoke testing. (although for
> sake of sanity, lets keep "interested" to be "interested in working on"
> I was thinking perhaps the hour right before the normal openstack one...
> anybody have strong thoughts? Of the topics/actions to be discussed are
> managing/moving forward with the current work on the testing
> infrastructure, and having discussions about how/if various work by
> folks can be leveraged by the Jenkins that is the gatekeeper over
> openstack branches.
> 
> 
> I suppose I should be on that list.

Done. Thanks!

> d) Make sure people know what we're doing over here in OpenStack
> CI/Tools land, so that they are aware of the tools already present so
> that they don't duplicate effort, and also of the current todo list in
> case anyone wants to help. Our Jenkins already does several things that
> it turns out nobody knows about. That's gotta get fixed. (Speaking of-
> if there's any Java folks laying around out there sick of all the
> python, there are a couple of tasks that can use love - more to come on
> that)
> 
> The meeting should be simple this time around - I propose meeting the
> hour before the openstack meeting, starting next week. I don't think we
> really need a separate mailing list, but I'm open to opinions... Also,
> in the mean time, ping me back and let me know if you're wanting to be
> part of this.
> 
> Thanks!
> Monty
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> 
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 
> 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Overview of CI/Testing

[Monty Taylor]

On 06/07/2011 03:03 PM, Andy Smith wrote:
> 
> 
> On Tue, Jun 7, 2011 at 12:59 PM, Monty Taylor  > wrote:
> 
> On 06/07/2011 02:38 PM, Andy Smith wrote:
> > Thanks for the update Monty :)
> 
> My pleasure as always. :)
> 
> >  That's just testing API in a VM though, and doesn't get us to
> testing
> > actual bare-metal deployment or integration testing. At
> Rackspace, we
> > have some machines set aside at the moment, and have had
> others offer
> > chunks of machines to test various combinations of things. At
> its heart,
> > the abstract version of this looks fairly identical to the
> smoketests
> > job - pxe boot machines, shove version to be tested on them,
> run tests.
> > However, there are several moving bits on the best way to
> actually do
> > the how. At the moment, the fine folks at rPath have a Jenkins
> > installing and testing rPath OpenStack images, so Mihai and I
> are going
> > to look at getting that setup ported to our Jenkins. However,
> although
> > that will be an excellent test of code, as our main target
> platform is
> > Ubuntu, we're also looking at doing a straight-up cobbler
> install using
> > generated .debs.
> >
> >
> > Jesse and I had already gotten quite far along using chef to do the
> > provisioning of baremetal boxes once we'd pxe booted them into ubuntu,
> > it seems like chef or puppet (our current preference is chef)
> should be
> > used there as well instead of generated .debs.
> 
> I have every intention of moving the current work that is running to be
> based on the chef work you did... but I do not view chef and .debs to be
> mutually exclusive options. The goal here is to be able to use chef to
> install and configure the official debs. If this is not possible, then
> there are fundamental flaws that must be fixed.
> 
> > At the moment the two closest things to being "official" installations
> > for us (me? are the chef recipes and the nova.sh script (the nova.sh
> > script obviously being only targeted at testing and dev though), those
> > are what we use to verify that the system is functional and I
> think we'd
> > like to use chef or puppet for baremetal deployments as well.
> >
> > TL;DR: Can we focus on the chef recipes instead of on .debs?
> 
> nova.sh is great for devs, but isn't really something I'd imagine would
> be used as the basis of a production deployment (which is kind of the
> point of doing post-install smoke testing)
> 
> 
> (I'm pretty sure that is what I said above)

Yup. I think I was obtusely just agreeing with you there.

> And again, chef can happily
> install the software from the produced debs.
> 
> 
> Agreed, I think, maybe we're just talking past each other, it sounded
> form your email that you would be generating additional debs to handle
> the install rather than continuing to use the existing debs (and related
> deb generation process). If that is not the case and you instead to use
> the packages mostly as they exist today then I think we're already agreeing.

AH Yes. Definitely talking past each other. Definitely using existing
debs. We agree with each other. That's much better!

> It's not really just about debs - for the rPath based testing backend,
> we'll obviously be testing RPMs. But in general, testing the packaged
> software that we ship is kind of important.
> 
> To sum up: yes to using the chef recipes, no to "instead of".
> 
> Monty
> 
> > In any case, this is the bit which is still in the
> > planning and discussion phase, but so far all of the
> conversations I've
> > had with folks have been great - and I'd love to get more
> folks involved
> > in that (thus this email)
> >
> > However- latent goal here is that whatever mechanism we're having
> > Jenkins use to deploy OpenStack onto real hardware should be
> consumable
> > and one that actual people might actually use - otherwise what
> the heck
> > are we testing?
> >
> > Additionally, as you may have surmised, it is also a goal to
> run as much
> > of this as possible from the OpenStack Jenkins, because that
> way we can
> > as a project choose to incorporate as much of the
> feedback/results of
> > various forms of testing directly in to branch
> testing/approval as we
> > want. For some things (spinning up 20 node OpenStack clusters)
> doing it
> > on every merge proposal or giving all devs the ability to
> click a button
> > and have it run on their branch will likely be overkill - but
> if it
> > turns out not to be, it would be great to have the ability to
> do it.
>

Re: [Openstack] Overview of CI/Testing

[Jay Pipes]

I'd like to propose that you two agree that you've agreed to agree on agreeing.

All in agreement?

-jay

On Tue, Jun 7, 2011 at 4:11 PM, Monty Taylor  wrote:
>
>
> On 06/07/2011 03:03 PM, Andy Smith wrote:
>>
>>
>> On Tue, Jun 7, 2011 at 12:59 PM, Monty Taylor > > wrote:
>>
>>     On 06/07/2011 02:38 PM, Andy Smith wrote:
>>     > Thanks for the update Monty :)
>>
>>     My pleasure as always. :)
>>
>>     >      That's just testing API in a VM though, and doesn't get us to
>>     testing
>>     >     actual bare-metal deployment or integration testing. At
>>     Rackspace, we
>>     >     have some machines set aside at the moment, and have had
>>     others offer
>>     >     chunks of machines to test various combinations of things. At
>>     its heart,
>>     >     the abstract version of this looks fairly identical to the
>>     smoketests
>>     >     job - pxe boot machines, shove version to be tested on them,
>>     run tests.
>>     >     However, there are several moving bits on the best way to
>>     actually do
>>     >     the how. At the moment, the fine folks at rPath have a Jenkins
>>     >     installing and testing rPath OpenStack images, so Mihai and I
>>     are going
>>     >     to look at getting that setup ported to our Jenkins. However,
>>     although
>>     >     that will be an excellent test of code, as our main target
>>     platform is
>>     >     Ubuntu, we're also looking at doing a straight-up cobbler
>>     install using
>>     >     generated .debs.
>>     >
>>     >
>>     > Jesse and I had already gotten quite far along using chef to do the
>>     > provisioning of baremetal boxes once we'd pxe booted them into ubuntu,
>>     > it seems like chef or puppet (our current preference is chef)
>>     should be
>>     > used there as well instead of generated .debs.
>>
>>     I have every intention of moving the current work that is running to be
>>     based on the chef work you did... but I do not view chef and .debs to be
>>     mutually exclusive options. The goal here is to be able to use chef to
>>     install and configure the official debs. If this is not possible, then
>>     there are fundamental flaws that must be fixed.
>>
>>     > At the moment the two closest things to being "official" installations
>>     > for us (me? are the chef recipes and the nova.sh script (the nova.sh
>>     > script obviously being only targeted at testing and dev though), those
>>     > are what we use to verify that the system is functional and I
>>     think we'd
>>     > like to use chef or puppet for baremetal deployments as well.
>>     >
>>     > TL;DR: Can we focus on the chef recipes instead of on .debs?
>>
>>     nova.sh is great for devs, but isn't really something I'd imagine would
>>     be used as the basis of a production deployment (which is kind of the
>>     point of doing post-install smoke testing)
>>
>>
>> (I'm pretty sure that is what I said above)
>
> Yup. I think I was obtusely just agreeing with you there.
>
>>     And again, chef can happily
>>     install the software from the produced debs.
>>
>>
>> Agreed, I think, maybe we're just talking past each other, it sounded
>> form your email that you would be generating additional debs to handle
>> the install rather than continuing to use the existing debs (and related
>> deb generation process). If that is not the case and you instead to use
>> the packages mostly as they exist today then I think we're already agreeing.
>
> AH Yes. Definitely talking past each other. Definitely using existing
> debs. We agree with each other. That's much better!
>
>>     It's not really just about debs - for the rPath based testing backend,
>>     we'll obviously be testing RPMs. But in general, testing the packaged
>>     software that we ship is kind of important.
>>
>>     To sum up: yes to using the chef recipes, no to "instead of".
>>
>>     Monty
>>
>>     >     In any case, this is the bit which is still in the
>>     >     planning and discussion phase, but so far all of the
>>     conversations I've
>>     >     had with folks have been great - and I'd love to get more
>>     folks involved
>>     >     in that (thus this email)
>>     >
>>     >     However- latent goal here is that whatever mechanism we're having
>>     >     Jenkins use to deploy OpenStack onto real hardware should be
>>     consumable
>>     >     and one that actual people might actually use - otherwise what
>>     the heck
>>     >     are we testing?
>>     >
>>     >     Additionally, as you may have surmised, it is also a goal to
>>     run as much
>>     >     of this as possible from the OpenStack Jenkins, because that
>>     way we can
>>     >     as a project choose to incorporate as much of the
>>     feedback/results of
>>     >     various forms of testing directly in to branch
>>     testing/approval as we
>>     >     want. For some things (spinning up 20 node OpenStack clusters)
>>     doing it
>>     

Re: [Openstack] Overview of CI/Testing

[Andy Smith]

On Tue, Jun 7, 2011 at 1:29 PM, Jay Pipes  wrote:

> I'd like to propose that you two agree that you've agreed to agree on
> agreeing.
>
> All in agreement?
>

NO.

P.S. Maybe?


>
> -jay
>
> On Tue, Jun 7, 2011 at 4:11 PM, Monty Taylor  wrote:
> >
> >
> > On 06/07/2011 03:03 PM, Andy Smith wrote:
> >>
> >>
> >> On Tue, Jun 7, 2011 at 12:59 PM, Monty Taylor  >> > wrote:
> >>
> >> On 06/07/2011 02:38 PM, Andy Smith wrote:
> >> > Thanks for the update Monty :)
> >>
> >> My pleasure as always. :)
> >>
> >> >  That's just testing API in a VM though, and doesn't get us to
> >> testing
> >> > actual bare-metal deployment or integration testing. At
> >> Rackspace, we
> >> > have some machines set aside at the moment, and have had
> >> others offer
> >> > chunks of machines to test various combinations of things. At
> >> its heart,
> >> > the abstract version of this looks fairly identical to the
> >> smoketests
> >> > job - pxe boot machines, shove version to be tested on them,
> >> run tests.
> >> > However, there are several moving bits on the best way to
> >> actually do
> >> > the how. At the moment, the fine folks at rPath have a Jenkins
> >> > installing and testing rPath OpenStack images, so Mihai and I
> >> are going
> >> > to look at getting that setup ported to our Jenkins. However,
> >> although
> >> > that will be an excellent test of code, as our main target
> >> platform is
> >> > Ubuntu, we're also looking at doing a straight-up cobbler
> >> install using
> >> > generated .debs.
> >> >
> >> >
> >> > Jesse and I had already gotten quite far along using chef to do
> the
> >> > provisioning of baremetal boxes once we'd pxe booted them into
> ubuntu,
> >> > it seems like chef or puppet (our current preference is chef)
> >> should be
> >> > used there as well instead of generated .debs.
> >>
> >> I have every intention of moving the current work that is running to
> be
> >> based on the chef work you did... but I do not view chef and .debs
> to be
> >> mutually exclusive options. The goal here is to be able to use chef
> to
> >> install and configure the official debs. If this is not possible,
> then
> >> there are fundamental flaws that must be fixed.
> >>
> >> > At the moment the two closest things to being "official"
> installations
> >> > for us (me? are the chef recipes and the nova.sh script (the
> nova.sh
> >> > script obviously being only targeted at testing and dev though),
> those
> >> > are what we use to verify that the system is functional and I
> >> think we'd
> >> > like to use chef or puppet for baremetal deployments as well.
> >> >
> >> > TL;DR: Can we focus on the chef recipes instead of on .debs?
> >>
> >> nova.sh is great for devs, but isn't really something I'd imagine
> would
> >> be used as the basis of a production deployment (which is kind of
> the
> >> point of doing post-install smoke testing)
> >>
> >>
> >> (I'm pretty sure that is what I said above)
> >
> > Yup. I think I was obtusely just agreeing with you there.
> >
> >> And again, chef can happily
> >> install the software from the produced debs.
> >>
> >>
> >> Agreed, I think, maybe we're just talking past each other, it sounded
> >> form your email that you would be generating additional debs to handle
> >> the install rather than continuing to use the existing debs (and related
> >> deb generation process). If that is not the case and you instead to use
> >> the packages mostly as they exist today then I think we're already
> agreeing.
> >
> > AH Yes. Definitely talking past each other. Definitely using existing
> > debs. We agree with each other. That's much better!
> >
> >> It's not really just about debs - for the rPath based testing
> backend,
> >> we'll obviously be testing RPMs. But in general, testing the
> packaged
> >> software that we ship is kind of important.
> >>
> >> To sum up: yes to using the chef recipes, no to "instead of".
> >>
> >> Monty
> >>
> >> > In any case, this is the bit which is still in the
> >> > planning and discussion phase, but so far all of the
> >> conversations I've
> >> > had with folks have been great - and I'd love to get more
> >> folks involved
> >> > in that (thus this email)
> >> >
> >> > However- latent goal here is that whatever mechanism we're
> having
> >> > Jenkins use to deploy OpenStack onto real hardware should be
> >> consumable
> >> > and one that actual people might actually use - otherwise what
> >> the heck
> >> > are we testing?
> >> >
> >> > Additionally, as you may have surmised, it is also a goal to
> >> run as much
> >> > of this as possible from the

Re: [Openstack] Overview of CI/Testing

[Gregory_Althaus]

Matt Ray and I have extended/modified some of the Anso-based chef scripts to 
configure the debs on systems.  I think Matt's focused on getting a systems 
built from bare metal using his spiceweasel tool and mine are focused on 
inclusion in crowbar that includes the pxe/install environment for bare metal 
and virtual environments.

I think Dan Prince had some chef scripts that included the ability to pull a 
branch to use as the nova base on top of the debs.

All three trees are up on github now.

Thanks,
Greg Althaus

From: openstack-bounces+gregory_althaus=dell@lists.launchpad.net 
[mailto:openstack-bounces+gregory_althaus=dell@lists.launchpad.net] On 
Behalf Of Devin Carlen
Sent: Tuesday, June 07, 2011 2:50 PM
To: Andy Smith
Cc: Peter J. Pouliot; Mihai Ibanescu; openstack@lists.launchpad.net
Subject: Re: [Openstack] Overview of CI/Testing

+1 for chef over debs!


On Jun 7, 2011, at 12:38 PM, Andy Smith wrote:


Thanks for the update Monty :)

 That's just testing API in a VM though, and doesn't get us to testing
actual bare-metal deployment or integration testing. At Rackspace, we
have some machines set aside at the moment, and have had others offer
chunks of machines to test various combinations of things. At its heart,
the abstract version of this looks fairly identical to the smoketests
job - pxe boot machines, shove version to be tested on them, run tests.
However, there are several moving bits on the best way to actually do
the how. At the moment, the fine folks at rPath have a Jenkins
installing and testing rPath OpenStack images, so Mihai and I are going
to look at getting that setup ported to our Jenkins. However, although
that will be an excellent test of code, as our main target platform is
Ubuntu, we're also looking at doing a straight-up cobbler install using
generated .debs.

Jesse and I had already gotten quite far along using chef to do the 
provisioning of baremetal boxes once we'd pxe booted them into ubuntu, it seems 
like chef or puppet (our current preference is chef) should be used there as 
well instead of generated .debs.

At the moment the two closest things to being "official" installations for us 
(me? are the chef recipes and the nova.sh script (the nova.sh script obviously 
being only targeted at testing and dev though), those are what we use to verify 
that the system is functional and I think we'd like to use chef or puppet for 
baremetal deployments as well.

TL;DR: Can we focus on the chef recipes instead of on .debs?


In any case, this is the bit which is still in the
planning and discussion phase, but so far all of the conversations I've
had with folks have been great - and I'd love to get more folks involved
in that (thus this email)

However- latent goal here is that whatever mechanism we're having
Jenkins use to deploy OpenStack onto real hardware should be consumable
and one that actual people might actually use - otherwise what the heck
are we testing?

Additionally, as you may have surmised, it is also a goal to run as much
of this as possible from the OpenStack Jenkins, because that way we can
as a project choose to incorporate as much of the feedback/results of
various forms of testing directly in to branch testing/approval as we
want. For some things (spinning up 20 node OpenStack clusters) doing it
on every merge proposal or giving all devs the ability to click a button
and have it run on their branch will likely be overkill - but if it
turns out not to be, it would be great to have the ability to do it.

End goal is to have:
 - publicly accessible and usable system for testing and build automation
 - resources that it uses to spin up clouds in order to test them are
themselves usable by people to spin up clouds
 - tooling around this is done in a manner that makes us of and
contributes back to existing projects (jenkins plugins, patches back to
cobbler/orchestra/whatever)

If you didn't read my _other_ long email from a few moments ago, actual
discussion of getting this done - and figuring out other people's
needs/tools and how to integrate them - is hopefully happening next week
right before the regular openstack-meeting. In the mean time, please
either flame on right here in list, or ping me back personally.

Thanks everyone!
Monty

___
Mailing list: https://launchpad.net/~openstack
Post to : 
openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : 
openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/

Re: [Openstack] Overview of CI/Testing

[Soren Hansen]

2011/6/7 Andy Smith :
> At the moment the two closest things to being "official" installations for
> us (me? are the chef recipes and the nova.sh script (the nova.sh script
> obviously being only targeted at testing and dev though), those are what we
> use to verify that the system is functional and I think we'd like to use
> chef or puppet for baremetal deployments as well.
> TL;DR: Can we focus on the chef recipes instead of on .debs?

I must have missed a memo at some point. Really.

-- 
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [SPAM] Re: Coalescing and Coordinating Continuous Integration, Developer Tools, Function and Smoke Testings Efforts

[Thierry Carrez]

John Purrier wrote:
> Correct, testing/CI is at 12:00 PDT, PPB meeting is at 1:00 PDT, and the
> weekly OpenStack meeting is at 2:00 PDT on Tuesdays.

All set on the common meetings one:

https://www.google.com/calendar/ical/bj05mroquq28jhud58esggqmh4%40group.calendar.google.com/public/basic.ics

(the one previously known as tinyurl.com/openstack-meetings

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Overview of CI/Testing

[Andy Smith]

On Tue, Jun 7, 2011 at 8:45 PM, Soren Hansen  wrote:

> 2011/6/7 Andy Smith :
> > At the moment the two closest things to being "official" installations
> for
> > us (me? are the chef recipes and the nova.sh script (the nova.sh script
> > obviously being only targeted at testing and dev though), those are what
> we
> > use to verify that the system is functional and I think we'd like to use
> > chef or puppet for baremetal deployments as well.
> > TL;DR: Can we focus on the chef recipes instead of on .debs?
>
> I must have missed a memo at some point. Really.
>

No memo, just what we've been using for lack (or ignorance) of an
alternative that allows us to adequately deploy and configure a system
quickly, and they are what we run our own continuous builds and smoketests
against for existing deployments.

We're at the same place as everybody else, we want better CI support, to
stop duplicating work and to have an easy-as-eating-pie tool for deployment,
so don't take the word "official" to mean we're putting any words in your
mouth, it was in quotes for a reason.


>
> --
> Soren Hansen| http://linux2go.dk/
> Ubuntu Developer| http://www.ubuntu.com/
> OpenStack Developer | http://www.openstack.org/
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] GlusterFS project proposal online

[Sherman Boyd]

Proposal looks great!

I'm wondering if you are aware of Redhat's CloudFS project.  It adds
namespace isolation, ID isolation, storage encryption and network encryption
to Gluster.  Having these kind of features takes a step towards addressing
cloud computing privacy issues.

I'm not sure it's in the scope of your plans, but when I saw your proposal I
thought that it would be nice to have some of these features as well, or at
least space for them on the roadmap.


Best regards,


*Sherman Boyd*






On Mon, Jun 6, 2011 at 10:42 PM, Shehjar Tikoo  wrote:

> For those who havent seen this already, GlusterFS proposal is now available
> at:
>
>
> http://wiki.openstack.org/Governance/Proposed/OpenStack%20Cloud%20Storage%20(Gluster)
>
> Inputs welcome. Please do CC openst...@gluster.com
>
> Thanks
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] suggestion for data backup/recovery in swift

[Rostyslav Slipetskyy]

> What if a container could be marked as "versioning enabled" 
> and therefore the proxy server would rewrite uploads to 
> distinct versioned objects and translate plain gets to the latest version?
>
> Example:
>
> Upload sample.txt, becomes sample.txt-1306882418.68949
> Upload sample.txt again, becomes sample.txt-1306882422.82929
>
> Ask for sample.txt, get latest sample.txt-1306882422.82929
> Ask for specific sample.txt-1306882418.68949 for older version.
>
> There are some annoyances such as only retaining x old copies, etc. 
> but those could be handled by a smarter proxy, 
> as long as it's okay if they're not /guaranteed/ to be perfect.

The idea to append timestamp (or maybe version number) to filename in order to 
get different storage nodes for different versions of the same file looks 
nice. Actually, file metadata can keep information about the previous 
timestamps/versions of a file. The location of file metadata can be determined 
from the Ring
using "account/container/object" path. The location of actual version of the 
file can be 
determined from the Ring using "account/container/object-version" 
(or "account/container/object-timestamp") path.

The problem with such an approach is that it adds transactional processing -
both metadata has to be updated AND file has to be stored successfully. Deletion
of previous versions of a file can be done in a separate process to lessen 
transaction
length.

Besides, the user-provided name of the file is not necessarily needed to become 
a name of the file in the cluster. It is only needed as an input to the Ring so 
that it 

returns different storage nodes. If the user-provided name becomes a part of 
the 

physical name it impacts privacy a bit. Imagine an administrator of a storage 
node who amuses 

himself by performing a search using a query "find all jpg 
files where 'naked' is part of the filename" (filename is also stored in 
container database,
but I don't know why is it really needed?)


- Rostik

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp