Re: [openstack-dev] [tripleo] Contributing to TripleO is challenging
On Sat, Mar 5, 2016 at 10:31 AM, Giulio Fidentewrote: > On 03/04/2016 03:23 PM, Emilien Macchi wrote: > >> That's not the name of any Summit's talk, it's just an e-mail I wanted >> to write for a long time. >> >> It is an attempt to expose facts or things I've heard a lot; and bring >> constructive thoughts about why it's challenging to contribute in >> TripleO project. >> > > hi Emilien, > > thanks for bringing this up, it's not an easy topic and yet of most > crucial. As a core contributors I feel, to some extent, responsible for the > current status of things and I think it's time for us to reflect more about > what we can, individually, do. > > I have some ideas but I want to start by commenting to your points. > > 1/ "I don't review this patch, we don't have CI coverage." >> >> One thing I've noticed in TripleO is that a very few people are involved >> in CI work. >> In my opinion, CI system is more critical than any feature in a product. >> Developing Software without tests is a bit like http://goo.gl/OlgFRc >> All people - specially core - in the project should be involved in CI >> work. If you are TripleO core and you don't contribute on CI, you might >> ask yourself why. >> > > Agreed, we need more 'eyes' on out CI to cope with both the infra and the > inavoidable failures due to changes/bugs in the puppet modules or openstack > itself. > > But there is more hiding behind this problem ... we already have quite a > number of optional and even pluggable features in TripleO and we're even > designing an interface to make this easier; testing them all isn't going to > happen. So we'll always hit something we don't have coverage for. > > Let's have a conversation on how we can improve coverage at the summit! > Maybe we can make simply make our CI scenarios more variegated/complex in > the attempt to touch more features? > > 2/ "I don't review this patch, CI is broken." >> >> Another thing I've noticed in TripleO is that when CI is broken, again, >> a very few people are actually working on fixing failures. >> My experience over the last years taught me to stop my daily work when >> CI is broken and fix it asap. >> > > Agreed. More eyes and more coverage to increase its dependability. > > 3/ "I don't review it, because this feature / code is not my area". >> >> My first though is "Aren't we supposed to be engineers and learn new >> areas?" >> My second though is that I think we have a problem with TripleO Heat >> Templates. >> THT or TripleO Heat Templates's code is 80% of Puppet / Hiera. If >> TripleO core say "I'm not familiar with Puppet", we have a problem here, >> isn't? >> Maybe should we split this repository? Or revisit the list of people who >> can +2 patches on THT. >> > > Not sure here, I find that manifests and templates are pretty much "meant > to go together" so I am worried that a split could solve some problems but > also cause others. > This is pretty much what I proposed last week ( https://blueprints.launchpad.net/tripleo/+spec/refactor-puppet-manifests) and I noticed Dan approved the blueprint yesterday (cheers). It's definitely going to cause problems in that THT defines the data interface and puppet-tripleo is going to have to keep up with that interface in lock-step in some cases so be prepared to deal with that as a patch author. This isn't really any different to non-tripleo puppet module situations where a change to the repo holding hiera data will be tied to changes in modules. Ideally I'd like to incrementally decouple the puppet-tripleo profiles from the data heat provides but for the first cut they'll be joined at the hip. So given a new home (puppet-tripleo) for a large portion of the code (starting with overcloud controller and controller_pacemaker), hopefully this paves the way for giving those who know puppet well the opportunity to take on responsibility for the manifests without necessarily being intimately familiar with the rest of the system, which I guess helps with Emilien's original concern that there's a skill split across the tooling lines. > > This said, let's be honest, an effective patch for THT requires a good > understanding of many different problems which can be TripleO specific (eg. > implications on upgrades), tooling specific (eg. Heat/Puppet), OpenStack > specific (eg. cooperation with other, optional, features) so I have myself > skipped changes when I didn't feel comfortable with it. > > But one problem which I think is more recently slowing reviews and which > is somewhat concause of 3) is that we're not dealing too well with code > duplication in the yamls and with conditional logic in the manifests. > > Maybe we could stop and think a together about new HOT functionalities > which could help us? Interesting for the summit as well? > > 4/ Patches are stalled. Most of the time. >> >> Over the last 12 months, I've pushed a lot of patches in TripleO and one >> thing I've noticed is that if I don't ping people, my patch
Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting
On Fri, Oct 3, 2014 at 4:05 AM, Soren Hansen so...@linux2go.dk wrote: I'm sorry about my slow responses. For some reason, gmail didn't think this was an important e-mail :( 2014-09-30 18:41 GMT+02:00 Jay Pipes jaypi...@gmail.com: On 09/30/2014 08:03 AM, Soren Hansen wrote: 2014-09-12 1:05 GMT+02:00 Jay Pipes jaypi...@gmail.com: How would I go about getting the associated fixed IPs for a network? The query to get associated fixed IPs for a network [1] in Nova looks like this: SELECT fip.address, fip.instance_uuid, [...] AND fip.instance_uuid IS NOT NULL AND i.host = :host would I have a Riak container for virtual_interfaces that would also have instance information, network information, fixed_ip information? How would I accomplish the query against a derived table that gets the minimum virtual interface ID for each instance UUID? What's a minimum virtual interface ID? Anyway, I think Clint answered this quite well. I've said it before, and I'll say it again. In Nova at least, the SQL schema is complex because the problem domain is complex. That means lots of relations, lots of JOINs, and that means the best way to query for that data is via an RDBMS. [...] I don't think relying on a central data store is in any conceivable way appropriate for a project like OpenStack. Least of all Nova. I don't see how we can build a highly available, distributed service on top of a centralized data store like MySQL. [...] I don't disagree with anything you say above. At all. Really? How can you agree that we can't build a highly available, distributed service on top of a centralized data store like MySQL while also saying that the best way to handle data in Nova is in an RDBMS? For complex control plane software like Nova, though, an RDBMS is the best tool for the job given the current lay of the land in open source data storage solutions matched with Nova's complex query and transactional requirements. What transactional requirements? https://github.com/openstack/nova/blob/stable/icehouse/nova/db/sqlalchemy/api.py#L1654 When you delete an instance, you don't want the delete to just stop half-way through the transaction and leave around a bunch of orphaned children. Similarly, when you reserve something, it helps to not have a half-finished state change that you need to go clean up if something goes boom. Looking at that particular example, it's about deleting an instance and all its associated metadata. As we established earlier, these are things that would just be in the same key as the instance itself, so it'd just be a single key that would get deleted. Easy. That said, there will certainly be situations where there'll be a need for some sort of anti-entropy mechanism. It just so happens that those situations already exist. We're dealing with about a complex distributed system. We're kidding ourselves if we think that any kind of consistency is guaranteed, just because our data store favours consistency over availability. I apologize if I'm missing something, but doesn't denormalization to add join support put the same value in many places, such that an update to that value is no longer a single atomic transaction? This would appear to counteract the requirement for strong consistency. If updating a single value is atomic (as in Riak's consistent mode) then it might be possible to construct a way to make multiple updates appear atomic, but it would add many more transactions and many more quorum checks, which would reduce performance to a crawl. I also don't really see how a NoSQL system in strong consistency mode is any different from running MySQL with galera in its failure modes. The requirement for quorum makes the addition of nodes increase the potential latency of writes (and reads in some cases) so having large scale doesn't grant much benefit, if any. Quorum will also prevent nodes on the wrong side of a partition from being able to access system state (or it will give them stale state, which is probably just as bad in our case). I think your goal of having state management that's able to handle network partitions is a good one, but I don't think the solution is as simple as swapping out where the state is stored. Maybe in some cases like split-racks the system needs to react to a network partition by forming its own independent cell with its own state storage, and when the network heals it then merges back into the other cluster cleanly? That would be very difficult to implement, but fun (for some definition of fun). As a thought experiment, a while ago I considered what would happen if instead of using a central store, I put a sqlite database behind every daemon and allowed them to query each other for the data they needed, and cluster if needed (using raft). Services like nova-scheduler need strong consistency and would have to cluster to perform their role, but services like nova-compute would
Re: [openstack-dev] [all] The future of the integrated release
On Fri, Aug 22, 2014 at 2:57 AM, Jay Pipes jaypi...@gmail.com wrote: On 08/19/2014 11:28 PM, Robert Collins wrote: On 20 August 2014 02:37, Jay Pipes jaypi...@gmail.com wrote: ... I'd like to see more unification of implementations in TripleO - but I still believe our basic principle of using OpenStack technologies that already exist in preference to third party ones is still sound, and offers substantial dogfood and virtuous circle benefits. No doubt Triple-O serves a valuable dogfood and virtuous cycle purpose. However, I would move that the Deployment Program should welcome the many projects currently in the stackforge/ code namespace that do deployment of OpenStack using traditional configuration management tools like Chef, Puppet, and Ansible. It cannot be argued that these configuration management systems are the de-facto way that OpenStack is deployed outside of HP, and they belong in the Deployment Program, IMO. I think you mean it 'can be argued'... ;). No, I definitely mean cannot be argued :) HP is the only company I know of that is deploying OpenStack using Triple-O. The vast majority of deployers I know of are deploying OpenStack using configuration management platforms and various systems or glue code for baremetal provisioning. Note that I am not saying that Triple-O is bad in any way! I'm only saying that it does not represent the way that the majority of real-world deployments are done. And I'd be happy if folk in those communities want to join in the deployment program and have code repositories in openstack/. To date, none have asked. My point in this thread has been and continues to be that by having the TC bless a certain project as The OpenStack Way of X, that we implicitly are saying to other valid alternatives Sorry, no need to apply here.. As a TC member, I would welcome someone from the Chef community proposing the Chef cookbooks for inclusion in the Deployment program, to live under the openstack/ code namespace. Same for the Puppet modules. While you may personally welcome the Chef community to propose joining the deployment Program and living under the openstack/ code namespace, I'm just saying that the impression our governance model and policies create is one of exclusion, not inclusion. Hope that clarifies better what I've been getting at. (As one of the core reviewers for the Puppet modules) Without a standardised package build process it's quite difficult to test trunk Puppet modules vs trunk official projects. This means we cut release branches some time after the projects themselves to give people a chance to test. Until this changes and the modules can be released with the same cadence as the integrated release I believe they should remain on Stackforge. In addition and perhaps as a consequence, there isn't any public integration testing at this time for the modules, although I know some parties have developed and maintain their own. The Chef modules may be in a different state, but it's hard for me to recommend the Puppet modules become part of an official program at this stage. All the best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] The future of the integrated release
On Fri, Aug 22, 2014 at 9:51 PM, Sean Dague s...@dague.net wrote: On 08/22/2014 01:30 AM, Michael Chapman wrote: On Fri, Aug 22, 2014 at 2:57 AM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 08/19/2014 11:28 PM, Robert Collins wrote: On 20 August 2014 02:37, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: ... I'd like to see more unification of implementations in TripleO - but I still believe our basic principle of using OpenStack technologies that already exist in preference to third party ones is still sound, and offers substantial dogfood and virtuous circle benefits. No doubt Triple-O serves a valuable dogfood and virtuous cycle purpose. However, I would move that the Deployment Program should welcome the many projects currently in the stackforge/ code namespace that do deployment of OpenStack using traditional configuration management tools like Chef, Puppet, and Ansible. It cannot be argued that these configuration management systems are the de-facto way that OpenStack is deployed outside of HP, and they belong in the Deployment Program, IMO. I think you mean it 'can be argued'... ;). No, I definitely mean cannot be argued :) HP is the only company I know of that is deploying OpenStack using Triple-O. The vast majority of deployers I know of are deploying OpenStack using configuration management platforms and various systems or glue code for baremetal provisioning. Note that I am not saying that Triple-O is bad in any way! I'm only saying that it does not represent the way that the majority of real-world deployments are done. And I'd be happy if folk in those communities want to join in the deployment program and have code repositories in openstack/. To date, none have asked. My point in this thread has been and continues to be that by having the TC bless a certain project as The OpenStack Way of X, that we implicitly are saying to other valid alternatives Sorry, no need to apply here.. As a TC member, I would welcome someone from the Chef community proposing the Chef cookbooks for inclusion in the Deployment program, to live under the openstack/ code namespace. Same for the Puppet modules. While you may personally welcome the Chef community to propose joining the deployment Program and living under the openstack/ code namespace, I'm just saying that the impression our governance model and policies create is one of exclusion, not inclusion. Hope that clarifies better what I've been getting at. (As one of the core reviewers for the Puppet modules) Without a standardised package build process it's quite difficult to test trunk Puppet modules vs trunk official projects. This means we cut release branches some time after the projects themselves to give people a chance to test. Until this changes and the modules can be released with the same cadence as the integrated release I believe they should remain on Stackforge. In addition and perhaps as a consequence, there isn't any public integration testing at this time for the modules, although I know some parties have developed and maintain their own. The Chef modules may be in a different state, but it's hard for me to recommend the Puppet modules become part of an official program at this stage. Is the focus of the Puppet modules only stable releases with packages? We try to target puppet module master at upstream OpenStack master, but without CI/CD we fall behind. The missing piece is building packages and creating a local repo before doing the puppet run, which I'm working on slowly as I want a single system for both deb and rpm that doesn't make my eyes bleed. fpm and pleaserun are the two key tools here. Puppet + git based deploys would be honestly a really handy thing (especially as lots of people end up having custom fixes for their site). The lack of CM tools for git based deploys is I think one of the reasons we seen people using DevStack as a generic installer. It's possible but it's also straight up a poor thing to do in my opinion. If you're going to install nova from source, maybe you also want libvirt from source to test a new feature, then you want some of libvirt's deps and so on. Puppet isn't equipped to deal with this effectively. It runs yum install x, and that brings in the dependencies. It's much better to automate the package building process
Re: [openstack-dev] [all] sample config files should be ignored in git...
On Thu, Mar 27, 2014 at 4:10 PM, Robert Collins robe...@robertcollins.netwrote: On 27 March 2014 17:30, Tom Fifield t...@openstack.org wrote: Does anyone disagree? /me raises hand When I was an operator, I regularly referred to the sample config files in the git repository. If there weren't generated versions of the sample config in the repo, I would probably grep the code (not an ideal user experience!). Running some random script that I don't know about the existence and might depend on having something else installed of is probably not something that would happen. So, I think its important you have sample configs to refer to. Do they need to be in the git repo? Note that because libraries now export config options (which is the root of this problem!) you cannot ever know from the source all the options for a service - you *must* know the library versions you are running, to interrogate them for their options. We can - and should - have a discussion about the appropriateness of the layering leak we have today, but in the meantime this is breaking multiple projects every time any shared library that uses oslo.config changes any config option... so we need to solve the workflow aspect. How about we make a copy of the latest config for each project for each series - e.g. trunk of everything, Icehouse of servers with trunk of everything else, etc and make that easily acccessible? -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev There are already some samples in the 'configuration reference' section of docs.openstack, eg: http://docs.openstack.org/havana/config-reference/content/ch_configuring-openstack-identity.html#sample-configuration-files However the compute and image sections opt for a formatted table, and the network section is more like an installation guide. If the samples are to be removed from github, perhaps our configuration reference section could be first and foremost the set of sample configuration files for each project + plugin, rather than them being merely a part of the reference doc as it currently exists. I fairly consistently refer to the github copies of the samples. They also allow me to refer to specific lines of the config when explaining concepts over text. I am not against their removal, but if we were to remove them I'd disappointed if I had to search very far on docs.openstack.org to get to them, and I would want the raw files instead of something formatted. - Michael ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] bad default values in conf files
Have the folks creating our puppet modules and install recommendations taken a close look at all the options and determined that the defaults are appropriate for deploying RHEL OSP in the configurations we are recommending? If by our puppet modules you mean the ones in stackforge, in the vast majority of cases they follow the defaults provided. I check that this is the case during review, and the only exceptions should be stuff like the db and mq locations that have to change for almost every install. - Michael On Sat, Feb 15, 2014 at 10:15 AM, Dirk Müller d...@dmllr.de wrote: were not appropriate for real deployment, and our puppet modules were not providing better values https://bugzilla.redhat.com/show_bug.cgi?id=1064061. I'd agree that raising the caching timeout is a not a good production default choice. I'd also argue that the underlying issue is fixed with https://review.openstack.org/#/c/69884/ In our testing this patch has speed up the revocation retrieval by factor 120. The default probably is too low, but raising it too high will cause concern with those who want revoked tokens to take effect immediately and are willing to scale the backend to get that result. I agree, and changing defaults has a cost as well: Every deployment solution out there has to detect the value change, update their config templates and potentially also migrate the setting from the old to the new default for existing deployments. Being in that situation, it has happened that we were surprised by default changes that had undesireable side effects, just because we chose to overwrite a different default elsewhere. I'm totally on board with having production ready defaults, but that also includes that they seldomly change and change only for a very good, possibly documented reason. Greetings, Dirk ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev