[openstack-dev] [tripleo] Posibilities to aggregate/merge configs across templates
Hi all, I want to start discussion on: how to solve issue with merging environment values in TripleO. Description: In TripleO we experience some issues related to setting parameters in heat templates. First, it isn't possible to set some params as ultimate source of truth (disallow to overwrite param in other heat templates). Second it isn't possible to merge values from different templates [0][1]. Both features are implemented in heat and can be easly used in templates.[2][3] This doesn't work in TripleO because we overwrite all values in template in python client instead of aggregating them etc. orsimply let heat do the job .[4][5] Solution: Example solutions are: we can fix how python tripleo client works with env and templates and enable heat features or we can write some puppet code that will work similar to firewall code [6] and will support aggregate and merge values that we point out. Both solutions have pros and cons but IMHO solution which give heat to do job is preferable. But solution with merging give us possibilities to have full control on merging of environments. Problems: Only few as a start: With both solutions we will have the same problem, porting new patches which will use this functionalities to older version of rhel. Also upgrades can be really problematic to new version. Also changes which will enable heat feature will totally change how templates work and we will need to change all templates and change default behavior (which is merge params) to override behavior and also add posibilities to run temporaly old behavior. On the end, I prepared two patchsets with two PoC in progress. First one with merging env in tripleo client but with using heat merging functionality: https://review.openstack.org/#/c/599322/ . And second where we ignore merget env and move all files and add them into deployment plan enviroments. https://review.openstack.org/#/c/599559/ What do you think about each of solution?Which solution should be used in TripleO? Best, Kamil Sambor [0] https://bugs.launchpad.net/tripleo/+bug/1716391 [1] https://bugs.launchpad.net/heat/+bug/1635409 [2] https://docs.openstack.org/heat/pike/template_guide/environment.html#restrict-update-or-replace-of-a-given-resource [3] https://docs.openstack.org/heat/pike/template_guide/environment.html#environment-merging [4] https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/utils.py#L1019 [5] https://github.com/openstack/python-heatclient/blob/f73c2a4177377b710a02577feea38560b00a24bf/heatclient/common/template_utils.py#L191 [6] https://github.com/openstack/puppet-tripleo/tree/master/manifests/firewall __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Deprecation warnings in python-fuelclient-6.1.*
Hi all, IMHO deprecation warning should be added only to commands that we recently changed (because users can switch to new interface when they see deprecation error) or eventually solution #2 sounds ok but is not ideal because people can forget about warning that they saw in previous release. Also we discuss 4th solution, simply we should inform users about deprecation of client and encourage users to use fuel_v2 client with new commands and parameters. Best regards, Kamil Sambor On Wed, Mar 4, 2015 at 9:28 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: Maybe add a Changelog in the repo and maintain it? http://keepachangelog.com/ Option #2 is OK but it can cause pain when testing -- upon each fresh installation from ISO we would get that message and it might break some tests though that is fixable. Option #3 is OK too. #1 is worst and I wouldn't do it. Or maybe display that info when showing all the commands (typing 'fuel' or 'fuel -h')? We already have a deprecation warning there concerning client/config.yaml, it is not very disturbing and shouldn't break any currently used automation scripts. P. On 03/03/2015 03:52 PM, Roman Prykhodchenko wrote: Hi folks! According to the refactoring plan [1] we are going to release the 6.1 version of python-fuelclient which is going to contain recent changes but will keep backwards compatibility with what was before. However, the next major release will bring users the fresh CLI that won’t be compatible with the old one and the new, actually usable IRL API library that also will be different. The issue this message is about is the fact that there is a strong need to let both CLI and API users about those changes. At the moment I can see 3 ways of resolving it: 1. Show deprecation warning for commands and parameters which are going to be different. Log deprecation warnings for deprecated library methods. The problem with this approach is that the structure of both CLI and the library will be changed, so deprecation warning will be raised for mostly every command for the whole release cycle. That does not look very user friendly, because users will have to run all commands with --quiet for the whole release cycle to mute deprecation warnings. 2. Show the list o the deprecated stuff and planned changes on the first run. Then mute it. The disadvantage of this approach if that there is a need of storing the info about the first run to a file. However, it may be cleaned after the upgrade. 3. The same as #2 but publish the warning online. I personally prefer #2, but I’d like to get more opinions on this topic. References: 1. https://blueprints.launchpad.net/fuel/+spec/re-thinking-fuel-client - romcheg __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Deprecation warnings in python-fuelclient-6.1.*
@romcheg - the idea is to switch partially to new client so keeping one package with two entry points: fuel and fuel_v2. It will be convenient for us to add new commands to fuel_v2 only and switch slowly old commands to new version and add warnings in old client commands. It will give users time to switch to new client and it will be easy for us to migrate only old commands. Now when we add new command we add to old client and then in future still need to migrate it. SO keeping two entry points for fuel-client IMHO will be convenient for developers and for users. Best regards, Kamil Sambor On Wed, Mar 4, 2015 at 10:54 AM, Roman Prykhodchenko m...@romcheg.me wrote: I’d like to resolve some questions: @Przemysław: - We can avoid that message by supplying --quiet. - Changelog is currently managed automatically by PBR so as soon as there is a release there will be a change log - I think #2 can be done along with #3 @Kamil: - The issue is that it’s not possible to release commands in this release because it will immediately make the CLI incompatible.For 7.0 there is a plan to get rid of the old CLI completely and replace it with Cliff-based one. I agree that people may forget the deprecation warning before 7.1 ISO is available but that is partially solvable by a change log. Besides, python-fuelclient-7.0 will be available on PyPi much earlier than 7.0 ISO is released. - ^ is basically the reason why we cannot use #4, because there will be nothing new to use, at least in the 6.1 ISO. Keeping both CLIs in the source tree will create more mess and will be terribly hard to test. - romcheg 4 бер. 2015 о 10:11 Kamil Sambor ksam...@mirantis.com написав(ла): Hi all, IMHO deprecation warning should be added only to commands that we recently changed (because users can switch to new interface when they see deprecation error) or eventually solution #2 sounds ok but is not ideal because people can forget about warning that they saw in previous release. Also we discuss 4th solution, simply we should inform users about deprecation of client and encourage users to use fuel_v2 client with new commands and parameters. Best regards, Kamil Sambor On Wed, Mar 4, 2015 at 9:28 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: Maybe add a Changelog in the repo and maintain it? http://keepachangelog.com/ Option #2 is OK but it can cause pain when testing -- upon each fresh installation from ISO we would get that message and it might break some tests though that is fixable. Option #3 is OK too. #1 is worst and I wouldn't do it. Or maybe display that info when showing all the commands (typing 'fuel' or 'fuel -h')? We already have a deprecation warning there concerning client/config.yaml, it is not very disturbing and shouldn't break any currently used automation scripts. P. On 03/03/2015 03:52 PM, Roman Prykhodchenko wrote: Hi folks! According to the refactoring plan [1] we are going to release the 6.1 version of python-fuelclient which is going to contain recent changes but will keep backwards compatibility with what was before. However, the next major release will bring users the fresh CLI that won’t be compatible with the old one and the new, actually usable IRL API library that also will be different. The issue this message is about is the fact that there is a strong need to let both CLI and API users about those changes. At the moment I can see 3 ways of resolving it: 1. Show deprecation warning for commands and parameters which are going to be different. Log deprecation warnings for deprecated library methods. The problem with this approach is that the structure of both CLI and the library will be changed, so deprecation warning will be raised for mostly every command for the whole release cycle. That does not look very user friendly, because users will have to run all commands with --quiet for the whole release cycle to mute deprecation warnings. 2. Show the list o the deprecated stuff and planned changes on the first run. Then mute it. The disadvantage of this approach if that there is a need of storing the info about the first run to a file. However, it may be cleaned after the upgrade. 3. The same as #2 but publish the warning online. I personally prefer #2, but I’d like to get more opinions on this topic. References: 1. https://blueprints.launchpad.net/fuel/+spec/re-thinking-fuel-client - romcheg __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions
Re: [openstack-dev] [Fuel] Separating granular tasks validator
Thank you guys for response. There is no cons, so we migrate validation to separate repo. Best regards, Kamil Sambor On Wed, Feb 18, 2015 at 9:34 AM, Evgeniy L e...@mirantis.com wrote: +1 to extract validators for granular deployment tasks Dmitry, do you mean that we should create some cli to generate graph picture? Or just make it as a module and then use it in Nailgun? Thanks, On Tue, Feb 17, 2015 at 4:31 PM, Dmitriy Shulyak dshul...@mirantis.com wrote: +1 for separate tasks/graph validation library In my opinion we may even migrate graph visualizer to this library, cause it is most usefull during development and to demand installed fuel with nailgun feels a bit suboptimal On Tue, Feb 17, 2015 at 12:58 PM, Kamil Sambor ksam...@mirantis.com wrote: Hi all, I want to discuss separating validation from our repositories to one. On this moment in fuel we have validation for granular deployment tasks in 3 separate repositories so we need to maintain very similar code in all of them. New idea that we discussed with guys assumes to keep this code in one place. Below are more details. Schema validator should be in separate repo, we will install validator in fuel-plugin, fuel-lib, fuel-nailgun. Validator should support versions (return schemas and validate them for selected version). Reasons why we should have validation in all three repositories: nailgun: we need validation in api because we are able to send our own tasks to nailgun and execute them (now we validate type of tasks in deployment graph and during installation of plugin) fuel-library: we need to check if tasks schema is correct defined in task.yaml files and if tasks not create cycles (actually we do both things) fuel-plugin: we need to check if defined tasks are supported by selected version of nailgun (now we check if task type are the same with hardcoded types in fuel-plugin, we not update this part since a while and now there are only 2 type of tasks: shell and puppet) With versioning we shouldn't have conflicts between nailgun serialization and fuel-plugin because plugin will be able to use schemas for specified version of nailgun. As a core reviewers of repositories we should keep the same reviewers as we have in fuel-core. How validator should looks like: separate repo, to install using pip need to return correct schema for selected version of fuel should be able to validate schema for selected version and ignore selected fields validate graph from selected tasks Pros and cons of this solution: pros: one place to keep validation less error prone - we will eliminate errors caused by not updating one of the repos, also it will be easy to test if changes are correct and compatible with all repos easier to develop (less changes in cases when we add new type of task or we change schemas of tasks - we edit just one place) easy distribution of code between repositories and easy to use by external developers cons: new repository that needs to be managed (and included into CI/QA/release cycle) new dependency for fuel-library, fuel-web, fuel-plugins (fuel plugin builder) of which developer need to be aware of Please comments and give opinions. Best regards, Kamil Sambor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel-client and Nailgun API
Hi all, I don't know nothing about 'holywar' so I'm interested where it had place. According to the fuel-client tests I think that it will be good idea to run some integration tests on nailgun API to check if client really works with nailgun and if it works as expected, but unit test can have mocks if it is necessary. If we will have test run on nailgun we can be sure that even if responses has been changed, client can still work or when we add new API our client realy works with API, not only with mocked responses. Best regards, Kamil Sambor On Mon, Feb 9, 2015 at 1:57 PM, Nikolay Markov nmar...@mirantis.com wrote: Hello colleagues, They say, there is some kind of holywar around the topic on if fuel-client tests should rely on working Nailgun API without mocking it. This is also connected with API stabilizing and finally moving fuel-client to a separate library which may be used by any third-party projects. I just wanted to start this thread so everyone can share his opinion on both Nailgun API stabilizing and further fate of fuel-client as a separate library (how they do it in OpenStack projects). Everyone is welcome to participate. -- Best regards, Nick Markov __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Dropping Python-2.6 support
Tomasz we are not using ssl in our client so now we not gain anything from moving to 2.7 . Best regards, – Kamil S. On Wed, Jan 14, 2015 at 8:32 AM, Bartłomiej Piotrowski bpiotrow...@mirantis.com wrote: On 01/13/2015 11:16 PM, Tomasz Napierala wrote: On 13 Jan 2015, at 10:51, Przemyslaw Kaminski pkamin...@mirantis.com wrote: For example https://www.python.org/download/releases/2.6.9/ All official maintenance for Python 2.6, including security patches, has ended. https://hg.python.org/cpython/raw-file/v2.7.9/Misc/NEWS Especially the SSL stuff is interesting http://bugs.python.org/issue22935 This looks like final word here. We cannot provide software, that has no security support. Regards, I can hardly see it as a justification for maintaining yet another package on our own while Red Hat is supposed to provide backports of security fixes to python 2.6 until 2020. I wanted to hear exact use cases of 2.7 features that allow us to accomplish things easier than it is now with 2.6. As Doug already said, clients and Oslo libraries will maintain compatibility with 2.6. So what is the real gain? Regards, Bartłomiej __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel][Nailgun] Web framework
Additionaly to what Przemek wrote, also Pecan is released more often, IMHO documentation is better written, and described a lot of possibilities of modification, also as Lukasz wrote in previous thread that Pecan is used in openstack. So I'm also for Pecan Best regards, Kamil S. On Wed, Dec 3, 2014 at 12:45 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: The only useful paradigm to write in Flask is MethodView's for me [1] because decorators seem hard to refactor for large projects. Please look at adding URLs -- one has to additionally specify methods to match those from the MethodView -- this is code duplication and looks ugly. It seems though that Fask-RESTful [2] fixes this but then we're dependent on 2 projects. I don't like that Flask uses a global request object [3]. From Flask documentation Basically you can completely ignore that this is the case unless you are doing something like unit testing. You will notice that code which depends on a request object will suddenly break because there is no request object. The solution is creating a request object yourself and binding it to the context. Yeah, let's make testing even harder... Pecan looks better in respect of RESTful services [4]. POST parameters are cleanly passed as arguments to the post method. It also provides custom JSON serialization hooks [5] so we can forget about explicit serialization in handlers. So from these 2 choices I'm for Pecan. [1] http://flask.pocoo.org/docs/0.10/views/#method-views-for-apis [2] https://flask-restful.readthedocs.org/en/0.3.0/ [3] http://flask.pocoo.org/docs/0.10/quickstart/#accessing-request-data [4] http://pecan.readthedocs.org/en/latest/rest.html [5] http://pecan.readthedocs.org/en/latest/jsonify.html P. On 12/03/2014 10:57 AM, Alexander Kislitsky wrote: We had used Flask in the fuel-stats. It was easy and pleasant and all project requirements was satisfied. And I saw difficulties and workarounds with Pecan, when Nick integrated it into Nailgun. So +1 for Flask. On Tue, Dec 2, 2014 at 11:00 PM, Nikolay Markov nmar...@mirantis.com wrote: Michael, we already solved all issues I described, and I just don't want to solve them once again after moving to another framework. Also, I think, nothing of these wishes contradicts with good API design. On Tue, Dec 2, 2014 at 10:49 PM, Michael Krotscheck krotsch...@gmail.com wrote: This sounds more like you need to pay off technical debt and clean up your API. Michael On Tue Dec 02 2014 at 10:58:43 AM Nikolay Markov nmar...@mirantis.com wrote: Hello all, I actually tried to use Pecan and even created a couple of PoCs, but there due to historical reasons of how our API is organized it will take much more time to implement all workarounds we need to issues Pecan doesn't solve out of the box, like working with non-RESTful URLs, reverse URL lookup, returning custom body in 404 response, wrapping errors to JSON automatically, etc. As far as I see, each OpenStack project implements its own workarounds for these issues, but still it requires much less men and hours for us to move to Flask-Restful instead of Pecan, because all these problems are already solved there. BTW, I know a lot of pretty big projects using Flask (it's the second most popular Web framework after Django in Python Web community), they even have their own hall of fame: http://flask.pocoo.org/community/poweredby/ . On Tue, Dec 2, 2014 at 7:13 PM, Ryan Brown rybr...@redhat.com wrote: On 12/02/2014 09:55 AM, Igor Kalnitsky wrote: Hi, Sebastian, Thank you for raising this topic again. [snip] Personally, I'd like to use Flask instead of Pecan, because first one is more production-ready tool and I like its design. But I believe this should be resolved by voting. Thanks, Igor On Tue, Dec 2, 2014 at 4:19 PM, Sebastian Kalinowski skalinow...@mirantis.com wrote: Hi all, [snip explanation+history] Best, Sebastian Given that Pecan is used for other OpenStack projects and has plenty of builtin functionality (REST support, sessions, etc) I'd prefer it for a number of reasons. 1) Wouldn't have to pull in plugins for standard (in Pecan) things 2) Pecan is built for high traffic, where Flask is aimed at much smaller projects 3) Already used by other OpenStack projects, so common patterns can be reused as oslo libs Of course, the Flask community seems larger (though the average flask project seems pretty small). I'm not sure what determines production readiness, but it seems to me like Fuel developers fall more in Pecan's target audience than in Flask's. My $0.02, Ryan -- Ryan Brown / Software Engineer, Openstack / Red Hat, Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
[openstack-dev] [Fuel] Authentication is turned on - Fuel API and UI
Hi folks, All parts of code related to stage I and II from blueprint http://docs-draft.openstack.org/29/96429/11/gate/gate-fuel-specs-docs/2807f30/doc/build/html/specs/5.1/access-control-master-node.htm http://docs-draft.openstack.org/29/96429/11/gate/gate-fuel-specs-docs/2807f30/doc/build/html/specs/5.1/access-control-master-node.html are merged. In result of that, fuel (api and UI) we now have authentication via keystone and now is required as default. Keystone is installed in new container during master installation. We can configure password via fuelmenu during installation (default user:password - admin:admin). Password is saved in astute.yaml, also admin_token is stored here. Almost all endpoints in fuel are protected and they required authentication token. We made exception for few endpoints and they are defined in nailgun/middleware/keystone.py in public_url . Default password can be changed via UI or via fuel-cli. In case of changing password via UI or fuel-cli password is not stored in any file only in keystone, so if you forgot password you can change it using keystone client from master node and admin_token from astute.yaml using command: keystone --os-endpoint=http://10.20.0.2:35357/v2.0 --os-token=admin_token password-update . Fuel client now use for authentication user and passwords which are stored in /etc/fuel/client/config.yaml. Password in this file is not changed during changing via fuel-cli or UI, user must change this password manualy. If user don't want use config file can provide user and password to fuel-cli by flags: --os-username=admin --os-password=test. We added also possibilities to change password via fuel-cli, to do this we should execute: fuel user --change-password --new-pass=new . To run or disable authentication we should change /etc/nailgun/settings.yaml (AUTHENTICATION_METHOD) in nailgun container. Best regards, Kamil S. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
