Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 5 January 2014 02:02, Sean Dague s...@dague.net wrote: So we used to do that the apps against release libraries. And the result was more and more full day gate breaks. We did 2 consecutive ones in 2 weeks. Basically, once you get to be a certain level of coupled in OpenStack we can no longer let you manage your own requirements file. We need a global lever on it. Because people were doing it wrong, and slowly (we could go through specific examples about how bad this was. This was a top issue at nearly every summit I'd been at going back to Essex. .. (It was about 14 days to resolve the python client issue, there was a django issue around the same time that never made it to the list, as we did it all under fire in IRC) And we have a solution now. Which is one list of requirements that we can test everything with, that we can propose requirements updates speculatively, and see what works and what doesn't. And *after* we know they work, we propose the changes back into the projects, now automatically. So the flip-flop thing is certainly very interesting. We wouldn't want that to happen again. I do see the issue Sean is pointing at, which is that we have to fix the libraries first and then the things that use them. OTOH thats normal in the software world, I don't see anything unique about it. Well, as the person that normally gets stuck figuring this out when .eu has been gate blocked for a day, and I'm one of the first people up on the east coast, I find the normal state of affairs unsatisfying. :) :) I also think that what we are basically dealing with is the classical N^2 comms problem. With N git trees that we need to all get working together, this gets exponentially more difficult over time. Which is why we created the integrated gate and the global requirements lever. I don't think we are - I think we're dealing with the fact that we've had no signal - no backpressure - on projects that have upper caps set to remove those caps. So they stick there and we all suffer. Another solution would be reduce the number of OpenStack git trees to make N^2 more manageable, and let us with single commits affect multiple components. But that's not the direction we've taken. I don't think thats necessary. What I'd like to see is: A) two test sets for every commit: - commit with latest-release of all deps - commit with latest-trunk [or dependent zuul ref] of all deps B) *if* there are upper version caps for any reason, some signal back to developers that this exists and that we need to fix our code to work with that newer release. - Possibly we should allow major version caps where major releases are anticipated to be incompatible without warning about that *until* there is a [pre-]release of the new major version available -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 01/10/2014 04:13 AM, Robert Collins wrote: On 5 January 2014 02:02, Sean Dague s...@dague.net wrote: So we used to do that the apps against release libraries. And the result was more and more full day gate breaks. We did 2 consecutive ones in 2 weeks. Basically, once you get to be a certain level of coupled in OpenStack we can no longer let you manage your own requirements file. We need a global lever on it. Because people were doing it wrong, and slowly (we could go through specific examples about how bad this was. This was a top issue at nearly every summit I'd been at going back to Essex. .. (It was about 14 days to resolve the python client issue, there was a django issue around the same time that never made it to the list, as we did it all under fire in IRC) And we have a solution now. Which is one list of requirements that we can test everything with, that we can propose requirements updates speculatively, and see what works and what doesn't. And *after* we know they work, we propose the changes back into the projects, now automatically. So the flip-flop thing is certainly very interesting. We wouldn't want that to happen again. I do see the issue Sean is pointing at, which is that we have to fix the libraries first and then the things that use them. OTOH thats normal in the software world, I don't see anything unique about it. Well, as the person that normally gets stuck figuring this out when .eu has been gate blocked for a day, and I'm one of the first people up on the east coast, I find the normal state of affairs unsatisfying. :) :) I also think that what we are basically dealing with is the classical N^2 comms problem. With N git trees that we need to all get working together, this gets exponentially more difficult over time. Which is why we created the integrated gate and the global requirements lever. I don't think we are - I think we're dealing with the fact that we've had no signal - no backpressure - on projects that have upper caps set to remove those caps. So they stick there and we all suffer. Another solution would be reduce the number of OpenStack git trees to make N^2 more manageable, and let us with single commits affect multiple components. But that's not the direction we've taken. I don't think thats necessary. What I'd like to see is: A) two test sets for every commit: - commit with latest-release of all deps - commit with latest-trunk [or dependent zuul ref] of all deps B) *if* there are upper version caps for any reason, some signal back to developers that this exists and that we need to fix our code to work with that newer release. - Possibly we should allow major version caps where major releases are anticipated to be incompatible without warning about that *until* there is a [pre-]release of the new major version available Honestly, I would too. I actually proposed just this at Summit. :) But it's a ton of work, that is lacking for volunteers right now. Earliest I'm going to look at it is Juno, but I'm not really sure I can really commit on this one. And I expect this all by itself needs two or three people where this is a top priority. So consider this a call for volunteers. If you want to be an OpenStack hero, here is a great way to do so. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On Mon, Jan 6, 2014 at 6:21 PM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-01-06 17:23:31 -0500 (-0500), Doug Hellmann wrote: [...] The global requirements syncing seems to have fixed the issue for apps, although it just occurred to me that I'm not sure we check that the requirements lists are the same when we cut a release. Do we do that already? Not yet in any automated fashion but keep in mind that we've only gotten the requirements update proposal job working reliably this cycle, so it could still take some time for the various projects to decide how to finish syncing up. Makes sense. I was just thinking about some of the assumptions we're making elsewhere in this thread w.r.t. syncing requirements. Doug -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On Sat, Jan 4, 2014 at 8:02 AM, Sean Dague s...@dague.net wrote: On 01/03/2014 08:27 PM, Robert Collins wrote: On 4 January 2014 08:44, Doug Hellmann doug.hellm...@dreamhost.com wrote: It seems safer to gate changes to libraries against the apps' trunk (to avoid making backwards-incompatible changes), and then gate changes to the apps against the released libraries (to ensure they work with something available to be packaged by the distros). There are lots and lots of version numbers available to us, so I see no problem with releasing new versions of libraries frequently. So we used to do that the apps against release libraries. And the result was more and more full day gate breaks. We did 2 consecutive ones in 2 weeks. Basically, once you get to be a certain level of coupled in OpenStack we can no longer let you manage your own requirements file. We need a global lever on it. Because people were doing it wrong, and slowly (we could go through specific examples about how bad this was. This was a top issue at nearly every summit I'd been at going back to Essex. Some reading from the break times: * http://lists.openstack.org/pipermail/openstack-dev/2013- July/011660.html * http://lists.openstack.org/pipermail/openstack-dev/2013- July/011708.html * http://lists.openstack.org/pipermail/openstack-dev/2013- July/012342.html (It was about 14 days to resolve the python client issue, there was a django issue around the same time that never made it to the list, as we did it all under fire in IRC) And we have a solution now. Which is one list of requirements that we can test everything with, that we can propose requirements updates speculatively, and see what works and what doesn't. And *after* we know they work, we propose the changes back into the projects, now automatically. I do see the issue Sean is pointing at, which is that we have to fix the libraries first and then the things that use them. OTOH thats normal in the software world, I don't see anything unique about it. Well, as the person that normally gets stuck figuring this out when .eu has been gate blocked for a day, and I'm one of the first people up on the east coast, I find the normal state of affairs unsatisfying. :) I appreciate you going through the details with me. As I said at the beginning, I'm not opposed to bringing taskflow into the oslo program at all -- and we can still do that, if Josh wants to, retaining the existing core review team, permissions, etc. -- but since we plan to release more libraries over time I want to make sure that we're set up to handle cases like this where we have access to the developers and source for a library, but don't call it an openstack library. I also think that what we are basically dealing with is the classical N^2 comms problem. With N git trees that we need to all get working together, this gets exponentially more difficult over time. Which is why we created the integrated gate and the global requirements lever. Another solution would be reduce the number of OpenStack git trees to make N^2 more manageable, and let us with single commits affect multiple components. But that's not the direction we've taken. The global requirements syncing seems to have fixed the issue for apps, although it just occurred to me that I'm not sure we check that the requirements lists are the same when we cut a release. Do we do that already? Doug -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On Sun, Jan 5, 2014 at 5:02 PM, James E. Blair jebl...@openstack.orgwrote: Joshua Harlow harlo...@yahoo-inc.com writes: It seems simple to have variations of venvs (or something similar) that taskflow tox.ini can have that specify the different 0.7, 0.8, 0.9, when sqlalchemy 1.0 comes out then this should become a nonissue (hopefully). I will bug the infra folks to see what can be done here (hopefully this is as simple as it sounds). It is. See pecan for an example: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml#n1636 toxgen is another tool that might be useful for setting up your tests. It tries to simplify creating a tox.ini with variations on different axes (like changing versions of sqlalchemy but keeping all the other dependencies the same). There's source for toxgen at https://bitbucket.org/cdevienne/toxgen and there's a copy in the WSME tree (I don't know if they're the same) with a tox-tmpl.ini file to serve as an example. Doug And thanks to Ryan Petrello for setting that system up! -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 2014-01-06 17:23:31 -0500 (-0500), Doug Hellmann wrote: [...] The global requirements syncing seems to have fixed the issue for apps, although it just occurred to me that I'm not sure we check that the requirements lists are the same when we cut a release. Do we do that already? Not yet in any automated fashion but keep in mind that we've only gotten the requirements update proposal job working reliably this cycle, so it could still take some time for the various projects to decide how to finish syncing up. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
With regards to the futures module it should just work fine with the packaging of https://pypi.python.org/pypi/futures which is a backport of the 3.2 concurrent futures package into 2.6,2.7, so that's the package there. Feel free to bug me on irc if u want any other help with dependencies, the entrypoint failures shouldn't happen (possibly due to this). If u need help just bug harlowja on irc or jump in #openstack-state-management where others can help to... Sent from my really tiny device... On Jan 4, 2014, at 7:26 AM, Thomas Goirand z...@debian.org wrote: Sean, Before everything, I'd like to thank you for insisting in making the transition to SQLA 0.8.x. Since it has been uploaded to Sid, this SQLA 0.7.99 has been without any doubt the biggest reoccurring pain in the but with the packaging of OpenStack. Without people like you, insisting again and again, I would have loose hope that progress could happen in OpenStack! So thanks again, Sean. On 01/03/2014 11:26 PM, Sean Dague wrote: Given that sqla 0.9 just came out, I wanted to explore, again, what the state of the world was with sqla 0.8 (especially given that Ubuntu and Red Hat are both shipping 0.8 in their OpenStack bundles) - https://review.openstack.org/#/c/64831/ The answer is not great. But more importantly, the answer is actually worse than the last time we checked, which I think gives this some urgency to move forward. For me, it's been urgent since the 6th of July... SQLA 0.8.2 was uploaded to Sid on the 6th of July. Since then, I've been bugging everyone on this list about it, explaining that I have no choice but to have Debian packages to support it (since I upload in Sid, and Sid has SQL 0.8.x). It seems I still haven't been heard. Now, we're 6 months after that, and after the release of Havana which happened more than 3 months after this, and after everything was fixed in all core packages (the last one was heat, at the end of August). So, in January 2014, I'm still fixing manually most requirements.txt, which still advertize for support of only SQL 0.7.99. I currently have fixes-SQLAlchemy-requirement.patch in Cinder, Neutron, and Nova (for Havana), just to allow it and stop the software to break because it was decided it is the case, even though they perfectly work with SQLA 0.8. Some other project have SQLAlchemy=0.7.8,=0.7.99 in the requirements.txt, but do not break as badly as these 3 just because of the bad declaration that doesn't reflect the reality (that's the case for Heat Keystone and Glance, for example). Something is going extremely wrong here. And seeing the actual result of the SQLA transition, I am really leaning toward thinking we have a process problem. What I believe is wrong, is that instead of having project wide decisions imposing some constraints, we have leaf packages that do. Until absolutely all of OpenStack is ready and fixed, then there's no constraint applied. This is the thing that must change. It shouldn't be this way. It should be from top to bottom. While I do understand that we do need the gate to be able to continue working with all projects at any given time, we still must find a solution so that this kind of 6 months transition never happens again. This really goes against the culture that we have inside OpenStack, and our common belief that things must be able to move fast. On 01/04/2014 04:13 AM, Sean Dague wrote: Because of entry points any library that specifies any dependencies that OpenStack components specify, at incompatible levels, means that library effectively puts a hold on the rest of OpenStack and prevents it from being able to move forward. That's exactly the kind of *very bad* habits that needs to stop in OpenStack. On 01/04/2014 04:13 AM, Sean Dague wrote: The only other option is that libraries we own (stackforge / oslo), for condition to be included in global- requirements, *can never* specify a maximum version of any dependency (and I really mean never), and can never specify a minimum greater than current global requirements. PLEASE !!! Let's do this !!! :) On 01/04/2014 05:29 AM, Sean Dague wrote: It actually tells us that a human, somewhere, decided that their software did not work with some combination of other software Often it's even worse. Sometimes, a human decide that, just it case, the software will declare itself incompatible with some non-existent future version of another software, even if there's no way to know. We're even more into sci-fi when we see stuff like: pbr=0.5.21,1 Monty, did you decide you would release 1.0 with lots of backward incompatibility? Has the topic been raised and I missed it??? I'm convinced this isn't the case (and let's pretend it isn't, just until the end of this message). So, how does one know that the thing he's using in PBR will be the thing that will cause trouble later on? For a version which hasn't been released
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
I've skimmed the rest of the thread and not seen something mentioned that seems like it matters a lot. If I missed this suggestion buried deep in the ensuing discussion, I apologize for that. Since TaskFlow wants to be generally consumable and not only driven as an OpenStack component, it should not be following global requirements. It should actually expand its SQL Alchemy compatibility to all supported versions of SQLAlchemy. Ideally it would have a gate for each major version of SQL Alchemy that upstream supports. Otherwise it will never even be able to work with any project that doesn't share its SQL Alchemy version pin. Excerpts from Joshua Harlow's message of 2014-01-03 08:37:17 -0800: So taskflow was tested with the version of sqlalchemy that was available and in the requirements at the time of its 0.1 release (taskflow syncs it's requirements from the same global requirements). From what I remember this is the same requirement that everyone else is bound to: SQLAlchemy=0.7.8,=0.7.99 I can unpin it if this is desired (the openstack requirements repo has the same version restriction). What would be recommended here? As more code moves to pypi reusable libraries (oslo.db when it arrives comes to mind) I think this will be hit more often. Let's come up with a good strategy to follow. Thoughts?? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
Agreed, we are going to expand it and work on figuring out how to test against multiple versions. It does work with 0.8 and it seems even like 0.9 works fine also. But all compatible also means I can't guarantee 0.10 (if it comes out) will work since afaik semver means sqlalchemy could still break things when it's 1.0. Anyone got a time machine I can use to check the future ;-) It seems simple to have variations of venvs (or something similar) that taskflow tox.ini can have that specify the different 0.7, 0.8, 0.9, when sqlalchemy 1.0 comes out then this should become a nonissue (hopefully). I will bug the infra folks to see what can be done here (hopefully this is as simple as it sounds). Sent from my really tiny device... On Jan 5, 2014, at 8:22 AM, Clint Byrum cl...@fewbar.com wrote: I've skimmed the rest of the thread and not seen something mentioned that seems like it matters a lot. If I missed this suggestion buried deep in the ensuing discussion, I apologize for that. Since TaskFlow wants to be generally consumable and not only driven as an OpenStack component, it should not be following global requirements. It should actually expand its SQL Alchemy compatibility to all supported versions of SQLAlchemy. Ideally it would have a gate for each major version of SQL Alchemy that upstream supports. Otherwise it will never even be able to work with any project that doesn't share its SQL Alchemy version pin. Excerpts from Joshua Harlow's message of 2014-01-03 08:37:17 -0800: So taskflow was tested with the version of sqlalchemy that was available and in the requirements at the time of its 0.1 release (taskflow syncs it's requirements from the same global requirements). From what I remember this is the same requirement that everyone else is bound to: SQLAlchemy=0.7.8,=0.7.99 I can unpin it if this is desired (the openstack requirements repo has the same version restriction). What would be recommended here? As more code moves to pypi reusable libraries (oslo.db when it arrives comes to mind) I think this will be hit more often. Let's come up with a good strategy to follow. Thoughts?? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
Joshua Harlow harlo...@yahoo-inc.com writes: It seems simple to have variations of venvs (or something similar) that taskflow tox.ini can have that specify the different 0.7, 0.8, 0.9, when sqlalchemy 1.0 comes out then this should become a nonissue (hopefully). I will bug the infra folks to see what can be done here (hopefully this is as simple as it sounds). It is. See pecan for an example: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml#n1636 And thanks to Ryan Petrello for setting that system up! -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 01/03/2014 08:27 PM, Robert Collins wrote: On 4 January 2014 08:44, Doug Hellmann doug.hellm...@dreamhost.com wrote: It seems safer to gate changes to libraries against the apps' trunk (to avoid making backwards-incompatible changes), and then gate changes to the apps against the released libraries (to ensure they work with something available to be packaged by the distros). There are lots and lots of version numbers available to us, so I see no problem with releasing new versions of libraries frequently. So we used to do that the apps against release libraries. And the result was more and more full day gate breaks. We did 2 consecutive ones in 2 weeks. Basically, once you get to be a certain level of coupled in OpenStack we can no longer let you manage your own requirements file. We need a global lever on it. Because people were doing it wrong, and slowly (we could go through specific examples about how bad this was. This was a top issue at nearly every summit I'd been at going back to Essex. Some reading from the break times: * http://lists.openstack.org/pipermail/openstack-dev/2013-July/011660.html * http://lists.openstack.org/pipermail/openstack-dev/2013-July/011708.html * http://lists.openstack.org/pipermail/openstack-dev/2013-July/012342.html (It was about 14 days to resolve the python client issue, there was a django issue around the same time that never made it to the list, as we did it all under fire in IRC) And we have a solution now. Which is one list of requirements that we can test everything with, that we can propose requirements updates speculatively, and see what works and what doesn't. And *after* we know they work, we propose the changes back into the projects, now automatically. I do see the issue Sean is pointing at, which is that we have to fix the libraries first and then the things that use them. OTOH thats normal in the software world, I don't see anything unique about it. Well, as the person that normally gets stuck figuring this out when .eu has been gate blocked for a day, and I'm one of the first people up on the east coast, I find the normal state of affairs unsatisfying. :) I also think that what we are basically dealing with is the classical N^2 comms problem. With N git trees that we need to all get working together, this gets exponentially more difficult over time. Which is why we created the integrated gate and the global requirements lever. Another solution would be reduce the number of OpenStack git trees to make N^2 more manageable, and let us with single commits affect multiple components. But that's not the direction we've taken. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
Sean, Before everything, I'd like to thank you for insisting in making the transition to SQLA 0.8.x. Since it has been uploaded to Sid, this SQLA 0.7.99 has been without any doubt the biggest reoccurring pain in the but with the packaging of OpenStack. Without people like you, insisting again and again, I would have loose hope that progress could happen in OpenStack! So thanks again, Sean. On 01/03/2014 11:26 PM, Sean Dague wrote: Given that sqla 0.9 just came out, I wanted to explore, again, what the state of the world was with sqla 0.8 (especially given that Ubuntu and Red Hat are both shipping 0.8 in their OpenStack bundles) - https://review.openstack.org/#/c/64831/ The answer is not great. But more importantly, the answer is actually worse than the last time we checked, which I think gives this some urgency to move forward. For me, it's been urgent since the 6th of July... SQLA 0.8.2 was uploaded to Sid on the 6th of July. Since then, I've been bugging everyone on this list about it, explaining that I have no choice but to have Debian packages to support it (since I upload in Sid, and Sid has SQL 0.8.x). It seems I still haven't been heard. Now, we're 6 months after that, and after the release of Havana which happened more than 3 months after this, and after everything was fixed in all core packages (the last one was heat, at the end of August). So, in January 2014, I'm still fixing manually most requirements.txt, which still advertize for support of only SQL 0.7.99. I currently have fixes-SQLAlchemy-requirement.patch in Cinder, Neutron, and Nova (for Havana), just to allow it and stop the software to break because it was decided it is the case, even though they perfectly work with SQLA 0.8. Some other project have SQLAlchemy=0.7.8,=0.7.99 in the requirements.txt, but do not break as badly as these 3 just because of the bad declaration that doesn't reflect the reality (that's the case for Heat Keystone and Glance, for example). Something is going extremely wrong here. And seeing the actual result of the SQLA transition, I am really leaning toward thinking we have a process problem. What I believe is wrong, is that instead of having project wide decisions imposing some constraints, we have leaf packages that do. Until absolutely all of OpenStack is ready and fixed, then there's no constraint applied. This is the thing that must change. It shouldn't be this way. It should be from top to bottom. While I do understand that we do need the gate to be able to continue working with all projects at any given time, we still must find a solution so that this kind of 6 months transition never happens again. This really goes against the culture that we have inside OpenStack, and our common belief that things must be able to move fast. On 01/04/2014 04:13 AM, Sean Dague wrote: Because of entry points any library that specifies any dependencies that OpenStack components specify, at incompatible levels, means that library effectively puts a hold on the rest of OpenStack and prevents it from being able to move forward. That's exactly the kind of *very bad* habits that needs to stop in OpenStack. On 01/04/2014 04:13 AM, Sean Dague wrote: The only other option is that libraries we own (stackforge / oslo), for condition to be included in global- requirements, *can never* specify a maximum version of any dependency (and I really mean never), and can never specify a minimum greater than current global requirements. PLEASE !!! Let's do this !!! :) On 01/04/2014 05:29 AM, Sean Dague wrote: It actually tells us that a human, somewhere, decided that their software did not work with some combination of other software Often it's even worse. Sometimes, a human decide that, just it case, the software will declare itself incompatible with some non-existent future version of another software, even if there's no way to know. We're even more into sci-fi when we see stuff like: pbr=0.5.21,1 Monty, did you decide you would release 1.0 with lots of backward incompatibility? Has the topic been raised and I missed it??? I'm convinced this isn't the case (and let's pretend it isn't, just until the end of this message). So, how does one know that the thing he's using in PBR will be the thing that will cause trouble later on? For a version which hasn't been released yet?!? Who's the person with such a looking glass, who can predict the future? I'd like to know as well some stuff in my personal future... Please, let's stop this kind of non-sense and stop pretending we can tell if something is incompatible with a version of something else that doesn't even exist yet. It's hurting and slowing down the whole of OpenStack for no reason. One of the key problems is taskflow, which has an sqla pin, which breaks all the cinder entry points. This was actually entirely the problem global requirements was meant to address, but it really can't when there are nested requirements like this, in stackforge projects
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 01/04/2014 07:53 AM, Joshua Harlow wrote: So another idea that was talked about on IRC. Taskflow exposes entrypoints for these storage backends (like your storage callback/interface idea). It currently provides 3 such 'default' backends [sqlalchemy, file/dir based, in-memory -- mainly for testing]. A 4th one is in progress for icehouse (zookeeper based). These backends are used to store intermediary 'flow/task' state (allowing the taskflow engine to resume if an app crashes/stopped while doing stuff). A couple ideas about this, since its already pluggable. Split the sqlalchemy backend - 'taskflow-sa' repo/new package. For those projects that want to use this backend, they include it (still means 'taskflow-sa' package has a dependency on sqlalchemy of some version). Another idea is to move the sqlalchemy dependency currently in taskflow requirements.txt - taskflow test-requirements.txt and for those projects that want to use the sqlalchemy backend, they include the sqlalchemy version themselves in there requirements.txt (taskflow keeps it in test-requirements.txt so that it can run its unit/integration tests, making sure the backend still works). I'm not really sure which is the best, none seem super-great, but sometimes u just work with what u got :-P Feel free to explore all kinds of direction you want, but *AFTER* we've moved to SQLA 0.8. It's been really too long already... Thanks for your understanding. Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
Such a bad state seems like FUD. Taskflow was just syncing its requirements with the same requirements that everyone else is... Those global requirements have 0.7.99 in them as we speak (which is why taskflow picked up that version). The issue here will be worked through and fixed, it won't be the last time a library that is used in various projects causes dependency issues, so we are working through this process as we learn what works best. 1, don't sync with that requirements file to attempt to use the same version as integrating projects, or become more integrated with the gate... or a few other solutions that have been discussed... New release will happen early next week of taskflow with adjusted sqlalchemy upper bound. Sent from my really tiny device... On Jan 4, 2014, at 7:27 AM, Thomas Goirand z...@debian.org wrote: On 01/04/2014 06:10 AM, Ivan Melnikov wrote: On 04.01.2014 01:29, Sean Dague wrote: On 01/03/2014 04:17 PM, Doug Hellmann wrote: [...] That's what made me think of the solution. But isn't setuptools in fact telling us that somehow the versions of things we expected to have installed are no longer installed and so something *is* broken (even if the versions of the installed libraries work together). It actually tells us that a human, somewhere, decided that their software did not work with some combination of other software, and that we are no longer able to correct their mistaken assumptions. [...] But sometimes humans are not wrong. For example, no released TaskFlow version really works with SQLAlchemy = 0.8 -- that was fixed only recently (https://review.openstack.org/#/c/62661/). What's wrong is to allow taskflow to be added to the global-requirements if it is in such a bad state, blocking such an important transition that has been needed for more than 6 months. Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
I was more of referring to general dependency issues, sqlalchemy hopefully never again but one never knows :P Sent from my really tiny device... On Jan 4, 2014, at 8:40 AM, Thomas Goirand z...@debian.org wrote: On 01/05/2014 12:12 AM, Joshua Harlow wrote: it won't be the last time a library that is used in various projects causes dependency issues Please, tell me the opposite thing. Please tell me that this is the last time we're having a discussion about problems with SQLA 0.8. Please tell me that we're actually learning from these mistakes and that we'll see improvements... Thomas Db Csq ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-devsaws ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 5 January 2014 04:22, Thomas Goirand z...@debian.org wrote: Sean, Before everything, I'd like to thank you for insisting in making the transition to SQLA 0.8.x. Since it has been uploaded to Sid, this SQLA 0.7.99 has been without any doubt the biggest reoccurring pain in the but with the packaging of OpenStack. Without people like you, insisting again and again, I would have loose hope that progress could happen in OpenStack! So thanks again, Sean. We're even more into sci-fi when we see stuff like: pbr=0.5.21,1 Monty, did you decide you would release 1.0 with lots of backward incompatibility? Has the topic been raised and I missed it??? I'm convinced this isn't the case (and let's pretend it isn't, just until the end of this message). Strictly speak, yes. More generously, this should be pbr=0.5.21,2 Because pbr is using semver, and 0.x has no stability guarantees, so the point when we will hit an incompatible change to a stable API is the 2 transition. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
Given that sqla 0.9 just came out, I wanted to explore, again, what the state of the world was with sqla 0.8 (especially given that Ubuntu and Red Hat are both shipping 0.8 in their OpenStack bundles) - https://review.openstack.org/#/c/64831/ The answer is not great. But more importantly, the answer is actually worse than the last time we checked, which I think gives this some urgency to move forward. One of the key problems is taskflow, which has an sqla pin, which breaks all the cinder entry points. This was actually entirely the problem global requirements was meant to address, but it really can't when there are nested requirements like this, in stackforge projects that aren't installing via git (so we can rewrite their requirements). So why does taskflow have the SQLA pin? And if the answer is global requirements, then taskflow can't be installing via pypi like it is now, because it's by nature taking a wedge. So we need a real solution here to un bind us, because right now it's actually impossible to upgrade sqla in requirements because of taskflow. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
So taskflow was tested with the version of sqlalchemy that was available and in the requirements at the time of its 0.1 release (taskflow syncs it's requirements from the same global requirements). From what I remember this is the same requirement that everyone else is bound to: SQLAlchemy=0.7.8,=0.7.99 I can unpin it if this is desired (the openstack requirements repo has the same version restriction). What would be recommended here? As more code moves to pypi reusable libraries (oslo.db when it arrives comes to mind) I think this will be hit more often. Let's come up with a good strategy to follow. Thoughts?? Sent from my really tiny device... On Jan 3, 2014, at 7:30 AM, Sean Dague s...@dague.netmailto:s...@dague.net wrote: Given that sqla 0.9 just came out, I wanted to explore, again, what the state of the world was with sqla 0.8 (especially given that Ubuntu and Red Hat are both shipping 0.8 in their OpenStack bundles) - https://review.openstack.org/#/c/64831/ The answer is not great. But more importantly, the answer is actually worse than the last time we checked, which I think gives this some urgency to move forward. One of the key problems is taskflow, which has an sqla pin, which breaks all the cinder entry points. This was actually entirely the problem global requirements was meant to address, but it really can't when there are nested requirements like this, in stackforge projects that aren't installing via git (so we can rewrite their requirements). So why does taskflow have the SQLA pin? And if the answer is global requirements, then taskflow can't be installing via pypi like it is now, because it's by nature taking a wedge. So we need a real solution here to un bind us, because right now it's actually impossible to upgrade sqla in requirements because of taskflow. -Sean -- Sean Dague Samsung Research America s...@dague.netmailto:s...@dague.net / sean.da...@samsung.commailto:sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 01/03/2014 11:37 AM, Joshua Harlow wrote: So taskflow was tested with the version of sqlalchemy that was available and in the requirements at the time of its 0.1 release (taskflow syncs it's requirements from the same global requirements). From what I remember this is the same requirement that everyone else is bound to: SQLAlchemy=0.7.8,=0.7.99 I can unpin it if this is desired (the openstack requirements repo has the same version restriction). What would be recommended here? As more code moves to pypi reusable libraries (oslo.db when it arrives comes to mind) I think this will be hit more often. Let's come up with a good strategy to follow. Thoughts?? So I think that given taskflow's usage, it really needs to live under the Oslo program, and follow the same rules that are applied to oslo libraries. (https://wiki.openstack.org/wiki/Oslo#Graduation) Which means it needs to be part of the integrated gate, so we can update it's requirements globally. It also means that changes to it will be gated on full devstack runs. We can work through the details on #openstack-infra. ttx has been doing the same for oslo.rootwrap this week. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
Ok, I think I'm fine with that (although not really sure what that entails). What does the living under the 'oslo program' change? Does that entail getting sucked into the incubator (which seems to be what your graduating link is about). I don't think its a good idea for taskflow to be in the 'incubator'. Taskflow is meant to be just like any other 3rd party library. Or were u mainly referring to the 'devstack-gate integration' section? I'd be interested in hearing dougs opinion here (cc'd him) as https://etherpad.openstack.org/p/icehouse-oslo-splitting-the-incubator would seem to cause even more of these types of new 3rd party libraries to appear on pypi (and therefore causing similar issues of transitive dependencies as taskflow). Will bug u on #openstack-infra soon :-) On 1/3/14, 9:05 AM, Sean Dague s...@dague.net wrote: On 01/03/2014 11:37 AM, Joshua Harlow wrote: So taskflow was tested with the version of sqlalchemy that was available and in the requirements at the time of its 0.1 release (taskflow syncs it's requirements from the same global requirements). From what I remember this is the same requirement that everyone else is bound to: SQLAlchemy=0.7.8,=0.7.99 I can unpin it if this is desired (the openstack requirements repo has the same version restriction). What would be recommended here? As more code moves to pypi reusable libraries (oslo.db when it arrives comes to mind) I think this will be hit more often. Let's come up with a good strategy to follow. Thoughts?? So I think that given taskflow's usage, it really needs to live under the Oslo program, and follow the same rules that are applied to oslo libraries. (https://wiki.openstack.org/wiki/Oslo#Graduation) Which means it needs to be part of the integrated gate, so we can update it's requirements globally. It also means that changes to it will be gated on full devstack runs. We can work through the details on #openstack-infra. ttx has been doing the same for oslo.rootwrap this week. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 01/03/2014 12:45 PM, Joshua Harlow wrote: Ok, I think I'm fine with that (although not really sure what that entails). What does the living under the 'oslo program' change? Does that entail getting sucked into the incubator (which seems to be what your graduating link is about). I don't think its a good idea for taskflow to be in the 'incubator'. Taskflow is meant to be just like any other 3rd party library. I didn't mean the incubator, I meant like oslo.* libs that we've spun out already. Or were u mainly referring to the 'devstack-gate integration' section? Correct. Just to understand what the libs live under. Basically taskflow is getting deeply integrated into projects in the same way oslo.* libs are, and as such, given it has non trivial requirements of it's own, we have to treat it like all the other OpenStack components and symmetrically gate on it. That will guaruntee you can't release a taskflow library that can break OpenStack, because we'll be testing every commit, which is goodness. I'd be interested in hearing dougs opinion here (cc'd him) as https://etherpad.openstack.org/p/icehouse-oslo-splitting-the-incubator would seem to cause even more of these types of new 3rd party libraries to appear on pypi (and therefore causing similar issues of transitive dependencies as taskflow). Will bug u on #openstack-infra soon :-) Definitely think doug should weigh in as well. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
Sounds good to me. Talked on #openstack-infra with some folks there and just awaiting next steps. Doesn't seem like should be anything to hard to adjust/move/... -Josh On 1/3/14, 11:27 AM, Sean Dague s...@dague.net wrote: On 01/03/2014 12:45 PM, Joshua Harlow wrote: Ok, I think I'm fine with that (although not really sure what that entails). What does the living under the 'oslo program' change? Does that entail getting sucked into the incubator (which seems to be what your graduating link is about). I don't think its a good idea for taskflow to be in the 'incubator'. Taskflow is meant to be just like any other 3rd party library. I didn't mean the incubator, I meant like oslo.* libs that we've spun out already. Or were u mainly referring to the 'devstack-gate integration' section? Correct. Just to understand what the libs live under. Basically taskflow is getting deeply integrated into projects in the same way oslo.* libs are, and as such, given it has non trivial requirements of it's own, we have to treat it like all the other OpenStack components and symmetrically gate on it. That will guaruntee you can't release a taskflow library that can break OpenStack, because we'll be testing every commit, which is goodness. I'd be interested in hearing dougs opinion here (cc'd him) as https://etherpad.openstack.org/p/icehouse-oslo-splitting-the-incubator would seem to cause even more of these types of new 3rd party libraries to appear on pypi (and therefore causing similar issues of transitive dependencies as taskflow). Will bug u on #openstack-infra soon :-) Definitely think doug should weigh in as well. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow harlo...@yahoo-inc.comwrote: Ok, I think I'm fine with that (although not really sure what that entails). What does the living under the 'oslo program' change? Does that entail getting sucked into the incubator (which seems to be what your graduating link is about). I don't think its a good idea for taskflow to be in the 'incubator'. Taskflow is meant to be just like any other 3rd party library. No, as we discussed in Hong Kong, there's no reason to add taskflow to the incubator. Whether or not it needs to be part of the oslo program (or any other program) is a separate question. I'm not opposed to bringing it in, but didn't see the point when it came up at the summit. I understand that moving taskflow into oslo would avoid the policy decision we have in place to not do symmetric gating on unreleased versions of things not owned by the OpenStack project. However, I don't know if we want to be testing against the git head of libraries no matter where they live. As fungi pointed out on IRC, gating against pre-release versions of libraries may allow us to reach a state where the software works when installed from git, but not from the released packages. It seems safer to gate changes to libraries against the apps' trunk (to avoid making backwards-incompatible changes), and then gate changes to the apps against the released libraries (to ensure they work with something available to be packaged by the distros). There are lots and lots of version numbers available to us, so I see no problem with releasing new versions of libraries frequently. Am I missing something that makes that not work? Doug Or were u mainly referring to the 'devstack-gate integration' section? I'd be interested in hearing dougs opinion here (cc'd him) as https://etherpad.openstack.org/p/icehouse-oslo-splitting-the-incubator would seem to cause even more of these types of new 3rd party libraries to appear on pypi (and therefore causing similar issues of transitive dependencies as taskflow). Will bug u on #openstack-infra soon :-) On 1/3/14, 9:05 AM, Sean Dague s...@dague.net wrote: On 01/03/2014 11:37 AM, Joshua Harlow wrote: So taskflow was tested with the version of sqlalchemy that was available and in the requirements at the time of its 0.1 release (taskflow syncs it's requirements from the same global requirements). From what I remember this is the same requirement that everyone else is bound to: SQLAlchemy=0.7.8,=0.7.99 I can unpin it if this is desired (the openstack requirements repo has the same version restriction). What would be recommended here? As more code moves to pypi reusable libraries (oslo.db when it arrives comes to mind) I think this will be hit more often. Let's come up with a good strategy to follow. Thoughts?? So I think that given taskflow's usage, it really needs to live under the Oslo program, and follow the same rules that are applied to oslo libraries. (https://wiki.openstack.org/wiki/Oslo#Graduation) Which means it needs to be part of the integrated gate, so we can update it's requirements globally. It also means that changes to it will be gated on full devstack runs. We can work through the details on #openstack-infra. ttx has been doing the same for oslo.rootwrap this week. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
I'd be up for that 'dual' gating. It would help make sure that nothing major is breaking in the next version as well as the version released on pypi isn't also causing problems. Although git head gating does seem odd, especially as git head/trunk is where things change (and should be allowed to break and change, change is good) and the level of false positives that would be raised might become to much of a pain. I'd rather not discourage innovation on trunk if we can; since this is what stable releases are for. Btw the sqlalchemy changes (unpinning) should be fine. Ivan did a couple of tests (basic stuff, not full integration). - https://review.openstack.org/#/c/64869/ - https://review.openstack.org/#/c/64881/ Both passed basic unit tests (full integration against real mysql, not sqlite will happen soon I hope with https://bugs.launchpad.net/taskflow/+bug/1265886). If we really need to I can push out a 0.1.2 with the unpinned version (one of the above reviews). -Josh From: Doug Hellmann doug.hellm...@dreamhost.commailto:doug.hellm...@dreamhost.com Date: Friday, January 3, 2014 at 11:44 AM To: Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com Cc: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org, Sean Dague s...@dague.netmailto:s...@dague.net Subject: Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote: Ok, I think I'm fine with that (although not really sure what that entails). What does the living under the 'oslo program' change? Does that entail getting sucked into the incubator (which seems to be what your graduating link is about). I don't think its a good idea for taskflow to be in the 'incubator'. Taskflow is meant to be just like any other 3rd party library. No, as we discussed in Hong Kong, there's no reason to add taskflow to the incubator. Whether or not it needs to be part of the oslo program (or any other program) is a separate question. I'm not opposed to bringing it in, but didn't see the point when it came up at the summit. I understand that moving taskflow into oslo would avoid the policy decision we have in place to not do symmetric gating on unreleased versions of things not owned by the OpenStack project. However, I don't know if we want to be testing against the git head of libraries no matter where they live. As fungi pointed out on IRC, gating against pre-release versions of libraries may allow us to reach a state where the software works when installed from git, but not from the released packages. It seems safer to gate changes to libraries against the apps' trunk (to avoid making backwards-incompatible changes), and then gate changes to the apps against the released libraries (to ensure they work with something available to be packaged by the distros). There are lots and lots of version numbers available to us, so I see no problem with releasing new versions of libraries frequently. Am I missing something that makes that not work? Doug Or were u mainly referring to the 'devstack-gate integration' section? I'd be interested in hearing dougs opinion here (cc'd him) as https://etherpad.openstack.org/p/icehouse-oslo-splitting-the-incubator would seem to cause even more of these types of new 3rd party libraries to appear on pypi (and therefore causing similar issues of transitive dependencies as taskflow). Will bug u on #openstack-infra soon :-) On 1/3/14, 9:05 AM, Sean Dague s...@dague.netmailto:s...@dague.net wrote: On 01/03/2014 11:37 AM, Joshua Harlow wrote: So taskflow was tested with the version of sqlalchemy that was available and in the requirements at the time of its 0.1 release (taskflow syncs it's requirements from the same global requirements). From what I remember this is the same requirement that everyone else is bound to: SQLAlchemy=0.7.8,=0.7.99 I can unpin it if this is desired (the openstack requirements repo has the same version restriction). What would be recommended here? As more code moves to pypi reusable libraries (oslo.db when it arrives comes to mind) I think this will be hit more often. Let's come up with a good strategy to follow. Thoughts?? So I think that given taskflow's usage, it really needs to live under the Oslo program, and follow the same rules that are applied to oslo libraries. (https://wiki.openstack.org/wiki/Oslo#Graduation) Which means it needs to be part of the integrated gate, so we can update it's requirements globally. It also means that changes to it will be gated on full devstack runs. We can work through the details on #openstack-infra. ttx has been doing the same for oslo.rootwrap this week. -Sean -- Sean Dague Samsung Research America s...@dague.netmailto:s...@dague.net / sean.da...@samsung.commailto:sean.da...@samsung.com http
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 01/03/2014 02:44 PM, Doug Hellmann wrote: On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com wrote: Ok, I think I'm fine with that (although not really sure what that entails). What does the living under the 'oslo program' change? Does that entail getting sucked into the incubator (which seems to be what your graduating link is about). I don't think its a good idea for taskflow to be in the 'incubator'. Taskflow is meant to be just like any other 3rd party library. No, as we discussed in Hong Kong, there's no reason to add taskflow to the incubator. Whether or not it needs to be part of the oslo program (or any other program) is a separate question. I'm not opposed to bringing it in, but didn't see the point when it came up at the summit. I understand that moving taskflow into oslo would avoid the policy decision we have in place to not do symmetric gating on unreleased versions of things not owned by the OpenStack project. However, I don't know if we want to be testing against the git head of libraries no matter where they live. As fungi pointed out on IRC, gating against pre-release versions of libraries may allow us to reach a state where the software works when installed from git, but not from the released packages. It seems safer to gate changes to libraries against the apps' trunk (to avoid making backwards-incompatible changes), and then gate changes to the apps against the released libraries (to ensure they work with something available to be packaged by the distros). There are lots and lots of version numbers available to us, so I see no problem with releasing new versions of libraries frequently. Am I missing something that makes that not work? Requirements wedging. Because of entry points any library that specifies any dependencies that OpenStack components specify, at incompatible levels, means that library effectively puts a hold on the rest of OpenStack and prevents it from being able to move forward. Which means every time we need to then change a library dependency it's going to require triggering a release, solely to change library dependencies. This is the giant disaster that we created global requirements to avoid, so that we have one nob to turn. The only other option is that libraries we own (stackforge / oslo), for condition to be included in global- requirements, *can never* specify a maximum version of any dependency (and I really mean never), and can never specify a minimum greater than current global requirements. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 2014-01-03 14:44:40 -0500 (-0500), Doug Hellmann wrote: [...] It seems safer to gate changes to libraries against the apps' trunk (to avoid making backwards-incompatible changes), and then gate changes to the apps against the released libraries (to ensure they work with something available to be packaged by the distros). There are lots and lots of version numbers available to us, so I see no problem with releasing new versions of libraries frequently. [...] Well, for the benefit of the pure OpenStack CD crowd, it would continue to make sense to also test trunk of everything together. I think what we had arrived at during the last summit was that we wanted to make sure that server changes are being gated against released libraries/clients *in addition to trunk* so that we ensure features are released in those before we start relying on them. We similarly probably need to test changes for a library/client against released versions of any other libraries or clients on which they depend as well, for those same reasons. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On Fri, Jan 3, 2014 at 3:13 PM, Sean Dague s...@dague.net wrote: On 01/03/2014 02:44 PM, Doug Hellmann wrote: On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com wrote: Ok, I think I'm fine with that (although not really sure what that entails). What does the living under the 'oslo program' change? Does that entail getting sucked into the incubator (which seems to be what your graduating link is about). I don't think its a good idea for taskflow to be in the 'incubator'. Taskflow is meant to be just like any other 3rd party library. No, as we discussed in Hong Kong, there's no reason to add taskflow to the incubator. Whether or not it needs to be part of the oslo program (or any other program) is a separate question. I'm not opposed to bringing it in, but didn't see the point when it came up at the summit. I understand that moving taskflow into oslo would avoid the policy decision we have in place to not do symmetric gating on unreleased versions of things not owned by the OpenStack project. However, I don't know if we want to be testing against the git head of libraries no matter where they live. As fungi pointed out on IRC, gating against pre-release versions of libraries may allow us to reach a state where the software works when installed from git, but not from the released packages. It seems safer to gate changes to libraries against the apps' trunk (to avoid making backwards-incompatible changes), and then gate changes to the apps against the released libraries (to ensure they work with something available to be packaged by the distros). There are lots and lots of version numbers available to us, so I see no problem with releasing new versions of libraries frequently. Am I missing something that makes that not work? Requirements wedging. Because of entry points any library that specifies any dependencies that OpenStack components specify, at incompatible levels, means that library effectively puts a hold on the rest of OpenStack and prevents it from being able to move forward. I have considered changing stevedore so it uses entry points to find plugins, but then handles the import itself specifically to eliminate the requirements checks for plugin dependencies enforced by pkg_resources. So far I haven't convinced myself that it's a good idea, but I'm open to discussion. :-) Which means every time we need to then change a library dependency it's going to require triggering a release, solely to change library dependencies. This is the giant disaster that we created global requirements to avoid, so that we have one nob to turn. The only other option is that libraries we own (stackforge / oslo), for condition to be included in global- requirements, *can never* specify a maximum version of any dependency (and I really mean never), and can never specify a minimum greater than current global requirements. That seems like a more appealing solution than testing against unreleased versions via git checkouts. It is is also an approach we could encourage authors of libraries we don't own to use. Doug -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 01/03/2014 03:30 PM, Doug Hellmann wrote: On Fri, Jan 3, 2014 at 3:13 PM, Sean Dague s...@dague.net mailto:s...@dague.net wrote: On 01/03/2014 02:44 PM, Doug Hellmann wrote: On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com__ wrote: Ok, I think I'm fine with that (although not really sure what that entails). What does the living under the 'oslo program' change? Does that entail getting sucked into the incubator (which seems to be what your graduating link is about). I don't think its a good idea for taskflow to be in the 'incubator'. Taskflow is meant to be just like any other 3rd party library. No, as we discussed in Hong Kong, there's no reason to add taskflow to the incubator. Whether or not it needs to be part of the oslo program (or any other program) is a separate question. I'm not opposed to bringing it in, but didn't see the point when it came up at the summit. I understand that moving taskflow into oslo would avoid the policy decision we have in place to not do symmetric gating on unreleased versions of things not owned by the OpenStack project. However, I don't know if we want to be testing against the git head of libraries no matter where they live. As fungi pointed out on IRC, gating against pre-release versions of libraries may allow us to reach a state where the software works when installed from git, but not from the released packages. It seems safer to gate changes to libraries against the apps' trunk (to avoid making backwards-incompatible changes), and then gate changes to the apps against the released libraries (to ensure they work with something available to be packaged by the distros). There are lots and lots of version numbers available to us, so I see no problem with releasing new versions of libraries frequently. Am I missing something that makes that not work? Requirements wedging. Because of entry points any library that specifies any dependencies that OpenStack components specify, at incompatible levels, means that library effectively puts a hold on the rest of OpenStack and prevents it from being able to move forward. I have considered changing stevedore so it uses entry points to find plugins, but then handles the import itself specifically to eliminate the requirements checks for plugin dependencies enforced by pkg_resources. So far I haven't convinced myself that it's a good idea, but I'm open to discussion. :-) Well, about 1/3 of our requirements wedges are actually completely entry point related. They were a class of problems we never had before we switched to entry points. This entire taskflow block wouldn't have been a problem if it wasn't loaded with entry points. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On Fri, Jan 3, 2014 at 4:08 PM, Sean Dague s...@dague.net wrote: On 01/03/2014 03:30 PM, Doug Hellmann wrote: On Fri, Jan 3, 2014 at 3:13 PM, Sean Dague s...@dague.net mailto:s...@dague.net wrote: On 01/03/2014 02:44 PM, Doug Hellmann wrote: On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com__ wrote: Ok, I think I'm fine with that (although not really sure what that entails). What does the living under the 'oslo program' change? Does that entail getting sucked into the incubator (which seems to be what your graduating link is about). I don't think its a good idea for taskflow to be in the 'incubator'. Taskflow is meant to be just like any other 3rd party library. No, as we discussed in Hong Kong, there's no reason to add taskflow to the incubator. Whether or not it needs to be part of the oslo program (or any other program) is a separate question. I'm not opposed to bringing it in, but didn't see the point when it came up at the summit. I understand that moving taskflow into oslo would avoid the policy decision we have in place to not do symmetric gating on unreleased versions of things not owned by the OpenStack project. However, I don't know if we want to be testing against the git head of libraries no matter where they live. As fungi pointed out on IRC, gating against pre-release versions of libraries may allow us to reach a state where the software works when installed from git, but not from the released packages. It seems safer to gate changes to libraries against the apps' trunk (to avoid making backwards-incompatible changes), and then gate changes to the apps against the released libraries (to ensure they work with something available to be packaged by the distros). There are lots and lots of version numbers available to us, so I see no problem with releasing new versions of libraries frequently. Am I missing something that makes that not work? Requirements wedging. Because of entry points any library that specifies any dependencies that OpenStack components specify, at incompatible levels, means that library effectively puts a hold on the rest of OpenStack and prevents it from being able to move forward. I have considered changing stevedore so it uses entry points to find plugins, but then handles the import itself specifically to eliminate the requirements checks for plugin dependencies enforced by pkg_resources. So far I haven't convinced myself that it's a good idea, but I'm open to discussion. :-) Well, about 1/3 of our requirements wedges are actually completely entry point related. They were a class of problems we never had before we switched to entry points. That's what made me think of the solution. But isn't setuptools in fact telling us that somehow the versions of things we expected to have installed are no longer installed and so something *is* broken (even if the versions of the installed libraries work together). Doug This entire taskflow block wouldn't have been a problem if it wasn't loaded with entry points. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 01/03/2014 04:17 PM, Doug Hellmann wrote: On Fri, Jan 3, 2014 at 4:08 PM, Sean Dague s...@dague.net mailto:s...@dague.net wrote: On 01/03/2014 03:30 PM, Doug Hellmann wrote: On Fri, Jan 3, 2014 at 3:13 PM, Sean Dague s...@dague.net mailto:s...@dague.net mailto:s...@dague.net mailto:s...@dague.net wrote: On 01/03/2014 02:44 PM, Doug Hellmann wrote: On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com__ mailto:harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com mailto:harlo...@yahoo-inc.com wrote: Ok, I think I'm fine with that (although not really sure what that entails). What does the living under the 'oslo program' change? Does that entail getting sucked into the incubator (which seems to be what your graduating link is about). I don't think its a good idea for taskflow to be in the 'incubator'. Taskflow is meant to be just like any other 3rd party library. No, as we discussed in Hong Kong, there's no reason to add taskflow to the incubator. Whether or not it needs to be part of the oslo program (or any other program) is a separate question. I'm not opposed to bringing it in, but didn't see the point when it came up at the summit. I understand that moving taskflow into oslo would avoid the policy decision we have in place to not do symmetric gating on unreleased versions of things not owned by the OpenStack project. However, I don't know if we want to be testing against the git head of libraries no matter where they live. As fungi pointed out on IRC, gating against pre-release versions of libraries may allow us to reach a state where the software works when installed from git, but not from the released packages. It seems safer to gate changes to libraries against the apps' trunk (to avoid making backwards-incompatible changes), and then gate changes to the apps against the released libraries (to ensure they work with something available to be packaged by the distros). There are lots and lots of version numbers available to us, so I see no problem with releasing new versions of libraries frequently. Am I missing something that makes that not work? Requirements wedging. Because of entry points any library that specifies any dependencies that OpenStack components specify, at incompatible levels, means that library effectively puts a hold on the rest of OpenStack and prevents it from being able to move forward. I have considered changing stevedore so it uses entry points to find plugins, but then handles the import itself specifically to eliminate the requirements checks for plugin dependencies enforced by pkg_resources. So far I haven't convinced myself that it's a good idea, but I'm open to discussion. :-) Well, about 1/3 of our requirements wedges are actually completely entry point related. They were a class of problems we never had before we switched to entry points. That's what made me think of the solution. But isn't setuptools in fact telling us that somehow the versions of things we expected to have installed are no longer installed and so something *is* broken (even if the versions of the installed libraries work together). It actually tells us that a human, somewhere, decided that their software did not work with some combination of other software, and that we are no longer able to correct their mistaken assumptions. There is a reason the Linux distros don't blindly follow what libraries say they depend on, because humans are foulable, and can only look backwards and not forwards in time. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 04.01.2014 01:29, Sean Dague wrote: On 01/03/2014 04:17 PM, Doug Hellmann wrote: [...] That's what made me think of the solution. But isn't setuptools in fact telling us that somehow the versions of things we expected to have installed are no longer installed and so something *is* broken (even if the versions of the installed libraries work together). It actually tells us that a human, somewhere, decided that their software did not work with some combination of other software, and that we are no longer able to correct their mistaken assumptions. [...] But sometimes humans are not wrong. For example, no released TaskFlow version really works with SQLAlchemy = 0.8 -- that was fixed only recently (https://review.openstack.org/#/c/62661/). I consider requirements to be part of the code, so if they are too strict (or too broad), that must be fixed, in a usual opensource way: via filing bug report, suggesting patch and so on. Requirements should reflect reality, especially for libraries that are intended to be useful outside of OpenStack, too. For example, current TaskFlow requirement for SQLAlchemy is too strict, so we'll fix that, and release new version with that fix. -- WBR, Ivan A. Melnikov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 01/03/2014 05:10 PM, Ivan Melnikov wrote: On 04.01.2014 01:29, Sean Dague wrote: On 01/03/2014 04:17 PM, Doug Hellmann wrote: [...] That's what made me think of the solution. But isn't setuptools in fact telling us that somehow the versions of things we expected to have installed are no longer installed and so something *is* broken (even if the versions of the installed libraries work together). It actually tells us that a human, somewhere, decided that their software did not work with some combination of other software, and that we are no longer able to correct their mistaken assumptions. [...] But sometimes humans are not wrong. For example, no released TaskFlow version really works with SQLAlchemy = 0.8 -- that was fixed only recently (https://review.openstack.org/#/c/62661/). I consider requirements to be part of the code, so if they are too strict (or too broad), that must be fixed, in a usual opensource way: via filing bug report, suggesting patch and so on. Requirements should reflect reality, especially for libraries that are intended to be useful outside of OpenStack, too. For example, current TaskFlow requirement for SQLAlchemy is too strict, so we'll fix that, and release new version with that fix. Well part of the problem is because of it being a dependency of a dependency, our global requirements process couldn't explore whether or not it functioned in a real environment. Which brings us back to the idea of making this a project that works in the integrated gate. It's also kind of problematic that apparently the introduction of taskflow actually caused a regression from Havana (which the distros managed to make work with sqla 0.8 even though it wasn't in global requirements, but this apparently would have broken). And the next question is when is a sqla 0.9 compatible version going to be out there? Because we can't even explore allowing openstack to use sqla 0.9 until taskflow does, again because it's a blocking requirement. It's exactly these kind of lock step problems that we avoid with the rest of openstack by making it an integrated gate with global requirements sync. But that we can't really handle with the library model. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
Since the library model is what most everyone else uses outside of openstack (I assume?) what can we do to get there so that this model works as well? Expanding dependencies recursively seems like it could help? This could then detect transitive dependency issues (and doesn't seem so hard to do). I like the idea of 'gate on all the things' (that I've heard come up before) but I don't know if its possible? If we hooked into the pypi upload 'stream' it would seem like we could automatically trigger openstack builds when a known dependency (or dependency of a dependency...) is uploaded (maybe?). That would be pretty neat. In general it really seems like having more libraries, not less is ideal (making us solve this issue whether we want to or not really). As libraries can then be used outside of openstack (taskflow is already being used elsewhere also), libraries create well-defined apis and boundaries between programs (...). I know they also create this dependency hell problem (anvil has been hitting these same issues for a while to). But I think we can figure out a solution that fits both worlds, the world of things we can gate on and the world of things we can't (3rd party libraries that aren't under openstack control). Taskflow is in-between those worlds, so it allows us to explore how to make both of those worlds work. -Josh On 1/3/14, 2:54 PM, Sean Dague s...@dague.net wrote: On 01/03/2014 05:10 PM, Ivan Melnikov wrote: On 04.01.2014 01:29, Sean Dague wrote: On 01/03/2014 04:17 PM, Doug Hellmann wrote: [...] That's what made me think of the solution. But isn't setuptools in fact telling us that somehow the versions of things we expected to have installed are no longer installed and so something *is* broken (even if the versions of the installed libraries work together). It actually tells us that a human, somewhere, decided that their software did not work with some combination of other software, and that we are no longer able to correct their mistaken assumptions. [...] But sometimes humans are not wrong. For example, no released TaskFlow version really works with SQLAlchemy = 0.8 -- that was fixed only recently (https://review.openstack.org/#/c/62661/). I consider requirements to be part of the code, so if they are too strict (or too broad), that must be fixed, in a usual opensource way: via filing bug report, suggesting patch and so on. Requirements should reflect reality, especially for libraries that are intended to be useful outside of OpenStack, too. For example, current TaskFlow requirement for SQLAlchemy is too strict, so we'll fix that, and release new version with that fix. Well part of the problem is because of it being a dependency of a dependency, our global requirements process couldn't explore whether or not it functioned in a real environment. Which brings us back to the idea of making this a project that works in the integrated gate. It's also kind of problematic that apparently the introduction of taskflow actually caused a regression from Havana (which the distros managed to make work with sqla 0.8 even though it wasn't in global requirements, but this apparently would have broken). And the next question is when is a sqla 0.9 compatible version going to be out there? Because we can't even explore allowing openstack to use sqla 0.9 until taskflow does, again because it's a blocking requirement. It's exactly these kind of lock step problems that we avoid with the rest of openstack by making it an integrated gate with global requirements sync. But that we can't really handle with the library model. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 01/03/2014 06:14 PM, Joshua Harlow wrote: Since the library model is what most everyone else uses outside of openstack (I assume?) what can we do to get there so that this model works as well? Expanding dependencies recursively seems like it could help? This could then detect transitive dependency issues (and doesn't seem so hard to do). We actually talked about having pip be able to help us here with a --requirements-override piece of function. dstufft seemed positive on the concept. I like the idea of 'gate on all the things' (that I've heard come up before) but I don't know if its possible? If we hooked into the pypi upload 'stream' it would seem like we could automatically trigger openstack builds when a known dependency (or dependency of a dependency...) is uploaded (maybe?). That would be pretty neat. In general it really seems like having more libraries, not less is ideal (making us solve this issue whether we want to or not really). As libraries can then be used outside of openstack (taskflow is already being used elsewhere also), libraries create well-defined apis and boundaries between programs (...). I know they also create this dependency hell problem (anvil has been hitting these same issues for a while to). But I think we can figure out a solution that fits both worlds, the world of things we can gate on and the world of things we can't (3rd party libraries that aren't under openstack control). Taskflow is in-between those worlds, so it allows us to explore how to make both of those worlds work. In general I agree however, if you get between OpenStack and SQLA you've now touched the 3rd rail. Because we have deep experience about how bad the compatibility between versions can be, and we can't be beholden to another project about our SQLA upgrade timeframe. So I think that generalities aside, if are a library, and use SQLA, we probably need to really think about putting it in the integrated gate. Because otherwise what we are saying is taskflow is completely dictating the SQLA version in the Icehouse release of OpenStack. And that's the wrong place for that decision to be. If taskflow worked with a storage callback mechanism, and got a storage interface from the program that was calling it, then things would be much different. But because it's going straight to the database and managing tables directly, through a known unstable library, that OpenStack itself needs some control over, it's definitely a different case. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
So another idea that was talked about on IRC. Taskflow exposes entrypoints for these storage backends (like your storage callback/interface idea). It currently provides 3 such 'default' backends [sqlalchemy, file/dir based, in-memory -- mainly for testing]. A 4th one is in progress for icehouse (zookeeper based). These backends are used to store intermediary 'flow/task' state (allowing the taskflow engine to resume if an app crashes/stopped while doing stuff). A couple ideas about this, since its already pluggable. Split the sqlalchemy backend - 'taskflow-sa' repo/new package. For those projects that want to use this backend, they include it (still means 'taskflow-sa' package has a dependency on sqlalchemy of some version). Another idea is to move the sqlalchemy dependency currently in taskflow requirements.txt - taskflow test-requirements.txt and for those projects that want to use the sqlalchemy backend, they include the sqlalchemy version themselves in there requirements.txt (taskflow keeps it in test-requirements.txt so that it can run its unit/integration tests, making sure the backend still works). I'm not really sure which is the best, none seem super-great, but sometimes u just work with what u got :-P On 1/3/14, 3:32 PM, Sean Dague s...@dague.net wrote: On 01/03/2014 06:14 PM, Joshua Harlow wrote: Since the library model is what most everyone else uses outside of openstack (I assume?) what can we do to get there so that this model works as well? Expanding dependencies recursively seems like it could help? This could then detect transitive dependency issues (and doesn't seem so hard to do). We actually talked about having pip be able to help us here with a --requirements-override piece of function. dstufft seemed positive on the concept. I like the idea of 'gate on all the things' (that I've heard come up before) but I don't know if its possible? If we hooked into the pypi upload 'stream' it would seem like we could automatically trigger openstack builds when a known dependency (or dependency of a dependency...) is uploaded (maybe?). That would be pretty neat. In general it really seems like having more libraries, not less is ideal (making us solve this issue whether we want to or not really). As libraries can then be used outside of openstack (taskflow is already being used elsewhere also), libraries create well-defined apis and boundaries between programs (...). I know they also create this dependency hell problem (anvil has been hitting these same issues for a while to). But I think we can figure out a solution that fits both worlds, the world of things we can gate on and the world of things we can't (3rd party libraries that aren't under openstack control). Taskflow is in-between those worlds, so it allows us to explore how to make both of those worlds work. In general I agree however, if you get between OpenStack and SQLA you've now touched the 3rd rail. Because we have deep experience about how bad the compatibility between versions can be, and we can't be beholden to another project about our SQLA upgrade timeframe. So I think that generalities aside, if are a library, and use SQLA, we probably need to really think about putting it in the integrated gate. Because otherwise what we are saying is taskflow is completely dictating the SQLA version in the Icehouse release of OpenStack. And that's the wrong place for that decision to be. If taskflow worked with a storage callback mechanism, and got a storage interface from the program that was calling it, then things would be much different. But because it's going straight to the database and managing tables directly, through a known unstable library, that OpenStack itself needs some control over, it's definitely a different case. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
