On Fri, Nov 13, 2015 at 10:17 AM, Alexander Kostrikov <akostri...@mirantis.com> wrote: > Hi, Alex! > Good to know someone is doing that. > > One question about Your initiative: > Are You going to reimplement all MOS HA features? > Seems first implementation of 'upstream on Fuel' will be HA-less. >
My first pass included the MOS HA features as we will still be leveraging the MOS packages for these items (mysql/haproxy). My PoC allows for a different set of OpenStack packages to be set as the preferred package to use over the current MOS packages which are the Fuel default. I think eventually we'll need to reevaluate how we can split out the MOS packages for these features and allow a user to select which package set to use and still have it 'work'. Thanks, -Alex > > On Thu, Nov 12, 2015 at 1:51 AM, Alex Schultz <aschu...@mirantis.com> wrote: >> >> On Tue, Nov 10, 2015 at 11:10 AM, Vladimir Kuklin <vkuk...@mirantis.com> >> wrote: >> > Alex >> > >> > Thanks for your very detailed answer - it clarified things a bit. So, if >> > you >> > would allow me to rephrase it - you are actually conducting a research >> > on >> > what is the actual gap between our downstream/fork and upstream >> > UCA/puppet-openstack. This seems to be a very promising initiative. Are >> > you >> > going to come up with some external-user readable report soon? >> > >> > Also, regarding multiple distros support. I think, we need to come up >> > with >> > an approach of making 'release manager' piece of Nailgun data driven and >> > just allow a user to run any distribution he or she wants. Just register >> > a >> > set of repos with packages and run it. Actually, we already have it - we >> > need to make base-image generation for RPM more flexible and any RPM/DEB >> > based distro should work ok with it. >> > >> > The remaining piece is to actually support distro-specific things, e.g. >> > CentOS/RHEL networking configuration, e.g. l23 network stored config >> > puppet >> > providers. But this is a distro-supporter/community burden. >> > >> > >> >> Yes I hope to have something together by the end of the week. I've >> managed to get a controller and compute/cinder nodes up (and passing >> basic OSTF tests) with what appears to be some minor adjustments to >> the fuel-library code. The one thing that gets dropped because of >> lack of upstream support is nova floating network ranges. There's a >> pending review that'll get that back in but I also don't know how >> important it would be to support for this type of configuration. >> Another issue is the upstream murano module is still a work in >> progress so that won't work right now either. Hopefully that'll get >> sorted out in time for the official release of the liberty puppet >> modules. >> >> As I've been working through this, I've noticed that it would be >> possible to use fuel-plugins to only apply UCA packages to specific >> nodes via a plugin role. An interesting follow on to this effort would >> be to use MOS packages for controllers and UCA for Compute or vice >> versa. But that should probably be more an academic exercise rather >> than production one. >> >> -Alex >> >> > >> > On Tue, Nov 10, 2015 at 6:25 PM, Alex Schultz <aschu...@mirantis.com> >> > wrote: >> >> >> >> Hey Vladimir, >> >> >> >> On Tue, Nov 10, 2015 at 5:56 AM, Vladimir Kuklin <vkuk...@mirantis.com> >> >> wrote: >> >> > Alex >> >> > >> >> > That's great to hear that. But be aware - making all of the >> >> > components >> >> > work >> >> > exactly the way they work within MOS is actually identical to >> >> > upstreaming >> >> > MOS. We are using some components of different versions to satisfy >> >> > many >> >> > requirements for our Reference Architecutre implementation. It will >> >> > not >> >> > be >> >> > so easy to base them upon existing 3rd party distributions. For >> >> > example, >> >> > `read timeout` for SQL requests is crucial for our HA as it handles >> >> > cases >> >> > when an SQL node dies while serving the request. And you will find >> >> > myriads >> >> > of them. And as we do not control things in upstream, we will always >> >> > have >> >> > some downstream divergence. >> >> > >> >> >> >> Yes, I'm aware that it'll be an effort to make it work identically to >> >> MOS. Currently that's not my goal. My goal is to get it working at >> >> all and be able to document the deficiencies when using upstream >> >> packages/modules vs MOS provided ones. Once we have documented these >> >> differences we will be able to make decisions as to what efforts >> >> should be made if we choose to address the differences. The read >> >> timeout thing seems to be an issue with what mysql python driver is >> >> used so that could easily be configurable based on a packages or a >> >> configuration option. >> >> >> >> > I guess, the optimal solution is to figure out the actual divergence >> >> > between >> >> > upstream and downstream and try to push things upstream as hard as we >> >> > can, >> >> > while retaining overrides for some packages and manifests on top of >> >> > upstream >> >> > versions. Do not get me wrong, but it seems there is exactly 0 (zero) >> >> > ways >> >> > you can get Fuel working with upstream packages unless they support >> >> > exactly >> >> > the same feature set and fix the same bugs in various components that >> >> > Fuel >> >> > expect them to support or fix. By 'working' I mean passing the same >> >> > set >> >> > of >> >> > at least smoke and destructive tests, let alone passing scale tests. >> >> > >> >> >> >> So I think this is where we are currently backwards in the way we're >> >> doings. As we hope to push Fuel as a community project, we need to be >> >> more open to supporting the distributions versions of these packages >> >> as being supported. If we want to continue with specific versions of >> >> things we need to be able to modularize them and apply them to a >> >> downstream version of Fuel that we can promote with the MOS package >> >> set. I agree that right now it's highly unlikely that one would be >> >> able to use Fuel without any MOS packages. Because I don't think it's >> >> possible right now, what I'm attempting to do is be able to deploy an >> >> alternate OpenStack package set but not all of the other pieces as >> >> they relate to our reference architecture. That seems to be a more >> >> achievable goal right now and allows us to document the gaps where >> >> Fuel/MOS are a head (or behind depending on your views) with upstream. >> >> >> >> > Simon >> >> > >> >> > AFAIK, this patch is really simple and can be easily applied to any >> >> > version >> >> > of HAProxy. So it is not too hard handle it while it provides >> >> > sufficient >> >> > benefits to our deployment engineers. If you have any better >> >> > solution, >> >> > please feel free to share the code with us. >> >> > >> >> >> >> Unfortunately the haproxy fork that we're currently running with also >> >> includes our forked version of the puppet haproxy module. I'm not sure >> >> the includes configs patch is really worth carrying the additional >> >> work/forks but I think that is a conversation for another day. >> >> Personally I think it would better if fuel-library supported the >> >> currently available upstream versions of haproxy & haproxy puppet >> >> module and if we want to continue using our copies of haproxy & puppet >> >> haproxy module that we turn that into a downstream that is applied for >> >> the MOS specific version of Fuel. That's the advantage of the >> >> Puppetfile we've been working on. We can change out the modules with >> >> our own version downstream. >> >> >> >> Just to add some additional updates issues that I've run into. >> >> >> >> - nova-vncproxy package names (nova-consoleproxy vs nova-novncproxy) >> >> but that's easily solved with the proposed os_package_type fact. >> >> - heat-docker package does not exist in UCA. Haven't looked at what >> >> exactly is in this package yet. >> >> - horizon UCA package runs into >> >> https://bugs.launchpad.net/cloud-archive/+bug/1479340 (we solved this >> >> for MOS but it's still broken in UCA) >> >> - neutron UCA package seems to suffer from >> >> https://bugs.launchpad.net/mos/+bug/1510844 >> >> - Need to solve for the url issues with the openstack command that are >> >> showing up in the CI as I just hit it with my deployment. >> >> >> >> Thanks, >> >> -Alex >> >> >> >> > On Tue, Nov 10, 2015 at 12:54 PM, Stanislaw Bogatkin >> >> > <sbogat...@mirantis.com> wrote: >> >> >> >> >> >> Hi Simon, >> >> >> >> >> >> using 'include' in HAProxy is damn conveniently, I don't think it >> >> >> should >> >> >> die. There is just one way I know to use haproxy with several >> >> >> different >> >> >> conf's - to construct looooooooooong command line with all of them - >> >> >> and >> >> >> it's really inconvenient. >> >> >> >> >> >> On Tue, Nov 10, 2015 at 12:20 PM, Simon Pasquier >> >> >> <spasqu...@mirantis.com> >> >> >> wrote: >> >> >>> >> >> >>> Hello Alex! >> >> >>> >> >> >>> On Mon, Nov 9, 2015 at 9:29 PM, Alex Schultz >> >> >>> <aschu...@mirantis.com> >> >> >>> wrote: >> >> >>>> >> >> >>>> Hey folks, >> >> >>>> >> >> >>>> I'm testing[0] out flipping our current method of consuming >> >> >>>> upstream >> >> >>>> puppet modules from using pinned versions hosted on fuel-infra to >> >> >>>> be >> >> >>>> able to use the ones directly from upstream (master). This work >> >> >>>> is >> >> >>>> primarily to be closer aligned with the other OpenStack projects >> >> >>>> as >> >> >>>> well as switching the current way we manage Fuel into a downstream >> >> >>>> of >> >> >>>> the upstream community version. As part of this work we have also >> >> >>>> been >> >> >>>> working towards improving the upstream modules support different >> >> >>>> package sets. Specifically running Debian packages on >> >> >>>> Ubuntu[1][2]. >> >> >>>> This work is the start of being able to allow a user of Fuel to be >> >> >>>> able to specify a specific package set and having it be able to >> >> >>>> work. >> >> >>>> If we can properly split out the puppet modules and package >> >> >>>> dependencies this will make Fuel a more flexible deployment engine >> >> >>>> as >> >> >>>> I believe we would be better positioned to support multiple >> >> >>>> versions >> >> >>>> of OpenStack for a given Fuel release. >> >> >>>> >> >> >>>> I'm currently working to get a PoC of Fuel consuming upstream >> >> >>>> puppet >> >> >>>> modules and the UCA packages together and documenting all of the >> >> >>>> issues so that we can address them. So far I have been able to >> >> >>>> deploy >> >> >>>> the upstream modules via a custom ISO using the MOS package set >> >> >>>> and >> >> >>>> it >> >> >>>> works locally. Unfortunately the CI seems to be hitting some >> >> >>>> issues >> >> >>>> that I think might be related to recently merged keystone changes >> >> >>>> but >> >> >>>> I did not run into the same problem when running a manual >> >> >>>> deployment. >> >> >>>> As I work through this PoC, I'm also attempting to develop a small >> >> >>>> plugin that could be used to capture the work arounds to the >> >> >>>> deployment process. I've run into a few issues so far as I work to >> >> >>>> switch out the package sets. >> >> >>>> >> >> >>>> For the sake of providing additional visibility into this work, >> >> >>>> here >> >> >>>> are the issues that I've hit so far. >> >> >>>> >> >> >>>> The first issue I ran across is that currently the MOS >> >> >>>> repositories >> >> >>>> contain packages for both OpenStack and other system dependencies >> >> >>>> for >> >> >>>> creating our HA implementation. This is problematic when we want >> >> >>>> to >> >> >>>> switch out the OpenStack packages but still want the MOS packages >> >> >>>> for >> >> >>>> our HA items. I'm working around this by adding the UCA >> >> >>>> repository >> >> >>>> as >> >> >>>> a higher priorities for deployments. As such I've run into an >> >> >>>> issue >> >> >>>> with the haproxy package that MOS provides vs the upstream Ubuntu >> >> >>>> package. To get around this, I've pinned the MOS version for now >> >> >>>> until I can circle back around and figure out if the difference is >> >> >>>> a >> >> >>>> config or functionality issue. >> >> >>> >> >> >>> >> >> >>> I know at least for one difference between the MOS and Ubuntu >> >> >>> versions: >> >> >>> HAProxy from MOS has a patch to support the "include" configuration >> >> >>> parameter. >> >> >>> This is unrelated to your mail but I think this fork should die >> >> >>> since >> >> >>> it >> >> >>> will never be accepted upstream and there are other ways to address >> >> >>> the use >> >> >>> case [0]. >> >> >>> >> >> >>> HTH, >> >> >>> Simon >> >> >>> >> >> >>> [0] http://marc.info/?l=haproxy&m=130817444025140&w=2 >> >> >>> >> >> >>>> >> >> >>>> >> >> >>>> The second issue that I have run across is that we are appending >> >> >>>> read_timeout=60 to our mysql connection strings for our >> >> >>>> configurations. This seems to be unsupported by the libraries and >> >> >>>> OpenStack components provided by the UCA package set. I'm not >> >> >>>> sure >> >> >>>> how the priority of python-pymsql vs python-mysqldb is resolved as >> >> >>>> I >> >> >>>> had both packages installed but it continued to fail on >> >> >>>> read_timeout >> >> >>>> being in the connection string. For now I've updated the >> >> >>>> fuel-library >> >> >>>> code to remove this item from the connection strings and will be >> >> >>>> circling back around to figure out the correct 'fix' for this >> >> >>>> issue. >> >> >>>> >> >> >>>> I'm hoping to be able to have a working ISO, plugin and a set of >> >> >>>> instructions that can be used to deploy a basic cloud using Fuel >> >> >>>> by >> >> >>>> the end of this week. >> >> >>>> >> >> >>>> Thanks, >> >> >>>> -Alex >> >> >>>> >> >> >>>> [0] https://review.openstack.org/#/c/240325/ >> >> >>>> [1] https://review.openstack.org/#/c/241615/ >> >> >>>> [2] https://review.openstack.org/#/c/241741/ >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> __________________________________________________________________________ >> >> >>>> OpenStack Development Mailing List (not for usage questions) >> >> >>>> Unsubscribe: >> >> >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> __________________________________________________________________________ >> >> >>> OpenStack Development Mailing List (not for usage questions) >> >> >>> Unsubscribe: >> >> >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> __________________________________________________________________________ >> >> >> OpenStack Development Mailing List (not for usage questions) >> >> >> Unsubscribe: >> >> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > Yours Faithfully, >> >> > Vladimir Kuklin, >> >> > Fuel Library Tech Lead, >> >> > Mirantis, Inc. >> >> > +7 (495) 640-49-04 >> >> > +7 (926) 702-39-68 >> >> > Skype kuklinvv >> >> > 35bk3, Vorontsovskaya Str. >> >> > Moscow, Russia, >> >> > www.mirantis.com >> >> > www.mirantis.ru >> >> > vkuk...@mirantis.com >> >> > >> >> > >> >> > >> >> > __________________________________________________________________________ >> >> > OpenStack Development Mailing List (not for usage questions) >> >> > Unsubscribe: >> >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > >> >> >> >> >> >> __________________________________________________________________________ >> >> OpenStack Development Mailing List (not for usage questions) >> >> Unsubscribe: >> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > >> > >> > >> > -- >> > Yours Faithfully, >> > Vladimir Kuklin, >> > Fuel Library Tech Lead, >> > Mirantis, Inc. >> > +7 (495) 640-49-04 >> > +7 (926) 702-39-68 >> > Skype kuklinvv >> > 35bk3, Vorontsovskaya Str. >> > Moscow, Russia, >> > www.mirantis.com >> > www.mirantis.ru >> > vkuk...@mirantis.com >> > >> > >> > __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > -- > > Kind Regards, > > Alexandr Kostrikov, > > > Mirantis, Inc. > > 35b/3, Vorontsovskaya St., 109147, Moscow, Russia > > > Tel.: +7 (495) 640-49-04 > Tel.: +7 (925) 716-64-52 > > Skype: akostrikov_mirantis > > E-mail: akostri...@mirantis.com > > www.mirantis.com > www.mirantis.ru > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev