Re: [openstack-dev] Cookiecutter repo for ease in making new projects
On Thu, Sep 12, 2013 at 10:08 PM, Monty Taylor wrote: > And boom, you'll have a directory all set up with your new project Awesome. I tried it and ran into a couple of small issues. I don't see a launchpad yet, so I'm not sure where to report bugs. Something is stripping all the new lines at EOF, so flake8 fails. Also, this is annoying because vim's default is to add one when it doesn't exist. Also, foo/test/__init__.py needs to call super(TestCase, self).setUp() otherwise you get an error that is something like this when you run tox: ValueError: TestCase.setUp was not called. Have you upcalled all the way up the hierarchy from your setUp? e.g. Call super(TestFoo, self).setUp() from your setUp(). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Code review study
On Tue, Aug 20, 2013 at 8:02 AM, Mark McLoughlin wrote: > On Tue, 2013-08-20 at 11:26 +0100, Mark McLoughlin wrote: > > The full study is here: > > > > > http://support.smartbear.com/resources/cc/book/code-review-cisco-case-study.pdf > I can't find the data they based their numbers on, nor their definition for a line of code, so I feel like I have to take that study with a grain of salt. > We should not enforce rules like this blindly. I agree with the sentiment: less complex patches are easier to review. Reviewers should use their judgement and push back on complex patches to be split up. Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Code review study
On Wed, Aug 14, 2013 at 7:12 PM, Robert Collins wrote: > Note specifically the citation of 200-400 lines as the knee of the review > effectiveness curve: that's lower than I thought - I thought 200 was > clearly fine - but no. > This is really interesting. I wish they would have explicitly defined "lines of code." Is that "git show |wc -l"? Just the new lines which were added? The sum of the lines changed, removed and added? You can get vastly different numbers depending on how you count it. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] General Question about CentOS
On Fri, Aug 16, 2013 at 2:51 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) wrote: > Is OpenStack supported on CentOS running Python 2.6? > Oh, I forgot to mention, keystone's py2.6 support seems to currently be broken because of this bug: https://bugs.launchpad.net/keystone/+bug/1213284/ Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] General Question about CentOS
We're doing continuous deployment on Oracle Linux 6.3, using a RHEL kernel making it almost identical to CentOS 6.3. We're using keystone, glance, neutron and nova. The neutron metadata service doesn't work, nor do overlapping ips: both need network namespace support in the kernel which wasn't introduced until 3.0.0. We use the nova metadata service with an iptables hack. We disabled overlapping ips. There are also some libvirt issues. We're running the stock libvirt 0.9.10, which, for reasons I can't recall, required running the nova compute process as root. Also, there were some issues relating to libvirt's host-passthrough, which were actually back ported in to the libvirt 0.9.10 by redhat, but the way nova does the check it doesn't see this backported capability (see nova's commit 2a236d24a). Jay On Fri, Aug 16, 2013 at 2:51 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) wrote: > Is OpenStack supported on CentOS running Python 2.6? > > ** ** > > Thanks, > > ** ** > > Mark > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Why is "network" and "subnet" modeled separately?
On Wed, Aug 14, 2013 at 1:41 PM, Jay Pipes wrote: > On 08/14/2013 04:31 PM, Jay Buffington wrote: > >> A common use case would be that one tenant creates a network with a >> > 10.1.1.0/24 <http://10.1.1.0/24> subnet. Another tenant wants to use >> that same network, so they create a new network and can also create the >> 10.1.1.0/24 <http://10.1.1.0/24> subnet. >> > I made an error In that second sentence. "Another tenant wants to use that same network" should read "Another tenant wants to use that same subnet" > Why not just call them all networks? A network is a container for subnets. Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Why is "network" and "subnet" modeled separately?
network is layer 2 and subnet is layer 3. A common use case would be that one tenant creates a network with a 10.1.1.0/24 subnet. Another tenant wants to use that same network, so they create a new network and can also create the 10.1.1.0/24 subnet. On Wed, Aug 14, 2013 at 1:12 PM, Lorin Hochstein wrote: > Here's a neutron implementation question: why does neutron model "network" > and "subnet" as separate entities? > > Or, to ask another way, are there are any practical use cases where you > would *not* have a one-to-one relationship between neutron networks and > neutron subnets in an OpenStack deployment? (e.g. one neutron network > associated with multiple neutron subnets, or one neutron network associated > with zero neutron subnets)? > > Lorin > -- > Lorin Hochstein > Lead Architect - Cloud Services > Nimbis Services, Inc. > www.nimbisservices.com > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages
On Tue, Aug 6, 2013 at 12:00 PM, Monty Taylor wrote: > On 08/06/2013 02:44 PM, Mate Lakat wrote: > > I would say, use a separated virtual environment in devstack - without > > the --system-site-packages switch, of course, and set it up as a user. > > Install the packages that are needed in order to be able to pip install > > them (like libxslt-dev). It's a development environment. I think my > > email is equivalent to a +1 to (Monty's change + virtualenv). > > YUP - except we can't do it without system-site-packages, because of > nova and libvirt. We should still be ok though with a single venv, as > long as all of the installs into the venv use the -U flag to pip, which > will override the system env with teh venv one. > I'm building my virtualenv without system site-packages. It's super lame, but my nova rpm has a dep on a specific version of libvirt-python and then includes these symlinks: ln -s /usr/lib64/python2.6/site-packages/libvirt.py $(VENV)/lib/python2.6/site-packages/ ln -s /usr/lib64/python2.6/site-packages/libvirt_qemu.py $(VENV)/lib/python2.6/site-packages/ ln -s /usr/lib64/python2.6/site-packages/libvirtmod.so $(VENV)/lib/python2.6/site-packages/ ln -s /usr/lib64/python2.6/site-packages/libvirtmod_qemu.so $(VENV)/lib/python2.6/site-packages/ Why isn't libvirt-python on pypi? AFAICT, nothing is stopping us from uploading it. Maybe we should just stick it on there and this issue will be resolved once and for all. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Python overhead for rootwrap
> Personally I'm of the opinion that from an architectural POV, use of > either rootwrap or sudo is a bad solution, so arguing about which is > better is really missing the bigger picture. In Linux, there has been > a move away from use of sudo or similar approaches, towards the idea > of having privileged separated services. So if you wanted todo stuff > related to storage, you'd have some small daemon running privilegd, > which exposed APIs over DBus, which the non-privileged thing would > call to make storage changes. Operations exposed by the service would > have access control configured via something like PolicyKit, and/or > SELinux/AppArmour. > The more I think about this problem the more I'm convinced that rootwrap simply exists to work around a fundamental design flaw in most (all?) of the components: a single, threaded, process with poor job/workflow mnemonics. If the architecture was such that every operation was a task which could be carried out by any process (even a process running on a different host) and each process publishes job types that it can do, then you could do proper isolation for just those processes that need to do privileged tasks. https://wiki.openstack.org/wiki/TaskFlow looks to be headed in the right direction, but I worry it's got a very long road ahead. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages
On Tue, Aug 6, 2013 at 8:35 AM, Joshua Harlow wrote: > I think jay your usage also was before anvil started to build all the > *missing* dependencies automatically (something u inspired me to get going > in the first place) so hopefully said updates to rhel.yaml are only now > needed for exceptions and not the common path :) > +1 I'm glad to hear that. I don't think any of the problems I saw with Anvil were intractable, but I do think packaging up the venv is much less error prone. My major complaints are issues related to yum. First off, yum can't resolve a version range (see my message to the yum list: http://lists.baseurl.org/pipermail/yum/2013-February/023922.html) which seems like a pretty basic task for a package manager. Also, Yum seems to do dependency resolution in a single pass. So yum can't provide an install solution for cases like this even when a valid one exists: package foo depends on pkgdep <= 2.0 package bar depends on pkgdep >= 1.0, <= 2.0 pkgdep 2.1 and 2.0 are available for install "sudo yum install foo" when none of those three packages are installed results in an error. Yum tries to install pkgdep 2.1 and then gives up when it sees foo doesn't work with pkgdep 2.1 rather than doing a second pass at building a dependency graph and finding that pkgdep 2.0 would satisfy all dependencies. Cases like these come up regularly with the 100+ python packages that openstack depends on (in requirements.txt) when building packages continuously. I came to the conclusion that yum is bad software and I should avoid it globbing up most everything into one big rpm. Three months into making the switch I'm happy with that decision. Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages
On Tue, Aug 6, 2013 at 4:36 AM, Ian Wienand wrote: > On Mon, Aug 05, 2013 at 03:37:24PM -0700, Jay Buffington wrote: > > I used Anvil for the first three months, but it required constant > > updating of dependency versions and it didn't support quantum. > > What do you mean by "updating" here? The rpm packages being updated > causing a lot of churn, or some manual intervention required on your > behalf? > Updating Anvil's conf/distros/rhel.yaml ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages
I'd like to open source them. They are dependent on some other internal tools, so I they wouldn't work out of the box, but I'll see if I can rip them apart and then get corporate sign off on open sourcing them. On Mon, Aug 5, 2013 at 4:02 PM, Joshua Harlow wrote: > This does sound like a neat approach, a hybrid if u will. Might be > something to try :-) > > Sometimes I wish package managers were better, especially with regard to > 'complex' (not really that complex in reality) dependencies. > > Any possibility of opening up said makefiles?? Be interesting to look at > :-) > > From: Jay Buffington > Reply-To: OpenStack Development Mailing List < > openstack-dev@lists.openstack.org> > Date: Monday, August 5, 2013 3:37 PM > To: OpenStack Development Mailing List > Subject: Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs > distro packages > > I've been doing continuous deployment via rpms on RHEL 6u3 of glance, > keystone, > neutron and nova for about six months now. I used Anvil for the first > three > months, but it required constant updating of dependency versions and it > didn't > support quantum. Also, yum is terrible at managing dependency ranges, > which > openstack components often have (e.g. SQLAlchemy>=0.7.8,<=0.7.9). I'm not > convinced DNF[1] will be much better. > > I found it easier to just build a virtualenv and then put all those > files in an > rpm. Then make non-python dependencies (like libxslt) dependencies of > that rpm. > Since we did that the number of daily packaging headaches have dropped > significantly. We have a bunch of Makefiles and jenkins jobs that drive > this > on every commit to components we care about. > > I was unaware of RDO (or it hadn't been released) when I switched our > builds > over. I'd take a serious look at it, but right now I'm happy with our > solution. > > [1] http://fedoraproject.org/wiki/Features/DNF > > > On Mon, Aug 5, 2013 at 10:03 AM, Dean Troyer wrote: > >> [Moving a discussion from https://review.openstack.org/40019 to the ML >> to get a wider audience] >> >> We've been around this block more than once so let's get it all >> documented in one place and see where to go next. Skip down to >> # for more actual discussion... >> >> Given: >> * OpenStack now has an official list of Python package versions >> required in https://review.openstack.org/p/openstack/requirements >> * This list is rarely completely available in any packaged Linux distro >> * Developers like new packages that fix their immediate problem >> * Packagers dislike the treadmill of constantly upgrading packages for >> many reasons (stability, effort, etc) >> * Running OpenStack on certain long-term-stability distros (RHEL6) is >> seriously a challenge due to the number of out-of-date components, >> specifically here many of the python-* packages. >> * Fedora and RHEL6 have a nasty configuration of telling pip to >> install packages into the same location as RPM-installed packages >> setting up hard-to-diagnose problems and irritating sysadmins >> everywhere. FTR, Debian/Ubuntu configure pip to install into >> /usr/local and put '/usr/local/lib/python2.7/dist-packages' ahead of >> '/usr/lib/python2.7/dist-packages' in sys.path. >> * Replacing setuptools on RHEL6 illustrated another issue: removing >> python-setuptools took with it a number of other packages that >> depended on it. >> * OpenStack devs are not in the packaging business. This has been >> decided [citation required?]. Fortunately those in the packaging >> business do contribute to OpenStack (THANKS!) and do make a huge >> effort to keep up with things like the Ububntu cloud Archive and Red >> Hat's RDO. >> >> The last week or so of attempting to install Python prereqs from >> requirements.txt and installing pip 1.4 to support that rather than >> re-invent the wheel and all of the fallout from that illustrates >> clearly that neither approach is going to solve our problem. >> >> Summary of the discussion in the review (paraphrased, read the review >> to see where I got it wrong): >> >> * packages are evil: we develop and gate based on what is available in >> requirements.txt and a non-zero number of those are only in PyPI >> * Anvil solved this already: resolves requirements into the RPM >> package list and packages anything required from PyPI >> * ongoing worries about pip and apt/rpm overwriting each other as >> additional things are installed >> * packages
Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages
I've been doing continuous deployment via rpms on RHEL 6u3 of glance, keystone, neutron and nova for about six months now. I used Anvil for the first three months, but it required constant updating of dependency versions and it didn't support quantum. Also, yum is terrible at managing dependency ranges, which openstack components often have (e.g. SQLAlchemy>=0.7.8,<=0.7.9). I'm not convinced DNF[1] will be much better. I found it easier to just build a virtualenv and then put all those files in an rpm. Then make non-python dependencies (like libxslt) dependencies of that rpm. Since we did that the number of daily packaging headaches have dropped significantly. We have a bunch of Makefiles and jenkins jobs that drive this on every commit to components we care about. I was unaware of RDO (or it hadn't been released) when I switched our builds over. I'd take a serious look at it, but right now I'm happy with our solution. [1] http://fedoraproject.org/wiki/Features/DNF On Mon, Aug 5, 2013 at 10:03 AM, Dean Troyer wrote: > [Moving a discussion from https://review.openstack.org/40019 to the ML > to get a wider audience] > > We've been around this block more than once so let's get it all > documented in one place and see where to go next. Skip down to > # for more actual discussion... > > Given: > * OpenStack now has an official list of Python package versions > required in https://review.openstack.org/p/openstack/requirements > * This list is rarely completely available in any packaged Linux distro > * Developers like new packages that fix their immediate problem > * Packagers dislike the treadmill of constantly upgrading packages for > many reasons (stability, effort, etc) > * Running OpenStack on certain long-term-stability distros (RHEL6) is > seriously a challenge due to the number of out-of-date components, > specifically here many of the python-* packages. > * Fedora and RHEL6 have a nasty configuration of telling pip to > install packages into the same location as RPM-installed packages > setting up hard-to-diagnose problems and irritating sysadmins > everywhere. FTR, Debian/Ubuntu configure pip to install into > /usr/local and put '/usr/local/lib/python2.7/dist-packages' ahead of > '/usr/lib/python2.7/dist-packages' in sys.path. > * Replacing setuptools on RHEL6 illustrated another issue: removing > python-setuptools took with it a number of other packages that > depended on it. > * OpenStack devs are not in the packaging business. This has been > decided [citation required?]. Fortunately those in the packaging > business do contribute to OpenStack (THANKS!) and do make a huge > effort to keep up with things like the Ububntu cloud Archive and Red > Hat's RDO. > > The last week or so of attempting to install Python prereqs from > requirements.txt and installing pip 1.4 to support that rather than > re-invent the wheel and all of the fallout from that illustrates > clearly that neither approach is going to solve our problem. > > Summary of the discussion in the review (paraphrased, read the review > to see where I got it wrong): > > * packages are evil: we develop and gate based on what is available in > requirements.txt and a non-zero number of those are only in PyPI > * Anvil solved this already: resolves requirements into the RPM > package list and packages anything required from PyPI > * ongoing worries about pip and apt/rpm overwriting each other as > additional things are installed > * packages are necessary: > > # > > My specific responses: > > * proposals to use a tool to automatically decide between package and > PyPI (harlowja, sdague): this works well on the surface, but anything > that does not take in to account the dependencies in these packages > going BOTH ways is going to fail. For example: on RHEL6 setuptools is > 0.6.10, we want 0.9.8 (the merged release w/distribute). Removing > python-setuptools will also remove python-nose, numpy and other > packages depending on what is installed. So fine, those can be > re-installed with pip. But a) we don't want to rebuild numpy (just > bear with me here), and b) the packaged python-nose 0.10.4 meets the > version requirement in requirements.txt so the package will be > re-installed, bringing with it python-setuptools 0.6.10 overwriting > the pip installation of 0.9.8. > * separate proposal to build meta-packages (iwienand, > https://review.openstack.org/39862): this would be an effective > work-around for the problems on Fedora and RHEL6 and protect against > the re-install problem, but this is solving a distro packaging problem > in an upstream testing tool. We would also need to manage these > individually and not in a single large package, or at least split > python-crypto and python-lxml out of the proposed package as they may > not be installed in all configurations. > * about-face on all-packages to all-PyPI (mtaylor): I'm still on the > use packages where possible side but DevStack specificall
Re: [openstack-dev] Python overhead for rootwrap
I haven't closely looked at rootwrap, but it seems to me that you could use the rootwrap config files to generate a gigantic sudoers config file which would not necessarily be human readable. That would have the flexibility and maintainability of rootwrap with the speed and audibility sudo. On Thu, Jul 25, 2013 at 1:21 PM, Joe Gordon wrote: > Hi All, > > We have recently hit some performance issues with nova-network. It turns > out the root cause of this was we do roughly 20 rootwrapped shell commands, > many inside of global locks. (https://bugs.launchpad.net/oslo/+bug/1199433 > ) > > It turns out starting python itself, has a fairly significant overhead > when compared to the run time of many of the binary commands we execute. > > For example: > > $ time python -c "print 'test'" > test > > real 0m0.023s > user 0m0.016s > sys 0m0.004s > > > $ time ip a > <...> > > real 0m0.003s > user 0m0.000s > sys 0m0.000s > > > While we have removed the extra overhead of using entry points, we are now > hitting the overhead of just shelling out to python. > > > While there are many possible ways to reduce this issue, such as reducing > the number of rootwrapped calls and making locks finer grain, I think its > worth exploring alternates to the current rootwrap model. > > Any ideas? I am sending this email out to get the discussion started. > > > best, > Joe Gordon > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [quantum] Deadlock on quantum port-create
I'm moving a thread we had with some vmware guys to this list to make it public. We had a problem with quantum deadlocking when it got several requests in quick succession. Aaron suggested we set sql_dbpool_enable = True. We did and it seemed to resolve our issue. What are the downsides of turning on sql_dbpool_enable? Should it be on by default? Thanks, Jay >> We are currently experience the following problem in our environment: >> issuing 5 'quantum port-create' commands in parallel effectively deadlocks quantum: >> >> $ for n in $(seq 5); do echo 'quantum --insecure port-create stage-net1'; done | parallel >> An unknown exception occurred. >> Request Failed: internal server error while processing your request. >> An unexpected error occurred in the NVP Plugin:Unable to get logical switches On Jun 21, 2013, at 9:36 AM, Aaron Rosen wrote: > We've encountered this issue as well. I'd try enabling: > # Enable the use of eventlet's db_pool for MySQL. The flags sql_min_pool_size, > # sql_max_pool_size and sql_idle_timeout are relevant only if this is enabled. > > sql_dbpool_enable = True > > in nvp.ini to see if that helps at all. In our internal cloud we removed the > creations of the lports in nvp from the transaction. Salvatore is working on > an async back-end to the plugin that will solve this and improve the plugin > performance. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quantum's new name is….
On Wed, Jun 19, 2013 at 9:18 AM, Qing He wrote: > The rationale behind it would be interesting ... My understanding is that the Quantum corporation (harddrive manufacturer) asserted their trademark rights, so the name had to be changed. See https://lists.launchpad.net/openstack/msg22544.html Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
