Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

I'd like to bring the attention back to this topic: Mark, could you reconsider removing the -2 here? https://review.openstack.org/#/c/93889/ Your reason was: """Until the upstream blueprint (https://blueprints.launchpad.net/oslo/+spec/rootwrap-daemon-mode ) merges in Oslo it does not

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/24/2014 07:23 PM, Yuriy Taraday wrote: On Mon, Mar 24, 2014 at 9:51 PM, Carl Baldwin mailto:c...@ecbaldwin.net>> wrote: Don't discard the first number so quickly. For example, say we use a timeout mechanism for the daemon running inside namespaces to avoid using too much mem

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

I was thinking that we could document the information about sudo and iproute2 patches with the upcoming release. How would I go about doing this? Is there any section in our documentation about OS level tweaks or requirements such as these that could present this information as part of the releas

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Mon, Mar 24, 2014 at 9:51 PM, Carl Baldwin wrote: > Don't discard the first number so quickly. > > For example, say we use a timeout mechanism for the daemon running > inside namespaces to avoid using too much memory with a daemon in > every namespace. That means we'll pay the startup cost re

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Don't discard the first number so quickly. For example, say we use a timeout mechanism for the daemon running inside namespaces to avoid using too much memory with a daemon in every namespace. That means we'll pay the startup cost repeatedly but in a way that amortizes it down. Even if it is rea

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

It's the first call starting the daemon / loading config files, etc?, May be that first sample should be discarded from the mean for all processes (it's an outlier value). On 03/21/2014 05:32 PM, Yuriy Taraday wrote: On Fri, Mar 21, 2014 at 2:01 PM, Thierry Carrez mailto:thie...@openstack

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Fri, Mar 21, 2014 at 2:01 PM, Thierry Carrez wrote: > Yuriy Taraday wrote: > > Benchmark included showed on my machine these numbers (average over 100 > > iterations): > > > > Running 'ip a': > > ip a : 4.565ms > >

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Sean Dague wrote: > Sounds great. One of the things I hope happens with this is a look at > some place rootwrap is used with such an open policy, that it's > completely moot. For instance the nova-cpu policy includes tee & dd with > no arg limitting (which has been that way forever from my look in

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/21/2014 05:42 AM, Thierry Carrez wrote: > Yuriy Taraday wrote: >> On Thu, Mar 20, 2014 at 5:41 PM, Miguel Angel Ajo > > wrote: >>>If this coupled to neutron in a way that it can be accepted for >>> Icehouse (we're killing a performance bug), or that at

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/21/2014 11:01 AM, Thierry Carrez wrote: Yuriy Taraday wrote: Benchmark included showed on my machine these numbers (average over 100 iterations): Running 'ip a': ip a : 4.565ms sudo ip a : 13.744

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/21/2014 10:42 AM, Thierry Carrez wrote: Yuriy Taraday wrote: On Thu, Mar 20, 2014 at 5:41 PM, Miguel Angel Ajo mailto:majop...@redhat.com>> wrote: If this coupled to neutron in a way that it can be accepted for Icehouse (we're killing a performance bug), or that at least it

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Yuriy Taraday wrote: > Benchmark included showed on my machine these numbers (average over 100 > iterations): > > Running 'ip a': > ip a : 4.565ms > sudo ip a : 13.744ms >sudo root

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Yuriy Taraday wrote: > On Thu, Mar 20, 2014 at 5:41 PM, Miguel Angel Ajo > wrote: >>If this coupled to neutron in a way that it can be accepted for >> Icehouse (we're killing a performance bug), or that at least it can >> be y backported, you'd be coveri

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Thu, Mar 20, 2014 at 8:23 PM, Rick Jones wrote: > On 03/20/2014 09:07 AM, Yuriy Taraday wrote: > >> On Thu, Mar 20, 2014 at 7:28 PM, Rick Jones > > wrote: >> Interesting result. Which versions of sudo and ip and with how many >> interfaces on the system? >>

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/20/2014 09:07 AM, Yuriy Taraday wrote: On Thu, Mar 20, 2014 at 7:28 PM, Rick Jones mailto:rick.jon...@hp.com>> wrote: Interesting result. Which versions of sudo and ip and with how many interfaces on the system? Here are the numbers: % sudo -V Sudo version 1.8.6p7 Sudoers policy

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Thu, Mar 20, 2014 at 5:41 PM, Miguel Angel Ajo wrote: > >Wow Yuriy, amazing and fast :-), benchmarks included ;-) > >The daemon solution only adds 4.5ms, good work. I'll add some comments > in a while. > >Recently I talked with another engineer in Red Hat (working > in ovirt/vdsm),

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Thu, Mar 20, 2014 at 7:28 PM, Rick Jones wrote: > On 03/20/2014 05:41 AM, Yuriy Taraday wrote: > >> Benchmark included showed on my machine these numbers (average over 100 >> iterations): >> >> Running 'ip a': >>ip a : 4.565ms >>

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/20/2014 05:41 AM, Yuriy Taraday wrote: On Tue, Mar 18, 2014 at 7:38 PM, Yuriy Taraday mailto:yorik@gmail.com>> wrote: I'm aiming at ~100 new lines of code for daemon. Of course I'll use some batteries included with Python stdlib but they should be safe already. It shoul

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Tue, Mar 11, 2014 at 12:58 AM, Carl Baldwin wrote: > > https://etherpad.openstack.org/p/neutron-agent-exec-performance > I've added info on how we can speedup work with namespaces by setting namespaces by ourselves using setns() without "ip netns exec" overhead. -- Kind regards, Yuriy.

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/20/2014 12:32 PM, Monty Taylor wrote: On 03/20/2014 05:31 AM, Miguel Angel Ajo wrote: On 03/19/2014 10:54 PM, Joe Gordon wrote: On Wed, Mar 19, 2014 at 9:25 AM, Miguel Angel Ajo mailto:majop...@redhat.com>> wrote: An advance on the changes that it's requiring to have a

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Wow Yuriy, amazing and fast :-), benchmarks included ;-) The daemon solution only adds 4.5ms, good work. I'll add some comments in a while. Recently I talked with another engineer in Red Hat (working in ovirt/vdsm), and they have something like this daemon, and they are using BaseMan

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Tue, Mar 18, 2014 at 7:38 PM, Yuriy Taraday wrote: > I'm aiming at ~100 new lines of code for daemon. Of course I'll use some > batteries included with Python stdlib but they should be safe already. > It should be rather easy to audit them. > Here's my take on this: https://review.openstack.o

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/20/2014 05:31 AM, Miguel Angel Ajo wrote: On 03/19/2014 10:54 PM, Joe Gordon wrote: On Wed, Mar 19, 2014 at 9:25 AM, Miguel Angel Ajo mailto:majop...@redhat.com>> wrote: An advance on the changes that it's requiring to have a py->c++ compiled rootwrap as a mitigation POC fo

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/19/2014 10:54 PM, Joe Gordon wrote: On Wed, Mar 19, 2014 at 9:25 AM, Miguel Angel Ajo mailto:majop...@redhat.com>> wrote: An advance on the changes that it's requiring to have a py->c++ compiled rootwrap as a mitigation POC for havana/icehouse. https://github.com/mange

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Wed, Mar 19, 2014 at 9:25 AM, Miguel Angel Ajo wrote: > > > An advance on the changes that it's requiring to have a > py->c++ compiled rootwrap as a mitigation POC for havana/icehouse. > > https://github.com/mangelajo/shedskin.rootwrap/commit/ > e4167a6491dfbc71e2d0f6e28ba93bc8a1dd66c0 > > The

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

An advance on the changes that it's requiring to have a py->c++ compiled rootwrap as a mitigation POC for havana/icehouse. https://github.com/mangelajo/shedskin.rootwrap/commit/e4167a6491dfbc71e2d0f6e28ba93bc8a1dd66c0 The current translation output is included. It looks like doable (almost ki

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Mon, Mar 17, 2014 at 1:01 PM, IWAMOTO Toshihiro wrote: > > I've added a couple of security-related comments (pickle decoding and > token leak) on the etherpad. > Please check. > Hello. Thanks for your input. - We can avoid pickle using xmlrpclib. - Token won't leak because we have direct pipe

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Hello, Thierry. On Mon, Mar 17, 2014 at 6:04 PM, Thierry Carrez wrote: > Note that the whole concept behind rootwrap is to limit the amount of > code that runs with elevated privileges. If you end up running a full > service as root which imports as many libraries as the rest of OpenStack > servi

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Joe Gordon wrote: >> And this is a test with shedskin, I suppose that in more complicated >> dependecy scenarios it should perform better. >> >> [majopela@redcylon tmp]$ cat > > import sys >> > print "hello world" >> > sys.exit(0) >> > EOF >> >> [majopela

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Hi Joe, thank you very much for the positive feedback, I plan to spend a day during this week on the shedskin-compatibility for rootwrap (I'll branch it, and tune/cut down as necessary) to make it compile under shedskin [1] : nothing done yet. It's a short-term alternative until we can hav

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Mon, Mar 17, 2014 at 4:48 PM, Joe Gordon wrote: > > > > On Tue, Mar 11, 2014 at 1:46 AM, Miguel Angel Ajo Pelayo < > mangel...@redhat.com> wrote: > >> >> I have included on the etherpad, the option to write a sudo >> plugin (or several), specific for openstack. >> >> >> And this is a test with

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Tue, Mar 11, 2014 at 1:46 AM, Miguel Angel Ajo Pelayo < mangel...@redhat.com> wrote: > > I have included on the etherpad, the option to write a sudo > plugin (or several), specific for openstack. > > > And this is a test with shedskin, I suppose that in more complicated > dependecy scenarios it

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Yuriy Taraday wrote: > Another option would be to allow rootwrap to run in daemon mode and > provide RPC interface. This way Neutron can spawn rootwrap (with its > CPython startup overhead) once and send new commands to be run later > over UNIX socket. > This way we won't need learn new language (C

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

At Thu, 13 Mar 2014 07:48:53 -0700, Aaron Rosen wrote: > > [1 ] > [1.1 ] > The easiest/quickest thing to do for ice house would probably be to run the > initial sync in parallel like the dhcp-agent does for this exact reason. > See: https://review.openstack.org/#/c/28914/ which did this for thr

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

As we said on the Thursday meeting, I've filled a bug with the details https://bugs.launchpad.net/neutron/+bug/1292598 Feel free to add / ask for any missing details. Best, Miguel Ángel. On 03/13/2014 10:52 PM, Carl Baldwin wrote: Right, the L3 agent does do this already. Agreed that the lim

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Right, the L3 agent does do this already. Agreed that the limiting factor is the cumulative effect of the wrappers and executables' start up overhead. Carl On Thu, Mar 13, 2014 at 9:47 AM, Brian Haley wrote: > Aaron, > > I thought the l3-agent already did this if doing a "full sync"? > > _sync_

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Aaron, I thought the l3-agent already did this if doing a "full sync"? _sync_routers_task()->_process_routers()->spawn_n(self.process_router, ri) So each router gets processed in a greenthread. It seems like the other calls - sudo/rootwrap, /sbin/ip, etc are now the limiting factor, at least on

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

The easiest/quickest thing to do for ice house would probably be to run the initial sync in parallel like the dhcp-agent does for this exact reason. See: https://review.openstack.org/#/c/28914/ which did this for thr dhcp-agent. Best, Aaron On Thu, Mar 13, 2014 at 12:18 PM, Miguel Angel Ajo wrote

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Thu, Mar 13, 2014 at 12:18 PM, Miguel Angel Ajo wrote: > > Yuri, could you elaborate your idea in detail? , I'm lost at some > points with your unix domain / token authentication. > > Where does the token come from?, > > Who starts rootwrap the first time? > > If you could write a full interacti

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Yuri, could you elaborate your idea in detail? , I'm lost at some points with your unix domain / token authentication. Where does the token come from?, Who starts rootwrap the first time? If you could write a full interaction sequence, on the etherpad, from rootwrap daemon start ,to a simple

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Tue, Mar 11, 2014 at 12:58 AM, Carl Baldwin wrote: > All, > > I was writing down a summary of all of this and decided to just do it > on an etherpad. Will you help me capture the big picture there? I'd > like to come up with some actions this week to try to address at least > part of the pro

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Mon, Mar 10, 2014 at 3:26 PM, Miguel Angel Ajo wrote: > I'm not familiar with unix domain sockets at low level, but , I wonder > if authentication could be achieved just with permissions (only users in > group "neutron" or group "rootwrap" accessing this service. > It can be enforced, but it i

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

I have included on the etherpad, the option to write a sudo plugin (or several), specific for openstack. And this is a test with shedskin, I suppose that in more complicated dependecy scenarios it should perform better. [majopela@redcylon tmp]$ cat import sys > print "hello world" >

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

We had this same issue with the dhcp-agent. Code was added that paralleled the initial sync here: https://review.openstack.org/#/c/28914/ that made things a good bit faster if I remember correctly. Might be worth doing something similar for the l3-agent. Best, Aaron On Mon, Mar 10, 2014 at 5:

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Mon, Mar 10, 2014 at 3:57 PM, Joe Gordon wrote: > I looked into the python to C options and haven't found anything promising > yet. > > > I tried Cython, and RPython, on a trivial hello world app, but git similar > startup times to standard python. > > The one thing that did work was adding a

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

I looked into the python to C options and haven't found anything promising yet. I tried Cython, and RPython, on a trivial hello world app, but git similar startup times to standard python. The one thing that did work was adding a '-S' when starting python. -S Disable the import of th

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Hi Carl, thank you, good idea. I started reviewing it, but I will do it more carefully tomorrow morning. - Original Message - > All, > > I was writing down a summary of all of this and decided to just do it > on an etherpad. Will you help me capture the big picture there? I'd > like

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

All, I was writing down a summary of all of this and decided to just do it on an etherpad. Will you help me capture the big picture there? I'd like to come up with some actions this week to try to address at least part of the problem before Icehouse releases. https://etherpad.openstack.org/p/ne

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Hi Yuri & Stephen, thanks a lot for the clarification. I'm not familiar with unix domain sockets at low level, but , I wonder if authentication could be achieved just with permissions (only users in group "neutron" or group "rootwrap" accessing this service. I find it an interesting alternativ

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Fri, Mar 7, 2014 at 5:41 PM, Stephen Gran wrote: > Hi, > > Given that Yuriy says explicitly 'unix socket', I dont think he means 'MQ' > when he says 'RPC'. I think he just means a daemon listening on a unix > socket for execution requests. This seems like a reasonably sensible idea > to me. >

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

I had a reply drafted up to Miguel's original post and now I realize that I never actually sent it. :( So, I'll clean up and update my draft and send it. This is a huge impediment to scaling Neutron and I believe this needs some attention before Icehouse releases. I believe this problem needs t

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Mar 6, 2014, at 3:31 AM, Miguel Angel Ajo wrote: > > Yes, one option could be to coalesce all calls that go into > a namespace into a shell script and run this in the > ootwrap > ip netns exec > > But we might find a mechanism to determine if some of the steps failed, and > what was the re

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Hi, Given that Yuriy says explicitly 'unix socket', I dont think he means 'MQ' when he says 'RPC'. I think he just means a daemon listening on a unix socket for execution requests. This seems like a reasonably sensible idea to me. Cheers, On 07/03/14 12:52, Miguel Angel Ajo wrote: I tho

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

I thought of this option, but didn't consider it, as It's somehow risky to expose an RPC end executing priviledged (even filtered) commands. If I'm not wrong, once you have credentials for messaging, you can send messages to any end, even filtered, I somehow see this as a higher risk option. An

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Hello. On Wed, Mar 5, 2014 at 6:42 PM, Miguel Angel Ajo wrote: > 2) What alternatives can we think about to improve this situation. > >0) already being done: coalescing system calls. But I'm unsure that's > enough. (if we coalesce 15 calls to 3 on this system we get: 192*3*0.3/60 > ~=3 minute

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/06/2014 07:57 AM, IWAMOTO Toshihiro wrote: At Wed, 05 Mar 2014 15:42:54 +0100, Miguel Angel Ajo wrote: 3) I also find 10 minutes a long time to setup 192 networks/basic tenant structures, I wonder if that time could be reduced by conversion of system process calls into system library call

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

At Wed, 05 Mar 2014 15:42:54 +0100, Miguel Angel Ajo wrote: > 3) I also find 10 minutes a long time to setup 192 networks/basic tenant > structures, I wonder if that time could be reduced by conversion > of system process calls into system library calls (I know we don't have > libraries for iprout

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

: Wednesday, March 5, 2014 1:13:33 PM Subject: Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo? On Mar 5, 2014, at 6:42 AM, Miguel Angel Ajo wrote: > >Hello, > >Recently, I found a serious issue about network-nodes startup time, > neutron-root

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Mar 5, 2014, at 6:42 AM, Miguel Angel Ajo wrote: > >Hello, > >Recently, I found a serious issue about network-nodes startup time, > neutron-rootwrap eats a lot of cpu cycles, much more than the processes it's > wrapping itself. > >On a database with 1 public network, 192 priva

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On Wed, Mar 5, 2014 at 8:51 AM, Miguel Angel Ajo Pelayo wrote: > > > - Original Message - >> Miguel Angel Ajo wrote: >> > [...] >> >The overhead comes from python startup time + rootwrap loading. >> > >> >I suppose that rootwrap was designed for lower amount of system calls >> > (n

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

On 03/05/2014 06:42 AM, Miguel Angel Ajo wrote: Hello, Recently, I found a serious issue about network-nodes startup time, neutron-rootwrap eats a lot of cpu cycles, much more than the processes it's wrapping itself. On a database with 1 public network, 192 private networks, 192

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

- Original Message - > Miguel Angel Ajo wrote: > > [...] > >The overhead comes from python startup time + rootwrap loading. > > > >I suppose that rootwrap was designed for lower amount of system calls > > (nova?). > > Yes, it was not really designed to escalate rights on hundred

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

This has actually come up before, too: http://lists.openstack.org/pipermail/openstack-dev/2013-July/012539.html -Ben On 2014-03-05 08:42, Miguel Angel Ajo wrote: Hello, Recently, I found a serious issue about network-nodes startup time, neutron-rootwrap eats a lot of cpu cycles, much more

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Miguel Angel Ajo wrote: > [...] >The overhead comes from python startup time + rootwrap loading. > >I suppose that rootwrap was designed for lower amount of system calls > (nova?). Yes, it was not really designed to escalate rights on hundreds of separate shell commands in a row. >An

[openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

Hello, Recently, I found a serious issue about network-nodes startup time, neutron-rootwrap eats a lot of cpu cycles, much more than the processes it's wrapping itself. On a database with 1 public network, 192 private networks, 192 routers, and 192 nano VMs, with OVS plugin: N