Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
On 3 March 2014 19:33, David Peraza david_per...@persistentsys.com wrote: Thanks John, What I'm trying to do is to run an asynchronous task that pre-organizes the target hosts for an image. Then scheduler only need to read the top of the list or priority queue. We have a paper proposed for the summit that will explain the approach, hopefully it gets accepted so we can have a conversation on this at the summit. I suspect the DB overhead will go away if we try our approach. Still theory though, that is why I want to get a significant test environment to appreciate the performance better. I attempted something similar as part of the caching scheduler work. When picking the size of the slot cache, I found I got the best performance when I turned it off. Small bursts of builds were slightly quicker, but would get delayed if they came in when the cache was being populated. Large bursts of requests very quickly depleted the cache, and filling it back up was quite expensive, and you queue up other requests while you do that. So choosing the cache size was very tricky. All the time, you end up making some bad choices because you are only looking at a subset of the nodes. I am however very interested in seeing if you have found a balance that works well. It feels like some combination would help in certain situations. I just couldn't find either myself. My current approach is just to cache the lists of hosts you get from the DB, and update the host state with each decision you make, so those requests don't race each other. Some simple optimisations to the filter and weights system seemed to be a much better route to improving the performance. (I had some patches up for that, will refresh them when Juno opens). But until we get the move to conductor work complete (using select destination instead of run_instance), the DB calls locking all the eventlet threads seems like the biggest issue. Anyways, looking forward to a good discussion at the summit. John Regards, David Peraza -Original Message- From: John Garbutt [mailto:j...@johngarbutt.com] Sent: Tuesday, February 25, 2014 5:45 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing On 24 February 2014 20:13, David Peraza david_per...@persistentsys.com wrote: Thanks John, I also think it is a good idea to test the algorithm at unit test level, but I will like to try out over amqp as well, that is, we process and threads talking to each other over rabbit or qpid. I'm trying to test out performance as well. Nothing beats testing the thing for real, of course. As a heads up, the overheads of DB calls turned out to dwarf any algorithmic improvements I managed. There will clearly be some RPC overhead, but it didn't stand out as much as the DB issue. The move to conductor work should certainly stop the scheduler making those pesky DB calls to update the nova instance. And then, improvements like no-db-scheduler and improvements to scheduling algorithms should shine through much more. Thanks, John -Original Message- From: John Garbutt [mailto:j...@johngarbutt.com] Sent: Monday, February 24, 2014 11:51 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing On 24 February 2014 16:24, David Peraza david_per...@persistentsys.com wrote: Hello all, I have been trying some new ideas on scheduler and I think I'm reaching a resource issue. I'm running 6 compute service right on my 4 CPU 4 Gig VM, and I started to get some memory allocation issues. Keystone and Nova are already complaining there is not enough memory. The obvious solution to add more candidates is to get another VM and set another 6 Fake compute service. I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? It depends what you want to test, but I was able to look at tuning the filters and weights using the test at the end of this file: https://review.openstack.org/#/c/67855/33/nova/tests/scheduler/test_ca ching_scheduler.py Cheers, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
Well I use a 8 cores 128G RAM physical host :) I did not see much of the CPU consumption for these 100 containers, so I suspect we can use less resources. -Message d'origine- De : David Peraza [mailto:david_per...@persistentsys.com] Envoyé : lundi 3 mars 2014 20:27 À : OpenStack Development Mailing List (not for usage questions) Objet : Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing Thanks Khanh, I see the potential issue with using threads. Thanks for pointing out. On using containers, that sounds like a cool configuration but that should have a bigger footprint on the host resources than just a separate service instance like I'm doing. I have to admit that 100 fake computes per physical host is good though. How big is your physical host. I'm running a 4 Gig 4 CPU VM. I suspect your physical system is much more equipped. Regards, David Peraza | Openstack Solutions Architect david_per...@persistentsys.com | Cell: (305)766-2520 Persistent Systems Inc. | Partners in Innovation | www.persistentsys.com -Original Message- From: Khanh-Toan Tran [mailto:khanh-toan.t...@cloudwatt.com] Sent: Tuesday, February 25, 2014 3:49 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? I'm not sure using threads is a good idea. We need a dedicated resources pool for each compute. If the threads share the same resources pool, then every new VM will change the available resources on all computes, which may lead to unexpected unpredicted scheduling result. For instance, RamWeigher may return the same compute twice instead of spreading, because at each time it finds out that the computes have the same free_ram. Using compute inside LXC, I created 100 computes per physical host. Here is what I did, it's very simple: - Creating a LXC with logical volume - Installing a fake nova-compute inside the LXC - Make a booting script that modifies its nova.conf to use its IP address starts nova-compute - Using the LXC above as the master, clone as many compute as you like! (Note that while cloning the LXC, the nova.conf is copied with the former's IP address, that's why we need the booting script.) Best regards, Toan -Message d'origine- De : David Peraza [mailto:david_per...@persistentsys.com] Envoyé : lundi 24 février 2014 21:13 À : OpenStack Development Mailing List (not for usage questions) Objet : Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing Thanks John, I also think it is a good idea to test the algorithm at unit test level, but I will like to try out over amqp as well, that is, we process and threads talking to each other over rabbit or qpid. I'm trying to test out performance as well. Regards, David Peraza -Original Message- From: John Garbutt [mailto:j...@johngarbutt.com] Sent: Monday, February 24, 2014 11:51 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing On 24 February 2014 16:24, David Peraza david_per...@persistentsys.com wrote: Hello all, I have been trying some new ideas on scheduler and I think I'm reaching a resource issue. I'm running 6 compute service right on my 4 CPU 4 Gig VM, and I started to get some memory allocation issues. Keystone and Nova are already complaining there is not enough memory. The obvious solution to add more candidates is to get another VM and set another 6 Fake compute service. I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? It depends what you want to test, but I was able to look at tuning the filters and weights using the test at the end of this file: https://review.openstack.org/#/c/67855/33/nova/tests/scheduler/test_cachin g _scheduler.py Cheers, John
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
On Mon, Mar 3, 2014 at 3:10 PM, Sergey Skripnick sskripn...@mirantis.comwrote: I can run multiple compute service in same hosts without containers. Containers give you a nice isolation and another way to try a more realistic scenario, but my initial goal now is to be able to simulate many fake compute node scenario with as little resources as possible. I believe it is impossible to use threads without changes in the code. Having gone the threads route once myself, I can say from experience that it requires changes to the code. I was able to get threads up and running with a few modifications, but there were other issues that I never fully resolved that make me lean more towards the container model that has been discussed earlier in the thread. Btw, I would suggest having a look at Rally, the Openstack Benchmarking Service. They have deployment frameworks that use LXC that you might be able to write a thread model for. -Mike -- Regards, Sergey Skripnick ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
Thanks Khanh, I see the potential issue with using threads. Thanks for pointing out. On using containers, that sounds like a cool configuration but that should have a bigger footprint on the host resources than just a separate service instance like I'm doing. I have to admit that 100 fake computes per physical host is good though. How big is your physical host. I'm running a 4 Gig 4 CPU VM. I suspect your physical system is much more equipped. Regards, David Peraza | Openstack Solutions Architect david_per...@persistentsys.com | Cell: (305)766-2520 Persistent Systems Inc. | Partners in Innovation | www.persistentsys.com -Original Message- From: Khanh-Toan Tran [mailto:khanh-toan.t...@cloudwatt.com] Sent: Tuesday, February 25, 2014 3:49 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? I'm not sure using threads is a good idea. We need a dedicated resources pool for each compute. If the threads share the same resources pool, then every new VM will change the available resources on all computes, which may lead to unexpected unpredicted scheduling result. For instance, RamWeigher may return the same compute twice instead of spreading, because at each time it finds out that the computes have the same free_ram. Using compute inside LXC, I created 100 computes per physical host. Here is what I did, it's very simple: - Creating a LXC with logical volume - Installing a fake nova-compute inside the LXC - Make a booting script that modifies its nova.conf to use its IP address starts nova-compute - Using the LXC above as the master, clone as many compute as you like! (Note that while cloning the LXC, the nova.conf is copied with the former's IP address, that's why we need the booting script.) Best regards, Toan -Message d'origine- De : David Peraza [mailto:david_per...@persistentsys.com] Envoyé : lundi 24 février 2014 21:13 À : OpenStack Development Mailing List (not for usage questions) Objet : Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing Thanks John, I also think it is a good idea to test the algorithm at unit test level, but I will like to try out over amqp as well, that is, we process and threads talking to each other over rabbit or qpid. I'm trying to test out performance as well. Regards, David Peraza -Original Message- From: John Garbutt [mailto:j...@johngarbutt.com] Sent: Monday, February 24, 2014 11:51 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing On 24 February 2014 16:24, David Peraza david_per...@persistentsys.com wrote: Hello all, I have been trying some new ideas on scheduler and I think I'm reaching a resource issue. I'm running 6 compute service right on my 4 CPU 4 Gig VM, and I started to get some memory allocation issues. Keystone and Nova are already complaining there is not enough memory. The obvious solution to add more candidates is to get another VM and set another 6 Fake compute service. I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? It depends what you want to test, but I was able to look at tuning the filters and weights using the test at the end of this file: https://review.openstack.org/#/c/67855/33/nova/tests/scheduler/test_cachin g _scheduler.py Cheers, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
Thanks John, What I'm trying to do is to run an asynchronous task that pre-organizes the target hosts for an image. Then scheduler only need to read the top of the list or priority queue. We have a paper proposed for the summit that will explain the approach, hopefully it gets accepted so we can have a conversation on this at the summit. I suspect the DB overhead will go away if we try our approach. Still theory though, that is why I want to get a significant test environment to appreciate the performance better. Regards, David Peraza -Original Message- From: John Garbutt [mailto:j...@johngarbutt.com] Sent: Tuesday, February 25, 2014 5:45 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing On 24 February 2014 20:13, David Peraza david_per...@persistentsys.com wrote: Thanks John, I also think it is a good idea to test the algorithm at unit test level, but I will like to try out over amqp as well, that is, we process and threads talking to each other over rabbit or qpid. I'm trying to test out performance as well. Nothing beats testing the thing for real, of course. As a heads up, the overheads of DB calls turned out to dwarf any algorithmic improvements I managed. There will clearly be some RPC overhead, but it didn't stand out as much as the DB issue. The move to conductor work should certainly stop the scheduler making those pesky DB calls to update the nova instance. And then, improvements like no-db-scheduler and improvements to scheduling algorithms should shine through much more. Thanks, John -Original Message- From: John Garbutt [mailto:j...@johngarbutt.com] Sent: Monday, February 24, 2014 11:51 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing On 24 February 2014 16:24, David Peraza david_per...@persistentsys.com wrote: Hello all, I have been trying some new ideas on scheduler and I think I'm reaching a resource issue. I'm running 6 compute service right on my 4 CPU 4 Gig VM, and I started to get some memory allocation issues. Keystone and Nova are already complaining there is not enough memory. The obvious solution to add more candidates is to get another VM and set another 6 Fake compute service. I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? It depends what you want to test, but I was able to look at tuning the filters and weights using the test at the end of this file: https://review.openstack.org/#/c/67855/33/nova/tests/scheduler/test_ca ching_scheduler.py Cheers, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
David Peraza david_per...@persistentsys.com писал(а) в своём письме Mon, 03 Mar 2014 21:27:01 +0200: Using compute inside LXC, I created 100 computes per physical host. Here is what I did, it's very simple: - Creating a LXC with logical volume - Installing a fake nova-compute inside the LXC - Make a booting script that modifies its nova.conf to use its IP address starts nova-compute - Using the LXC above as the master, clone as many compute as you like! (Note that while cloning the LXC, the nova.conf is copied with the former's IP address, that's why we need the booting script.) Also btrfs can be used to reduce disk space usage. Scenario is looks like this: * create btrfs filesystem on /var/lib/lxc * create first container with btrfs backingstore (lxc-create -B btrfs ...) * setup devstack with fakevirt * stop container, and make N snapshotted clones (lxc-clone --snapshot ...) * start containers and run booting script (here script used by Rally [0]) If you do not want to do all this manually, there is MultihostEngine/LxcEngine coming soon in Rally [1] [0] https://review.openstack.org/#/c/56222/25/rally/deploy/engines/lxc/start.sh [1] https://wiki.openstack.org/wiki/Rally -- Regards, Sergey Skripnick ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
Thanks Sergey, Will look into this, but still have the same question, would introducing containers increase my footprint per fake compute service? I can run multiple compute service in same hosts without containers. Containers give you a nice isolation and another way to try a more realistic scenario, but my initial goal now is to be able to simulate many fake compute node scenario with as little resources as possible. Regards, David Peraza -Original Message- From: Sergey Skripnick [mailto:sskripn...@mirantis.com] Sent: Monday, March 03, 2014 3:18 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing David Peraza david_per...@persistentsys.com писал(а) в своём письме Mon, 03 Mar 2014 21:27:01 +0200: Using compute inside LXC, I created 100 computes per physical host. Here is what I did, it's very simple: - Creating a LXC with logical volume - Installing a fake nova-compute inside the LXC - Make a booting script that modifies its nova.conf to use its IP address starts nova-compute - Using the LXC above as the master, clone as many compute as you like! (Note that while cloning the LXC, the nova.conf is copied with the former's IP address, that's why we need the booting script.) Also btrfs can be used to reduce disk space usage. Scenario is looks like this: * create btrfs filesystem on /var/lib/lxc * create first container with btrfs backingstore (lxc-create -B btrfs ...) * setup devstack with fakevirt * stop container, and make N snapshotted clones (lxc-clone --snapshot ...) * start containers and run booting script (here script used by Rally [0]) If you do not want to do all this manually, there is MultihostEngine/LxcEngine coming soon in Rally [1] [0] https://review.openstack.org/#/c/56222/25/rally/deploy/engines/lxc/start.sh [1] https://wiki.openstack.org/wiki/Rally -- Regards, Sergey Skripnick ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
David Peraza david_per...@persistentsys.com писал(а) в своём письме Mon, 03 Mar 2014 23:52:12 +0200: Thanks Sergey, Will look into this, but still have the same question, would introducing containers increase my footprint per fake compute service? Yes. There is more then one processes running in the container. I can run multiple compute service in same hosts without containers. Containers give you a nice isolation and another way to try a more realistic scenario, but my initial goal now is to be able to simulate many fake compute node scenario with as little resources as possible. I believe it is impossible to use threads without changes in the code. -- Regards, Sergey Skripnick ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? I'm not sure using threads is a good idea. We need a dedicated resources pool for each compute. If the threads share the same resources pool, then every new VM will change the available resources on all computes, which may lead to unexpected unpredicted scheduling result. For instance, RamWeigher may return the same compute twice instead of spreading, because at each time it finds out that the computes have the same free_ram. Using compute inside LXC, I created 100 computes per physical host. Here is what I did, it's very simple: - Creating a LXC with logical volume - Installing a fake nova-compute inside the LXC - Make a booting script that modifies its nova.conf to use its IP address starts nova-compute - Using the LXC above as the master, clone as many compute as you like! (Note that while cloning the LXC, the nova.conf is copied with the former's IP address, that's why we need the booting script.) Best regards, Toan -Message d'origine- De : David Peraza [mailto:david_per...@persistentsys.com] Envoyé : lundi 24 février 2014 21:13 À : OpenStack Development Mailing List (not for usage questions) Objet : Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing Thanks John, I also think it is a good idea to test the algorithm at unit test level, but I will like to try out over amqp as well, that is, we process and threads talking to each other over rabbit or qpid. I'm trying to test out performance as well. Regards, David Peraza -Original Message- From: John Garbutt [mailto:j...@johngarbutt.com] Sent: Monday, February 24, 2014 11:51 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing On 24 February 2014 16:24, David Peraza david_per...@persistentsys.com wrote: Hello all, I have been trying some new ideas on scheduler and I think I'm reaching a resource issue. I'm running 6 compute service right on my 4 CPU 4 Gig VM, and I started to get some memory allocation issues. Keystone and Nova are already complaining there is not enough memory. The obvious solution to add more candidates is to get another VM and set another 6 Fake compute service. I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? It depends what you want to test, but I was able to look at tuning the filters and weights using the test at the end of this file: https://review.openstack.org/#/c/67855/33/nova/tests/scheduler/test_cachin g _scheduler.py Cheers, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
On 24 February 2014 20:13, David Peraza david_per...@persistentsys.com wrote: Thanks John, I also think it is a good idea to test the algorithm at unit test level, but I will like to try out over amqp as well, that is, we process and threads talking to each other over rabbit or qpid. I'm trying to test out performance as well. Nothing beats testing the thing for real, of course. As a heads up, the overheads of DB calls turned out to dwarf any algorithmic improvements I managed. There will clearly be some RPC overhead, but it didn't stand out as much as the DB issue. The move to conductor work should certainly stop the scheduler making those pesky DB calls to update the nova instance. And then, improvements like no-db-scheduler and improvements to scheduling algorithms should shine through much more. Thanks, John -Original Message- From: John Garbutt [mailto:j...@johngarbutt.com] Sent: Monday, February 24, 2014 11:51 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing On 24 February 2014 16:24, David Peraza david_per...@persistentsys.com wrote: Hello all, I have been trying some new ideas on scheduler and I think I'm reaching a resource issue. I'm running 6 compute service right on my 4 CPU 4 Gig VM, and I started to get some memory allocation issues. Keystone and Nova are already complaining there is not enough memory. The obvious solution to add more candidates is to get another VM and set another 6 Fake compute service. I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? It depends what you want to test, but I was able to look at tuning the filters and weights using the test at the end of this file: https://review.openstack.org/#/c/67855/33/nova/tests/scheduler/test_caching_scheduler.py Cheers, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
On Tue, 2014-02-25 at 10:45 +, John Garbutt wrote: As a heads up, the overheads of DB calls turned out to dwarf any algorithmic improvements I managed. There will clearly be some RPC overhead, but it didn't stand out as much as the DB issue. The move to conductor work should certainly stop the scheduler making those pesky DB calls to update the nova instance. And then, improvements like no-db-scheduler and improvements to scheduling algorithms should shine through much more. Although DB access is sure the key for performance, but do we really want to pursue conductor-based scheduler? --jyh ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
Hello all, I have been trying some new ideas on scheduler and I think I'm reaching a resource issue. I'm running 6 compute service right on my 4 CPU 4 Gig VM, and I started to get some memory allocation issues. Keystone and Nova are already complaining there is not enough memory. The obvious solution to add more candidates is to get another VM and set another 6 Fake compute service. I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? Regards, David Peraza DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
On 24 February 2014 16:24, David Peraza david_per...@persistentsys.com wrote: Hello all, I have been trying some new ideas on scheduler and I think I'm reaching a resource issue. I'm running 6 compute service right on my 4 CPU 4 Gig VM, and I started to get some memory allocation issues. Keystone and Nova are already complaining there is not enough memory. The obvious solution to add more candidates is to get another VM and set another 6 Fake compute service. I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? It depends what you want to test, but I was able to look at tuning the filters and weights using the test at the end of this file: https://review.openstack.org/#/c/67855/33/nova/tests/scheduler/test_caching_scheduler.py Cheers, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing
Thanks John, I also think it is a good idea to test the algorithm at unit test level, but I will like to try out over amqp as well, that is, we process and threads talking to each other over rabbit or qpid. I'm trying to test out performance as well. Regards, David Peraza -Original Message- From: John Garbutt [mailto:j...@johngarbutt.com] Sent: Monday, February 24, 2014 11:51 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing On 24 February 2014 16:24, David Peraza david_per...@persistentsys.com wrote: Hello all, I have been trying some new ideas on scheduler and I think I'm reaching a resource issue. I'm running 6 compute service right on my 4 CPU 4 Gig VM, and I started to get some memory allocation issues. Keystone and Nova are already complaining there is not enough memory. The obvious solution to add more candidates is to get another VM and set another 6 Fake compute service. I could do that but I think I need to be able to scale more without the need to use this much resources. I will like to simulate a cloud of 100 maybe 1000 compute nodes that do nothing (Fake driver) this should not take this much memory. Anyone knows of a more efficient way to simulate many computes? I was thinking changing the Fake driver to report many compute services in different threads instead of having to spawn a process per compute service. Any other ideas? It depends what you want to test, but I was able to look at tuning the filters and weights using the test at the end of this file: https://review.openstack.org/#/c/67855/33/nova/tests/scheduler/test_caching_scheduler.py Cheers, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
