Re: [openstack-dev] [Fuel][Docker] Master node bootstrapping issues
4) do lrzipping as weak as possible during the development phase and lrzip it strongly only when we do release We create lrzip archive with compression level 2 (from 1-9 range, default is 7). So we don't have much potential for improvement here. Here are some tests: == Bare metal 12 cpus, 32 G RAM == level 1 decompression total time: 00:00:27.70 level 2 decompression total time: 00:00:29.64 == Virtual server 1 cpu, 1G RAM == level 1 decompression total time: 00:05:17.12 level 2 decompression total time: 00:05:22.86 I did some further research on this matter, increasing RAM to 4G did not help much as well: == Virtual server 1 cpu, 4G RAM == level 1 decompression total time: 00:05:08.70 level 2 decompression total time: 00:05:12.57 So it looks like 'lrzuntar' command itself causes this problem on a weak hardware, because it uses lrzcat | tar which works very fast if we have enough memory. If we extract archive in two separate steps (lrzip -d tar -xf) instead of single lrzuntar, then they take only ~1m30s summary (comparing to 5+ minutes with single lrzuntar command). Here are some test results for lrzip+tar commands: == Bare metal 12 cpus, 32 G RAM == level 2 decompression total time: 00:00:50.12 == Virtual server 1 cpu, 1G RAM == level 2 decompression total time: 00:01:35.95 So lrzuntar works faster on a powerful hardware, lrzip+tar works faster on a weak hardware (like VMs). I suggest to switch to lrzip+tar. On Sat, May 10, 2014 at 8:01 PM, Dmitry Borodaenko dborodae...@mirantis.com wrote: FWIW 1GB works fine for me on my laptop, I run the master setup manually. So I'm against increasing RAM requirement, we have better things to spend that RAM on. On May 10, 2014 1:37 AM, Mike Scherbakov mscherba...@mirantis.com wrote: It is not related to RAM or CPU. I run installation on my Mac with 1Gb of RAM for master node, and experience the following: - yes, it needs time to bootstrap admin node - As soon as I have message that master node is installed, I immediately open 10.20.0.2:8000 and try to generate diag snapshot. And it is failed. - If I wait a few more minutes, and try again - it is passed. It actually seems to me that we simply still do not have https://bugs.launchpad.net/fuel/+bug/1315865 fixed, I'll add more details there as well as logs. When I checked logs, I saw: - for about a minute, astute was not able to connect to MQ. It means it is still started before MQ is ready? - shotgun -c /tmp/dump_config /var/log/dump.log 21 cat /var/www/nailgun/dump/last returned 1 When I tried to run diag_snapshot for a second time, the command above succeeded with 0 return code. So it obviously needs further debugging and in my opinion even if we need to increase VCPU or RAM, then no more than 2 VCPU / 2 Gb. Vladimir, as you and Matt were changing the code which should run containers in a certain order, I'm looking forward to hear from both of you suggestions on where and how we should hack it. Thanks, On Sat, May 10, 2014 at 1:04 AM, Vladimir Kuklin vkuk...@mirantis.comwrote: Hi all We are still experiencing some issues with master node bootstrapping after moving to container-based installation. First of all, these issues are related to our system tests. We have rather small nodes running as master node - only 1 GB of RAM and 1 virtual CPU. As we are using strongly lrzipped archive, this seems quite not enough and leads to timeouts during deployment of the master node. I have several suggestions: 1) Increase amount of RAM for master node to at least 8 Gigabytes (or do some pci virtual memory hotplug during master node bootstrapping) and add additional vCPU for the master node. 2) Run system tests with non-containerized environment (env variable PRODUCTION=prod set) 3) Split our system tests in that way not allowing more than 2 master nodes to bootstrap simulteneously on the single hardware node. 4) do lrzipping as weak as possible during the development phase and lrzip it strongly only when we do release 5) increase bootstrap timeout for the master node in system tests Any input would be appreciated. -- Yours Faithfully, Vladimir Kuklin, Fuel Library Tech Lead, Mirantis, Inc. +7 (495) 640-49-04 +7 (926) 702-39-68 Skype kuklinvv 45bk3, Vorontsovskaya Str. Moscow, Russia, www.mirantis.com http://www.mirantis.ru/ www.mirantis.ru vkuk...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [Fuel][Docker] Master node bootstrapping issues
It is not related to RAM or CPU. I run installation on my Mac with 1Gb of RAM for master node, and experience the following: - yes, it needs time to bootstrap admin node - As soon as I have message that master node is installed, I immediately open 10.20.0.2:8000 and try to generate diag snapshot. And it is failed. - If I wait a few more minutes, and try again - it is passed. It actually seems to me that we simply still do not have https://bugs.launchpad.net/fuel/+bug/1315865 fixed, I'll add more details there as well as logs. When I checked logs, I saw: - for about a minute, astute was not able to connect to MQ. It means it is still started before MQ is ready? - shotgun -c /tmp/dump_config /var/log/dump.log 21 cat /var/www/nailgun/dump/last returned 1 When I tried to run diag_snapshot for a second time, the command above succeeded with 0 return code. So it obviously needs further debugging and in my opinion even if we need to increase VCPU or RAM, then no more than 2 VCPU / 2 Gb. Vladimir, as you and Matt were changing the code which should run containers in a certain order, I'm looking forward to hear from both of you suggestions on where and how we should hack it. Thanks, On Sat, May 10, 2014 at 1:04 AM, Vladimir Kuklin vkuk...@mirantis.comwrote: Hi all We are still experiencing some issues with master node bootstrapping after moving to container-based installation. First of all, these issues are related to our system tests. We have rather small nodes running as master node - only 1 GB of RAM and 1 virtual CPU. As we are using strongly lrzipped archive, this seems quite not enough and leads to timeouts during deployment of the master node. I have several suggestions: 1) Increase amount of RAM for master node to at least 8 Gigabytes (or do some pci virtual memory hotplug during master node bootstrapping) and add additional vCPU for the master node. 2) Run system tests with non-containerized environment (env variable PRODUCTION=prod set) 3) Split our system tests in that way not allowing more than 2 master nodes to bootstrap simulteneously on the single hardware node. 4) do lrzipping as weak as possible during the development phase and lrzip it strongly only when we do release 5) increase bootstrap timeout for the master node in system tests Any input would be appreciated. -- Yours Faithfully, Vladimir Kuklin, Fuel Library Tech Lead, Mirantis, Inc. +7 (495) 640-49-04 +7 (926) 702-39-68 Skype kuklinvv 45bk3, Vorontsovskaya Str. Moscow, Russia, www.mirantis.com http://www.mirantis.ru/ www.mirantis.ru vkuk...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel][Docker] Master node bootstrapping issues
FWIW 1GB works fine for me on my laptop, I run the master setup manually. So I'm against increasing RAM requirement, we have better things to spend that RAM on. On May 10, 2014 1:37 AM, Mike Scherbakov mscherba...@mirantis.com wrote: It is not related to RAM or CPU. I run installation on my Mac with 1Gb of RAM for master node, and experience the following: - yes, it needs time to bootstrap admin node - As soon as I have message that master node is installed, I immediately open 10.20.0.2:8000 and try to generate diag snapshot. And it is failed. - If I wait a few more minutes, and try again - it is passed. It actually seems to me that we simply still do not have https://bugs.launchpad.net/fuel/+bug/1315865 fixed, I'll add more details there as well as logs. When I checked logs, I saw: - for about a minute, astute was not able to connect to MQ. It means it is still started before MQ is ready? - shotgun -c /tmp/dump_config /var/log/dump.log 21 cat /var/www/nailgun/dump/last returned 1 When I tried to run diag_snapshot for a second time, the command above succeeded with 0 return code. So it obviously needs further debugging and in my opinion even if we need to increase VCPU or RAM, then no more than 2 VCPU / 2 Gb. Vladimir, as you and Matt were changing the code which should run containers in a certain order, I'm looking forward to hear from both of you suggestions on where and how we should hack it. Thanks, On Sat, May 10, 2014 at 1:04 AM, Vladimir Kuklin vkuk...@mirantis.comwrote: Hi all We are still experiencing some issues with master node bootstrapping after moving to container-based installation. First of all, these issues are related to our system tests. We have rather small nodes running as master node - only 1 GB of RAM and 1 virtual CPU. As we are using strongly lrzipped archive, this seems quite not enough and leads to timeouts during deployment of the master node. I have several suggestions: 1) Increase amount of RAM for master node to at least 8 Gigabytes (or do some pci virtual memory hotplug during master node bootstrapping) and add additional vCPU for the master node. 2) Run system tests with non-containerized environment (env variable PRODUCTION=prod set) 3) Split our system tests in that way not allowing more than 2 master nodes to bootstrap simulteneously on the single hardware node. 4) do lrzipping as weak as possible during the development phase and lrzip it strongly only when we do release 5) increase bootstrap timeout for the master node in system tests Any input would be appreciated. -- Yours Faithfully, Vladimir Kuklin, Fuel Library Tech Lead, Mirantis, Inc. +7 (495) 640-49-04 +7 (926) 702-39-68 Skype kuklinvv 45bk3, Vorontsovskaya Str. Moscow, Russia, www.mirantis.com http://www.mirantis.ru/ www.mirantis.ru vkuk...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Fuel][Docker] Master node bootstrapping issues
Hi all We are still experiencing some issues with master node bootstrapping after moving to container-based installation. First of all, these issues are related to our system tests. We have rather small nodes running as master node - only 1 GB of RAM and 1 virtual CPU. As we are using strongly lrzipped archive, this seems quite not enough and leads to timeouts during deployment of the master node. I have several suggestions: 1) Increase amount of RAM for master node to at least 8 Gigabytes (or do some pci virtual memory hotplug during master node bootstrapping) and add additional vCPU for the master node. 2) Run system tests with non-containerized environment (env variable PRODUCTION=prod set) 3) Split our system tests in that way not allowing more than 2 master nodes to bootstrap simulteneously on the single hardware node. 4) do lrzipping as weak as possible during the development phase and lrzip it strongly only when we do release 5) increase bootstrap timeout for the master node in system tests Any input would be appreciated. -- Yours Faithfully, Vladimir Kuklin, Fuel Library Tech Lead, Mirantis, Inc. +7 (495) 640-49-04 +7 (926) 702-39-68 Skype kuklinvv 45bk3, Vorontsovskaya Str. Moscow, Russia, www.mirantis.com http://www.mirantis.ru/ www.mirantis.ru vkuk...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
