Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
We've been through this discussion a bit before, and we believe the repoquery needs to be there. I'm a bit more curious about why you are spending so much time in the operation and most people are not. When using yum in any sort of important setup, I almost always create a yum mirror with reposync, etc, and even in our testing, we're not seeing any major timing issues with the yum options at all. yum_rhn_plugin can sometimes be a very very different story (hence even more reason to mirror content). On Wed, Feb 26, 2014 at 11:49 PM, Marc Trudel mtru...@wizcorp.jp wrote: Some feedback I tried a few things still to make it perform better, including mirror repositories, but the fact that repoquery is forced on the user is perhaps limiting... any ways to make that optional instead of using it if it is present? On Thursday, February 20, 2014 2:39:48 PM UTC+9, Marc Trudel wrote: @cove_s nice :) I didn't get to go down that much, but that reflects pretty well what I am experiencing. @Adam @Michael at least for updates, NOT using repoquery made things faster for me. What I did is change the code for the yum module to undefine the repoquery path. On Thursday, February 20, 2014 3:51:00 AM UTC+9, cove_s wrote: I just happen to add some crude log traces to my yum module last night to see if I could figure out what it's doing. On RPMs that are already installed it will use up all the CPU/IO for a while, on a small instance it can take a long time. The instance I was testing with was an m1.small, so it's slow anyway, but for just testing if an RPM is already installed, it's pretty intense. The what_provides() appeared to be the worst, however I didn't log the exit time of the function to get a good measurement. I'm also not sure why it would need to call that if I just gave it an RPM name instead of a path to look up. This RPM from an onsite repo cache, and we do run yum clear all before hand... 2014-02-19T07:21:07.245573+00:00 myserver-01 ansible-yum: Invoked with name=MyRPM list=None disable_gpg_check=False conf_file=None state=latest disablerepo=None enablerepo=None 2014-02-19T07:21:07.245761+00:00 myserver-01 ansible-yum: ensure() MyRPM 2014-02-19T07:21:07.381131+00:00 myserver-01 ansible-yum: latest() ['MyRPM'] 2014-02-19T07:21:07.381283+00:00 myserver-01 ansible-yum: is_installed() MyRPM 2014-02-19T07:21:07.393524+00:00 myserver-01 ansible-yum: what_provides() MyRPM On Feb 19, 2014, at 10:26 , Adam Morris zwac...@gmail.com wrote: I'm manually adding yum-utils in my RedHat installs as I am performing a minimal install. I figured that this was my fault for trying to install as little as possible. It might make some sense to document that dependency in the yum module page though. Adam On Wednesday, February 19, 2014 5:03:18 AM UTC-8, Michael DeHaan wrote: +1 Also, what (remote) OS is this? We'd have this discussion before, where yum-utils we were pretty sure was only excluded in @core installs. That might not b e true though -- need to check. I have no problem making the yum module self-add yum-utils if not already there if it resolves problems in those environments as it should be there anyway. On Wed, Feb 19, 2014 at 6:17 AM, Walid walid@gmail.com wrote: I am away from my Ansible machine and test, however in my playbook the first thing i do is update yum, and yum-utils to the latest update as i had similar issues with older releases. On 19 February 2014 11:59, Marc Trudel mtr...@wizcorp.jp wrote: Quick note. My playbooks break if I do not have repoquery... the code seems to suggest this is optional, but I just found a case, for instance, where checking for an already installed package gave me a recursion error, while another fresh install failed on failed to parse: SUDO-SUCCESS-whatever On Wednesday, February 19, 2014 5:22:07 PM UTC+9, Marc Trudel wrote: I think I found the issue - seems to be related to repoquery Following tests were done as suggested with the test-module on the host With repoquery: real0m21.014s user0m4.094s sys0m1.337s Without repoquery: real0m8.130s user0m1.914s sys0m0.449s I guess it is then no longer an ansible issue (never really were), but has anyone experienced this in the past? On Wednesday, February 19, 2014 10:22:54 AM UTC+9, Michael DeHaan wrote: ./hacking/test-module in the checkout is pretty useful for things like this. Do a checkout on a machine with yum and even inserting some basic print statements or logging could be a useful start to find out what functions or commands are taking the most time. On Tue, Feb 18, 2014 at 7:47 PM, Marc Trudel mtr...@wizcorp.jpwrote: I'll remove fastest-mirror, it indeed looks like it made things slower (this is in fact what I was adding to my stack as an experiment to make YUM faster - at first I thought it was purely YUM-related issue). I will try to find some information as to how
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
For yum, I disable fastestmirror, set hard-coded repo sites, then configure an http_proxy. For apt, I set hard-coded repo sites, then configure an http_proxy. This seems much lighter weight then cloning an entire OS distribution, when most packages aren't going to be installed anyways. ps: if you leave fastestmirror enabled, then the download site will change randomly, so a proxy is worthless. Also, the centralized site that fastestmirror talks to seems to be highly unstable, and returns spurious errors, which cause the ansible yum module to abort, but only sometimes. This isn't a bug in ansible, but in the yum python module that ansible uses. On 02/27/2014 07:15 AM, Michael DeHaan wrote: We've been through this discussion a bit before, and we believe the repoquery needs to be there. I'm a bit more curious about why you are spending so much time in the operation and most people are not. When using yum in any sort of important setup, I almost always create a yum mirror with reposync, etc, and even in our testing, we're not seeing any major timing issues with the yum options at all. yum_rhn_plugin can sometimes be a very very different story (hence even more reason to mirror content). -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/530F7A50.8070902%40brainfood.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
Yep, that can be a valid approach. This seems much lighter weight then cloning an entire OS distribution, It's much smaller than the apt repo, however. The other bonus is being able to control the package versions on all of your hosts and update when you choose while still coding only state=latest in the ansible content. On Thu, Feb 27, 2014 at 12:48 PM, Adam Heath a...@brainfood.com wrote: For yum, I disable fastestmirror, set hard-coded repo sites, then configure an http_proxy. For apt, I set hard-coded repo sites, then configure an http_proxy. This seems much lighter weight then cloning an entire OS distribution, when most packages aren't going to be installed anyways. ps: if you leave fastestmirror enabled, then the download site will change randomly, so a proxy is worthless. Also, the centralized site that fastestmirror talks to seems to be highly unstable, and returns spurious errors, which cause the ansible yum module to abort, but only sometimes. This isn't a bug in ansible, but in the yum python module that ansible uses. On 02/27/2014 07:15 AM, Michael DeHaan wrote: We've been through this discussion a bit before, and we believe the repoquery needs to be there. I'm a bit more curious about why you are spending so much time in the operation and most people are not. When using yum in any sort of important setup, I almost always create a yum mirror with reposync, etc, and even in our testing, we're not seeing any major timing issues with the yum options at all. yum_rhn_plugin can sometimes be a very very different story (hence even more reason to mirror content). -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/ansible-project/530F7A50.8070902%40brainfood.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEVJ8QN0brc_hhHwJ7AdH499AKpffOra9zBdN%2Ba3kvMbES-jow%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
Was talking about yum. On Thu, Feb 27, 2014 at 1:17 PM, Adam Heath a...@brainfood.com wrote: On 02/27/2014 12:10 PM, Michael DeHaan wrote: Yep, that can be a valid approach. This seems much lighter weight then cloning an entire OS distribution, It's much smaller than the apt repo, however. The other bonus is being able to control the package versions on all of your hosts and update when you choose while still coding only state=latest in the ansible content. Debian has snapshot support, so I could always add that line to sources.list, and then configure version pinning, if I really wanted that. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/ansible-project/530F811F.80504%40brainfood.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEVJ8QOy4pKYw7gBB5EEM4ZGN%3DsO2B5Y2HAPCxX2aCvH%2B0bJkw%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
Sure, but it's easier in debian, as snapshot/backports are also mirrored. On 02/27/2014 12:17 PM, Michael DeHaan wrote: Was talking about yum. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/530F818E.8060006%40brainfood.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
This is not the place for which-distro-is-best and is off topic, see the subject of this thread and what we are discussing. On Thu, Feb 27, 2014 at 1:18 PM, Adam Heath a...@brainfood.com wrote: Sure, but it's easier in debian, as snapshot/backports are also mirrored. On 02/27/2014 12:17 PM, Michael DeHaan wrote: Was talking about yum. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/ansible-project/530F818E.8060006%40brainfood.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEVJ8QOXgtWLdJpB6uquxXz52WxeJ%3DbWDq-_mvraZixWqpcsDw%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
Some feedback I tried a few things still to make it perform better, including mirror repositories, but the fact that repoquery is forced on the user is perhaps limiting... any ways to make that optional instead of using it if it is present? On Thursday, February 20, 2014 2:39:48 PM UTC+9, Marc Trudel wrote: @cove_s nice :) I didn't get to go down that much, but that reflects pretty well what I am experiencing. @Adam @Michael at least for updates, NOT using repoquery made things faster for me. What I did is change the code for the yum module to undefine the repoquery path. On Thursday, February 20, 2014 3:51:00 AM UTC+9, cove_s wrote: I just happen to add some crude log traces to my yum module last night to see if I could figure out what it’s doing. On RPMs that are already installed it will use up all the CPU/IO for a while, on a small instance it can take a long time. The instance I was testing with was an m1.small, so it’s slow anyway, but for just testing if an RPM is already installed, it’s pretty intense. The what_provides() appeared to be the worst, however I didn’t log the exit time of the function to get a good measurement. I’m also not sure why it would need to call that if I just gave it an RPM name instead of a path to look up. This RPM from an onsite repo cache, and we do run yum clear all” before hand... 2014-02-19T07:21:07.245573+00:00 myserver-01 ansible-yum: Invoked with name=MyRPM list=None disable_gpg_check=False conf_file=None state=latest disablerepo=None enablerepo=None 2014-02-19T07:21:07.245761+00:00 myserver-01 ansible-yum: ensure() MyRPM 2014-02-19T07:21:07.381131+00:00 myserver-01 ansible-yum: latest() ['MyRPM'] 2014-02-19T07:21:07.381283+00:00 myserver-01 ansible-yum: is_installed() MyRPM 2014-02-19T07:21:07.393524+00:00 myserver-01 ansible-yum: what_provides() MyRPM On Feb 19, 2014, at 10:26 , Adam Morris zwac...@gmail.com wrote: I'm manually adding yum-utils in my RedHat installs as I am performing a minimal install. I figured that this was my fault for trying to install as little as possible. It might make some sense to document that dependency in the yum module page though. Adam On Wednesday, February 19, 2014 5:03:18 AM UTC-8, Michael DeHaan wrote: +1 Also, what (remote) OS is this? We'd have this discussion before, where yum-utils we were pretty sure was only excluded in @core installs. That might not b e true though -- need to check. I have no problem making the yum module self-add yum-utils if not already there if it resolves problems in those environments as it should be there anyway. On Wed, Feb 19, 2014 at 6:17 AM, Walid walid@gmail.com wrote: I am away from my Ansible machine and test, however in my playbook the first thing i do is update yum, and yum-utils to the latest update as i had similar issues with older releases. On 19 February 2014 11:59, Marc Trudel mtr...@wizcorp.jp wrote: Quick note. My playbooks break if I do not have repoquery... the code seems to suggest this is optional, but I just found a case, for instance, where checking for an already installed package gave me a recursion error, while another fresh install failed on failed to parse: SUDO-SUCCESS-whatever On Wednesday, February 19, 2014 5:22:07 PM UTC+9, Marc Trudel wrote: I think I found the issue - seems to be related to repoquery Following tests were done as suggested with the test-module on the host With repoquery: real0m21.014s user0m4.094s sys0m1.337s Without repoquery: real0m8.130s user0m1.914s sys0m0.449s I guess it is then no longer an ansible issue (never really were), but has anyone experienced this in the past? On Wednesday, February 19, 2014 10:22:54 AM UTC+9, Michael DeHaan wrote: ./hacking/test-module in the checkout is pretty useful for things like this. Do a checkout on a machine with yum and even inserting some basic print statements or logging could be a useful start to find out what functions or commands are taking the most time. On Tue, Feb 18, 2014 at 7:47 PM, Marc Trudel mtr...@wizcorp.jpwrote: I'll remove fastest-mirror, it indeed looks like it made things slower (this is in fact what I was adding to my stack as an experiment to make YUM faster - at first I thought it was purely YUM-related issue). I will try to find some information as to how to benchmark, but would you have any recommendation as to how I should proceed? On Tuesday, February 18, 2014 9:20:58 PM UTC+9, Michael DeHaan wrote: I see from the above that you said 50 seconds above and I misread. In your case this is definitely slower than the actual command by a very decent margin. I'm still not seeing this. If you can benchmark where it is spending it's time that would be appreciated. I noticed you were installing fastest-mirror though, which you probably don't want to do :) On Tue, Feb
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
I think I found the issue - seems to be related to repoquery Following tests were done as suggested with the test-module on the host With repoquery: real0m21.014s user0m4.094s sys0m1.337s Without repoquery: real0m8.130s user0m1.914s sys0m0.449s I guess it is then no longer an ansible issue (never really were), but has anyone experienced this in the past? On Wednesday, February 19, 2014 10:22:54 AM UTC+9, Michael DeHaan wrote: ./hacking/test-module in the checkout is pretty useful for things like this. Do a checkout on a machine with yum and even inserting some basic print statements or logging could be a useful start to find out what functions or commands are taking the most time. On Tue, Feb 18, 2014 at 7:47 PM, Marc Trudel mtr...@wizcorp.jpjavascript: wrote: I'll remove fastest-mirror, it indeed looks like it made things slower (this is in fact what I was adding to my stack as an experiment to make YUM faster - at first I thought it was purely YUM-related issue). I will try to find some information as to how to benchmark, but would you have any recommendation as to how I should proceed? On Tuesday, February 18, 2014 9:20:58 PM UTC+9, Michael DeHaan wrote: I see from the above that you said 50 seconds above and I misread. In your case this is definitely slower than the actual command by a very decent margin. I'm still not seeing this. If you can benchmark where it is spending it's time that would be appreciated. I noticed you were installing fastest-mirror though, which you probably don't want to do :) On Tue, Feb 18, 2014 at 7:06 AM, Michael DeHaan mic...@ansible.comwrote: It runs some extra ops to ensure it doesn't need to run change-inducing commands up front. However I would disagree that 20% is much slower. Do make sure you have fastest mirror disabled, BTW, the module usually isn't faster. Local mirroring is also always a fantastic idea! Check out yum reposync. On Tue, Feb 18, 2014 at 6:33 AM, Marc Trudel mtr...@wizcorp.jp wrote: I notice that --enablerepos should be --enablerepo - no worries, I tested with the right flags. Also, just to make sure its not ssh related, I also tried: time ssh 123.1322.0.453 sudo yum install yum-presto yum-fastestmirror yum-fast-downloader --enablerepo=personalrepo, rpmforge Which gave me comparable times as with running it straight on the server. On Tuesday, February 18, 2014 8:28:23 PM UTC+9, Marc Trudel wrote: The yum module looks heaps slower than the actual yum command. For instance, when I check if a set of three packages are installed in ansible (timestamps are mine) [19:44:05] TASK: [common | Install presto, fastdownloader and yum-fast-downloader] *** [19:44:56] ok: [someserver] = (item=yum-presto,yum-fastestmi rror,yum-fast-downloader) But if I run: time yum install yum-presto yum-fastestmirror yum-fast-downloader --enablerepos=personalrepo,rpmforge It runs in: First time: real0m7.956s user0m0.829s sys0m0.190s Second time: real0m5.031s user0m1.136s sys0m0.269s If I run the previous command from ansible: [20:27:21] TASK: [common | Install presto, fastdownloader and yum-fast-downloader] *** [20:27:28] changed: [someserver] Any reasons why ansible's yum module run are that much slower? I have tested on 1.4.5. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com. To post to this group, send email to ansible...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com javascript:. To post to this group, send email to ansible...@googlegroups.comjavascript: . For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
Quick note. My playbooks break if I do not have repoquery... the code seems to suggest this is optional, but I just found a case, for instance, where checking for an already installed package gave me a recursion error, while another fresh install failed on failed to parse: SUDO-SUCCESS-whatever On Wednesday, February 19, 2014 5:22:07 PM UTC+9, Marc Trudel wrote: I think I found the issue - seems to be related to repoquery Following tests were done as suggested with the test-module on the host With repoquery: real0m21.014s user0m4.094s sys0m1.337s Without repoquery: real0m8.130s user0m1.914s sys0m0.449s I guess it is then no longer an ansible issue (never really were), but has anyone experienced this in the past? On Wednesday, February 19, 2014 10:22:54 AM UTC+9, Michael DeHaan wrote: ./hacking/test-module in the checkout is pretty useful for things like this. Do a checkout on a machine with yum and even inserting some basic print statements or logging could be a useful start to find out what functions or commands are taking the most time. On Tue, Feb 18, 2014 at 7:47 PM, Marc Trudel mtr...@wizcorp.jp wrote: I'll remove fastest-mirror, it indeed looks like it made things slower (this is in fact what I was adding to my stack as an experiment to make YUM faster - at first I thought it was purely YUM-related issue). I will try to find some information as to how to benchmark, but would you have any recommendation as to how I should proceed? On Tuesday, February 18, 2014 9:20:58 PM UTC+9, Michael DeHaan wrote: I see from the above that you said 50 seconds above and I misread. In your case this is definitely slower than the actual command by a very decent margin. I'm still not seeing this. If you can benchmark where it is spending it's time that would be appreciated. I noticed you were installing fastest-mirror though, which you probably don't want to do :) On Tue, Feb 18, 2014 at 7:06 AM, Michael DeHaan mic...@ansible.comwrote: It runs some extra ops to ensure it doesn't need to run change-inducing commands up front. However I would disagree that 20% is much slower. Do make sure you have fastest mirror disabled, BTW, the module usually isn't faster. Local mirroring is also always a fantastic idea! Check out yum reposync. On Tue, Feb 18, 2014 at 6:33 AM, Marc Trudel mtr...@wizcorp.jpwrote: I notice that --enablerepos should be --enablerepo - no worries, I tested with the right flags. Also, just to make sure its not ssh related, I also tried: time ssh 123.1322.0.453 sudo yum install yum-presto yum-fastestmirror yum-fast-downloader --enablerepo=personalrepo, rpmforge Which gave me comparable times as with running it straight on the server. On Tuesday, February 18, 2014 8:28:23 PM UTC+9, Marc Trudel wrote: The yum module looks heaps slower than the actual yum command. For instance, when I check if a set of three packages are installed in ansible (timestamps are mine) [19:44:05] TASK: [common | Install presto, fastdownloader and yum-fast-downloader] *** [19:44:56] ok: [someserver] = (item=yum-presto,yum-fastestmi rror,yum-fast-downloader) But if I run: time yum install yum-presto yum-fastestmirror yum-fast-downloader --enablerepos=personalrepo,rpmforge It runs in: First time: real0m7.956s user0m0.829s sys0m0.190s Second time: real0m5.031s user0m1.136s sys0m0.269s If I run the previous command from ansible: [20:27:21] TASK: [common | Install presto, fastdownloader and yum-fast-downloader] *** [20:27:28] changed: [someserver] Any reasons why ansible's yum module run are that much slower? I have tested on 1.4.5. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com. To post to this group, send email to ansible...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com. To post to this group, send email to ansible...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
I am away from my Ansible machine and test, however in my playbook the first thing i do is update yum, and yum-utils to the latest update as i had similar issues with older releases. On 19 February 2014 11:59, Marc Trudel mtru...@wizcorp.jp wrote: Quick note. My playbooks break if I do not have repoquery... the code seems to suggest this is optional, but I just found a case, for instance, where checking for an already installed package gave me a recursion error, while another fresh install failed on failed to parse: SUDO-SUCCESS-whatever On Wednesday, February 19, 2014 5:22:07 PM UTC+9, Marc Trudel wrote: I think I found the issue - seems to be related to repoquery Following tests were done as suggested with the test-module on the host With repoquery: real0m21.014s user0m4.094s sys0m1.337s Without repoquery: real0m8.130s user0m1.914s sys0m0.449s I guess it is then no longer an ansible issue (never really were), but has anyone experienced this in the past? On Wednesday, February 19, 2014 10:22:54 AM UTC+9, Michael DeHaan wrote: ./hacking/test-module in the checkout is pretty useful for things like this. Do a checkout on a machine with yum and even inserting some basic print statements or logging could be a useful start to find out what functions or commands are taking the most time. On Tue, Feb 18, 2014 at 7:47 PM, Marc Trudel mtr...@wizcorp.jp wrote: I'll remove fastest-mirror, it indeed looks like it made things slower (this is in fact what I was adding to my stack as an experiment to make YUM faster - at first I thought it was purely YUM-related issue). I will try to find some information as to how to benchmark, but would you have any recommendation as to how I should proceed? On Tuesday, February 18, 2014 9:20:58 PM UTC+9, Michael DeHaan wrote: I see from the above that you said 50 seconds above and I misread. In your case this is definitely slower than the actual command by a very decent margin. I'm still not seeing this. If you can benchmark where it is spending it's time that would be appreciated. I noticed you were installing fastest-mirror though, which you probably don't want to do :) On Tue, Feb 18, 2014 at 7:06 AM, Michael DeHaan mic...@ansible.comwrote: It runs some extra ops to ensure it doesn't need to run change-inducing commands up front. However I would disagree that 20% is much slower. Do make sure you have fastest mirror disabled, BTW, the module usually isn't faster. Local mirroring is also always a fantastic idea! Check out yum reposync. On Tue, Feb 18, 2014 at 6:33 AM, Marc Trudel mtr...@wizcorp.jpwrote: I notice that --enablerepos should be --enablerepo - no worries, I tested with the right flags. Also, just to make sure its not ssh related, I also tried: time ssh 123.1322.0.453 sudo yum install yum-presto yum-fastestmirror yum-fast-downloader --enablerepo=personalrepo,rpmf orge Which gave me comparable times as with running it straight on the server. On Tuesday, February 18, 2014 8:28:23 PM UTC+9, Marc Trudel wrote: The yum module looks heaps slower than the actual yum command. For instance, when I check if a set of three packages are installed in ansible (timestamps are mine) [19:44:05] TASK: [common | Install presto, fastdownloader and yum-fast-downloader] *** [19:44:56] ok: [someserver] = (item=yum-presto,yum-fastestmi rror,yum-fast-downloader) But if I run: time yum install yum-presto yum-fastestmirror yum-fast-downloader --enablerepos=personalrepo,rpmforge It runs in: First time: real0m7.956s user0m0.829s sys0m0.190s Second time: real0m5.031s user0m1.136s sys0m0.269s If I run the previous command from ansible: [20:27:21] TASK: [common | Install presto, fastdownloader and yum-fast-downloader] *** [20:27:28] changed: [someserver] Any reasons why ansible's yum module run are that much slower? I have tested on 1.4.5. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com. To post to this group, send email to ansible...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com. To post to this group, send email to ansible...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group,
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
I just happen to add some crude log traces to my yum module last night to see if I could figure out what it's doing. On RPMs that are already installed it will use up all the CPU/IO for a while, on a small instance it can take a long time. The instance I was testing with was an m1.small, so it's slow anyway, but for just testing if an RPM is already installed, it's pretty intense. The what_provides() appeared to be the worst, however I didn't log the exit time of the function to get a good measurement. I'm also not sure why it would need to call that if I just gave it an RPM name instead of a path to look up. This RPM from an onsite repo cache, and we do run yum clear all before hand... 2014-02-19T07:21:07.245573+00:00 myserver-01 ansible-yum: Invoked with name=MyRPM list=None disable_gpg_check=False conf_file=None state=latest disablerepo=None enablerepo=None 2014-02-19T07:21:07.245761+00:00 myserver-01 ansible-yum: ensure() MyRPM 2014-02-19T07:21:07.381131+00:00 myserver-01 ansible-yum: latest() ['MyRPM'] 2014-02-19T07:21:07.381283+00:00 myserver-01 ansible-yum: is_installed() MyRPM 2014-02-19T07:21:07.393524+00:00 myserver-01 ansible-yum: what_provides() MyRPM On Feb 19, 2014, at 10:26 , Adam Morris zwack...@gmail.com wrote: I'm manually adding yum-utils in my RedHat installs as I am performing a minimal install. I figured that this was my fault for trying to install as little as possible. It might make some sense to document that dependency in the yum module page though. Adam On Wednesday, February 19, 2014 5:03:18 AM UTC-8, Michael DeHaan wrote: +1 Also, what (remote) OS is this? We'd have this discussion before, where yum-utils we were pretty sure was only excluded in @core installs. That might not b e true though -- need to check. I have no problem making the yum module self-add yum-utils if not already there if it resolves problems in those environments as it should be there anyway. On Wed, Feb 19, 2014 at 6:17 AM, Walid walid@gmail.com wrote: I am away from my Ansible machine and test, however in my playbook the first thing i do is update yum, and yum-utils to the latest update as i had similar issues with older releases. On 19 February 2014 11:59, Marc Trudel mtr...@wizcorp.jp wrote: Quick note. My playbooks break if I do not have repoquery... the code seems to suggest this is optional, but I just found a case, for instance, where checking for an already installed package gave me a recursion error, while another fresh install failed on failed to parse: SUDO-SUCCESS-whatever On Wednesday, February 19, 2014 5:22:07 PM UTC+9, Marc Trudel wrote: I think I found the issue - seems to be related to repoquery Following tests were done as suggested with the test-module on the host With repoquery: real0m21.014s user0m4.094s sys0m1.337s Without repoquery: real0m8.130s user0m1.914s sys0m0.449s I guess it is then no longer an ansible issue (never really were), but has anyone experienced this in the past? On Wednesday, February 19, 2014 10:22:54 AM UTC+9, Michael DeHaan wrote: ./hacking/test-module in the checkout is pretty useful for things like this. Do a checkout on a machine with yum and even inserting some basic print statements or logging could be a useful start to find out what functions or commands are taking the most time. On Tue, Feb 18, 2014 at 7:47 PM, Marc Trudel mtr...@wizcorp.jp wrote: I'll remove fastest-mirror, it indeed looks like it made things slower (this is in fact what I was adding to my stack as an experiment to make YUM faster - at first I thought it was purely YUM-related issue). I will try to find some information as to how to benchmark, but would you have any recommendation as to how I should proceed? On Tuesday, February 18, 2014 9:20:58 PM UTC+9, Michael DeHaan wrote: I see from the above that you said 50 seconds above and I misread. In your case this is definitely slower than the actual command by a very decent margin. I'm still not seeing this. If you can benchmark where it is spending it's time that would be appreciated. I noticed you were installing fastest-mirror though, which you probably don't want to do :) On Tue, Feb 18, 2014 at 7:06 AM, Michael DeHaan mic...@ansible.com wrote: It runs some extra ops to ensure it doesn't need to run change-inducing commands up front. However I would disagree that 20% is much slower. Do make sure you have fastest mirror disabled, BTW, the module usually isn't faster. Local mirroring is also always a fantastic idea! Check out yum reposync. On Tue, Feb 18, 2014 at 6:33 AM, Marc Trudel mtr...@wizcorp.jp wrote: I notice that --enablerepos should be --enablerepo - no worries, I tested with the right flags. Also, just to make sure its not ssh related, I also tried: time ssh 123.1322.0.453 sudo yum install yum-presto
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
@cove_s nice :) I didn't get to go down that much, but that reflects pretty well what I am experiencing. @Adam @Michael at least for updates, NOT using repoquery made things faster for me. What I did is change the code for the yum module to undefine the repoquery path. On Thursday, February 20, 2014 3:51:00 AM UTC+9, cove_s wrote: I just happen to add some crude log traces to my yum module last night to see if I could figure out what it’s doing. On RPMs that are already installed it will use up all the CPU/IO for a while, on a small instance it can take a long time. The instance I was testing with was an m1.small, so it’s slow anyway, but for just testing if an RPM is already installed, it’s pretty intense. The what_provides() appeared to be the worst, however I didn’t log the exit time of the function to get a good measurement. I’m also not sure why it would need to call that if I just gave it an RPM name instead of a path to look up. This RPM from an onsite repo cache, and we do run yum clear all” before hand... 2014-02-19T07:21:07.245573+00:00 myserver-01 ansible-yum: Invoked with name=MyRPM list=None disable_gpg_check=False conf_file=None state=latest disablerepo=None enablerepo=None 2014-02-19T07:21:07.245761+00:00 myserver-01 ansible-yum: ensure() MyRPM 2014-02-19T07:21:07.381131+00:00 myserver-01 ansible-yum: latest() ['MyRPM'] 2014-02-19T07:21:07.381283+00:00 myserver-01 ansible-yum: is_installed() MyRPM 2014-02-19T07:21:07.393524+00:00 myserver-01 ansible-yum: what_provides() MyRPM On Feb 19, 2014, at 10:26 , Adam Morris zwac...@gmail.com javascript: wrote: I'm manually adding yum-utils in my RedHat installs as I am performing a minimal install. I figured that this was my fault for trying to install as little as possible. It might make some sense to document that dependency in the yum module page though. Adam On Wednesday, February 19, 2014 5:03:18 AM UTC-8, Michael DeHaan wrote: +1 Also, what (remote) OS is this? We'd have this discussion before, where yum-utils we were pretty sure was only excluded in @core installs. That might not b e true though -- need to check. I have no problem making the yum module self-add yum-utils if not already there if it resolves problems in those environments as it should be there anyway. On Wed, Feb 19, 2014 at 6:17 AM, Walid walid@gmail.com wrote: I am away from my Ansible machine and test, however in my playbook the first thing i do is update yum, and yum-utils to the latest update as i had similar issues with older releases. On 19 February 2014 11:59, Marc Trudel mtr...@wizcorp.jp wrote: Quick note. My playbooks break if I do not have repoquery... the code seems to suggest this is optional, but I just found a case, for instance, where checking for an already installed package gave me a recursion error, while another fresh install failed on failed to parse: SUDO-SUCCESS-whatever On Wednesday, February 19, 2014 5:22:07 PM UTC+9, Marc Trudel wrote: I think I found the issue - seems to be related to repoquery Following tests were done as suggested with the test-module on the host With repoquery: real0m21.014s user0m4.094s sys0m1.337s Without repoquery: real0m8.130s user0m1.914s sys0m0.449s I guess it is then no longer an ansible issue (never really were), but has anyone experienced this in the past? On Wednesday, February 19, 2014 10:22:54 AM UTC+9, Michael DeHaan wrote: ./hacking/test-module in the checkout is pretty useful for things like this. Do a checkout on a machine with yum and even inserting some basic print statements or logging could be a useful start to find out what functions or commands are taking the most time. On Tue, Feb 18, 2014 at 7:47 PM, Marc Trudel mtr...@wizcorp.jpwrote: I'll remove fastest-mirror, it indeed looks like it made things slower (this is in fact what I was adding to my stack as an experiment to make YUM faster - at first I thought it was purely YUM-related issue). I will try to find some information as to how to benchmark, but would you have any recommendation as to how I should proceed? On Tuesday, February 18, 2014 9:20:58 PM UTC+9, Michael DeHaan wrote: I see from the above that you said 50 seconds above and I misread. In your case this is definitely slower than the actual command by a very decent margin. I'm still not seeing this. If you can benchmark where it is spending it's time that would be appreciated. I noticed you were installing fastest-mirror though, which you probably don't want to do :) On Tue, Feb 18, 2014 at 7:06 AM, Michael DeHaan mic...@ansible.com wrote: It runs some extra ops to ensure it doesn't need to run change-inducing commands up front. However I would disagree that 20% is much slower. Do make sure you have fastest mirror disabled, BTW, the module usually isn't faster. Local
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
It runs some extra ops to ensure it doesn't need to run change-inducing commands up front. However I would disagree that 20% is much slower. Do make sure you have fastest mirror disabled, BTW, the module usually isn't faster. Local mirroring is also always a fantastic idea! Check out yum reposync. On Tue, Feb 18, 2014 at 6:33 AM, Marc Trudel mtru...@wizcorp.jp wrote: I notice that --enablerepos should be --enablerepo - no worries, I tested with the right flags. Also, just to make sure its not ssh related, I also tried: time ssh 123.1322.0.453 sudo yum install yum-presto yum-fastestmirror yum-fast-downloader --enablerepo=personalrepo,rpmforge Which gave me comparable times as with running it straight on the server. On Tuesday, February 18, 2014 8:28:23 PM UTC+9, Marc Trudel wrote: The yum module looks heaps slower than the actual yum command. For instance, when I check if a set of three packages are installed in ansible (timestamps are mine) [19:44:05] TASK: [common | Install presto, fastdownloader and yum-fast-downloader] *** [19:44:56] ok: [someserver] = (item=yum-presto,yum- fastestmirror,yum-fast-downloader) But if I run: time yum install yum-presto yum-fastestmirror yum-fast-downloader --enablerepos=personalrepo,rpmforge It runs in: First time: real0m7.956s user0m0.829s sys0m0.190s Second time: real0m5.031s user0m1.136s sys0m0.269s If I run the previous command from ansible: [20:27:21] TASK: [common | Install presto, fastdownloader and yum-fast-downloader] *** [20:27:28] changed: [someserver] Any reasons why ansible's yum module run are that much slower? I have tested on 1.4.5. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [ansible-project] Re: YUM: Much slower in ansible than on the cli
I see from the above that you said 50 seconds above and I misread. In your case this is definitely slower than the actual command by a very decent margin. I'm still not seeing this. If you can benchmark where it is spending it's time that would be appreciated. I noticed you were installing fastest-mirror though, which you probably don't want to do :) On Tue, Feb 18, 2014 at 7:06 AM, Michael DeHaan mich...@ansible.com wrote: It runs some extra ops to ensure it doesn't need to run change-inducing commands up front. However I would disagree that 20% is much slower. Do make sure you have fastest mirror disabled, BTW, the module usually isn't faster. Local mirroring is also always a fantastic idea! Check out yum reposync. On Tue, Feb 18, 2014 at 6:33 AM, Marc Trudel mtru...@wizcorp.jp wrote: I notice that --enablerepos should be --enablerepo - no worries, I tested with the right flags. Also, just to make sure its not ssh related, I also tried: time ssh 123.1322.0.453 sudo yum install yum-presto yum-fastestmirror yum-fast-downloader --enablerepo=personalrepo,rpmforge Which gave me comparable times as with running it straight on the server. On Tuesday, February 18, 2014 8:28:23 PM UTC+9, Marc Trudel wrote: The yum module looks heaps slower than the actual yum command. For instance, when I check if a set of three packages are installed in ansible (timestamps are mine) [19:44:05] TASK: [common | Install presto, fastdownloader and yum-fast-downloader] *** [19:44:56] ok: [someserver] = (item=yum-presto,yum- fastestmirror,yum-fast-downloader) But if I run: time yum install yum-presto yum-fastestmirror yum-fast-downloader --enablerepos=personalrepo,rpmforge It runs in: First time: real0m7.956s user0m0.829s sys0m0.190s Second time: real0m5.031s user0m1.136s sys0m0.269s If I run the previous command from ansible: [20:27:21] TASK: [common | Install presto, fastdownloader and yum-fast-downloader] *** [20:27:28] changed: [someserver] Any reasons why ansible's yum module run are that much slower? I have tested on 1.4.5. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Ansible Project group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
