The second time it runs (with pluginsync enabled), it only pauses at
the "Info: Retrieving plugin" notice for a few seconds. So, it sounds like
the md5sum is not the bottleneck.
The specific setup that I'm using is a pretty beefy Xeon workstation with
24 GB of RAM acting as the puppetmaster for itself and all of the local
vagrant VMs. It is running webrick, which I hadn't thought would be a
problem, since it's only serving to itself and 1-6 VMs. During "vagrant
up", I make sure to do only one machine at at time, so that I can watch
them go by. So, it is only really doing any heavy lifting for one VM at a
time.
We do use passenger on our production system, so I'll be happy to try
passenger on my workstation. I'll let you know if I see any improvement.
On Monday, January 7, 2013 2:01:00 PM UTC-5, Ken Barber wrote:
>
> There are two primary bottlenecks for pluginsync:
>
> * the md5 sum it needs to do on both ends to compare and see if a
> resync is needed
> * the transfer itself
>
> If you run puppet a second time, with pluginsync enabled (but this
> time nothing should be transferred) - how much time does it take
> compared to the first run? I'm only curious as it might rule out
> transfer versus the 'md5' operation. *shrug*. Alas though, I haven't
> seen this specific problem myself - are you running the puppetmaster
> using webrick or some sort of application server like 'passenger' or
> 'unicorn'? Its a long-shot, but webrick is single-threaded which has
> been known to be a problem - alas it shouldn't be a problem during a
> sequential pluginsync :-(.
>
> On Mon, Jan 7, 2013 at 6:28 PM, Kirk Steffensen
> <ki...@steffensenfamily.com <javascript:>> wrote:
> > Ken,
> >
> > Thanks. I agree with your gut feeling, but after running the first
> round of
> > tests you suggested, I don't think that's it. Bandwidth is good and
> there
> > are no netstat errors. (Test results at the end of this email.)
> >
> > Actually, I just realized that the Puppet client on the Ubuntu box is
> > running Puppet 2.7.11, which does not have pluginsync set to true by
> default
> > (The CentOS boxes have Puppet 3.0.2, which has pluginsync set to true by
> > default.) I watched the Ubuntu box start up, and it didn't give me a
> "Info:
> > Retrieving plugin" notice like the CentOS box does. When I changed
> > pluginsync to true on the Ubuntu box, it also took many minutes to sync
> the
> > stdlib module. (I didn't time it during this run, but it was about 10
> > minutes. I'd be happy to actually time it, if it would help
> troubleshoot.)
> >
> > While I didn't time the entire stdlib sync on the Ubuntu box, I did time
> > several of the individual files. It's taking about 10 seconds per file
> to
> > copy it over to the client from the Puppet Master. Since there are 90
> files
> > in stdlib, it is taking many minutes to sync the plugin, even though
> there
> > are only 852KB of total data. Each file is resulting in a Puppet notice
> > like the following output:
> >
> > notice:
> > /File[/var/lib/puppet/lib/puppet/parser/functions/str2bool.rb]/ensure:
> > defined content as '{md5}ab045013031d01a0e9335af92580dde6'
> >
> > For my short-term needs, my solution is to change pluginsync to false on
> the
> > CentOS boxes, since I'm only using the stdlib module to get the anchor
> > definition, and that doesn't need to be on the client side. But this
> will
> > not be a long-term solution, as I will probably end up developing
> modules
> > that need pluginsync to be true.
> >
> > Is anyone else who is using stdlib with pluginsync = true seeing this
> same
> > 13-14 minute sync time during the first puppet run on a machine?
> >
> > Here are the results of the tests:
> >
> > iperf on CentOS: 298 Mbits/sec
> > iperf on Ubuntu: 2.55 Gbits/sec
> >
> > $ netstat -in [on CentOS - no errors]
> > Kernel Interface table
> > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP
> TX-OVR
> > Flg
> > eth0 1500 0 4813366 0 0 0 3076279 0 0
> > 0 BMRU
> > lo 16436 0 1140661 0 0 0 1140661 0 0
> > 0 LRU
> > vboxnet0 1500 0 0 0 0 0 866 0 0
> > 0 BMRU
> >
> > The VBoxGuestAdditions are added in the basebox generation (using
> veewee,
> > another great tool!) with the following in postinstall.sh. This is
> > resulting in the most current (and more importantly, the matching
> version to
> > the host):
> >
> > ***********
> > # Install vagrant keys
> > mkdir /home/vagrant/.ssh
> > chmod 700 /home/vagrant/.ssh
> > cd /home/vagrant/.ssh
> > wget --no-check-certificate
> > 'https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub' -O
> > authorized_keys
> > chown -R vagrant /home/vagrant/.ssh
> >
> > # Install the virtualbox guest additions
> > VBOX_VERSION=$(cat /home/vagrant/.vbox_version)
> > cd /tmp
> > wget
> >
> http://download.virtualbox.org/virtualbox/$VBOX_VERSION/VBoxGuestAdditions_$VBOX_VERSION.iso
>
> > mount -o loop VBoxGuestAdditions_$VBOX_VERSION.iso /mnt
> > sh /mnt/VBoxLinuxAdditions.run
> > umount /mnt
> >
> > rm VBoxGuestAdditions_$VBOX_VERSION.iso
> > ***********
> >
> > Thanks,
> > Kirk
> >
> >
> > On Monday, January 7, 2013 12:20:34 PM UTC-5, Ken Barber wrote:
> >>
> >> My immediate gut feeling on this would be network not Puppet being the
> >> issue actually, especially if another client is having success at
> >> doing the sync.
> >>
> >> Its a virt so it could be hypervisor drivers or some other issue, its
> >> an old version of the kernel as well - its more likely to happen -
> >> although I haven't seen it myself lately :-). Run something like
> >> 'iperf' on both ends to confirm your getting the right kind of network
> >> performance, compare that performance to your Ubuntu target ... and
> >> check out netstat -in for errors etc. etc.
> >>
> >> Check to make sure the right VirtualBox hypervisor drivers were
> >> installed when building your Centos vagrant box btw
> >> (VBoxGuestAdditions) ... I'm not sure of its origin but this can be an
> >> important step for network performance.
> >>
> >>
> >> On Mon, Jan 7, 2013 at 4:50 PM, Kirk Steffensen
> >> <ki...@steffensenfamily.com> wrote:
> >> > Hi,
> >> >
> >> > I have a fresh CentOS 5.8 Vagrant VM that I'm using to emulate a
> >> > customer's
> >> > server. During the first Puppet run, it takes 13 minutes and 48
> seconds
> >> > to
> >> > sync the Puppet Labs stdlib module. On a similar Ubuntu 12.04.1
> Vagrant
> >> > VM,
> >> > Puppet starts up, and almost instantly goes from plugin sync to
> facter
> >> > load.
> >> >
> >> > Is this load time for stdlib on a RHEL variant normal? It's only
> 580K
> >> > of
> >> > data to transfer. It seems like it should be almost instantaneous,
> even
> >> > on
> >> > an older kernel like CentOS 5.8.
> >> >
> >> > I've posted the relevant part of the Puppet run's output to this
> gist:
> >> > https://gist.github.com/4475110
> >> >
> >> > If anyone has any insight or any troubleshooting steps to try, I'd
> >> > appreciate it.
> >> >
> >> > Thanks,
> >> > Kirk
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "Puppet Users" group.
> >> > To view this discussion on the web visit
> >> > https://groups.google.com/d/msg/puppet-users/-/jH71g1du7icJ.
> >> > To post to this group, send email to puppet...@googlegroups.com.
> >> > To unsubscribe from this group, send email to
> >> > puppet-users...@googlegroups.com.
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/puppet-users?hl=en.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "Puppet Users" group.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msg/puppet-users/-/gdGOknbXSW0J.
> >
> > To post to this group, send email to
> > puppet...@googlegroups.com<javascript:>.
>
> > To unsubscribe from this group, send email to
> > puppet-users...@googlegroups.com <javascript:>.
> > For more options, visit this group at
> > http://groups.google.com/group/puppet-users?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/Fw2yD-CqBPsJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
