On Fri, Jun 30, 2017 at 2:14 AM, SCHAER Frederic <frederic.sch...@cea.fr> wrote:
> Hi, > > > > I was (up to now) mirroring the puppetlabs repositories to both : > > - Make sure I have a local copy in case your repos are down, or > our internet link is too weak > > - Not hammer on your infrastructure with our servers > Much appreciated! > Unfortunately, we just noticed our mirroring suddenly got broken just > after your changes. > > We use lftp, but I just tried with a recursive wget and see the same > issue, depending on the max depth I ask. > > > > What I see after a simple wget –r –l1 yum.puppetlabs.com is this : > > > > # ll yum.puppetlabs.com/el/6Server/ > > total 28 > > -rw-r--r-- 1 root root 1635 Jun 28 20:01 dependencies > > -rw-r--r-- 1 root root 1603 Jun 28 20:01 devel > > -rw-r--r-- 1 root root 1832 Jun 28 20:01 index_by_lastModified.html > > -rw-r--r-- 1 root root 1824 Jun 28 20:01 index_by_name_reverse.html > > -rw-r--r-- 1 root root 1832 Jun 28 20:01 index_by_size.html > > -rw-r--r-- 1 root root 1592 Jun 28 20:01 PC1 > > -rw-r--r-- 1 root root 1618 Jun 28 20:01 products > > > > Symlinks or directories now are files, and those are html files with > relative hrefs to the corresponding files/directories > Symlinks don't exist anymore because we're using S3 as the backend to CloudFront, and S3 has no concept of symlinks. We're still shipping symlinks to the rsync servers, so if you "rsync -a rsync:// rsync.puppet.com/packages/yum /tmp/yum" you'll get the symlinks, which is a lot more space efficient than mirroring the fully-resolved filesystems via HTTPS. > > > Is there a « now recommended » way of mirroring the repos ? > > (I’ve tried the lftp –dereference option to no avail…) > Each of the following should work: Rsync: rsync -a rsync://rsync.puppet.com/packages/yum /tmp/yum Directly from S3: aws s3 sync --exclude '*.html' s3://yum.puppetlabs.com /tmp/yum HTTPS via CloudFront: wget -r https://yum.puppetlabs.com Of them, rsync and aws s3 sync are both incremental, so they'll be more bandwidth efficient and faster compared to using wget. If any of those options aren't working, I'd like to know about it. I'm seeing the same behavior you described when wget max depth is set, but I don't understand the use case where you'd limit depth on it; it seems like you'd end up with a broken repository if you didn't sync down pretty much everything. > > > I see rsync is still sort of supported, but I have the impression you > would prefer that we use your new CDN (hence, get the files through > cloudfront using http …) > rsync is absolutely still supported. After we discontinued it last time and got a lot of pushback, I've been working on an HA rsync. > > > Thanks && regards > > Frederic Schaer > > > > *De :* puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] > *De la part de* Daniel Dreier > *Envoyé :* mercredi 28 juin 2017 20:26 > *À :* puppet-users@googlegroups.com > *Objet :* [Puppet Users] apt, yum, downloads, and rsync infrastructure > improvements > > > > Today we're making improvements to apt.puppetlabs.com, yum.puppetlabs.com, > and downloads.puppetlabs.com. I don't anticipate any user-visible > changes, and this notification is just to let folks know so that you can > report problems to me. > > > > Specifically, we're switching out the CDN backend - it's been running off > a server in Linode, and we're switching to an S3 backend, which will be > faster and more reliable. > > > > If you see any problems with the apt/yum download infrastructure, please > let me know. > > > > -- > > Daniel Dreier > > Technical Operations Engineer > > GPG: BA4379FD > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/puppet-users/CAGk8subLmT0QDSh79Jd9KOCxfzRXO > TqK4i0Padvhk3Abq0iQTg%40mail.gmail.com > <https://groups.google.com/d/msgid/puppet-users/CAGk8subLmT0QDSh79Jd9KOCxfzRXOTqK4i0Padvhk3Abq0iQTg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/puppet-users/CAB586860327CB498EF79903967FEA2317B21A78%40E-EXDAGE-A0. > extra.cea.fr > <https://groups.google.com/d/msgid/puppet-users/CAB586860327CB498EF79903967FEA2317B21A78%40E-EXDAGE-A0.extra.cea.fr?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Daniel Dreier Technical Operations Engineer GPG: BA4379FD -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAGk8suZaso1PSeQUvKQ3QYKRuPhGUagBCgiOEQJmKR-eGRBFFQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
