Re: Share /var/cache/pkg/ between machines
On Tue, Feb 26, 2013 at 3:57 AM, Aristedes Maniatis wrote: > I'd like to share packages between a couple of nearly identical machines in > a server farm. I think I have the following options: > > 1. Set up apache httpd on one primary machine to serve the packages to the > others by pointing website root to to /var/cache/pkg/ and setting > PACKAGESITE in the other servers. This looks like it might work except that > repo.txz is missing from /var/cache/pkg/ > > 2. rsync /var/cache/pkg/ from the primary machine to the others. Set > PACKAGESITE on all machines to point to some central repository where all > these packages originally were built (we run poudriere in another location). > > 3. Something else > > > How do other people cache/proxy built packages under pkgng? I don't want to > have to pull the same 80Mb JDK package onto 10 machines across the internet. > > > Thanks > > Ari > > One option is to use poudriere, and set WITH_PKGNG=yes in /usr/local/etc/poudriere.d/make.conf (or the appropriate one for one specific jail, if you have multiple). That should make it build pkgng-style packages, including a repo.txz - file. Then host the corresponding package directory with some http server, and set it as a pkgng packagesite on the others. I was playing with this a few weeks ago, and it took a few tries (including more than one "why doesn't it create a repo.txz") before I got it. -- Daniel Nebdal ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Share /var/cache/pkg/ between machines
On 26/02/13 5:22pm, Alexander Yerenkow wrote: I don't understand how do you imagine "magical appearing" 80 Mb JDK from any one place to other without pulling data from internet? :) In any case, with any kind of sharing you probably will have almost same traffic, maybe even greater than if you choose simple `pkg install` traffic. Perhaps I oversimplified our setup. Here is what we have: - pkg.office.example.com (poudriere): builds all packages (pkgng). Runs apache to serve repository to other servers. -- server1.office.example.com pulls files from build server (PACKAGESITE points to pkg.office.example.com) -- server2.office.example.com, etc.. as above office <-- VPN tunnel --> colo (data centre) - pkg.colo.example.com also pulls files from build server (PACKAGESITE points to pkg.office.example.com) -- server[2-10].colo.example.com... rather than pull packages over the slow VPN tunnel, I want to pull them from pkg.colo.example.com Now, one approach is to rsync the poudriere output folder on pkg.office.example.com to pkg.colo.example.com. And given the complexity of other options, that's probably what I'll do. It means more network traffic than optimal, because poudriere will bulk many packages which aren't actually needed in the colo (and more often than the colo machines are actually updated as well since poudriere in the office might do a run twice a week, but the colo is updated once every 3 months or when there is a security patch we care about). So I was hoping to rely on "pkg install" to pull the package over the VPN network, and then replicate the data out to the other servers (NFS/rsync/etc) in the colo. Sorry for confusing you with my abbreviated problem description. Hopefully this is clearer. Ari 1 - pointless, you can set up simple nginx, pointing to poudriere build packages directory and set up packagesite in /u/l/e/pkg.conf, without any proxy machine, which must download these cached packages in any case to be served by apache. AFAIK that machine will download something only while installing, right? So you overcomplicate your setup. 2. rsync will produce packagesize traffic + constant little more, due to checks of alteration. You'll lose in that case in long shot. My advice - put nginx to serve poudriere's built packages dir, and trust to xz compression level = easy to setup, not big traffic really. This is of course based on sentence "... onto 10 machines across the internet" - if you have some clusters with PCs in some nets, then you could think about rsync with primary poudriere, to serve locally packages for few other PCs. Hope this help :) -- Regards, Alexander Yerenkow -- --> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Share /var/cache/pkg/ between machines
I don't understand how do you imagine "magical appearing" 80 Mb JDK from any one place to other without pulling data from internet? :) In any case, with any kind of sharing you probably will have almost same traffic, maybe even greater than if you choose simple `pkg install` traffic. 1 - pointless, you can set up simple nginx, pointing to poudriere build packages directory and set up packagesite in /u/l/e/pkg.conf, without any proxy machine, which must download these cached packages in any case to be served by apache. AFAIK that machine will download something only while installing, right? So you overcomplicate your setup. 2. rsync will produce packagesize traffic + constant little more, due to checks of alteration. You'll lose in that case in long shot. My advice - put nginx to serve poudriere's built packages dir, and trust to xz compression level = easy to setup, not big traffic really. This is of course based on sentence "... onto 10 machines across the internet" - if you have some clusters with PCs in some nets, then you could think about rsync with primary poudriere, to serve locally packages for few other PCs. Hope this help :) -- Regards, Alexander Yerenkow ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
RE: Share /var/cache/pkg/ between machines
> -Original Message- > From: owner-freebsd-po...@freebsd.org > [mailto:owner-freebsd-po...@freebsd.org] On Behalf Of > Aristedes Maniatis > Sent: Tuesday, 26 February 2013 1:58 PM > To: freebsd-ports@freebsd.org > Subject: Share /var/cache/pkg/ between machines > > I'd like to share packages between a couple of nearly > identical machines in a server farm. I think I have the > following options: > > 1. Set up apache httpd on one primary machine to serve the > packages to the others by pointing website root to to > /var/cache/pkg/ and setting PACKAGESITE in the other servers. > This looks like it might work except that repo.txz is missing > from /var/cache/pkg/ > > 2. rsync /var/cache/pkg/ from the primary machine to the > others. Set PACKAGESITE on all machines to point to some > central repository where all these packages originally were > built (we run poudriere in another location). > > 3. Something else > > > How do other people cache/proxy built packages under pkgng? I > don't want to have to pull the same 80Mb JDK package onto 10 > machines across the internet. > > > Thanks > > Ari > > > > -- > --> > Aristedes Maniatis > ish > http://www.ish.com.au > Level 1, 30 Wilson Street Newtown 2042 Australia > phone +61 2 9550 5001 fax +61 2 9550 4001 > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 > 102A ___ > freebsd-ports@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to > "freebsd-ports-unsubscr...@freebsd.org" Ari, We've been doing 1. for a few years, including remote servers. The pkg_ suite suits our needs and will be retained until retired. To reduce package sizes, we remove content that isn't required, such as: doc, include files, examples, content of share etc. But this requires effort to get right and may not suit your needs. Portmaster maintains the ports collection. Regards, Dewayne. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Share /var/cache/pkg/ between machines
You could put /var/cache/pkg on NFS, writable by all clients. However, my preference would be to put the pkg repo on NFS and not use cache at all. At the moment I clean out cache daily via periodic(8). -- James. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Share /var/cache/pkg/ between machines
Just to add a 'something else', unsure how fully it may suffice... [details at the bottom] --- On Mon, 2/25/13, Aristedes Maniatis wrote: From: Aristedes Maniatis Subject: Share /var/cache/pkg/ between machines To: freebsd-ports@freebsd.org Date: Monday, February 25, 2013, 6:57 PM I'd like to share packages between a couple of nearly identical machines in a server farm. I think I have the following options: 1. Set up apache httpd on one primary machine to serve the packages to the others by pointing website root to to /var/cache/pkg/ and setting PACKAGESITE in the other servers. This looks like it might work except that repo.txz is missing from /var/cache/pkg/ 2. rsync /var/cache/pkg/ from the primary machine to the others. Set PACKAGESITE on all machines to point to some central repository where all these packages originally were built (we run poudriere in another location). 3. Something else How do other people cache/proxy built packages under pkgng? I don't want to <>-- --> >>Aristedes Maniatis portmaster? [Cannot directly answer the post question, but...] If you put /portmatster-download/ on /da0 (a thumbdrive) mount -t unionfs /dev/da0 /usr/ports/packages... then the thumbdrive packages will appear to be already downloaded to portmaster, for migrating between machines. [I set up an ftp server for similar functionality, but find this method quicker and reconfigurable. YMMV of course depending upon the number/physical placment of your servers] Sorry to not answer about /pkg/, not fully implemented on most machines, here. [portmaster -d -B -P -i -g category/port category/port... or scripted equivalent in pkg or shell syntax.] I've seen it handily upgrade thirty p5 ports at a stretch using a pipe... just because a thumbdrive was in place, where otherwise it would mean duplicate builds etc. J. Bouquet ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Share /var/cache/pkg/ between machines
I'd like to share packages between a couple of nearly identical machines in a server farm. I think I have the following options: 1. Set up apache httpd on one primary machine to serve the packages to the others by pointing website root to to /var/cache/pkg/ and setting PACKAGESITE in the other servers. This looks like it might work except that repo.txz is missing from /var/cache/pkg/ 2. rsync /var/cache/pkg/ from the primary machine to the others. Set PACKAGESITE on all machines to point to some central repository where all these packages originally were built (we run poudriere in another location). 3. Something else How do other people cache/proxy built packages under pkgng? I don't want to have to pull the same 80Mb JDK package onto 10 machines across the internet. Thanks Ari -- --> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
