Re: [Users] New vzpkg, templates and package caching daemon
Robert Nelson wrote: Kir Kolyshkin wrote: A few more questions to Robert (sorry for top-posting) 1. Can you explain how you deal with chicken-and-egg problem? I mean, to run yum/rpm inside a VE you need to have those packages (and their deps) inside the VE. In old vzpkg, we use external rpm/yum (vzrpm and vzyum) to do that. Since you are not relying on that anymore, how do you perform an initial installation of rpm/yum/their dependencies in a newly created VE? For creating the cache I use the standard yum/rpm on the HN. Once a CT is created the yum/rpm installed in it is used. Hmm. Here comes a problem: try to create CentOS 4 (which uses rpm-4.3) template on a host system running CentOS 5 (rpm-4.4). Not really sure how to deal with that in a less hackish manner. ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users
Re: [Users] New vzpkg, templates and package caching daemon
Kir Kolyshkin wrote: Robert Nelson wrote: Kir Kolyshkin wrote: A few more questions to Robert (sorry for top-posting) 1. Can you explain how you deal with chicken-and-egg problem? I mean, to run yum/rpm inside a VE you need to have those packages (and their deps) inside the VE. In old vzpkg, we use external rpm/yum (vzrpm and vzyum) to do that. Since you are not relying on that anymore, how do you perform an initial installation of rpm/yum/their dependencies in a newly created VE? For creating the cache I use the standard yum/rpm on the HN. Once a CT is created the yum/rpm installed in it is used. Hmm. Here comes a problem: try to create CentOS 4 (which uses rpm-4.3) template on a host system running CentOS 5 (rpm-4.4). Not really sure how to deal with that in a less hackish manner. That is part of my standard test matrix, works fine. I test creation of templates and CT on both CentOS 5 HN and a Debian Etch HN for all of the following: CentOS 4, i386 and x86_64 CentOS 5, i386 and x86_64 Fedora 7, i386 and x86_64 Fedora 8, i386 and x86_64 Fedora 9, i386 and x86_64 Debian Sarge, i386 Debian Etch, i386 and amd64 As I said earlier, the only problem between rpm 4.3 and 4.4 is the _db.00? files and they are automatically created if missing the first time rpm is executed. Once the templates are created and before doing the tar I delete the files. ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users
Re: [Users] New vzpkg, templates and package caching daemon
A few more questions to Robert (sorry for top-posting) 1. Can you explain how you deal with chicken-and-egg problem? I mean, to run yum/rpm inside a VE you need to have those packages (and their deps) inside the VE. In old vzpkg, we use external rpm/yum (vzrpm and vzyum) to do that. Since you are not relying on that anymore, how do you perform an initial installation of rpm/yum/their dependencies in a newly created VE? (My own solution to this is to have a list of URLs to a few packages comprising a minimal system in which rpm could work, and then download/unpack (using rpm2cpio | cpio -id) those into a newly created empty container. When we can run rpm --initdb inside and do 'yum install full list of packages. The bad thing about that is (per-distro per-version list of) hard-coded package names and inability to use packages from updates repo since they are ever-changing (but packages in base repo should be OK). 2. Is it possible to have opensuse template metadata? AFAIK opensuse lacks yum and you (YaST Online Updater) should be used instead. Robert Nelson wrote: Roberto Mello wrote: On Fri, Sep 5, 2008 at 9:27 PM, Robert Nelson [EMAIL PROTECTED] wrote: I have a preliminary version of the new vzpkg utilities ready for testing. These new tools support creating templates for 32 and 64 bit flavours of the following: Centos 4 and 5, Fedora 7, 8 and 9, Debian Sarge and Etch, Ubuntu Feisty, Gutsy and Hardy. They are extensible and will eventually support OpenSUSE and Gentoo. Fantastic! Thanks for doing this and for sharing. Will definitely be taking a look at this. In addition, I've created pkg-cacher, a transparent caching proxy daemon optimized for Debian and RPM packages. It is based on apt-cacher version 1.6.4 available with Debian. This version has been modified to understand RedHat RPM repositiories. It also adds support for the HTTP Range header and deals correctly with files which have the same name but different content on different distributions or in some cases different versions of the same distribution. In my experience I've found that just setting up Squid and telling my different machines to use the proxy (Acquire::http::Proxy http://proxy.hostname:3128/;;) has been the best pacakge caching solution. Works accross different packaging systems too. There are a few advantages of using pkg-cacher versus squid: It understands the difference between packages (static content) and metadata files (dynamic content). It only keeps a single copy of a package even if it is shared across multiple distributions. versions or accessed from multiple mirrors. Removal of stale packages can be based on whether the package is referenced by any repositories using the metadata. This means less downloads and less disk usage. Even a 250 GB disk starts looking small once you deal with multiple distros, versions and mirrors :-) In order to make installation easy I've setup repositories for 32 and 64 bit versions of CentOS 4, 5 and Fedora 7, 8 and 9. I'll be producing packages for Debian and Ubuntu as HN later. I can help with Debian packages if you need. Roberto Thanks for the offer, the only reason I didn't release it at this time is I didn't have a Debian HN set up for testing yet. Should be available in the next day or so. ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users
Re: [Users] New vzpkg, templates and package caching daemon
Kir Kolyshkin wrote: A few more questions to Robert (sorry for top-posting) 1. Can you explain how you deal with chicken-and-egg problem? I mean, to run yum/rpm inside a VE you need to have those packages (and their deps) inside the VE. In old vzpkg, we use external rpm/yum (vzrpm and vzyum) to do that. Since you are not relying on that anymore, how do you perform an initial installation of rpm/yum/their dependencies in a newly created VE? For creating the cache I use the standard yum/rpm on the HN. Once a CT is created the yum/rpm installed in it is used. (My own solution to this is to have a list of URLs to a few packages comprising a minimal system in which rpm could work, and then download/unpack (using rpm2cpio | cpio -id) those into a newly created empty container. When we can run rpm --initdb inside and do 'yum install full list of packages. The bad thing about that is (per-distro per-version list of) hard-coded package names and inability to use packages from updates repo since they are ever-changing (but packages in base repo should be OK). 2. Is it possible to have opensuse template metadata? AFAIK opensuse lacks yum and you (YaST Online Updater) should be used instead. I'm working on that now :-) OpenSUSE uses rpms and supports yum, but their command line tool is zypper, gui is YaST. Both YaST and zypper use yum/rpm compatible repositories. The problem I've hit is a slight difference in version checking. If you have a requirement that is EQ and it doesn't include a revision just a version then zypper accepts any revision yum barfs. Robert Nelson wrote: Roberto Mello wrote: On Fri, Sep 5, 2008 at 9:27 PM, Robert Nelson [EMAIL PROTECTED] wrote: I have a preliminary version of the new vzpkg utilities ready for testing. These new tools support creating templates for 32 and 64 bit flavours of the following: Centos 4 and 5, Fedora 7, 8 and 9, Debian Sarge and Etch, Ubuntu Feisty, Gutsy and Hardy. They are extensible and will eventually support OpenSUSE and Gentoo. Fantastic! Thanks for doing this and for sharing. Will definitely be taking a look at this. In addition, I've created pkg-cacher, a transparent caching proxy daemon optimized for Debian and RPM packages. It is based on apt-cacher version 1.6.4 available with Debian. This version has been modified to understand RedHat RPM repositiories. It also adds support for the HTTP Range header and deals correctly with files which have the same name but different content on different distributions or in some cases different versions of the same distribution. In my experience I've found that just setting up Squid and telling my different machines to use the proxy (Acquire::http::Proxy http://proxy.hostname:3128/;;) has been the best pacakge caching solution. Works accross different packaging systems too. There are a few advantages of using pkg-cacher versus squid: It understands the difference between packages (static content) and metadata files (dynamic content). It only keeps a single copy of a package even if it is shared across multiple distributions. versions or accessed from multiple mirrors. Removal of stale packages can be based on whether the package is referenced by any repositories using the metadata. This means less downloads and less disk usage. Even a 250 GB disk starts looking small once you deal with multiple distros, versions and mirrors :-) In order to make installation easy I've setup repositories for 32 and 64 bit versions of CentOS 4, 5 and Fedora 7, 8 and 9. I'll be producing packages for Debian and Ubuntu as HN later. I can help with Debian packages if you need. Roberto Thanks for the offer, the only reason I didn't release it at this time is I didn't have a Debian HN set up for testing yet. Should be available in the next day or so. ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users
Re: [Users] New vzpkg, templates and package caching daemon
On Fri, Sep 5, 2008 at 9:27 PM, Robert Nelson [EMAIL PROTECTED] wrote: I have a preliminary version of the new vzpkg utilities ready for testing. These new tools support creating templates for 32 and 64 bit flavours of the following: Centos 4 and 5, Fedora 7, 8 and 9, Debian Sarge and Etch, Ubuntu Feisty, Gutsy and Hardy. They are extensible and will eventually support OpenSUSE and Gentoo. Fantastic! Thanks for doing this and for sharing. Will definitely be taking a look at this. In addition, I've created pkg-cacher, a transparent caching proxy daemon optimized for Debian and RPM packages. It is based on apt-cacher version 1.6.4 available with Debian. This version has been modified to understand RedHat RPM repositiories. It also adds support for the HTTP Range header and deals correctly with files which have the same name but different content on different distributions or in some cases different versions of the same distribution. In my experience I've found that just setting up Squid and telling my different machines to use the proxy (Acquire::http::Proxy http://proxy.hostname:3128/;;) has been the best pacakge caching solution. Works accross different packaging systems too. In order to make installation easy I've setup repositories for 32 and 64 bit versions of CentOS 4, 5 and Fedora 7, 8 and 9. I'll be producing packages for Debian and Ubuntu as HN later. I can help with Debian packages if you need. Roberto -- http://blog.divisiblebyfour.org/ ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users
Re: [Users] New vzpkg, templates and package caching daemon
Roberto Mello wrote: On Fri, Sep 5, 2008 at 9:27 PM, Robert Nelson [EMAIL PROTECTED] wrote: I have a preliminary version of the new vzpkg utilities ready for testing. These new tools support creating templates for 32 and 64 bit flavours of the following: Centos 4 and 5, Fedora 7, 8 and 9, Debian Sarge and Etch, Ubuntu Feisty, Gutsy and Hardy. They are extensible and will eventually support OpenSUSE and Gentoo. Fantastic! Thanks for doing this and for sharing. Will definitely be taking a look at this. In addition, I've created pkg-cacher, a transparent caching proxy daemon optimized for Debian and RPM packages. It is based on apt-cacher version 1.6.4 available with Debian. This version has been modified to understand RedHat RPM repositiories. It also adds support for the HTTP Range header and deals correctly with files which have the same name but different content on different distributions or in some cases different versions of the same distribution. In my experience I've found that just setting up Squid and telling my different machines to use the proxy (Acquire::http::Proxy http://proxy.hostname:3128/;;) has been the best pacakge caching solution. Works accross different packaging systems too. There are a few advantages of using pkg-cacher versus squid: It understands the difference between packages (static content) and metadata files (dynamic content). It only keeps a single copy of a package even if it is shared across multiple distributions. versions or accessed from multiple mirrors. Removal of stale packages can be based on whether the package is referenced by any repositories using the metadata. This means less downloads and less disk usage. Even a 250 GB disk starts looking small once you deal with multiple distros, versions and mirrors :-) In order to make installation easy I've setup repositories for 32 and 64 bit versions of CentOS 4, 5 and Fedora 7, 8 and 9. I'll be producing packages for Debian and Ubuntu as HN later. I can help with Debian packages if you need. Roberto Thanks for the offer, the only reason I didn't release it at this time is I didn't have a Debian HN set up for testing yet. Should be available in the next day or so. ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users
Re: [Users] New vzpkg, templates and package caching daemon
Hello Robert, i've built a debian vzpkg2 package, but need the source of pkg-cacher as well, but cannot be download from http://gforge.opensource-sw.net/projects/pkg_cacher. So please fix, that the packge depends are working, Thanks and by, Thorsten -- Thorsten Schifferdecker [EMAIL PROTECTED] Am Sa, 6.09.2008, 05:27, schrieb Robert Nelson: I have a preliminary version of the new vzenpkg utilities ready for testing. These new tools support creating templates for 32 and 64 bit flavours of the following: Centos 4 and 5, Fedora 7, 8 and 9, Debian Sarge and Etch, Ubuntu Feisty, Gutsy and Hardy. They are extensible and will eventually support OpenSUSE and Gentoo. In addition, I've created pkg-cacher, a transparent caching proxy daemon optimized for Debian and RPM packages. It is based on apt-cacher version 1.6.4 available with Debian. This version has been modified to understand RedHat RPM repositiories. It also adds support for the HTTP Range header and deals correctly with files which have the same name but different content on different distributions or in some cases different versions of the same distribution. The pkg-cacher source is available on http://gforge.opensource-sw.net/projects/pkg_cacher. The source for the new vzpkg and templates is available on git.opensource-sw.net. You can browse the source using the url http://git.opensource-sw.net/. In order to clone the updated source use the command git clone http://git.opensource-sw.net/git/vzpkg;. The templates are available using git clone http://git.opensource-sw.net/git/vztmpl;. In order to make installation easy I've setup repositories for 32 and 64 bit versions of CentOS 4, 5 and Fedora 7, 8 and 9. I'll be producing packages for Debian and Ubuntu as HN later. Please let me know of any issues or suggestions. Step by step installation instructions are below: Existing install) Backup your existing /vz/template directory Uninstall vzpkg and any vztmpl-* packages. New install) wget http://download.openvz.org/openvz.repo -P /etc/yum.repos.d Enable appropriate OpenVZ kernel yum install vzctl service vz start echo -n 1 /proc/sys/net/ipv4/ip_forward For CentOS: wget http://repo.opensource-sw.net/ossw_centos.repo -P /etc/yum.repos.d For Fedora: wget http://repo.opensource-sw.net/ossw_fedora.repo -P /etc/yum.repos.d yum install vzpkg2 vztmpl2-centos vztmpl2-fedora vztmpl2-debian vztmpl2-ubuntu yum install pkg-cacher Add the following to /etc/vz/vz.conf # # Replace hostname with the IP address or name of the server running pkg-cacher. # Don't use localhost, it must be a name resolvable from within the container. VZPKG_CACHE_HOST=hostname:3142 # # Network information for updating templates for Debian, Ubuntu # and other distributions that need network access. # # Adjust these to suit your network. # # Temporary static IP address used by the container while updating the # cached template. VZPKG_IPADDR=192.168.1.100 # # Default domain suffix to append to host names that aren't fully qualified. VZPKG_SEARCH=example.com # # List of DNS servers VZPKG_DNS=192.168.1.2,192.168.1.3 # cd /etc/vz/dists cp fedora-8.conf fedora-9.conf ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users
Re: [Users] New vzpkg, templates and package caching daemon
Thorsten Schifferdecker wrote: Hello Robert, i've built a debian vzpkg2 package, but need the source of pkg-cacher as well, but cannot be download from http://gforge.opensource-sw.net/projects/pkg_cacher. So please fix, that the packge depends are working, Oops, sorry about that. Fixed now. Thanks and by, Thorsten -- Thorsten Schifferdecker [EMAIL PROTECTED] Am Sa, 6.09.2008, 05:27, schrieb Robert Nelson: I have a preliminary version of the new vzenpkg utilities ready for testing. These new tools support creating templates for 32 and 64 bit flavours of the following: Centos 4 and 5, Fedora 7, 8 and 9, Debian Sarge and Etch, Ubuntu Feisty, Gutsy and Hardy. They are extensible and will eventually support OpenSUSE and Gentoo. In addition, I've created pkg-cacher, a transparent caching proxy daemon optimized for Debian and RPM packages. It is based on apt-cacher version 1.6.4 available with Debian. This version has been modified to understand RedHat RPM repositiories. It also adds support for the HTTP Range header and deals correctly with files which have the same name but different content on different distributions or in some cases different versions of the same distribution. The pkg-cacher source is available on http://gforge.opensource-sw.net/projects/pkg_cacher. The source for the new vzpkg and templates is available on git.opensource-sw.net. You can browse the source using the url http://git.opensource-sw.net/. In order to clone the updated source use the command git clone http://git.opensource-sw.net/git/vzpkg;. The templates are available using git clone http://git.opensource-sw.net/git/vztmpl;. In order to make installation easy I've setup repositories for 32 and 64 bit versions of CentOS 4, 5 and Fedora 7, 8 and 9. I'll be producing packages for Debian and Ubuntu as HN later. Please let me know of any issues or suggestions. Step by step installation instructions are below: Existing install) Backup your existing /vz/template directory Uninstall vzpkg and any vztmpl-* packages. New install) wget http://download.openvz.org/openvz.repo -P /etc/yum.repos.d Enable appropriate OpenVZ kernel yum install vzctl service vz start echo -n 1 /proc/sys/net/ipv4/ip_forward For CentOS: wget http://repo.opensource-sw.net/ossw_centos.repo -P /etc/yum.repos.d For Fedora: wget http://repo.opensource-sw.net/ossw_fedora.repo -P /etc/yum.repos.d yum install vzpkg2 vztmpl2-centos vztmpl2-fedora vztmpl2-debian vztmpl2-ubuntu yum install pkg-cacher Add the following to /etc/vz/vz.conf # # Replace hostname with the IP address or name of the server running pkg-cacher. # Don't use localhost, it must be a name resolvable from within the container. VZPKG_CACHE_HOST=hostname:3142 # # Network information for updating templates for Debian, Ubuntu # and other distributions that need network access. # # Adjust these to suit your network. # # Temporary static IP address used by the container while updating the # cached template. VZPKG_IPADDR=192.168.1.100 # # Default domain suffix to append to host names that aren't fully qualified. VZPKG_SEARCH=example.com # # List of DNS servers VZPKG_DNS=192.168.1.2,192.168.1.3 # cd /etc/vz/dists cp fedora-8.conf fedora-9.conf ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users
[Users] New vzpkg, templates and package caching daemon
I have a preliminary version of the new vzpkg utilities ready for testing. These new tools support creating templates for 32 and 64 bit flavours of the following: Centos 4 and 5, Fedora 7, 8 and 9, Debian Sarge and Etch, Ubuntu Feisty, Gutsy and Hardy. They are extensible and will eventually support OpenSUSE and Gentoo. In addition, I've created pkg-cacher, a transparent caching proxy daemon optimized for Debian and RPM packages. It is based on apt-cacher version 1.6.4 available with Debian. This version has been modified to understand RedHat RPM repositiories. It also adds support for the HTTP Range header and deals correctly with files which have the same name but different content on different distributions or in some cases different versions of the same distribution. The pkg-cacher source is available on http://gforge.opensource-sw.net/projects/pkg_cacher. The source for the new vzpkg and templates is available on git.opensource-sw.net. You can browse the source using the url http://git.opensource-sw.net/. In order to clone the updated source use the command git clone http://git.opensource-sw.net/git/vzpkg;. The templates are available using git clone http://git.opensource-sw.net/git/vztmpl;. In order to make installation easy I've setup repositories for 32 and 64 bit versions of CentOS 4, 5 and Fedora 7, 8 and 9. I'll be producing packages for Debian and Ubuntu as HN later. Please let me know of any issues or suggestions. Step by step installation instructions are below: Existing install) Backup your existing /vz/template directory Uninstall vzpkg and any vztmpl-* packages. New install) wget http://download.openvz.org/openvz.repo -P /etc/yum.repos.d Enable appropriate OpenVZ kernel yum install vzctl service vz start echo -n 1 /proc/sys/net/ipv4/ip_forward For CentOS: wget http://repo.opensource-sw.net/ossw_centos.repo -P /etc/yum.repos.d For Fedora: wget http://repo.opensource-sw.net/ossw_fedora.repo -P /etc/yum.repos.d yum install vzpkg2 vztmpl2-centos vztmpl2-fedora vztmpl2-debian vztmpl2-ubuntu yum install pkg-cacher Add the following to /etc/vz/vz.conf # # Replace hostname with the IP address or name of the server running pkg-cacher. # Don't use localhost, it must be a name resolvable from within the container. VZPKG_CACHE_HOST=hostname:3142 # # Network information for updating templates for Debian, Ubuntu # and other distributions that need network access. # # Adjust these to suit your network. # # Temporary static IP address used by the container while updating the # cached template. VZPKG_IPADDR=192.168.1.100 # # Default domain suffix to append to host names that aren't fully qualified. VZPKG_SEARCH=example.com # # List of DNS servers VZPKG_DNS=192.168.1.2,192.168.1.3 # cd /etc/vz/dists cp fedora-8.conf fedora-9.conf ___ Users mailing list Users@openvz.org https://openvz.org/mailman/listinfo/users
