Bug#620824: ITP: Percona XtraBackup - hot backup utility for MySQL
Paul Wise writes: > On Tue, Jul 2, 2013 at 7:35 PM, Stewart Smith wrote: > >> I'll investigate this first thing in the morning (it's getting late), >> although it looks as though the tar.gz didn't fully download (yes, it's >> really 135MB) > > Looks like your tarball is named incorrectly (tar.gz instead of > orig.tar.gz) and your website returns a HTTP 302 code (redirect) and > some HTML for the orig.tar.gz URL rather than a 404. I'll ensure our process includes putting the tarball up named correctly. The tarball in that directory should be the right one when named correctly though (unless I brain-farted and got it wrong :) >> Yep, we typically do this. We use a key for the company rather than our >> individual ones for our current repositories. Is this an acceptable >> approach for packages going into Debian? Or should we just have the few >> individuals sign it if they're involved? > > It is a reasonable approach for upstream tarballs. Uploads to Debian > need to be signed by one of the keys in the developer keyring or a > subkey of one of those keys. My guess is the reasonable thing to do is ensure that a few of us have keys in the keyring then so that there's some redundancy (after all, we have to let people take vacation :) >> We'd need to depend on very specific 5.1, 5.5 and 5.6 versions of both >> MySQL and Percona Server (which isn't yet packaged) and we'd also have >> to patch it. >> >> Our short/medium term plan is to do away with having multiple server >> source trees and binaries. We instead plan to just make a modified MySQL >> branch where we just include the needed (modified) source. In an ideal >> world the needed code would be buildable as a library and we'd get the >> needed patches upstream. It is not an ideal world however :( > > I see, maybe you have chosen the way of least pain. We try to :) >> I'm not really attached either way... It appears that it may not be >> worth having it. I'll defer to the experts on this one. > > Maybe ask your userbase if they need it? I'll reach out through our support org. I suspect we'd be fine without it. >> I'll bring it up there. We would see that as simply a patch that's >> needed for that version to be included in that Os and thus not an issue. > > The danger is if that approach to interpretation if the policy were to > change. Okay. I'll bring it up on debian-devel and see if we need changes (it just means I probably get to talk trademark law more... which is generally something I try to avoid :) >> It's certainly something we may be interested at looking at... I guess >> we'll see what the buildds say :) >> >> We do have a test suite. It does (of course) depend on having server >> binaries around to run backup and restore with. Our CI infrastructure >> currently uses just binary tarballs of various server versions, but this >> probably isn't ideal for Debain so we'll have to come up with another >> solution. One thing to consider is that (like MySQL) running the test >> suite takes a non-trivial amount of time. > > Please do enable the test suite for Debian builds and run them with > the binaries built during the build process. The problem we'll have to solve is running against MySQL (and Percona Server) versions... and I'm not sure the best way to wrangle that (we probably have to fix a bug or two in the test suite to ensure it works against an installed binary - we currently download binary tarballs and run against them). -- Stewart Smith pgpJNloePVzJP.pgp Description: PGP signature
Bug#620824: ITP: Percona XtraBackup - hot backup utility for MySQL
Vincent Bernat writes: > ❦ 2 juillet 2013 09:20 CEST, Paul Wise : > >>> Why is the tarball so big and why did it require an internet connection? >>> Well, XtraBackup uses *part* of the MySQL code (actually, mostly parts >>> of InnoDB) >> >> An alternative to including the MySQL InnoDB code might be to >> build-depend on mysql-source-5.5, unpack and patch it during the build >> process and add Built-Using: mysql-5.5 (= $version) to the resulting >> binary package. > > The problem is that it should Build-Depends on Percona MySQL which is > not yet available in Debian. I think the first step would be to provide > Percona MySQL packages. Currently, it seems that there is no easy way to > provide packages as alternative to MySQL but maybe work has already > started with MariaDB. It would need to build-depend on specific MySQL and Percona Server versions (which end up being on-disk compatible with various MariaDB versions). We're working on getting Percona Server ready to be included too, but we'd be going for the current Percona Server versions not the ones we patch heavily to then build XtraBackup with. But, the main problem is that we need to patch the InnoDB source to build XtraBackup and the patch isn't trivial to port to the latest versions (and there's usually no benefit in doing so). It's better to think of XtraBackup patching and including the parts of InnoDB it needs and for reasons that are largely historical it's multiple binaries for multiple versions of InnoDB. Our short/medium term goal is to stop doing this and actually just have one set of InnoDB code from the most recent MySQL/Percona Server version and one XtraBackup binary that works on all server versions. In an ideal world there'd be some library we could link against and have the patches be accepted upstream.. but we're not in an ideal world. This would make it more like something like xfsprogs which does end up including some of the same code as the kernel (although we patch InnoDB while xfsprogs seems to be about at the point where that isn't necessary). So while we do understand how it'd be great to build against the source in mysql-source-5.5, it's just not really feasible to do so and maintain quality and maintainability. We're hoping that our short/medium term of just including the patched InnoDB bits we need in our source tree to mostly solve the problem. I hope this helps clarify. -- Stewart Smith pgpVemmzIwodh.pgp Description: PGP signature
Bug#620824: ITP: Percona XtraBackup - hot backup utility for MySQL
On Tue, Jul 2, 2013 at 7:35 PM, Stewart Smith wrote: > I'll investigate this first thing in the morning (it's getting late), > although it looks as though the tar.gz didn't fully download (yes, it's > really 135MB) Looks like your tarball is named incorrectly (tar.gz instead of orig.tar.gz) and your website returns a HTTP 302 code (redirect) and some HTML for the orig.tar.gz URL rather than a 404. > I think there's some good ideas here that we can add to our CI > infrastructure. I don't think there's anything that would be considered > a blocker though. Indeed. > Yep, we typically do this. We use a key for the company rather than our > individual ones for our current repositories. Is this an acceptable > approach for packages going into Debian? Or should we just have the few > individuals sign it if they're involved? It is a reasonable approach for upstream tarballs. Uploads to Debian need to be signed by one of the keys in the developer keyring or a subkey of one of those keys. > We'd need to depend on very specific 5.1, 5.5 and 5.6 versions of both > MySQL and Percona Server (which isn't yet packaged) and we'd also have > to patch it. > > Our short/medium term plan is to do away with having multiple server > source trees and binaries. We instead plan to just make a modified MySQL > branch where we just include the needed (modified) source. In an ideal > world the needed code would be buildable as a library and we'd get the > needed patches upstream. It is not an ideal world however :( I see, maybe you have chosen the way of least pain. > I'm not really attached either way... It appears that it may not be > worth having it. I'll defer to the experts on this one. Maybe ask your userbase if they need it? > I'll bring it up there. We would see that as simply a patch that's > needed for that version to be included in that Os and thus not an issue. The danger is if that approach to interpretation if the policy were to change. > It's certainly something we may be interested at looking at... I guess > we'll see what the buildds say :) > > We do have a test suite. It does (of course) depend on having server > binaries around to run backup and restore with. Our CI infrastructure > currently uses just binary tarballs of various server versions, but this > probably isn't ideal for Debain so we'll have to come up with another > solution. One thing to consider is that (like MySQL) running the test > suite takes a non-trivial amount of time. Please do enable the test suite for Debian builds and run them with the binaries built during the build process. -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#620824: ITP: Percona XtraBackup - hot backup utility for MySQL
Paul Wise writes: > On Tue, Jul 2, 2013 at 2:41 PM, Stewart Smith wrote: > >> I'm Stewart and I work for Percona. One of the things I'm currently >> working on is ensuring all our free and open source software projects >> are packaged for all the major linux distributions - including my >> beloved Debian. > > Since you are part of upstream, please review our upstream guide and > the links in the external advice section: > > http://wiki.debian.org/UpstreamGuide Thanks for the pointer. It appears that we're pretty good on most things mentioned there and some things we've got plans to address. >> There is an open bug for Percona XtraBackup packaging: >> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620824 > > Since you intend to package it, you should retitle this bug to intent > to package and set yourself as the owner of it: > > http://www.debian.org/devel/wnpp/#l3 > http://www.debian.org/Bugs/server-control#retitle > http://www.debian.org/Bugs/server-control#owner Done. >> I've put up source tarball and packaging for unstable up at: >> http://www.percona.com/downloads/TESTING/XtraBackup/xtrabackup-2.1.3/release-2.1.3/610/source/ > > I'm unable to unpack the source package: > > $ dpkg-source -x percona-xtrabackup_2.1.3-610-1.dsc > dpkg-source: warning: extracting unsigned source package > (percona-xtrabackup_2.1.3-610-1.dsc) > dpkg-source: error: File ./percona-xtrabackup_2.1.3-610.orig.tar.gz > has size 39459 instead of expected 141074103 I'll investigate this first thing in the morning (it's getting late), although it looks as though the tar.gz didn't fully download (yes, it's really 135MB) > You may want to run some commands from the source tree after a build: > > http://wiki.debian.org/HowToPackageForDebian#Check_points_for_any_package I think there's some good ideas here that we can add to our CI infrastructure. I don't think there's anything that would be considered a blocker though. > Also, I encourage you to sign your Debian packages and also your > upstream tarballs using OpenPGP keys. Here are some best practices for > OpenPGP: > > https://we.riseup.net/riseuplabs+paow/openpgp-best-practices Yep, we typically do this. We use a key for the company rather than our individual ones for our current repositories. Is this an acceptable approach for packages going into Debian? Or should we just have the few individuals sign it if they're involved? >> Why is the tarball so big and why did it require an internet connection? >> Well, XtraBackup uses *part* of the MySQL code (actually, mostly parts >> of InnoDB) > > An alternative to including the MySQL InnoDB code might be to > build-depend on mysql-source-5.5, unpack and patch it during the build > process and add Built-Using: mysql-5.5 (= $version) to the resulting > binary package. We'd need to depend on very specific 5.1, 5.5 and 5.6 versions of both MySQL and Percona Server (which isn't yet packaged) and we'd also have to patch it. Our short/medium term plan is to do away with having multiple server source trees and binaries. We instead plan to just make a modified MySQL branch where we just include the needed (modified) source. In an ideal world the needed code would be buildable as a library and we'd get the needed patches upstream. It is not an ideal world however :( >> 1) We have a transitional package called 'xtrabackup' that was (prior to >>2.0) the package name was in our own APT repositories. Does it make >>sense to keep this in the Debian packaging? > > Seems reasonable to include it if you really want to. There seem to be > more folks using percona-xtrabackup than xtrabackup though: I'm not really attached either way... It appears that it may not be worth having it. I'll defer to the experts on this one. >> 2) We have trademarks. We've tried to come up with a trademark policy >>that makes everybody happy (well, as happy as you can be when you're >>dealing with trademark law). >>It's at: >>http://www.percona.com/doc/percona-xtrabackup/2.1/trademark-policy.html >>We believe this should satisfy any requirements of Linux >>distributions, but if there's any concerns, don't hesitate to ask. > > I'd suggest mailing debian-legal about this question but based on the > trademark policy it appears that Debian would have to rename percona > if we wanted to add a security patch in stable? I'll bring it up there. We would see that as simply a patch that's needed for that version to be included in that Os and thus not an issue. > You may want to listen to this FOSDEM talk entitled 'How to Share a > Trademark': > > http://faif.us/cast/2013/may/07/0x3C/ I'll have a look. We tried to think of linux distros and Debian when we were forming the trademark policy. We'll see what Debian-legal says :) >> 3) Supported architectures. We see x86-64, x86 and very, very rarely, >>SPARC. I'd say that both people running XtraBackup on a non-x86 >>architecture are happy, but I'm pretty sure there's not even that
Bug#620824: ITP: Percona XtraBackup - hot backup utility for MySQL
❦ 2 juillet 2013 09:20 CEST, Paul Wise : >> Why is the tarball so big and why did it require an internet connection? >> Well, XtraBackup uses *part* of the MySQL code (actually, mostly parts >> of InnoDB) > > An alternative to including the MySQL InnoDB code might be to > build-depend on mysql-source-5.5, unpack and patch it during the build > process and add Built-Using: mysql-5.5 (= $version) to the resulting > binary package. The problem is that it should Build-Depends on Percona MySQL which is not yet available in Debian. I think the first step would be to provide Percona MySQL packages. Currently, it seems that there is no easy way to provide packages as alternative to MySQL but maybe work has already started with MariaDB. -- /* Fuck me gently with a chainsaw... */ 2.0.38 /usr/src/linux/arch/sparc/kernel/ptrace.c pgp_MOOSUFCkt.pgp Description: PGP signature
Bug#620824: ITP: Percona XtraBackup - hot backup utility for MySQL
On Tue, Jul 2, 2013 at 2:41 PM, Stewart Smith wrote: > I'm Stewart and I work for Percona. One of the things I'm currently > working on is ensuring all our free and open source software projects > are packaged for all the major linux distributions - including my > beloved Debian. Since you are part of upstream, please review our upstream guide and the links in the external advice section: http://wiki.debian.org/UpstreamGuide > We're wanting to have Percona XtraBackup be part of Debian. Excellent. > There is an open bug for Percona XtraBackup packaging: > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620824 Since you intend to package it, you should retitle this bug to intent to package and set yourself as the owner of it: http://www.debian.org/devel/wnpp/#l3 http://www.debian.org/Bugs/server-control#retitle http://www.debian.org/Bugs/server-control#owner > We are going to make "upload release to Debian" be part of our release > process We want to be an active and involved upstream. Great. > I've put up source tarball and packaging for unstable up at: > http://www.percona.com/downloads/TESTING/XtraBackup/xtrabackup-2.1.3/release-2.1.3/610/source/ I'm unable to unpack the source package: $ dpkg-source -x percona-xtrabackup_2.1.3-610-1.dsc dpkg-source: warning: extracting unsigned source package (percona-xtrabackup_2.1.3-610-1.dsc) dpkg-source: error: File ./percona-xtrabackup_2.1.3-610.orig.tar.gz has size 39459 instead of expected 141074103 You may want to run some commands from the source tree after a build: http://wiki.debian.org/HowToPackageForDebian#Check_points_for_any_package Also, I encourage you to sign your Debian packages and also your upstream tarballs using OpenPGP keys. Here are some best practices for OpenPGP: https://we.riseup.net/riseuplabs+paow/openpgp-best-practices > Why is the tarball so big and why did it require an internet connection? > Well, XtraBackup uses *part* of the MySQL code (actually, mostly parts > of InnoDB) An alternative to including the MySQL InnoDB code might be to build-depend on mysql-source-5.5, unpack and patch it during the build process and add Built-Using: mysql-5.5 (= $version) to the resulting binary package. > 1) We have a transitional package called 'xtrabackup' that was (prior to >2.0) the package name was in our own APT repositories. Does it make >sense to keep this in the Debian packaging? Seems reasonable to include it if you really want to. There seem to be more folks using percona-xtrabackup than xtrabackup though: $ GET http://popcon.debian.org/unknown/by_inst | grep -i xtrab 1590 percona-xtrabackup 150368232 0 (Not in sid) 2170 xtrabackup98 529 064 (Not in sid) 23575 percona-xtrabackup-test3 0 2 1 0 (Not in sid) 31597 percona-xtrabackup-dbg 2 0 2 0 0 (Not in sid) 74903 szn-mysql-xtrabackup 1 0 1 0 0 (Not in sid) > 2) We have trademarks. We've tried to come up with a trademark policy >that makes everybody happy (well, as happy as you can be when you're >dealing with trademark law). >It's at: >http://www.percona.com/doc/percona-xtrabackup/2.1/trademark-policy.html >We believe this should satisfy any requirements of Linux >distributions, but if there's any concerns, don't hesitate to ask. I'd suggest mailing debian-legal about this question but based on the trademark policy it appears that Debian would have to rename percona if we wanted to add a security patch in stable? You may want to listen to this FOSDEM talk entitled 'How to Share a Trademark': http://faif.us/cast/2013/may/07/0x3C/ > 3) Supported architectures. We see x86-64, x86 and very, very rarely, >SPARC. I'd say that both people running XtraBackup on a non-x86 >architecture are happy, but I'm pretty sure there's not even that >many. While XtraBackup may compile (or even run) on other >architectures, it currently makes approximately zero business sense >for us to spend any time on it - although we're happy to take >patches. > >Given this, how much "works and passes tests on all archs" is >required to have packages accepted? There is no requirement to work on all arches but regressions in architecture support are release-critical. You need to have a good test suite so that binary packages are not produced on architectures where the software doesn't work. I expect you will want to make it work on ARM since arm64 is coming soon and may become a popular architecture for servers. http://wiki.debian.org/Arm64Port -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#620824: ITP: Percona XtraBackup - hot backup utility for MySQL
Hi! I'm Stewart and I work for Percona. One of the things I'm currently working on is ensuring all our free and open source software projects are packaged for all the major linux distributions - including my beloved Debian. We're wanting to have Percona XtraBackup be part of Debian. Percona XtraBackup is an online backup tool for MySQL, MariaDB, Percona Server and Percona XtraDB Cluster. In the near future, we will also wanting our packages for Percona Server and Percona XtraDB Cluster to be part of Debian. There is an open bug for Percona XtraBackup packaging: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620824 We are going to make "upload release to Debian" be part of our release process. We tend to make a new minor bugfix release (i.e. 2.1.x) every 2-3 months and a new major release every 12 (2.0 in April 2012, 2.1 in May 2013). We want to be an active and involved upstream. There have been a couple of issues with our source tarballs and build procedure that I've had to fix up before proposing that Percona XtraBackup be included. I've made changes on top of our 2.1.3 release that mean we now generate source tarballs that do not require an active internet connection to build. I've put up source tarball and packaging for unstable up at: http://www.percona.com/downloads/TESTING/XtraBackup/xtrabackup-2.1.3/release-2.1.3/610/source/ All future releases will produce tarballs like this, so they'll be in a more normal area of our downloads site - I'll also get one up in an official place for the current 2.1.3 release. I (and we, the rest of Percona) would really appreciate any review/comments/suggestions on this packaging - we're really keen to have Percona XtraBackup be in the Debian archives. Why is the tarball so big and why did it require an internet connection? Well, XtraBackup uses *part* of the MySQL code (actually, mostly parts of InnoDB) to help it read InnoDB pages from disk and back them up safely. It also uses part of the code to replay REDO logs to prepare a backup before it can be restored. Due to various not-that-interesting reasons (all of which we do consider bugs) we currently produce different binaries for different MySQL versions (hence linking in code From different MySQL versions). It's important to note that we patch the InnoDB code inside MySQL to make it usable by xtrabackup and that we cannot just use the MySQL code already shipped in Debian for this without creating a nightmare for the Debian MySQL packaging team, the security team and ourselves (it's generally non-trivial to port this patch to new versions). There should never be any security implications of linking against this (not always up to date) MySQL code as we only use it to read (and write) files to local disk. Questions: 1) We have a transitional package called 'xtrabackup' that was (prior to 2.0) the package name was in our own APT repositories. Does it make sense to keep this in the Debian packaging? 2) We have trademarks. We've tried to come up with a trademark policy that makes everybody happy (well, as happy as you can be when you're dealing with trademark law). It's at: http://www.percona.com/doc/percona-xtrabackup/2.1/trademark-policy.html We believe this should satisfy any requirements of Linux distributions, but if there's any concerns, don't hesitate to ask. 3) Supported architectures. We see x86-64, x86 and very, very rarely, SPARC. I'd say that both people running XtraBackup on a non-x86 architecture are happy, but I'm pretty sure there's not even that many. While XtraBackup may compile (or even run) on other architectures, it currently makes approximately zero business sense for us to spend any time on it - although we're happy to take patches. Given this, how much "works and passes tests on all archs" is required to have packages accepted? 4) We plan on continuing to provide our own repositories as well and are wanting to not diverge in packaging from our repos and what's in Debian. Any tips/tricks are very welcome - we have "build debian packages" as part of or continuous integration efforts and plan to soon have "run the whole test suite on those packages" as part of it. Thanks in advance for the feedback! -- Stewart Smith pgpbTINQwmimG.pgp Description: PGP signature
