deb/rpm package purge
Should the package purge remove /var/lib/ceph/* (potential mon data, osd data) and/or /var/log/ceph/* (logs)? Right now it does, but mysql, for example, leaves /var/lib/mysql where it is (not sure about logs). sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On Tuesday, March 19, 2013 at 12:48 PM, Sage Weil wrote: > Should the package purge remove /var/lib/ceph/* (potential mon data, osd > data) and/or /var/log/ceph/* (logs)? Right now it does, but mysql, for > example, leaves /var/lib/mysql where it is (not sure about logs). > I'm with Mark's ticket on this (http://tracker.ceph.com/issues/4505). Config data in /etc/ceph and logs in var/log/ceph is fine to remove, but storage data isn't. That's essentially user-generated and not something that can be recovered or rebuilt following the purge. Keyrings might not be unreasonable to delete, but I don't think they're necessary and certainly aren't worth putting in the work to separate from the other user data in var/lib/ceph. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On 03/19/2013 02:48 PM, Sage Weil wrote: Should the package purge remove /var/lib/ceph/* (potential mon data, osd data) and/or /var/log/ceph/* (logs)? Right now it does, but mysql, for example, leaves /var/lib/mysql where it is (not sure about logs). I'm definitely for leaving mon/osd data in place. Those files are created at cluster creation time, not when the packages are installed. They may have been created by a totally different installation of Ceph than the packaged version. What's worse is that you can't get the files back simply by reinstalling the package. With the way things currently are, a package purge will effectively permanently destroy your cluster. Purge should get rid of configuration files, but I don't think it should destroy user data which is what it effectively is doing now. sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
Is there a way out if you remove the keyrings? On 03/19/2013 01:05 PM, Mark Nelson wrote: On 03/19/2013 02:48 PM, Sage Weil wrote: Should the package purge remove /var/lib/ceph/* (potential mon data, osd data) and/or /var/log/ceph/* (logs)? Right now it does, but mysql, for example, leaves /var/lib/mysql where it is (not sure about logs). I'm definitely for leaving mon/osd data in place. Those files are created at cluster creation time, not when the packages are installed. They may have been created by a totally different installation of Ceph than the packaged version. What's worse is that you can't get the files back simply by reinstalling the package. With the way things currently are, a package purge will effectively permanently destroy your cluster. Purge should get rid of configuration files, but I don't think it should destroy user data which is what it effectively is doing now. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On Tue, 19 Mar 2013, Dan Mick wrote: > Is there a way out if you remove the keyrings? Yeah; you can use the keyring in the mon data dir to refetch any keyrings you deleted from /etc/ceph/. ... I don't disagree with any of this discussion, BTW; I ask the question because I want to understand what the packaging best-practices are before we put our foot down for what is "right". sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote: > On Tue, 2013-03-19 at 13:51 -0700, Sage Weil wrote: > > On Tue, 19 Mar 2013, Dan Mick wrote: > > > Is there a way out if you remove the keyrings? > > > > Yeah; you can use the keyring in the mon data dir to refetch any keyrings > > you deleted from /etc/ceph/. > But if packages are purged on mon as well... > > > I don't disagree with any of this discussion, BTW; I ask the question > > because I want to understand what the packaging best-practices are before > > we put our foot down for what is "right". > Actually there are two methods to uninstall a package from the system. > One is '-r' or '--remove' options of dpkg . As it's name and the > manpage confirms: "Remove an installed package. -r or --remove remove > everything _except conffiles_. This may avoid having to reconfigure the > package if it is reinstalled later.". The second is '-P' or '--purge', > which stands for '[...] removes everything, including conffiles.". > Debian policy chapter 10 [1] writes about files and their handling. Its > subchapter 10.7 [2] talks about configuration files. Their handling is > expressed in subchapter 6.8 [3]. > > In short, if user asks to purge the package, then the keys have to be > removed as well. If someone thinks about a reinstallation, s/he should > use remove instead. To be safe, I see one option. On purge, a debconf > message can be displayed notifying the user about the keys are going to > be removed as well. Abort of purge or backup of keys can be offered, > based on the choice. The keys aren't a problem; they are still in the mon database (/var/lib/ceph/mon/...). The real question is whether purge should remove /var/lib/ceph and /var/log/ceph... sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On Tue, 2013-03-19 at 13:51 -0700, Sage Weil wrote: > On Tue, 19 Mar 2013, Dan Mick wrote: > > Is there a way out if you remove the keyrings? > > Yeah; you can use the keyring in the mon data dir to refetch any keyrings > you deleted from /etc/ceph/. But if packages are purged on mon as well... > I don't disagree with any of this discussion, BTW; I ask the question > because I want to understand what the packaging best-practices are before > we put our foot down for what is "right". Actually there are two methods to uninstall a package from the system. One is '-r' or '--remove' options of dpkg . As it's name and the manpage confirms: "Remove an installed package. -r or --remove remove everything _except conffiles_. This may avoid having to reconfigure the package if it is reinstalled later.". The second is '-P' or '--purge', which stands for '[...] removes everything, including conffiles.". Debian policy chapter 10 [1] writes about files and their handling. Its subchapter 10.7 [2] talks about configuration files. Their handling is expressed in subchapter 6.8 [3]. In short, if user asks to purge the package, then the keys have to be removed as well. If someone thinks about a reinstallation, s/he should use remove instead. To be safe, I see one option. On purge, a debconf message can be displayed notifying the user about the keys are going to be removed as well. Abort of purge or backup of keys can be offered, based on the choice. Hope this helps, Laszlo/GCS [1] http://www.debian.org/doc/debian-policy/ch-files.html [2] http://www.debian.org/doc/debian-policy/ch-files.html#s-config-files [3] http://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#s-removedetails -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On Tue, 2013-03-19 at 15:29 -0700, Sage Weil wrote: > On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote: > > In short, if user asks to purge the package, then the keys have to be > > removed as well. If someone thinks about a reinstallation, s/he should > > use remove instead. > The keys aren't a problem; they are still in the mon database > (/var/lib/ceph/mon/...). > > The real question is whether purge should remove /var/lib/ceph and > /var/log/ceph... Sorry if I was not clear and/or generic enough. Use of 'dpkg --remove' is to remove binaries of the package, but leave other things, runtime data (configuration, logs, database settings/users and so on) to be left untouched. On the other hand, 'dpkg --purge' is to remove everything the package has installed and/or generated. This includes debconf answers as well. With other words, purge is used to make the system totally clean of the package. As such, if the sysadmin install the package again, all debconf questions will be asked again and all generated files will be generated again from scratch. Laszlo/GCS -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote: > Sorry if I was not clear and/or generic enough. Use of 'dpkg --remove' > is to remove binaries of the package, but leave other things, runtime > data (configuration, logs, database settings/users and so on) to be left > untouched. > On the other hand, 'dpkg --purge' is to remove everything the package > has installed and/or generated. This includes debconf answers as well. > With other words, purge is used to make the system totally clean of the > package. As such, if the sysadmin install the package again, all debconf > questions will be asked again and all generated files will be generated > again from scratch. I understand that part, but the policy isn't very clear about files that are not part of the package but are generated as a result of the package being installed (i.e., user data). /var/lib/ceph/ has all of the *user data*, like critical monitor files that are necessary for the cluster to function (/var/lib/ceph/mon/*), and actual ceph objects (/var/lib/ceph/osd/*, although these are usually mount points that we would unmount, not delete). Doing a purge on the monitor nodes would destroy the cluster (and all of its data) in a not-easily-recoverable way. As a point of comparison, mysql removes the config files but not /var/lib/mysql. The question is, is that okay/typical/desireable/recommended/a bad idea? The less important question is whether /var/log/ceph should be removed; I'm assuming yes? Thanks! sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On 03/19/2013 05:51 PM, Laszlo Boszormenyi (GCS) wrote: On Tue, 2013-03-19 at 15:29 -0700, Sage Weil wrote: On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote: In short, if user asks to purge the package, then the keys have to be removed as well. If someone thinks about a reinstallation, s/he should use remove instead. The keys aren't a problem; they are still in the mon database (/var/lib/ceph/mon/...). The real question is whether purge should remove /var/lib/ceph and /var/log/ceph... Sorry if I was not clear and/or generic enough. Use of 'dpkg --remove' is to remove binaries of the package, but leave other things, runtime data (configuration, logs, database settings/users and so on) to be left untouched. On the other hand, 'dpkg --purge' is to remove everything the package has installed and/or generated. This includes debconf answers as well. With other words, purge is used to make the system totally clean of the package. As such, if the sysadmin install the package again, all debconf questions will be asked again and all generated files will be generated again from scratch. Hi Lazlo, Thanks for the answers! The question is whether or not the data in /var/lib/ceph should be removed when that data was not generated as part of the package installation process, but rather as part of the cluster configuration after the package was installed. IE the package creates the directory, but the user creates the data that resides in it. What should purge do with the directory now that the user has placed (critical) data in it that was not generated as part of the package installation process? Laszlo/GCS -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On Tue, 2013-03-19 at 15:59 -0700, Sage Weil wrote: > On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote: > > On the other hand, 'dpkg --purge' is to remove everything the package > > has installed and/or generated. This includes debconf answers as well. > > With other words, purge is used to make the system totally clean of the > > package. As such, if the sysadmin install the package again, all debconf > > questions will be asked again and all generated files will be generated > > again from scratch. > > I understand that part, but the policy isn't very clear about files that > are not part of the package but are generated as a result of the package > being installed (i.e., user data). Forgive me, I just learnt English and my wording may not be that clear for a natural speaker. > As a point of comparison, mysql removes the config files but not > /var/lib/mysql. As I remember, MySQL asks if /var/lib/mysql/ should be purged or not; I may mix with an other (database related) package. > The question is, is that okay/typical/desireable/recommended/a bad idea? I can rephrase my words. Purge removes the (binary) package files, its configuration and logs (its generated files). To emphasis, user files are _not_ fall into this category and must remain as-is, _intact_. Some packages writes a console message that 'your files remain at xxx, they were not removed' on purge. Others just leave the dpkg warning "directory not empty so not removed" which means user files may have left there and that may be the reason the directory is not empty. I'm in a rush, but hopefully will be able to note policy parts in the afternoon (CET). > The less important question is whether /var/log/ceph should be removed; > I'm assuming yes? Yes, logs are going to be removed. Hope I could answer your question now. Please note me if I should clear more parts of my answer. Laszlo/GCS -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On Wed, 20 Mar 2013, Laszlo Boszormenyi (GCS) wrote: > On Tue, 2013-03-19 at 15:59 -0700, Sage Weil wrote: > > On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote: > > > On the other hand, 'dpkg --purge' is to remove everything the package > > > has installed and/or generated. This includes debconf answers as well. > > > With other words, purge is used to make the system totally clean of the > > > package. As such, if the sysadmin install the package again, all debconf > > > questions will be asked again and all generated files will be generated > > > again from scratch. > > > > I understand that part, but the policy isn't very clear about files that > > are not part of the package but are generated as a result of the package > > being installed (i.e., user data). > Forgive me, I just learnt English and my wording may not be that clear > for a natural speaker. > > > As a point of comparison, mysql removes the config files but not > > /var/lib/mysql. > As I remember, MySQL asks if /var/lib/mysql/ should be purged or not; I > may mix with an other (database related) package. > > > The question is, is that okay/typical/desireable/recommended/a bad idea? > I can rephrase my words. Purge removes the (binary) package files, its > configuration and logs (its generated files). To emphasis, user files > are _not_ fall into this category and must remain as-is, _intact_. > Some packages writes a console message that 'your files remain at xxx, > they were not removed' on purge. Others just leave the dpkg warning > "directory not empty so not removed" which means user files may have > left there and that may be the reason the directory is not empty. > I'm in a rush, but hopefully will be able to note policy parts in the > afternoon (CET). Thanks, Laszlo, that's exactly what I was after! Sorry for the confusing exchange. :) Sounds like in this case, the fix is simply to leave /var/lib/ceph untouched. We'll need to update teuthology ceph.py and nuke to clean up /var/lib/ceph (for qa runs), and I think we should add a ceph-deploy 'purgedata' command to clean out /var/lib/ceph on a given host. Thanks! sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On 03/20/2013 07:48 AM, Sage Weil wrote: On Wed, 20 Mar 2013, Laszlo Boszormenyi (GCS) wrote: On Tue, 2013-03-19 at 15:59 -0700, Sage Weil wrote: On Tue, 19 Mar 2013, Laszlo Boszormenyi (GCS) wrote: On the other hand, 'dpkg --purge' is to remove everything the package has installed and/or generated. This includes debconf answers as well. With other words, purge is used to make the system totally clean of the package. As such, if the sysadmin install the package again, all debconf questions will be asked again and all generated files will be generated again from scratch. I understand that part, but the policy isn't very clear about files that are not part of the package but are generated as a result of the package being installed (i.e., user data). Forgive me, I just learnt English and my wording may not be that clear for a natural speaker. As a point of comparison, mysql removes the config files but not /var/lib/mysql. As I remember, MySQL asks if /var/lib/mysql/ should be purged or not; I may mix with an other (database related) package. The question is, is that okay/typical/desireable/recommended/a bad idea? I can rephrase my words. Purge removes the (binary) package files, its configuration and logs (its generated files). To emphasis, user files are _not_ fall into this category and must remain as-is, _intact_. Some packages writes a console message that 'your files remain at xxx, they were not removed' on purge. Others just leave the dpkg warning "directory not empty so not removed" which means user files may have left there and that may be the reason the directory is not empty. I'm in a rush, but hopefully will be able to note policy parts in the afternoon (CET). Thanks, Laszlo, that's exactly what I was after! Sorry for the confusing exchange. :) Sounds like in this case, the fix is simply to leave /var/lib/ceph untouched. We'll need to update teuthology ceph.py and nuke to clean up /var/lib/ceph (for qa runs), and I think we should add a ceph-deploy 'purgedata' command to clean out /var/lib/ceph on a given host. It's not as important given that it won't outright destroy the cluster, but perhaps we should also leave /etc/ceph untouched on purge if a ceph.conf file has been placed in it (since that also was not installed by the package, but rather by a user?). I figure we should probably try to get it right now. The message about the directory not being empty sounds good. My thought here is: - remove anything created by the packages in /var/lib/ceph that has been untouched since package installation. - remove /var/lib/ceph if it has been untouched - remove /etc/ceph if it has been untouched Thoughts? Thanks! sage Mark -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 20/03/13 14:48, Mark Nelson wrote: >> >> We'll need to update teuthology ceph.py and nuke to clean up >> /var/lib/ceph (for qa runs), and I think we should add a >> ceph-deploy 'purgedata' command to clean out /var/lib/ceph on a >> given host. > > It's not as important given that it won't outright destroy the > cluster, but perhaps we should also leave /etc/ceph untouched on > purge if a ceph.conf file has been placed in it (since that also > was not installed by the package, but rather by a user?). I figure > we should probably try to get it right now. The message about the > directory not being empty sounds good. > > My thought here is: > > - remove anything created by the packages in /var/lib/ceph that has > been untouched since package installation. - remove /var/lib/ceph > if it has been untouched - remove /etc/ceph if it has been > untouched If those directories are created by dpkg rather than maintainer scripts (i.e. in ceph.dirs rather than ceph.postinst) you should should get that all for free; if the directories contain anything dpkg does not know about it will just not removed them. - -- James Page Ubuntu Core Developer Debian Maintainer james.p...@ubuntu.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCAAGBQJRSfFoAAoJEL/srsug59jDJgsP/0zD4aKFtimFPh02/zdfJ3X0 BEY8Jmnmt3HcCxSPbGleZ+2p/38iLfLz8HdM1ZpOwGDVIv13N45vG0wW9zF/843R 8vKoGHJY7gAt/uY1fqa115m9txeNXAIZoaxwjrd6Zd31pgPvTBmfZhVsFKUnk7E5 9JmUs/K8gjjAZajhkUKgddp2ID70n/WGdHR+iu5cy72TyuvXVBQV1OmyYi9lMIxM yHGnCM/X7x5DE1g61x532VP2D0gAegA2WWURoqQ6vAM3IZfVGVuvat+HdzZ8Ej8z LfTk+8n2alTj6s1Xp3KGbb/D231MIi3VBaFMQx5pBlM29lAv8OYKidRQpZc9bZe2 5m5vhDutp4ZOZmqxDvDdayZgb/s8uVuodT2XK7qn4KbBRDEJN5aJKiUzXH0wTVTZ Lg/A/criFuzRP+ZH/Sh1pfSnLkNrrLMbdTglUv4krM2L6ZPOmEU3fh+UIXkr2u9t f6lnu4fVBwikDy/4hDztVL76IqB3wjnxYlJ1uHMHOrCugDeLRsGHdbCdFcoZonRW rUhdrtqzuuSbPHkzs/dEMCm4vF439YdmuL4WGsfzxEu+djcESDtzuAw4D1LRO12V JbEc9s8+L84oBJUv5dCSDG333jWc/8eihSs1ZG33NKZsWsNheppLR6aeXbg/nGi1 53R63uM1Lv7f1UaTxvH8 =6adU -END PGP SIGNATURE- -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
On Wed, 2013-03-20 at 05:48 -0700, Sage Weil wrote: > On Wed, 20 Mar 2013, Laszlo Boszormenyi (GCS) wrote: > > On Tue, 2013-03-19 at 15:59 -0700, Sage Weil wrote: > > > As a point of comparison, mysql removes the config files but not > > > /var/lib/mysql. > > > The question is, is that okay/typical/desireable/recommended/a bad idea? I should have asked this sooner. Do you know _any_ program that removes your favorite music collection, your family photos or your business emails when you do uninstall it? I suspect that your question was theoretical instead. On Wed, 2013-03-20 at 09:48 -0500, Mark Nelson wrote: On 03/20/2013 07:48 AM, Sage Weil wrote: > It's not as important given that it won't outright destroy the cluster, > but perhaps we should also leave /etc/ceph untouched on purge if a > ceph.conf file has been placed in it (since that also was not installed > by the package, but rather by a user?). I figure we should probably try > to get it right now. The message about the directory not being empty > sounds good. Sure, personal user data must be kept. If it's a big amount of data and left under a non-standard location (ie, not under his/her $HOME) then s/he should be informed where those files are located on purge. > My thought here is: > - remove anything created by the packages in /var/lib/ceph that has been > untouched since package installation. > - remove /var/lib/ceph if it has been untouched Please note that you have to store some kind of checksum for the files then. Probably md5sum is enough. > - remove /etc/ceph if it has been untouched This is an other case. dpkg itself handle package files here, called conffiles. I should check the method (md5sum and/or sha1 variants) used for the checksum on these files. On upgrade it's used not to overwrite local changes by the user. It may worth to read a bit more about it[1] from Raphaël Hertzog. He is the co-author of Debian Administrator's handbook[2] BTW. On purge dpkg will remove the package conffiles no matter what. It won't check if those were changed or not. You may not mark the files under /etc as conffiles, but then you'll lose the mentioned merge logic on upgrades; dpkg will just overwrite those. In short, files under /var/lib/ceph are the only candidates for in-package checksumming. How many files under there that essential for the packages? Laszlo/GCS [1] http://raphaelhertzog.com/2010/09/21/debian-conffile-configuration-file-managed-by-dpkg/ [2] http://debian-handbook.info/ -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: deb/rpm package purge
As a point of comparison, mysql removes the config files but not /var/lib/mysql. The question is, is that okay/typical/desireable/recommended/a bad idea? I should have asked this sooner. Do you know _any_ program that removes your favorite music collection, your family photos or your business emails when you do uninstall it? I suspect that your question was theoretical instead. It's somewhat different in that the data is not owned by one user, but there are clear parallels. The thing to be careful about here, IMO, is not only to preserve the data, but the associated files that allow (reasonably-easy) access to that data. (It's no good preserving the OSD filestore if the keys, monmap, or osdmap are gone or hard to recover.) - remove /etc/ceph if it has been untouched This is an other case. dpkg itself handle package files here, called conffiles. I should check the method (md5sum and/or sha1 variants) used for the checksum on these files. On upgrade it's used not to overwrite local changes by the user. It may worth to read a bit more about it[1] from Raphaël Hertzog. He is the co-author of Debian Administrator's handbook[2] BTW. Excellent reference; thanks for the pointer. On purge dpkg will remove the package conffiles no matter what. It won't check if those were changed or not. You may not mark the files under /etc as conffiles, but then you'll lose the mentioned merge logic on upgrades; dpkg will just overwrite those. In short, files under /var/lib/ceph are the only candidates for in-package checksumming. How many files under there that essential for the packages? Laszlo/GCS [1] http://raphaelhertzog.com/2010/09/21/debian-conffile-configuration-file-managed-by-dpkg/ [2] http://debian-handbook.info/ -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
