Re: [ceph-users] normalizing radosgw
I'm a big fan of /etc/*.d/ configs. Basically if the package maintained /etc/ceph.conf includes all files in /etc/ceph.d/ then I can break up the files however I'd like (mon, ods, mds, client, one per daemon, etc). Then when upgrading, I don't have to worry about the new packages trying to overwrite my conf file. If you have the include order be /var/lib/ceph/, /etc/ceph.conf, /etc/ceph.d/ then package maintainers can put in things so that it just works and can be easily overridden with /etc/ceph.d/. Just my $0.02. On Sat, Dec 6, 2014 at 11:39 AM, Sage Weil sw...@redhat.com wrote: Several things are different/annoying with radosgw than with other Ceph daemons: - binary/package are named 'radosgw' instead of 'ceph-rgw'. This is cosmetic, but it also makes it fit less well into the new /var/lib/ceph/* view of things. - default log location is /var/log/radosgw/$cluster-$name.log instead of /var/lib/ceph/$cluster-$rgw.$name.log (or similar). - rgw_data default is /var/lib/ceph/radosgw instead of /var/lib/ceph/rgw (not sure if 3 letters for consistency is better?) - rgw usually authenticates as a client.something user, which means if you do use more standard rgw log names, then you get /var/log/ceph/client.something.$pid.$uniqueid.log. There is a loose convention that 'something' is 'rgw.hostname' (i.e., client.rgw.hostname). - radosgw has it's own separate sysvinit script that enumerates daemons from /etc/ceph/ceph.conf sections that start with client.radosgw.* - radosgw upstart script is called radosgw (not ceph-rgw) and enumerates daemons from /var/lib/ceph/radosgw/* (totally different than sysvinit!) - radosgw instances usually need some stuff in ceph.conf to make them behave properly, which means they need a section in the shared /etc/ceph/ceph.conf. more work for admins or config management systems. - on rpm-based systems we have a separate sysvinit script that is slightly different (mostly because the username is different, apache instead of www-data). --- There is enough wrong here and little enough convention that my proposal is to essentially start fresh: - rename package ceph-rgw. obsoletes/replaces radosgw. - merge rgw start/stop into standard sysvinit script (just another daemon type?). - normalize upstart start/stop (just rename radosgw-* to ceph-rgw-*, basically). - make upstart use /var/lib/ceph/rgw instead of /var/lib/ceph/radosgw (do this automagically on upgrade? that will at least avoid the upgrade pain for ubuntu users) - create new systemd start/stop that are correct the first time around. - move log file to /var/log/ceph The part I'm not sure about is how to handle config. I would really like the ability to put per-daemon config stuff in /var/lib/ceph/rgw/foo/conf or config, but there is no 'include file' function in our ini-file parser (or in other typical ini-file implementations that i can find) and I don't like the idea the per-daemon config would completely supplant the /etc/ceph/ one. Basically, I'd like to get to a point where we can programatically deploy an rgw without any wonky code that edits the ceph.conf. Perhaps the way to accomplish this is to have the admin create a generic client.rgw user and generic [client.rgw] section in the global ceph.conf? If there were an include file function, we could make the per-daemon config file something like include /etc/ceph/$cluster.conf [client.rgw.hostname] rgw foo = bar rgw baz = blip or whatever. Thoughts? Suggestions? sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] normalizing radosgw
On Wed, 10 Dec 2014, Robert LeBlanc wrote: I'm a big fan of /etc/*.d/ configs. Basically if the package maintained /etc/ceph.conf includes all files in /etc/ceph.d/ then I can break up the files however I'd like (mon, ods, mds, client, one per daemon, etc). Then when upgrading, I don't have to worry about the new packages trying to overwrite my conf file. If you have the include order be /var/lib/ceph/, /etc/ceph.conf, /etc/ceph.d/ then package maintainers can put in things so that it just works and can be easily overridden with /etc/ceph.d/. Just my $0.02. How about /etc/ceph/ceph.conf.d/* ? In reality it's /etc/ceph/$cluster.conf.d/* sage On Sat, Dec 6, 2014 at 11:39 AM, Sage Weil sw...@redhat.com wrote: Several things are different/annoying with radosgw than with other Ceph daemons: - binary/package are named 'radosgw' instead of 'ceph-rgw'. This is cosmetic, but it also makes it fit less well into the new /var/lib/ceph/* view of things. - default log location is /var/log/radosgw/$cluster-$name.log instead of /var/lib/ceph/$cluster-$rgw.$name.log (or similar). - rgw_data default is /var/lib/ceph/radosgw instead of /var/lib/ceph/rgw (not sure if 3 letters for consistency is better?) - rgw usually authenticates as a client.something user, which means if you do use more standard rgw log names, then you get /var/log/ceph/client.something.$pid.$uniqueid.log. There is a loose convention that 'something' is 'rgw.hostname' (i.e., client.rgw.hostname). - radosgw has it's own separate sysvinit script that enumerates daemons from /etc/ceph/ceph.conf sections that start with client.radosgw.* - radosgw upstart script is called radosgw (not ceph-rgw) and enumerates daemons from /var/lib/ceph/radosgw/* (totally different than sysvinit!) - radosgw instances usually need some stuff in ceph.conf to make them behave properly, which means they need a section in the shared /etc/ceph/ceph.conf. more work for admins or config management systems. - on rpm-based systems we have a separate sysvinit script that is slightly different (mostly because the username is different, apache instead of www-data). --- There is enough wrong here and little enough convention that my proposal is to essentially start fresh: - rename package ceph-rgw. obsoletes/replaces radosgw. - merge rgw start/stop into standard sysvinit script (just another daemon type?). - normalize upstart start/stop (just rename radosgw-* to ceph-rgw-*, basically). - make upstart use /var/lib/ceph/rgw instead of /var/lib/ceph/radosgw (do this automagically on upgrade? that will at least avoid the upgrade pain for ubuntu users) - create new systemd start/stop that are correct the first time around. - move log file to /var/log/ceph The part I'm not sure about is how to handle config. I would really like the ability to put per-daemon config stuff in /var/lib/ceph/rgw/foo/conf or config, but there is no 'include file' function in our ini-file parser (or in other typical ini-file implementations that i can find) and I don't like the idea the per-daemon config would completely supplant the /etc/ceph/ one. Basically, I'd like to get to a point where we can programatically deploy an rgw without any wonky code that edits the ceph.conf. Perhaps the way to accomplish this is to have the admin create a generic client.rgw user and generic [client.rgw] section in the global ceph.conf? If there were an include file function, we could make the per-daemon config file something like include /etc/ceph/$cluster.conf [client.rgw.hostname] rgw foo = bar rgw baz = blip or whatever. Thoughts? Suggestions? sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] normalizing radosgw
On Wed, 10 Dec 2014, Robert LeBlanc wrote: I guess you would have to specify the cluster name in /etc/ceph/ceph.conf? That would be my only concern. ceph.conf is $cluster.conf (default cluster name is 'ceph'). Unfortunately under systemd it's not possible to parameterize daemons with two variables (cluster and id) so the cluster is fixed per-host in /etc/defaults/ceph (or something like that). At least that's how the current units work.. sage On Wed, Dec 10, 2014 at 1:28 PM, Sage Weil sw...@redhat.com wrote: On Wed, 10 Dec 2014, Robert LeBlanc wrote: I'm a big fan of /etc/*.d/ configs. Basically if the package maintained /etc/ceph.conf includes all files in /etc/ceph.d/ then I can break up the files however I'd like (mon, ods, mds, client, one per daemon, etc). Then when upgrading, I don't have to worry about the new packages trying to overwrite my conf file. If you have the include order be /var/lib/ceph/, /etc/ceph.conf, /etc/ceph.d/ then package maintainers can put in things so that it just works and can be easily overridden with /etc/ceph.d/. Just my $0.02. How about /etc/ceph/ceph.conf.d/* ? In reality it's /etc/ceph/$cluster.conf.d/* sage On Sat, Dec 6, 2014 at 11:39 AM, Sage Weil sw...@redhat.com wrote: Several things are different/annoying with radosgw than with other Ceph daemons: - binary/package are named 'radosgw' instead of 'ceph-rgw'. This is cosmetic, but it also makes it fit less well into the new /var/lib/ceph/* view of things. - default log location is /var/log/radosgw/$cluster-$name.log instead of /var/lib/ceph/$cluster-$rgw.$name.log (or similar). - rgw_data default is /var/lib/ceph/radosgw instead of /var/lib/ceph/rgw (not sure if 3 letters for consistency is better?) - rgw usually authenticates as a client.something user, which means if you do use more standard rgw log names, then you get /var/log/ceph/client.something.$pid.$uniqueid.log. There is a loose convention that 'something' is 'rgw.hostname' (i.e., client.rgw.hostname). - radosgw has it's own separate sysvinit script that enumerates daemons from /etc/ceph/ceph.conf sections that start with client.radosgw.* - radosgw upstart script is called radosgw (not ceph-rgw) and enumerates daemons from /var/lib/ceph/radosgw/* (totally different than sysvinit!) - radosgw instances usually need some stuff in ceph.conf to make them behave properly, which means they need a section in the shared /etc/ceph/ceph.conf. more work for admins or config management systems. - on rpm-based systems we have a separate sysvinit script that is slightly different (mostly because the username is different, apache instead of www-data). --- There is enough wrong here and little enough convention that my proposal is to essentially start fresh: - rename package ceph-rgw. obsoletes/replaces radosgw. - merge rgw start/stop into standard sysvinit script (just another daemon type?). - normalize upstart start/stop (just rename radosgw-* to ceph-rgw-*, basically). - make upstart use /var/lib/ceph/rgw instead of /var/lib/ceph/radosgw (do this automagically on upgrade? that will at least avoid the upgrade pain for ubuntu users) - create new systemd start/stop that are correct the first time around. - move log file to /var/log/ceph The part I'm not sure about is how to handle config. I would really like the ability to put per-daemon config stuff in /var/lib/ceph/rgw/foo/conf or config, but there is no 'include file' function in our ini-file parser (or in other typical ini-file implementations that i can find) and I don't like the idea the per-daemon config would completely supplant
Re: [ceph-users] normalizing radosgw
On Wed, 10 Dec 2014, Robert LeBlanc wrote: If cluster is specified in /etc/default/ceph than I don't have any other reservations to your proposal. Great. I like it much better than putting something in /var/lib/ceph/* (because it's bad policy) and this nicely sidesteps the 'include file' issue. Hmm, what is supposed to happen in general if you have /etc/foo /etc/foo.d/00-a /etc/foo.d/01-b Is the result a concatentation of /etc/foo + the .d files? I would guess at the beginning? sage On Wed, Dec 10, 2014 at 1:39 PM, Sage Weil sw...@redhat.com wrote: On Wed, 10 Dec 2014, Robert LeBlanc wrote: I guess you would have to specify the cluster name in /etc/ceph/ceph.conf? That would be my only concern. ceph.conf is $cluster.conf (default cluster name is 'ceph'). Unfortunately under systemd it's not possible to parameterize daemons with two variables (cluster and id) so the cluster is fixed per-host in /etc/defaults/ceph (or something like that). At least that's how the current units work.. sage On Wed, Dec 10, 2014 at 1:28 PM, Sage Weil sw...@redhat.com wrote: On Wed, 10 Dec 2014, Robert LeBlanc wrote: I'm a big fan of /etc/*.d/ configs. Basically if the package maintained /etc/ceph.conf includes all files in /etc/ceph.d/ then I can break up the files however I'd like (mon, ods, mds, client, one per daemon, etc). Then when upgrading, I don't have to worry about the new packages trying to overwrite my conf file. If you have the include order be /var/lib/ceph/, /etc/ceph.conf, /etc/ceph.d/ then package maintainers can put in things so that it just works and can be easily overridden with /etc/ceph.d/. Just my $0.02. How about /etc/ceph/ceph.conf.d/* ? In reality it's /etc/ceph/$cluster.conf.d/* sage On Sat, Dec 6, 2014 at 11:39 AM, Sage Weil sw...@redhat.com wrote: Several things are different/annoying with radosgw than with other Ceph daemons: - binary/package are named 'radosgw' instead of 'ceph-rgw'. This is cosmetic, but it also makes it fit less well into the new /var/lib/ceph/* view of things. - default log location is /var/log/radosgw/$cluster-$name.log instead of /var/lib/ceph/$cluster-$rgw.$name.log (or similar). - rgw_data default is /var/lib/ceph/radosgw instead of /var/lib/ceph/rgw (not sure if 3 letters for consistency is better?) - rgw usually authenticates as a client.something user, which means if you do use more standard rgw log names, then you get /var/log/ceph/client.something.$pid.$uniqueid.log. There is a loose convention that 'something' is 'rgw.hostname' (i.e., client.rgw.hostname). - radosgw has it's own separate sysvinit script that enumerates daemons from /etc/ceph/ceph.conf sections that start with client.radosgw.* - radosgw upstart script is called radosgw (not ceph-rgw) and enumerates daemons from /var/lib/ceph/radosgw/* (totally different than sysvinit!) - radosgw instances usually need some stuff in ceph.conf to make them behave properly, which means they need a section in the shared /etc/ceph/ceph.conf. more work for admins or config management systems. - on rpm-based systems we have a separate sysvinit script that is slightly different (mostly because the username is different, apache instead of www-data).
Re: [ceph-users] normalizing radosgw
I would suggest that it would be: /etc/ceph/$cluster.conf /etc/ceph/$cluster.d/00-a.conf /etc/ceph/$cluster.d/01-b.conf To make sure there is no conflict with anything else in /etc/ and it is easy to always find the ceph configs instead of being mixed in with everything else. I also like the idea of the .conf extension in $cluster.d/. It makes it easy to disable a part of the config by appending .disabled or changing the extension in some way. Kind of similar to /etc/sudoers.d/ (look at the README file in that directory). If it is like other config files, options later in the file override options earlier in the file. That makes it easy to have defaults in /etc/ceph/$cluster.conf and then overrode in /etc/ceph/$cluster.d/*, but I don't think this is such a big deal for Ceph. Are you thinking about hard coding the concatenation in the code, or adding and include directive? Adding the directive could be useful for people who would like their configs somewhere else and would provide the most flexibility (but it sounds like it was there before and removed). It would be nice if it were a directory that it would just lexically cat all files in that directory that end in .conf. I thought there would be more input on this topic. I know that some people are vehemently opposed to *.d/, but I have really come to like it and cringe when something doesn't support it. On Wed, Dec 10, 2014 at 1:48 PM, Sage Weil sw...@redhat.com wrote: On Wed, 10 Dec 2014, Robert LeBlanc wrote: If cluster is specified in /etc/default/ceph than I don't have any other reservations to your proposal. Great. I like it much better than putting something in /var/lib/ceph/* (because it's bad policy) and this nicely sidesteps the 'include file' issue. Hmm, what is supposed to happen in general if you have /etc/foo /etc/foo.d/00-a /etc/foo.d/01-b Is the result a concatentation of /etc/foo + the .d files? I would guess at the beginning? sage On Wed, Dec 10, 2014 at 1:39 PM, Sage Weil sw...@redhat.com wrote: On Wed, 10 Dec 2014, Robert LeBlanc wrote: I guess you would have to specify the cluster name in /etc/ceph/ceph.conf? That would be my only concern. ceph.conf is $cluster.conf (default cluster name is 'ceph'). Unfortunately under systemd it's not possible to parameterize daemons with two variables (cluster and id) so the cluster is fixed per-host in /etc/defaults/ceph (or something like that). At least that's how the current units work.. sage On Wed, Dec 10, 2014 at 1:28 PM, Sage Weil sw...@redhat.com wrote: On Wed, 10 Dec 2014, Robert LeBlanc wrote: I'm a big fan of /etc/*.d/ configs. Basically if the package maintained /etc/ceph.conf includes all files in /etc/ceph.d/ then I can break up the files however I'd like (mon, ods, mds, client, one per daemon, etc). Then when upgrading, I don't have to worry about the new packages trying to overwrite my conf file. If you have the include order be /var/lib/ceph/, /etc/ceph.conf, /etc/ceph.d/ then package maintainers can put in things so that it just works and can be easily overridden with /etc/ceph.d/. Just my $0.02. How about /etc/ceph/ceph.conf.d/* ? In reality it's /etc/ceph/$cluster.conf.d/* sage On Sat, Dec 6, 2014 at 11:39 AM, Sage Weil sw...@redhat.com wrote: Several things are different/annoying with radosgw than with other Ceph daemons: - binary/package are named 'radosgw' instead of 'ceph-rgw'. This is cosmetic, but it also makes it fit less well into the new /var/lib/ceph/* view of things. - default log location is /var/log/radosgw/$cluster-$name.log instead of /var/lib/ceph/$cluster-$rgw.$name.log (or similar). - rgw_data default is /var/lib/ceph/radosgw instead of /var/lib/ceph/rgw (not sure if 3 letters for consistency is better?) - rgw usually authenticates as a client.something user, which means if you do use more standard rgw log names, then you
Re: [ceph-users] normalizing radosgw
On Wed, 10 Dec 2014, Robert LeBlanc wrote: I would suggest that it would be: /etc/ceph/$cluster.conf /etc/ceph/$cluster.d/00-a.conf /etc/ceph/$cluster.d/01-b.conf Yeah If it is like other config files, options later in the file override options earlier in the file. That makes it easy to have defaults in /etc/ceph/$cluster.conf and then overrode in /etc/ceph/$cluster.d/*, but I don't think this is such a big deal for Ceph. Are you thinking about hard coding the concatenation in the code, or adding and include directive? Adding the directive could be useful for people who would like their configs somewhere else and would provide the most flexibility (but it sounds like it was there before and removed). It would be nice if it were a directory that it would just lexically cat all files in that directory that end in .conf. Yeah. I'm wondering if it is well-defined practice that foo comes before foo.d/* is all. It looks like sudoers does it explicitly with an #includdir dirname line... perhaps that is what we should do too. I thought there would be more input on this topic. I know that some people are vehemently opposed to *.d/, but I have really come to like it and cringe when something doesn't support it. +1 sage On Wed, Dec 10, 2014 at 1:48 PM, Sage Weil sw...@redhat.com wrote: On Wed, 10 Dec 2014, Robert LeBlanc wrote: If cluster is specified in /etc/default/ceph than I don't have any other reservations to your proposal. Great. I like it much better than putting something in /var/lib/ceph/* (because it's bad policy) and this nicely sidesteps the 'include file' issue. Hmm, what is supposed to happen in general if you have /etc/foo /etc/foo.d/00-a /etc/foo.d/01-b Is the result a concatentation of /etc/foo + the .d files? I would guess at the beginning? sage On Wed, Dec 10, 2014 at 1:39 PM, Sage Weil sw...@redhat.com wrote: On Wed, 10 Dec 2014, Robert LeBlanc wrote: I guess you would have to specify the cluster name in /etc/ceph/ceph.conf? That would be my only concern. ceph.conf is $cluster.conf (default cluster name is 'ceph'). Unfortunately under systemd it's not possible to parameterize daemons with two variables (cluster and id) so the cluster is fixed per-host in /etc/defaults/ceph (or something like that). At least that's how the current units work.. sage On Wed, Dec 10, 2014 at 1:28 PM, Sage Weil sw...@redhat.com wrote: On Wed, 10 Dec 2014, Robert LeBlanc wrote: I'm a big fan of /etc/*.d/ configs. Basically if the package maintained /etc/ceph.conf includes all files in /etc/ceph.d/ then I can break up the files however I'd like (mon, ods, mds, client, one per daemon, etc). Then when upgrading, I don't have to worry about the new packages trying to overwrite my conf file. If you have the include order be /var/lib/ceph/, /etc/ceph.conf, /etc/ceph.d/ then package maintainers can put in things so that it just works and can be easily overridden with /etc/ceph.d/. Just my $0.02. How about /etc/ceph/ceph.conf.d/* ? In reality it's /etc/ceph/$cluster.conf.d/* sage On Sat, Dec 6, 2014 at 11:39 AM, Sage Weil sw...@redhat.com wrote: Several things are different/annoying with radosgw than with other Ceph daemons: - binary/package are named 'radosgw' instead of 'ceph-rgw'. This is cosmetic, but it also makes it fit less well into the new /var/lib/ceph/* view of things. - default log location is /var/log/radosgw/$cluster-$name.log instead of
Re: [ceph-users] normalizing radosgw
Suggestion: can we look at normalizing custom cluster names as well for RHEL-like systems? /etc/init.d/ceph: how do you pass --cluster myname at system startup? systemd service file uses EnvironmentFile. What about . /etc/sysconfig/ceph #near the top and /etc/sysconfig/ceph contains cluster=myname For ceph-radosgw even this does not work. You need export CEPH_CONF=/etc/ceph/myname.conf as well. Anthony ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] normalizing radosgw
On Sat, Dec 6, 2014 at 10:39 AM, Sage Weil sw...@redhat.com wrote: Several things are different/annoying with radosgw than with other Ceph daemons: - binary/package are named 'radosgw' instead of 'ceph-rgw'. This is cosmetic, but it also makes it fit less well into the new /var/lib/ceph/* view of things. - default log location is /var/log/radosgw/$cluster-$name.log instead of /var/lib/ceph/$cluster-$rgw.$name.log (or similar). - rgw_data default is /var/lib/ceph/radosgw instead of /var/lib/ceph/rgw (not sure if 3 letters for consistency is better?) - rgw usually authenticates as a client.something user, which means if you do use more standard rgw log names, then you get /var/log/ceph/client.something.$pid.$uniqueid.log. There is a loose convention that 'something' is 'rgw.hostname' (i.e., client.rgw.hostname). Maybe we can make it so that unless user name starts with mon, osd, or mds, it's going to be treated as 'client'? Or something along these lines. - radosgw has it's own separate sysvinit script that enumerates daemons from /etc/ceph/ceph.conf sections that start with client.radosgw.* - radosgw upstart script is called radosgw (not ceph-rgw) and enumerates daemons from /var/lib/ceph/radosgw/* (totally different than sysvinit!) - radosgw instances usually need some stuff in ceph.conf to make them behave properly, which means they need a section in the shared /etc/ceph/ceph.conf. more work for admins or config management systems. - on rpm-based systems we have a separate sysvinit script that is slightly different (mostly because the username is different, apache instead of www-data). --- There is enough wrong here and little enough convention that my proposal is to essentially start fresh: - rename package ceph-rgw. obsoletes/replaces radosgw. - merge rgw start/stop into standard sysvinit script (just another daemon type?). - normalize upstart start/stop (just rename radosgw-* to ceph-rgw-*, basically). - make upstart use /var/lib/ceph/rgw instead of /var/lib/ceph/radosgw (do this automagically on upgrade? that will at least avoid the upgrade pain for ubuntu users) - create new systemd start/stop that are correct the first time around. - move log file to /var/log/ceph The part I'm not sure about is how to handle config. I would really like the ability to put per-daemon config stuff in /var/lib/ceph/rgw/foo/conf or config, but there is no 'include file' function in our ini-file parser (or in other typical ini-file implementations that i can find) and I don't The samba configuration files, which our ceph conf was heavily influenced by does have that capability. We also used to be able to do: include = other include file but sadly that was taken out during some cleanup years ago. I don't think it's going to be too complicated to revive this functionality. like the idea the per-daemon config would completely supplant the /etc/ceph/ one. Basically, I'd like to get to a point where we can programatically deploy an rgw without any wonky code that edits the ceph.conf. Perhaps the way to accomplish this is to have the admin create a generic client.rgw user and generic [client.rgw] section in the global ceph.conf? If there were an include file function, we could make the per-daemon config file something like include /etc/ceph/$cluster.conf [client.rgw.hostname] rgw foo = bar rgw baz = blip or whatever. Thoughts? Suggestions? sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] normalizing radosgw
On 07/12/14 07:39, Sage Weil wrote: Thoughts? Suggestions? Would kit make sense to include radosgw-agent package in this normalization too? Regards Mark ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
