Re: [ceph-users] RGW lifecycle not expiring objects

Wed, 28 Jun 2017 07:48:27 -0700

This is almost certainly because it's using v4 auth, which is not well supported in RGW yet. Can you try with v3 auth? 

Daniel

On 06/27/2017 07:57 PM, Graham Allan wrote:

I upgraded my test cluster to Luminous 12.1.0, and a separate problem
made me realize a possible cause for the lifecycle failure.

After the upgrade, radosgw spammed its logfile with "failed to list
reshard log entries" errors. I realized that the radosgw user only had
authorization to access specific osd pools - and not the newly-created
"default.rgw.reshard" pool.

Which made me also realize, it had no access to the new-for-kraken
"default.rgw.lc" pool. That may explain the failure to process lifecycle
below...

The behavior has now changed. Listing lifecycle on my test bucket gives
different output:

% aws --endpoint-url https://xxx.xxx.xxx.xxx s3api \
    get-bucket-lifecycle-configuration --bucket=testgta
{
    "Rules": [
        {
            "Status": "Enabled",
            "Prefix": "",
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "ID": "test"
        }
    ]
}

Editing this lifecycle to add back the now-missing "Days" parameter
results in a NotImplemented error:

% aws --endpoint-url https://xxx.xxx.xxx.xxx s3api \
    put-bucket-lifecycle-configuration --bucket=testgta \
    --lifecycle-configuration  file://lifecycle.json

An error occurred (NotImplemented) when calling the
PutBucketLifecycleConfiguration operation: Unknown

and the lifecycle is not updated.

So, I think I understand why it may not have worked before, but the
goalposts seem to have changed to a new problem.

Would appreciate any ideas...

Graham

On 06/08/2017 11:54 AM, Graham Allan wrote:

Sorry I didn't get to reply until now. The thing is I believe I *do*
have a lifecycle configured on at least one bucket. As noted in that
issue, I get an error returned when trying to set the lifecycle, but
it does appear to get stored:

% aws --endpoint-url https://xxx.xxx.xxx.xxx s3api \
   get-bucket-lifecycle-configuration --bucket=testgta
{
     "Rules": [
         {
             "Status": "Enabled",
             "Prefix": "",
             "Expiration": {
                 "Days": 3
             },
             "ID": "test"
         }
     ]
}

Does that look valid? Is there some other configuration which is
needed besides this? The rgw config values for lifecycle look as
expected.

Thanks,

Graham

On 06/06/2017 01:43 PM, Ben Hines wrote:

If you have nothing listed in 'lc list', you probably need to add a
lifecycle configuration using the S3 API. It's not automatic and has
to be added per-bucket.


Here's some sample code for doing so:
http://tracker.ceph.com/issues/19587

-Ben

On Tue, Jun 6, 2017 at 9:07 AM, Graham Allan <g...@umn.edu
<mailto:g...@umn.edu>> wrote:

    I still haven't seen anything get expired from our kraken (11.2.0)
    system.

    When I run "radosgw-admin lc list" I get no output, besides debug
    output (I have "debug rgw = 10" at present):

    # radosgw-admin lc list
    2017-06-06 10:57:49.319576 7f2b26ffd700  2
    RGWDataChangesLog::ChangesRenewThread: start
    2017-06-06 10:57:49.350646 7f2b49558c80 10 Cannot find current
    period zone using local zone
    2017-06-06 10:57:49.379065 7f2b49558c80  2 all 8 watchers are set,
    enabling cache
    []
    2017-06-06 10:57:49.399538 7f2b49558c80  2 removed watcher,
    disabling cache

    Unclear to me whether the debug message about "Cannot find current
    period zone using local zone" is related or indicates a problem.

    Currently all the lc config is more or less default, eg a few
values:

    # ceph --show-config|grep rgw_|grep lifecycle
    rgw_lifecycle_enabled = true
    rgw_lifecycle_thread = 1
    rgw_lifecycle_work_time = 00:00-06:00


    Graham

    On 06/05/2017 01:07 PM, Ben Hines wrote:

        FWIW lifecycle is working for us. I did have to research to find
        the appropriate lc config file settings, the documentation for
        which is found in a git pull request (waiting for another
        release?) rather than on the Ceph docs site.
        https://github.com/ceph/ceph/pull/13990
        <https://github.com/ceph/ceph/pull/13990>


        Try these:
        debug rgw = 20
        rgw lifecycle work time = 00:01-23:59


        and see if you have lifecycles listed when you run:


        radosgw-admin lc list


        2017-06-05 10:58:00.473957 7f3429f77c80  0 System already
converted
        [
              {
                  "bucket": ":bentest:default.653959.6",
                  "status": "COMPLETE"
              },
              {
                  "bucket": ":<redacted>:default.24713983.1",
                  "status": "PROCESSING"
              },
              {
                  "bucket": ":<redacted>:default.24713983.2",
                  "status": "PROCESSING"
              },

        ....


        At 10 loglevel, the lifecycle processor logs 'DELETED' each time
        it deletes something:
        https://github.com/ceph/ceph/blob/master/src/rgw/rgw_lc.cc#L388

<https://github.com/ceph/ceph/blob/master/src/rgw/rgw_lc.cc#L388>

           grep --text DELETED client.<hostname>.log | wc -l
        121853


        -Ben

        On Mon, Jun 5, 2017 at 6:16 AM, Daniel Gryniewicz
        <d...@redhat.com <mailto:d...@redhat.com>
        <mailto:d...@redhat.com <mailto:d...@redhat.com>>> wrote:

             Kraken has lifecycle, Jewel does not.

             Daniel


             On 06/04/2017 07:16 PM, ceph.nov...@habmalnefrage.de
        <mailto:ceph.nov...@habmalnefrage.de>
             <mailto:ceph.nov...@habmalnefrage.de
        <mailto:ceph.nov...@habmalnefrage.de>> wrote:


                 grrr... sorry && and again as text :|


                 Gesendet: Montag, 05. Juni 2017 um 01:12 Uhr
                 Von: ceph.nov...@habmalnefrage.de
        <mailto:ceph.nov...@habmalnefrage.de>
                 <mailto:ceph.nov...@habmalnefrage.de
        <mailto:ceph.nov...@habmalnefrage.de>>
                 An: "Yehuda Sadeh-Weinraub" <yeh...@redhat.com
        <mailto:yeh...@redhat.com>
                 <mailto:yeh...@redhat.com <mailto:yeh...@redhat.com>>>
                 Cc: "ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>
                 <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>" <ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>
                 <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>>, ceph-de...@vger.kernel.org
        <mailto:ceph-de...@vger.kernel.org>
                 <mailto:ceph-de...@vger.kernel.org
        <mailto:ceph-de...@vger.kernel.org>>
                 Betreff: Re: [ceph-users] RGW lifecycle not expiring
        objects



                 Hi (again) Yehuda.

                 Looping in ceph-devel...

                 Could it be that lifecycle is still not implemented
        neither in
                 Jewel nor in Kraken, even if release notes and other
        places say so?

        https://www.spinics.net/lists/ceph-devel/msg34492.html
        <https://www.spinics.net/lists/ceph-devel/msg34492.html>
                 <https://www.spinics.net/lists/ceph-devel/msg34492.html
        <https://www.spinics.net/lists/ceph-devel/msg34492.html>>

https://github.com/ceph/ceph-ci/commit/7d48f62f5c86913d8f00b44d46a04a52d338907c


<https://github.com/ceph/ceph-ci/commit/7d48f62f5c86913d8f00b44d46a04a52d338907c>


<https://github.com/ceph/ceph-ci/commit/7d48f62f5c86913d8f00b44d46a04a52d338907c


<https://github.com/ceph/ceph-ci/commit/7d48f62f5c86913d8f00b44d46a04a52d338907c>>


https://github.com/ceph/ceph-ci/commit/9162bd29594d34429a09562ed60a32a0703940ea


<https://github.com/ceph/ceph-ci/commit/9162bd29594d34429a09562ed60a32a0703940ea>


<https://github.com/ceph/ceph-ci/commit/9162bd29594d34429a09562ed60a32a0703940ea


<https://github.com/ceph/ceph-ci/commit/9162bd29594d34429a09562ed60a32a0703940ea>>


                 Thanks & regards
                   Anton


                 Gesendet: Sonntag, 04. Juni 2017 um 21:34 Uhr
                 Von: ceph.nov...@habmalnefrage.de
        <mailto:ceph.nov...@habmalnefrage.de>
                 <mailto:ceph.nov...@habmalnefrage.de
        <mailto:ceph.nov...@habmalnefrage.de>>
                 An: "Yehuda Sadeh-Weinraub" <yeh...@redhat.com
        <mailto:yeh...@redhat.com>
                 <mailto:yeh...@redhat.com <mailto:yeh...@redhat.com>>>
                 Cc: "ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>
                 <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>" <ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>
                 <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>>
                 Betreff: Re: [ceph-users] RGW lifecycle not expiring
        objects
                 Hi Yahuda.

                 Well, here we go:

http://tracker.ceph.com/issues/20177[http://tracker.ceph.com/issues/20177]


<http://tracker.ceph.com/issues/20177%5Bhttp://tracker.ceph.com/issues/20177%5D>


<http://tracker.ceph.com/issues/20177%5Bhttp://tracker.ceph.com/issues/20177%5D


<http://tracker.ceph.com/issues/20177%5Bhttp://tracker.ceph.com/issues/20177%5D>>


                 As it's my first one, hope it's ok as it is...

                 Thanks & regards
                 Anton


                 Gesendet: Samstag, 03. Juni 2017 um 00:14 Uhr
                 Von: "Yehuda Sadeh-Weinraub" <yeh...@redhat.com
        <mailto:yeh...@redhat.com>
                 <mailto:yeh...@redhat.com <mailto:yeh...@redhat.com>>>
                 An: ceph.nov...@habmalnefrage.de
        <mailto:ceph.nov...@habmalnefrage.de>
                 <mailto:ceph.nov...@habmalnefrage.de
        <mailto:ceph.nov...@habmalnefrage.de>>
                 Cc: "Graham Allan" <g...@umn.edu <mailto:g...@umn.edu>
        <mailto:g...@umn.edu <mailto:g...@umn.edu>>>,
                 "ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>
        <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>"
                 <ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>
        <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>>
                 Betreff: Re: [ceph-users] RGW lifecycle not expiring
        objects
                 Have you opened a ceph tracker issue, so that we don't
        lose track of
                 the problem?

                 Thanks,
                 Yehuda

                 On Fri, Jun 2, 2017 at 3:05 PM,
        <ceph.nov...@habmalnefrage.de
<mailto:ceph.nov...@habmalnefrage.de>
                 <mailto:ceph.nov...@habmalnefrage.de
        <mailto:ceph.nov...@habmalnefrage.de>>> wrote:

                     Hi Graham.

                     We are on Kraken and have the same problem with
        "lifecycle".
                     Various (other) tools like s3cmd or CyberDuck do
        show the
                     applied "expiration" settings, but objects seem
        never to be
                     purged.

                     If you should have new findings, hints,... PLEASE
        share/let
                     me know.

                     Thanks a lot!
                     Anton


                     Gesendet: Freitag, 19. Mai 2017 um 22:44 Uhr
                     Von: "Graham Allan" <g...@umn.edu
        <mailto:g...@umn.edu> <mailto:g...@umn.edu <mailto:g...@umn.edu>>>
                     An: ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>
        <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>

                     Betreff: [ceph-users] RGW lifecycle not expiring
        objects
                     I've been having a hard time getting the s3 object
        lifecycle
                     to do
                     anything here. I was able to set a lifecycle on
a test
                     bucket. As others
                     also seem to have found, I do get an EACCES error
        on setting the
                     lifecycle, but it does however get stored:

                         % aws --endpoint-url
        https://xxx.xxx.xxx.xxx[https://xxx.xxx.xxx.xxx
        <https://xxx.xxx.xxx.xxx>
        <https://xxx.xxx.xxx.xxx>][https://xxx.xxx.xxx.xxx
        <https://xxx.xxx.xxx.xxx>
        <https://xxx.xxx.xxx.xxx>[https://xxx.xxx.xxx.xxx
        <https://xxx.xxx.xxx.xxx>
                         <https://xxx.xxx.xxx.xxx>]] s3api
                         get-bucket-lifecycle-configuration
--bucket=testgta
                         {
                         "Rules": [
                         {
                         "Status": "Enabled",
                         "Prefix": "",
                         "Expiration": {
                         "Days": 3
                         },
                         "ID": "test"
                         }
                         ]
                         }


                     but many days later I have yet to see any object
        actually
                     get expired.
                     There are some hints in the rgw log that the expiry
        thread
                     does run
                     periodically:

                         2017-05-19 03:49:03.281347 7f74f1134700 2
                         RGWDataChangesLog::ChangesRenewThread: start
                         2017-05-19 03:49:16.356022 7f74ef931700 2
object
                         expiration: start
                         2017-05-19 03:49:16.356036 7f74ef931700 20
        proceeding
                         shard = obj_delete_at_hint.0000000000
                         2017-05-19 03:49:16.359785 7f74ef931700 20
        proceeding
                         shard = obj_delete_at_hint.0000000001
                         2017-05-19 03:49:16.364667 7f74ef931700 20
        proceeding
                         shard = obj_delete_at_hint.0000000002
                         2017-05-19 03:49:16.369636 7f74ef931700 20
        proceeding
                         shard = obj_delete_at_hint.0000000003

                     ...

                         2017-05-19 03:49:16.803270 7f74ef931700 20
        proceeding
                         shard = obj_delete_at_hint.0000000126
                         2017-05-19 03:49:16.806423 7f74ef931700 2
object
                         expiration: stop


                     "radosgw-admin lc process" gives me no output
unless I
                     enable debug, then:

                         ]# radosgw-admin lc process
                         2017-05-19 15:28:46.383049 7fedb9ffb700 2
                         RGWDataChangesLog::ChangesRenewThread: start
                         2017-05-19 15:28:46.421806 7feddc240c80 10
        Cannot find
                         current period zone using local zone
                         2017-05-19 15:28:46.453431 7feddc240c80 2 all 8
        watchers
                         are set, enabling cache
                         2017-05-19 15:28:46.614991 7feddc240c80 2
removed
                         watcher, disabling cache


                     "radosgw-admin lc list" seems to return "empty"
output:

                         # radosgw-admin lc list
                         []


                     Is there anything obvious that I might be missing?

                     Graham
                     --
                     Graham Allan
                     Minnesota Supercomputing Institute - g...@umn.edu
        <mailto:g...@umn.edu>
                     <mailto:g...@umn.edu <mailto:g...@umn.edu>>
                     _______________________________________________
                     ceph-users mailing list
        ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
        <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]]]


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5D>


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5D


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5D>>



                     _______________________________________________
                     ceph-users mailing list
        ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
        <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]]


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D>


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D>>


                 _______________________________________________
                 ceph-users mailing list
        ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
        <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D>


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D>>


                   _______________________________________________
        ceph-users
                 mailing list ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>
                 <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D>


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D


<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D>>

                 _______________________________________________
                 ceph-users mailing list
        ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
        <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>
        http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
        <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
                 <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
        <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>>


             _______________________________________________
             ceph-users mailing list
        ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
        <mailto:ceph-users@lists.ceph.com
        <mailto:ceph-users@lists.ceph.com>>
        http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
        <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
             <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
        <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>>




        _______________________________________________
        ceph-users mailing list
        ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
        http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
        <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>


    --     Graham Allan
    Minnesota Supercomputing Institute - g...@umn.edu
<mailto:g...@umn.edu>
    _______________________________________________
    ceph-users mailing list
    ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
    http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
    <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>








_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to