Re: [ceph-users] RGW Swift metadata dropped when S3 bucket versioning enabled

[Yehuda Sadeh-Weinraub]

On Wed, Nov 28, 2018 at 10:07 AM Maxime Guyot  wrote:
>
> Hi Florian,
>
> You assumed correctly, the "test" container (private) was created with the 
> "openstack container create test", then I am using the S3 API to 
> enable/disable object versioning on it.
> I use the following Python snippet to enable/disable S3 bucket versioning:
>
> import boto, boto.s3, boto.s3.connection
> conn = conn = boto.connect_s3(aws_access_key_id='***', 
> aws_secret_access_key='***', host='***', port=8080, 
> calling_format=boto.s3.connection.OrdinaryCallingFormat())
> bucket = conn.get_bucket('test')
> bucket.configure_versioning(True) # Or False to disable S3 bucket versioning
> bucket.get_versioning_status()
>
> > Semi-related: I've seen some interesting things when mucking around with
> > a single container/bucket while switching APIs, when it comes to
> > container properties and metadata. For example, if you set a public read
> > ACL on an S3 bucket, the the corresponding Swift container is also
> > publicly readable but its read ACL looks empty (i.e. private) when you
> > ask via the Swift API.
>
> This can definitely become a problem if Swift API says "private" but data is 
> actually publicly available.
> Since the doc says "S3 and Swift APIs share a common namespace, so you may 
> write data with one API and retrieve it with the other", it might be useful 
> to document this kind of limitations somewhere.

Note that swift acls and S3 acls don't quite map perfectly to each
other. When S3 public read acl on a bucket doesn't mean that data is
accessible, but rather that bucket can be listed. In swift the
container acls are about the objects inside. Not sure that there is an
equivalent swift acl that would only deal with ability to list objects
in the container.

Yehuda
>
> Cheers,
> / Maxime
>
> On Wed, 28 Nov 2018 at 17:58 Florian Haas  wrote:
>>
>> On 27/11/2018 20:28, Maxime Guyot wrote:
>> > Hi,
>> >
>> > I'm running into an issue with the RadosGW Swift API when the S3 bucket
>> > versioning is enabled. It looks like it silently drops any metadata sent
>> > with the "X-Object-Meta-foo" header (see example below).
>> > This is observed on a Luminous 12.2.8 cluster. Is that a normal thing?
>> > Am I misconfiguring something here?
>> >
>> >
>> > With S3 bucket versioning OFF:
>> > $ openstack object set --property foo=bar test test.dat
>> > $ os object show test test.dat
>> > ++--+
>> > | Field  | Value|
>> > ++--+
>> > | account| v1   |
>> > | container  | test |
>> > | content-length | 507904   |
>> > | content-type   | binary/octet-stream  |
>> > | etag   | 03e8a398f343ade4e1e1d7c81a66e400 |
>> > | last-modified  | Tue, 27 Nov 2018 13:53:54 GMT|
>> > | object | test.dat |
>> > | properties | Foo='bar'|  <= Metadata is here
>> > ++--+
>> >
>> > With S3 bucket versioning ON:
>>
>> Can you elaborate on what exactly you're doing here to enable S3 bucket
>> versioning? Do I assume correctly that you are creating the "test"
>> container using the swift or openstack client, then sending a
>> VersioningConfiguration request against the "test" bucket, as explained
>> in
>> https://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html#how-to-enable-disable-versioning-intro?
>>
>> > $ openstack object set --property foo=bar test test2.dat
>> > $ openstack object show test test2.dat
>> > ++--+
>> > | Field  | Value|
>> > ++--+
>> > | account| v1   |
>> > | container  | test |
>> > | content-length | 507904   |
>> > | content-type   | binary/octet-stream  |
>> > | etag   | 03e8a398f343ade4e1e1d7c81a66e400 |
>> > | last-modified  | Tue, 27 Nov 2018 13:56:50 GMT|
>> > | object | test2.dat| <= Metadata is absent
>> > ++--+
>>
>> Semi-related: I've seen some interesting things when mucking around with
>> a single container/bucket while switching APIs, when it comes to
>> container properties and metadata. For example, if you set a public read
>> ACL on an S3 bucket, the the corresponding Swift container is also
>> publicly readable but its read ACL looks empty (i.e. private) when you
>> ask via the Swift API.
>>
>> Cheers,
>> Florian
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users 

Re: [ceph-users] Radosgw index has been inconsistent with reality

[Yehuda Sadeh-Weinraub]

On Wed, Oct 17, 2018 at 1:14 AM Yang Yang  wrote:
>
> Hi,
> A few weeks ago I found radosgw index has been inconsistent with reality. 
> Some object I can not list, but I can get them by key. Please see the details 
> below:
>
> BACKGROUND：
> Ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous 
> (stable)
> Index pool is on ssd.
> There is a very big bucket with more than 10 million object and 500TB 
> data.
> Ceph health is OK.
> I use s3 api on radosgw.
>
> DESCRIBE:
> When use s3 list_object() to list, some uploaded object can not be listed 
> and some uploaded object have an old lastModified time.
> But at the same time, we can get this object by an exact key. And if I 
> put a new object into this bucket, it can be listed.
> It seems that some indexes during a period of time have been lost.
>
> I try to run "radosgw-admin bucket check --bucket  --fix 
> --check-objects" and I get nothing at all.
>
> SOME ELSE：
> I found that one bucket will have many indexes, and we can use 
> "radosgw-admin metadata list bucket.instance | grep "{bucket name}" to show 
> them. But I can not found a doc to describe this feature. And we can use 
> "radosgw-admin bucket stats --bucket {bucket_name}" to get id as the active 
> instance id.
> I use "rados listomapkeys" at active(or latest) index to get all object 
> in a index, it is really lost. But when I use "rados listomapkeys" at another 
> index which is not active as mentioned above, I found the lost object index.
>
> Resharding is within my consideration. Listomapkeys means do this action 
> on all shards(more than 300).
> In my understanding, a big bucket has one latest index and many old 
> indexes. Every index has many shards. So listomapkeys on a index means 
> listomapkeys on many shards.
>
> QUESTION:
> Why my index lost?
> How to recover？

I don't really know what happened, haven't seen this exact issue
before. You can try copying objects into themselves. That should
recreate their bucket index entry.

> Why radosgw has many index instances, how do radosgw use them and how to 
> change active index?

Could be related to an existing bug. You can unlink the bucket and
then link a specific bucket instance version (to the user), however,
I'm not sure I recommend going this path if it isn't necessary.

Regards,
Yehuda
>
>
> Thanks,
>
> Inksink
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

[Yehuda Sadeh-Weinraub]

There was an existing bug reported for this one, and it's fixed on master:

http://tracker.ceph.com/issues/23801

It will be backport to luminous and mimic.

On Mon, Aug 20, 2018 at 9:25 AM, Yehuda Sadeh-Weinraub
 wrote:
> That message has been there since 2014. We should lower the log level though.
>
> Yehuda
>
> On Mon, Aug 20, 2018 at 6:08 AM, David Turner  wrote:
>> In luminous they consolidated a lot of the rgw metadata pools by using
>> namespace inside of the pools. I would say that the GC pool was consolidated
>> into the log pool based on the correlation you've found with the primary
>> osds.  At least that mystery is solved as to why those 8 osds. I don't know
>> why there logs are being spammed with GC messages though. Hopefully someone
>> else can shed a light on that. I cc'd Yehuda on this, the primary RGW dev.
>>
>>
>> On Mon, Aug 20, 2018, 7:09 AM Jakub Jaszewski 
>> wrote:
>>>
>>> Hi David,
>>>
>>> Right, we use this cluster (v12.2.5, fresh installation) for RGW, however,
>>> I don't see default.rgw.gc pool like we have on other cluster which was
>>> upgraded to Luminous, 10.2.9 -> 10.2.10 -> 12.2.2 (I believe that
>>> default.rgw.gc pool is there from the time of setting up RGW on Jewel
>>> version and the pool was automatically created).
>>>
>>> On impacted cluster we have below pools
>>> pool 1 '.rgw.root' replicated size 3 min_size 2 crush_rule 2 object_hash
>>> rjenkins pg_num 8 pgp_num 8 last_change 1499 flags hashpspool stripe_width 0
>>> application rgw
>>> pool 2 'rbd' replicated size 3 min_size 2 crush_rule 2 object_hash
>>> rjenkins pg_num 2048 pgp_num 2048 last_change 2230 flags hashpspool
>>> stripe_width 0 application rbd
>>> pool 3 'default.rgw.control' replicated size 3 min_size 2 crush_rule 2
>>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1501 flags hashpspool
>>> stripe_width 0 application rgw
>>> pool 4 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 2
>>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1491 flags hashpspool
>>> stripe_width 0 application rgw
>>> pool 5 'default.rgw.log' replicated size 3 min_size 2 crush_rule 2
>>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1486 flags hashpspool
>>> stripe_width 0 application rgw
>>> pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule
>>> 2 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1483 owner
>>> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
>>> pool 8 'default.rgw.buckets.data' erasure size 12 min_size 10 crush_rule 3
>>> object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 2228 flags
>>> hashpspool max_bytes 879609302220800 stripe_width 36864 application rgw
>>> pool 9 'default.rgw.buckets.non-ec' replicated size 3 min_size 2
>>> crush_rule 2 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1506 owner
>>> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
>>>
>>> Eight annoying OSDs match to primary OSDs of PGs that make default.rgw.log
>>> pool.
>>>
>>> Many thanks
>>> Jakub
>>>
>>>
>>> On Mon, Aug 20, 2018 at 11:54 AM David Turner 
>>> wrote:
>>>>
>>>> I'm assuming you use RGW and that you have a GC pool for RGW. It also
>>>> might beat assumed that your GC pool only has 8 PGs.  Are any of those
>>>> guesses correct?
>>>>
>>>> On Mon, Aug 20, 2018, 5:13 AM Jakub Jaszewski 
>>>> wrote:
>>>>>
>>>>> Issue tracker http://tracker.ceph.com/issues/23801.
>>>>> Still don't know why only particular OSDs write this information to log
>>>>> files.
>>>>>
>>>>> Jakub
>>>>>
>>>>> On Wed, Aug 8, 2018 at 12:02 PM Jakub Jaszewski
>>>>>  wrote:
>>>>>>
>>>>>> Hi All, exactly the same story today, same 8 OSDs and a lot of garbage
>>>>>> collection objects to process
>>>>>>
>>>>>> Below is the number of "cls_rgw.cc:3284: gc_iterate_entries end_key="
>>>>>> entries per OSD log file
>>>>>> hostA:
>>>>>>   /var/log/ceph/ceph-osd.58.log
>>>>>>   1826467
>>>>>> hostB:
>>>>>>   /var/log/ceph/ceph-osd.88.log
>>>>>>   2924241
>>>>>> hostC:
>>>>>>   /var/log/ceph/ceph-osd.153.log
>>>&

Re: [ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

[Yehuda Sadeh-Weinraub]

That message has been there since 2014. We should lower the log level though.

Yehuda

On Mon, Aug 20, 2018 at 6:08 AM, David Turner  wrote:
> In luminous they consolidated a lot of the rgw metadata pools by using
> namespace inside of the pools. I would say that the GC pool was consolidated
> into the log pool based on the correlation you've found with the primary
> osds.  At least that mystery is solved as to why those 8 osds. I don't know
> why there logs are being spammed with GC messages though. Hopefully someone
> else can shed a light on that. I cc'd Yehuda on this, the primary RGW dev.
>
>
> On Mon, Aug 20, 2018, 7:09 AM Jakub Jaszewski 
> wrote:
>>
>> Hi David,
>>
>> Right, we use this cluster (v12.2.5, fresh installation) for RGW, however,
>> I don't see default.rgw.gc pool like we have on other cluster which was
>> upgraded to Luminous, 10.2.9 -> 10.2.10 -> 12.2.2 (I believe that
>> default.rgw.gc pool is there from the time of setting up RGW on Jewel
>> version and the pool was automatically created).
>>
>> On impacted cluster we have below pools
>> pool 1 '.rgw.root' replicated size 3 min_size 2 crush_rule 2 object_hash
>> rjenkins pg_num 8 pgp_num 8 last_change 1499 flags hashpspool stripe_width 0
>> application rgw
>> pool 2 'rbd' replicated size 3 min_size 2 crush_rule 2 object_hash
>> rjenkins pg_num 2048 pgp_num 2048 last_change 2230 flags hashpspool
>> stripe_width 0 application rbd
>> pool 3 'default.rgw.control' replicated size 3 min_size 2 crush_rule 2
>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1501 flags hashpspool
>> stripe_width 0 application rgw
>> pool 4 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 2
>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1491 flags hashpspool
>> stripe_width 0 application rgw
>> pool 5 'default.rgw.log' replicated size 3 min_size 2 crush_rule 2
>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 1486 flags hashpspool
>> stripe_width 0 application rgw
>> pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule
>> 2 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1483 owner
>> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
>> pool 8 'default.rgw.buckets.data' erasure size 12 min_size 10 crush_rule 3
>> object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 2228 flags
>> hashpspool max_bytes 879609302220800 stripe_width 36864 application rgw
>> pool 9 'default.rgw.buckets.non-ec' replicated size 3 min_size 2
>> crush_rule 2 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1506 owner
>> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
>>
>> Eight annoying OSDs match to primary OSDs of PGs that make default.rgw.log
>> pool.
>>
>> Many thanks
>> Jakub
>>
>>
>> On Mon, Aug 20, 2018 at 11:54 AM David Turner 
>> wrote:
>>>
>>> I'm assuming you use RGW and that you have a GC pool for RGW. It also
>>> might beat assumed that your GC pool only has 8 PGs.  Are any of those
>>> guesses correct?
>>>
>>> On Mon, Aug 20, 2018, 5:13 AM Jakub Jaszewski 
>>> wrote:

 Issue tracker http://tracker.ceph.com/issues/23801.
 Still don't know why only particular OSDs write this information to log
 files.

 Jakub

 On Wed, Aug 8, 2018 at 12:02 PM Jakub Jaszewski
  wrote:
>
> Hi All, exactly the same story today, same 8 OSDs and a lot of garbage
> collection objects to process
>
> Below is the number of "cls_rgw.cc:3284: gc_iterate_entries end_key="
> entries per OSD log file
> hostA:
>   /var/log/ceph/ceph-osd.58.log
>   1826467
> hostB:
>   /var/log/ceph/ceph-osd.88.log
>   2924241
> hostC:
>   /var/log/ceph/ceph-osd.153.log
>   581002
>   /var/log/ceph/ceph-osd.164.log
>   3278606
> hostD:
>   /var/log/ceph/ceph-osd.95.log
>   1426963
> hostE:
>   /var/log/ceph/ceph-osd.4.log
>   2716914
>   /var/log/ceph/ceph-osd.53.log
>   943749
> hostF:
>   /var/log/ceph/ceph-osd.172.log
>   4085334
>
>
> # radosgw-admin gc list --include-all|grep oid |wc -l
> 302357
> #
>
> Can anyone please explain what is going on ?
>
> Thanks!
> Jakub
>
> On Tue, Aug 7, 2018 at 3:03 PM Jakub Jaszewski
>  wrote:
>>
>> Hi,
>>
>> 8 out of 192 OSDs in our cluster (version 12.2.5) write plenty of
>> records like "cls_rgw.cc:3284: gc_iterate_entries end_key=" to the
>> corresponding log files, e.g.
>>
>> 2018-08-07 04:34:06.000585 7fdd8f012700  0 
>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>> end_key=1_01533616446.000580407
>> 2018-08-07 04:34:06.001888 7fdd8f012700  0 
>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>> end_key=1_01533616446.001886318
>> 2018-08-07 04:34:06.003395 7fdd8f012700  0 
>> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
>> 

Re: [ceph-users] RGW problems after upgrade to Luminous

[Yehuda Sadeh-Weinraub]

Oh, also -- one thing that might work is running bucket check --fix on
the bucket. That should overwrite the reshard status field in the
bucket index.

Let me know if it happens to fix the issue for you.

Yehuda.

On Fri, Aug 3, 2018 at 9:46 AM, Yehuda Sadeh-Weinraub  wrote:
> Is it actually resharding, or is it just stuck in that state?
>
> On Fri, Aug 3, 2018 at 7:55 AM, David Turner  wrote:
>> I am currently unable to write any data to this bucket in this current
>> state.  Does anyone have any ideas for reverting to the original index
>> shards and cancel the reshard processes happening to the bucket?
>>
>> On Thu, Aug 2, 2018 at 12:32 PM David Turner  wrote:
>>>
>>> I upgraded my last cluster to Luminous last night.  It had some very large
>>> bucket indexes on Jewel which caused a couple problems with the upgrade, but
>>> finally everything finished and we made it to the other side, but now I'm
>>> having problems with [1] these errors populating a lot of our RGW logs and
>>> clients seeing the time skew error responses.  The time stamps between the
>>> client nodes, rgw nodes, and the rest of the ceph cluster match perfectly
>>> and actually build off of the same ntp server.
>>>
>>> I tried disabling dynamic resharding for the RGW daemons by placing this
>>> in the ceph.conf for the affected daemons `rgw_dynamic_resharding = false`
>>> and restarting them as well as issuing a reshard cancel for the bucket, but
>>> nothing seems to actually stop the reshard from processing.  Here's the
>>> output of a few commands.  [2] reshard list [3] reshard status
>>>
>>> Are there any things we can do to actually disable bucket resharding or
>>> let it finish?  I'm stuck on ideas.  I've tried quite a few things I've
>>> found around except for manually resharding which is a last resort here.
>>> This bucket won't exist in a couple months and the performance is good
>>> enough without resharding, but I don't know how to get it to stop.  Thanks.
>>>
>>>
>>> [1] 2018-08-02 16:22:16.047387 7fbe82e61700  0 NOTICE: resharding
>>> operation on bucket index detected, blocking
>>> 2018-08-02 16:22:16.206950 7fbe8de77700  0 block_while_resharding ERROR:
>>> bucket is still resharding, please retry
>>> 2018-08-02 16:22:12.253734 7fbe4f5fa700  0 NOTICE: request time skew too
>>> big now=2018-08-02 16:22:12.00 req_time=2018-08-02 16:06:03.00
>>>
>>> [2] $ radosgw-admin reshard list
>>> [2018-08-02 16:13:19.082172 7f3ca4163c80 -1 ERROR: failed to list reshard
>>> log entries, oid=reshard.10
>>> 2018-08-02 16:13:19.082757 7f3ca4163c80 -1 ERROR: failed to list reshard
>>> log entries, oid=reshard.11
>>> 2018-08-02 16:13:19.083941 7f3ca4163c80 -1 ERROR: failed to list reshard
>>> log entries, oid=reshard.12
>>> 2018-08-02 16:13:19.085170 7f3ca4163c80 -1 ERROR: failed to list reshard
>>> log entries, oid=reshard.13
>>> 2018-08-02 16:13:19.085898 7f3ca4163c80 -1 ERROR: failed to list reshard
>>> log entries, oid=reshard.14
>>> ]
>>> 2018-08-02 16:13:19.086476 7f3ca4163c80 -1 ERROR: failed to list reshard
>>> log entries, oid=reshard.15
>>>
>>> [3] $ radosgw-admin reshard status --bucket my-bucket
>>> [
>>> {
>>> "reshard_status": 1,
>>> "new_bucket_instance_id":
>>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>>> "num_shards": 32
>>> },
>>> {
>>> "reshard_status": 1,
>>> "new_bucket_instance_id":
>>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>>> "num_shards": 32
>>> },
>>> {
>>> "reshard_status": 1,
>>> "new_bucket_instance_id":
>>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>>> "num_shards": 32
>>> },
>>> {
>>> "reshard_status": 1,
>>> "new_bucket_instance_id":
>>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>>> "num_shards": 32
>>> },
>>> {
>>> "reshard_status": 1,
>>> "new_bucket_instance_id":
>>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>>> "num_shards": 32
>>> },

Re: [ceph-users] RGW problems after upgrade to Luminous

[Yehuda Sadeh-Weinraub]

Is it actually resharding, or is it just stuck in that state?

On Fri, Aug 3, 2018 at 7:55 AM, David Turner  wrote:
> I am currently unable to write any data to this bucket in this current
> state.  Does anyone have any ideas for reverting to the original index
> shards and cancel the reshard processes happening to the bucket?
>
> On Thu, Aug 2, 2018 at 12:32 PM David Turner  wrote:
>>
>> I upgraded my last cluster to Luminous last night.  It had some very large
>> bucket indexes on Jewel which caused a couple problems with the upgrade, but
>> finally everything finished and we made it to the other side, but now I'm
>> having problems with [1] these errors populating a lot of our RGW logs and
>> clients seeing the time skew error responses.  The time stamps between the
>> client nodes, rgw nodes, and the rest of the ceph cluster match perfectly
>> and actually build off of the same ntp server.
>>
>> I tried disabling dynamic resharding for the RGW daemons by placing this
>> in the ceph.conf for the affected daemons `rgw_dynamic_resharding = false`
>> and restarting them as well as issuing a reshard cancel for the bucket, but
>> nothing seems to actually stop the reshard from processing.  Here's the
>> output of a few commands.  [2] reshard list [3] reshard status
>>
>> Are there any things we can do to actually disable bucket resharding or
>> let it finish?  I'm stuck on ideas.  I've tried quite a few things I've
>> found around except for manually resharding which is a last resort here.
>> This bucket won't exist in a couple months and the performance is good
>> enough without resharding, but I don't know how to get it to stop.  Thanks.
>>
>>
>> [1] 2018-08-02 16:22:16.047387 7fbe82e61700  0 NOTICE: resharding
>> operation on bucket index detected, blocking
>> 2018-08-02 16:22:16.206950 7fbe8de77700  0 block_while_resharding ERROR:
>> bucket is still resharding, please retry
>> 2018-08-02 16:22:12.253734 7fbe4f5fa700  0 NOTICE: request time skew too
>> big now=2018-08-02 16:22:12.00 req_time=2018-08-02 16:06:03.00
>>
>> [2] $ radosgw-admin reshard list
>> [2018-08-02 16:13:19.082172 7f3ca4163c80 -1 ERROR: failed to list reshard
>> log entries, oid=reshard.10
>> 2018-08-02 16:13:19.082757 7f3ca4163c80 -1 ERROR: failed to list reshard
>> log entries, oid=reshard.11
>> 2018-08-02 16:13:19.083941 7f3ca4163c80 -1 ERROR: failed to list reshard
>> log entries, oid=reshard.12
>> 2018-08-02 16:13:19.085170 7f3ca4163c80 -1 ERROR: failed to list reshard
>> log entries, oid=reshard.13
>> 2018-08-02 16:13:19.085898 7f3ca4163c80 -1 ERROR: failed to list reshard
>> log entries, oid=reshard.14
>> ]
>> 2018-08-02 16:13:19.086476 7f3ca4163c80 -1 ERROR: failed to list reshard
>> log entries, oid=reshard.15
>>
>> [3] $ radosgw-admin reshard status --bucket my-bucket
>> [
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> "b7567cda-7d6f-4feb-86d6-bbd9da36b14d.141873449.1",
>> "num_shards": 32
>> },
>> {
>> "reshard_status": 1,
>> "new_bucket_instance_id":
>> 

Re: [ceph-users] radosgw multizone not syncing large bucket completly to other zone

[Yehuda Sadeh-Weinraub]

On Fri, Jun 29, 2018 at 8:48 AM, Enrico Kern  wrote:

> also when i try to sync the bucket manual i get this error:
>
> ERROR: sync.run() returned ret=-16
> 2018-06-29 15:47:50.137268 7f54b7e4ecc0  0 data sync: ERROR: failed to
> read sync status for bucketname:6a9448d2-bdba-4bec-
> aad6-aba72cd8eac6.27150814.1
>
> it works flawless with all other buckets.
>

error 16 is EBUSY: meaning it can't take a lease to do work on the bucket.
This usually happens when another entity (e.g., a running radosgw process)
is working on it at the same time. Either something took the lease and
never gave it back (leases shouldn't be indefinite, usually are being taken
for a short period but are renewed periodically), or there might be some
other bug related to the lease itself. I would start by first figuring out
whether it's the first case or the second one. On the messenger log there
should be a message prior to that that shows the operation that got the -16
as a response (should have something like "...=-16 (Device or resource
busy)" in it). The same line would also contain the name of the rados
object that is used to manage the lease. Try to look at the running radosgw
log at the same time when this happens, and check whether there are other
operations on that object.
One thing to note is that if you run a sync on a bucket and stop it
uncleanly in the middle (e.g., like killing the process), the leak will
stay locked for a period of time (Something in the order of 1 to 2 minutes).

Yehuda

>
>
> On Fri, Jun 29, 2018 at 5:39 PM Enrico Kern 
> wrote:
>
>> Hello,
>>
>> thanks for the reply.
>>
>> We have around 200k objects in the bucket. It is not automatic resharded
>> (is that even supported in multisite?)
>>
>> What i see when i run a complete data sync with the debug logs after a
>> while i see alot of informations that it is unable to perform some log and
>> also some device or resource busy (also with alot of different osds,
>> restarting the osds also doesnt make this error going away):
>>
>>
>> 018-06-29 15:18:30.391085 7f38bf882cc0 20 cr:s=0x55de55700b20:op=
>> 0x55de55717010:20RGWContinuousLeaseCR: couldn't lock
>> amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-
>> bdba-4bec-aad6-aba72cd8eac6.59:sync_lock: retcode=-16
>>
>> 2018-06-29 15:18:30.391094 7f38bf882cc0 20 cr:s=0x55de55732750:op=
>> 0x55de5572d970:20RGWContinuousLeaseCR: couldn't lock
>> amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-
>> bdba-4bec-aad6-aba72cd8eac6.10:sync_lock: retcode=-16
>>
>> 2018-06-29 15:22:01.618744 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604
>> <== osd.43 10.30.3.44:6800/29982 13272  osd_op_reply(258628
>> datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.52 [call]
>> v14448'24265315 uv24265266 ondisk = -16 ((16) Device or resource busy)) v8
>>  209+0+0 (2379682838 0 0) 0x7f38a8005110 con 0x7f3868003380
>>
>> 2018-06-29 15:22:01.618829 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604
>> <== osd.43 10.30.3.44:6800/29982 13273  osd_op_reply(258629
>> datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.105
>> [call] v14448'24265316 uv24265256 ondisk = -16 ((16) Device or resource
>> busy)) v8  210+0+0 (4086289880 0 0) 0x7f38a8005110 con 0x7f3868003380
>>
>>
>> There are no issues with the OSDs all other stuff in the cluster works
>> (rbd, images to openstack etc.)
>>
>>
>> Also that command with appending debug never finishes.
>>
>> On Tue, Jun 26, 2018 at 5:45 PM Yehuda Sadeh-Weinraub 
>> wrote:
>>
>>>
>>>
>>> On Sun, Jun 24, 2018 at 12:59 AM, Enrico Kern <
>>> enrico.k...@glispamedia.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> We have two ceph luminous clusters (12.2.5).
>>>>
>>>> recently one of our big buckets stopped syncing properly. We have a one
>>>> specific bucket which is around 30TB in size consisting of alot of
>>>> directories with each one having files of 10-20MB.
>>>>
>>>> The secondary zone is often completly missing multiple days of data in
>>>> this bucket, while all other smaller buckets sync just fine.
>>>>
>>>> Even with the complete data missing radosgw-admin sync status always
>>>> says everything is fine.
>>>>
>>>> the sync error log doesnt show anything for those days.
>>>>
>>>> Running
>>>>
>>>> radosgw-admin metadata sync and data sync also doesnt solve the issue.
>>>> The only way of making it sync again is to disable and re-eanble the sync.
>>>&

Re: [ceph-users] radosgw multizone not syncing large bucket completly to other zone

[Yehuda Sadeh-Weinraub]

On Sun, Jun 24, 2018 at 12:59 AM, Enrico Kern 
wrote:

> Hello,
>
> We have two ceph luminous clusters (12.2.5).
>
> recently one of our big buckets stopped syncing properly. We have a one
> specific bucket which is around 30TB in size consisting of alot of
> directories with each one having files of 10-20MB.
>
> The secondary zone is often completly missing multiple days of data in
> this bucket, while all other smaller buckets sync just fine.
>
> Even with the complete data missing radosgw-admin sync status always says
> everything is fine.
>
> the sync error log doesnt show anything for those days.
>
> Running
>
> radosgw-admin metadata sync and data sync also doesnt solve the issue. The
> only way of making it sync again is to disable and re-eanble the sync. That
> needs to be done as often as like 10 times in an hour to make it sync
> properly.
>
> radosgw-admin bucket sync disable
> radosgw-admin bucket sync enable
>
> when i run data init i sometimes get this:
>
>  radosgw-admin data sync init --source-zone berlin
> 2018-06-24 07:55:46.337858 7fe7557fa700  0 ERROR: failed to distribute
> cache for amsterdam.rgw.log:datalog.sync-status.6a9448d2-bdba-
> 4bec-aad6-aba72cd8eac6
>
> Sometimes when really alot of data is missing (yesterday it was more then
> 1 month) this helps making them get in sync again when run on the secondary
> zone:
>
> radosgw-admin bucket check --fix --check-objects
>
> how can i debug that problem further? We have so many requests on the
> cluster that is is hard to dig something out of the log files..
>
> Given all the smaller buckets are perfectly in sync i suspect some problem
> because of the size of the bucket.
>

How many objects in the bucket? Is it getting automatically resharded?


>
> Any points to the right direction are greatly appreciated.
>

A few things to look at that might help identify the issue.

What does this show (I think the luminous command is as follows):

$ radosgw-admin bucket sync status --source-zone=

You can try manually syncing the bucket, and get specific logs for that
operation:

$ radosgw-admin bucket sync run -source-zone= --debug-rgw=20
--debug-ms=1

And you can try getting more info from the sync trace module:

$ ceph --admin-daemon  sync trace history


You can also try the 'sync trace show' command.


Yehuda



>
> Regards,
>
> Enrico
>
> --
>
> *Enrico Kern*
> VP IT Operations
>
> enrico.k...@glispa.com
> +49 (0) 30 555713017 / +49 (0) 152 26814501
> skype: flyersa
> LinkedIn Profile 
>
>
>  
>
> *Glispa GmbH* | Berlin Office
> Sonnenburger Straße 73
> 
> 10437 Berlin
> 
> |
> 
>  Germany
> 
>
> Managing Director Din Karol-Gavish
> Registered in Berlin
> AG Charlottenburg |
> 
>  HRB
> 114678B
> –
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW Dynamic bucket index resharding keeps resharding all buckets

[Yehuda Sadeh-Weinraub]

(resending)

Sounds like a bug. Can you open a ceph tracker issue?

Thanks,
Yehuda

On Mon, Jun 18, 2018 at 7:24 AM, Sander van Schie / True
 wrote:
> While Ceph was resharding buckets over and over again, the maximum available 
> storage as reported by 'ceph df' also decreased by about 20%, while usage 
> stayed the same, we have yet to find out where the missing storage went. The 
> decreasing stopped once we disabled resharding.
>
> Any help would be greatly appreciated.
>
> Thanks,
>
> Sander
>
> 
> From: Sander van Schie / True
> Sent: Friday, June 15, 2018 14:19
> To: ceph-users@lists.ceph.com
> Subject: RGW Dynamic bucket index resharding keeps resharding all buckets
>
> Hello,
>
> We're into some problems with dynamic bucket index resharding. After an 
> upgrade from Ceph 12.2.2 to 12.2.5, which fixed an issue with the resharding 
> when using tenants (which we do), the cluster was busy resharding for 2 days 
> straight, resharding the same buckets over and over again.
>
> After disabling it and re-enabling it a while later, it resharded all buckets 
> again and then kept quiet for a bit. Later on it started resharding buckets 
> over and over again, even buckets which didn't have any data added in the 
> meantime. In the reshard list it always says 'old_num_shards: 1' for every 
> bucket, even though I can confirm with 'bucket stats' there's already the 
> desired amount of shards present. It looks like the background process which 
> scans buckets doesn't properly recognize the amount of shards a bucket 
> currently has. When I manually add a reshard job, it does properly recognize 
> the current amount of shards.
>
> On a sidenote, we had two buckets in the reshard list which were removed a 
> long while ago. We were unable to cancel the reshard job for those buckets. 
> After recreating the users and buckets we were able to remove them from the 
> list though, so they are no longer present. Probably not relevant, but you 
> never know.
>
> Are we missing something, or are we running into a bug?
>
> Thanks,
>
> Sander
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to configure s3 bucket acl so that one user's bucket is visible to another.

[Yehuda Sadeh-Weinraub]

You can't. A user can only list the buckets that it owns, it cannot
list other users' buckets.

Yehuda

On Sat, Apr 28, 2018 at 11:10 AM, Безруков Илья Алексеевич
 wrote:
> Hello,
>
> How to configure s3 bucket acl so that one user's bucket is visible to
> another.
>
>
> I can create a bucket, objects in it and give another user access to it.
> But another user does not see this bucket in the list of available buckets.
>
>
> ## User1
>
> ```
> s3cmd -c s3cfg_user1 ls s3://
>
> 2018-04-28 07:50  s3://example1
>
> #set ACL
> s3cmd -c s3cfg_user1 setacl --acl-grant=all:user2 s3://example1
> s3://example1/: ACL updated
>
> # Check
> s3cmd -c s3cfg_user1 info s3://example1
> s3://example1/ (bucket):
>Location:  us-east-1
>Payer: BucketOwner
>Expiration Rule: none
>Policy:none
>CORS:  none
>ACL:   User1: FULL_CONTROL
>ACL:   User2: FULL_CONTROL
>
> # Put some data
> s3cmd -c s3cfg_user1 put /tmp/dmesg s3://example1
> upload: '/tmp/dmesg' -> 's3://example1/dmesg'  [1 of 1]
>  5305 of 5305   100% in0s27.28 kB/s  done
>
> #set ACL
> s3cmd -c s3cfg_user1 setacl --acl-grant=all:bondarenko s3://example1/dmesg
> s3://example1/dmesg: ACL updated
>
> ```
>
> ## User2
> ```
> s3cmd -c ~/.s3cfg_user2 ls s3://
> 2018-04-27 14:23  s3://only_itself_dir
>
> # Check info
> s3cmd -c ~/.s3cfg_user2 info s3://example1
> ERROR: Access to bucket 'example1' was denied
> ERROR: S3 error: 403 (AccessDenied)
>
> # ls bucket
> s3cmd -c ~/.s3cfg_user2 ls s3://example1
> 2018-04-28 07:58  5305   s3://example1/dmesg
>
> #Get info
> s3cmd -c ~/.s3cfg_user2 info s3://example1/dmesg
> s3://example1/dmesg (object):
>File size: 5305
>Last mod:  Sat, 28 Apr 2018 07:58:03 GMT
>MIME type: text/plain
>Storage:   STANDARD
>MD5 sum:   47ddc4780956cb55abe27e851aa02cfa
>SSE:   none
>Policy:none
> ERROR: Access to bucket 'example1' was denied
> ERROR: S3 error: 403 (AccessDenied)
>
> #Get object
> s3cmd -c ~/.s3cfg_user2 get s3://example1/dmesg /tmp/test
> download: 's3://example1/dmesg' -> '/tmp/test'  [1 of 1]
>  5305 of 5305   100% in0s   160.54 kB/s  done
>
> #Put some oject to bucket
> s3cmd -c ~/.s3cfg_user2 put /tmp/dmesg2 s3://example1/dmesg2
> upload: '/tmp/dmesg2' -> 's3://example1/dmesg2'  [1 of 1]
>  38136 of 38136   100% in0s   455.18 kB/s  done
> ```
>
> Best regards,
>
> Ilya
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How much damage have I done to RGW hardcore-wiping a bucket out of its existence?

[Yehuda Sadeh-Weinraub]

On Fri, Apr 13, 2018 at 5:09 PM, Katie Holly <8ld3j...@meo.ws> wrote:
> Hi everyone,
>
> I found myself in a situation where dynamic sharding and writing data to a 
> bucket containing a little more than 5M objects at the same time caused 
> corruption on the data rendering the entire bucket unusable, I tried several 
> solutions to fix this bucket and ended up ditching it.
>
> What I tried before going the hardcore way:
>
> * radosgw-admin reshard list -> didn't list any reshard process going on at 
> the time, but
> * radosgw-admin reshard cancel --bucket $bucket -> canceled the reshard 
> process going on in the background, overall load on the cluster dropped after 
> a few minutes
>
> At this point I decided to start from scratch since a lot of the data was 
> corrupted due to a broken application version writing to this bucket.
>
> * aws s3 rm --recursive s3://$bucket -> Deleted most files, but 13k files 
> consuming around 500G total weren't deleted, re-running the same command 
> didn't fix that
> * aws s3 rb s3://$bucket -> That obviously didn't work since the bucket isn't 
> empty
> * radosgw-admin bucket rm --bucket $bucket -> "ERROR: could not remove 
> non-empty bucket $bucket" and "ERROR: unable to remove bucket(39) Directory 
> not empty"
> * radosgw-admin bucket rm --bucket $bucket --purge-objects -> "No such file 
> or directory"
>
> After some days of helpless Googling and trying various combinations of 
> radosgw-admin bucket, bi, reshard and other commands that all did pretty much 
> nothing, I did
>
> * rados -p $pool ls | tr '\t' '\n' | fgrep $bucket_marker_id | tr '\n' '\0' | 
> xargs -0 -n 128 -P 32 rados -p $pool rm
>

Should have probably used "${bucket_marker_id}_" here

> That deleted the orphan objects from the rados pool cleaning up the used 
> ~500G of data.
>
> * radosgw-admin bucket check --bucket $bucket -> listed some objects in an 
> array, probably the lost ones that weren't deleted
> * radosgw-admin bucket check --bucket $bucket --fix ( --check-objects) -> 
> didn't do anything
>
> * radosgw-admin bi purge --bucket=$bucket --yes-i-really-mean-it -> This 
> deleted the bucket index
> * radosgw-admin bucket list -> Bucket still appeared in the list
> * aws s3 ls -> Bucket was still appearing in the list
> * aws s3 rb $bucket -> "NoSuchBucket"
> * aws s3 rm --recursive s3://$bucket -> No error or output
> * aws s3 rb $bucket -> No error
> * aws s3 ls -> Bucket is no longer in the list
>
> At this point, I decided to restart all RGW frontend instances to make sure 
> nothing is being cached. To confirm that it's really gone now, let's check 
> everything...
>
> * aws s3 ls -> Check.
> * radosgw-admin bucket list -> Check.
> * radosgw-admin metadata get bucket:$bucket -> Check.
> * radosgw-admin bucket stats --bucket $bucket -> Check.
> But:
> * radosgw-admin reshard list -> It's doing a reshard, I stopped that for now. 
> However, all RGW frontend instances were logging this repeatedly for some 
> minutes:
>
>> block_while_resharding ERROR: bucket is still resharding, please retry
>> RGWWatcher::handle_error cookie xxx err (107) Transport endpoint is not 
>> connected
>> NOTICE: resharding operation on bucket index detected, blocking
>> NOTICE: resharding operation on bucket index detected, blocking
>> block_while_resharding ERROR: bucket is still resharding, please retry
>> block_while_resharding ERROR: bucket is still resharding, please retry
>> NOTICE: resharding operation on bucket index detected, blocking
>> NOTICE: resharding operation on bucket index detected, blocking
>> RGWWatcher::handle_error cookie xxx err (107) Transport endpoint is not 
>> connected
>> RGWWatcher::handle_error cookie xxx err (107) Transport endpoint is not 
>> connected
>> block_while_resharding ERROR: bucket is still resharding, please retry
>> RGWWatcher::handle_error cookie xxx err (107) Transport endpoint is not 
>> connected
>> NOTICE: resharding operation on bucket index detected, blocking
>> block_while_resharding ERROR: bucket is still resharding, please retry
>> NOTICE: resharding operation on bucket index detected, blocking
>
> One of the RGW frontend instances crashed during this, all others seem to be 
> running fine at the moment:
>
>> 2018-04-13 23:19:41.599307 7f35c6e00700  0 ERROR: flush_read_list(): 
>> d->client_cb->handle_data() returned -5
>> terminate called after throwing an instance of 'ceph::buffer::bad_alloc'
>>   what():  buffer::bad_alloc
>> *** Caught signal (Aborted) **
>>  in thread 7f35f341d700 thread_name:msgr-worker-0
>
>
> * aws s3 mb s3://$bucket -> This command succeeded
> * aws s3 cp $file s3://$bucket/$file -> This command succeeded as well
>
> My question at this point would be, how much have I damaged this cluster on 
> an RGW pov and is it possible to undo those damages? If I want to proceed 
> with cleaning up the old bucket data, where should I continue and how would I 
> verify that everything, that might further damage the cluster at a later 
> 

Re: [ceph-users] Civetweb log format

[Yehuda Sadeh-Weinraub]

On Thu, Mar 8, 2018 at 2:22 PM, David Turner  wrote:
> I remember some time ago Yehuda had commented on a thread like this saying
> that it would make sense to add a logging/auditing feature like this to RGW.
> I haven't heard much about it since then, though.  Yehuda, do you remember
> that and/or think that logging like this might become viable.

I vaguely remember Matt was working on this. Matt?

Yehuda

>
>
> On Thu, Mar 8, 2018 at 4:17 PM Aaron Bassett 
> wrote:
>>
>> Yea thats what I was afraid of. I'm looking at possibly patching to add
>> it, but i really dont want to support my own builds. I suppose other
>> alternatives are to use proxies to log stuff, but that makes me sad.
>>
>> Aaron
>>
>>
>> On Mar 8, 2018, at 12:36 PM, David Turner  wrote:
>>
>> Setting radosgw debug logging to 10/10 is the only way I've been able to
>> get the access key in the logs for requests.  It's very unfortunate as it
>> DRASTICALLY increases the amount of log per request, but it's what we needed
>> to do to be able to have the access key in the logs along with the request.
>>
>> On Tue, Mar 6, 2018 at 3:09 PM Aaron Bassett 
>> wrote:
>>>
>>> Hey all,
>>> I'm trying to get something of an audit log out of radosgw. To that end I
>>> was wondering if theres a mechanism to customize the log format of civetweb.
>>> It's already writing IP, HTTP Verb, path, response and time, but I'm hoping
>>> to get it to print the Authorization header of the request, which containers
>>> the access key id which we can tie back into the systems we use to issue
>>> credentials. Any thoughts?
>>>
>>> Thanks,
>>> Aaron
>>> CONFIDENTIALITY NOTICE
>>> This e-mail message and any attachments are only for the use of the
>>> intended recipient and may contain information that is privileged,
>>> confidential or exempt from disclosure under applicable law. If you are not
>>> the intended recipient, any disclosure, distribution or other use of this
>>> e-mail message or attachments is prohibited. If you have received this
>>> e-mail message in error, please delete and notify the sender immediately.
>>> Thank you.
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] change radosgw object owner

[Yehuda Sadeh-Weinraub]

On Tue, Mar 6, 2018 at 11:40 AM, Ryan Leimenstoll
 wrote:
> Hi all,
>
> We are trying to move a bucket in radosgw from one user to another in an 
> effort both change ownership and attribute the storage usage of the data to 
> the receiving user’s quota.
>
> I have unlinked the bucket and linked it to the new user using:
>
> radosgw-admin bucket unlink —bucket=$MYBUCKET —uid=$USER
> radosgw-admin bucket link —bucket=$MYBUCKET —bucket-id=$BUCKET_ID 
> —uid=$NEWUSER
>
> However, perhaps as expected, the owner of all the objects in the bucket 
> remain as $USER. I don’t believe changing the owner is a supported operation 
> from the S3 protocol, however it would be very helpful to have the ability to 
> do this on the radosgw backend. This is especially useful for large 
> buckets/datasets where copying the objects out and into radosgw could be time 
> consuming.
>
>  Is this something that is currently possible within radosgw? We are running 
> Ceph 12.2.2.

Maybe try to copy objects into themselves with the new owner (as long
as it can read it, if not then you first need to change the objects'
acls to allow read)? Note that you need to do a copy that would retain
the old meta attributes of the old object.

Yehuda

>
> Thanks,
> Ryan Leimenstoll
> rleim...@umiacs.umd.edu
> University of Maryland Institute for Advanced Computer Studies
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[Yehuda Sadeh-Weinraub]

I don't know why 'us' works for you, but it could be that s3cmd is
just not sending any location constraint when 'us' is set. You can try
looking at the capture for this. You can try using wireshark for the
capture (assuming http endpoint and not https).

Yehuda

On Mon, Feb 26, 2018 at 1:21 PM, David Turner <drakonst...@gmail.com> wrote:
> I set it to that for randomness.  I don't have a zonegroup named 'us'
> either, but that works fine.  I don't see why 'cn' should be any different.
> The bucket_location that triggered me noticing this was 'gd1'.  I don't know
> where that one came from, but I don't see why we should force people setting
> it to 'us' when that has nothing to do with the realm.  If it needed to be
> set to 'local-atl' that would make sense, but 'us' works just fine.  Perhaps
> 'us' working is what shouldn't work as opposed to allowing whatever else to
> be able to work.
>
> I tested setting bucket_location to 'local-atl' and it did successfully
> create the bucket.  So the question becomes, why do my other realms not care
> what that value is set to and why does this realm allow 'us' to be used when
> it isn't correct?
>
> On Mon, Feb 26, 2018 at 4:12 PM Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> If that's what you set in the config file, I assume that's what passed
>> in. Why did you set that in your config file? You don't have a
>> zonegroup named 'cn', right?
>>
>> On Mon, Feb 26, 2018 at 1:10 PM, David Turner <drakonst...@gmail.com>
>> wrote:
>> > I'm also not certain how to do the tcpdump for this.  Do you have any
>> > pointers to how to capture that for you?
>> >
>> > On Mon, Feb 26, 2018 at 4:09 PM David Turner <drakonst...@gmail.com>
>> > wrote:
>> >>
>> >> That's what I set it to in the config file.  I probably should have
>> >> mentioned that.
>> >>
>> >> On Mon, Feb 26, 2018 at 4:07 PM Yehuda Sadeh-Weinraub
>> >> <yeh...@redhat.com>
>> >> wrote:
>> >>>
>> >>> According to the log here, it says that the location constraint it got
>> >>> is "cn", can you take a look at a tcpdump, see if that's actually
>> >>> what's passed in?
>> >>>
>> >>> On Mon, Feb 26, 2018 at 12:02 PM, David Turner <drakonst...@gmail.com>
>> >>> wrote:
>> >>> > I run with `debug rgw = 10` and was able to find these lines at the
>> >>> > end
>> >>> > of a
>> >>> > request to create the bucket.
>> >>> >
>> >>> > Successfully creating a bucket with `bucket_location = US` looks
>> >>> > like
>> >>> > [1]this.  Failing to create a bucket has "ERROR: S3 error: 400
>> >>> > (InvalidLocationConstraint): The specified location-constraint is
>> >>> > not
>> >>> > valid"
>> >>> > on the CLI and [2]this (excerpt from the end of the request) in the
>> >>> > rgw
>> >>> > log
>> >>> > (debug level 10).  "create bucket location constraint" was not found
>> >>> > in
>> >>> > the
>> >>> > log for successfully creating the bucket.
>> >>> >
>> >>> >
>> >>> > [1]
>> >>> > 2018-02-26 19:52:36.419251 7f4bc9bc8700 10 cache put:
>> >>> >
>> >>> >
>> >>> > name=local-atl.rgw.data.root++.bucket.meta.testerton:bef43c26-daf3-47ef-a3a5-e1167e3f88ac.39099765.1
>> >>> > info.flags=0x17
>> >>> > 2018-02-26 19:52:36.419262 7f4bc9bc8700 10 adding
>> >>> >
>> >>> >
>> >>> > local-atl.rgw.data.root++.bucket.meta.testerton:bef43c26-daf3-47ef-a3a5-e1167e3f88ac.39099765.1
>> >>> > to cache LRU end
>> >>> > 2018-02-26 19:52:36.419266 7f4bc9bc8700 10 updating xattr:
>> >>> > name=user.rgw.acl
>> >>> > bl.length()=141
>> >>> > 2018-02-26 19:52:36.423863 7f4bc9bc8700 10
>> >>> > RGWWatcher::handle_notify()
>> >>> > notify_id 344855809097728 cookie 139963970426880 notifier 39099765
>> >>> > bl.length()=361
>> >>> > 2018-02-26 19:52:36.423875 7f4bc9bc8700 10 cache put:
>> >>> > name=local-atl.rgw.data.root++testerton info.flags=0x17
>> >>> > 2018-02-26 19:52:36.423882 7f4bc9bc8700 10 adding
>> >>> >

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[Yehuda Sadeh-Weinraub]

If that's what you set in the config file, I assume that's what passed
in. Why did you set that in your config file? You don't have a
zonegroup named 'cn', right?

On Mon, Feb 26, 2018 at 1:10 PM, David Turner <drakonst...@gmail.com> wrote:
> I'm also not certain how to do the tcpdump for this.  Do you have any
> pointers to how to capture that for you?
>
> On Mon, Feb 26, 2018 at 4:09 PM David Turner <drakonst...@gmail.com> wrote:
>>
>> That's what I set it to in the config file.  I probably should have
>> mentioned that.
>>
>> On Mon, Feb 26, 2018 at 4:07 PM Yehuda Sadeh-Weinraub <yeh...@redhat.com>
>> wrote:
>>>
>>> According to the log here, it says that the location constraint it got
>>> is "cn", can you take a look at a tcpdump, see if that's actually
>>> what's passed in?
>>>
>>> On Mon, Feb 26, 2018 at 12:02 PM, David Turner <drakonst...@gmail.com>
>>> wrote:
>>> > I run with `debug rgw = 10` and was able to find these lines at the end
>>> > of a
>>> > request to create the bucket.
>>> >
>>> > Successfully creating a bucket with `bucket_location = US` looks like
>>> > [1]this.  Failing to create a bucket has "ERROR: S3 error: 400
>>> > (InvalidLocationConstraint): The specified location-constraint is not
>>> > valid"
>>> > on the CLI and [2]this (excerpt from the end of the request) in the rgw
>>> > log
>>> > (debug level 10).  "create bucket location constraint" was not found in
>>> > the
>>> > log for successfully creating the bucket.
>>> >
>>> >
>>> > [1]
>>> > 2018-02-26 19:52:36.419251 7f4bc9bc8700 10 cache put:
>>> >
>>> > name=local-atl.rgw.data.root++.bucket.meta.testerton:bef43c26-daf3-47ef-a3a5-e1167e3f88ac.39099765.1
>>> > info.flags=0x17
>>> > 2018-02-26 19:52:36.419262 7f4bc9bc8700 10 adding
>>> >
>>> > local-atl.rgw.data.root++.bucket.meta.testerton:bef43c26-daf3-47ef-a3a5-e1167e3f88ac.39099765.1
>>> > to cache LRU end
>>> > 2018-02-26 19:52:36.419266 7f4bc9bc8700 10 updating xattr:
>>> > name=user.rgw.acl
>>> > bl.length()=141
>>> > 2018-02-26 19:52:36.423863 7f4bc9bc8700 10 RGWWatcher::handle_notify()
>>> > notify_id 344855809097728 cookie 139963970426880 notifier 39099765
>>> > bl.length()=361
>>> > 2018-02-26 19:52:36.423875 7f4bc9bc8700 10 cache put:
>>> > name=local-atl.rgw.data.root++testerton info.flags=0x17
>>> > 2018-02-26 19:52:36.423882 7f4bc9bc8700 10 adding
>>> > local-atl.rgw.data.root++testerton to cache LRU end
>>> >
>>> > [2]
>>> > 2018-02-26 19:43:37.340289 7f466bbca700  2 req 428078:0.004204:s3:PUT
>>> > /testraint/:create_bucket:executing
>>> > 2018-02-26 19:43:37.340366 7f466bbca700  5 NOTICE: call to
>>> > do_aws4_auth_completion
>>> > 2018-02-26 19:43:37.340472 7f466bbca700 10 v4 auth ok --
>>> > do_aws4_auth_completion
>>> > 2018-02-26 19:43:37.340715 7f466bbca700 10 create bucket location
>>> > constraint: cn
>>> > 2018-02-26 19:43:37.340766 7f466bbca700  0 location constraint (cn)
>>> > can't be
>>> > found.
>>> > 2018-02-26 19:43:37.340794 7f466bbca700  2 req 428078:0.004701:s3:PUT
>>> > /testraint/:create_bucket:completing
>>> > 2018-02-26 19:43:37.341782 7f466bbca700  2 req 428078:0.005689:s3:PUT
>>> > /testraint/:create_bucket:op status=-2208
>>> > 2018-02-26 19:43:37.341792 7f466bbca700  2 req 428078:0.005707:s3:PUT
>>> > /testraint/:create_bucket:http status=400
>>> >
>>> > On Mon, Feb 26, 2018 at 2:36 PM Yehuda Sadeh-Weinraub
>>> > <yeh...@redhat.com>
>>> > wrote:
>>> >>
>>> >> I'm not sure if the rgw logs (debug rgw = 20) specify explicitly why a
>>> >> bucket creation is rejected in these cases, but it might be worth
>>> >> trying to look at these. If not, then a tcpdump of the specific failed
>>> >> request might shed some light (would be interesting to look at the
>>> >> generated LocationConstraint).
>>> >>
>>> >> Yehuda
>>> >>
>>> >> On Mon, Feb 26, 2018 at 11:29 AM, David Turner <drakonst...@gmail.com>
>>> >> wrote:
>>> >> > Our problem only appeared to be present in bucket creation.
>>> >> > Listing,

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[Yehuda Sadeh-Weinraub]

According to the log here, it says that the location constraint it got
is "cn", can you take a look at a tcpdump, see if that's actually
what's passed in?

On Mon, Feb 26, 2018 at 12:02 PM, David Turner <drakonst...@gmail.com> wrote:
> I run with `debug rgw = 10` and was able to find these lines at the end of a
> request to create the bucket.
>
> Successfully creating a bucket with `bucket_location = US` looks like
> [1]this.  Failing to create a bucket has "ERROR: S3 error: 400
> (InvalidLocationConstraint): The specified location-constraint is not valid"
> on the CLI and [2]this (excerpt from the end of the request) in the rgw log
> (debug level 10).  "create bucket location constraint" was not found in the
> log for successfully creating the bucket.
>
>
> [1]
> 2018-02-26 19:52:36.419251 7f4bc9bc8700 10 cache put:
> name=local-atl.rgw.data.root++.bucket.meta.testerton:bef43c26-daf3-47ef-a3a5-e1167e3f88ac.39099765.1
> info.flags=0x17
> 2018-02-26 19:52:36.419262 7f4bc9bc8700 10 adding
> local-atl.rgw.data.root++.bucket.meta.testerton:bef43c26-daf3-47ef-a3a5-e1167e3f88ac.39099765.1
> to cache LRU end
> 2018-02-26 19:52:36.419266 7f4bc9bc8700 10 updating xattr: name=user.rgw.acl
> bl.length()=141
> 2018-02-26 19:52:36.423863 7f4bc9bc8700 10 RGWWatcher::handle_notify()
> notify_id 344855809097728 cookie 139963970426880 notifier 39099765
> bl.length()=361
> 2018-02-26 19:52:36.423875 7f4bc9bc8700 10 cache put:
> name=local-atl.rgw.data.root++testerton info.flags=0x17
> 2018-02-26 19:52:36.423882 7f4bc9bc8700 10 adding
> local-atl.rgw.data.root++testerton to cache LRU end
>
> [2]
> 2018-02-26 19:43:37.340289 7f466bbca700  2 req 428078:0.004204:s3:PUT
> /testraint/:create_bucket:executing
> 2018-02-26 19:43:37.340366 7f466bbca700  5 NOTICE: call to
> do_aws4_auth_completion
> 2018-02-26 19:43:37.340472 7f466bbca700 10 v4 auth ok --
> do_aws4_auth_completion
> 2018-02-26 19:43:37.340715 7f466bbca700 10 create bucket location
> constraint: cn
> 2018-02-26 19:43:37.340766 7f466bbca700  0 location constraint (cn) can't be
> found.
> 2018-02-26 19:43:37.340794 7f466bbca700  2 req 428078:0.004701:s3:PUT
> /testraint/:create_bucket:completing
> 2018-02-26 19:43:37.341782 7f466bbca700  2 req 428078:0.005689:s3:PUT
> /testraint/:create_bucket:op status=-2208
> 2018-02-26 19:43:37.341792 7f466bbca700  2 req 428078:0.005707:s3:PUT
> /testraint/:create_bucket:http status=400
>
> On Mon, Feb 26, 2018 at 2:36 PM Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> I'm not sure if the rgw logs (debug rgw = 20) specify explicitly why a
>> bucket creation is rejected in these cases, but it might be worth
>> trying to look at these. If not, then a tcpdump of the specific failed
>> request might shed some light (would be interesting to look at the
>> generated LocationConstraint).
>>
>> Yehuda
>>
>> On Mon, Feb 26, 2018 at 11:29 AM, David Turner <drakonst...@gmail.com>
>> wrote:
>> > Our problem only appeared to be present in bucket creation.  Listing,
>> > putting, etc objects in a bucket work just fine regardless of the
>> > bucket_location setting.  I ran this test on a few different realms to
>> > see
>> > what would happen and only 1 of them had a problem.  There isn't an
>> > obvious
>> > thing that steps out about it.  The 2 local realms do not have
>> > multi-site,
>> > the internal realm has multi-site and the operations were performed on
>> > the
>> > primary zone for the zonegroup.
>> >
>> > Worked with non 'US' bucket_location for s3cmd to create bucket:
>> > realm=internal
>> > zonegroup=internal-ga
>> > zone=internal-atl
>> >
>> > Failed with non 'US' bucket_location for s3cmd to create bucket:
>> > realm=local-atl
>> > zonegroup=local-atl
>> > zone=local-atl
>> >
>> > Worked with non 'US' bucket_location for s3cmd to create bucket:
>> > realm=local
>> > zonegroup=local
>> > zone=local
>> >
>> > I was thinking it might have to do with all of the parts being named the
>> > same, but I made sure to do the last test to confirm.  Interestingly
>> > it's
>> > only bucket creation that has a problem and it's fine as long as I put
>> > 'US'
>> > as the bucket_location.
>> >
>> > On Mon, Feb 19, 2018 at 6:48 PM F21 <f21.gro...@gmail.com> wrote:
>> >>
>> >> I am using the official ceph/daemon docker image. It starts RGW and
>> >> creates a zonegroup and zone with their names set to an empty string:
>&g

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[Yehuda Sadeh-Weinraub]

I'm not sure if the rgw logs (debug rgw = 20) specify explicitly why a
bucket creation is rejected in these cases, but it might be worth
trying to look at these. If not, then a tcpdump of the specific failed
request might shed some light (would be interesting to look at the
generated LocationConstraint).

Yehuda

On Mon, Feb 26, 2018 at 11:29 AM, David Turner <drakonst...@gmail.com> wrote:
> Our problem only appeared to be present in bucket creation.  Listing,
> putting, etc objects in a bucket work just fine regardless of the
> bucket_location setting.  I ran this test on a few different realms to see
> what would happen and only 1 of them had a problem.  There isn't an obvious
> thing that steps out about it.  The 2 local realms do not have multi-site,
> the internal realm has multi-site and the operations were performed on the
> primary zone for the zonegroup.
>
> Worked with non 'US' bucket_location for s3cmd to create bucket:
> realm=internal
> zonegroup=internal-ga
> zone=internal-atl
>
> Failed with non 'US' bucket_location for s3cmd to create bucket:
> realm=local-atl
> zonegroup=local-atl
> zone=local-atl
>
> Worked with non 'US' bucket_location for s3cmd to create bucket:
> realm=local
> zonegroup=local
> zone=local
>
> I was thinking it might have to do with all of the parts being named the
> same, but I made sure to do the last test to confirm.  Interestingly it's
> only bucket creation that has a problem and it's fine as long as I put 'US'
> as the bucket_location.
>
> On Mon, Feb 19, 2018 at 6:48 PM F21 <f21.gro...@gmail.com> wrote:
>>
>> I am using the official ceph/daemon docker image. It starts RGW and
>> creates a zonegroup and zone with their names set to an empty string:
>>
>> https://github.com/ceph/ceph-container/blob/master/ceph-releases/luminous/ubuntu/16.04/daemon/start_rgw.sh#L36:54
>>
>> $RGW_ZONEGROUP and $RGW_ZONE are both empty strings by default:
>>
>> https://github.com/ceph/ceph-container/blob/master/ceph-releases/luminous/ubuntu/16.04/daemon/variables_entrypoint.sh#L46
>>
>> Here's what I get when I query RGW:
>>
>> $ radosgw-admin zonegroup list
>> {
>>  "default_info": "",
>>  "zonegroups": [
>>  "default"
>>  ]
>> }
>>
>> $ radosgw-admin zone list
>> {
>>  "default_info": "",
>>  "zones": [
>>  "default"
>>  ]
>> }
>>
>> On 20/02/2018 10:33 AM, Yehuda Sadeh-Weinraub wrote:
>> > What is the name of your zonegroup?
>> >
>> > On Mon, Feb 19, 2018 at 3:29 PM, F21 <f21.gro...@gmail.com> wrote:
>> >> I've done some debugging and the LocationConstraint is not being set by
>> >> the
>> >> SDK by default.
>> >>
>> >> I do, however, need to set the region on the client to us-east-1 for it
>> >> to
>> >> work. Anything else will return an InvalidLocationConstraint error.
>> >>
>> >> Francis
>> >>
>> >>
>> >> On 20/02/2018 8:40 AM, Yehuda Sadeh-Weinraub wrote:
>> >>> Sounds like the go sdk adds a location constraint to requests that
>> >>> don't go to us-east-1. RGW itself is definitely isn't tied to
>> >>> us-east-1, and does not know anything about it (unless you happen to
>> >>> have a zonegroup named us-east-1). Maybe there's a way to configure
>> >>> the sdk to avoid doing that?
>> >>>
>> >>> Yehuda
>> >>>
>> >>> On Sun, Feb 18, 2018 at 1:54 PM, F21 <f21.gro...@gmail.com> wrote:
>> >>>> I am using the AWS Go SDK v2 (https://github.com/aws/aws-sdk-go-v2)
>> >>>> to
>> >>>> talk
>> >>>> to my RGW instance using the s3 interface. I am running ceph in
>> >>>> docker
>> >>>> using
>> >>>> the ceph/daemon docker images in demo mode. The RGW is started with a
>> >>>> zonegroup and zone with their names set to an empty string by the
>> >>>> scripts
>> >>>> in
>> >>>> the image.
>> >>>>
>> >>>> I have ForcePathStyle for the client set to true, because I want to
>> >>>> access
>> >>>> all my buckets using the path: myrgw.instance:8080/somebucket.
>> >>>>
>> >>>> I noticed that if I set the region for the client to anything other
>> >>>> than
>> >>>> u

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[Yehuda Sadeh-Weinraub]

What is the name of your zonegroup?

On Mon, Feb 19, 2018 at 3:29 PM, F21 <f21.gro...@gmail.com> wrote:
> I've done some debugging and the LocationConstraint is not being set by the
> SDK by default.
>
> I do, however, need to set the region on the client to us-east-1 for it to
> work. Anything else will return an InvalidLocationConstraint error.
>
> Francis
>
>
> On 20/02/2018 8:40 AM, Yehuda Sadeh-Weinraub wrote:
>>
>> Sounds like the go sdk adds a location constraint to requests that
>> don't go to us-east-1. RGW itself is definitely isn't tied to
>> us-east-1, and does not know anything about it (unless you happen to
>> have a zonegroup named us-east-1). Maybe there's a way to configure
>> the sdk to avoid doing that?
>>
>> Yehuda
>>
>> On Sun, Feb 18, 2018 at 1:54 PM, F21 <f21.gro...@gmail.com> wrote:
>>>
>>> I am using the AWS Go SDK v2 (https://github.com/aws/aws-sdk-go-v2) to
>>> talk
>>> to my RGW instance using the s3 interface. I am running ceph in docker
>>> using
>>> the ceph/daemon docker images in demo mode. The RGW is started with a
>>> zonegroup and zone with their names set to an empty string by the scripts
>>> in
>>> the image.
>>>
>>> I have ForcePathStyle for the client set to true, because I want to
>>> access
>>> all my buckets using the path: myrgw.instance:8080/somebucket.
>>>
>>> I noticed that if I set the region for the client to anything other than
>>> us-east-1, I get this error when creating a bucket:
>>> InvalidLocationConstraint: The specified location-constraint is not
>>> valid.
>>>
>>> If I set the region in the client to something made up, such as "ceph"
>>> and
>>> the LocationConstraint to "ceph", I still get the same error.
>>>
>>> The only way to get my buckets to create successfully is to set the
>>> client's
>>> region to us-east-1. I have grepped the ceph code base and cannot find
>>> any
>>> references to us-east-1. In addition, I looked at the AWS docs for
>>> calculating v4 signatures and us-east-1 is the default region but I can
>>> see
>>> that the region string is used in the calculation (i.e. the region is not
>>> ignored when calculating the signature if it is set to us-east-1).
>>>
>>> Why do my buckets create successfully if I set the region in my s3 client
>>> to
>>> us-east-1, but not otherwise? If I do not want to use us-east-1 as my
>>> default region, for example, if I want us-west-1 as my default region,
>>> what
>>> should I be configuring in ceph?
>>>
>>> Thanks,
>>>
>>> Francis
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[Yehuda Sadeh-Weinraub]

Sounds like the go sdk adds a location constraint to requests that
don't go to us-east-1. RGW itself is definitely isn't tied to
us-east-1, and does not know anything about it (unless you happen to
have a zonegroup named us-east-1). Maybe there's a way to configure
the sdk to avoid doing that?

Yehuda

On Sun, Feb 18, 2018 at 1:54 PM, F21  wrote:
> I am using the AWS Go SDK v2 (https://github.com/aws/aws-sdk-go-v2) to talk
> to my RGW instance using the s3 interface. I am running ceph in docker using
> the ceph/daemon docker images in demo mode. The RGW is started with a
> zonegroup and zone with their names set to an empty string by the scripts in
> the image.
>
> I have ForcePathStyle for the client set to true, because I want to access
> all my buckets using the path: myrgw.instance:8080/somebucket.
>
> I noticed that if I set the region for the client to anything other than
> us-east-1, I get this error when creating a bucket:
> InvalidLocationConstraint: The specified location-constraint is not valid.
>
> If I set the region in the client to something made up, such as "ceph" and
> the LocationConstraint to "ceph", I still get the same error.
>
> The only way to get my buckets to create successfully is to set the client's
> region to us-east-1. I have grepped the ceph code base and cannot find any
> references to us-east-1. In addition, I looked at the AWS docs for
> calculating v4 signatures and us-east-1 is the default region but I can see
> that the region string is used in the calculation (i.e. the region is not
> ignored when calculating the signature if it is set to us-east-1).
>
> Why do my buckets create successfully if I set the region in my s3 client to
> us-east-1, but not otherwise? If I do not want to use us-east-1 as my
> default region, for example, if I want us-west-1 as my default region, what
> should I be configuring in ceph?
>
> Thanks,
>
> Francis
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw: Moving index objects to the right index_pool

[Yehuda Sadeh-Weinraub]

On Wed, Feb 14, 2018 at 10:53 PM, Ingo Reimann <ireim...@dunkel.de> wrote:
> Hi Yehuda,
>
> Thanks for you help.
>
> No, listing does not work, if I remove the old index objects.
>
> I guessed, I could use the resharding for my purpose. I just tried
> * copy the index object
> * rewrite bucket metadata
> * reshard
> => I get new index objects at the old place. Metadata gets turned back
> again.
>
> Maybe this is more complicated as expected?
>

You probably need this change for you to be able to modify the pool placement:

https://github.com/yehudasa/ceph/commit/0baebec32e388f4cb7bdf1fee9afe2144eeeb354

> Best regards,
>
> Ingo
>
>
> -Ursprüngliche Nachricht-
> Von: Yehuda Sadeh-Weinraub [mailto:yeh...@redhat.com]
> Gesendet: Donnerstag, 15. Februar 2018 00:21
> An: Ingo Reimann
> Cc: ceph-users
> Betreff: Re: [ceph-users] rgw: Moving index objects to the right index_pool
>
> On Tue, Feb 13, 2018 at 11:27 PM, Ingo Reimann <ireim...@dunkel.de> wrote:
>> Hi List,
>>
>> we want to brush up our cluster and correct things, that have been
>> changed over time. When we started with bobtail, we put all index
>> objects together with data into the pool rgw.buckets:
>>
>> root@cephadmin:~# radosgw-admin metadata get bucket:some-bucket {
>> "key": "bucket:some-bucket",
>> "ver": {
>> "tag": "_zgv1FXm604BQtdiZnLkaiXN",
>> "ver": 1
>> },
>> "mtime": "2016-01-08 04:53:38.00Z",
>> "data": {
>> "bucket": {
>> "name": "some-bucket",
>> "pool": "rgw.buckets",
>> "data_extra_pool": "",
>> "index_pool": "rgw.buckets",
>> "marker": "default.101387371.6",
>> "bucket_id": "default.101387371.6",
>> "tenant": ""
>>   [...]
>>
>> With Jewel, we introduced a default-placement and put the index of new
>> buckets into the pool rgw.buckets.index. Now, we`d like to correct the
>> old buckets and shift all the indices where they belong to.
>> I can
>> * copy the .dir.MARKER.SHARD# objects to the index pool
>> * modify the buckets metadata
>> But:
>> * when I try to modify the metadata of the bucket instance, the
>> index_pool does not get changed
>> * radosgw-admin bucket stats still shows the old index pool
>>
>> Did I miss something?
>
> You didn't miss much. There is a guard in the code that prevents you from
> modifying the placement pools. I have this commit that changes that (but
> that probably doesn't help you much):
>
> https://github.com/yehudasa/ceph/commit/0baebec32e388f4cb7bdf1fee9afe2144eeeb354
>
> The way to go forward for you I think would be by reshrding the buckets,
> which will put the bucket indexes in the correct place. Can you actually
> list the bucket indexes right now?
>
> Yehuda
>
>>
>> NB: Just now, I perform all the operations with a jewel 10.2.2 rgw.
>> Luminous is available but not active yet.
>>
>> Best regards,
>>
>> Ingo Reimann
>> Dunkel GmbH
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw gives MethodNotAllowed for OPTIONS?

[Yehuda Sadeh-Weinraub]

The CORS related operations are working on specific buckets, not on
the service root. You'll need to set CORS on a bucket, and specify it
in the path.

Yehuda

On Mon, Feb 12, 2018 at 5:17 PM, Piers Haken  wrote:
> I’m trying to do direct-from-browser upload to rgw using pre-signed urls,
> and I’m getting stuck because the browser is doing a pre-flight OPTIONS
> request and rgw is giving me a MethodNotAllowed response.
>
>
>
> Is this supported?
>
>
>
> OPTIONS http://storage-test01:7480/ HTTP/1.1
>
> Host: storage-test01:7480
>
> Connection: keep-alive
>
> Origin: http://localhost:3032
>
> Access-Control-Request-Method: POST
>
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
>
> Accept: */*
>
> Accept-Encoding: gzip, deflate
>
> Accept-Language: en-US,en;q=0.9,es;q=0.8
>
>
>
>
>
> HTTP/1.1 405 Method Not Allowed
>
> Content-Length: 189
>
> x-amz-request-id: txf-005a823b66-5e3e-default
>
> Accept-Ranges: bytes
>
> Content-Type: application/xml
>
> Date: Tue, 13 Feb 2018 01:12:06 GMT
>
> Connection: Keep-Alive
>
>
>
>  encoding="UTF-8"?>MethodNotAllowedtxf-005a823b66-5e3e-default5e3e-default-default
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw: Moving index objects to the right index_pool

[Yehuda Sadeh-Weinraub]

On Tue, Feb 13, 2018 at 11:27 PM, Ingo Reimann  wrote:
> Hi List,
>
> we want to brush up our cluster and correct things, that have been changed
> over time. When we started with bobtail, we put all index objects together
> with data into the pool rgw.buckets:
>
> root@cephadmin:~# radosgw-admin metadata get bucket:some-bucket
> {
> "key": "bucket:some-bucket",
> "ver": {
> "tag": "_zgv1FXm604BQtdiZnLkaiXN",
> "ver": 1
> },
> "mtime": "2016-01-08 04:53:38.00Z",
> "data": {
> "bucket": {
> "name": "some-bucket",
> "pool": "rgw.buckets",
> "data_extra_pool": "",
> "index_pool": "rgw.buckets",
> "marker": "default.101387371.6",
> "bucket_id": "default.101387371.6",
> "tenant": ""
>   [...]
>
> With Jewel, we introduced a default-placement and put the index of new
> buckets into the pool rgw.buckets.index. Now, we`d like to correct the old
> buckets and shift all the indices where they belong to.
> I can
> * copy the .dir.MARKER.SHARD# objects to the index pool
> * modify the buckets metadata
> But:
> * when I try to modify the metadata of the bucket instance, the index_pool
> does not get changed
> * radosgw-admin bucket stats still shows the old index pool
>
> Did I miss something?

You didn't miss much. There is a guard in the code that prevents you
from modifying the placement pools. I have this commit that changes
that (but that probably doesn't help you much):

https://github.com/yehudasa/ceph/commit/0baebec32e388f4cb7bdf1fee9afe2144eeeb354

The way to go forward for you I think would be by reshrding the
buckets, which will put the bucket indexes in the correct place. Can
you actually list the bucket indexes right now?

Yehuda

>
> NB: Just now, I perform all the operations with a jewel 10.2.2 rgw.
> Luminous is available but not active yet.
>
> Best regards,
>
> Ingo Reimann
> Dunkel GmbH
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW Metadata Search - Elasticserver

[Yehuda Sadeh-Weinraub]

On Wed, Feb 14, 2018 at 2:54 AM, Amardeep Singh  wrote:
> Hi,
>
> I am trying to setup RGW Metadata Search with Elastic server tier type as
> per blog post here. https://ceph.com/rgw/new-luminous-rgw-metadata-search/
>
> The environment setup is done using ceph-ansible  docker containers.
>
> Containers running on Node 1 - rgw, mds, mgr, mon , 5 osds
>
> Containers running on Node 2 - rgw, mon
>
> The steps performed are for setting up according to blog post are:
>
> docker exec ceph-rgw radosgw-admin --cluster test realm create
> --rgw-realm=gold --default
> docker exec ceph-rgw radosgw-admin --cluster test zonegroup delete
> --rgw-zonegroup=default
> docker exec ceph-rgw radosgw-admin --cluster test zonegroup create
> --rgw-zonegroup=uk --endpoints=http://rgw1:8080 --master --default
> docker exec ceph-rgw radosgw-admin --cluster test zone create
> --rgw-zonegroup=uk --rgw-zone=uk-west --endpoints=http://rgw1:8080
> --access-key=xxx --secret= --default --master
> docker exec ceph-rgw radosgw-admin --cluster test user create
> --uid=zone.user --display-name="Zone User" --access-key=xx
> --secret=xx --system
> docker exec ceph-rgw radosgw-admin --cluster test period update --commit
>
> Restart Docker Container and then perform the following :
>
> docker exec ceph-rgw radosgw-admin --cluster test zone create
> --rgw-zonegroup=uk --rgw-zone=uk-west-es --endpoints=http://rgw2:8080
> --access-key=xxx --secret=xx
> docker exec ceph-rgw radosgw-admin --cluster test zone modify
> --rgw-zone=uk-west-es --tier-type=elasticsearch
> --tier-config=endpoint=http://elasticserver:9200,num_shards=10,num_replicas=1
> docker exec ceph-rgw radosgw-admin --cluster test period update --commit
>
> After completing the above I can add documents but not able to search
> metadata using obo tool.
>
> Checking sync status I get
>
> realm 281ba7e8-3bd1-47de-981a-c94914bdf54f (gold)
> zonegroup 0d9efa23-09d6-4adf-a486-0858f3261d7b (uk)
> zone 3c1aae3e-4252-47cc-8e66-ae1cb6275158 (uk-west)
> metadata sync no sync (zone is master)
> data sync source: 13ee3cfa-10ba-45ef-aec0-e42d8f55e3b6 (uk-west-es)
> not syncing from zone
>
> Enabled Debug on RGW
>
> Tue Feb 13 11:51:49 2018
> /admin/realm/period
> 2018-02-13 17:21:49.500561 7fd4c5989700 15 generated auth header: AWS
> HCTJCLF1F1E857N09X1Y:Yn03Gr+aJJf+CqQxgaWeLqskTH8=
> 2018-02-13 17:21:49.500576 7fd4c5989700 20 sending request to
> http://rgw2:8080/admin/realm/period?period=74d8f220-1165-4092-8042-14734c27364c=2=94f8340b-9fdc-4277-8e55-3dd6fe878f48
> 2018-02-13 17:21:49.500587 7fd4c5989700 20 register_request
> mgr=0x56386f2d5770 req_data->id=47, easy_handle=0x563870068000
> 2018-02-13 17:21:49.500620 7fd4c5989700 20 run: stack=0x56386f37b750 is io
> blocked
> 2018-02-13 17:21:49.500874 7fd4c618a700 20 link_request
> req_data=0x56386f5d7cc0 req_data->id=47, easy_handle=0x563870068000
> 2018-02-13 17:21:50.342999 7fd518c36700 2
> RGWDataChangesLog::ChangesRenewThread: start
> 2018-02-13 17:21:50.503796 7fd4c618a700 10 receive_http_header
> 2018-02-13 17:21:50.503809 7fd4c618a700 10 received header:HTTP/1.1 400 Bad
> Request
> 2018-02-13 17:21:50.503814 7fd4c618a700 10 receive_http_header
> 2018-02-13 17:21:50.503815 7fd4c618a700 10 received header:Content-Length:
> 115
> 2018-02-13 17:21:50.503818 7fd4c618a700 10 receive_http_header
> 2018-02-13 17:21:50.503819 7fd4c618a700 10 received header:x-amz-request-id:
> tx0004b-005a82d155-3729-uk-west
> 2018-02-13 17:21:50.503821 7fd4c618a700 10 receive_http_header
> 2018-02-13 17:21:50.503821 7fd4c618a700 10 received header:Accept-Ranges:
> bytes
> 2018-02-13 17:21:50.503823 7fd4c618a700 10 receive_http_header
> 2018-02-13 17:21:50.503823 7fd4c618a700 10 received header:Content-Type:
> application/json
> 2018-02-13 17:21:50.503825 7fd4c618a700 10 receive_http_header
> 2018-02-13 17:21:50.503825 7fd4c618a700 10 received header:Date: Tue, 13 Feb
> 2018 11:51:50 GMT
> 2018-02-13 17:21:50.503826 7fd4c618a700 10 receive_http_header
> 2018-02-13 17:21:50.503827 7fd4c618a700 10 received header:
> 2018-02-13 17:21:50.504056 7fd4c5989700 20
> cr:s=0x56386f37b750:op=0x563870044900:21RGWPostRESTResourceCRI9RGWPeriodiE:
> operate()
> 2018-02-13 17:21:50.504075 7fd4c5989700 5 failed to wait for op, ret=-22:
> POST
> http://rgw2:8080/admin/realm/period?period=74d8f220-1165-4092-8042-14734c27364c=2=94f8340b-9fdc-4277-8e55-3dd6fe878f48
> 2018-02-13 17:21:50.504173 7fd4c5989700 20
> cr:s=0x56386f37b750:op=0x563870044900:21RGWPostRESTResourceCRI9RGWPeriodiE:
> operate() returned r=-22
> 2018-02-13 17:21:50.504190 7fd4c5989700 20
> cr:s=0x56386f37b750:op=0x56386f588e00:14PushAndRetryCR: operate()
> 2018-02-13 17:21:50.504192 7fd4c5989700 10 rgw period pusher: waiting
> 30.00s for retry..
> 2018-02-13 17:21:50.504205 7fd4c5989700 20 run: stack=0x56386f37b750 is io
> blocked
> 2018-02-13 17:22:12.343154 7fd518c36700 2
> RGWDataChangesLog::ChangesRenewThread: start


This request goes from 

Re: [ceph-users] Luminous RGW Metadata Search

[Yehuda Sadeh-Weinraub]

Yes, you're definitely right, docs can be improved. We'd be happy to
get a pull request with any improvements if someone wants to pick it
up.

Thanks,
Uejida

On Tue, Jan 16, 2018 at 1:30 PM, Youzhong Yang <youzh...@gmail.com> wrote:
> My bad ... Once I sent config request to us-east-1 (the master zone), it
> works, and 'obo mdsearch' against "us-east-es" zone works like a charm.
>
> May I suggest that the following page be modified to reflect this
> requirement so that someone else won't run into the same issue? I understand
> it may sound obvious to experienced users ...
>
> http://ceph.com/rgw/new-luminous-rgw-metadata-search/
>
> Thanks a lot.
>
>
> On Tue, Jan 16, 2018 at 3:59 PM, Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> On Tue, Jan 16, 2018 at 12:20 PM, Youzhong Yang <youzh...@gmail.com>
>> wrote:
>> > Hi Yehuda,
>> >
>> > I can use your tool obo to create a bucket, and upload a file to the
>> > object
>> > store, but when I tried to run the following command, it failed:
>> >
>> > # obo mdsearch buck --config='x-amz-meta-foo; string, x-amz-meta-bar;
>> > integer'
>> > ERROR: {"status": 405, "resource": null, "message": "", "error_code":
>> > "MethodNotAllowed", "reason": "Method Not Allowed"}
>> >
>> > How to make the method 'Allowed'?
>>
>>
>> Which rgw are you sending this request to?
>>
>> >
>> > Thanks in advance.
>> >
>> > On Fri, Jan 12, 2018 at 7:25 PM, Yehuda Sadeh-Weinraub
>> > <yeh...@redhat.com>
>> > wrote:
>> >>
>> >> The errors you're seeing there don't look like related to
>> >> elasticsearch. It's a generic radosgw related error that says that it
>> >> failed to reach the rados (ceph) backend. You can try bumping up the
>> >> messenger log (debug ms =1) and see if there's any hint in there.
>> >>
>> >> Yehuda
>> >>
>> >> On Fri, Jan 12, 2018 at 12:54 PM, Youzhong Yang <youzh...@gmail.com>
>> >> wrote:
>> >> > So I did the exact same thing using Kraken and the same set of VMs,
>> >> > no
>> >> > issue. What is the magic to make it work in Luminous? Anyone lucky
>> >> > enough to
>> >> > have this RGW ElasticSearch working using Luminous?
>> >> >
>> >> > On Mon, Jan 8, 2018 at 10:26 AM, Youzhong Yang <youzh...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi Yehuda,
>> >> >>
>> >> >> Thanks for replying.
>> >> >>
>> >> >> >radosgw failed to connect to your ceph cluster. Does the rados
>> >> >> > command
>> >> >> >with the same connection params work?
>> >> >>
>> >> >> I am not quite sure what to do by running rados command to test.
>> >> >>
>> >> >> So I tried again, could you please take a look and check what could
>> >> >> have
>> >> >> gone wrong?
>> >> >>
>> >> >> Here are what I did:
>> >> >>
>> >> >>  On ceph admin node, I removed installation on ceph-rgw1 and
>> >> >> ceph-rgw2, reinstalled rgw on ceph-rgw1, stoped rgw service, removed
>> >> >> all rgw
>> >> >> pools. Elasticsearch is running on ceph-rgw2 node on port 9200.
>> >> >>
>> >> >> ceph-deploy purge ceph-rgw1
>> >> >> ceph-deploy purge ceph-rgw2
>> >> >> ceph-deploy purgedata ceph-rgw2
>> >> >> ceph-deploy purgedata ceph-rgw1
>> >> >> ceph-deploy install --release luminous ceph-rgw1
>> >> >> ceph-deploy admin ceph-rgw1
>> >> >> ceph-deploy rgw create ceph-rgw1
>> >> >> ssh ceph-rgw1 sudo systemctl stop ceph-rado...@rgw.ceph-rgw1
>> >> >> rados rmpool default.rgw.log default.rgw.log
>> >> >> --yes-i-really-really-mean-it
>> >> >> rados rmpool default.rgw.meta default.rgw.meta
>> >> >> --yes-i-really-really-mean-it
>> >> >> rados rmpool default.rgw.control default.rgw.control
>> >> >> --yes-i-really-really-mean-it
>> >> >> rados rmpool .rgw.root .rgw.root --yes-i-really-really-mean-it
>> >> >>
>> >> >&g

Re: [ceph-users] Luminous RGW Metadata Search

[Yehuda Sadeh-Weinraub]

On Tue, Jan 16, 2018 at 12:20 PM, Youzhong Yang <youzh...@gmail.com> wrote:
> Hi Yehuda,
>
> I can use your tool obo to create a bucket, and upload a file to the object
> store, but when I tried to run the following command, it failed:
>
> # obo mdsearch buck --config='x-amz-meta-foo; string, x-amz-meta-bar;
> integer'
> ERROR: {"status": 405, "resource": null, "message": "", "error_code":
> "MethodNotAllowed", "reason": "Method Not Allowed"}
>
> How to make the method 'Allowed'?


Which rgw are you sending this request to?

>
> Thanks in advance.
>
> On Fri, Jan 12, 2018 at 7:25 PM, Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> The errors you're seeing there don't look like related to
>> elasticsearch. It's a generic radosgw related error that says that it
>> failed to reach the rados (ceph) backend. You can try bumping up the
>> messenger log (debug ms =1) and see if there's any hint in there.
>>
>> Yehuda
>>
>> On Fri, Jan 12, 2018 at 12:54 PM, Youzhong Yang <youzh...@gmail.com>
>> wrote:
>> > So I did the exact same thing using Kraken and the same set of VMs, no
>> > issue. What is the magic to make it work in Luminous? Anyone lucky
>> > enough to
>> > have this RGW ElasticSearch working using Luminous?
>> >
>> > On Mon, Jan 8, 2018 at 10:26 AM, Youzhong Yang <youzh...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Yehuda,
>> >>
>> >> Thanks for replying.
>> >>
>> >> >radosgw failed to connect to your ceph cluster. Does the rados command
>> >> >with the same connection params work?
>> >>
>> >> I am not quite sure what to do by running rados command to test.
>> >>
>> >> So I tried again, could you please take a look and check what could
>> >> have
>> >> gone wrong?
>> >>
>> >> Here are what I did:
>> >>
>> >>  On ceph admin node, I removed installation on ceph-rgw1 and
>> >> ceph-rgw2, reinstalled rgw on ceph-rgw1, stoped rgw service, removed
>> >> all rgw
>> >> pools. Elasticsearch is running on ceph-rgw2 node on port 9200.
>> >>
>> >> ceph-deploy purge ceph-rgw1
>> >> ceph-deploy purge ceph-rgw2
>> >> ceph-deploy purgedata ceph-rgw2
>> >> ceph-deploy purgedata ceph-rgw1
>> >> ceph-deploy install --release luminous ceph-rgw1
>> >> ceph-deploy admin ceph-rgw1
>> >> ceph-deploy rgw create ceph-rgw1
>> >> ssh ceph-rgw1 sudo systemctl stop ceph-rado...@rgw.ceph-rgw1
>> >> rados rmpool default.rgw.log default.rgw.log
>> >> --yes-i-really-really-mean-it
>> >> rados rmpool default.rgw.meta default.rgw.meta
>> >> --yes-i-really-really-mean-it
>> >> rados rmpool default.rgw.control default.rgw.control
>> >> --yes-i-really-really-mean-it
>> >> rados rmpool .rgw.root .rgw.root --yes-i-really-really-mean-it
>> >>
>> >>  On ceph-rgw1 node:
>> >>
>> >> export RGWHOST="ceph-rgw1"
>> >> export ELASTICHOST="ceph-rgw2"
>> >> export REALM="demo"
>> >> export ZONEGRP="zone1"
>> >> export ZONE1="zone1-a"
>> >> export ZONE2="zone1-b"
>> >> export SYNC_AKEY="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20
>> >> |
>> >> head -n 1 )"
>> >> export SYNC_SKEY="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40
>> >> |
>> >> head -n 1 )"
>> >>
>> >> radosgw-admin realm create --rgw-realm=${REALM} --default
>> >> radosgw-admin zonegroup create --rgw-realm=${REALM}
>> >> --rgw-zonegroup=${ZONEGRP} --endpoints=http://${RGWHOST}:8000 --master
>> >> --default
>> >> radosgw-admin zone create --rgw-realm=${REALM}
>> >> --rgw-zonegroup=${ZONEGRP}
>> >> --rgw-zone=${ZONE1} --endpoints=http://${RGWHOST}:8000
>> >> --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --master --default
>> >> radosgw-admin user create --uid=sync --display-name="zone sync"
>> >> --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --system
>> >> radosgw-admin period update --commit
>> >> sudo systemctl start ceph-radosgw@rgw.${RGWHOST}
>> >>
>> >> radosgw-admin zone create --rgw-realm=

Re: [ceph-users] Luminous RGW Metadata Search

[Yehuda Sadeh-Weinraub]

The errors you're seeing there don't look like related to
elasticsearch. It's a generic radosgw related error that says that it
failed to reach the rados (ceph) backend. You can try bumping up the
messenger log (debug ms =1) and see if there's any hint in there.

Yehuda

On Fri, Jan 12, 2018 at 12:54 PM, Youzhong Yang  wrote:
> So I did the exact same thing using Kraken and the same set of VMs, no
> issue. What is the magic to make it work in Luminous? Anyone lucky enough to
> have this RGW ElasticSearch working using Luminous?
>
> On Mon, Jan 8, 2018 at 10:26 AM, Youzhong Yang  wrote:
>>
>> Hi Yehuda,
>>
>> Thanks for replying.
>>
>> >radosgw failed to connect to your ceph cluster. Does the rados command
>> >with the same connection params work?
>>
>> I am not quite sure what to do by running rados command to test.
>>
>> So I tried again, could you please take a look and check what could have
>> gone wrong?
>>
>> Here are what I did:
>>
>>  On ceph admin node, I removed installation on ceph-rgw1 and
>> ceph-rgw2, reinstalled rgw on ceph-rgw1, stoped rgw service, removed all rgw
>> pools. Elasticsearch is running on ceph-rgw2 node on port 9200.
>>
>> ceph-deploy purge ceph-rgw1
>> ceph-deploy purge ceph-rgw2
>> ceph-deploy purgedata ceph-rgw2
>> ceph-deploy purgedata ceph-rgw1
>> ceph-deploy install --release luminous ceph-rgw1
>> ceph-deploy admin ceph-rgw1
>> ceph-deploy rgw create ceph-rgw1
>> ssh ceph-rgw1 sudo systemctl stop ceph-rado...@rgw.ceph-rgw1
>> rados rmpool default.rgw.log default.rgw.log --yes-i-really-really-mean-it
>> rados rmpool default.rgw.meta default.rgw.meta
>> --yes-i-really-really-mean-it
>> rados rmpool default.rgw.control default.rgw.control
>> --yes-i-really-really-mean-it
>> rados rmpool .rgw.root .rgw.root --yes-i-really-really-mean-it
>>
>>  On ceph-rgw1 node:
>>
>> export RGWHOST="ceph-rgw1"
>> export ELASTICHOST="ceph-rgw2"
>> export REALM="demo"
>> export ZONEGRP="zone1"
>> export ZONE1="zone1-a"
>> export ZONE2="zone1-b"
>> export SYNC_AKEY="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 |
>> head -n 1 )"
>> export SYNC_SKEY="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 |
>> head -n 1 )"
>>
>> radosgw-admin realm create --rgw-realm=${REALM} --default
>> radosgw-admin zonegroup create --rgw-realm=${REALM}
>> --rgw-zonegroup=${ZONEGRP} --endpoints=http://${RGWHOST}:8000 --master
>> --default
>> radosgw-admin zone create --rgw-realm=${REALM} --rgw-zonegroup=${ZONEGRP}
>> --rgw-zone=${ZONE1} --endpoints=http://${RGWHOST}:8000
>> --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --master --default
>> radosgw-admin user create --uid=sync --display-name="zone sync"
>> --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --system
>> radosgw-admin period update --commit
>> sudo systemctl start ceph-radosgw@rgw.${RGWHOST}
>>
>> radosgw-admin zone create --rgw-realm=${REALM} --rgw-zonegroup=${ZONEGRP}
>> --rgw-zone=${ZONE2} --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY}
>> --endpoints=http://${RGWHOST}:8002
>> radosgw-admin zone modify --rgw-realm=${REALM} --rgw-zonegroup=${ZONEGRP}
>> --rgw-zone=${ZONE2} --tier-type=elasticsearch
>> --tier-config=endpoint=http://${ELASTICHOST}:9200,num_replicas=1,num_shards=10
>> radosgw-admin period update --commit
>>
>> sudo systemctl restart ceph-radosgw@rgw.${RGWHOST}
>> sudo radosgw --keyring /etc/ceph/ceph.client.admin.keyring -f
>> --rgw-zone=${ZONE2} --rgw-frontends="civetweb port=8002"
>> 2018-01-08 00:21:54.389432 7f0fe9cd2e80 -1 Couldn't init storage provider
>> (RADOS)
>>
>>  As you can see, starting rgw on port 8002 failed, but rgw on port
>> 8000 was started successfully.
>>  Here are some more info which may be useful for diagnosis:
>>
>> $ cat /etc/ceph/ceph.conf
>> [global]
>> fsid = 3e5a32d4-e45e-48dd-a3c5-f6f28fef8edf
>> mon_initial_members = ceph-mon1, ceph-osd1, ceph-osd2, ceph-osd3
>> mon_host = 172.30.212.226,172.30.212.227,172.30.212.228,172.30.212.250
>> auth_cluster_required = cephx
>> auth_service_required = cephx
>> auth_client_required = cephx
>> osd_pool_default_size = 2
>> osd_pool_default_min_size = 2
>> osd_pool_default_pg_num = 100
>> osd_pool_default_pgp_num = 100
>> bluestore_compression_algorithm = zlib
>> bluestore_compression_mode = force
>> rgw_max_put_size = 21474836480
>> [osd]
>> osd_max_object_size = 1073741824
>> [mon]
>> mon_allow_pool_delete = true
>> [client.rgw.ceph-rgw1]
>> host = ceph-rgw1
>> rgw frontends = civetweb port=8000
>>
>> $ wget -O - -q http://ceph-rgw2:9200/
>> {
>>   "name" : "Hippolyta",
>>   "cluster_name" : "elasticsearch",
>>   "version" : {
>> "number" : "2.3.1",
>> "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
>> "build_timestamp" : "2016-04-04T12:25:05Z",
>> "build_snapshot" : false,
>> "lucene_version" : "5.5.0"
>>   },
>>   "tagline" : "You Know, for Search"
>> }
>>
>> $ ceph df
>> GLOBAL:
>> SIZE AVAIL RAW USED %RAW USED
>> 719G  705G   14473M   

Re: [ceph-users] Luminous RGW Metadata Search

[Yehuda Sadeh-Weinraub]

On Fri, Dec 22, 2017 at 11:49 PM, Youzhong Yang  wrote:
> I followed the exact steps of the following page:
>
> http://ceph.com/rgw/new-luminous-rgw-metadata-search/
>
> "us-east-1" zone is serviced by host "ceph-rgw1" on port 8000, no issue, the
> service runs successfully.
>
> "us-east-es" zone is serviced by host "ceph-rgw2" on port 8002, the service
> was unable to start:
>
> # /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-rgw2 --setuser
> ceph --setgroup ceph   2017-12-22 16:35:48.513912 7fc54e98ee80 -1
> Couldn't init storage provider (RADOS)
>
> It's this mysterious error message "Couldn't init storage provider (RADOS)",
> there's no any clue what is wrong, what is mis-configured or anything like
> that.

radosgw failed to connect to your ceph cluster. Does the rados command
with the same connection params work?

Yehuda
>
> Yes, I have elasticsearch installed and running on host 'ceph-rgw2'. Is
> there any additional configuration required for ElasticSearch?
>
> Did I miss anything? what is the magic to make this basic stuff work?
>
> Thanks,
>
> --Youzhong
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW Logging pool

[Yehuda Sadeh-Weinraub]

(+matt)

On Fri, Dec 15, 2017 at 7:21 PM, David Turner  wrote:
> We're trying to build an auditing system for when a user key pair performs
> an operation on a bucket (put, delete, creating a bucket, etc) and so far
> were only able to find this information in the level 10 debug logging in the
> rgw systems logs.

Yeah, there's no readily available single log line for that currently.
Log level 2 has the operation type and the resource, so you're missing
the cross reference to the user. This might not be too complicated to
add, so maybe someone would like to try implementing that. This will
need to fit with the whole identity work that Matt's team is currently
working on.

>
> We noticed that our rgw log pool has been growing somewhat indefinitely and
> we had to move it off of the nvme's and put it to HDD's due to it's growing
> size.  What is in that pool and how can it be accessed?  I haven't found the
> right terms to search for to find anything about what's in this pool on the
> ML or on Google.

The log pool contains data for multiple different rgw functionalities:
garbage collector, lifecycle, multisite sync, usage logging. Some of
it is being stored in separate rados namespaces, so you need to run
'rados ls -p  --all' to see all the objects there.

>
> What I would like to do is export the log to ElasticSearch, cleanup the log
> on occasion, and hopefully find the information we're looking for to fulfill
> our user auditing without having our RGW daemons running on debug level 10
> (which is a lot of logging!).


Have you looked at the usage logging?

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Running Jewel and Luminous mixed for a longer period

[Yehuda Sadeh-Weinraub]

It's hard to say, we don't really test your specific scenario so use
it with your own risk. There was a change in cls_refcount that we had
issues with in the upgrade suite, but looking at it I'm not sure it'll
actually be a problem for you (you'll still hit the original problem
though).
Other problematic area is the osd limit on large omap operations, for
which we added 'truncated' flag for the relevant objclass operations.
Running older rgw against newer osds might cause issues when listing
omaps. You should make sure that bucket listing works correctly, but
there may be other issues (garbage collector, listing of user's
buckets, multipart upload completion). It could be that you can
configure that osd limit to be a higher number, so that you won't hit
that issue (RGW probably never requests more than 1000 entries off
omap, so setting it to 1k should be fine).

Yehuda

On Wed, Dec 6, 2017 at 2:09 PM, Wido den Hollander <w...@42on.com> wrote:
>
>> Op 6 december 2017 om 10:25 schreef Yehuda Sadeh-Weinraub 
>> <yeh...@redhat.com>:
>>
>>
>> Are you using rgw? There are certain compatibility issues that you
>> might hit if you run mixed versions.
>>
>
> Yes, it is. So would it hurt if OSDs are running Luminous but the RGW is 
> still Jewel?
>
> Multisite isn't used, it's just a local RGW.
>
> Wido
>
>> Yehuda
>>
>> On Tue, Dec 5, 2017 at 3:20 PM, Wido den Hollander <w...@42on.com> wrote:
>> > Hi,
>> >
>> > I haven't tried this before but I expect it to work, but I wanted to check 
>> > before proceeding.
>> >
>> > I have a Ceph cluster which is running with manually formatted FileStore 
>> > XFS disks, Jewel, sysvinit and Ubuntu 14.04.
>> >
>> > I would like to upgrade this system to Luminous, but since I have to 
>> > re-install all servers and re-format all disks I'd like to move it to 
>> > BlueStore at the same time.
>> >
>> > This system however has 768 3TB disks and has a utilization of about 60%. 
>> > You can guess, it will take a long time before all the backfills complete.
>> >
>> > The idea is to take a machine down, wipe all disks, re-install it with 
>> > Ubuntu 16.04 and Luminous and re-format the disks with BlueStore.
>> >
>> > The OSDs get back, start to backfill and we wait.
>> >
>> > My estimation is that we can do one machine per day, but we have 48 
>> > machines to do. Realistically this will take ~60 days to complete.
>> >
>> > Afaik running Jewel (10.2.10) mixed with Luminous (12.2.2) should work 
>> > just fine I wanted to check if there are any caveats I don't know about.
>> >
>> > I'll upgrade the MONs to Luminous first before starting to upgrade the 
>> > OSDs. Between each machine I'll wait for a HEALTH_OK before proceeding 
>> > allowing the MONs to trim their datastore.
>> >
>> > The question is: Does it hurt to run Jewel and Luminous mixed for ~60 days?
>> >
>> > I think it won't, but I wanted to double-check.
>> >
>> > Wido
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Running Jewel and Luminous mixed for a longer period

[Yehuda Sadeh-Weinraub]

Are you using rgw? There are certain compatibility issues that you
might hit if you run mixed versions.

Yehuda

On Tue, Dec 5, 2017 at 3:20 PM, Wido den Hollander  wrote:
> Hi,
>
> I haven't tried this before but I expect it to work, but I wanted to check 
> before proceeding.
>
> I have a Ceph cluster which is running with manually formatted FileStore XFS 
> disks, Jewel, sysvinit and Ubuntu 14.04.
>
> I would like to upgrade this system to Luminous, but since I have to 
> re-install all servers and re-format all disks I'd like to move it to 
> BlueStore at the same time.
>
> This system however has 768 3TB disks and has a utilization of about 60%. You 
> can guess, it will take a long time before all the backfills complete.
>
> The idea is to take a machine down, wipe all disks, re-install it with Ubuntu 
> 16.04 and Luminous and re-format the disks with BlueStore.
>
> The OSDs get back, start to backfill and we wait.
>
> My estimation is that we can do one machine per day, but we have 48 machines 
> to do. Realistically this will take ~60 days to complete.
>
> Afaik running Jewel (10.2.10) mixed with Luminous (12.2.2) should work just 
> fine I wanted to check if there are any caveats I don't know about.
>
> I'll upgrade the MONs to Luminous first before starting to upgrade the OSDs. 
> Between each machine I'll wait for a HEALTH_OK before proceeding allowing the 
> MONs to trim their datastore.
>
> The question is: Does it hurt to run Jewel and Luminous mixed for ~60 days?
>
> I think it won't, but I wanted to double-check.
>
> Wido
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] S3 object notifications

[Yehuda Sadeh-Weinraub]

On Wed, Nov 29, 2017 at 12:43 AM,  <ceph.nov...@habmalnefrage.de> wrote:
> Hi Yehuda.
>
> Are there any examples (doc's, blog posts, ...):
> - how to use that "framework" and especially for the "callbacks"

There's a minimal sync module implementation that does nothing other
than write a debug log for each sync event:

https://github.com/ceph/ceph/blob/master/src/rgw/rgw_sync_module_log.cc

Docs-wise, the elasticsearch luminous blog has a concrete config
example that can be applied to other sync modules:
http://ceph.com/rgw/new-luminous-rgw-metadata-search/


> - for the latest "Metasearch" feature / usage with a S3 client/tools like 
> CyberDuck, s3cmd, AWSCLI or at least boto3?
>   - i.e. is an external ELK still needed or is this somehow included in RGW 
> now?
>

You still need an external elasticsearch server, it's not part of
ceph. However, search can be done by sending requests through rgw's
RESTful api. We have a test that uses boto to generate such requests,
but might not be exactly what you're looking for:
https://github.com/ceph/ceph/blob/master/src/test/rgw/rgw_multi/zone_es.py

Yehuda

> Thanks & regards
>
>
> Gesendet: Dienstag, 28. November 2017 um 13:52 Uhr
> Von: "Yehuda Sadeh-Weinraub" <yeh...@redhat.com>
> An: "Sean Purdy" <s.pu...@cv-library.co.uk>
> Cc: "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com>
> Betreff: Re: [ceph-users] S3 object notifications
> rgw has a sync modules framework that allows you to write your own
> sync plugins. The system identifies objects changes and triggers
> callbacks that can then act on those changes. For example, the
> metadata search feature that was added recently is using this to send
> objects metadata into elasticsearch for indexing.
>
> Yehuda
>
> On Tue, Nov 28, 2017 at 2:22 PM, Sean Purdy <s.pu...@cv-library.co.uk> wrote:
>> Hi,
>>
>>
>> http://docs.ceph.com/docs/master/radosgw/s3/ says that S3 object 
>> notifications are not supported. I'd like something like object 
>> notifications so that we can backup new objects in realtime, instead of 
>> trawling the whole object list for what's changed.
>>
>> Is there anything similar I can use? I've found Spreadshirt's haproxy fork 
>> which traps requests and updates redis - 
>> https://github.com/spreadshirt/s3gw-haproxy[https://github.com/spreadshirt/s3gw-haproxy][https://github.com/spreadshirt/s3gw-haproxy[https://github.com/spreadshirt/s3gw-haproxy]]
>>  Anybody used that?
>>
>>
>> Thanks,
>>
>> Sean Purdy
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] S3 object notifications

[Yehuda Sadeh-Weinraub]

rgw has a sync modules framework that allows you to write your own
sync plugins. The system identifies objects changes and triggers
callbacks that can then act on those changes. For example, the
metadata search feature that was added recently is using this to send
objects metadata into elasticsearch for indexing.

Yehuda

On Tue, Nov 28, 2017 at 2:22 PM, Sean Purdy  wrote:
> Hi,
>
>
> http://docs.ceph.com/docs/master/radosgw/s3/ says that S3 object 
> notifications are not supported.  I'd like something like object 
> notifications so that we can backup new objects in realtime, instead of 
> trawling the whole object list for what's changed.
>
> Is there anything similar I can use?  I've found Spreadshirt's haproxy fork 
> which traps requests and updates redis - 
> https://github.com/spreadshirt/s3gw-haproxy  Anybody used that?
>
>
> Thanks,
>
> Sean Purdy
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: What's the fastest way to try out object classes?

[Yehuda Sadeh-Weinraub]

On Thu, Nov 9, 2017 at 10:05 AM, Zheyuan Chen  wrote:
> I installed rados-objclass-dev and objclass.h was installed successfully.
> However, I failed to run the objclass following the steps as below:
>
> 1. copy https://github.com/ceph/ceph/blob/master/src/cls/sdk/cls_sdk.cc into
> my machine. (cls_test.cpp)
> 2. make some changes on cls_test.cpp: 1) rename all "sdk" into "test". 2)
> add "namespace ceph {..}" wrapping the whole code.
> 3. compile it using the g++: g++ -std=c++11 -fPIC cls_test.cpp --shared -o
> libcls_test.so
> 4. copy libcls_test.so to all osds:/usr/lib/rados-classes
> 5. add two lines in ceph.conf: "osd class load list = *" and "osd class
> default list = *" and copy to all nodes.
> 6. restart all nodes in the cluster
> 7. call the objclass from python code
> ~~~
> ioctx.execute('oid', 'test', 'test_coverage_write', "test")
> ~~~
> I got this error:
> ~~~
> ...
> File "rados.pyx", line 498, in rados.requires.wrapper.validate_func
> (/build/ceph-12.2.1/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:4922)
> File "rados.pyx", line 2751, in rados.Ioctx.execute
> (/build/ceph-12.2.1/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:35467)
> rados.OSError: [errno 95] Ioctx.read(test): failed to read oid


errno 95: Not supported. Either the test object class failed to load,
or it couldn't find the test_coverage_write method. Try looking at
your osd logs (set 'debug objclass = 20' in your ceph.conf).

Yehuda

> ~~~
> 8. calling sdk gave me no error
> ~~~
> ioctx.execute('oid', 'sdk', 'test_coverage_write', "test")
> ~~~
>
> Did I do anything wrong here? I hope anyone can help me with this.
>
> Thank you very much,
> Zheyuan
>
> On Mon, Oct 30, 2017 at 4:20 PM, Neha Ojha  wrote:
>>
>> Should be rados-objclass-dev or rados-objclass-devel. Try and let me
>> know how it goes. Honestly, I've always done it from source :)
>>
>> On Mon, Oct 30, 2017 at 4:12 PM, Zheyuan Chen  wrote:
>> > Do you know which package should I install?
>> >
>> > On Mon, Oct 30, 2017 at 3:54 PM, Neha Ojha  wrote:
>> >>
>> >> I am not sure about a docker image, but you should be able to install
>> >> it through packages.
>> >>
>> >> On Mon, Oct 30, 2017 at 3:20 PM, Zheyuan Chen 
>> >> wrote:
>> >> > Hi Neha,
>> >> >
>> >> > Thanks for answering.
>> >> > Building from source just takes too much time. So I was wondering if
>> >> > there's
>> >> > any docker image or prebuilt package already containing objclass.h
>> >> > If that's the only way, I have to go ahead with it.
>>
>> >> >
>> >> > On Mon, Oct 30, 2017 at 3:05 PM, Neha Ojha  wrote:
>> >> >>
>> >> >> Hi Zheyuan,
>> >> >>
>> >> >> You can build Ceph from source and run make install. This should
>> >> >> place
>> >> >> objclass.h in /include/rados/ .
>> >> >>
>> >> >> Thanks,
>> >> >> Neha
>> >> >>
>> >> >> On Mon, Oct 30, 2017 at 2:18 PM, Zheyuan Chen 
>> >> >> wrote:
>> >> >> >
>> >> >> > -- Forwarded message --
>> >> >> > From: Zheyuan Chen 
>> >> >> > Date: Mon, Oct 30, 2017 at 2:16 PM
>> >> >> > Subject: What's the fastest way to try out object classes?
>> >> >> > To: ceph-users@lists.ceph.com
>> >> >> >
>> >> >> >
>> >> >> > Hi All,
>> >> >> >
>> >> >> > I'd like to try out object classes.
>> >> >> > http://docs.ceph.com/docs/master/rados/api/objclass-sdk/
>> >> >> > I used this docker image: https://hub.docker.com/r/ceph/demo/, but
>> >> >> > found
>> >> >> > the
>> >> >> > object class sdk is not included (couldn't find
>> >> >> > /usr/local/include/rados/objectclass.h) even after I installed
>> >> >> > librados-devel manually.
>> >> >> >
>> >> >> > Do I have to build from the source code if I want to have
>> >> >> > objectclass.h?
>> >> >> > What is the fastest way to set up the environment if I want to try
>> >> >> > out
>> >> >> > object classes?
>> >> >> >
>> >> >> > Thank you very much!
>> >> >> > Zheyuan
>> >> >> >
>> >> >> >
>> >> >> > ___
>> >> >> > ceph-users mailing list
>> >> >> > ceph-users@lists.ceph.com
>> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW: ERROR: failed to distribute cache

[Yehuda Sadeh-Weinraub]

On Mon, Nov 6, 2017 at 7:29 AM, Wido den Hollander  wrote:
> Hi,
>
> On a Ceph Luminous (12.2.1) environment I'm seeing RGWs stall and about the 
> same time I see these errors in the RGW logs:
>
> 2017-11-06 15:50:24.859919 7f8f5fa1a700  0 ERROR: failed to distribute cache 
> for 
> gn1-pf.rgw.data.root:.bucket.meta.X:eb32b1ca-807a-4867-aea5-ff43ef7647c6.20755572.20
> 2017-11-06 15:50:41.768881 7f8f7824b700  0 ERROR: failed to distribute cache 
> for gn1-pf.rgw.data.root:X
> 2017-11-06 15:55:15.781739 7f8f7824b700  0 ERROR: failed to distribute cache 
> for 
> gn1-pf.rgw.meta:.meta:bucket.instance:X:eb32b1ca-807a-4867-aea5-ff43ef7647c6.20755572.32:_XK5LExyX6EEIXxCD5Cws:1
> 2017-11-06 15:55:25.784404 7f8f7824b700  0 ERROR: failed to distribute cache 
> for 
> gn1-pf.rgw.data.root:.bucket.meta.X:eb32b1ca-807a-4867-aea5-ff43ef7647c6.20755572.32
>
> I see one message from a year ago: 
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-June/010531.html
>
> The setup has two RGWs running:
>
> - ceph-rgw1
> - ceph-rgw2
>
> While trying to figure this out I see that a "radosgw-admin period pull" 
> hangs for ever.
>
> I don't know if that is related, but it's something I've noticed.
>
> Mainly I see that at random times the RGW stalls for about 30 seconds and 
> while that happens these messages show up in the RGW's log.
>

do you happen to know if there's a dynamic resharding happening? The
dynamic resharding should only affect the writes to the specific
bucket, and should not affect cache distribution though. Originally I
thought it could be HUP signal related issue, but that seem to be
fixed in 12.2.1.

Yehuda

> Is anybody else running into this issue?
>
> Wido
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Speeding up garbage collection in RGW

[Yehuda Sadeh-Weinraub]

On Wed, Oct 25, 2017 at 2:32 PM, Bryan Stillwell <bstillw...@godaddy.com> wrote:
> That helps a little bit, but overall the process would take years at this
> rate:
>
>
>
> # for i in {1..3600}; do ceph df -f json-pretty |grep -A7 '".rgw.buckets"'
> |grep objects; sleep 60; done
>
> "objects": 1660775838
>
> "objects": 1660775733
>
> "objects": 1660775548
>
> "objects": 1660774825
>
> "objects": 1660774790
>
> "objects": 1660774735
>
>
>
> This is on a hammer cluster.  Would upgrading to Jewel or Luminous speed up
> this process at all?
>

I'm not sure it's going to help much, although the omap performance
might improve there. The big problem is that the omaps are just too
big, so that every operation on them take considerable time. I think
the best way forward there is to take a list of all the rados objects
that need to be removed from the gc omaps, and then get rid of the gc
objects themselves (newer ones will be created, this time using the
new configurable). Then remove the objects manually (and concurrently)
using the rados command line tool.
The one problem I see here is that even just removal of objects with
large omaps can affect the availability of the osds that hold these
objects. I discussed that now with Josh, and we think the best way to
deal with that is not to remove the gc objects immediatly, but to
rename the gc pool, and create a new one (with appropriate number of
pgs). This way new gc entries will now go into the new gc pool (with
higher number of gc shards), and you don't need to remove the old gc
objects (thus no osd availability problem). Then you can start
trimming the old gc objects (on the old renamed pool) by using the
rados command. It'll take a very very long time, but the process
should pick up speed slowly, as the objects shrink.

Yehuda

>
>
> Bryan
>
>
>
> From: Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> Date: Wednesday, October 25, 2017 at 11:32 AM
> To: Bryan Stillwell <bstillw...@godaddy.com>
> Cc: David Turner <drakonst...@gmail.com>, Ben Hines <bhi...@gmail.com>,
> "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com>
>
>
> Subject: Re: [ceph-users] Speeding up garbage collection in RGW
>
>
>
> Some of the options there won't do much for you as they'll only affect
>
> newer object removals. I think the default number of gc objects is
>
> just inadequate for your needs. You can try manually running
>
> 'radosgw-admin gc process' concurrently (for the start 2 or 3
>
> processes), see if it makes any dent there. I think one of the problem
>
> is that the gc omaps grew so much that operations on them are too
>
> slow.
>
>
>
> Yehuda
>
>
>
> On Wed, Oct 25, 2017 at 9:05 AM, Bryan Stillwell <bstillw...@godaddy.com>
> wrote:
>
> We tried various options like the one's Ben mentioned to speed up the
> garbage collection process and were unsuccessful.  Luckily, we had the
> ability to create a new cluster and move all the data that wasn't part of
> the POC which created our problem.
>
>
>
> One of the things we ran into was the .rgw.gc pool became too large to
> handle drive failures without taking down the cluster.  We eventually had to
> move that pool to SSDs just to get the cluster healthy.  It was not obvious
> it was getting large though, because this is what it looked like in the
> 'ceph df' output:
>
>
>
>  NAME   ID USED  %USED MAX AVAIL OBJECTS
>
>  .rgw.gc17 0 0  235G
> 2647
>
>
>
> However, if you look at the SSDs we used (repurposed journal SSDs to get out
> of the disaster) in 'ceph osd df' you can see quite a bit of data is being
> used:
>
>
>
> 410 0.2  1.0  181G 23090M   158G 12.44 0.18
>
> 411 0.2  1.0  181G 29105M   152G 15.68 0.22
>
> 412 0.2  1.0  181G   110G 72223M 61.08 0.86
>
> 413 0.2  1.0  181G 42964M   139G 23.15 0.33
>
> 414 0.2  1.0  181G 33530M   148G 18.07 0.26
>
> 415 0.2  1.0  181G 38420M   143G 20.70 0.29
>
> 416 0.2  1.0  181G 92215M 93355M 49.69 0.70
>
> 417 0.2  1.0  181G 64730M   118G 34.88 0.49
>
> 418 0.2  1.0  181G 61353M   121G 33.06 0.47
>
> 419 0.2  1.0  181G 77168M   105G 41.58 0.59
>
>
>
> That's ~560G of omap data for the .rgw.gc pool that isn't being reported in
> 'ceph df'.
>
>
>
> Right now the cluster is still around while we wait to verify the new
> cluster isn't missing anything.  So if the

Re: [ceph-users] Speeding up garbage collection in RGW

[Yehuda Sadeh-Weinraub]

Some of the options there won't do much for you as they'll only affect
newer object removals. I think the default number of gc objects is
just inadequate for your needs. You can try manually running
'radosgw-admin gc process' concurrently (for the start 2 or 3
processes), see if it makes any dent there. I think one of the problem
is that the gc omaps grew so much that operations on them are too
slow.

Yehuda

On Wed, Oct 25, 2017 at 9:05 AM, Bryan Stillwell  wrote:
> We tried various options like the one's Ben mentioned to speed up the garbage 
> collection process and were unsuccessful.  Luckily, we had the ability to 
> create a new cluster and move all the data that wasn't part of the POC which 
> created our problem.
>
> One of the things we ran into was the .rgw.gc pool became too large to handle 
> drive failures without taking down the cluster.  We eventually had to move 
> that pool to SSDs just to get the cluster healthy.  It was not obvious it was 
> getting large though, because this is what it looked like in the 'ceph df' 
> output:
>
> NAME   ID USED  %USED MAX AVAIL OBJECTS
> .rgw.gc17 0 0  235G   2647
>
> However, if you look at the SSDs we used (repurposed journal SSDs to get out 
> of the disaster) in 'ceph osd df' you can see quite a bit of data is being 
> used:
>
> 410 0.2  1.0  181G 23090M   158G 12.44 0.18
> 411 0.2  1.0  181G 29105M   152G 15.68 0.22
> 412 0.2  1.0  181G   110G 72223M 61.08 0.86
> 413 0.2  1.0  181G 42964M   139G 23.15 0.33
> 414 0.2  1.0  181G 33530M   148G 18.07 0.26
> 415 0.2  1.0  181G 38420M   143G 20.70 0.29
> 416 0.2  1.0  181G 92215M 93355M 49.69 0.70
> 417 0.2  1.0  181G 64730M   118G 34.88 0.49
> 418 0.2  1.0  181G 61353M   121G 33.06 0.47
> 419 0.2  1.0  181G 77168M   105G 41.58 0.59
>
> That's ~560G of omap data for the .rgw.gc pool that isn't being reported in 
> 'ceph df'.
>
> Right now the cluster is still around while we wait to verify the new cluster 
> isn't missing anything.  So if there is anything the RGW developers would 
> like to try on it to speed up the gc process, we should be able to do that.
>
> Bryan
>
> From: ceph-users  on behalf of David 
> Turner 
> Date: Tuesday, October 24, 2017 at 4:07 PM
> To: Ben Hines 
> Cc: "ceph-users@lists.ceph.com" 
> Subject: Re: [ceph-users] Speeding up garbage collection in RGW
>
> Thank you so much for chiming in, Ben.
>
> Can you explain what each setting value means? I believe I understand min 
> wait, that's just how long to wait before allowing the object to be cleaned 
> up.  gc max objs is how many will be cleaned up during each period?  gc 
> processor period is how often it will kick off gc to clean things up?  And gc 
> processor max time is the longest the process can run after the period 
> starts?  Is that about right for that?  I read somewhere saying that prime 
> numbers are optimal for gc max objs.  Do you know why that is?  I notice 
> you're using one there.  What is lc max objs?  I couldn't find a reference 
> for that setting.
>
> Additionally, do you know if the radosgw-admin gc list is ever cleaned up, or 
> is it an ever growing list?  I got up to 3.6 Billion objects in the list 
> before I killed the gc list command.
>
> On Tue, Oct 24, 2017 at 4:47 PM Ben Hines  wrote:
> I agree the settings are rather confusing. We also have many millions of 
> objects and had this trouble, so i set these rather aggressive gc settings on 
> our cluster which result in gc almost always running. We also use lifecycles 
> to expire objects.
>
> rgw lifecycle work time = 00:01-23:59
> rgw gc max objs = 2647
> rgw lc max objs = 2647
> rgw gc obj min wait = 300
> rgw gc processor period = 600
> rgw gc processor max time = 600
>
>
> -Ben
>
> On Tue, Oct 24, 2017 at 9:25 AM, David Turner  wrote:
> As I'm looking into this more and more, I'm realizing how big of a problem 
> garbage collection has been in our clusters.  The biggest cluster has over 1 
> billion objects in its gc list (the command is still running, it just 
> recently passed by the 1B mark).  Does anyone have any guidance on what to do 
> to optimize the gc settings to hopefully/eventually catch up on this as well 
> as stay caught up once we are?  I'm not expecting an overnight fix, but 
> something that could feasibly be caught up within 6 months would be wonderful.
>
> On Mon, Oct 23, 2017 at 11:18 AM David Turner  wrote:
> We recently deleted a bucket that was no longer needed that had 400TB of data 
> in it to help as our cluster is getting quite full.  That should free up 
> about 30% of our cluster used space, but in the last week we haven't seen 
> nearly a fraction of that 

Re: [ceph-users] Luminous 12.2.1 - RadosGW Multisite doesnt replicate multipart uploads

[Yehuda Sadeh-Weinraub]

What is the size of the object? Is it only this one?

Try this command: 'radosgw-admin sync error list'. Does it show anything
related to that object?

Thanks,
Yehuda


On Wed, Oct 11, 2017 at 3:26 PM, Enrico Kern <enrico.k...@glispamedia.com>
wrote:

> if i change permissions the sync status shows that it is syncing 1 shard,
> but no files ends up in the pool (testing with empty data pool). after a
> while it shows that data is back in sync but there is no file
>
> On Wed, Oct 11, 2017 at 11:26 PM, Yehuda Sadeh-Weinraub <yeh...@redhat.com
> > wrote:
>
>> Thanks for your report. We're looking into it. You can try to see if
>> touching the object (e.g., modifying its permissions) triggers the sync.
>>
>> Yehuda
>>
>> On Wed, Oct 11, 2017 at 1:36 PM, Enrico Kern <enrico.k...@glispamedia.com
>> > wrote:
>>
>>> Hi David,
>>>
>>> yeah seems you are right, they are stored as different filenames in the
>>> data bucket when using multisite upload. But anyway it stil doesnt get
>>> replicated. As example i have files like
>>>
>>> 6a9448d2-bdba-4bec-aad6-aba72cd8eac6.21344646.1__multipart_W
>>> ireshark-win64-2.2.7.exe.2~0LAfq93OMdk7hrijvyzW_EBRkVQLX37.6
>>>
>>> in the data pool on one zone. But its not replicated to the other zone.
>>> naming is not relevant, the other data bucket doesnt have any file
>>> multipart or not.
>>>
>>> im really missing the file on the other zone.
>>>
>>>
>>> <http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>>  Virenfrei.
>>> www.avg.com
>>> <http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>> <#m_-2933057183600238029_m_-2032373670326744902_m_1117069282601502036_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>> On Wed, Oct 11, 2017 at 10:25 PM, David Turner <drakonst...@gmail.com>
>>> wrote:
>>>
>>>> Multipart is a client side setting when uploading.  Multisite in and of
>>>> itself is a client and it doesn't use multipart (at least not by default).
>>>> I have a Jewel RGW Multisite cluster and one site has the object as
>>>> multi-part while the second site just has it as a single object.  I had to
>>>> change from looking at the objects in the pool for monitoring to looking at
>>>> an ls of the buckets to see if they were in sync.
>>>>
>>>> I don't know if multisite has the option to match if an object is
>>>> multipart between sites, but it definitely doesn't seem to be the default
>>>> behavior.
>>>>
>>>> On Wed, Oct 11, 2017 at 3:56 PM Enrico Kern <
>>>> enrico.k...@glispamedia.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> i just setup multisite replication according to the docs from
>>>>> http://docs.ceph.com/docs/master/radosgw/multisite/ and everything
>>>>> works except that if a client uploads via multipart the files dont get
>>>>> replicated.
>>>>>
>>>>> If i in one zone rename a file that was uploaded via multipart it gets
>>>>> replicated, but not if i left it untouched. Any ideas why? I remember 
>>>>> there
>>>>> was a similar bug with jewel a while back.
>>>>>
>>>>> On the slave node i also permanently get this error (unrelated to the
>>>>> replication) in the radosgw log:
>>>>>
>>>>>  meta sync: ERROR: failed to read mdlog info with (2) No such file or
>>>>> directory
>>>>>
>>>>> we didnt run radosgw before the luminous upgrade of our clusters.
>>>>>
>>>>> after a finished multipart upload which is only visible at one zone
>>>>> "radosgw-admin sync status" just shows that metadata and data is caught up
>>>>> with the source.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Enrico Kern
>>>>> *Lead System Engineer*
>>>>>
>>>>> *T* +49 (0) 30 555713017 <+49%2030%20555713017>  | *M *+49 (0)152
>>>>> 26814501 <+49%201522%206814501>
>>>>> *E*  enrico.k...@glispa.com |  *Skype flyersa* |  LinkedIn View my
>>>>> Profile <https://www.linkedin.com/in/enricokern/>
>>>>>
>>>>>
>>>>>
>>>&g

Re: [ceph-users] Luminous 12.2.1 - RadosGW Multisite doesnt replicate multipart uploads

[Yehuda Sadeh-Weinraub]

Thanks for your report. We're looking into it. You can try to see if
touching the object (e.g., modifying its permissions) triggers the sync.

Yehuda

On Wed, Oct 11, 2017 at 1:36 PM, Enrico Kern 
wrote:

> Hi David,
>
> yeah seems you are right, they are stored as different filenames in the
> data bucket when using multisite upload. But anyway it stil doesnt get
> replicated. As example i have files like
>
> 6a9448d2-bdba-4bec-aad6-aba72cd8eac6.21344646.1__
> multipart_Wireshark-win64-2.2.7.exe.2~0LAfq93OMdk7hrijvyzW_EBRkVQLX37.6
>
> in the data pool on one zone. But its not replicated to the other zone.
> naming is not relevant, the other data bucket doesnt have any file
> multipart or not.
>
> im really missing the file on the other zone.
>
>
> 
>  Virenfrei.
> www.avg.com
> 
> <#m_1117069282601502036_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Wed, Oct 11, 2017 at 10:25 PM, David Turner 
> wrote:
>
>> Multipart is a client side setting when uploading.  Multisite in and of
>> itself is a client and it doesn't use multipart (at least not by default).
>> I have a Jewel RGW Multisite cluster and one site has the object as
>> multi-part while the second site just has it as a single object.  I had to
>> change from looking at the objects in the pool for monitoring to looking at
>> an ls of the buckets to see if they were in sync.
>>
>> I don't know if multisite has the option to match if an object is
>> multipart between sites, but it definitely doesn't seem to be the default
>> behavior.
>>
>> On Wed, Oct 11, 2017 at 3:56 PM Enrico Kern 
>> wrote:
>>
>>> Hi all,
>>>
>>> i just setup multisite replication according to the docs from
>>> http://docs.ceph.com/docs/master/radosgw/multisite/ and everything
>>> works except that if a client uploads via multipart the files dont get
>>> replicated.
>>>
>>> If i in one zone rename a file that was uploaded via multipart it gets
>>> replicated, but not if i left it untouched. Any ideas why? I remember there
>>> was a similar bug with jewel a while back.
>>>
>>> On the slave node i also permanently get this error (unrelated to the
>>> replication) in the radosgw log:
>>>
>>>  meta sync: ERROR: failed to read mdlog info with (2) No such file or
>>> directory
>>>
>>> we didnt run radosgw before the luminous upgrade of our clusters.
>>>
>>> after a finished multipart upload which is only visible at one zone
>>> "radosgw-admin sync status" just shows that metadata and data is caught up
>>> with the source.
>>>
>>>
>>>
>>> --
>>>
>>> Enrico Kern
>>> *Lead System Engineer*
>>>
>>> *T* +49 (0) 30 555713017 <+49%2030%20555713017>  | *M *+49 (0)152
>>> 26814501 <+49%201522%206814501>
>>> *E*  enrico.k...@glispa.com |  *Skype flyersa* |  LinkedIn View my
>>> Profile 
>>>
>>>
>>>
>>> *Glispa GmbH* - Berlin Office
>>> Sonnenburger Str. 73 10437 Berlin, Germany
>>> 
>>>
>>> Managing Director: David Brown, Registered in Berlin, AG Charlottenburg
>>> HRB 114678B
>>>    
>>> 
>>> 
>>>
>>> 
>>> 
>>>
>>>
>>> 
>>>  Virenfrei.
>>> www.avg.com
>>> 
>>> <#m_1117069282601502036_m_-8782081303072922711_m_-7647428735749890284_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>
>
> --
>
> Enrico Kern
> *Lead System Engineer*
>
> *T* +49 (0) 30 555713017 <+49%2030%20555713017>  | *M *+49 (0)152 26814501
> <+49%201522%206814501>
> *E*  enrico.k...@glispa.com |  *Skype flyersa* |  LinkedIn View my Profile
> 
>
>
>
> *Glispa GmbH* - Berlin Office
> Sonnenburger Str. 73 10437 Berlin, Germany
> Managing Director: Dina Karol-Gavish, Registered in Berlin, AG
> Charlottenburg HRB 114678B
>    
> 
> 
>   
>
>
> 

Re: [ceph-users] rgw resharding operation seemingly won't end

[Yehuda Sadeh-Weinraub]

On Mon, Oct 9, 2017 at 1:59 PM, Ryan Leimenstoll
 wrote:
> Hi all,
>
> We recently upgraded to Ceph 12.2.1 (Luminous) from 12.2.0 however are now 
> seeing issues running radosgw. Specifically, it appears an automatically 
> triggered resharding operation won’t end, despite the jobs being cancelled 
> (radosgw-admin reshard cancel). I have also disabled dynamic sharding for the 
> time being in the ceph.conf.
>
>
> [root@objproxy02 ~]# radosgw-admin reshard list
> []
>
> The two buckets were also reported in the `radosgw-admin reshard list` before 
> our RGW frontends paused recently (and only came back after a service 
> restart). These two buckets cannot currently be written to at this point 
> either.
>
> 2017-10-06 22:41:19.547260 7f90506e9700 0 block_while_resharding ERROR: 
> bucket is still resharding, please retry
> 2017-10-06 22:41:19.547411 7f90506e9700 0 WARNING: set_req_state_err 
> err_no=2300 resorting to 500
> 2017-10-06 22:41:19.547729 7f90506e9700 0 ERROR: 
> RESTFUL_IO(s)->complete_header() returned err=Input/output error
> 2017-10-06 22:41:19.548570 7f90506e9700 1 == req done req=0x7f90506e3180 
> op status=-2300 http_status=500 ==
> 2017-10-06 22:41:19.548790 7f90506e9700 1 civetweb: 0x55766d111000: 
> $MY_IP_HERE$ - - [06/Oct/2017:22:33:47 -0400] "PUT /
> $REDACTED_BUCKET_NAME$/$REDACTED_KEY_NAME$ HTTP/1.1" 1 0 - Boto3/1.4.7 
> Python/2.7.12 Linux/4.9.43-17.3
> 9.amzn1.x86_64 exec-env/AWS_Lambda_python2.7 Botocore/1.7.2 Resource
> [.. slightly later in the logs..]
> 2017-10-06 22:41:53.516272 7f90406c9700 1 rgw realm reloader: Frontends paused
> 2017-10-06 22:41:53.528703 7f907893f700 0 ERROR: failed to clone shard, 
> completion_mgr.get_next() returned ret=-125
> 2017-10-06 22:44:32.049564 7f9074136700 0 ERROR: keystone revocation 
> processing returned error r=-22
> 2017-10-06 22:59:32.059222 7f9074136700 0 ERROR: keystone revocation 
> processing returned error r=-22
>
> Can anyone advise on the best path forward to stop the current sharding 
> states and avoid this moving forward?
>

What does 'radosgw-admin reshard status --bucket=' return?
I think just manually resharding the buckets should clear this flag,
is that not an option?
manual reshard: radosgw-admin bucket reshard --bucket=
--num-shards=

also, the 'radosgw-admin bucket check --fix' might clear that flag.

For some reason it seems that the reshard cancellation code is not
clearing that flag on the bucket index header (pretty sure it used to
do it at one point). I'll open a tracker ticket.

Thanks,
Yehuda

>
> Some other details:
>  - 3 rgw instances
>  - Ceph Luminous 12.2.1
>  - 584 active OSDs, rgw bucket index is on Intel NVMe OSDs
>
>
> Thanks,
> Ryan Leimenstoll
> rleim...@umiacs.umd.edu
> University of Maryland Institute for Advanced Computer Studies
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw notify on creation/deletion of file in bucket

[Yehuda Sadeh-Weinraub]

On Tue, Oct 3, 2017 at 8:59 AM, Sean Purdy  wrote:
> Hi,
>
>
> Is there any way that radosgw can ping something when a file is removed or 
> added to a bucket?
>

That depends on what exactly you're looking for. You can't get that
info as a user. but there is a mechanism for remote zones to detect
changes that happen on the zone.

> Or use its sync facility to sync files to AWS/Google buckets?
>

Not at the moment, in the works. Unless you want to write your own sync plugin.

Yehuda

> Just thinking about backups.  What do people use for backups?  Been looking 
> at rclone.
>
>
> Thanks,
>
> Sean
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Ceph-maintainers] Ceph release cadence

[Yehuda Sadeh-Weinraub]

I'm not a huge fan of train releases, as they tend to never quite make
it on time and it always feels a bit artificial timeline anyway. OTOH,
I do see and understand the need of a predictable schedule with a
roadmap attached to it. There are many that need to have at least a
vague idea on what we're going to ship and when, so that they can plan
ahead. We need it ourselves, as sometimes the schedule can dictate the
engineering decision that we're going to make.
On the other hand, I think that working towards a release that come
out after 9 or 12 months is a bit too long. This is a recipe for more
delays as the penalty for missing a feature is painful. Maybe we can
consider returning to shorter iterations for *dev* releases. These are
checkpoints that need to happen after a short period (2-3 weeks),
where we end up with a minimally tested release that passes some smoke
test. Features are incrementally added to the dev release. The idea
behind a short term dev release is that it minimizes the window where
master is completely unusable, thus reduces the time to stabilization.
Then it's easier to enforce a train schedule if we want to. It might
be easier to let go of a feature that doesn't make it, as it will be
there soon, and maybe if really needed we (or the downstream
maintainer) can make the decision to backport it. This makes me think
that we could revisit the our backport policy/procedure/tooling, so
that we can do it in a sane and safe way when needed and possible.

Yehuda

On Fri, Sep 8, 2017 at 7:59 PM, Gregory Farnum  wrote:
> I think I'm the resident train release advocate so I'm sure my
> advocating that model will surprise nobody. I'm not sure I'd go all
> the way to Lars' multi-release maintenance model (although it's
> definitely something I'm interested in), but there are two big reasons
> I wish we were on a train with more frequent real releases:
>
> 1) It reduces the cost of features missing a release. Right now if
> something misses an LTS release, that's it for a year. And nobody
> likes releasing an LTS without a bunch of big new features, so each
> LTS is later than the one before as we scramble to get features merged
> in.
>
> ...and then we deal with the fact that we scrambled to get a bunch of
> features merged in and they weren't quite baked. (Luminous so far
> seems to have gone much better in this regard! Hurray! But I think
> that has a lot to do with our feature-release-scramble this year being
> mostly peripheral stuff around user interfaces that got tacked on
> about the time we'd initially planned the release to occur.)
>
> 2) Train releases increase predictability for downstreams, partners,
> and users around when releases will happen. Right now, the release
> process and schedule is entirely opaque to anybody who's not involved
> in every single upstream meeting we have; and it's unpredictable even
> to those who are. That makes things difficult, as Xiaoxi said.
>
> There are other peripheral but serious benefits I'd expect to see from
> fully-validated train releases as well. It would be *awesome* to have
> more frequent known-stable points to do new development against. If
> you're an external developer and you want a new feature, you have to
> either keep it rebased against a fast-changing master branch, or you
> need to settle for writing it against a long-out-of-date LTS and then
> forward-porting it for merge. If you're an FS developer writing a very
> small new OSD feature and you try to validate it against RADOS, you've
> no idea if bugs that pop up and look random are because you really did
> something wrong or if there's currently an intermittent issue in RADOS
> master. I would have *loved* to be able to maintain CephFS integration
> branches for features that didn't touch RADOS and were built on top of
> the latest release instead of master, but it was utterly infeasible
> because there were too many missing features with the long delays.
>
> On Fri, Sep 8, 2017 at 9:16 AM, Sage Weil  wrote:
>> I'm going to pick on Lars a bit here...
>>
>> On Thu, 7 Sep 2017, Lars Marowsky-Bree wrote:
>>> On 2017-09-06T15:23:34, Sage Weil  wrote:
>>> > Other options we should consider?  Other thoughts?
>>>
>>> With about 20-odd years in software development, I've become a big
>>> believer in schedule-driven releases. If it's feature-based, you never
>>> know when they'll get done.
>>>
>>> If the schedule intervals are too long though, the urge to press too
>>> much in (so as not to miss the next merge window) is just too high,
>>> meaning the train gets derailed. (Which cascades into the future,
>>> because the next time the pressure will be even higher based on the
>>> previous experience.) This requires strictness.
>>>
>>> We've had a few Linux kernel releases that were effectively feature
>>> driven and never quite made it. 1.3.x? 1.5.x? My memory is bad, but they
>>> were a disaster than eventually led Linus to evolve 

Re: [ceph-users] RGW Multisite metadata sync init

[Yehuda Sadeh-Weinraub]

On Thu, Sep 7, 2017 at 11:37 PM, David Turner <drakonst...@gmail.com> wrote:
> I'm pretty sure I'm using the cluster admin user/keyring.  Is there any
> output that would be helpful?  Period, zonegroup get, etc?

 - radosgw-admin period get
 - radosgw-admin zone list
 - radosgw-admin zonegroup list

For each zone, zonegroup in result:
 - radosgw-admin zone get --rgw-zone=
 - radosgw-admin zonegroup get --rgw-zonegroup=

 - rados lspools

Also, create a user with --debug-rgw=20 --debug-ms=1, need to look at the log.

Yehuda


>
> On Thu, Sep 7, 2017 at 4:27 PM Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> On Thu, Sep 7, 2017 at 11:02 PM, David Turner <drakonst...@gmail.com>
>> wrote:
>> > I created a test user named 'ice' and then used it to create a bucket
>> > named
>> > ice.  The bucket ice can be found in the second dc, but not the user.
>> > `mdlog list` showed ice for the bucket, but not for the user.  I
>> > performed
>> > the same test in the internal realm and it showed the user and bucket
>> > both
>> > in `mdlog list`.
>> >
>>
>> Maybe your radosgw-admin command is running with a ceph user that
>> doesn't have permissions to write to the log pool? (probably not,
>> because you are able to run the sync init commands).
>> Another very slim explanation would be if you had for some reason
>> overlapping zones configuration that shared some of the config but not
>> all of it, having radosgw running against the correct one and
>> radosgw-admin against the bad one. I don't think it's the second
>> option.
>>
>> Yehuda
>>
>> >
>> >
>> > On Thu, Sep 7, 2017 at 3:27 PM Yehuda Sadeh-Weinraub <yeh...@redhat.com>
>> > wrote:
>> >>
>> >> On Thu, Sep 7, 2017 at 10:04 PM, David Turner <drakonst...@gmail.com>
>> >> wrote:
>> >> > One realm is called public with a zonegroup called public-zg with a
>> >> > zone
>> >> > for
>> >> > each datacenter.  The second realm is called internal with a
>> >> > zonegroup
>> >> > called internal-zg with a zone for each datacenter.  they each have
>> >> > their
>> >> > own rgw's and load balancers.  The needs of our public facing rgw's
>> >> > and
>> >> > load
>> >> > balancers vs internal use ones was different enough that we split
>> >> > them
>> >> > up
>> >> > completely.  We also have a local realm that does not use multisite
>> >> > and
>> >> > a
>> >> > 4th realm called QA that mimics the public realm as much as possible
>> >> > for
>> >> > staging configuration stages for the rgw daemons.  All 4 realms have
>> >> > their
>> >> > own buckets, users, etc and that is all working fine.  For all of the
>> >> > radosgw-admin commands I am using the proper identifiers to make sure
>> >> > that
>> >> > each datacenter and realm are running commands on exactly what I
>> >> > expect
>> >> > them
>> >> > to (--rgw-realm=public --rgw-zonegroup=public-zg
>> >> > --rgw-zone=public-dc1
>> >> > --source-zone=public-dc2).
>> >> >
>> >> > The data sync issue was in the internal realm but running a data sync
>> >> > init
>> >> > and kickstarting the rgw daemons in each datacenter fixed the data
>> >> > discrepancies (I'm thinking it had something to do with a power
>> >> > failure
>> >> > a
>> >> > few months back that I just noticed recently).  The metadata sync
>> >> > issue
>> >> > is
>> >> > in the public realm.  I have no idea what is causing this to not sync
>> >> > properly since running a `metadata sync init` catches it back up to
>> >> > the
>> >> > primary zone, but then it doesn't receive any new users created after
>> >> > that.
>> >> >
>> >>
>> >> Sounds like an issue with the metadata log in the primary master zone.
>> >> Not sure what could go wrong there, but maybe the master zone doesn't
>> >> know that it is a master zone, or it's set to not log metadata. Or
>> >> maybe there's a problem when the secondary is trying to fetch the
>> >> metadata log. Maybe some kind of # of shards mismatch (though not
>> >> likely).
>> >

Re: [ceph-users] RGW Multisite metadata sync init

[Yehuda Sadeh-Weinraub]

On Thu, Sep 7, 2017 at 11:02 PM, David Turner <drakonst...@gmail.com> wrote:
> I created a test user named 'ice' and then used it to create a bucket named
> ice.  The bucket ice can be found in the second dc, but not the user.
> `mdlog list` showed ice for the bucket, but not for the user.  I performed
> the same test in the internal realm and it showed the user and bucket both
> in `mdlog list`.
>

Maybe your radosgw-admin command is running with a ceph user that
doesn't have permissions to write to the log pool? (probably not,
because you are able to run the sync init commands).
Another very slim explanation would be if you had for some reason
overlapping zones configuration that shared some of the config but not
all of it, having radosgw running against the correct one and
radosgw-admin against the bad one. I don't think it's the second
option.

Yehuda

>
>
> On Thu, Sep 7, 2017 at 3:27 PM Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> On Thu, Sep 7, 2017 at 10:04 PM, David Turner <drakonst...@gmail.com>
>> wrote:
>> > One realm is called public with a zonegroup called public-zg with a zone
>> > for
>> > each datacenter.  The second realm is called internal with a zonegroup
>> > called internal-zg with a zone for each datacenter.  they each have
>> > their
>> > own rgw's and load balancers.  The needs of our public facing rgw's and
>> > load
>> > balancers vs internal use ones was different enough that we split them
>> > up
>> > completely.  We also have a local realm that does not use multisite and
>> > a
>> > 4th realm called QA that mimics the public realm as much as possible for
>> > staging configuration stages for the rgw daemons.  All 4 realms have
>> > their
>> > own buckets, users, etc and that is all working fine.  For all of the
>> > radosgw-admin commands I am using the proper identifiers to make sure
>> > that
>> > each datacenter and realm are running commands on exactly what I expect
>> > them
>> > to (--rgw-realm=public --rgw-zonegroup=public-zg --rgw-zone=public-dc1
>> > --source-zone=public-dc2).
>> >
>> > The data sync issue was in the internal realm but running a data sync
>> > init
>> > and kickstarting the rgw daemons in each datacenter fixed the data
>> > discrepancies (I'm thinking it had something to do with a power failure
>> > a
>> > few months back that I just noticed recently).  The metadata sync issue
>> > is
>> > in the public realm.  I have no idea what is causing this to not sync
>> > properly since running a `metadata sync init` catches it back up to the
>> > primary zone, but then it doesn't receive any new users created after
>> > that.
>> >
>>
>> Sounds like an issue with the metadata log in the primary master zone.
>> Not sure what could go wrong there, but maybe the master zone doesn't
>> know that it is a master zone, or it's set to not log metadata. Or
>> maybe there's a problem when the secondary is trying to fetch the
>> metadata log. Maybe some kind of # of shards mismatch (though not
>> likely).
>> Try to see if the master logs any changes: should use the
>> 'radosgw-admin mdlog list' command.
>>
>> Yehuda
>>
>> > On Thu, Sep 7, 2017 at 2:52 PM Yehuda Sadeh-Weinraub <yeh...@redhat.com>
>> > wrote:
>> >>
>> >> On Thu, Sep 7, 2017 at 7:44 PM, David Turner <drakonst...@gmail.com>
>> >> wrote:
>> >> > Ok, I've been testing, investigating, researching, etc for the last
>> >> > week
>> >> > and
>> >> > I don't have any problems with data syncing.  The clients on one side
>> >> > are
>> >> > creating multipart objects while the multisite sync is creating them
>> >> > as
>> >> > whole objects and one of the datacenters is slower at cleaning up the
>> >> > shadow
>> >> > files.  That's the big discrepancy between object counts in the pools
>> >> > between datacenters.  I created a tool that goes through for each
>> >> > bucket
>> >> > in
>> >> > a realm and does a recursive listing of all objects in it for both
>> >> > datacenters and compares the 2 lists for any differences.  The data
>> >> > is
>> >> > definitely in sync between the 2 datacenters down to the modified
>> >> > time
>> >> > and
>> >> > byte of each file in s3.
>> >> >
>

Re: [ceph-users] RGW Multisite metadata sync init

[Yehuda Sadeh-Weinraub]

On Thu, Sep 7, 2017 at 10:04 PM, David Turner <drakonst...@gmail.com> wrote:
> One realm is called public with a zonegroup called public-zg with a zone for
> each datacenter.  The second realm is called internal with a zonegroup
> called internal-zg with a zone for each datacenter.  they each have their
> own rgw's and load balancers.  The needs of our public facing rgw's and load
> balancers vs internal use ones was different enough that we split them up
> completely.  We also have a local realm that does not use multisite and a
> 4th realm called QA that mimics the public realm as much as possible for
> staging configuration stages for the rgw daemons.  All 4 realms have their
> own buckets, users, etc and that is all working fine.  For all of the
> radosgw-admin commands I am using the proper identifiers to make sure that
> each datacenter and realm are running commands on exactly what I expect them
> to (--rgw-realm=public --rgw-zonegroup=public-zg --rgw-zone=public-dc1
> --source-zone=public-dc2).
>
> The data sync issue was in the internal realm but running a data sync init
> and kickstarting the rgw daemons in each datacenter fixed the data
> discrepancies (I'm thinking it had something to do with a power failure a
> few months back that I just noticed recently).  The metadata sync issue is
> in the public realm.  I have no idea what is causing this to not sync
> properly since running a `metadata sync init` catches it back up to the
> primary zone, but then it doesn't receive any new users created after that.
>

Sounds like an issue with the metadata log in the primary master zone.
Not sure what could go wrong there, but maybe the master zone doesn't
know that it is a master zone, or it's set to not log metadata. Or
maybe there's a problem when the secondary is trying to fetch the
metadata log. Maybe some kind of # of shards mismatch (though not
likely).
Try to see if the master logs any changes: should use the
'radosgw-admin mdlog list' command.

Yehuda

> On Thu, Sep 7, 2017 at 2:52 PM Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> On Thu, Sep 7, 2017 at 7:44 PM, David Turner <drakonst...@gmail.com>
>> wrote:
>> > Ok, I've been testing, investigating, researching, etc for the last week
>> > and
>> > I don't have any problems with data syncing.  The clients on one side
>> > are
>> > creating multipart objects while the multisite sync is creating them as
>> > whole objects and one of the datacenters is slower at cleaning up the
>> > shadow
>> > files.  That's the big discrepancy between object counts in the pools
>> > between datacenters.  I created a tool that goes through for each bucket
>> > in
>> > a realm and does a recursive listing of all objects in it for both
>> > datacenters and compares the 2 lists for any differences.  The data is
>> > definitely in sync between the 2 datacenters down to the modified time
>> > and
>> > byte of each file in s3.
>> >
>> > The metadata is still not syncing for the other realm, though.  If I run
>> > `metadata sync init` then the second datacenter will catch up with all
>> > of
>> > the new users, but until I do that newly created users on the primary
>> > side
>> > don't exist on the secondary side.  `metadata sync status`, `sync
>> > status`,
>> > `metadata sync run` (only left running for 30 minutes before I ctrl+c
>> > it),
>> > etc don't show any problems... but the new users just don't exist on the
>> > secondary side until I run `metadata sync init`.  I created a new bucket
>> > with the new user and the bucket shows up in the second datacenter, but
>> > no
>> > objects because the objects don't have a valid owner.
>> >
>> > Thank you all for the help with the data sync issue.  You pushed me into
>> > good directions.  Does anyone have any insight as to what is preventing
>> > the
>> > metadata from syncing in the other realm?  I have 2 realms being sync
>> > using
>> > multi-site and it's only 1 of them that isn't getting the metadata
>> > across.
>> > As far as I can tell it is configured identically.
>>
>> What do you mean you have two realms? Zones and zonegroups need to
>> exist in the same realm in order for meta and data sync to happen
>> correctly. Maybe I'm misunderstanding.
>>
>> Yehuda
>>
>> >
>> > On Thu, Aug 31, 2017 at 12:46 PM David Turner <drakonst...@gmail.com>
>> > wrote:
>> >>
>> >> All of the messages from sync error list are listed below.  The number
>> >> 

Re: [ceph-users] RGW Multisite metadata sync init

[Yehuda Sadeh-Weinraub]

On Thu, Sep 7, 2017 at 7:44 PM, David Turner  wrote:
> Ok, I've been testing, investigating, researching, etc for the last week and
> I don't have any problems with data syncing.  The clients on one side are
> creating multipart objects while the multisite sync is creating them as
> whole objects and one of the datacenters is slower at cleaning up the shadow
> files.  That's the big discrepancy between object counts in the pools
> between datacenters.  I created a tool that goes through for each bucket in
> a realm and does a recursive listing of all objects in it for both
> datacenters and compares the 2 lists for any differences.  The data is
> definitely in sync between the 2 datacenters down to the modified time and
> byte of each file in s3.
>
> The metadata is still not syncing for the other realm, though.  If I run
> `metadata sync init` then the second datacenter will catch up with all of
> the new users, but until I do that newly created users on the primary side
> don't exist on the secondary side.  `metadata sync status`, `sync status`,
> `metadata sync run` (only left running for 30 minutes before I ctrl+c it),
> etc don't show any problems... but the new users just don't exist on the
> secondary side until I run `metadata sync init`.  I created a new bucket
> with the new user and the bucket shows up in the second datacenter, but no
> objects because the objects don't have a valid owner.
>
> Thank you all for the help with the data sync issue.  You pushed me into
> good directions.  Does anyone have any insight as to what is preventing the
> metadata from syncing in the other realm?  I have 2 realms being sync using
> multi-site and it's only 1 of them that isn't getting the metadata across.
> As far as I can tell it is configured identically.

What do you mean you have two realms? Zones and zonegroups need to
exist in the same realm in order for meta and data sync to happen
correctly. Maybe I'm misunderstanding.

Yehuda

>
> On Thu, Aug 31, 2017 at 12:46 PM David Turner  wrote:
>>
>> All of the messages from sync error list are listed below.  The number on
>> the left is how many times the error message is found.
>>
>>1811 "message": "failed to sync bucket instance:
>> (16) Device or resource busy"
>>   7 "message": "failed to sync bucket instance:
>> (5) Input\/output error"
>>  65 "message": "failed to sync object"
>>
>> On Tue, Aug 29, 2017 at 10:00 AM Orit Wasserman 
>> wrote:
>>>
>>>
>>> Hi David,
>>>
>>> On Mon, Aug 28, 2017 at 8:33 PM, David Turner 
>>> wrote:

 The vast majority of the sync error list is "failed to sync bucket
 instance: (16) Device or resource busy".  I can't find anything on Google
 about this error message in relation to Ceph.  Does anyone have any idea
 what this means? and/or how to fix it?
>>>
>>>
>>> Those are intermediate errors resulting from several radosgw trying to
>>> acquire the same sync log shard lease. It doesn't effect the sync progress.
>>> Are there any other errors?
>>>
>>> Orit


 On Fri, Aug 25, 2017 at 2:48 PM Casey Bodley  wrote:
>
> Hi David,
>
> The 'data sync init' command won't touch any actual object data, no.
> Resetting the data sync status will just cause a zone to restart a full 
> sync
> of the --source-zone's data changes log. This log only lists which
> buckets/shards have changes in them, which causes radosgw to consider them
> for bucket sync. So while the command may silence the warnings about data
> shards being behind, it's unlikely to resolve the issue with missing 
> objects
> in those buckets.
>
> When data sync is behind for an extended period of time, it's usually
> because it's stuck retrying previous bucket sync failures. The 'sync error
> list' may help narrow down where those failures are.
>
> There is also a 'bucket sync init' command to clear the bucket sync
> status. Following that with a 'bucket sync run' should restart a full sync
> on the bucket, pulling in any new objects that are present on the
> source-zone. I'm afraid that those commands haven't seen a lot of polish 
> or
> testing, however.
>
> Casey
>
>
> On 08/24/2017 04:15 PM, David Turner wrote:
>
> Apparently the data shards that are behind go in both directions, but
> only one zone is aware of the problem.  Each cluster has objects in their
> data pool that the other doesn't have.  I'm thinking about initiating a
> `data sync init` on both sides (one at a time) to get them back on the 
> same
> page.  Does anyone know if that command will overwrite any local data that
> the zone has that the other doesn't if you run `data sync init` on it?
>
> On Thu, Aug 24, 2017 at 1:51 PM David Turner 

Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4

[Yehuda Sadeh-Weinraub]

On Wed, Aug 30, 2017 at 5:44 PM, Bryan Banister
 wrote:
> Not sure what’s happening but we started to but a decent load on the RGWs we
> have setup and we were seeing failures with the following kind of
> fingerprint:
>
>
>
> 2017-08-29 17:06:22.072361 7ffdc501a700  1 rgw realm reloader: Frontends
> paused
>

Are you modifying configuration? Could be that something is sending
HUP singal to the radosgw process. We disabled this behavior (process
dynamic reconfig after HUP) in 12.2.0.

Yehuda

> 2017-08-29 17:06:22.072359 7fffacbe9700  1 civetweb: 0x56add000:
> 7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT
> /blah?partNumber=8=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0
> - Minio (linux; amd64) minio-go/3.0.0
>
> 2017-08-29 17:06:22.072438 7fffcb426700  0 ERROR: failed to clone shard,
> completion_mgr.get_next() returned ret=-125
>
> 2017-08-29 17:06:23.689610 7ffdc501a700  1 rgw realm reloader: Store closed
>
> 2017-08-29 17:06:24.117630 7ffdc501a700  1 failed to decode the mdlog
> history: buffer::end_of_buffer
>
> 2017-08-29 17:06:24.117635 7ffdc501a700  1 failed to read mdlog history: (5)
> Input/output error
>
> 2017-08-29 17:06:24.118711 7ffdc501a700  1 rgw realm reloader: Creating new
> store
>
> 2017-08-29 17:06:24.118901 7ffdc501a700  1 mgrc service_daemon_register
> rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4
> (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R)
> Xeon(R) CPU E5-2680 v4 @ 2.40GHz,distro=rhel,distro_description=Red Hat
> Enterprise Linux Server 7.3
> (Maipo),distro_version=7.3,frontend_config#0=civetweb port=80
> num_threads=1024,frontend_type#0=civetweb,hos
>
> tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT
> 2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}
>
> 2017-08-29 17:06:24.118925 7ffdc501a700  1 rgw realm reloader: Finishing
> initialization of new store
>
> 2017-08-29 17:06:24.118927 7ffdc501a700  1 rgw realm reloader:  - REST
> subsystem init
>
> 2017-08-29 17:06:24.118943 7ffdc501a700  1 rgw realm reloader:  - user
> subsystem init
>
> 2017-08-29 17:06:24.118947 7ffdc501a700  1 rgw realm reloader:  - user
> subsystem init
>
> 2017-08-29 17:06:24.118950 7ffdc501a700  1 rgw realm reloader:  - usage
> subsystem init
>
> 2017-08-29 17:06:24.118985 7ffdc501a700  1 rgw realm reloader: Resuming
> frontends with new realm configuration.
>
> 2017-08-29 17:06:24.119018 7fffad3ea700  1 == starting new request
> req=0x7fffad3e4190 =
>
> 2017-08-29 17:06:24.119039 7fffacbe9700  1 == starting new request
> req=0x7fffacbe3190 =
>
> 2017-08-29 17:06:24.120163 7fffacbe9700  1 == req done
> req=0x7fffacbe3190 op status=0 http_status=403 ==
>
> 2017-08-29 17:06:24.120200 7fffad3ea700  1 == req done
> req=0x7fffad3e4190 op status=0 http_status=403 ==
>
>
>
> Any help understanding how to fix this would be greatly appreciated!
>
> -Bryan
>
>
> 
>
> Note: This email is for the confidential use of the named addressee(s) only
> and may contain proprietary, confidential or privileged information. If you
> are not the intended recipient, you are hereby notified that any review,
> dissemination or copying of this email is strictly prohibited, and to please
> notify the sender immediately and destroy this email and any attachments.
> Email transmission cannot be guaranteed to be secure or error-free. The
> Company, therefore, does not make any guarantees as to the completeness or
> accuracy of this email or any attachments. This email is for informational
> purposes only and does not constitute a recommendation, offer, request or
> solicitation of any kind to buy, sell, subscribe, redeem or perform any type
> of transaction of a financial product.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to replicate metadata only on RGW multisite?

[Yehuda Sadeh-Weinraub]

On Fri, Jun 30, 2017 at 4:49 AM, Henrik Korkuc  wrote:
> Hello,
>
> I have RGW multisite setup on Jewel and I would like to turn off data
> replication there so that only metadata (users, created buckets, etc) would
> be synced but not the data.
>
>

FWIW, not in jewel, but in kraken the zone info has two params:
 - sync_from_all
 - sync_from

The first specifies whether zone should sync from all its peers
(within the same zonegroup), and the second one specifies what zones
to sync from (in case sync_from_all is false).

However, as I said, this doesn't exist in jewel. In jewel you can
still just put the zone in a separate zonegroup as data is synced only
within the same zonegroup. Metadata syncs from the master zone to any
other zones.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw versioning S3 compatible?

[Yehuda Sadeh-Weinraub]

On Wed, Jun 28, 2017 at 8:13 AM, Martin Emrich
 wrote:
> Correction: It’s about the Version expiration, not the versioning itself.
>
> We send this rule:
>
>
>
>   Rules: [
>
> {
>
>   Status: 'Enabled',
>
>   Prefix: '',
>
>   NoncurrentVersionExpiration: {
>
> NoncurrentDays: 60
>
>   },
>
>   Expiration: {
>
> ExpiredObjectDeleteMarker: true
>
>   },
>
>   ID: 'expire-60days'
>
> }
>
>   ]
>
>
>
> Should that be supported?
>

Currently rgw bucket lifecycle rules do not versioning.

Yehuda

>
>
> Thanks
>
>
>
> Martin
>
>
>
> Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag von
> Martin Emrich
> Gesendet: Mittwoch, 28. Juni 2017 16:13
> An: ceph-users@lists.ceph.com
> Betreff: [ceph-users] Radosgw versioning S3 compatible?
>
>
>
> Hi!
>
>
>
> Is the Object Gateway S3 API supposed to be compatible with Amazon S3
> regarding versioning?
>
>
>
> Object Versioning is listed as supported in Ceph 12.1, but using the
> standard Node.js aws-sdk module (s3.putBucketVersioning()) results in
> “NotImplemented”.
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Living with huge bucket sizes

[Yehuda Sadeh-Weinraub]

On Fri, Jun 9, 2017 at 2:21 AM, Dan van der Ster  wrote:
> Hi Bryan,
>
> On Fri, Jun 9, 2017 at 1:55 AM, Bryan Stillwell  
> wrote:
>> This has come up quite a few times before, but since I was only working with
>> RBD before I didn't pay too close attention to the conversation.  I'm
>> looking
>> for the best way to handle existing clusters that have buckets with a large
>> number of objects (>20 million) in them.  The cluster I'm doing test on is
>> currently running hammer (0.94.10), so if things got better in jewel I would
>> love to hear about it!
>> ...
>> Has anyone found a good solution for this for existing large buckets?  I
>> know sharding is the solution going forward, but afaik it can't be done
>> on existing buckets yet (although the dynamic resharding work mentioned
>> on today's performance call sounds promising).
>
> I haven't tried it myself, but 0.94.10 should have the (offline)
> resharding feature. From the release notes:
>

Right. We did add automatic dynamic resharding to Luminous, but
offline resharding should be enough.


>> * In RADOS Gateway, it is now possible to reshard an existing bucket's index
>> using an off-line tool.
>>
>> Usage:
>>
>> $ radosgw-admin bucket reshard --bucket= 
>> --num_shards=
>>
>> This will create a new linked bucket instance that points to the newly 
>> created
>> index objects. The old bucket instance still exists and currently it's up to
>> the user to manually remove the old bucket index objects. (Note that bucket
>> resharding currently requires that all IO (especially writes) to the specific
>> bucket is quiesced.)

Once resharding is done, use the radosgw-admin bi purge command to
remove the old bucket indexes.

Yehuda

>
> -- Dan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW lifecycle not expiring objects

[Yehuda Sadeh-Weinraub]

Have you opened a ceph tracker issue, so that we don't lose track of
the problem?

Thanks,
Yehuda

On Fri, Jun 2, 2017 at 3:05 PM,   wrote:
> Hi Graham.
>
> We are on Kraken and have the same problem with "lifecycle". Various (other) 
> tools like s3cmd or CyberDuck do show the applied "expiration" settings, but 
> objects seem never to be purged.
>
> If you should have new findings, hints,... PLEASE share/let me know.
>
> Thanks a lot!
> Anton
>
>
> Gesendet: Freitag, 19. Mai 2017 um 22:44 Uhr
> Von: "Graham Allan" 
> An: ceph-users@lists.ceph.com
> Betreff: [ceph-users] RGW lifecycle not expiring objects
> I've been having a hard time getting the s3 object lifecycle to do
> anything here. I was able to set a lifecycle on a test bucket. As others
> also seem to have found, I do get an EACCES error on setting the
> lifecycle, but it does however get stored:
>
>> % aws --endpoint-url https://xxx.xxx.xxx.xxx s3api 
>> get-bucket-lifecycle-configuration --bucket=testgta
>> {
>> "Rules": [
>> {
>> "Status": "Enabled",
>> "Prefix": "",
>> "Expiration": {
>> "Days": 3
>> },
>> "ID": "test"
>> }
>> ]
>> }
>
> but many days later I have yet to see any object actually get expired.
> There are some hints in the rgw log that the expiry thread does run
> periodically:
>
>> 2017-05-19 03:49:03.281347 7f74f1134700 2 
>> RGWDataChangesLog::ChangesRenewThread: start
>> 2017-05-19 03:49:16.356022 7f74ef931700 2 object expiration: start
>> 2017-05-19 03:49:16.356036 7f74ef931700 20 proceeding shard = 
>> obj_delete_at_hint.00
>> 2017-05-19 03:49:16.359785 7f74ef931700 20 proceeding shard = 
>> obj_delete_at_hint.01
>> 2017-05-19 03:49:16.364667 7f74ef931700 20 proceeding shard = 
>> obj_delete_at_hint.02
>> 2017-05-19 03:49:16.369636 7f74ef931700 20 proceeding shard = 
>> obj_delete_at_hint.03
> ...
>> 2017-05-19 03:49:16.803270 7f74ef931700 20 proceeding shard = 
>> obj_delete_at_hint.000126
>> 2017-05-19 03:49:16.806423 7f74ef931700 2 object expiration: stop
>
> "radosgw-admin lc process" gives me no output unless I enable debug, then:
>
>> ]# radosgw-admin lc process
>> 2017-05-19 15:28:46.383049 7fedb9ffb700 2 
>> RGWDataChangesLog::ChangesRenewThread: start
>> 2017-05-19 15:28:46.421806 7feddc240c80 10 Cannot find current period zone 
>> using local zone
>> 2017-05-19 15:28:46.453431 7feddc240c80 2 all 8 watchers are set, enabling 
>> cache
>> 2017-05-19 15:28:46.614991 7feddc240c80 2 removed watcher, disabling cache
>
> "radosgw-admin lc list" seems to return "empty" output:
>
>> # radosgw-admin lc list
>> []
>
> Is there anything obvious that I might be missing?
>
> Graham
> --
> Graham Allan
> Minnesota Supercomputing Institute - g...@umn.edu
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW: removal of support for fastcgi

[Yehuda Sadeh-Weinraub]

On Mon, May 15, 2017 at 8:35 AM, Ken Dreyer <kdre...@redhat.com> wrote:
> On Fri, May 5, 2017 at 1:51 PM, Yehuda Sadeh-Weinraub <yeh...@redhat.com> 
> wrote:
>>
>> TL;DR: Does anyone care if we remove support for fastcgi in rgw?
>
> Please remove it as soon as possible. The old libfcgi project's code
> is a security liability. When upstream died, there was a severe lack
> of coordination around distributing patches to fix CVE-2012-6687. I
> expect a similar level of chaos if another CVE surfaces in this
> library. There are also unanswered questions about libfcgi's continued
> use of poll vs select, see
> https://bugs.launchpad.net/ubuntu/+source/libfcgi/+bug/933417/comments/5
> .

Didn't see any compelling reason to keep it. It seems to me that the
SSL issue that Roger was pointing at could be solved either through a
different apache config, or something that we could fix (and might
have already fixed) in civetweb. As a first go I think we should still
keep the code around but not build it for Luminous.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RGW: removal of support for fastcgi

[Yehuda Sadeh-Weinraub]

RGW has supported since forever. Originally it was the only supported
frontend, and nowadays it is the least preferred one.

Rgw was first developed over fastcgi + lighttpd, but there were some
issues with this setup, so we switched to fastcgi + apache as our main
supported configuration. This was also sub-optimal, as there wasn't a
good, supported, and mainained fastcgi module for apache that we could
find. At the time there were two modules: mod_fcgid, and mod_fastcgi.
The former had a major flaw, which it buffered all PUTs before sending
them to the backend (rgw). It also didn't really support 100-continue.
The latter was mostly unmaintained, and didn't quite support
100-continue either. We ended up maintaining a fork of mod_fastcgi for
quite a while, which was a pain. Later came mod-proxy-fcgi, which also
didn't fully support 100-continue (iirc), but it was maintained, and
was good enough to use out of the box, so we settled on it. At that
time we already had civetweb as a frontend, so we didn't really worry
about it.
Now, I'd like to know whether anyone actually uses and needs fastcgi.
I get an occasional request to remove it altogether, and I'd like to
have some more info before we go and do it. The requests for removal
usually cite security reasons, but I can also add 'we don't really
want to maintain it anymore'.
A valid replacement for fastcgi could be using civetweb directly, or
using mod-proxy (in apache, I'd expect similar solution in other
webservers) with civetweb as the rgw frontend.

TL;DR: Does anyone care if we remove support for fastcgi in rgw?

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw leaking objects

[Yehuda Sadeh-Weinraub]

On Mon, Apr 3, 2017 at 1:32 AM, Luis Periquito  wrote:
>> Right. The tool isn't removing objects (yet), because we wanted to
>> have more confidence in the tool before having it automatically
>> deleting all the found objects. The process currently is to manually
>> move these objects to a different backup pool (via rados cp, rados
>> rm), then when you're confident that no needed data was lost in the
>> process remove the backup pool. In the future we'll automate that.
>
> My problem exactly. I don't have enough confidence in myself to just
> delete a bunch of random objects... Any idea as to when will be
> available such tool?

Why random? The objects are the ones that the orphan tool pointed at.
And the idea is to move these objects to a safe place before removal,
so that even if the wrong objects are removed, they can be recovered.
There is no current ETA for the tool, but the tool will probably have
the same two steps as reflected here: 1. backup, 2. remove backup.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] S3 Multi-part upload broken with newer AWS Java SDK and Kraken RGW

[Yehuda Sadeh-Weinraub]

This sounds like this bug:
http://tracker.ceph.com/issues/17076

Will be fixed in 10.2.6. It's triggered by aws4 auth, so a workaround
would be to use aws2 instead.

Yehuda


On Wed, Mar 1, 2017 at 10:46 AM, John Nielsen  wrote:
> Hi all-
>
> We use Amazon S3 quite a bit at $WORK but are evaluating Ceph+radosgw as an 
> alternative for some things. We have an "S3 smoke test" written using the AWS 
> Java SDK that we use to validate a number of operations. On my Kraken 
> cluster, multi-part uploads work fine for s3cmd. Our smoke test also passes 
> fine using version 1.9.27 of the AWS SDK. However in SDK 1.11.69 the 
> multi-part upload fails. The initial POST (to reserve the object name and 
> start the upload) succeeds, but the first PUT fails with a 403 error.
>
> So, does anyone know offhand what might be going on here? If not, how can I 
> get more details about the 403 error and what is causing it?
>
> The cluster was installed with Jewel and recently updated to Kraken. Using 
> the built-in civetweb server.
>
> Here is the log output for three multi-part uploads. The first two are s3cmd 
> and the older SDK, respectively. The last is the failing one with the newer 
> SDK.
>
> S3cmd, Succeeds.
> 2017-03-01 17:33:16.845613 7f80b06de700  1 == starting new request 
> req=0x7f80b06d8340 =
> 2017-03-01 17:33:16.856522 7f80b06de700  1 == req done req=0x7f80b06d8340 
> op status=0 http_status=200 ==
> 2017-03-01 17:33:16.856628 7f80b06de700  1 civetweb: 0x7f81131fd000: 
> 10.251.50.7 - - [01/Mar/2017:17:33:16 +] "POST 
> /testdomobucket10x3x104x64250438/multipartStreamTest?uploads HTTP/1.1" 1 0 - -
> 2017-03-01 17:33:16.953967 7f80b06de700  1 == starting new request 
> req=0x7f80b06d8340 =
> 2017-03-01 17:33:24.094134 7f80b06de700  1 == req done req=0x7f80b06d8340 
> op status=0 http_status=200 ==
> 2017-03-01 17:33:24.094211 7f80b06de700  1 civetweb: 0x7f81131fd000: 
> 10.251.50.7 - - [01/Mar/2017:17:33:16 +] "PUT 
> /testdomobucket10x3x104x64250438/multipartStreamTest?partNumber=1=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB
>  HTTP/1.1" 1 0 - -
> 2017-03-01 17:33:24.193747 7f80b06de700  1 == starting new request 
> req=0x7f80b06d8340 =
> 2017-03-01 17:33:30.002050 7f80b06de700  1 == req done req=0x7f80b06d8340 
> op status=0 http_status=200 ==
> 2017-03-01 17:33:30.002124 7f80b06de700  1 civetweb: 0x7f81131fd000: 
> 10.251.50.7 - - [01/Mar/2017:17:33:16 +] "PUT 
> /testdomobucket10x3x104x64250438/multipartStreamTest?partNumber=2=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB
>  HTTP/1.1" 1 0 - -
> 2017-03-01 17:33:30.085033 7f80b06de700  1 == starting new request 
> req=0x7f80b06d8340 =
> 2017-03-01 17:33:30.104944 7f80b06de700  1 == req done req=0x7f80b06d8340 
> op status=0 http_status=200 ==
> 2017-03-01 17:33:30.105007 7f80b06de700  1 civetweb: 0x7f81131fd000: 
> 10.251.50.7 - - [01/Mar/2017:17:33:16 +] "POST 
> /testdomobucket10x3x104x64250438/multipartStreamTest?uploadId=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB
>  HTTP/1.1" 1 0 - -
>
> AWS SDK (1.9.27). Succeeds.
> 2017-03-01 17:54:50.720093 7f80c0eff700  1 == starting new request 
> req=0x7f80c0ef9340 =
> 2017-03-01 17:54:50.733109 7f80c0eff700  1 == req done req=0x7f80c0ef9340 
> op status=0 http_status=200 ==
> 2017-03-01 17:54:50.733188 7f80c0eff700  1 civetweb: 0x7f811314c000: 
> 10.251.50.7 - - [01/Mar/2017:17:54:42 +] "POST 
> /testdomobucket10x3x104x6443285/multipartStreamTest?uploads HTTP/1.1" 1 0 - 
> aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 
> Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71
> 2017-03-01 17:54:50.831618 7f80c0eff700  1 == starting new request 
> req=0x7f80c0ef9340 =
> 2017-03-01 17:54:58.057011 7f80c0eff700  1 == req done req=0x7f80c0ef9340 
> op status=0 http_status=200 ==
> 2017-03-01 17:54:58.057082 7f80c0eff700  1 civetweb: 0x7f811314c000: 
> 10.251.50.7 - - [01/Mar/2017:17:54:42 +] "PUT 
> /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo=1
>  HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 
> Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71
> 2017-03-01 17:54:58.143235 7f80c0eff700  1 == starting new request 
> req=0x7f80c0ef9340 =
> 2017-03-01 17:54:58.328351 7f80c0eff700  1 == req done req=0x7f80c0ef9340 
> op status=0 http_status=200 ==
> 2017-03-01 17:54:58.328437 7f80c0eff700  1 civetweb: 0x7f811314c000: 
> 10.251.50.7 - - [01/Mar/2017:17:54:42 +] "PUT 
> /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo=2
>  HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 
> Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71
> 2017-03-01 17:54:58.415890 7f80c0eff700  1 == starting new request 
> req=0x7f80c0ef9340 =
> 2017-03-01 17:54:58.438199 7f80c0eff700  1 == req done req=0x7f80c0ef9340 
> op status=0 http_status=200 ==
> 2017-03-01 17:54:58.438253 7f80c0eff700  1 

Re: [ceph-users] rgw leaking data, orphan search loop

[Yehuda Sadeh-Weinraub]

On Tue, Feb 28, 2017 at 5:44 AM, George Mihaiescu <lmihaie...@gmail.com> wrote:
> Hi Yehuda,
>
> I've ran the "radosgw-admin orphans find" command again, but captured its
> output this time.
>
> There are both "shadow" files and "multipart" files detected as leaked.
>
> leaked:
> default.34461213.1__multipart_data/d2a14aeb-a384-51b1-8704-fe76a9a6f5f5.-j0vqDrC0wr44bii2ytrtpcrlnspSyE.44
> leaked:
> default.34461213.1__multipart_data/dda8a18a-1d99-50b7-a397-b876811bdf94.Mpjqa2RwKirM9Ae_HTsttGiJEHAvdxc.113
> leaked:
> default.34461213.1__multipart_data/dda8a18a-1d99-50b7-a397-b876811bdf94.bRtExqhbdw_J0gcT-xADwUBQJFOjKmG.111
> leaked:
> default.34461213.1__multipart_data/f080cdc1-0826-5ac9-a9f7-21700ceeebf3.BqR5aBMpJDmO1U5xKSrEe3EvPmEHNq8.96
> leaked:
> default.34461213.1__multipart_data/f793a1ec-5c7d-5b59-845a-9d280325bb25.LcCH4ia_LwWV4MyVzwhTv_PAxrpSPpM.52
> leaked:
> default.34461213.1__multipart_data/f793a1ec-5c7d-5b59-845a-9d280325bb25.gbrMfo0bWww2nEe2x4LL146BwtLMkA6.37
> leaked:
> default.34461213.1__multipart_data/f793a1ec-5c7d-5b59-845a-9d280325bb25.rIZlATigZEwP6FVW66m-YhcmgIiJihM.48
> leaked:
> default.34461213.1__shadow_data/0181bbd5-4202-57a0-a1f3-07007d043660.2~gy8NGkx7YmMzHwv8_ordh7u_TNk7_4c.1_1
> leaked:
> default.34461213.1__shadow_data/0181bbd5-4202-57a0-a1f3-07007d043660.2~gy8NGkx7YmMzHwv8_ordh7u_TNk7_4c.1_2
> leaked:
> default.34461213.1__shadow_data/0181bbd5-4202-57a0-a1f3-07007d043660.2~gy8NGkx7YmMzHwv8_ordh7u_TNk7_4c.1_3
> leaked:
> default.34461213.1__shadow_data/0181bbd5-4202-57a0-a1f3-07007d043660.2~gy8NGkx7YmMzHwv8_ordh7u_TNk7_4c.1_4
> leaked:
> default.34461213.1__shadow_data/0181bbd5-4202-57a0-a1f3-07007d043660.2~gy8NGkx7YmMzHwv8_ordh7u_TNk7_4c.1_5
> leaked:
> default.34461213.1__shadow_data/0181bbd5-4202-57a0-a1f3-07007d043660.2~gy8NGkx7YmMzHwv8_ordh7u_TNk7_4c.1_6
> leaked:
> default.34461213.1__shadow_data/02aca392-6d6b-536c-ae17-fdffe164e05a.2~lXP-3WDlbF5MSPYuE7JHLNK1z1hr1Y4.100_1
> leaked:
> default.34461213.1__shadow_data/02aca392-6d6b-536c-ae17-fdffe164e05a.2~lXP-3WDlbF5MSPYuE7JHLNK1z1hr1Y4.100_10
> leaked:
> default.34461213.1__shadow_data/02aca392-6d6b-536c-ae17-fdffe164e05a.2~lXP-3WDlbF5MSPYuE7JHLNK1z1hr1Y4.100_11
>
> I deleted both the multipart and shadow leaked files for one of the S3
> objects, and then the object couldn't be retrieved anymore.
>
> I deleted just the shadow leaked files for another S3 object, and then that
> object couldn't be retrieved anymore after either.
>
> I think the "radosgw-admin orphans find" command again doesn't work as
> expected, is there anything else I can do?
>

The orphans find command will point at tail objects that aren't
referenced by any bucket index entry. In this case you have rgw object
that doesn't appear on the bucket index, so it is natural that its
tail (composed of both multipart and shadow entries) is marked as
leaked.

Yehuda

> Thank you,
> George
>
>
>
> On Fri, Feb 24, 2017 at 1:22 PM, Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> oid is object id. The orphan find command generates a list of objects
>> that needs to be removed at the end of the run (if finishes
>> successfully). If you didn't catch that, you should be able to still
>> run the same scan (using the same scan id) and retrieve that info
>> again.
>>
>> Yehuda
>>
>> On Fri, Feb 24, 2017 at 9:48 AM, George Mihaiescu <lmihaie...@gmail.com>
>> wrote:
>> > Hi Yehuda,
>> >
>> > Thank you for the quick reply.
>> >
>> > What is the  you're referring to that I should backup and then
>> > delete?
>> > I extracted the files from the ".log" pool where the "orphan find" tool
>> > stored the results, but they are zero bytes files.
>> >
>> >
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.rados.52
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.rados.58
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 obj_delete_at_hint.000122
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 obj_delete_at_hint.57
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.bck1.rados.53
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.buckets.20
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.buckets.25
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.bck1.rados.0
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.rados.2
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.linked.19
>> > -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.rados.38
>> > -rw-r--r-- 1 root root 0 Feb 

Re: [ceph-users] rgw leaking data, orphan search loop

[Yehuda Sadeh-Weinraub]

oid is object id. The orphan find command generates a list of objects
that needs to be removed at the end of the run (if finishes
successfully). If you didn't catch that, you should be able to still
run the same scan (using the same scan id) and retrieve that info
again.

Yehuda

On Fri, Feb 24, 2017 at 9:48 AM, George Mihaiescu <lmihaie...@gmail.com> wrote:
> Hi Yehuda,
>
> Thank you for the quick reply.
>
> What is the  you're referring to that I should backup and then delete?
> I extracted the files from the ".log" pool where the "orphan find" tool
> stored the results, but they are zero bytes files.
>
>
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.rados.52
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.rados.58
> -rw-r--r-- 1 root root 0 Feb 24 12:45 obj_delete_at_hint.000122
> -rw-r--r-- 1 root root 0 Feb 24 12:45 obj_delete_at_hint.57
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.bck1.rados.53
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.buckets.20
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.buckets.25
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.bck1.rados.0
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.rados.2
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.linked.19
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.rados.38
> -rw-r--r-- 1 root root 0 Feb 24 12:45 obj_delete_at_hint.18
> -rw-r--r-- 1 root root 0 Feb 24 12:45 obj_delete_at_hint.92
> -rw-r--r-- 1 root root 0 Feb 24 12:45 obj_delete_at_hint.000108
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.bck1.rados.13
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.linked.20
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.rados.18
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.bck1.rados.11
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.rados.50
> -rw-r--r-- 1 root root 0 Feb 24 12:45 orphan.scan.orphans.buckets.33
>
>
> George
>
>
>
> On Fri, Feb 24, 2017 at 12:12 PM, Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> Hi,
>>
>> we wanted to have more confidence in the orphans search tool before
>> providing a functionality that actually remove the objects. One thing
>> that you can do is create a new pool, copy these objects to the new
>> pool (as a backup, rados -p  --target-pool=
>> cp  ), and remove these objects (rados -p  rm ).
>> Then when you're confident enough that this didn't break existing
>> objects, you can remove the backup pool.
>>
>> Yehuda
>>
>> On Fri, Feb 24, 2017 at 8:23 AM, George Mihaiescu <lmihaie...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I updated http://tracker.ceph.com/issues/18331 with my own issue, and I
>> > am
>> > hoping Orit or Yehuda could give their opinion on what to do next.
>> > What was the purpose of the "orphan find" tool and how to actually clean
>> > up
>> > these files?
>> >
>> > Thank you,
>> > George
>> >
>> >
>> > On Fri, Jan 13, 2017 at 2:22 PM, Wido den Hollander <w...@42on.com>
>> > wrote:
>> >>
>> >>
>> >> > Op 24 december 2016 om 13:47 schreef Wido den Hollander
>> >> > <w...@42on.com>:
>> >> >
>> >> >
>> >> >
>> >> > > Op 23 december 2016 om 16:05 schreef Wido den Hollander
>> >> > > <w...@42on.com>:
>> >> > >
>> >> > >
>> >> > >
>> >> > > > Op 22 december 2016 om 19:00 schreef Orit Wasserman
>> >> > > > <owass...@redhat.com>:
>> >> > > >
>> >> > > >
>> >> > > > HI Maruis,
>> >> > > >
>> >> > > > On Thu, Dec 22, 2016 at 12:00 PM, Marius Vaitiekunas
>> >> > > > <mariusvaitieku...@gmail.com> wrote:
>> >> > > > > On Thu, Dec 22, 2016 at 11:58 AM, Marius Vaitiekunas
>> >> > > > > <mariusvaitieku...@gmail.com> wrote:
>> >> > > > >>
>> >> > > > >> Hi,
>> >> > > > >>
>> >> > > > >> 1) I've written before into mailing list, but one more time.
>> >> > > > >> We
>> >> > > > >> have big
>> >> > > > >> issues recently with rgw on jewel. because of leaked data -
>> >> > > > >> the
>&

Re: [ceph-users] rgw leaking data, orphan search loop

[Yehuda Sadeh-Weinraub]

Hi,

we wanted to have more confidence in the orphans search tool before
providing a functionality that actually remove the objects. One thing
that you can do is create a new pool, copy these objects to the new
pool (as a backup, rados -p  --target-pool=
cp  ), and remove these objects (rados -p  rm ).
Then when you're confident enough that this didn't break existing
objects, you can remove the backup pool.

Yehuda

On Fri, Feb 24, 2017 at 8:23 AM, George Mihaiescu  wrote:
> Hi,
>
> I updated http://tracker.ceph.com/issues/18331 with my own issue, and I am
> hoping Orit or Yehuda could give their opinion on what to do next.
> What was the purpose of the "orphan find" tool and how to actually clean up
> these files?
>
> Thank you,
> George
>
>
> On Fri, Jan 13, 2017 at 2:22 PM, Wido den Hollander  wrote:
>>
>>
>> > Op 24 december 2016 om 13:47 schreef Wido den Hollander :
>> >
>> >
>> >
>> > > Op 23 december 2016 om 16:05 schreef Wido den Hollander
>> > > :
>> > >
>> > >
>> > >
>> > > > Op 22 december 2016 om 19:00 schreef Orit Wasserman
>> > > > :
>> > > >
>> > > >
>> > > > HI Maruis,
>> > > >
>> > > > On Thu, Dec 22, 2016 at 12:00 PM, Marius Vaitiekunas
>> > > >  wrote:
>> > > > > On Thu, Dec 22, 2016 at 11:58 AM, Marius Vaitiekunas
>> > > > >  wrote:
>> > > > >>
>> > > > >> Hi,
>> > > > >>
>> > > > >> 1) I've written before into mailing list, but one more time. We
>> > > > >> have big
>> > > > >> issues recently with rgw on jewel. because of leaked data - the
>> > > > >> rate is
>> > > > >> about 50GB/hour.
>> > > > >>
>> > > > >> We've hitted these bugs:
>> > > > >> rgw: fix put_acls for objects starting and ending with underscore
>> > > > >> (issue#17625, pr#11669, Orit Wasserman)
>> > > > >>
>> > > > >> Upgraded to jewel 10.2.5 - no luck.
>> > > > >>
>> > > > >> Also we've hitted this one:
>> > > > >> rgw: RGW loses realm/period/zonegroup/zone data: period
>> > > > >> overwritten if
>> > > > >> somewhere in the cluster is still running Hammer (issue#17371,
>> > > > >> pr#11519,
>> > > > >> Orit Wasserman)
>> > > > >>
>> > > > >> Fixed zonemaps - also no luck.
>> > > > >>
>> > > > >> We do not use multisite - only default realm, zonegroup, zone.
>> > > > >>
>> > > > >> We have no more ideas, how these data leak could happen. gc is
>> > > > >> working -
>> > > > >> we can see it in rgw logs.
>> > > > >>
>> > > > >> Maybe, someone could give any hint about this? Where should we
>> > > > >> look?
>> > > > >>
>> > > > >>
>> > > > >> 2) Another story is about removing all the leaked/orphan objects.
>> > > > >> radosgw-admin orphans find enters the loop state on stage when it
>> > > > >> starts
>> > > > >> linking objects.
>> > > > >>
>> > > > >> We've tried to change the number of shards to 16, 64 (default),
>> > > > >> 512. At
>> > > > >> the moment it's running with shards number 1.
>> > > > >>
>> > > > >> Again, any ideas how to make orphan search happen?
>> > > > >>
>> > > > >>
>> > > > >> I could provide any logs, configs, etc. if someone is ready to
>> > > > >> help on
>> > > > >> this case.
>> > > > >>
>> > > > >>
>> > > >
>> > > > How many buckets do you have ? how many object in each?
>> > > > Can you provide the output of rados ls -p .rgw.buckets ?
>> > >
>> > > Marius asked me to look into this for him, so I did.
>> > >
>> > > What I found is that at *least* three buckets have way more RADOS
>> > > objects then they should.
>> > >
>> > > The .rgw.buckets pool has 35.651.590 objects totaling 76880G.
>> > >
>> > > I listed all objects in the .rgw.buckets pool and summed them per
>> > > bucket, the top 5:
>> > >
>> > >  783844 default.25918901.102486
>> > >  876013 default.25918901.3
>> > > 3325825 default.24201682.7
>> > > 6324217 default.84795862.29891
>> > > 7805208 default.25933378.233873
>> > >
>> > > So I started to rados_stat() (using Python) all the objects in the
>> > > last three pools. While these stat() calls are still running. I statted
>> > > about 30% of the objects and their total size is already 17511GB/17TB.
>> > >
>> > > size_kb_actual summed up for bucket default.24201682.7,
>> > > default.84795862.29891 and default.25933378.233873 sums up to 12TB.
>> > >
>> > > So I'm currently at 30% of statting the objects and I'm already 5TB
>> > > over the total size of these buckets.
>> > >
>> >
>> > The stat calls have finished. The grant total is 65TB.
>> >
>> > So while the buckets should consume only 12TB they seems to occupy 65TB
>> > of storage.
>> >
>> > > What I noticed is that it's mainly *shadow* objects which are all 4MB
>> > > in size.
>> > >
>> > > I know that 'radosgw-admin orphans find --pool=.rgw.buckets
>> > > --job-id=xyz' should also do this for me, but as mentioned, this keeps
>> > > looping and hangs.
>> > >
>> >
>> > I started this tool about 20 hours ago:
>> >
>> > # radosgw-admin orphans find --pool=.rgw.buckets 

Re: [ceph-users] rgw multisite resync only one bucket

[Yehuda Sadeh-Weinraub]

On Fri, Feb 24, 2017 at 3:59 AM, Marius Vaitiekunas
<mariusvaitieku...@gmail.com> wrote:
>
>
> On Wed, Feb 22, 2017 at 8:33 PM, Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> On Wed, Feb 22, 2017 at 6:19 AM, Marius Vaitiekunas
>> <mariusvaitieku...@gmail.com> wrote:
>> > Hi Cephers,
>> >
>> > We are testing rgw multisite solution between to DC. We have one
>> > zonegroup
>> > and to zones. At the moment all writes/deletes are done only to primary
>> > zone.
>> >
>> > Sometimes not all the objects are replicated.. We've written prometheus
>> > exporter to check replication status. It gives us each bucket object
>> > count
>> > from user perspective, because we have millions of objects and hundreds
>> > of
>> > buckets. We just want to be sure, that everything is replicated without
>> > using ceph internals like rgw admin api for now.
>> >
>> > Is it possible to initiate full resync of only one rgw bucket from
>> > master
>> > zone? What are the options about resync when things go wrong and
>> > replication
>> > misses some objects?
>> >
>> > We run latest jewel 10.2.5.
>>
>>
>> There's the 'radosgw-admin bucket sync init' command that you can run
>> on the specific bucket on the target zone. This will reinitialize the
>> sync state, so that when it starts syncing it will go through the
>> whole full sync process. Note that it shouldn't actually copy data
>> that already exists on the target. Also, in order to actually start
>> the sync, you'll need to have some change that would trigger the sync
>> on that bucket, e.g., create a new object there.
>>
>> Yehuda
>>
>
> Hi,
>
> I've tried to resync a bucket, but it didn't manage to resync a missing
> object. If I try to copy missing object by hand into secondary zone, i get
> asked to overwrite existing object.. It looks like the object is replicated,
> but is not in a bucket index. I've tried to check bucket index with --fix
> and --check-objects flags, but nothing changes. What else should i try?
>

That's weird. Do you see anything when you run 'radosgw-admin bi list
--bucket='?

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw multisite resync only one bucket

[Yehuda Sadeh-Weinraub]

On Wed, Feb 22, 2017 at 11:41 AM, Marius Vaitiekunas
<mariusvaitieku...@gmail.com> wrote:
>
>
> On Wed, Feb 22, 2017 at 8:33 PM, Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> On Wed, Feb 22, 2017 at 6:19 AM, Marius Vaitiekunas
>> <mariusvaitieku...@gmail.com> wrote:
>> > Hi Cephers,
>> >
>> > We are testing rgw multisite solution between to DC. We have one
>> > zonegroup
>> > and to zones. At the moment all writes/deletes are done only to primary
>> > zone.
>> >
>> > Sometimes not all the objects are replicated.. We've written prometheus
>> > exporter to check replication status. It gives us each bucket object
>> > count
>> > from user perspective, because we have millions of objects and hundreds
>> > of
>> > buckets. We just want to be sure, that everything is replicated without
>> > using ceph internals like rgw admin api for now.
>> >
>> > Is it possible to initiate full resync of only one rgw bucket from
>> > master
>> > zone? What are the options about resync when things go wrong and
>> > replication
>> > misses some objects?
>> >
>> > We run latest jewel 10.2.5.
>>
>>
>> There's the 'radosgw-admin bucket sync init' command that you can run
>> on the specific bucket on the target zone. This will reinitialize the
>> sync state, so that when it starts syncing it will go through the
>> whole full sync process. Note that it shouldn't actually copy data
>> that already exists on the target. Also, in order to actually start
>> the sync, you'll need to have some change that would trigger the sync
>> on that bucket, e.g., create a new object there.
>>
>> Yehuda
>>
>
>
> Great! I guess 'radosgw-admin data sync init' initiates whole zone data
> sync,  'radosgw-admin metadata sync init' metadata sync on some trigger?

Yes

> And when should we use 'radosgw-admin data/metadata sync run' ?
>

Don't really need to run it, it's for doing the same sync work that
the radosgw process does. Can be used maybe if radosgw process itself
is down but still want to have metadata sync. This was a useful tool
when developing the feature, but not too useful at this point.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw multisite resync only one bucket

[Yehuda Sadeh-Weinraub]

On Wed, Feb 22, 2017 at 6:19 AM, Marius Vaitiekunas
 wrote:
> Hi Cephers,
>
> We are testing rgw multisite solution between to DC. We have one zonegroup
> and to zones. At the moment all writes/deletes are done only to primary
> zone.
>
> Sometimes not all the objects are replicated.. We've written prometheus
> exporter to check replication status. It gives us each bucket object count
> from user perspective, because we have millions of objects and hundreds of
> buckets. We just want to be sure, that everything is replicated without
> using ceph internals like rgw admin api for now.
>
> Is it possible to initiate full resync of only one rgw bucket from master
> zone? What are the options about resync when things go wrong and replication
> misses some objects?
>
> We run latest jewel 10.2.5.


There's the 'radosgw-admin bucket sync init' command that you can run
on the specific bucket on the target zone. This will reinitialize the
sync state, so that when it starts syncing it will go through the
whole full sync process. Note that it shouldn't actually copy data
that already exists on the target. Also, in order to actually start
the sync, you'll need to have some change that would trigger the sync
on that bucket, e.g., create a new object there.

Yehuda

>
>
> --
> Marius Vaitiekūnas
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw swift api long term support

[Yehuda Sadeh-Weinraub]

On Tue, Jan 10, 2017 at 1:35 AM, Marius Vaitiekunas
 wrote:
> Hi,
>
> I would like to ask ceph developers if there any chance that swift api
> support for rgw is going to be dropped in the future (like in 5 years).
>
> Why am I asking? :)
>
> We were happy openstack glance users on ceph s3 api until openstack decided
> to drop glance s3 support.. So, we need to switch our image backend. Swift
> api on ceph looks quite good solution.
>

I'm not aware of any plans (current or future) to drop swift api.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw: how to prevent rgw user from creating a new bucket?

[Yehuda Sadeh-Weinraub]

On Fri, Dec 2, 2016 at 3:18 AM, Yang Joseph  wrote:
> Hello,
>
> I would like only to allow the user to read the object in a already existed
> bucket, and not allow users
> to create new bucket. It supposed to execute the following command:
>
> $ radosgw-admin metadata put user:test3 < ...
>   ...
> "caps": [
> {
> "type": "buckets",
> "perm": "read"
> }
>
> But why user test3 can still create new bucket after I have set its caps to
> "buckets=read"?
>


Because this cap is unrelated. iirc starting at jewel you can do:

$ radosgw-admin user modify --uid=test3 --max-buckets=-1

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] "Lost" buckets on radosgw

[Yehuda Sadeh-Weinraub]

On Mon, Nov 21, 2016 at 3:33 PM, Graham Allan <g...@umn.edu> wrote:
>
>
> On 11/21/2016 05:23 PM, Yehuda Sadeh-Weinraub wrote:
>>
>>
>> Seems like bucket was sharded, but for some reason the bucket instance
>> info does not specify that. I don't know why that would happen, but
>> maybe running mixed versions could be the culprit. You can try
>> modifying the bucket instance info for this bucket, change the num
>> shards param to 32.
>>
>> Yehuda
>
>
> Yes! That seems to do it...
>
> And of course in retrospect I can see num_shards = 32 for my working bucket
> but 0 for this one.
>
> so I did:
>>
>> rados get --pool .rgw.buckets.ec42
>> default.712449.19__shadow_tcga_test/rnaseq/Aligned.out.sam.2~10nUehK5BnyXdhhiOqTL2JdpLfDCd0k.11_76
>> lemming
>
>
> edit "num_shards": 32
>
>> # radosgw-admin metadata put bucket.instance:tcga:default.712449.19 <
>> tcga.json
>
>
> and this bucket is now visible again! Thanks so much!
>
> I wonder how this happened. It looks like this affects ~25/680 buckets.
>
>

Could be that this is a secondary issue, maybe trying to fix the
buckets because of the original issue caused this to happen
(radosgw-admin bucket check maybe broke it?).

Yehuda

> Graham
> --
> Graham Allan
> Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] "Lost" buckets on radosgw

[Yehuda Sadeh-Weinraub]

On Mon, Nov 21, 2016 at 3:14 PM, Graham Allan <g...@umn.edu> wrote:
>
>
> On 11/21/2016 04:44 PM, Yehuda Sadeh-Weinraub wrote:
>>
>> On Mon, Nov 21, 2016 at 2:42 PM, Graham Allan <g...@umn.edu> wrote:
>>>
>>> Following up to this (same problem, looking at it with Jeff)...
>>>
>>> There was definite confusion with the zone/zonegroup/realm/period changes
>>> during the hammer->jewel upgrade. It's possible that our placement
>>> settings
>>> were misplaced at this time.
>>>
>>> However what I find puzzling is that different buckets from the same pool
>>> seem affected - if this were placement related, I'd rather expect all
>>> buckets from one pool to be affected, those in another not. Am I
>>> interpreting this wrongly?
>>>
>>> For example here is one bucket which remains accessible:
>>>
>>>> # radosgw-admin metadata get bucket.instance:gta:default.691974.1
>>>> {
>>>> "key": "bucket.instance:gta:default.691974.1",
>>>> "ver": {
>>>> "tag": "_3Z9nfFjZn97aV2YJ4nFhVuk",
>>>> "ver": 85
>>>> },
>>>> "mtime": "2016-11-11 16:48:02.950760Z",
>>>> "data": {
>>>> "bucket_info": {
>>>> "bucket": {
>>>> "name": "gta",
>>>> "pool": ".rgw.buckets.ec42",
>>>> "data_extra_pool": ".rgw.buckets.extra",
>>>> "index_pool": ".rgw.buckets.index",
>>>> "marker": "default.691974.1",
>>>> "bucket_id": "default.691974.1",
>>>> "tenant": ""
>>>> },
>>>> "creation_time": "2015-11-13 20:05:26.00Z",
>>>> "owner": "gta",
>>>> "flags": 0,
>>>> "zonegroup": "default",
>>>> "placement_rule": "ec42-placement",
>>>> "has_instance_obj": "true",
>>>> "quota": {
>>>> "enabled": false,
>>>> "max_size_kb": -1,
>>>> "max_objects": -1
>>>> },
>>>> "num_shards": 32,
>>>> "bi_shard_hash_type": 0,
>>>> "requester_pays": "false",
>>>> "has_website": "false",
>>>> "swift_versioning": "false",
>>>> "swift_ver_location": ""
>>>> },
>>>> "attrs": [
>>>> {
>>>> "key": "user.rgw.acl",
>>>> "val":
>>>>
>>>> "AgJ\/AgIXAwAAAGd0YQwAAABHcmFoYW0gQWxsYW4DA1wBAQMAAABndGEPAQMAAABndGEDAzcCAgQAAwAAAGd0YQAAAgIEDwwAAABHcmFoYW0gQWxsYW4AAA=="
>>>> },
>>>> {
>>>> "key": "user.rgw.idtag",
>>>> "val": ""
>>>> },
>>>> {
>>>> "key": "user.rgw.manifest",
>>>> "val": ""
>>>> }
>>>> ]
>>>> }
>>>> }
>>>
>>>
>>>
>>> while here is another, located in the same pool, which is not accessible:
>>>
>>>> # radosgw-admin metadata get bucket.instance:tcga:default.712449.19
>>>> {
>>>> "key": "bucket.instance:tcga:default.712449.19",
>>>> "ver": {
>>>> "tag": "_vm0Og31XbhhtmnuQVZ6cYJP",
>>>> "ver": 2010
>>>> },
>>>> "mtime": "2016-11-19 03:49:03.406938Z",
>>>> "data":

Re: [ceph-users] "Lost" buckets on radosgw

[Yehuda Sadeh-Weinraub]

On Mon, Nov 21, 2016 at 2:42 PM, Graham Allan  wrote:
> Following up to this (same problem, looking at it with Jeff)...
>
> There was definite confusion with the zone/zonegroup/realm/period changes
> during the hammer->jewel upgrade. It's possible that our placement settings
> were misplaced at this time.
>
> However what I find puzzling is that different buckets from the same pool
> seem affected - if this were placement related, I'd rather expect all
> buckets from one pool to be affected, those in another not. Am I
> interpreting this wrongly?
>
> For example here is one bucket which remains accessible:
>
>> # radosgw-admin metadata get bucket.instance:gta:default.691974.1
>> {
>> "key": "bucket.instance:gta:default.691974.1",
>> "ver": {
>> "tag": "_3Z9nfFjZn97aV2YJ4nFhVuk",
>> "ver": 85
>> },
>> "mtime": "2016-11-11 16:48:02.950760Z",
>> "data": {
>> "bucket_info": {
>> "bucket": {
>> "name": "gta",
>> "pool": ".rgw.buckets.ec42",
>> "data_extra_pool": ".rgw.buckets.extra",
>> "index_pool": ".rgw.buckets.index",
>> "marker": "default.691974.1",
>> "bucket_id": "default.691974.1",
>> "tenant": ""
>> },
>> "creation_time": "2015-11-13 20:05:26.00Z",
>> "owner": "gta",
>> "flags": 0,
>> "zonegroup": "default",
>> "placement_rule": "ec42-placement",
>> "has_instance_obj": "true",
>> "quota": {
>> "enabled": false,
>> "max_size_kb": -1,
>> "max_objects": -1
>> },
>> "num_shards": 32,
>> "bi_shard_hash_type": 0,
>> "requester_pays": "false",
>> "has_website": "false",
>> "swift_versioning": "false",
>> "swift_ver_location": ""
>> },
>> "attrs": [
>> {
>> "key": "user.rgw.acl",
>> "val":
>> "AgJ\/AgIXAwAAAGd0YQwAAABHcmFoYW0gQWxsYW4DA1wBAQMAAABndGEPAQMAAABndGEDAzcCAgQAAwAAAGd0YQAAAgIEDwwAAABHcmFoYW0gQWxsYW4AAA=="
>> },
>> {
>> "key": "user.rgw.idtag",
>> "val": ""
>> },
>> {
>> "key": "user.rgw.manifest",
>> "val": ""
>> }
>> ]
>> }
>> }
>
>
> while here is another, located in the same pool, which is not accessible:
>
>> # radosgw-admin metadata get bucket.instance:tcga:default.712449.19
>> {
>> "key": "bucket.instance:tcga:default.712449.19",
>> "ver": {
>> "tag": "_vm0Og31XbhhtmnuQVZ6cYJP",
>> "ver": 2010
>> },
>> "mtime": "2016-11-19 03:49:03.406938Z",
>> "data": {
>> "bucket_info": {
>> "bucket": {
>> "name": "tcga",
>> "pool": ".rgw.buckets.ec42",
>> "data_extra_pool": ".rgw.buckets.extra",
>> "index_pool": ".rgw.buckets.index",
>> "marker": "default.712449.19",
>> "bucket_id": "default.712449.19",
>> "tenant": ""
>> },
>> "creation_time": "2016-01-21 20:51:21.00Z",
>> "owner": "jmcdonal",
>> "flags": 0,
>> "zonegroup": "default",
>> "placement_rule": "ec42-placement",
>> "has_instance_obj": "true",
>> "quota": {
>> "enabled": false,
>> "max_size_kb": -1,
>> "max_objects": -1
>> },
>> "num_shards": 0,
>> "bi_shard_hash_type": 0,
>> "requester_pays": "false",
>> "has_website": "false",
>> "swift_versioning": "false",
>> "swift_ver_location": ""
>> },
>> "attrs": [
>> {
>> "key": "user.rgw.acl",
>> "val":
>> "AgKbAgIgCGptY2RvbmFsEEplZmZyZXkgTWNEb25hbGQDA28BAQgAAABqbWNkb25hbA8BCGptY2RvbmFsAwNAAgIEAAgAAABqbWNkb25hbgIEDwAAABBKZWZmcmV5IE1jRG9uYWxkAAA="
>> },
>> {
>> "key": "user.rgw.idtag",
>> "val": ""
>> },
>> {
>> "key": "user.rgw.manifest",
>> "val": ""
>> }
>> ]
>> }
>> }
>
>
> if I do "ls --pool .rgw.buckets.ec42|grep default.712449.19" I can see
> objects with the above bucket ID, and fetch them, so I know the data is
> there...
>
> Does this seem like a placement_pool issue, or maybe some other unrelated
> issue?
>

Could be another semi-related issue. Can you provide output of the
commands that fail with 'debug rgw = 20' and 'debug ms = 1'?

Thanks,
Yehuda

Re: [ceph-users] "Lost" buckets on radosgw

[Yehuda Sadeh-Weinraub]

On Fri, Nov 18, 2016 at 1:14 PM, Jeffrey McDonald  wrote:
> Hi,
>
> MSI has an erasure coded ceph pool accessible by the radosgw interface.
> We recently upgraded to Jewel from Hammer.   Several days ago, we
> experienced issues with a couple of the rados gateway servers and
> inadvertently deployed older Hammer versions of the radosgw instances.
> This configuration was running for a couple of days.   We removed the Hammer
> versions and re-deployed the Jewel versions of the radosgw.S3cmd
> querying of some of the buckets are now reporting 'NoSuchKey'.   This seems
> to have started when the Hammer versions were deployed and this is now
> persistent.   While radosgw-admin seems to know about the bucket, checks on
> the buckets itself now fail:
>
> # radosgw-admin bucket list --uid=jmcdonal
> [
> "bigbucket",
> "hpmesabiinfo",
> "jmarchive",
> "jmcdon",
> "jmcdonal",
> "jmcdontest3",
> "jmtestbigfiles",
> "laptopbackup",
> "mesabihpsite",
> "msisoftware",
> "tcga"
> ]
> # radosgw-admin bucket check --check-head-obj-locator --bucket=jmarchive
> ERROR: store->list_objects(): (2) No such file or directory
>
> It seems like some metadata about the buckets or objects has been lost, is
> there a way to recover the bucket?
>
> I searched through the archives but didn't find anything exactly like this.
> Please point me to the documentation if this has been seen before.
>

Do you have access to other buckets? Not sure what could have gone
wrong, but start by looking at:

$ radosgw-admin metadata get bucket:

and then, using the bucket_id from this command:

$ radosgw-admin metadata get bucket.instance::

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw print continue and civetweb

[Yehuda Sadeh-Weinraub]

On Mon, Nov 14, 2016 at 9:20 AM, Brian Andrus
 wrote:
> Hi William,
>
> "rgw print continue = true" is an apache specific setting, as mentioned
> here:
>
> http://docs.ceph.com/docs/master/install/install-ceph-gateway/#migrating-from-apache-to-civetweb
>
> I do not believe it is needed for civetweb. For documentation, you can see
> or change the version branch in the URL to make sure you are consuming the
> latest version.
>

That is, no need to touch this settings. The configurable was used to
turn off 100-continue support when using fcgi frontends that didn't
support it. Civetweb does support it.

Yehuda

>
> On Sun, Nov 13, 2016 at 8:03 PM, William Josefsson
>  wrote:
>>
>> Hi list, can anyone please clarify if the default 'rgw print continue
>> = true', is supported by civetweb?
>>
>> I'm using radosgw with civetweb, and this document (may be outdated?)
>> mentions to install apache,
>> http://docs.ceph.com/docs/hammer/install/install-ceph-gateway/. This
>> ticket seems to keep 'print continue' with civetweb enabled.
>> http://tracker.ceph.com/issues/12640
>>
>> Can anyone please clarify whether civetweb support the default
>> 100-continue setting? thx will
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
> --
> Brian Andrus
> Cloud Systems Engineer
> DreamHost, LLC
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw - http status 400 while creating a bucket

[Yehuda Sadeh-Weinraub]

On Wed, Nov 9, 2016 at 1:30 AM, Andrei Mikhailovsky <and...@arhont.com> wrote:
> Hi Yehuda,
>
> just tried to run the command to set the master_zone to default followed by 
> the bucket create without doing the restart and I still have the same error 
> on the client:
>
>  encoding="UTF-8"?>InvalidArgumentmy-new-bucket-31337tx00010-005822ebbd-9951ad8-default9951ad8-default-default
>

After setting the master zone, try running:

$ radosgw-admin period update --commit

Yehuda

>
> Andrei
>
> - Original Message -
>> From: "Yehuda Sadeh-Weinraub" <yeh...@redhat.com>
>> To: "Andrei Mikhailovsky" <and...@arhont.com>
>> Cc: "ceph-users" <ceph-users@lists.ceph.com>
>> Sent: Wednesday, 9 November, 2016 01:13:48
>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket
>
>> On Tue, Nov 8, 2016 at 5:05 PM, Andrei Mikhailovsky <and...@arhont.com> 
>> wrote:
>>> Hi Yehuda,
>>>
>>> I don't have a multizone setup. The radosgw service was configured about two
>>> years ago according to the documentation on ceph.com and haven't changed 
>>> with
>>> numerous version updates. All was working okay until i've upgraded to 
>>> version
>>> 10.2.x.
>>>
>>> Could you please point me in the right direction what exactly needs to be 
>>> done?
>>>
>>>
>>> # radosgw-admin zonegroup get --rgw-zonegroup=default
>>> {
>>> "id": "default",
>>> "name": "default",
>>> "api_name": "",
>>> "is_master": "true",
>>> "endpoints": [],
>>> "hostnames": [],
>>> "hostnames_s3website": [],
>>> "master_zone": "",
>>> "zones": [
>>> {
>>> "id": "default",
>>> "name": "default",
>>> "endpoints": [],
>>> "log_meta": "false",
>>> "log_data": "false",
>>> "bucket_index_max_shards": 0,
>>> "read_only": "false"
>>> }
>>> ],
>>> "placement_targets": [
>>> {
>>> "name": "default-placement",
>>> "tags": []
>>> }
>>> ],
>>> "default_placement": "default-placement",
>>> "realm_id": ""
>>> }
>>
>> Try:
>>
>> $ radosgw-admin zonegroup get --rgw-zonegroup=default > zonegroup.json
>>
>> ... modify the master_zone to be "default"
>>
>> $ radosgw-admin zonegroup set --rgw-zonegroup=default < zonegroup.json
>>
>> (restart radosgw)
>>
>> Yehuda
>>
>>>
>>>
>>>
>>> # radosgw-admin zone get --rgw-zone=default
>>> {
>>> "id": "default",
>>> "name": "default",
>>> "domain_root": ".rgw",
>>> "control_pool": ".rgw.control",
>>> "gc_pool": ".rgw.gc",
>>> "log_pool": ".log",
>>> "intent_log_pool": ".intent-log",
>>> "usage_log_pool": ".usage",
>>> "user_keys_pool": ".users",
>>> "user_email_pool": ".users.email",
>>> "user_swift_pool": ".users.swift",
>>> "user_uid_pool": ".users.uid",
>>> "system_key": {
>>> "access_key": "",
>>> "secret_key": ""
>>> },
>>> "placement_pools": [
>>> {
>>> "key": "default-placement",
>>> "val": {
>>> "index_pool": ".rgw.buckets.index",
>>> "data_pool": ".rgw.buckets",
>>> "data_extra_pool": "default.rgw.buckets.non-ec",
>>> "index_type": 0
>>> }
>>> }
>>> ],
>>> "metad

Re: [ceph-users] radosgw - http status 400 while creating a bucket

[Yehuda Sadeh-Weinraub]

On Tue, Nov 8, 2016 at 5:05 PM, Andrei Mikhailovsky <and...@arhont.com> wrote:
> Hi Yehuda,
>
> I don't have a multizone setup. The radosgw service was configured about two 
> years ago according to the documentation on ceph.com and haven't changed with 
> numerous version updates. All was working okay until i've upgraded to version 
> 10.2.x.
>
> Could you please point me in the right direction what exactly needs to be 
> done?
>
>
> # radosgw-admin zonegroup get --rgw-zonegroup=default
> {
> "id": "default",
> "name": "default",
> "api_name": "",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "",
> "zones": [
> {
> "id": "default",
> "name": "default",
> "endpoints": [],
> "log_meta": "false",
> "log_data": "false",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": ""
> }

Try:

$ radosgw-admin zonegroup get --rgw-zonegroup=default > zonegroup.json

... modify the master_zone to be "default"

$ radosgw-admin zonegroup set --rgw-zonegroup=default < zonegroup.json

(restart radosgw)

Yehuda

>
>
>
> # radosgw-admin zone get --rgw-zone=default
> {
> "id": "default",
> "name": "default",
> "domain_root": ".rgw",
> "control_pool": ".rgw.control",
> "gc_pool": ".rgw.gc",
> "log_pool": ".log",
> "intent_log_pool": ".intent-log",
> "usage_log_pool": ".usage",
> "user_keys_pool": ".users",
>     "user_email_pool": ".users.email",
> "user_swift_pool": ".users.swift",
> "user_uid_pool": ".users.uid",
> "system_key": {
> "access_key": "",
> "secret_key": ""
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": ".rgw.buckets.index",
> "data_pool": ".rgw.buckets",
> "data_extra_pool": "default.rgw.buckets.non-ec",
> "index_type": 0
> }
> }
> ],
> "metadata_heap": ".rgw.meta",
> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c"
> }
>
>
>
> Thanks
>
>
>
> - Original Message -
>> From: "Yehuda Sadeh-Weinraub" <yeh...@redhat.com>
>> To: "Andrei Mikhailovsky" <and...@arhont.com>
>> Cc: "ceph-users" <ceph-users@lists.ceph.com>
>> Sent: Wednesday, 9 November, 2016 00:48:50
>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket
>
>> On Tue, Nov 8, 2016 at 3:36 PM, Andrei Mikhailovsky <and...@arhont.com> 
>> wrote:
>>> Hello
>>>
>>> I am having issues with creating buckets in radosgw. It started with an
>>> upgrade to version 10.2.x
>>>
>>> When I am creating a bucket I get the following error on the client side:
>>>
>>>
>>> boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
>>> >> encoding="UTF-8"?>InvalidArgumentmy-new-bucket-31337tx2-0058225bae-994d148-default994d148-default-default
>>>
>>>
>>> The radosgw logs are (redacted):
>>>
>>> ###
>>>
>>> 2016-11-08 23:11:42.876862 7f026d953700 20 enqueued request
>>> req=0x7f02ba07b0e0
>>> 2016-11-08 23:11:42.876892 7f026d953700 20 RGWWQ:
>>> 2016-11-08 23:11:42.876897 7f026d953700 20 req: 0x7f02ba07b0e0
>>> 2016-11-08 23:11:42.876912 7f026d953700 10 allocated request
>>> req=0x7f02ba07b140
>>> 2016-1

Re: [ceph-users] Migrate pre-Jewel RGW data to Jewel realm/zonegroup/zone?

[Yehuda Sadeh-Weinraub]

Generally you need to create a new realm, and add the 'default'
zonegroup into it. I think you can achieve this via the 'radosgw-admin
zonegroup modify' command.
The zonegroup and zone can be renamed (their id will still be
'default', but you can change their names).

Yehuda

On Thu, Oct 6, 2016 at 3:07 AM, Richard Chan
 wrote:
> Upgraded from Hammer 0.94.9 to Jewel 10.2.3 while all RGW data survived in a
> realm-less, setup (no realm, "default" zonegroup, "default" zone).
>
> Is it possible to "move" this into a single realm/zonegroup/zone in
> preparation for multisite (i.e before add the 2nd zone).
>
> I don't need more than one realm, but would like to
>
> 1. create a single realm"default"
> 2. move zonegroup "default" as "us", say
> 3. move zone to zonegroup as "us-east"
>
> This must preserve all existing data. After which I would like to add a 2nd
> zone (in a different ceph cluster) "us-west".
>
>
> # radosgw-admin realm list
> {
> "default_info": "",
> "realms": []
> }
>
> # radosgw-admin zonegroup list
> read_default_id : -2
> {
> "default_info": "",
> "zonegroups": [
> "default"
> ]
> }
>
>
> # radosgw-admin zone list
> {
> "default_info": "",
> "zones": [
> "default"
> ]
> }
>
>
>
>
>
>
> --
> Richard Chan
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Question on RGW MULTISITE and librados

[Yehuda Sadeh-Weinraub]

On Thu, Sep 22, 2016 at 1:52 PM, Paul Nimbley  wrote:
> Fairly new to ceph so please excuse any misused terminology.  We’re
> currently exploring the use of ceph as a replacement storage backend for an
> existing application.  The existing application has 2 requirements which
> seemingly can be met individually by using librados and the Ceph Object
> Gateway multisite support, but it seems cannot be met together.  These are:
>
>
>
> 1.   The ability to append to an existing object and read from any
> offset/length within the object, (the librados API allows this, the S3 and
> Swift APIs do not appear to support this).
>
> 2.   The ability to replicate data between 2 geographically separate
> locations.  I.e. 2 separate ceph clusters using the multisite support of the
> Ceph Object Gateway to replicate between them.
>
>
>
> Specifically we’re testing using librados to write directly to the object
> store because we need the ability to append to objects, which using the
> librados API allows.  However if one writes to the object store directly
> using the librados API is it correct to assume that those objects will not
> be replicated to the other zone by the Ceph Object Gateway since its being
> taken out of the data path?
>

The rgw multisite feature is an rgw only feature, and as such it
doesn't apply to raw rados object operations. The rados gateway only
handles its own data's replication, and it depends on its internal
data structures and its different mechanics, so for raw rados
replication there needs to be a different system in place.

Yehuda

>
>
> Thanks,
>
> Paul
>
> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
> for the use of the addressee(s).
> If you are not the intended recipient, please notify so to the sender by
> e-mail and delete the original message.
> In such cases, please notify us immediately at i...@infinite.com . Further,
> you are not to copy,
> disclose, or distribute this e-mail or its contents to any unauthorized
> person(s) .Any such actions are
> considered unlawful. This e-mail may contain viruses. Infinite has taken
> every reasonable precaution to minimize
> this risk, but is not liable for any damage you may sustain as a result of
> any virus in this e-mail. You should
> carry out your own virus checks before opening the e-mail or attachments.
> Infinite reserves the right to monitor
> and review the content of all messages sent to or from this e-mail address.
> Messages sent to or from this e-mail
> address may be stored on the Infinite e-mail system.
> ***INFINITE End of DisclaimerINFINITE
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High CPU load with radosgw instances

[Yehuda Sadeh-Weinraub]

On Thu, Sep 15, 2016 at 4:53 PM, lewis.geo...@innoscale.net
 wrote:
> Hi,
> So, maybe someone has an idea of where to go on this.
>
> I have just setup 2 rgw instances in a multisite setup. They are working
> nicely. I have add a couple of test buckets and some files to make sure it
> works is all. The status shows both are caught up. Nobody else is accessing
> or using them.
>
> However, the CPU load on both hosts is sitting at like 3.00, with the
> radosgw process taking up 99% CPU constantly. I do not see anything in the
> logs happening at all.
>
> Any thoughts or direction?
>

We've seen that happening when running on a system with older version
of libcurl (e.g., 7.29). If that's the case upgrading to a newer
version should fix it for you.

Yehuda


> Have a good day,
>
> Lewis George
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW zonegroup id error

[Yehuda Sadeh-Weinraub]

On Fri, Sep 2, 2016 at 12:54 AM, Yoann Moulin  wrote:
> Hello,
>
>> I have an issue with the default zonegroup on my cluster (Jewel 10.2.2), I 
>> don't
>> know when this occured, but I think I did a wrong command during the
>> manipulation of zones and regions. Now the ID of my zonegroup is "default"
>> instead of "4d982760-7853-4174-8c05-cec2ef148cf0", I cannot update zones or
>> regions anymore.
>>
>> Is that possible to change the ID of the zonegroup, I try to update the json
>> then set the zonegroup but it doesn't work (certainly because it's not the 
>> same
>> ID...)
>
> if I create a new zonegroup then set as the default zonegroup, update the 
> zonegroup-map, zone etc, then delete the zonegroup with the ID
> "default" it should work ?
>
It should work. Do you have any existing data on the zone group?

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cleaning Up Failed Multipart Uploads

[Yehuda Sadeh-Weinraub]

On Wed, Aug 3, 2016 at 10:57 AM, Brian Felton <bjfel...@gmail.com> wrote:

> I should clarify:
>
> There doesn't seem to be a problem with list_multipart_parts -- upon
> further review, it seems to be doing the right thing.  What tipped me off
> is that when one aborts a multipart upload where parts have been uploaded
> more than once, the last copy of each part uploaded is successfully removed
> (not just removed from the bucket's stats, as with complete multipart, but
> submitted for garbage collection).  The difference seems to be in the
> following:
>
> In RGWCompleteMultipart::execute, the removal doesn't occur on the entries
> returned from list_mutlpart_parts; instead, we initialize a 'src_obj'
> rgw_obj structure and grab its index key
> (src_obj.get_index_key(_key)), which is then pushed onto remove_objs.
>

iirc, we don't really remove the objects there. Only remove the entries
from the index.


>
> In RGWAbortMultipart::execute, we operate directly on the
> RGWUploadPartInfo value in the obj_parts map, submitting it for deletion
> (gc) if its manifest is empty.
>
> If this is correct, there is no "fix" for list_multipart_parts; instead,
> it would seem that the only fix is to not allow an upload part to generate
> a new prefix in RGWPutObj::execute().
>

The problem is that operations can happen concurrently, so the decision
whether to remove or not to remove an entry is not very easy. We have seen
before that application initiated multiple uploads of the same part, but
the one that actually complete the last was not the last to upload (e.g.,
due to networking timeouts and retries that happen in different layers).


> Since I don't really have any context on why a new prefix would be
> generated if the object already exists, I'm not the least bit confident
> that changing it will not have all sorts of unforeseen consequences.  That
> said, since all knowledge of an uploaded part seems to vanish from
> existence once it has been replaced, I don't see how the accounting of
> multipart data will ever be correct.
>

Having a mutable part is problematic, since different uploads might step on
each other (as with the example I provided above), and you end up with
corrupted data.


>
> And yes, I've tried the orphan find, but I'm not really sure what to do
> with the results.  The post I could find in the mailing list (mostly from
> you), seemed to conclude that no action should be taken on the things that
> it finds are orphaned.  Also, I have removed a significant number of
> multipart and shadow files that are not valid, but none of that actually
>

The tool is not removing data, only reporting about possible leaked rados
objects.


> updates the buckets stats to the correct values.  If I had some mechanism
> for forcing that, this would be much less of a big deal.
>

Right, this is a separate issue. Did you try running 'radosgw-admin bucket
check --fix'?

Yehuda


>
>
> Brian
>
> On Wed, Aug 3, 2016 at 12:46 PM, Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>
>>
>>
>> On Wed, Aug 3, 2016 at 10:10 AM, Brian Felton <bjfel...@gmail.com> wrote:
>>
>>> This may just be me having a conversation with myself, but maybe this
>>> will be helpful to someone else.
>>>
>>> Having dug and dug and dug through the code, I've come to the following
>>> realizations:
>>>
>>>1. When a multipart upload is completed, the function
>>>list_multipart_parts in rgw_op.cc is called.  This seems to be the start 
>>> of
>>>the problems, as it will only return those parts in the 'multipart'
>>>namespace that include the upload id in the name, irrespective of how 
>>> many
>>>copies of parts exist on the system with non-upload id prefixes
>>>2. In the course of writing to the OSDs, a list (remove_objs) is
>>>processed in cls_rgw.cc:unaccount_entry(), causing bucket stats to be
>>>decremented
>>>3. These decremented stats are written to the bucket's index
>>>entry/entries in .rgw.buckets.index via the CEPH_OSD_OP_OMAPSETHEADER 
>>> case
>>>in ReplicatedPG::do_osd_ops
>>>
>>> So this explains why manually removing the multipart entries from
>>> .rgw.buckets and cleaning the shadow entries in .rgw.buckets.index does not
>>> cause the bucket's stats to be updated.  What I don't know how to do is
>>> force an update of the bucket's stats from the CLI.  I can retrieve the
>>> omap header from each of the bucket's shards in .rgw.buckets.index, but I
>>> don't have the first clue how to read the data or rebuild it into something
>>> valid.  I'v

Re: [ceph-users] Cleaning Up Failed Multipart Uploads

[Yehuda Sadeh-Weinraub]

On Wed, Aug 3, 2016 at 10:10 AM, Brian Felton  wrote:

> This may just be me having a conversation with myself, but maybe this will
> be helpful to someone else.
>
> Having dug and dug and dug through the code, I've come to the following
> realizations:
>
>1. When a multipart upload is completed, the function
>list_multipart_parts in rgw_op.cc is called.  This seems to be the start of
>the problems, as it will only return those parts in the 'multipart'
>namespace that include the upload id in the name, irrespective of how many
>copies of parts exist on the system with non-upload id prefixes
>2. In the course of writing to the OSDs, a list (remove_objs) is
>processed in cls_rgw.cc:unaccount_entry(), causing bucket stats to be
>decremented
>3. These decremented stats are written to the bucket's index
>entry/entries in .rgw.buckets.index via the CEPH_OSD_OP_OMAPSETHEADER case
>in ReplicatedPG::do_osd_ops
>
> So this explains why manually removing the multipart entries from
> .rgw.buckets and cleaning the shadow entries in .rgw.buckets.index does not
> cause the bucket's stats to be updated.  What I don't know how to do is
> force an update of the bucket's stats from the CLI.  I can retrieve the
> omap header from each of the bucket's shards in .rgw.buckets.index, but I
> don't have the first clue how to read the data or rebuild it into something
> valid.  I've searched the docs and mailing list archives, but I didn't find
> any solution to this problem.  For what it's worth, I've tried 'bucket
> check' with all combinations of '--check-objects' and '--fix' after
> cleaning up .rgw.buckets and .rgw.buckets.index.
>
> From a long-term perspective, it seems there are two possible fixes here:
>
>1. Update the logic in list_multipart_parts to return all the parts
>for a multipart object, so that *all* parts in the 'multipart' namespace
>can be properly removed
>2. Update the logic in RGWPutObj::execute() to not restart a write if
>the put_data_and_throttle() call returns -EEXIST but instead put the data
>in the original file(s)
>
> While I think 2 would involve the least amount of yak shaving with the
> multipart logic since the MP logic already assumes a happy path where all
> objects have a prefix of the multipart upload id, I'm all but certain this
> is going to horribly break many other parts of the system that I don't
> fully understand.
>

#2 is dangerous. That was the original behavior, and it is racy and *will*
lead to data corruption.  OTOH, I don't think #1 is an easy option. We only
keep a single entry per part, so we don't really have a good way to see all
the uploaded pieces. We could extend the meta object to keep record of all
the uploaded parts, and at the end, when assembling everything remove the
parts that aren't part of the final assembly.

> The good news is that the assembly of the multipart object is being done
> correctly; what I can't figure out is how it knows about the non-upload id
> prefixes when creating the metadata on the multipart object in
> .rgw.buckets.  My best guess is that it's copying the metadata from the
> 'meta' object in .rgw.buckets.extra (which is correctly updated with the
> new part prefixes after each successful upload), but I haven't absolutely
> confirmed that.
>

Yeah, something along these lines.


> If one of the developer folk that are more familiar with this could weigh
> in, I would be greatly appreciative.
>

btw, did you try to run the radosgw-admin orphan find tool?

Yehuda

> Brian
>
> On Tue, Aug 2, 2016 at 8:59 AM, Brian Felton  wrote:
>
>> I am actively working through the code and debugging everything.  I
>> figure the issue is with how RGW is listing the parts of a multipart upload
>> when it completes or aborts the upload (read: it's not getting *all* the
>> parts, just those that are either most recent or tagged with the upload
>> id).  As soon as I can figure out a patch, or, more importantly, how to
>> manually address the problem, I will respond with instructions.
>>
>> The reported bug contains detailed instructions on reproducing the
>> problem, so it's trivial to reproduce and test on a small and/or new
>> cluster.
>>
>> Brian
>>
>>
>> On Tue, Aug 2, 2016 at 8:53 AM, Tyler Bishop <
>> tyler.bis...@beyondhosting.net> wrote:
>>
>>> We're having the same issues.   I have a 1200TB pool at 90% utilization
>>> however disk utilization is only 40%
>>>
>>>
>>>
>>>  [image: http://static.beyondhosting.net/img/bh-small.png]
>>>
>>>
>>> *Tyler Bishop *Chief Technical Officer
>>> 513-299-7108 x10
>>>
>>> tyler.bis...@beyondhosting.net
>>>
>>> If you are not the intended recipient of this transmission you are
>>> notified that disclosing, copying, distributing or taking any action in
>>> reliance on the contents of this information is strictly prohibited.
>>>
>>>
>>>
>>> --
>>> *From: *"Brian Felton" 

Re: [ceph-users] [jewel][rgw]why the usage log record date is 16 hours later than the real operate time

[Yehuda Sadeh-Weinraub]

On Thu, Jul 28, 2016 at 5:53 PM, Leo Yu  wrote:
> hi all,
>   i want get the usage of user,so i use the command radosgw-admin usage show
> ,but i can not get the usage when i use the --start-date unless  minus 16
> hours
>
> i have rgw both on ceph01 and ceph03,civeweb:7480 port ,and the ceph version
> is jewel 10.2.2
>
> the time zone of ceph01 and ceph03
> [root@ceph03 ~]# ls -l /etc/localtime
> lrwxrwxrwx 1 root root 35 Jul 25 07:40 /etc/localtime ->
> ../usr/share/zoneinfo/Asia/Shanghai
> [root@ceph03 ~]# ssh ceph01 ls -l /etc/localtime
> lrwxrwxrwx 1 root root 35 Jul 25 14:14 /etc/localtime ->
> ../usr/share/zoneinfo/Asia/Shanghai
>
>
> the timedateclt of ceph03
>
> [root@ceph03 ~]# timedatectl
> Warning: Ignoring the TZ variable. Reading the system's time zone setting
> only.
>
>   Local time: Fri 2016-07-29 08:28:44 CST
>   Universal time: Fri 2016-07-29 00:28:44 UTC
> RTC time: Fri 2016-07-29 00:28:44
>Time zone: Asia/Shanghai (CST, +0800)
>  NTP enabled: yes
> NTP synchronized: yes
>  RTC in local TZ: no
>   DST active: n/a
>
> the timedate of ceph01
> [root@ceph03 ~]# ssh ceph01 timedatectl
>   Local time: Fri 2016-07-29 08:32:43 CST
>   Universal time: Fri 2016-07-29 00:32:43 UTC
> RTC time: Fri 2016-07-29 08:32:43
>Time zone: Asia/Shanghai (CST, +0800)
>  NTP enabled: yes
> NTP synchronized: yes
>  RTC in local TZ: no
>   DST active: n/a
>
>
> i create bucket use the python script test2.py
>
> [root@ceph01 ~]# cat  test2.py
> import requests
> import logging
> from datetime import *
> from requests_toolbelt.utils import dump
> from awsauth import S3Auth
> # host = 'yuliyangdebugwebjewel.tunnel.qydev.com'
> #host = 'yuliyangdebugweb68.tunnel.qydev.com'
> #host = '10.254.9.20:7480'
> host = '10.254.3.68:7480' #ceph03
> #host = '127.0.0.1:7480'  #ceph01
> logging.basicConfig(level=logging.DEBUG)
> access_key = 'date2'
> secret_key = 'date2'
> cmd = '/%s' % '{:%m_%d.%H_%M_%S}'.format(datetime.now())
> #cmd = '/%s' % '{:%m_%d.%H_%M_%S}'.format(datetime.now() -
> timedelta(hours=16))
> url = 'http://%s%s' % (host,cmd)
> response = requests.put(url,auth=S3Auth(access_key,
> secret_key,service_url=host))
>
> data = dump.dump_all(response)
> print(data.decode('utf-8'))
>
>
> and it's output
>
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection
> (1): 10.254.3.68
> DEBUG:requests.packages.urllib3.connectionpool:"PUT /07_29.08_34_42
> HTTP/1.1" 200 0
> < PUT /07_29.08_34_42 HTTP/1.1
> < Host: 10.254.3.68:7480
> < Content-Length: 0
> < Accept-Encoding: gzip, deflate
> < Accept: */*
> < User-Agent: python-requests/2.6.0 CPython/2.7.5
> Linux/3.10.0-327.el7.x86_64
> < Connection: keep-alive
> < date: Fri, 29 Jul 2016 00:34:42 GMT
> < Authorization: AWS date2:F+KLuenLNP42e25P/My/VWoUkeA=
> <
>
>> HTTP/1.1 200 OK
>> date: Fri, 29 Jul 2016 00:34:42 GMT
>> content-length: 0
>> x-amz-request-id: tx16112-00579aa4a2-c4c4f-default
>> connection: Keep-Alive
>>
>
>
>
> but the usage show the bucket create 16 hours before
>
> [root@ceph01 ~]# radosgw-admin usage show --uid=date2 |grep 07_29.08_34_42
> -A30
> "bucket": "07_29.08_34_42",
> "time": "2016-07-28 16:00:00.00Z",
> "epoch": 1469721600,
> "owner": "date2",
> "categories": [
> {
> "category": "create_bucket",
> "bytes_sent": 19,
> "bytes_received": 0,
> "ops": 1,
> "successful_ops": 1
> }
> ]
> },
>
>
> the time("time": "2016-07-28 16:00:00.00Z",) is 16 hours later than
> Local time: Fri 2016-07-29 08:28:44 CST
>
> i can get the usage show by radosgw-admin usage show --uid=date2
> --start-date="2016-07-28 16:00:00"
> [root@ceph03 ~]# radosgw-admin usage show --uid=date2
> --start-date="2016-07-28 16:00:00"
> {
> "entries": [
> {
> "user": "date2",
> "buckets": [
> {
> "bucket": "07_29.08_34_42",
> "time": "2016-07-28 16:00:00.00Z",
> "epoch": 1469721600,
> "owner": "date2",
> "categories": [
> {
> "category": "create_bucket",
> "bytes_sent": 19,
> "bytes_received": 0,
> "ops": 1,
> "successful_ops": 1
> }
> ]
> }
> ]
> }
> ],
> "summary": [
> {
> "user": "date2",
> "categories": [
> {
> "category": "create_bucket",
> 

Re: [ceph-users] blind buckets

[Yehuda Sadeh-Weinraub]

On Thu, Jul 28, 2016 at 12:11 PM, Tyler Bischel
<tyler.bisc...@sendgrid.com> wrote:
> Can I not update an existing placement target's index_type?  I had tried to
> update the default pool's index type:
>
> radosgw-admin zone get --rgw-zone=default > default-zone.json
>
> #replace index_type:0 to index_type:1 in the default zone file, under the
> default-placement entry of the placement_pools
>
> radosgw-admin zone set --rgw-zone=default --infile default-zone.json
>
> However, it seems like I can still access bucket lists of objects after
> additional objects added, which makes me think this setting isn't being
> respected in the way I thought it would.
>

It only affects newly created buckets.

Yehuda

> On Thu, Jul 28, 2016 at 9:59 AM, Yehuda Sadeh-Weinraub <yeh...@redhat.com>
> wrote:
>>
>> In order to use indexless (blind) buckets, you need to create a new
>> placement target, and then set the placement target's index_type param
>> to 1.
>>
>> Yehuda
>>
>> On Tue, Jul 26, 2016 at 10:30 AM, Tyler Bischel
>> <tyler.bisc...@sendgrid.com> wrote:
>> > Hi there,
>> >   We are looking at using Ceph (Jewel) for a use case that is very write
>> > heavy strictly as an object store.  We've been working with Rados
>> > Gateway
>> > because we can easily integrate with existing S3 libraries... but we
>> > will
>> > never be doing any of the bucket listing operations.  I am concerned
>> > about
>> > the potential bottleneck of the RGW index files.
>> >   I've read here that Jewel now supports "Blind Buckets"... with some
>> > reference to setting a the RGWBucketIndexType to RGWBIType_Indexless...
>> > and
>> > I'm guessing its set as "index_type" here.  In the docs, the only
>> > "index_type" reference I see is here, under the placement pools.
>> > However,
>> > the Pools documentation doesn't really give a clue how to set this
>> > value, or
>> > even if this is the proper index_type field that I'm guessing.
>> >   So the two things I'm interested in figuring out is:
>> > 1) Are "Blind Buckets" actually production ready
>> > 2) How do I configure Rados Gateway to omit the index files?
>> > --Tyler
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] blind buckets

[Yehuda Sadeh-Weinraub]

In order to use indexless (blind) buckets, you need to create a new
placement target, and then set the placement target's index_type param
to 1.

Yehuda

On Tue, Jul 26, 2016 at 10:30 AM, Tyler Bischel
 wrote:
> Hi there,
>   We are looking at using Ceph (Jewel) for a use case that is very write
> heavy strictly as an object store.  We've been working with Rados Gateway
> because we can easily integrate with existing S3 libraries... but we will
> never be doing any of the bucket listing operations.  I am concerned about
> the potential bottleneck of the RGW index files.
>   I've read here that Jewel now supports "Blind Buckets"... with some
> reference to setting a the RGWBucketIndexType to RGWBIType_Indexless... and
> I'm guessing its set as "index_type" here.  In the docs, the only
> "index_type" reference I see is here, under the placement pools.  However,
> the Pools documentation doesn't really give a clue how to set this value, or
> even if this is the proper index_type field that I'm guessing.
>   So the two things I'm interested in figuring out is:
> 1) Are "Blind Buckets" actually production ready
> 2) How do I configure Rados Gateway to omit the index files?
> --Tyler
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw pool names

[Yehuda Sadeh-Weinraub]

On Fri, Jun 10, 2016 at 11:44 AM, Deneau, Tom  wrote:
> When I start radosgw, I create the pool .rgw.buckets manually to control
> whether it is replicated or erasure coded and I let the other pools be
> created automatically.
>
> However, I have noticed that sometimes the pools get created with the 
> "default"
> prefix, thus
> rados lspools
>   .rgw.root
>   default.rgw.control
>   default.rgw.data.root
>   default.rgw.gc
>   default.rgw.log
>   .rgw.buckets  # the one I created
>   default.rgw.users.uid
>   default.rgw.users.keys
>   default.rgw.meta
>   default.rgw.buckets.index
>   default.rgw.buckets.data  # the one actually being used
>
> What controls whether these pools have the "default" prefix or not?
>

The prefix is the name of the zone ('default' by default). This was
added for the jewel release, as well as dropping the requirement of
having the pool names starts with a dot.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw s3website issue

[Yehuda Sadeh-Weinraub]

On Sun, May 29, 2016 at 4:47 AM, Gaurav Bafna  wrote:
> Hi Cephers,
>
> I am unable to create bucket hosting a webstite in my vstart cluster.
>
> When I do this in boto :
>
> website_bucket.configure_website('index.html','error.html')
>
> I get :
>
> boto.exception.S3ResponseError: S3ResponseError: 405 Method Not Allowed
>
>
> Here is my ceph.conf for radosgw:
>
> rgw frontends = fastcgi, civetweb port=8010
>
> rgw enable static website = true
>
> rgw dns name = 10.140.13.22
>
> rgw dns s3website name = 10.140.13.22
>
>
> Here are the logs in rgw :
>
> 2016-05-29 00:00:47.191297 7ff404ff9700  1 == starting new request
> req=0x7ff404ff37d0 =
>
> 2016-05-29 00:00:47.191325 7ff404ff9700  2 req 1:0.28::PUT
> /s3website/::initializing for trans_id =
> tx1-005749967f-101f-default
>
> 2016-05-29 00:00:47.191330 7ff404ff9700 10 host=10.140.13.22
>
> 2016-05-29 00:00:47.191338 7ff404ff9700 20 subdomain=
> domain=10.140.13.22 in_hosted_domain=1 in_hosted_domain_s3website=1
>

Could it be that the endpoint is configured to serve both S3 and
static websites?

Yehuda

> 2016-05-29 00:00:47.191350 7ff404ff9700  5 the op is PUT
>
> 2016-05-29 00:00:47.191395 7ff404ff9700 20 get_handler
> handler=32RGWHandler_REST_Bucket_S3Website
>
> 2016-05-29 00:00:47.191399 7ff404ff9700 10
> handler=32RGWHandler_REST_Bucket_S3Website
>
> 2016-05-29 00:00:47.191401 7ff404ff9700  2 req 1:0.000104:s3:PUT
> /s3website/::getting op 1
>
> 2016-05-29 00:00:47.191410 7ff404ff9700 10
> RGWHandler_REST_S3Website::error_handler err_no=-2003 http_ret=405
>
> 2016-05-29 00:00:47.191412 7ff404ff9700 20 No special error handling today!
>
> 2016-05-29 00:00:47.191415 7ff404ff9700 20 handler->ERRORHANDLER:
> err_no=-2003 new_err_no=-2003
>
> 2016-05-29 00:00:47.191504 7ff404ff9700  2 req 1:0.000207:s3:PUT
> /s3website/::op status=0
>
> 2016-05-29 00:00:47.191510 7ff404ff9700  2 req 1:0.000213:s3:PUT
> /s3website/::http status=405
>
> 2016-05-29 00:00:47.191511 7ff404ff9700  1 == req done
> req=0x7ff404ff37d0 op status=0 http_status=405 ==
>
>
> Code wise I see that put_op is not defined for
> RGWHandler_REST_S3Website class but is defined for
> RGWHandler_REST_Bucket_S3 class .
>
> Can somebody please help me out ?
>
>
>
>
> --
> Gaurav Bafna
> 9540631400
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw hammer -> jewel upgrade (default zone & region config)

[Yehuda Sadeh-Weinraub]

On Fri, May 20, 2016 at 9:03 AM, Jonathan D. Proulx  wrote:
> Hi All,
>
> I saw the previous thread on this related to
> http://tracker.ceph.com/issues/15597
>
> and Yehuda's fix script
> https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone
>
> Running this seems to have landed me in a weird state.
>
> I can create and get new buckets and objects but I've "lost" all my
> old buckets.  I'm fairly confident the "lost" data is in the
> .rgw.buckets pool but my current zone is set to use .rgw.buckets_
>
>
>
> root@ceph-mon0:~# radosgw-admin zone get
> {
> "id": "default",
> "name": "default",
> "domain_root": ".rgw_",
> "control_pool": ".rgw.control_",
> "gc_pool": ".rgw.gc_",
> "log_pool": ".log_",
> "intent_log_pool": ".intent-log_",
> "usage_log_pool": ".usage_",
> "user_keys_pool": ".users_",
> "user_email_pool": ".users.email_",
> "user_swift_pool": ".users.swift_",
> "user_uid_pool": ".users.uid_",
> "system_key": {
> "access_key": "",
> "secret_key": ""
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": ".rgw.buckets.index_",
> "data_pool": ".rgw.buckets_",
> "data_extra_pool": ".rgw.buckets.extra_",
> "index_type": 0
> }
> }
> ],
> "metadata_heap": "default.rgw.meta",
> "realm_id": "a935d12f-14b7-4bf8-a24f-596d5ddd81be"
> }
>
>
> root@ceph-mon0:~# ceph osd pool ls |grep rgw|sort
> default.rgw.meta
> .rgw
> .rgw_
> .rgw.buckets
> .rgw.buckets_
> .rgw.buckets.index
> .rgw.buckets.index_
> .rgw.control
> .rgw.control_
> .rgw.gc
> .rgw.gc_
> .rgw.root
> .rgw.root.backup
>
> Should I just adjust the zone to use the pools without trailing
> slashes?  I'm a bit lost.  the last I could see from running the

Yes. The trailing slashes were needed when upgrading for 10.2.0, as
there was another bug, and I needed to add these to compensate for it.
I should update the script now to reflect that fix. You should just
update the json and set the zone appropriately.

Yehuda

> script didn't seem to indicate any errors (though I lost the to to
> scroll back buffer before i noticed the issue)
>
> Tail of output from running script:
> https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone
>
> + radosgw-admin zone set --rgw-zone=default
> zone id default{
> "id": "default",
> "name": "default",
> "domain_root": ".rgw_",
> "control_pool": ".rgw.control_",
> "gc_pool": ".rgw.gc_",
> "log_pool": ".log_",
> "intent_log_pool": ".intent-log_",
> "usage_log_pool": ".usage_",
> "user_keys_pool": ".users_",
> "user_email_pool": ".users.email_",
> "user_swift_pool": ".users.swift_",
> "user_uid_pool": ".users.uid_",
> "system_key": {
> "access_key": "",
> "secret_key": ""
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": ".rgw.buckets.index_",
> "data_pool": ".rgw.buckets_",
> "data_extra_pool": ".rgw.buckets.extra_",
> "index_type": 0
> }
> }
> ],
> "metadata_heap": "default.rgw.meta",
> "realm_id": "a935d12f-14b7-4bf8-a24f-596d5ddd81be"
> }
> + radosgw-admin zonegroup default --rgw-zonegroup=default
> + radosgw-admin zone default --rgw-zone=default
> root@ceph-mon0:~# radosgw-admin region get --rgw-zonegroup=default
> {
> "id": "default",
> "name": "default",
> "api_name": "",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "default",
> "zones": [
> {
> "id": "default",
> "name": "default",
> "endpoints": [],
> "log_meta": "false",
> "log_data": "false",
> "bucket_index_max_shards": 0,
> "read_only": "false"}
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "a935d12f-14b7-4bf8-a24f-596d5ddd81be"}
>
> root@ceph-mon0:~# ceph -v
> ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
>
> Thanks,
> -Jon
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW - Problems running the S3 and SWIFT API at the same time

[Yehuda Sadeh-Weinraub]

On Thu, May 12, 2016 at 12:29 AM, Saverio Proto  wrote:
>> While I'm usually not fond of blaming the client application, this is
>> really the swift command line tool issue. It tries to be smart by
>> comparing the md5sum of the object's content with the object's etag,
>> and it breaks with multipart objects. Multipart objects is calculated
>> differently (md5sum of the md5sum of each part). I think the swift
>> tool has a special handling for swift large objects (which are not the
>> same as s3 multipart objects), so that's why it works in that specific
>> use case.
>
> Well but I tried also with rclone and I have the same issue.
>
> Clients I tried
> rclone (both SWIFT and S3)
> s3cmd (S3)
> python-swiftclient (SWIFT).
>
> I can reproduce the issue with different clients.
> Once a multipart object is uploaded via S3 (with rclone or s3cmd) I
> cannot read it anymore via SWIFT (either with rclone or
> pythonswift-client).
>
> Are you saying that all SWIFT clients implementations are wrong ?

Yes.

>
> Or should the radosgw be configured with only 1 API active ?
>
> Saverio
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW - Problems running the S3 and SWIFT API at the same time

[Yehuda Sadeh-Weinraub]

While I'm usually not fond of blaming the client application, this is
really the swift command line tool issue. It tries to be smart by
comparing the md5sum of the object's content with the object's etag,
and it breaks with multipart objects. Multipart objects is calculated
differently (md5sum of the md5sum of each part). I think the swift
tool has a special handling for swift large objects (which are not the
same as s3 multipart objects), so that's why it works in that specific
use case.

Yehuda

On Wed, May 11, 2016 at 7:15 AM, Saverio Proto  wrote:
> It does not work also the way around:
>
> If I upload a file with the swift client with the -S options to force
> swift to make multipart:
>
> swift upload -S 100 multipart 180.mp4
>
> Then I am not able to read the file with S3
>
> s3cmd get s3://multipart/180.mp4
> download: 's3://multipart/180.mp4' -> './180.mp4'  [1 of 1]
> download: 's3://multipart/180.mp4' -> './180.mp4'  [1 of 1]
>  38818503 of 38818503   100% in1s27.32 MB/s  done
> WARNING: MD5 signatures do not match:
> computed=961f154cc78c7bf1be3b4009c29e5a68,
> received=d41d8cd98f00b204e9800998ecf8427e
>
> Saverio
>
>
> 2016-05-11 16:07 GMT+02:00 Saverio Proto :
>> Thank you.
>>
>> It is exactly a problem with multipart.
>>
>> So I tried two clients (s3cmd and rclone). When you upload a file in
>> S3 using multipart, you are not able to read anymore this object with
>> the SWIFT API because the md5 check fails.
>>
>> Saverio
>>
>>
>>
>> 2016-05-09 12:00 GMT+02:00 Xusangdi :
>>> Hi,
>>>
>>> I'm not running a cluster as yours, but I don't think the issue is caused 
>>> by you using 2 APIs at the same time.
>>> IIRC the dash thing is append by S3 multipart upload, with a following 
>>> digit indicating the number of parts.
>>> You may want to check this reported in s3cmd community:
>>> https://sourceforge.net/p/s3tools/bugs/123/
>>>
>>> and some basic info from Amazon:
>>> http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
>>>
>>> Hope this helps :D
>>>
>>> Regards,
>>> ---Sandy
>>>
 -Original Message-
 From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
 Saverio Proto
 Sent: Monday, May 09, 2016 4:42 PM
 To: ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] RadosGW - Problems running the S3 and SWIFT API 
 at the same time

 I try to simplify the question to get some feedback.

 Is anyone running the RadosGW in production with S3 and SWIFT API active 
 at the same time ?

 thank you !

 Saverio


 2016-05-06 11:39 GMT+02:00 Saverio Proto :
 > Hello,
 >
 > We have been running the Rados GW with the S3 API and we did not have
 > problems for more than a year.
 >
 > We recently enabled also the SWIFT API for our users.
 >
 > radosgw --version
 > ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
 >
 > The idea is that each user of the system is free of choosing the S3
 > client or the SWIFT client to access the same container/buckets.
 >
 > Please tell us if this is possible by design or if we are doing 
 > something wrong.
 >
 > We have now a problem that some files wrote in the past with S3,
 > cannot be read with the SWIFT API because the md5sum always fails.
 >
 > I am able to reproduce the bug in this way:
 >
 > We have this file googlebooks-fre-all-2gram-20120701-ts.gz and we know
 > the correct md5 is 1c8113d2bd21232688221ec74dccff3a You can download
 > the same file here:
 > https://www.dropbox.com/s/auq16vdv2maw4p7/googlebooks-fre-all-2gram-20
 > 120701-ts.gz?dl=0
 >
 > rclone mkdir lss3:bugreproduce
 > rclone copy googlebooks-fre-all-2gram-20120701-ts.gz lss3:bugreproduce
 >
 > The file is successfully uploaded.
 >
 > At this point I can succesfully download again the file:
 > rclone copy lss3:bugreproduce/googlebooks-fre-all-2gram-20120701-ts.gz
 > test.gz
 >
 > but not with swift:
 >
 > swift download googlebooks-ngrams-gz
 > fre/googlebooks-fre-all-2gram-20120701-ts.gz
 > Error downloading object
 > 'googlebooks-ngrams-gz/fre/googlebooks-fre-all-2gram-20120701-ts.gz':
 > u'Error downloading fre/googlebooks-fre-all-2gram-20120701-ts.gz:
 > md5sum != etag, 1c8113d2bd21232688221ec74dccff3a !=
 > 1a209a31b4ac3eb923fac5e8d194d9d3-2'
 >
 > Also I found strange the dash character '-' at the end of the md5 that
 > is trying to compare.
 >
 > Of course upload a file with the swift client and redownloading the
 > same file just works.
 >
 > Should I open a bug for the radosgw on http://tracker.ceph.com/ ?
 >
 > thank you
 >
 > Saverio
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 

Re: [ceph-users] removing 'rados cppool' command

[Yehuda Sadeh-Weinraub]

On Fri, May 6, 2016 at 2:27 PM, Sage Weil <sw...@redhat.com> wrote:
> On Fri, 6 May 2016, Yehuda Sadeh-Weinraub wrote:
>> On Fri, May 6, 2016 at 12:41 PM, Sage Weil <sw...@redhat.com> wrote:
>> > This PR
>> >
>> > https://github.com/ceph/ceph/pull/8975
>> >
>> > removes the 'rados cppool' command.  The main problem is that the command
>> > does not make a faithful copy of all data because it doesn't preserve the
>> > snapshots (and snapshot related metadata).  That means if you copy an RBD
>> > pool it will render the images somewhat broken (snaps won't be present and
>> > won't work properly).  It also doesn't preserve the user_version field
>> > that some librados users may rely on.
>> >
>> > Since it's obscure and of limited use, this PR just removes it.
>> >
>> > Alternatively, we could add safeguards so that it refuses to make a copy
>> > if there are any selfmanaged_snaps, and/or generate some warnings.
>> >
>> > Any objections?
>>
>> I prefer the alternative. I found this command pretty useful for
>> testing config upgrade scenarios with rgw. After generating config
>> scenarios in older versions, I used this command to store the config
>> on another pool, and then I could get the different config whenever
>> needed.
>
> Keep in mind that all of these calls in rgw
>
> rgw/rgw_rados.cc:  epoch = ref.ioctx.get_last_version();
>
> may be subtley broken by cppool because user_version is not preserved...
>

Right, but these calls are done in the bucket index pool, not at the
root or metadata pools that are relevant to the system config. While
it may not be the perfect tool for everything, I still find it useful
from time to time.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] removing 'rados cppool' command

[Yehuda Sadeh-Weinraub]

On Fri, May 6, 2016 at 12:41 PM, Sage Weil  wrote:
> This PR
>
> https://github.com/ceph/ceph/pull/8975
>
> removes the 'rados cppool' command.  The main problem is that the command
> does not make a faithful copy of all data because it doesn't preserve the
> snapshots (and snapshot related metadata).  That means if you copy an RBD
> pool it will render the images somewhat broken (snaps won't be present and
> won't work properly).  It also doesn't preserve the user_version field
> that some librados users may rely on.
>
> Since it's obscure and of limited use, this PR just removes it.
>
> Alternatively, we could add safeguards so that it refuses to make a copy
> if there are any selfmanaged_snaps, and/or generate some warnings.
>
> Any objections?

I prefer the alternative. I found this command pretty useful for
testing config upgrade scenarios with rgw. After generating config
scenarios in older versions, I used this command to store the config
on another pool, and then I could get the different config whenever
needed.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW not start after upgrade to Jewel

[Yehuda Sadeh-Weinraub]

On Tue, Apr 26, 2016 at 6:50 AM, Abhishek Lekshmanan  wrote:
>
> Ansgar Jazdzewski writes:
>
>> Hi,
>>
>> After plaing with the setup i got some output that looks wrong
>>
>> # radosgw-admin zone get
>>
>> "placement_pools": [
>> {
>> "key": "default-placement",
>> "val": {
>> "index_pool": ".eu-qa.rgw.buckets.inde",
>> "data_pool": ".eu-qa.rgw.buckets.dat",
>> "data_extra_pool": ".eu-qa.rgw.buckets.non-e",
>> "index_type": 0
>> }
>> }
>> ],
>>
>> i think it sould be
>>
>> index_pool = .eu-qa.rgw.buckets.index.
>> data_pool = .eu-qa.rgw.buckets
>> data_extra_pool = .eu-qa.rgw.buckets.extra
>>
>> how can i fix it?
>
> Not sure how it reached this state, but given a zone get json, you can

There's an issue now when doing radosgw-admin zone set, and the pool
names start with a period (http://tracker.ceph.com/issues/15597). The
pool name is getting truncated by one character. We will have this
fixed for the next point release, but the workaround now would be to
add an extra character in each pool name before running the zone set
command.

Yehuda

> edit this and set it back using zone set for eg
> # radosgw-admin zone get > zone.json # now edit this file
> # radosgw-admin zone set --rgw-zone="eu-qa" < zone.json
>>
>> Thanks
>> Ansgar
>>
>> 2016-04-26 13:07 GMT+02:00 Ansgar Jazdzewski :
>>> Hi all,
>>>
>>> i got an answer, that pointed me to:
>>> https://github.com/ceph/ceph/blob/master/doc/radosgw/multisite.rst
>>>
>>> 2016-04-25 16:02 GMT+02:00 Karol Mroz :
 On Mon, Apr 25, 2016 at 02:23:28PM +0200, Ansgar Jazdzewski wrote:
> Hi,
>
> we test Jewel in our  QA environment (from Infernalis to Hammer) the
> upgrade went fine but the Radosgw did not start.
>
> the error appears also with radosgw-admin
>
> # radosgw-admin user info --uid="images" --rgw-region=eu --rgw-zone=eu-qa
> 2016-04-25 12:13:33.425481 7fc757fad900  0 error in read_id for id  :
> (2) No such file or directory
> 2016-04-25 12:13:33.425494 7fc757fad900  0 failed reading zonegroup
> info: ret -2 (2) No such file or directory
> couldn't init storage provider
>
> do i have to change some settings, also for upgrade of the radosgw?

 Hi,

 Testing a recent master build (with only default region and zone),
 I'm able to successfully run the command you specified:

 % ./radosgw-admin user info --uid="testid" --rgw-region=default 
 --rgw-zone=default
 ...
 {
 "user_id": "testid",
 "display_name": "M. Tester",
 ...
 }

 Are you certain the region and zone you specified exist?

 What do the following report:

 radosgw-admin zone list
 radosgw-admin region list

 --
 Regards,
 Karol
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> --
> Abhishek Lekshmanan
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 
> 21284 (AG Nürnberg)
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can Jewel read Hammer radosgw buckets?

[Yehuda Sadeh-Weinraub]

I managed to reproduce the issue, and there seem to be multiple
problems. Specifically we have an issue when upgrading a default
cluster that hasn't had a zone (and region) explicitly configured
before. There is another bug that I found
(http://tracker.ceph.com/issues/15597) that makes things even a bit
more complicated.

I created the following script that might be able to fix things for you:
https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone

For future reference, this script shouldn't be used if there are any
zones configured other than the default one. It also makes some ninja
patching to the zone config because of a bug that exists currently,
but will probably not apply to any next versions.

Please let me know if you have any issues, or if this actually does its magic.

Thanks,
Yehuda

On Mon, Apr 25, 2016 at 4:10 PM, Richard Chan
 wrote:
>
>> > How do you actually do that?
>>
>> What does 'radosgw-admin zone get' return?
>>
>> Yehuda
>
>
>
> [root@node1 ceph]# radosgw-admin zone get
> unable to initialize zone: (2) No such file or directory
>
> (I don't have any rgw configuration in /etc/ceph/ceph.conf; this is from a
> clean
>
> ceph-deploy rgw create node1
>
> ## user created under Hammer
> [root@node1 ceph]# radosgw-admin user info --uid=testuser
> 2016-04-26 07:07:06.159497 7f410c33ca40  0 RGWZoneParams::create(): error
> creating default zone params: (17) File exists
> could not fetch user info: no user info saved
>
> "rgw_max_chunk_size": "524288",
> "rgw_max_put_size": "5368709120",
> "rgw_override_bucket_index_max_shards": "0",
> "rgw_bucket_index_max_aio": "8",
> "rgw_enable_quota_threads": "true",
> "rgw_enable_gc_threads": "true",
> "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-rgw.node1",
> "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin",
> "rgw_cache_enabled": "true",
> "rgw_cache_lru_size": "1",
> "rgw_socket_path": "",
> "rgw_host": "",
> "rgw_port": "",
> "rgw_dns_name": "",
> "rgw_dns_s3website_name": "",
> "rgw_content_length_compat": "false",
> "rgw_script_uri": "",
> "rgw_request_uri": "",
> "rgw_swift_url": "",
> "rgw_swift_url_prefix": "swift",
> "rgw_swift_auth_url": "",
> "rgw_swift_auth_entry": "auth",
> "rgw_swift_tenant_name": "",
> "rgw_swift_account_in_url": "false",
> "rgw_swift_enforce_content_length": "false",
> "rgw_keystone_url": "",
> "rgw_keystone_admin_token": "",
> "rgw_keystone_admin_user": "",
> "rgw_keystone_admin_password": "",
> "rgw_keystone_admin_tenant": "",
> "rgw_keystone_admin_project": "",
> "rgw_keystone_admin_domain": "",
> "rgw_keystone_api_version": "2",
> "rgw_keystone_accepted_roles": "Member, admin",
> "rgw_keystone_token_cache_size": "1",
> "rgw_keystone_revocation_interval": "900",
> "rgw_keystone_verify_ssl": "true",
> "rgw_keystone_implicit_tenants": "false",
> "rgw_s3_auth_use_rados": "true",
> "rgw_s3_auth_use_keystone": "false",
> "rgw_ldap_uri": "ldaps:\/\/",
> "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com",
> "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com",
> "rgw_ldap_dnattr": "uid",
> "rgw_ldap_secret": "\/etc\/openldap\/secret",
> "rgw_s3_auth_use_ldap": "false",
> "rgw_admin_entry": "admin",
> "rgw_enforce_swift_acls": "true",
> "rgw_swift_token_expiration": "86400",
> "rgw_print_continue": "true",
> "rgw_remote_addr_param": "REMOTE_ADDR",
> "rgw_op_thread_timeout": "600",
> "rgw_op_thread_suicide_timeout": "0",
> "rgw_thread_pool_size": "100",
> "rgw_num_control_oids": "8",
> "rgw_num_rados_handles": "1",
> "rgw_nfs_lru_lanes": "5",
> "rgw_nfs_lru_lane_hiwat": "911",
> "rgw_nfs_fhcache_partitions": "3",
> "rgw_nfs_fhcache_size": "2017",
> "rgw_zone": "",
> "rgw_zone_root_pool": ".rgw.root",
> "rgw_default_zone_info_oid": "default.zone",
> "rgw_region": "",
> "rgw_default_region_info_oid": "default.region",
> "rgw_zonegroup": "",
> "rgw_zonegroup_root_pool": ".rgw.root",
> "rgw_default_zonegroup_info_oid": "default.zonegroup",
> "rgw_realm": "",
> "rgw_realm_root_pool": ".rgw.root",
> "rgw_default_realm_info_oid": "default.realm",
> "rgw_period_root_pool": ".rgw.root",
> "rgw_period_latest_epoch_info_oid": ".latest_epoch",
> "rgw_log_nonexistent_bucket": "false",
> "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
> "rgw_log_object_name_utc": "false",
> "rgw_usage_max_shards": "32",
> "rgw_usage_max_user_shards": "1",
> "rgw_enable_ops_log": "false",
> "rgw_enable_usage_log": "false",
> "rgw_ops_log_rados": "true",
> "rgw_ops_log_socket_path": "",
> "rgw_ops_log_data_backlog": "5242880",
> "rgw_usage_log_flush_threshold": "1024",
> "rgw_usage_log_tick_interval": "30",
> 

Re: [ceph-users] Can Jewel read Hammer radosgw buckets?

[Yehuda Sadeh-Weinraub]

(sorry for resubmission, adding ceph-users)

On Mon, Apr 25, 2016 at 9:47 AM, Richard Chan
 wrote:
> Hi Yehuda
>
> I created a test 3xVM setup with Hammer and one radosgw on the (separate)
> admin node; creating one user and buckets.
>
> I upgraded the VMs to jewel and created a new radosgw on one of the nodes.
>
> The object store didn't seem to survive the upgrade
>
> # radosgw-admin user info --uid=testuser
> 2016-04-26 00:41:50.713069 7fcdcc6fca40  0 RGWZoneParams::create(): error
> creating default zone params: (17) File exists
> could not fetch user info: no user info saved
>
> rados lspools
> rbd
> .rgw.root
> .rgw.control
> .rgw
> .rgw.gc
> .users.uid
> .users
> .rgw.buckets.index
> .rgw.buckets
> default.rgw.control
> default.rgw.data.root
> default.rgw.gc
> default.rgw.log
> default.rgw.users.uid
> default.rgw.users.keys
>
> Do I have to configure radosgw to use the pools with default.*?

No. Need to get it to play along nicely with the old pools.

> How do you actually do that?

What does 'radosgw-admin zone get' return?

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can Jewel read Hammer radosgw buckets?

[Yehuda Sadeh-Weinraub]

On Sat, Apr 23, 2016 at 6:22 AM, Richard Chan
 wrote:
> Hi Cephers,
>
> I upgraded to Jewel and noted the is massive radosgw multisite rework
> in the release notes.
>
> Can Jewel radosgw be configured to present existing Hammer buckets?
> On  a test system, jewel didn't recognise my Hammer buckets;
>
> Hammer used pools .rgw.*
> Jewel created by default: .rgw.root and default.rgw*
>
>
>

Yes, jewel should be able to read hammer buckets. If it detects that
there's an old config, it should migrate existing setup into the new
config. It seemsthat something didn't work as expected here. One way
to fix it would be to create a new zone and set its pools to point at
the old config's pools. We'll need to figure out what went wrong
though.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw bucket deletion woes

[Yehuda Sadeh-Weinraub]

On Tue, Mar 15, 2016 at 11:36 PM, Pavan Rallabhandi
 wrote:
> Hi,
>
> I find this to be discussed here before, but couldn¹t find any solution
> hence the mail. In RGW, for a bucket holding objects in the range of ~
> millions, one can find it to take for ever to delete the bucket(via
> radosgw-admin). I understand the gc(and its parameters) that would reclaim
> the space eventually, but am looking more at the bucket deletion options
> that can possibly speed up the operation.
>
> I realize, currently rgw_remove_bucket(), does it 1000 objects at a time,
> serially. Wanted to know if there is a reason(that am possibly missing and
> discussed) for this to be left that way, otherwise I was considering a
> patch to make it happen better.
>

There is no real reason. You might want to have a version of that
command that doesn't schedule the removal to gc, but rather removes
all the object parts by itself. Otherwise, you're just going to flood
the gc. You'll need to iterate through all the objects, and for each
object you'll need to remove all of it's rados objects (starting with
the tail, then the head). Removal of each rados object can be done
asynchronously, but you'll need to throttle the operations, not send
everything to the osds at once (which will be impossible, as the
objecter will throttle the requests anyway, which will lead to a high
memory consumption).

Thanks,
Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem: silently corrupted RadosGW objects caused by slow requests

[Yehuda Sadeh-Weinraub]

On Fri, Mar 4, 2016 at 7:26 AM, Ritter Sławomir
 wrote:
>> From: Robin H. Johnson [mailto:robb...@gentoo.org]
>> Sent: Friday, March 04, 2016 12:40 AM
>> To: Ritter Sławomir
>> Cc: ceph-us...@ceph.com; ceph-devel
>> Subject: Re: [ceph-users] Problem: silently corrupted RadosGW objects caused
>> by slow requests
>>
>> On Thu, Mar 03, 2016 at 01:55:13PM +0100, Ritter Sławomir wrote:
>> > Hi,
>> >
>> > I think this is really serious problem - again:
>> >
>> > - we silently lost S3/RGW objects in clusters
>> >
>> > Moreover, it our situation looks very similiar to described in
>> > uncorrected bug #13764 (Hammer) and in corrected #8269 (Dumpling).
>> FYI fix in #8269 _is_ present in Hammer:
>> commit bd8e026f88b rgw: don't allow multiple writers to same multiobject part
>>
>> --
>> Robin Hugh Johnson
>> Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
>> E-Mail : robb...@gentoo.org
>> GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
> Yes,
>
> fix for #8269 also has been included in our version: Dumpling 0.67.11.
> Guys from #13764 are using patched Hammer version

I didn't notice that you were actually running Dumpling (which we
haven't supported and backported fixes for a while). Here's one issue
that you might have hit:

http://tracker.ceph.com/issues/11604

Yehuda

>
> Both situations with corrupted files are very similiar to that described in 
> #8269.
> There was a problem with 2 threads writing to the same RADOS objects.
>
> Maybe there is another one uknown and specific exception to fix?
>
> Cheers,
> SR
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem: silently corrupted RadosGW objects caused by slow requests

[Yehuda Sadeh-Weinraub]

On Thu, Feb 25, 2016 at 7:17 AM, Ritter Sławomir
 wrote:
> Hi,
>
>
>
> We have two CEPH clusters running on Dumpling 0.67.11 and some of our
> "multipart objects" are incompleted. It seems that some slow requests could
> cause corruption of related S3 objects. Moveover GETs for that objects are
> working without any error messages. There are only HTTP 200 in logs as well
> as no information about problems from popular client tools/libs.
>
>
>
> The situation looks very similiar to described in bug #8269, but we are
> using fixed 0.67.11 version:  http://tracker.ceph.com/issues/8269
>
>
>
> Regards,
>
>
>
> Sławomir Ritter
>
>
>
>
>
>
>
> EXAMPLE#1
>
>
>
> slow_request
>
> 
>
> 2016-02-23 13:49:58.818640 osd.260 10.176.67.27:6800/688083 2119 : [WRN] 4
> slow requests, 4 included below; oldest blocked for > 30.727096 secs
>
> 2016-02-23 13:49:58.818673 osd.260 10.176.67.27:6800/688083 2120 : [WRN]
> slow request 30.727096 seconds old, received at 2016-02-23 13:49:28.091460:
> osd_op(c
>
> lient.47792965.0:185007087
> default.14654.445__shadow_c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv.b73N3OmW4OhCjDYSR-RTkZNNIKA1C9Z.57_2
> [writef
>
> ull 0~524288] 10.ce729ebe e107594) v4 currently waiting for subops from
> [469,9]
>

Did these requests ever finish?

>
>
>
>
> HTTP_500 in apache.log
>
> ==
>
> 127.0.0.1 - - [23/Feb/2016:13:49:27 +0100] "PUT
> /video-shbc/c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv?uploadId=b73N3OmW4OhCjDYSR-RTkZNNIKA1C9Z=56
> HTTP/1.0" 200 221 "-" "Boto/2.31.1 Python/2.7.3
> Linux/3.13.0-39-generic(syncworker)"
>
> 127.0.0.1 - - [23/Feb/2016:13:49:28 +0100] "PUT
> /video-shbc/c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv?uploadId=b73N3OmW4OhCjDYSR-RTkZNNIKA1C9Z=57
> HTTP/1.0" 500 751 "-" "Boto/2.31.1 Python/2.7.3
> Linux/3.13.0-39-generic(syncworker)"
>
> 127.0.0.1 - - [23/Feb/2016:13:49:58 +0100] "PUT
> /video-shbc/c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv?uploadId=b73N3OmW4OhCjDYSR-RTkZNNIKA1C9Z=57
> HTTP/1.0" 200 221 "-" "Boto/2.31.1 Python/2.7.3
> Linux/3.13.0-39-generic(syncworker)"
>
> 127.0.0.1 - - [23/Feb/2016:13:49:59 +0100] "PUT
> /video-shbc/c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv?uploadId=b73N3OmW4OhCjDYSR-RTkZNNIKA1C9Z=58
> HTTP/1.0" 200 221 "-" "Boto/2.31.1 Python/2.7.3
> Linux/3.13.0-39-generic(syncworker)"
>
>
>
>
>
> Empty RADOS object (real size = 0 bytes), list generated basis on MANIFEST
>
> ==
>
> found
> default.14654.445__shadow_c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv.b73N3OmW4OhCjDYSR-RTkZNNIKA1C9Z.56_2
> 2097152   ok  2097152   10.7acc9476 (10.1476) [278,142,436]
> [278,142,436]
>
> found
> default.14654.445__multipart_c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv.b73N3OmW4OhCjDYSR-RTkZNNIKA1C9Z.57
> 0 diff4194304   10.4f5be025 (10.25)   [57,310,428]
> [57,310,428]
>
> found
> default.14654.445__shadow_c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv.b73N3OmW4OhCjDYSR-RTkZNNIKA1C9Z.57_1
> 4194304   ok  4194304   10.81191602 (10.1602) [441,109,420]
> [441,109,420]
>
> found
> default.14654.445__shadow_c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv.b73N3OmW4OhCjDYSR-RTkZNNIKA1C9Z.57_2
> 2097152   ok  2097152   10.ce729ebe (10.1ebe) [260,469,9]
> [260,469,9]
>
>
>
>
>
> "Silent" GETs
>
> =
>
> # object size from headers
>
> $ s3 -u head
> video-shbc/c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv
> Content-Type: binary/octet-stream
>
> Content-Length: 641775701
>
> Server: nginx
>
>
>
> # but GETs only 637581397 (641775701 - missing 4194304 = 637581397)
>
> $ s3 -u get
> video-shbc/c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv >
> /tmp/test
>
> $  ls -al /tmp/test
>
> -rw-r--r-- 1 root root 637581397 Feb 23 17:05 /tmp/test
>
>
>
> # no error in logs
>
> 127.0.0.1 - - [23/Feb/2016:17:05:00 +0100] "GET
> /video-shbc/c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv
> HTTP/1.0" 200 637581711 "-" "Mozilla/4.0 (Compatible; s3; libs3 2.0; Linux
> x86_64)"
>
>
>
> # wget - retry for missing part, but there is no missing part, so it GETs
> head/tail of the file again
>
> $ wget
> http://127.0.0.1:88/video-shbc/c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv
>
> --2016-02-23 17:10:11--
> http://127.0.0.1:88/video-shbc/c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv
>
> Connecting to 127.0.0.1:88... connected.
>
> HTTP request sent, awaiting response... 200 OK
>
> Length: 641775701 (612M) [binary/octet-stream]
>
> Saving to: `c9f8db1b-cee2-4ec8-8fb3-8b4bc7585d80.1456231572.877051.ismv'
>
>
>
> 99%
> [==>
> ] 637,581,397 63.9M/s   in 9.5s
>
>
>
> 2016-02-23 17:10:20 

Re: [ceph-users] radosgw flush_read_list(): d->client_c->handle_data() returned -5

[Yehuda Sadeh-Weinraub]

On Wed, Feb 24, 2016 at 5:48 PM, Ben Hines  wrote:
> Any idea what is going on here? I get these intermittently, especially with
> very large file.
>
> The client is doing RANGE requests on this >51 GB file, incrementally
> fetching later chunks.
>
> 2016-02-24 16:30:59.669561 7fd33b7fe700  1 == starting new request
> req=0x7fd32c0879c0 =
> 2016-02-24 16:30:59.669675 7fd33b7fe700  2 req 3648804:0.000114::GET
> //int8-0.181.4-1654016.2016-02-23_03-53-42.pkg::initializing for
> trans_id = tx00037ad24-0056ce4b43-259914b-default
> 2016-02-24 16:30:59.669687 7fd33b7fe700 10 host=
> 2016-02-24 16:30:59.669757 7fd33b7fe700 10
> s->object=/int8-0.181.4-1654016.2016-02-23_03-53-42.pkg
> s->bucket=
> 2016-02-24 16:30:59.669767 7fd33b7fe700  2 req 3648804:0.000206:s3:GET
> //int8-0.181.4-1654016.2016-02-23_03-53-42.pkg::getting op
> 2016-02-24 16:30:59.669776 7fd33b7fe700  2 req 3648804:0.000215:s3:GET
> //int8-0.181.4-1654016.2016-02-23_03-53-42.pkg:get_obj:authorizing
> 2016-02-24 16:30:59.669785 7fd33b7fe700  2 req 3648804:0.000224:s3:GET
> //int8-0.181.4-1654016.2016-02-23_03-53-42.pkg:get_obj:reading
> permissions
> 2016-02-24 16:30:59.673797 7fd33b7fe700 10 manifest: total_size =
> 50346000384
> 2016-02-24 16:30:59.673841 7fd33b7fe700  2 req 3648804:0.004280:s3:GET
> //int8-0.181.4-1654016.2016-02-23_03-53-42.pkg:get_obj:init op
> 2016-02-24 16:30:59.673867 7fd33b7fe700 10 cache get:
> name=.users.uid+ : hit
> 2016-02-24 16:30:59.673881 7fd33b7fe700 10 cache get:
> name=.users.uid+ : hit
> 2016-02-24 16:30:59.673921 7fd33b7fe700  2 req 3648804:0.004360:s3:GET
> //int8-0.181.4-1654016.2016-02-23_03-53-42.pkg:get_obj:verifying
> op mask
> 2016-02-24 16:30:59.673929 7fd33b7fe700  2 req 3648804:0.004369:s3:GET
> //int8-0.181.4-1654016.2016-02-23_03-53-42.pkg:get_obj:verifying
> op permissions
> 2016-02-24 16:30:59.673941 7fd33b7fe700  5 Searching permissions for
> uid=anonymous mask=49
> 2016-02-24 16:30:59.673944 7fd33b7fe700  5 Permissions for user not found
> 2016-02-24 16:30:59.673946 7fd33b7fe700  5 Searching permissions for group=1
> mask=49
> 2016-02-24 16:30:59.673949 7fd33b7fe700  5 Found permission: 1
> 2016-02-24 16:30:59.673951 7fd33b7fe700  5 Searching permissions for group=2
> mask=49
> 2016-02-24 16:30:59.673953 7fd33b7fe700  5 Permissions for group not found
> 2016-02-24 16:30:59.673955 7fd33b7fe700  5 Getting permissions id=anonymous
> owner= perm=1
> 2016-02-24 16:30:59.673957 7fd33b7fe700 10  uid=anonymous requested perm
> (type)=1, policy perm=1, user_perm_mask=15, acl perm=1
> 2016-02-24 16:30:59.673961 7fd33b7fe700  2 req 3648804:0.004400:s3:GET
> //int8-0.181.4-1654016.2016-02-23_03-53-42.pkg:get_obj:verifying
> op params
> 2016-02-24 16:30:59.673965 7fd33b7fe700  2 req 3648804:0.004404:s3:GET
> //int8-0.181.4-1654016.2016-02-23_03-53-42.pkg:get_obj:executing
> 2016-02-24 16:30:59.674107 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=130023424 stripe_ofs=130023424 part_ofs=104857600
> rule->part_size=52428800
> 2016-02-24 16:30:59.674193 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=134217728 stripe_ofs=134217728 part_ofs=104857600
> rule->part_size=52428800
> 2016-02-24 16:30:59.674317 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=138412032 stripe_ofs=138412032 part_ofs=104857600
> rule->part_size=52428800
> 2016-02-24 16:30:59.674433 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=142606336 stripe_ofs=142606336 part_ofs=104857600
> rule->part_size=52428800
> 2016-02-24 16:31:00.046110 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=146800640 stripe_ofs=146800640 part_ofs=104857600
> rule->part_size=52428800
> 2016-02-24 16:31:00.150966 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=150994944 stripe_ofs=150994944 part_ofs=104857600
> rule->part_size=52428800
> 2016-02-24 16:31:00.151118 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=155189248 stripe_ofs=155189248 part_ofs=104857600
> rule->part_size=52428800
> 2016-02-24 16:31:00.161000 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=157286400 stripe_ofs=157286400 part_ofs=157286400
> rule->part_size=52428800
> 2016-02-24 16:31:00.199553 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=161480704 stripe_ofs=161480704 part_ofs=157286400
> rule->part_size=52428800
> 2016-02-24 16:31:00.278308 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=165675008 stripe_ofs=165675008 part_ofs=157286400
> rule->part_size=52428800
> 2016-02-24 16:31:00.312306 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=169869312 stripe_ofs=169869312 part_ofs=157286400
> rule->part_size=52428800
> 2016-02-24 16:31:00.751626 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=174063616 stripe_ofs=174063616 part_ofs=157286400
> rule->part_size=52428800
> 2016-02-24 16:31:00.833570 7fd33b7fe700  0 RGWObjManifest::operator++():
> result: ofs=178257920 stripe_ofs=178257920 part_ofs=157286400
> 

Re: [ceph-users] Problem create user RGW

[Yehuda Sadeh-Weinraub]

try running:

$ radosgw-admin --name client.rgw.servergw001 metadata list user


Yehuda

On Wed, Feb 24, 2016 at 8:41 AM, Andrea Annoè  wrote:
> I don’t see any user create in RGW
>
>
>
> sudo radosgw-admin metadata list user
>
> [
>
> ]
>
>
>
>
>
> sudo radosgw-admin user create --uid="user1site1" --display-name="User test
> replica site1" --name client.rgw.servergw001 --access-key=user1site1
> --secret=pwd1
>
> {
>
> "user_id": "user1site1",
>
> "display_name": "User test replica site1",
>
> "email": "",
>
> "suspended": 0,
>
> "max_buckets": 1000,
>
> "auid": 0,
>
> "subusers": [],
>
> "keys": [
>
> {
>
> "user": "user1site1",
>
> "access_key": "user1site1",
>
> "secret_key": "pwd1"
>
> }
>
> ],
>
> "swift_keys": [],
>
> "caps": [],
>
> "op_mask": "read, write, delete",
>
> "default_placement": "",
>
> "placement_tags": [],
>
> "bucket_quota": {
>
> "enabled": false,
>
> "max_size_kb": -1,
>
> "max_objects": -1
>
> },
>
> "user_quota": {
>
> "enabled": false,
>
> "max_size_kb": -1,
>
> "max_objects": -1
>
> },
>
> "temp_url_keys": []
>
> }
>
>
>
> sudo radosgw-admin metadata list user
>
> [
>
> ]
>
>
>
>
>
> The list of user don’t change… what’s the problem? Command, keyring… ??
>
> The command for create user don’t report error if I try to retry more time.
>
>
>
> Please help me.
>
>
>
> Best regards.
>
> Andrea
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Idea for speedup RadosGW for buckets with many objects.

[Yehuda Sadeh-Weinraub]

On Wed, Feb 17, 2016 at 12:51 PM, Krzysztof Księżyk  wrote:
> Hi,
>
> I'm experiencing problem with poor performance of RadosGW while operating on
> bucket with many object. That's known issue with LevelDB and can be
> partially resolved using shrading but I have one more idea. As I see in ceph
> osd logs all slow requests are while making call to rgw.bucket_list:
>
> 2016-02-17 03:17:56.846694 7f5396f63700  0 log_channel(cluster) log [WRN] :
> slow request 30.272904 seconds old, received at 2016-02-17 03:17:26.573742:
> osd_op(client.12611484.0:15137332 .dir.default.4162.3 [call rgw.bucket_list]
> 9.2955279 ack+read+known_if_redirected e3252) currently started
>
> I don't know exactly how Ceph internally works but maybe data required to
> return results for rgw.bucket_list could be cached for some time. Cache TTL
> would be parametrized and could be disabled to keep the same behaviour as
> current one. There can be 3 cases when there's a call to rgw.bucket_list:
> 1. no cached data
> 2. up-to-date cache
> 3. outdated cache
>
> Ad 1. First call starts generating full list. All new requests are put on
> hold. When list is ready it's saved to cache
> Ad 2. All calls are served from cache
> Ad 3. First request starts generating full list. All new requests are served
> from outdated cache until new cached data is ready
>
> This can be even optimized by periodically generating fresh cache, even if
> it's not expired yet to reduce cases when cache is outdated.

Where is the cache going to live in? Note that for it to be on rgw, it
will need to be shared among all rgw instances (serving the same
zone). On the other hand, I'm not exactly sure how the osd could cache
it (there's not mechanism at the moment that would allow that). And
the cache itself will need to be part of the osd that serves the
specific bucket index, otherwise you'd need to go to multiple osds for
that operation, which will slow down things for the general case.
Note that we need for things to be durable, otherwise we might end up
with inconsistencies when things don't go as expected (e.g., when rgw
/ osd went down).

We did some thinking recently around the bucket index area, see how
things can be improved. One way would be (for some use cases) to drop
it altogether. This could work in environment where 1. you don't need
to list objects in the bucket, and 2. no multi-zone sync. Another
possible mechanisms would be to relax the bucket index update, and
replace it with some kind of a lazy update (maybe similar to what you
suggested), and some way to rebuild the index out of the raw pool data
(maybe combining it with rados namespace).

>
> Maybe this idea is stupid, maybe not, but if it's doable it would be nice to
> have choice.

Thanks for the suggestions!

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw anonymous write

[Yehuda Sadeh-Weinraub]

On Tue, Feb 9, 2016 at 5:15 AM, Jacek Jarosiewicz
 wrote:
> Hi list,
>
> My setup is: ceph 0.94.5, ubuntu 14.04, tengine (patched nginx).
>
> I'm trying to migrate from our old file storage (MogileFS) to the new ceph
> radosgw. The problem is that the old storage had no access control - no
> authorization, so the access to read and/or write was controlled by the web
> server (ie per IP/network).
>
> I want to keep the clients using old storage, but get rid of the MogileFS so
> I don't have to maintain two different storage solutions.
>
> Basically MogileFS http API is similar to S3, except for the authorization
> part - so the methods are the same (PUT, GET, DELETE..).
>
> I've created a bucket with public-read-write access and tried to connect
> MogileFS client to it - the uploads work fine, and the files get acl
> public-read so are readable, but they don't have an owner.
>
> So after upload I can't manage them (ie modify acl) - I can only remove
> objects.
>
> Is there a way to force files that are uploaded anonymously to have an
> owner? Is there a way maybe to have them inherit owner from the bucket?
>

Currently there's no way to change it. I'm not sure though that we're
doing the correct thing. Did you try it with Amazon S3 by any chance?

> Cheers,
> J
>
> --
> Jacek Jarosiewicz
> Administrator Systemów Informatycznych
>
> 
> SUPERMEDIA Sp. z o.o. z siedzibą w Warszawie
> ul. Senatorska 13/15, 00-075 Warszawa
> Sąd Rejonowy dla m.st.Warszawy, XII Wydział Gospodarczy Krajowego Rejestru
> Sądowego,
> nr KRS 029537; kapitał zakładowy 42.756.000 zł
> NIP: 957-05-49-503
> Adres korespondencyjny: ul. Jubilerska 10, 04-190 Warszawa
>
> 
> SUPERMEDIA ->   http://www.supermedia.pl
> dostep do internetu - hosting - kolokacja - lacza - telefonia
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW: oddity when creating users via admin api

[Yehuda Sadeh-Weinraub]

On Wed, Jan 27, 2016 at 4:20 PM, seapasu...@uchicago.edu
 wrote:
> So when I create a new user with the admin api. If the user already exists
> it just generates a new keypair for that user. Shouldn't the admin api
> report that the user already exists? I ask because I can end up with
> multiple keypairs for the same user unintentionally which could be an issue.
> I was not sure if this was a feature or a bug so I thought I would ask here
> prior to filing a bug.

It's definitely a bug. But note that it sounds familiar, and we might
have already fixed it for the next major version.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW :: bucket quota not enforced below 1

[Yehuda Sadeh-Weinraub]

On Wed, Jan 27, 2016 at 4:18 PM, seapasu...@uchicago.edu
 wrote:
> if you set a RGW user to have abucket quota of 0 buckets you can still
> create buckets. The only way I have found to prevent a user from being able
> to create buckets is to set the op_mask to read. 1.) it looks like
> bucket_policy is not enforced when you have it set to anything below 1. It
> looks like the only way to prevent a user from creating buckets is to set
> the op_mask but this is not documented. How would I set the op_mask via the
> radosgw admin api? I keep getting a 200 success code but the op_mask of the
> user stays the same.
>
> relavent pastebins:
> http://pastebin.com/Rbzdy52c -- shows user info with bucket quota set but
> shows ability to create buckets.
> http://pastebin.com/J9K3dgdF -- shows inability to set op_mask from admin
> api (that or I don't know how)
>
>
> 1.) does anyone know how to set the op_mask via the admin api? 2.) why can I
> create what seems like an infinite amount of buckets when my bucket quota is
> set to 0 objects and 0 size? Shouldn't it be enforced for anything above -1?

That's not bucket quota, that's the user's max_buckets param. When
this value is set to '0' it means the user has no limit on the number
of the buckets. Sadly due to backward compatibility issue, having 0
mean something else is a bit problematic. We can probably add a new
bool param that will specify whether bucket creation is allowed at
all.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 411 Content-Length required error

[Yehuda Sadeh-Weinraub]

On Wed, Jan 27, 2016 at 3:31 PM, John Hogenmiller  wrote:
> I did end up switching to civetweb and I also found that rgw content length
> compat, which I set to true. I am still getting the 411 Length required
> issue.
>
> I have had more discussions with our testing team, and I am still trying to
> ascertain how valid this issue is.
>
> With AWS Sig v4, you use a different method to do chunked transfers. With
> the sigv2, you do it as a "Transfer-Encoding: chunked" (as detailed in my
> s3curl example).  However, that v2 method may only apply to the
> implementation we have have (we have a proprietary implementation of s3 that
> I am hoping to replace with Ceph, if I can match our acceptance testing).
>
> The reason I think that this is a valid issue is because of this commit
>
> http://tracker.ceph.com/projects/ceph/repository/revisions/14fa77d9277b5ef5d0c6683504b368773b39ccc4
>
>> Fixes: #2878
>> We now allow complete multipart upload to use chunked encoding
>> when sending request data. With chunked encoding the HTTP_LENGTH
>> header is not required.
>
>
> What I would like to see is the test code for this (ideally in a curl or
> s3curl format) so that I can compare locally to see if we're saying the same
> thing, or if that commit from 3 years ago is still valid.
>

I don't think it's related. Try bumping up the rgw debug log, (debug
rgw = 20), and see what are the http header fields that are being sent
for the specific request. It could be that apache is not passing on
the Transfer-Encoding header, or does something else to it.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How-to doc: hosting a static website on radosgw

[Yehuda Sadeh-Weinraub]

On Tue, Jan 26, 2016 at 2:37 PM, Florian Haas  wrote:
> On Tue, Jan 26, 2016 at 8:56 PM, Wido den Hollander  wrote:
>> On 01/26/2016 08:29 PM, Florian Haas wrote:
>>> Hi everyone,
>>>
>>> we recently worked a bit on running a full static website just on
>>> radosgw (akin to
>>> http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html),
>>> and didn't find a good how-to writeup out there. So we did a bit of
>>> fiddling with radosgw and HAproxy, and wrote one:
>>> https://www.hastexo.com/resources/hints-and-kinks/hosting-website-radosgw/#.VqfGx99vFhG
>>>
>>> Hopefully some of you find this useful. If you spot errors or
>>> omissions, just let us know in the comments at the bottom of the page.
>>> Thanks!
>>>
>>
>> Thanks!
>>
>> Were you aware of this work going on:
>> https://github.com/ceph/ceph/tree/wip-static-website
>>
>> This might be in the RADOS Gateway soon and then you don't need HAProxy
>> anymore.
>
> The moment this lands in a release, we'll be more than happy to ditch
> the HAProxy request/response mangling bits. But that WIP branch hasn't
> seen commits in 4 months, so we took it as an exercise in coming up

Here's a more up-to-date branch:

https://github.com/ceph/ceph/tree/wip-rgw-static-website-yehuda

We're currently testing it, and the plan is to get it in before jewel.
One caveat though, the error page handling still has some issues so at
the moment so the feature will be disabled by default for now.

Yehuda

> with something workable as an interim solution. :)
>
> Cheers,
> Florian
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com