Re: [ceph-users] radosgw multizone not syncing large bucket completly to other zone

[Orit Wasserman]

Hi Enrico,

On Fri, Jun 29, 2018 at 7:50 PM Enrico Kern  wrote:

> hmm that also pops up right away when i restart all radosgw instances. But
> i will check further and see if i can find something. Maybe doing the
> upgrade to mimic too.
>
> That bucket is basically under load on the master zone all the time as we
> use it as historical storage for druid, so there is constantly data written
> to it. I just dont get why disabling/enabling sync on the bucket flawless
> syncs everything while if i just keep it enabled it stops syncing at all.
> For the last days i was just running disabling/enabling for the bucket in a
> while loop with 30 minute interval, but thats no persistent fix ;)
>
>
Are you using Haproxy? we have seen sync stales with it.
The simplest work around is to configure the radosgw's addresses as the
sync endpoints not the haproxy's.

Regards,
Orit



> On Fri, Jun 29, 2018 at 6:15 PM Yehuda Sadeh-Weinraub 
> wrote:
>
>>
>>
>> On Fri, Jun 29, 2018 at 8:48 AM, Enrico Kern 
>> wrote:
>>
>>> also when i try to sync the bucket manual i get this error:
>>>
>>> ERROR: sync.run() returned ret=-16
>>> 2018-06-29 15:47:50.137268 7f54b7e4ecc0  0 data sync: ERROR: failed to
>>> read sync status for
>>> bucketname:6a9448d2-bdba-4bec-aad6-aba72cd8eac6.27150814.1
>>>
>>> it works flawless with all other buckets.
>>>
>>
>> error 16 is EBUSY: meaning it can't take a lease to do work on the
>> bucket. This usually happens when another entity (e.g., a running radosgw
>> process) is working on it at the same time. Either something took the lease
>> and never gave it back (leases shouldn't be indefinite, usually are being
>> taken for a short period but are renewed periodically), or there might be
>> some other bug related to the lease itself. I would start by first figuring
>> out whether it's the first case or the second one. On the messenger log
>> there should be a message prior to that that shows the operation that got
>> the -16 as a response (should have something like "...=-16 (Device or
>> resource busy)" in it). The same line would also contain the name of the
>> rados object that is used to manage the lease. Try to look at the running
>> radosgw log at the same time when this happens, and check whether there are
>> other operations on that object.
>> One thing to note is that if you run a sync on a bucket and stop it
>> uncleanly in the middle (e.g., like killing the process), the leak will
>> stay locked for a period of time (Something in the order of 1 to 2 minutes).
>>
>> Yehuda
>>
>>>
>>>
>>> On Fri, Jun 29, 2018 at 5:39 PM Enrico Kern 
>>> wrote:
>>>
 Hello,

 thanks for the reply.

 We have around 200k objects in the bucket. It is not automatic
 resharded (is that even supported in multisite?)

 What i see when i run a complete data sync with the debug logs after a
 while i see alot of informations that it is unable to perform some log and
 also some device or resource busy (also with alot of different osds,
 restarting the osds also doesnt make this error going away):


 018-06-29 15:18:30.391085 7f38bf882cc0 20
 cr:s=0x55de55700b20:op=0x55de55717010:20RGWContinuousLeaseCR: couldn't lock
 amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.59:sync_lock:
 retcode=-16

 2018-06-29 15:18:30.391094 7f38bf882cc0 20
 cr:s=0x55de55732750:op=0x55de5572d970:20RGWContinuousLeaseCR: couldn't lock
 amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.10:sync_lock:
 retcode=-16

 2018-06-29 15:22:01.618744 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604
 <== osd.43 10.30.3.44:6800/29982 13272  osd_op_reply(258628
 datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.52 [call]
 v14448'24265315 uv24265266 ondisk = -16 ((16) Device or resource busy)) v8
  209+0+0 (2379682838 0 0) 0x7f38a8005110 con 0x7f3868003380

 2018-06-29 15:22:01.618829 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604
 <== osd.43 10.30.3.44:6800/29982 13273  osd_op_reply(258629
 datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.105 [call]
 v14448'24265316 uv24265256 ondisk = -16 ((16) Device or resource busy)) v8
  210+0+0 (4086289880 0 0) 0x7f38a8005110 con 0x7f3868003380


 There are no issues with the OSDs all other stuff in the cluster works
 (rbd, images to openstack etc.)


 Also that command with appending debug never finishes.

 On Tue, Jun 26, 2018 at 5:45 PM Yehuda Sadeh-Weinraub <
 yeh...@redhat.com> wrote:

>
>
> On Sun, Jun 24, 2018 at 12:59 AM, Enrico Kern <
> enrico.k...@glispamedia.com> wrote:
>
>> Hello,
>>
>> We have two ceph luminous clusters (12.2.5).
>>
>> recently one of our big buckets stopped syncing properly. We have a
>> one specific bucket which is around 30TB in size consisting of alot of
>> directories 

Re: [ceph-users] Infinite loop in radosgw-usage show

[Orit Wasserman]

Hi,

On Tue, Feb 6, 2018 at 10:04 AM, Ingo Reimann  wrote:
> Just to add -
>
> We wrote a little wrapper, that reads the output of "radosgw-admin usage
> show" and stops, when the loop starts. When we add all entries by
> ourselves, the result is correct. Moreover - the duplicate timestamp, that
> we detect to break the loop, is not the last taken into  account. Eg:
> ./radosgw-admin-break-loop --uid=TestUser --start-date=2017-12-01
> --end-date=2018-01-01 [...]
> "bytes_received": 1472051975516, Loop detected at
> "2017-12-21 08:00:00.00Z"
>
> ./radosgw-admin-break-loop --uid=TestUser --start-date=2017-12-01
> --end-date=2017-12-22 [...]
> "bytes_received": 1245051973424, Loop detected at
> "2017-12-21 08:00:00.00Z"
>
>  This leads to the assumption, that the loop occurs after processing of
> raw data.
>
> Looks like a bug?

Yes :(
Please open a tracker issue please include your ceph version, special
configuration and logs.

Thanks,
Orit

>
> Best regards,
>
> Ingo Reimann
> Dunkel GmbH
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW default.rgw.meta pool

[Orit Wasserman]

On Mon, Feb 5, 2018 at 12:45 PM, Thomas Bennett  wrote:
> Hi,
>
> In trying to understand RGW pool usage I've noticed the pool called
> default.rgw.meta pool has a large number of objects in it. Suspiciously
> about twice as many objects in my default.rgw.buckets.index pool.
>
> As I delete and add buckets, the number of objects in both pools decrease
> and increase proportionally.
>
> However when I try to list the objects in the default.rgw.meta pool, it
> returns nothing.
>
> I.e  'rados -p default.rgw.buckets.index ls' returns nothing.
>
> Is this expected behaviour for this pool?
>
> What are all those objects and why can I not list them?
>

default.rgw.bucket.index store the bucket index objects which are omap objects.
You cannot see the omap size using rados ls but need to use rados omap commands.
You can use this script to calculate the bucket index size:
https://github.com/mkogan1/ceph-utils/blob/master/scripts/get_omap_kv_size.sh

> From my understanding default.rgw.buckets.index should contain thinks like:
you probably meant default.rgw.meta.
It is a namespace not a pool try using:
rados ls -p default.rgw.meta --all

Regards,
Orit

> domain_root, user_keys_pool, user_email_pool, user_swift_pool,
> user_uid_pool.
>


> Cheers,
> Tom
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bug in RadosGW resharding? Hangs again...

[Orit Wasserman]

On Wed, Jan 17, 2018 at 3:16 PM, Martin Emrich <martin.emr...@empolis.com>
wrote:

> Hi!
>
>
>
> I created a tracker ticket: http://tracker.ceph.com/issues/22721
>
> It also happens without a lifecycle rule (only versioning).
>
>
>
Thanks.


> I collected a log from the resharding process, after 10 minutes I canceled
> it. Got 500MB log (gzipped still 20MB), so I cannot upload it to the bug
> tracker.
>
>
>
I will try to reproduce it on my setup it should be simpler now I am sure
it is the versioning.

Orit


> Regards,
>
>
>
> Martin
>
>
>
>
>
> *Von: *Orit Wasserman <owass...@redhat.com>
> *Datum: *Mittwoch, 17. Januar 2018 um 11:57
>
> *An: *Martin Emrich <martin.emr...@empolis.com>
> *Cc: *ceph-users <ceph-users@lists.ceph.com>
> *Betreff: *Re: [ceph-users] Bug in RadosGW resharding? Hangs again...
>
>
>
>
>
>
>
> On Wed, Jan 17, 2018 at 11:45 AM, Martin Emrich <martin.emr...@empolis.com>
> wrote:
>
> Hi Orit!
>
>
>
> I did some tests, and indeed the combination of Versioning/Lifecycle with
> Resharding is the problem:
>
>
>
>- If I do not enable Versioning/Lifecycle, Autoresharding works fine.
>- If I disable Autoresharding but enable Versioning+Lifecycle, pushing
>data works fine, until I manually reshard. This hangs also.
>
>
>
> Thanks for testing :) This is very helpful!
>
>
>
> My lifecycle rule (which shall remove all versions older than 60 days):
>
>
>
> {
>
> "Rules": [{
>
> "Status": "Enabled",
>
> "Prefix": "",
>
> "NoncurrentVersionExpiration": {
>
> "NoncurrentDays": 60
>
> },
>
> "Expiration": {
>
> "ExpiredObjectDeleteMarker": true
>
> },
>
> "ID": "expire-60days"
>
> }]
>
> }
>
>
>
> I am currently testing with an application containing customer data, but I
> am also creating some random test data to create logs I can share.
>
> I will also test whether the versioning itself is the culprit, or if it is
> the lifecycle rule.
>
>
>
>
>
> I am suspecting versioning (never tried it with resharding).
>
> Can you open a tracker issue with all the information?
>
>
>
> Thanks,
>
> Orit
>
>
>
> Regards,
>
> Martin
>
>
>
> *Von: *Orit Wasserman <owass...@redhat.com>
> *Datum: *Dienstag, 16. Januar 2018 um 18:38
> *An: *Martin Emrich <martin.emr...@empolis.com>
> *Cc: *ceph-users <ceph-users@lists.ceph.com>
> *Betreff: *Re: [ceph-users] Bug in RadosGW resharding? Hangs again...
>
>
>
> Hi Martin,
>
>
>
> On Mon, Jan 15, 2018 at 6:04 PM, Martin Emrich <martin.emr...@empolis.com>
> wrote:
>
> Hi!
>
> After having a completely broken radosgw setup due to damaged buckets, I
> completely deleted all rgw pools, and started from scratch.
>
> But my problem is reproducible. After pushing ca. 10 objects into a
> bucket, the resharding process appears to start, and the bucket is now
> unresponsive.
>
>
>
> Sorry to hear that.
>
> Can you share radosgw logs with --debug_rgw=20 --debug_ms=1?
>
>
>
> I just see lots of these messages in all rgw logs:
>
> 2018-01-15 16:57:45.108826 7fd1779b1700  0 block_while_resharding ERROR:
> bucket is still resharding, please retry
> 2018-01-15 16:57:45.119184 7fd1779b1700  0 NOTICE: resharding operation on
> bucket index detected, blocking
> 2018-01-15 16:57:45.260751 7fd1120e6700  0 block_while_resharding ERROR:
> bucket is still resharding, please retry
> 2018-01-15 16:57:45.280410 7fd1120e6700  0 NOTICE: resharding operation on
> bucket index detected, blocking
> 2018-01-15 16:57:45.300775 7fd15b979700  0 block_while_resharding ERROR:
> bucket is still resharding, please retry
> 2018-01-15 16:57:45.300971 7fd15b979700  0 WARNING: set_req_state_err
> err_no=2300 resorting to 500
> 2018-01-15 16:57:45.301042 7fd15b979700  0 ERROR: 
> RESTFUL_IO(s)->complete_header()
> returned err=Input/output error
>
> One radosgw process and two OSDs housing the bucket index/metadata are
> still busy, but it seems to be stuck again.
>
> How long is this resharding process supposed to take? I cannot believe
> that an application is supposed to block for more than half an hour...
>
> I feel inclined to open a bug report, but I am yet unshure where the
> problem lies.
>
> Some information:
>
> * 3 RGW processes, 3 OSD hosts with 12 HDD OSDs and 6 SSD OSDs
> * Ceph 12.2.2
> * Auto-Resharding on, Bucket Versioning & Lifecycle rule enabled.
>
>
>
> What life cycle rules do you use?
>
>
>
> Regards,
>
> Orit
>
> Thanks,
>
> Martin
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bug in RadosGW resharding? Hangs again...

[Orit Wasserman]

On Wed, Jan 17, 2018 at 11:45 AM, Martin Emrich <martin.emr...@empolis.com>
wrote:

> Hi Orit!
>
>
>
> I did some tests, and indeed the combination of Versioning/Lifecycle with
> Resharding is the problem:
>
>
>
>- If I do not enable Versioning/Lifecycle, Autoresharding works fine.
>- If I disable Autoresharding but enable Versioning+Lifecycle, pushing
>data works fine, until I manually reshard. This hangs also.
>
>
>
Thanks for testing :) This is very helpful!

My lifecycle rule (which shall remove all versions older than 60 days):
>
>
>
> {
>
> "Rules": [{
>
> "Status": "Enabled",
>
> "Prefix": "",
>
> "NoncurrentVersionExpiration": {
>
> "NoncurrentDays": 60
>
> },
>
> "Expiration": {
>
> "ExpiredObjectDeleteMarker": true
>
> },
>
> "ID": "expire-60days"
>
> }]
>
> }
>
>
>
> I am currently testing with an application containing customer data, but I
> am also creating some random test data to create logs I can share.
>
> I will also test whether the versioning itself is the culprit, or if it is
> the lifecycle rule.
>
>
>

I am suspecting versioning (never tried it with resharding).
Can you open a tracker issue with all the information?

Thanks,
Orit

Regards,
>
> Martin
>
>
>
> *Von: *Orit Wasserman <owass...@redhat.com>
> *Datum: *Dienstag, 16. Januar 2018 um 18:38
> *An: *Martin Emrich <martin.emr...@empolis.com>
> *Cc: *ceph-users <ceph-users@lists.ceph.com>
> *Betreff: *Re: [ceph-users] Bug in RadosGW resharding? Hangs again...
>
>
>
> Hi Martin,
>
>
>
> On Mon, Jan 15, 2018 at 6:04 PM, Martin Emrich <martin.emr...@empolis.com>
> wrote:
>
> Hi!
>
> After having a completely broken radosgw setup due to damaged buckets, I
> completely deleted all rgw pools, and started from scratch.
>
> But my problem is reproducible. After pushing ca. 10 objects into a
> bucket, the resharding process appears to start, and the bucket is now
> unresponsive.
>
>
>
> Sorry to hear that.
>
> Can you share radosgw logs with --debug_rgw=20 --debug_ms=1?
>
>
>
> I just see lots of these messages in all rgw logs:
>
> 2018-01-15 16:57:45.108826 7fd1779b1700  0 block_while_resharding ERROR:
> bucket is still resharding, please retry
> 2018-01-15 16:57:45.119184 7fd1779b1700  0 NOTICE: resharding operation on
> bucket index detected, blocking
> 2018-01-15 16:57:45.260751 7fd1120e6700  0 block_while_resharding ERROR:
> bucket is still resharding, please retry
> 2018-01-15 16:57:45.280410 7fd1120e6700  0 NOTICE: resharding operation on
> bucket index detected, blocking
> 2018-01-15 16:57:45.300775 7fd15b979700  0 block_while_resharding ERROR:
> bucket is still resharding, please retry
> 2018-01-15 16:57:45.300971 7fd15b979700  0 WARNING: set_req_state_err
> err_no=2300 resorting to 500
> 2018-01-15 16:57:45.301042 7fd15b979700  0 ERROR: 
> RESTFUL_IO(s)->complete_header()
> returned err=Input/output error
>
> One radosgw process and two OSDs housing the bucket index/metadata are
> still busy, but it seems to be stuck again.
>
> How long is this resharding process supposed to take? I cannot believe
> that an application is supposed to block for more than half an hour...
>
> I feel inclined to open a bug report, but I am yet unshure where the
> problem lies.
>
> Some information:
>
> * 3 RGW processes, 3 OSD hosts with 12 HDD OSDs and 6 SSD OSDs
> * Ceph 12.2.2
> * Auto-Resharding on, Bucket Versioning & Lifecycle rule enabled.
>
>
>
> What life cycle rules do you use?
>
>
>
> Regards,
>
> Orit
>
> Thanks,
>
> Martin
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bug in RadosGW resharding? Hangs again...

[Orit Wasserman]

Hi Martin,

On Mon, Jan 15, 2018 at 6:04 PM, Martin Emrich 
wrote:

> Hi!
>
> After having a completely broken radosgw setup due to damaged buckets, I
> completely deleted all rgw pools, and started from scratch.
>
> But my problem is reproducible. After pushing ca. 10 objects into a
> bucket, the resharding process appears to start, and the bucket is now
> unresponsive.
>
>
Sorry to hear that.
Can you share radosgw logs with --debug_rgw=20 --debug_ms=1?


> I just see lots of these messages in all rgw logs:
>
> 2018-01-15 16:57:45.108826 7fd1779b1700  0 block_while_resharding ERROR:
> bucket is still resharding, please retry
> 2018-01-15 16:57:45.119184 7fd1779b1700  0 NOTICE: resharding operation on
> bucket index detected, blocking
> 2018-01-15 16:57:45.260751 7fd1120e6700  0 block_while_resharding ERROR:
> bucket is still resharding, please retry
> 2018-01-15 16:57:45.280410 7fd1120e6700  0 NOTICE: resharding operation on
> bucket index detected, blocking
> 2018-01-15 16:57:45.300775 7fd15b979700  0 block_while_resharding ERROR:
> bucket is still resharding, please retry
> 2018-01-15 16:57:45.300971 7fd15b979700  0 WARNING: set_req_state_err
> err_no=2300 resorting to 500
> 2018-01-15 16:57:45.301042 7fd15b979700  0 ERROR:
> RESTFUL_IO(s)->complete_header() returned err=Input/output error
>
> One radosgw process and two OSDs housing the bucket index/metadata are
> still busy, but it seems to be stuck again.
>
> How long is this resharding process supposed to take? I cannot believe
> that an application is supposed to block for more than half an hour...
>
> I feel inclined to open a bug report, but I am yet unshure where the
> problem lies.
>
> Some information:
>
> * 3 RGW processes, 3 OSD hosts with 12 HDD OSDs and 6 SSD OSDs
> * Ceph 12.2.2
> * Auto-Resharding on, Bucket Versioning & Lifecycle rule enabled.
>
>
What life cycle rules do you use?

Regards,
Orit

> Thanks,
>
> Martin
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Resharding issues / How long does it take?

[Orit Wasserman]

Hi,

On Mon, Dec 11, 2017 at 11:45 AM, Martin Emrich
<martin.emr...@empolis.com> wrote:
> Hi!
>
> Am 10.12.17, 11:54 schrieb "Orit Wasserman" <owass...@redhat.com>:
>
> Hi Martin,
>
> On Thu, Dec 7, 2017 at 5:05 PM, Martin Emrich <martin.emr...@empolis.com> 
> wrote:
>
> It could be issue: http://tracker.ceph.com/issues/21619
> The workaround is running radosgw-admin bucket check --fix , it will
> reset the resharding flag.
> If you can update the tracker with your logs it will be very helpful.
>
> I already tried that two times, and after working hard for a few minutes it 
> hangs. Each time it seems to have created a new "set" of objects in the pool 
> (the bucket had ca. 11 objects, "radosgw-admin bucket limit check" 
> reports ca. 33 "num_objects" now).
>

This is after resharding the bucket?

> Which logs would be helpful?
>

rgw logs , if you can increase the debug level debug_rgw=20 and
debug_ms=1 that will be great.

> > I have a feeling that the bucket index is still 
> damaged/incomplete/inconsistent. What does the message
> >
> > *** NOTICE: operation will not remove old bucket index objects ***
> > *** these will need to be removed manually ***
> >
> > mean? How can I clean up manually?
> >
>
> Resharding creates a new bucket index with the new number of shards.
> It doesn't remove the old bucket index, you will need to do it manually.
>
> How do I do that? Does it just involve identifying the right objects in the 
> RADOS pools to delete? Or is there more to it?
>

When resharding completes it prints the old bucket instance id, you
will need to remove it.
I think this is the warning in the start of resharding I believe in
your case resharding hasn't completed.
you can get the old bucket instance id from the resharding log or the
bucket info.
Than you will need to delete it using rados command.

> My primary goal is now to completely remove the damaged bucket without a 
> trace, but I'd also love to find out what went wrong in the first place.
> Could I have a "multisite setup" without knowing it? I did not knowingly set 
> up anything in this regard, It's just one cluster, with three identically 
> configured radosgw behind a load balancer...
>
Probably not you need to setup multisite ...

As to remove the damaged bucket, if the bucket index is no consistent
you will need to manually remove it:
first unlink the bucket from the user: radosgw-admin bucket unlink

Than you will need manually remove the bucket:
1. if this is you only bucket I would go for deleting the bucket pools
and using new one
2. Try to fix the bucket using bucket check --fix
3. try this procedure
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-August/020012.html
4. Another options is to remove the deleted objects entries from the
bucket index and try to delete it

Good Luck,
Orit

> Thanks,
>
> Martin
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous, RGW bucket resharding

[Orit Wasserman]

On Mon, Dec 11, 2017 at 5:44 PM, Sam Wouters <s...@ericom.be> wrote:
> On 11-12-17 16:23, Orit Wasserman wrote:
>> On Mon, Dec 11, 2017 at 4:58 PM, Sam Wouters <s...@ericom.be> wrote:
>>> Hi Orrit,
>>>
>>>
>>> On 04-12-17 18:57, Orit Wasserman wrote:
>>>> Hi Andreas,
>>>>
>>>> On Mon, Dec 4, 2017 at 11:26 AM, Andreas Calminder
>>>> <andreas.calmin...@klarna.com> wrote:
>>>>> Hello,
>>>>> With release 12.2.2 dynamic resharding bucket index has been disabled
>>>>> when running a multisite environment
>>>>> (http://tracker.ceph.com/issues/21725). Does this mean that resharding
>>>>> of bucket indexes shouldn't be done at all, manually, while running
>>>>> multisite as there's a risk of corruption?
>>>>>
>>>> You will need to stop the sync on the bucket before doing the
>>>> resharding and start it again after the resharding completes.
>>>> It will start a full sync on the bucket (it doesn't mean we copy the
>>>> objects but we go over on all of them to check if the need to be
>>>> synced).
>>>> We will automate this as part of the reshard admin command in the next
>>>> Luminous release.
>>> Does this also apply to Jewel? Stop sync and restart after resharding.
>>> (I don't know if there is any way to disable sync for a specific bucket.)
>>>
>> In Jewel we only support offline bucket resharding, you have to stop
>> both zones gateways before resharding.
>> Do:
>> Execute the resharding radosgw-admin command.
>> Run full sync on the bucket using: radosgw-admin bucket sync init on the 
>> bucket.
>> Start the gateways.
>>
>> This should work but I have not tried it ...
>> Regards,
>> Orit
> Is it necessary to really stop the gateways? We tend to block all
> traffic to the bucket being resharded with the use of ACLs in the
> haproxy in front, to avoid downtime for non related buckets.
>

You need to stop the sync process for the bucket and the most simple
way is to stop the gateways.
In Luminous we added the capability to stop the sync per bucket but
you don't have it in Jewel.

> Would a:
>
> - restart gws with sync thread disabled
> - block traffic to bucket
> - reshard
> - unblock traffic
> - bucket sync init
> - restart gws with sync enabled
>
> work as well?
>
> r,
> Sam
>
>>> r,
>>> Sam
>>>>> Also, as dynamic bucket resharding was/is the main motivator moving to
>>>>> Luminous (for me at least) is dynamic reshardning something that is
>>>>> planned to be fixed for multisite environments later in the Luminous
>>>>> life-cycle or will it be left disabled forever?
>>>>>
>>>> We are planning to enable it in Luminous time.
>>>>
>>>> Regards,
>>>> Orit
>>>>
>>>>> Thanks!
>>>>> /andreas
>>>>> ___
>>>>> ceph-users mailing list
>>>>> ceph-users@lists.ceph.com
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>> ___
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous, RGW bucket resharding

[Orit Wasserman]

On Mon, Dec 11, 2017 at 4:58 PM, Sam Wouters <s...@ericom.be> wrote:
> Hi Orrit,
>
>
> On 04-12-17 18:57, Orit Wasserman wrote:
>> Hi Andreas,
>>
>> On Mon, Dec 4, 2017 at 11:26 AM, Andreas Calminder
>> <andreas.calmin...@klarna.com> wrote:
>>> Hello,
>>> With release 12.2.2 dynamic resharding bucket index has been disabled
>>> when running a multisite environment
>>> (http://tracker.ceph.com/issues/21725). Does this mean that resharding
>>> of bucket indexes shouldn't be done at all, manually, while running
>>> multisite as there's a risk of corruption?
>>>
>> You will need to stop the sync on the bucket before doing the
>> resharding and start it again after the resharding completes.
>> It will start a full sync on the bucket (it doesn't mean we copy the
>> objects but we go over on all of them to check if the need to be
>> synced).
>> We will automate this as part of the reshard admin command in the next
>> Luminous release.
> Does this also apply to Jewel? Stop sync and restart after resharding.
> (I don't know if there is any way to disable sync for a specific bucket.)
>

In Jewel we only support offline bucket resharding, you have to stop
both zones gateways before resharding.
Do:
Execute the resharding radosgw-admin command.
Run full sync on the bucket using: radosgw-admin bucket sync init on the bucket.
Start the gateways.

This should work but I have not tried it ...
Regards,
Orit

> r,
> Sam
>>> Also, as dynamic bucket resharding was/is the main motivator moving to
>>> Luminous (for me at least) is dynamic reshardning something that is
>>> planned to be fixed for multisite environments later in the Luminous
>>> life-cycle or will it be left disabled forever?
>>>
>> We are planning to enable it in Luminous time.
>>
>> Regards,
>> Orit
>>
>>> Thanks!
>>> /andreas
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous, RGW bucket resharding

[Orit Wasserman]

Hi Andreas,

On Mon, Dec 4, 2017 at 11:26 AM, Andreas Calminder
 wrote:
> Hello,
> With release 12.2.2 dynamic resharding bucket index has been disabled
> when running a multisite environment
> (http://tracker.ceph.com/issues/21725). Does this mean that resharding
> of bucket indexes shouldn't be done at all, manually, while running
> multisite as there's a risk of corruption?
>

You will need to stop the sync on the bucket before doing the
resharding and start it again after the resharding completes.
It will start a full sync on the bucket (it doesn't mean we copy the
objects but we go over on all of them to check if the need to be
synced).
We will automate this as part of the reshard admin command in the next
Luminous release.

> Also, as dynamic bucket resharding was/is the main motivator moving to
> Luminous (for me at least) is dynamic reshardning something that is
> planned to be fixed for multisite environments later in the Luminous
> life-cycle or will it be left disabled forever?
>

We are planning to enable it in Luminous time.

Regards,
Orit

> Thanks!
> /andreas
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs Hadoop Plugin and CEPH integration

[Orit Wasserman]

On Wed, Nov 29, 2017 at 6:52 PM, Aristeu Gil Alves Jr
 wrote:
>> > Does s3 or swifta (for hadoop or spark) have integrated data-layout APIs
>> > for
>> > local processing data as have cephfs hadoop plugin?
>> >
>> With s3 and swift you won't have data locality as it was designed for
>> public cloud.
>> We recommend disable locality based scheduling in Hadoop when running
>> with those connectors.
>> There is on going work on to optimize those connectors to work with
>> object storage.
>> Hadoop community works on the s3a connector.
>> There is also https://github.com/SparkTC/stocator which is a swift
>> based connector IBM wrote  for their cloud.
>
>
>
> Assuming this cases, how would be a mapreduce process without data locality?
> How the processors get the data? Still there's the need to split the data,
> no?
The s3/swift storage splits the data.

> Doesn't it severely impact the performance of big files (not just the
> network)?
>
There is a facebook research paper showing locality is not as good as
expected, if I remember correctly it was around 30%.
The users that use s3/swift with Hadoop are already using object
storage (for other usages) or have a very very big data set that fits
object storage better.

> --
> Aristeu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs Hadoop Plugin and CEPH integration

[Orit Wasserman]

On Wed, Nov 29, 2017 at 6:54 PM, Gregory Farnum  wrote:
> On Wed, Nov 29, 2017 at 8:52 AM Aristeu Gil Alves Jr 
> wrote:
>>>
>>> > Does s3 or swifta (for hadoop or spark) have integrated data-layout
>>> > APIs for
>>> > local processing data as have cephfs hadoop plugin?
>>> >
>>> With s3 and swift you won't have data locality as it was designed for
>>> public cloud.
>>> We recommend disable locality based scheduling in Hadoop when running
>>> with those connectors.
>>> There is on going work on to optimize those connectors to work with
>>> object storage.
>>> Hadoop community works on the s3a connector.
>>> There is also https://github.com/SparkTC/stocator which is a swift
>>> based connector IBM wrote  for their cloud.
>>
>>
>>
>> Assuming this cases, how would be a mapreduce process without data
>> locality?
>> How the processors get the data? Still there's the need to split the data,
>> no?
>> Doesn't it severely impact the performance of big files (not just the
>> network)?
>>
>
> Given that you already have your data in CephFS (and have been using it
> successfully for two years!), I'd try using its Hadoop plugin and seeing if
> it suits your needs. Trying a less-supported plugin is a lot easier than
> rolling out a new storage stack! :)

completely agree :)

> -Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs Hadoop Plugin and CEPH integration

[Orit Wasserman]

Hi,

On Wed, Nov 29, 2017 at 5:32 PM, Aristeu Gil Alves Jr
<aristeu...@gmail.com> wrote:
> Orit,
>
> As I mentioned, I have cephfs in production for almost two years.
> Can I use this installed filesystem or I need to start from scratch? If the
> first is true, is there any tutorial that you recommend on adding s3 on an
> installed base, or to ceph in general?

Radosgw is the service that provide s3 and swift compatible object storage.
http://docs.ceph.com/docs/master/radosgw/
You can use you existing ceph cluster (monitors and OSDS) but will
need to add the radosgw demon.
It will have it's one separate pools.

> Does s3 or swifta (for hadoop or spark) have integrated data-layout APIs for
> local processing data as have cephfs hadoop plugin?
>
With s3 and swift you won't have data locality as it was designed for
public cloud.
We recommend disable locality based scheduling in Hadoop when running
with those connectors.
There is on going work on to optimize those connectors to work with
object storage.
Hadoop community works on the s3a connector.
There is also https://github.com/SparkTC/stocator which is a swift
based connector IBM wrote  for their cloud.

> Sorry for my lack of knowledge on the matter. As I was exclusively a CephFS
> user, I didn't touch RGW yet. Gonna learn everything now. Any hint is going
> to be welcome.
>

CephFS is great and depending on your dataset and workload it maybe
the right storage for you :)

Cheers,
Orit

>
> Thanks and regards,
> Aristeu
>
> 2017-11-29 4:19 GMT-02:00 Orit Wasserman <owass...@redhat.com>:
>>
>> On Tue, Nov 28, 2017 at 7:26 PM, Aristeu Gil Alves Jr
>> <aristeu...@gmail.com> wrote:
>> > Greg and Donny,
>> >
>> > Thanks for the answers. It helped a lot!
>> >
>> > I just watched the swifta presentation and it looks quite good!
>> >
>>
>> I would highly recommend using s3a and not swifta as it is much more
>> mature and is more used.
>>
>> Cheers,
>> Orit
>>
>> > Due the lack of updates/development, and the fact that we can choose
>> > spark
>> > also, I think maybe swift/swifta with ceph is a good strategy too.
>> > I need to study it more, tho.
>> >
>> > Can I get the same results (performance and integrated data-layout APIs)
>> > with it?
>> >
>> > Is there a migration cases/tutorials from a cephfs to a swift with ceph
>> > scenario that you could suggest?
>> >
>> > Best regards,
>> > --
>> > Aristeu
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs Hadoop Plugin and CEPH integration

[Orit Wasserman]

On Tue, Nov 28, 2017 at 7:26 PM, Aristeu Gil Alves Jr
 wrote:
> Greg and Donny,
>
> Thanks for the answers. It helped a lot!
>
> I just watched the swifta presentation and it looks quite good!
>

I would highly recommend using s3a and not swifta as it is much more
mature and is more used.

Cheers,
Orit

> Due the lack of updates/development, and the fact that we can choose spark
> also, I think maybe swift/swifta with ceph is a good strategy too.
> I need to study it more, tho.
>
> Can I get the same results (performance and integrated data-layout APIs)
> with it?
>
> Is there a migration cases/tutorials from a cephfs to a swift with ceph
> scenario that you could suggest?
>
> Best regards,
> --
> Aristeu
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issues with dynamic bucket indexing resharding and tenants

[Orit Wasserman]

On Wed, Nov 8, 2017 at 9:45 PM, Mark Schouten <m...@tuxis.nl> wrote:
> I see you fixed this (with a rather trivial patch :)), great!
>
:)

> I am wondering though, should I be able to remove the invalid entry using
> this patch too?
>
It should work.

> Regards,
>
> Mark
>
>
> On 5 Nov 2017, at 07:33, Orit Wasserman <owass...@redhat.com> wrote:
>
> Hi Mark,
>
> On Fri, Oct 20, 2017 at 4:26 PM, Mark Schouten <m...@tuxis.nl> wrote:
>
> Hi,
>
> I see issues with resharding. rgw logging shows the following:
> 2017-10-20 15:17:30.018807 7fa1b219a700 -1 ERROR: failed to get entry from
> reshard log, oid=reshard.13 tenant= bucket=qnapnas
>
> radosgw-admin shows me there is one bucket in the queue to do resharding
> for:
> radosgw-admin reshard list
> [
>{
>"time": "2017-10-20 12:37:28.575096Z",
>"tenant": "DB0339",
>"bucket_name": "qnapnas",
>"bucket_id": "1c19a332-7ffc-4472-b852-ec4a143785cc.19675875.3",
>"new_instance_id": "",
>"old_num_shards": 1,
>"new_num_shards": 4
>}
> ]
>
> But, the tenant field in the logging entry is emtpy, which makes me expect
> that the tenant part is partially implemented.
>
> Also, I can add "DB0339/qnapnas" to the list:
> radosgw-admin reshard add --bucket DB0339/qnapnas --num-shards 4
>
> But not like this:
> radosgw-admin reshard add --bucket qnapnas --tenant DB0339 --num-shards 4
> ERROR: --tenant is set, but there's no user ID
>
>
> Please advise.
>
>
> Looks like a bug.
> Can you open a tracker issue for this ?
>
> Thanks,
> Orit
>
>
> Met vriendelijke groeten,
>
> --
> Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
> Mark Schouten | Tuxis Internet Engineering
> KvK: 61527076 | http://www.tuxis.nl/
> T: 0318 200208 | i...@tuxis.nl
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issues with dynamic bucket indexing resharding and tenants

[Orit Wasserman]

Hi Mark,

On Fri, Oct 20, 2017 at 4:26 PM, Mark Schouten  wrote:
> Hi,
>
> I see issues with resharding. rgw logging shows the following:
> 2017-10-20 15:17:30.018807 7fa1b219a700 -1 ERROR: failed to get entry from
> reshard log, oid=reshard.13 tenant= bucket=qnapnas
>
> radosgw-admin shows me there is one bucket in the queue to do resharding
> for:
> radosgw-admin reshard list
> [
> {
> "time": "2017-10-20 12:37:28.575096Z",
> "tenant": "DB0339",
> "bucket_name": "qnapnas",
> "bucket_id": "1c19a332-7ffc-4472-b852-ec4a143785cc.19675875.3",
> "new_instance_id": "",
> "old_num_shards": 1,
> "new_num_shards": 4
> }
> ]
>
> But, the tenant field in the logging entry is emtpy, which makes me expect
> that the tenant part is partially implemented.
>
> Also, I can add "DB0339/qnapnas" to the list:
> radosgw-admin reshard add --bucket DB0339/qnapnas --num-shards 4
>
> But not like this:
> radosgw-admin reshard add --bucket qnapnas --tenant DB0339 --num-shards 4
> ERROR: --tenant is set, but there's no user ID
>
>
> Please advise.

Looks like a bug.
Can you open a tracker issue for this ?

Thanks,
Orit

>
> Met vriendelijke groeten,
>
> --
> Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
> Mark Schouten | Tuxis Internet Engineering
> KvK: 61527076 | http://www.tuxis.nl/
> T: 0318 200208 | i...@tuxis.nl
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] zone, zonegroup and resharding bucket on luminous

[Orit Wasserman]

On Fri, Sep 29, 2017 at 5:56 PM, Yoann Moulin  wrote:
> Hello,
>
> I'm doing some tests on the radosgw on luminous (12.2.1), I have a few 
> questions.
>
> In the documentation[1], there is a reference to "radosgw-admin region get" 
> but it seems not to be available anymore.
> It should be "radosgw-admin zonegroup get" I guess.
>
> 1. http://docs.ceph.com/docs/luminous/install/install-ceph-gateway/
>
> I have installed my luminous cluster with ceph-ansible playbook.
>
> but when I try to manipulate zonegroup or zone, I have this
>
>> # radosgw-admin zonegroup get
>> failed to init zonegroup: (2) No such file or directory
>

try with --rgw-zonegroup=default

>> # radosgw-admin  zone get
>> unable to initialize zone: (2) No such file or directory
>
try with --rgw-zone=default

> I guessed it's because I don't have a realm set and not default zone and 
> zonegroup ?
>

The default zone and zonegroup are  part of the realm so without a
realm you cannot set them as defaults.
This means you have to specifiy --rgw-zonegroup=default and --rgw-zone=default
 I am guessing our documentation needs updating :(
I think we can improve our behavior and make those command works
without a realm , i.e return the default zonegroup and zone. I will
open a tracker issue for that.

>> # radosgw-admin realm list
>> {
>> "default_info": "",
>> "realms": []
>> }
>
>> # radosgw-admin zonegroup list
>> {
>> "default_info": "",
>> "zonegroups": [
>> "default"
>> ]
>> }
>
>> # radosgw-admin zone list
>> {
>> "default_info": "",
>> "zones": [
>> "default"
>> ]
>> }
>
> Is that the default behaviour not to create default realm on a fresh radosgw 
> ? Or is it a side effect of ceph-ansible installation ?
>
It is the default behavior, there is no default realm.

> I have a bucket that referred to a zonegroup but without realm. Can I create 
> a default realm ? Is that safe for the bucket that has already been
> uploaded ?
>
Yes You can create a realm and add the zonegroup to it.
Don't forgot to do "radosgw-admin period update --commit" to commit the changes.

> On the "default" zonegroup (which is not set as default), the  
> "bucket_index_max_shards" is set to "0", can I modify it without reaml ?
>
I just updated this section in this pr: https://github.com/ceph/ceph/pull/18063

Regards,
Orit
> some useful information (I guess) :
>
>> # radosgw-admin zonegroup get --rgw-zonegroup=default
>> {
>> "id": "43d23097-56b9-48a6-ad52-de42341be4bd",
>> "name": "default",
>> "api_name": "",
>> "is_master": "true",
>> "endpoints": [],
>> "hostnames": [],
>> "hostnames_s3website": [],
>> "master_zone": "69d2fd65-fcf9-461b-865f-3dbb053803c4",
>> "zones": [
>> {
>> "id": "69d2fd65-fcf9-461b-865f-3dbb053803c4",
>> "name": "default",
>> "endpoints": [],
>> "log_meta": "false",
>> "log_data": "false",
>> "bucket_index_max_shards": 0,
>> "read_only": "false",
>> "tier_type": "",
>> "sync_from_all": "true",
>> "sync_from": []
>> }
>> ],
>> "placement_targets": [
>> {
>> "name": "default-placement",
>> "tags": []
>> }
>> ],
>> "default_placement": "default-placement",
>> "realm_id": ""
>> }
>
>> # radosgw-admin zone get --rgw-zone=default
>> {
>> "id": "69d2fd65-fcf9-461b-865f-3dbb053803c4",
>> "name": "default",
>> "domain_root": "default.rgw.meta:root",
>> "control_pool": "default.rgw.control",
>> "gc_pool": "default.rgw.log:gc",
>> "lc_pool": "default.rgw.log:lc",
>> "log_pool": "default.rgw.log",
>> "intent_log_pool": "default.rgw.log:intent",
>> "usage_log_pool": "default.rgw.log:usage",
>> "reshard_pool": "default.rgw.log:reshard",
>> "user_keys_pool": "default.rgw.meta:users.keys",
>> "user_email_pool": "default.rgw.meta:users.email",
>> "user_swift_pool": "default.rgw.meta:users.swift",
>> "user_uid_pool": "default.rgw.meta:users.uid",
>> "system_key": {
>> "access_key": "",
>> "secret_key": ""
>> },
>> "placement_pools": [
>> {
>> "key": "default-placement",
>> "val": {
>> "index_pool": "default.rgw.buckets.index",
>> "data_pool": "default.rgw.buckets.data",
>> "data_extra_pool": "default.rgw.buckets.non-ec",
>> "index_type": 0,
>> "compression": ""
>> }
>> }
>> ],
>> "metadata_heap": "",
>> "tier_config": [],
>> "realm_id": ""
>> }
>
>> # radosgw-admin metadata get bucket:image-net
>> {
>> "key": "bucket:image-net",
>> "ver": {
>> "tag": "_2_RFnI5pKQV7XEc5s2euJJW",
>> "ver": 1
>> },
>> "mtime": "2017-08-28 12:27:35.629882Z",
>> "data": {
>> "bucket": {
>> "name": 

Re: [ceph-users] RGW Multisite metadata sync init

[Orit Wasserman]

Hi David,

On Mon, Aug 28, 2017 at 8:33 PM, David Turner  wrote:

> The vast majority of the sync error list is "failed to sync bucket
> instance: (16) Device or resource busy".  I can't find anything on Google
> about this error message in relation to Ceph.  Does anyone have any idea
> what this means? and/or how to fix it?
>

Those are intermediate errors resulting from several radosgw trying to
acquire the same sync log shard lease. It doesn't effect the sync progress.
Are there any other errors?

Orit

>
> On Fri, Aug 25, 2017 at 2:48 PM Casey Bodley  wrote:
>
>> Hi David,
>>
>> The 'data sync init' command won't touch any actual object data, no.
>> Resetting the data sync status will just cause a zone to restart a full
>> sync of the --source-zone's data changes log. This log only lists which
>> buckets/shards have changes in them, which causes radosgw to consider them
>> for bucket sync. So while the command may silence the warnings about data
>> shards being behind, it's unlikely to resolve the issue with missing
>> objects in those buckets.
>>
>> When data sync is behind for an extended period of time, it's usually
>> because it's stuck retrying previous bucket sync failures. The 'sync error
>> list' may help narrow down where those failures are.
>>
>> There is also a 'bucket sync init' command to clear the bucket sync
>> status. Following that with a 'bucket sync run' should restart a full sync
>> on the bucket, pulling in any new objects that are present on the
>> source-zone. I'm afraid that those commands haven't seen a lot of polish or
>> testing, however.
>>
>> Casey
>>
>> On 08/24/2017 04:15 PM, David Turner wrote:
>>
>> Apparently the data shards that are behind go in both directions, but
>> only one zone is aware of the problem.  Each cluster has objects in their
>> data pool that the other doesn't have.  I'm thinking about initiating a
>> `data sync init` on both sides (one at a time) to get them back on the same
>> page.  Does anyone know if that command will overwrite any local data that
>> the zone has that the other doesn't if you run `data sync init` on it?
>>
>> On Thu, Aug 24, 2017 at 1:51 PM David Turner 
>> wrote:
>>
>>> After restarting the 2 RGW daemons on the second site again, everything
>>> caught up on the metadata sync.  Is there something about having 2 RGW
>>> daemons on each side of the multisite that might be causing an issue with
>>> the sync getting stale?  I have another realm set up the same way that is
>>> having a hard time with its data shards being behind.  I haven't told them
>>> to resync, but yesterday I noticed 90 shards were behind.  It's caught back
>>> up to only 17 shards behind, but the oldest change not applied is 2 months
>>> old and no order of restarting RGW daemons is helping to resolve this.
>>>
>>> On Thu, Aug 24, 2017 at 10:59 AM David Turner 
>>> wrote:
>>>
 I have a RGW Multisite 10.2.7 set up for bi-directional syncing.  This
 has been operational for 5 months and working fine.  I recently created a
 new user on the master zone, used that user to create a bucket, and put in
 a public-acl object in there.  The Bucket created on the second site, but
 the user did not and the object errors out complaining about the access_key
 not existing.

 That led me to think that the metadata isn't syncing, while bucket and
 data both are.  I've also confirmed that data is syncing for other buckets
 as well in both directions. The sync status from the second site was this.


1.

  metadata sync syncing

2.

full sync: 0/64 shards

3.

incremental sync: 64/64 shards

4.

metadata is caught up with master

5.

  data sync source: f4c12327-4721-47c9-a365-86332d84c227 
 (public-atl01)

6.

syncing

7.

full sync: 0/128 shards

8.

incremental sync: 128/128 shards

9.

data is caught up with source



 Sync status leads me to think that the second site believes it is up to
 date, even though it is missing a freshly created user.  I restarted all of
 the rgw daemons for the zonegroup, but it didn't trigger anything to fix
 the missing user in the second site.  I did some googling and found the
 sync init commands mentioned in a few ML posts and used metadata sync init
 and now have this as the sync status.


1.

  metadata sync preparing for full sync

2.

full sync: 64/64 shards

3.

full sync: 0 entries to sync

4.

Re: [ceph-users] Linear space complexity or memory leak in `Radosgw-admin bucket check --fix`

[Orit Wasserman]

Hi Hans,

On Wed, Jul 26, 2017 at 10:24 AM, Ben Hines  wrote:
> Which version of Ceph?
>
> On Tue, Jul 25, 2017 at 4:19 AM, Hans van den Bogert 
> wrote:
>>
>> Hi All,
>>
>> I don't seem to be able to fix a bucket, a bucket which has become
>> inconsistent due to the use of the `inconsistent-index` flag 8).
>>
>> My ceph-admin VM has 4GB of RAM, but that doesn't seem to be enough to do
>> a `radosgw-admin bucket check --fix` which holds 6M items, as the
>> radosgw-admin process is killed eventually by the Out-Of-Memory-Manager. Is
>> this high RAM usage to be expected, or should I file a bug?
>>

At the current implementation we send all entries needing fixing to
the OSD in one go.
In case of a large bucket this it can cause high memory usage or even
http://tracker.ceph.com/issues/20772.
I am working on fix it.

Regards,
Orit

>> Regards,
>>
>> Hans
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Object store Swift and S3 interface

[Orit Wasserman]

On Fri, Jul 7, 2017 at 4:20 PM, Murali Balcha 
wrote:

> Hi,
> We tried to use Swift interface for Ceph object store and soon found out
> that it does not support SLO/DLO. We are planning to move to use S3
> interface. Are there any known limitations with the support of S3 with Ceph
> object store?
>
>
Ceph Radosgw supports swift slo/dlo.

For S3 Bucket policy is new in Luminous

Regard,
Orit

Best,
> Murali Balcha
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bucket resharding: "radosgw-admin bi list" ERROR

[Orit Wasserman]

Hi Maarten,

On Tue, Jul 4, 2017 at 9:46 PM, Maarten De Quick 
wrote:

> Hi,
>
> Background: We're having issues with our index pool (slow requests / time
> outs causes crashing of an OSD and a recovery -> application issues). We
> know we have very big buckets (eg. bucket of 77 million objects with only
> 16 shards) that need a reshard so we were looking at the resharding process.
>
> First thing we would like to do is making a backup of the bucket index,
> but this failed with:
>
> # radosgw-admin -n client.radosgw.be-west-3 bi list
> --bucket=priv-prod-up-alex > /var/backup/priv-prod-up-alex.list.backup
> 2017-07-03 21:28:30.325613 7f07fb8bc9c0  0 System already converted
> ERROR: bi_list(): (4) Interrupted system call
>
>
What version of are you using?
Can you rerun the command with --debug-rgw=20 --debug-ms=1?
Also please open a tracker issue (for rgw) with all the information.

Thanks,
Orit

When I grep for "idx" and I count these:
>  # grep idx priv-prod-up-alex.list.backup | wc -l
> 2294942
> When I do a bucket stats for that bucket I get:
> # radosgw-admin -n client.radosgw.be-west-3 bucket stats
> --bucket=priv-prod-up-alex | grep num_objects
> 2017-07-03 21:33:05.776499 7faca49b89c0  0 System already converted
> "num_objects": 20148575
>
> It looks like there are 18 million objects missing and the backup is not
> complete (not sure if that's a correct assumption?). We're also afraid that
> the resharding command will face the same issue.
> Has anyone seen this behaviour before or any thoughts on how to fix it?
>
> We were also wondering if we really need the backup. As the resharding
> process creates a complete new index and keeps the old bucket, is there
> maybe a possibility to relink your bucket to the old bucket in case of
> issues? Or am I missing something important here?
>
> Any help would be greatly appreciated, thanks!
>
> Regards,
> Maarten
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] FW: radosgw: stale/leaked bucket index entries

[Orit Wasserman]

Hi Pavan,

On Tue, Jun 20, 2017 at 8:29 AM, Pavan Rallabhandi <
prallabha...@walmartlabs.com> wrote:

> Trying one more time with ceph-users
>
> On 19/06/17, 11:07 PM, "Pavan Rallabhandi" 
> wrote:
>
> On many of our clusters running Jewel (10.2.5+), am running into a
> strange problem of having stale bucket index entries left over for (some of
> the) objects deleted. Though it is not reproducible at will, it has been
> pretty consistent of late and am clueless at this point for the possible
> reasons to happen so.
>
> The symptoms are that the actual delete operation of an object is
> reported successful in the RGW logs, but a bucket list on the container
> would still show the deleted object. An attempt to download/stat of the
> object appropriately results in a failure. No failures are seen in the
> respective OSDs where the bucket index object is located. And rebuilding
> the bucket index by running ‘radosgw-admin bucket check –fix’ would fix the
> issue.
>
> Though I could simulate the problem by instrumenting the code, to not
> to have invoked `complete_del` on the bucket index op
> https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rados.cc#L8793, but
> that call is always seem to be made unless there is a cascading error from
> the actual delete operation of the object, which doesn’t seem to be the
> case here.
>
> I wanted to know the possible reasons where the bucket index would be
> left in such limbo, any pointers would be much appreciated. FWIW, we are
> not sharding the buckets and very recently I’ve seen this happen with
> buckets having as low as
> < 10 objects, and we are using swift for all the operations.
>
>
Do you use multisite?

Regards,
Orit


> Thanks,
> -Pavan.
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help! how to create multiple zonegroups in single realm?

[Orit Wasserman]

On Wed, May 3, 2017 at 12:13 PM, Orit Wasserman <owass...@redhat.com> wrote:

>
>
> On Wed, May 3, 2017 at 12:05 PM, yiming xie <plato...@gmail.com> wrote:
>
>>  Cluster c2 have not *zone:us-1*
>>
>> ./bin/radosgw-admin -c ./run/c2/ceph.conf period update --commit
>> --rgw-zonegroup=us --rgw-zone=us-1
>>
>
> try --rgw-zonegroup==default --rgw-zone=default.
>
> Could you open a tracker (tracker.ceph.com) issue about this ,include all
> the configuration and all the commands you tried.
> it should have worked without parameters or with just zone us-2.
>

Apparently there is already an issue and a fix:
http://tracker.ceph.com/issues/19554.
At the moment you need to use the url:
radosgw-admin period commit --url=http://localhost:8001
--access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY


> Orit
>
>
>> 2017-05-03 05:01:30.219721 7efcff2606c0  1 Cannot find zone id=
>> (name=us-1), switching to local zonegroup configuration
>> 2017-05-03 05:01:30.222956 7efcff2606c0 -1 Cannot find zone id=
>> (name=us-1)
>> couldn't init storage provider
>>
>> ./bin/radosgw-admin -c ./run/c2/ceph.conf zone get --rgw-zone=us-1
>> unable to initialize zone: (2) No such file or directory
>>
>> ./bin/radosgw-admin -c ./run/c2/ceph.conf zone list
>> {
>> "default_info": "0cae32e6-82d5-489f-adf5-99e92c70f86f",
>> "zones": [
>> "us-2"
>> ]
>> }
>>
>> ./bin/radosgw-admin -c ./run/c2/ceph.conf zonegroup list
>> {
>> "default_info": "6cc7889a-3f00-4fcd-b4dd-0f5951fbd561",
>> "zonegroups": [
>> "us2",
>> "us"
>> ]
>> }
>>
>>
>> 在 2017年5月3日，下午4:57，Orit Wasserman <owass...@redhat.com> 写道：
>>
>>
>>
>> On Wed, May 3, 2017 at 11:51 AM, yiming xie <plato...@gmail.com> wrote:
>>
>>> I run
>>> ./bin/radosgw-admin -c ./run/c2/ceph.conf period update --commit
>>>
>>>
>> try adding --rgw-zonegroup=us1 --rgw-zone=us-1
>>
>>
>>> the error：
>>> 2017-05-03 04:46:10.298103 7fdb2e4226c0  1 Cannot find zone
>>> id=0cae32e6-82d5-489f-adf5-99e92c70f86f (name=us-2), switching to local
>>> zonegroup configuration
>>> 2017-05-03 04:46:10.300145 7fdb2e4226c0 -1 Cannot find zone
>>> id=0cae32e6-82d5-489f-adf5-99e92c70f86f (name=us-2)
>>> couldn't init storage provider
>>>
>>>
>>> 在 2017年5月3日，下午4:45，Orit Wasserman <owass...@redhat.com> 写道：
>>>
>>>
>>>
>>> On Wed, May 3, 2017 at 11:36 AM, yiming xie <plato...@gmail.com> wrote:
>>>
>>>> Hi Orit:
>>>>   Thanks for your reply.
>>>>
>>>>  when I recreate secondary zone group, there is still a error！
>>>>
>>>>   radosgw-admin realm pull --url=http://localhost:8001 
>>>> --access-key=$SYSTEM_ACCESS_KEY
>>>> --secret=$SYSTEM_SECRET_KEY --default
>>>>   radosgw-admin period pull --url=http://localhost:8001 
>>>> --access-key=$SYSTEM_ACCESS_KEY
>>>> --secret=$SYSTEM_SECRET_KEY --default
>>>>   radosge-admin zonegroup create --rgw-zonegroup=us2 --endpoints=
>>>> http://localhost:8002 --rgw-realm=earth
>>>>   radosgw-admin zone create --rgw-zonegroup=us2 --rgw-zone=us-2
>>>> --endpoints=http://localhost:8002 --master
>>>> --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY
>>>>  radosgw-admin period update --commit --rgw-zonegroup=us2
>>>> --rgw-zone=us-2
>>>>
>>>
>>> Remove  "--rgw-zonegroup=us2 --rgw-zone=us-2" - they don't exist yet
>>>
>>>
>>>>
>>> 2017-05-03 04:31:57.319796 7f87dab4e6c0  1 error read_lastest_epoch
>>>> .rgw.root:periods.894eeaf6-4c1f-4478-88eb-413e58f1a4a4:stagi
>>>> ng.latest_epoch
>>>> Sending period to new master zone cb8fd49d-9789-4cb3-8010-2523bf46a650
>>>> could not find connection for zone or zonegroup id:
>>>> cb8fd49d-9789-4cb3-8010-2523bf46a650
>>>> request failed: (2) No such file or directory
>>>> failed to commit period: (2) No such file or directory
>>>>
>>>> ceph version 11.1.0-7421-gd25b355 (d25b3550dae243f6868a526632e97
>>>> 405866e76d4)
>>>>
>>>>
>>>>
>>>>
>>>> 在 2017年5月3日，下午4:07，Orit Wasserman <owass...@redhat.com> 写道：
>>>>
>>>> Hi,
>>>>
>&

Re: [ceph-users] Help! how to create multiple zonegroups in single realm?

[Orit Wasserman]

On Wed, May 3, 2017 at 12:05 PM, yiming xie <plato...@gmail.com> wrote:

>  Cluster c2 have not *zone:us-1*
>
> ./bin/radosgw-admin -c ./run/c2/ceph.conf period update --commit
> --rgw-zonegroup=us --rgw-zone=us-1
>

try --rgw-zonegroup==default --rgw-zone=default.

Could you open a tracker (tracker.ceph.com) issue about this ,include all
the configuration and all the commands you tried.
it should have worked without parameters or with just zone us-2.

Orit


> 2017-05-03 05:01:30.219721 7efcff2606c0  1 Cannot find zone id=
> (name=us-1), switching to local zonegroup configuration
> 2017-05-03 05:01:30.222956 7efcff2606c0 -1 Cannot find zone id= (name=us-1)
> couldn't init storage provider
>
> ./bin/radosgw-admin -c ./run/c2/ceph.conf zone get --rgw-zone=us-1
> unable to initialize zone: (2) No such file or directory
>
> ./bin/radosgw-admin -c ./run/c2/ceph.conf zone list
> {
> "default_info": "0cae32e6-82d5-489f-adf5-99e92c70f86f",
> "zones": [
> "us-2"
> ]
> }
>
> ./bin/radosgw-admin -c ./run/c2/ceph.conf zonegroup list
> {
> "default_info": "6cc7889a-3f00-4fcd-b4dd-0f5951fbd561",
> "zonegroups": [
> "us2",
> "us"
> ]
> }
>
>
> 在 2017年5月3日，下午4:57，Orit Wasserman <owass...@redhat.com> 写道：
>
>
>
> On Wed, May 3, 2017 at 11:51 AM, yiming xie <plato...@gmail.com> wrote:
>
>> I run
>> ./bin/radosgw-admin -c ./run/c2/ceph.conf period update --commit
>>
>>
> try adding --rgw-zonegroup=us1 --rgw-zone=us-1
>
>
>> the error：
>> 2017-05-03 04:46:10.298103 7fdb2e4226c0  1 Cannot find zone
>> id=0cae32e6-82d5-489f-adf5-99e92c70f86f (name=us-2), switching to local
>> zonegroup configuration
>> 2017-05-03 04:46:10.300145 7fdb2e4226c0 -1 Cannot find zone
>> id=0cae32e6-82d5-489f-adf5-99e92c70f86f (name=us-2)
>> couldn't init storage provider
>>
>>
>> 在 2017年5月3日，下午4:45，Orit Wasserman <owass...@redhat.com> 写道：
>>
>>
>>
>> On Wed, May 3, 2017 at 11:36 AM, yiming xie <plato...@gmail.com> wrote:
>>
>>> Hi Orit:
>>>   Thanks for your reply.
>>>
>>>  when I recreate secondary zone group, there is still a error！
>>>
>>>   radosgw-admin realm pull --url=http://localhost:8001 
>>> --access-key=$SYSTEM_ACCESS_KEY
>>> --secret=$SYSTEM_SECRET_KEY --default
>>>   radosgw-admin period pull --url=http://localhost:8001 
>>> --access-key=$SYSTEM_ACCESS_KEY
>>> --secret=$SYSTEM_SECRET_KEY --default
>>>   radosge-admin zonegroup create --rgw-zonegroup=us2 --endpoints=
>>> http://localhost:8002 --rgw-realm=earth
>>>   radosgw-admin zone create --rgw-zonegroup=us2 --rgw-zone=us-2
>>> --endpoints=http://localhost:8002 --master
>>> --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY
>>>  radosgw-admin period update --commit --rgw-zonegroup=us2 --rgw-zone=us-2
>>>
>>
>> Remove  "--rgw-zonegroup=us2 --rgw-zone=us-2" - they don't exist yet
>>
>>
>>>
>> 2017-05-03 04:31:57.319796 7f87dab4e6c0  1 error read_lastest_epoch
>>> .rgw.root:periods.894eeaf6-4c1f-4478-88eb-413e58f1a4a4:stagi
>>> ng.latest_epoch
>>> Sending period to new master zone cb8fd49d-9789-4cb3-8010-2523bf46a650
>>> could not find connection for zone or zonegroup id:
>>> cb8fd49d-9789-4cb3-8010-2523bf46a650
>>> request failed: (2) No such file or directory
>>> failed to commit period: (2) No such file or directory
>>>
>>> ceph version 11.1.0-7421-gd25b355 (d25b3550dae243f6868a526632e97
>>> 405866e76d4)
>>>
>>>
>>>
>>>
>>> 在 2017年5月3日，下午4:07，Orit Wasserman <owass...@redhat.com> 写道：
>>>
>>> Hi,
>>>
>>> On Wed, May 3, 2017 at 11:00 AM, yiming xie <plato...@gmail.com> wrote:
>>>
>>>> Hi orit:
>>>> I try to create multiple zonegroups in single realm, but failed. Pls
>>>> tell me the correct way about creating multiple zonegroups
>>>> Tks a lot!!
>>>>
>>>> 1.create the firstr zone group on the c1 cluster
>>>> ./bin/radosgw-admin -c ./run/c1/ceph.conf realm create
>>>> --rgw-realm=earth --default
>>>> ./bin/radosgw-admin -c ./run/c1/ceph.conf zonegroup create
>>>> --rgw-zonegroup=us --endpoints=http://localhost:8001 --master --default
>>>>
>>>> ./bin/radosgw-admin -c ./run/c1/ceph.conf zone create
>>>> --rgw-zonegroup=us --rgw-zone=us-1

Re: [ceph-users] Help! how to create multiple zonegroups in single realm?

[Orit Wasserman]

On Wed, May 3, 2017 at 11:51 AM, yiming xie <plato...@gmail.com> wrote:

> I run
> ./bin/radosgw-admin -c ./run/c2/ceph.conf period update --commit
>
>
try adding --rgw-zonegroup=us1 --rgw-zone=us-1


> the error：
> 2017-05-03 04:46:10.298103 7fdb2e4226c0  1 Cannot find zone
> id=0cae32e6-82d5-489f-adf5-99e92c70f86f (name=us-2), switching to local
> zonegroup configuration
> 2017-05-03 04:46:10.300145 7fdb2e4226c0 -1 Cannot find zone
> id=0cae32e6-82d5-489f-adf5-99e92c70f86f (name=us-2)
> couldn't init storage provider
>
>
> 在 2017年5月3日，下午4:45，Orit Wasserman <owass...@redhat.com> 写道：
>
>
>
> On Wed, May 3, 2017 at 11:36 AM, yiming xie <plato...@gmail.com> wrote:
>
>> Hi Orit:
>>   Thanks for your reply.
>>
>>  when I recreate secondary zone group, there is still a error！
>>
>>   radosgw-admin realm pull --url=http://localhost:8001 
>> --access-key=$SYSTEM_ACCESS_KEY
>> --secret=$SYSTEM_SECRET_KEY --default
>>   radosgw-admin period pull --url=http://localhost:8001 
>> --access-key=$SYSTEM_ACCESS_KEY
>> --secret=$SYSTEM_SECRET_KEY --default
>>   radosge-admin zonegroup create --rgw-zonegroup=us2 --endpoints=
>> http://localhost:8002 --rgw-realm=earth
>>   radosgw-admin zone create --rgw-zonegroup=us2 --rgw-zone=us-2
>> --endpoints=http://localhost:8002 --master --access-key=$SYSTEM_ACCESS_KEY
>> --secret=$SYSTEM_SECRET_KEY
>>  radosgw-admin period update --commit --rgw-zonegroup=us2 --rgw-zone=us-2
>>
>
> Remove  "--rgw-zonegroup=us2 --rgw-zone=us-2" - they don't exist yet
>
>
>>
> 2017-05-03 04:31:57.319796 7f87dab4e6c0  1 error read_lastest_epoch
>> .rgw.root:periods.894eeaf6-4c1f-4478-88eb-413e58f1a4a4:stagi
>> ng.latest_epoch
>> Sending period to new master zone cb8fd49d-9789-4cb3-8010-2523bf46a650
>> could not find connection for zone or zonegroup id:
>> cb8fd49d-9789-4cb3-8010-2523bf46a650
>> request failed: (2) No such file or directory
>> failed to commit period: (2) No such file or directory
>>
>> ceph version 11.1.0-7421-gd25b355 (d25b3550dae243f6868a526632e97
>> 405866e76d4)
>>
>>
>>
>>
>> 在 2017年5月3日，下午4:07，Orit Wasserman <owass...@redhat.com> 写道：
>>
>> Hi,
>>
>> On Wed, May 3, 2017 at 11:00 AM, yiming xie <plato...@gmail.com> wrote:
>>
>>> Hi orit:
>>> I try to create multiple zonegroups in single realm, but failed. Pls
>>> tell me the correct way about creating multiple zonegroups
>>> Tks a lot!!
>>>
>>> 1.create the firstr zone group on the c1 cluster
>>> ./bin/radosgw-admin -c ./run/c1/ceph.conf realm create --rgw-realm=earth
>>> --default
>>> ./bin/radosgw-admin -c ./run/c1/ceph.conf zonegroup create
>>> --rgw-zonegroup=us --endpoints=http://localhost:8001 --master --default
>>>
>>> ./bin/radosgw-admin -c ./run/c1/ceph.conf zone create --rgw-zonegroup=us
>>> --rgw-zone=us-1 --endpoints=http://localhost:8001 --master --default
>>> --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY
>>> ./bin/radosgw-admin -c ./run/c1/ceph.conf user create --uid=zone.user
>>> --display-name="Zone User" --access-key=$SYSTEM_ACCESS_KEY
>>> --secret=$SYSTEM_SECRET_KEY --system
>>> ./bin/radosgw-admin -c ./run/c1/ceph.conf period update --commit
>>> //start rgw
>>> ./bin/radosgw -c ./run/c1/ceph.conf --log-file=./run/c1/out/rgw.log
>>> --debug-rgw=20 --debug-ms=1 -i client.rgw.us-1 -rgw-zone=us-1
>>>
>>> 2.create the scondary zone group on the c2 cluster
>>> ./bin/radosgw-admin -c ./run/c2/ceph.conf realm pull --url=
>>> http://localhost:8001 --access-key=$SYSTEM_ACCESS_KEY
>>> --secret=$SYSTEM_SECRET_KEY
>>>
>>> I recommend adding --default that set this realm as default otherwise
>> you need to run:
>> radosgw-admin realm default --rgw-realm=earth
>>
>> You are missing the period pull command:
>>  radosgw-admin period pull --url=http://localhost:8001 
>> --access-key=$SYSTEM_ACCESS_KEY
>> --secret=$SYSTEM_SECRET_KEY --default
>>
>> Orit
>>
>>> ./bin/radosgw-admin -c ./run/c2/ceph.conf zonegroup create
>>> --rgw-zonegroup=us2 --endpoints=http://localhost:8002 --rgw-realm=earth
>>> ./bin/radosgw-admin -c ./run/c2/ceph.conf period update --commit
>>> --rgw-zonegroup=us2 --rgw-realm=earth
>>>
>>> 2017-05-03 00:51:20.190417 7f538dbbb6c0  1 Cannot find zone id= (name=),
>>> switching to local zonegroup configuration
>>> 2017-05-03 00:51:20.192342 7f538dbbb6c0 -1 Cannot find zone id= (name=)
>>> couldn't init storage provider
>>> .
>>>
>>> ./bin/radosgw-admin -c ./run/c2/ceph.conf zone create ..
>>>
>>>
>>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help! how to create multiple zonegroups in single realm?

[Orit Wasserman]

On Wed, May 3, 2017 at 11:36 AM, yiming xie <plato...@gmail.com> wrote:

> Hi Orit:
>   Thanks for your reply.
>
>  when I recreate secondary zone group, there is still a error！
>
>   radosgw-admin realm pull --url=http://localhost:8001 
> --access-key=$SYSTEM_ACCESS_KEY
> --secret=$SYSTEM_SECRET_KEY --default
>   radosgw-admin period pull --url=http://localhost:8001 
> --access-key=$SYSTEM_ACCESS_KEY
> --secret=$SYSTEM_SECRET_KEY --default
>   radosge-admin zonegroup create --rgw-zonegroup=us2 --endpoints=
> http://localhost:8002 --rgw-realm=earth
>   radosgw-admin zone create --rgw-zonegroup=us2 --rgw-zone=us-2
> --endpoints=http://localhost:8002 --master --access-key=$SYSTEM_ACCESS_KEY
> --secret=$SYSTEM_SECRET_KEY
>  radosgw-admin period update --commit --rgw-zonegroup=us2 --rgw-zone=us-2
>

Remove  "--rgw-zonegroup=us2 --rgw-zone=us-2" - they don't exist yet


>
2017-05-03 04:31:57.319796 7f87dab4e6c0  1 error read_lastest_epoch
> .rgw.root:periods.894eeaf6-4c1f-4478-88eb-413e58f1a4a4:
> staging.latest_epoch
> Sending period to new master zone cb8fd49d-9789-4cb3-8010-2523bf46a650
> could not find connection for zone or zonegroup id:
> cb8fd49d-9789-4cb3-8010-2523bf46a650
> request failed: (2) No such file or directory
> failed to commit period: (2) No such file or directory
>
> ceph version 11.1.0-7421-gd25b355 (d25b3550dae243f6868a526632e974
> 05866e76d4)
>
>
>
>
> 在 2017年5月3日，下午4:07，Orit Wasserman <owass...@redhat.com> 写道：
>
> Hi,
>
> On Wed, May 3, 2017 at 11:00 AM, yiming xie <plato...@gmail.com> wrote:
>
>> Hi orit:
>> I try to create multiple zonegroups in single realm, but failed. Pls tell
>> me the correct way about creating multiple zonegroups
>> Tks a lot!!
>>
>> 1.create the firstr zone group on the c1 cluster
>> ./bin/radosgw-admin -c ./run/c1/ceph.conf realm create --rgw-realm=earth
>> --default
>> ./bin/radosgw-admin -c ./run/c1/ceph.conf zonegroup create
>> --rgw-zonegroup=us --endpoints=http://localhost:8001 --master --default
>>
>> ./bin/radosgw-admin -c ./run/c1/ceph.conf zone create --rgw-zonegroup=us
>> --rgw-zone=us-1 --endpoints=http://localhost:8001 --master --default
>> --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY
>> ./bin/radosgw-admin -c ./run/c1/ceph.conf user create --uid=zone.user
>> --display-name="Zone User" --access-key=$SYSTEM_ACCESS_KEY
>> --secret=$SYSTEM_SECRET_KEY --system
>> ./bin/radosgw-admin -c ./run/c1/ceph.conf period update --commit
>> //start rgw
>> ./bin/radosgw -c ./run/c1/ceph.conf --log-file=./run/c1/out/rgw.log
>> --debug-rgw=20 --debug-ms=1 -i client.rgw.us-1 -rgw-zone=us-1
>>
>> 2.create the scondary zone group on the c2 cluster
>> ./bin/radosgw-admin -c ./run/c2/ceph.conf realm pull --url=
>> http://localhost:8001 --access-key=$SYSTEM_ACCESS_KEY
>> --secret=$SYSTEM_SECRET_KEY
>>
>> I recommend adding --default that set this realm as default otherwise you
> need to run:
> radosgw-admin realm default --rgw-realm=earth
>
> You are missing the period pull command:
>  radosgw-admin period pull --url=http://localhost:8001 
> --access-key=$SYSTEM_ACCESS_KEY
> --secret=$SYSTEM_SECRET_KEY --default
>
> Orit
>
>> ./bin/radosgw-admin -c ./run/c2/ceph.conf zonegroup create
>> --rgw-zonegroup=us2 --endpoints=http://localhost:8002 --rgw-realm=earth
>> ./bin/radosgw-admin -c ./run/c2/ceph.conf period update --commit
>> --rgw-zonegroup=us2 --rgw-realm=earth
>>
>> 2017-05-03 00:51:20.190417 7f538dbbb6c0  1 Cannot find zone id= (name=),
>> switching to local zonegroup configuration
>> 2017-05-03 00:51:20.192342 7f538dbbb6c0 -1 Cannot find zone id= (name=)
>> couldn't init storage provider
>> .
>>
>> ./bin/radosgw-admin -c ./run/c2/ceph.conf zone create ..
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help! how to create multiple zonegroups in single realm?

[Orit Wasserman]

Hi,

On Wed, May 3, 2017 at 11:00 AM, yiming xie  wrote:

> Hi orit:
> I try to create multiple zonegroups in single realm, but failed. Pls tell
> me the correct way about creating multiple zonegroups
> Tks a lot!!
>
> 1.create the firstr zone group on the c1 cluster
> ./bin/radosgw-admin -c ./run/c1/ceph.conf realm create --rgw-realm=earth
> --default
> ./bin/radosgw-admin -c ./run/c1/ceph.conf zonegroup create
> --rgw-zonegroup=us --endpoints=http://localhost:8001 --master --default
>
> ./bin/radosgw-admin -c ./run/c1/ceph.conf zone create --rgw-zonegroup=us
> --rgw-zone=us-1 --endpoints=http://localhost:8001 --master --default
> --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY
> ./bin/radosgw-admin -c ./run/c1/ceph.conf user create --uid=zone.user
> --display-name="Zone User" --access-key=$SYSTEM_ACCESS_KEY
> --secret=$SYSTEM_SECRET_KEY --system
> ./bin/radosgw-admin -c ./run/c1/ceph.conf period update --commit
> //start rgw
> ./bin/radosgw -c ./run/c1/ceph.conf --log-file=./run/c1/out/rgw.log
> --debug-rgw=20 --debug-ms=1 -i client.rgw.us-1 -rgw-zone=us-1
>
> 2.create the scondary zone group on the c2 cluster
> ./bin/radosgw-admin -c ./run/c2/ceph.conf realm pull --url=
> http://localhost:8001 --access-key=$SYSTEM_ACCESS_KEY
> --secret=$SYSTEM_SECRET_KEY
>
> I recommend adding --default that set this realm as default otherwise you
need to run:
radosgw-admin realm default --rgw-realm=earth

You are missing the period pull command:
 radosgw-admin period pull --url=http://localhost:8001
--access-key=$SYSTEM_ACCESS_KEY
--secret=$SYSTEM_SECRET_KEY --default

Orit

> ./bin/radosgw-admin -c ./run/c2/ceph.conf zonegroup create
> --rgw-zonegroup=us2 --endpoints=http://localhost:8002 --rgw-realm=earth
> ./bin/radosgw-admin -c ./run/c2/ceph.conf period update --commit
> --rgw-zonegroup=us2 --rgw-realm=earth
>
> 2017-05-03 00:51:20.190417 7f538dbbb6c0  1 Cannot find zone id= (name=),
> switching to local zonegroup configuration
> 2017-05-03 00:51:20.192342 7f538dbbb6c0 -1 Cannot find zone id= (name=)
> couldn't init storage provider
> .
>
> ./bin/radosgw-admin -c ./run/c2/ceph.conf zone create ..
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Orit Wasserman]

Hi Ben,

On Thu, Apr 20, 2017 at 6:08 PM, Ben Morrice  wrote:
> Hi all,
>
> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
> and authentication is in a very bad state. This installation is part of a
> multigw configuration, and I have just updated one host in the secondary
> zone (all other hosts/zones are running 10.2.5).
>
> On the 10.2.7 server I cannot authenticate as a user (normally backed by
> OpenStack Keystone), but even worse I can also not authenticate with an
> admin user.
>
> Please see [1] for the results of performing a list bucket operation with
> python boto (script works against rgw 10.2.5)
>
> Also, if I try to authenticate from the 'master' rgw zone with a
> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
>
> "ERROR: failed to fetch datalog info"
>
> "failed to retrieve sync info: (13) Permission denied"
>
> The above errors correlates to the errors in the log on the server running
> 10.2.7 (debug level 20) at [2]
>
> I'm not sure what I have done wrong or can try next?
>
> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
> authentication functionality

Can you provide the following info:
radosgw-admin period get
radsogw-admin zonegroup get
radsogw-admin zone get

Can you provide your ceph.conf?

Thanks,
Orit

>
> [1]
> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>  encoding="UTF-8"?>SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva
>
> [2]
> /bbpsrvc15.cscs.ch/admin/log
> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15
> auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
> 2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
> 2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
> err_no=-2027 new_err_no=-2027
> 2017-04-20 16:43:04.916329 7ff87c6c0700  2 req 354:0.052585:s3:GET
> /admin/log:get_obj:op status=0
> 2017-04-20 16:43:04.916339 7ff87c6c0700  2 req 354:0.052595:s3:GET
> /admin/log:get_obj:http status=403
> 2017-04-20 16:43:04.916343 7ff87c6c0700  1 == req done
> req=0x7ff87c6ba710 op status=0 http_status=403 ==
> 2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
> 2017-04-20 16:43:04.916390 7ff87c6c0700  1 civetweb: 0x7ff990015610:
> 10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
> - -
> 2017-04-20 16:43:04.917212 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate()
> 2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1544: shard_id=20
> mdlog_marker=1_1492686039.901886_5551978.1
> sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
> 2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
> 2017-04-20 16:43:04.917236 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: init request
> 2017-04-20 16:43:04.917240 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status
> 2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 is io
> blocked
> 2017-04-20 16:43:04.918285 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status complete
> 2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20
> marker=1_1492686039.901886_5551978.1 last_update=2017-04-20
> 13:00:39.0.901886s
> 2017-04-20 16:43:04.918316 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: sending rest request
> 2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): HTTP_DATE: Thu Apr
> 20 14:43:04 2017
> 2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20
> 14:43:04 2017
> 2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource():
> dest=/admin/log
> 2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical header: GET
>
> --
> Kind regards,
>
> Ben Morrice
>
> __
> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
> EPFL / BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list

Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API has been added. Note that currently it only supports object expiration."

[Orit Wasserman]

I see : acct_user=foo, acct_name=foo,
Are you using radosgw with tenants?
If not it could be the problem

Orit

On Sat, Apr 1, 2017 at 7:43 AM, Ben Hines  wrote:

> I'm also trying to use lifecycles (via boto3) but i'm getting permission
> denied trying to create the lifecycle. I'm bucket owner with full_control
> and WRITE_ACP for good measure. Any ideas?
>
> This is debug ms=20 debug radosgw=20
>
>
>
> 2017-03-31 21:28:18.382217 7f50d0010700  2 req 8:0.000693:s3:PUT
> /bentest:put_lifecycle:verifying op permissions
> 2017-03-31 21:28:18.38 7f50d0010700  5 Searching permissions for
> identity=RGWThirdPartyAccountAuthApplier() ->
> RGWLocalAuthApplier(acct_user=foo, acct_name=foo, subuser=, perm_mask=15,
> is_admin=) mask=56
> 2017-03-31 21:28:18.382232 7f50d0010700  5 Searching permissions for
> uid=foo
> 2017-03-31 21:28:18.382235 7f50d0010700  5 Found permission: 15
> 2017-03-31 21:28:18.382237 7f50d0010700  5 Searching permissions for
> group=1 mask=56
> 2017-03-31 21:28:18.382297 7f50d0010700  5 Found permission: 3
> 2017-03-31 21:28:18.382307 7f50d0010700  5 Searching permissions for
> group=2 mask=56
> 2017-03-31 21:28:18.382313 7f50d0010700  5 Permissions for group not found
> 2017-03-31 21:28:18.382318 7f50d0010700  5 Getting permissions
> identity=RGWThirdPartyAccountAuthApplier() ->
> RGWLocalAuthApplier(acct_user=foo, acct_name=foo, subuser=, perm_mask=15,
> is_admin=) owner=foo perm=8
> 2017-03-31 21:28:18.382325 7f50d0010700 10  
> identity=RGWThirdPartyAccountAuthApplier()
> -> RGWLocalAuthApplier(acct_user=foo, acct_name=foo, subuser=,
> perm_mask=15, is_admin=) requested perm (type)=8, policy perm=8,
> user_perm_mask=8, acl perm=8
> 2017-03-31 21:28:18.382330 7f50d0010700  2 req 8:0.000808:s3:PUT
> /bentest:put_lifecycle:verifying op params
> 2017-03-31 21:28:18.382334 7f50d0010700  2 req 8:0.000813:s3:PUT
> /bentest:put_lifecycle:pre-executing
> 2017-03-31 21:28:18.382339 7f50d0010700  2 req 8:0.000817:s3:PUT
> /bentest:put_lifecycle:executing
> 2017-03-31 21:28:18.382361 7f50d0010700 15 read len=183
> data=http://s3.amazonaws.com
> /doc/2006-03-01/">Enabled
> 10
> 2017-03-31 21:28:18.382439 7f50d0010700  2 req 8:0.000917:s3:PUT
> /bentest:put_lifecycle:completing
> 2017-03-31 21:28:18.382594 7f50d0010700  2 req 8:0.001072:s3:PUT
> /bentest:put_lifecycle:op status=-13
> 2017-03-31 21:28:18.382620 7f50d0010700  2 req 8:0.001098:s3:PUT
> /bentest:put_lifecycle:http status=403
> 2017-03-31 21:28:18.382665 7f50d0010700  1 == req done
> req=0x7f50d000a340 op status=-13 http_status=403 ==
>
>
> -Ben
>
> On Tue, Mar 28, 2017 at 6:42 AM, Daniel Gryniewicz 
> wrote:
>
>> On 03/27/2017 04:28 PM, ceph.nov...@habmalnefrage.de wrote:
>>
>>> Hi Cephers.
>>>
>>> Couldn't find any special documentation about the "S3 object expiration"
>>> so I assume it should work "AWS S3 like" (?!?) ...  BUT ...
>>> we have a test cluster based on 11.2.0 - Kraken and I set some object
>>> expiration dates via CyberDuck and DragonDisk, but the objects are still
>>> there, days after the applied date/time. Do I miss something?
>>>
>>> Thanks & regards
>>>
>>>
>> It is intended to work like AWS S3, yes.  Not every feature of AWS
>> lifecycle is supported, (for example no moving between storage tiers), but
>> deletion works, and is tested in teuthology runs.
>>
>> Did you somehow turn it off?  The config option rgw_enable_lc_threads
>> controls it, but it defaults to "on".  Also make sure rgw_lc_debug_interval
>> is not set, and that rgw_lifecycle_work_time isn't set to some interval too
>> small scan your objects...
>>
>> Daniel
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW listing users' quota and usage painfully slow

[Orit Wasserman]

On Thu, Mar 9, 2017 at 1:28 PM, Matthew Vernon  wrote:

> On 09/03/17 10:45, Abhishek Lekshmanan wrote:
>
> On 03/09/2017 11:26 AM, Matthew Vernon wrote:
>>
>>>
>>> I'm using Jewel / 10.2.3-0ubuntu0.16.04.2 . We want to keep track of our
>>> S3 users' quota and usage. Even with a relatively small number of users
>>> (23) it's taking ~23 seconds.
>>>
>>> What we do is (in outline):
>>> radosgw-admin metadata list user
>>> for each user X:
>>>   radosgw-admin user info --uid=X  #has quota details
>>>   radosgw-admin user stats --uid=X #has usage details
>>>
>>> None of these calls is particularly slow (~0.5s), but the net result is
>>> not very satisfactory.
>>>
>>> What am I doing wrong? :)
>>>
>>
>> Is this a single site or a multisite cluster? If you're only trying to
>> read info you could try disabling the cache (it is not recommended to
>> use this if you're trying to write/modify info) for eg:
>>
>
> It's a single site.
>
> $ radosgw-admin user info --uid=x --rgw-cache-enabled=false
>>
>
> That doesn't noticably change the execution time (perhaps it improves it a
> little)
>
> also you could run the info command with higher debug (--debug-rgw=20
>> --debug-ms=1) and paste that somewhere (its very verbose) to help
>> identify where we're slowing down
>>
>
> https://drive.google.com/drive/folders/0B4TV1iNptBAdMEdUaGJI
> a3U1QVE?usp=sharing
>
>
A quick look at the logs seems to indicate that most of the time is spent
on the bootstrap , I will investigate it further.

Should let you see the output from running this with that cache-disabling
> option (and without).
>
> Naiively, I find myself wondering if some sort of all-users flag to the
> info and stats command or a "tell me usage and quota with one call" command
> would be quicker.
>

That sounds a good idea :), can you open a tracker feature issue for this

Regards,
Orit

>
> Thanks,
>
> Matthew
>
>
> --
> The Wellcome Trust Sanger Institute is operated by Genome Research
> Limited, a charity registered in England with number 1021457 and a company
> registered in England with number 2742969, whose registered office is 215
> Euston Road, London, NW1 2BE. __
> _
>
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] unable to do regionmap update

[Orit Wasserman]

On Wed, Jan 11, 2017 at 2:53 PM, Marko Stojanovic  wrote:
>
> Hello all,
>
>  I have issue with radosgw-admin regionmap update . It doesn't update map.
>
> With zone configured like this:
>
> radosgw-admin zone get
> {
> "id": "fc12ac44-e27e-44e3-9b13-347162d3c1d2",
> "name": "oak-1",
> "domain_root": "oak-1.rgw.data.root",
> "control_pool": "oak-1.rgw.control",
> "gc_pool": "oak-1.rgw.gc",
> "log_pool": "oak-1.rgw.log",
> "intent_log_pool": "oak-1.rgw.intent-log",
> "usage_log_pool": "oak-1.rgw.usage",
> "user_keys_pool": "oak-1.rgw.users.keys",
> "user_email_pool": "oak-1.rgw.users.email",
> "user_swift_pool": "oak-1.rgw.users.swift",
> "user_uid_pool": "oak-1.rgw.users.uid",
> "system_key": {
> "access_key": "XX",
> "secret_key": "XX"
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": "oak-1.rgw.buckets.index",
> "data_pool": "oak-1.rgw.buckets.data",
> "data_extra_pool": "oak-1.rgw.buckets.non-ec",
> "index_type": 0
> }
> },
> {
> "key": "ssd-placement",
> "val": {
> "index_pool": "oak-1.rgw.buckets.index-ssd",
> "data_pool": "oak-1.rgw.buckets.data-ssd",
> "data_extra_pool": "oak-1.rgw.buckets.non-ec-ssd",
> "index_type": 0
> }
> }
> ],
> "metadata_heap": "oak-1.rgw.meta",
> "realm_id": "67e26f6b-4774-4b14-9668-a5cf76b9e9ce"
> }
>
> And region
>
> radosgw-admin region get
> {
> "id": "dbec3557-87bb-4460-8546-b59b4fde7e10",
> "name": "oak",
> "api_name": "oak",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "fc12ac44-e27e-44e3-9b13-347162d3c1d2",
> "zones": [
> {
> "id": "fc12ac44-e27e-44e3-9b13-347162d3c1d2",
> "name": "oak-1",
> "endpoints": [
> "http:\/\/ceph1.oak.vast.com:7480"
> ],
> "log_meta": "true",
> "log_data": "false",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": [
> "default-placement"
> ]
> },
> {
> "name": "ssd-placement",
> "tags": [
> "ssd-placement"
> ]
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "67e26f6b-4774-4b14-9668-a5cf76b9e9ce"
>
>
>
> When I run radosgw-admin regionmap update I don't get ssd-placement as
> placement_target:
>
> {
> "zonegroups": [
> {
> "key": "dbec3557-87bb-4460-8546-b59b4fde7e10",
> "val": {
> "id": "dbec3557-87bb-4460-8546-b59b4fde7e10",
> "name": "oak",
> "api_name": "oak",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "fc12ac44-e27e-44e3-9b13-347162d3c1d2",
> "zones": [
> {
> "id": "fc12ac44-e27e-44e3-9b13-347162d3c1d2",
> "name": "oak-1",
> "endpoints": [
> "http:\/\/ceph1.oak.vast.com:7480"
> ],
> "log_meta": "true",
> "log_data": "false",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "67e26f6b-4774-4b14-9668-a5cf76b9e9ce"
> }
> }
> ],
> "master_zonegroup": "dbec3557-87bb-4460-8546-b59b4fde7e10",
> "bucket_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> },
> "user_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> }
> }
>
> Ceph version is:
> ceph --version
> ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
>
> Any advises?
>

First I recommend using zonegroup for jewel as region was renamed to zonegroup.
How did you create/update the zones and zonegroup?
Did you executed period update?

Orit

>
> Thanks in advance
>
> Marko Stojanovic
>
>
>
> ___
> ceph-users mailing list

Re: [ceph-users] radosgw setup issue

[Orit Wasserman]

On Wed, Jan 4, 2017 at 7:08 PM, Brian Andrus <brian.and...@dreamhost.com> wrote:
> Regardless of whether it worked before, have you verified your RadosGWs have
> write access to monitors? They will need it if you want the RadosGW to
> create its own pools.
>
> ceph auth get 
>

I agree, it could be permissions issue

> On Wed, Jan 4, 2017 at 8:59 AM, Kamble, Nitin A <nitin.kam...@teradata.com>
> wrote:
>>
>>
>> > On Dec 26, 2016, at 2:48 AM, Orit Wasserman <owass...@redhat.com> wrote:
>> >
>> > On Fri, Dec 23, 2016 at 3:42 AM, Kamble, Nitin A
>> > <nitin.kam...@teradata.com> wrote:
>> >> I am trying to setup radosgw on a ceph cluster, and I am seeing some
>> >> issues where google is not helping. I hope some of the developers would be
>> >> able to help here.
>> >>
>> >>
>> >> I tried to create radosgw as mentioned here [0] on a jewel cluster. And
>> >> it gives the following error in log file after starting radosgw.
>> >>
>> >>
>> >> 2016-12-22 17:36:46.755786 7f084beeb9c0  0 set uid:gid to 167:167
>> >> (ceph:ceph)
>> >> 2016-12-22 17:36:46.755849 7f084beeb9c0  0 ceph version
>> >> 10.2.2-118-g894a5f8 (894a5f8d878d4b267f80b90a4bffce157f2b4ba7), process
>> >> radosgw, pid 10092
>> >> 2016-12-22 17:36:46.763821 7f084beeb9c0  1 -- :/0 messenger.start
>> >> 2016-12-22 17:36:46.764731 7f084beeb9c0  1 -- :/1011033520 -->
>> >> 39.0.16.7:6789/0 -- auth(proto 0 40 bytes epoch 0) v1 -- ?+0 
>> >> 0x7f084c8e9f60
>> >> con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.765055 7f084beda700  1 -- 39.0.16.9:0/1011033520
>> >> learned my addr 39.0.16.9:0/1011033520
>> >> 2016-12-22 17:36:46.765492 7f082a7fc700  1 -- 39.0.16.9:0/1011033520
>> >> <== mon.0 39.0.16.7:6789/0 1  mon_map magic: 0 v1  195+0+0
>> >> (146652916 0 0) 0x7f0814000a60 con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.765562 7f082a7fc700  1 -- 39.0.16.9:0/1011033520
>> >> <== mon.0 39.0.16.7:6789/0 2  auth_reply(proto 2 0 (0) Success) v1 
>> >> 
>> >> 33+0+0 (1206278719 0 0) 0x7f0814000ee0 con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.765697 7f082a7fc700  1 -- 39.0.16.9:0/1011033520
>> >> --> 39.0.16.7:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0
>> >> 0x7f08180013b0 con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.765968 7f082a7fc700  1 -- 39.0.16.9:0/1011033520
>> >> <== mon.0 39.0.16.7:6789/0 3  auth_reply(proto 2 0 (0) Success) v1 
>> >> 
>> >> 222+0+0 (4230455906 0 0) 0x7f0814000ee0 con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.766053 7f082a7fc700  1 -- 39.0.16.9:0/1011033520
>> >> --> 39.0.16.7:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- ?+0
>> >> 0x7f0818001830 con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.766315 7f082a7fc700  1 -- 39.0.16.9:0/1011033520
>> >> <== mon.0 39.0.16.7:6789/0 4  auth_reply(proto 2 0 (0) Success) v1 
>> >> 
>> >> 425+0+0 (3179848142 0 0) 0x7f0814001180 con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.766383 7f082a7fc700  1 -- 39.0.16.9:0/1011033520
>> >> --> 39.0.16.7:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 
>> >> 0x7f084c8ea440
>> >> con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.766452 7f084beeb9c0  1 -- 39.0.16.9:0/1011033520
>> >> --> 39.0.16.7:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x7f084c8ea440
>> >> con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.766518 7f082a7fc700  1 -- 39.0.16.9:0/1011033520
>> >> <== mon.0 39.0.16.7:6789/0 5  mon_map magic: 0 v1  195+0+0
>> >> (146652916 0 0) 0x7f0814001110 con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.766671 7f08227fc700  2
>> >> RGWDataChangesLog::ChangesRenewThread: start
>> >> 2016-12-22 17:36:46.766691 7f084beeb9c0 20 get_system_obj_state:
>> >> rctx=0x7ffec2850d00 obj=.rgw.root:default.realm state=0x7f084c8efdf8
>> >> s->prefetch_data=0
>> >> 2016-12-22 17:36:46.766750 7f082a7fc700  1 -- 39.0.16.9:0/1011033520
>> >> <== mon.0 39.0.16.7:6789/0 6  osd_map(9506..9506 src has 8863..9506) 
>> >> v3
>> >>  66915+0+0 (689048617 0 0) 0x7f0814011680 con 0x7f084c8e9480
>> >> 2016-12-22 17:36:46.767029 7f084beeb9c0  1 -- 39.0.16.9:0/1011033520
>> >> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0
>> >> 0x7f084c8f05f0 con 

Re: [ceph-users] radosgw setup issue

[Orit Wasserman]

On Fri, Dec 23, 2016 at 3:42 AM, Kamble, Nitin A
 wrote:
> I am trying to setup radosgw on a ceph cluster, and I am seeing some issues 
> where google is not helping. I hope some of the developers would be able to 
> help here.
>
>
> I tried to create radosgw as mentioned here [0] on a jewel cluster. And it 
> gives the following error in log file after starting radosgw.
>
>
> 2016-12-22 17:36:46.755786 7f084beeb9c0  0 set uid:gid to 167:167 (ceph:ceph)
> 2016-12-22 17:36:46.755849 7f084beeb9c0  0 ceph version 10.2.2-118-g894a5f8 
> (894a5f8d878d4b267f80b90a4bffce157f2b4ba7), process radosgw, pid 10092
> 2016-12-22 17:36:46.763821 7f084beeb9c0  1 -- :/0 messenger.start
> 2016-12-22 17:36:46.764731 7f084beeb9c0  1 -- :/1011033520 --> 
> 39.0.16.7:6789/0 -- auth(proto 0 40 bytes epoch 0) v1 -- ?+0 0x7f084c8e9f60 
> con 0x7f084c8e9480
> 2016-12-22 17:36:46.765055 7f084beda700  1 -- 39.0.16.9:0/1011033520 learned 
> my addr 39.0.16.9:0/1011033520
> 2016-12-22 17:36:46.765492 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 <== 
> mon.0 39.0.16.7:6789/0 1  mon_map magic: 0 v1  195+0+0 (146652916 0 
> 0) 0x7f0814000a60 con 0x7f084c8e9480
> 2016-12-22 17:36:46.765562 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 <== 
> mon.0 39.0.16.7:6789/0 2  auth_reply(proto 2 0 (0) Success) v1  
> 33+0+0 (1206278719 0 0) 0x7f0814000ee0 con 0x7f084c8e9480
> 2016-12-22 17:36:46.765697 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 --> 
> 39.0.16.7:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7f08180013b0 
> con 0x7f084c8e9480
> 2016-12-22 17:36:46.765968 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 <== 
> mon.0 39.0.16.7:6789/0 3  auth_reply(proto 2 0 (0) Success) v1  
> 222+0+0 (4230455906 0 0) 0x7f0814000ee0 con 0x7f084c8e9480
> 2016-12-22 17:36:46.766053 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 --> 
> 39.0.16.7:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- ?+0 0x7f0818001830 
> con 0x7f084c8e9480
> 2016-12-22 17:36:46.766315 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 <== 
> mon.0 39.0.16.7:6789/0 4  auth_reply(proto 2 0 (0) Success) v1  
> 425+0+0 (3179848142 0 0) 0x7f0814001180 con 0x7f084c8e9480
> 2016-12-22 17:36:46.766383 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 --> 
> 39.0.16.7:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f084c8ea440 con 
> 0x7f084c8e9480
> 2016-12-22 17:36:46.766452 7f084beeb9c0  1 -- 39.0.16.9:0/1011033520 --> 
> 39.0.16.7:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x7f084c8ea440 con 
> 0x7f084c8e9480
> 2016-12-22 17:36:46.766518 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 <== 
> mon.0 39.0.16.7:6789/0 5  mon_map magic: 0 v1  195+0+0 (146652916 0 
> 0) 0x7f0814001110 con 0x7f084c8e9480
> 2016-12-22 17:36:46.766671 7f08227fc700  2 
> RGWDataChangesLog::ChangesRenewThread: start
> 2016-12-22 17:36:46.766691 7f084beeb9c0 20 get_system_obj_state: 
> rctx=0x7ffec2850d00 obj=.rgw.root:default.realm state=0x7f084c8efdf8 
> s->prefetch_data=0
> 2016-12-22 17:36:46.766750 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 <== 
> mon.0 39.0.16.7:6789/0 6  osd_map(9506..9506 src has 8863..9506) v3  
> 66915+0+0 (689048617 0 0) 0x7f0814011680 con 0x7f084c8e9480
> 2016-12-22 17:36:46.767029 7f084beeb9c0  1 -- 39.0.16.9:0/1011033520 --> 
> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 
> 0x7f084c8f05f0 con 0x7f084c8e9480
> 2016-12-22 17:36:46.767163 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 <== 
> mon.0 39.0.16.7:6789/0 7  mon_get_version_reply(handle=1 version=9506) v2 
>  24+0+0 (2817198406 0 0) 0x7f0814001110 con 0x7f084c8e9480
> 2016-12-22 17:36:46.767214 7f084beeb9c0 20 get_system_obj_state: 
> rctx=0x7ffec2850210 obj=.rgw.root:default.realm state=0x7f084c8efdf8 
> s->prefetch_data=0
> 2016-12-22 17:36:46.767231 7f084beeb9c0  1 -- 39.0.16.9:0/1011033520 --> 
> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=2) v1 -- ?+0 
> 0x7f084c8f0ac0 con 0x7f084c8e9480
> 2016-12-22 17:36:46.767341 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 <== 
> mon.0 39.0.16.7:6789/0 8  mon_get_version_reply(handle=2 version=9506) v2 
>  24+0+0 (1826043941 0 0) 0x7f0814001110 con 0x7f084c8e9480
> 2016-12-22 17:36:46.767367 7f084beeb9c0 10 could not read realm id: (2) No 
> such file or directory
> 2016-12-22 17:36:46.767390 7f084beeb9c0  1 -- 39.0.16.9:0/1011033520 --> 
> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=3) v1 -- ?+0 
> 0x7f084c8efe50 con 0x7f084c8e9480
> 2016-12-22 17:36:46.767496 7f082a7fc700  1 -- 39.0.16.9:0/1011033520 <== 
> mon.0 39.0.16.7:6789/0 9  mon_get_version_reply(handle=3 version=9506) v2 
>  24+0+0 (3600349867 0 0) 0x7f0814001110 con 0x7f084c8e9480
> 2016-12-22 17:36:46.767518 7f084beeb9c0 10 failed to list objects 
> pool_iterate_begin() returned r=-2
> 2016-12-22 17:36:46.767542 7f084beeb9c0 20 get_system_obj_state: 
> rctx=0x7ffec2850420 obj=.rgw.root:zone_names.default state=0x7f084c8f0f38 
> s->prefetch_data=0
> 2016-12-22 17:36:46.767554 7f084beeb9c0  1 -- 39.0.16.9:0/1011033520 

Re: [ceph-users] rgw leaking data, orphan search loop

[Orit Wasserman]

HI Maruis,

On Thu, Dec 22, 2016 at 12:00 PM, Marius Vaitiekunas
<mariusvaitieku...@gmail.com> wrote:
> On Thu, Dec 22, 2016 at 11:58 AM, Marius Vaitiekunas
> <mariusvaitieku...@gmail.com> wrote:
>>
>> Hi,
>>
>> 1) I've written before into mailing list, but one more time. We have big
>> issues recently with rgw on jewel. because of leaked data - the rate is
>> about 50GB/hour.
>>
>> We've hitted these bugs:
>> rgw: fix put_acls for objects starting and ending with underscore
>> (issue#17625, pr#11669, Orit Wasserman)
>>
>> Upgraded to jewel 10.2.5 - no luck.
>>
>> Also we've hitted this one:
>> rgw: RGW loses realm/period/zonegroup/zone data: period overwritten if
>> somewhere in the cluster is still running Hammer (issue#17371, pr#11519,
>> Orit Wasserman)
>>
>> Fixed zonemaps - also no luck.
>>
>> We do not use multisite - only default realm, zonegroup, zone.
>>
>> We have no more ideas, how these data leak could happen. gc is working -
>> we can see it in rgw logs.
>>
>> Maybe, someone could give any hint about this? Where should we look?
>>
>>
>> 2) Another story is about removing all the leaked/orphan objects.
>> radosgw-admin orphans find enters the loop state on stage when it starts
>> linking objects.
>>
>> We've tried to change the number of shards to 16, 64 (default), 512. At
>> the moment it's running with shards number 1.
>>
>> Again, any ideas how to make orphan search happen?
>>
>>
>> I could provide any logs, configs, etc. if someone is ready to help on
>> this case.
>>
>>

How many buckets do you have ? how many object in each?
Can you provide the output of rados ls -p .rgw.buckets ?

Orit

>
> Sorry. I forgot to mention, that we've registered two issues on tracker:
> http://tracker.ceph.com/issues/18331
> http://tracker.ceph.com/issues/18258
>
> --
> Marius Vaitiekūnas
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cannot commit period: period does not have a master zone of a master zonegroup

[Orit Wasserman]

On Tue, Dec 20, 2016 at 5:39 PM, Wido den Hollander <w...@42on.com> wrote:
>
>> Op 15 december 2016 om 17:10 schreef Orit Wasserman <owass...@redhat.com>:
>>
>>
>> Hi Wido,
>>
>> This looks like you are hitting http://tracker.ceph.com/issues/17364
>> The fix is being backported to jewel: https://github.com/ceph/ceph/pull/12315
>>
>> A workaround:
>> save the realm, zonegroup and zones json file
>> make a copy of .rgw.root (the pool contain the multisite config)
>> remove .rgw.root
>> stop the gateway
>> radosgw-admin realm set < json
>> radosgw-admin zonegroup set < json
>> raodsgw-admin zone set < json
>> radosgw-admin period update --commit
>> start the gateway
>>
>> If the realm set will give you problems you can create a new realm
>> and will need to update the realm id in the zonegroup and zones json
>> files before using them
>>
>
> I eventually ended up doing that indeed. Setting a realm from a backup 
> doesn't work.
>

I suspect that, can you open an tracker issue?

> So my sequence in commands:
>
> NOTE THE UUID OF THE REALM AND CHANGE IN JSON FILES
>
> nano zm1.json
> nano zonegroup.json
>
> radosgw-admin zonegroup set --rgw-zonegroup gn < zonegroup.json
> radosgw-admin zone set --rgw-zonegroup gn --rgw-zone zm1 < zm1.json
> radosgw-admin zonegroup default --rgw-zonegroup gn
> radosgw-admin zone default --rgw-zone zm1
> radosgw-admin period update
> radosgw-admin period update --commit
>
> This eventually got things working again.
>
Good

> The only thing I keep seeing everywhere:
>
> root@alpha:~# radosgw-admin period update --commit
> 2016-12-20 16:38:07.958860 7f9571697a00  0 error in read_id for id  : (2) No 
> such file or directory
> 2016-12-20 16:38:07.960035 7f9571697a00  0 error in read_id for id  : (2) No 
> such file or directory
>

I am guessing this is not an error just a message that should have
higher log level,
can you open an issue?

> Brought me to:
>
> - http://tracker.ceph.com/issues/15776
> - https://github.com/ceph/ceph/pull/8994
>
> Doesn't seem to be backported to 10.2.5 however.
>

Strange it should be part of jewel , I will look into it.

> Wido
>
>> Orit
>>
>>
>> On Thu, Dec 15, 2016 at 4:47 PM, Wido den Hollander <w...@42on.com> wrote:
>> > Hi,
>> >
>> > On a Ceph cluster running Jewel 10.2.5 I'm running into a problem.
>> >
>> > I want to change the amount of shards:
>> >
>> > # radosgw-admin zonegroup-map get > zonegroup.json
>> > # nano zonegroup.json
>> > # radosgw-admin zonegroup-map set --infile zonegroup.json
>> > # radosgw-admin period update --commit
>> >
>> > Now, the error arrises:
>> >
>> > cannot commit period: period does not have a master zone of a master 
>> > zonegroup
>> > failed to commit period: (22) Invalid argument
>> >
>> > Looking at the output:
>> >
>> > # radosgw-admin period update
>> >
>> > {
>> > ...
>> > "master_zonegroup": "",
>> > "master_zone": "",
>> > ...
>> > }
>> >
>> > # radosgw-admin zone list
>> >
>> > {
>> > "default_info": "zm1",
>> > "zones": [
>> > "default",
>> > "zm1"
>> > ]
>> > }
>> >
>> > To me it seems like there is something wrong with the period since there 
>> > is no UUID present in master_zone/zonegroup.
>> >
>> > Any idea on how to fix this?
>> >
>> > Wido
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cannot commit period: period does not have a master zone of a master zonegroup

[Orit Wasserman]

Hi Wido,

This looks like you are hitting http://tracker.ceph.com/issues/17364
The fix is being backported to jewel: https://github.com/ceph/ceph/pull/12315

A workaround:
save the realm, zonegroup and zones json file
make a copy of .rgw.root (the pool contain the multisite config)
remove .rgw.root
stop the gateway
radosgw-admin realm set < json
radosgw-admin zonegroup set < json
raodsgw-admin zone set < json
radosgw-admin period update --commit
start the gateway

If the realm set will give you problems you can create a new realm
and will need to update the realm id in the zonegroup and zones json
files before using them

Orit


On Thu, Dec 15, 2016 at 4:47 PM, Wido den Hollander  wrote:
> Hi,
>
> On a Ceph cluster running Jewel 10.2.5 I'm running into a problem.
>
> I want to change the amount of shards:
>
> # radosgw-admin zonegroup-map get > zonegroup.json
> # nano zonegroup.json
> # radosgw-admin zonegroup-map set --infile zonegroup.json
> # radosgw-admin period update --commit
>
> Now, the error arrises:
>
> cannot commit period: period does not have a master zone of a master zonegroup
> failed to commit period: (22) Invalid argument
>
> Looking at the output:
>
> # radosgw-admin period update
>
> {
> ...
> "master_zonegroup": "",
> "master_zone": "",
> ...
> }
>
> # radosgw-admin zone list
>
> {
> "default_info": "zm1",
> "zones": [
> "default",
> "zm1"
> ]
> }
>
> To me it seems like there is something wrong with the period since there is 
> no UUID present in master_zone/zonegroup.
>
> Any idea on how to fix this?
>
> Wido
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OpenStack Keystone with RadosGW

[Orit Wasserman]

radosgw supports keystone v3 in Jewel.
Can you give more details about the error? what is the exact command
are you trying?
radosgw log with debug_rgw=20 and debug_ms=5 will be most helpfull

On Tue, Nov 22, 2016 at 10:24 AM, 한승진  wrote:
> I've figured out the main reason is.
>
> When swift client request through keystone user like 'admin', keystone
> returned with X-Auth-Token header.
>
> After that, the swift client requests with X-Auth-Token to radosgw, but
> radosgw returned 'AccessDenied'
>
> Some people says radosgw doesn't support keystone identity version 3 yet.
>
>
> 2016-11-22 15:41 GMT+09:00 한승진 :
>>
>> Hi All,
>>
>> I am trying to implement radosgw with Openstack as an object storage
>> service.
>>
>> I think there are 2 cases for using radosgw as an object storage
>>
>> First, Keystone <-> Ceph connect directly.
>>
>> like below guide..
>>
>> http://docs.ceph.com/docs/master/radosgw/keystone/
>>
>> Second, use ceph as a back-end of swift.
>>
>> like below guide..
>>
>> https://github.com/openstack/swift-ceph-backend#installation
>>
>> In first case, It issues always 405 error therefore I cannot go forward
>> any more.
>>
>> In second case, I don't know how to make ring builder in ceph backend
>> environment.
>>
>> Is anybody use radosgw with OpenStack? Please give me a guide.
>>
>> Thanks.
>>
>> John.
>>
>> =
>> Here is my ceph.conf configurations
>>
>> [client.radosgw.cephmon01]
>> rgw keystone api version = 3
>> rgw keystone url =  http://controller:35357
>> rgw keystone admin user = swift
>> rgw keystone admin password = *
>> rgw keystone admin project = service
>> rgw keystone admin domain = default
>> rgw keystone accepted roles =  admin,user
>>
>> rgw s3 auth use keystone = true
>> rgw keystone verify ssl = false
>>
>>
>>
>>
>>
>>
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] index-sharding on existing bucket ?

[Orit Wasserman]

Hi,
We have support for offline bucket resharding admin command:
https://github.com/ceph/ceph/pull/11230.
It will be available in Jewel 10.2.5.

Orit



On Thu, Nov 17, 2016 at 9:11 PM, Yoann Moulin  wrote:
> Hello,
>
> is that possible to shard the index of existing buckets ?
>
> I have more than 100TB of data in a couples of buckets, I'd like to avoid to 
> re upload everythings.
>
> Thanks for your help,
>
> --
> Yoann Moulin
> EPFL IC-IT
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw - http status 400 while creating a bucket

[Orit Wasserman]

On Fri, Nov 11, 2016 at 11:27 PM, Andrei Mikhailovsky <and...@arhont.com> wrote:
> Hi Orit,
>
> Many thanks. I will try that over the weekend and let you know.
>
> Are you sure removing the pool will not destroy my data, user info and 
> buckets?
>

No this pool (.rgw.root) contains only the configuration (realm,
zonegroups and zones).
If you want to back it up run:
# rados mkpool .rgw.root.backup
# rados cppool .rgw.root .rgw.root.backup

Orit
> Thanks
>
> ----- Original Message -
>> From: "Orit Wasserman" <owass...@redhat.com>
>> To: "andrei" <and...@arhont.com>
>> Cc: "Yoann Moulin" <yoann.mou...@epfl.ch>, "ceph-users" 
>> <ceph-users@lists.ceph.com>
>> Sent: Friday, 11 November, 2016 11:24:51
>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket
>
>> I have a workaround:
>>
>> 1. Use  zonegroup and zone jsons you have from before (default-zg.json
>> and default-zone.json)
>> 2. Make sure the realm id in the jsons is ""
>> 3. Stop the gateways
>> 4. Remove .rgw.root  pool(you can back it up if you want to by using
>> mkpool and cppool commands):
>>rados rm .rgw.root
>> 5. radosgw-admin realm create --rgw-realm=myrealm
>> 6. radosgw-admin zonegroup set --rgw-zonegroup=default  --default <
>> default-zg.json
>> 7. radosgw-admin zone set --rgw-zone=default --deault < default-zone.json
>> 8. radosgw-admin period update --commit
>>
>> Good luck,
>> Orit
>>
>> On Thu, Nov 10, 2016 at 7:08 PM, Andrei Mikhailovsky <and...@arhont.com> 
>> wrote:
>>>
>>> Orit, here is the output:
>>>
>>> root@arh-ibstorage2-ib:~# rados ls -p .rgw.root
>>> region_map
>>> default.zone.5b41b1b2-0f92-463d-b582-07552f83e66c
>>> realms.5b41b1b2-0f92-463d-b582-07552f83e66c
>>> zonegroups_names.default
>>> zone_names.default
>>> periods.a9543371-a073-4d73-ab6d-0f54991c7ad9.1
>>> realms_names.default
>>> realms_names.london-ldex
>>> realms.f08592f2-5d53-4701-a895-b780b16b5374
>>> periods.286475fa-625b-4fdb-97bf-dcec4b437960.latest_epoch
>>> periods.5b41b1b2-0f92-463d-b582-07552f83e66c:staging
>>> periods.286475fa-625b-4fdb-97bf-dcec4b437960.1
>>> default.realm
>>> default.zonegroup.5b41b1b2-0f92-463d-b582-07552f83e66c
>>> periods.a9543371-a073-4d73-ab6d-0f54991c7ad9.latest_epoch
>>> periods.5b41b1b2-0f92-463d-b582-07552f83e66c:staging.latest_epoch
>>> realms.f08592f2-5d53-4701-a895-b780b16b5374.control
>>> zone_info.default
>>> zonegroup_info.default
>>> realms.5b41b1b2-0f92-463d-b582-07552f83e66c.control
>>>
>>>
>>> Thanks
>>>
>>> - Original Message -
>>>> From: "Orit Wasserman" <owass...@redhat.com>
>>>> To: "Andrei Mikhailovsky" <and...@arhont.com>
>>>> Cc: "Yoann Moulin" <yoann.mou...@epfl.ch>, "ceph-users"
>>>> <ceph-users@lists.ceph.com>
>>>> Sent: Thursday, 10 November, 2016 15:22:16
>>>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket
>>>
>>>> On Thu, Nov 10, 2016 at 3:32 PM, Andrei Mikhailovsky <and...@arhont.com> 
>>>> wrote:
>>>>> Orit, true.
>>>>>
>>>>> yeah, all my servers are running 10.2.3-1xenial or 10.2.3-1trusty. I have 
>>>>> a
>>>>> small cluster and I always update all servers at once.
>>>>>
>>>>> I don't have any Hammer releases of ceph anywhere on the network.
>>>>>
>>>>
>>>> can you run: rados ls .rgw.root?
>>>>
>>>>> Is 10.2.4 out already? I didn't see an update package to that.
>>>>>
>>>>
>>>> It should be out soon
>>>>
>>>>> Thanks
>>>>>
>>>>> Andrei
>>>>>
>>>>> - Original Message -
>>>>>> From: "Orit Wasserman" <owass...@redhat.com>
>>>>>> To: "Andrei Mikhailovsky" <and...@arhont.com>
>>>>>> Cc: "Yoann Moulin" <yoann.mou...@epfl.ch>, "ceph-users"
>>>>>> <ceph-users@lists.ceph.com>
>>>>>> Sent: Thursday, 10 November, 2016 13:58:32
>>>>>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a 
>>>>>> bucket
>>>>>
>>>

Re: [ceph-users] radosgw - http status 400 while creating a bucket

[Orit Wasserman]

On Fri, Nov 11, 2016 at 12:24 PM, Orit Wasserman <owass...@redhat.com> wrote:
> I have a workaround:
>
> 1. Use  zonegroup and zone jsons you have from before (default-zg.json
> and default-zone.json)
> 2. Make sure the realm id in the jsons is ""
> 3. Stop the gateways
> 4. Remove .rgw.root  pool(you can back it up if you want to by using
> mkpool and cppool commands):
> rados rm .rgw.root
it should be:
rados rmpool .rgw.root

> 5. radosgw-admin realm create --rgw-realm=myrealm
> 6. radosgw-admin zonegroup set --rgw-zonegroup=default  --default <
> default-zg.json
> 7. radosgw-admin zone set --rgw-zone=default --deault < default-zone.json
> 8. radosgw-admin period update --commit
>
> Good luck,
> Orit
>
> On Thu, Nov 10, 2016 at 7:08 PM, Andrei Mikhailovsky <and...@arhont.com> 
> wrote:
>>
>> Orit, here is the output:
>>
>> root@arh-ibstorage2-ib:~# rados ls -p .rgw.root
>> region_map
>> default.zone.5b41b1b2-0f92-463d-b582-07552f83e66c
>> realms.5b41b1b2-0f92-463d-b582-07552f83e66c
>> zonegroups_names.default
>> zone_names.default
>> periods.a9543371-a073-4d73-ab6d-0f54991c7ad9.1
>> realms_names.default
>> realms_names.london-ldex
>> realms.f08592f2-5d53-4701-a895-b780b16b5374
>> periods.286475fa-625b-4fdb-97bf-dcec4b437960.latest_epoch
>> periods.5b41b1b2-0f92-463d-b582-07552f83e66c:staging
>> periods.286475fa-625b-4fdb-97bf-dcec4b437960.1
>> default.realm
>> default.zonegroup.5b41b1b2-0f92-463d-b582-07552f83e66c
>> periods.a9543371-a073-4d73-ab6d-0f54991c7ad9.latest_epoch
>> periods.5b41b1b2-0f92-463d-b582-07552f83e66c:staging.latest_epoch
>> realms.f08592f2-5d53-4701-a895-b780b16b5374.control
>> zone_info.default
>> zonegroup_info.default
>> realms.5b41b1b2-0f92-463d-b582-07552f83e66c.control
>>
>>
>> Thanks
>>
>> - Original Message -
>>> From: "Orit Wasserman" <owass...@redhat.com>
>>> To: "Andrei Mikhailovsky" <and...@arhont.com>
>>> Cc: "Yoann Moulin" <yoann.mou...@epfl.ch>, "ceph-users" 
>>> <ceph-users@lists.ceph.com>
>>> Sent: Thursday, 10 November, 2016 15:22:16
>>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket
>>
>>> On Thu, Nov 10, 2016 at 3:32 PM, Andrei Mikhailovsky <and...@arhont.com> 
>>> wrote:
>>>> Orit, true.
>>>>
>>>> yeah, all my servers are running 10.2.3-1xenial or 10.2.3-1trusty. I have a
>>>> small cluster and I always update all servers at once.
>>>>
>>>> I don't have any Hammer releases of ceph anywhere on the network.
>>>>
>>>
>>> can you run: rados ls .rgw.root?
>>>
>>>> Is 10.2.4 out already? I didn't see an update package to that.
>>>>
>>>
>>> It should be out soon
>>>
>>>> Thanks
>>>>
>>>> Andrei
>>>>
>>>> - Original Message -
>>>>> From: "Orit Wasserman" <owass...@redhat.com>
>>>>> To: "Andrei Mikhailovsky" <and...@arhont.com>
>>>>> Cc: "Yoann Moulin" <yoann.mou...@epfl.ch>, "ceph-users"
>>>>> <ceph-users@lists.ceph.com>
>>>>> Sent: Thursday, 10 November, 2016 13:58:32
>>>>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a 
>>>>> bucket
>>>>
>>>>> On Thu, Nov 10, 2016 at 2:55 PM, Andrei Mikhailovsky <and...@arhont.com> 
>>>>> wrote:
>>>>>> Orit,
>>>>>>
>>>>>> Here is what i've done just now:
>>>>>>
>>>>>> root@arh-ibstorage1-ib:~# service ceph-radosgw@radosgw.gateway stop
>>>>>>
>>>>>> (the above command was ran on both radosgw servers). Checked with ps and 
>>>>>> no
>>>>>> radosgw services were running. After that I've done:
>>>>>>
>>>>>>
>>>>>>
>>>>>> root@arh-ibstorage1-ib:~# ./ceph-zones-fix.sh
>>>>>> + RADOSGW_ADMIN=radosgw-admin
>>>>>> + echo Exercise initialization code
>>>>>> Exercise initialization code
>>>>>> + radosgw-admin user info --uid=foo
>>>>>> could not fetch user info: no user info saved
>>>>>> + echo Get default zonegroup
>>>>>> Get default zonegroup
>>>>>> + rad

Re: [ceph-users] radosgw - http status 400 while creating a bucket

[Orit Wasserman]

I have a workaround:

1. Use  zonegroup and zone jsons you have from before (default-zg.json
and default-zone.json)
2. Make sure the realm id in the jsons is ""
3. Stop the gateways
4. Remove .rgw.root  pool(you can back it up if you want to by using
mkpool and cppool commands):
rados rm .rgw.root
5. radosgw-admin realm create --rgw-realm=myrealm
6. radosgw-admin zonegroup set --rgw-zonegroup=default  --default <
default-zg.json
7. radosgw-admin zone set --rgw-zone=default --deault < default-zone.json
8. radosgw-admin period update --commit

Good luck,
Orit

On Thu, Nov 10, 2016 at 7:08 PM, Andrei Mikhailovsky <and...@arhont.com> wrote:
>
> Orit, here is the output:
>
> root@arh-ibstorage2-ib:~# rados ls -p .rgw.root
> region_map
> default.zone.5b41b1b2-0f92-463d-b582-07552f83e66c
> realms.5b41b1b2-0f92-463d-b582-07552f83e66c
> zonegroups_names.default
> zone_names.default
> periods.a9543371-a073-4d73-ab6d-0f54991c7ad9.1
> realms_names.default
> realms_names.london-ldex
> realms.f08592f2-5d53-4701-a895-b780b16b5374
> periods.286475fa-625b-4fdb-97bf-dcec4b437960.latest_epoch
> periods.5b41b1b2-0f92-463d-b582-07552f83e66c:staging
> periods.286475fa-625b-4fdb-97bf-dcec4b437960.1
> default.realm
> default.zonegroup.5b41b1b2-0f92-463d-b582-07552f83e66c
> periods.a9543371-a073-4d73-ab6d-0f54991c7ad9.latest_epoch
> periods.5b41b1b2-0f92-463d-b582-07552f83e66c:staging.latest_epoch
> realms.f08592f2-5d53-4701-a895-b780b16b5374.control
> zone_info.default
> zonegroup_info.default
> realms.5b41b1b2-0f92-463d-b582-07552f83e66c.control
>
>
> Thanks
>
> - Original Message -
>> From: "Orit Wasserman" <owass...@redhat.com>
>> To: "Andrei Mikhailovsky" <and...@arhont.com>
>> Cc: "Yoann Moulin" <yoann.mou...@epfl.ch>, "ceph-users" 
>> <ceph-users@lists.ceph.com>
>> Sent: Thursday, 10 November, 2016 15:22:16
>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket
>
>> On Thu, Nov 10, 2016 at 3:32 PM, Andrei Mikhailovsky <and...@arhont.com> 
>> wrote:
>>> Orit, true.
>>>
>>> yeah, all my servers are running 10.2.3-1xenial or 10.2.3-1trusty. I have a
>>> small cluster and I always update all servers at once.
>>>
>>> I don't have any Hammer releases of ceph anywhere on the network.
>>>
>>
>> can you run: rados ls .rgw.root?
>>
>>> Is 10.2.4 out already? I didn't see an update package to that.
>>>
>>
>> It should be out soon
>>
>>> Thanks
>>>
>>> Andrei
>>>
>>> - Original Message -
>>>> From: "Orit Wasserman" <owass...@redhat.com>
>>>> To: "Andrei Mikhailovsky" <and...@arhont.com>
>>>> Cc: "Yoann Moulin" <yoann.mou...@epfl.ch>, "ceph-users"
>>>> <ceph-users@lists.ceph.com>
>>>> Sent: Thursday, 10 November, 2016 13:58:32
>>>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket
>>>
>>>> On Thu, Nov 10, 2016 at 2:55 PM, Andrei Mikhailovsky <and...@arhont.com> 
>>>> wrote:
>>>>> Orit,
>>>>>
>>>>> Here is what i've done just now:
>>>>>
>>>>> root@arh-ibstorage1-ib:~# service ceph-radosgw@radosgw.gateway stop
>>>>>
>>>>> (the above command was ran on both radosgw servers). Checked with ps and 
>>>>> no
>>>>> radosgw services were running. After that I've done:
>>>>>
>>>>>
>>>>>
>>>>> root@arh-ibstorage1-ib:~# ./ceph-zones-fix.sh
>>>>> + RADOSGW_ADMIN=radosgw-admin
>>>>> + echo Exercise initialization code
>>>>> Exercise initialization code
>>>>> + radosgw-admin user info --uid=foo
>>>>> could not fetch user info: no user info saved
>>>>> + echo Get default zonegroup
>>>>> Get default zonegroup
>>>>> + radosgw-admin zonegroup get --rgw-zonegroup=default
>>>>> + sed s/"id":.*/"id": "default",/g
>>>>> + sed s/"master_zone.*/"master_zone": "default",/g
>>>>> + echo Get default zone
>>>>> Get default zone
>>>>> + radosgw-admin zone get --zone-id=default
>>>>> + echo Creating realm
>>>>> Creating realm
>>>>> + radosgw-admin realm create --rgw-realm=london-ldex
>>>>> ERROR: couldn't create realm london-ldex: (17) F

Re: [ceph-users] radosgw - http status 400 while creating a bucket

[Orit Wasserman]

On Thu, Nov 10, 2016 at 3:32 PM, Andrei Mikhailovsky <and...@arhont.com> wrote:
> Orit, true.
>
> yeah, all my servers are running 10.2.3-1xenial or 10.2.3-1trusty. I have a 
> small cluster and I always update all servers at once.
>
> I don't have any Hammer releases of ceph anywhere on the network.
>

can you run: rados ls .rgw.root?

> Is 10.2.4 out already? I didn't see an update package to that.
>

It should be out soon

> Thanks
>
> Andrei
>
> - Original Message -
>> From: "Orit Wasserman" <owass...@redhat.com>
>> To: "Andrei Mikhailovsky" <and...@arhont.com>
>> Cc: "Yoann Moulin" <yoann.mou...@epfl.ch>, "ceph-users" 
>> <ceph-users@lists.ceph.com>
>> Sent: Thursday, 10 November, 2016 13:58:32
>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket
>
>> On Thu, Nov 10, 2016 at 2:55 PM, Andrei Mikhailovsky <and...@arhont.com> 
>> wrote:
>>> Orit,
>>>
>>> Here is what i've done just now:
>>>
>>> root@arh-ibstorage1-ib:~# service ceph-radosgw@radosgw.gateway stop
>>>
>>> (the above command was ran on both radosgw servers). Checked with ps and no
>>> radosgw services were running. After that I've done:
>>>
>>>
>>>
>>> root@arh-ibstorage1-ib:~# ./ceph-zones-fix.sh
>>> + RADOSGW_ADMIN=radosgw-admin
>>> + echo Exercise initialization code
>>> Exercise initialization code
>>> + radosgw-admin user info --uid=foo
>>> could not fetch user info: no user info saved
>>> + echo Get default zonegroup
>>> Get default zonegroup
>>> + radosgw-admin zonegroup get --rgw-zonegroup=default
>>> + sed s/"id":.*/"id": "default",/g
>>> + sed s/"master_zone.*/"master_zone": "default",/g
>>> + echo Get default zone
>>> Get default zone
>>> + radosgw-admin zone get --zone-id=default
>>> + echo Creating realm
>>> Creating realm
>>> + radosgw-admin realm create --rgw-realm=london-ldex
>>> ERROR: couldn't create realm london-ldex: (17) File exists
>>> 2016-11-10 13:44:48.872839 7f87a13d9a00  0 ERROR creating new realm object
>>> london-ldex: (17) File exists
>>> + echo Creating default zonegroup
>>> Creating default zonegroup
>>> + radosgw-admin zonegroup set --rgw-zonegroup=default
>>> {
>>> "id": "default",
>>> "name": "default",
>>> "api_name": "",
>>> "is_master": "true",
>>> "endpoints": [],
>>> "hostnames": [],
>>> "hostnames_s3website": [],
>>> "master_zone": "default",
>>> "zones": [
>>> {
>>> "id": "default",
>>> "name": "default",
>>> "endpoints": [],
>>> "log_meta": "false",
>>> "log_data": "false",
>>> "bucket_index_max_shards": 0,
>>> "read_only": "false"
>>> }
>>> ],
>>> "placement_targets": [
>>> {
>>> "name": "default-placement",
>>> "tags": []
>>> }
>>> ],
>>> "default_placement": "default-placement",
>>> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c"
>>> }
>>> + echo Creating default zone
>>> Creating default zone
>>> + radosgw-admin zone set --rgw-zone=default
>>> zone id default{
>>> "id": "default",
>>> "name": "default",
>>> "domain_root": ".rgw",
>>> "control_pool": ".rgw.control",
>>> "gc_pool": ".rgw.gc",
>>> "log_pool": ".log",
>>> "intent_log_pool": ".intent-log",
>>> "usage_log_pool": ".usage",
>>> "user_keys_pool": ".users",
>>> "user_email_pool": ".users.email",
>>> "user_swift_pool": ".users.swift",
>>> "user_uid_pool": "

Re: [ceph-users] radosgw - http status 400 while creating a bucket

[Orit Wasserman]

ot;,
> "intent_log_pool": ".intent-log",
> "usage_log_pool": ".usage",
> "user_keys_pool": ".users",
> "user_email_pool": ".users.email",
> "user_swift_pool": ".users.swift",
> "user_uid_pool": ".users.uid",
> "system_key": {
> "access_key": "",
> "secret_key": ""
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": ".rgw.buckets.index",
> "data_pool": ".rgw.buckets",
> "data_extra_pool": "default.rgw.buckets.non-ec",
> "index_type": 0
> }
> }
> ],
> "metadata_heap": ".rgw.meta",
> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c"
> }
>
>
>
> root@arh-ibstorage1-ib:~# radosgw-admin zonegroup get --rgw-zonegroup=default
> {
> "id": "default",
> "name": "default",
> "api_name": "",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "default",
> "zones": [
> {
> "id": "default",
> "name": "default",
> "endpoints": [],
> "log_meta": "false",
> "log_data": "false",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c"
> }
>
>
>
>
> As far as I can see, the master_zone is now set to default.
>
> Now I start the radosgw service:
>
>
> root@arh-ibstorage1-ib:~# service ceph-radosgw@radosgw.gateway start
> root@arh-ibstorage1-ib:~#
> root@arh-ibstorage1-ib:~#
> root@arh-ibstorage1-ib:~#
> root@arh-ibstorage1-ib:~#
> root@arh-ibstorage1-ib:~#
> root@arh-ibstorage1-ib:~# radosgw-admin zone get --rgw-zone=default
> {
> "id": "default",
> "name": "default",
> "domain_root": ".rgw",
> "control_pool": ".rgw.control",
> "gc_pool": ".rgw.gc",
> "log_pool": ".log",
> "intent_log_pool": ".intent-log",
> "usage_log_pool": ".usage",
> "user_keys_pool": ".users",
> "user_email_pool": ".users.email",
> "user_swift_pool": ".users.swift",
> "user_uid_pool": ".users.uid",
> "system_key": {
> "access_key": "",
> "secret_key": ""
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": ".rgw.buckets.index",
> "data_pool": ".rgw.buckets",
> "data_extra_pool": "default.rgw.buckets.non-ec",
> "index_type": 0
> }
> }
> ],
> "metadata_heap": ".rgw.meta",
> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c"
>
>
> root@arh-ibstorage1-ib:~# radosgw-admin zonegroup get --rgw-zonegroup=default
> {
> "id": "default",
> "name": "default",
> "api_name": "",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "",
> "zones": [
> {
> "id": "default",
> "name": "default",
> "endpoints": [],
> "log_meta": "f

Re: [ceph-users] radosgw - http status 400 while creating a bucket

[Orit Wasserman]

On Wed, Nov 9, 2016 at 10:20 PM, Yoann Moulin  wrote:
> Hello,
>
>> many thanks for your help. I've tried setting the zone to master, followed 
>> by the period update --commit command. This is what i've had:
>
> maybe it's related to this issue :
>
> http://tracker.ceph.com/issues/16839 (fixe in Jewel 10.2.3)
>
> or this one :
>
> http://tracker.ceph.com/issues/17239
>
> the "id" of the zonegroup shouldn't be "default" but an uuid afaik
>
> Best regards
>
> Yoann Moulin
>
>> root@arh-ibstorage1-ib:~# radosgw-admin zonegroup get --rgw-zonegroup=default
>> {
>> "id": "default",
>> "name": "default",
>> "api_name": "",
>> "is_master": "true",
>> "endpoints": [],
>> "hostnames": [],
>> "hostnames_s3website": [],
>> "master_zone": "default",
>> "zones": [
>> {
>> "id": "default",
>> "name": "default",
>> "endpoints": [],
>> "log_meta": "false",
>> "log_data": "false",
>> "bucket_index_max_shards": 0,
>> "read_only": "false"
>> }
>> ],
>> "placement_targets": [
>> {
>> "name": "default-placement",
>> "tags": []
>> }
>> ],
>> "default_placement": "default-placement",
>> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c"
>> }
>>
>>
>> root@arh-ibstorage1-ib:~# radosgw-admin period update --commit
>> cannot commit period: period does not have a master zone of a master 
>> zonegroup
>> failed to commit period: (22) Invalid argument
>>
>>

This is issue http://tracker.ceph.com/issues/17364 (I am working on it
at he moment).

Do the same procedure as before without the period update --commit command,
it should fix the master zone problem.
see also 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-July/011157.html

This looks like an upgraded system (the id equals the name after an upgrade).

Orit

>> root@arh-ibstorage1-ib:~# radosgw-admin zonegroup get --rgw-zonegroup=default
>> {
>> "id": "default",
>> "name": "default",
>> "api_name": "",
>> "is_master": "true",
>> "endpoints": [],
>> "hostnames": [],
>> "hostnames_s3website": [],
>> "master_zone": "",
>> "zones": [
>> {
>> "id": "default",
>> "name": "default",
>> "endpoints": [],
>> "log_meta": "false",
>> "log_data": "false",
>> "bucket_index_max_shards": 0,
>> "read_only": "false"
>> }
>> ],
>> "placement_targets": [
>> {
>> "name": "default-placement",
>> "tags": []
>> }
>> ],
>> "default_placement": "default-placement",
>> "realm_id": ""
>> }
>>
>>
>>
>>
>>
>> The strange thing as you can see, following the "radosgw-admin period update 
>> --commit" command, the master_zone and the realm_id values reset to blank. 
>> What could be causing this?
>>
>> Here is my ceph infrastructure setup, perhaps it will help with finding the 
>> issue?:
>>
>> ceph osd and mon servers:
>> arh-ibstorage1-ib (also radosgw server)
>> arh-ibstorage2-ib (also radosgw server)
>> arh-ibstorage3-ib
>>
>> ceph mon server:
>> arh-cloud13-ib
>>
>>
>>
>> Thus, overall, i have 4 mon servers, 3 osd servers and 2 radosgw servers
>>
>> Thanks
>>
>>
>>
>> - Original Message -
>>> From: "Yehuda Sadeh-Weinraub" 
>>> To: "Andrei Mikhailovsky" 
>>> Cc: "ceph-users" 
>>> Sent: Wednesday, 9 November, 2016 17:12:30
>>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket
>>
>>> On Wed, Nov 9, 2016 at 1:30 AM, Andrei Mikhailovsky  
>>> wrote:
 Hi Yehuda,

 just tried to run the command to set the master_zone to default followed 
 by the
 bucket create without doing the restart and I still have the same error on 
 the
 client:

 >>> encoding="UTF-8"?>InvalidArgumentmy-new-bucket-31337tx00010-005822ebbd-9951ad8-default9951ad8-default-default

>>>
>>> After setting the master zone, try running:
>>>
>>> $ radosgw-admin period update --commit
>>>
>>> Yehuda
>>>

 Andrei

 - Original Message -
> From: "Yehuda Sadeh-Weinraub" 
> To: "Andrei Mikhailovsky" 
> Cc: "ceph-users" 
> Sent: Wednesday, 9 November, 2016 01:13:48
> Subject: Re: [ceph-users] radosgw - http status 400 while creating a 
> bucket

> On Tue, Nov 8, 2016 at 5:05 PM, Andrei Mikhailovsky  
> wrote:
>> Hi Yehuda,
>>
>> I don't have a multizone setup. The radosgw service was configured about 
>> two
>> years ago according to the documentation on ceph.com and haven't changed 
>> with
>> numerous version updates. All was working okay until i've upgraded to 
>> version
>> 10.2.x.
>>

Re: [ceph-users] RGW: Delete orphan period for non-existent realm

[Orit Wasserman]

On Thu, Oct 27, 2016 at 12:30 PM, Richard Chan
 wrote:
> Hi Cephers,
>
> In my period list I am seeing an orphan period
>
> {
> "periods": [
> "24dca961-5761-4bd1-972b-685a57e2fcf7:staging",
> "a5632c6c4001615e57e587c129c1ad93:staging",
> "fac3496d-156f-4c09-9654-179ad44091b9"
> ]
> }
>
>
> The realm a5632c6c4001615e57e587c129c1ad93 no longer exists.
>
> How do I clean this up?
>
> radosgw-admin --cluster flash period delete --realm-id
> a5632c6c4001615e57e587c129c1ad93
> missing period id
>

you need to provide period id not realm id for this command.

try:
radosgw-admin --cluster flash period delete --period 

> doesn't work as it wants a period id, instead of deleting by realm-id
>
>
>
>
>
>
>
> --
> Richard Chan
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Calc the nuber of shards needed for a pucket

[Orit Wasserman]

Hi Ansgar,

We recommend 100,000 object per shard, for 50M objects you will need 512 shards.

Orit

On Fri, Oct 14, 2016 at 1:44 PM, Ansgar Jazdzewski
 wrote:
> Hi,
>
> I like to know if someone of you have some kind of a formula to set
> the right number of shards for a bucket.
> We have currently a Bucket with 30M objects and expect that it will go
> up to 50M.
> At the moment we have 64 Shards configured, but it was told me that
> this is much to less.
>
> Any hints / Formulars for me, thanks
> Ansgar
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] unable to start radosgw after upgrade from 10.2.2 to 10.2.3

[Orit Wasserman]

Hi Graham,
Is there a chance you have old radosgw-admin (hammer) running?
You may encountered http://tracker.ceph.com/issues/17371
If an hammer radosgw-admin runs on the jewel radosgw it corrupts the
configuration.
We are working on a fix for that.

Orit


On Fri, Oct 7, 2016 at 9:37 PM, Graham Allan <g...@umn.edu> wrote:
> Dear Orit,
>
> On 10/07/2016 04:21 AM, Orit Wasserman wrote:
>>
>> Hi,
>>
>> On Wed, Oct 5, 2016 at 11:23 PM, Andrei Mikhailovsky <and...@arhont.com>
>> wrote:
>>>
>>> Hello everyone,
>>>
>>> I've just updated my ceph to version 10.2.3 from 10.2.2 and I am no
>>> longer
>>> able to start the radosgw service. When executing I get the following
>>> error:
>>>
>>> 2016-10-05 22:14:10.735883 7f1852d26a00  0 ceph version 10.2.3
>>> (ecc23778eb545d8dd55e2e4735b53cc93f92e65b), process radosgw, pid 2711
>>> 2016-10-05 22:14:10.765648 7f1852d26a00  0 pidfile_write: ignore empty
>>> --pid-file
>>> 2016-10-05 22:14:11.287772 7f1852d26a00  0 zonegroup default missing zone
>>> for master_zone=
>>
>>
>> This means you are missing a master zone , you can get here only if
>> you configured a realm.
>> Is that the case?
>>
>> Can you provide:
>> radosgw-admin realm get
>> radosgw-admin zonegroupmap get
>> radosgw-admin zonegroup get
>> radosgw-admin zone get  -rgw-zone=default
>>
>> Orit
>
>
> I have not yet modified anything since the jewel upgrade - do you mind if I
> post the output for these from our cluster for your opinion? There is
> apparently no realm configured (which is what I expect for this cluster),
> but it sounds like you think this situation shouldn't arise in that case.
>
> root@cephgw04:~# radosgw-admin realm get
> missing realm name or id, or default realm not found
> root@cephgw04:~# radosgw-admin realm list
> {
> "default_info": "",
> "realms": []
> }
>
> root@cephgw04:~# radosgw-admin zonegroupmap get
> failed to read current period info: (2) No such file or directory{
> "zonegroups": [],
> "master_zonegroup": "",
> "bucket_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> },
> "user_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> }
> }
> 2016-10-07 14:33:15.24 7fecf5cf4900  0 RGWPeriod::init failed to init
> realm  id  : (2) No such file or directory
> root@cephgw04:~# radosgw-admin zonegroup get
> failed to init zonegroup: (2) No such file or directory
> root@cephgw04:~# radosgw-admin zonegroup get --rgw-zonegroup=default
> {
> "id": "default",
> "name": "default",
> "api_name": "",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "",
> "zones": [
> {
> "id": "default",
> "name": "default",
> "endpoints": [],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 32,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> },
> {
> "name": "ec42-placement",
> "tags": []
> }
> ],
> "default_placement": "ec42-placement",
> "realm_id": ""
> }
>
> root@cephgw04:~# radosgw-admin zone get --rgw-zone=default
> {
> "id": "default",
> "name": "default",
> "domain_root": ".rgw",
> "control_pool": ".rgw.control",
> "gc_pool": ".rgw.gc",
> "log_pool": ".log",
> "intent_log_pool": ".intent-log",
> "usage_log_pool": ".usage",
> "user_keys_pool": ".users",
> "user_email_pool": ".users.email",
> "user_swift_pool": ".users.swift",
> "user_uid_pool": ".users.uid",
> "system_key": {
> "access_key": "",
> "secret_key": ""
> },
> "placement_pools": [],
> "metadata_heap": ".rgw.meta",
> "realm_id": ""
> }
>
>
> --
> Graham Allan
> Minnesota Supercomputing Institute - g...@umn.edu
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] unable to start radosgw after upgrade from 10.2.2 to 10.2.3

[Orit Wasserman]

On Fri, Oct 7, 2016 at 9:37 PM, Graham Allan <g...@umn.edu> wrote:
> Dear Orit,
>
> On 10/07/2016 04:21 AM, Orit Wasserman wrote:
>>
>> Hi,
>>
>> On Wed, Oct 5, 2016 at 11:23 PM, Andrei Mikhailovsky <and...@arhont.com>
>> wrote:
>>>
>>> Hello everyone,
>>>
>>> I've just updated my ceph to version 10.2.3 from 10.2.2 and I am no
>>> longer
>>> able to start the radosgw service. When executing I get the following
>>> error:
>>>
>>> 2016-10-05 22:14:10.735883 7f1852d26a00  0 ceph version 10.2.3
>>> (ecc23778eb545d8dd55e2e4735b53cc93f92e65b), process radosgw, pid 2711
>>> 2016-10-05 22:14:10.765648 7f1852d26a00  0 pidfile_write: ignore empty
>>> --pid-file
>>> 2016-10-05 22:14:11.287772 7f1852d26a00  0 zonegroup default missing zone
>>> for master_zone=
>>
>>
>> This means you are missing a master zone , you can get here only if
>> you configured a realm.
>> Is that the case?
>>
>> Can you provide:
>> radosgw-admin realm get
>> radosgw-admin zonegroupmap get
>> radosgw-admin zonegroup get
>> radosgw-admin zone get  -rgw-zone=default
>>
>> Orit
>
>
> I have not yet modified anything since the jewel upgrade - do you mind if I
> post the output for these from our cluster for your opinion? There is
> apparently no realm configured (which is what I expect for this cluster),
> but it sounds like you think this situation shouldn't arise in that case.
>
> root@cephgw04:~# radosgw-admin realm get
> missing realm name or id, or default realm not found
> root@cephgw04:~# radosgw-admin realm list
> {
> "default_info": "",
> "realms": []
> }
>
> root@cephgw04:~# radosgw-admin zonegroupmap get
> failed to read current period info: (2) No such file or directory{
> "zonegroups": [],
> "master_zonegroup": "",
> "bucket_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> },
> "user_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> }
> }
> 2016-10-07 14:33:15.24 7fecf5cf4900  0 RGWPeriod::init failed to init
> realm  id  : (2) No such file or directory
> root@cephgw04:~# radosgw-admin zonegroup get
> failed to init zonegroup: (2) No such file or directory
> root@cephgw04:~# radosgw-admin zonegroup get --rgw-zonegroup=default
> {
> "id": "default",
> "name": "default",
> "api_name": "",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "",

you are missing the master zone (it should be default) so you will
need to fix it

> "zones": [
> {
> "id": "default",
> "name": "default",
> "endpoints": [],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 32,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> },
> {
> "name": "ec42-placement",
> "tags": []
> }
> ],
> "default_placement": "ec42-placement",
> "realm_id": ""
> }
>
> root@cephgw04:~# radosgw-admin zone get --rgw-zone=default
> {
> "id": "default",
> "name": "default",
> "domain_root": ".rgw",
> "control_pool": ".rgw.control",
> "gc_pool": ".rgw.gc",
> "log_pool": ".log",
> "intent_log_pool": ".intent-log",
> "usage_log_pool": ".usage",
> "user_keys_pool": ".users",
> "user_email_pool": ".users.email",
> "user_swift_pool": ".users.swift",
> "user_uid_pool": ".users.uid",
> "system_key": {
> "access_key": "",
> "secret_key": ""
> },
> "placement_pools": [],
> "metadata_heap": ".rgw.meta",
> "realm_id": ""
> }
>
>
> --
> Graham Allan
> Minnesota Supercomputing Institute - g...@umn.edu
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] unable to start radosgw after upgrade from 10.2.2 to 10.2.3

[Orit Wasserman]

On Fri, Oct 7, 2016 at 12:24 PM, Andrei Mikhailovsky <and...@arhont.com> wrote:
> Hi Orit,
>
> The radosgw service has been configured about two years ago using the 
> documentation on the ceph.com website. No changes to configuration has been 
> done since. The service was working fine until the 10.2.3 update recently. I 
> have been updating ceph to include every major release and practically every 
> minor release (apart from maybe one or two about a year ago.
>
> I have followed the insructions in the following link 
> (https://www.mail-archive.com/ceph-users@lists.ceph.com/msg31764.html) and 
> that has solved the issue with radosgw not able to start at all.
>
> To answer your question, I now have the following from the commands that 
> you've listed. Please let me know if something is missing.
>
>
> # radosgw-admin realm get
> {
> "id": "5b41b1b2-0f92-463d-b582-07552f83e66c",
> "name": "default",
> "current_period": "286475fa-625b-4fdb-97bf-dcec4b437960",
> "epoch": 1
> }
>
>
>
> # radosgw-admin zonegroupmap get
> {
> "zonegroups": [],
> "master_zonegroup": "",
> "bucket_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> },
> "user_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> }
> }
>
>
>
> # radosgw-admin zonegroup get
> {
> "id": "default",
> "name": "default",
> "api_name": "",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "",

You need to set the master zone to be default zone id (which is also
default becuase you upgraded from an older version)

try: radosgw-admin zone modify --master --rgw-zone default.

If that doesn't work there is a more complicated procedure that I hope
we can avoid.

> "zones": [
> {
> "id": "default",
> "name": "default",
> "endpoints": [],
> "log_meta": "false",
> "log_data": "false",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": ""
> }
>
>
>
> # radosgw-admin zone get  --rgw-zone=default
> {
> "id": "default",
> "name": "default",
> "domain_root": ".rgw",
> "control_pool": ".rgw.control",
> "gc_pool": ".rgw.gc",
> "log_pool": ".log",
> "intent_log_pool": ".intent-log",
> "usage_log_pool": ".usage",
> "user_keys_pool": ".users",
> "user_email_pool": ".users.email",
> "user_swift_pool": ".users.swift",
> "user_uid_pool": ".users.uid",
> "system_key": {
> "access_key": "",
> "secret_key": ""
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": ".rgw.buckets.index",
> "data_pool": ".rgw.buckets",
> "data_extra_pool": "",
> "index_type": 0
> }
> }
> ],
> "metadata_heap": ".rgw.meta",
> "realm_id": ""
> }
>
>
>
>
>
> I did notice that I am missing the .usage pool as per "usage_log_pool": 
> ".usage", setting. Should I create it with the same settings as the .users 
> pool?

no need , rgw creates it when it needs it

Orit
> Cheers
>
> Andrei
>
>
>
>
> - Original Message -
>> From: "Orit Wasserman" <owass...@redhat.com>
>> To

Re: [ceph-users] unable to start radosgw after upgrade from 10.2.2 to 10.2.3

[Orit Wasserman]

Hi,

On Wed, Oct 5, 2016 at 11:23 PM, Andrei Mikhailovsky  wrote:
> Hello everyone,
>
> I've just updated my ceph to version 10.2.3 from 10.2.2 and I am no longer
> able to start the radosgw service. When executing I get the following error:
>
> 2016-10-05 22:14:10.735883 7f1852d26a00  0 ceph version 10.2.3
> (ecc23778eb545d8dd55e2e4735b53cc93f92e65b), process radosgw, pid 2711
> 2016-10-05 22:14:10.765648 7f1852d26a00  0 pidfile_write: ignore empty
> --pid-file
> 2016-10-05 22:14:11.287772 7f1852d26a00  0 zonegroup default missing zone
> for master_zone=

This means you are missing a master zone , you can get here only if
you configured a realm.
Is that the case?

Can you provide:
radosgw-admin realm get
radosgw-admin zonegroupmap get
radosgw-admin zonegroup get
radosgw-admin zone get  -rgw-zone=default

Orit

> 2016-10-05 22:14:11.294141 7f1852d26a00 -1 Couldn't init storage provider
> (RADOS)
>
>
>
> I had no issues starting rados on 10.2.2 and all versions prior to that.
>
> I am running ceph 10.2.3 on Ubuntu 16.04 LTS servers.
>
> Could someone please help me with fixing the problem?
>
> Thanks
>
> Andrei
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Troubles seting up radosgw

[Orit Wasserman]

On Wed, Sep 28, 2016 at 10:32 AM, Iban Cabrillo  wrote:
> Dear Admins,
>During last day I have been trying to deploy a new radosgw, following
> jewel guide, ceph cluster is healthy (3 mon and 2 osd servers )
>root@cephrgw ceph]# ceph -v
> ceph version 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b)
>[root@cephrgw ceph]# rpm -qa | grep ceph
> ceph-common-10.2.3-0.el7.x86_64
> libcephfs1-10.2.3-0.el7.x86_64
> ceph-deploy-1.5.36-0.noarch
> ceph-release-1-1.el7.noarch
> ceph-base-10.2.3-0.el7.x86_64
> ceph-radosgw-10.2.3-0.el7.x86_64
> python-cephfs-10.2.3-0.el7.x86_64
> ceph-selinux-10.2.3-0.el7.x86_64
>
> Cityweb is running on default port:
> [root@cephrgw ceph]# systemctl status ceph-radosgw@rgw.cephrgw.service
> ● ceph-radosgw@rgw.cephrgw.service - Ceph rados gateway
>Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled;
> vendor preset: disabled)
>Active: active (running) since mié 2016-09-28 10:20:34 CEST; 2s ago
>  Main PID: 29311 (radosgw)
>CGroup:
> /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.cephrgw.service
>└─29311 /usr/bin/radosgw -f --cluster ceph --name
> client.rgw.cephrgw --setuser ceph --setgroup ceph
>
> sep 28 10:20:34 cephrgw.ifca.es systemd[1]: Started Ceph rados gateway.
> sep 28 10:20:34 cephrgw.ifca.es systemd[1]: Starting Ceph rados gateway...
>
> And this pools were created on ceph storage:
> .rgw.root  2400
> 0  252  19545
> default.rgw.control0800
> 00000
> default.rgw.data.root0000
> 00000
> default.rgw.gc 0   3200
> 0 2112 2080 14080
> default.rgw.log0  12700
> 04762547498317500
> default.rgw.users.uid0000
> 00000
>
> But seems that zones are not well defined.
>
> radosgw-admin zone get --zone-id=default
> 2016-09-28 10:24:07.142478 7fd810b219c0  0 failed reading obj info from
> .rgw.root:zone_info.default: (2) No such file or directory
>
The zone id is not default this is the name try:
 radosgw-admin zone get --rgw-zone default

>
> [root@cephrgw ~]# radosgw-admin zone get
> 2016-09-28 10:25:41.740162 7f18072799c0  1 -- :/0 messenger.start
> 2016-09-28 10:25:41.741262 7f18072799c0  1 -- :/945549824 -->
> 10.10.3.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x7f18085c9850
> con 0x7f18085c85a0
> 2016-09-28 10:25:41.742048 7f180726f700  1 -- 10.10.3.4:0/945549824 learned
> my addr 10.10.3.4:0/945549824
> 2016-09-28 10:25:41.743168 7f17eae03700  1 -- 10.10.3.4:0/945549824 <==
> mon.2 10.10.3.3:6789/0 1  mon_map magic: 0 v1  495+0+0 (2693174994 0
> 0) 0x7f17d4000b90 con 0x7f18085c85a0
> 2016-09-28 10:25:41.743380 7f17eae03700  1 -- 10.10.3.4:0/945549824 <==
> mon.2 10.10.3.3:6789/0 2  auth_reply(proto 2 0 (0) Success) v1 
> 33+0+0 (3801669063 0 0) 0x7f17d4001010 con 0x7f18085c85a0
> 2016-09-28 10:25:41.743696 7f17eae03700  1 -- 10.10.3.4:0/945549824 -->
> 10.10.3.3:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7f17e0001730
> con 0x7f18085c85a0
> 2016-09-28 10:25:41.744541 7f17eae03700  1 -- 10.10.3.4:0/945549824 <==
> mon.2 10.10.3.3:6789/0 3  auth_reply(proto 2 0 (0) Success) v1 
> 206+0+0 (1705741500 0 0) 0x7f17d4001010 con 0x7f18085c85a0
> 2016-09-28 10:25:41.744765 7f17eae03700  1 -- 10.10.3.4:0/945549824 -->
> 10.10.3.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x7f17e0001bf0
> con 0x7f18085c85a0
> 2016-09-28 10:25:41.745619 7f17eae03700  1 -- 10.10.3.4:0/945549824 <==
> mon.2 10.10.3.3:6789/0 4  auth_reply(proto 2 0 (0) Success) v1 
> 393+0+0 (482591267 0 0) 0x7f17d40008c0 con 0x7f18085c85a0
> 2016-09-28 10:25:41.745783 7f17eae03700  1 -- 10.10.3.4:0/945549824 -->
> 10.10.3.3:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f18085cd560 con
> 0x7f18085c85a0
> 2016-09-28 10:25:41.745967 7f18072799c0  1 -- 10.10.3.4:0/945549824 -->
> 10.10.3.3:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x7f18085c9850 con
> 0x7f18085c85a0
> 2016-09-28 10:25:41.746521 7f17eae03700  1 -- 10.10.3.4:0/945549824 <==
> mon.2 10.10.3.3:6789/0 5  mon_map magic: 0 v1  495+0+0 (2693174994 0
> 0) 0x7f17d40012b0 con 0x7f18085c85a0
> 2016-09-28 10:25:41.746669 7f17daffd700  2
> RGWDataChangesLog::ChangesRenewThread: start
> 2016-09-28 10:25:41.746882 7f18072799c0 20 get_system_obj_state:
> rctx=0x7ffe0afd57e0 obj=.rgw.root:default.realm state=0x7f18085cf4e8
> s->prefetch_data=0
> 2016-09-28 10:25:41.746962 7f17eae03700  1 -- 10.10.3.4:0/945549824 <==
> mon.2 10.10.3.3:6789/0 6  osd_map(5792..5792 src has 

Re: [ceph-users] fixing zones

[Orit Wasserman]

On Wed, Sep 28, 2016 at 5:27 PM, Michael Parson <mpar...@bl.org> wrote:
> On Wed, 28 Sep 2016, Orit Wasserman wrote:
>>
>> see blow
>>
>> On Tue, Sep 27, 2016 at 8:31 PM, Michael Parson <mpar...@bl.org> wrote:
>
>
> 
>
>>> We googled around a bit and found the fix-zone script:
>>>
>>>
>>> https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone
>>>
>>> Which ran fine until the last command, which errors out with:
>>>
>>> + radosgw-admin zone default --rgw-zone=default
>>> WARNING: failed to initialize zonegroup
>>>
>>
>> That is a known issue (default zone is a realm propety) it should not
>> effect you because radosgw uses the "default" zone
>> if it doesn't find any zone.
>>
>>> the 'default' rgw-zone seems OK:
>>>
>>> $ sudo radosgw-admin zone get --zone-id=default
>>> {
>>> "id": "default",
>>> "name": "default",
>>> "domain_root": ".rgw_",
>>
>>
>> the underscore doesn't look good here and in the other pools
>> are you sure this are the pools you used before?
>
>
> The underscores were done by the script referenced above, but you're
> right, I don't see the underscores in my osd pool list:
>
> $ sudo ceph osd pool ls | grep rgw | sort
> default.rgw.buckets.data
> .rgw
> .rgw.buckets
> .rgw.buckets.extra
> .rgw.buckets.index
> .rgw.control
> .rgw.gc
> .rgw.meta
> .rgw.root
> .rgw.root.backup
>
>

You need to fix the pools in the zone:
1. get zone json: radosgw-admin zone get --rgw-zone default > default.json
2. shutdown the radosgw (very important)
3. fix pools names in default.json
4. run: radosgw-admin zone set --rgw-zone default < default.zone
5. run: radosgw-admin period update --commit
6. start the radosgw

Orit
> --
> Michael Parson
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw multi-site replication issues

[Orit Wasserman]

On Tue, Sep 27, 2016 at 10:19 PM, John Rowe <john.r...@salesforce.com> wrote:
> Hi Orit,
>
> It appears it must have been one of the known bugs in 10.2.2.  I just
> upgraded to 10.2.3 and bi-directional syncing now works.
>

Good

> I am still seeing some errors when I run synch-related commands but they
> don't seem to be affecting operations as of yet:
>
> radosgw-admin sync status
> 2016-09-27 16:17:15.270331 7fe5e83ad9c0  0 error in read_id for id  : (2) No
> such file or directory
> 2016-09-27 16:17:15.270883 7fe5e83ad9c0  0 error in read_id for id  : (2) No
> such file or directory
>   realm 3af93a86-916a-490f-b38f-17922b472b19 (my_realm)
>   zonegroup 235b010c-22e2-4b43-8fcc-8ae01939273e (us)
>zone 58aa3eef-fc1f-492c-a08e-9c6019e7c266 (us-phx)
>   metadata sync preparing for full sync
> full sync: 0/64 shards
> metadata is caught up with master
> incremental sync: 64/64 shards
>   data sync source: 6c830b44-4e39-4e19-9bd8-03c37c2021f2 (us-dfw)
> preparing for full sync
> full sync: 18/128 shards
> full sync: 0 buckets to sync
> incremental sync: 110/128 shards
> data is behind on 20 shards
> oldest incremental change not applied: 2016-09-27
> 16:17:08.0.922757s
>
>

data sync can take time if you have lots of data,
make sure it catches up

> I've also verified all of the existing data within each bucket has sync'd
> over as well
>
>
> On Tue, Sep 27, 2016 at 9:21 AM John Rowe <john.r...@salesforce.com> wrote:
>>
>> Hi Orit,
>>
>> That was my failed attempt at sanitizing :)
>>
>> They are actually all identical:
>>
>> Periods:
>> MD5 (cephrgw1-1-dfw-period.json) = 12ed481381c1f2937a27b57db0473d6d
>> MD5 (cephrgw1-1-phx-period.json) = 12ed481381c1f2937a27b57db0473d6d
>> MD5 (cephrgw1-2-dfw-period.json) = 12ed481381c1f2937a27b57db0473d6d
>> MD5 (cephrgw1-2-phx-period.json) = 12ed481381c1f2937a27b57db0473d6d
>> MD5 (cephrgw1-3-dfw-period.json) = 12ed481381c1f2937a27b57db0473d6d
>> MD5 (cephrgw1-3-phx-period.json) = 12ed481381c1f2937a27b57db0473d6d
>>
>> Realms:
>> MD5 (cephrgw1-1-dfw-realm.json) = 39a4e63bab64ed756961117d3629b109
>> MD5 (cephrgw1-1-phx-realm.json) = 39a4e63bab64ed756961117d3629b109
>> MD5 (cephrgw1-2-dfw-realm.json) = 39a4e63bab64ed756961117d3629b109
>> MD5 (cephrgw1-2-phx-realm.json) = 39a4e63bab64ed756961117d3629b109
>> MD5 (cephrgw1-3-dfw-realm.json) = 39a4e63bab64ed756961117d3629b109
>> MD5 (cephrgw1-3-phx-realm.json) = 39a4e63bab64ed756961117d3629b109
>>
>>
>> On Tue, Sep 27, 2016 at 5:32 AM Orit Wasserman <owass...@redhat.com>
>> wrote:
>>>
>>> see comment below
>>>
>>> On Mon, Sep 26, 2016 at 10:00 PM, John Rowe <john.r...@salesforce.com>
>>> wrote:
>>> > Hi Orit,
>>> >
>>> > Sure thing, please see below.
>>> >
>>> > Thanks!
>>> >
>>> >
>>> > DFW (Primary)
>>> > radosgw-admin zonegroupmap get
>>> > {
>>> > "zonegroups": [
>>> > {
>>> > "key": "235b010c-22e2-4b43-8fcc-8ae01939273e",
>>> > "val": {
>>> > "id": "235b010c-22e2-4b43-8fcc-8ae01939273e",
>>> > "name": "us",
>>> > "api_name": "us",
>>> > "is_master": "true",
>>> > "endpoints": [
>>> > "http:\/\/ELB_FQDN:80"
>>> > ],
>>> > "hostnames": [],
>>> > "hostnames_s3website": [],
>>> > "master_zone": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
>>> > "zones": [
>>> > {
>>> > "id": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
>>> > "name": "us-phx",
>>> > "endpoints": [
>>> > "http:\/\/cephrgw1-1:80"
>>> > ],
>>> > "log_meta": "false",
>>> >   

Re: [ceph-users] fixing zones

[Orit Wasserman]

see blow

On Tue, Sep 27, 2016 at 8:31 PM, Michael Parson  wrote:
> (I tried to start this discussion on irc, but I wound up with the wrong
> paste buffer and wound up getting kicked off for a paste flood, sorry,
> that was on me :(  )
>
> We were having some weirdness with our Ceph and did an upgrade up to
> 10.2.3, which fixed some, but not all of our problems.
>
> It looked like our users pool might have been corrupt, so we moved it
> aside and created a new set:
>
> $ sudo ceph osd pool rename .users old.users
> $ sudo ceph osd pool rename .users.email old.users.email
> $ sudo ceph osd pool rename .users.swift old.users.swift
> $ sudo ceph osd pool rename .users.uid old.users.uid
>
>
> $ sudo ceph osd pool create .users 16 16
> $ sudo ceph osd pool create .users.email 16 16
> $ sudo ceph osd pool create .users.swift 16 16
> $ sudo ceph osd pool create .users.uid 16 16
>
> This allowed me to create new users and swift subusers under them, but
> only the first one is allowing auth, all others are getting 403s when
> attempting to auth.
>
> We googled around a bit and found the fix-zone script:
>
> https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone
>
> Which ran fine until the last command, which errors out with:
>
> + radosgw-admin zone default --rgw-zone=default
> WARNING: failed to initialize zonegroup
>

That is a known issue (default zone is a realm propety) it should not
effect you because radosgw uses the "default" zone
if it doesn't find any zone.

> the 'default' rgw-zone seems OK:
>
> $ sudo radosgw-admin zone get --zone-id=default
> {
> "id": "default",
> "name": "default",
> "domain_root": ".rgw_",

the underscore doesn't look good here and in the other pools
are you sure this are the pools you used before?

Orit

> "control_pool": ".rgw.control_",
> "gc_pool": ".rgw.gc_",
> "log_pool": ".log_",
> "intent_log_pool": ".intent-log_",
> "usage_log_pool": ".usage_",
> "user_keys_pool": ".users_",
> "user_email_pool": ".users.email_",
> "user_swift_pool": ".users.swift_",
> "user_uid_pool": ".users.uid_",
> "system_key": {
> "access_key": "",
> "secret_key": ""
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": ".rgw.buckets.index_",
> "data_pool": ".rgw.buckets_",
> "data_extra_pool": ".rgw.buckets.extra_",
> "index_type": 0
> }
> }
> ],
> "metadata_heap": ".rgw.meta",
> "realm_id": "a113de3d-c506-4112-b419-0d5c94ded7af"
> }
>

> Not really sure where to go from here, any help would be appreciated.
>
> --
> Michael Parson
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw multi-site replication issues

[Orit Wasserman]

uot;name": "my_realm",
> "current_period": "75e5f8df-5d53-4ac9-b87d-625eee5455c1",
> "epoch": 2
> }
> radosgw-admin period get
> {
> "id": "75e5f8df-5d53-4ac9-b87d-625eee5455c1",
> "epoch": 4,
> "predecessor_uuid": "ab3ffbbb-0fc9-49f7-a44f-e804f9ed7697",
> "sync_status": [
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> "",
> ""
> ],
> "period_map": {
> "id": "75e5f8df-5d53-4ac9-b87d-625eee5455c1",
> "zonegroups": [
> {
> "id": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> "name": "us",
> "api_name": "us",
> "is_master": "true",
> "endpoints": [
> "http:\/\/ELB_FQDN:80"
> ],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> "zones": [
> {
> "id": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
> "name": "us-phx",
> "endpoints": [
> "http:\/\/cephrgw1-1-phx:80"
> ],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> },
> {
> "id": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> "name": "us-dfw",
> "endpoints": [
> "http:\/\/cephrgw1-1-dfw:80"
> ],
> "log_meta": "true",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "3af93a86-916a-490f-b38f-17922b472b19"
> }
> ],
> "short_zone_ids": [
> {
> "key": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
> "val": 1694436416
> },
>

Re: [ceph-users] rgw multi-site replication issues

[Orit Wasserman]

quot;metadata_heap": "us-phx.rgw.meta",
> "realm_id": "3af93a86-916a-490f-b38f-17922b472b19"
> }
>
> ---
> rgw-secondary-2:
> radosgw-admin zonegroup get
> {
> "id": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> "name": "us",
> "api_name": "us",
> "is_master": "true",
> "endpoints": [
> "http:\/\/LB_FQDN:80"
> ],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> "zones": [
> {
> "id": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
> "name": "us-phx",
> "endpoints": [
> "http:\/\/PHX-RGW-1:80"
> ],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> },
> {
> "id": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> "name": "us-dfw",
> "endpoints": [
> "http:\/\/DFW-RGW-1:80"
> ],
> "log_meta": "true",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "3af93a86-916a-490f-b38f-17922b472b19"
> }
> radosgw-admin zone get
> {
> "id": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
> "name": "us-phx",
> "domain_root": "us-phx.rgw.data.root",
> "control_pool": "us-phx.rgw.control",
> "gc_pool": "us-phx.rgw.gc",
> "log_pool": "us-phx.rgw.log",
> "intent_log_pool": "us-phx.rgw.intent-log",
> "usage_log_pool": "us-phx.rgw.usage",
> "user_keys_pool": "us-phx.rgw.users.keys",
> "user_email_pool": "us-phx.rgw.users.email",
> "user_swift_pool": "us-phx.rgw.users.swift",
> "user_uid_pool": "us-phx.rgw.users.uid",
> "system_key": {
> "access_key": "SYSTEM_ACCESS_KEY",
> "secret_key": "SYSTEM_SECRET_KEY"
> },
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": "us-phx.rgw.buckets.index",
> "data_pool": "us-phx.rgw.buckets.data",
> "data_extra_pool": "us-phx.rgw.buckets.non-ec",
> "index_type": 0
> }
> }
> ],
> "metadata_heap": "us-phx.rgw.meta",
> "realm_id": "3af93a86-916a-490f-b38f-17922b472b19"
> }
>
> ---
> rgw-secondary-3
> radosgw-admin zonegroup get
> {
> "id": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> "name": "us",
> "api_name": "us",
> "is_master": "true",
> "endpoints": [
> "http:\/\/LB_FQDN:80"
> ],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> "zones": [
> {
> "id": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
> "name": "us-phx",
> "endpoints": [
> "http:\/\/PHX-RGW-1:80"
> ],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> },
> {
> "id": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> "name"

Re: [ceph-users] RGW multisite replication failures

[Orit Wasserman]

Hi Ben,
It seems to be http://tracker.ceph.com/issues/16742.
It is being backported to jewel http://tracker.ceph.com/issues/16794,
you can try apply it and see if it helps you.

Regards,
Orit

On Fri, Sep 23, 2016 at 9:21 AM, Ben Morrice  wrote:
> Hello all,
>
> I have two separate ceph (10.2.2) clusters and have configured multisite
> replication between the two. I can see some buckets get synced, however
> others do not.
>
> Both clusters are RHEL7, and I have upgraded libcurl from 7.29 to 7.50
> (to avoid http://tracker.ceph.com/issues/15915).
>
> Below is some debug output on the 'secondary' zone (bbp-gva-secondary)
> after uploading a file to the bucket 'bentest1' from onto the master
> zone (bbp-gva-master).
>
> This appears to to be happening very frequently. The size of my bucket
> pool in the master is ~120GB, however on the secondary site it's only
> 5GB so things are not very happy at the moment.
>
> What steps can I take to work out why RGW cannot create a lock in the
> log pool?
>
> Is there a way to force a full sync, starting fresh (the secondary site
> is not advertised to users, thus it's okay to even clean pools to start
> again)?
>
>
> 2016-09-23 09:03:28.498292 7f992e664700 20 execute(): read data:
> [{"key":6,"val":["bentest1:bbp-gva-master.85732351.16:-1"]}]
> 2016-09-23 09:03:28.498453 7f992e664700 20 execute(): modified
> key=bentest1:bbp-gva-master.85732351.16:-1
> 2016-09-23 09:03:28.498456 7f992e664700 20 wakeup_data_sync_shards:
> source_zone=bbp-gva-master,
> shard_ids={6=bentest1:bbp-gva-master.85732351.16:-1}
> 2016-09-23 09:03:28.498547 7f9a72ffd700 20 incremental_sync(): async
> update notification: bentest1:bbp-gva-master.85732351.16:-1
> 2016-09-23 09:03:28.499137 7f9a7dffb700 20 get_system_obj_state:
> rctx=0x7f9a3c5f8e08
> obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16
> state=0x7f9a0c069848 s->prefetch_data=0
> 2016-09-23 09:03:28.501379 7f9a72ffd700 20 operate(): sync status for
> bucket bentest1:bbp-gva-master.85732351.16:-1: 2
> 2016-09-23 09:03:28.501433 7f9a877fe700 20 reading from
> .bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16
> 2016-09-23 09:03:28.501447 7f9a877fe700 20 get_system_obj_state:
> rctx=0x7f9a877fc6d0
> obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16
> state=0x7f9a340cfbe8 s->prefetch_data=0
> 2016-09-23 09:03:28.503269 7f9a877fe700 20 get_system_obj_state:
> rctx=0x7f9a877fc6d0
> obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16
> state=0x7f9a340cfbe8 s->prefetch_data=0
> 2016-09-23 09:03:28.510428 7f9a72ffd700 20 sending request to
> https://bbpobjectstorage.epfl.ch:443/admin/log?bucket-instance=bentest1%3Abbp-gva-master.85732351.16=json=034.4578.3=bucket-index=bbp-gva
> 2016-09-23 09:03:28.625755 7f9a72ffd700 20 [inc sync] skipping object:
> bentest1:bbp-gva-master.85732351.16:-1/1m: non-complete operation
> 2016-09-23 09:03:28.625759 7f9a72ffd700 20 [inc sync] syncing object:
> bentest1:bbp-gva-master.85732351.16:-1/1m
> 2016-09-23 09:03:28.625831 7f9a72ffd700 20 bucket sync single entry
> (source_zone=bbp-gva-master)
> b=bentest1(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-master.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.85732351.16]):-1/1m[0]
> log_entry=036.4586.3 op=0 op_state=1
> 2016-09-23 09:03:28.625857 7f9a72ffd700  5 bucket sync: sync obj:
> bbp-gva-master/bentest1(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-master.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.85732351.16])/1m[0]
> 2016-09-23 09:03:28.626092 7f9a85ffb700 20 get_obj_state:
> rctx=0x7f9a85ff96a0 obj=bentest1:1m state=0x7f9a30051cf8 s->prefetch_data=0
> 2016-09-23 09:03:28.626119 7f9a72ffd700 20 sending request to
> https://bbpobjectstorage.epfl.ch:443/admin/log?bucket-instance=bentest1%3Abbp-gva-master.85732351.16=json=036.4586.3=bucket-index=bbp-gva
> 2016-09-23 09:03:28.627560 7f9a85ffb700 10 get_canon_resource():
> dest=/bentest1/1m
> /bentest1/1m
> 2016-09-23 09:03:28.627612 7f9a85ffb700 20 sending request to
> https://bbpobjectstorage.epfl.ch:443/bentest1/1m?rgwx-zonegroup=bbp-gva=bbp-gva
> 2016-09-23 09:03:28.725185 7f9a72ffd700 20 incremental_sync:1067:
> shard_id=6 log_entry: 1_1474614207.373384_1713810.1:2016-09-23
> 09:03:27.0.373384s:bentest1:bbp-gva-master.85732351.16
> 2016-09-23 09:03:28.725477 7f9a9affd700 20 get_system_obj_state:
> rctx=0x7f9a3c5f8e08
> obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16
> state=0x7f9a741bb0a8 s->prefetch_data=0
> 2016-09-23 09:03:28.728404 7f9a72ffd700 20 operate(): sync status for
> bucket bentest1:bbp-gva-master.85732351.16:-1: 2
> 2016-09-23 09:03:28.728462 7f9a7b7f6700 20 reading from
> .bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16
> 2016-09-23 09:03:28.728490 7f9a7b7f6700 20 

Re: [ceph-users] rgw multi-site replication issues

[Orit Wasserman]

Hi John,
Can you provide your zonegroup and zones configurations on all 3 rgw?
(run the commands on each rgw)

Thanks,
Orit

On Wed, Sep 21, 2016 at 11:14 PM, John Rowe  wrote:
> Hello,
>
> We have 2 Ceph clusters running in two separate data centers, each one with
> 3 mons, 3 rgws, and 5 osds. I am attempting to get bi-directional multi-site
> replication setup as described in the ceph documentation here:
> http://docs.ceph.com/docs/jewel/radosgw/multisite/
>
> We are running Jewel v 10.2.2:
> rpm -qa | grep ceph
> ceph-base-10.2.2-0.el7.x86_64
> ceph-10.2.2-0.el7.x86_64
> ceph-radosgw-10.2.2-0.el7.x86_64
> libcephfs1-10.2.2-0.el7.x86_64
> python-cephfs-10.2.2-0.el7.x86_64
> ceph-selinux-10.2.2-0.el7.x86_64
> ceph-mon-10.2.2-0.el7.x86_64
> ceph-osd-10.2.2-0.el7.x86_64
> ceph-release-1-1.el7.noarch
> ceph-common-10.2.2-0.el7.x86_64
> ceph-mds-10.2.2-0.el7.x86_64
>
> It appears syncing is happening, however it is not able to sync the
> metadata, and therefore no users/buckets from the primary are going to the
> secondary.
> Primary sync status:
>  radosgw-admin sync status
>   realm 3af93a86-916a-490f-b38f-17922b472b19 (my_realm)
>   zonegroup 235b010c-22e2-4b43-8fcc-8ae01939273e (us)
>zone 6c830b44-4e39-4e19-9bd8-03c37c2021f2 (us-dfw)
>   metadata sync no sync (zone is master)
>   data sync source: 58aa3eef-fc1f-492c-a08e-9c6019e7c266 (us-phx)
> syncing
> full sync: 0/128 shards
> incremental sync: 128/128 shards
> data is caught up with source
>
> radosgw-admin data sync status --source-zone=us-phx
> {
> "sync_status": {
> "info": {
> "status": "sync",
> "num_shards": 128
> },
> "markers": [
> ...
> }
>
> radosgw-admin metadata sync status
> {
> "sync_status": {
> "info": {
> "status": "init",
> "num_shards": 0,
> "period": "",
> "realm_epoch": 0
> },
> "markers": []
> },
> "full_sync": {
> "total": 0,
> "complete": 0
> }
> }
> Secondary sync status:
> radosgw-admin sync status
>   realm 3af93a86-916a-490f-b38f-17922b472b19 (pardot)
>   zonegroup 235b010c-22e2-4b43-8fcc-8ae01939273e (us)
>zone 58aa3eef-fc1f-492c-a08e-9c6019e7c266 (us-phx)
>   metadata sync failed to read sync status: (2) No such file or directory
>   data sync source: 6c830b44-4e39-4e19-9bd8-03c37c2021f2 (us-dfw)
> syncing
> full sync: 0/128 shards
> incremental sync: 128/128 shards
> data is behind on 10 shards
> oldest incremental change not applied: 2016-09-20
> 15:00:17.0.330225s
>
> radosgw-admin data sync status --source-zone=us-dfw
> {
> "sync_status": {
> "info": {
> "status": "building-full-sync-maps",
> "num_shards": 128
> },
> 
> }
>
> radosgw-admin metadata sync status --source-zone=us-dfw
> ERROR: sync.read_sync_status() returned ret=-2
>
>
>
> In the logs I am seeing (date stamps not in order to pick out non-dupes):
> Primary logs:
> 2016-09-20 15:02:44.313204 7f2a2dffb700  0 ERROR:
> client_io->complete_request() returned -5
> 2016-09-20 10:31:57.501247 7faf4bfff700  0 ERROR: failed to wait for op,
> ret=-11: POST
> http://pardot0-cephrgw1-3-phx.ops.sfdc.net:80/admin/realm/period?period=385c44c7-0506-4204-90d7-9d26a6cbaad2=12=f46ce11b-ee5d-489b-aa30-752fc5353931
> 2016-09-20 10:32:03.391118 7fb12affd700  0 ERROR: failed to fetch datalog
> info
> 2016-09-20 10:32:03.491520 7fb12affd700  0 ERROR: lease cr failed, done
> early
>
> Secondary logs;
> 2016-09-20 10:28:15.290050 7faab2fed700  0 ERROR: failed to get bucket
> instance info for bucket
> id=BUCKET1:78301214-35bb-41df-a77f-24968ee4b3ff.104293.1
> 2016-09-20 10:28:15.290108 7faab5ff3700  0 ERROR: failed to get bucket
> instance info for bucket
> id=BUCKET1:78301214-35bb-41df-a77f-24968ee4b3ff.104293.1
> 2016-09-20 10:28:15.290571 7faab77f6700  0 ERROR: failed to get bucket
> instance info for bucket
> id=BUCKET1:78301214-35bb-41df-a77f-24968ee4b3ff.104293.1
> 2016-09-20 10:28:15.304619 7faaad7e2700  0 ERROR: failed to get bucket
> instance info for bucket
> id=BUCKET1:78301214-35bb-41df-a77f-24968ee4b3ff.104293.1
> 2016-09-20 10:28:38.169629 7fa98bfff700  0 ERROR: failed to distribute cache
> for .rgw.root:periods.385c44c7-0506-4204-90d7-9d26a6cbaad2.12
> 2016-09-20 10:28:38.169642 7fa98bfff700 -1 period epoch 12 is not newer than
> current epoch 12, discarding update
> 2016-09-21 03:19:01.550808 7fe10bfff700  0 rgw meta sync: ERROR: failed to
> fetch mdlog info
> 2016-09-21 15:45:09.799195 7fcd677fe700  0 ERROR: failed to fetch remote
> data log info: ret=-11
>
> Each of those logs are repeated several times each, and constantly.
>
> Any help would be greatly appreciated.  

Re: [ceph-users] RGWZoneParams::create(): error creating default zone params: (17) File exists

[Orit Wasserman]

Everything is fine,
This is a debugging message and it was removed in newer jewel versions

Orit

On Sat, Sep 10, 2016 at 7:20 PM, Helmut Garrison
 wrote:
> Hi
>
> i installed ceph and created an object storage from documents but when i
> want to create a user this message appears and after  one or two seconds
> system shows my new user details .
>
> RGWZoneParams::create(): error creating default zone params: (17) File
> exists
>
> what is this message for ? did i do something wrong or i have to do
> configurations ?!! if so tell me what should i do please . thank u so much
>
> regards
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW Error : Error updating periodmap, multiple master zonegroups configured

[Orit Wasserman]

you can try:
radosgw-admin zonegroup modify --zonegroup-id  --master=false

On Tue, Sep 6, 2016 at 11:08 AM, Yoann Moulin  wrote:
> Hello Orit,
>
>> you have two (or more) zonegroups that are set as master.
>
> Yes I know, but I don't know how to fix this
>
>> First detect which zonegroup are the problematic
>> get zonegroup list by running: radosgw-admin zonegroup list
>
> I only see one zonegroup :
>
> $ radosgw-admin zonegroup list
> read_default_id : 0
> {
> "default_info": "default",
> "zonegroups": [
> "default"
> ]
> }
>
>> than on each zonegroup run:
>> radosgw-admin zonegroup get --rgw-zonegroup 
>> see in which is_master is true.
>
> $ radosgw-admin zonegroup get --rgw-zonegroup default
> {
> "id": "default",
> "name": "default",
> "api_name": "",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "",
> "zones": [
> {
> "id": "default",
> "name": "default",
> "endpoints": [],
> "log_meta": "false",
> "log_data": "false",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "ccc2e663-66d3-49a6-9e3a-f257785f2d9a"
> }
>
>
>> Now you need to clear the master flag for all zonegroups except one,
>> this can be done by running:
>> radsogw-admin zonegroup modify --rgw-zonegroup  --master=false
>
> if you check in files in my previous mail in metadata_zonegroup-map.json and 
> metadata_zonegroup.json, there is only one zonegroup with name
> "default" but in metadata_zonegroup.json, the id is "default" and in 
> metadata_zonegroup-map.json it is "4d982760-7853-4174-8c05-cec2ef148cf0"
>
> so for the zonegroup with the name "default", I have 2 differents ID, I guess 
> the problem is there
>
> Thanks for your help
>
> Best regards
>
> Yoann Moulin
>
>> On Tue, Sep 6, 2016 at 9:22 AM, Yoann Moulin  wrote:
>>> Dear List,
>>>
>>> I have an issue with my radosGW.
>>>
>>> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
>>> Linux cluster002 4.2.0-42-generic #49~14.04.1-Ubuntu SMP Wed Jun 29 
>>> 20:22:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>>> Ubuntu 16.04 LTS
>>>
 $ ceph -s
 cluster f9dfd27f-c704-4d53-9aa0-4a23d655c7c4
  health HEALTH_OK
  monmap e1: 3 mons at 
 {cluster002.localdomain=10.90.37.3:6789/0,cluster010.localdomain=10.90.37.11:6789/0,cluster018.localdomain=10.90.37.19:6789/0}
 election epoch 40, quorum 0,1,2 
 cluster002.localdomain,cluster010.localdomain,cluster018.localdomain
   fsmap e47: 1/1/1 up {0=cluster006.localdomain=up:active}, 2 
 up:standby
  osdmap e3784: 144 osds: 144 up, 120 in
 flags sortbitwise
   pgmap v1146863: 7024 pgs, 26 pools, 71470 GB data, 41466 kobjects
 209 TB used, 443 TB / 653 TB avail
 7013 active+clean
7 active+clean+scrubbing+deep
4 active+clean+scrubbing
>>>
>>> Example of the error message I have :
>>>
 $ radosgw-admin bucket list
 2016-09-06 09:04:14.810198 7fcbb01d5900  0 Error updating periodmap, 
 multiple master zonegroups configured
 2016-09-06 09:04:14.810213 7fcbb01d5900  0 master zonegroup: 
 4d982760-7853-4174-8c05-cec2ef148cf0 and  default
 2016-09-06 09:04:14.810215 7fcbb01d5900  0 ERROR: updating period map: 
 (22) Invalid argument
 2016-09-06 09:04:14.810230 7fcbb01d5900  0 failed to add zonegroup to 
 current_period: (22) Invalid argument
 2016-09-06 09:04:14.810238 7fcbb01d5900 -1 failed converting region to 
 zonegroup : ret -22 (22) Invalid argument
>>>
>>> in attachment, you have the result of those commands :
>>>
 $ radosgw-admin metadata zonegroup-map get > metadata_zonegroup-map.json
 $ radosgw-admin metadata zonegroup get > metadata_zonegroup.json
 $ radosgw-admin metadata zone get > metadata_zone.json
 $ radosgw-admin metadata region-map get > metadata_region-map.json
 $ radosgw-admin metadata region get >  metadata_region.json
 $ radosgw-admin zonegroup-map get > zonegroup-map.json
 $ radosgw-admin zonegroup get > zonegroup.json
 $ radosgw-admin zone get > zone.json
 $ radosgw-admin region-map get > region-map.json
 $ radosgw-admin region get > region.json
 $ radosgw-admin period get > period.json
 $ radosgw-admin period list > period_list.json
>>>
>>> I have 60TB of data in this RadosGW, can I fix this issue without having to 
>>> repupload all those data ?
>>>
>>> Thanks for you help !
>>>
>>> Best regards
>>>
>>> --
>>> Yoann Moulin
>>> EPFL IC-IT
>>>
>>> 

Re: [ceph-users] RadosGW Error : Error updating periodmap, multiple master zonegroups configured

[Orit Wasserman]

Hi Yoann,
you have two (or more) zonegroups that are set as master.
First detect which zonegroup are the problematic
get zonegroup list by running: radosgw-admin zonegroup list
than on each zonegroup run:
radosgw-admin zonegroup get --rgw-zonegroup 
see in which is_master is true.

Now you need to clear the master flag for all zonegroups except one,
this can be done by running:
radsogw-admin zonegroup modify --rgw-zonegroup  --master=false

Orit

On Tue, Sep 6, 2016 at 9:22 AM, Yoann Moulin  wrote:
> Dear List,
>
> I have an issue with my radosGW.
>
> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
> Linux cluster002 4.2.0-42-generic #49~14.04.1-Ubuntu SMP Wed Jun 29 20:22:11 
> UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> Ubuntu 16.04 LTS
>
>> $ ceph -s
>> cluster f9dfd27f-c704-4d53-9aa0-4a23d655c7c4
>>  health HEALTH_OK
>>  monmap e1: 3 mons at 
>> {cluster002.localdomain=10.90.37.3:6789/0,cluster010.localdomain=10.90.37.11:6789/0,cluster018.localdomain=10.90.37.19:6789/0}
>> election epoch 40, quorum 0,1,2 
>> cluster002.localdomain,cluster010.localdomain,cluster018.localdomain
>>   fsmap e47: 1/1/1 up {0=cluster006.localdomain=up:active}, 2 up:standby
>>  osdmap e3784: 144 osds: 144 up, 120 in
>> flags sortbitwise
>>   pgmap v1146863: 7024 pgs, 26 pools, 71470 GB data, 41466 kobjects
>> 209 TB used, 443 TB / 653 TB avail
>> 7013 active+clean
>>7 active+clean+scrubbing+deep
>>4 active+clean+scrubbing
>
> Example of the error message I have :
>
>> $ radosgw-admin bucket list
>> 2016-09-06 09:04:14.810198 7fcbb01d5900  0 Error updating periodmap, 
>> multiple master zonegroups configured
>> 2016-09-06 09:04:14.810213 7fcbb01d5900  0 master zonegroup: 
>> 4d982760-7853-4174-8c05-cec2ef148cf0 and  default
>> 2016-09-06 09:04:14.810215 7fcbb01d5900  0 ERROR: updating period map: (22) 
>> Invalid argument
>> 2016-09-06 09:04:14.810230 7fcbb01d5900  0 failed to add zonegroup to 
>> current_period: (22) Invalid argument
>> 2016-09-06 09:04:14.810238 7fcbb01d5900 -1 failed converting region to 
>> zonegroup : ret -22 (22) Invalid argument
>
> in attachment, you have the result of those commands :
>
>> $ radosgw-admin metadata zonegroup-map get > metadata_zonegroup-map.json
>> $ radosgw-admin metadata zonegroup get > metadata_zonegroup.json
>> $ radosgw-admin metadata zone get > metadata_zone.json
>> $ radosgw-admin metadata region-map get > metadata_region-map.json
>> $ radosgw-admin metadata region get >  metadata_region.json
>> $ radosgw-admin zonegroup-map get > zonegroup-map.json
>> $ radosgw-admin zonegroup get > zonegroup.json
>> $ radosgw-admin zone get > zone.json
>> $ radosgw-admin region-map get > region-map.json
>> $ radosgw-admin region get > region.json
>> $ radosgw-admin period get > period.json
>> $ radosgw-admin period list > period_list.json
>
> I have 60TB of data in this RadosGW, can I fix this issue without having to 
> repupload all those data ?
>
> Thanks for you help !
>
> Best regards
>
> --
> Yoann Moulin
> EPFL IC-IT
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem with RGW after update to Jewel

[Orit Wasserman]

can you get the print out of radosgw-admin zonegroupmap?
and radosgw-admin zonegroup get --rgw-zonegroup=default

On Tue, Jul 26, 2016 at 12:36 PM, Frank Enderle
<frank.ende...@anamica.de> wrote:
> ok - i did now the following:
>
> radosgw-admin --cluster=pbs realm create --rgw-realm=pbs --default
> 2016-07-26 10:34:15.216404 7fdf346bc9c0  0 error read_lastest_epoch
> .rgw.root:periods.d94c5208-fc1f-4e02-9773-bc709e4d8a34.latest_epoch
> {
> "id": "98089a5c-6c61-4cc2-a5d8-fce0cb0a9704",
> "name": "pbs",
> "current_period": "d94c5208-fc1f-4e02-9773-bc709e4d8a34",
> "epoch": 1
> }
>
> radosgw-admin --cluster=pbs zonegroup get --rgw-zonegroup=default >zg.json
>
> set master_zone to default
>
> radosgw-admin --cluster=pbs zonegroup set --rgw-zonegroup=default  {
> "id": "default",
> "name": "default",
> "api_name": "",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [
> "***REDACTED***",
> "***REDACTED***",
> "***REDACTED***"
> ],
> "hostnames_s3website": [],
> "master_zone": "default",
> "zones": [
> {
> "id": "default",
> "name": "default",
> "endpoints": [],
> "log_meta": "false",
> "log_data": "false",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "98089a5c-6c61-4cc2-a5d8-fce0cb0a9704"
> }
>
> radosgw-admin --cluster=pbs period update --commit
> 2016-07-26 10:34:56.160525 7f0e22ccf9c0  0 RGWZoneParams::create(): error
> creating default zone params: (17) File exists
> 2016-07-26 10:34:56.264927 7f0e22ccf9c0  0 error read_lastest_epoch
> .rgw.root:periods.98089a5c-6c61-4cc2-a5d8-fce0cb0a9704:staging.latest_epoch
> cannot commit period: period does not have a master zone of a master
> zonegroup
> failed to commit period: (22) Invalid argument
>
>
> --
>
> anamica GmbH
> Heppacher Str. 39
> 71404 Korb
>
> Telefon:   +49 7151 1351565 0
> Telefax: +49 7151 1351565 9
> E-Mail: frank.ende...@anamica.de
> Internet: www.anamica.de
>
>
> Handelsregister: AG Stuttgart HRB 732357
> Geschäftsführer: Yvonne Holzwarth, Frank Enderle
>
>
> From: Orit Wasserman <owass...@redhat.com>
> Date: 26 July 2016 at 12:32:58
> To: Frank Enderle <frank.ende...@anamica.de>
> Cc: ceph-users@lists.ceph.com <ceph-users@lists.ceph.com>, Shilpa Manjarabad
> Jagannath <smanj...@redhat.com>
>
> Subject:  Re: [ceph-users] Problem with RGW after update to Jewel
>
> it doesn't matter, you can call it gold like in the documentation
>
> On Tue, Jul 26, 2016 at 12:15 PM, Frank Enderle
> <frank.ende...@anamica.de> wrote:
>> What should I choose for realm name? I never selected one - does it matter
>> what I put there?
>>
>> --
>>
>> anamica GmbH
>> Heppacher Str. 39
>> 71404 Korb
>>
>> Telefon: +49 7151 1351565 0
>> Telefax: +49 7151 1351565 9
>> E-Mail: frank.ende...@anamica.de
>> Internet: www.anamica.de
>>
>>
>> Handelsregister: AG Stuttgart HRB 732357
>> Geschäftsführer: Yvonne Holzwarth, Frank Enderle
>>
>>
>> From: Orit Wasserman <owass...@redhat.com>
>> Date: 26 July 2016 at 12:13:21
>>
>> To: Frank Enderle <frank.ende...@anamica.de>
>> Cc: ceph-users@lists.ceph.com <ceph-users@lists.ceph.com>, Shilpa
>> Manjarabad
>> Jagannath <smanj...@redhat.com>
>> Subject: Re: [ceph-users] Problem with RGW after update to Jewel
>>
>> Lets try:
>>
>> radosgw-admin realm create --rgw-realm= --default
>>
>> radosgw-admin zonegroup set --rgw-zonegroup=default < json
>>
>> radosgw-admin period update --commit
>>
>> In the next jewel release the upgrade will be smoother.
>>
>> Orit
>>
>> On Tue, Jul 26, 2016 at 11:34 AM, Frank Enderle
>> <frank.ende...@anamica.de> wrote:
>>> Yes! that worked :-)
>>>
>>> now I cha

Re: [ceph-users] Problem with RGW after update to Jewel

[Orit Wasserman]

it doesn't matter, you can call it gold like in the documentation

On Tue, Jul 26, 2016 at 12:15 PM, Frank Enderle
<frank.ende...@anamica.de> wrote:
> What should I choose for realm name? I never selected one - does it matter
> what I put there?
>
> --
>
> anamica GmbH
> Heppacher Str. 39
> 71404 Korb
>
> Telefon:   +49 7151 1351565 0
> Telefax: +49 7151 1351565 9
> E-Mail: frank.ende...@anamica.de
> Internet: www.anamica.de
>
>
> Handelsregister: AG Stuttgart HRB 732357
> Geschäftsführer: Yvonne Holzwarth, Frank Enderle
>
>
> From: Orit Wasserman <owass...@redhat.com>
> Date: 26 July 2016 at 12:13:21
>
> To: Frank Enderle <frank.ende...@anamica.de>
> Cc: ceph-users@lists.ceph.com <ceph-users@lists.ceph.com>, Shilpa Manjarabad
> Jagannath <smanj...@redhat.com>
> Subject:  Re: [ceph-users] Problem with RGW after update to Jewel
>
> Lets try:
>
> radosgw-admin realm create --rgw-realm= --default
>
> radosgw-admin zonegroup set --rgw-zonegroup=default < json
>
> radosgw-admin period update --commit
>
> In the next jewel release the upgrade will be smoother.
>
> Orit
>
> On Tue, Jul 26, 2016 at 11:34 AM, Frank Enderle
> <frank.ende...@anamica.de> wrote:
>> Yes! that worked :-)
>>
>> now I changed the master_zone to default like so:
>>
>> {
>> "id": "default",
>> "name": "default",
>> "api_name": "",
>> "is_master": "true",
>> "endpoints": [],
>> "hostnames": [
>> "***REDACTED***",
>> "***REDACTED***",
>> "***REDACTED***"
>> ],
>> "hostnames_s3website": [],
>> "master_zone": "default",
>> "zones": [
>> {
>> "id": "default",
>> "name": "default",
>> "endpoints": [],
>> "log_meta": "false",
>> "log_data": "false",
>> "bucket_index_max_shards": 0,
>> "read_only": "false"
>> }
>> ],
>> "placement_targets": [
>> {
>> "name": "default-placement",
>> "tags": []
>> }
>> ],
>> "default_placement": "default-placement",
>> "realm_id": ""
>> }
>>
>> and
>>
>> radosgw-admin --cluster=pbs zonegroup set --rgw-zonegroup=default
>>
>> gives me
>>
>> failed to init realm: (2) No such file or directory
>>
>>
>> --
>>
>> anamica GmbH
>> Heppacher Str. 39
>> 71404 Korb
>>
>> Telefon: +49 7151 1351565 0
>> Telefax: +49 7151 1351565 9
>> E-Mail: frank.ende...@anamica.de
>> Internet: www.anamica.de
>>
>>
>> Handelsregister: AG Stuttgart HRB 732357
>> Geschäftsführer: Yvonne Holzwarth, Frank Enderle
>>
>>
>> From: Orit Wasserman <owass...@redhat.com>
>> Date: 26 July 2016 at 11:27:43
>> To: Frank Enderle <frank.ende...@anamica.de>
>> Cc: ceph-users@lists.ceph.com <ceph-users@lists.ceph.com>, Shilpa
>> Manjarabad
>> Jagannath <smanj...@redhat.com>
>>
>> Subject: Re: [ceph-users] Problem with RGW after update to Jewel
>>
>> does adding --rgw-zonegroup=default helps?
>>
>> On Tue, Jul 26, 2016 at 11:09 AM, Frank Enderle
>> <frank.ende...@anamica.de> wrote:
>>> I get this error when I try to execute the command:
>>>
>>> radosgw-admin --cluster=pbs zonegroup get
>>> failed to init zonegroup: (2) No such file or directory
>>>
>>> also with
>>>
>>> radosgw-admin --cluster=pbs zonegroup get --rgw-zone=default
>>> failed to init zonegroup: (2) No such file or directory
>>>
>>>
>>> --
>>>
>>> anamica GmbH
>>> Heppacher Str. 39
>>> 71404 Korb
>>>
>>> Telefon: +49 7151 1351565 0
>>> Telefax: +49 7151 1351565 9
>>> E-Mail: frank.ende...@anamica.de
>>> Internet: www.anamica.de
>>>
>>>
>>> Handelsregister: AG Stuttgart HRB 732357
>>> Geschäftsführer: Yvonne Holzwarth, Frank Enderle
>>>
>>>
>>> From: Orit Wasserman <owass...@redhat.com>
>>> Date: 26 July 2016 at 09:55:58
>>> To: Frank Enderle <frank.ende...@anamica.de>
>>> Cc: Shilpa Manjarabad Jagannath <smanj...@redhat.com>,
>>> ceph-users@lists.ceph.com <ceph-users@lists.ceph.com>
>>

Re: [ceph-users] Problem with RGW after update to Jewel

[Orit Wasserman]

Lets try:

radosgw-admin realm create --rgw-realm= --default

radosgw-admin zonegroup set --rgw-zonegroup=default  < json

radosgw-admin period update --commit

In the next jewel release the upgrade will be smoother.

Orit

On Tue, Jul 26, 2016 at 11:34 AM, Frank Enderle
<frank.ende...@anamica.de> wrote:
> Yes! that worked :-)
>
> now I changed the master_zone to default like so:
>
> {
> "id": "default",
> "name": "default",
> "api_name": "",
> "is_master": "true",
> "endpoints": [],
> "hostnames": [
> "***REDACTED***",
> "***REDACTED***",
> "***REDACTED***"
> ],
> "hostnames_s3website": [],
> "master_zone": "default",
> "zones": [
> {
> "id": "default",
> "name": "default",
> "endpoints": [],
> "log_meta": "false",
> "log_data": "false",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
>     "realm_id": ""
> }
>
> and
>
> radosgw-admin --cluster=pbs zonegroup set --rgw-zonegroup=default
>
> gives me
>
> failed to init realm: (2) No such file or directory
>
>
> --
>
> anamica GmbH
> Heppacher Str. 39
> 71404 Korb
>
> Telefon:   +49 7151 1351565 0
> Telefax: +49 7151 1351565 9
> E-Mail: frank.ende...@anamica.de
> Internet: www.anamica.de
>
>
> Handelsregister: AG Stuttgart HRB 732357
> Geschäftsführer: Yvonne Holzwarth, Frank Enderle
>
>
> From: Orit Wasserman <owass...@redhat.com>
> Date: 26 July 2016 at 11:27:43
> To: Frank Enderle <frank.ende...@anamica.de>
> Cc: ceph-users@lists.ceph.com <ceph-users@lists.ceph.com>, Shilpa Manjarabad
> Jagannath <smanj...@redhat.com>
>
> Subject:  Re: [ceph-users] Problem with RGW after update to Jewel
>
> does adding --rgw-zonegroup=default helps?
>
> On Tue, Jul 26, 2016 at 11:09 AM, Frank Enderle
> <frank.ende...@anamica.de> wrote:
>> I get this error when I try to execute the command:
>>
>> radosgw-admin --cluster=pbs zonegroup get
>> failed to init zonegroup: (2) No such file or directory
>>
>> also with
>>
>> radosgw-admin --cluster=pbs zonegroup get --rgw-zone=default
>> failed to init zonegroup: (2) No such file or directory
>>
>>
>> --
>>
>> anamica GmbH
>> Heppacher Str. 39
>> 71404 Korb
>>
>> Telefon: +49 7151 1351565 0
>> Telefax: +49 7151 1351565 9
>> E-Mail: frank.ende...@anamica.de
>> Internet: www.anamica.de
>>
>>
>> Handelsregister: AG Stuttgart HRB 732357
>> Geschäftsführer: Yvonne Holzwarth, Frank Enderle
>>
>>
>> From: Orit Wasserman <owass...@redhat.com>
>> Date: 26 July 2016 at 09:55:58
>> To: Frank Enderle <frank.ende...@anamica.de>
>> Cc: Shilpa Manjarabad Jagannath <smanj...@redhat.com>,
>> ceph-users@lists.ceph.com <ceph-users@lists.ceph.com>
>>
>> Subject: Re: [ceph-users] Problem with RGW after update to Jewel
>>
>> you need to set the default zone as master zone.
>> you can try:
>> radosgw-admin zonegroup set < zg.json
>> where the json is the json return from radosgw-admin zonegroup get
>> with master_zone field set to "default"
>>
>> Orit
>>
>> On Mon, Jul 25, 2016 at 11:17 PM, Frank Enderle
>> <frank.ende...@anamica.de> wrote:
>>> It most certainly looks very much like the same problem.. Is there a way
>>> to
>>> patch the configuration by hand to get the cluster back in a working
>>> state?
>>>
>>> --
>>>
>>> From: Shilpa Manjarabad Jagannath <smanj...@redhat.com>
>>> Date: 25 July 2016 at 10:34:42
>>> To: Frank Enderle <frank.ende...@anamica.de>
>>> Cc: ceph-users@lists.ceph.com <ceph-users@lists.ceph.com>
>>> Subject: Re: [ceph-users] Problem with RGW after update to Jewel
>>>
>>>
>>> - Original Message -
>>>> From: "Frank Enderle" <frank.ende...@anamica.de>
>>

Re: [ceph-users] Problem with RGW after update to Jewel

[Orit Wasserman]

does adding --rgw-zonegroup=default helps?

On Tue, Jul 26, 2016 at 11:09 AM, Frank Enderle
<frank.ende...@anamica.de> wrote:
> I get this error when I try to execute the command:
>
> radosgw-admin --cluster=pbs zonegroup get
> failed to init zonegroup: (2) No such file or directory
>
> also with
>
> radosgw-admin --cluster=pbs zonegroup get --rgw-zone=default
> failed to init zonegroup: (2) No such file or directory
>
>
> --
>
> anamica GmbH
> Heppacher Str. 39
> 71404 Korb
>
> Telefon:   +49 7151 1351565 0
> Telefax: +49 7151 1351565 9
> E-Mail: frank.ende...@anamica.de
> Internet: www.anamica.de
>
>
> Handelsregister: AG Stuttgart HRB 732357
> Geschäftsführer: Yvonne Holzwarth, Frank Enderle
>
>
> From: Orit Wasserman <owass...@redhat.com>
> Date: 26 July 2016 at 09:55:58
> To: Frank Enderle <frank.ende...@anamica.de>
> Cc: Shilpa Manjarabad Jagannath <smanj...@redhat.com>,
> ceph-users@lists.ceph.com <ceph-users@lists.ceph.com>
>
> Subject:  Re: [ceph-users] Problem with RGW after update to Jewel
>
> you need to set the default zone as master zone.
> you can try:
> radosgw-admin zonegroup set < zg.json
> where the json is the json return from radosgw-admin zonegroup get
> with master_zone field set to "default"
>
> Orit
>
> On Mon, Jul 25, 2016 at 11:17 PM, Frank Enderle
> <frank.ende...@anamica.de> wrote:
>> It most certainly looks very much like the same problem.. Is there a way
>> to
>> patch the configuration by hand to get the cluster back in a working
>> state?
>>
>> --
>>
>> From: Shilpa Manjarabad Jagannath <smanj...@redhat.com>
>> Date: 25 July 2016 at 10:34:42
>> To: Frank Enderle <frank.ende...@anamica.de>
>> Cc: ceph-users@lists.ceph.com <ceph-users@lists.ceph.com>
>> Subject: Re: [ceph-users] Problem with RGW after update to Jewel
>>
>>
>> - Original Message -
>>> From: "Frank Enderle" <frank.ende...@anamica.de>
>>> To: ceph-users@lists.ceph.com
>>> Sent: Monday, July 25, 2016 1:28:10 AM
>>> Subject: [ceph-users] Problem with RGW after update to Jewel
>>>
>>> Hi,
>>>
>>> a while ago I updated a cluster from Infernalis to Jewel. After the
>>> update
>>> some problems occured, which I fixed (I had to create some additional
>>> pool
>>> which I was helped with in the IRC channel) - so the cluster now ran fine
>>> until we tried to add an addtional bucket. Now I get the following error
>>> in
>>> the error log:
>>>
>>> 2016-07-24 19:50:45.978005 7f6ce97fa700 1 == starting new request
>>> req=0x7f6ce97f4710 =
>>> 2016-07-24 19:50:46.021122 7f6ce97fa700 0 sending create_bucket request
>>> to
>>> master zonegroup
>>> 2016-07-24 19:50:46.021135 7f6ce97fa700 0 ERROR: endpoints not configured
>>> for
>>> upstream zone
>>> 2016-07-24 19:50:46.021148 7f6ce97fa700 0 WARNING: set_req_state_err
>>> err_no=5
>>> resorting to 500
>>> 2016-07-24 19:50:46.021249 7f6ce97fa700 1 == req done
>>> req=0x7f6ce97f4710
>>> op status=-5 http_status=500 ==
>>> 2016-07-24 19:50:46.021304 7f6ce97fa700 1 civetweb: 0x7f6dac001420:
>>> 10.42.20.5 - - [24/Jul/2016: 19:50:45 + ] "PUT /abc/ HTTP/1.1" 500 0
>>> -
>>> Cyberduck/4.7.3.18402 (Mac OS X/10.11.6) (x86_64)
>>>
>>> I already tried to fix the problem using the script at
>>>
>>> https://www.mail-archive.com/ceph-users@lists.ceph.com/msg28620.html
>>>
>>> with the outcome that all users disappeared and no bucket could be
>>> access.
>>> So
>>> I restored the backup .rgw.root and it now works again, but still I can't
>>> create buckets. Obviously something has been mixed up with the
>>> zone/zonegroup stuff during the update.
>>>
>>> Would be somebody able to take a look at this? I'm happy to provide all
>>> the
>>> required files; just name them.
>>>
>>> Thanks,
>>>
>>> Frank
>>>
>>
>> It looks like http://tracker.ceph.com/issues/16627, pending backport.
>>
>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem with RGW after update to Jewel

[Orit Wasserman]

you need to set the default zone as master zone.
you can try:
radosgw-admin zonegroup set < zg.json
where the json is the json return from radosgw-admin zonegroup get
with master_zone field set to  "default"

Orit

On Mon, Jul 25, 2016 at 11:17 PM, Frank Enderle
 wrote:
> It most certainly looks very much like the same problem.. Is there a way to
> patch the configuration by hand to get the cluster back in a working state?
>
> --
>
> From: Shilpa Manjarabad Jagannath 
> Date: 25 July 2016 at 10:34:42
> To: Frank Enderle 
> Cc: ceph-users@lists.ceph.com 
> Subject:  Re: [ceph-users] Problem with RGW after update to Jewel
>
>
> - Original Message -
>> From: "Frank Enderle" 
>> To: ceph-users@lists.ceph.com
>> Sent: Monday, July 25, 2016 1:28:10 AM
>> Subject: [ceph-users] Problem with RGW after update to Jewel
>>
>> Hi,
>>
>> a while ago I updated a cluster from Infernalis to Jewel. After the update
>> some problems occured, which I fixed (I had to create some additional pool
>> which I was helped with in the IRC channel) - so the cluster now ran fine
>> until we tried to add an addtional bucket. Now I get the following error
>> in
>> the error log:
>>
>> 2016-07-24 19:50:45.978005 7f6ce97fa700 1 == starting new request
>> req=0x7f6ce97f4710 =
>> 2016-07-24 19:50:46.021122 7f6ce97fa700 0 sending create_bucket request to
>> master zonegroup
>> 2016-07-24 19:50:46.021135 7f6ce97fa700 0 ERROR: endpoints not configured
>> for
>> upstream zone
>> 2016-07-24 19:50:46.021148 7f6ce97fa700 0 WARNING: set_req_state_err
>> err_no=5
>> resorting to 500
>> 2016-07-24 19:50:46.021249 7f6ce97fa700 1 == req done
>> req=0x7f6ce97f4710
>> op status=-5 http_status=500 ==
>> 2016-07-24 19:50:46.021304 7f6ce97fa700 1 civetweb: 0x7f6dac001420:
>> 10.42.20.5 - - [24/Jul/2016: 19:50:45 + ] "PUT /abc/ HTTP/1.1" 500 0 -
>> Cyberduck/4.7.3.18402 (Mac OS X/10.11.6) (x86_64)
>>
>> I already tried to fix the problem using the script at
>>
>> https://www.mail-archive.com/ceph-users@lists.ceph.com/msg28620.html
>>
>> with the outcome that all users disappeared and no bucket could be access.
>> So
>> I restored the backup .rgw.root and it now works again, but still I can't
>> create buckets. Obviously something has been mixed up with the
>> zone/zonegroup stuff during the update.
>>
>> Would be somebody able to take a look at this? I'm happy to provide all
>> the
>> required files; just name them.
>>
>> Thanks,
>>
>> Frank
>>
>
> It looks like http://tracker.ceph.com/issues/16627, pending backport.
>
>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com