Re: [ceph-users] RGW pool contents

2015-12-23 Thread Florian Haas 
On Tue, Nov 24, 2015 at 8:48 PM, Somnath Roy  wrote:
> Hi Yehuda/RGW experts,
>
> I have one cluster with RGW up and running in the customer site.
>
> I did some heavy performance testing on that with CosBench and as a result
> written significant amount of data to showcase performance on that.
>
> Over time, customer also wrote significant amount of data using S3 api into
> the cluster.
>
> Now, I want to remove the buckets/objects created by CosBench and need some
> help on that.

Just intercepting you here. At the risk of asking a silly question,
why do you think you need to poke around with rados commands to get
this cleaned up?

If you just want to remove buckets including the objects they contain,
this is the fastest way I am aware of:

radosgw-admin bucket rm --bucket= --purge-objects

Please let me know if that information is helpful. Thank you!

Cheers,
Florian
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW pool contents

2015-12-22 Thread Somnath Roy 
Thanks for responding back, unfortunately Cosbench setup is not there..
Good to know that there are cleanup steps for Cosbench data.

Regards
Somnath

From: ghislain.cheval...@orange.com [mailto:ghislain.cheval...@orange.com]
Sent: Tuesday, December 22, 2015 11:28 PM
To: Somnath Roy; ceph-users@lists.ceph.com
Subject: RE: RGW pool contents

Hi,
Did you try to use the cleanup and dispose steps of cosbench?
brgds

De : ceph-users [mailto:ceph-users-boun...@lists.ceph.com] De la part de 
Somnath Roy
Envoyé : mardi 24 novembre 2015 20:49
À : ceph-users@lists.ceph.com
Objet : [ceph-users] RGW pool contents

Hi Yehuda/RGW experts,
I have one cluster with RGW up and running in the customer site.
I did some heavy performance testing on that with CosBench and as a result 
written significant amount of data to showcase performance on that.
Over time, customer also wrote significant amount of data using S3 api into the 
cluster.
Now, I want to remove the buckets/objects created by CosBench and need some 
help on that.
I ran the following command to list the buckets.

"radosgw-admin bucket list"

The output is the following snippet..

"rgwdef42",
"rgwdefghijklmnop79",
"rgwyzabc43",
"rgwdefgh43",
"rgwdefghijklm200",

..
..

My understanding is , cosbench should create containers with "mycontainers_" 
 and objects with format "myobjects_" prefix (?). But, it's not there in the 
output of the above command.

Next, I tried to list the contents of the different rgw pools..

rados -p .rgw.buckets.index ls

.dir.default.5407.17
.dir.default.6063.24
.dir.default.6068.23
.dir.default.6046.7
.dir.default.6065.44
.dir.default.5409.3
...
...

Nothing with rgw prefix...Shouldn't the bucketindex objects having similar 
prefix with bucket names ?


Now, tried to get the actual objects...
rados -p .rgw.buckets ls

default.6662.5_myobjects57862
default.5193.18_myobjects6615
default.5410.5_myobjects68518
default.6661.8_myobjects7407
default.5410.22_myobjects54939
default.6651.6_myobjects23790


...

So, looking at these, it seems cosbench run is creating the .dir.default.* 
buckets and the default._myobjects* objects (?)

But, these buckets are not listed by the first "radosgw-admin" command, why ?

Next, I listed the contents of the .rgw pool and here is the output..

rados -p .rgw ls

.bucket.meta.rgwdefghijklm78:default.6069.18
rgwdef42
rgwdefghijklmnop79
rgwyzabc43
.bucket.meta.rgwdefghijklmnopqr71:default.6655.3
rgwdefgh43
.bucket.meta.rgwdefghijklm119:default.6066.25
rgwdefghijklm200
.bucket.meta.rgwxghi2:default.5203.4
rgwxjk17
rgwdefghijklm196

...
...

It seems this pool has the buckets listed by the radosgw-admin command.

Can anybody explain what is .rgw pool supposed to contain ?

Also, what is the difference between .users.uid and .users pool ?


Appreciate any help on this.

Thanks & Regards
Somnath

_



Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.



This message and its attachments may contain confidential or privileged 
information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete 
this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.

Thank you.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW pool contents

2015-12-22 Thread ghislain.chevalier 
Hi,
Did you try to use the cleanup and dispose steps of cosbench?
brgds

De : ceph-users [mailto:ceph-users-boun...@lists.ceph.com] De la part de 
Somnath Roy
Envoyé : mardi 24 novembre 2015 20:49
À : ceph-users@lists.ceph.com
Objet : [ceph-users] RGW pool contents

Hi Yehuda/RGW experts,
I have one cluster with RGW up and running in the customer site.
I did some heavy performance testing on that with CosBench and as a result 
written significant amount of data to showcase performance on that.
Over time, customer also wrote significant amount of data using S3 api into the 
cluster.
Now, I want to remove the buckets/objects created by CosBench and need some 
help on that.
I ran the following command to list the buckets.

"radosgw-admin bucket list"

The output is the following snippet..

"rgwdef42",
"rgwdefghijklmnop79",
"rgwyzabc43",
"rgwdefgh43",
"rgwdefghijklm200",

..
..

My understanding is , cosbench should create containers with "mycontainers_" 
 and objects with format "myobjects_" prefix (?). But, it's not there in the 
output of the above command.

Next, I tried to list the contents of the different rgw pools..

rados -p .rgw.buckets.index ls

.dir.default.5407.17
.dir.default.6063.24
.dir.default.6068.23
.dir.default.6046.7
.dir.default.6065.44
.dir.default.5409.3
...
...

Nothing with rgw prefix...Shouldn't the bucketindex objects having similar 
prefix with bucket names ?


Now, tried to get the actual objects...
rados -p .rgw.buckets ls

default.6662.5_myobjects57862
default.5193.18_myobjects6615
default.5410.5_myobjects68518
default.6661.8_myobjects7407
default.5410.22_myobjects54939
default.6651.6_myobjects23790


...

So, looking at these, it seems cosbench run is creating the .dir.default.* 
buckets and the default._myobjects* objects (?)

But, these buckets are not listed by the first "radosgw-admin" command, why ?

Next, I listed the contents of the .rgw pool and here is the output..

rados -p .rgw ls

.bucket.meta.rgwdefghijklm78:default.6069.18
rgwdef42
rgwdefghijklmnop79
rgwyzabc43
.bucket.meta.rgwdefghijklmnopqr71:default.6655.3
rgwdefgh43
.bucket.meta.rgwdefghijklm119:default.6066.25
rgwdefghijklm200
.bucket.meta.rgwxghi2:default.5203.4
rgwxjk17
rgwdefghijklm196

...
...

It seems this pool has the buckets listed by the radosgw-admin command.

Can anybody explain what is .rgw pool supposed to contain ?

Also, what is the difference between .users.uid and .users pool ?


Appreciate any help on this.

Thanks & Regards
Somnath

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW pool contents

2015-11-28 Thread Wido den Hollander 
On 11/26/2015 09:03 AM, Somnath Roy wrote:
> Thanks Wido !
> Could you please explain a bit more on the relationship between user created 
> buckets and the objects within .bucket.index pool ?
> I am not seeing for each bucket one entry is created within .bucket.index 
> pool.
> 

I don't think the bucket index is created when you actually create a
bucket. IIRC it creates the index when the first object is written to
the bucket. But I could be wrong.

> Regards
> Somnath
> 
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido 
> den Hollander
> Sent: Wednesday, November 25, 2015 10:56 PM
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] RGW pool contents
> 
> On 11/24/2015 08:48 PM, Somnath Roy wrote:
>> Hi Yehuda/RGW experts,
>>
>> I have one cluster with RGW up and running in the customer site.
>>
>> I did some heavy performance testing on that with CosBench and as a 
>> result written significant amount of data to showcase performance on that.
>>
>> Over time, customer also wrote significant amount of data using S3 api 
>> into the cluster.
>>
>> Now, I want to remove the buckets/objects created by CosBench and need 
>> some help on that.
>>
>> I ran the following command to list the buckets.
>>
>>  
>>
>> "radosgw-admin bucket list"
>>
>>  
>>
>> The output is the following snippet..
>>
>>  
>>
>> "rgwdef42",
>>
>> "rgwdefghijklmnop79",
>>
>> "rgwyzabc43",
>>
>> "rgwdefgh43",
>>
>> "rgwdefghijklm200",
>>
>>  
>>
>> ..
>>
>> ..
>>
>>  
>>
>> My understanding is , cosbench should create containers with 
>> "*mycontainers_*"  and objects with format "*myobjects*_" prefix 
>> (?). But, it's not there in the output of the above command.
>>
>>  
> 
> Well, if it did, they should show up there.
> 
>>
>> Next, I tried to list the contents of the different rgw pools..
>>
>>  
>>
>> *rados -p .rgw.buckets.index ls*
>>
>>  
>>
>> .dir.default.5407.17
>>
>> .dir.default.6063.24
>>
>> .dir.default.6068.23
>>
>> .dir.default.6046.7
>>
>> .dir.default.6065.44
>>
>> .dir.default.5409.3
>>
>> ...
>>
>> ...
>>
>>  
>>
>> Nothing with rgw prefix...Shouldn't the bucketindex objects having 
>> similar prefix with bucket names ?
>>
> 
> No, there are the internal IDs of the buckets. You can find the actual bucket 
> objects in the ".rgw" pool.
> 
>>  
>>
>>  
>>
>> Now, tried to get the actual objects...
>>
>> *rados -p .rgw.buckets ls*
>>
>>  
>>
>> default.6662.5_myobjects57862
>>
>> default.5193.18_myobjects6615
>>
>> default.5410.5_myobjects68518
>>
>> default.6661.8_myobjects7407
>>
>> default.5410.22_myobjects54939
>>
>> default.6651.6_myobjects23790
>>
>>  
>>
>> 
>>
>> ...
>>
>>  
>>
>> So, looking at these, it seems cosbench run is creating the
>> .dir.default.* buckets and the default._myobjects* objects 
>> (?)
>>
> 
> No, again, the .dir.default.X is the internal ID of the bucket. It creates 
> "myobject" object on those buckets.
> 
>>  
>>
>> But, these buckets are not listed by the first "radosgw-admin" 
>> command, *why ?*
>>
>>  
>>
>> Next, I listed the contents of the .rgw pool and here is the output..
>>
>>  
>>
>> *rados -p .rgw ls*
>>
>>  
>>
>> .bucket.meta.rgwdefghijklm78:default.6069.18
>>
>> rgwdef42
>>
>> rgwdefghijklmnop79
>>
>> rgwyzabc43
>>
>> .bucket.meta.rgwdefghijklmnopqr71:default.6655.3
>>
>> rgwdefgh43
>>
>> .bucket.meta.rgwdefghijklm119:default.6066.25
>>
>> rgwdefghijklm200
>>
>> .bucket.meta.rgwxghi2:default.5203.4
>>
>> rgwxjk17
>>
>> rgwdefghijklm196
>>
>>  
>>
>> ...
>>
>> ...
>>
>>  
>>
>> It seems this pool has the buckets listed by the radosgw-admin command.
>>
>

Re: [ceph-users] RGW pool contents

2015-11-26 Thread Somnath Roy 
Thanks Wido !
Could you please explain a bit more on the relationship between user created 
buckets and the objects within .bucket.index pool ?
I am not seeing for each bucket one entry is created within .bucket.index pool.

Regards
Somnath

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido 
den Hollander
Sent: Wednesday, November 25, 2015 10:56 PM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] RGW pool contents

On 11/24/2015 08:48 PM, Somnath Roy wrote:
> Hi Yehuda/RGW experts,
> 
> I have one cluster with RGW up and running in the customer site.
> 
> I did some heavy performance testing on that with CosBench and as a 
> result written significant amount of data to showcase performance on that.
> 
> Over time, customer also wrote significant amount of data using S3 api 
> into the cluster.
> 
> Now, I want to remove the buckets/objects created by CosBench and need 
> some help on that.
> 
> I ran the following command to list the buckets.
> 
>  
> 
> "radosgw-admin bucket list"
> 
>  
> 
> The output is the following snippet..
> 
>  
> 
> "rgwdef42",
> 
> "rgwdefghijklmnop79",
> 
> "rgwyzabc43",
> 
> "rgwdefgh43",
> 
> "rgwdefghijklm200",
> 
>  
> 
> ..
> 
> ..
> 
>  
> 
> My understanding is , cosbench should create containers with 
> "*mycontainers_*"  and objects with format "*myobjects*_" prefix 
> (?). But, it's not there in the output of the above command.
> 
>  

Well, if it did, they should show up there.

> 
> Next, I tried to list the contents of the different rgw pools..
> 
>  
> 
> *rados -p .rgw.buckets.index ls*
> 
>  
> 
> .dir.default.5407.17
> 
> .dir.default.6063.24
> 
> .dir.default.6068.23
> 
> .dir.default.6046.7
> 
> .dir.default.6065.44
> 
> .dir.default.5409.3
> 
> ...
> 
> ...
> 
>  
> 
> Nothing with rgw prefix...Shouldn't the bucketindex objects having 
> similar prefix with bucket names ?
> 

No, there are the internal IDs of the buckets. You can find the actual bucket 
objects in the ".rgw" pool.

>  
> 
>  
> 
> Now, tried to get the actual objects...
> 
> *rados -p .rgw.buckets ls*
> 
>  
> 
> default.6662.5_myobjects57862
> 
> default.5193.18_myobjects6615
> 
> default.5410.5_myobjects68518
> 
> default.6661.8_myobjects7407
> 
> default.5410.22_myobjects54939
> 
> default.6651.6_myobjects23790
> 
>  
> 
> 
> 
> ...
> 
>  
> 
> So, looking at these, it seems cosbench run is creating the
> .dir.default.* buckets and the default._myobjects* objects 
> (?)
> 

No, again, the .dir.default.X is the internal ID of the bucket. It creates 
"myobject" object on those buckets.

>  
> 
> But, these buckets are not listed by the first "radosgw-admin" 
> command, *why ?*
> 
>  
> 
> Next, I listed the contents of the .rgw pool and here is the output..
> 
>  
> 
> *rados -p .rgw ls*
> 
>  
> 
> .bucket.meta.rgwdefghijklm78:default.6069.18
> 
> rgwdef42
> 
> rgwdefghijklmnop79
> 
> rgwyzabc43
> 
> .bucket.meta.rgwdefghijklmnopqr71:default.6655.3
> 
> rgwdefgh43
> 
> .bucket.meta.rgwdefghijklm119:default.6066.25
> 
> rgwdefghijklm200
> 
> .bucket.meta.rgwxghi2:default.5203.4
> 
> rgwxjk17
> 
> rgwdefghijklm196
> 
>  
> 
> ...
> 
> ...
> 
>  
> 
> It seems this pool has the buckets listed by the radosgw-admin command.
> 
>  
> 
> Can anybody explain what is *.rgw pool* supposed to contain ?
> 
>  

This pool contains only the bucket metadata objects, here it references to the 
internal IDs.

You can fetch this with 'radosgw-admin metadata get bucket:XX'

> 
> Also, what is the difference between .*users.uid and .users pool* ?
> 
>  

In the .user.uid pool the RGW can do a quick query for users IDs since that is 
required for matching ACLs which might be on a bucket and/or object.

Wido

> 
>  
> 
> Appreciate any help on this.
> 
>  
> 
> Thanks & Regards
> 
> Somnath
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


--
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW pool contents

2015-11-25 Thread Wido den Hollander 
On 11/24/2015 08:48 PM, Somnath Roy wrote:
> Hi Yehuda/RGW experts,
> 
> I have one cluster with RGW up and running in the customer site.
> 
> I did some heavy performance testing on that with CosBench and as a
> result written significant amount of data to showcase performance on that.
> 
> Over time, customer also wrote significant amount of data using S3 api
> into the cluster.
> 
> Now, I want to remove the buckets/objects created by CosBench and need
> some help on that.
> 
> I ran the following command to list the buckets.
> 
>  
> 
> “radosgw-admin bucket list”
> 
>  
> 
> The output is the following snippet..
> 
>  
> 
> "rgwdef42",
> 
> "rgwdefghijklmnop79",
> 
> "rgwyzabc43",
> 
> "rgwdefgh43",
> 
> "rgwdefghijklm200",
> 
>  
> 
> ..
> 
> ..
> 
>  
> 
> My understanding is , cosbench should create containers with
> “*mycontainers_*”  and objects with format “*myobjects*_” prefix
> (?). But, it’s not there in the output of the above command.
> 
>  

Well, if it did, they should show up there.

> 
> Next, I tried to list the contents of the different rgw pools..
> 
>  
> 
> *rados -p .rgw.buckets.index ls*
> 
>  
> 
> .dir.default.5407.17
> 
> .dir.default.6063.24
> 
> .dir.default.6068.23
> 
> .dir.default.6046.7
> 
> .dir.default.6065.44
> 
> .dir.default.5409.3
> 
> ………..
> 
> ………..
> 
>  
> 
> Nothing with rgw prefix…Shouldn’t the bucketindex objects having similar
> prefix with bucket names ?
> 

No, there are the internal IDs of the buckets. You can find the actual
bucket objects in the ".rgw" pool.

>  
> 
>  
> 
> Now, tried to get the actual objects…
> 
> *rados -p .rgw.buckets ls*
> 
>  
> 
> default.6662.5_myobjects57862
> 
> default.5193.18_myobjects6615
> 
> default.5410.5_myobjects68518
> 
> default.6661.8_myobjects7407
> 
> default.5410.22_myobjects54939
> 
> default.6651.6_myobjects23790
> 
>  
> 
> ….
> 
> …
> 
>  
> 
> So, looking at these, it seems cosbench run is creating the
> .dir.default.* buckets and the default._myobjects* objects (?)
> 

No, again, the .dir.default.X is the internal ID of the bucket. It
creates "myobject" object on those buckets.

>  
> 
> But, these buckets are not listed by the first “radosgw-admin” command,
> *why ?*
> 
>  
> 
> Next, I listed the contents of the .rgw pool and here is the output..
> 
>  
> 
> *rados -p .rgw ls*
> 
>  
> 
> .bucket.meta.rgwdefghijklm78:default.6069.18
> 
> rgwdef42
> 
> rgwdefghijklmnop79
> 
> rgwyzabc43
> 
> .bucket.meta.rgwdefghijklmnopqr71:default.6655.3
> 
> rgwdefgh43
> 
> .bucket.meta.rgwdefghijklm119:default.6066.25
> 
> rgwdefghijklm200
> 
> .bucket.meta.rgwxghi2:default.5203.4
> 
> rgwxjk17
> 
> rgwdefghijklm196
> 
>  
> 
> …….
> 
> …….
> 
>  
> 
> It seems this pool has the buckets listed by the radosgw-admin command.
> 
>  
> 
> Can anybody explain what is *.rgw pool* supposed to contain ?
> 
>  

This pool contains only the bucket metadata objects, here it references
to the internal IDs.

You can fetch this with 'radosgw-admin metadata get bucket:XX'

> 
> Also, what is the difference between .*users.uid and .users pool* ?
> 
>  

In the .user.uid pool the RGW can do a quick query for users IDs since
that is required for matching ACLs which might be on a bucket and/or object.

Wido

> 
>  
> 
> Appreciate any help on this.
> 
>  
> 
> Thanks & Regards
> 
> Somnath
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com