[ceph-users] french ceph meetup

[eric mourgaya]

Hi,

The first unofficial French ceph user meetup took place in Nantes on Friday
29th 2013.  There were 8 participants.

Yann from the University of Nantes related his use case of ceph. During his
presentation we talked about hardware choices, about using SSD and problems
he has experienced in his infrastructure. We also talked about his choice
of cephfs and rbd to access his ceph cluster. With Q/A this occupied our
morning.

In the afternoon, we talked about the supervision and administration of
ceph. Alain from Orange Labs showed us a first version of a web interface
based on ceph-rest-api to create ceph objects and a d3js view like 'osd
tree' and 'osd status'; it was great. We discussed what this interface
should look like, and proposed to create a github projet to boost the
developpement of this interface. That should happen soon.

We think that it is a pity that we can not specify the ruleset when
creating a pool with ceph_rest_api:
osd/pool/create?pool=pool()&pg
_num=pg_num()&pgp_num={pgp_num( )}. We will list all the
differences between ceph_rest_api and the ceph cli and keep inktank team
informed.

We also had a discussion about how to aggregate informations needed for
supervision, administration and d3js visualization.  Pierre from Affilae
proposed using mongoDB instead of redis  to store all the information, not
only ceph information but also hardware and system information. We are
going to test this idea. We will propose a solution to collect and analyse
informations for nagios, d3js and administration interfaces. In conclusion,
the community needs an administration interface to help ceph get adopted by
more companies. So let's do it!
I enjoyed this meeting and I 'm looking forward to the next one.
People who are interested by the next French meetup can send me an e-mail.


Best Regards,
-- 
Eric Mourgaya,


Respectons la planete!
Luttons contre la mediocrite!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Yehuda Sadeh]

For bobtail at this point yes. You can try the unofficial version with
that fix off the gitbuilder. Another option is to upgrade everything
to dumpling.

Yehuda

On Mon, Dec 2, 2013 at 10:24 PM, Dominik Mostowiec
 wrote:
> Thanks
> Workaround, don't use multipart when obj size == 0 ?
>
> On Dec 3, 2013 6:43 AM, "Yehuda Sadeh"  wrote:
>>
>> I created earlier an issue (6919) and updated it with the relevant
>> issue. This has been fixed in dumpling, although I don't remember
>> hitting the scenario that you did. Was probably hitting it as part of
>> the development work that was done then.
>> In any case I created a branch with the relevant fixes in it (wip-6919).
>>
>> Thanks,
>> Yehuda
>>
>> On Mon, Dec 2, 2013 at 8:39 PM, Dominik Mostowiec
>>  wrote:
>> > for another object.
>> > http://pastebin.com/VkVAYgwn
>> >
>> >
>> > 2013/12/3 Yehuda Sadeh :
>> >> I see. Do you have backtrace for the crash?
>> >>
>> >> On Mon, Dec 2, 2013 at 6:19 PM, Dominik Mostowiec
>> >>  wrote:
>> >>> 0.56.7
>> >>>
>> >>> W dniu poniedziałek, 2 grudnia 2013 użytkownik Yehuda Sadeh napisał:
>> >>>
>>  I'm having trouble reproducing the issue. What version are you using?
>> 
>>  Thanks,
>>  Yehuda
>> 
>>  On Mon, Dec 2, 2013 at 2:16 PM, Yehuda Sadeh 
>>  wrote:
>>  > Actually, I read that differently. It only says that if there's
>>  > more
>>  > than 1 part, all parts except for the last one need to be > 5M.
>>  > Which
>>  > means that for uploads that are smaller than 5M there should be
>>  > zero
>>  > or one parts.
>>  >
>>  > On Mon, Dec 2, 2013 at 12:54 PM, Dominik Mostowiec
>>  >  wrote:
>>  >> You're right.
>>  >>
>>  >> S3 api doc:
>>  >>
>>  >> http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
>>  >> "Err:EntityTooSmall
>>  >> Your proposed upload is smaller than the minimum allowed object
>>  >> size.
>>  >> Each part must be at least 5 MB in size, except the last part."
>>  >>
>>  >> Thanks.
>>  >>
>>  >> This error should be triggered from radosgw also.
>>  >>
>>  >> --
>>  >> Regards
>>  >> Dominik
>>  >>
>>  >> 2013/12/2 Yehuda Sadeh :
>>  >>> Looks like it. There should be a guard against it (mulitpart
>>  >>> upload
>>  >>> minimum is 5M).
>>  >>>
>>  >>> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
>>  >>>  wrote:
>>   Yes, this is probably upload empty file.
>>   This is the problem?
>>  
>>   --
>>   Regards
>>   Dominik
>>  
>>  
>>   2013/12/2 Yehuda Sadeh :
>>  > By any chance are you uploading empty objects through the
>>  > multipart
>>  > upload api?
>>  >
>>  > On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
>>  >  wrote:
>>  >> Hi,
>>  >> Another file with the same problems:
>>  >>
>>  >> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new
>>  >> request
>>  >> req=0x25406d0 =
>>  >> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req
>>  >> 1314:0.52initializing
>>  >> 2013-12-01 11:37:15.556789 7f7891fd3700 10
>>  >> s->object=files/192.txt
>>  >> s->bucket=testbucket
>>  >> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req
>>  >> 1314:0.000112:s3:POST
>>  >> /testbucket/files/192.txt::getting op
>>  >> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req
>>  >> 1314:0.000118:s3:POST
>>  >> /testbucket/files/192.txt:complete_multipart:authorizing
>>  >> 2013-12-01 11:37:15.560013 7f7891fd3700 10
>>  >> get_canon_resource():
>>  >>
>>  >>
>>  >> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>>  >> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
>>  >> POST
>>  >>
>>  >> application/xml
>>  >> Sun, 01 Dec 2013 10:37:10 GMT
>>  >>
>>  >> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>>  >> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req
>>  >> 1314:0.003399:s3:POST
>>  >> /testbucket/files/192.txt:complete_multipart:reading
>>  >> permissions
>>  >> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req
>>  >> 1314:0.005670:s3:POST
>>  >> /testbucket/files/192.txt:complete_multipart:verifying op
>>  >> permissions
>>  >> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching
>>  >> permissions
>>  >> for
>>  >> uid=0 mask=2
>>  >> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission:
>>  >> 15
>>  >> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested
>>  >> perm
>>  >> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
>>  >> 2013-12-01 11:37:15.562381 7f78

Re: [ceph-users] Adding new OSDs, need to increase PGs?

[Robert van Leeuwen]

> On 2 dec. 2013, at 18:26, "Brian Andrus"  wrote:

>  Setting your pg_num and pgp_num to say... 1024 would A) increase data 
> granularity, B) likely lend no noticeable increase to resource consumption, 
> and C) allow some room for future OSDs two be added while still within range 
> of acceptable pg numbers. You could probably safely double even that number 
> if you plan on expanding at a rapid rate and want to avoid splitting PGs 
> every time a node is added.
> 
> In general, you can conservatively err on the larger side when it comes to 
> pg/p_num. Any excess resource utilization will be negligible (up to a certain 
> point). If you have a comfortable amount of available RAM, you could 
> experiment with increasing the multiplier in the equation you are using and 
> see how it affects your final number.
> 
> The pg_num and pgp_num parameters can safely be changed before or after your 
> new nodes are integrated.

I would be a bit conservative with the PGs / PGPs.
I've experimented with the PG number a bit and noticed the following random IO 
performance drop.
( this could be something to our specific setup but since the PG is easily 
increased and impossible to decrease I would be conservative)

 The setup:
3 OSD nodes with 128GB ram, 2 * 6 core CPU (12 with ht).
Nodes have 10 OSDs running on 1 tb disks and 2 SSDs for Journals.

We use a replica count of 3 so optimum according to formula is about 1000
With 1000 PGs I got about 2000-2500 random 4k IOPS.

Because the nodes are fast enough and I expect the cluster to be expanded with 
3 more nodes I set the PGs to 2000.
Performance dropped to about 1200-1400 IOPS.

I noticed that the spinning disks where no longer maxing out on 100% usage.
Memory and CPU did not seem to be a problem.
Since had the option to recreate the pool and I was not using the recommended 
settings I did not really dive into the issue.
I will not stray to far from the recommended settings in the future though :)

Cheers,
Robert van Leeuwen
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

Thanks
Workaround, don't use multipart when obj size == 0 ?
On Dec 3, 2013 6:43 AM, "Yehuda Sadeh"  wrote:

> I created earlier an issue (6919) and updated it with the relevant
> issue. This has been fixed in dumpling, although I don't remember
> hitting the scenario that you did. Was probably hitting it as part of
> the development work that was done then.
> In any case I created a branch with the relevant fixes in it (wip-6919).
>
> Thanks,
> Yehuda
>
> On Mon, Dec 2, 2013 at 8:39 PM, Dominik Mostowiec
>  wrote:
> > for another object.
> > http://pastebin.com/VkVAYgwn
> >
> >
> > 2013/12/3 Yehuda Sadeh :
> >> I see. Do you have backtrace for the crash?
> >>
> >> On Mon, Dec 2, 2013 at 6:19 PM, Dominik Mostowiec
> >>  wrote:
> >>> 0.56.7
> >>>
> >>> W dniu poniedziałek, 2 grudnia 2013 użytkownik Yehuda Sadeh napisał:
> >>>
>  I'm having trouble reproducing the issue. What version are you using?
> 
>  Thanks,
>  Yehuda
> 
>  On Mon, Dec 2, 2013 at 2:16 PM, Yehuda Sadeh 
> wrote:
>  > Actually, I read that differently. It only says that if there's more
>  > than 1 part, all parts except for the last one need to be > 5M.
> Which
>  > means that for uploads that are smaller than 5M there should be zero
>  > or one parts.
>  >
>  > On Mon, Dec 2, 2013 at 12:54 PM, Dominik Mostowiec
>  >  wrote:
>  >> You're right.
>  >>
>  >> S3 api doc:
>  >>
> http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
>  >> "Err:EntityTooSmall
>  >> Your proposed upload is smaller than the minimum allowed object
> size.
>  >> Each part must be at least 5 MB in size, except the last part."
>  >>
>  >> Thanks.
>  >>
>  >> This error should be triggered from radosgw also.
>  >>
>  >> --
>  >> Regards
>  >> Dominik
>  >>
>  >> 2013/12/2 Yehuda Sadeh :
>  >>> Looks like it. There should be a guard against it (mulitpart
> upload
>  >>> minimum is 5M).
>  >>>
>  >>> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
>  >>>  wrote:
>   Yes, this is probably upload empty file.
>   This is the problem?
>  
>   --
>   Regards
>   Dominik
>  
>  
>   2013/12/2 Yehuda Sadeh :
>  > By any chance are you uploading empty objects through the
> multipart
>  > upload api?
>  >
>  > On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
>  >  wrote:
>  >> Hi,
>  >> Another file with the same problems:
>  >>
>  >> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new
>  >> request
>  >> req=0x25406d0 =
>  >> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req
>  >> 1314:0.52initializing
>  >> 2013-12-01 11:37:15.556789 7f7891fd3700 10
> s->object=files/192.txt
>  >> s->bucket=testbucket
>  >> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req
>  >> 1314:0.000112:s3:POST
>  >> /testbucket/files/192.txt::getting op
>  >> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req
>  >> 1314:0.000118:s3:POST
>  >> /testbucket/files/192.txt:complete_multipart:authorizing
>  >> 2013-12-01 11:37:15.560013 7f7891fd3700 10
> get_canon_resource():
>  >>
>  >>
> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>  >> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
>  >> POST
>  >>
>  >> application/xml
>  >> Sun, 01 Dec 2013 10:37:10 GMT
>  >>
> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>  >> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req
>  >> 1314:0.003399:s3:POST
>  >> /testbucket/files/192.txt:complete_multipart:reading
> permissions
>  >> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req
>  >> 1314:0.005670:s3:POST
>  >> /testbucket/files/192.txt:complete_multipart:verifying op
>  >> permissions
>  >> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching
> permissions
>  >> for
>  >> uid=0 mask=2
>  >> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
>  >> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested
> perm
>  >> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
>  >> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req
>  >> 1314:0.005695:s3:POST
>  >> /testbucket/files/192.txt:complete_multipart:verifying op
> params
>  >> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req
>  >> 1314:0.005698:s3:POST
>  >> /testbucket/files/192.txt:complete_multipart:executing
>  >> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
>  >> d41d8cd98f00b204e9800998ecf8427e-0
>  >> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
>  >> testbucket:

Re: [ceph-users] optimal setup with 4 x ethernet ports

[Kyle Bader]

> Is having two cluster networks like this a supported configuration? Every
osd and mon can reach every other so I think it should be.

Maybe. If your back end network is a supernet and each cluster network is a
subnet of that supernet. For example:

Ceph.conf cluster network (supernet): 10.0.0.0/8

Cluster network #1:  10.1.1.0/24
Cluster network #2: 10.1.2.0/24

With that configuration OSD address autodection *should* just work.

> 1. move osd traffic to eth1. This obviously limits maximum throughput to
~100Mbytes/second, but I'm getting nowhere near that right now anyway.

Given three links I would probably do this if your replication factor is >=
3. Keep in mind 100Mbps links could very well end up being a limiting
factor.

What are you backing each OSD with storage wise and how many OSDs do you
expect to participate in this cluster?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Yehuda Sadeh]

I created earlier an issue (6919) and updated it with the relevant
issue. This has been fixed in dumpling, although I don't remember
hitting the scenario that you did. Was probably hitting it as part of
the development work that was done then.
In any case I created a branch with the relevant fixes in it (wip-6919).

Thanks,
Yehuda

On Mon, Dec 2, 2013 at 8:39 PM, Dominik Mostowiec
 wrote:
> for another object.
> http://pastebin.com/VkVAYgwn
>
>
> 2013/12/3 Yehuda Sadeh :
>> I see. Do you have backtrace for the crash?
>>
>> On Mon, Dec 2, 2013 at 6:19 PM, Dominik Mostowiec
>>  wrote:
>>> 0.56.7
>>>
>>> W dniu poniedziałek, 2 grudnia 2013 użytkownik Yehuda Sadeh napisał:
>>>
 I'm having trouble reproducing the issue. What version are you using?

 Thanks,
 Yehuda

 On Mon, Dec 2, 2013 at 2:16 PM, Yehuda Sadeh  wrote:
 > Actually, I read that differently. It only says that if there's more
 > than 1 part, all parts except for the last one need to be > 5M. Which
 > means that for uploads that are smaller than 5M there should be zero
 > or one parts.
 >
 > On Mon, Dec 2, 2013 at 12:54 PM, Dominik Mostowiec
 >  wrote:
 >> You're right.
 >>
 >> S3 api doc:
 >> http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
 >> "Err:EntityTooSmall
 >> Your proposed upload is smaller than the minimum allowed object size.
 >> Each part must be at least 5 MB in size, except the last part."
 >>
 >> Thanks.
 >>
 >> This error should be triggered from radosgw also.
 >>
 >> --
 >> Regards
 >> Dominik
 >>
 >> 2013/12/2 Yehuda Sadeh :
 >>> Looks like it. There should be a guard against it (mulitpart upload
 >>> minimum is 5M).
 >>>
 >>> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
 >>>  wrote:
  Yes, this is probably upload empty file.
  This is the problem?
 
  --
  Regards
  Dominik
 
 
  2013/12/2 Yehuda Sadeh :
 > By any chance are you uploading empty objects through the multipart
 > upload api?
 >
 > On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
 >  wrote:
 >> Hi,
 >> Another file with the same problems:
 >>
 >> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new
 >> request
 >> req=0x25406d0 =
 >> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req
 >> 1314:0.52initializing
 >> 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
 >> s->bucket=testbucket
 >> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req
 >> 1314:0.000112:s3:POST
 >> /testbucket/files/192.txt::getting op
 >> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req
 >> 1314:0.000118:s3:POST
 >> /testbucket/files/192.txt:complete_multipart:authorizing
 >> 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
 >>
 >> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
 >> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
 >> POST
 >>
 >> application/xml
 >> Sun, 01 Dec 2013 10:37:10 GMT
 >> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
 >> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req
 >> 1314:0.003399:s3:POST
 >> /testbucket/files/192.txt:complete_multipart:reading permissions
 >> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req
 >> 1314:0.005670:s3:POST
 >> /testbucket/files/192.txt:complete_multipart:verifying op
 >> permissions
 >> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions
 >> for
 >> uid=0 mask=2
 >> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
 >> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
 >> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
 >> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req
 >> 1314:0.005695:s3:POST
 >> /testbucket/files/192.txt:complete_multipart:verifying op params
 >> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req
 >> 1314:0.005698:s3:POST
 >> /testbucket/files/192.txt:complete_multipart:executing
 >> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
 >> d41d8cd98f00b204e9800998ecf8427e-0
 >> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
 >> testbucket:files/192.txt to shadow object, tag/shadow_obj haven't
 >> been
 >> set
 >> 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
 >> tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
 >> 2013-12-01 11:37:15.678973 7f7891fd3700  2 req
 >> 1314:0.122286:s3:POST
 >> /testbucket/files/192.txt:complete_multipart:http status=200
>>>

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

for another object.
http://pastebin.com/VkVAYgwn


2013/12/3 Yehuda Sadeh :
> I see. Do you have backtrace for the crash?
>
> On Mon, Dec 2, 2013 at 6:19 PM, Dominik Mostowiec
>  wrote:
>> 0.56.7
>>
>> W dniu poniedziałek, 2 grudnia 2013 użytkownik Yehuda Sadeh napisał:
>>
>>> I'm having trouble reproducing the issue. What version are you using?
>>>
>>> Thanks,
>>> Yehuda
>>>
>>> On Mon, Dec 2, 2013 at 2:16 PM, Yehuda Sadeh  wrote:
>>> > Actually, I read that differently. It only says that if there's more
>>> > than 1 part, all parts except for the last one need to be > 5M. Which
>>> > means that for uploads that are smaller than 5M there should be zero
>>> > or one parts.
>>> >
>>> > On Mon, Dec 2, 2013 at 12:54 PM, Dominik Mostowiec
>>> >  wrote:
>>> >> You're right.
>>> >>
>>> >> S3 api doc:
>>> >> http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
>>> >> "Err:EntityTooSmall
>>> >> Your proposed upload is smaller than the minimum allowed object size.
>>> >> Each part must be at least 5 MB in size, except the last part."
>>> >>
>>> >> Thanks.
>>> >>
>>> >> This error should be triggered from radosgw also.
>>> >>
>>> >> --
>>> >> Regards
>>> >> Dominik
>>> >>
>>> >> 2013/12/2 Yehuda Sadeh :
>>> >>> Looks like it. There should be a guard against it (mulitpart upload
>>> >>> minimum is 5M).
>>> >>>
>>> >>> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
>>> >>>  wrote:
>>>  Yes, this is probably upload empty file.
>>>  This is the problem?
>>> 
>>>  --
>>>  Regards
>>>  Dominik
>>> 
>>> 
>>>  2013/12/2 Yehuda Sadeh :
>>> > By any chance are you uploading empty objects through the multipart
>>> > upload api?
>>> >
>>> > On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
>>> >  wrote:
>>> >> Hi,
>>> >> Another file with the same problems:
>>> >>
>>> >> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new
>>> >> request
>>> >> req=0x25406d0 =
>>> >> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req
>>> >> 1314:0.52initializing
>>> >> 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
>>> >> s->bucket=testbucket
>>> >> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req
>>> >> 1314:0.000112:s3:POST
>>> >> /testbucket/files/192.txt::getting op
>>> >> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req
>>> >> 1314:0.000118:s3:POST
>>> >> /testbucket/files/192.txt:complete_multipart:authorizing
>>> >> 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
>>> >>
>>> >> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>>> >> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
>>> >> POST
>>> >>
>>> >> application/xml
>>> >> Sun, 01 Dec 2013 10:37:10 GMT
>>> >> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>>> >> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req
>>> >> 1314:0.003399:s3:POST
>>> >> /testbucket/files/192.txt:complete_multipart:reading permissions
>>> >> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req
>>> >> 1314:0.005670:s3:POST
>>> >> /testbucket/files/192.txt:complete_multipart:verifying op
>>> >> permissions
>>> >> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions
>>> >> for
>>> >> uid=0 mask=2
>>> >> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
>>> >> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
>>> >> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
>>> >> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req
>>> >> 1314:0.005695:s3:POST
>>> >> /testbucket/files/192.txt:complete_multipart:verifying op params
>>> >> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req
>>> >> 1314:0.005698:s3:POST
>>> >> /testbucket/files/192.txt:complete_multipart:executing
>>> >> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
>>> >> d41d8cd98f00b204e9800998ecf8427e-0
>>> >> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
>>> >> testbucket:files/192.txt to shadow object, tag/shadow_obj haven't
>>> >> been
>>> >> set
>>> >> 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
>>> >> tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
>>> >> 2013-12-01 11:37:15.678973 7f7891fd3700  2 req
>>> >> 1314:0.122286:s3:POST
>>> >> /testbucket/files/192.txt:complete_multipart:http status=200
>>> >> 2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
>>> >> req=0x25406d0 http_status=200 ==
>>> >>
>>> >> Yes, I can read oryginal object.
>>> >>
>>> >> --
>>> >> Regards
>>> >> Dominik
>>> >>
>>> >> 2013/12/2 Yehuda Sadeh :
>>> >>> That's unknown bug. I have a guess as to how the original object
>>> >>> was
>>> >>> created. Can you read the original object, but only copy fails?
>>> >>>
>>> >>> On Dec 2, 2013 4:53 AM, "Dominik Mostow

Re: [ceph-users] odd performance graph

[James Harper]

> 
> I also noticed a graph like this once i benchmarked w2k8 guest on ceph with
> rbd.
> To me it looked like when the space on the drive is used, the throughput is
> lower, when the space read by rbd on the drive is unused, the reads are
> superfast.
> 
> I don't know how rbd works inside, but i think ceph rbd here returns zeros
> without real osd disk read if the block/sector of the rbd-disk is unused. That
> would explain the graph you see. You can try adding a second rbd image and
> not
> format/use it and benchmark this disk, then make a filesystem on it and
> write
> some data and benchmark again...
> 

That makes a lot of sense, although I wonder why the performance is that 
different in the 'used' vs 'unused' areas of the disk...

James
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Real size of rbd image

[Josh Durgin]

On 12/02/2013 08:10 AM, Stephen Taylor wrote:

I had not enabled discard/trim for the filesystem using the image, but I have 
done so this morning. I doesn't appear to make a difference.

The extents I'm seeing reported as "zero" are not actually discarded extents. They are extents that 
contain data that was added after the ending snapshot I'm giving to the diff command. If I specify a later 
ending snapshot or none, then the same extents are reported as "data" rather than as 
"zero" extents. Ignoring those seems to give me the correct size when I'm calculating a diff size 
from one snapshot to another, but I wanted to make sure that this situation is expected before I rely on it, 
although continuing to ignore these extents in the future if they simply go away seems easy enough.


Ah, I see where these are coming from now. It's confusing but
technically correct output for objects that do not exist in the
given snapshots, but do in the current version of the image.
We should probably omit these extents from the output. They don't
affect correctness of size calculations or applying diffs, so it
should be backwards compatible to do so. Issue created [1].

Like you said, ignoring any extents marked 'zero' is always fine
for this size calculation.

Josh

[1] http://tracker.ceph.com/issues/6926


Again, I appreciate your help.

Steve

-Original Message-
From: Josh Durgin [mailto:josh.dur...@inktank.com]
Sent: Wednesday, November 27, 2013 7:51 PM
To: Stephen Taylor; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Real size of rbd image

On 11/26/2013 02:22 PM, Stephen Taylor wrote:

  From ceph-users archive 08/27/2013:

On 08/27/2013 01:39 PM, Timofey Koolin wrote:


/Is way to know real size of rbd image and rbd snapshots?/



/rbd ls -l write declared size of image, but I want to know real
size./


You can sum the sizes of the extents reported by:

   rbd diff pool/image[@snap] [--format json]

That's the difference since the beginning of time, so it reports all

used extents.

Josh

I don't seem to be able to find any documentation supporting the
[@snap] parameter for this call, but it seems to work, at least in
part. I have a requirement to find the size of a snapshot relative to
another snapshot. Here is what I've used:

  rbd diff pool/image@snap2 --from-snap snap1


Most rbd commands work on snapshots too. The help text could certainly be 
improved - suggestions welcome!


The returned list of extents seems to include all changes since snap1,
not just those up to snap2, but those that have been written after
snap2 are labeled "zero" rather than as "data" extents. If I ignore the "zero"
extents and sum the lengths of the "data" extents, it seems to give me
an accurate relative snapshot size. Is this expected behavior and the
correct way to calculate the size I'm looking for?


Do you have discard/trim enabled for whatever's using the image?
The diff will include discarded extents as "zero". For calculating size, it's 
fine to ignore them. It would be unexpected if these aren't listed when you leave out the 
@snap2 portion though.

Josh



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Yehuda Sadeh]

I see. Do you have backtrace for the crash?

On Mon, Dec 2, 2013 at 6:19 PM, Dominik Mostowiec
 wrote:
> 0.56.7
>
> W dniu poniedziałek, 2 grudnia 2013 użytkownik Yehuda Sadeh napisał:
>
>> I'm having trouble reproducing the issue. What version are you using?
>>
>> Thanks,
>> Yehuda
>>
>> On Mon, Dec 2, 2013 at 2:16 PM, Yehuda Sadeh  wrote:
>> > Actually, I read that differently. It only says that if there's more
>> > than 1 part, all parts except for the last one need to be > 5M. Which
>> > means that for uploads that are smaller than 5M there should be zero
>> > or one parts.
>> >
>> > On Mon, Dec 2, 2013 at 12:54 PM, Dominik Mostowiec
>> >  wrote:
>> >> You're right.
>> >>
>> >> S3 api doc:
>> >> http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
>> >> "Err:EntityTooSmall
>> >> Your proposed upload is smaller than the minimum allowed object size.
>> >> Each part must be at least 5 MB in size, except the last part."
>> >>
>> >> Thanks.
>> >>
>> >> This error should be triggered from radosgw also.
>> >>
>> >> --
>> >> Regards
>> >> Dominik
>> >>
>> >> 2013/12/2 Yehuda Sadeh :
>> >>> Looks like it. There should be a guard against it (mulitpart upload
>> >>> minimum is 5M).
>> >>>
>> >>> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
>> >>>  wrote:
>>  Yes, this is probably upload empty file.
>>  This is the problem?
>> 
>>  --
>>  Regards
>>  Dominik
>> 
>> 
>>  2013/12/2 Yehuda Sadeh :
>> > By any chance are you uploading empty objects through the multipart
>> > upload api?
>> >
>> > On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
>> >  wrote:
>> >> Hi,
>> >> Another file with the same problems:
>> >>
>> >> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new
>> >> request
>> >> req=0x25406d0 =
>> >> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req
>> >> 1314:0.52initializing
>> >> 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
>> >> s->bucket=testbucket
>> >> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req
>> >> 1314:0.000112:s3:POST
>> >> /testbucket/files/192.txt::getting op
>> >> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req
>> >> 1314:0.000118:s3:POST
>> >> /testbucket/files/192.txt:complete_multipart:authorizing
>> >> 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
>> >>
>> >> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>> >> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
>> >> POST
>> >>
>> >> application/xml
>> >> Sun, 01 Dec 2013 10:37:10 GMT
>> >> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>> >> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req
>> >> 1314:0.003399:s3:POST
>> >> /testbucket/files/192.txt:complete_multipart:reading permissions
>> >> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req
>> >> 1314:0.005670:s3:POST
>> >> /testbucket/files/192.txt:complete_multipart:verifying op
>> >> permissions
>> >> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions
>> >> for
>> >> uid=0 mask=2
>> >> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
>> >> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
>> >> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
>> >> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req
>> >> 1314:0.005695:s3:POST
>> >> /testbucket/files/192.txt:complete_multipart:verifying op params
>> >> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req
>> >> 1314:0.005698:s3:POST
>> >> /testbucket/files/192.txt:complete_multipart:executing
>> >> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
>> >> d41d8cd98f00b204e9800998ecf8427e-0
>> >> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
>> >> testbucket:files/192.txt to shadow object, tag/shadow_obj haven't
>> >> been
>> >> set
>> >> 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
>> >> tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
>> >> 2013-12-01 11:37:15.678973 7f7891fd3700  2 req
>> >> 1314:0.122286:s3:POST
>> >> /testbucket/files/192.txt:complete_multipart:http status=200
>> >> 2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
>> >> req=0x25406d0 http_status=200 ==
>> >>
>> >> Yes, I can read oryginal object.
>> >>
>> >> --
>> >> Regards
>> >> Dominik
>> >>
>> >> 2013/12/2 Yehuda Sadeh :
>> >>> That's unknown bug. I have a guess as to how the original object
>> >>> was
>> >>> created. Can you read the original object, but only copy fails?
>> >>>
>> >>> On Dec 2, 2013 4:53 AM, "Dominik Mostowiec"
>> >>> 
>> >>> wrote:
>> 
>>  Hi,
>>  I found that issue is related with "ETag: -0" (ends -0)
>>  This is known bug ?
>> 
>>  --
>>  

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

0.56.7

W dniu poniedziałek, 2 grudnia 2013 użytkownik Yehuda Sadeh napisał:

> I'm having trouble reproducing the issue. What version are you using?
>
> Thanks,
> Yehuda
>
> On Mon, Dec 2, 2013 at 2:16 PM, Yehuda Sadeh 
> >
> wrote:
> > Actually, I read that differently. It only says that if there's more
> > than 1 part, all parts except for the last one need to be > 5M. Which
> > means that for uploads that are smaller than 5M there should be zero
> > or one parts.
> >
> > On Mon, Dec 2, 2013 at 12:54 PM, Dominik Mostowiec
> > > wrote:
> >> You're right.
> >>
> >> S3 api doc:
> http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
> >> "Err:EntityTooSmall
> >> Your proposed upload is smaller than the minimum allowed object size.
> >> Each part must be at least 5 MB in size, except the last part."
> >>
> >> Thanks.
> >>
> >> This error should be triggered from radosgw also.
> >>
> >> --
> >> Regards
> >> Dominik
> >>
> >> 2013/12/2 Yehuda Sadeh >:
> >>> Looks like it. There should be a guard against it (mulitpart upload
> >>> minimum is 5M).
> >>>
> >>> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
> >>> > wrote:
>  Yes, this is probably upload empty file.
>  This is the problem?
> 
>  --
>  Regards
>  Dominik
> 
> 
>  2013/12/2 Yehuda Sadeh >:
> > By any chance are you uploading empty objects through the multipart
> upload api?
> >
> > On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
> > > wrote:
> >> Hi,
> >> Another file with the same problems:
> >>
> >> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new
> request
> >> req=0x25406d0 =
> >> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req
> 1314:0.52initializing
> >> 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
> >> s->bucket=testbucket
> >> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req 1314:0.000112:s3:POST
> >> /testbucket/files/192.txt::getting op
> >> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req 1314:0.000118:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:authorizing
> >> 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
> >>
> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
> >> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
> >> POST
> >>
> >> application/xml
> >> Sun, 01 Dec 2013 10:37:10 GMT
> >> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
> >> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req 1314:0.003399:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:reading permissions
> >> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req 1314:0.005670:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:verifying op
> permissions
> >> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions for
> >> uid=0 mask=2
> >> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
> >> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
> >> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
> >> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req 1314:0.005695:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:verifying op params
> >> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req 1314:0.005698:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:executing
> >> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
> >> d41d8cd98f00b204e9800998ecf8427e-0
> >> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
> >> testbucket:files/192.txt to shadow object, tag/shadow_obj haven't
> been
> >> set
> >> 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
> >> tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
> >> 2013-12-01 11:37:15.678973 7f7891fd3700  2 req 1314:0.122286:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:http status=200
> >> 2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
> >> req=0x25406d0 http_status=200 ==
> >>
> >> Yes, I can read oryginal object.
> >>
> >> --
> >> Regards
> >> Dominik
> >>
> >> 2013/12/2 Yehuda Sadeh >:
> >>> That's unknown bug. I have a guess as to how the original object
> was
> >>> created. Can you read the original object, but only copy fails?
> >>>
> >>> On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" <
> dominikmostow...@gmail.com >
> >>> wrote:
> 
>  Hi,
>  I found that issue is related with "ETag: -0" (ends -0)
>  This is known bug ?
> 
>  --
>  Regards
>  Dominik
> 
>  2013/12/2 Dominik Mostowiec 
> >:
>  > Hi,
>  > I have strange problem.
>  > Obj copy (0 size) killing radosgw.
>  >
>  > Head for this file:
>  > Content-Type: application/octet-stream
>  > Server: Apache/2.2.22 (Ubuntu)

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

0.56.7
On Dec 2, 2013 11:30 PM, "Yehuda Sadeh"  wrote:

> I'm having trouble reproducing the issue. What version are you using?
>
> Thanks,
> Yehuda
>
> On Mon, Dec 2, 2013 at 2:16 PM, Yehuda Sadeh  wrote:
> > Actually, I read that differently. It only says that if there's more
> > than 1 part, all parts except for the last one need to be > 5M. Which
> > means that for uploads that are smaller than 5M there should be zero
> > or one parts.
> >
> > On Mon, Dec 2, 2013 at 12:54 PM, Dominik Mostowiec
> >  wrote:
> >> You're right.
> >>
> >> S3 api doc:
> http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
> >> "Err:EntityTooSmall
> >> Your proposed upload is smaller than the minimum allowed object size.
> >> Each part must be at least 5 MB in size, except the last part."
> >>
> >> Thanks.
> >>
> >> This error should be triggered from radosgw also.
> >>
> >> --
> >> Regards
> >> Dominik
> >>
> >> 2013/12/2 Yehuda Sadeh :
> >>> Looks like it. There should be a guard against it (mulitpart upload
> >>> minimum is 5M).
> >>>
> >>> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
> >>>  wrote:
>  Yes, this is probably upload empty file.
>  This is the problem?
> 
>  --
>  Regards
>  Dominik
> 
> 
>  2013/12/2 Yehuda Sadeh :
> > By any chance are you uploading empty objects through the multipart
> upload api?
> >
> > On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
> >  wrote:
> >> Hi,
> >> Another file with the same problems:
> >>
> >> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new
> request
> >> req=0x25406d0 =
> >> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req
> 1314:0.52initializing
> >> 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
> >> s->bucket=testbucket
> >> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req 1314:0.000112:s3:POST
> >> /testbucket/files/192.txt::getting op
> >> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req 1314:0.000118:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:authorizing
> >> 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
> >>
> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
> >> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
> >> POST
> >>
> >> application/xml
> >> Sun, 01 Dec 2013 10:37:10 GMT
> >> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
> >> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req 1314:0.003399:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:reading permissions
> >> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req 1314:0.005670:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:verifying op
> permissions
> >> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions for
> >> uid=0 mask=2
> >> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
> >> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
> >> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
> >> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req 1314:0.005695:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:verifying op params
> >> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req 1314:0.005698:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:executing
> >> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
> >> d41d8cd98f00b204e9800998ecf8427e-0
> >> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
> >> testbucket:files/192.txt to shadow object, tag/shadow_obj haven't
> been
> >> set
> >> 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
> >> tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
> >> 2013-12-01 11:37:15.678973 7f7891fd3700  2 req 1314:0.122286:s3:POST
> >> /testbucket/files/192.txt:complete_multipart:http status=200
> >> 2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
> >> req=0x25406d0 http_status=200 ==
> >>
> >> Yes, I can read oryginal object.
> >>
> >> --
> >> Regards
> >> Dominik
> >>
> >> 2013/12/2 Yehuda Sadeh :
> >>> That's unknown bug. I have a guess as to how the original object
> was
> >>> created. Can you read the original object, but only copy fails?
> >>>
> >>> On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" <
> dominikmostow...@gmail.com>
> >>> wrote:
> 
>  Hi,
>  I found that issue is related with "ETag: -0" (ends -0)
>  This is known bug ?
> 
>  --
>  Regards
>  Dominik
> 
>  2013/12/2 Dominik Mostowiec :
>  > Hi,
>  > I have strange problem.
>  > Obj copy (0 size) killing radosgw.
>  >
>  > Head for this file:
>  > Content-Type: application/octet-stream
>  > Server: Apache/2.2.22 (Ubuntu)
>  > ETag: "d41d8cd98f00b204e98

Re: [ceph-users] issues with Swift and ceph

[Yehuda Sadeh]

On Mon, Dec 2, 2013 at 4:39 PM, jheryl williams  wrote:
> Hello ,
>
> I have been looking all over the net to find a solution to my problem and I
> came across one that you helped someone else with. I was wondering if you
> can assist me as well. I pretty much created a ceph cluster from the step by
> step procedure off of ceph.com. My issue is when enter the swift command to
> test, i get the following message:
>
>
> esguser@ubuntu-ceph01:~/esg-cluster$ swift -V 1.0 -A
> http://radosgw.esg-ceph.com/auth -U esguser:swift -K
> Jb5aADPzf9R0yKLm+wXGHsV3Db+e6X0nV7E6mvFz post test
> Auth GET failed: http://radosgw.esg-ceph.com:80/auth/ 404 Not Found
>
> I made sure I was using the swift keys for authentication...
>
>
> I checked my apache log and this is what it says
>
> [Mon Dec 02 16:23:46 2013] [error] [client 10.8.12.122] File does not exist:
> /var/www/auth
>
> I checked the /var/www/auth and I do not see a auth file or folder there.
>
>
> my rgw.conf file is as follows.. (/etc/apache2/sites-enabled/rgw.conf)
>
> FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock
>
> 
> ServerName esg-ceph.com
> ServerAdmin jheryl.willi...@wdc.com
> DocumentRoot /var/www
> 
>
> RewriteEngine On
> RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*)
> /s3gw.fcgi?page=$1¶ms=$2&%{QUERY_STRING}
> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]

The rewrite rule needs to be within the VirtualHost section.

Yehuda

>
> 
>
> 
> 
> Options +ExecCGI
> AllowOverride All
> SetHandler fastcgi-script
> Order allow,deny
> Allow from all
> AuthBasicAuthoritative Off
> 
> 
>
> 
>
> 
>
> AllowEncodedSlashes On
> ErrorLog /var/log/apache2/error.log
> CustomLog /var/log/apache2/access.log combined
> ServerSignature Off
> 
>
> Any assistance would be greatly appreciated.
>
> Thank you in advance,
>
> Jheryl Williams
>
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Nearly full OSDs with very little (apparent) FS utilization

[Yan, Zheng]

On Mon, Dec 2, 2013 at 9:03 PM, Miguel Afonso Oliveira
 wrote:
> Hi,
>
> Sorry for the very late reply. I have been trying a lot of things...
>
>
> On 25/10/13 22:40, Yan, Zheng wrote:
>>
>> On Sat, Oct 26, 2013 at 2:05 AM, Gregory Farnum  wrote:
>>>
>>> Are you sure you're using only CephFS? Do you have any snapshots?
>>> -Greg
>>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>>
>>>
>>> On Fri, Oct 25, 2013 at 2:59 AM, Miguel Afonso Oliveira
>>>  wrote:

 Hi,

 I have a recent ceph deployment with version:

 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)

 on 4 12TB OSDs:

 GLOBAL:
  SIZE   AVAIL RAW USED %RAW USED
  49143G 8285G 40858G   83.14

 POOLS:
  NAME ID USED   %USED OBJECTS
  data 0  20396G 41.50 7342052
  metadata 1  276M   0 81826
  rbd  2  0  0 0

 and this morning I started to get a warning about a full OSD:

cluster 14320bfb-8b8c-4280-afee-df63172b1d0c
 health HEALTH_WARN 1 near full osd(s)
 monmap e3: 3 mons at

 {gridio1=10.112.0.148:6789/0,gridio2=10.112.0.149:6789/0,gridio3=10.112.0.150:6789/0},
 election epoch 44, quorum 0,1,2 gridio1,gridio2,gridio3
 osdmap e498: 4 osds: 4 up, 4 in
  pgmap v485463: 6144 pgs: 6142 active+clean, 2
 active+clean+scrubbing+deep; 20396 GB data, 40858 GB used, 8285 GB /
 49143
 GB avail; 2252B/s wr, 0op/s
 mdsmap e54: 1/1/1 up {0=gridio4=up:active}

 However when I use a du on the mount point I get:

 [root@ce01 /]# du -bsh grid/
 31Ggrid/
>>
>> [root@ce01 /]# du -bsh grid/
>> 31Ggrid/

 what is the output of 'getfattr -d -m - grid/' ?
>
>
> [root@ce01 ~]# getfattr -d -m -  /grid
> getfattr: Removing leading '/' from absolute path names
> # file: grid
> ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304
> pool=data"
>
>
>
>> sounds like the 'purge strays' bug. try umounting all clients and
>> restarting the mds.
>
>
> I think you have nailed it on the head! I have now updated to:
>
> ceph version 0.72.1 (4d923861868f6a15dcb33fef7f50f674997322de)
>
> but I still see the same behavior. Is there anything else I can do other
> than
> keep having to do this every time until the bug is solved? Any idea when
> will
> that be? Next release?

If your issue is caused by the bug I presume, you need to use the
newest client (0.72 ceph-fuse or 3.12 kernel)

Regards
Yan, Zheng


>
> Cheers,
>
> MAO
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Granularity/efficiency of copy-on-write?

[Bill Eldridge]

Hi all,

We're looking at using Ceph's copy-on-write for a ton of users' 
replicated cloud image environments,
and are wondering how efficient Ceph is for adding user data to base 
images -
is data added in normal 4kB or 64kB sizes, or can you specify block size 
for volumes
(so you can have video partitions with large content and email/chat/web 
cache partitions with small files)


Is Ceph's behavior & configuration for copy-on-write documented well 
somewhere?


Thanks.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] odd performance graph

[Mark Nelson]

On 12/02/2013 05:06 PM, Gruher, Joseph R wrote:

I don't know how rbd works inside, but i think ceph rbd here returns zeros
without real osd disk read if the block/sector of the rbd-disk is unused. That
would explain the graph you see. You can try adding a second rbd image and
not format/use it and benchmark this disk, then make a filesystem on it and
write some data and benchmark again...



When performance testing RBDs I generally write in the whole area before doing 
any testing to avoid this problem.  It would be interesting to have 
confirmation this is a real concern with Ceph.  I know it is in other thin 
provisioned storage, for example, VMWare.  Perhaps someone more expert can 
comment.

Also, is there any way to shortcut the write-in process?  Writing in TBs of RBD 
image can really extend the length of our performance test cycle.  It would be 
great if there was some shortcut to cause Ceph to treat the whole RBD as having 
already been written, or just go fetch data from disk on all reads regardless 
of whether that area had been written, just for testing purposes.


For our internal testing, we always write data out in it's entirety 
before doing reads as well.  Not doing so will show inaccurate results 
as you've noticed.


Mark


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] odd performance graph

[Gruher, Joseph R]

>I don't know how rbd works inside, but i think ceph rbd here returns zeros
>without real osd disk read if the block/sector of the rbd-disk is unused. That
>would explain the graph you see. You can try adding a second rbd image and
>not format/use it and benchmark this disk, then make a filesystem on it and
>write some data and benchmark again...
>

When performance testing RBDs I generally write in the whole area before doing 
any testing to avoid this problem.  It would be interesting to have 
confirmation this is a real concern with Ceph.  I know it is in other thin 
provisioned storage, for example, VMWare.  Perhaps someone more expert can 
comment.

Also, is there any way to shortcut the write-in process?  Writing in TBs of RBD 
image can really extend the length of our performance test cycle.  It would be 
great if there was some shortcut to cause Ceph to treat the whole RBD as having 
already been written, or just go fetch data from disk on all reads regardless 
of whether that area had been written, just for testing purposes.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] odd performance graph

[Smart Weblications GmbH - Florian Wiessner]

Hi,

Am 02.12.2013 04:27, schrieb James Harper:
>> Hi,
>>
 The low points are all ~35Mbytes/sec and the high points are all
 ~60Mbytes/sec. This is very reproducible.
>>>
>>> It occurred to me that just stopping the OSD's selectively would allow me to
>>> see if there was a change when one
>>> was ejected, but at no time was there a change to the graph...
>>
>> did you configure the pool with 3 copies and try to run the benchmark test
>> with one OSD only ?
>> Can you reproduce the values for each OSD ?
> 
> I'll have to do that after hours. I'm not seeing this across all VM's though 
> so I think it's a bit hit and miss (the graph is constant for that one VM 
> though). 
> 
> What I did do was to shut down each OSD selectively, and there was no change 
> to the graph.
> 
> One thing I hadn't considered is that this VM is running on a physical host 
> which has a different network set up - using LACP across 2 ports. I suspect 
> that the combination of connections and the way LACP works means that 
> sometimes the data goes across one network port (although I don't understand 
> why I'm only getting 30mbytes/second per port in that case).
> 
> I'm going to recable things at some point soon, so I'll revisit it after that.
> 
> James
> 

I also noticed a graph like this once i benchmarked w2k8 guest on ceph with rbd.
To me it looked like when the space on the drive is used, the throughput is
lower, when the space read by rbd on the drive is unused, the reads are 
superfast.

I don't know how rbd works inside, but i think ceph rbd here returns zeros
without real osd disk read if the block/sector of the rbd-disk is unused. That
would explain the graph you see. You can try adding a second rbd image and not
format/use it and benchmark this disk, then make a filesystem on it and write
some data and benchmark again...

-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Yehuda Sadeh]

I'm having trouble reproducing the issue. What version are you using?

Thanks,
Yehuda

On Mon, Dec 2, 2013 at 2:16 PM, Yehuda Sadeh  wrote:
> Actually, I read that differently. It only says that if there's more
> than 1 part, all parts except for the last one need to be > 5M. Which
> means that for uploads that are smaller than 5M there should be zero
> or one parts.
>
> On Mon, Dec 2, 2013 at 12:54 PM, Dominik Mostowiec
>  wrote:
>> You're right.
>>
>> S3 api doc: 
>> http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
>> "Err:EntityTooSmall
>> Your proposed upload is smaller than the minimum allowed object size.
>> Each part must be at least 5 MB in size, except the last part."
>>
>> Thanks.
>>
>> This error should be triggered from radosgw also.
>>
>> --
>> Regards
>> Dominik
>>
>> 2013/12/2 Yehuda Sadeh :
>>> Looks like it. There should be a guard against it (mulitpart upload
>>> minimum is 5M).
>>>
>>> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
>>>  wrote:
 Yes, this is probably upload empty file.
 This is the problem?

 --
 Regards
 Dominik


 2013/12/2 Yehuda Sadeh :
> By any chance are you uploading empty objects through the multipart 
> upload api?
>
> On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
>  wrote:
>> Hi,
>> Another file with the same problems:
>>
>> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new request
>> req=0x25406d0 =
>> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req 
>> 1314:0.52initializing
>> 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
>> s->bucket=testbucket
>> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req 1314:0.000112:s3:POST
>> /testbucket/files/192.txt::getting op
>> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req 1314:0.000118:s3:POST
>> /testbucket/files/192.txt:complete_multipart:authorizing
>> 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
>> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
>> POST
>>
>> application/xml
>> Sun, 01 Dec 2013 10:37:10 GMT
>> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req 1314:0.003399:s3:POST
>> /testbucket/files/192.txt:complete_multipart:reading permissions
>> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req 1314:0.005670:s3:POST
>> /testbucket/files/192.txt:complete_multipart:verifying op permissions
>> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions for
>> uid=0 mask=2
>> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
>> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
>> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
>> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req 1314:0.005695:s3:POST
>> /testbucket/files/192.txt:complete_multipart:verifying op params
>> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req 1314:0.005698:s3:POST
>> /testbucket/files/192.txt:complete_multipart:executing
>> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
>> d41d8cd98f00b204e9800998ecf8427e-0
>> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
>> testbucket:files/192.txt to shadow object, tag/shadow_obj haven't been
>> set
>> 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
>> tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
>> 2013-12-01 11:37:15.678973 7f7891fd3700  2 req 1314:0.122286:s3:POST
>> /testbucket/files/192.txt:complete_multipart:http status=200
>> 2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
>> req=0x25406d0 http_status=200 ==
>>
>> Yes, I can read oryginal object.
>>
>> --
>> Regards
>> Dominik
>>
>> 2013/12/2 Yehuda Sadeh :
>>> That's unknown bug. I have a guess as to how the original object was
>>> created. Can you read the original object, but only copy fails?
>>>
>>> On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" 
>>> wrote:

 Hi,
 I found that issue is related with "ETag: -0" (ends -0)
 This is known bug ?

 --
 Regards
 Dominik

 2013/12/2 Dominik Mostowiec :
 > Hi,
 > I have strange problem.
 > Obj copy (0 size) killing radosgw.
 >
 > Head for this file:
 > Content-Type: application/octet-stream
 > Server: Apache/2.2.22 (Ubuntu)
 > ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
 > Last-Modified: 2013-12-01T10:37:15Z
 >
 > rgw log.
 > 2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new 
 > request
 > req=0x2be6fa0 =
 > 2013-12-02 08:18:59.196709 7f5308ff1700  2 req
 > 237:0.58initializing

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Yehuda Sadeh]

Actually, I read that differently. It only says that if there's more
than 1 part, all parts except for the last one need to be > 5M. Which
means that for uploads that are smaller than 5M there should be zero
or one parts.

On Mon, Dec 2, 2013 at 12:54 PM, Dominik Mostowiec
 wrote:
> You're right.
>
> S3 api doc: 
> http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
> "Err:EntityTooSmall
> Your proposed upload is smaller than the minimum allowed object size.
> Each part must be at least 5 MB in size, except the last part."
>
> Thanks.
>
> This error should be triggered from radosgw also.
>
> --
> Regards
> Dominik
>
> 2013/12/2 Yehuda Sadeh :
>> Looks like it. There should be a guard against it (mulitpart upload
>> minimum is 5M).
>>
>> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
>>  wrote:
>>> Yes, this is probably upload empty file.
>>> This is the problem?
>>>
>>> --
>>> Regards
>>> Dominik
>>>
>>>
>>> 2013/12/2 Yehuda Sadeh :
 By any chance are you uploading empty objects through the multipart upload 
 api?

 On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
  wrote:
> Hi,
> Another file with the same problems:
>
> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new request
> req=0x25406d0 =
> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req 
> 1314:0.52initializing
> 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
> s->bucket=testbucket
> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req 1314:0.000112:s3:POST
> /testbucket/files/192.txt::getting op
> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req 1314:0.000118:s3:POST
> /testbucket/files/192.txt:complete_multipart:authorizing
> 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
> POST
>
> application/xml
> Sun, 01 Dec 2013 10:37:10 GMT
> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req 1314:0.003399:s3:POST
> /testbucket/files/192.txt:complete_multipart:reading permissions
> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req 1314:0.005670:s3:POST
> /testbucket/files/192.txt:complete_multipart:verifying op permissions
> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions for
> uid=0 mask=2
> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req 1314:0.005695:s3:POST
> /testbucket/files/192.txt:complete_multipart:verifying op params
> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req 1314:0.005698:s3:POST
> /testbucket/files/192.txt:complete_multipart:executing
> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
> d41d8cd98f00b204e9800998ecf8427e-0
> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
> testbucket:files/192.txt to shadow object, tag/shadow_obj haven't been
> set
> 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
> tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
> 2013-12-01 11:37:15.678973 7f7891fd3700  2 req 1314:0.122286:s3:POST
> /testbucket/files/192.txt:complete_multipart:http status=200
> 2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
> req=0x25406d0 http_status=200 ==
>
> Yes, I can read oryginal object.
>
> --
> Regards
> Dominik
>
> 2013/12/2 Yehuda Sadeh :
>> That's unknown bug. I have a guess as to how the original object was
>> created. Can you read the original object, but only copy fails?
>>
>> On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" 
>> wrote:
>>>
>>> Hi,
>>> I found that issue is related with "ETag: -0" (ends -0)
>>> This is known bug ?
>>>
>>> --
>>> Regards
>>> Dominik
>>>
>>> 2013/12/2 Dominik Mostowiec :
>>> > Hi,
>>> > I have strange problem.
>>> > Obj copy (0 size) killing radosgw.
>>> >
>>> > Head for this file:
>>> > Content-Type: application/octet-stream
>>> > Server: Apache/2.2.22 (Ubuntu)
>>> > ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
>>> > Last-Modified: 2013-12-01T10:37:15Z
>>> >
>>> > rgw log.
>>> > 2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new request
>>> > req=0x2be6fa0 =
>>> > 2013-12-02 08:18:59.196709 7f5308ff1700  2 req
>>> > 237:0.58initializing
>>> > 2013-12-02 08:18:59.196752 7f5308ff1700 10 meta>>
>>> > HTTP_X_AMZ_ACL=public-read
>>> > 2013-12-02 08:18:59.196760 7f5308ff1700 10 meta>>
>>> > HTTP_X_AMZ_COPY_SOURCE=/testbucket/testfile.xml
>>> > 2013-12-02 08:18:59.196766 7f5308ff1700 10 meta>>
>>> > HTTP_X_

Re: [ceph-users] qemu-kvm packages for centos

[Dan Van Der Ster]

You need the rhev package.

https://www.mail-archive.com/ceph-users@lists.ceph.com/msg05962.html

On Dec 2, 2013 10:18 PM, Alvin Starr  wrote:
I have been looking at the src rpm qemu-img-0.12.1.2-2.415.el6.x86_64
It looks like it there are a number of comments implying that it supprts
rdb but

qemu -h returns.
Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2
qed vhdx parallels nbd blkdebug host_cdrom host_floppy host_device file
gluster gluster gluster gluster

I most of all like the gluster gluster gluster gluster Is that a hint??

I am trying to track down if rdb is actually enabled.
when/if I figure it out I will post it to the list.


  On 12/02/2013 10:46 AM, Dan Van Der Ster wrote:
> Hi,
> See this one also: http://tracker.ceph.com/issues/6365
> But I’m not sure the Inktank patched qemu-kvm is relevant any longer since 
> RedHat just released qemu-kvm-rhev with RBD support.
> Cheers, Dan
>
> On 02 Dec 2013, at 15:36, Darren Birkett  wrote:
>
>> Hi List,
>>
>> Any chance the following will be updated with the latest packages for 
>> dumpling/emperor:
>>
>> http://ceph.com/packages/qemu-kvm/centos/x86_64/
>>
>> Using CentOS 6.4 and dumpling with OpenStack Havana, I am unable to boot 
>> from rbd volumes until I install an rbd-ified qemu-kvm.  I have grabbed the 
>> latest (cuttlefish) and it seems to work ok, but I can't be 100% sure.
>>
>> I noted the following ticket where the original packaging work was done:
>>
>> http://tracker.ceph.com/issues/4834
>>
>> ...but nothing seems to have been done since.
>>
>> Also, is this the official place for that package to live?  I'm in the 
>> process of working with the ceph chef cookbooks, as well as our own 
>> openstack cookbooks, and want to make sure the package is pulled from the 
>> right place.
>>
>> Thanks
>> Darren
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
Alvin Starr   ||   voice: (905)513-7688
Netvel Inc.   ||   Cell:  (416)806-0133
al...@netvel.net  ||

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-kvm packages for centos

[Alvin Starr]

I have been looking at the src rpm qemu-img-0.12.1.2-2.415.el6.x86_64
It looks like it there are a number of comments implying that it supprts 
rdb but


qemu -h returns.
Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2 
qed vhdx parallels nbd blkdebug host_cdrom host_floppy host_device file 
gluster gluster gluster gluster


I most of all like the gluster gluster gluster gluster Is that a hint??

I am trying to track down if rdb is actually enabled.
when/if I figure it out I will post it to the list.


 On 12/02/2013 10:46 AM, Dan Van Der Ster wrote:

Hi,
See this one also: http://tracker.ceph.com/issues/6365
But I’m not sure the Inktank patched qemu-kvm is relevant any longer since 
RedHat just released qemu-kvm-rhev with RBD support.
Cheers, Dan

On 02 Dec 2013, at 15:36, Darren Birkett  wrote:


Hi List,

Any chance the following will be updated with the latest packages for 
dumpling/emperor:

http://ceph.com/packages/qemu-kvm/centos/x86_64/

Using CentOS 6.4 and dumpling with OpenStack Havana, I am unable to boot from 
rbd volumes until I install an rbd-ified qemu-kvm.  I have grabbed the latest 
(cuttlefish) and it seems to work ok, but I can't be 100% sure.

I noted the following ticket where the original packaging work was done:

http://tracker.ceph.com/issues/4834

...but nothing seems to have been done since.

Also, is this the official place for that package to live?  I'm in the process 
of working with the ceph chef cookbooks, as well as our own openstack 
cookbooks, and want to make sure the package is pulled from the right place.

Thanks
Darren
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Alvin Starr   ||   voice: (905)513-7688
Netvel Inc.   ||   Cell:  (416)806-0133
al...@netvel.net  ||

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

You're right.

S3 api doc: http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
"Err:EntityTooSmall
Your proposed upload is smaller than the minimum allowed object size.
Each part must be at least 5 MB in size, except the last part."

Thanks.

This error should be triggered from radosgw also.

--
Regards
Dominik

2013/12/2 Yehuda Sadeh :
> Looks like it. There should be a guard against it (mulitpart upload
> minimum is 5M).
>
> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
>  wrote:
>> Yes, this is probably upload empty file.
>> This is the problem?
>>
>> --
>> Regards
>> Dominik
>>
>>
>> 2013/12/2 Yehuda Sadeh :
>>> By any chance are you uploading empty objects through the multipart upload 
>>> api?
>>>
>>> On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
>>>  wrote:
 Hi,
 Another file with the same problems:

 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new request
 req=0x25406d0 =
 2013-12-01 11:37:15.556739 7f7891fd3700  2 req 
 1314:0.52initializing
 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
 s->bucket=testbucket
 2013-12-01 11:37:15.556799 7f7891fd3700  2 req 1314:0.000112:s3:POST
 /testbucket/files/192.txt::getting op
 2013-12-01 11:37:15.556804 7f7891fd3700  2 req 1314:0.000118:s3:POST
 /testbucket/files/192.txt:complete_multipart:authorizing
 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
 dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
 POST

 application/xml
 Sun, 01 Dec 2013 10:37:10 GMT
 /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
 2013-12-01 11:37:15.560085 7f7891fd3700  2 req 1314:0.003399:s3:POST
 /testbucket/files/192.txt:complete_multipart:reading permissions
 2013-12-01 11:37:15.562356 7f7891fd3700  2 req 1314:0.005670:s3:POST
 /testbucket/files/192.txt:complete_multipart:verifying op permissions
 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions for
 uid=0 mask=2
 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
 (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
 2013-12-01 11:37:15.562381 7f7891fd3700  2 req 1314:0.005695:s3:POST
 /testbucket/files/192.txt:complete_multipart:verifying op params
 2013-12-01 11:37:15.562384 7f7891fd3700  2 req 1314:0.005698:s3:POST
 /testbucket/files/192.txt:complete_multipart:executing
 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
 d41d8cd98f00b204e9800998ecf8427e-0
 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
 testbucket:files/192.txt to shadow object, tag/shadow_obj haven't been
 set
 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
 tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
 2013-12-01 11:37:15.678973 7f7891fd3700  2 req 1314:0.122286:s3:POST
 /testbucket/files/192.txt:complete_multipart:http status=200
 2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
 req=0x25406d0 http_status=200 ==

 Yes, I can read oryginal object.

 --
 Regards
 Dominik

 2013/12/2 Yehuda Sadeh :
> That's unknown bug. I have a guess as to how the original object was
> created. Can you read the original object, but only copy fails?
>
> On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" 
> wrote:
>>
>> Hi,
>> I found that issue is related with "ETag: -0" (ends -0)
>> This is known bug ?
>>
>> --
>> Regards
>> Dominik
>>
>> 2013/12/2 Dominik Mostowiec :
>> > Hi,
>> > I have strange problem.
>> > Obj copy (0 size) killing radosgw.
>> >
>> > Head for this file:
>> > Content-Type: application/octet-stream
>> > Server: Apache/2.2.22 (Ubuntu)
>> > ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
>> > Last-Modified: 2013-12-01T10:37:15Z
>> >
>> > rgw log.
>> > 2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new request
>> > req=0x2be6fa0 =
>> > 2013-12-02 08:18:59.196709 7f5308ff1700  2 req
>> > 237:0.58initializing
>> > 2013-12-02 08:18:59.196752 7f5308ff1700 10 meta>>
>> > HTTP_X_AMZ_ACL=public-read
>> > 2013-12-02 08:18:59.196760 7f5308ff1700 10 meta>>
>> > HTTP_X_AMZ_COPY_SOURCE=/testbucket/testfile.xml
>> > 2013-12-02 08:18:59.196766 7f5308ff1700 10 meta>>
>> > HTTP_X_AMZ_METADATA_DIRECTIVE=COPY
>> > 2013-12-02 08:18:59.196771 7f5308ff1700 10 x>> x-amz-acl:public-read
>> > 2013-12-02 08:18:59.196772 7f5308ff1700 10 x>>
>> > x-amz-copy-source:/testbucket/testfile.xml
>> > 2013-12-02 08:18:59.196773 7f5308ff1700 10 x>>
>> > x-amz-metadata-directive:COPY
>> > 2013-12-02 08:18:59.196786 7f5308ff1700 10
>> > s->object=/testbucket/new_testfile.ini s->bucket=t

Re: [ceph-users] Website related discussions

[Patrick McGarry]

Hey Loic,

That's great!  As for ceph-user I was referring more to a way for
people to drop content or requests on the community at large.  For a
chatty discussion about website changes we can just include the
interested parties (message to follow) and others can chime in with
their desire to help here and join the discussion that way.  Thanks!


Best Regards,

Patrick McGarry
Director, Community || Inktank
http://ceph.com  ||  http://inktank.com
@scuttlemonkey || @ceph || @inktank


On Mon, Dec 2, 2013 at 3:42 PM, Loic Dachary  wrote:
> Hi Patrick,
>
> Aaron and Nathan are willing to take an active role in editing 
> http://ceph.com/. During the Ceph Developer Summit, it was suggested that we 
> use ceph-users@ until it's too much noise and only then create a dedicated 
> mailing list. This is a friendly warning that such a noisy conversation is 
> about to begin on the topic of webmastering ceph.com :-) Whenever you think 
> this is too much, we will stand ready to divert the discussions to a 
> dedicated mailing list.
>
> Cheers
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Yehuda Sadeh]

Looks like it. There should be a guard against it (mulitpart upload
minimum is 5M).

On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
 wrote:
> Yes, this is probably upload empty file.
> This is the problem?
>
> --
> Regards
> Dominik
>
>
> 2013/12/2 Yehuda Sadeh :
>> By any chance are you uploading empty objects through the multipart upload 
>> api?
>>
>> On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
>>  wrote:
>>> Hi,
>>> Another file with the same problems:
>>>
>>> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new request
>>> req=0x25406d0 =
>>> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req 1314:0.52initializing
>>> 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
>>> s->bucket=testbucket
>>> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req 1314:0.000112:s3:POST
>>> /testbucket/files/192.txt::getting op
>>> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req 1314:0.000118:s3:POST
>>> /testbucket/files/192.txt:complete_multipart:authorizing
>>> 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
>>> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>>> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
>>> POST
>>>
>>> application/xml
>>> Sun, 01 Dec 2013 10:37:10 GMT
>>> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>>> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req 1314:0.003399:s3:POST
>>> /testbucket/files/192.txt:complete_multipart:reading permissions
>>> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req 1314:0.005670:s3:POST
>>> /testbucket/files/192.txt:complete_multipart:verifying op permissions
>>> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions for
>>> uid=0 mask=2
>>> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
>>> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
>>> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
>>> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req 1314:0.005695:s3:POST
>>> /testbucket/files/192.txt:complete_multipart:verifying op params
>>> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req 1314:0.005698:s3:POST
>>> /testbucket/files/192.txt:complete_multipart:executing
>>> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
>>> d41d8cd98f00b204e9800998ecf8427e-0
>>> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
>>> testbucket:files/192.txt to shadow object, tag/shadow_obj haven't been
>>> set
>>> 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
>>> tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
>>> 2013-12-01 11:37:15.678973 7f7891fd3700  2 req 1314:0.122286:s3:POST
>>> /testbucket/files/192.txt:complete_multipart:http status=200
>>> 2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
>>> req=0x25406d0 http_status=200 ==
>>>
>>> Yes, I can read oryginal object.
>>>
>>> --
>>> Regards
>>> Dominik
>>>
>>> 2013/12/2 Yehuda Sadeh :
 That's unknown bug. I have a guess as to how the original object was
 created. Can you read the original object, but only copy fails?

 On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" 
 wrote:
>
> Hi,
> I found that issue is related with "ETag: -0" (ends -0)
> This is known bug ?
>
> --
> Regards
> Dominik
>
> 2013/12/2 Dominik Mostowiec :
> > Hi,
> > I have strange problem.
> > Obj copy (0 size) killing radosgw.
> >
> > Head for this file:
> > Content-Type: application/octet-stream
> > Server: Apache/2.2.22 (Ubuntu)
> > ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
> > Last-Modified: 2013-12-01T10:37:15Z
> >
> > rgw log.
> > 2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new request
> > req=0x2be6fa0 =
> > 2013-12-02 08:18:59.196709 7f5308ff1700  2 req
> > 237:0.58initializing
> > 2013-12-02 08:18:59.196752 7f5308ff1700 10 meta>>
> > HTTP_X_AMZ_ACL=public-read
> > 2013-12-02 08:18:59.196760 7f5308ff1700 10 meta>>
> > HTTP_X_AMZ_COPY_SOURCE=/testbucket/testfile.xml
> > 2013-12-02 08:18:59.196766 7f5308ff1700 10 meta>>
> > HTTP_X_AMZ_METADATA_DIRECTIVE=COPY
> > 2013-12-02 08:18:59.196771 7f5308ff1700 10 x>> x-amz-acl:public-read
> > 2013-12-02 08:18:59.196772 7f5308ff1700 10 x>>
> > x-amz-copy-source:/testbucket/testfile.xml
> > 2013-12-02 08:18:59.196773 7f5308ff1700 10 x>>
> > x-amz-metadata-directive:COPY
> > 2013-12-02 08:18:59.196786 7f5308ff1700 10
> > s->object=/testbucket/new_testfile.ini s->bucket=testbucket
> > 2013-12-02 08:18:59.196792 7f5308ff1700  2 req 237:0.000141:s3:PUT
> > /testbucket/new_testfile.ini::getting op
> > 2013-12-02 08:18:59.196797 7f5308ff1700  2 req 237:0.000146:s3:PUT
> > /testbucket/new_testfile.ini:copy_obj:authorizing
> > 2013-12-02 08:18:59.200648 7f5308ff1700 10 get_canon_resource():
> > dest=/testbucket/new_testfile.ini
> > 2013-12-02 08:18:59.200661 7f5308ff1700 10 auth_hdr:
> > PUT
> > 1B2M2Y8AsgTpgAmY7

[ceph-users] Website related discussions

[Loic Dachary]

Hi Patrick,

Aaron and Nathan are willing to take an active role in editing 
http://ceph.com/. During the Ceph Developer Summit, it was suggested that we 
use ceph-users@ until it's too much noise and only then create a dedicated 
mailing list. This is a friendly warning that such a noisy conversation is 
about to begin on the topic of webmastering ceph.com :-) Whenever you think 
this is too much, we will stand ready to divert the discussions to a dedicated 
mailing list.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

Yes, this is probably upload empty file.
This is the problem?

--
Regards
Dominik


2013/12/2 Yehuda Sadeh :
> By any chance are you uploading empty objects through the multipart upload 
> api?
>
> On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
>  wrote:
>> Hi,
>> Another file with the same problems:
>>
>> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new request
>> req=0x25406d0 =
>> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req 1314:0.52initializing
>> 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
>> s->bucket=testbucket
>> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req 1314:0.000112:s3:POST
>> /testbucket/files/192.txt::getting op
>> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req 1314:0.000118:s3:POST
>> /testbucket/files/192.txt:complete_multipart:authorizing
>> 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
>> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
>> POST
>>
>> application/xml
>> Sun, 01 Dec 2013 10:37:10 GMT
>> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req 1314:0.003399:s3:POST
>> /testbucket/files/192.txt:complete_multipart:reading permissions
>> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req 1314:0.005670:s3:POST
>> /testbucket/files/192.txt:complete_multipart:verifying op permissions
>> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions for
>> uid=0 mask=2
>> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
>> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
>> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
>> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req 1314:0.005695:s3:POST
>> /testbucket/files/192.txt:complete_multipart:verifying op params
>> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req 1314:0.005698:s3:POST
>> /testbucket/files/192.txt:complete_multipart:executing
>> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
>> d41d8cd98f00b204e9800998ecf8427e-0
>> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
>> testbucket:files/192.txt to shadow object, tag/shadow_obj haven't been
>> set
>> 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
>> tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
>> 2013-12-01 11:37:15.678973 7f7891fd3700  2 req 1314:0.122286:s3:POST
>> /testbucket/files/192.txt:complete_multipart:http status=200
>> 2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
>> req=0x25406d0 http_status=200 ==
>>
>> Yes, I can read oryginal object.
>>
>> --
>> Regards
>> Dominik
>>
>> 2013/12/2 Yehuda Sadeh :
>>> That's unknown bug. I have a guess as to how the original object was
>>> created. Can you read the original object, but only copy fails?
>>>
>>> On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" 
>>> wrote:

 Hi,
 I found that issue is related with "ETag: -0" (ends -0)
 This is known bug ?

 --
 Regards
 Dominik

 2013/12/2 Dominik Mostowiec :
 > Hi,
 > I have strange problem.
 > Obj copy (0 size) killing radosgw.
 >
 > Head for this file:
 > Content-Type: application/octet-stream
 > Server: Apache/2.2.22 (Ubuntu)
 > ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
 > Last-Modified: 2013-12-01T10:37:15Z
 >
 > rgw log.
 > 2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new request
 > req=0x2be6fa0 =
 > 2013-12-02 08:18:59.196709 7f5308ff1700  2 req
 > 237:0.58initializing
 > 2013-12-02 08:18:59.196752 7f5308ff1700 10 meta>>
 > HTTP_X_AMZ_ACL=public-read
 > 2013-12-02 08:18:59.196760 7f5308ff1700 10 meta>>
 > HTTP_X_AMZ_COPY_SOURCE=/testbucket/testfile.xml
 > 2013-12-02 08:18:59.196766 7f5308ff1700 10 meta>>
 > HTTP_X_AMZ_METADATA_DIRECTIVE=COPY
 > 2013-12-02 08:18:59.196771 7f5308ff1700 10 x>> x-amz-acl:public-read
 > 2013-12-02 08:18:59.196772 7f5308ff1700 10 x>>
 > x-amz-copy-source:/testbucket/testfile.xml
 > 2013-12-02 08:18:59.196773 7f5308ff1700 10 x>>
 > x-amz-metadata-directive:COPY
 > 2013-12-02 08:18:59.196786 7f5308ff1700 10
 > s->object=/testbucket/new_testfile.ini s->bucket=testbucket
 > 2013-12-02 08:18:59.196792 7f5308ff1700  2 req 237:0.000141:s3:PUT
 > /testbucket/new_testfile.ini::getting op
 > 2013-12-02 08:18:59.196797 7f5308ff1700  2 req 237:0.000146:s3:PUT
 > /testbucket/new_testfile.ini:copy_obj:authorizing
 > 2013-12-02 08:18:59.200648 7f5308ff1700 10 get_canon_resource():
 > dest=/testbucket/new_testfile.ini
 > 2013-12-02 08:18:59.200661 7f5308ff1700 10 auth_hdr:
 > PUT
 > 1B2M2Y8AsgTpgAmY7PhCfg==
 > application/octet-stream
 > Mon, 02 Dec 2013 07:18:55 GMT
 > x-amz-acl:public-read
 > x-amz-copy-source:/testbucket/testfile.xml
 > x-amz-metadata-directive:COPY
 > /testbucket/new_testfile.ini
 > 2013-12-02 08:18:59.200717 7f530

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Yehuda Sadeh]

By any chance are you uploading empty objects through the multipart upload api?

On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
 wrote:
> Hi,
> Another file with the same problems:
>
> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new request
> req=0x25406d0 =
> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req 1314:0.52initializing
> 2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
> s->bucket=testbucket
> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req 1314:0.000112:s3:POST
> /testbucket/files/192.txt::getting op
> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req 1314:0.000118:s3:POST
> /testbucket/files/192.txt:complete_multipart:authorizing
> 2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
> POST
>
> application/xml
> Sun, 01 Dec 2013 10:37:10 GMT
> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req 1314:0.003399:s3:POST
> /testbucket/files/192.txt:complete_multipart:reading permissions
> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req 1314:0.005670:s3:POST
> /testbucket/files/192.txt:complete_multipart:verifying op permissions
> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions for
> uid=0 mask=2
> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
> 2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
> 2013-12-01 11:37:15.562381 7f7891fd3700  2 req 1314:0.005695:s3:POST
> /testbucket/files/192.txt:complete_multipart:verifying op params
> 2013-12-01 11:37:15.562384 7f7891fd3700  2 req 1314:0.005698:s3:POST
> /testbucket/files/192.txt:complete_multipart:executing
> 2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
> d41d8cd98f00b204e9800998ecf8427e-0
> 2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
> testbucket:files/192.txt to shadow object, tag/shadow_obj haven't been
> set
> 2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
> tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
> 2013-12-01 11:37:15.678973 7f7891fd3700  2 req 1314:0.122286:s3:POST
> /testbucket/files/192.txt:complete_multipart:http status=200
> 2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
> req=0x25406d0 http_status=200 ==
>
> Yes, I can read oryginal object.
>
> --
> Regards
> Dominik
>
> 2013/12/2 Yehuda Sadeh :
>> That's unknown bug. I have a guess as to how the original object was
>> created. Can you read the original object, but only copy fails?
>>
>> On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" 
>> wrote:
>>>
>>> Hi,
>>> I found that issue is related with "ETag: -0" (ends -0)
>>> This is known bug ?
>>>
>>> --
>>> Regards
>>> Dominik
>>>
>>> 2013/12/2 Dominik Mostowiec :
>>> > Hi,
>>> > I have strange problem.
>>> > Obj copy (0 size) killing radosgw.
>>> >
>>> > Head for this file:
>>> > Content-Type: application/octet-stream
>>> > Server: Apache/2.2.22 (Ubuntu)
>>> > ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
>>> > Last-Modified: 2013-12-01T10:37:15Z
>>> >
>>> > rgw log.
>>> > 2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new request
>>> > req=0x2be6fa0 =
>>> > 2013-12-02 08:18:59.196709 7f5308ff1700  2 req
>>> > 237:0.58initializing
>>> > 2013-12-02 08:18:59.196752 7f5308ff1700 10 meta>>
>>> > HTTP_X_AMZ_ACL=public-read
>>> > 2013-12-02 08:18:59.196760 7f5308ff1700 10 meta>>
>>> > HTTP_X_AMZ_COPY_SOURCE=/testbucket/testfile.xml
>>> > 2013-12-02 08:18:59.196766 7f5308ff1700 10 meta>>
>>> > HTTP_X_AMZ_METADATA_DIRECTIVE=COPY
>>> > 2013-12-02 08:18:59.196771 7f5308ff1700 10 x>> x-amz-acl:public-read
>>> > 2013-12-02 08:18:59.196772 7f5308ff1700 10 x>>
>>> > x-amz-copy-source:/testbucket/testfile.xml
>>> > 2013-12-02 08:18:59.196773 7f5308ff1700 10 x>>
>>> > x-amz-metadata-directive:COPY
>>> > 2013-12-02 08:18:59.196786 7f5308ff1700 10
>>> > s->object=/testbucket/new_testfile.ini s->bucket=testbucket
>>> > 2013-12-02 08:18:59.196792 7f5308ff1700  2 req 237:0.000141:s3:PUT
>>> > /testbucket/new_testfile.ini::getting op
>>> > 2013-12-02 08:18:59.196797 7f5308ff1700  2 req 237:0.000146:s3:PUT
>>> > /testbucket/new_testfile.ini:copy_obj:authorizing
>>> > 2013-12-02 08:18:59.200648 7f5308ff1700 10 get_canon_resource():
>>> > dest=/testbucket/new_testfile.ini
>>> > 2013-12-02 08:18:59.200661 7f5308ff1700 10 auth_hdr:
>>> > PUT
>>> > 1B2M2Y8AsgTpgAmY7PhCfg==
>>> > application/octet-stream
>>> > Mon, 02 Dec 2013 07:18:55 GMT
>>> > x-amz-acl:public-read
>>> > x-amz-copy-source:/testbucket/testfile.xml
>>> > x-amz-metadata-directive:COPY
>>> > /testbucket/new_testfile.ini
>>> > 2013-12-02 08:18:59.200717 7f5308ff1700  2 req 237:0.004066:s3:PUT
>>> > /testbucket/new_testfile.ini:copy_obj:reading permissions
>>> > 2013-12-02 08:18:59.203330 7f5308ff1700  2 req 237:0.006679:s3:PUT
>>> > /testbucket/new_testfile.ini:copy_obj:verifying op p

Re: [ceph-users] CEPH HA with Ubuntu OpenStack and Highly Available Controller Nodes

[Mike Dawson]

Gary,

Yes, Ceph can provide a highly available infrastructure. A few pointers:

- IO will stop if anything less than a majority of the monitors are 
functioning properly. So, run a minimum of 3 Ceph monitors distributed 
across your data center's failure domains. If you choose to add more 
monitors, it is advisable to add them in pairs to maintain an odd number.


- Run at least a replication size of 2 (lots of operators choose 3 for 
more redundancy). Ceph will gracefully handle failure conditions of the 
primary OSD once it is automatically or manually marked as "down".


- Design your CRUSH map to mimic the failure domains in your datacenter 
(root, room, row, rack, chassis, host, etc). Use the CRUSH chooseleaf 
rules to spread replicas across the largest failure domain that will 
have more entries than your replication factor. For instance, try to 
replicate across racks rather than hosts if your cluster will be large 
enough.


- Don't set the "ceph osd set nodown" flag on your cluster, as it will 
prevent osds from being marked as down automatically if unavailable, 
substantially diminishing the HA capabilities.


Cheers,
Mike Dawson


On 12/2/2013 11:43 AM, Gary Harris (gharris) wrote:

Hi, I have a question about CEP high availability when integrated with
OpenStack.


Assuming you have all openstack controller nodes in HA mode would your
actually
have an HA CEPH implementation as well meaning two Primary OSDs or both
pointing to the primary?

Or,  do the client requests get forwarded automatically to the secondary
OSD should the primary fail?
(Excuse the simplifications):)

So my assumption is that if the primary fails, the mon would detect and
a new primary OSD candidate would be presented to clients?

Thanks,
Gary




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH HA with Ubuntu OpenStack and Highly Available Controller Nodes

[Randy Smith]

On Mon, Dec 2, 2013 at 9:43 AM, Gary Harris (gharris) wrote:

>  Hi, I have a question about CEP high availability when integrated with
> OpenStack.
>
>
>  Assuming you have all openstack controller nodes in HA mode would your
> actually
>   have an HA CEPH implementation as well meaning two Primary OSDs or both
> pointing to the primary?
>

Ceph is clustered, replicated, distributed storage so the concepts of
primary/secondary are bit different. Each osd will have its own local
storage. Ceph will take care of replication between osds based on how you
configure it. If an osd fails, clients will look for one of the replicas.
Ceph will also take care of re-replicating the data around the cluster as
needed.


>
>   Or,  do the client requests get forwarded automatically to the
> secondary OSD should the primary fail?
>   (Excuse the simplifications):)
>
>  So my assumption is that if the primary fails, the mon would detect and
> a new primary OSD candidate would be presented to clients?
>

The mon will detect the error and update the crush map as needed so that
clients can figure out where their data is.

You can read about the crush map at
http://ceph.com/docs/master/rados/operations/crush-map/.


>
>  Thanks,
> Gary
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
Randall Smith
Computing Services
Adams State University
http://www.adams.edu/
719-587-7411
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

Hi,
Another file with the same problems:

2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new request
req=0x25406d0 =
2013-12-01 11:37:15.556739 7f7891fd3700  2 req 1314:0.52initializing
2013-12-01 11:37:15.556789 7f7891fd3700 10 s->object=files/192.txt
s->bucket=testbucket
2013-12-01 11:37:15.556799 7f7891fd3700  2 req 1314:0.000112:s3:POST
/testbucket/files/192.txt::getting op
2013-12-01 11:37:15.556804 7f7891fd3700  2 req 1314:0.000118:s3:POST
/testbucket/files/192.txt:complete_multipart:authorizing
2013-12-01 11:37:15.560013 7f7891fd3700 10 get_canon_resource():
dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
POST

application/xml
Sun, 01 Dec 2013 10:37:10 GMT
/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
2013-12-01 11:37:15.560085 7f7891fd3700  2 req 1314:0.003399:s3:POST
/testbucket/files/192.txt:complete_multipart:reading permissions
2013-12-01 11:37:15.562356 7f7891fd3700  2 req 1314:0.005670:s3:POST
/testbucket/files/192.txt:complete_multipart:verifying op permissions
2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching permissions for
uid=0 mask=2
2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission: 15
2013-12-01 11:37:15.562378 7f7891fd3700 10  uid=0 requested perm
(type)=2, policy perm=2, user_perm_mask=2, acl perm=2
2013-12-01 11:37:15.562381 7f7891fd3700  2 req 1314:0.005695:s3:POST
/testbucket/files/192.txt:complete_multipart:verifying op params
2013-12-01 11:37:15.562384 7f7891fd3700  2 req 1314:0.005698:s3:POST
/testbucket/files/192.txt:complete_multipart:executing
2013-12-01 11:37:15.565461 7f7891fd3700 10 calculated etag:
d41d8cd98f00b204e9800998ecf8427e-0
2013-12-01 11:37:15.566718 7f7891fd3700 10 can't clone object
testbucket:files/192.txt to shadow object, tag/shadow_obj haven't been
set
2013-12-01 11:37:15.566777 7f7891fd3700  0 setting object
tag=_leyAzxCw7YxpKv8P3v3QGwcsw__9VmP
2013-12-01 11:37:15.678973 7f7891fd3700  2 req 1314:0.122286:s3:POST
/testbucket/files/192.txt:complete_multipart:http status=200
2013-12-01 11:37:15.679192 7f7891fd3700  1 == req done
req=0x25406d0 http_status=200 ==

Yes, I can read oryginal object.

--
Regards
Dominik

2013/12/2 Yehuda Sadeh :
> That's unknown bug. I have a guess as to how the original object was
> created. Can you read the original object, but only copy fails?
>
> On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" 
> wrote:
>>
>> Hi,
>> I found that issue is related with "ETag: -0" (ends -0)
>> This is known bug ?
>>
>> --
>> Regards
>> Dominik
>>
>> 2013/12/2 Dominik Mostowiec :
>> > Hi,
>> > I have strange problem.
>> > Obj copy (0 size) killing radosgw.
>> >
>> > Head for this file:
>> > Content-Type: application/octet-stream
>> > Server: Apache/2.2.22 (Ubuntu)
>> > ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
>> > Last-Modified: 2013-12-01T10:37:15Z
>> >
>> > rgw log.
>> > 2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new request
>> > req=0x2be6fa0 =
>> > 2013-12-02 08:18:59.196709 7f5308ff1700  2 req
>> > 237:0.58initializing
>> > 2013-12-02 08:18:59.196752 7f5308ff1700 10 meta>>
>> > HTTP_X_AMZ_ACL=public-read
>> > 2013-12-02 08:18:59.196760 7f5308ff1700 10 meta>>
>> > HTTP_X_AMZ_COPY_SOURCE=/testbucket/testfile.xml
>> > 2013-12-02 08:18:59.196766 7f5308ff1700 10 meta>>
>> > HTTP_X_AMZ_METADATA_DIRECTIVE=COPY
>> > 2013-12-02 08:18:59.196771 7f5308ff1700 10 x>> x-amz-acl:public-read
>> > 2013-12-02 08:18:59.196772 7f5308ff1700 10 x>>
>> > x-amz-copy-source:/testbucket/testfile.xml
>> > 2013-12-02 08:18:59.196773 7f5308ff1700 10 x>>
>> > x-amz-metadata-directive:COPY
>> > 2013-12-02 08:18:59.196786 7f5308ff1700 10
>> > s->object=/testbucket/new_testfile.ini s->bucket=testbucket
>> > 2013-12-02 08:18:59.196792 7f5308ff1700  2 req 237:0.000141:s3:PUT
>> > /testbucket/new_testfile.ini::getting op
>> > 2013-12-02 08:18:59.196797 7f5308ff1700  2 req 237:0.000146:s3:PUT
>> > /testbucket/new_testfile.ini:copy_obj:authorizing
>> > 2013-12-02 08:18:59.200648 7f5308ff1700 10 get_canon_resource():
>> > dest=/testbucket/new_testfile.ini
>> > 2013-12-02 08:18:59.200661 7f5308ff1700 10 auth_hdr:
>> > PUT
>> > 1B2M2Y8AsgTpgAmY7PhCfg==
>> > application/octet-stream
>> > Mon, 02 Dec 2013 07:18:55 GMT
>> > x-amz-acl:public-read
>> > x-amz-copy-source:/testbucket/testfile.xml
>> > x-amz-metadata-directive:COPY
>> > /testbucket/new_testfile.ini
>> > 2013-12-02 08:18:59.200717 7f5308ff1700  2 req 237:0.004066:s3:PUT
>> > /testbucket/new_testfile.ini:copy_obj:reading permissions
>> > 2013-12-02 08:18:59.203330 7f5308ff1700  2 req 237:0.006679:s3:PUT
>> > /testbucket/new_testfile.ini:copy_obj:verifying op permissions
>> > 2013-12-02 08:18:59.207627 7f5308ff1700 10 manifest: total_size = 0
>> > 2013-12-02 08:18:59.207649 7f5308ff1700  5 Searching permissions for
>> > uid=0 mask=1
>> > 2013-12-02 08:18:59.207652 7f5308ff1700  5 Found permission: 15
>> > 2013-12-02 08:18:59.207654 7f5308ff1700 10  uid=0 r

[ceph-users] CEPH HA with Ubuntu OpenStack and Highly Available Controller Nodes

[Gary Harris (gharris)]

Hi, I have a question about CEP high availability when integrated with 
OpenStack.


Assuming you have all openstack controller nodes in HA mode would your actually
have an HA CEPH implementation as well meaning two Primary OSDs or both 
pointing to the primary?

Or,  do the client requests get forwarded automatically to the secondary OSD 
should the primary fail?
(Excuse the simplifications):)

So my assumption is that if the primary fails, the mon would detect and a new 
primary OSD candidate would be presented to clients?

Thanks,
Gary

[cid:51B27B33-7251-4BC1-812D-9059FF8EBF2D]

<>___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

Yes I can read it.  Oryginal object is 0 size.

Regards
Dominik
On Dec 2, 2013 6:14 PM, "Yehuda Sadeh"  wrote:

> That's unknown bug. I have a guess as to how the original object was
> created. Can you read the original object, but only copy fails?
> On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" 
> wrote:
>
>> Hi,
>> I found that issue is related with "ETag: -0" (ends -0)
>> This is known bug ?
>>
>> --
>> Regards
>> Dominik
>>
>> 2013/12/2 Dominik Mostowiec :
>> > Hi,
>> > I have strange problem.
>> > Obj copy (0 size) killing radosgw.
>> >
>> > Head for this file:
>> > Content-Type: application/octet-stream
>> > Server: Apache/2.2.22 (Ubuntu)
>> > ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
>> > Last-Modified: 2013-12-01T10:37:15Z
>> >
>> > rgw log.
>> > 2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new request
>> > req=0x2be6fa0 =
>> > 2013-12-02 08:18:59.196709 7f5308ff1700  2 req
>> 237:0.58initializing
>> > 2013-12-02 08:18:59.196752 7f5308ff1700 10 meta>>
>> HTTP_X_AMZ_ACL=public-read
>> > 2013-12-02 08:18:59.196760 7f5308ff1700 10 meta>>
>> > HTTP_X_AMZ_COPY_SOURCE=/testbucket/testfile.xml
>> > 2013-12-02 08:18:59.196766 7f5308ff1700 10 meta>>
>> > HTTP_X_AMZ_METADATA_DIRECTIVE=COPY
>> > 2013-12-02 08:18:59.196771 7f5308ff1700 10 x>> x-amz-acl:public-read
>> > 2013-12-02 08:18:59.196772 7f5308ff1700 10 x>>
>> > x-amz-copy-source:/testbucket/testfile.xml
>> > 2013-12-02 08:18:59.196773 7f5308ff1700 10 x>>
>> x-amz-metadata-directive:COPY
>> > 2013-12-02 08:18:59.196786 7f5308ff1700 10
>> > s->object=/testbucket/new_testfile.ini s->bucket=testbucket
>> > 2013-12-02 08:18:59.196792 7f5308ff1700  2 req 237:0.000141:s3:PUT
>> > /testbucket/new_testfile.ini::getting op
>> > 2013-12-02 08:18:59.196797 7f5308ff1700  2 req 237:0.000146:s3:PUT
>> > /testbucket/new_testfile.ini:copy_obj:authorizing
>> > 2013-12-02 08:18:59.200648 7f5308ff1700 10 get_canon_resource():
>> > dest=/testbucket/new_testfile.ini
>> > 2013-12-02 08:18:59.200661 7f5308ff1700 10 auth_hdr:
>> > PUT
>> > 1B2M2Y8AsgTpgAmY7PhCfg==
>> > application/octet-stream
>> > Mon, 02 Dec 2013 07:18:55 GMT
>> > x-amz-acl:public-read
>> > x-amz-copy-source:/testbucket/testfile.xml
>> > x-amz-metadata-directive:COPY
>> > /testbucket/new_testfile.ini
>> > 2013-12-02 08:18:59.200717 7f5308ff1700  2 req 237:0.004066:s3:PUT
>> > /testbucket/new_testfile.ini:copy_obj:reading permissions
>> > 2013-12-02 08:18:59.203330 7f5308ff1700  2 req 237:0.006679:s3:PUT
>> > /testbucket/new_testfile.ini:copy_obj:verifying op permissions
>> > 2013-12-02 08:18:59.207627 7f5308ff1700 10 manifest: total_size = 0
>> > 2013-12-02 08:18:59.207649 7f5308ff1700  5 Searching permissions for
>> > uid=0 mask=1
>> > 2013-12-02 08:18:59.207652 7f5308ff1700  5 Found permission: 15
>> > 2013-12-02 08:18:59.207654 7f5308ff1700 10  uid=0 requested perm
>> > (type)=1, policy perm=1, user_perm_mask=15, acl perm=1
>> > 2013-12-02 08:18:59.207669 7f5308ff1700  5 Searching permissions for
>> > uid=0 mask=2
>> > 2013-12-02 08:18:59.207670 7f5308ff1700  5 Found permission: 15
>> > 2013-12-02 08:18:59.207671 7f5308ff1700 10  uid=0 requested perm
>> > (type)=2, policy perm=2, user_perm_mask=15, acl perm=2
>> > 2013-12-02 08:18:59.207681 7f5308ff1700  2 req 237:0.011030:s3:PUT
>> > /testbucket/new_testfile.ini:copy_obj:verifying op params
>> > 2013-12-02 08:18:59.207686 7f5308ff1700  2 req 237:0.011035:s3:PUT
>> > /testbucket/new_testfile.ini:copy_obj:executing
>> > 2013-12-02 08:18:59.207699 7f5308ff1700 10 x>> x-amz-acl:public-read
>> > 2013-12-02 08:18:59.207704 7f5308ff1700 10 x>>
>> > x-amz-copy-source:/testbucket/testfile.xml
>> > 2013-12-02 08:18:59.207709 7f5308ff1700 10 x>>
>> x-amz-metadata-directive:COPY
>> > 2013-12-02 08:18:59.207759 7f5308ff1700  5 Copy object
>> > testbucket(@.rgw.buckets[406250.1]):testfile.ini =>
>> > testbucket(@.rgw.buckets[406250.1]):new_testfile.ini
>> > 2013-12-02 08:18:59.208903 7f5308ff1700 -1 *** Caught signal
>> > (Segmentation fault) **
>> >  in thread 7f5308ff1700
>> >
>> >
>> > --
>> > Regards
>> > Dominik
>>
>>
>>
>> --
>> Pozdrawiam
>> Dominik
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding new OSDs, need to increase PGs?

[Brian Andrus]

Yes, I would recommend increasing PGs in your case.

The pg_num and pgp_num recommendations are designed to be fairly broad to
cover a wide range of different hardware that a ceph user might be
utilizing. You basically should be using a number that will ensure data
granularity across all your OSDs. Setting your pg_num and pgp_num to say...
1024 would A) increase data granularity, B) likely lend no noticeable
increase to resource consumption, and C) allow some room for future OSDs to
be added while still within range of acceptable pg numbers. You could
probably safely double even that number if you plan on expanding at a rapid
rate and want to avoid splitting PGs every time a node is added.

In general, you can conservatively err on the larger side when it comes to
pg/p_num. Any excess resource utilization will be negligible (up to a
certain point). If you have a comfortable amount of available RAM, you
could experiment with increasing the multiplier in the equation you are
using and see how it affects your final number.

The pg_num and pgp_num parameters can safely be changed before or after
your new nodes are integrated.

~Brian

On Sat, Nov 30, 2013 at 11:35 PM, Indra Pramana  wrote:

> Dear all,
>
> Greetings to all, I am new to this list. Please mind my newbie question. :)
>
> I am running a Ceph cluster with 3 servers and 4 drives / OSDs per server.
> So total currently there are 12 OSDs running on the cluster. I set PGs
> (Placement Groups) value to 600 based on recommendation of calculating
> number of PGs = number of OSDs * 100 / number of replicas, which is 2.
>
> Now I am going to add one more OSD node with 4 drives (OSDs) to make up to
> 16 OSDs in total.
>
> My question, do I need to increase the PGs value to 800? Or leave it at
> 600? If I need to increase, on which step I need to increase the value
> (pg_num and pgp_num) during the insertion of the new node?
>
> Any advice is greatly appreciated, thank you.
>
> Cheers.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Yehuda Sadeh]

That's unknown bug. I have a guess as to how the original object was
created. Can you read the original object, but only copy fails?
On Dec 2, 2013 4:53 AM, "Dominik Mostowiec" 
wrote:

> Hi,
> I found that issue is related with "ETag: -0" (ends -0)
> This is known bug ?
>
> --
> Regards
> Dominik
>
> 2013/12/2 Dominik Mostowiec :
> > Hi,
> > I have strange problem.
> > Obj copy (0 size) killing radosgw.
> >
> > Head for this file:
> > Content-Type: application/octet-stream
> > Server: Apache/2.2.22 (Ubuntu)
> > ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
> > Last-Modified: 2013-12-01T10:37:15Z
> >
> > rgw log.
> > 2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new request
> > req=0x2be6fa0 =
> > 2013-12-02 08:18:59.196709 7f5308ff1700  2 req
> 237:0.58initializing
> > 2013-12-02 08:18:59.196752 7f5308ff1700 10 meta>>
> HTTP_X_AMZ_ACL=public-read
> > 2013-12-02 08:18:59.196760 7f5308ff1700 10 meta>>
> > HTTP_X_AMZ_COPY_SOURCE=/testbucket/testfile.xml
> > 2013-12-02 08:18:59.196766 7f5308ff1700 10 meta>>
> > HTTP_X_AMZ_METADATA_DIRECTIVE=COPY
> > 2013-12-02 08:18:59.196771 7f5308ff1700 10 x>> x-amz-acl:public-read
> > 2013-12-02 08:18:59.196772 7f5308ff1700 10 x>>
> > x-amz-copy-source:/testbucket/testfile.xml
> > 2013-12-02 08:18:59.196773 7f5308ff1700 10 x>>
> x-amz-metadata-directive:COPY
> > 2013-12-02 08:18:59.196786 7f5308ff1700 10
> > s->object=/testbucket/new_testfile.ini s->bucket=testbucket
> > 2013-12-02 08:18:59.196792 7f5308ff1700  2 req 237:0.000141:s3:PUT
> > /testbucket/new_testfile.ini::getting op
> > 2013-12-02 08:18:59.196797 7f5308ff1700  2 req 237:0.000146:s3:PUT
> > /testbucket/new_testfile.ini:copy_obj:authorizing
> > 2013-12-02 08:18:59.200648 7f5308ff1700 10 get_canon_resource():
> > dest=/testbucket/new_testfile.ini
> > 2013-12-02 08:18:59.200661 7f5308ff1700 10 auth_hdr:
> > PUT
> > 1B2M2Y8AsgTpgAmY7PhCfg==
> > application/octet-stream
> > Mon, 02 Dec 2013 07:18:55 GMT
> > x-amz-acl:public-read
> > x-amz-copy-source:/testbucket/testfile.xml
> > x-amz-metadata-directive:COPY
> > /testbucket/new_testfile.ini
> > 2013-12-02 08:18:59.200717 7f5308ff1700  2 req 237:0.004066:s3:PUT
> > /testbucket/new_testfile.ini:copy_obj:reading permissions
> > 2013-12-02 08:18:59.203330 7f5308ff1700  2 req 237:0.006679:s3:PUT
> > /testbucket/new_testfile.ini:copy_obj:verifying op permissions
> > 2013-12-02 08:18:59.207627 7f5308ff1700 10 manifest: total_size = 0
> > 2013-12-02 08:18:59.207649 7f5308ff1700  5 Searching permissions for
> > uid=0 mask=1
> > 2013-12-02 08:18:59.207652 7f5308ff1700  5 Found permission: 15
> > 2013-12-02 08:18:59.207654 7f5308ff1700 10  uid=0 requested perm
> > (type)=1, policy perm=1, user_perm_mask=15, acl perm=1
> > 2013-12-02 08:18:59.207669 7f5308ff1700  5 Searching permissions for
> > uid=0 mask=2
> > 2013-12-02 08:18:59.207670 7f5308ff1700  5 Found permission: 15
> > 2013-12-02 08:18:59.207671 7f5308ff1700 10  uid=0 requested perm
> > (type)=2, policy perm=2, user_perm_mask=15, acl perm=2
> > 2013-12-02 08:18:59.207681 7f5308ff1700  2 req 237:0.011030:s3:PUT
> > /testbucket/new_testfile.ini:copy_obj:verifying op params
> > 2013-12-02 08:18:59.207686 7f5308ff1700  2 req 237:0.011035:s3:PUT
> > /testbucket/new_testfile.ini:copy_obj:executing
> > 2013-12-02 08:18:59.207699 7f5308ff1700 10 x>> x-amz-acl:public-read
> > 2013-12-02 08:18:59.207704 7f5308ff1700 10 x>>
> > x-amz-copy-source:/testbucket/testfile.xml
> > 2013-12-02 08:18:59.207709 7f5308ff1700 10 x>>
> x-amz-metadata-directive:COPY
> > 2013-12-02 08:18:59.207759 7f5308ff1700  5 Copy object
> > testbucket(@.rgw.buckets[406250.1]):testfile.ini =>
> > testbucket(@.rgw.buckets[406250.1]):new_testfile.ini
> > 2013-12-02 08:18:59.208903 7f5308ff1700 -1 *** Caught signal
> > (Segmentation fault) **
> >  in thread 7f5308ff1700
> >
> >
> > --
> > Regards
> > Dominik
>
>
>
> --
> Pozdrawiam
> Dominik
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Real size of rbd image

[Stephen Taylor]

I had not enabled discard/trim for the filesystem using the image, but I have 
done so this morning. I doesn't appear to make a difference.

The extents I'm seeing reported as "zero" are not actually discarded extents. 
They are extents that contain data that was added after the ending snapshot I'm 
giving to the diff command. If I specify a later ending snapshot or none, then 
the same extents are reported as "data" rather than as "zero" extents. Ignoring 
those seems to give me the correct size when I'm calculating a diff size from 
one snapshot to another, but I wanted to make sure that this situation is 
expected before I rely on it, although continuing to ignore these extents in 
the future if they simply go away seems easy enough.

Again, I appreciate your help.

Steve

-Original Message-
From: Josh Durgin [mailto:josh.dur...@inktank.com] 
Sent: Wednesday, November 27, 2013 7:51 PM
To: Stephen Taylor; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Real size of rbd image

On 11/26/2013 02:22 PM, Stephen Taylor wrote:
>  From ceph-users archive 08/27/2013:
>
> On 08/27/2013 01:39 PM, Timofey Koolin wrote:
>
>>/Is way to know real size of rbd image and rbd snapshots?/
>
>>/rbd ls -l write declared size of image, but I want to know real 
>>size./
>
> You can sum the sizes of the extents reported by:
>
>   rbd diff pool/image[@snap] [--format json]
>
> That's the difference since the beginning of time, so it reports all
>
> used extents.
>
> Josh
>
> I don't seem to be able to find any documentation supporting the 
> [@snap] parameter for this call, but it seems to work, at least in 
> part. I have a requirement to find the size of a snapshot relative to 
> another snapshot. Here is what I've used:
>
>  rbd diff pool/image@snap2 --from-snap snap1

Most rbd commands work on snapshots too. The help text could certainly be 
improved - suggestions welcome!

> The returned list of extents seems to include all changes since snap1, 
> not just those up to snap2, but those that have been written after 
> snap2 are labeled "zero" rather than as "data" extents. If I ignore the "zero"
> extents and sum the lengths of the "data" extents, it seems to give me 
> an accurate relative snapshot size. Is this expected behavior and the 
> correct way to calculate the size I'm looking for?

Do you have discard/trim enabled for whatever's using the image?
The diff will include discarded extents as "zero". For calculating size, it's 
fine to ignore them. It would be unexpected if these aren't listed when you 
leave out the @snap2 portion though.

Josh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-kvm packages for centos

[Dan Van Der Ster]

Hi,
See this one also: http://tracker.ceph.com/issues/6365
But I’m not sure the Inktank patched qemu-kvm is relevant any longer since 
RedHat just released qemu-kvm-rhev with RBD support. 
Cheers, Dan

On 02 Dec 2013, at 15:36, Darren Birkett  wrote:

> Hi List,
> 
> Any chance the following will be updated with the latest packages for 
> dumpling/emperor:
> 
> http://ceph.com/packages/qemu-kvm/centos/x86_64/
> 
> Using CentOS 6.4 and dumpling with OpenStack Havana, I am unable to boot from 
> rbd volumes until I install an rbd-ified qemu-kvm.  I have grabbed the latest 
> (cuttlefish) and it seems to work ok, but I can't be 100% sure.
> 
> I noted the following ticket where the original packaging work was done:
> 
> http://tracker.ceph.com/issues/4834
> 
> ...but nothing seems to have been done since.
> 
> Also, is this the official place for that package to live?  I'm in the 
> process of working with the ceph chef cookbooks, as well as our own openstack 
> cookbooks, and want to make sure the package is pulled from the right place.
> 
> Thanks
> Darren
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] adding another mon failed

[Alfredo Deza]

On Fri, Nov 29, 2013 at 1:24 PM, German Anders  wrote:
> Hi, i'm having issues while trying to add another monitor to my cluster:
>
>  ceph@ceph-deploy01:~/ceph-cluster$ ceph-deploy mon create ceph-node02
> [ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy mon create
> ceph-node02
> [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node02
> [ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node02 ...
> [ceph-node02][DEBUG ] connected to host: ceph-node02
> [ceph-node02][DEBUG ] detect platform information from remote host
> [ceph-node02][DEBUG ] detect machine type
> [ceph_deploy.mon][INFO  ] distro info: Ubuntu 12.10 quantal
> [ceph-node02][DEBUG ] determining if provided host has same hostname in
> remote
> [ceph-node02][DEBUG ] get remote short hostname
> [ceph-node02][DEBUG ] deploying mon to ceph-node02
> [ceph-node02][DEBUG ] get remote short hostname
> [ceph-node02][DEBUG ] remote hostname: ceph-node02
> [ceph-node02][DEBUG ] write cluster configuration to
> /etc/ceph/{cluster}.conf
> [ceph-node02][DEBUG ] create the mon path if it does not exist
> [ceph-node02][DEBUG ] checking for done path:
> /var/lib/ceph/mon/ceph-ceph-node02/done
> [ceph-node02][DEBUG ] create a done file to avoid re-doing the mon
> deployment
> [ceph-node02][DEBUG ] create the init path if it does not exist
> [ceph-node02][DEBUG ] locating the `service` executable...
> [ceph-node02][INFO  ] Running command: sudo initctl emit ceph-mon
> cluster=ceph id=ceph-node02
> [ceph-node02][INFO  ] Running command: sudo ceph --cluster=ceph
> --admin-daemon /var/run/ceph/ceph-mon.ceph-node02.asok mon_status
> [ceph-node02][ERROR ] admin_socket: exception getting command descriptions:
> [Errno 2] No such file or directory
> [ceph-node02][WARNIN] monitor: mon.ceph-node02, might not be running yet
> [ceph-node02][INFO  ] Running command: sudo ceph --cluster=ceph
> --admin-daemon /var/run/ceph/ceph-mon.ceph-node02.asok mon_status
> [ceph-node02][ERROR ] admin_socket: exception getting command descriptions:
> [Errno 2] No such file or directory
> [ceph-node02][WARNIN] ceph-node02 is not defined in `mon initial members`
> [ceph-node02][WARNIN] monitor ceph-node02 does not exist in monmap
> [ceph-node02][WARNIN] neither `public_addr` nor `public_network` keys are
> defined for monitors
> [ceph-node02][WARNIN] monitors may not be able to form quorum
> ceph@ceph-deploy01:~/ceph-cluster$
>
> # ceph status
> cluster cd60ab37-23bd-4c17-9470-404cb3b31112
>  health HEALTH_OK
>  monmap e1: 1 mons at {ceph-node01=10.111.82.242:6789/0}, election epoch
> 2, quorum 0 ceph-node01
>  mdsmap e4: 1/1/1 up {0=ceph-node01=up:active}
>  osdmap e13: 3 osds: 3 up, 3 in
>   pgmap v23: 192 pgs, 3 pools, 9470 bytes data, 21 objects
> 110 MB used, 834 GB / 834 GB avail
>  192 active+clean
>
> Anyone have any idea what could be the issue here?

The last 3 lines of the ceph-deploy logs are pointing to the root of
the problem: you have not defined "public_add" nor "public_network",
that monitor does not exist in the monmap and is not defined in
mon_initial_members.

It may be that you are running a mix of different network interfaces
and the monitors are using the incorrect one
to communicate.

Can you try and fix those warnings and see if it fixes the problem?
>
> Thanks in advance,
>
> Best regards,
>
>
> German Anders
>
>
>
>
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph User Committee Formal Announcement Format

[Regola, Nathan (Contractor)]

I'm looking forward to working with everyone involved with the Ceph User
Committee 
(http://wiki.ceph.com/01Planning/02Blueprints/Firefly/Ceph_User_Committee#D
etailed_Description). I believe that all of the members of the Ceph User
Committee should have received an email from Loic asking them to confirm
their organization's interest in being named a founding member. The formal
announcement is currently being planned for 10 December and we are working
on drafting it.

Would members prefer a single general announcement or a personalized
announcement? A personalized announcement would probably be something like
an automatically generated PDF file containing a letter (with the member's
name/affiliation) so that members could distribute it. We are open to
suggestions. If you have a preference for a general announcement listing
all of the members or a personalized announcement welcoming the user
(which obviously could include a list of all members), please reply.

Best Regards,
Nate Regola

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] does ceph-deploy adding of osds automatically update ceph.conf? It seems no...

[Alfredo Deza]

On Fri, Nov 29, 2013 at 9:46 AM, Gautam Saxena  wrote:
> I've got ceph up and running on a 3-node centos 6.4 cluster. However, after
> I
>
> a) set the cluster to nout as follows: ceph osd set noout
> b) rebooted 1 node
> c) logged into that 1 node, I tried to do: service ceph start osd.12
>
> but it returned with error message:
>
> /etc/init.d/ceph: osd.12 not found (/etc/ceph/ceph.conf defines ,
> /var/lib/ceph defines )
>
> Now, it *IS* true that the /etc/ceph/ceph.conf does NOT make any reference
> to osd.12.
>
> Am I supposed to *manually* update the master ceph.conf file to put each and
> ever osd reference into (osd.1 through osd 14 in my case) and then copy the
> ceph.conf file to each node?

Yes. ceph-deploy does not create specific sections for daemons. You
can create those and then push the config everywhere though with:

ceph-deploy config push {nodes}

I would have thought that when I add a new osd
> (using ceph-deploy) that it would *automatically* update either a) the
> master ceph.conf file on my admin machine (where ceph-deploy runs) or least
> it would update the ceph.conf file on the node that contains the osd that's
> being added.

Since ceph-deploy doesn't even create them (the daemon sections) then
no, it doesn't update those.

>
> It feels as if ceph-deploy will allow you to add osds, but this addition
> only updates the "in-memory" definitions of ceph.conf, and that after server
> reboot, it looses the in-memory definitions and attempts to re-read info
> from ceph.conf. Is this correct?

That is not entirely correct if I understand you correctly. If you
don't have daemons specified in your configuration file it means that
you would need to specify them in the command line.

For example:

sudo /etc/init.d/ceph start osd.0




If so, is there a way when using
> ceph-deploy to get it to automtically create the necessary entries for the
> ceph.conf file? If so, how? If not, then am we supposed to both a) use
> ceph-deploy to add osds and then b) manually edit ceph.conf? If so, isn't
> that dangerous/error prone?

A lot of questions here :)

* There is currently no way to make ceph-deploy to automatically
create daemon sections in ceph.conf

* Yes, you need to add those yourself and then push the config (using
the example I added above)

* That may be error prone, but it is also complexity in ceph-deploy
that is not needed for someone to get started with a Ceph cluster. For
granular operations of ceph clusters (including configuration updates)
I would recommend using a configuration management tool (Chef, Puppet,
Ansible, etc...)

That is also documented in "Why feature X is not implemented" in the
documentation:

http://ceph.com/ceph-deploy/docs/#why-is-feature-x-not-implemented


>
> Or am I missing something fundamental?
>
> (I'm using ceph version 0.72.1).
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] qemu-kvm packages for centos

[Darren Birkett]

Hi List,

Any chance the following will be updated with the latest packages for
dumpling/emperor:

http://ceph.com/packages/qemu-kvm/centos/x86_64/

Using CentOS 6.4 and dumpling with OpenStack Havana, I am unable to boot
from rbd volumes until I install an rbd-ified qemu-kvm.  I have grabbed the
latest (cuttlefish) and it seems to work ok, but I can't be 100% sure.

I noted the following ticket where the original packaging work was done:

http://tracker.ceph.com/issues/4834

...but nothing seems to have been done since.

Also, is this the official place for that package to live?  I'm in the
process of working with the ceph chef cookbooks, as well as our own
openstack cookbooks, and want to make sure the package is pulled from the
right place.

Thanks
Darren
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Nearly full OSDs with very little (apparent) FS utilization

[Miguel Afonso Oliveira]

Hi,

Sorry for the very late reply. I have been trying a lot of things...

On 25/10/13 22:40, Yan, Zheng wrote:

On Sat, Oct 26, 2013 at 2:05 AM, Gregory Farnum  wrote:

Are you sure you're using only CephFS? Do you have any snapshots?
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Oct 25, 2013 at 2:59 AM, Miguel Afonso Oliveira
 wrote:

Hi,

I have a recent ceph deployment with version:

ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)

on 4 12TB OSDs:

GLOBAL:
 SIZE   AVAIL RAW USED %RAW USED
 49143G 8285G 40858G   83.14

POOLS:
 NAME ID USED   %USED OBJECTS
 data 0  20396G 41.50 7342052
 metadata 1  276M   0 81826
 rbd  2  0  0 0

and this morning I started to get a warning about a full OSD:

   cluster 14320bfb-8b8c-4280-afee-df63172b1d0c
health HEALTH_WARN 1 near full osd(s)
monmap e3: 3 mons at
{gridio1=10.112.0.148:6789/0,gridio2=10.112.0.149:6789/0,gridio3=10.112.0.150:6789/0},
election epoch 44, quorum 0,1,2 gridio1,gridio2,gridio3
osdmap e498: 4 osds: 4 up, 4 in
 pgmap v485463: 6144 pgs: 6142 active+clean, 2
active+clean+scrubbing+deep; 20396 GB data, 40858 GB used, 8285 GB / 49143
GB avail; 2252B/s wr, 0op/s
mdsmap e54: 1/1/1 up {0=gridio4=up:active}

However when I use a du on the mount point I get:

[root@ce01 /]# du -bsh grid/
31Ggrid/

[root@ce01 /]# du -bsh grid/
31Ggrid/
what is the output of 'getfattr -d -m - grid/' ? 


[root@ce01 ~]# getfattr -d -m -  /grid
getfattr: Removing leading '/' from absolute path names
# file: grid
ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 
pool=data"




sounds like the 'purge strays' bug. try umounting all clients and
restarting the mds.


I think you have nailed it on the head! I have now updated to:

ceph version 0.72.1 (4d923861868f6a15dcb33fef7f50f674997322de)

but I still see the same behavior. Is there anything else I can do other 
than
keep having to do this every time until the bug is solved? Any idea when 
will

that be? Next release?

Cheers,

MAO



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

Hi,
I found that issue is related with "ETag: -0" (ends -0)
This is known bug ?

--
Regards
Dominik

2013/12/2 Dominik Mostowiec :
> Hi,
> I have strange problem.
> Obj copy (0 size) killing radosgw.
>
> Head for this file:
> Content-Type: application/octet-stream
> Server: Apache/2.2.22 (Ubuntu)
> ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
> Last-Modified: 2013-12-01T10:37:15Z
>
> rgw log.
> 2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new request
> req=0x2be6fa0 =
> 2013-12-02 08:18:59.196709 7f5308ff1700  2 req 237:0.58initializing
> 2013-12-02 08:18:59.196752 7f5308ff1700 10 meta>> HTTP_X_AMZ_ACL=public-read
> 2013-12-02 08:18:59.196760 7f5308ff1700 10 meta>>
> HTTP_X_AMZ_COPY_SOURCE=/testbucket/testfile.xml
> 2013-12-02 08:18:59.196766 7f5308ff1700 10 meta>>
> HTTP_X_AMZ_METADATA_DIRECTIVE=COPY
> 2013-12-02 08:18:59.196771 7f5308ff1700 10 x>> x-amz-acl:public-read
> 2013-12-02 08:18:59.196772 7f5308ff1700 10 x>>
> x-amz-copy-source:/testbucket/testfile.xml
> 2013-12-02 08:18:59.196773 7f5308ff1700 10 x>> x-amz-metadata-directive:COPY
> 2013-12-02 08:18:59.196786 7f5308ff1700 10
> s->object=/testbucket/new_testfile.ini s->bucket=testbucket
> 2013-12-02 08:18:59.196792 7f5308ff1700  2 req 237:0.000141:s3:PUT
> /testbucket/new_testfile.ini::getting op
> 2013-12-02 08:18:59.196797 7f5308ff1700  2 req 237:0.000146:s3:PUT
> /testbucket/new_testfile.ini:copy_obj:authorizing
> 2013-12-02 08:18:59.200648 7f5308ff1700 10 get_canon_resource():
> dest=/testbucket/new_testfile.ini
> 2013-12-02 08:18:59.200661 7f5308ff1700 10 auth_hdr:
> PUT
> 1B2M2Y8AsgTpgAmY7PhCfg==
> application/octet-stream
> Mon, 02 Dec 2013 07:18:55 GMT
> x-amz-acl:public-read
> x-amz-copy-source:/testbucket/testfile.xml
> x-amz-metadata-directive:COPY
> /testbucket/new_testfile.ini
> 2013-12-02 08:18:59.200717 7f5308ff1700  2 req 237:0.004066:s3:PUT
> /testbucket/new_testfile.ini:copy_obj:reading permissions
> 2013-12-02 08:18:59.203330 7f5308ff1700  2 req 237:0.006679:s3:PUT
> /testbucket/new_testfile.ini:copy_obj:verifying op permissions
> 2013-12-02 08:18:59.207627 7f5308ff1700 10 manifest: total_size = 0
> 2013-12-02 08:18:59.207649 7f5308ff1700  5 Searching permissions for
> uid=0 mask=1
> 2013-12-02 08:18:59.207652 7f5308ff1700  5 Found permission: 15
> 2013-12-02 08:18:59.207654 7f5308ff1700 10  uid=0 requested perm
> (type)=1, policy perm=1, user_perm_mask=15, acl perm=1
> 2013-12-02 08:18:59.207669 7f5308ff1700  5 Searching permissions for
> uid=0 mask=2
> 2013-12-02 08:18:59.207670 7f5308ff1700  5 Found permission: 15
> 2013-12-02 08:18:59.207671 7f5308ff1700 10  uid=0 requested perm
> (type)=2, policy perm=2, user_perm_mask=15, acl perm=2
> 2013-12-02 08:18:59.207681 7f5308ff1700  2 req 237:0.011030:s3:PUT
> /testbucket/new_testfile.ini:copy_obj:verifying op params
> 2013-12-02 08:18:59.207686 7f5308ff1700  2 req 237:0.011035:s3:PUT
> /testbucket/new_testfile.ini:copy_obj:executing
> 2013-12-02 08:18:59.207699 7f5308ff1700 10 x>> x-amz-acl:public-read
> 2013-12-02 08:18:59.207704 7f5308ff1700 10 x>>
> x-amz-copy-source:/testbucket/testfile.xml
> 2013-12-02 08:18:59.207709 7f5308ff1700 10 x>> x-amz-metadata-directive:COPY
> 2013-12-02 08:18:59.207759 7f5308ff1700  5 Copy object
> testbucket(@.rgw.buckets[406250.1]):testfile.ini =>
> testbucket(@.rgw.buckets[406250.1]):new_testfile.ini
> 2013-12-02 08:18:59.208903 7f5308ff1700 -1 *** Caught signal
> (Segmentation fault) **
>  in thread 7f5308ff1700
>
>
> --
> Regards
> Dominik



-- 
Pozdrawiam
Dominik
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

Hi,
I have strange problem.
Obj copy (0 size) killing radosgw.

Head for this file:
Content-Type: application/octet-stream
Server: Apache/2.2.22 (Ubuntu)
ETag: "d41d8cd98f00b204e9800998ecf8427e-0"
Last-Modified: 2013-12-01T10:37:15Z

rgw log.
2013-12-02 08:18:59.196651 7f5308ff1700  1 == starting new request
req=0x2be6fa0 =
2013-12-02 08:18:59.196709 7f5308ff1700  2 req 237:0.58initializing
2013-12-02 08:18:59.196752 7f5308ff1700 10 meta>> HTTP_X_AMZ_ACL=public-read
2013-12-02 08:18:59.196760 7f5308ff1700 10 meta>>
HTTP_X_AMZ_COPY_SOURCE=/testbucket/testfile.xml
2013-12-02 08:18:59.196766 7f5308ff1700 10 meta>>
HTTP_X_AMZ_METADATA_DIRECTIVE=COPY
2013-12-02 08:18:59.196771 7f5308ff1700 10 x>> x-amz-acl:public-read
2013-12-02 08:18:59.196772 7f5308ff1700 10 x>>
x-amz-copy-source:/testbucket/testfile.xml
2013-12-02 08:18:59.196773 7f5308ff1700 10 x>> x-amz-metadata-directive:COPY
2013-12-02 08:18:59.196786 7f5308ff1700 10
s->object=/testbucket/new_testfile.ini s->bucket=testbucket
2013-12-02 08:18:59.196792 7f5308ff1700  2 req 237:0.000141:s3:PUT
/testbucket/new_testfile.ini::getting op
2013-12-02 08:18:59.196797 7f5308ff1700  2 req 237:0.000146:s3:PUT
/testbucket/new_testfile.ini:copy_obj:authorizing
2013-12-02 08:18:59.200648 7f5308ff1700 10 get_canon_resource():
dest=/testbucket/new_testfile.ini
2013-12-02 08:18:59.200661 7f5308ff1700 10 auth_hdr:
PUT
1B2M2Y8AsgTpgAmY7PhCfg==
application/octet-stream
Mon, 02 Dec 2013 07:18:55 GMT
x-amz-acl:public-read
x-amz-copy-source:/testbucket/testfile.xml
x-amz-metadata-directive:COPY
/testbucket/new_testfile.ini
2013-12-02 08:18:59.200717 7f5308ff1700  2 req 237:0.004066:s3:PUT
/testbucket/new_testfile.ini:copy_obj:reading permissions
2013-12-02 08:18:59.203330 7f5308ff1700  2 req 237:0.006679:s3:PUT
/testbucket/new_testfile.ini:copy_obj:verifying op permissions
2013-12-02 08:18:59.207627 7f5308ff1700 10 manifest: total_size = 0
2013-12-02 08:18:59.207649 7f5308ff1700  5 Searching permissions for
uid=0 mask=1
2013-12-02 08:18:59.207652 7f5308ff1700  5 Found permission: 15
2013-12-02 08:18:59.207654 7f5308ff1700 10  uid=0 requested perm
(type)=1, policy perm=1, user_perm_mask=15, acl perm=1
2013-12-02 08:18:59.207669 7f5308ff1700  5 Searching permissions for
uid=0 mask=2
2013-12-02 08:18:59.207670 7f5308ff1700  5 Found permission: 15
2013-12-02 08:18:59.207671 7f5308ff1700 10  uid=0 requested perm
(type)=2, policy perm=2, user_perm_mask=15, acl perm=2
2013-12-02 08:18:59.207681 7f5308ff1700  2 req 237:0.011030:s3:PUT
/testbucket/new_testfile.ini:copy_obj:verifying op params
2013-12-02 08:18:59.207686 7f5308ff1700  2 req 237:0.011035:s3:PUT
/testbucket/new_testfile.ini:copy_obj:executing
2013-12-02 08:18:59.207699 7f5308ff1700 10 x>> x-amz-acl:public-read
2013-12-02 08:18:59.207704 7f5308ff1700 10 x>>
x-amz-copy-source:/testbucket/testfile.xml
2013-12-02 08:18:59.207709 7f5308ff1700 10 x>> x-amz-metadata-directive:COPY
2013-12-02 08:18:59.207759 7f5308ff1700  5 Copy object
testbucket(@.rgw.buckets[406250.1]):testfile.ini =>
testbucket(@.rgw.buckets[406250.1]):new_testfile.ini
2013-12-02 08:18:59.208903 7f5308ff1700 -1 *** Caught signal
(Segmentation fault) **
 in thread 7f5308ff1700


-- 
Regards
Dominik
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com