Re: [ceph-users] mds damage detected - Jewel

[John Spray]

On Thu, Sep 15, 2016 at 10:30 PM, Jim Kilborn  wrote:
> I have a replicated cache pool and metadata pool which reside on ssd drives, 
> with a size of 2, backed by a erasure coded data pool
> The cephfs filesystem was in a healthy state. I pulled an SSD drive, to 
> perform an exercise in osd failure.
>
> The cluster recognized the ssd failure, and replicated back to a healthy 
> state, but I got a message saying the mds0 Metadata damage detected.
>
>
>cluster 62ed97d6-adf4-12e4-8fd5-3d9701b22b86
>  health HEALTH_ERR
> mds0: Metadata damage detected
> mds0: Client master01.div18.swri.org failing to respond to cache 
> pressure
>  monmap e2: 3 mons at 
> {ceph01=192.168.19.241:6789/0,ceph02=192.168.19.242:6789/0,ceph03=192.168.19.243:6789/0}
> election epoch 24, quorum 0,1,2 
> ceph01,darkjedi-ceph02,darkjedi-ceph03
>   fsmap e25: 1/1/1 up {0=-ceph04=up:active}, 1 up:standby
>  osdmap e1327: 20 osds: 20 up, 20 in
> flags sortbitwise
>   pgmap v11630: 1536 pgs, 3 pools, 100896 MB data, 442 kobjects
> 201 GB used, 62915 GB / 63116 GB avail
> 1536 active+clean
>
> In the mds logs of the active mds, I see the following:
>
> 7fad0c4b2700  0 -- 192.168.19.244:6821/1 >> 192.168.19.243:6805/5090 
> pipe(0x7fad25885400 sd=56 :33513 s=1 pgs=0 cs=0 l=1 c=0x7fad2585f980).fault
> 7fad14add700  0 mds.beacon.darkjedi-ceph04 handle_mds_beacon no longer laggy
> 7fad101d3700  0 mds.0.cache.dir(1016c08) _fetched missing object for [dir 
> 1016c08 /usr/ [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741952 f() n() 
> hs=0+0,ss=0+0 | waiter=1 authpin=1 0x7fad25ced500]
> 7fad101d3700 -1 log_channel(cluster) log [ERR] : dir 1016c08 object 
> missing on disk; some files may be lost
> 7fad0f9d2700  0 -- 192.168.19.244:6821/1 >> 192.168.19.242:6800/3746 
> pipe(0x7fad25a4e800 sd=42 :0 s=1 pgs=0 cs=0 l=1 c=0x7fad25bd5180).fault
> 7fad14add700 -1 log_channel(cluster) log [ERR] : unmatched fragstat size on 
> single dirfrag 1016c08, inode has f(v0 m2016-09-14 14:00:36.654244 
> 13=1+12), dirfrag has f(v0 m2016-09-14 14:00:36.654244 1=0+1)
> 7fad14add700 -1 log_channel(cluster) log [ERR] : unmatched rstat rbytes on 
> single dirfrag 1016c08, inode has n(v77 rc2016-09-14 14:00:36.654244 
> b1533163206 48173=43133+5040), dirfrag has n(v77 rc2016-09-14 14:00:36.654244 
> 1=0+1)
> 7fad101d3700 -1 log_channel(cluster) log [ERR] : unmatched rstat on 
> 1016c08, inode has n(v78 rc2016-09-14 14:00:36.656244 2=0+2), dirfrags 
> have n(v0 rc2016-09-14 14:00:36.656244 3=0+3)
>
> I’m not sure why the metadata got damaged, since its being replicated, but I 
> want to fix the issue, and test again. However, I cant figure out the steps 
> to repair the metadata.

Losing an object like that is almost certainly a sign that you've hit
a bug -- probably an OSD bug if it was the OSDs being disrupted while
the MDS daemons continued to run.

The subsequent "unmatched fragstat" etc messages are probably a red
herring where the stats are only bad because the object is missing,
not because of some other issue (http://tracker.ceph.com/issues/17284)

> I saw something about running a damage ls, but I can’t seem to find a more 
> detailed repair document. Any pointers to get the metadata fixed? Seems both 
> my mds daemons are running correctly, but that error bothers me. Shouldn’t 
> happen I think.

You can get the detail on what's damaged with "ceph tell mds.
damage ls" -- this spits out JSON that you may well want to parse with
a tiny python script.

>
> I tried the following command, but it doesn’t understand it….
> ceph --admin-daemon /var/run/ceph/ceph-mds. ceph03.asok damage ls
>
>
> I then rebooted all 4 ceph servers simultaneously (another stress test), and 
> the ceph cluster came back up healthy, and the mds damaged status has been 
> cleared!!  I  then replaced the ssd, put it back into service, and let the 
> backfill complete. The cluster was fully healthy. I pulled another ssd, and 
> repeated this process, yet I never got the damaged mds messages. Was this 
> just a random metadata damage due to yanking a drive out? Is there any 
> lingering affects of the metadata that I need to address?

The MDS damage table is an ephemeral structure, so when you reboot it
will forget about the damage.  I would expect that doing a "ls -R" on
your filsystem will cause the damage to be detected again as it
traverses the filesystem, although if it doesn't then that would be a
sign that the "missing object" was actually a bug failing to find it
one time, rather than a bug where the object has really been lost.

John

>
>
> -  Jim
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-

Re: [ceph-users] CephFS: Upper limit for number of files in adirectory?

[John Spray]

On Thu, Sep 15, 2016 at 4:19 PM, Burkhard Linke
 wrote:
> Hi,
>
>
> On 09/15/2016 12:00 PM, John Spray wrote:
>>
>> On Thu, Sep 15, 2016 at 2:20 PM, Burkhard Linke
>>  wrote:
>>>
>>> Hi,
>>>
>>> does CephFS impose an upper limit on the number of files in a directory?
>>>
>>>
>>> We currently have one directory with a large number of subdirectories:
>>>
>>> $ ls | wc -l
>>> 158141
>>>
>>> Creating a new subdirectory fails:
>>>
>>> $ touch foo
>>> touch: cannot touch 'foo': No space left on device
>
> thansk for the fast reply.
>>
>> This limit was added recently: it's a limit on the size of a directory
>> fragment.
>>
>> Previously folks were hitting nasty OSD issues with very large
>> directory fragments, so we added this limit to give a clean failure
>> instead.
>
> I remember seeing a thread on the devel mailing list about this issue.
>>
>>
>> Directory fragmentation (mds_bal_frag setting) is turned off by
>> default in Jewel: I was planning to get this activated by default in
>> Kraken, but haven't quite got there yet.  Once fragmentation is
>> enabled you should find that the threshold for splitting dirfrags is
>> hit well before you hit the safety limit that gives you ENOSPC.
>
> Does enabling directory fragmentation require a MDS restart? And are
> directories processed at restart or on demand during the first access? Are
> there known problems with fragmentation?

You could change it at runtime using `injectargs` as with other configs.

Directories are considered for fragmentation when they are touched by
a client operation, so you won't see any fragmentation happen until
the client does something.

I would recommend that you don't enable fragmenation unless you have a
100% need for an unbounded number of files in one directory.  I just
tried a test run with frags enabled and saw a new failure!

John

>>
>> Note that if you set mds_bal_frag then you also need to use the "ceph
>> fs set  allow_dirfrags true" (that command from memory so check
>> the help if it's wrong), or the MDSs will ignore the setting.
>
> So its allowing fragmentation first and changing the MDS configuration
> afterwards.
>
>
> Regards,
> Burkhard
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Erasure coding general information Openstack+kvm virtual machine block storage

[Wes Dillingham]

Erick,

You can use erasure coding but it has to be fronted by a replicated cache
tier, or so states the documentation, I have never set up this
configuration, and always opt to use RBD directly on replicated pools.

https://access.redhat.com/documentation/en/red-hat-ceph-storage/1.3/paged/storage-strategies/chapter-31-erasure-coded-pools-and-cache-tiering

On Fri, Sep 16, 2016 at 1:15 AM, Josh Durgin  wrote:

> On 09/16/2016 09:46 AM, Erick Perez - Quadrian Enterprises wrote:
>
>> Can someone point me to a thread or site that uses ceph+erasure coding
>> to serve block storage for Virtual Machines running with Openstack+KVM?
>> All references that I found are using erasure coding for cold data or
>> *not* VM block access.
>>
>
> Erasure coding is not supported by RBD currently, since EC pools only
> support append operations. There's work in progress to make it
> possible, by allowing overwrites for EC pools, but it won't be usable
> until at earliest Luminous [0].
>
> Josh
>
> [0] http://tracker.ceph.com/issues/14031
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Respectfully,

Wes Dillingham
wes_dilling...@harvard.edu
Research Computing | Infrastructure Engineer
Harvard University | 38 Oxford Street, Cambridge, Ma 02138 | Room 210
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mds damage detected - Jewel

[Jim Kilborn]

John,



thanks for the tips.



I ran a recursive long listing of the cephfs volume, and didn’t receive any 
errors. So I guess it wasn’t serious.



I also tried running the following from



ceph tell mds.0 damage ls

2016-09-16 07:11:36.824330 7fc2ff00e700  0 client.224234 ms_handle_reset on 
192.168.19.243:6804/3448

Error EPERM: problem getting command descriptions from mds.0



I’m guessing this output means something didn’t work. Any other suggestions 
with this command?



Thanks,

Jim





Sent from Mail for Windows 10



From: John Spray
Sent: Friday, September 16, 2016 2:37 AM
To: Jim Kilborn
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] mds damage detected - Jewel



On Thu, Sep 15, 2016 at 10:30 PM, Jim Kilborn  wrote:
> I have a replicated cache pool and metadata pool which reside on ssd drives, 
> with a size of 2, backed by a erasure coded data pool
> The cephfs filesystem was in a healthy state. I pulled an SSD drive, to 
> perform an exercise in osd failure.
>
> The cluster recognized the ssd failure, and replicated back to a healthy 
> state, but I got a message saying the mds0 Metadata damage detected.
>
>
>cluster 62ed97d6-adf4-12e4-8fd5-3d9701b22b86
>  health HEALTH_ERR
> mds0: Metadata damage detected
> mds0: Client master01.div18.swri.org failing to respond to cache 
> pressure
>  monmap e2: 3 mons at 
> {ceph01=192.168.19.241:6789/0,ceph02=192.168.19.242:6789/0,ceph03=192.168.19.243:6789/0}
> election epoch 24, quorum 0,1,2 
> ceph01,darkjedi-ceph02,darkjedi-ceph03
>   fsmap e25: 1/1/1 up {0=-ceph04=up:active}, 1 up:standby
>  osdmap e1327: 20 osds: 20 up, 20 in
> flags sortbitwise
>   pgmap v11630: 1536 pgs, 3 pools, 100896 MB data, 442 kobjects
> 201 GB used, 62915 GB / 63116 GB avail
> 1536 active+clean
>
> In the mds logs of the active mds, I see the following:
>
> 7fad0c4b2700  0 -- 192.168.19.244:6821/1 >> 192.168.19.243:6805/5090 
> pipe(0x7fad25885400 sd=56 :33513 s=1 pgs=0 cs=0 l=1 c=0x7fad2585f980).fault
> 7fad14add700  0 mds.beacon.darkjedi-ceph04 handle_mds_beacon no longer laggy
> 7fad101d3700  0 mds.0.cache.dir(1016c08) _fetched missing object for [dir 
> 1016c08 /usr/ [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741952 f() n() 
> hs=0+0,ss=0+0 | waiter=1 authpin=1 0x7fad25ced500]
> 7fad101d3700 -1 log_channel(cluster) log [ERR] : dir 1016c08 object 
> missing on disk; some files may be lost
> 7fad0f9d2700  0 -- 192.168.19.244:6821/1 >> 192.168.19.242:6800/3746 
> pipe(0x7fad25a4e800 sd=42 :0 s=1 pgs=0 cs=0 l=1 c=0x7fad25bd5180).fault
> 7fad14add700 -1 log_channel(cluster) log [ERR] : unmatched fragstat size on 
> single dirfrag 1016c08, inode has f(v0 m2016-09-14 14:00:36.654244 
> 13=1+12), dirfrag has f(v0 m2016-09-14 14:00:36.654244 1=0+1)
> 7fad14add700 -1 log_channel(cluster) log [ERR] : unmatched rstat rbytes on 
> single dirfrag 1016c08, inode has n(v77 rc2016-09-14 14:00:36.654244 
> b1533163206 48173=43133+5040), dirfrag has n(v77 rc2016-09-14 14:00:36.654244 
> 1=0+1)
> 7fad101d3700 -1 log_channel(cluster) log [ERR] : unmatched rstat on 
> 1016c08, inode has n(v78 rc2016-09-14 14:00:36.656244 2=0+2), dirfrags 
> have n(v0 rc2016-09-14 14:00:36.656244 3=0+3)
>
> I’m not sure why the metadata got damaged, since its being replicated, but I 
> want to fix the issue, and test again. However, I cant figure out the steps 
> to repair the metadata.

Losing an object like that is almost certainly a sign that you've hit
a bug -- probably an OSD bug if it was the OSDs being disrupted while
the MDS daemons continued to run.

The subsequent "unmatched fragstat" etc messages are probably a red
herring where the stats are only bad because the object is missing,
not because of some other issue (http://tracker.ceph.com/issues/17284)

> I saw something about running a damage ls, but I can’t seem to find a more 
> detailed repair document. Any pointers to get the metadata fixed? Seems both 
> my mds daemons are running correctly, but that error bothers me. Shouldn’t 
> happen I think.

You can get the detail on what's damaged with "ceph tell mds.
damage ls" -- this spits out JSON that you may well want to parse with
a tiny python script.

>
> I tried the following command, but it doesn’t understand it….
> ceph --admin-daemon /var/run/ceph/ceph-mds. ceph03.asok damage ls
>
>
> I then rebooted all 4 ceph servers simultaneously (another stress test), and 
> the ceph cluster came back up healthy, and the mds damaged status has been 
> cleared!!  I  then replaced the ssd, put it back into service, and let the 
> backfill complete. The cluster was fully healthy. I pulled another ssd, and 
> repeated this process, yet I never got the damaged mds messages. Was this 
> just a random metadata damage due to yanking 

Re: [ceph-users] High CPU load with radosgw instances

[lewis.geo...@innoscale.net]

Hi Yehuda,
 Thank you for the idea. I will try to test that and see if it helps.
  
 If that is the case, would that be considered a bug with radosgw? I ask 
because, that version of curl seems to be what is currently standard on 
RHEL/CentOS 7 (fully updated). I will have to either compile it or search 
3rd-party repos for newer version, which is not usually something that is 
great. 
  
 Have a good day,
  
 Lewis George
  
  


 From: "Yehuda Sadeh-Weinraub" 
Sent: Thursday, September 15, 2016 10:42 PM
To: lewis.geo...@innoscale.net
Cc: "ceph-users@lists.ceph.com" 
Subject: Re: [ceph-users] High CPU load with radosgw instances   
On Thu, Sep 15, 2016 at 4:53 PM, lewis.geo...@innoscale.net
 wrote:
> Hi,
> So, maybe someone has an idea of where to go on this.
>
> I have just setup 2 rgw instances in a multisite setup. They are working
> nicely. I have add a couple of test buckets and some files to make sure 
it
> works is all. The status shows both are caught up. Nobody else is 
accessing
> or using them.
>
> However, the CPU load on both hosts is sitting at like 3.00, with the
> radosgw process taking up 99% CPU constantly. I do not see anything in 
the
> logs happening at all.
>
> Any thoughts or direction?
>

We've seen that happening when running on a system with older version
of libcurl (e.g., 7.29). If that's the case upgrading to a newer
version should fix it for you.

Yehuda

> Have a good day,
>
> Lewis George
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High CPU load with radosgw instances

[lewis.geo...@innoscale.net]

Hi Yehuda,
 Well, again, thank you! 
  
 I was able to get a package built from the latest curl release, and after 
upgrading on my radosgw hosts, the load is no longer running high. The load 
is just sitting at almost nothing and I only see the radosgw process using 
CPU when it is actually doing something now.
  
 So, I am still curious if this would be considered a bug or not, since the 
curl version from the base CentOS repo seems to have an issue. 
  
 Have a good day,
  
 Lewis George
  
  
  


 From: "lewis.geo...@innoscale.net" 
Sent: Friday, September 16, 2016 7:28 AM
To: "Yehuda Sadeh-Weinraub" 
Cc: "ceph-users@lists.ceph.com" 
Subject: Re: [ceph-users] High CPU load with radosgw instances   
 Hi Yehuda,
   Thank you for the idea. I will try to test that and see if it helps.

   If that is the case, would that be considered a bug with radosgw? I ask 
because, that version of curl seems to be what is currently standard on 
RHEL/CentOS 7 (fully updated). I will have to either compile it or search 
3rd-party repos for newer version, which is not usually something that is 
great. 

   Have a good day,

   Lewis George




   From: "Yehuda Sadeh-Weinraub" 
Sent: Thursday, September 15, 2016 10:42 PM
To: lewis.geo...@innoscale.net
Cc: "ceph-users@lists.ceph.com" 
Subject: Re: [ceph-users] High CPU load with radosgw instances
 On Thu, Sep 15, 2016 at 4:53 PM, lewis.geo...@innoscale.net
 wrote:
> Hi,
> So, maybe someone has an idea of where to go on this.
>
> I have just setup 2 rgw instances in a multisite setup. They are working
> nicely. I have add a couple of test buckets and some files to make sure 
it
> works is all. The status shows both are caught up. Nobody else is 
accessing
> or using them.
>
> However, the CPU load on both hosts is sitting at like 3.00, with the
> radosgw process taking up 99% CPU constantly. I do not see anything in 
the
> logs happening at all.
>
> Any thoughts or direction?
>

We've seen that happening when running on a system with older version
of libcurl (e.g., 7.29). If that's the case upgrading to a newer
version should fix it for you.

Yehuda

> Have a good day,
>
> Lewis George
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High CPU load with radosgw instances

[Ken Dreyer]

Hi Lewis,

This sounds a lot like https://bugzilla.redhat.com/1347904 , currently
slated for the upcoming RHEL 7.3 (and CentOS 7.3).

There's an SRPM in that BZ that you can rebuild and test out. This
method won't require you to keep chasing upstream curl versions
forever (curl has a lot of CVEs).

Mind testing that out and reporting back?

- Ken


On Fri, Sep 16, 2016 at 11:06 AM, lewis.geo...@innoscale.net
 wrote:
> Hi Yehuda,
> Well, again, thank you!
>
> I was able to get a package built from the latest curl release, and after
> upgrading on my radosgw hosts, the load is no longer running high. The load
> is just sitting at almost nothing and I only see the radosgw process using
> CPU when it is actually doing something now.
>
> So, I am still curious if this would be considered a bug or not, since the
> curl version from the base CentOS repo seems to have an issue.
>
> Have a good day,
>
> Lewis George
>
>
>
> 
> From: "lewis.geo...@innoscale.net" 
> Sent: Friday, September 16, 2016 7:28 AM
> To: "Yehuda Sadeh-Weinraub" 
>
> Cc: "ceph-users@lists.ceph.com" 
> Subject: Re: [ceph-users] High CPU load with radosgw instances
>
> Hi Yehuda,
> Thank you for the idea. I will try to test that and see if it helps.
>
> If that is the case, would that be considered a bug with radosgw? I ask
> because, that version of curl seems to be what is currently standard on
> RHEL/CentOS 7 (fully updated). I will have to either compile it or search
> 3rd-party repos for newer version, which is not usually something that is
> great.
>
> Have a good day,
>
> Lewis George
>
>
> 
> From: "Yehuda Sadeh-Weinraub" 
> Sent: Thursday, September 15, 2016 10:42 PM
> To: lewis.geo...@innoscale.net
> Cc: "ceph-users@lists.ceph.com" 
> Subject: Re: [ceph-users] High CPU load with radosgw instances
>
> On Thu, Sep 15, 2016 at 4:53 PM, lewis.geo...@innoscale.net
>  wrote:
>> Hi,
>> So, maybe someone has an idea of where to go on this.
>>
>> I have just setup 2 rgw instances in a multisite setup. They are working
>> nicely. I have add a couple of test buckets and some files to make sure it
>> works is all. The status shows both are caught up. Nobody else is
>> accessing
>> or using them.
>>
>> However, the CPU load on both hosts is sitting at like 3.00, with the
>> radosgw process taking up 99% CPU constantly. I do not see anything in the
>> logs happening at all.
>>
>> Any thoughts or direction?
>>
>
> We've seen that happening when running on a system with older version
> of libcurl (e.g., 7.29). If that's the case upgrading to a newer
> version should fix it for you.
>
> Yehuda
>
>
>> Have a good day,
>>
>> Lewis George
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High CPU load with radosgw instances

[Casey Bodley]

In the meantime, we've made changes to radosgw so that it can detect and 
work around this libcurl bug. You can track the progress of this 
workaround (currently in master and pending backport to jewel) at 
http://tracker.ceph.com/issues/16695.


Casey


On 09/16/2016 01:38 PM, Ken Dreyer wrote:

Hi Lewis,

This sounds a lot like https://bugzilla.redhat.com/1347904 , currently
slated for the upcoming RHEL 7.3 (and CentOS 7.3).

There's an SRPM in that BZ that you can rebuild and test out. This
method won't require you to keep chasing upstream curl versions
forever (curl has a lot of CVEs).

Mind testing that out and reporting back?

- Ken


On Fri, Sep 16, 2016 at 11:06 AM, lewis.geo...@innoscale.net
 wrote:

Hi Yehuda,
Well, again, thank you!

I was able to get a package built from the latest curl release, and after
upgrading on my radosgw hosts, the load is no longer running high. The load
is just sitting at almost nothing and I only see the radosgw process using
CPU when it is actually doing something now.

So, I am still curious if this would be considered a bug or not, since the
curl version from the base CentOS repo seems to have an issue.

Have a good day,

Lewis George




From: "lewis.geo...@innoscale.net" 
Sent: Friday, September 16, 2016 7:28 AM
To: "Yehuda Sadeh-Weinraub" 

Cc: "ceph-users@lists.ceph.com" 
Subject: Re: [ceph-users] High CPU load with radosgw instances

Hi Yehuda,
Thank you for the idea. I will try to test that and see if it helps.

If that is the case, would that be considered a bug with radosgw? I ask
because, that version of curl seems to be what is currently standard on
RHEL/CentOS 7 (fully updated). I will have to either compile it or search
3rd-party repos for newer version, which is not usually something that is
great.

Have a good day,

Lewis George



From: "Yehuda Sadeh-Weinraub" 
Sent: Thursday, September 15, 2016 10:42 PM
To: lewis.geo...@innoscale.net
Cc: "ceph-users@lists.ceph.com" 
Subject: Re: [ceph-users] High CPU load with radosgw instances

On Thu, Sep 15, 2016 at 4:53 PM, lewis.geo...@innoscale.net
 wrote:

Hi,
So, maybe someone has an idea of where to go on this.

I have just setup 2 rgw instances in a multisite setup. They are working
nicely. I have add a couple of test buckets and some files to make sure it
works is all. The status shows both are caught up. Nobody else is
accessing
or using them.

However, the CPU load on both hosts is sitting at like 3.00, with the
radosgw process taking up 99% CPU constantly. I do not see anything in the
logs happening at all.

Any thoughts or direction?


We've seen that happening when running on a system with older version
of libcurl (e.g., 7.29). If that's the case upgrading to a newer
version should fix it for you.

Yehuda



Have a good day,

Lewis George


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High CPU load with radosgw instances

[lewis.geo...@innoscale.net]

Hi Casey,
 Thank you for the follow-up. I had just found that one while searching in 
the tracker. I probably should have done that first(though I guess I was 
hoping/assuming google would have brought it up(but of course it didn't). 
  
 Anyway, it is working nicely for me now with the newer version of curl, so 
that is what really matters. Still, kind of a pain having to go build the 
new package. 
  
 Have a good day,
  
 Lewis George
  
  
  


 From: "Casey Bodley" 
Sent: Friday, September 16, 2016 11:02 AM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] High CPU load with radosgw instances   
In the meantime, we've made changes to radosgw so that it can detect and
work around this libcurl bug. You can track the progress of this
workaround (currently in master and pending backport to jewel) at
http://tracker.ceph.com/issues/16695.

Casey

On 09/16/2016 01:38 PM, Ken Dreyer wrote:
> Hi Lewis,
>
> This sounds a lot like https://bugzilla.redhat.com/1347904 , currently
> slated for the upcoming RHEL 7.3 (and CentOS 7.3).
>
> There's an SRPM in that BZ that you can rebuild and test out. This
> method won't require you to keep chasing upstream curl versions
> forever (curl has a lot of CVEs).
>
> Mind testing that out and reporting back?
>
> - Ken
>
>
> On Fri, Sep 16, 2016 at 11:06 AM, lewis.geo...@innoscale.net
>  wrote:
>> Hi Yehuda,
>> Well, again, thank you!
>>
>> I was able to get a package built from the latest curl release, and 
after
>> upgrading on my radosgw hosts, the load is no longer running high. The 
load
>> is just sitting at almost nothing and I only see the radosgw process 
using
>> CPU when it is actually doing something now.
>>
>> So, I am still curious if this would be considered a bug or not, since 
the
>> curl version from the base CentOS repo seems to have an issue.
>>
>> Have a good day,
>>
>> Lewis George
>>
>>
>>
>> 
>> From: "lewis.geo...@innoscale.net" 
>> Sent: Friday, September 16, 2016 7:28 AM
>> To: "Yehuda Sadeh-Weinraub" 
>>
>> Cc: "ceph-users@lists.ceph.com" 
>> Subject: Re: [ceph-users] High CPU load with radosgw instances
>>
>> Hi Yehuda,
>> Thank you for the idea. I will try to test that and see if it helps.
>>
>> If that is the case, would that be considered a bug with radosgw? I ask
>> because, that version of curl seems to be what is currently standard on
>> RHEL/CentOS 7 (fully updated). I will have to either compile it or 
search
>> 3rd-party repos for newer version, which is not usually something that 
is
>> great.
>>
>> Have a good day,
>>
>> Lewis George
>>
>>
>> 
>> From: "Yehuda Sadeh-Weinraub" 
>> Sent: Thursday, September 15, 2016 10:42 PM
>> To: lewis.geo...@innoscale.net
>> Cc: "ceph-users@lists.ceph.com" 
>> Subject: Re: [ceph-users] High CPU load with radosgw instances
>>
>> On Thu, Sep 15, 2016 at 4:53 PM, lewis.geo...@innoscale.net
>>  wrote:
>>> Hi,
>>> So, maybe someone has an idea of where to go on this.
>>>
>>> I have just setup 2 rgw instances in a multisite setup. They are 
working
>>> nicely. I have add a couple of test buckets and some files to make sure 
it
>>> works is all. The status shows both are caught up. Nobody else is
>>> accessing
>>> or using them.
>>>
>>> However, the CPU load on both hosts is sitting at like 3.00, with the
>>> radosgw process taking up 99% CPU constantly. I do not see anything in 
the
>>> logs happening at all.
>>>
>>> Any thoughts or direction?
>>>
>> We've seen that happening when running on a system with older version
>> of libcurl (e.g., 7.29). If that's the case upgrading to a newer
>> version should fix it for you.
>>
>> Yehuda
>>
>>
>>> Have a good day,
>>>
>>> Lewis George
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Erasure coding general information Openstack+kvm virtual machine block storage

[Erick Perez - Quadrian Enterprises]

Thanks Wes and Josh for your answers. So, for more production-like
environments and more tested procedures in case of failures the default
replication seems to be the way to go. Perhaps in next release we will add
a storage node with EC.

Thanks,


On Fri, Sep 16, 2016 at 7:25 AM, Wes Dillingham 
wrote:

> Erick,
>
> You can use erasure coding but it has to be fronted by a replicated cache
> tier, or so states the documentation, I have never set up this
> configuration, and always opt to use RBD directly on replicated pools.
>
> https://access.redhat.com/documentation/en/red-hat-ceph-
> storage/1.3/paged/storage-strategies/chapter-31-erasure-
> coded-pools-and-cache-tiering
>
> On Fri, Sep 16, 2016 at 1:15 AM, Josh Durgin  wrote:
>
>> On 09/16/2016 09:46 AM, Erick Perez - Quadrian Enterprises wrote:
>>
>>> Can someone point me to a thread or site that uses ceph+erasure coding
>>> to serve block storage for Virtual Machines running with Openstack+KVM?
>>> All references that I found are using erasure coding for cold data or
>>> *not* VM block access.
>>>
>>
>> Erasure coding is not supported by RBD currently, since EC pools only
>> support append operations. There's work in progress to make it
>> possible, by allowing overwrites for EC pools, but it won't be usable
>> until at earliest Luminous [0].
>>
>> Josh
>>
>> [0] http://tracker.ceph.com/issues/14031
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
>
> --
> Respectfully,
>
> Wes Dillingham
> wes_dilling...@harvard.edu
> Research Computing | Infrastructure Engineer
> Harvard University | 38 Oxford Street, Cambridge, Ma 02138 | Room 210
>
>


-- 

-
Erick Perez
Soluciones Tacticas Pasivas/Activas de Inteligencia y Analitica de Datos
para Gobiernos
Quadrian Enterprises S.A. - Panama, Republica de Panama
Skype chat: eaperezh
WhatsApp IM: +507-6675-5083
POBOX 0819-12679, Panama
Tel. (507) 391.8174 / (507) 391.8175
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Full OSD halting a cluster - isn't this violating the "no single point of failure" promise?

[Christian Theune]

Hi,

(just in case: this isn’t intended as a rant and I hope it doesn’t get read at 
it. I’m trying to understand what some perspectives towards potential future 
improvements are and I think it would be valuable to have this discoverable in 
the archives)

We’ve had a “good" time recently balancing our growing cluster and did a lot of 
reweighting after a full OSD actually did bite us once.

Apart from paying our dues (tight monitoring, reweighting and generally hedging 
the cluster) I was wondering whether this behaviour is a violation of the “no 
single point of failure” promise: independent of how big your setup grows, a 
single OSD can halt practically everything. Even just stopping the OSD would 
unblock your cluster (assuming that Crush made a particular pathological choice 
and that 1 OSD being extremely off the curve compared to the others) and keep 
going.

I haven’t found much whether this is “it’s the way it is and we don’t see a way 
forward” or whether this behaviour is considered something that could be 
improved in the future and whether there are strategies around already?

From my perspective this is directly related to how well Crush weighting works 
with respect to placing data evenly. (I would expect that in certain situations 
like a single RBD cluster where all objects are identically sized that this 
should be something that Crush can perform well in, but my last weeks tells me 
that isn’t the case. :) )

An especially interesting edge case is if your cluster consists of 2 pools 
where each runs using a completely disjoint set of OSDs: I guess it’s an 
accidental (not intentional) behaviour that the one pool would be affecting the 
other, right?

Thoughts?

Hugs,
Christian

--
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High CPU load with radosgw instances

[Ken Dreyer]

Hi Casey,

That warning message tells users to upgrade to a new version of
libcurl. Telling users to upgrade to a newer version of a base system
package like that sets the user on a trajectory to have to maintain
their own curl packages forever, decreasing the security of their
overall system in the long run. For example ceph.com itself shipped a
newer el6 curl package for a while in "ceph-extras", until it fell of
everyone's radar, no one updated it, and it had many outstanding CVEs
until we finally dropped el6 support altogether.

Would you please remove this last sentence from the RGW log message?

- Ken


On Fri, Sep 16, 2016 at 12:02 PM, Casey Bodley  wrote:
> In the meantime, we've made changes to radosgw so that it can detect and
> work around this libcurl bug. You can track the progress of this workaround
> (currently in master and pending backport to jewel) at
> http://tracker.ceph.com/issues/16695.
>
> Casey
>
>
>
> On 09/16/2016 01:38 PM, Ken Dreyer wrote:
>>
>> Hi Lewis,
>>
>> This sounds a lot like https://bugzilla.redhat.com/1347904 , currently
>> slated for the upcoming RHEL 7.3 (and CentOS 7.3).
>>
>> There's an SRPM in that BZ that you can rebuild and test out. This
>> method won't require you to keep chasing upstream curl versions
>> forever (curl has a lot of CVEs).
>>
>> Mind testing that out and reporting back?
>>
>> - Ken
>>
>>
>> On Fri, Sep 16, 2016 at 11:06 AM, lewis.geo...@innoscale.net
>>  wrote:
>>>
>>> Hi Yehuda,
>>> Well, again, thank you!
>>>
>>> I was able to get a package built from the latest curl release, and after
>>> upgrading on my radosgw hosts, the load is no longer running high. The
>>> load
>>> is just sitting at almost nothing and I only see the radosgw process
>>> using
>>> CPU when it is actually doing something now.
>>>
>>> So, I am still curious if this would be considered a bug or not, since
>>> the
>>> curl version from the base CentOS repo seems to have an issue.
>>>
>>> Have a good day,
>>>
>>> Lewis George
>>>
>>>
>>>
>>> 
>>> From: "lewis.geo...@innoscale.net" 
>>> Sent: Friday, September 16, 2016 7:28 AM
>>> To: "Yehuda Sadeh-Weinraub" 
>>>
>>> Cc: "ceph-users@lists.ceph.com" 
>>> Subject: Re: [ceph-users] High CPU load with radosgw instances
>>>
>>> Hi Yehuda,
>>> Thank you for the idea. I will try to test that and see if it helps.
>>>
>>> If that is the case, would that be considered a bug with radosgw? I ask
>>> because, that version of curl seems to be what is currently standard on
>>> RHEL/CentOS 7 (fully updated). I will have to either compile it or search
>>> 3rd-party repos for newer version, which is not usually something that is
>>> great.
>>>
>>> Have a good day,
>>>
>>> Lewis George
>>>
>>>
>>> 
>>> From: "Yehuda Sadeh-Weinraub" 
>>> Sent: Thursday, September 15, 2016 10:42 PM
>>> To: lewis.geo...@innoscale.net
>>> Cc: "ceph-users@lists.ceph.com" 
>>> Subject: Re: [ceph-users] High CPU load with radosgw instances
>>>
>>> On Thu, Sep 15, 2016 at 4:53 PM, lewis.geo...@innoscale.net
>>>  wrote:

 Hi,
 So, maybe someone has an idea of where to go on this.

 I have just setup 2 rgw instances in a multisite setup. They are working
 nicely. I have add a couple of test buckets and some files to make sure
 it
 works is all. The status shows both are caught up. Nobody else is
 accessing
 or using them.

 However, the CPU load on both hosts is sitting at like 3.00, with the
 radosgw process taking up 99% CPU constantly. I do not see anything in
 the
 logs happening at all.

 Any thoughts or direction?

>>> We've seen that happening when running on a system with older version
>>> of libcurl (e.g., 7.29). If that's the case upgrading to a newer
>>> version should fix it for you.
>>>
>>> Yehuda
>>>
>>>
 Have a good day,

 Lewis George


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High CPU load with radosgw instances

[Ken Dreyer]

On Fri, Sep 16, 2016 at 2:03 PM, Ken Dreyer  wrote:
> Hi Casey,
>
> That warning message tells users to upgrade to a new version of
> libcurl. Telling users to upgrade to a newer version of a base system
> package like that sets the user on a trajectory to have to maintain
> their own curl packages forever, decreasing the security of their
> overall system in the long run. For example ceph.com itself shipped a
> newer el6 curl package for a while in "ceph-extras", until it fell of
> everyone's radar, no one updated it, and it had many outstanding CVEs
> until we finally dropped el6 support altogether.
>

I got the details wrong here - in ceph-extras, it was qemu-kvm on el6
that had a bunch of unfixed security issues, and on Fedora it was
libcurl :)

- Ken
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Segmentation fault in ceph-authtool (FIPS=1)

[Jean Christophe “JC” Martin]

Hi,

I’m trying to run ceph jewel on centos 7.2 with fips mode=1and got a 
segmentation fault running ceph-authtool.
I’m pretty sure that this is a result of the fips mode.
Obviously I would hope that this would work. I will have to try again, but I 
think that firefly does not have this issue.

rpm -qa ceph
ceph-10.2.2-0.el7.x86_64

ceph-authtool -v
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)

kernel : 3.10.0-327.28.3.el7.x86_64

here is the stack:

# ceph-authtool --create-keyring "ceph.mon.keyring" --gen-key -n mon. --cap mon 
'allow *'
creating ceph.mon.keyring
*** Caught signal (Segmentation fault) **
 in thread 7ff3dcf64d00 thread_name:ceph-authtool
 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
 1: (()+0xe2faa) [0x7ff3dd05bfaa]
 2: (()+0xf100) [0x7ff3dc1c9100]
 3: (_FreeSymKey const*+0x55) [0x7ff3dc662155]
 4: (CryptoAESKeyHandler::~CryptoAESKeyHandler()+0x29) [0x7ff3dd0672a9]
 5: (CryptoAES::get_key_handler(ceph::buffer::ptr const&, std::string&)+0x41a) 
[0x7ff3dd0662ba]
 6: (CryptoKey::_set_secret(int, ceph::buffer::ptr const&)+0xec) 
[0x7ff3dd064f6c]
 7: (CryptoKey::create(CephContext*, int)+0x8a) [0x7ff3dd0653fa]
 8: (main()+0xc09) [0x7ff3dd052339]
 9: (__libc_start_main()+0xf5) [0x7ff3dafd2b15]
 10: (()+0xdbb59) [0x7ff3dd054b59]
2016-09-17 00:31:01.236632 7ff3dcf64d00 -1 *** Caught signal (Segmentation 
fault) **
 in thread 7ff3dcf64d00 thread_name:ceph-authtool

 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
 1: (()+0xe2faa) [0x7ff3dd05bfaa]
 2: (()+0xf100) [0x7ff3dc1c9100]
 3: (_FreeSymKey const*+0x55) [0x7ff3dc662155]
 4: (CryptoAESKeyHandler::~CryptoAESKeyHandler()+0x29) [0x7ff3dd0672a9]
 5: (CryptoAES::get_key_handler(ceph::buffer::ptr const&, std::string&)+0x41a) 
[0x7ff3dd0662ba]
 6: (CryptoKey::_set_secret(int, ceph::buffer::ptr const&)+0xec) 
[0x7ff3dd064f6c]
 7: (CryptoKey::create(CephContext*, int)+0x8a) [0x7ff3dd0653fa]
 8: (main()+0xc09) [0x7ff3dd052339]
 9: (__libc_start_main()+0xf5) [0x7ff3dafd2b15]
 10: (()+0xdbb59) [0x7ff3dd054b59]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.

--- begin dump of recent events ---
   -14> 2016-09-17 00:31:01.183767 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command perfcounters_dump hook 0x7ff3e7547980
   -13> 2016-09-17 00:31:01.183781 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command 1 hook 0x7ff3e7547980
   -12> 2016-09-17 00:31:01.183784 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command perf dump hook 0x7ff3e7547980
   -11> 2016-09-17 00:31:01.183789 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command perfcounters_schema hook 0x7ff3e7547980
   -10> 2016-09-17 00:31:01.183791 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command 2 hook 0x7ff3e7547980
-9> 2016-09-17 00:31:01.183792 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command perf schema hook 0x7ff3e7547980
-8> 2016-09-17 00:31:01.183794 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command perf reset hook 0x7ff3e7547980
-7> 2016-09-17 00:31:01.183796 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command config show hook 0x7ff3e7547980
-6> 2016-09-17 00:31:01.183798 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command config set hook 0x7ff3e7547980
-5> 2016-09-17 00:31:01.183802 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command config get hook 0x7ff3e7547980
-4> 2016-09-17 00:31:01.183804 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command config diff hook 0x7ff3e7547980
-3> 2016-09-17 00:31:01.183807 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command log flush hook 0x7ff3e7547980
-2> 2016-09-17 00:31:01.183809 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command log dump hook 0x7ff3e7547980
-1> 2016-09-17 00:31:01.183811 7ff3dcf64d00  5 asok(0x7ff3e7542d50) 
register_command log reopen hook 0x7ff3e7547980
 0> 2016-09-17 00:31:01.236632 7ff3dcf64d00 -1 *** Caught signal 
(Segmentation fault) **
 in thread 7ff3dcf64d00 thread_name:ceph-authtool

 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
 1: (()+0xe2faa) [0x7ff3dd05bfaa]
 2: (()+0xf100) [0x7ff3dc1c9100]
 3: (_FreeSymKey const*+0x55) [0x7ff3dc662155]
 4: (CryptoAESKeyHandler::~CryptoAESKeyHandler()+0x29) [0x7ff3dd0672a9]
 5: (CryptoAES::get_key_handler(ceph::buffer::ptr const&, std::string&)+0x41a) 
[0x7ff3dd0662ba]
 6: (CryptoKey::_set_secret(int, ceph::buffer::ptr const&)+0xec) 
[0x7ff3dd064f6c]
 7: (CryptoKey::create(CephContext*, int)+0x8a) [0x7ff3dd0653fa]
 8: (main()+0xc09) [0x7ff3dd052339]
 9: (__libc_start_main()+0xf5) [0x7ff3dafd2b15]
 10: (()+0xdbb59) [0x7ff3dd054b59]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   

Re: [ceph-users] RADOSGW and LDAP

[Matt Benjamin]

Hi Brian,

This issue is fixed upstream in commit 08d54291435e.  It looks like this did 
not make it to Jewel, we're prioritizing this, and will follow up when this and 
any related LDAP and NFS commits make it there.

Thanks for bringing this to our attention!

Matt

- Original Message -
> From: "Brian Contractor Andrus" 
> To: ceph-users@lists.ceph.com
> Sent: Thursday, September 15, 2016 12:56:29 PM
> Subject: [ceph-users] RADOSGW and LDAP
> 
> 
> 
> All,
> 
> I have been making some progress on troubleshooting this.
> 
> I am seeing that when rgw is configured for LDAP, I am getting an error in my
> slapd log:
> 
> 
> 
> Sep 14 06:56:21 mgmt1 slapd[23696]: conn=1762 op=0 RESULT tag=97 err=2
> text=historical protocol version requested, use LDAPv3 instead
> 
> 
> 
> Am I correct with an interpretation that rgw does not do LDAPv3?
> Is there a way to enable this, or must I allow older versions in my OpenLDAP
> configuration?
> 
> 
> 
> Brian Andrus
> 
> ITACS/Research Computing
> 
> Naval Postgraduate School
> 
> Monterey, California
> 
> voice: 831-656-6238
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com