Re: [ceph-users] cephfs(Kraken 11.2.1), Unable to write more file when one dir more than 100000 files, mds_bal_fragment_size_max = 5000000

ZhengYan, I'm sorry, just a description of some questions. when one dir more than 10 files, I can continue to write it , but I don't find file which wrote in the past. for example: 1. I write 10 files named 512k.file$i 2. I continue to write 1 files named aaa.file$i 3. I

Re: [ceph-users] cephfs(Kraken 11.2.1), Unable to write more file when one dir more than 100000 files, mds_bal_fragment_size_max = 5000000

Its a feature ... http://docs.ceph.com/docs/master/cephfs/dirfrags/ https://www.spinics.net/lists/ceph-users/msg31473.html Marcus Haarmann Von: donglifec...@gmail.com An: "zyan" CC: "ceph-users" Gesendet: Freitag, 8. September 2017 07:30:53

[ceph-users] cephfs(Kraken 11.2.1), Unable to write more file when one dir more than 100000 files, mds_bal_fragment_size_max = 5000000

ZhengYan, I test cephfs(Kraken 11.2.1), I can't write more files when one dir more than 10 files, I have already set up "mds_bal_fragment_size_max = 500". why is this case? Is it a bug? Thanks a lot. donglifec...@gmail.com ___ ceph-users

Re: [ceph-users] Blocked requests

Is it this? https://bugzilla.redhat.com/show_bug.cgi?id=1430588 On Fri, Sep 8, 2017 at 7:01 AM, Matthew Stroud wrote: > After some troubleshooting, the issues appear to be caused by gnocchi using > rados. I’m trying to figure out why. > > > > Thanks, > > Matthew Stroud

Re: [ceph-users] Ceph release cadence

>> Maybe I missed something, but I think Ceph does not support LTS releases for >> 3 years. Yes, you are correct but it averages to 18mths sometime I see 20mths(Hammer). But anything with 1yr release cycle is not worth the time and having near 3yr support model is best for PROD.

Re: [ceph-users] Luminous BlueStore EC performance

What type of EC config (k+m) was used if I may ask? On Fri, Sep 8, 2017 at 1:34 AM, Mohamad Gebai wrote: > Hi, > > These numbers are probably not as detailed as you'd like, but it's > something. They show the overhead of reading and/or writing to EC pools as > compared to 3x

Re: [ceph-users] RGW Multisite metadata sync init

I sent the output of all of the files including the logs to you. Thank you for your help so far. On Thu, Sep 7, 2017 at 4:48 PM Yehuda Sadeh-Weinraub wrote: > On Thu, Sep 7, 2017 at 11:37 PM, David Turner > wrote: > > I'm pretty sure I'm using the

Re: [ceph-users] Blocked requests

After some troubleshooting, the issues appear to be caused by gnocchi using rados. I’m trying to figure out why. Thanks, Matthew Stroud From: Brian Andrus Date: Thursday, September 7, 2017 at 1:53 PM To: Matthew Stroud Cc: David Turner

[ceph-users] Vote re release cadence

One vote for: * Drop the odd releases, and aim for a ~9 month cadence. This splits the difference between the current even/odd pattern we've been doing. We've already been bit by gotchas with upgrades even between point releases, so I favor strategies that limit the number of upgrade paths in

Re: [ceph-users] RGW Multisite metadata sync init

On Thu, Sep 7, 2017 at 11:37 PM, David Turner wrote: > I'm pretty sure I'm using the cluster admin user/keyring. Is there any > output that would be helpful? Period, zonegroup get, etc? - radosgw-admin period get - radosgw-admin zone list - radosgw-admin zonegroup

Re: [ceph-users] RBD: How many snapshots is too many?

On 2017-09-05 02:41 PM, Gregory Farnum wrote: > On Tue, Sep 5, 2017 at 1:44 PM, Florian Haas > wrote: > >> Hi everyone, >> >> with the Luminous release out the door and the Labor Day weekend >> over, I hope I can kick off a discussion on another issue that has >> irked me a

Re: [ceph-users] RGW Multisite metadata sync init

I'm pretty sure I'm using the cluster admin user/keyring. Is there any output that would be helpful? Period, zonegroup get, etc? On Thu, Sep 7, 2017 at 4:27 PM Yehuda Sadeh-Weinraub wrote: > On Thu, Sep 7, 2017 at 11:02 PM, David Turner > wrote: > >

Re: [ceph-users] RGW Multisite metadata sync init

On Thu, Sep 7, 2017 at 11:02 PM, David Turner wrote: > I created a test user named 'ice' and then used it to create a bucket named > ice. The bucket ice can be found in the second dc, but not the user. > `mdlog list` showed ice for the bucket, but not for the user. I

[ceph-users] Significant uptick in inconsistent pgs in Jewel 10.2.9

Hi, Our clusters were upgraded to v10.2.9, from ~v10.2.7 (actually a local git snapshot that was not quite 10.2.7), and since then, we're seeing a LOT more scrub errors than previously. The digest logging on the scrub errors, in some cases, is also now maddeningly short: it doesn't contain ANY

Re: [ceph-users] RGW Multisite metadata sync init

I created a test user named 'ice' and then used it to create a bucket named ice. The bucket ice can be found in the second dc, but not the user. `mdlog list` showed ice for the bucket, but not for the user. I performed the same test in the internal realm and it showed the user and bucket both

Re: [ceph-users] Blocked requests

"ceph osd blocked-by" can do the same thing as that provided script. Can you post relevant osd.10 logs and a pg dump of an affected placement group? Specifically interested in recovery_state section. Hopefully you were careful in how you were rebooting OSDs, and not rebooting multiple in the

Re: [ceph-users] Blocked requests

Here is the output of your snippet: [root@mon01 ceph-conf]# bash /tmp/ceph_foo.sh 6 osd.10 52 ops are blocked > 4194.3 sec on osd.17 9 ops are blocked > 2097.15 sec on osd.10 4 ops are blocked > 1048.58 sec on osd.10 39 ops are blocked > 262.144 sec on osd.10 19 ops are blocked >

Re: [ceph-users] Client features by IP?

On 09/07/2017 11:31 AM, Bryan Stillwell wrote: On 09/07/2017 10:47 AM, Josh Durgin wrote: On 09/06/2017 04:36 PM, Bryan Stillwell wrote: I was reading this post by Josh Durgin today and was pretty happy to see we can get a summary of features that clients are using with the 'ceph features'

Re: [ceph-users] RGW Multisite metadata sync init

On Thu, Sep 7, 2017 at 10:04 PM, David Turner wrote: > One realm is called public with a zonegroup called public-zg with a zone for > each datacenter. The second realm is called internal with a zonegroup > called internal-zg with a zone for each datacenter. they each have

Re: [ceph-users] Client features by IP?

On 09/07/2017 10:47 AM, Josh Durgin wrote: > On 09/06/2017 04:36 PM, Bryan Stillwell wrote: > > I was reading this post by Josh Durgin today and was pretty happy to > > see we can get a summary of features that clients are using with the > > 'ceph features' command: > > > >

Re: [ceph-users] PCIe journal benefit for SSD OSDs

Hi Stefan >>Have you already done tests how he performance changes with bluestore >>while putting all 3 block devices on the same ssd? I'm going to test bluestore with 3 nodes , 18 x intel s3610 1,6TB in coming weeks. I'll send results on the mailing. - Mail original - De: "Stefan

Re: [ceph-users] Luminous BlueStore EC performance

Hi, These numbers are probably not as detailed as you'd like, but it's something. They show the overhead of reading and/or writing to EC pools as compared to 3x replicated pools using 1, 2, 8 and 16 threads (single client): Rep EC Diff Slowdown IOPS IOPS

Re: [ceph-users] RGW Multisite metadata sync init

One realm is called public with a zonegroup called public-zg with a zone for each datacenter. The second realm is called internal with a zonegroup called internal-zg with a zone for each datacenter. they each have their own rgw's and load balancers. The needs of our public facing rgw's and load

Re: [ceph-users] RGW Multisite metadata sync init

On Thu, Sep 7, 2017 at 7:44 PM, David Turner wrote: > Ok, I've been testing, investigating, researching, etc for the last week and > I don't have any problems with data syncing. The clients on one side are > creating multipart objects while the multisite sync is creating

Re: [ceph-users] Blocked requests

Well in the meantime things have gone from bad to worse now the cluster isn’t rebuilding and clients are unable to pass IO to the cluster. When this first took place, we started rolling back to 10.2.7, though that was successful, it didn’t help with the issue. Here is the command output:

Re: [ceph-users] Blocked requests

To be fair, other times I have to go in and tweak configuration settings and timings to resolve chronic blocked requests. On Thu, Sep 7, 2017 at 1:32 PM David Turner wrote: > `ceph health detail` will give a little more information into the blocked > requests.

Re: [ceph-users] Blocked requests

`ceph health detail` will give a little more information into the blocked requests. Specifically which OSDs are the requests blocked on and how long have they actually been blocked (as opposed to '> 32 sec'). I usually find a pattern after watching that for a time and narrow things down to an

Re: [ceph-users] Separate WAL and DB Partitions for existing OSDs ?

On Filestore you would flush the journal and then after mapping the new journal device use the command to create the journal. I'm sure there's something similar for bluestore, but I don't have any experience with it yet. Is there a new command similar to flush and create for the WAL and DB? On

Re: [ceph-users] Client features by IP?

On 09/06/2017 04:36 PM, Bryan Stillwell wrote: I was reading this post by Josh Durgin today and was pretty happy to see we can get a summary of features that clients are using with the 'ceph features' command: http://ceph.com/community/new-luminous-upgrade-complete/ However, I haven't found

Re: [ceph-users] RGW Multisite metadata sync init

Ok, I've been testing, investigating, researching, etc for the last week and I don't have any problems with data syncing. The clients on one side are creating multipart objects while the multisite sync is creating them as whole objects and one of the datacenters is slower at cleaning up the

Re: [ceph-users] Separate WAL and DB Partitions for existing OSDs ?

To be mor eprecise, what I want to know is: I have a lot of bluestore osds and now I want to add separate wal and db on new nvme partitions. Would it be enough to just generate empty partitions with parted and make symlinks on the osd partition like this: $ sudo ln -sf

[ceph-users] Blocked requests

After updating from 10.2.7 to 10.2.9 I have a bunch of blocked requests for ‘currently waiting for missing object’. I have tried bouncing the osds and rebooting the osd nodes, but that just moves the problems around. Previous to this upgrade we had no issues. Any ideas of what to look at?

Re: [ceph-users] ceph mgr unknown version

Thanks for explanation. W dniu 07.09.2017 o 12:06, John Spray pisze: On Wed, Sep 6, 2017 at 4:47 PM, Piotr Dzionek wrote: Oh, I see that this is probably a bug: http://tracker.ceph.com/issues/21260 I also noticed following error in mgr logs: 2017-09-06

Re: [ceph-users] [Ceph-maintainers] Ceph release cadence

On 2017-09-06T15:23:34, Sage Weil wrote: Hi Sage, thanks for kicking off this discussion - after the L experience, it was on my hot list to talk about too. I do agree that we need predictable releases more than feature-rich releases. Distributors like to plan, but that's not

Re: [ceph-users] ceph mgr unknown version

On Wed, Sep 6, 2017 at 4:47 PM, Piotr Dzionek wrote: > Oh, I see that this is probably a bug: http://tracker.ceph.com/issues/21260 > > I also noticed following error in mgr logs: > > 2017-09-06 16:41:08.537577 7f34c0a7a700 1 mgr send_beacon active > 2017-09-06

Re: [ceph-users] ceph mgr unknown version

On Wed, Sep 6, 2017 at 4:31 PM, Piotr Dzionek wrote: > Hi, > I ran a small test two node ceph cluster - 12.2.0 version. It has 28 osds, 1 > mon and 2 mgr. It runs fine, however I noticed this strange thing in output > of ceph versions command: > > # ceph versions > { >

[ceph-users] Separate WAL and DB Partitions for existing OSDs ?

Hi there, is it possible to move WAL and DB Data for Existing bluestore OSDs to separate partitions ? I am looking for a method to maybe take an OSD out, do some magic and move some data to new SSD Devices and then take the OSD back in. Any Ideas ? Thanks Christoph

Re: [ceph-users] PCIe journal benefit for SSD OSDs

Am 07.09.2017 um 10:44 schrieb Christian Balzer: > > Hello, > > On Thu, 7 Sep 2017 08:03:31 +0200 Stefan Priebe - Profihost AG wrote: > >> Hello, >> Am 07.09.2017 um 03:53 schrieb Christian Balzer: >>> >>> Hello, >>> >>> On Wed, 6 Sep 2017 09:09:54 -0400 Alex Gorbachev wrote: >>> We are

[ceph-users] radosgw-admin orphans find -- Hammer

Hello, we need to reclaim a lot of wasted space by RGW orphans in our production Hammer cluster (0.94.10 on Ubuntu 14.04). According to http://tracker.ceph.com/issues/18258 there is a bug in the radosgw-admin orphans find command, that causes it to get

Re: [ceph-users] PCIe journal benefit for SSD OSDs

Am 07.09.2017 um 10:22 schrieb Marc Roos: > > Sorry to cut in your thread. > >> Have you disabled te FLUSH command for the Samsung ones? > > We have a test cluster currently only with spinners pool, but we have > SM863 available to create the ssd pool. Is there something specific that >

Re: [ceph-users] PCIe journal benefit for SSD OSDs

Hello, On Thu, 7 Sep 2017 08:03:31 +0200 Stefan Priebe - Profihost AG wrote: > Hello, > Am 07.09.2017 um 03:53 schrieb Christian Balzer: > > > > Hello, > > > > On Wed, 6 Sep 2017 09:09:54 -0400 Alex Gorbachev wrote: > > > >> We are planning a Jewel filestore based cluster for a performance

Re: [ceph-users] PCIe journal benefit for SSD OSDs

Sorry to cut in your thread. > Have you disabled te FLUSH command for the Samsung ones? We have a test cluster currently only with spinners pool, but we have SM863 available to create the ssd pool. Is there something specific that needs to be done for the SM863? -Original

[ceph-users] Nimmst du meine Einladung an und kommst auch zu Ceph Berlin?

Ceph Berlin Begleite Robert Sander und 406 weitere Cephalopods zu Berlin. Bleib immer auf dem Laufenden über neue Events in Deiner Gegend. This is a group for anyone interested in Ceph (

Re: [ceph-users] Ceph release cadence

On 17-09-06 18:23, Sage Weil wrote: Hi everyone, Traditionally, we have done a major named "stable" release twice a year, and every other such release has been an "LTS" release, with fixes backported for 1-2 years. With kraken and luminous we missed our schedule by a lot: instead of releasing

Re: [ceph-users] Ceph release cadence

> * Drop the odd releases, and aim for a ~9 month cadence. This splits the > difference between the current even/odd pattern we've been doing. > > + eliminate the confusing odd releases with dubious value > + waiting for the next release isn't quite as bad > - required upgrades every 9

Re: [ceph-users] PCIe journal benefit for SSD OSDs

Hello, Am 07.09.2017 um 03:53 schrieb Christian Balzer: > > Hello, > > On Wed, 6 Sep 2017 09:09:54 -0400 Alex Gorbachev wrote: > >> We are planning a Jewel filestore based cluster for a performance >> sensitive healthcare client, and the conservative OSD choice is >> Samsung SM863A. >> > >