w.buckets.data' root_id -1 using 0.705080476146 of space,
> bias 1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
>
> Why the balancing does not happen is still nebulous to me.
>
>
>
> Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der Ster
> :
>>
>&
; 'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid', 'eu-msg-1.rgw.control',
> '.rgw.root', 'eu-msg-1.rgw.buckets.data', 'default.rgw.control',
> 'fra-1.rgw.log', 'default.rgw.data.root', 'whitespace-again-2021-03
osd.57
> 58 hdd 7.32619 1.0 7.3 TiB 5.2 TiB 5.2 TiB 604 MiB 13 GiB 2.1
> TiB 71.23 0.97 98 up osd.58
> 59 hdd 7.32619 1.0 7.3 TiB 5.1 TiB 5.1 TiB 414 MiB 13 GiB 2.2
> TiB 70.03 0.95 88 up osd.59
> 60 hdd 7.32619 1.0
Hi Sam,
The daemons restart (for *some* releases) because of this:
https://tracker.ceph.com/issues/21672
In short, if the selinux module changes, and if you have selinux
enabled, then midway through yum update, there will be a systemctl
restart ceph.target issued.
For the rest -- I think you shou
ey wrote:
>>
>> Hi Dan:
>>
>> Thanks for the reply - at present, our mons and mgrs are off [because of the
>> unsustainable nature of the filesystem usage]. We'll try putting them on
>> again for long enough to get "ceph status" out of them, bu
one blocked for 901 sec, mon.cephs01 has
> slow ops
>
> services:
> mon: 3 daemons, quorum cephs01,cephs02,cephs03 (age 2h)
> mgr: cephs01(active, since 77m)
> osd: 329 osds: 98 up (since 4s), 328 in (since 4d)
> flags pauserd,pausewr,noout,nobackfill,noreba
7;s changed
> about how we specify stuff?
>
> This might correlate with the other person on the IRC list who has
> problems with 14.2.18 and their OSDs deciding they don't work sometimes
> until they forcibly restart their network links...
>
>
> Sam
>
> On Mon, 22
ever
>
> (where here p2p2 is the only active network link, and is also the private
> and public network for the ceph cluster)
>
> The output is similar on other hosts - with p2p2 either at position 3 or 5
> depending on the order the interfaces were enumerated.
>
> Sam
>
&g
nd address explicitly does seem to
> help, so we'll iterate on that and get to a suitable ceph.conf.
>
> Thanks for the help [and it was the network all along]!
>
>
> Sam
>
> On Mon, 22 Mar 2021 at 19:12, Dan van der Ster wrote:
>>
>> There are two commi
Sam, see https://tracker.ceph.com/issues/49938 and
https://github.com/ceph/ceph/pull/40334
On Tue, Mar 23, 2021 at 8:29 AM Dan van der Ster wrote:
>
> Hi Sam,
>
> Yeah somehow `lo:` is not getting skipped, probably due to those
> patches. (I guess it is because the 2nd patch look
y the same client, sequentially:
>
> ]# getfattr -n ceph.dir.rctime *
> # file: 0001
> ceph.dir.rctime="1614858289.09988958262"
> # file: 0002
> ceph.dir.rctime="2140551942.090"
> # file: 0003
> ceph.dir.rctime="1614876495.09878190535"
>
>
n Tue, Mar 23, 2021 at 2:45 PM Sam Skipsey wrote:
>
> I should note that our cluster is entirely IPv4 [because it's on a private
> network, so there's no need to go IPv6], which maybe influences matters?
>
> Sam
>
> On Tue, 23 Mar 2021 at 11:43, Stefan Kooman wro
000 3.7 TiB 3.0 TiB 2.9 TiB 440 MiB 8.1 GiB 704
>> GiB 81.35 1.08 48 up osd.75
>> 71 hdd 3.68750 1.0 3.7 TiB 3.0 TiB 3.0 TiB 7.5 MiB 8.0 GiB 663
>> GiB 82.44 1.09 47 up osd.71
>> 76 hdd 3.68750 1.0 3.7 TiB 3.1 TiB 3.0 TiB 251 M
that the data will go to other PGs than the ones that are currently on filled
> OSDs.
>
>
>
> Am Di., 23. März 2021 um 18:58 Uhr schrieb Dan van der Ster
> :
>>
>> Hi,
>>
>> backfill_toofull is not a bad thing when the cluster is really full
>> like yo
ver...)
-- dan
On Tue, Mar 23, 2021 at 7:17 PM Boris Behrens wrote:
>
> Ok, then I will try to reweight the most filled OSDs to .95 and see if this
> helps.
>
> Am Di., 23. März 2021 um 19:13 Uhr schrieb Dan van der Ster
> :
>>
>> Data goes to *all* PGs unifor
You can use:
osd_crush_initial_weight = 0.0
-- Dan
On Wed, Mar 24, 2021 at 2:23 PM Boris Behrens wrote:
>
> Hi people,
>
> I currently try to add ~30 OSDs to our cluster and wanted to use the
> gentle-rerweight script for that.
> I use ceph-colume lvm prepare --data /dev/sdX to create the osd
Not sure why, without looking at your crush map in detail.
But to be honest, I don't think you need such a tool anymore. It was
written back in the filestore days when backfilling could be much more
disruptive than today.
You have only ~10 osds to fill up: just mark them fully in, or increment by
Here's a crush ruleset for 8+2 that will choose 2 osds per host:
rule cephfs_data_82 {
id 4
type erasure
min_size 3
max_size 10
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step choose indep 5 typ
netstat -anp | grep LISTEN | grep mgr
has it bound to 127.0.0.1 ?
(also check the other daemons).
If so this is another case of https://tracker.ceph.com/issues/49938
-- dan
On Thu, Mar 25, 2021 at 8:34 PM Simon Oosthoek wrote:
>
> Hi
>
> I'm in a bit of a panic :-(
>
> Recently we started att
So do this. Get the ip of the host running the mgr, and put it this in
the config file:
[global]
public addr =
cluster addr =
Then restart your mgr.
IMHO we really need a 14.2.19 with this fixed asap.
On Thu, Mar 25, 2021 at 8:47 PM Simon Oosthoek wrote:
>
> On 25/03/2021 20:42, D
In each host's ceph.conf, put:
[global]
public addr =
cluster addr =
Or use the public network / cluster network options = a.b.c.0/24 or
however your network is defined.
-- dan
On Thu, Mar 25, 2021 at 8:50 PM Simon Oosthoek wrote:
>
> On 25/03/2021 20:42, Dan van der
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking
that if you have already backfilled the degraded PGs, and can afford
to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because
https://github.com/facebook/rocksdb/issue
39 0.95000 7.3 TiB 6.7 TiB 6.7 TiB 322 MiB 16 GiB 548
> GiB 92.64 1.18 121 up osd.66
> 46 hdd 7.27739 1.0 7.3 TiB 6.8 TiB 6.7 TiB 316 MiB 16 GiB 536
> GiB 92.81 1.18 119 up osd.46
>
> Am Di., 23. März 2021 um 19:59 Uhr schrieb Boris B
We wanted to insert another OSD node with
> 7x8TB disks anyway, but postponed it due to the rebalancing.
>
> Am Di., 30. März 2021 um 14:23 Uhr schrieb Dan van der Ster
> :
>>
>> Are those PGs backfilling due to splitting or due to balancing?
>> If it's the former
u-central-1.rgw.buckets.data' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 2048 pgp_num 1946 pgp_num_target
> 2048 autoscale_mode warn last_change 320966 lfor 0/263549/317774 flags
> hashpspool,backfillfull stripe_width 0 application rgw
> ...
>
> Am Di., 30. M
Hi Robert,
We get a handful of verify_authorizer warnings on some of our clusters
too but they don't seem to pose any problems.
I've tried without success to debug this in the past -- IIRC I started
to suspect it was coming from old cephfs kernel clients but got
distracted and never reached the bo
Hi Andras.
Assuming that you've already tightened the
mgr/balancer/upmap_max_deviation to 1, I suspect that this cluster
already has too many upmaps.
Last time I checked, the balancer implementation is not able to
improve a pg-upmap-items entry if one already exists for a PG. (It can
add an OSD m
ld be used to remove
any upmaps which are directing PGs *to* those toofull OSDs. Or maybe
it will be enough to just reweight those OSDs to 0.9.
-- Dan
On Fri, Apr 2, 2021 at 10:47 AM Dan van der Ster wrote:
>
> Hi Andras.
>
> Assuming that you've already tighte
Hi Robert,
Have you checked a log with debug_mon=20 yet to try to see what it's doing?
.. Dan
On Fri, Apr 9, 2021, 7:02 PM Robert LeBlanc wrote:
> The only step not yet taken was to move to straw2. That was the last
> step we were going to do next.
>
> Robert LeBlanc
> PGP F
On Fri, Apr 9, 2021 at 7:24 PM Robert LeBlanc wrote:
>
> On Fri, Apr 9, 2021 at 11:05 AM Dan van der Ster wrote:
> >
> > Hi Robert,
> >
> > Have you checked a log with debug_mon=20 yet to try to see what it's doing?
> >
> I've posted the logs with
On Fri, Apr 9, 2021 at 8:39 PM Robert LeBlanc wrote:
>
> On Fri, Apr 9, 2021 at 11:49 AM Dan van der Ster wrote:
> >
> > Thanks. I didn't see anything ultra obvious to me.
> >
> > But I did notice the nearfull warnings so I wonder if this cluster is
> >
On Fri, Apr 9, 2021 at 9:37 PM Dan van der Ster wrote:
>
> On Fri, Apr 9, 2021 at 8:39 PM Robert LeBlanc wrote:
> >
> > On Fri, Apr 9, 2021 at 11:49 AM Dan van der Ster
> > wrote:
> > >
> > > Thanks. I didn't see anything ultra obvious to me.
>
On Fri, Apr 9, 2021 at 11:50 PM Robert LeBlanc wrote:
>
> On Fri, Apr 9, 2021 at 2:04 PM Dan van der Ster wrote:
> >
> > On Fri, Apr 9, 2021 at 9:37 PM Dan van der Ster wrote:
> > >
> > > On Fri, Apr 9, 2021 at 8:39 PM Robert LeBlanc
> > > wrote:
> Thanks for the idea, I've tried it with 1 thread, and it shredded another OSD.
> I've updated the tracker ticket :)
>
> At least non-racecondition bugs are hopefully easier to spot...
>
> I wouldn't just disable the fsck and upgrade anyway until the cause is rooted
&
Hi all,
bdev_enable_discard has been in ceph for several major releases now
but it is still off by default.
Did anyone try it recently -- is it safe to use? And do you have perf
numbers before and after enabling?
Cheers, Dan
___
ceph-users mailing list
On Tue, Apr 13, 2021 at 9:00 AM Wido den Hollander wrote:
>
>
>
> On 4/12/21 5:46 PM, Dan van der Ster wrote:
> > Hi all,
> >
> > bdev_enable_discard has been in ceph for several major releases now
> > but it is still off by default.
> > Did anyone try it
On Tue, Apr 13, 2021 at 12:35 PM Mark Nelson wrote:
>
> On 4/13/21 4:07 AM, Dan van der Ster wrote:
>
> > On Tue, Apr 13, 2021 at 9:00 AM Wido den Hollander wrote:
> >>
> >>
> >> On 4/12/21 5:46 PM, Dan van der Ster wrote:
> >>> Hi all
Hi Igor,
After updating to 14.2.19 and then moving some PGs around we have a
few warnings related to the new efficient PG removal code, e.g. [1].
Is that something to worry about?
Best Regards,
Dan
[1]
/var/log/ceph/ceph-osd.792.log:2021-04-14 20:34:34.353 7fb2439d4700 0
osd.792 pg_epoch: 409
a paranoid
> > check for an unexpected situation which has actually triggered. Hence
> > IMO no need to worry at this point but developers might want to validate
> > why this is happening
> >
> >
> > Thanks,
> >
> > Igor
> >
> > On 4/
This should help:
ceph config set mgr mgr/balancer/upmap_max_deviation 1
On Mon, Apr 19, 2021 at 10:17 AM Ml Ml wrote:
>
> Anyone an idea? :)
>
> On Fri, Apr 16, 2021 at 3:09 PM Ml Ml wrote:
> >
> > Hello List,
> >
> > any ideas why my OSDs are that unbalanced ?
> >
> > root@ceph01:~# ceph -s
>
On Tue, Apr 20, 2021 at 11:26 AM Ilya Dryomov wrote:
>
> On Tue, Apr 20, 2021 at 2:01 AM David Galloway wrote:
> >
> > This is the 20th bugfix release in the Nautilus stable series. It
> > addresses a security vulnerability in the Ceph authentication framework.
> > We recommend users to update t
=
0x5573928a7340
2021-04-21 15:23:50.473 7f51b2498700 0 log_channel(cluster) log [WRN]
: Monitor daemon marked osd.941 down, but it is still running
-- dan
On Thu, Apr 15, 2021 at 10:32 AM Dan van der Ster wrote:
>
> Thanks Igor and Neha for the quick responses.
>
> I posted an osd
to backfill new OSD's on 14.2.19: pool have 2048PG with 7.14G
> objects...
> avg number of PG is 3486328 objects
>
>
>
> [1] https://tracker.ceph.com/issues/47044
>
> k
>
>
> On 21 Apr 2021, at 16:37, Dan van der Ster wrote:
>
> Do we have a tracker for
ote:
>
> Hi Dan,
>
> I recall no relevant tracker, feel free to create.
>
> Curious if you had bluefs_buffered_io set to true when faced that?
>
>
> Thanks,
>
> Igor
>
> On 4/21/2021 4:37 PM, Dan van der Ster wrote:
> > Do we have a tracker for
s is hdd or hybrids OSD's? How much obj per PG avg?
>
>
> k
>
> Sent from my iPhone
>
> > On 21 Apr 2021, at 17:44, Dan van der Ster wrote:
> >
> > Here's a tracker: https://tracker.ceph.com/issues/50466
> >
> > bluefs_buffered_io is indeed enable
Are you currently doing IO on the relevant pool? Maybe nearfull isn't
reported until some pgstats are reported.
Otherwise sorry I haven't seen this.
Dan
On Wed, Apr 21, 2021, 8:05 PM Konstantin Shalygin wrote:
> Hi,
>
> On the adopted cluster Prometheus was triggered for "osd full > 90%"
>
Did this eventually clear?
We had something like this happen once when we changed an md export pin for
a very top level directory from mds.3 to mds.0. This triggered so much
subtree export work that it took something like 30 minutes to complete. In
our case the md segments kept growing into a few 1
normally I'd follow the advice of
> restarting the mds:
> https://docs.ceph.com/en/latest/cephfs/troubleshooting/#slow-requests-mds
>
> ... but I just did that and replaying 1200 segments took two hours, so if
> the same happens again, then I'm looking at 12 hours of downtime, which
| [.dir.path, .auth_first,
.export_pin]'
Dan
On Wed, Apr 21, 2021, 9:05 PM Flemming Frandsen wrote:
> No, I don't.
>
> I guess I could pin a large part of the tree, if that's something that's
> likely to help.
>
> On Wed, 21 Apr 2021 at 21:02, Dan van der S
Hi Robert,
That's not very good !
You can probably set that back to false on each mon with:
ceph daemon mon.`hostname -s` config set
auth_allow_insecure_global_id_reclaim true
-- Dan
On Thu, Apr 22, 2021 at 9:08 AM Robert Sander
wrote:
>
> Hi,
>
> I upgraded a test cluster to 15.2.11 and aft
Hi,
Just pinging to check if this issue was understood yet?
Cheers, Dan
On Mon, Apr 12, 2021 at 9:12 PM Jonas Jelten wrote:
>
> Hi Igor!
>
> I have plenty of OSDs to loose, as long as the recovery works well afterward,
> so I can go ahead with it :D
>
> What debug flags should I activate? osd=
Hi all,
In 14.2.20, when re-creating a mixed OSD after device replacement,
ceph-volume batch is no longer able to find any available space for a
block_db.
Below I have shown a zap [1] which frees up the HDD and one LV on the
block-dbs VG.
But then we try to recreate, and none of the block-dbs are
y
> /dev/sdf" command should give you the reason why the device was
> rejected.
> If not legit that's probably a bug...
>
> Thanks!
> –
> Sébastien Han
> Senior Principal Software Engineer, Storage Architect
>
> "Always give 100%. Unless
On Sat, Apr 10, 2021 at 2:10 AM Robert LeBlanc wrote:
>
> On Fri, Apr 9, 2021 at 4:04 PM Dan van der Ster wrote:
> >
> > Here's what you should look for, with debug_mon=10. It shows clearly
> > that it takes the mon 23 seconds to run through
> > get_rem
Wait, first just restart the leader mon.
See: https://tracker.ceph.com/issues/47380 for a related issue.
-- dan
On Mon, May 3, 2021 at 2:55 PM Vladimir Sigunov
wrote:
>
> Hi Frank,
> Yes, I would purge the osd. The cluster looks absolutely healthy except of
> this osd.584 Probably, the purge
Hi,
This sounds a lot like the negative progress bug we just found last
week: https://tracker.ceph.com/issues/50591
That bug makes the mon enter a very long loop rendering a progress bar
if the mgr incorrectly sends a message to the mon that the progress is
negative.
Octopus and later don't have
On Tue, May 4, 2021 at 4:21 PM Janne Johansson wrote:
>
> Den tis 4 maj 2021 kl 16:10 skrev Rainer Krienke :
> > Hello,
> > I am playing around with a test ceph 14.2.20 cluster. The cluster
> > consists of 4 VMs, each VM has 2 OSDs. The first three VMs vceph1,
> > vceph2 and vceph3 are monitors. v
On Tue, May 4, 2021 at 4:34 PM Janne Johansson wrote:
>
> Den tis 4 maj 2021 kl 16:29 skrev Dan van der Ster :
> > BTW, if you find that this is indeed what's blocking your mons, you
> > can workaround by setting `ceph progress off` until the fixes are
> > released.
Hi Bryan,
I had to do something similar, and never found a rule to place "up to"
2 chunks per host, so I stayed with the placement of *exactly* 2
chunks per host.
But I did this slightly differently to what you wrote earlier: my rule
chooses exactly 4 hosts, then chooses exactly 2 osds on each:
Hi,
Do you have the mds log from the initial crash?
Also, I don't see the new global_id warnings in your status output --
did you change any settings from the defaults during this upgrade?
Cheers, Dan
On Tue, May 18, 2021 at 10:22 AM Eugen Block wrote:
>
> Hi *,
>
> I tried a minor update (14.
advanced mon_warn_on_insecure_global_id_reclaim_allowed false
but if you changed these other settings to false, it might have caused the
old mds to error:
auth_allow_insecure_global_id_reclaim
auth_expose_insecure_global_id_reclaim
Cheers, Dan
>
> Thanks!
> Eugen
>
&g
ue to some temporary unavailability of the host at
> some point? As all my servers are now up and running, is there a way to
> force the placement rule to rerun?
>
>Thanks!
>
> Fulvio
>
>
> Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
>
y unavailability of the host at
> some point? As all my servers are now up and running, is there a way to
> force the placement rule to rerun?
>
>Thanks!
>
> Fulvio
>
>
> Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
> > Hi Bryan,
> >
&g
I can confirm that we still occasionally see stored==used even with
14.2.21, but I didn't have time yet to debug the pattern behind the
observations. I'll let you know if we find anything useful.
.. Dan
On Thu, May 20, 2021, 6:56 PM Konstantin Shalygin wrote:
>
>
> > On 20 May 2021, at 19:47,
Hi,
The mds_cache_memory_limit should be set to something relative to the
RAM size of the MDS -- maybe 50% is a good rule of thumb, because
there are a few cases where the RSS can exceed this limit. Your
experience will help guide what size you need (metadata pool IO
activity will be really high i
ephFS behaved when changing one of the parameters.
> Although I have achieved improvement the procedure does not convince me
> at all and that's why I was asking if there was something more reliable ...
>
>
>
>
> El 26/5/21 a las 12:15, Dan van der Ster escribió:
> > Hi,
>
gt;
> Degrading the filesystem and I have assumed that the problem is due to
> the memory consumption of the MDS process, which can reach around 80% or
> more of the total memory.
>
>
>
>
>
> El 26/5/21 a las 13:21, Dan van der Ster escribió:
> > I've seen you
On Thu, May 27, 2021 at 9:21 AM Andres Rojas Guerrero wrote:
>
>
>
> El 26/5/21 a las 16:51, Dan van der Ster escribió:
> > I see you have two active MDSs. Is your cluster more stable if you use
> > only one single active MDS?
>
> Good question!! I read form Ceph
an we consider that
> there are many clients?
>
>
>
> El 27/5/21 a las 9:24, Dan van der Ster escribió:
> > On Thu, May 27, 2021 at 9:21 AM Andres Rojas Guerrero
> > wrote:
> >>
> >>
> >>
> >> El 26/5/21 a las 16:51, Dan van der Ster escr
ndeed, Dan:
>
> [cephmgr@cephAdmPA1.cephAdmPA1 ~]$ ceph osd dump | grep upmap | grep 116.453
> pg_upmap_items 116.453 [76,49,129,108]
>
> Don't think I ever set such an upmap myself. Do you think it would be
> good to try and remove all upmaps, let the upmap balancer do its magi
Hi,
I've never encountered this before.
To troubleshoot you could try to identify if this was caused by the
MDS writing to the metadata pool (e.g. maybe the mds log?), or if it
was some operation in the OSD which consumed too much space (e.g.
something like compaction?).
Can you find any unusual
Hi,
client_force_lazyio only works for ceph-fuse and libcephfs:
https://github.com/ceph/ceph/pull/26976/files
You can use the ioctl to enable per file with the kernel mount, but
you might run into the same problem we did:
https://tracker.ceph.com/issues/44166
Please share if it works for you.
C
Hi,
At which point in the update procedure did you
ceph osd require-osd-release octopus
?
And are you sure it was set to nautilus before the update? (`ceph osd dump`
will show)
Cheers , Dan
On Thu, Jun 10, 2021, 5:45 PM Manuel Lausch wrote:
> Hi Peter,
>
> your suggestion pointed me t
t this point.
> I did set this to octopus, after all components was updatated.
>
>
> Manuel
>
>
>
> On Thu, 10 Jun 2021 17:54:49 +0200
> Dan van der Ster wrote:
>
> > Hi,
> >
> > At which point in the update procedure did you
> >
> > ce
On Fri, Jun 11, 2021 at 11:08 AM Peter Lieven wrote:
>
> Am 10.06.21 um 17:45 schrieb Manuel Lausch:
> > Hi Peter,
> >
> > your suggestion pointed me to the right spot.
> > I didn't know about the feature, that ceph will read from replica
> > PGs.
> >
> > So on. I found two functions in the osd/Pr
Hi all,
The cluster here is running v14.2.20 and is used for RBD images.
We have a PG in recovery_unfound state and since this is the first
time we've had this occur, we wanted to get your advice on the best
course of action.
PG 4.1904 went into state active+recovery_unfound+degraded+repair [1]
-- I think we need a way to tell the
cluster to use the object at v 3593555'368312656.
Cheers, Dan
On Sun, Jun 13, 2021 at 8:58 AM Dan van der Ster wrote:
>
> Hi all,
>
> The cluster here is running v14.2.20 and is used for RBD images.
>
> We have a PG in recovery_unfound s
is bug and
hopeful find a fix so it doesn't happen again.
Cheers, dan
On Sun, Jun 13, 2021 at 3:36 PM Dan van der Ster wrote:
>
> Hi again,
>
> We haven't taken any actions yet, but this seems like it might be a bug.
> We compared the version numbers with the osdmap
Looks like this: https://tracker.ceph.com/issues/51112
On Tue, Jun 15, 2021 at 5:48 PM Ackermann, Christoph
wrote:
>
> Hello all,
>
> after upgrading Centos clients to version 8.4 CephFS ( Kernel
> 4.18.0-305.3.1.el8 ) mount did fail. Message: *mount error 110 =
> Connection timed out*
> ..unf
Hi Ilya,
We're now hitting this on CentOS 8.4.
The "setmaxosd" workaround fixed access to one of our clusters, but
isn't working for another, where we have gaps in the osd ids, e.g.
# ceph osd getmaxosd
max_osd = 553 in epoch 691642
# ceph osd tree | sort -n -k1 | tail
541 ssd 0.87299
Replying to own mail...
On Tue, Jun 15, 2021 at 7:54 PM Dan van der Ster wrote:
>
> Hi Ilya,
>
> We're now hitting this on CentOS 8.4.
>
> The "setmaxosd" workaround fixed access to one of our clusters, but
> isn't working for another, where we have
ient libceph.
>
> Christoph
>
>
>
>
>
> Am Di., 15. Juni 2021 um 20:26 Uhr schrieb Dan van der Ster
> :
>>
>> Replying to own mail...
>>
>> On Tue, Jun 15, 2021 at 7:54 PM Dan van der Ster wrote:
>> >
>> > Hi Ilya,
>> &g
Hi,
Stuck activating could be an old known issue: if the cluster has many
(>100) PGs per OSD, they may temporarily need to hold more than the
max (300) and therefore PGs get stuck activating.
We always use this option as a workaround:
osd max pg per osd hard ratio = 10.0
I suggest giving thi
On Thu, Jul 1, 2021 at 12:53 AM Patrick Donnelly wrote:
>
> Hi Dan,
>
> Sorry for the very late reply -- I'm going through old unanswered email.
>
> On Mon, Nov 9, 2020 at 4:13 PM Dan van der Ster wrote:
> >
> > Hi,
> >
> > Today while debugging s
Hi,
That's interesting -- yes on a lightly loaded cluster the metadata IO
should be almost nil.
You can debug what is happening using ceph daemonperf on the active
MDS, e.g. https://pastebin.com/raw/n0iD8zXY
(Use a wide terminal to show all the columns).
Normally, lots of md io would indicate t
till taking 64 MB/s writes.
>
> We have two active MDS servers, without pinning.
>
> mds_cache_memory_limit is set to 20 GB, which ought to be enough for
> anyone(tm) as only 24 GB of data is used in the metadata pool.
>
> Does that offer any kind of clue?
>
> On Thu, 8 Ju
Hi Boris,
Does the bucket have object versioning enabled?
We saw something like this once a while ago: `s3cmd ls` showed an
entry for an object, but when we tried to get it we had 404.
We didn't find a good explanation in the end -- our user was able to
re-upload the object and it didn't recur so
Behrens, wrote:
> Hi Dan,
> how do I find out if a bucket got versioning enabled?
>
> Am Mo., 19. Juli 2021 um 17:00 Uhr schrieb Dan van der Ster
> :
> >
> > Hi Boris,
> >
> > Does the bucket have object versioning enabled?
> > We saw something like this on
Hi Mark and all,
The key point is to consider your users' write requirements: do your
applications need to write concurrently to the same file from several
cephfs mounts? or does each job write to a separate file?
If your use-case is predominantly the latter, you'll have a lot of
success right now
Hi Rafael,
AFAIU, that gc issue was not relevant for N -- the bug is in the new
rgw_gc code which landed in Octopus and was not backported to N.
Well, RHCEPH had the new rgw_gc cls backported to it, and RHCEPH has
the bugfix you refer to:
* Wed Dec 02 2020 Ceph Jenkins 2:14.2.11-86
- rgw: during
h.com/issues/47866
-- Dan
On Thu, Jul 22, 2021 at 11:12 AM Dan van der Ster wrote:
>
> Hi Rafael,
>
> AFAIU, that gc issue was not relevant for N -- the bug is in the new
> rgw_gc code which landed in Octopus and was not backported to N.
>
> Well, RHCEPH had the new rgw_gc
With four mons total then only one can be down... mon.osd01 is down already
you're at the limit.
It's possible that whichever reason is preventing this mon from joining
will also prevent the new mon from joining.
I think you should:
1. Investigate why mon.osd01 isn't coming back into the quorum.
ndle_reset 0x560954d36000 -
> mon.osd01@0(probing) e1 probe_timeout 0x560954c9c420
> mon.osd01@0(probing) e1 bootstrap
> mon.osd01@0(probing) e1 sync_reset_requester
> mon.osd01@0(probing) e1 unregister_cluster_logger - not registered
> mon.osd01@0(probing) e1 cancel_probe_timeout
's stuck probing.
.. Dan
On Sun, 25 Jul 2021, 17:53 Ansgar Jazdzewski,
wrote:
> Am So., 25. Juli 2021 um 17:17 Uhr schrieb Dan van der Ster
> :
> >
> > > raise the min version to nautilus
> >
> > Are you referring to the min osd version or the min client
/issues/48033
-- Dan
On Sun, Jul 25, 2021 at 6:36 PM Ansgar Jazdzewski
wrote:
>
> Am So., 25. Juli 2021 um 18:02 Uhr schrieb Dan van der Ster
> :
> >
> > What do you have for the new global_id settings? Maybe set it to allow
> > insecure global_id auth and see if
5 06:46:52.042 7fe061f1c700 0 log_channel(cluster) log [WRN]
> : Health detail: HEALTH_WARN 7 slow ops, oldest one blocked for 36
> sec, daemons [mon.osd02,mon.osd03] have slow ops.
> 2021-07-25 06:46:52.042 7fe061f1c700 0 log_channel(cluster) log [WRN]
> : SLOW_OPS 7 slow ops, oldest
On Tue, Jul 27, 2021 at 1:46 PM Marcel Kuiper wrote:
>
> We recently upgraded one of our clusters from 14.2.21 to 15.2.13 and had
> some troubles with osd performance. After reading some threads we
> manually compacted the rocksdb's on all osds and hope that that will
> alleviate the problem
>
> I
Hi,
Start by setting:
ceph config set mgr mgr/balancer/upmap_max_deviation 1
This configures the balancer to squeeze the OSDs to within 1 PG of eachother.
I'm starting to think this should be the default.
Cheers, dan
On Wed, Jul 28, 2021 at 9:08 AM Manuel Holtgrewe wrote:
>
> Dear all,
anuel
>
> On Wed, Jul 28, 2021 at 9:15 AM Dan van der Ster wrote:
>>
>> Hi,
>>
>> Start by setting:
>>
>> ceph config set mgr mgr/balancer/upmap_max_deviation 1
>>
>> This configures the balancer to squeeze the OSDs to within 1 PG of
ot;oldest_map": 99775,
> "newest_map": 281713,
> "num_pgs": 77
> }
>
> I found this ticket (https://tracker.ceph.com/issues/38931 I believe you
> actually opened it ;-)) and tried to restart the osd.0 and now the OSD is
> scrubbing some of its
101 - 200 of 528 matches
Mail list logo
