On Sat, Mar 23, 2024, 4:26 AM Torkil Svensgaard wrote:
> Hi
>
> ... Using mclock with high_recovery_ops profile.
>
> What is the bottleneck here? I would have expected a huge number of
> simultaneous backfills. Backfill reservation logjam?
>
mClock is very buggy in my experience and frequently
> On 23.02.24 16:18, Christian Rohmann wrote:
> > I just noticed issues with ceph-crash using the Debian /Ubuntu
> > packages (package: ceph-base):
> >
> > While the /var/lib/ceph/crash/posted folder is created by the package
> > install,
> > it's not properly chowned to ceph:ceph by the postinst
On Tue, Dec 5, 2023 at 10:13 AM Zakhar Kirpichenko wrote:
>
> Any input from anyone?
>
> /Z
IIt's not clear whether or not these issues are related. I see three
things in this e-mail chain:
1) bdev() _aio_thread with EPERM, as in the subject of this e-mail chain
2) bdev() _aio_thread with the
On Sat, Nov 4, 2023, 6:44 AM Matthew Booth wrote:
> I have a 3 node ceph cluster in my home lab. One of the pools spans 3
> hdds, one on each node, and has size 2, min size 1. One of my nodes is
> currently down, and I have 160 pgs in 'unknown' state. The other 2
> hosts are up and the cluster
h mgr is active and/or try stopping all but the Octopus
mgr when stopping the mon as well?
Cheers,
Tyler
> On Thu, Oct 26, 2023 at 4:57 PM Tyler Stachecki
> wrote:
>
>> On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia
>> wrote:
>> >
>> > Hi Tyler,
>> >
On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia wrote:
>
> Hi Tyler,
>
> Maybe you didn't read the full message, but in the message you will notice
> that I'm doing exactly that, and the problem just occurred when I was doing
> the upgrade from Octopus to Pacific. I'm nowhere near Quincy yet. The
On Thu, Oct 26, 2023 at 6:36 PM Tyler Stachecki
wrote:
>
>
> On Thu, Oct 26, 2023, 6:16 PM Jorge Garcia wrote:
> >
> > from Centos 7 and Nautilus to
> > Rocky 9 and Quincy.
>
> I hate to be the bearer of bad news here, but:
> https://docs.ceph.com/en/latest
On Thu, Oct 26, 2023, 6:16 PM Jorge Garcia wrote:
>
> from Centos 7 and Nautilus to
> Rocky 9 and Quincy.
I hate to be the bearer of bad news here, but:
https://docs.ceph.com/en/latest/releases/quincy/#upgrading-from-pre-octopus-releases-like-nautilus
"You *must* first upgrade to Octopus
ar between
some of the operators and developers. Wallaby is still marked EM upstream,
but even so did not get this patch.
The story is, unfortunately, the same here: the only way out of some of
these holes is to upgrade...
Regards,
Tyler
>
> On Fri, 20 Oct 2023 at 15:39, Tyler Stachecki
> wr
On Fri, Oct 20, 2023, 8:11 AM Zakhar Kirpichenko wrote:
> Thank you, Igor.
>
> It is somewhat disappointing that fixing this bug in Pacific has such a low
> priority, considering its impact on existing clusters.
>
Unfortunately, the hard truth here is that Pacific (stable) was released
over 30
On Tue, Oct 17, 2023, 8:19 PM Dave Hall wrote:
> Hello,
>
> I have a Nautilus cluster built using Ceph packages from Debian 10
> Backports, deployed with Ceph-Ansible.
>
> I see that Debian does not offer Ceph 15/Octopus packages. However,
> download.ceph.com does offer such packages.
>
>
On Sat, Oct 14, 2023 at 5:52 PM Tyler Stachecki
wrote:
>
> On Sat, Oct 14, 2023 at 5:14 PM Dave Hall wrote:
> >
> > Hello.
> >
> > It's been a while. For the past couple years I've had a cluster running
> > Nautilus on Debian 10 using the Debian Ceph packages
On Sat, Oct 14, 2023 at 5:14 PM Dave Hall wrote:
>
> Hello.
>
> It's been a while. For the past couple years I've had a cluster running
> Nautilus on Debian 10 using the Debian Ceph packages, and deployed with
> Ceph-Ansible. It's not a huge cluster - 10 OSD nodes with 80 x 12TB HDD
> OSDs,
ctions so that it can render progress bars ETAs in e.g. `ceph -s` output.
> /Z
>
> On Fri, 29 Sept 2023 at 14:13, Tyler Stachecki
> wrote:
>
>> On Fri, Sep 29, 2023, 5:55 AM Zakhar Kirpichenko
>> wrote:
>>
>>> Thank you, Eugen.
>>>
>>> Indeed it l
On Fri, Sep 29, 2023, 5:55 AM Zakhar Kirpichenko wrote:
> Thank you, Eugen.
>
> Indeed it looks like the progress module had some stale events from the
> time when we added new OSDs and set a specific number of PGs for pools,
> while the autoscaler tried to scale them down. Somehow the
On Sat, Sep 9, 2023 at 10:48 AM Anthony D'Atri wrote:
> There was also at point an issue where clients wouldn’t get a runtime update
> of new mons.
There's also 8+ year old unresolved bugs like this in OpenStack Cinder
that will bite you if the relocated mons have new IP addresses:
On Sun, Aug 13, 2023 at 10:43 PM Anthony D'Atri wrote:
> Think you meant s/SSD/SAS|SATA/.
Indeed, (SATA/SAS) SSD - thanks!
> The OP implies that the cluster's performance *degraded* with the Quincy
> upgrade.I wonder if there was a kernel change at the same time.
Also a good point. OP: do you
On Sun, Aug 13, 2023 at 10:19 PM J David wrote:
>
> I have a cluster that is performing *very* poorly.
>
> It has 12 physical servers and 72 400GB SATA Intel DC mixed-use SSD
> OSDs (S3700's & S3710's). All servers have bonded 10GB NICs for 20Gbps
> aggregate networking to a dedicated switch. At
On Fri, Aug 4, 2023 at 11:33 AM Dave Hall wrote:
>
> Dave,
>
> Actually, my failure domain is OSD since I so far only have 9 OSD nodes but
> EC 8+2. However, the drives are still functioning, except that one has
> failed multiple times in the last few days, requiring a node power-cycle to
>
Hi all,
Has anyone else noticed any p99.99+ tail latency regression for RBD
workloads in Quincy vs. pre-Pacific, i.e., before the kv_onode cache
existed?
Some notes from what I have seen thus far:
* Restarting OSDs temporarily resolves the problem... then as activity
accrues over time, the
On Sun, Oct 23, 2022 at 11:04 PM can zhu wrote:
>
> crash info:
>
> {
> "backtrace": [
> "/lib64/libpthread.so.0(+0x12ce0) [0x7f82e87cece0]",
> "(BlueStore::Onode::put()+0x1a3) [0x55bd21a422e3]",
> "(std::_Hashtable boost::intrusive_ptr >,
>
Just a datapoint - we upgraded several large Mimic-born clusters straight
to 15.2.12 with the quick fsck disabled in ceph.conf, then did
require-osd-release, and finally did the omap conversion offline after the
cluster was upgraded using the bluestore tool while the OSDs were down (all
done in
You seem to have an OSD that's down and out (status says 9 OSDs, 8 up and
in). One possibility is that the pg is not able to fully recover because of
existing CRUSH rules and the virtue that the only OSD that could store the
last replica is down and out.
So, what do your CRUSH rules and
You're supposed to upgrade the mons first...
https://docs.ceph.com/en/quincy/releases/pacific/#upgrading-non-cephadm-clusters
Maybe try downgrading the mgrs back to Octopus? That's a bit of a scary
situation.
Tyler
On Wed, Jul 27, 2022, 1:24 PM wrote:
> Currently running Octopus 15.2.16,
Your first assumption was correct. You can set the 'size' parameter of the
pool to 2 (ceph osd pool set size 2), but you'll also want to either
want to drop min_size to 1 or accept the fact that you cannot ever have
either metadata OSD go down. It's fine for a toy cluster, but for any
production
Ceph always aims to provide high availability. So, if you do not set
cluster flags that prevent Ceph from trying to self-heal, it will self-heal.
Based on your description, it sounds like you want to consider the 'noout'
flag. By default, after 10(?) minutes of an OSD being down, Ceph will begin
What does 'ceph mon dump | grep min_mon_release' say? You're running
msgrv2 and all Ceph daemons are talking on v2, since you're on
Nautilus, right?
Was the cluster conceived on Nautilus, or something earlier?
Tyler
On Sun, Mar 20, 2022 at 10:30 PM Clippinger, Sam
wrote:
>
> Hello!
>
> I need
You can upgrade straight from Mimic to Octopus without a full outage window.
One thing to keep in mind that wasn't spelled out clearly in the docs (I
thought): you will have to bounce all components twice to get the full
benefits of MSGRv2. After you bounce everything in the correct order and
run
They maintain and distribute the OSD map, i.e. the state of the OSDs.
Generally 3 or 5 recommended as there always needs to be a quorum of mons
available or the cluster can stall.
With 3 mons, you can bring down one for servicing, patching, etc. but if
another mon fails while you intentionally
I would still set noout on relevant parts of the cluster in case something
goes south and it does take longer than 2 minutes. Otherwise OSDs will
start outing themselves after 10 minutes or so by default and then you have
a lot of churn going on.
The monitors monitors will be fine unless you lose
I'm curious to hear if anyone has looked into kernel scheduling tweaks
or changes in order to improve qd=1/bs=4k performance (while we
patiently wait for Seastar!)
Using this toy cluster:
3x OSD nodes: Atom C3758 (8 core, 2.2GHz) with 1x Intel S4500 SSD each
Debian Bullseye with Linux 5.15.14 and
31 matches
Mail list logo
