This is happening to our 16.2.14 cluster as well. I'm not sure whether this
was happening before the upgrade to 16.2.14.
/Z
On Thu, 31 Aug 2023, 17:49 Adrien Georget,
wrote:
> Hello,
>
> On our 16.2.14 CephFS cluster, all OSDs are spamming logs with messages
> like "log_channel(cluster) log
It looks much better, at least for Ubuntu focal, thanks!
/Z
On Thu, 31 Aug 2023 at 03:48, Yuri Weinstein wrote:
> We redeployed all packages again.
>
> Please confirm that the issue is resolved.
>
> Thank you for your help and patience!
>
> On Wed, Aug 30, 2023 at 3:44
Now the release email comes and the repositories are still missing
packages. What a mess.
/Z
On Wed, 30 Aug 2023 at 19:27, Yuri Weinstein wrote:
> 16.2.14 has not been released yet.
>
> Please don't do any upgrades before we send an announcement email.
>
> TIA
>
> On Wed, Aug 30, 2023 at 8:45
Hi,
Please note that some packages have been pushed for Ubuntu focal as well,
but the repo is incomplete. I think it would be good if such things could
be avoided in the future.
/Z
On Wed, 30 Aug 2023 at 19:27, Yuri Weinstein wrote:
> 16.2.14 has not been released yet.
>
> Please don't do any
Hi,
You can use the following formula to roughly calculate the IOPS you can get
from a cluster: (Drive_IOPS * Number_of_Drives * 0.75) / Cluster_Size.
For example, for 60 10K rpm SAS drives each capable of 200 4K IOPS and a
replicated pool with size 3: (~200 * 60 * 0.75) / 3 = ~3000 IOPS with
to
167:167, which in my system is the user ID crash process uses inside the
crash container.
/Z
On Mon, 5 Jun 2023 at 11:24, Zakhar Kirpichenko wrote:
> Any other thoughts on this, please? Should I file a bug report?
>
> /Z
>
> On Fri, 2 Jun 2023 at 06:11, Zakhar Kirpichenko wro
Any other thoughts on this, please? Should I file a bug report?
/Z
On Fri, 2 Jun 2023 at 06:11, Zakhar Kirpichenko wrote:
> Thanks, Josh. The cluster is managed by cephadm.
>
> On Thu, 1 Jun 2023, 23:07 Josh Baergen, wrote:
>
>> Hi Zakhar,
>>
>> I'm going to
e the directory permissions, assuming that you manage
> the directories yourself. If this is managed by cephadm or something like
> that, then that seems like some sort of missing migration in the upgrade.
>
> Josh
>
> On Thu, Jun 1, 2023 at 12:34 PM Zakhar Kirpichenko
> wrote
Hi,
I'm having an issue with crash daemons on Pacific 16.2.13 hosts. ceph-crash
throws the following error on all hosts:
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist;
please create
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist;
please create
Thanks!
An upgrade from 16.2.12 on Ubuntu 20.04 LTS went smoothly.
/Z
On Wed, 10 May 2023 at 00:45, Yuri Weinstein wrote:
> We're happy to announce the 13th backport release in the Pacific series.
>
> https://ceph.io/en/news/blog/2023/v16-2-13-pacific-released/
>
> Notable Changes
>
he perf counters and leave the cluster
> run for 30 mins and collect perf counters again.
>
> - wait 24 hours and collect the counters one more time
>
> - share all four counters snapshots.
>
>
> Thanks,
>
> Igor
>
> On 5/8/2023 11:31 PM, Zakhar Kirpichenko wrote:
&g
Don't mean to hijack the thread, but I may be observing something similar
with 16.2.12: OSD performance noticeably peaks after OSD restart and then
gradually reduces over 10-14 days, while commit and apply latencies
increase across the board.
Non-default settings are:
Periodically we observe OSD perf drop due to degraded RocksDB
> performance, particularly after bulk data removal/migration.. Compaction
> is quite helpful in this case.
>
>
> Thanks,
>
> Igor
>
>
>
> On 26/04/2023 20:22, Zakhar Kirpichenko wrote:
> > Hi,
>
-primary and others would take over. If the new
> primary OSDs show increased latencies as well there might be something
> else going on.
>
> Zitat von Zakhar Kirpichenko :
>
> > Thanks, Eugen. I very much appreciate your time and replies.
> >
> > It's a hybrid OSD
at OSD (ceph pg
> ls-by-osd )? Have you tried to restart and/or compact the OSD and
> see if anything improves?
> You could set its primary-affinity to 0, or the worst case rebuild
> that OSD. And there are no smart errors or anything in dmesg reported
> about this disk?
>
> Zi
at": "2023-04-27T07:37:35.046036+",
> "age": 54.80501619997,
> "duration": 0.5819846869995,
> ...
>
> The output contains the PG (so you know which pool is involved) and
> the duration of the operation, not su
f
13:f6c9079e:::rbd_data.eed629ecc1f946.001c:head [stat,write
4018176~4096] snapc 0=[] ondisk+write+known_if_redirected e118835)",
"initiated_at": "2023-04-26T07:00:58.299270+",
There's a lot more information there ofc. I also tried to
`dump_o
Hi,
I have a Ceph 16.2.12 cluster with hybrid OSDs (HDD block storage, DB/WAL
on NVME). All OSD settings are default except, cache-related settings are
as follows:
osd.14dev bluestore_cache_autotune true
osd.14dev bluestore_cache_size_hdd
Hi,
I have a Ceph 16.2.12 cluster with uniform hardware, same drive make/model,
etc. A particular OSD is showing higher latency than usual in `ceph osd
perf`, usually mid to high tens of milliseconds while other OSDs show low
single digits, although its drive's I/O stats don't look different from
ultiple times after the reboot.
/Z
On Thu, 26 Jan 2023 at 05:41, Konstantin Shalygin wrote:
> Hi Zakhar,
>
> On 26 Jan 2023, at 08:33, Zakhar Kirpichenko wrote:
>
> Jan 25 23:07:53 ceph01 bash[2553123]:
>
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AV
Hi,
Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started
crashing in a loop on the very 1st host:
Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval
Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval
0.000
Jan 25 23:07:53 ceph01
erver and client versions is
unsupported and may lead to anomalous behavior."
As pointed out by a kind soul on reddit.
/Z
On Mon, 5 Dec 2022 at 06:01, Zakhar Kirpichenko wrote:
> Hi!
>
> I'm planning to upgrade our Ceph cluster from Pacific (16.2.10) to Quincy
> (17.2.x). The cluster
Hi!
I'm planning to upgrade our Ceph cluster from Pacific (16.2.10) to Quincy
(17.2.x). The cluster is used for Openstack block storage (RBD), Openstack
version is Wallaby built on Ubuntu 20.04.
Is anyone using Ceph Quincy (17.2.x) with Openstack Wallaby? If you are,
please let me know if you've
Thanks, Stefan. I've read this very well.
My question is whether there's anything not covered by the available
documentation that we should be aware of.
/Z
On Fri, 25 Nov 2022 at 09:11, Stefan Kooman wrote:
> On 11/24/22 18:53, Zakhar Kirpichenko wrote:
> > Hi!
> >
> > I
how much snapshots in chain for this image.
> More than 10 snaps for one image can increase client ops latency to tens
> milliseconds even for NVMe drives, that usually operates at usec or 1-2msec
>
>
> k
> Sent from my iPhone
>
> > On 7 Oct 2022, at 14:35, Zakhar Kirpichen
t;
> Sent from my iPhone
>
> > On 7 Oct 2022, at 07:33, Zakhar Kirpichenko wrote:
> >
> > Anyone, please?
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
Hi,
I'm having a peculiar "issue" in my cluster, which I'm not sure whether
it's real: a particular OSD always shows significant latency in `ceph osd
perf` report, an order of magnitude higher than any other OSD.
I traced this OSD to a particular drive in a particular host. OSD logs
don't look
Many thanks, Dan. Much appreciated!
/Z
On Thu, 14 Jul 2022 at 08:43, Dan van der Ster wrote:
> Yes, that is correct. No need to restart the osds.
>
> .. Dan
>
>
> On Thu., Jul. 14, 2022, 07:04 Zakhar Kirpichenko,
> wrote:
>
>> Hi!
>>
>> M
Hi!
My apologies for butting in. Please confirm
that bluestore_prefer_deferred_size_hdd is a runtime option, which doesn't
require OSDs to be stopped or rebuilt?
Best regards,
Zakhar
On Tue, 12 Jul 2022 at 14:46, Dan van der Ster wrote:
> Hi Igor,
>
> Thank you for the reply and information.
> > I remember this has been discussed a couple of times in the past on
> > this list, but I'm wondering if this still happens in newer releases.
> > I assume there's no way of preventing that, so we'll probably go with
> > the safe approach on the next node. It's a pr
Hi Eugen,
Can you please elaborate on what you mean by "restarting the primary PG"?
Best regards,
Zakhar
On Wed, Apr 6, 2022 at 5:15 PM Eugen Block wrote:
> Update: Restarting the primary PG helped to bring the PGs back to
> active state. Consider this thread closed.
>
>
> Zitat von Eugen
Hi!
I run cephadm-based 16.2.x cluster in production. It's been mostly fine,
but not without quirks. Hope this helps.
/Z
On Tue, Mar 8, 2022 at 6:17 AM norman.kern wrote:
> Dear Ceph folks,
>
> Anyone is using cephadm in product(Version: Pacific)？ I found several bugs
> on it and
> I really
you have some pools with size = 2 and min_size = 2?
>
> Regards,
> Eugen
>
>
> Zitat von Zakhar Kirpichenko :
>
> > Hi!
> >
> > Sometimes when we upgrade our cephadm-managed 16.2.x cluster, cephadm
> > decides that it's safe to upgrade a bunch of OSDs at a ti
Hi!
Sometimes when we upgrade our cephadm-managed 16.2.x cluster, cephadm
decides that it's safe to upgrade a bunch of OSDs at a time, as a result
sometimes RBD-backed Openstack VMs appear to get I/O stalls and read-only
filesystems. Is there a way to make cephadm upgrade fewer OSDs at a time,
or
Hi!
I have a Ceph cluster version 16.2.7 built with cephadm. Ceph logs in
/var/log/ceph/ all rotate nicely, but container logs in
/var/lib/docker/containers/ keep growing. For example, a log of a monitor
container has reached almost 3 GB since December 2021:
09:18 [root@ceph02
e block db.
>
> Currently we are planning to install new OSD with SATA-SSD. so block-db
> will not be required.
>
> we will use device-class and different replicated_rule
>
> $ceph osd crush set-device-class SSD osd.10
> $ceph osd crush rule create-replicated replicated_rule_SSD default hos
Hi!
If you use SSDs for OSDs, there's no real benefit from putting DB/WAL on a
separate drive.
Best regards,
Z
On Sun, Dec 5, 2021 at 10:15 AM Md. Hejbul Tawhid MUNNA
wrote:
> Hi,
>
> We are running openstack cloud with backend ceph storage. Currently we have
> only HDD storage in our ceph
;
> Weiwen Hu
>
>
>
> 从 Windows 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>发送
>
>
>
> *发件人: *Zakhar Kirpichenko
> *发送时间: *2021年11月11日 20:54
> *收件人: *ceph-users
> *主题: *[ceph-users] Pacific: parallel PG reads?
>
>
>
> Hi,
>
> I'm sti
Hi,
On Thu, Nov 11, 2021 at 3:26 PM Janne Johansson wrote:
> Den tors 11 nov. 2021 kl 13:54 skrev Zakhar Kirpichenko >:
> > I'm still trying to combat really bad read performance from HDD-backed
> > replicated pools, which is under 100 MB/s most of the time with 1 thread
>
Hi,
I'm still trying to combat really bad read performance from HDD-backed
replicated pools, which is under 100 MB/s most of the time with 1 thread
and QD=1. I don't quite understand why the reads are that slow, i.e. much
slower than a single HDD, but do understand that Ceph clients read a PG
Hi,
I have a 6-node cluster running Pacific 16.2.6 with 54 x 10TB HDD and 12 x
6.4 TB NVME drives. By default, the autoscaler appears to scale down each
pool to 32 PGs, which causes a very uneven data distribution and somewhat
lower performance.
Although I knew that previously there was a neat
oda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
>
> -----Original Message-
> From: Zakhar Kirpichenko
> Sent: Friday, November 5, 2021 6:45 PM
> To: ceph-users
> Subject: [ceph-users] Re: Optimal Erasure Code
the simple Host failure domain on 6 aktive servers and “k=4, m=2”
> erasure coding.
> But I don't advise you to do that!
>
>
> Best
> Sebastian
>
>
> > On 05.11.2021, at 06:14, Zakhar Kirpichenko wrote:
> >
> > Hi!
> >
> > I've got a CEPH 16.
more clear that
adding a cache tier isn't a good idea.
/Z
On Fri, Nov 5, 2021 at 7:55 AM Anthony D'Atri
wrote:
> Cache tiers are kind of deprecated, they’re finicky and easy to run into
> trouble with. Suggest avoiding.
>
> > On Nov 4, 2021, at 10:42 PM, Zakhar Kirpichenko
> wrote:
&g
Hi,
I seem to have some stale monitoring alerts in my Mgr UI, which do not want
to go away. For example (I'm also attaching an image for your convenience):
MTU Mismatch: Node ceph04 has a different MTU size (9000) than the median
value on device storage-int.
The alerts appears to be active, but
Hi,
I'm trying to figure out if setting auth caps and/or adding a cache pool
are I/O-disruptive operations, i.e. if caps reset to 'none' for a brief
moment or client I/O momentarily stops for other reasons.
For example, I had the following auth setting in my 16.2.x cluster:
client.cinder
Hi!
I've got a CEPH 16.2.6 cluster, the hardware is 6 x Supermicro SSG-6029P
nodes, each equipped with:
2 x Intel(R) Xeon(R) Gold 5220R CPUs
384 GB RAM
2 x boot drives
2 x 1.6 TB enterprise NVME drives (DB/WAL)
2 x 6.4 TB enterprise drives (storage tier)
9 x 9TB HDDs (storage tier)
2 x Intel
> From: Etienne Menguy
> Sent: Tuesday, November 2, 2021 3:17 PM
> To: Zakhar Kirpichenko
> Cc: ceph-users
> Subject: [ceph-users] Re: Best way to add multiple nodes to a cluster?
>
> Email received from the internet. If in doubt, don't click any link nor
> open any atta
Hi!
I have a 3-node 16.2.6 cluster with 33 OSDs, and plan to add another 3
nodes of the same configuration to it. What is the best way to add the new
nodes and OSDs so that I can avoid a massive rebalance and performance hit
until all new nodes and OSDs are in place and operational?
I would very
Hi,
It is my experience with Bluestore that even if you put DB/WAL on a very
fast SSD (NVME in my case), the OSD will still refuse to write faster than
the storage device (HDD) can write on average. This is a bummer because the
behavior is very different from Filestore, which could buffer write
Hi!
Although I'm not sure why you'd want to do it, there's a thick provision
option for RBD create, i.e. it is possible:
$ rbd help create | grep thick
--thick-provision fully allocate storage and zero image
Note that fully allocating and zeroing the image means writing data to
every
Hi,
RBD is a block device and is suitable for running any application, but its
performance depends on many factors and it has significant overheads. What
you need to determine is whether RBD on your particular hardware with your
particular settings provides satisfactory random I/O performance and
he risk of an in place upgrade.
>
>
>
> On Tue, Oct 12, 2021 at 2:22 PM Igor Fedotov
> wrote:
>
>> Zakhar,
>>
>> could you please point me to the similar reports at Proxmox forum?
>>
>> Curious what's the Ceph release mentioned there...
>>
x
> forum posts as well. I'm not sure if going to the 5.4 kernel will create
> any other challenges for us, as we're using dual port mellanox connectx-6
> 200G nics in the hosts, but it is definitely something we can try.
>
> Marco
>
> On Tue, Oct 12, 2021 at 1:53 PM Zakhar
Hi,
This could be kernel-related, as I've seen similar reports in Proxmox
forum. Specifically, 5.11.x with Ceph seems to be hitting kernel NULL
pointer dereference. Perhaps a newer kernel would help. If not, I'm running
16.2.6 with kernel 5.4.x without any issues.
Best regards,
Z
On Tue, Oct
Hi,
I tried experimenting with RBD striping feature:
rbd image 'volume-bd873c3f-c8c7-4270-81f8-951f65fc860c':
size 50 GiB in 12800 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 187607b9ebf28a
block_name_prefix: rbd_data.187607b9ebf28a
format: 2
features: layering, striping,
These are valid points, thank you for the input!
/Z
On Wed, Oct 6, 2021 at 11:39 AM Stefan Kooman wrote:
> On 10/6/21 09:23, Zakhar Kirpichenko wrote:
> > Hi,
> >
> > Indeed, that's a lot of CPU and RAM, the idea was to put sufficient
> > resources in case we want to
, 2021 at 11:16 AM Janne Johansson wrote:
> Den ons 6 okt. 2021 kl 10:11 skrev Zakhar Kirpichenko :
> >
> > I've initially disabled power-saving features, which nicely improved the
> > network latency.
> >
> > Btw, the first interesting find: I enabled 'rbd_balance
I've initially disabled power-saving features, which nicely improved the
network latency.
Btw, the first interesting find: I enabled 'rbd_balance_parent_reads' on
the clients, and single-thread reads now scale much better, I routinely get
similar readings from a single disk doing 4k reads with 1
:06, Zakhar Kirpichenko wrote:
> > Hi,
> >
> > I built a CEPH 16.2.x cluster with relatively fast and modern hardware,
> and
> > its performance is kind of disappointing. I would very much appreciate an
> > advice and/or pointers :-)
> >
> > The hardware
disabling the HDD write cache can
> actually improve latency quite substantially (e.g.
> https://ceph-users.ceph.narkive.com/UU9QMu9W/disabling-write-cache-on-sata-hdds-reduces-write-latency-7-times)
> - might be worth a try
>
>
> On Wed, 6 Oct 2021 at 10:07, Zakhar Kirpichenko w
Actually recent versions of virtio-blk support discard as well. In general,
I found that virtio-scsi isn't worth it. I appreciate your input though,
thanks!
/Z
On Wed, 6 Oct 2021, 09:03 Janne Johansson, wrote:
> Den ons 6 okt. 2021 kl 06:37 skrev Zakhar Kirpichenko :
> > Got it. I d
ume / attachment / VM, or enables the client-side admin socket, that
> sort of throttling can be very visually apparent.
>
>
> > On Oct 5, 2021, at 8:35 PM, Zakhar Kirpichenko wrote:
> >
> > Hi!
> >
> > The clients are KVM VMs, there's QEMU/libvirt impact for sure. I
ndreds of PGs. But QD=1 and small block sizes are going to
> limit your throughput.
>
> What are your clients? Are they bare metal? Are they VMs? If they’re
> VMs, do you have QEMU/libvirt throttling in play? I see that a lot.
>
> > On Oct 5, 2021, at 2:06 PM, Zakhar Kirp
y math 7k IOPS at 4k should only be
>> 27MiB/sec - not sure how the 120MiB/sec was achieved
>> The read benchmark seems in line with 13k IOPS at 4k making around
>> 52MiB/sec bandwidth which again is expected.
>>
>>
>> On Wed, 6 Oct 2021 at 04:08, Zakhar Kirpichenko
eved
> The read benchmark seems in line with 13k IOPS at 4k making around
> 52MiB/sec bandwidth which again is expected.
>
>
> On Wed, 6 Oct 2021 at 04:08, Zakhar Kirpichenko wrote:
>
>> Hi,
>>
>> I built a CEPH 16.2.x cluster with relatively fast and
Aren't all writes to bluestore turned into sequential writes?
/Z
On Tue, 5 Oct 2021, 20:05 Marc, wrote:
>
> Hi Zakhar,
>
> > using 16 threads) are not. Literally every storage device in my setup
> > can read and write at least 200+ MB/s sequentially, so I'm trying to
> > find an explanation
Hi Marc,
Many thanks for your comment! As I mentioned, rados bench results are more
or less acceptable and explainable. RBD clients writing at ~120 MB/st tops
(regardless of the number of threads or block size btw) and reading ~50
MB/s in a single thread (I managed to read over 500 MB/s using 16
Hi,
I built a CEPH 16.2.x cluster with relatively fast and modern hardware, and
its performance is kind of disappointing. I would very much appreciate an
advice and/or pointers :-)
The hardware is 3 x Supermicro SSG-6029P nodes, each equipped with:
2 x Intel(R) Xeon(R) Gold 5220R CPUs
384 GB
our VMs was almost impossible. But this
> is an RBD cache so I can't tell how the other protocols perform with a
> cache tier.
>
>
> Zitat von Zakhar Kirpichenko :
>
> > Hi,
> >
> > You can arbitrarily add or remove the cache tier, there's no problem with
> &
Hi,
You can arbitrarily add or remove the cache tier, there's no problem with
that. The problem is that cache tier doesn't work well, I tried it in front
of replicated and EC-pools with very mixed results: when it worked there
wasn't as much of a speed/latency benefit as one would expect from
101 - 171 of 171 matches
Mail list logo
