_recovery_max_active is default to 3.
osd_recovery_op_priority is default to 3, a lower number increases the
priority for recovery.
All of them can be runtime adjusted.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe s
,
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iamuser/"
]
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::*"
}
]
}
And then run
s3cmd setpolicy s3-policy.json s3://
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
On Mon, Mar 25, 2024 at 10:58:24PM +0100, Kai Stian Olstad wrote:
On Mon, Mar 25, 2024 at 09:28:01PM +0100, Torkil Svensgaard wrote:
My tally came to 412 out of 539 OSDs showing up in a blocked_by list
and that is about every OSD with data prior to adding ~100 empty
OSDs. How 400 read targets
to the new OSD but also between the existing OSD, but
they are busy with backfilling so they have no free backfill reservation.
I do recommend this document by Joshua Baergen at Digital Ocean that explains
backfilling and the problem with it and there solution, a tool called
pgremapper.
--
Kai Stian
On Sat, Mar 23, 2024 at 12:09:29PM +0100, Torkil Svensgaard wrote:
The other output is too big for pastebin and I'm not familiar with
paste services, any suggestion for a preferred way to share such
output?
You can attached files to the mail here on the list.
--
Kai Stian Olstad
ecause of the 500 limit.
So only the SSD got the osd_mclock_max_capacity_iops_ssd set.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
osd_mclock_max_capacity_iops_{hdd,ssd} every time
the OSD is started.
If you check the OSD log you'll see it does the bench.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
, wait for recovery and the run deep-scrub to
check if things look better.
But would it then use corrupted data on osd 269 to rebuild.
-
Kai Stian Olstad
On 26.02.2024 10:19, Eugen Block wrote:
Hi,
I think your approach makes sense. But I'm wondering if moving only
the problematic PGs
, wait for recovery and the run deep-scrub to
check if things look better.
But would it then use corrupted data on osd 269 to rebuild.
-
Kai Stian Olstad
On 26.02.2024 10:19, Eugen Block wrote:
Hi,
I think your approach makes sense. But I'm wondering if moving only
the problematic PGs
they are now down from the remaining
shards
s1(osd.297), s3(osd.276), s4(osd.136) and s5(osd.197)
_Any_ comment is highly appreciated.
-
Kai Stian Olstad
On 21.02.2024 13:27, Kai Stian Olstad wrote:
Hi,
Short summary
PG 404.bc is an EC 4+2 where s0 and s2 report hash mismtach for 698
objects
.
- I am asking because I need to use a local registry to deploy
those
images.
You need to check the cephadm source for the version you would like to
use
https://github.com/ceph/ceph/blob/v17.2.7/src/cephadm/cephadm#L46
--
Kai Stian Olstad
2.0.46 - test1 [21/Feb/2024:08:27:06.021 +] "GET
/benchfiles/2021-11-08T19%3A43%3A50%2C145489260%2B00%3A00 HTTP/1.1" 500
226 - - - latency=0.02568s
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/IPHBE3DLW5ABCZHSNYOBUBSI3TLWVD22/#OE3QXLAJIY6
is 4+2 jerasure blaum_roth and the HDD is hybrid
where DB is on SSD shared by 5 HDD.
And in your cases?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
can find some documentation on octupus
(https://docs.ceph.com/en/octopus/install/ceph-deploy/install-ceph-gateway/)
ceph-deploy is deprecated
https://docs.ceph.com/en/reef/install/
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
/IPHBE3DLW5ABCZHSNYOBUBSI3TLWVD22/#OE3QXLAJIY6NU7PNMGHP47UK2CBZJPUG
-
Kai Stian Olstad
On Tue, Feb 06, 2024 at 06:35:26AM -, LeonGao wrote:
Hi community
We have a new Ceph cluster deployment with 100 nodes. When we are draining an
OSD host from the cluster, we see a small amount of PGs that cannot
On 31.01.2024 09:38, garcetto wrote:
good morning,
how can i install latest dev release using cephadm?
Have you looked at this page?
https://docs.ceph.com/en/latest/install/containers/#development-builds
--
Kai Stian Olstad
___
ceph-users mailing
cluster.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
nk you Wes, switching to wpq and restart the OSDs fixed it for me
too.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
Hi,
This is a cluster running 17.2.7 upgraded from 16.2.6 on the 15 January
2024.
On Monday 22 January we had 4 HDD all on different server with I/O-error
because of some damage sectors, the OSD is hybrid so the DB is on SSD, 5
HDD share 1 SSD.
I set the OSD out, ceph osd out 223 269 290
and specify it,
some details are here
https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-environment
The containers themself don't need to download anything at start.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
and
the admin key.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
incy/rados/operations/monitoring-osd-pg/#monitoring-pg-states
https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-pg/
https://docs.ceph.com/en/latest/dev/placement-group/#user-visible-pg-states
--
Kai Stian Olstad
___
ceph-users mailing li
the whole host. I would very much
appreciate any suggestions.
If you look closely at 53906 you'll see it's a duplicate of
https://tracker.ceph.com/issues/53907
In there you have the fix and a workaround until next minor is released.
--
Kai Stian Olstad
in install
What should I do?
The easiest is use tell, then you can run it on any node that have access to
ceph.
ceph tell osd.8 dump_historic_ops
ceph tell osd.8 help
will give you all you can do with tell.
--
Kai Stian Olstad
___
ceph-users ma
in the documentation, have you tried that?
https://docs.ceph.com/en/reef/radosgw/config-ref/#lifecycle-settings
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
-10-09T07:59:01.303+
This log say that it's bash with PID 4019 that is creating the log
entry.
Maybe start there, check what what other thing you are running on the
server that creates this messages.
--
Kai Stian Olstad
___
ceph-users mailing list
nice script that do the
steps Wes explained above
https://github.com/cernceph/ceph-scripts/blob/master/tools/scrubbing/autorepair.sh
that you might want to take a look at.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe s
, but with so large disk you could or maybe should
consider having a higher number of PGs per OSD.
[1]
https://docs.ceph.com/en/quincy/rados/operations/placement-groups/#viewing-pg-scaling-recommendations
--
Kai Stian Olstad
___
ceph-users mailing list
domain set to
host, it's my understanding no data will be migrated/backfilled when a node
fails.
The reason is that there is nowhere to copy the data to, to fulfill the crush
rule
one copy on 3 different hosts.
--
Kai Stian Olstad
___
ceph-users mailing
mand to do this
ceph config-key get mgr/cephadm/spex.rgw.pech | jq -r
.spec.spec.rgw_frontend_ssl_certificate | ceph config-key set rgw/cert/rgw.pech
-
ceph orch restart rgw.pech
My claim is that Ceph should update "rgw/cert/rgw.pech" when
"mgr/cephadm/spex.rgw.pech" is
rgw_max_concurrent_requests to 10240
https://www.seagate.com/content/dam/seagate/migrated-assets/www-content/solutions/partners/red-hat/_shared/files/st-seagate-rhcs5-detail-f29951wg-202110-en.pdf
So I think the only way to find out it to increase it and see what happens.
--
Kai Stian Olstad
PLACEMENT
rgw.pech ?:443 7/7 4m ago 22h label:cog
But running
ceph orch redeploy rgw.pech
will redeploy all 7 RGW, and would be the same as
ceph orch daemon redeploy rgw.pech.pech-mon-3.upnvrd
but only redeploy one of them.
From: Kai Stian Olstad
The certificate is about
, but could be increase it to
2048.
But the recommended action was to add more gateways instead.
[1]
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_max_concurrent_requests
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
ones about to expire.
I have looked in the documentation and can't find how to update the certificate
for RGW.
Can anyone shed some light on how to replace the certificate?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
On Wed, Mar 01, 2023 at 08:39:56AM -0500, Daniel Gryniewicz wrote:
We're actually writing this for RGW right now. It'll be a bit before
it's productized, but it's in the works.
Just curious, what is the use cases for this feature?
S3 against CephFS?
--
Kai Stian Olstad
Hi
Just a follow up, the issue was solved by running command
ceph pg 404.1ff mark_unfound_lost delete
-
Kai Stian Olstad
On 04.01.2023 13:00, Kai Stian Olstad wrote:
Hi
We are running Ceph 16.2.6 deployed with Cephadm.
Around Christmas OSD 245 and 327 had about 20 read error so I set
oes anyone have any idea on how to resolv the problem?
Any help is much appreciated.
-
Kai Stian Olstad
::
ceph-s.txt
::
ceph -s
---
cluster:
id: d13c6b81-51ee-4d22-84e9-456f9307296c
health: HEALTH_ERR
1/416287125 objects
this by looking at how Openstack does it and some trial and
error.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
On 16.11.2022 13:21, E Taka wrote:
gmail marks too many messages on this mailing list as spam.
You can fix that by creating a filter in Gmail for ceph-users@ceph.io
and check the "Never send it to Spam".
--
Kai Stian Olstad
___
ceph-use
On 16.11.2022 00:25, Daniel Brunner wrote:
are my mails not getting through?
is anyone receiving my emails?
You can check this yourself by checking the archives
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/
If you see your mail there, they are getting through.
--
Kai Stian
ort SMART or any status on SAS disk today, I only get
the message "No SMART data available".
I have gathered "smartctl -x --json=vo" log for the 6 types of SAS this
I have in my possession.
You can find them here if interested [1]
[1] https://gitlab.com/-/s
eyring?
And is the keyring file readable for the user running the ceph commands?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
.")[0] }}
This works because the first part of the instance name is the hostname.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
quot; and check status every 2 seconds for 2 minutes, no
degraded messages.
The only new messages in ceph -s was 12 remapped pgs and "11
active-remapped+backfilling" and "1 active+remapped+backfill_wait"
Previously I had to set all osd(15 disks) on a host to out and there was
n
, but the degradation problems
also
go away when we turn off the balancer.
So this goes a long way of confirming there are something wrong with the
balancer since we now see it on two different installation.
--
Kai Stian Olstad
___
ceph-users
it will set the correct
permissions.
# ceph auth get-or-create client.user2 mon 'profile rbd' osd 'profile rbd
pool=rbd namespace=user2' -o /etc/ceph/client.user2.keyring
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
request to me :)
Of course, I would never refer someone to send a feature request to a
person even if you were a Ceph developer, I would consider that rude,
the tracker exist for that :-)
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users
anyone know it's correct to return 416 on 0 byte object with range
or should this be considered a bug in Ceph.
[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users
On 15.03.2022 10:10, Jimmy Spets wrote:
Thanks for your reply.
I have two things that I am unsure of:
- Is the OSD UUID the same for all OSD:s or should it be unique for
each?
It's unique and generated when you run ceph-volume lvm prepare or add an
OSD.
You can find OSD UUID/FSID for
On 11.03.2022 14:04, Ilya Dryomov wrote:
On Fri, Mar 11, 2022 at 8:04 AM Kai Stian Olstad
wrote:
Isn't namespace supported with erasure encoded pools?
RBD images can't be created in EC pools, so attempting to create RBD
namespaces there is pointless. The way to store RBD image data
son --osd-fsid
517213f3-0715-4d23-8103-6a34b1f8ef08
And then the OSD should be back up and running.
This is the way I have found to do OSD replacement, it might be an
easier way of doing it but I have not found that.
--
Kai Stian Olstad
___
ceph-users
to add namespace: (95) Operation not supported
Isn't namespace supported with erasure encoded pools?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
> no user data at all.
When you have DB on a separate disk the DB size count towards total size of the
osd. But this DB space is considered used so you will see a lot of used space.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@cep
the change
I can't find the feed URL.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
/install/index.html
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
sue
https://tracker.ceph.com/issues/51667
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
etwork.
[1] https://github.com/ceph/ceph/blob/v16.2.7/src/cephadm/cephadm#L50-L61
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
and is empty except for one pool with one
PG created by Cephadm.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
of the cores was
saturated,
and network statistics show no errors or drops.
So perhaps there is a bug in the balancer code?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
On 16.09.2021 15:51, Josh Baergen wrote:
I assume it's the balancer module. If you write lots of data quickly
into the cluster the distribution can vary and the balancer will try
to even out the placement.
The balancer won't cause degradation, only misplaced objects.
Since I'm trying to test
data redundancy: 4315/7005845 objects degraded (0.062%), 2
pgs degraded, 4 pgs undersized
So my question is, it normal that Ceph report degraded under normal use?
or do I have a problem somewhere that I need to investigate?
--
Kai Stian Olstad
___
c
issues/new ?
Found a issue reported under MGR
https://tracker.ceph.com/issues/52028 - mgr/dashboard: Incorrect MTU
mismatch warning
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
a difference but haven’t gotten around to
it.
If you look at the message it says which interface it is.
It does check and report on all the interfaces, even those that is in
DOWN state which it shouldn't.
--
Kai Stian Olstad
___
ceph-users mailing list
Hi Peter
Please remember to include the list address in your reply.
I will not trim so people on the list can read you answer.
On 29.07.2021 12:43, Peter Childs wrote:
On Thu, 29 Jul 2021 at 10:37, Kai Stian Olstad
wrote:
A little disclaimer, I have never used multipath with Ceph
tipath device
paths and not the individual disk the OS sees.
[1] https://docs.ceph.com/en/latest/cephadm/osd/#creating-new-osds
[2]
https://docs.ceph.com/en/latest/cephadm/osd/#advanced-osd-service-specifications
--
Kai Stian Olstad
___
ceph-users
On 22.07.2021 13:56, Kai Stian Olstad wrote:
Hi
I have a warning that says
"1 stray daemon(s) not managed by cephadm"
What i did is the following.
I have 3 nodes that the mon should run on, but because of a bug in
16.2.4 I couldn't run on then since they are in differ
"osd|node|crash" | sort
"alertmanager.pech-mds-1"
"grafana.pech-cog-2"
"mgr.pech-mon-1.ptrsea"
"mgr.pech-mon-2.mfdanx"
"mon.pech-mon-1"
"mon.pech-mon-2"
"mon.pech-mon-3"
"prometheus.pech-mds-1"
No stray daemo
install/upgrade issue like this
has been allowed to sit)
You and me both.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
3-8952-eec3e4a2c178 -wi-ao 12.47t
osd-block-2da790bc-a74c-41da-8772-3b8aac77001c
ceph-block-1b5ad7e7-2e24-4315-8a05-7439ab782b45 -wi-ao 12.47t
The fist one is the new OSD and the second one is one that cephadm
itself created.
--
Kai Stian Olstad
___
on one node. This spec file results in three
standalone OSDs! Without the other filters this won't work as
expected, it seems. I'll try again on Pacific with the same test and
see where that goes.
This spec did worked for me when I initially deployed with Octopus
15.2.5.
--
Kai Stian Olstad
g volume group
backup
"/etc/lvm/backup/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b"
(seqno 62).
INFO:cephadm:/usr/bin/podman:stderr --> Zapping successful for: /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d69>
-
ocked
pech-hd-7 /dev/sds hdd SEAGATE ST14000NM016813.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
|DB |WAL |
+-+--+-+--++-+
|osd |hdd |pech-hd-7|/dev/sdt |- |-|
+-+--+-+--++-+
To me I looks like Cephadm can't use/find the free space on the VG and
use that as a new LV for the OSD.
--
Kai Stian Olstad
___
tried overwriting the LV with
/dev/zero, but that did solve it.
So I guess with wipe do you mean delete the LV with lvremove?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
is the wrong way to do it, but I can't find the answer in
the documentation.
So how can I replace this failed disk in Cephadm?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
On 11.03.2021 15:47, Sebastian Wagner wrote:
yes
Am 11.03.21 um 15:46 schrieb Kai Stian Olstad:
To resolve it, could I just remove it with "cephadm rm-daemon"?
That worked like a charm, and the upgrade is resumed.
Thank you Sebastian.
--
Kai St
ot;stopped",
"container_id": null,
"container_image_name":
"goharbor.example.com/library/ceph/ceph:v15.2.5",
"container_image_id": null,
"version": null,
"started": null,
"created": "2021-01-20T09:5
at the source code it looks like it get daemons to upgrade from
mgr cache, so I restarted both mgr but still it tries to upgrade
osd.355.
Does anyone know how I can get the upgrade to continue?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users
78 matches
Mail list logo
