Hi Joshua,
I asked like 1 year ago, and didn’t have good sense for setup from ceph
foundation.
Looks like all must be mailist or IRC but anyway I think the forum can be
setup without any permission. The community is free to share technical question
whatever we want or share resoures in a
Hi guys,
I remember some CEPH member deploy CEPH over nodes of 1U and 16 HDD 3,5.
Currently with the chip shortage Supermicro deliver new nodes in October...
Im looking for the model and brand of the 1U and 16 HDD I think is Asus or
Asrock server nodes... but someone can post the server
This same error caused us to wipe a full cluster of 300TB... will be related to
some rados index/database bug not to s3.
As Janek exposed is a mayor issue, because the error silent happend and you can
only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping
NoSuchKey.
Just ignore rgw.none is a old bug as far I investigated just a representation
bug .
New versions and newer buckets doesn't have anymore rgw.none, and right now
there's no way to remove section rgw.none.
Im at Nautilus 14.2.11 rgw.none is not present since several versions ago...
-Mensaje
, no previusly Smart errors...
Regards
Manuel
-Mensaje original-
De: Dan van der Ster
Enviado el: lunes, 17 de agosto de 2020 17:31
Para: EDH - Manuel Rios
CC: ceph-users
Asunto: Re: [ceph-users] OSD RGW Index 14.2.11 crash
Hi,
Do you have scsi errors around the time of the crash?
`journalctl
Hi , Today one of our SSD dedicated to RGW index crashed, maybe a bug or just
osd crashed.
Our current versión 14.2.11, today we're under heavy object process... aprox
60TB data.
ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus
(stable)
1:
Hi,
We got our cluster updated to the lastest versión 14.2.10
Checking rgw logs after 14.2.10 upgrade
2020-08-10 10:21:49.186 7f74cd7db700 1
RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing
requires read #1
2020-08-10 10:21:49.188 7f75eca19700 1
Hi,
Maybe this help, You can increase the osd_op_tp thread in ceph conf to
something similar to:
[osd]
osd_op_thread_suicide_timeout = 900
osd_op_thread_timeout = 300
osd_recovery_thread_timeout = 300
Regards
-Mensaje original-
De: Ingo Reimann
Enviado el:
You'r not the only one affected by this issue
As far as i know several huge companies hitted this bug too, but private
patches or tools are not public released.
This is caused for the a resharding process during upload in previous versions.
Workarround for us.:
- Delete objects of the bucket
Hi ,
Today checking the osd logs at boot after upgrade to 14.2.10 we found that:
set_numa_affinity unable to identify public interface 'p3p1.4094' numa node:
(2) No such file or directory
"2020-07-20 20:41:41.134 7f2cd15ca700 -1 osd.12 1120769 set_numa_affinity
unable to identify public
You can ignore rgw.none details, it dont make sense today from our experience
Still dont know why dev dont cleanup bucket with those rgw.none stats...
Some of our buckets got it others new ones no.
-Mensaje original-
De: Janne Johansson
Enviado el: martes, 30 de junio de 2020 8:40
, 29 de junio de 2020 6:14
Para: EDH - Manuel Rios
Asunto: Re: [ceph-users] rgw : unable to find part(s) of aborted multipart
upload of [object].meta
You can also check in default.rgw.buckets.non-ec pool for unmerged multipart or
bucket indexes that need to be fix.
On Mon, Jun 29, 2020 at 5:56 AM
Hi Dev's
With the failures of the previous versions in the buckets due to the shardings.
We have started a copy of the buckets to new buckets to clean our ceph cluster.
After synchronizing the bucket with the AWS cli, we are in the phase of
deleting the old buckets.
We have tried
Hi Andrei,
Orphans find code is not running. Will be deprecated in next reléase maybe
14.2.10
Check: https://docs.ceph.com/docs/master/radosgw/orphans/
Stop progress is bugged.
You got the same issue than us, multiparts are not being clean due a sharding
bugs.
Or fast solution for recover
Hi,
ATS ( Automatic Transfer Switch ) Works well.
We use in other services for single PSU server, they transfer the power from
source B to UPS in nano secs , preventing all services from going down.
You can get for 8A / 16A or 32 A , always monitorizable by SNMP , webinterface.
-Mensaje
Anyone can share their table with other MTU values?
Also interested into Switch CPU load
KR,
Manuel
-Mensaje original-
De: Marc Roos
Enviado el: miércoles, 27 de mayo de 2020 12:01
Para: chris.palmer ; paul.emmerich
CC: amudhan83 ; anthony.datri ;
ceph-users ; doustar ; kdhall
;
Hi Kamil,
We got a similar setup, and thats our config:
osd advanced osd_max_scrubs
1
osd advanced osd_recovery_max_active
4
osd advanced
Thx Mat for fast response, today night at datacenter adding more OSD for S3.
Will change the params and come back for share experience.
Regards
Manuel
-Mensaje original-
De: Matt Benjamin
Enviado el: domingo, 24 de mayo de 2020 22:47
Para: EDH - Manuel Rios
CC: ceph-users@ceph.io
Hi,
Im looking for any experience optimizing garbage collector with the next
configs:
global advanced rgw_gc_obj_min_wait
global advanced rgw_gc_processor_max_time
global advanced
: EDH - Manuel Rios
Enviado el: martes, 5 de mayo de 2020 15:34
Para: Katarzyna Myrek ; Eric Ivancich
CC: ceph-users@ceph.io
Asunto: [ceph-users] Re: RGW and the orphans
Hi Eric,
Expected version to be included your tool in Nautilus? Maybe next reléase?
Best Regards
Manuel
-Mensaje
Hi Eric,
Expected version to be included your tool in Nautilus? Maybe next reléase?
Best Regards
Manuel
-Mensaje original-
De: Katarzyna Myrek
Enviado el: lunes, 20 de abril de 2020 12:19
Para: Eric Ivancich
CC: EDH - Manuel Rios ; ceph-users@ceph.io
Asunto: Re: [ceph-users] RGW
Im prettty sure that you got the same issue than we already reported :
https://tracker.ceph.com/issues/43756
Garbage and garbage stored into our OSD without be able to cleanup wasting a
lot of space.
As you can see its solved in the new versions but... the last versión didnt
have any "scrub"
Hi Eric,
Are there any ETA for get those script backported maybe in 14.2.10?
Regards
Manuel
De: Eric Ivancich
Enviado el: jueves, 16 de abril de 2020 19:05
Para: Katarzyna Myrek ; EDH - Manuel Rios
CC: ceph-users@ceph.io
Asunto: Re: [ceph-users] RGW and the orphans
There is currently a PR
Hi,
>From my experience orphans find didn't work since several releases ago, and
>command should be re-coded or deprecated because its not running.
Im our cases it loops over generated shards until RGW daemon crash.
Interested into this post, in our case orphans find takes more than 24 hours
Hi,
Looks like you too hitted the bug:
https://tracker.ceph.com/issues/43756
&
https://tracker.ceph.com/issues/43583
We upgraded to 14.2.8 , but in our case we're still unable to clean multiparts.
Waiting to RGW team to provide a fix/workaround for the multiparts issue.
Regards
Manuel
Hi,
We're at 14.2.8, still got problems with abort multiparts.
This night we created a full list of objects with the string multipart like
Hi,
We have updated our cluster to 14.2.8 since we suffered the bug
https://tracker.ceph.com/issues/43583, now life cycle policies give more
information than before.
In 14.2.7 they ended instantly so something we have advanced. But they are not
yet able to eliminate multipart.
Just a line of
de 2020 20:47
Para: EDH - Manuel Rios
CC: ceph-users@ceph.io
Asunto: Re: [ceph-users] Bucket rename with
On 2/4/20 12:29 PM, EDH - Manuel Rios wrote:
> Hi
>
> Some Customer asked us for a normal easy problem, they want rename a bucket.
>
> Checking the Nautilus documentation l
Hi
Some Customer asked us for a normal easy problem, they want rename a bucket.
Checking the Nautilus documentation looks by now its not possible, but I
checked master documentation and a CLI should be accomplish this apparently.
$ radosgw-admin bucket link --bucket=foo --bucket-new-name=bar
rebalancing disk, that cause some objects
moves to other osd but you can so fast space available.
Regards
De: German Anders
Enviado el: martes, 4 de febrero de 2020 14:20
Para: EDH - Manuel Rios
CC: ceph-us...@ceph.com
Asunto: Re: [ceph-users] Doubt about AVAIL space on df
Hi Manuel,
Sure thing
Hi German,
Can you post , ceph osd df tree ?
Looks like your usage distribution is not perfect and that's why you got less
space than real.
Regards
-Mensaje original-
De: German Anders
Enviado el: martes, 4 de febrero de 2020 14:00
Para: ceph-us...@ceph.com
Asunto: [ceph-users]
Enviado el: viernes, 31 de enero de 2020 14:32
Para: EDH - Manuel Rios
CC: ceph-users
Asunto: Re: Micron SSD/Basic Config
Appreciate the input.
Looking at those articles they make me feel like the 40G they are talking about
is 4x Bonded 10G connections.
Im looking at 40Gbps without bonding
Hmm change 40Gbps to 100Gbps networking.
40Gbps technology its just a bond of 4x10 Links with some latency due link
aggregation.
100 Gbps and 25Gbps got less latency and Good performance. In ceph a 50% of the
latency comes from Network commits and the other 50% from disk commits.
A fast graph
As update,
It continues..
2019-10-29 19:36:48.787 7fc5ae22c700 0 abort_bucket_multiparts WARNING :
aborted 2437000 incomplete multipart uploads
How to get debug for upload ?
Regards
De: EDH - Manuel Rios Fernandez
Enviado el: lunes, 28 de octubre de 2019 14:18
Para: ceph
Hi Ceph's!
We started deteling a bucket several days ago. Total size 47TB / 8.5M
objects.
Now we see the cli bucket rm stucked and by console drop this messages.
[root@ceph-rgw03 ~]# 2019-10-28 13:55:43.880 7f0dd92c9700 0
abort_bucket_multiparts WARNING : aborted 1000 incomplete
My radosgw-admin orphans find generated +64 shards and it show a lot of
_shadow_ , _multipart and other undefined object type.
Waiting for someone clarify what to do with the output.
Regards
De: P. O.
Enviado el: martes, 24 de septiembre de 2019 11:26
Para: ceph-users@ceph.io
Hi!
We're looking to mantain our rgw pools out of orphans objects, checking the
documentation and mailist is not really clear how it works and what will do.
Radosgw-admin orphands find -pool= --job-id=
Loops over all objects in the cluster looking for leaked objects and add it
to a
Checking the shards, bucket got 64.
But shard 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.16313306.1.1 seems missing.
Radosgw-admin fix wont recreate again, any recommendation? Maybe change
shard numbers to lower?
Regards
Manuel
De: EDH - Manuel Rios Fernandez
Enviado el: jueves, 5
Hi,
Were at 14.2.2
We just found a broken bucket index, trying to repair with the common
commands ]# radosgw-admin bucket check --check-objects fix finish
instantly, but bucket should have near 60-70TB info.
[root@CEPH-MON01 home]# radosgw-admin bucket check --check-objects --bucket
There's no patch notes at ceph.com I suggest dont update until update
changelog.
-Mensaje original-
De: Fyodor Ustinov
Enviado el: miércoles, 4 de septiembre de 2019 14:16
Para: ceph-users
Asunto: [ceph-users] Re: CEPH 14.2.3
Hi!
And by the way, I confirm - the installation of the
Hi Fyodor
Whats the model of SSD?
Regards
-Mensaje original-
De: Fyodor Ustinov
Enviado el: martes, 3 de septiembre de 2019 13:13
Para: ceph-users
Asunto: [ceph-users] Strange hardware behavior
Hi!
I understand that this question is not quite for this mailing list, but
Just a note:
With 7+5 you will need 13 host for access your data in case one goes down.
Expected in the nexts version allow access data with the EC numbers.
-Mensaje original-
De: Yoann Moulin
Enviado el: martes, 3 de septiembre de 2019 11:28
Para: ceph-users@ceph.io
Asunto:
42 matches
Mail list logo
