[ceph-users] Re: Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1

[duluxoz]

Sorry, let me qualify things / try to make them simpler:

When upgrading from a Rocky Linux 8.6 Server running Ceph-Quincy to 
Rocky Linux 9.1 Server running Ceph-Quincy (ie an in-place upgrade of a 
host-node in an existing cluster):


- What is the update procedure?

- Can we use the "standard(?)" update methodology as per numerous blog 
posts available on-line?


- Is this procedure documented anywhere?

- Are there any special actions we need to be aware of?

- Are there any "gotchas", etc that we need to be aware of?

Thanks in advance

Cheers

Dulux-Oz
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1

[duluxoz]

As I said in the initial post the servers are currently Rocky v8.6. 
Obviously there's the migrate2rocky.sh and migrate2rocky9.sh scripts, 
but I was wondering if there was anything "special" that we need to do 
when running them with quincy ie any "gotchas"?  :-)


On 11/02/2023 16:43, Konstantin Shalygin wrote:
You are mentioned that your cluster is Quincy, the el9 package are 
also for Quincy. What exactly upgrade you mean?



k
Sent from my iPhone


On 11 Feb 2023, at 12:29, duluxoz  wrote:



That's great - thanks.

Any idea if there are any upgrade instructions? Any "gotchas", etc?

I mean, having the new rpm is great for a fresh install, but we were 
wanting to upgrade an existing cluster  :-)


Cheers

Dulux-Oz

On 11/02/2023 15:02, Konstantin Shalygin wrote:

Hi,

Seems packages el9_quincy are available [1]
You can try


k
[1] https://download.ceph.com/rpm-quincy/el9/x86_64/


On 10 Feb 2023, at 13:23, duluxoz  wrote:

Sorry if this was mentioned previously (I obviously missed it if it 
was) but can we upgrade a Ceph Quincy Host/Cluster from Rocky Linux 
(RHEL) v8.6/8.7 to v9.1 (yet), and if so, what is / where can I 
find the procedure to do this - ie is there anything "special" that 
needs to be done because of Ceph, or can we just do a "simple" v8.x 
+> v9.1 upgrade?



--
Peregrine IT Signature

*Matthew J BLACK*
  M.Inf.Tech.(Data Comms)
  MBA
  B.Sc.
  MACS (Snr), CP, IP3P

When you want it done /right/ ‒ the first time!

Phone:  +61 4 0411 0089
Email:  matt...@peregrineit.net 
Web:www.peregrineit.net 

View Matthew J BLACK's profile on LinkedIn 



This Email is intended only for the addressee.  Its use is limited to 
that intended by the author at the time and it is not to be 
distributed without the author’s consent.  You must not use or 
disclose the contents of this Email, or add the sender’s Email 
address to any database, list or mailing list unless you are 
expressly authorised to do so.  Unless otherwise stated, Peregrine 
I.T. Pty Ltd accepts no liability for the contents of this Email 
except where subsequently confirmed in writing.  The opinions 
expressed in this Email are those of the author and do not 
necessarily represent the views of Peregrine I.T. Pty Ltd.  This 
Email is confidential and may be subject to a claim of legal privilege.


If you have received this Email in error, please notify the author 
and delete this message immediately.



--
Peregrine IT Signature

*Matthew J BLACK*
  M.Inf.Tech.(Data Comms)
  MBA
  B.Sc.
  MACS (Snr), CP, IP3P

When you want it done /right/ ‒ the first time!

Phone:  +61 4 0411 0089
Email:  matt...@peregrineit.net 
Web:www.peregrineit.net 

View Matthew J BLACK's profile on LinkedIn 



This Email is intended only for the addressee.  Its use is limited to 
that intended by the author at the time and it is not to be distributed 
without the author’s consent.  You must not use or disclose the contents 
of this Email, or add the sender’s Email address to any database, list 
or mailing list unless you are expressly authorised to do so.  Unless 
otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the 
contents of this Email except where subsequently confirmed in 
writing.  The opinions expressed in this Email are those of the author 
and do not necessarily represent the views of Peregrine I.T. Pty 
Ltd.  This Email is confidential and may be subject to a claim of legal 
privilege.


If you have received this Email in error, please notify the author and 
delete this message immediately.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1

[Konstantin Shalygin]

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1

[duluxoz]

That's great - thanks.

Any idea if there are any upgrade instructions? Any "gotchas", etc?

I mean, having the new rpm is great for a fresh install, but we were 
wanting to upgrade an existing cluster  :-)


Cheers

Dulux-Oz

On 11/02/2023 15:02, Konstantin Shalygin wrote:

Hi,

Seems packages el9_quincy are available [1]
You can try


k
[1] https://download.ceph.com/rpm-quincy/el9/x86_64/


On 10 Feb 2023, at 13:23, duluxoz  wrote:

Sorry if this was mentioned previously (I obviously missed it if it 
was) but can we upgrade a Ceph Quincy Host/Cluster from Rocky Linux 
(RHEL) v8.6/8.7 to v9.1 (yet), and if so, what is / where can I find 
the procedure to do this - ie is there anything "special" that needs 
to be done because of Ceph, or can we just do a "simple" v8.x +> v9.1 
upgrade?



--
Peregrine IT Signature

*Matthew J BLACK*
  M.Inf.Tech.(Data Comms)
  MBA
  B.Sc.
  MACS (Snr), CP, IP3P

When you want it done /right/ ‒ the first time!

Phone:  +61 4 0411 0089
Email:  matt...@peregrineit.net 
Web:www.peregrineit.net 

View Matthew J BLACK's profile on LinkedIn 



This Email is intended only for the addressee.  Its use is limited to 
that intended by the author at the time and it is not to be distributed 
without the author’s consent.  You must not use or disclose the contents 
of this Email, or add the sender’s Email address to any database, list 
or mailing list unless you are expressly authorised to do so.  Unless 
otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the 
contents of this Email except where subsequently confirmed in 
writing.  The opinions expressed in this Email are those of the author 
and do not necessarily represent the views of Peregrine I.T. Pty 
Ltd.  This Email is confidential and may be subject to a claim of legal 
privilege.


If you have received this Email in error, please notify the author and 
delete this message immediately.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1

[Konstantin Shalygin]

Hi,

Seems packages el9_quincy are available [1]
You can try


k
[1] https://download.ceph.com/rpm-quincy/el9/x86_64/

> On 10 Feb 2023, at 13:23, duluxoz  wrote:
> 
> Sorry if this was mentioned previously (I obviously missed it if it was) but 
> can we upgrade a Ceph Quincy Host/Cluster from Rocky Linux (RHEL) v8.6/8.7 to 
> v9.1 (yet), and if so, what is / where can I find the procedure to do this - 
> ie is there anything "special" that needs to be done because of Ceph, or can 
> we just do a "simple" v8.x +> v9.1 upgrade?

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Exit yolo mode by increasing size/min_size does not (really) work

[Eugen Block]

Okay, so your applied crush rule has failure domain „room“ which you  
have three of, but the third has no OSDs available. Check your osd  
tree output, that’s why ceph fails to create a third replica. To  
resolve this you can either change the rule to a different failure  
domain (for example „host“) and then increase the size. Or you create  
a new rule and apply it to the pool(s). Either way you’ll have to  
decide how to place three replicas, e. g. move hosts within the crush  
tree (and probably the third room into the default root) to enable an  
even distribution. Note that moving buckets within the crush tree will  
cause rebalancing.


Regards,
Eugen

Zitat von stefan.pin...@bearingpoint.com:


sure!

ceph osd pool ls detail
https://privatebin.net/?85105578dd50f65f#4oNunvNfLoNbnqJwuXoWXrB1idt4zMGnBXdQ8Lkwor8p
i guess this needs some cleaning up regarding snapshots - could this  
be a problem?


ceph osd crush rule dump
https://privatebin.net/?bd589bc9d7800dd3#3PFS3659qXqbxfaXSUcKot3ynmwRG2mDjpxSmhCxQzAB
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Extremally need help. Openshift cluster is down :c

[kreept . sama]

Hello everyone and sorry. Maybe someone has already faced this problem. 
A day ago, we restored our Openshift cluster, however, at the moment, the PVCs 
cannot connect to the pod. We looked at the status of the ceph and found that 
our MDS were in standby mode, then found that the metadata was corrupted. After 
some manipulations, we were able to turn on our MDS daemons, but there is still 
no record on the cluster, the ceph status command shows the following.

sh-4.4$ ceph -s
  cluster:
id: 9213604e-b0b6-49d5-bcb3-f55ab3d79119
health: HEALTH_ERR
1 MDSs report damaged metadata
1 MDSs are read only
6 daemons have recently crashed
  services:
mon: 5 daemons, quorum bd,bj,bm,bn,bo (age 26h)
mgr: a(active, since 25h)
mds: 1/1 daemons up, 1 hot standby
osd: 9 osds: 9 up (since 41h), 9 in (since 42h)
rgw: 1 daemon active (1 hosts, 1 zones)
  data:
volumes: 1/1 healthy
pools:   10 pools, 225 pgs
objects: 1.60M objects, 234 GiB
usage:   606 GiB used, 594 GiB / 1.2 TiB avail
pgs: 225 active+clean
  io:
client:   852 B/s rd, 1 op/s rd, 0 op/s wr

Now we trying to follow this instructions:
https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#recovery-from-missing-metadata-objects

What else have we tried:

cephfs-journal-tool --rank=1:0 event recover_dentries summary
cephfs-journal-tool --rank=1:0 journal reset
cephfs-table-tool all reset session
ceph tell mds.gml--cephfs-a scrub start / recursive repair force
ceph tell mds.gml--cephfs-b scrub start / recursive repair force
ceph mds repaired 0

ceph tell mds.gml--cephfs-a damage ls

[
{
"damage_type": "dir_frag",
"id": 26851730,
"ino": 1100162409473,
"frag": "*",
"path": 
"/volumes/csi/csi-vol-5ad18c03-3205-11ed-9ba7-0a580a810206/e5664004-51e0-4bff-85c8-029944b431d8/store/096/096a1497-78ab-4802-a5a7-d09e011fd3a5/202301_1027796_1027796_0"
},
………

{
"damage_type": "dir_frag",
"id": 118336643,
"ino": 1100162424469,
"frag": "*",
"path": 
"/volumes/csi/csi-vol-5ad18c03-3205-11ed-9ba7-0a580a810206/e5664004-51e0-4bff-85c8-029944b431d8/store/096/096a1497-78ab-4802-a5a7-d09e011fd3a5/202301_1027832_1027832_0"
},

Now we trying: 

# Session table
cephfs-table-tool 0 reset session
# SnapServer
cephfs-table-tool 0 reset snap
# InoTable
cephfs-table-tool 0 reset inode
# Journal
cephfs-journal-tool --rank=0 journal reset
# Root inodes ("/" and MDS directory)
cephfs-data-scan init

cephfs-data-scan scan_extents 
cephfs-data-scan scan_inodes 
cephfs-data-scan scan_links

Is it right way and cant it be our salvation? 
Thank you!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Exit yolo mode by increasing size/min_size does not (really) work

[stefan . pinter]

sure!

ceph osd pool ls detail
https://privatebin.net/?85105578dd50f65f#4oNunvNfLoNbnqJwuXoWXrB1idt4zMGnBXdQ8Lkwor8p
i guess this needs some cleaning up regarding snapshots - could this be a 
problem?

ceph osd crush rule dump
https://privatebin.net/?bd589bc9d7800dd3#3PFS3659qXqbxfaXSUcKot3ynmwRG2mDjpxSmhCxQzAB
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: issue in connecting Openstack(Kolla-ansible) manila with external ceph (cephadm)

[Eugen Block]

Hi,

I found this bug (won’t fix):

https://tracker.ceph.com/issues/51039

Which openstack version is this? With cephadm your ceph version is at  
least Octopus, but it might be an older openstack version so the  
backend can’t parse the newer mon-mgr target but expects only mon.


Zitat von Haitham Abdulaziz :


i deployed kolla-ansible & cephadm on virtual machines (kvm) .
My ceph cluster is on 3 vms with 12 vCPU each and 24gb of ram i used  
cephadm to deploy ceph


ceph -s :

--

  cluster:
id: a0e5ad36-a54c-11ed-9aea-5254008c2a3e
health: HEALTH_OK

  services:
mon: 3 daemons, quorum ceph0,ceph1,ceph2 (age 6h)
mgr: ceph0.dzutak(active, since 24h), standbys: ceph1.aizuyc
mds: 3/3 daemons up, 6 standby
osd: 9 osds: 9 up (since 24h), 9 in (since 24h)

  data:
volumes: 3/3 healthy
pools:   9 pools, 257 pgs
objects: 70 objects, 7.3 KiB
usage:   76 MiB used, 780 GiB / 780 GiB avail
pgs: 257 active+clean

--

my openstack deployment is AIO on a single node , now i wanna link  
them together so i started with manila & native cephfs thinking its  
the easist following this doc :


https://docs.openstack.org/manila/latest/admin/cephfs_driver.html#authorizing-the-driver-to-communicate-with-ceph


i created the user
--

client.manila
  key: AQC7ot9jfiDsIxAA57fb7S6bVMnr5IadsnukHQ==
  caps: [mgr] allow rw
  caps: [mon] allow r
  caps: [osd] allow rw pool=ganesha_rados_store
and created a file system called manila

--

my ceph.conf

--

[global]
  fsid = a0e5ad36-a54c-11ed-9aea-5254008c2a3e
  mon_host = [v2:192.168.122.25:3300/0,v1:192.168.122.25:6789/0]  
[v2:192.168.122.115:3300/0,v1:192.168.122.115:6789/0]  
[v2:192.168.122.14:3300/0,v1:192.168.122.14:6789/0]


--
i moved the files as to the openstack node and trying to connect  
them together but it didnt go will , Viewing the logs shows

--

json_command failed - prefix=fs volume ls, argdict={'format':  
'json'} - exception message: Bad target type 'mon-mgr'.


--where should i start to fix this issue ?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] issue in connecting Openstack(Kolla-ansible) manila with external ceph (cephadm)

[Haitham Abdulaziz]

i deployed kolla-ansible & cephadm on virtual machines (kvm) .
My ceph cluster is on 3 vms with 12 vCPU each and 24gb of ram i used cephadm to 
deploy ceph

ceph -s :

--

  cluster:
id: a0e5ad36-a54c-11ed-9aea-5254008c2a3e
health: HEALTH_OK

  services:
mon: 3 daemons, quorum ceph0,ceph1,ceph2 (age 6h)
mgr: ceph0.dzutak(active, since 24h), standbys: ceph1.aizuyc
mds: 3/3 daemons up, 6 standby
osd: 9 osds: 9 up (since 24h), 9 in (since 24h)

  data:
volumes: 3/3 healthy
pools:   9 pools, 257 pgs
objects: 70 objects, 7.3 KiB
usage:   76 MiB used, 780 GiB / 780 GiB avail
pgs: 257 active+clean

--

my openstack deployment is AIO on a single node , now i wanna link them 
together so i started with manila & native cephfs thinking its the easist 
following this doc :

https://docs.openstack.org/manila/latest/admin/cephfs_driver.html#authorizing-the-driver-to-communicate-with-ceph


i created the user
--

client.manila
  key: AQC7ot9jfiDsIxAA57fb7S6bVMnr5IadsnukHQ==
  caps: [mgr] allow rw
  caps: [mon] allow r
  caps: [osd] allow rw pool=ganesha_rados_store
and created a file system called manila

--

my ceph.conf

--

[global]
  fsid = a0e5ad36-a54c-11ed-9aea-5254008c2a3e
  mon_host = [v2:192.168.122.25:3300/0,v1:192.168.122.25:6789/0] 
[v2:192.168.122.115:3300/0,v1:192.168.122.115:6789/0] 
[v2:192.168.122.14:3300/0,v1:192.168.122.14:6789/0]

--
i moved the files as to the openstack node and trying to connect them together 
but it didnt go will , Viewing the logs shows
--

[ceph-users] Re: RadosGW - Performance Expectations

[Mark Nelson]

For reference, with parallel writes using the S3 Go API (via hsbench: 
https://github.com/markhpc/hsbench), I was recently doing about 600ish 
MB/s to a single RGW instance from one client.  RadosGW used around 3ish 
HW threads from a 2016 era Xeon to do that.  Didn't try single-file 
tests in that case though which likely would have been slower.


Mark

On 2/10/23 09:59, Shawn Weeks wrote:

With this options I still see around 38-40MB/s for my 16gb test file. So far my 
testing is mostly synthetic, I’m going to be using some programs like GitLab 
and Sonatype Nexus that store their data in object storage. At work I deal with 
real S3 and regular see upload speeds in the 100s of MB/s so I was kinda 
surprised that the aws cli was only doing 25 or so.

Thanks
Shawn


On Feb 10, 2023, at 8:46 AM, Janne Johansson  wrote:


The problem I’m seeing is after setting up RadosGW I can only upload to “S3” at 
around 25MBs with the official AWS CLI. Using s3cmd is slightly better at 
around 45MB/s. I’m going directly to the RadosGW instance with no load 
balancers in between and no ssl enabled. Just trying to figure out if this is 
normal. I’m not expecting it to be as fast as writing directly to a RBD but I 
was kinda hoping for more than this.

So what should I expect in performance from the RadosGW?


For s3cmd, I have some perf options I use,

multipart_chunk_size_mb = 256
send_chunk = 262144
recv_chunk = 262144
and frequently see 100-150MB/s for well connected client runs,
especially if you repeat uploads and use s3cmd's   --cache-file=FILE
option so that you don't benchmark your local computers ability to
checksum the object(s).

But I would also consider using rclone and/or something that actually
makes sure to split up large files/objects and uploads them in
parallel. We have hdd+nvme clusters on 25GE networks that ingest some
1.5-2 GB/s using lots of threads and many clients, but the totals are
in that vicinity. Several load balancers and some 6-9 rgws to share
the load helps there.

--
May the most significant bit of your life be positive.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW - Performance Expectations

[Shawn Weeks]

With this options I still see around 38-40MB/s for my 16gb test file. So far my 
testing is mostly synthetic, I’m going to be using some programs like GitLab 
and Sonatype Nexus that store their data in object storage. At work I deal with 
real S3 and regular see upload speeds in the 100s of MB/s so I was kinda 
surprised that the aws cli was only doing 25 or so.

Thanks
Shawn

> On Feb 10, 2023, at 8:46 AM, Janne Johansson  wrote:
> 
>> The problem I’m seeing is after setting up RadosGW I can only upload to “S3” 
>> at around 25MBs with the official AWS CLI. Using s3cmd is slightly better at 
>> around 45MB/s. I’m going directly to the RadosGW instance with no load 
>> balancers in between and no ssl enabled. Just trying to figure out if this 
>> is normal. I’m not expecting it to be as fast as writing directly to a RBD 
>> but I was kinda hoping for more than this.
>> 
>> So what should I expect in performance from the RadosGW?
> 
> For s3cmd, I have some perf options I use,
> 
> multipart_chunk_size_mb = 256
> send_chunk = 262144
> recv_chunk = 262144
> and frequently see 100-150MB/s for well connected client runs,
> especially if you repeat uploads and use s3cmd's   --cache-file=FILE
> option so that you don't benchmark your local computers ability to
> checksum the object(s).
> 
> But I would also consider using rclone and/or something that actually
> makes sure to split up large files/objects and uploads them in
> parallel. We have hdd+nvme clusters on 25GE networks that ingest some
> 1.5-2 GB/s using lots of threads and many clients, but the totals are
> in that vicinity. Several load balancers and some 6-9 rgws to share
> the load helps there.
> 
> -- 
> May the most significant bit of your life be positive.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW - Performance Expectations

[Shawn Weeks]

With s5cmd and its defaults I got around 127MB/s for a single 16gb test file. 
Is there any way to make s5cmd give feedback while it’s running. At first I 
didn’t think it was working because it just sat there for a while.

Thanks
Shawn

On Feb 10, 2023, at 8:45 AM, Matt Benjamin  wrote:

Hi Shawn,

To get another S3 upload baseline, I'd recommend doing some upload testing with 
s5cmd [1].

1. https://github.com/peak/s5cmd

Matt


On Fri, Feb 10, 2023 at 9:38 AM Shawn Weeks 
mailto:swe...@weeksconsulting.us>> wrote:
Good morning everyone, been running a small Ceph cluster with Proxmox for a 
while now and I’ve finally run across an issue I can’t find any information on. 
I have a 3 node cluster with 9 Samsung PM983 960GB NVME drives running on a 
dedicated 10gb network. RBD and CephFS performance have been great, most of the 
time I see over 500MBs writes and a rados benchmark shows 951 MB/s write and 
1140 MB/s read bandwidth.

The problem I’m seeing is after setting up RadosGW I can only upload to “S3” at 
around 25MBs with the official AWS CLI. Using s3cmd is slightly better at 
around 45MB/s. I’m going directly to the RadosGW instance with no load 
balancers in between and no ssl enabled. Just trying to figure out if this is 
normal. I’m not expecting it to be as fast as writing directly to a RBD but I 
was kinda hoping for more than this.

So what should I expect in performance from the RadosGW?

Here are some rados bench results and my ceph report

https://gist.github.com/shawnweeks/f6ef028284b5cdb10d80b8dc0654eec5

https://gist.github.com/shawnweeks/7cfe94c08adbc24f2a3d8077688df438

Thanks
Shawn
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io


--

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW - Performance Expectations

[Janne Johansson]

> The problem I’m seeing is after setting up RadosGW I can only upload to “S3” 
> at around 25MBs with the official AWS CLI. Using s3cmd is slightly better at 
> around 45MB/s. I’m going directly to the RadosGW instance with no load 
> balancers in between and no ssl enabled. Just trying to figure out if this is 
> normal. I’m not expecting it to be as fast as writing directly to a RBD but I 
> was kinda hoping for more than this.
>
> So what should I expect in performance from the RadosGW?

For s3cmd, I have some perf options I use,

multipart_chunk_size_mb = 256
send_chunk = 262144
recv_chunk = 262144
and frequently see 100-150MB/s for well connected client runs,
especially if you repeat uploads and use s3cmd's   --cache-file=FILE
option so that you don't benchmark your local computers ability to
checksum the object(s).

But I would also consider using rclone and/or something that actually
makes sure to split up large files/objects and uploads them in
parallel. We have hdd+nvme clusters on 25GE networks that ingest some
1.5-2 GB/s using lots of threads and many clients, but the totals are
in that vicinity. Several load balancers and some 6-9 rgws to share
the load helps there.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW - Performance Expectations

[Matt Benjamin]

Hi Shawn,

To get another S3 upload baseline, I'd recommend doing some upload testing
with s5cmd [1].

1. https://github.com/peak/s5cmd

Matt


On Fri, Feb 10, 2023 at 9:38 AM Shawn Weeks 
wrote:

> Good morning everyone, been running a small Ceph cluster with Proxmox for
> a while now and I’ve finally run across an issue I can’t find any
> information on. I have a 3 node cluster with 9 Samsung PM983 960GB NVME
> drives running on a dedicated 10gb network. RBD and CephFS performance have
> been great, most of the time I see over 500MBs writes and a rados benchmark
> shows 951 MB/s write and 1140 MB/s read bandwidth.
>
> The problem I’m seeing is after setting up RadosGW I can only upload to
> “S3” at around 25MBs with the official AWS CLI. Using s3cmd is slightly
> better at around 45MB/s. I’m going directly to the RadosGW instance with no
> load balancers in between and no ssl enabled. Just trying to figure out if
> this is normal. I’m not expecting it to be as fast as writing directly to a
> RBD but I was kinda hoping for more than this.
>
> So what should I expect in performance from the RadosGW?
>
> Here are some rados bench results and my ceph report
>
> https://gist.github.com/shawnweeks/f6ef028284b5cdb10d80b8dc0654eec5
>
> https://gist.github.com/shawnweeks/7cfe94c08adbc24f2a3d8077688df438
>
> Thanks
> Shawn
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RadosGW - Performance Expectations

[Shawn Weeks]

Good morning everyone, been running a small Ceph cluster with Proxmox for a 
while now and I’ve finally run across an issue I can’t find any information on. 
I have a 3 node cluster with 9 Samsung PM983 960GB NVME drives running on a 
dedicated 10gb network. RBD and CephFS performance have been great, most of the 
time I see over 500MBs writes and a rados benchmark shows 951 MB/s write and 
1140 MB/s read bandwidth.

The problem I’m seeing is after setting up RadosGW I can only upload to “S3” at 
around 25MBs with the official AWS CLI. Using s3cmd is slightly better at 
around 45MB/s. I’m going directly to the RadosGW instance with no load 
balancers in between and no ssl enabled. Just trying to figure out if this is 
normal. I’m not expecting it to be as fast as writing directly to a RBD but I 
was kinda hoping for more than this.

So what should I expect in performance from the RadosGW?

Here are some rados bench results and my ceph report

https://gist.github.com/shawnweeks/f6ef028284b5cdb10d80b8dc0654eec5

https://gist.github.com/shawnweeks/7cfe94c08adbc24f2a3d8077688df438

Thanks
Shawn
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: No such file or directory when issuing "rbd du"

[Mehmet]

Am 2023-02-10 09:13, schrieb Victor Rodriguez:

I've seen that happen when a rbd image or a snapshot is being removed
and you cancel the operation, specially if they are big or storage is
relatively slow. The rbd image will stay "half removed" in the pool.

Check "rbd ls -p POOL" vs "rbd ls -l -p POOL" outputs: the first may
have one or more lines in it's output. Those extra lines are the half
removed images that rbd du or rbd ls -l are complaining about. Make
absolutely sure that you don't need them and remove them manually with
"rbd rm IMAGE -p POOL".


Hello Victor,

thank you very much!
Your commands was an easy solution to see which "vdisk" is affected:

root@node35:~# rbd ls -p cephhdd-001-mypool > rbdlsA.txt
root@node35:~# rbd ls -l -p cephhdd-001-mypool > rbdlaB.txt
rbd: error opening vm-44815-disk-0: (2) No such file or directory
rbd: error opening vm-44815-disk-1: (2) No such file or directory
rbd: listing images failed: (2) No such file or directory

I don't know why we have an issue with this disk, but i guess that a 
coleague had done a mistake ^^


if removed the disks like "rbd rm cephhdd-001-mypool/vm-44815-disk-0" 
and my rbd du doesnt show any error anymore.


Have a nice weekend
Mehmet
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Yet another question about OSD memory usage ...

[Ulrich Klein]

Hi,

Yet another question about OSD memory usage ...

I have a test cluster running. When I do a ceph orch ps I see for my osd.11:
ceph orch ps --refresh
NAME HOSTPORTSSTATUSREFRESHED  AGE  MEM 
USE  MEM LIM  VERSION  IMAGE ID  CONTAINER ID
osd.11   ceph01   running (2h)97s ago   2h
23.0G13.1G  17.2.5   cc65afd6173a  5d1062e8d392

When I chek via top on the machine I see:
PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+ COMMAND
  39807 ceph  20   0 6254956   3.7g   9228 S  31.2   3.0 846:21.63 
/usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph 
--default-log-to-file=false --default-lo+

Now, where does ceph orch ps get those 23.0G from, when top just shows 3.7G 
resident and 6.2G virtual for osd.11?
(I do understand that the MEM LIM n the ceph orch ps list is not really the 
limit)

Anyone know where that discrepancy comes from?

Ciao, Uli

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Frequent calling monitor election

[Stefan Kooman]

On 2/9/23 16:55, Frank Schilder wrote:



We moved a switch from one rack to another and after the switch came beck up, 
the monitors frequently bitch about who is the alpha. How do I get them to 
focus more on their daily duties again?


Just checking here, do you use xfs as monitor database filesystem? We 
encountered monitor elections when monthly trim (discard unused blocks) 
would run. Disabling the trims solved the issue for us. If you have 
"discard" option enabled this might hurt you more often ...


Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: No such file or directory when issuing "rbd du"

[Victor Rodriguez]

I've seen that happen when a rbd image or a snapshot is being removed 
and you cancel the operation, specially if they are big or storage is 
relatively slow. The rbd image will stay "half removed" in the pool.


Check "rbd ls -p POOL" vs "rbd ls -l -p POOL" outputs: the first may 
have one or more lines in it's output. Those extra lines are the half 
removed images that rbd du or rbd ls -l are complaining about. Make 
absolutely sure that you don't need them and remove them manually with 
"rbd rm IMAGE -p POOL".




On 2/9/23 17:04, Mehmet wrote:

Hello Friends,

i have a strange output when issuing following command

root@node35:~# rbd du -p cephhdd-001-mypool
NAME  PROVISIONED  USED
...
vm-99936587-disk-0@H202302091535  400 GiB  5.2 GiB
vm-99936587-disk-0@H202302091635  400 GiB  1.2 GiB
vm-99936587-disk-0    400 GiB  732 MiB
vm-104-cloudinit    4 MiB    4 MiB
vm-104-disk-0 600 GiB  586 GiB
    49 TiB   44 TiB
rbd: du failed: (2) No such file or directory
root@node35:~#

I do not know why i receive "rbd: du failed: (2) No such file or 
directory".


How can i find the origin for this?

My Ceph-Version 17.2.3 installed with "cephadm".
Cluster is "HEALTH_OK" with 108 OSDs distributed over 3 Nodes where 
mgr/mon also resides.


Hope you can help
Mehmet
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSD fail to authenticate after node outage

[Eugen Block]

Hi,

I believe this question already has been answered on [1]. The failing  
OSDs had an old monmap and were able to start after modifying their  
config.


[1]  
https://stackoverflow.com/questions/75366436/ceph-osd-authenticate-timed-out-after-node-restart


Zitat von tsmg...@gmail.com:


Release: 16.2.7 (pacific)
Infra: 4 x Nodes (4xOSD HDD), 3 x Nodes (mon/mds, 1 x OSD NVMe)

We recently had a couple of node which went offline unexpectedly  
triggering a rebalance which is still ongoing.
The OSDs on the restarted node are marked as down and they keep  
showing in the log `authenticated timed out`, after a period of time  
they get marked `autoout`.
We tried setting `noout` on the cluster which has stopped them being  
marked out but they still never authenticate.
We can access all the ceph tooling from those nodes which indicates  
connection to mons.

The node keyring/time are both in sync.
We are at a loss to why we can not get the OSDs to authenticate.

Any help would be apreciated.

```
  cluster:
id: d5126e5a-882e-11ec-954e-90e2baec3d2c
health: HEALTH_WARN
7 failed cephadm daemon(s)
2 stray daemon(s) not managed by cephadm
insufficient standby MDS daemons available
nodown,noout flag(s) set
8 osds down
2 hosts (8 osds) down
Degraded data redundancy: 195930251/392039621 objects  
degraded (49.977%), 160 pgs degraded, 160 pgs undersized

2 pgs not deep-scrubbed in time

  services:
mon: 3 daemons, quorum ceph5,ceph7,ceph6 (age 38h)
mgr: ceph2.tofizp(active, since 9M), standbys: ceph1.vnkagp
mds: 3/3 daemons up
osd: 19 osds: 11 up (since 38h), 19 in (since 45h); 5 remapped pgs
 flags nodown,noout

  data:
volumes: 1/1 healthy
pools:   6 pools, 257 pgs
objects: 102.94M objects, 67 TiB
usage:   68 TiB used, 50 TiB / 118 TiB avail
pgs: 195930251/392039621 objects degraded (49.977%)
 3205811/392039621 objects misplaced (0.818%)
 155 active+undersized+degraded
 97  active+clean
 3   active+undersized+degraded+remapped+backfill_wait
 2   active+undersized+degraded+remapped+backfilling

  io:
client:   511 B/s rd, 102 KiB/s wr, 0 op/s rd, 2 op/s wr
recovery: 13 MiB/s, 16 objects/s
```
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io