[ceph-users] Re: mgr finish mon failed to return metadata for mds

[Venky Shankar]

On Tue, Dec 12, 2023 at 7:58 PM Eugen Block  wrote:
>
> Can you restart the primary MDS (not sure which one it currently is,
> should be visible from the mds daemon log) and see if this resolves at
> least temporarily? Because after we recovered the cluster and cephfs
> we did have output in 'ceph fs status' and I can't remember seeing
> these error messages.
> This old bug [1] describes the same issue, it seems, and it was just
> recently updated. Not sure though if the backport made it into latest
> quincy.

No it didn't - https://tracker.ceph.com/issues/63415

Scheduled for next quincy though.

> And 'ceph mds metadata 0' also results in missing output:
>
> root@node01:~# ceph mds metadata 0
> {}
> Error ENOENT:
>
> [1] https://tracker.ceph.com/issues/24403
>
>
> Zitat von Manolis Daramas :
>
> > The current ceph version that we use is 17.2.7.
> >
> > We see in the Manager logs the below errors:
> >
> > 2 mgr.server handle_open ignoring open from
> > mds.storage.node01.zjltbu v2:10.40.99.11:6800/1327026642; not ready
> > for session (expect reconnect)
> > 0 7faf43715700  1 mgr finish mon failed to return metadata for
> > mds.storage.node01.zjltbu: (2) No such file or directory
> >
> > and in
> >
> > # ceph fs status
> > Error EINVAL: Traceback (most recent call last):
> >   File "/usr/share/ceph/mgr/mgr_module.py", line 1759, in _handle_command
> > return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
> >   File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
> > return self.func(mgr, **kwargs)
> >   File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status
> > assert metadata
> > AssertionError
> >
> > Does anyone know what they are and what we can do to fix them ?
> >
> > Thanks,
> >
> > Manolis
> > Under the General Data Protection Regulation (GDPR) (EU) 2016/679,
> > Motivian as Data Controller has a legal duty to protect any
> > information collected from you via email. Information contained in
> > this email and any attachments may be privileged or confidential and
> > intended for the exclusive use of the original recipient. If you
> > have received this email by mistake, please advise the sender
> > immediately and delete the email, including emptying your deleted
> > email box. Information included in this email is reserved to named
> > addressee's eyes only. You may not share this message or any of its
> > attachments to anyone. Please note that as the recipient, it is your
> > responsibility to check the email for malicious software. Motivian
> > puts the security of the client at a high priority. Therefore, we
> > have put efforts into ensuring that the message is error and
> > virus-free. Unfortunately, full security of the email cannot be
> > ensured as, despite our efforts, the data included in emails could
> > be infected,
> >  intercepted, or corrupted. Therefore, the recipient should check
> > the email for threats with proper software, as the sender does not
> > accept liability for any damage inflicted by viewing the content of
> > this email.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Terrible cephfs rmdir performance

[Venky Shankar]

Hi Paul,

On Wed, Dec 13, 2023 at 9:50 PM Paul Mezzanini  wrote:
>
> Long story short, we've got a lot of empty directories that I'm working on 
> removing.  While removing directories, using "perf top -g" we can watch the 
> MDS daemon go to 100% cpu usage with "SnapRealm:: split_at" and 
> "CInode::is_ancestor_of".
>
> It's this 2 year old bug that still is around.
> https://tracker.ceph.com/issues/53192

Unfortunately the fix isn't straightforward as it was attempted, so
lately, we've been working around these issues by pinning
to-be-deleted directories to a (separate) active MDS. This might need
some tuning at the application level to move stuff inside this
"special" pinned directory and then delete it.

HTH.

>
> To help combat this, we've moved our snapshot schedule down the tree one 
> level so the snaprealm is significantly smaller.  Our luck with multiple 
> active MDSs hasn't been great so we are still on a single MDS.  To help split 
> the load, I'm working on moving different workloads to different filesytems 
> within ceph.
>
> A user can still fairly easily overwhelm the MDS's finisher thread and 
> basically stop all cephfs io through that MDS. I'm hoping we can get some 
> other people chiming in with "Me Too!" so there can be some traction behind 
> fixing this.
>
> It's a longstanding bug so the version is less important, but we are on 
> 17.2.7.
>
> Thoughts?
> -paul
>
> --
>
> Paul Mezzanini
> Platform Engineer III
> Research Computing
>
> Rochester Institute of Technology
>
>  “End users is a description, not a goal.”
>
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS crashing repeatedly

[Venky Shankar]

Hi Thomas,

On Wed, Dec 13, 2023 at 8:46 PM Thomas Widhalm  wrote:
>
> Hi,
>
> I have a 18.2.0 Ceph cluster and my MDS are now crashing repeatedly.
> After a few automatic restart, every MDS is removed and only one stays
> active. But it's flagged "laggy" and I can't even start a scrub on it.
>
> In the log I have this during crashes:
>
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
> 2023-12-13T14:54:02.721+ 7f15ea108700 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.0/rpm/el8/BUILD/ceph-18.2.0/src/mds/MDCache.cc:
> In function 'void MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*,
> CDentry*, snapid_t, CInode**, CDentry::linkage_t*)' thread 7f15ea108700
> time 2023-12-13T14:54:02.720383+
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.0/rpm/el8/BUILD/ceph-18.2.0/src/mds/MDCache.cc:
> 1638: FAILED ceph_assert(follows >= realm->get_newest_seq())
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef
> (stable)
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x135) [0x7f15f5ef9dbb]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   2: /usr/lib64/ceph/libceph-common.so.2(+0x2a8f81) [0x7f15f5ef9f81]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   3: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*,
> snapid_t, CInode**, CDentry::linkage_t*)+0xae2) [0x55727bc0c672]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   4: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*,
> snapid_t)+0xc5) [0x55727bc0d0d5]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   5: (Locker::scatter_writebehind(ScatterLock*)+0x5f6) [0x55727bce40f6]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   6: (Locker::simple_sync(SimpleLock*, bool*)+0x388) [0x55727bceb908]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   7: (Locker::scatter_nudge(ScatterLock*, MDSContext*, bool)+0x30d)
> [0x55727bcef25d]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   8: (Locker::scatter_tick()+0x1e7) [0x55727bd0bc37]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   9: (Locker::tick()+0xd) [0x55727bd0c0ed]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   10: (MDSRankDispatcher::tick()+0x1ef) [0x55727bb08e9f]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   11: (Context::complete(int)+0xd) [0x55727bade2cd]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   12: (CommonSafeTimer::timer_thread()+0x16d)
> [0x7f15f5fea1cd]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   13: (CommonSafeTimerThread::entry()+0x11)
> [0x7f15f5feb2a1]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   14: /lib64/libpthread.so.0(+0x81ca) [0x7f15f4ca11ca]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   15: clone()
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
> *** Caught signal (Aborted) **
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   in thread 7f15ea108700 thread_name:safe_timer
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef
> (stable)
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   1: /lib64/libpthread.so.0(+0x12cf0) [0x7f15f4cabcf0]
> Dec 13 15:54:02 ceph04
> ceph-ff6e50de-ed72-11ec-881c-dca6325c2cc4-mds-mds01-ceph04-krxszj[33486]:
>   2: gsignal()
> Dec 13 15:54:02 ceph04
> 

[ceph-users] Re: v18.2.1 Reef released

[Eugen Block]

Hi,

I thought the fix for that would have made it into 18.2.1. It was  
marked as resolved two months ago  
(https://tracker.ceph.com/issues/63150,  
https://github.com/ceph/ceph/pull/53922).



Zitat von "Robert W. Eckert" :


Hi- I tried to start the upgrade using

ceph orch upgrade start --ceph-version 18.2.1
Initiating upgrade to quay.io/ceph/ceph:v18:v18.2.1
And checked on the status
  [root@rhel1 ~]# ceph orch upgrade status
{
"target_image": "quay.io/ceph/ceph:v18:v18.2.1",
"in_progress": true,
"which": "Upgrading all daemon types on all hosts",
"services_complete": [],
"progress": "",
	"message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull  
target image",

"is_paused": true
}

However this does work:

[root@rhel1 ~]# ceph orch upgrade start --image 
quay.io/ceph/ceph:v18.2.1
Initiating upgrade to quay.io/ceph/ceph:v18.2.1

It looks like something is inserting an incorrect v18 when I tried  
to do the update by version.


Once I used the image, the upgrade finished quickly

-Rob


-Original Message-
From: Yuri Weinstein 
Sent: Monday, December 18, 2023 4:20 PM
To: ceph-annou...@ceph.io; ceph-users ; dev  
; ceph-maintain...@ceph.io

Subject: [ceph-users] v18.2.1 Reef released

We're happy to announce the 1st backport release in the Reef series.
This is the first backport release in the Reef series, and the first  
with Debian packages, for Debian Bookworm.

We recommend all users update to this release.

https://ceph.io/en/news/blog/2023/v18-2-1-reef-released/

Notable Changes
---
* RGW: S3 multipart uploads using Server-Side Encryption now  
replicate correctly in
  multi-site. Previously, the replicas of such objects were  
corrupted on decryption.
  A new tool, ``radosgw-admin bucket resync encrypted multipart``,  
can be used to
  identify these original multipart uploads. The ``LastModified``  
timestamp of any
  identified object is incremented by 1ns to cause peer zones to  
replicate it again.

  For multi-site deployments that make any use of Server-Side Encryption, we
  recommended running this command against every bucket in every  
zone after all

  zones have upgraded.

* CEPHFS: MDS evicts clients which are not advancing their request  
tids which causes
  a large buildup of session metadata resulting in the MDS going  
read-only due to

  the RADOS operation exceeding the size threshold.
`mds_session_metadata_threshold`
  config controls the maximum size that a (encoded) session metadata  
can grow.


* RGW: New tools have been added to radosgw-admin for identifying and
  correcting issues with versioned bucket indexes. Historical bugs with the
  versioned bucket index transaction workflow made it possible for the index
  to accumulate extraneous "book-keeping" olh entries and plain placeholder
  entries. In some specific scenarios where clients made concurrent requests
  referencing the same object key, it was likely that a lot of extra index
  entries would accumulate. When a significant number of these entries are
  present in a single bucket index shard, they can cause high bucket listing
  latencies and lifecycle processing failures. To check whether a versioned
  bucket has unnecessary olh entries, users can now run ``radosgw-admin
  bucket check olh``. If the ``--fix`` flag is used, the extra entries will
  be safely removed. A distinct issue from the one described thus far, it is
  also possible that some versioned buckets are maintaining extra unlinked
  objects that are not listable from the S3/ Swift APIs. These extra objects
  are typically a result of PUT requests that exited abnormally, in  
the middle

  of a bucket index transaction - so the client would not have received a
  successful response. Bugs in prior releases made these unlinked  
objects easy
  to reproduce with any PUT request that was made on a bucket that  
was actively

  resharding. Besides the extra space that these hidden, unlinked objects
  consume, there can be another side effect in certain scenarios, caused by
  the nature of the failure mode that produced them, where a client  
of a bucket
  that was a victim of this bug may find the object associated with  
the key to7fe91d5d5842e04be3b4f514d6dd990c54b29c76
  be in an inconsistent state. To check whether a versioned bucket  
has unlinked

  entries, users can now run ``radosgw-admin bucket check unlinked``. If the
  ``--fix`` flag is used, the unlinked objects will be safely  
removed. Finally,

  a third issue made it possible for versioned bucket index stats to be
  accounted inaccurately. The tooling for recalculating versioned  
bucket stats
  also had a bug, and was not previously capable of fixing these  
inaccuracies.
  This release resolves those issues and users can now expect that  
the existing

  ``radosgw-admin bucket check`` command will produce correct results. We
 

[ceph-users] Re: cephadm file "/sbin/cephadm", line 10098 PK ^

[Eugen Block]

The option ‚—image‘ is to be used for the cephadm command, not the  
bootstrap command. So it should be like this:


cephadm —image  bootstrap…

This is also covered by the link I provided (isolated environment).

Zitat von farhad kh :


hi,thank you for guidance

There is no ability to change the global image before launching,I  
need to download the images from the private registry during the  
initial setup.

i used option --image but it not worked.

# cephadm bootstrap --image  rgistry.test/ceph/ceph:v18  --mon-ip  
192.168.0.160  --initial-dashboard-password P@ssw0rd  
--dashboard-password-noupdate --allow-fqdn-hostname --ssh-user  
cephadmin

usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR]
   [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR]
   [--sysctl-dir SYSCTL_DIR] [--unit-dir UNIT_DIR] [--verbose]
   [--timeout TIMEOUT] [--retry RETRY] [--env ENV]
   [--no-container-init] [--no-cgroups-split]

{version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,zap-osds,unit,logs,bootstrap,deploy,_orch,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,host-maintenance,agent,disk-rescan}

   ...
cephadm: error: unrecognized arguments: --image rgistry.test/ceph/ceph:v18

also i used cephadm  registry-login and it show loged but when i  
bootstrap first node trying download image from quay registry.


cephadm bootstrap  --mon-ip 192.168.0.160 --registry-json  
/root/mylogin.json --initial-dashboard-password P@ssw0rd  
--dashboard-password-noupdate --allow-fqdn-hostname --ssh-user  
cephadmin

Creating directory /etc/ceph for ceph.conf
Verifying ssh connectivity using standard pubkey authentication ...
Adding key to cephadmin@localhost authorized_keys...
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 3c00e38c-9e2e-11ee-95cd-000c29e9f44e
Verifying IP 192.168.0.160 port 3300 ...
Verifying IP 192.168.0.160 port 6789 ...
Mon IP `192.168.0.160` is in CIDR network `192.168.0.0/24`
Mon IP `192.168.0.160` is in CIDR network `192.168.0.0/24`
Internal network (--cluster-network) has not been provided, OSD  
replication will default to the public_network

Pulling custom registry login info from /root/mylogin.json.
Logging into custom registry.
Pulling container image quay.io/ceph/ceph:v18...


befor i edited cephadm script but now file in coded and can't edited.
and i don't know how can fix it :(((
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm file "/sbin/cephadm", line 10098 PK ^

[farhad kh]

hi,thank you for guidance

There is no ability to change the global image before launching,I need to 
download the images from the private registry during the initial setup.
i used option --image but it not worked.

# cephadm bootstrap --image  rgistry.test/ceph/ceph:v18  --mon-ip 192.168.0.160 
 --initial-dashboard-password P@ssw0rd --dashboard-password-noupdate 
--allow-fqdn-hostname --ssh-user cephadmin
usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR]
   [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR]
   [--sysctl-dir SYSCTL_DIR] [--unit-dir UNIT_DIR] [--verbose]
   [--timeout TIMEOUT] [--retry RETRY] [--env ENV]
   [--no-container-init] [--no-cgroups-split]
   
{version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,zap-osds,unit,logs,bootstrap,deploy,_orch,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,host-maintenance,agent,disk-rescan}
   ...
cephadm: error: unrecognized arguments: --image rgistry.test/ceph/ceph:v18

also i used cephadm  registry-login and it show loged but when i bootstrap 
first node trying download image from quay registry.

cephadm bootstrap  --mon-ip 192.168.0.160 --registry-json /root/mylogin.json 
--initial-dashboard-password P@ssw0rd --dashboard-password-noupdate 
--allow-fqdn-hostname --ssh-user cephadmin
Creating directory /etc/ceph for ceph.conf
Verifying ssh connectivity using standard pubkey authentication ...
Adding key to cephadmin@localhost authorized_keys...
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 3c00e38c-9e2e-11ee-95cd-000c29e9f44e
Verifying IP 192.168.0.160 port 3300 ...
Verifying IP 192.168.0.160 port 6789 ...
Mon IP `192.168.0.160` is in CIDR network `192.168.0.0/24`
Mon IP `192.168.0.160` is in CIDR network `192.168.0.0/24`
Internal network (--cluster-network) has not been provided, OSD replication 
will default to the public_network
Pulling custom registry login info from /root/mylogin.json.
Logging into custom registry.
Pulling container image quay.io/ceph/ceph:v18...


befor i edited cephadm script but now file in coded and can't edited.
and i don't know how can fix it :(((
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Support of SNMP on CEPH ansible

[Lokendra Rathour]

Hi Team,
please help in the reference of the issue raised.


Best Regards,
Lokendra

On Wed, Dec 13, 2023 at 2:33 PM Kushagr Gupta 
wrote:

> Hi Team,
>
> *Environment:*
> We have deployed a ceph setup using ceph-ansible.
> Ceph-version: 18.2.0
> OS: Almalinux 8.8
> We have a 3 node-setup.
>
> *Queries:*
>
> 1. Is SNMP supported for ceph-ansible?Is there some other way to setup
> SNMP gateway for the ceph cluster?
> 2. Do we have a procedure to set backend for ceph-orchestrator via
> ceph-ansible? Which backend to use?
> 3. Are there any CEPH MIB files which work independent of prometheus.
>
>
> *Description:*
> We are trying to perform SNMP monitoring for the ceph cluster using the
> following link:
>
> 1.
> https://docs.ceph.com/en/quincy/cephadm/services/snmp-gateway/#:~:text=Ceph's%20SNMP%20integration%20focuses%20on,a%20designated%20SNMP%20management%20platform
> .
> 2.
> https://www.ibm.com/docs/en/storage-ceph/7?topic=traps-deploying-snmp-gateway
>
> But when we try to follow the steps mentioned in the above link, we get
> the following error when we try to run any "ceph orch" we get the following
> error:
> "Error ENOENT: No orchestrator configured (try `ceph orch set backend`)"
>
> After going through following links:
> 1.
> https://www.ibm.com/docs/en/storage-ceph/5?topic=operations-use-ceph-orchestrator
> 2.
> https://forum.proxmox.com/threads/ceph-mgr-orchestrator-enabled-but-showing-missing.119145/
> 3. https://docs.ceph.com/en/latest/mgr/orchestrator_modules/
> I think since we have deployed the cluster using ceph-ansible, we can't
> use the ceph-orch commands.
> When we checked in the cluster, the following are the enabled modules:
> "
> [root@storagenode1 ~]# ceph mgr module ls
> MODULE
> balancer   on (always on)
> crash  on (always on)
> devicehealth   on (always on)
> orchestrator   on (always on)
> pg_autoscaler  on (always on)
> progress   on (always on)
> rbd_supporton (always on)
> status on (always on)
> telemetry  on (always on)
> volumeson (always on)
> alerts on
> iostat on
> nfson
> prometheus on
> restfulon
> dashboard  -
> influx -
> insights   -
> localpool  -
> mds_autoscaler -
> mirroring  -
> osd_perf_query -
> osd_support-
> rgw-
> selftest   -
> snap_schedule  -
> stats  -
> telegraf   -
> test_orchestrator  -
> zabbix -
> [root@storagenode1 ~]#
> "
> As can be seen above, orchestrator is on.
>
> Also, We were exploring more about snmp and as per the file:
> "/etc/prometheus/ceph/ceph_default_alerts.yml" on the ceph storage, the
> OIDs in the file represents the OID for ceph components via prometheus.
> For example:
> for the following OID: 1.3.6.1.4.1.50495.1.2.1.2.1
> [root@storagenode3 ~]# snmpwalk -v 2c -c 209ijvfwer0df92jd -O e 10.0.1.36
> 1.3.6.1.4.1.50495.1.2.1.2.1
> CEPH-MIB::promHealthStatusError = No Such Object available on this agent
> at this OID
> [root@storagenode3 ~]#
>
> Kindly help us for the same.
>
> Thanks and regards,
> Kushagra Gupta
>


-- 
~ Lokendra
skype: lokendrarathour
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v18.2.1 Reef released

[Robert W. Eckert]

Hi- I tried to start the upgrade using

ceph orch upgrade start --ceph-version 18.2.1
Initiating upgrade to quay.io/ceph/ceph:v18:v18.2.1
And checked on the status
  [root@rhel1 ~]# ceph orch upgrade status
{
"target_image": "quay.io/ceph/ceph:v18:v18.2.1",
"in_progress": true,
"which": "Upgrading all daemon types on all hosts",
"services_complete": [],
"progress": "",
"message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull 
target image",
"is_paused": true
}

However this does work: 

[root@rhel1 ~]# ceph orch upgrade start --image 
quay.io/ceph/ceph:v18.2.1
Initiating upgrade to quay.io/ceph/ceph:v18.2.1

It looks like something is inserting an incorrect v18 when I tried to do the 
update by version.

Once I used the image, the upgrade finished quickly

-Rob


-Original Message-
From: Yuri Weinstein  
Sent: Monday, December 18, 2023 4:20 PM
To: ceph-annou...@ceph.io; ceph-users ; dev ; 
ceph-maintain...@ceph.io
Subject: [ceph-users] v18.2.1 Reef released

We're happy to announce the 1st backport release in the Reef series.
This is the first backport release in the Reef series, and the first with 
Debian packages, for Debian Bookworm.
We recommend all users update to this release.

https://ceph.io/en/news/blog/2023/v18-2-1-reef-released/

Notable Changes
---
* RGW: S3 multipart uploads using Server-Side Encryption now replicate 
correctly in
  multi-site. Previously, the replicas of such objects were corrupted on 
decryption.
  A new tool, ``radosgw-admin bucket resync encrypted multipart``, can be used 
to
  identify these original multipart uploads. The ``LastModified`` timestamp of 
any
  identified object is incremented by 1ns to cause peer zones to replicate it 
again.
  For multi-site deployments that make any use of Server-Side Encryption, we
  recommended running this command against every bucket in every zone after all
  zones have upgraded.

* CEPHFS: MDS evicts clients which are not advancing their request tids which 
causes
  a large buildup of session metadata resulting in the MDS going read-only due 
to
  the RADOS operation exceeding the size threshold.
`mds_session_metadata_threshold`
  config controls the maximum size that a (encoded) session metadata can grow.

* RGW: New tools have been added to radosgw-admin for identifying and
  correcting issues with versioned bucket indexes. Historical bugs with the
  versioned bucket index transaction workflow made it possible for the index
  to accumulate extraneous "book-keeping" olh entries and plain placeholder
  entries. In some specific scenarios where clients made concurrent requests
  referencing the same object key, it was likely that a lot of extra index
  entries would accumulate. When a significant number of these entries are
  present in a single bucket index shard, they can cause high bucket listing
  latencies and lifecycle processing failures. To check whether a versioned
  bucket has unnecessary olh entries, users can now run ``radosgw-admin
  bucket check olh``. If the ``--fix`` flag is used, the extra entries will
  be safely removed. A distinct issue from the one described thus far, it is
  also possible that some versioned buckets are maintaining extra unlinked
  objects that are not listable from the S3/ Swift APIs. These extra objects
  are typically a result of PUT requests that exited abnormally, in the middle
  of a bucket index transaction - so the client would not have received a
  successful response. Bugs in prior releases made these unlinked objects easy
  to reproduce with any PUT request that was made on a bucket that was actively
  resharding. Besides the extra space that these hidden, unlinked objects
  consume, there can be another side effect in certain scenarios, caused by
  the nature of the failure mode that produced them, where a client of a bucket
  that was a victim of this bug may find the object associated with the key 
to7fe91d5d5842e04be3b4f514d6dd990c54b29c76
  be in an inconsistent state. To check whether a versioned bucket has unlinked
  entries, users can now run ``radosgw-admin bucket check unlinked``. If the
  ``--fix`` flag is used, the unlinked objects will be safely removed. Finally,
  a third issue made it possible for versioned bucket index stats to be
  accounted inaccurately. The tooling for recalculating versioned bucket stats
  also had a bug, and was not previously capable of fixing these inaccuracies.
  This release resolves those issues and users can now expect that the existing
  ``radosgw-admin bucket check`` command will produce correct results. We
  recommend that users with versioned buckets, especially those that existed
  on prior releases, use these new tools to check whether their buckets are
  affected and to clean them up accordingly.

* mgr/snap-schedule: For clusters with multiple CephFS file systems, 

[ceph-users] v18.2.1 Reef released

[Yuri Weinstein]

We're happy to announce the 1st backport release in the Reef series.
This is the first backport release in the Reef series, and the first
with Debian packages, for Debian Bookworm.
We recommend all users update to this release.

https://ceph.io/en/news/blog/2023/v18-2-1-reef-released/

Notable Changes
---
* RGW: S3 multipart uploads using Server-Side Encryption now replicate
correctly in
  multi-site. Previously, the replicas of such objects were corrupted
on decryption.
  A new tool, ``radosgw-admin bucket resync encrypted multipart``, can
be used to
  identify these original multipart uploads. The ``LastModified``
timestamp of any
  identified object is incremented by 1ns to cause peer zones to
replicate it again.
  For multi-site deployments that make any use of Server-Side Encryption, we
  recommended running this command against every bucket in every zone after all
  zones have upgraded.

* CEPHFS: MDS evicts clients which are not advancing their request
tids which causes
  a large buildup of session metadata resulting in the MDS going
read-only due to
  the RADOS operation exceeding the size threshold.
`mds_session_metadata_threshold`
  config controls the maximum size that a (encoded) session metadata can grow.

* RGW: New tools have been added to radosgw-admin for identifying and
  correcting issues with versioned bucket indexes. Historical bugs with the
  versioned bucket index transaction workflow made it possible for the index
  to accumulate extraneous "book-keeping" olh entries and plain placeholder
  entries. In some specific scenarios where clients made concurrent requests
  referencing the same object key, it was likely that a lot of extra index
  entries would accumulate. When a significant number of these entries are
  present in a single bucket index shard, they can cause high bucket listing
  latencies and lifecycle processing failures. To check whether a versioned
  bucket has unnecessary olh entries, users can now run ``radosgw-admin
  bucket check olh``. If the ``--fix`` flag is used, the extra entries will
  be safely removed. A distinct issue from the one described thus far, it is
  also possible that some versioned buckets are maintaining extra unlinked
  objects that are not listable from the S3/ Swift APIs. These extra objects
  are typically a result of PUT requests that exited abnormally, in the middle
  of a bucket index transaction - so the client would not have received a
  successful response. Bugs in prior releases made these unlinked objects easy
  to reproduce with any PUT request that was made on a bucket that was actively
  resharding. Besides the extra space that these hidden, unlinked objects
  consume, there can be another side effect in certain scenarios, caused by
  the nature of the failure mode that produced them, where a client of a bucket
  that was a victim of this bug may find the object associated with
the key to7fe91d5d5842e04be3b4f514d6dd990c54b29c76
  be in an inconsistent state. To check whether a versioned bucket has unlinked
  entries, users can now run ``radosgw-admin bucket check unlinked``. If the
  ``--fix`` flag is used, the unlinked objects will be safely removed. Finally,
  a third issue made it possible for versioned bucket index stats to be
  accounted inaccurately. The tooling for recalculating versioned bucket stats
  also had a bug, and was not previously capable of fixing these inaccuracies.
  This release resolves those issues and users can now expect that the existing
  ``radosgw-admin bucket check`` command will produce correct results. We
  recommend that users with versioned buckets, especially those that existed
  on prior releases, use these new tools to check whether their buckets are
  affected and to clean them up accordingly.

* mgr/snap-schedule: For clusters with multiple CephFS file systems, all the
  snap-schedule commands now expect the '--fs' argument.

* RADOS: A POOL_APP_NOT_ENABLED health warning will now be reported  if
  the application is not enabled for the pool irrespective of whether
  the pool is in use or not. Always add ``application`` label to a pool
  to avoid reporting of POOL_APP_NOT_ENABLED health warning for that pool.
  The user might temporarilty mute this warning using
  ``ceph health mute POOL_APP_NOT_ENABLED``.


Getting Ceph

* Git at git://github.com/ceph/ceph.git
* Tarball at https://download.ceph.com/tarballs/ceph-18.2.1.tar.gz
* Containers at https://quay.io/repository/ceph/ceph
* For packages, see https://docs.ceph.com/en/latest/install/get-packages/
* Release git sha1: 7fe91d5d5842e04be3b4f514d6dd990c54b29c76
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm file "/sbin/cephadm", line 10098 PK ^

[Eugen Block]

Hi,

you don't need to change the cephadm file to point to your registry,  
just set the respective config value:


ceph config set global container_image 

or per daemon:

ceph config set mon container_image 
ceph config set rgw container_image 

which usually wouldn't be necessary since it's best to have the same  
version across all ceph daemons. You should also make sure all daemons  
are logged into the registry:


ceph cephadm registry-login [] [] []

Check the docs [1][2] for more information about using your own registry.

Regards,
Eugen

[1] https://docs.ceph.com/en/latest/install/containers/#containers
[2]  
https://docs.ceph.com/en/latest/cephadm/install/#deployment-in-an-isolated-environment


Zitat von farhad kh :


Hello, I downloaded cephadm from the link below.
https://download.ceph.com/rpm-18.2.0/el8/noarch/
I change the address of the images to the address of my private registry,
```
DEFAULT_IMAGE = 'opkbhfpspsp0101.fns/ceph/ceph:v18'
DEFAULT_IMAGE_IS_MAIN = False
DEFAULT_IMAGE_RELEASE = 'reef'
DEFAULT_PROMETHEUS_IMAGE = 'opkbhfpspsp0101.fns/ceph/prometheus:v2.43.0'
DEFAULT_LOKI_IMAGE = 'opkbhfpspsp0101.fns/ceph/loki:2.4.0'
DEFAULT_PROMTAIL_IMAGE = 'opkbhfpspsp0101.fns/ceph/promtail:2.4.0'
DEFAULT_NODE_EXPORTER_IMAGE =
'opkbhfpspsp0101.fns/ceph/node-exporter:v1.5.0'
DEFAULT_ALERT_MANAGER_IMAGE =
'opkbhfpspsp0101.fns/ceph/alertmanager:v0.25.0'
DEFAULT_GRAFANA_IMAGE = 'opkbhfpspsp0101.fns/ceph/ceph-grafana:9.4.7'
DEFAULT_HAPROXY_IMAGE = 'opkbhfpspsp0101.fns/ceph/haproxy:2.3'
DEFAULT_KEEPALIVED_IMAGE = 'opkbhfpspsp0101.fns/ceph/keepalived:2.2.4'
DEFAULT_SNMP_GATEWAY_IMAGE = 'opkbhfpspsp0101.fns/ceph/snmp-notifier:v1.2.1'
DEFAULT_ELASTICSEARCH_IMAGE =
'opkbhfpspsp0101.fns/ceph/elasticsearch:6.8.23'
DEFAULT_JAEGER_COLLECTOR_IMAGE =
'opkbhfpspsp0101.fns/ceph/jaeger-collector:1.29'
DEFAULT_JAEGER_AGENT_IMAGE = 'opkbhfpspsp0101.fns/ceph/jaeger-agent:1.29'
DEFAULT_JAEGER_QUERY_IMAGE = 'opkbhfpspsp0101.fns/ceph/jaeger-query:1.29'
DEFAULT_REGISTRY = 'opkbhfpspsp0101.fns'   # normalize unqualified digests
to this
```
but I encounter this error `
  File "/sbin/cephadm", line 10098
PK
  ^
`. Also, there is a wrong line at the beginning of cephadm file
PK^C^D^T^@^@^@^@^@¥<9a>^CW<8e>^[º^×Ü^E^@×Ü^E^@^K^@^@^@__main__.py#!/usr/bin/python3
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephadm file "/sbin/cephadm", line 10098 PK ^

[farhad kh]

 Hello, I downloaded cephadm from the link below.
https://download.ceph.com/rpm-18.2.0/el8/noarch/
I change the address of the images to the address of my private registry,
```
DEFAULT_IMAGE = 'opkbhfpspsp0101.fns/ceph/ceph:v18'
DEFAULT_IMAGE_IS_MAIN = False
DEFAULT_IMAGE_RELEASE = 'reef'
DEFAULT_PROMETHEUS_IMAGE = 'opkbhfpspsp0101.fns/ceph/prometheus:v2.43.0'
DEFAULT_LOKI_IMAGE = 'opkbhfpspsp0101.fns/ceph/loki:2.4.0'
DEFAULT_PROMTAIL_IMAGE = 'opkbhfpspsp0101.fns/ceph/promtail:2.4.0'
DEFAULT_NODE_EXPORTER_IMAGE =
'opkbhfpspsp0101.fns/ceph/node-exporter:v1.5.0'
DEFAULT_ALERT_MANAGER_IMAGE =
'opkbhfpspsp0101.fns/ceph/alertmanager:v0.25.0'
DEFAULT_GRAFANA_IMAGE = 'opkbhfpspsp0101.fns/ceph/ceph-grafana:9.4.7'
DEFAULT_HAPROXY_IMAGE = 'opkbhfpspsp0101.fns/ceph/haproxy:2.3'
DEFAULT_KEEPALIVED_IMAGE = 'opkbhfpspsp0101.fns/ceph/keepalived:2.2.4'
DEFAULT_SNMP_GATEWAY_IMAGE = 'opkbhfpspsp0101.fns/ceph/snmp-notifier:v1.2.1'
DEFAULT_ELASTICSEARCH_IMAGE =
'opkbhfpspsp0101.fns/ceph/elasticsearch:6.8.23'
DEFAULT_JAEGER_COLLECTOR_IMAGE =
'opkbhfpspsp0101.fns/ceph/jaeger-collector:1.29'
DEFAULT_JAEGER_AGENT_IMAGE = 'opkbhfpspsp0101.fns/ceph/jaeger-agent:1.29'
DEFAULT_JAEGER_QUERY_IMAGE = 'opkbhfpspsp0101.fns/ceph/jaeger-query:1.29'
DEFAULT_REGISTRY = 'opkbhfpspsp0101.fns'   # normalize unqualified digests
to this
```
but I encounter this error `
  File "/sbin/cephadm", line 10098
PK
  ^
`. Also, there is a wrong line at the beginning of cephadm file
PK^C^D^T^@^@^@^@^@¥<9a>^CW<8e>^[º^×Ü^E^@×Ü^E^@^K^@^@^@__main__.py#!/usr/bin/python3
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Cluster Deployment - Recommendation

[Amardeep Singh]

Hi Anthony,

What is recommended number of servers for 2 datahalls for quality cluster?  
replicated pools x 4 will not work here?

Thanks,

Amar

On 18/12/23 21:12, Anthony D'Atri wrote:

EXTERNAL: Do not click links or open attachments if you do not recognize the 
sender.

Four servers doth not a quality cluster make.  This setup will work, but you 
can't use a reasonable EC profile for your bucket pool.  Aim higher than the 
party line wrt PG counts esp. for the index pool.







On Dec 18, 2023, at 10:19, Amardeep Singh 
 wrote:

Hi Everyone,

We are in the process of planning a ceph cluster deployment for our data 
infrastructure.

To provide you with a bit of context, we have deployed hardware across two data 
halls in our data center, and they are connected via a 10Gb interconnect.

The hardware configuration for 4 x Ceph Cluster (2 x servers in both data halls)


*   2 x AMD EPYC 7513 - 32 Cores / 64 Threads

*   512GB RAM

*   2 x 960 GB (OS DISKS)

*   8x Micron 7450 PRO 7680GB NVMe - PCIe Gen4

*   Intel X550-T2 - 10GbE Dual-Port RJ45 Server Adaptor

Our primary usage is Object Gateway and will be running 4 x RGW Service.

We are aiming to deploy using cephadm and utilize all nodes for MON/MGR/RGW and 
OSD's

Given our limited experience with Ceph, we are reaching out to the 
knowledgeable members of this community for recommendations and best practices. 
We would greatly appreciate any insights or advice you can share regarding the 
following aspects:

Cluster Topology: Considering our hardware setup with two data halls connected 
via a 10Gb interconnect, what would be the recommended cluster topology for 
optimal performance and fault tolerance?

Best Practices for Deployment: Are there any recommended best practices for 
deploying Ceph in a similar environment? Any challenges we should be aware of?

Thank you in advance for your time and assistance.

Regards,
Amar

DISCLAIMER: This email and any files transmitted with it are confidential and 
intended solely for the use of the individual or entity to whom they are 
addressed. If you have received this email in error, please notify the sender. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee, you should not 
disseminate, distribute or copy this email. Please notify the sender 
immediately by email if you have received this email by mistake and delete this 
email from your system.

If you are not the intended recipient, you are notified that disclosing, 
copying, distributing or taking any action in reliance on the contents of this 
information is strictly prohibited. Thanks for your cooperation.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io





DISCLAIMER: This email and any files transmitted with it are confidential and 
intended solely for the use of the individual or entity to whom they are 
addressed. If you have received this email in error, please notify the sender. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee, you should not 
disseminate, distribute or copy this email. Please notify the sender 
immediately by email if you have received this email by mistake and delete this 
email from your system.

If you are not the intended recipient, you are notified that disclosing, 
copying, distributing or taking any action in reliance on the contents of this 
information is strictly prohibited. Thanks for your cooperation.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Cluster Deployment - Recommendation

[Zach Underwood]

You may want to think about 100gb between the datahalls and 100gb to the
server. The proxmox guys did some testing and released a paper this month
showing with fast NVMe drives anything short of dual 25gb links using FRR
for routing on this links or 100gb interface you will be bottlenecked by
the network.
https://proxmox.com/images/download/pve/docs/Proxmox-VE-Ceph-Benchmark-202312-rev0.pdf


On Mon, Dec 18, 2023 at 10:20 AM Amardeep Singh <
amardeep.si...@techblue.co.uk> wrote:

>  Hi Everyone,
>
> We are in the process of planning a ceph cluster deployment for our data
> infrastructure.
>
> To provide you with a bit of context, we have deployed hardware across two
> data halls in our data center, and they are connected via a 10Gb
> interconnect.
>
> The hardware configuration for 4 x Ceph Cluster (2 x servers in both data
> halls)
>
>
>   *   2 x AMD EPYC 7513 - 32 Cores / 64 Threads
>
>   *   512GB RAM
>
>   *   2 x 960 GB (OS DISKS)
>
>   *   8x Micron 7450 PRO 7680GB NVMe - PCIe Gen4
>
>   *   Intel X550-T2 - 10GbE Dual-Port RJ45 Server Adaptor
>
> Our primary usage is Object Gateway and will be running 4 x RGW Service.
>
> We are aiming to deploy using cephadm and utilize all nodes for
> MON/MGR/RGW and OSD's
>
> Given our limited experience with Ceph, we are reaching out to the
> knowledgeable members of this community for recommendations and best
> practices. We would greatly appreciate any insights or advice you can share
> regarding the following aspects:
>
> Cluster Topology: Considering our hardware setup with two data halls
> connected via a 10Gb interconnect, what would be the recommended cluster
> topology for optimal performance and fault tolerance?
>
> Best Practices for Deployment: Are there any recommended best practices
> for deploying Ceph in a similar environment? Any challenges we should be
> aware of?
>
> Thank you in advance for your time and assistance.
>
> Regards,
> Amar
>
> DISCLAIMER: This email and any files transmitted with it are confidential
> and intended solely for the use of the individual or entity to whom they
> are addressed. If you have received this email in error, please notify the
> sender. This message contains confidential information and is intended only
> for the individual named. If you are not the named addressee, you should
> not disseminate, distribute or copy this email. Please notify the sender
> immediately by email if you have received this email by mistake and delete
> this email from your system.
>
> If you are not the intended recipient, you are notified that disclosing,
> copying, distributing or taking any action in reliance on the contents of
> this information is strictly prohibited. Thanks for your cooperation.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 
Zach Underwood (RHCE,RHCSA,RHCT,UACA)
My website 
advance-networking.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Cluster Deployment - Recommendation

[Anthony D'Atri]

Four servers doth not a quality cluster make.  This setup will work, but you 
can't use a reasonable EC profile for your bucket pool.  Aim higher than the 
party line wrt PG counts esp. for the index pool.





> On Dec 18, 2023, at 10:19, Amardeep Singh  
> wrote:
> 
> Hi Everyone,
> 
> We are in the process of planning a ceph cluster deployment for our data 
> infrastructure.
> 
> To provide you with a bit of context, we have deployed hardware across two 
> data halls in our data center, and they are connected via a 10Gb interconnect.
> 
> The hardware configuration for 4 x Ceph Cluster (2 x servers in both data 
> halls)
> 
> 
>  *   2 x AMD EPYC 7513 - 32 Cores / 64 Threads
> 
>  *   512GB RAM
> 
>  *   2 x 960 GB (OS DISKS)
> 
>  *   8x Micron 7450 PRO 7680GB NVMe - PCIe Gen4
> 
>  *   Intel X550-T2 - 10GbE Dual-Port RJ45 Server Adaptor
> 
> Our primary usage is Object Gateway and will be running 4 x RGW Service.
> 
> We are aiming to deploy using cephadm and utilize all nodes for MON/MGR/RGW 
> and OSD's
> 
> Given our limited experience with Ceph, we are reaching out to the 
> knowledgeable members of this community for recommendations and best 
> practices. We would greatly appreciate any insights or advice you can share 
> regarding the following aspects:
> 
> Cluster Topology: Considering our hardware setup with two data halls 
> connected via a 10Gb interconnect, what would be the recommended cluster 
> topology for optimal performance and fault tolerance?
> 
> Best Practices for Deployment: Are there any recommended best practices for 
> deploying Ceph in a similar environment? Any challenges we should be aware of?
> 
> Thank you in advance for your time and assistance.
> 
> Regards,
> Amar
> 
> DISCLAIMER: This email and any files transmitted with it are confidential and 
> intended solely for the use of the individual or entity to whom they are 
> addressed. If you have received this email in error, please notify the 
> sender. This message contains confidential information and is intended only 
> for the individual named. If you are not the named addressee, you should not 
> disseminate, distribute or copy this email. Please notify the sender 
> immediately by email if you have received this email by mistake and delete 
> this email from your system.
> 
> If you are not the intended recipient, you are notified that disclosing, 
> copying, distributing or taking any action in reliance on the contents of 
> this information is strictly prohibited. Thanks for your cooperation.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph Cluster Deployment - Recommendation

[Amardeep Singh]

 Hi Everyone,

We are in the process of planning a ceph cluster deployment for our data 
infrastructure.

To provide you with a bit of context, we have deployed hardware across two data 
halls in our data center, and they are connected via a 10Gb interconnect.

The hardware configuration for 4 x Ceph Cluster (2 x servers in both data halls)


  *   2 x AMD EPYC 7513 - 32 Cores / 64 Threads

  *   512GB RAM

  *   2 x 960 GB (OS DISKS)

  *   8x Micron 7450 PRO 7680GB NVMe - PCIe Gen4

  *   Intel X550-T2 - 10GbE Dual-Port RJ45 Server Adaptor

Our primary usage is Object Gateway and will be running 4 x RGW Service.

We are aiming to deploy using cephadm and utilize all nodes for MON/MGR/RGW and 
OSD's

Given our limited experience with Ceph, we are reaching out to the 
knowledgeable members of this community for recommendations and best practices. 
We would greatly appreciate any insights or advice you can share regarding the 
following aspects:

Cluster Topology: Considering our hardware setup with two data halls connected 
via a 10Gb interconnect, what would be the recommended cluster topology for 
optimal performance and fault tolerance?

Best Practices for Deployment: Are there any recommended best practices for 
deploying Ceph in a similar environment? Any challenges we should be aware of?

Thank you in advance for your time and assistance.

Regards,
Amar

DISCLAIMER: This email and any files transmitted with it are confidential and 
intended solely for the use of the individual or entity to whom they are 
addressed. If you have received this email in error, please notify the sender. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee, you should not 
disseminate, distribute or copy this email. Please notify the sender 
immediately by email if you have received this email by mistake and delete this 
email from your system.

If you are not the intended recipient, you are notified that disclosing, 
copying, distributing or taking any action in reliance on the contents of this 
information is strictly prohibited. Thanks for your cooperation.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph 16.2.14: ceph-mgr getting oom-killed

[Zakhar Kirpichenko]

Hi,

Today after 3 weeks of normal operation the mgr reached memory usage of
1600 MB, quickly ballooned to over 100 GB for no apparent reason and got
oom-killed again. There were no suspicious messages in the logs until the
message indicating that the mgr failed to allocate more memory. Any
thoughts?

/Z

On Mon, 11 Dec 2023 at 12:34, Zakhar Kirpichenko  wrote:

> Hi,
>
> Another update: after 2 more weeks the mgr process grew to ~1.5 GB, which
> again was expected:
>
> mgr.ceph01.vankui ceph01  *:8443,9283  running (2w)102s ago   2y
>  1519M-  16.2.14  fc0182d6cda5  3451f8c6c07e
> mgr.ceph02.shsinf ceph02  *:8443,9283  running (2w)102s ago   7M
>   112M-  16.2.14  fc0182d6cda5  1c3d2d83b6df
>
> The cluster is healthy and operating normally, the mgr process is growing
> slowly. It's still unclear what caused the ballooning and OOM issue under
> very similar conditions.
>
> /Z
>
> On Sat, 25 Nov 2023 at 08:31, Zakhar Kirpichenko  wrote:
>
>> Hi,
>>
>> A small update: after disabling 'progress' module the active mgr (on
>> ceph01) used up ~1.3 GB of memory in 3 days, which was expected:
>>
>> mgr.ceph01.vankui ceph01  *:8443,9283  running (3d)  9m ago   2y
>>1284M-  16.2.14  fc0182d6cda5  3451f8c6c07e
>> mgr.ceph02.shsinf ceph02  *:8443,9283  running (3d)  9m ago   7M
>> 374M-  16.2.14  fc0182d6cda5  1c3d2d83b6df
>>
>> The cluster is healthy and operating normally. The mgr process is growing
>> slowly, at roughly about 1-2 MB per 10 minutes give or take, which is not
>> quick enough to balloon to over 100 GB RSS over several days, which likely
>> means that whatever triggers the issue happens randomly and quite suddenly.
>> I'll continue monitoring the mgr and get back with more observations.
>>
>> /Z
>>
>> On Wed, 22 Nov 2023 at 16:33, Zakhar Kirpichenko 
>> wrote:
>>
>>> Thanks for this. This looks similar to what we're observing. Although we
>>> don't use the API apart from the usage by Ceph deployment itself - which I
>>> guess still counts.
>>>
>>> /Z
>>>
>>> On Wed, 22 Nov 2023, 15:22 Adrien Georget, 
>>> wrote:
>>>
 Hi,

 This memory leak with ceph-mgr seems to be due to a change in Ceph
 16.2.12.
 Check this issue : https://tracker.ceph.com/issues/59580
 We are also affected by this, with or without containerized services.

 Cheers,
 Adrien

 Le 22/11/2023 à 14:14, Eugen Block a écrit :
 > One other difference is you use docker, right? We use podman, could
 it
 > be some docker restriction?
 >
 > Zitat von Zakhar Kirpichenko :
 >
 >> It's a 6-node cluster with 96 OSDs, not much I/O, mgr . Each node
 has
 >> 384
 >> GB of RAM, each OSD has a memory target of 16 GB, about 100 GB of
 >> memory,
 >> give or take, is available (mostly used by page cache) on each node
 >> during
 >> normal operation. Nothing unusual there, tbh.
 >>
 >> No unusual mgr modules or settings either, except for disabled
 progress:
 >>
 >> {
 >> "always_on_modules": [
 >> "balancer",
 >> "crash",
 >> "devicehealth",
 >> "orchestrator",
 >> "pg_autoscaler",
 >> "progress",
 >> "rbd_support",
 >> "status",
 >> "telemetry",
 >> "volumes"
 >> ],
 >> "enabled_modules": [
 >> "cephadm",
 >> "dashboard",
 >> "iostat",
 >> "prometheus",
 >> "restful"
 >> ],
 >>
 >> /Z
 >>
 >> On Wed, 22 Nov 2023, 14:52 Eugen Block,  wrote:
 >>
 >>> What does your hardware look like memory-wise? Just for comparison,
 >>> one customer cluster has 4,5 GB in use (middle-sized cluster for
 >>> openstack, 280 OSDs):
 >>>
 >>>  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM
 TIME+
 >>> COMMAND
 >>> 6077 ceph  20   0 6357560 4,522g  22316 S 12,00 1,797
 >>> 57022:54 ceph-mgr
 >>>
 >>> In our own cluster (smaller than that and not really heavily used)
 the
 >>> mgr uses almost 2 GB. So those numbers you have seem relatively
 small.
 >>>
 >>> Zitat von Zakhar Kirpichenko :
 >>>
 >>> > I've disabled the progress module entirely and will see how it
 goes.
 >>> > Otherwise, mgr memory usage keeps increasing slowly, from past
 >>> experience
 >>> > it will stabilize at around 1.5-1.6 GB. Other than this event
 >>> warning,
 >>> it's
 >>> > unclear what could have caused random memory ballooning.
 >>> >
 >>> > /Z
 >>> >
 >>> > On Wed, 22 Nov 2023 at 13:07, Eugen Block  wrote:
 >>> >
 >>> >> I see these progress messages all the time, I don't think they
 cause
 >>> >> it, but I might be wrong. You can disable it just to rule that
 out.
 >>> >>
 >>> >> Zitat von Zakhar Kirpichenko :
 >>> 

[ceph-users] Re: OSD has Rocksdb corruption that crashes ceph-bluestore-tool repair

[Igor Fedotov]

Hi Malcolm,

you might want to try ceph-objectstore-tool's export command to save the 
PG into a file and then import it to another OSD.



Thanks,

Igor

On 18/12/2023 02:59, Malcolm Haak wrote:

Hello all,

I had an OSD go offline due to UWE. When restarting the OSD service,
to try and at least get it to  drain cleanly of that data that wasn't
damaged, the ceph-osd process would crash.

I then attempted to repair it using ceph-bluestore-tool. I can run
fsck and it will complete without issue, however when attempting to
run repair it crashes in the exact same way that ceph-osd crashes.

I'll attach the tail end of the output here:

  2023-12-17T20:24:53.320+1000 7fdb7bf17740 -1 rocksdb: submit_common
error: Corruption: block checksum mismatch: stored = 1106056583,
computed = 657190205, type = 1  in db/020524.sst offset 21626321 size
4014 code =  Rocksdb transaction:
PutCF( prefix = S key = 'per_pool_omap' value size = 1)
   -442> 2023-12-17T20:24:53.386+1000 7fdb7bf17740 -1
/usr/src/debug/ceph/ceph-18.2.0/src/os/bluestore/BlueStore.cc: In
function 'unsigned int BlueStoreRepairer::apply(KeyValueDB*)' thread
7fdb7bf17740 time 2023-12-17T20:24:53.341999+1000
/usr/src/debug/ceph/ceph-18.2.0/src/os/bluestore/BlueStore.cc: 17982:
FAILED ceph_assert(ok)

  ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef
(stable)
  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x136) [0x7fdb7b6502c9]
  2: /usr/lib/ceph/libceph-common.so.2(+0x2504a4) [0x7fdb7b6504a4]
  3: (BlueStoreRepairer::apply(KeyValueDB*)+0x5af) [0x559afb98cc7f]
  4: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x45fc)
[0x559afba2436c]
  5: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x204)
[0x559afba31014]
  6: main()
  7: /usr/lib/libc.so.6(+0x27cd0) [0x7fdb7ae45cd0]
  8: __libc_start_main()
  9: _start()

   -441> 2023-12-17T20:24:53.390+1000 7fdb7bf17740 -1 *** Caught signal
(Aborted) **
  in thread 7fdb7bf17740 thread_name:ceph-bluestore-

  ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef
(stable)
  1: /usr/lib/libc.so.6(+0x3e710) [0x7fdb7ae5c710]
  2: /usr/lib/libc.so.6(+0x8e83c) [0x7fdb7aeac83c]
  3: raise()
  4: abort()
  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x191) [0x7fdb7b650324]
  6: /usr/lib/ceph/libceph-common.so.2(+0x2504a4) [0x7fdb7b6504a4]
  7: (BlueStoreRepairer::apply(KeyValueDB*)+0x5af) [0x559afb98cc7f]
  8: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x45fc)
[0x559afba2436c]
  9: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x204)
[0x559afba31014]
  10: main()
  11: /usr/lib/libc.so.6(+0x27cd0) [0x7fdb7ae45cd0]
  12: __libc_start_main()
  13: _start()
  NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.

The reason I need to get this OSD functioning is I had two other OSD's
fail causing a single PG to be in down state. The weird thing is, I
got one of those back up without issue (ceph-osd crashed due to root
filling and alert not sending) but the PG is still down. So I need to
get this other one back up (or the data extracted) to get that PG back
from down.

Thanks in advance
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io