[ceph-users] Dashboard and Object Gateway

[Tim Holloway]

First, an abject apology for the horrors I'm about to unveil. I made a
cold migration from GlusterFS to Ceph a few months back, so it was a
learn-/screwup/-as-you-go affair.

For reasons of presumed compatibility with some of my older servers, I
started with Ceph Octopus. Unfortunately, Octopus seems to have been a
nexus of transitions from older Ceph organization and management to a
newer (cephadm) system combined with a relocation of many ceph
resources and compounded by stale bits of documentation (notably some
references to SysV procedures and an obsolete installer that doesn't
even come with Octopus).

A far bigger problem was a known issue where actions would be scheduled
but never executed if the system was even slightly dirty. And of
course, since my system was hopelessly dirty, that was a major issue.
Finally I took a risk and bumped up to Pacific, where that issue no
longer exists. I won't say that I'm 100% clean even now, but at least
the remaining crud is in areas where it cannot do any harm. Presumably.

Given that, the only bar now remaining to total joy has been my
inability to connect via the Ceph Dashboard to the Object Gateway.

This seems to be an oft-reported problem, but generally referenced
relative to higher-level administrative interfaces like Kubernetes and
rook. I'm interfacing more directly, however. Regardless, the error
reported is notably familiar:

[quote]
The Object Gateway Service is not configured
Error connecting to Object Gateway: RGW REST API failed request with
status code 404
(b'{"Code":"NoSuchBucket","Message":"","BucketName":"default","RequestI
d":"tx00' b'000dd0c65b8bda685b4-00652d8e0f-5e3a9b-
default","HostId":"5e3a9b-default-defa' b'ult"}')
Please consult the documentation on how to configure and enable the
Object Gateway management functionality. 
[/quote]

In point of fact, what this REALLY means in my case is that the bucket
that is supposed to contain the necessary information for the dashboard
and rgw to communicate has not been created. Presumably that SHOULDhave
been done by the "ceph dashboard set-rgw-credentials" command, but
apparently isn't, because the default zone has no buckets at all, much
less one named "default".

By way of reference, the dashboard is definitely trying to interact
with the rgw container, because trying object gateway options on the
dashboard result in the container logging the following.

beast: 0x7efd29621620: 10.0.1.16 - dashboard [16/Oct/2023:19:25:03.678
+] "GET /default/metadata/user?myself HTTP/1.1" 404

To make everything happy, I'd be glad to accept instructions on how to
manually brute-force construct this bucket.

Of course, as a cleaner long-term solution, it would be nice if the
failure to create could be detected and logged.

And of course, the ultimate solution: something that would assist in
making whatever processes are unhappy be happy.

Thanks,
  Tim
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Dashboard and Object Gateway

[Tim Holloway]

Thank you, Ondřej!

Yes, I set the admin entry set to "default". It's just the latest
result of failed attempts ("admin" didn't work for me either). I did
say there were some horrors in there!

If I got your sample URL pattern right, the results of a GET on
"http://x.y.z/default"; return 404, NoSuchBucket. If that means that I
didn't properly set rgw_enable_apis, then I probably don't know how to
set it right.

   Best Regards,
  Tim

On Tue, 2023-10-17 at 08:35 +0200, Ondřej Kukla wrote:
> Hello Tim,
> 
> I was also struggling with this when I was configuring the object
> gateway for the first time.
> 
> There is a few things that you should check to make sure the
> dashboard would work.
> 
> 1. You need to have the admin api enabled on all rgws with the
> rgw_enable_apis option. (As far as I know you are not able to force
> the dashboard to use one rgw instance)
> 2. It seems that you have the rgw_admin_entry set to a non default
> value - the default is admin but it seems that you have “default" (by
> the name of the bucket) make sure that you have this also set on all
> rgws.
> 
> You can confirm that both of these settings are set properly by
> sending GET request to ${rgw-ip}:${port}/${rgw_admin_entry} 
> “default" in your case -> it should return 405 Method Not Supported
> 
> Btw there is actually no bucket that you would be able to see in the
> administration. It’s just abstraction on the rgw.
> 
> Reagards,
> 
> Ondrej
> 
> > On 16. 10. 2023, at 22:00, Tim Holloway  wrote:
> > 
> > First, an abject apology for the horrors I'm about to unveil. I
> > made a
> > cold migration from GlusterFS to Ceph a few months back, so it was
> > a
> > learn-/screwup/-as-you-go affair.
> > 
> > For reasons of presumed compatibility with some of my older
> > servers, I
> > started with Ceph Octopus. Unfortunately, Octopus seems to have
> > been a
> > nexus of transitions from older Ceph organization and management to
> > a
> > newer (cephadm) system combined with a relocation of many ceph
> > resources and compounded by stale bits of documentation (notably
> > some
> > references to SysV procedures and an obsolete installer that
> > doesn't
> > even come with Octopus).
> > 
> > A far bigger problem was a known issue where actions would be
> > scheduled
> > but never executed if the system was even slightly dirty. And of
> > course, since my system was hopelessly dirty, that was a major
> > issue.
> > Finally I took a risk and bumped up to Pacific, where that issue no
> > longer exists. I won't say that I'm 100% clean even now, but at
> > least
> > the remaining crud is in areas where it cannot do any harm.
> > Presumably.
> > 
> > Given that, the only bar now remaining to total joy has been my
> > inability to connect via the Ceph Dashboard to the Object Gateway.
> > 
> > This seems to be an oft-reported problem, but generally referenced
> > relative to higher-level administrative interfaces like Kubernetes
> > and
> > rook. I'm interfacing more directly, however. Regardless, the error
> > reported is notably familiar:
> > 
> > [quote]
> > The Object Gateway Service is not configured
> > Error connecting to Object Gateway: RGW REST API failed request
> > with
> > status code 404
> > (b'{"Code":"NoSuchBucket","Message":"","BucketName":"default","Requ
> > estI
> > d":"tx00' b'000dd0c65b8bda685b4-00652d8e0f-5e3a9b-
> > default","HostId":"5e3a9b-default-defa' b'ult"}')
> > Please consult the documentation on how to configure and enable the
> > Object Gateway management functionality. 
> > [/quote]
> > 
> > In point of fact, what this REALLY means in my case is that the
> > bucket
> > that is supposed to contain the necessary information for the
> > dashboard
> > and rgw to communicate has not been created. Presumably that
> > SHOULDhave
> > been done by the "ceph dashboard set-rgw-credentials" command, but
> > apparently isn't, because the default zone has no buckets at all,
> > much
> > less one named "default".
> > 
> > By way of reference, the dashboard is definitely trying to interact
> > with the rgw container, because trying object gateway options on
> > the
> > dashboard result in the container logging the following.
> > 
> > beast: 0x7efd29621620: 10.0.1.16 - dashboard
&

[ceph-users] Re: Dashboard and Object Gateway

[Tim Holloway]

Thanks, Casey!

I'm not really certain where to set this option. While Ceph is very
well-behaved once you know what to do, the nature of Internet-based
documentation (and occasionally incompletely-updated manuals) is that
stale information is often given equal weight to the obsolete
information. It's a problem I had as support for JavaServer Faces, in
fact. I spent literally years correcting people who'd got their
examples from obsoleted sources.

If I was to concoct a "Really, Really Newbies Intro to Ceph" I think
that the two most fundamental items explained would be "Ceph as
traditional services" versus "Ceph as Containerized services" (As far
as I can tell, both are still viable but containerization - at least
for me - is a preferable approach). And the ceph.conf file versus
storing operational parameters within Ceph entities (e.g. buckets or
pseudo-buckets like RGW is doing). While lots of stuff still reference
ceph.conf for configuration, I'm feeling like it's actually no longer
authoritative for some options, may be an alternative source for others
(with which source has priority being unclear) and stuff that Ceph no
longer even looks at because it has moved on.

Such is my plight.

I have no problem with making the administrative interface look
"bucket-like". Or for that matter, having the RGW report it as a
(missing) bucket if it isn't configured. But knowing where to inject
the magic that activates that interface eludes me and whether to do it
directly on the RGW container hos (and how) or on my master host is
totally unclear to me. It doesn't help that this is an item that has
multiple values, not just on/off or that by default the docs seem to
imply it should be already preset to standard values out of the box.

   Thanks,
  Tim

On Tue, 2023-10-17 at 09:11 -0400, Casey Bodley wrote:
> hey Tim,
> 
> your changes to rgw_admin_entry probably aren't taking effect on the
> running radosgws. you'd need to restart them in order to set up the
> new route
> 
> there also seems to be some confusion about the need for a bucket
> named 'default'. radosgw just routes requests with paths starting
> with
> '/{rgw_admin_entry}' to a separate set of admin-related rest apis.
> otherwise they fall back to the s3 api, which treats '/foo' as a
> request for bucket foo - that's why you see NoSuchBucket errors when
> it's misconfigured
> 
> also note that, because of how these apis are nested,
> rgw_admin_entry='default' would prevent users from creating and
> operating on a bucket named 'default'
> 
> On Tue, Oct 17, 2023 at 7:03 AM Tim Holloway 
> wrote:
> > 
> > Thank you, Ondřej!
> > 
> > Yes, I set the admin entry set to "default". It's just the latest
> > result of failed attempts ("admin" didn't work for me either). I
> > did
> > say there were some horrors in there!
> > 
> > If I got your sample URL pattern right, the results of a GET on
> > "http://x.y.z/default"; return 404, NoSuchBucket. If that means that
> > I
> > didn't properly set rgw_enable_apis, then I probably don't know how
> > to
> > set it right.
> > 
> >    Best Regards,
> >   Tim
> > 
> > On Tue, 2023-10-17 at 08:35 +0200, Ondřej Kukla wrote:
> > > Hello Tim,
> > > 
> > > I was also struggling with this when I was configuring the object
> > > gateway for the first time.
> > > 
> > > There is a few things that you should check to make sure the
> > > dashboard would work.
> > > 
> > > 1. You need to have the admin api enabled on all rgws with the
> > > rgw_enable_apis option. (As far as I know you are not able to
> > > force
> > > the dashboard to use one rgw instance)
> > > 2. It seems that you have the rgw_admin_entry set to a non
> > > default
> > > value - the default is admin but it seems that you have “default"
> > > (by
> > > the name of the bucket) make sure that you have this also set on
> > > all
> > > rgws.
> > > 
> > > You can confirm that both of these settings are set properly by
> > > sending GET request to ${rgw-ip}:${port}/${rgw_admin_entry}
> > > “default" in your case -> it should return 405 Method Not
> > > Supported
> > > 
> > > Btw there is actually no bucket that you would be able to see in
> > > the
> > > administration. It’s just abstraction on the rgw.
> > > 
> > > Reagards,
> > > 
> > > Ondrej
> > > 
> > > > On 16. 

[ceph-users] Re: Dashboard and Object Gateway

[Tim Holloway]

SOLVED!

OK, there was some last-minute flailing around so I can't quite report
a cookbook recipe, but it goes something like this:


1. ceph config set client.mousetech rgw_admin_entry admin

Note: the standard example is for client.rgw, but I named my RGW
"mousetech" to make it distinguishable from possible internal magic
names. Tried both "client.mousetech" and "client.rgw" just to be sure.

Peeve: the "ceph config" command is silent. You cannot tell if it
successfully updated (or added?) a value and when it simply threw away
a bad config request.

2. Re-issued "ceph dashboard set-rgw-credentials" just to jam in
anything that might not have been set right previously. 

3. Restarted the RGW container on the RGW host.

I got to the point where I got the "503" error, but still no dashboard
even after logging in and out. Noted that the RGW logs were reporting
my manual requests, but the dashboard requests weren't showing up. Huh?

Head scratching. Realized that I can't get a "404" error unless there's
a host on the other end, so SOMETHING was talking and it didn't make
any sense that the RGW only logged some requests and not others.

Finally got a suspicion and did a "ceph orch ps". Yup, I have TWO RGW
servers running now (it was intentional, partly to flush out some of
the older failures). So I dialed into the alternate RGW, restarted it
and Dashboard Joy!

So basically, everything I wanted Ceph to do is working and working
clean, from the Ceph filesystem to NFS to VM backing stores to RGW and
I'm delirious with joy.

  Thanks, guys!

On Tue, 2023-10-17 at 10:23 -0400, Casey Bodley wrote:
> you're right that many docs still mention ceph.conf, after the mimic
> release added a centralized config database to ceph-mon. you can read
> about the mon-based 'ceph config' commands in
> https://docs.ceph.com/en/reef/rados/configuration/ceph-conf/#commands
> 
> to modify rgw_admin_entry for all radosgw instances, you'd use a
> command like:
> 
> $ ceph config set client.rgw rgw_admin_entry admin
> 
> then restart radosgws because they only read that value on startup
> 
> On Tue, Oct 17, 2023 at 9:54 AM Tim Holloway 
> wrote:
> > 
> > Thanks, Casey!
> > 
> > I'm not really certain where to set this option. While Ceph is very
> > well-behaved once you know what to do, the nature of Internet-based
> > documentation (and occasionally incompletely-updated manuals) is
> > that
> > stale information is often given equal weight to the obsolete
> > information. It's a problem I had as support for JavaServer Faces,
> > in
> > fact. I spent literally years correcting people who'd got their
> > examples from obsoleted sources.
> > 
> > If I was to concoct a "Really, Really Newbies Intro to Ceph" I
> > think
> > that the two most fundamental items explained would be "Ceph as
> > traditional services" versus "Ceph as Containerized services" (As
> > far
> > as I can tell, both are still viable but containerization - at
> > least
> > for me - is a preferable approach). And the ceph.conf file versus
> > storing operational parameters within Ceph entities (e.g. buckets
> > or
> > pseudo-buckets like RGW is doing). While lots of stuff still
> > reference
> > ceph.conf for configuration, I'm feeling like it's actually no
> > longer
> > authoritative for some options, may be an alternative source for
> > others
> > (with which source has priority being unclear) and stuff that Ceph
> > no
> > longer even looks at because it has moved on.
> > 
> > Such is my plight.
> > 
> > I have no problem with making the administrative interface look
> > "bucket-like". Or for that matter, having the RGW report it as a
> > (missing) bucket if it isn't configured. But knowing where to
> > inject
> > the magic that activates that interface eludes me and whether to do
> > it
> > directly on the RGW container hos (and how) or on my master host is
> > totally unclear to me. It doesn't help that this is an item that
> > has
> > multiple values, not just on/off or that by default the docs seem
> > to
> > imply it should be already preset to standard values out of the
> > box.
> > 
> >    Thanks,
> >   Tim
> > 
> > On Tue, 2023-10-17 at 09:11 -0400, Casey Bodley wrote:
> > > hey Tim,
> > > 
> > > your changes to rgw_admin_entry probably aren't taking effect on
> > > the
> > > running radosgws. you'd need to restart them in order to set up
>

[ceph-users] Re: Nautilus - Octopus upgrade - more questions

[Tim Holloway]

I started with Octopus. It had one very serious flaw that I only fixed
by having Ceph self-upgrade to Pacific. Octopus required perfect health
to alter daemons and often the health problems were themselves issues
with daemons. Pacific can overlook most of those problems, so it's a
lot easier to repair stuff.

Nautilus is also the highest level supported natively by CentOS 7 (I
went with Octopus because I mis-read that). I can attest that a Ceph FS
mount under the Nautilus packages to Octopus and Pacific seem to work
just find, incidentally.

I can't use Ansible myself, because my hardware distribution is too
irregular, so I'm doing everything from Ceph commands.

If you are currently running on Ceph as installed services rather than
containerized, I should also note that apparently the version of Docker
for CentOS7 (which I'll take as a rough equivalent to what you've got)
cannot properly run the Ceph containers. AlmaLinux 8, however has no
problems.

Since I've yet to migrate most of my hosts off CentOS 7, I run most of
Ceph in VMs running AlmaLinux 8. The OSD storage I mount as raw disks
to avoid the extra layer that virtualizing them would entail. The raw
disks are themselves LVM logical volumes as I don't have dedicated
physical drives. It's a work in progress, and a messy one.

I don't know if you can do incremental migration with mixed
Nautilus/Pacific OSDs, if Nautilus supports Ceph's internal upgrade
(Octopus does), or if it's best to just crank up a fresh Ceph and
migrate the data via something like rsync (which I did, since I was
moving from glusterfs to Ceph). Maybe someone with more knowledge of
the internals can answer these questions.

   Tim

On Tue, 2023-10-17 at 20:18 -0400, Dave Hall wrote:
> Hello,
> 
> I have a Nautilus cluster built using Ceph packages from Debian 10
> Backports, deployed with Ceph-Ansible.
> 
> I see that Debian does not offer Ceph 15/Octopus packages.  However,
> download.ceph.com does offer such packages.
> 
> Question:  Is it a safe upgrade to install the download.ceph.com
> packages
> over top of the buster-backports packages?
> 
> If so, the next question is how to deploy this?  Should I pull down
> an
> appropriate version of Ceph-Ansible and use the rolling-upgrade
> playbook?
> Or just apg-get -f dist-upgrade the new Ceph packages into place?
> 
> BTW, in the long run I'll probably want to get to container-based
> Reef, but
> I need to keep a stable cluster throughout.
> 
> Any advice or reassurance much appreciated.
> 
> Thanks.
> 
> -Dave
> 
> --
> Dave Hall
> Binghamton University
> kdh...@binghamton.edu
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Logging control

[Tim Holloway]

Ceph version is Pacific (16.2.14), upgraded from a sloppy Octopus.

I ran afoul of all the best bugs in Octopus, and in the process
switched on a lot of stuff better left alone, including some detailed
debug logging. Now I can't turn it off.

I am confidently informed by the documentation that the first step
would be the command:

ceph daemon osd.1 config show | less

But instead of config information I get back:

Can't get admin socket path: unable to get conf option admin_socket for
osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid
types are: auth, mon, osd, mds, mgr, client\n"

Which seems to be kind of insane.

Attempting to get daemon config info on a monitor on that machine
gives:

admin_socket: exception getting command descriptions: [Errno 2] No such
file or directory

Which doesn't help either.

Anyone got an idea?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Logging control

[Tim Holloway]

The problem with "ceph daemon" is that the results I listed DID come
from running the command on the same machine as the daemon.

But "ceph tell" seems to be more promising.

There's more to the story, since I tried to do blind brute-force
adjustments and they failed also, but let me see if ceph tell gives a
better idea of what I should be doing.

   Tim
On Tue, 2023-12-19 at 16:21 -0500, Wesley Dillingham wrote:
> "ceph daemon" commands need to be run local to the machine where the
> daemon is running. So in this case if you arent on the node where
> osd.1 lives it wouldnt work. "ceph tell" should work anywhere there
> is a client.admin key.
> 
> 
> Respectfully,
> 
> Wes Dillingham
> w...@wesdillingham.com
> LinkedIn
> 
> 
> On Tue, Dec 19, 2023 at 4:02 PM Tim Holloway 
> wrote:
> > Ceph version is Pacific (16.2.14), upgraded from a sloppy Octopus.
> > 
> > I ran afoul of all the best bugs in Octopus, and in the process
> > switched on a lot of stuff better left alone, including some
> > detailed
> > debug logging. Now I can't turn it off.
> > 
> > I am confidently informed by the documentation that the first step
> > would be the command:
> > 
> > ceph daemon osd.1 config show | less
> > 
> > But instead of config information I get back:
> > 
> > Can't get admin socket path: unable to get conf option admin_socket
> > for
> > osd: b"error parsing 'osd': expected string of the form TYPE.ID,
> > valid
> > types are: auth, mon, osd, mds, mgr, client\n"
> > 
> > Which seems to be kind of insane.
> > 
> > Attempting to get daemon config info on a monitor on that machine
> > gives:
> > 
> > admin_socket: exception getting command descriptions: [Errno 2] No
> > such
> > file or directory
> > 
> > Which doesn't help either.
> > 
> > Anyone got an idea?
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Logging control

[Tim Holloway]

-
9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:11:01.548+
7fa1d9fc7700  0 log_channel(cluster) log [DBG] : pgmap v166: 649 pgs: 1
active+clean+scrubbi>
Dec 19 17:11:02 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-19T22:11:01.549680+
mgr.xyz1 (mgr.6889303) 183 : cluster [DBG] pgmap v166: 649 pgs: 1
active+clean+scrubbin>



On Tue, 2023-12-19 at 16:21 -0500, Wesley Dillingham wrote:
> "ceph daemon" commands need to be run local to the machine where the
> daemon
> is running. So in this case if you arent on the node where osd.1
> lives it
> wouldnt work. "ceph tell" should work anywhere there is a
> client.admin key.
> 
> 
> Respectfully,
> 
> *Wes Dillingham*
> w...@wesdillingham.com
> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
> 
> 
> On Tue, Dec 19, 2023 at 4:02 PM Tim Holloway 
> wrote:
> 
> > Ceph version is Pacific (16.2.14), upgraded from a sloppy Octopus.
> > 
> > I ran afoul of all the best bugs in Octopus, and in the process
> > switched on a lot of stuff better left alone, including some
> > detailed
> > debug logging. Now I can't turn it off.
> > 
> > I am confidently informed by the documentation that the first step
> > would be the command:
> > 
> > ceph daemon osd.1 config show | less
> > 
> > But instead of config information I get back:
> > 
> > Can't get admin socket path: unable to get conf option admin_socket
> > for
> > osd: b"error parsing 'osd': expected string of the form TYPE.ID,
> > valid
> > types are: auth, mon, osd, mds, mgr, client\n"
> > 
> > Which seems to be kind of insane.
> > 
> > Attempting to get daemon config info on a monitor on that machine
> > gives:
> > 
> > admin_socket: exception getting command descriptions: [Errno 2] No
> > such
> > file or directory
> > 
> > Which doesn't help either.
> > 
> > Anyone got an idea?
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> > 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Support of SNMP on CEPH ansible

[Tim Holloway]

I can't speak for details of ceph-ansible. I don't use it because from
what I can see, ceph-ansible requires a lot more symmetry in the server
farm than I have.

It is, however, my understanding that cephadm is the preferred
installation and management option these days and it certainly helped
me to do the migration.

The actual SNMP subsystem for Ceph is based on listening to Prometheus
alerts and translating them into SNMP traps
(https://github.com/maxwo/snmp_notifier). Should be fairly version-
independent I'd think. If ceph-ansible doesn't support it, I expect
than an ansible playbook to do the job would be relatively simple.

Just as an alternative, since I don't use SNMP for Ceph, and I do use
Nagios as my general systems monitor, I did a Nagios NRPE script that
simply runs "ceph health" and looks for "OK", "WARN" or "ERROR" in the
response, which translates to Nagios alert levels. If Nagios alerts me,
I then go to the Ceph Control panel for details. Which I'd think is a
lot less work than dissecting traps, though I can tolerate a 5-minute
delay for polling on alerts here. Your Mileage May Vary.

On Tue, 2023-12-19 at 08:10 +, Eugen Block wrote:
> Hi,
> 
> I don't have an answer for the SNMP part, I guess you could just
> bring  
> up your own snmp daemon and configure it to your needs. As for the  
> orchestrator backend you have these three options (I don't know what 
> "test_orchestrator" does but it doesn't sound like it should be used 
> in production):
> 
>  enum_allowed=['cephadm', 'rook', 'test_orchestrator'],
> 
> If you intend to use the orchestrator I suggest to move to cephadm  
> (you can convert an existing cluster by following this guide:  
> https://docs.ceph.com/en/latest/cephadm/adoption/). Although the  
> orchestrator is "on" it requires a backend.
> 
> Regards,
> Eugen
> 
> Zitat von Lokendra Rathour :
> 
> > Hi Team,
> > please help in the reference of the issue raised.
> > 
> > 
> > Best Regards,
> > Lokendra
> > 
> > On Wed, Dec 13, 2023 at 2:33 PM Kushagr Gupta
> > 
> > wrote:
> > 
> > > Hi Team,
> > > 
> > > *Environment:*
> > > We have deployed a ceph setup using ceph-ansible.
> > > Ceph-version: 18.2.0
> > > OS: Almalinux 8.8
> > > We have a 3 node-setup.
> > > 
> > > *Queries:*
> > > 
> > > 1. Is SNMP supported for ceph-ansible?Is there some other way to
> > > setup
> > > SNMP gateway for the ceph cluster?
> > > 2. Do we have a procedure to set backend for ceph-orchestrator
> > > via
> > > ceph-ansible? Which backend to use?
> > > 3. Are there any CEPH MIB files which work independent of
> > > prometheus.
> > > 
> > > 
> > > *Description:*
> > > We are trying to perform SNMP monitoring for the ceph cluster
> > > using the
> > > following link:
> > > 
> > > 1.
> > > https://docs.ceph.com/en/quincy/cephadm/services/snmp-gateway/#:~:text=Ceph's%20SNMP%20integration%20focuses%20on,a%20designated%20SNMP%20management%20platform
> > > .
> > > 2.
> > > https://www.ibm.com/docs/en/storage-ceph/7?topic=traps-deploying-snmp-gateway
> > > 
> > > But when we try to follow the steps mentioned in the above link,
> > > we get
> > > the following error when we try to run any "ceph orch" we get the
> > > following
> > > error:
> > > "Error ENOENT: No orchestrator configured (try `ceph orch set
> > > backend`)"
> > > 
> > > After going through following links:
> > > 1.
> > > https://www.ibm.com/docs/en/storage-ceph/5?topic=operations-use-ceph-orchestrator
> > > 2.
> > > https://forum.proxmox.com/threads/ceph-mgr-orchestrator-enabled-but-showing-missing.119145/
> > > 3. https://docs.ceph.com/en/latest/mgr/orchestrator_modules/
> > > I think since we have deployed the cluster using ceph-ansible, we
> > > can't
> > > use the ceph-orch commands.
> > > When we checked in the cluster, the following are the enabled
> > > modules:
> > > "
> > > [root@storagenode1 ~]# ceph mgr module ls
> > > MODULE
> > > balancer   on (always on)
> > > crash  on (always on)
> > > devicehealth   on (always on)
> > > orchestrator   on (always on)
> > > pg_autoscaler  on (always on)
> > > progress   on (always on)
> > > rbd_support    on (always on)
> > > status on (always on)
> > > telemetry  on (always on)
> > > volumes    on (always on)
> > > alerts on
> > > iostat on
> > > nfs    on
> > > prometheus on
> > > restful    on
> > > dashboard  -
> > > influx -
> > > insights   -
> > > localpool  -
> > > mds_autoscaler -
> > > mirroring  -
> > > osd_perf_query -
> > > osd_support    -
> > > rgw    -
> > > selftest   -
> > > snap_schedule  -
> > > stats  -
> > > telegraf   -
> > > test_orchestrator  -
> > > zabbix -
> > > [root@storagenode1 ~]#
> > > "
> > > As can be seen above, orchestrator is on.
> > > 
> > > Also, We were exploring more about snmp and as per the file:
>

[ceph-users] Re: Logging control

[Tim Holloway]

Thanks, that is most useful to know!

The Ceph docs are very good except when they propagate obsolete
information. For example, using "ceph-deploy" on Octopus (my copy
didn't come with ceph-deploy - it used cephadm).

And, alas, nothing has been written to delineate differences between
containerized resource management versus "bare" resource management.

Hopefully that will get fixed someday. Ideally, it will be both fixed
and applied backwards, where applicable.

   Tim

On Wed, 2023-12-20 at 12:32 +, Eugen Block wrote:
> Just to add a bit more information, the 'ceph daemon' command is
> still  
> valid, it just has to be issued inside of the containers:
> 
> quincy-1:~ # cephadm enter --name osd.0
> Inferring fsid 1e6e5cb6-73e8-11ee-b195-fa163ee43e22
> [ceph: root@quincy-1 /]# ceph daemon osd.0 config diff | head
> {
>  "diff": {
>  "bluestore_block_db_size": {
>  "default": "0",
>  "mon": "20",
>  "final": "20"
>  },
> 
> Or with the cephadm shell:
> 
> quincy-1:~ # cephadm shell --name osd.0 -- ceph daemon osd.0 config  
> diff | head
> 
> But 'ceph tell' should work as well, I just wanted to show some more 
> context a options.
> 
> As for the debug messages, there are a couple of things to tweak as  
> you may have noticed. For example, you could reduce the log level of 
> debug_rocksdb (default 4/5). If you want to reduce the
> mgr_tick_period  
> (the repeating health messages every two seconds) you can do that
> like  
> this:
> 
> quincy-1:~ # ceph config set mgr mgr_tick_period 10
> 
> But don't use too large periods, I had mentioned that in a recent  
> thread. 10 seconds seem to work just fine for me, though.
> 
> 
> 
> Zitat von Tim Holloway :
> 
> > OK. Found some loglevel overrides in the monitor and reset them.
> > 
> > Restarted the mgr and monitor just in case.
> > 
> > Still getting a lot of stuff that looks like this.
> > 
> > Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mon-xyz1[1906961]: debug 2023-12-19T22:10:51.314+
> > 7f36d7291700  4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443]
> > [default] Manual compact>
> > Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mon-xyz1[1906961]: debug 2023-12-19T22:10:51.314+
> > 7f36d7291700  4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443]
> > [default] Manual compact>
> > Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mon-xyz1[1906961]: debug 2023-12-19T22:10:51.314+
> > 7f36d7291700  4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443]
> > [default] Manual compact>
> > Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mon-xyz1[1906961]: debug 2023-12-19T22:10:51.314+
> > 7f36d7291700  4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443]
> > [default] Manual compact>
> > Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-
> > 19T22:10:49.542670+
> > mgr.xyz1 (mgr.6889303) 177 : cluster [DBG] pgmap v160: 649 pgs: 1
> > active+clean+scrubbin>
> > Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:10:51.542+
> > 7fa1d9fc7700  0 log_channel(cluster) log [DBG] : pgmap v161: 649
> > pgs: 1
> > active+clean+scrubbi>
> > Dec 19 17:10:52 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-
> > 19T22:10:51.543403+
> > mgr.xyz1 (mgr.6889303) 178 : cluster [DBG] pgmap v161: 649 pgs: 1
> > active+clean+scrubbin>
> > Dec 19 17:10:52 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:10:52.748+
> > 7fa1c74a2700  0 [progress INFO root] Processing OSDMap change
> > 20239..20239
> > Dec 19 17:10:53 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:10:53.544+
> > 7fa1d9fc7700  0 log_channel(cluster) log [DBG] : pgmap v162: 649
> > pgs: 1
> > active+clean+scrubbi>
> > Dec 19 17:10:54 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-
> > 19T22:10:53.545649+
> > mgr.xyz1 (mgr.6889303) 179 : cluster [DBG] pgmap v162: 649 pgs: 1
> > active+clean+scrubbin>
> > Dec 19 17:10:55 xyz1.mousetec

[ceph-users] Adding a new monitor fails

[Tim Holloway]

I just jacked in a completely new, clean server and I've been trying to
get a Ceph (Pacific) monitor running on it.

The "ceph orch daemon add" appears to install all/most of what's
necessary, but when the monitor starts, it shuts down immediately, and
in the manner of Ceph containers immediately erases itself and the
container log, so it's not possible to see what its problem is.

I looked at manual installation, but the docs appear to be oriented
towards old-style non-container implementation and don't account for
the newer /var/lib/ceph/*fsid*/ approach.

Any tips?

Last few lines in the system journal are like this:

Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.938+
7f26810ae700  4 rocksdb: (Original Log Time 2024/02/06-16:09:58.938432)
[compaction/compaction_job.cc:760] [default] compacted to: base level 6
level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 2]
max score 0.00, MB/sec: 351.7 rd, 351.7 wr, level 6, files in(4, 0)
out(2) MB in(92.8, 0.0) out(92.8), read-write-amplify(2.0) write-
amplify(1.0) OK, records in: 2858, records dropped: 0
output_compression: NoCompression
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: 
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.938+
7f26810ae700  4 rocksdb: (Original Log Time 2024/02/06-16:09:58.938452)
EVENT_LOG_v1 {"time_micros": 1707235798938446, "job": 6, "event":
"compaction_finished", "compaction_time_micros": 276718,
"compaction_time_cpu_micros": 73663, "output_level": 6,
"num_output_files": 2, "total_output_size": 97309398,
"num_input_records": 2858, "num_output_records": 2858,
"num_subcompactions": 1, "output_compression": "NoCompression",
"num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0,
"lsm_state": [0, 0, 0, 0, 0, 0, 2]}
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.940+
7f26810ae700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798941291,
"job": 6, "event": "table_file_deletion", "file_number": 14}
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.943+
7f26810ae700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798943980,
"job": 6, "event": "table_file_deletion", "file_number": 12}
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.946+
7f26810ae700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798946734,
"job": 6, "event": "table_file_deletion", "file_number": 10}
Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.946+
7f26810ae700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798946789,
"job": 6, "event": "table_file_deletion", "file_number": 4}
Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+
7f26818af700 -1 received  signal: Terminated from Kernel ( Could be
generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+
7f26818af700 -1 mon.dell02@-1(synchronizing) e161 *** Got Signal
Terminated ***
Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+
7f26818af700  1 mon.dell02@-1(synchronizing) e161 shutdown
Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.452+
7f2691a95880  4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling
all background work
Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.452+
7f2691a95880  4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete
Feb 06 11:09:59 dell02.mousetech.com bash[1357898]: ceph-278fcd86-0861-
11ee-a7df-9c5c8e86cf8f-mon-dell02
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Direct ceph mount on desktops

[Tim Holloway]

Back when I was battline Octopus, I had problems getting ganesha's NFS
to work reliably. I resolved this by doing a direct (ceph) mount on my
desktop machine instead of an NFS mount.

I've since been plagued by ceph "laggy OSD" complaints that appear to
be due to a non-responsive client and I'm suspecting that the client in
question is the desktop machine when it's suspended while the ceph
mount is in effect.

So the question is: Should ceph native mounts be used on general client
machines which may hibernate or otherwise go offline?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph as rootfs?

[Tim Holloway]

My €0.02 for what it's worth(less).

I've been doing RBD-based VMs under libvirt with no problem. In that
particular case, the ceph RBD base images are being overlaid cloud-
style with a an instance-specific qcow2 image and the RBD is just part
of my storage pools.

For a physical machine, I'd probably be content to use the Ceph NFS
share for the OS and PXE boot itself. I could always mount a ceph
filesystem once the OS was up if I wanted to.

If I'm not mistaken, getting ceph to boot directly would require a
custom ramdisk with ceph driver installed, whereas an NFS-supporting
ramdisk is pretty much off-the-shelf. Since ceph-ganesha NFS seems
pretty reliable and performant, I'd be happy with that.

   Tim

On Thu, 2024-01-04 at 17:16 +0100, Nico Schottelius wrote:
> 
> We also had a look at this a few years ago when we flashed almost all
> servers to boot iPXE from the nic directly. The idea was as follows:
> 
> iPXE -> http -> get kernel + initramfs -> mount rootfs
> 
> Our idea was to use RBD as a disk for the server, but to the last of
> my
> knowledge, there is no support in the linux kernel for something like
> 
>  linux root=ceph://mon1,mon2,mon3,key=keytouse/rbd/pool-x/image-y
> 
> Which would be needed for a ceph based rootfs.
> 
> It would be amazing to have that though, a decentralised storage
> system
> that is able to be booted from. Above description would actually be
> more
> suitable to the requested cephfs booting, as NFS root also often
> splits
> the kernel+initramfs.
> 
> The RBD case would potentially require boot loader support, if we
> assume
> that the RBD image contains something like grub - so a tiny bit more
> complicated.
> 
> BR and wishing you all the best for this approach,
> 
> Nico
> 
> 
> Alex Gorbachev  writes:
> 
> > A while ago I had done this with iSCSI, using iBFT, the advantage
> > being the
> > NIC itself supports such boot, the disadvantage is you need to use
> > iSCSI.
> > 
> > I recall PXE can bring an image over TFTP, so assuming you want to
> > load
> > that image into an RBD device for persistence?  Looks like this was
> > discussed here:
> > https://www.spinics.net/lists/ceph-users/msg66370.html
> 
> 
> --
> Sustainable and modern Infrastructures by ungleich.ch
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Adding a new monitor fails

[Tim Holloway]

ceph orch ls
NAME   PORTSRUNNING  REFRESHED  AGE
PLACEMENT 
alertmanager   ?:9093,9094  1/1  3m ago 8M
count:1   
crash   5/5  3m ago 8M
* 
grafana?:3000   1/1  3m ago 8M
count:1   
mds.ceefs   2/2  3m ago 4M
count:2   
mds.fs_name 3/3  3m ago 8M
count:3   
mgr 3/3  3m ago 4M
www6.mousetech.com;www2.mousetech.com;www7.mousetech.com  
mon 2/1  3m ago 4M
www6.mousetech.com;www2.mousetech.com;www7.mousetech.com;count:1  
nfs.foo?:2049   1/1  3m ago 4M
www7.mousetech.com
node-exporter  ?:9100   5/5  3m ago 8M
* 
osd   6  3m ago - 
   
osd.dashboard-admin-1686941775231 0  -  7M
* 
prometheus ?:9095   1/1  3m ago 8M
count:1   
rgw.mousetech  ?:80 2/2  3m ago 3M
www7.mousetech.com;www2.mousetech.com   


Note that the dell02 monitor doesn't show here although the "ceph orch
deamon add" returns success initially. And actually the www6 monitor is
not running nor does it list on the dashboard or "ceph orch ps". The
www6 machine is still somewhat messed up because it was the initial
launch machine for Octopus.

On Tue, 2024-02-06 at 17:22 +, Eugen Block wrote:
> So the orchestrator is working and you have a working ceph cluster?  
> Can you share the output of:
> ceph orch ls mon
> 
> If the orchestrator expects only one mon and you deploy another  
> manually via daemon add it can be removed. Try using a mon.yaml file 
> instead which contains the designated mon hosts and then run
> ceph orch apply -I mon.yaml
> 
> 
> 
> Zitat von Tim Holloway :
> 
> > I just jacked in a completely new, clean server and I've been
> > trying to
> > get a Ceph (Pacific) monitor running on it.
> > 
> > The "ceph orch daemon add" appears to install all/most of what's
> > necessary, but when the monitor starts, it shuts down immediately,
> > and
> > in the manner of Ceph containers immediately erases itself and the
> > container log, so it's not possible to see what its problem is.
> > 
> > I looked at manual installation, but the docs appear to be oriented
> > towards old-style non-container implementation and don't account
> > for
> > the newer /var/lib/ceph/*fsid*/ approach.
> > 
> > Any tips?
> > 
> > Last few lines in the system journal are like this:
> > 
> > Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-
> > 06T16:09:58.938+
> > 7f26810ae700  4 rocksdb: (Original Log Time 2024/02/06-
> > 16:09:58.938432)
> > [compaction/compaction_job.cc:760] [default] compacted to: base
> > level 6
> > level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0
> > 2]
> > max score 0.00, MB/sec: 351.7 rd, 351.7 wr, level 6, files in(4, 0)
> > out(2) MB in(92.8, 0.0) out(92.8), read-write-amplify(2.0) write-
> > amplify(1.0) OK, records in: 2858, records dropped: 0
> > output_compression: NoCompression
> > Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mon-dell02[1357545]:
> > Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df-
> > 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-
> > 06T16:09:58.938+
> > 7f26810ae700  4 rocksdb: (Original Log Time 2024/02/06-
> > 16:09:58.938452)
> > EVENT_LOG_v1 {"time_micros": 1707235798938446, "job": 6, "event":
> > "compaction_finished", "compaction_time_micros": 276718,
> > "compaction_time_cpu_micros": 73663, "output_level": 6,
> > "num_output_files&quo

[ceph-users] Re: Adding a new monitor fails

[Tim Holloway]

Ah, yes. Much better.

There are definitely some warts om there, as the monitor count was 1
but there were 2 monitors listed running.

I've mostly avoided docs that reference ceph config files and yaml
configs because the online docs are (as I've whined before) not always
trustworthy and often contain anachronisms. Were I sufficiently
knowledgeable, I'd offer to clean them up, but if that were the case, I
wouldn't have to come crying here.

All happy now, though.

   Tim


On Tue, 2024-02-06 at 19:22 +, Eugen Block wrote:
> Yeah, you have the „count:1“ in there, that’s why your manually
> added  
> daemons are rejected. Try my suggestion with a mon.yaml.
> 
> Zitat von Tim Holloway :
> 
> > ceph orch ls
> > NAME   PORTS    RUNNING  REFRESHED 
> > AGE
> > PLACEMENT
> > alertmanager   ?:9093,9094  1/1  3m ago
> > 8M
> > count:1
> > crash   5/5  3m ago
> > 8M
> > *
> > grafana    ?:3000   1/1  3m ago
> > 8M
> > count:1
> > mds.ceefs   2/2  3m ago
> > 4M
> > count:2
> > mds.fs_name 3/3  3m ago
> > 8M
> > count:3
> > mgr 3/3  3m ago
> > 4M
> > www6.mousetech.com;www2.mousetech.com;www7.mousetech.com
> > mon 2/1  3m ago
> > 4M
> > www6.mousetech.com;www2.mousetech.com;www7.mousetech.com;count:1
> > nfs.foo    ?:2049   1/1  3m ago
> > 4M
> > www7.mousetech.com
> > node-exporter  ?:9100   5/5  3m ago
> > 8M
> > *
> > osd   6  3m ago
> > -
> > 
> > osd.dashboard-admin-1686941775231 0  - 
> > 7M
> > *
> > prometheus ?:9095   1/1  3m ago
> > 8M
> > count:1
> > rgw.mousetech  ?:80 2/2  3m ago
> > 3M
> > www7.mousetech.com;www2.mousetech.com
> > 
> > 
> > Note that the dell02 monitor doesn't show here although the "ceph
> > orch
> > deamon add" returns success initially. And actually the www6
> > monitor is
> > not running nor does it list on the dashboard or "ceph orch ps".
> > The
> > www6 machine is still somewhat messed up because it was the initial
> > launch machine for Octopus.
> > 
> > On Tue, 2024-02-06 at 17:22 +, Eugen Block wrote:
> > > So the orchestrator is working and you have a working ceph
> > > cluster? 
> > > Can you share the output of:
> > > ceph orch ls mon
> > > 
> > > If the orchestrator expects only one mon and you deploy another 
> > > manually via daemon add it can be removed. Try using a mon.yaml
> > > file 
> > > instead which contains the designated mon hosts and then run
> > > ceph orch apply -I mon.yaml
> > > 
> > > 
> > > 
> > > Zitat von Tim Holloway :
> > > 
> > > > I just jacked in a completely new, clean server and I've been
> > > > trying to
> > > > get a Ceph (Pacific) monitor running on it.
> > > > 
> > > > The "ceph orch daemon add" appears to install all/most of
> > > > what's
> > > > necessary, but when the monitor starts, it shuts down
> > > > immediately,
> > > > and
> > > > in the manner of Ceph containers immediately erases itself and
> > > > the
> > > > container log, so it's not possible to see what its problem is.
> > > > 
> > > > I looked at manual installation, but the docs appear to be
> > > > oriented
> > > > towards old-style non-container implementation and don't
> > > > account
> > > > for
> > > > the newer /var/lib/ceph/*fsid*/ approach.
> > > > 
> > > > Any tips?
> > > > 
> > > > Last few lines in the system journal are like this:
> > > > 
> > > > Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-
> > > > a7df-
> > > > 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-
> > > > 06T16:09:58.938+
> > > > 7f26810ae700  4 rocksdb: (Original Log Time 2024/02/06-
> > > > 16:09:58.93843

[ceph-users] Re: Direct ceph mount on desktops

[Tim Holloway]

Unfortunately, it looks like the exact error text has already rolled
off my logs. Earlier today something jammed up and I restarted ceph,
which returned everything to a healthy state.

But I'll take that as an indicator that ceph isn't a good match for
sleeper systems. Fortunately, ganesha NFS is now working properly for
me since I upgraded Octopus to Pacific and I no longer feel the need
for a direct mount. NFS has had to deal with hibernation for years, so
I think I can depend on it to deal with it one way or the other.

   Tim
On Tue, 2024-02-06 at 13:00 -0500, Patrick Donnelly wrote:
> On Tue, Feb 6, 2024 at 12:09 PM Tim Holloway 
> wrote:
> > 
> > Back when I was battline Octopus, I had problems getting ganesha's
> > NFS
> > to work reliably. I resolved this by doing a direct (ceph) mount on
> > my
> > desktop machine instead of an NFS mount.
> > 
> > I've since been plagued by ceph "laggy OSD" complaints that appear
> > to
> > be due to a non-responsive client and I'm suspecting that the
> > client in
> > question is the desktop machine when it's suspended while the ceph
> > mount is in effect.
> 
> You should not see "laggy OSD" messages due to a client becoming
> unresponsive.
> 
> > So the question is: Should ceph native mounts be used on general
> > client
> > machines which may hibernate or otherwise go offline?
> 
> The mounts will eventually be evicted (generally) by the MDS if the
> machine hibernates/suspends. There are mechanisms for the mount to
> recover (see "recover_session" in the mount.ceph man page).  Any
> dirty
> data would be lost.
> 
> As for whether you should have clients that hibernate, it's not
> ideal.
> It could conceivably create problems if client machines hibernate
> longer than the blocklist duration (after eviction by the MDS).
> 
> -- 
> Patrick Donnelly, Ph.D.
> He / Him / His
> Red Hat Partner Engineer
> IBM, Inc.
> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problems adding a new host via orchestration.

[Tim Holloway]

Just FYI, I've seen this on CentOS systems as well, and I'm not even
sure that it was just for Ceph. Maybe some stuff like Ansible.

I THINK you can safely ignore that message or alternatively that it's
such an easy fix that senility has already driven it from my mind.

Tim

On Tue, 2024-02-06 at 14:44 -0500, Gary Molenkamp wrote:
> I confirmed selinux is disabled on all existing and new hosts.
> Likewise, 
> python3.6 is installed on all as well.  (3.9.16 on RL8, 3.9.18 on
> RL9).
> 
> I am running 16.2.12 on all containers, so it may be worth updating
> to 
> 16.2.14 to ensure I'm on the latest Pacific release.
> 
> Gary
> 
> 
> On 2024-02-05 08:17, Curt wrote:
> > 
> > 
> > You don't often get email from light...@gmail.com. Learn why this
> > is 
> > important 
> > 
> > 
> > I don't use rocky, so stab in the dark and probably not the issue,
> > but 
> > could selinux be blocking the process?  Really long shot, but
> > python3 
> > is in the standard location? So if you run python3 --version as
> > your 
> > ceph user it returns?
> > 
> > Probably not much help, but figured I'd throw it out there.
> > 
> > On Mon, 5 Feb 2024, 16:54 Gary Molenkamp,  wrote:
> > 
> >     I have verified the server's expected hostname (with
> > `hostname`)
> >     matches
> >     the hostname I am trying to use.
> >     Just to be sure, I also ran:
> >      cephadm check-host --expect-hostname 
> >     and it returns:
> >      Hostname "" matches what is expected.
> > 
> >     On the current admin server where I am trying to add the host,
> > the
> >     host
> >     is reachable, the shortname even matches proper IP with dns
> > search
> >     order.
> >     Likewise, on the server where the mgr is running, I am able to
> >     confirm
> >     reachability and DNS resolution for the new server as well.
> > 
> >     I thought this may be a DNS/name resolution issue as well, but
> > I
> >     don't
> >     see any errors in my setup wrt to host naming.
> > 
> >     Thanks
> >     Gary
> > 
> > 
> >     On 2024-02-03 06:46, Eugen Block wrote:
> >     > Hi,
> >     >
> >     > I found this blog post [1] which reports the same error
> > message. It
> >     > seems a bit misleading because it appears to be about DNS.
> > Can
> >     you check
> >     >
> >     > cephadm check-host --expect-hostname 
> >     >
> >     > Or is that what you already tried? It's not entirely clear
> > how you
> >     > checked the hostname.
> >     >
> >     > Regards,
> >     > Eugen
> >     >
> >     > [1]
> >     >
> >    
> > https://blog.mousetech.com/ceph-distributed-file-system-for-the-enterprise/ceph-bogus-error-cannot-allocate-memory/
> >     >
> >     > Zitat von Gary Molenkamp :
> >     >
> >     >> Happy Friday all.  I was hoping someone could point me in
> > the
> >     right
> >     >> direction or clarify any limitations that could be impacting
> > an
> >     issue
> >     >> I am having.
> >     >>
> >     >> I'm struggling to add a new set of hosts to my ceph cluster
> > using
> >     >> cephadm and orchestration.  When trying to add a host:
> >     >>     "ceph orch host add  172.31.102.41 --labels
> > _admin"
> >     >> returns:
> >     >>     "Error EINVAL: Can't communicate with remote host
> >     >> `172.31.102.41`, possibly because python3 is not installed
> > there:
> >     >> [Errno 12] Cannot allocate memory"
> >     >>
> >     >> I've verified that the ceph ssh key works to the remote
> > host,
> >     host's
> >     >> name matches that returned from `hostname`, python3 is
> >     installed, and
> >     >> "/usr/sbin/cephadm prepare-host" on the new hosts returns
> > "host is
> >     >> ok".    In addition, the cluster ssh key works between hosts
> >     and the
> >     >> existing hosts are able to ssh in using the ceph key.
> >     >>
> >     >> The existing ceph cluster is Pacific release using docker
> > based
> >     >> containerization on RockyLinux8 base OS.  The new hosts are
> >     >> RockyLinux9 based, with the cephadm being installed from
> > Quincy
> >     release:
> >     >>         ./cephadm add-repo --release quincy
> >     >>         ./cephadm install
> >     >> I did try installing cephadm from the Pacific release by
> >     changing the
> >     >> repo to el8,  but that did not work either.
> >     >>
> >     >> Is there a limitation is mixing RL8 and RL9 container hosts
> > under
> >     >> Pacific?  Does this same limitation exist under Quincy? Is
> > there a
> >     >> python version dependency?
> >     >> The reason for RL9 on the new hosts is to stage upgrading
> > the OS's
> >     >> for the cluster.  I did this under Octopus for moving from
> >     Centos7 to
> >     >> RL8.
> >     >>
> >     >> Thanks and I appreciate any feedback/pointers.
> >     >> Gary
> >     >>
> >     >>
> >     >> I've added the log trace here in case that helps (from `ceph
> >     log last
> >     >> cephadm`)
> >     >>
> >     >>
> >     >>
> 

[ceph-users] Re: Direct ceph mount on desktops

[Tim Holloway]

Followup.

Desktop system went to sleep overnight. I woke up to this:


HEALTH_WARN 1 client(s) laggy due to laggy OSDs; 1 clients failing to
respond to capability release; 1 MDSs report slow requests
[WRN] MDS_CLIENTS_LAGGY: 1 client(s) laggy due to laggy OSDs
mds.ceefs.www2.lzjqgd(mds.0): Client 7513643 is laggy; not evicted
because some OSD(s) is/are laggy
[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to
capability release
mds.ceefs.www2.lzjqgd(mds.0): Client a64.mousetech.com: failing to
respond to capability release client_id: 7513643
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
mds.ceefs.www2.lzjqgd(mds.0): 1 slow requests are blocked > 30 secs

a64 is the sleeping desktop.

I restarted the the ceph target and a whole bunch of stuck pg's popped
up, but the whole w2 machine is apparently scrambled and even a reboot
hasn't made its processes happy so I'm going to have to go in and fix
them one by one. May be unrelated to the original laggy OSD problem,
though. Just wanted to mention it for completeness.

Also, the desktop has weirdly managed to mount both native ceph and the
ceph NFS at the same mount point. Most likely because I'd added a
pre/post sleep script to unmount/remount ceph when the system went into
or out of suspension and I lost track of it so I haven't fixed it.

Again, this is mostly just for background not related to ceph's own
internals, although I had wondered if the ceph mount was happening
slowly and the suspension wasn't waiting properly for it to complete.

On Tue, 2024-02-06 at 13:00 -0500, Patrick Donnelly wrote:
> On Tue, Feb 6, 2024 at 12:09 PM Tim Holloway 
> wrote:
> > 
> > Back when I was battline Octopus, I had problems getting ganesha's
> > NFS
> > to work reliably. I resolved this by doing a direct (ceph) mount on
> > my
> > desktop machine instead of an NFS mount.
> > 
> > I've since been plagued by ceph "laggy OSD" complaints that appear
> > to
> > be due to a non-responsive client and I'm suspecting that the
> > client in
> > question is the desktop machine when it's suspended while the ceph
> > mount is in effect.
> 
> You should not see "laggy OSD" messages due to a client becoming
> unresponsive.
> 
> > So the question is: Should ceph native mounts be used on general
> > client
> > machines which may hibernate or otherwise go offline?
> 
> The mounts will eventually be evicted (generally) by the MDS if the
> machine hibernates/suspends. There are mechanisms for the mount to
> recover (see "recover_session" in the mount.ceph man page).  Any
> dirty
> data would be lost.
> 
> As for whether you should have clients that hibernate, it's not
> ideal.
> It could conceivably create problems if client machines hibernate
> longer than the blocklist duration (after eviction by the MDS).
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Adding a new monitor fails

[Tim Holloway]

Thanks,

I'll have to see if I come up with a suitable issue on documentation.
My biggest issue isn't a specific item (well, except for Octopus
telling me to use the not-included ceph-deploy command in lots of
places). It's more a case of needing attention paid to anachronisms in
general.

That and more attention could be paid to the distinction between
container-based and OS-native Ceph components.

So in short, not single issues, but more of a need for attention to the
overall details to assure that features described for a specific
release actually apply TO that release. Grunt work, but it can save a
lot on service calls.

I migrated to ceph from gluster because gluster is apparently going
unsupported at the end of this year. I moved to gluster from DR/BD
because I wanted triple redundancy on the data. While ceph is really
kind of overkill for my small R&D farm, it has proven to be about the
most solid network distributed filesystem I've worked with, No split
brains, no outright corruption, no data outages. Despite all the
atrocities I committed in setting it up, it has never failed at it
primary duty of delivering data service.

I started off with Octopus, and that has been the root of a lot of my
problems. Octopus introduced cephadm as a primary management tool, I
believe, but the documentation still referenced ceph-deploy. And
cephadm suffered from a bug that meant that if even one service was
down, scheduled work would not be done, so to repair anything I needed
an already-repaired system.

Migrating to Pacific cleared that up so a lot of what I'm doing now is
getting the lint out. I'm now staying consistently healthy between a
proper monitor configuration and having removed direct ceph mounts on
the desktops.

I very much appreciate all the help and insights you've provided. It's
nice to have laid my problems to rest.

   Tim

On Thu, 2024-02-08 at 14:41 +, Eugen Block wrote:
> Hi,
> 
> you're always welcome to report a documentation issue on  
> tracker.ceph.com, you don't need to clean them up by yourself. :-)
> There is a major restructuring in progress, but they will probably  
> never be perfect anyway.
> 
> > There are definitely some warts om there, as the monitor count was
> > 1
> > but there were 2 monitors listed running.
> 
> I don't know your mon history, but I assume that you've had more
> than  
> one mon (before converting to cephadm?). Then you might have updated 
> the mon specs via command line, containing "count:1". But the mgr  
> refuses to remove the second mon because it would break quorum.
> That's  
> why you had 2/1 running, this is reproducible in my test cluster.
> Adding more mons also failed because of the count:1 spec. You could  
> have just overwritten it in the cli as well without a yaml spec file 
> (omit the count spec):
> 
> ceph orch apply mon --placement="host1,host2,host3"
> 
> Regards,
> Eugen
> 
> Zitat von Tim Holloway :
> 
> > Ah, yes. Much better.
> > 
> > There are definitely some warts om there, as the monitor count was
> > 1
> > but there were 2 monitors listed running.
> > 
> > I've mostly avoided docs that reference ceph config files and yaml
> > configs because the online docs are (as I've whined before) not
> > always
> > trustworthy and often contain anachronisms. Were I sufficiently
> > knowledgeable, I'd offer to clean them up, but if that were the
> > case, I
> > wouldn't have to come crying here.
> > 
> > All happy now, though.
> > 
> >    Tim
> > 
> > 
> > On Tue, 2024-02-06 at 19:22 +, Eugen Block wrote:
> > > Yeah, you have the „count:1“ in there, that’s why your manually
> > > added 
> > > daemons are rejected. Try my suggestion with a mon.yaml.
> > > 
> > > Zitat von Tim Holloway :
> > > 
> > > > ceph orch ls
> > > > NAME   PORTS    RUNNING 
> > > > REFRESHED 
> > > > AGE
> > > > PLACEMENT
> > > > alertmanager   ?:9093,9094  1/1  3m
> > > > ago
> > > > 8M
> > > > count:1
> > > > crash   5/5  3m
> > > > ago
> > > > 8M
> > > > *
> > > > grafana    ?:3000   1/1  3m
> > > > ago
> > > > 8M
> > > > count:1
> > > > mds.ceefs   2/2  3m
> > > > ago
> > > > 4M
> > > > count:2
> > > > mds.fs_name

[ceph-users] Phhantom host

[Tim Holloway]

Ceph Pacific

Thanks to some misplaced thumbs-on-keyboard, I inadvertently managed to
alias a non-ceph system's ip as a ceph host and ceph adopted it
somehow.

I fixed the fat-fingered IP, and have gone through the usual motions to
delete a host, but some parts of the ceph ecosystem haven't caught up. 

The host no longer shows on "ceph orch host ls', but on the web control
panel, it's still there and thinks it has an OSD attached. Ditto for
the "ceph health detail". On the other hand, the webapp shows not one,
but THREE OSD's associated with the phantom host on the dashboard/hosts
detail expansion. It's claiming to own OSD daemons that are actually on
other machines.

Any assistance would be much appreciated!

Tim
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Phhantom host

[Tim Holloway]

Unfortunately, it's not just the dashboard. It shows up in "ceph health
detail" as strays:

[WRN] CEPHADM_STRAY_HOST: 1 stray host(s) with 1 daemon(s) not managed
by cephadm
stray host docker0 has 1 stray daemons: ['osd.7']


On Fri, 2024-06-21 at 11:46 -0400, Adam King wrote:
> I don't remember how connected the dashboard is to the orchestrator
> in
> pacific, but the only thing I could think to do here is just restart
> it.
> (ceph mgr module disable dashboard, ceph mgr module enable
> dashboard). You
> could also totally fail over the mgr (ceph mgr fail) although that
> might
> change the url you need for the dashboard by changing where the
> active mgr
> is.
> 
> On Fri, Jun 21, 2024 at 10:14 AM Tim Holloway 
> wrote:
> 
> > Ceph Pacific
> > 
> > Thanks to some misplaced thumbs-on-keyboard, I inadvertently
> > managed to
> > alias a non-ceph system's ip as a ceph host and ceph adopted it
> > somehow.
> > 
> > I fixed the fat-fingered IP, and have gone through the usual
> > motions to
> > delete a host, but some parts of the ceph ecosystem haven't caught
> > up.
> > 
> > The host no longer shows on "ceph orch host ls', but on the web
> > control
> > panel, it's still there and thinks it has an OSD attached. Ditto
> > for
> > the "ceph health detail". On the other hand, the webapp shows not
> > one,
> > but THREE OSD's associated with the phantom host on the
> > dashboard/hosts
> > detail expansion. It's claiming to own OSD daemons that are
> > actually on
> > other machines.
> > 
> > Any assistance would be much appreciated!
> > 
> > Tim
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> > 
> > 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Phantom hosts

[Tim Holloway]

It's getting worse.

As many may be aware, the venerable CentOS 7 OS is hitting end-of-life in a
matter of days.

The easiest way to upgrade my serves has been to simply create an alternate
disk with the new OS, turn my provisioning system loose on it, yank the old
OS system disk and jack in the new one.


However, Ceph is another matter. For that part, the simplest thing to do is
to destroy the Ceph node(s) on the affected box, do the OS upgrade, then
re-create the nodes.

But now I have even MORE strays. The OSD on my box lives on in Ceph in the
dashboard host view even though the documented removal procedures were
followed and the VM itself was destroyed.

Further, this last node is an RGW node and I cannot remove it from the RGW
configuration. It not only shows on the dashboard, it also lists as still
active on the command line and as entries in the config database no matter
what I do.


I really need some solution to this, as it's a major chokepoint in the
upgrade process


   Tim
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Phantom hosts

[Tim Holloway]

Hi Eugen,

It's gone now, although similar artefacts seems to linger.

The reason it's gone is that I've been upgrading all my machines to
AlamLinux 8 from CentOS 7 and AlmaLinux 7, as one is already EOL and
the other is within days of it. Rather than upgrade-in-place, I chose
to nuke/replace the entire system disks and provision from scratch. It
helped me clean up my network and get rid of years of cruft.

Ceph helped a lot there, since I'd do one machine at a time, and since
the provisioning data is on Ceph, it was always available even as
individual machines went up and down.

I lost the phantom host, although for a while one of the newer OSDs
gave me issues. The container would die while starting claiming that
the OSD block (badly-quoted) was "already in user". I believe this
happened right after I moved the _admin node to that machine.

I finally got the failed machine back online by manually stopping the
systemd service, waiting a while, then starting (not restarting) it.
But some other nodes may have been rebooted in the interim, so it's
hard to be certain what actually made it happy. Annoyingly, the
dashboard and OSD tree listed the failed node as "up" and "in" even
thoiug "ceph orch ps" showed it as "error". I couldn't persuade it to
go down and out, or I would have destroyed and re-created it.

I did clear up a major mess, though. My original install/admin machine
was littered with dead and mangled objects. Two long-deleted OSDs left
traces, and there was a mix of pre-cephadm components (including) OSDs
and newer stuff.

I did discover a node still running Octopus which I plan to upgrade
today, but overall things look pretty clean, excepting the ever-
frustrating "too many PGs per OSD". If autotuning was supposed to auto-
fix this, it's not doing so, even though autotuning is switched on.
Manual changes don't seem to take either.

Going back to the phantom host situation, one thing I have seen is that
on the dashboard, the hosts display lists OSDs that have been deleted
as belonging to that machine. "ceph osd tree" and the OSD view disagree
and show neither the phantom host nor the deleted OSDs.

Just to recap, the original phantom host was a non-ceph node that
accidentally got sucked in when I did a host add with the wrong IP
address. It then claimed to own another host's OSD.

  Thanks,
Tim

On Tue, 2024-07-09 at 06:08 +, Eugen Block wrote:
> Hi Tim,
> 
> is this still an issue? If it is, I recommend to add some more
> details  
> so it's easier to follow your train of thought.
> 
> ceph osd tree
> ceph -s
> ceph health detail
> ceph orch host ls
> 
> And then please point out which host you're trying to get rid of. I  
> would deal with the rgw thing later. Is it possible, that the
> phantom  
> host actually had OSDs on it? Maybe that needs to be cleaned up
> first.  
> I had something similar on a customer cluster recently where we
> hunted  
> failing OSDs but it turned out they were removed quite a while ago,  
> just not properly cleaned up yet on the filesystem.
> 
> Thanks,
> Eugen
> 
> Zitat von Tim Holloway :
> 
> > It's getting worse.
> > 
> > As many may be aware, the venerable CentOS 7 OS is hitting end-of-
> > life in a
> > matter of days.
> > 
> > The easiest way to upgrade my serves has been to simply create an
> > alternate
> > disk with the new OS, turn my provisioning system loose on it, yank
> > the old
> > OS system disk and jack in the new one.
> > 
> > 
> > However, Ceph is another matter. For that part, the simplest thing
> > to do is
> > to destroy the Ceph node(s) on the affected box, do the OS upgrade,
> > then
> > re-create the nodes.
> > 
> > But now I have even MORE strays. The OSD on my box lives on in Ceph
> > in the
> > dashboard host view even though the documented removal procedures
> > were
> > followed and the VM itself was destroyed.
> > 
> > Further, this last node is an RGW node and I cannot remove it from
> > the RGW
> > configuration. It not only shows on the dashboard, it also lists as
> > still
> > active on the command line and as entries in the config database no
> > matter
> > what I do.
> > 
> > 
> > I really need some solution to this, as it's a major chokepoint in
> > the
> > upgrade process
> > 
> > 
> >    Tim
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS MDS crashing during replay with standby MDSes crashing afterwards

[Tim Holloway]

Ivan,

This may be a little off-topic, but if you're still running AlmaLinux
8,9, it's worth noting that CentOS 8 actually end-of-lifed about 2
years ago, thanks to CentOS Stream.

Up until this last week, however, I had several AlmaLinux 8 machines
running myself, but apparently somewhere around May IBM Red Hat pulled
all of its CentOS8 enterprise sites offline, including Storage and
Ceph, which broke my yum updates.

While as far as I'm aware, once you've installed cephadm (whether via
yum/dnf or otherwise), there's no further need for the RPM repos,
losing yum support is not helping at the very least.

On the upside, it's possible to upgrade-in-place from AlmaLinux 8.9 to
AlmaLinux 9, although it may require temporarily disabling certain OS
services to appease the upgrade process.

Probably won't solve your problem, but at least you'll be able to move
fairly painlessly to a better-supported platform.

  Best Regards,
 Tim

On Tue, 2024-07-09 at 11:14 +0100, Ivan Clayson wrote:
> Hi Dhairya,
> 
> I would be more than happy to try and give as many details as
> possible 
> but the slack channel is private and requires my email to have an 
> account/ access to it.
> 
> Wouldn't taking the discussion about this error to a private channel 
> also stop other users who experience this error from learning about
> how 
> and why this happened as  well as possibly not be able to view the 
> solution? Would it not be possible to discuss this more publicly for
> the 
> benefit of the other users on the mailing list?
> 
> Kindest regards,
> 
> Ivan
> 
> On 09/07/2024 10:44, Dhairya Parmar wrote:
> > CAUTION: This email originated from outside of the LMB:
> > *.-dpar...@redhat.com-.*
> > Do not click links or open attachments unless you recognize the
> > sender 
> > and know the content is safe.
> > If you think this is a phishing email, please forward it to 
> > phish...@mrc-lmb.cam.ac.uk
> > 
> > 
> > --
> > 
> > Hey Ivan,
> > 
> > This is a relatively new MDS crash, so this would require some 
> > investigation but I was instructed to recommend disaster-recovery 
> > steps [0] (except session reset) to you to get the FS up again.
> > 
> > This crash is being discussed on upstream CephFS slack channel [1] 
> > with @Venky Shankar  and other CephFS 
> > devs. I'd encourage you to join the conversation, we can discuss
> > this 
> > in detail and maybe go through the incident step by step which
> > should 
> > help analyse the crash better.
> > 
> > [0] 
> > https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#disaster-recovery-experts
> > [1]
> > https://ceph-storage.slack.com/archives/C04LVQMHM9B/p1720443057919519
> > 
> > On Mon, Jul 8, 2024 at 7:37 PM Ivan Clayson
> >  
> > wrote:
> > 
> >     Hi Dhairya,
> > 
> >     Thank you ever so much for having another look at this so
> > quickly.
> >     I don't think I have any logs similar to the ones you
> > referenced
> >     this time as my MDSs don't seem to enter the replay stage when
> >     they crash (or at least don't now after I've thrown the logs
> > away)
> >     but those errors do crop up in the prior logs I shared when the
> >     system first crashed.
> > 
> >     Kindest regards,
> > 
> >     Ivan
> > 
> >     On 08/07/2024 14:08, Dhairya Parmar wrote:
> > >     CAUTION: This email originated from outside of the LMB:
> > >     *.-dpar...@redhat.com-.*
> > >     Do not click links or open attachments unless you recognize
> > > the
> > >     sender and know the content is safe.
> > >     If you think this is a phishing email, please forward it to
> > >     phish...@mrc-lmb.cam.ac.uk
> > > 
> > > 
> > >     --
> > > 
> > >     Ugh, something went horribly wrong. I've downloaded the MDS
> > > logs
> > >     that contain assertion failure and it looks relevant to this
> > > [0].
> > >     Do you have client logs for this?
> > > 
> > >     The other log that you shared is being downloaded right now,
> > > once
> > >     that's done and I'm done going through it, I'll update you.
> > > 
> > >     [0] https://tracker.ceph.com/issues/54546
> > > 
> > >     On Mon, Jul 8, 2024 at 4:49 PM Ivan Clayson
> > >      wrote:
> > > 
> > >     Hi Dhairya,
> > > 
> > >     Sorry to resurrect this thread again, but we still
> > >     unfortunately have an issue with our filesystem after we
> > >     attempted to write new backups to it.
> > > 
> > >     We finished the scrub of the filesystem on Friday and ran
> > > a
> > >     repair scrub on the 1 directory which had metadata
> > > damage.
> > >     After doing so and rebooting, the cluster reported no
> > > issues
> > >     and data was accessible again.
> > > 
> > >     We re-started the backups to run over the weekend and
> > >     unfortunately the filesystem crashed again where the log
> > > of
> > >     the failure is here:
> > >    
> > > https://www.mrc-lmb.cam.ac.uk/scicomp/data/uploads/ceph/ceph-mds.pebbles-s2.log-20240708

[ceph-users] Re: cephadm for Ubuntu 24.04

[Tim Holloway]

Just my €.02. There is, in fact a cephadm package for the Raspberry Pi
OS. If I read the synopis correctly, it's for ceph 16.2.11, which I
think is the same release of Ceph Pacific that I'm presently running my
own farm on. It appears to derive off Debian Bookworm.

Since cephadm is mainly a program to control other programs, I'd not be
surprised if it can at least partly manage newer ceph systems. Then
again, I don't know what it uses for interfaces beyond its ability to
pull down and setup ceph containers.

For my newer systems, cephadm is the only package I install and I use
cephadm shell. For general management, I do like to install the ceph-
common to get the "ceph" command down to the bash shell level without
invoking cephadm. Everything else, I haven't found a use for.

I do like managed packages because it's useful to have them show up in
a software inventory. However, once I have the cephadm package
installed, I prefer to let it manage the rest of the ceph
infrastructure.

  Best Regards,
 Tim
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Schödinger's OSD

[Tim Holloway]

Incidentally, I just noticed that my phantom host isn't completely
gone. It's not in the host list, either command-line or dashboard, but
it does list (with no assets) as a host under "ceph osd tree".

---

More seriously, I've been having problems with OSDs that report as
being both up and down at the same time.

This is on 2 new hosts. One host saw this when I made it the _admin
host. The other caught it because it's running in a VM with the OSD
mapped out as an imported disk and the host os managed to flip which
drive was sda and which was sbd, resulting in having to delete and re-
define the OSD in the VM.

But now the OSD on this VM reports as "UP/IN" on the dashboard while
it's "error" on "ceph orch ps" and on the actual vbox, the osd
container fails on startup. viz:

ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee-
a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+
7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block)
open open got: (16) De>
Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee-
a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+
7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object store
Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee-
a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+
7fc17dfb9380 -1  ** ERROR: osd init failed: (16) Device or resource
busy

Note that truncated message above reads

bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16)
Device or resource busy

Rebooting doesn't help. Nor does freeing us resources and
stopping/starting processes manually. The problem eventually cleared up
spontaneously on the admin box, but I have no idea why.

---

Also noted that now the OSD on the admin box shows in ceph orch ps as
"stopped", though again, the dashboard lists it as "UP/IN". 

Here's what systemctl thinks about it:

systemctl status ceph-osd@5.service
● ceph-osd@5.service - Ceph object storage daemon osd.5
 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service;
enabled-runtime; preset: disabled)
 Active: active (running) since Fri 2024-07-12 16:45:51 EDT; 1min
40s ago
Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh -
-cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS)
   Main PID: 8517 (ceph-osd)
  Tasks: 70
 Memory: 478.6M
CPU: 3.405s
 CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@5.service
 └─8517 /usr/bin/ceph-osd -f --cluster ceph --id 5 --
setuser ceph --setgroup ceph

Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting Ceph object
storage daemon osd.5...
Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started Ceph object
storage daemon osd.5.
Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024-07-
12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public interface
Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024-07-
12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 log_to_monitors
{default=true}
Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024-07-
12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 set_numa_affinity
unable to identify public interface '' numa node: (2) No such file or
directory

The actual container is not running.

Ceph version, incidentally, is 16.2.15. Except for that one node that
apparently didn't move up from Octupus (I'll be nuking that one
shortly).
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Schödinger's OSD

[Tim Holloway]

This particular system has it both ways and neither wants to work.

The peculiar thing was that when I first re-created the OSD with
cephadm, it was reported that this was an "unmanaged node". So I ran
the same cephadm agin and THAT time it showed up. So I suspect that the
ceph-osd@4.service was the first install and the
ceph-~@osd.4.servuce got added on the second try.

ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504

Here's the OSD tree:

ID   CLASS  WEIGHT   TYPE NAME STATUS  REWEIGHT  PRI-AFF
 -1 7.0  root default   
-25 1.0  host ceph01
  1hdd  1.0  osd.1 up   1.0  1.0
-28 1.0  host ceph03
  3hdd  1.0  osd.3 up   1.0  1.0
-11 1.0  host ceph04
  7hdd  1.0  osd.7 up   1.0  1.0
-31 1.0  host ceph05
  4hdd  1.0  osd.4 up   1.0  1.0
 -9 1.0  host ceph06
  2hdd  1.0  osd.2 up   1.0  1.0
-22 1.0  host dell02
  5ssd  1.0  osd.5 up   1.0  1.0
-13   0  host docker0   
 -5   0  host www2  
 -3   0  host www6  
 -7 1.0  host www7  
  0hdd  1.0  osd.0 up   1.0  1.0

Note that www2 was recommissioned and replaced by the ceph05 machine
and no longer physically exists. www6 was the original/admin ceph node
with all the accumulated glop. It ran on the base OS (not in a VM) and
I have not attempted to re-create that. Ceph remembers it from its own
internal memory, as the original www6 system drive was replaced with an
SSD and the OS was installed from scratch. The docker0 machine was
never a ceph node. It's the phantom that got yanked it because I
accidentally supplied its IP address when doing a ceph orch host add
for a completely different machine.

And here's the host list as the orchestrator sees it:

ceph orch host ls
HOST   ADDR   LABELS  STATUS  
ceph01.internal.mousetech.com  10.0.1.21  
ceph03.internal.mousetech.com  10.0.1.53  
ceph04.mousetech.com   10.0.1.14  
ceph05.internal.mousetech.com  10.0.1.54  
ceph06.internal.mousetech.com  10.0.1.56  
dell02.mousetech.com   10.0.1.52  _admin rgw  
www7.mousetech.com 10.0.1.7   rgw 
7 hosts in cluster

 
On Fri, 2024-07-12 at 22:15 +, Eugen Block wrote:
> Hi,
> 
> containerized daemons usually have the fsid in the systemd unit,
> like  
> ceph-{fsid}@osd.5
> 
> Is it possible that you have those confused? Check the  
> /var/lib/ceph/osd/ directory to find possible orphaned daemons and  
> clean them up.
> And as previously stated, it would help to see your osd tree and
> which  
> OSDs you’re talking about.
> 
> Zitat von Tim Holloway :
> 
> > Incidentally, I just noticed that my phantom host isn't completely
> > gone. It's not in the host list, either command-line or dashboard,
> > but
> > it does list (with no assets) as a host under "ceph osd tree".
> > 
> > ---
> > 
> > More seriously, I've been having problems with OSDs that report as
> > being both up and down at the same time.
> > 
> > This is on 2 new hosts. One host saw this when I made it the _admin
> > host. The other caught it because it's running in a VM with the OSD
> > mapped out as an imported disk and the host os managed to flip
> > which
> > drive was sda and which was sbd, resulting in having to delete and
> > re-
> > define the OSD in the VM.
> > 
> > But now the OSD on this VM reports as "UP/IN" on the dashboard
> > while
> > it's "error" on "ceph orch ps" and on the actual vbox, the osd
> > container fails on startup. viz:
> > 
> > ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-
> > 11ee-
> > a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+
> > 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block)
> > open open got: (16) De>
> > Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-
> > 11ee-
> > a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+
> > 7fc17dfb9380 -1 osd.4 0 OS

[ceph-users] Re: Schödinger's OSD

[Tim Holloway]

OK. Phantom hosts are gone. Many thanks! I'll have to review my 
checklist for decomissioning hosts to make sure that step is on it.


On the legacy/container OSD stuff, that is a complete puzzle.

While the first thing that I see when I look up "creating an OSD" in the 
system documentation is the manual process, I've been using cephadm long 
enough to know to dig past that. The manual process is sufficiently 
tedious that I cannot think that I'd have used it by accident. 
Especially since I set out with the explicit goal of using cephadm. Yet 
here it is. This isn't an upgraded machine, it was constructed within 
the last week from the ground up. So I have no idea how the legacy 
definition got there. On two separate systems.


The disable on the legacy OSD worked and the container is now running. 
Although I'm not sure that it will survive a reboot, since the legacy 
service is dynamically created on each reboot.


This is what happens when I try to adopt:

 cephadm adopt --style legacy --name osd.4
Pulling container image quay.io/ceph/ceph:v16...
Found online OSD at //var/lib/ceph/osd/ceph-4/fsid
objectstore_type is bluestore
Disabling old systemd unit ceph-osd@4...
Moving data...
Traceback (most recent call last):
  File "/usr/sbin/cephadm", line 9509, in 
    main()
  File "/usr/sbin/cephadm", line 9497, in main
    r = ctx.func(ctx)
  File "/usr/sbin/cephadm", line 2061, in _default_image
    return func(ctx)
  File "/usr/sbin/cephadm", line 6043, in command_adopt
    command_adopt_ceph(ctx, daemon_type, daemon_id, fsid)
  File "/usr/sbin/cephadm", line 6210, in command_adopt_ceph
    move_files(ctx, glob(os.path.join(data_dir_src, '*')),
  File "/usr/sbin/cephadm", line 2215, in move_files
    os.symlink(src_rl, dst_file)
FileExistsError: [Errno 17] File exists: '/dev/vg_ceph/ceph0504' -> 
'/var/lib/ceph/278fcd86-0861-11ee-a7df-9c5c8e86cf8f/osd.4/block'


I have considered simply doing a brute-force removal of the OSD files in 
/var/lib/ceph/osd but I'm not sure what ill effects might ensue. I 
discovered that my other offending machine actually has TWO legacy OSD 
directories, but only one of them is being used. The other OSD is the 
remnant of a deletion and it's just dead files now.




On 7/13/24 02:39, Eugen Block wrote:
Okay, it looks like you just need some further cleanup regarding your 
phantom hosts, for example:


ceph osd crush remove www2
ceph osd crush remove docker0

and so on.

Regarding the systemd unit (well, cephadm also generates one, but with 
the fsid as already mentioned), you could just stop and disable the 
old one:


systemctl disable --now ceph-osd@4

and see if the container takes over.

Was this your attempt to adopt an existing OSD from pre-cephadm?


ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504


The recommended way would have been to adopt the device:

cephadm [--image your-custom-image] adopt --style legacy --name osd.4

locally on that host. The --image parameter is optional. Did you 
follow the docs [1] when you moved to cephadm? Anyway, since it 
somehow seems to work already, it's probably not that relevant 
anymore, I just wanted to point to it anyway.


[1] https://docs.ceph.com/en/latest/cephadm/adoption/

Zitat von Tim Holloway :


This particular system has it both ways and neither wants to work.

The peculiar thing was that when I first re-created the OSD with
cephadm, it was reported that this was an "unmanaged node". So I ran
the same cephadm agin and THAT time it showed up. So I suspect that the
ceph-osd@4.service was the first install and the
ceph-~@osd.4.servuce got added on the second try.

ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504

Here's the OSD tree:

ID   CLASS  WEIGHT   TYPE NAME STATUS  REWEIGHT  PRI-AFF
 -1 7.0  root default
-25 1.0  host ceph01
  1    hdd  1.0  osd.1 up   1.0  1.0
-28 1.0  host ceph03
  3    hdd  1.0  osd.3 up   1.0  1.0
-11 1.0  host ceph04
  7    hdd  1.0  osd.7 up   1.0  1.0
-31 1.0  host ceph05
  4    hdd  1.0  osd.4 up   1.0  1.0
 -9 1.0  host ceph06
  2    hdd  1.0  osd.2 up   1.0  1.0
-22 1.0  host dell02
  5    ssd  1.0  osd.5 up   1.0  1.0
-13   0  host docker0
 -5   0  host www2
 -3   0  host www6
 -7 1.0  host www7
  0    hdd  1.0  osd.0 up   1.0  1.0

Note that www2 was recommissioned and replaced by the ceph05 machine
and no longer physically exists. www6 was the original/admin ceph node
with all the accumulated glop. It ran 

[ceph-users] Re: Schödinger's OSD

[Tim Holloway]

The problem with merely disabling or masking the non-cephadm OSD is
that the offending systemd service unit lives under /run, not under
/lib/systemd or /etc/systemd.

As far as I know, essentially the entire /run directory's contents get
destroyed when you reboot and that would include the disabled OSD unit.
Then a new copy would get created as the system boot proceeded. I
could, of course, then re-disable it, but that's not a very pretty
solution. Better to determine why Ceph feels the need to create this
Systemd service dynamically and persuade it not to.

I was kind of hoping that it came from finding that OSD directory that
under /var/lib/ceph/osd, but as I said, I have another machine with TWO
such directories and only one manifests as a systemd service. The other
doesn't run at all, doesn't list on an osd tree, orch ps or dashboard,
and since as far as I'm concerned doesn't exist anyway, I'll not
complain about that. I just need to get the invalid stuff excised
safely.

Oh wait, one difference between the two /var/lib/cep/osd's is that the
one that's running has files, the one that isn't is just an empty
directory. Which suggests that the cue for making the /run/ceph/osd
service may be the detection of one of the files there and maybe I
could risk ripping the unwanted directory out. I thinks there are some
softlinks, though, so I'll proceed with caution.

On the plus side, the autotuner seems to have finally kicked in. First
time I've seen "HEALTH OK" in a while!

Alwin: Thanks for your interest. Both the funny machines are dedicated
ceph host nodes, so it's no accident that cephadm is installed on the.
And I've never had more than 1 fsid, so no issue there.

If you're thinking about the "phantom host", that was just because of a
typing error when adding a new ceph host. That problem has now been
resolved.

  Tim

On Mon, 2024-07-15 at 05:49 +, Eugen Block wrote:
> If the OSD is already running in a container, adopting it won't
> work,  
> as you already noticed. I don't have an explanation how the  
> non-cephadm systemd unit has been created, but that should be fixed
> by  
> disabling it.
> 
> > I have considered simply doing a brute-force removal of the OSD  
> > files in /var/lib/ceph/osd but I'm not sure what ill effects might 
> > ensue. I discovered that my other offending machine actually has
> > TWO  
> > legacy OSD directories, but only one of them is being used. The  
> > other OSD is the remnant of a deletion and it's just dead files
> > now.
> 
> Check which OSDs are active and remove the remainders of the
> orphaned  
> directories, that should be fine. But be careful and check properly  
> before actually remocing anything and only remove one by one while  
> watching the cluster status.
> 
> Zitat von Tim Holloway :
> 
> > OK. Phantom hosts are gone. Many thanks! I'll have to review my  
> > checklist for decomissioning hosts to make sure that step is on it.
> > 
> > On the legacy/container OSD stuff, that is a complete puzzle.
> > 
> > While the first thing that I see when I look up "creating an OSD"
> > in  
> > the system documentation is the manual process, I've been using  
> > cephadm long enough to know to dig past that. The manual process
> > is  
> > sufficiently tedious that I cannot think that I'd have used it by  
> > accident. Especially since I set out with the explicit goal of
> > using  
> > cephadm. Yet here it is. This isn't an upgraded machine, it was  
> > constructed within the last week from the ground up. So I have no  
> > idea how the legacy definition got there. On two separate systems.
> > 
> > The disable on the legacy OSD worked and the container is now  
> > running. Although I'm not sure that it will survive a reboot,
> > since  
> > the legacy service is dynamically created on each reboot.
> > 
> > This is what happens when I try to adopt:
> > 
> >  cephadm adopt --style legacy --name osd.4
> > Pulling container image quay.io/ceph/ceph:v16...
> > Found online OSD at //var/lib/ceph/osd/ceph-4/fsid
> > objectstore_type is bluestore
> > Disabling old systemd unit ceph-osd@4...
> > Moving data...
> > Traceback (most recent call last):
> >   File "/usr/sbin/cephadm", line 9509, in 
> >     main()
> >   File "/usr/sbin/cephadm", line 9497, in main
> >     r = ctx.func(ctx)
> >   File "/usr/sbin/cephadm", line 2061, in _default_image
> >     return func(ctx)
> >   File "/usr/sbin/cephadm", line 6043, in command_adopt
> >     command_adopt_ceph(ctx,

[ceph-users] Re: Schödinger's OSD

[Tim Holloway]

Interesting. And thanks for the info.

I did a quick look-around. The admin node, which is one of the mixed-
osd machines has these packages installed:

centos-release-ceph-pacific-1.0-2.el9.noarch
cephadm-16.2.14-2.el9s.noarch
libcephfs2-16.2.15-1.el9s.x86_64
python3-ceph-common-16.2.15-1.el9s.x86_64
python3-ceph-argparse-16.2.15-1.el9s.x86_64
python3-cephfs-16.2.15-1.el9s.x86_64
libcephsqlite-16.2.15-1.el9s.x86_64
ceph-common-16.2.15-1.el9s.x86_64
ceph-base-16.2.15-1.el9s.x86_64
ceph-selinux-16.2.15-1.el9s.x86_64
ceph-mds-16.2.15-1.el9s.x86_64
ceph-mon-16.2.15-1.el9s.x86_64
ceph-osd-16.2.15-1.el9s.x86_64
ceph-prometheus-alerts-16.2.15-1.el9s.noarch
ceph-grafana-dashboards-16.2.15-1.el9s.noarch
ceph-mgr-dashboard-16.2.15-1.el9s.noarch
ceph-mgr-diskprediction-local-16.2.15-1.el9s.noarch
ceph-mgr-k8sevents-16.2.15-1.el9s.noarch
ceph-mgr-modules-core-16.2.15-1.el9s.noarch
ceph-mgr-16.2.15-1.el9s.x86_64
ceph-mgr-rook-16.2.15-1.el9s.noarch
ceph-16.2.15-1.el9s.x86_64

The other problem node has these:

centos-release-ceph-pacific-1.0-2.el9.noarch
python3-ceph-common-16.2.15-1.el9s.x86_64
libcephfs2-16.2.15-1.el9s.x86_64
python3-ceph-argparse-16.2.15-1.el9s.x86_64
python3-cephfs-16.2.15-1.el9s.x86_64
libcephsqlite-16.2.15-1.el9s.x86_64
ceph-mgr-modules-core-16.2.15-1.el9s.noarch
ceph-common-16.2.15-1.el9s.x86_64
ceph-base-16.2.15-1.el9s.x86_64
ceph-selinux-16.2.15-1.el9s.x86_64
ceph-mgr-diskprediction-local-16.2.15-1.el9s.noarch
ceph-mgr-16.2.15-1.el9s.x86_64
ceph-mds-16.2.15-1.el9s.x86_64
ceph-mon-16.2.15-1.el9s.x86_64
ceph-osd-16.2.15-1.el9s.x86_64
ceph-16.2.15-1.el9s.x86_64
cephadm-16.2.14-2.el9s.noarch

But a simple node hosting only and OSD and NFS did look like this:

centos-release-ceph-pacific-1.0-2.el9.noarch
cephadm-16.2.14-2.el9s.noarch

I did an in-place migration from AlmaLinux 8 to AlamaLinux 9 and that
may have had side-effects. But who knows?

I'll start ripping packages and see what happens.

On Tue, 2024-07-16 at 06:38 +, Eugen Block wrote:
> Do you have more ceph packages installed than just cephadm? If you  
> have ceph-osd packages (or ceph-mon, ceph-mds etc.), I would remove  
> them and clean up the directories properly. To me it looks like a  
> mixup of "traditional" package based installation and cephadm  
> deployment. Only you can tell how and why, but it's more important
> to  
> clean that up and keep it consistent. You should keep the cephadm
> and  
> optional the ceph-common package, but the rest isn't required to run
> a  
> cephadm cluster.
> 
> 
> Zitat von Tim Holloway :
> 
> > The problem with merely disabling or masking the non-cephadm OSD is
> > that the offending systemd service unit lives under /run, not under
> > /lib/systemd or /etc/systemd.
> > 
> > As far as I know, essentially the entire /run directory's contents
> > get
> > destroyed when you reboot and that would include the disabled OSD
> > unit.
> > Then a new copy would get created as the system boot proceeded. I
> > could, of course, then re-disable it, but that's not a very pretty
> > solution. Better to determine why Ceph feels the need to create
> > this
> > Systemd service dynamically and persuade it not to.
> > 
> > I was kind of hoping that it came from finding that OSD directory
> > that
> > under /var/lib/ceph/osd, but as I said, I have another machine with
> > TWO
> > such directories and only one manifests as a systemd service. The
> > other
> > doesn't run at all, doesn't list on an osd tree, orch ps or
> > dashboard,
> > and since as far as I'm concerned doesn't exist anyway, I'll not
> > complain about that. I just need to get the invalid stuff excised
> > safely.
> > 
> > Oh wait, one difference between the two /var/lib/cep/osd's is that
> > the
> > one that's running has files, the one that isn't is just an empty
> > directory. Which suggests that the cue for making the /run/ceph/osd
> > service may be the detection of one of the files there and maybe I
> > could risk ripping the unwanted directory out. I thinks there are
> > some
> > softlinks, though, so I'll proceed with caution.
> > 
> > On the plus side, the autotuner seems to have finally kicked in.
> > First
> > time I've seen "HEALTH OK" in a while!
> > 
> > Alwin: Thanks for your interest. Both the funny machines are
> > dedicated
> > ceph host nodes, so it's no accident that cephadm is installed on
> > the.
> > And I've never had more than 1 fsid, so no issue there.
> > 
> > If you're thinking about the "phantom host", that was just because
> > of a
> > typing error when adding a

[ceph-users] Re: Schödinger's OSD

[Tim Holloway]

OK. I deleted the questionable stuff with this command:

dnf erase ceph-mgr-modules-core-16.2.15-1.el9s.noarch  ceph-mgr-
diskprediction-local-16.2.15-1.el9s.noarch  ceph-mgr-16.2.15-
1.el9s.x86_64 ceph-mds-16.2.15-1.el9s.x86_64 ceph-mon-16.2.15-
1.el9s.x86_64

That left these two:

centos-release-ceph-pacific-1.0-2.el9.noarch
cephadm-16.2.14-2.el9s.noarch

At that point, I couldn't run cephadm because the /etc/ceph directory
was deleted as well by dnf. I copied it back in from the master copy
but no containers were running. A reboot brought everything back up.

So I'm optimistic that that machine is now clean (and has nearly 500MB
of newly-freed space from the deleted packages!)

One of the deleted items was the "ceph" meta-package. I'm thinking that
I installed this and it's what pulled in the others.

The final machine is operational and I'm going to leave it, but it does
show 1 quirk. Dashboard and osd tree show its OSD as up/running, but
"ceph orch ps" shows it as "stopped". My guess is that ceph orch is
looking for the container OSD and doesn't notice the legacy OSD.

Thanks again for all the help!

   Tim

On Tue, 2024-07-16 at 06:38 +, Eugen Block wrote:
> Do you have more ceph packages installed than just cephadm? If you  
> have ceph-osd packages (or ceph-mon, ceph-mds etc.), I would remove  
> them and clean up the directories properly. To me it looks like a  
> mixup of "traditional" package based installation and cephadm  
> deployment. Only you can tell how and why, but it's more important
> to  
> clean that up and keep it consistent. You should keep the cephadm
> and  
> optional the ceph-common package, but the rest isn't required to run
> a  
> cephadm cluster.
> 
(history deleted. See earlier copies)
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Cephadm has a small wart

[Tim Holloway]

I've been setting up a cookbook OSD creation process and as I walked
through the various stages, I noted that the /etc/redhat-release file
said "CentOS Stream 8". I panicked, because IBM has pulled the Ceph
archives for CentOS 8 and nuked the machine, then rebuilt it with more
attention to detail.

I consistently was seeing the correct redhat-release until I installed
cephadm and ran cephadm shell. Cephadm is running a chroot jail, it
seems, and it's plugging in the wrong value for that file. Presumably
from its internal image instead of copying the actual file. The false
release string does not leak outside of cephadm, though.

So far this appears to be only important as a possible confusion point,
but it is of concern since the repositories in /etc/yum.repos.d
download from repositories whose URLs are derived from the OS release
that YUM (dnf) thinks it's running under and I'd hate to find out the
hard way that something got confused.

Probably rpm and yum/dnf shouldn't be available from within cephadm,
but since cephadm pulls stuff internally that may not be possible.

This was seen on AlmaLinux 9.4, cephadm version 16.2.15.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm has a small wart

[Tim Holloway]

Ah. Makes sense. Might be nice if the container build appended
something like "cephadm container" to the redhat-release string,
though.

A more concerning item is that the container is based on CentOS 8
Stream. I'd feel more comfortable if the base OS was more release-
oriented. Especially since Centos8 Stream EOL'ed back around May.

More importantly, the Ceph archives for CentOS 8 went down the Memory
Hole. That's really worrisome. Makes it questionable that it would be
easy to build from source on a blank system now.

Yanking repos is not the Linux Way. I could probably spin up a copy of
CentOS 2, because repos are still online even if no sensible person
would want it for a production server. But not so the CentOS 8
enterprise extensions. They;re gone.


On Fri, 2024-07-19 at 09:00 +0200, Robert Sander wrote:
> On 7/18/24 21:50, Tim Holloway wrote:
> > I've been setting up a cookbook OSD creation process and as I
> > walked
> > through the various stages, I noted that the /etc/redhat-release
> > file
> > said "CentOS Stream 8".
> 
> This is the case because the orachestrator cephadm uses container
> images 
> based on CentOS 8.
> 
> When you execute "cephadm shell" it starts a container with that
> image 
> for you.
> 
> Regards
> -- 
> Robert Sander
> Heinlein Consulting GmbH
> Schwedter Str. 8/9b, 10119 Berlin
> 
> https://www.heinlein-support.de
> 
> Tel: 030 / 405051-43
> Fax: 030 / 405051-19
> 
> Amtsgericht Berlin-Charlottenburg - HRB 220009 B
> Geschäftsführer: Peer Heinlein - Sitz: Berlin
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm has a small wart

[Tim Holloway]

Well, for what it's worth, I'm fairly agnostic about the distro used,
more interested in its stability.

Alpine is Debian-based, I think and very popular as a minimalistic
based for containers. I prefer the Red Hat world myself, but can live
with either. Nominally Ceph is a Red Hat product, and that's about the
only other factor I can think of.


On Fri, 2024-07-19 at 14:50 +0200, Stefan Kooman wrote:
> On 19-07-2024 14:04, Tim Holloway wrote:
> > Ah. Makes sense. Might be nice if the container build appended
> > something like "cephadm container" to the redhat-release string,
> > though.
> > 
> > A more concerning item is that the container is based on CentOS 8
> > Stream. I'd feel more comfortable if the base OS was more release-
> > oriented. Especially since Centos8 Stream EOL'ed back around May.
> 
> Yeah. The ceph devs are aware of that. The new release for Reef
> 18.2.4 
> and Squid are based on CentOS stream 9. CentOS 8 stream has been (or
> is 
> being) phased out.
> 
> > 
> > More importantly, the Ceph archives for CentOS 8 went down the
> > Memory
> > Hole. That's really worrisome. Makes it questionable that it would
> > be
> > easy to build from source on a blank system now.
> > 
> > Yanking repos is not the Linux Way. I could probably spin up a copy
> > of
> > CentOS 2, because repos are still online even if no sensible person
> > would want it for a production server. But not so the CentOS 8
> > enterprise extensions. They;re gone.
> 
> There have been some discussons on ceph-dev ML if CentOS stream a
> good 
> build target for Ceph containers. Some people like to see debian 
> instead. Not sure if this will be point of discussion in the future 
> (again). But we all agree it should be based on a OS that is still 
> supported (and upgraded before it gets EOL).
> 
> Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] PSA: Bringing up an OSD really, really fast

[Tim Holloway]

While running Ceph in a VM may not be considered the most efficient
mode of operation, it's still fairly popular. I've put together a small
project that can spin up an OSD VM in just a few minutes.

Whereas Ceph given free rein will gladly acquire resources with all the
enthusiasm as a sprig of mint in damp earth, what I've developed is
more useful for a small farm with very irregular resources such as my
own. And for research purposes!

The project has 2 components: one of which gets deployed on the
intended VM host, the other being an Ansible role and model playbook.
Full instructions are provided. And incidentally, the Ansible role
should be able to run against a base OS host, even though it was
developed for VMs.

Comments and suggestions welcome.

https://gogs.mousetech.com/mtsinc7/instant_osd
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Stuck in remapped state?

[Tim Holloway]

I was in the  middle of tuning my OSDs when lightning blew me off the
Internet. Had to wait 5 days for my ISP to send a tech and replace a
fried cable. In the mean time, among other things. I had some serious
time drift between servers thanks to the OS upgrades replacing NTP with
chrony and me not having thought to re-establish a master in-house
timeserver.

Ceph tried really hard to keep up with all that, but eventually it was
just too much. Now I've got an offline filesystem and apparently it's
stuck trying to get back online again.

The forensics:
[ceph: root@www7 /]# ceph -s
  cluster:
id: 278fcd86-0861-11ee-a7df-9c5c8e86cf8f
health: HEALTH_WARN
failed to probe daemons or devices
1 filesystem is degraded
1/3 mons down, quorum www7,ceph03
 
  services:
mon: 3 daemons, quorum www7,ceph03 (age 2m), out of quorum: dell02
mgr: ceph08.tlocfi(active, since 58m), standbys: dell02.odtbqw,
www7.rxagfn
mds: 1/1 daemons up, 1 standby
osd: 7 osds: 7 up (since 12h), 7 in (since 18h); 308 remapped pgs
rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
volumes: 0/1 healthy, 1 recovering
pools:   22 pools, 681 pgs
objects: 125.10k objects, 36 GiB
usage:   91 GiB used, 759 GiB / 850 GiB avail
pgs: 47772/369076 objects misplaced (12.944%)
 373 active+clean
 308 active+clean+remapped
 
  io:
client:   170 B/s rd, 0 op/s rd, 0 op/s wr

[ceph: root@www7 /]# ceph health detail
HEALTH_WARN 1 filesystem is degraded; 25 client(s) laggy due to laggy
OSDs
[WRN] FS_DEGRADED: 1 filesystem is degraded
fs ceefs is degraded
[WRN] MDS_CLIENTS_LAGGY: 25 client(s) laggy due to laggy OSDs
mds.ceefs.www7.drnuyi(mds.0): Client 14019719 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14124385 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14144243 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14144375 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14224103 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14224523 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14234194 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14234545 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14236841 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14237837 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14238536 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14244124 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14264236 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14266870 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14294170 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14294434 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14296012 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14304212 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14316057 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14318379 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14325518 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14328956 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14334283 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14336104 is laggy; not evicted
because some OSD(s) is/are laggy
mds.ceefs.www7.drnuyi(mds.0): Client 14374237 is laggy; not evicted
because some OSD(s) is/are laggy

[ceph: root@www7 /]# ceph osd tree
ID   CLASS  WEIGHT   TYPE NAMESTATUS  REWEIGHT  PRI-AFF
 -1 2.79994  root default  
-25 0.15999  host ceph01   
  1hdd  0.15999  osd.1up   0.15999  1.0
-28 1.15999  host ceph03   
  3hdd  0.15999  osd.3up   0.15999  1.0
  5hdd  1.0  osd.5up   1.0  1.0
 -9 0.15999  host ceph06   
  2hdd  0.15999  osd.2up   0.15999  1.0
 -3 0.15999  host ceph07   

[ceph-users] Re: Stuck in remapped state?

[Tim Holloway]

Update on "ceph-s". A machine was in the process of crashing when I
took the original snapshot. Here it is after the reboot:

[root@dell02 ~]# ceph -s
  cluster:
id: 278fcd86-0861-11ee-a7df-9c5c8e86cf8f
health: HEALTH_WARN
1 filesystem is degraded
25 client(s) laggy due to laggy OSDs
 
  services:
mon: 3 daemons, quorum dell02,www7,ceph03 (age 8m)
mgr: ceph08.tlocfi(active, since 81m), standbys: www7.rxagfn,
dell02.odtbqw
mds: 1/1 daemons up, 2 standby
osd: 7 osds: 7 up (since 12h), 7 in (since 19h); 308 remapped pgs
rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
volumes: 0/1 healthy, 1 recovering
pools:   22 pools, 681 pgs
objects: 125.10k objects, 36 GiB
usage:   91 GiB used, 759 GiB / 850 GiB avail
pgs: 47772/369076 objects misplaced (12.944%)
 373 active+clean
 308 active+clean+remapped
 
  io:
client:   170 B/s rd, 0 op/s rd, 0 op/s wr

On Sat, 2024-07-27 at 11:31 -0400, Tim Holloway wrote:
> I was in the  middle of tuning my OSDs when lightning blew me off the
> Internet. Had to wait 5 days for my ISP to send a tech and replace a
> fried cable. In the mean time, among other things. I had some serious
> time drift between servers thanks to the OS upgrades replacing NTP
> with
> chrony and me not having thought to re-establish a master in-house
> timeserver.
> 
> Ceph tried really hard to keep up with all that, but eventually it
> was
> just too much. Now I've got an offline filesystem and apparently it's
> stuck trying to get back online again.
> 
> The forensics:
> [ceph: root@www7 /]# ceph -s
>   cluster:
>     id: 278fcd86-0861-11ee-a7df-9c5c8e86cf8f
>     health: HEALTH_WARN
>     failed to probe daemons or devices
>     1 filesystem is degraded
>     1/3 mons down, quorum www7,ceph03
>  
>   services:
>     mon: 3 daemons, quorum www7,ceph03 (age 2m), out of quorum:
> dell02
>     mgr: ceph08.tlocfi(active, since 58m), standbys: dell02.odtbqw,
> www7.rxagfn
>     mds: 1/1 daemons up, 1 standby
>     osd: 7 osds: 7 up (since 12h), 7 in (since 18h); 308 remapped pgs
>     rgw: 2 daemons active (2 hosts, 1 zones)
>  
>   data:
>     volumes: 0/1 healthy, 1 recovering
>     pools:   22 pools, 681 pgs
>     objects: 125.10k objects, 36 GiB
>     usage:   91 GiB used, 759 GiB / 850 GiB avail
>     pgs: 47772/369076 objects misplaced (12.944%)
>  373 active+clean
>  308 active+clean+remapped
>  
>   io:
>     client:   170 B/s rd, 0 op/s rd, 0 op/s wr
> 
> [ceph: root@www7 /]# ceph health detail
> HEALTH_WARN 1 filesystem is degraded; 25 client(s) laggy due to laggy
> OSDs
> [WRN] FS_DEGRADED: 1 filesystem is degraded
>     fs ceefs is degraded
> [WRN] MDS_CLIENTS_LAGGY: 25 client(s) laggy due to laggy OSDs
>     mds.ceefs.www7.drnuyi(mds.0): Client 14019719 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14124385 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14144243 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14144375 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14224103 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14224523 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14234194 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14234545 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14236841 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14237837 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14238536 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14244124 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14264236 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14266870 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14294170 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14294434 is laggy; not
> evicted
> because some OSD(s) is/are laggy
>     mds.ceefs.www7.drnuyi(mds.0): Client 14296012 is laggy; not
> evicted

[ceph-users] Re: Stuck in remapped state?

[Tim Holloway]

Solved, I think.

https://docs.ceph.com/en/latest/cephfs/eviction/

I're reached the point where I've seen the docs before, but when I go
to find them, instead I get the out-of-date stuff. And by out-of-date I
mean things like processes that when out with Octopus but still the
first point of entry for the Reef docs. Frustrating.

On Sat, 2024-07-27 at 11:44 -0400, Tim Holloway wrote:
> Update on "ceph-s". A machine was in the process of crashing when I
> took the original snapshot. Here it is after the reboot:
> 
> [root@dell02 ~]# ceph -s
>   cluster:
>     id: 278fcd86-0861-11ee-a7df-9c5c8e86cf8f
>     health: HEALTH_WARN
>     1 filesystem is degraded
>     25 client(s) laggy due to laggy OSDs
>  
>   services:
>     mon: 3 daemons, quorum dell02,www7,ceph03 (age 8m)
>     mgr: ceph08.tlocfi(active, since 81m), standbys: www7.rxagfn,
> dell02.odtbqw
>     mds: 1/1 daemons up, 2 standby
>     osd: 7 osds: 7 up (since 12h), 7 in (since 19h); 308 remapped pgs
>     rgw: 2 daemons active (2 hosts, 1 zones)
>  
>   data:
>     volumes: 0/1 healthy, 1 recovering
>     pools:   22 pools, 681 pgs
>     objects: 125.10k objects, 36 GiB
>     usage:   91 GiB used, 759 GiB / 850 GiB avail
>     pgs: 47772/369076 objects misplaced (12.944%)
>  373 active+clean
>  308 active+clean+remapped
>  
>   io:
>     client:   170 B/s rd, 0 op/s rd, 0 op/s wr
> 
> On Sat, 2024-07-27 at 11:31 -0400, Tim Holloway wrote:
> > I was in the  middle of tuning my OSDs when lightning blew me off
> > the
> > Internet. Had to wait 5 days for my ISP to send a tech and replace
> > a
> > fried cable. In the mean time, among other things. I had some
> > serious
> > time drift between servers thanks to the OS upgrades replacing NTP
> > with
> > chrony and me not having thought to re-establish a master in-house
> > timeserver.
> > 
> > Ceph tried really hard to keep up with all that, but eventually it
> > was
> > just too much. Now I've got an offline filesystem and apparently
> > it's
> > stuck trying to get back online again.
> > 
> > The forensics:
> > [ceph: root@www7 /]# ceph -s
> >   cluster:
> >     id: 278fcd86-0861-11ee-a7df-9c5c8e86cf8f
> >     health: HEALTH_WARN
> >     failed to probe daemons or devices
> >     1 filesystem is degraded
> >     1/3 mons down, quorum www7,ceph03
> >  
> >   services:
> >     mon: 3 daemons, quorum www7,ceph03 (age 2m), out of quorum:
> > dell02
> >     mgr: ceph08.tlocfi(active, since 58m), standbys: dell02.odtbqw,
> > www7.rxagfn
> >     mds: 1/1 daemons up, 1 standby
> >     osd: 7 osds: 7 up (since 12h), 7 in (since 18h); 308 remapped
> > pgs
> >     rgw: 2 daemons active (2 hosts, 1 zones)
> >  
> >   data:
> >     volumes: 0/1 healthy, 1 recovering
> >     pools:   22 pools, 681 pgs
> >     objects: 125.10k objects, 36 GiB
> >     usage:   91 GiB used, 759 GiB / 850 GiB avail
> >     pgs: 47772/369076 objects misplaced (12.944%)
> >  373 active+clean
> >  308 active+clean+remapped
> >  
> >   io:
> >     client:   170 B/s rd, 0 op/s rd, 0 op/s wr
> > 
> > [ceph: root@www7 /]# ceph health detail
> > HEALTH_WARN 1 filesystem is degraded; 25 client(s) laggy due to
> > laggy
> > OSDs
> > [WRN] FS_DEGRADED: 1 filesystem is degraded
> >     fs ceefs is degraded
> > [WRN] MDS_CLIENTS_LAGGY: 25 client(s) laggy due to laggy OSDs
> >     mds.ceefs.www7.drnuyi(mds.0): Client 14019719 is laggy; not
> > evicted
> > because some OSD(s) is/are laggy
> >     mds.ceefs.www7.drnuyi(mds.0): Client 14124385 is laggy; not
> > evicted
> > because some OSD(s) is/are laggy
> >     mds.ceefs.www7.drnuyi(mds.0): Client 14144243 is laggy; not
> > evicted
> > because some OSD(s) is/are laggy
> >     mds.ceefs.www7.drnuyi(mds.0): Client 14144375 is laggy; not
> > evicted
> > because some OSD(s) is/are laggy
> >     mds.ceefs.www7.drnuyi(mds.0): Client 14224103 is laggy; not
> > evicted
> > because some OSD(s) is/are laggy
> >     mds.ceefs.www7.drnuyi(mds.0): Client 14224523 is laggy; not
> > evicted
> > because some OSD(s) is/are laggy
> >     mds.ceefs.www7.drnuyi(mds.0): Client 14234194 is laggy; not
> > evicted
> > because some OSD(s) is/are laggy
> >     mds.ceefs.www7.drnuyi(mds.0): Client 14234545 is laggy; not
> > evicted
> > because some OSD(s) is/are laggy
> >     mds.ceefs.www7.drnuyi(mds.0): Client 14236841 is lagg

[ceph-users] Re: Cephadm Offline Bootstrapping Issue

[Tim Holloway]

You might want to to try my "bringing up an OSD really, really fast"
package (https://gogs.mousetech.com/mtsinc7/instant_osd).

It's actually for spinning up a VM with an OSD in it, although you can
skip the VM setup script if you're on a bare OS and just run the
Ansible part.

Apologies for anyone who tried to pull it last week, as lightning
destroyed the cable to my ISP and it took them 5 days to get me back on
the Internet. So much for having a business account.

One quirk. You may need to manually copy in a copy of you ceph osd-
bootstrap key to get the operation to complete. I'm not sure why, since
I'd expect cephadm to have dealt with that, and the key has to be
located in /etc/cepn, hot in /etc/ceph/. And may further
be impacted by the filesystem differences internal and external to the
cephadm shell. Which is mildly annoying, but not too bad. Someday I
hope to get that part locked down.

   Tim

On Fri, 2024-08-02 at 13:24 +, Eugen Block wrote:
> Hi,
> 
> I haven't seen that one yet. Can you show the output from these
> commands?
> 
> ceph orch client-keyring ls
> ceph orch client-keyring set client.admin label:_admin
> 
> Is there anything helpful in the mgr log?
> 
> Zitat von "Alex Hussein-Kershaw (HE/HIM)" :
> 
> > Hi,
> > 
> > I'm hitting an issue doing an offline install of Ceph 18.2.2 using
> > cephadm.
> > 
> > Long output below... any advice is appreciated.
> > 
> > Looks like we don't managed to add admin labels (but also trying  
> > with --skip-admin results in a similar health warning).
> > 
> > Subsequently trying to add an OSD fails quietly, I assume because  
> > cephadm is unhappy.
> > 
> > Thanks,
> > Alex
> > 
> > $  sudo  cephadm --image "ceph/ceph:v18.2.2" --docker bootstrap   
> > --mon-ip `hostname -I` --skip-pull --ssh-user qs-admin  
> > --ssh-private-key /home/qs-admin/.ssh/id_rsa --ssh-public-key  
> > /home/qs-admin/.ssh/id_rsa.pub  --skip-dashboard
> > Verifying ssh connectivity using standard pubkey authentication ...
> > Adding key to qs-admin@localhost authorized_keys...
> > key already in qs-admin@localhost authorized_keys...
> > Verifying podman|docker is present...
> > Verifying lvm2 is present...
> > Verifying time synchronization is in place...
> > Unit chronyd.service is enabled and running
> > Repeating the final host check...
> > docker (/usr/bin/docker) is present
> > systemctl is present
> > lvcreate is present
> > Unit chronyd.service is enabled and running
> > Host looks OK
> > Cluster fsid: 65bee110-3ae6-11ef-a1de-005056013d88
> > Verifying IP 10.235.22.8 port 3300 ...
> > Verifying IP 10.235.22.8 port 6789 ...
> > Mon IP `10.235.22.8` is in CIDR network `10.235.16.0/20`
> > Mon IP `10.235.22.8` is in CIDR network `10.235.16.0/20`
> > Internal network (--cluster-network) has not been provided, OSD  
> > replication will default to the public_network
> > Ceph version: ceph version 18.2.2  
> > (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)
> > Extracting ceph user uid/gid from container image...
> > Creating initial keys...
> > Creating initial monmap...
> > Creating mon...
> > Waiting for mon to start...
> > Waiting for mon...
> > mon is available
> > Assimilating anything we can from ceph.conf...
> > Generating new minimal ceph.conf...
> > Restarting the monitor...
> > Setting public_network to 10.235.16.0/20 in mon config section
> > Wrote config to /etc/ceph/ceph.conf
> > Wrote keyring to /etc/ceph/ceph.client.admin.keyring
> > Creating mgr...
> > Verifying port 0.0.0.0:9283 ...
> > Verifying port 0.0.0.0:8765 ...
> > Verifying port 0.0.0.0:8443 ...
> > Waiting for mgr to start...
> > Waiting for mgr...
> > mgr not available, waiting (1/15)...
> > mgr not available, waiting (2/15)...
> > mgr not available, waiting (3/15)...
> > mgr not available, waiting (4/15)...
> > mgr not available, waiting (5/15)...
> > mgr is available
> > Enabling cephadm module...
> > Waiting for the mgr to restart...
> > Waiting for mgr epoch 5...
> > mgr epoch 5 is available
> > Setting orchestrator backend to cephadm...
> > Using provided ssh keys...
> > Adding key to qs-admin@localhost authorized_keys...
> > key already in qs-admin@localhost authorized_keys...
> > Adding host starlight-1...
> > Deploying mon service with default placement...
> > Deploying mgr service with default placement...
> > Deploying crash service with default placement...
> > Deploying ceph-exporter service with default placement...
> > Deploying prometheus service with default placement...
> > Deploying grafana service with default placement...
> > Deploying node-exporter service with default placement...
> > Deploying alertmanager service with default placement...
> > Enabling client.admin keyring and conf on hosts with "admin" label
> > Non-zero exit code 5 from /usr/bin/docker run --rm --ipc=host  
> > --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host  
> > --entrypoint /usr/bin/ceph --init -e  
> > CONTAINER_IMAGE=ceph/ceph:v18.2.2 -e NODE_NAME=starlight-1 -e  
> > CEPH_USE

[ceph-users] Re: Accidentally created systemd units for OSDs

[Tim Holloway]

Been there/did that. Cried a lot. Fixed now.

Personally, I recommend the containerise/cephadm-managed approach. In a
lot of ways, it's simpler and it supports more than one fsid on a
single host.The downside is that the systemd names are really gnarly
(the full fsid is part of the unitname) and they're generated on-the-
fly so you won't find the unitnames in any of the systemd static
directories (/usr/lib/systemd/systemd and /etc/systemd/system).

As I recall, having 2 different managers for an OSD didn't hurt overall
operation that much, but it did make status reporting somewhat flakey.
Apparently different tools check OSDs differently.

The other problem is that the traditional (non-container) OSD
definitions are located under /var.lib/ceph, but the containerized OSDs
are under /var/lib/ceph/. There are some softlinks involved so
that, for example, the OSD data itself may be shared between the two
OSD controllers, and that has the potential for conflict.

I think that the safest way to handle that is to completely nuke that
OSD in both its forms, then re-create it according to your mode of
choice. That can take a long time, but shortcuts may make things worse.

Pity that Ceph can't detect and prevent that sort of stuff, but such is
life.

   Tim

On Fri, 2024-08-16 at 15:32 +, Dan O'Brien wrote:
> I was [poorly] following the instructions for migrating the wal/db to
> an SSD
> 
> https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-for-existing-osd/
> 
> and I didn't add the '--no-systemd' when I did 'ceph-volume lvm
> activate' command (3 f***ing times). The result is that I've
> "twinned" 3 of my OSDs: There's a container version managed by
> cephadm and there's an instantiated systemd unit that runs directly.
> Surprisingly, this has not done a lot of damage, but it does result
> in the dashboard reporting 3 failed cephadm daemons when the "native"
> OSDs start before the containerized ones.
> 
> I've disabled the systemd units for ceph-osd@9.service,
> ceph-osd@11.service and ceph-osd@25.service, but I'd like to remove
> them completely. I will eventually badger The Google into giving me
> an answer, but could someone tell me what I need to do? The semester
> starts soon and I don't really have the bandwidth for this right now.
> 
> Thanks in advance. I will forever be in your debt. (Seriously, I'm
> ready to give you a kidney, if you need it.)
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: squid release codename

[Tim Holloway]

I find the Spongebox ideas amusing, and I agree that in an isolated
world, "Squid" would be the logical next release name.

BUT

It's going to wreak havoc on search engines that can't tell when
someone's looking up Ceph versus the long-establish Squid Proxy.

If we're going to look to the cartoon world, may I suggest the name
"Surume"? It's the name that Luffy gave the Kraken, and if' I'm not
mistaken, actually is the Japanese word for squid!

   Tim



On Thu, 2024-08-15 at 10:22 -0700, Devender Singh wrote:
> Few More If you like. 
> 
> Metaverse 
> Jumbo
> 
> Regards
> Dev
> 
> > On Aug 15, 2024, at 10:12 AM, Tarrago, Eli (RIS-BCT)
> >  wrote:
> > 
> > Additional names that could be considered:
> > 
> > Sea Monster
> > Sea Bear
> > Star (for Patrick star)
> > 
> > And for the best underwater squirrel friend, Sandy.
> > 
> > 
> > From: Boris 
> > Date: Thursday, August 15, 2024 at 9:35 AM
> > To: ceph-users@ceph.io 
> > Subject: [ceph-users] Re: squid release codename
> > *** External email: use caution ***
> > 
> > 
> > 
> > I have to admit: all the Spongebob references are kinda cool :D
> > 
> > Am Do., 15. Aug. 2024 um 15:05 Uhr schrieb Patrick Donnelly <
> > pdonn...@redhat.com>:
> > 
> > > On Thu, Aug 15, 2024 at 8:29 AM Alfredo Rezinovsky
> > >  wrote:
> > > > 
> > > > I think is a very bad idea to name a release with the name of
> > > > the most
> > > > popular http cache.
> > > > It will difficult googling.
> > > 
> > > This was discussed in:
> > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpad.ceph.com%2Fp%2Fs&data=05%7C02%7Celi.tarrago%40lexisnexisrisk.com%7C4979743eff2f4a78329308dcbd2f1847%7C9274ee3f94254109a27f9fb15c10675d%7C0%7C0%7C638593257179433754%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=zdPk%2BtQZtas%2Bo6sAXLgDFAxNNFLKpgoAUHdKVypX118%3D&reserved=0
> > > 
> > > 
> > > That objection was raised and not considered sufficient
> > > justification
> > > to prevent the use of "squid".
> > > 
> > > --
> > > Patrick Donnelly, Ph.D.
> > > He / Him / His
> > > Red Hat Partner Engineer
> > > IBM, Inc.
> > > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > > 
> > 
> > 
> > --
> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
> > abweichend im
> > groüen Saal.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> > 
> > 
> > The information contained in this e-mail message is intended only
> > for the personal and confidential use of the recipient(s) named
> > above. This message may be an attorney-client communication and/or
> > work product and as such is privileged and confidential. If the
> > reader of this message is not the intended recipient or an agent
> > responsible for delivering it to the intended recipient, you are
> > hereby notified that you have received this document in error and
> > that any review, dissemination, distribution, or copying of this
> > message is strictly prohibited. If you have received this
> > communication in error, please notify us immediately by e-mail, and
> > delete the original message.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Accidentally created systemd units for OSDs

[Tim Holloway]

If it makes you feel better, that sounds exactly like what happened to
me and I have no idea how. Other than I'd started with Octopus and it
was a transitional release, there are conflicting instructions AND a
reference in the Octopus docs to procedures using a tool that was no
longer distributed with Octopus.

If by "templated" you mean what I think you mean, the cleanup I'd
recommend is as follows:

1. Follow documented processed for draining and deleting an OSD (the
dashboard seems to handle this well).

2. Make sure that your non-container service is shut down.

3. Also make sure that your OSD is completely removed from the "osd
tree" list. That may require a manual reweighting to 0 which I did from
the command line. My thanks to EEugen Block for that assist.

4. At this point, hopefully a "systemctl status" won't show either the
old-style or container-style OSD services as active. In which case, you
may manually delete the specific OSD unit file from /etc/systemd/system
and it probably won't hurt if you delete the template, but I'd
recommend leaving it, since it will just come back soon.

5. finally, completely erase the OSD directory under /var/lib/ceph and
its counterpart under /var/lib/ceph/, If they're already gone, so
much the better.

The template OSD is used by "ceph control" to dynamically generate the
systemd unit file under /run/ceph/ when it detects an OSD
definition. Which is probably keyed by finding a resource under
/var/lib/ceph//osd.x, if not actually the OSD directory itself.

Unlike /etc/systemd and /etc/systemd, the /run/systemd is effectively
destroyed when the system is down and its ceph files aren't permanent.
Manually erasing them thus has no benefit.

So, in other words, the templated OSD systemd unit won't get created
unless the system actually has cephadm OSDs defined and your problem
vanishes when the OSDs do!

If it's of any help, I've a quick-and-dirty way to spawn minimal VMs
and OSDs you can pull from https://gogs.mousetech.com. It has 2 parts.
One to construct a VM, the other is an Ansible playbooks for installing
the necessary cephadm infrastructure on a new machine, Which should
work on bare OS's as well, as far as I can see. Though I've not tested
that. Customize and/or plunder for whatever benefits it might give you.

Tim

On Fri, 2024-08-16 at 19:24 +, Dan O'Brien wrote:
> I am 100% using cephadm and containers and plan to continue to do so.
> 
> Our original setup was all spinners, but after going to Ceph Days
> NYC, I pushed for SSDs to use for the WAL/RocksDb and I'm in the
> process of migrating the WAL/RocksDb. In general, it's been fairly
> straightforward -- IF YOU FOLLOW THE DIRECTIONS EXACTLY. From what I
> can tell (and someone please correct me if I'm wrong), it appears
> that I've just introduced a bit of cruft into systemd that duplicates
> the configuration of the container OSD. If I can get rid of that bit
> of bellybutton-lint, I think I'm OK without rebuilding the OSD. (At
> least until I screw it up again on one of the remaining OSDs I need
> to migrate).
> 
> Anyone know how to get rid of an instance of a templated systemd
> unit? PLEEZE?
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Accidentally created systemd units for OSDs

[Tim Holloway]

It depends on your available resources, but I really do recommend
destroying and re-creating that OSD. If you have to spin up a VM and
set up a temporary OSD just to keep the overall system happy, even that
is a small price to pay.

As I said, you can't unlink/disable the container systemd, because it's
created and re-recreated dynamically. The non-container systemd unit is
easier to deal with, since it's static but if you're sharing resources
like I was, I still think it safer if you migrate everything out of
that OSD, totally destroy all traces of both realizations of it, then
re-create and migrate back. Ceph can do that, and although it can take
a while to do the migration, it's well worth the effort, Ceph is very
good at keeping data safe, but I'd not abuse the privilege.

Oh yeah, you have multiple OSDs with that problem. Do one at a time and
you'll go faster, Doing a mass shovel of data in and out of one OSD
takes long enough, but trying to run the process in parallel may well
bog down the whole Ceph network.

As I recall, as part of my own fumbling approach, I first stopped and
masked the osd-style systemd unit. But as I said, different subsystems
look in different places to monitor OSDs and I'd get conflicting status
reports. So while in theory, you could brute-force remove the systemd
unit and the /car/lib/ceph/osd.x directory subtree (preferably while
the container is also offline, I think it cleaner and safer overall to
simply go back to a blank slate.

On Fri, 2024-08-16 at 21:05 +, Dan O'Brien wrote:
> OK... I've been in the Circle of Hell where systemd lives and I
> *THINK* I have convinced myself I'm OK. I *REALLY* don't want to
> trash and rebuild the OSDs.
> 
> In the manpage for systemd.unit, I found
> 
> UNIT GARBAGE COLLECTION
> The system and service manager loads a unit's configuration
> automatically when a unit is referenced for the first time. It will
> automatically unload the unit configuration and state again when the
> unit is not needed anymore ("garbage collection").
> 
> I've disabled the systemd units (which removes the symlink from the
> target) for the non-cephadm OSDs I created by mistake and I'm PRETTY
> SURE if I wait long enough (or reboot) that I won't see them any
> more, since there won't be a unit for systemd to care about.
> 
> I *WILL* have to clean up /var/lib/ceph/osd eventually. I tried just
> now, but it says "device busy." I think that's because there's some
> OTHER systemd cruft that shows a mount:
> [root@ceph02 ~]# systemctl --all | grep ceph | grep mount
>   var-lib-ceph-osd-ceph\x2d11.mount loaded    active  
> mounted   /var/lib/ceph/osd/ceph-11
>   var-lib-ceph-osd-ceph\x2d25.mount    loaded    active  
> mounted   /var/lib/ceph/osd/ceph-25
>   var-lib-ceph-osd-ceph\x2d9.mount  loaded    active  
> mounted   /var/lib/ceph/osd/ceph-9
> 
> When things settle down, I *MIGHT* put in a RFE to change the default
> for ceph-volume to --no-systemd to save someone else from this
> anguish.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Accidentally created systemd units for OSDs

[Tim Holloway]

I'd put in an RFO to detect/prevent creation of mutually-exclusive OSD
definitions on a single OSD storage unit myself, since that's the real
problem. As Eugen has noted, you can up-convert a traditional OSD to
cephadm manaegment... unless there's already a managed instance
existing. I can attest that from experience.

the reasons I recommend complete destruction/rebuilding of the
offending OSDs boil down to this:

1. Not knowing the internals of either new or old OSD logic, I feel it
risky to just rip things out by brute force.

2. Because two independent and mutually-ignorant processes have been
working on the same OSD backing store, there is possibility for
corruption.I don't expect an OSD to have suitable interlocks to prevent
that, since normally a single OSD is the sole owner of its backing
store and interlocks would just slow it down for no purpose.

Thus, by cleanly shutting down the old-style OSD process, leaving just
the container-based OSD running, draining that OSD, wiping out
everything that automated cleanup missed and re-creating the OSD, all
of the data in the OSD is going to have passed through the migration
process twice, and I would expect migration to detect and clean up (or
at least report) any inconsistencies, so that they don't pop up months
or years later.

Granted, if you have triple redundancy on the pools in the OSD, the
likelihood of lost/damaged data is pretty low, but in my case, I was
still recovering from a fried Internet connection and didn't want any
more surprises.

So ultimately the choice is yours, Quick-fix or slow.

   Tim

On Sat, 2024-08-17 at 08:05 +, Eugen Block wrote:
> Hi,
> 
> > When things settle down, I *MIGHT* put in a RFE to change the  
> > default for ceph-volume to --no-systemd to save someone else from  
> > this anguish.
> 
> note that there are still users/operators/admins who don't use  
> containers. Changing the ceph-volume default might not be the best  
> idea in this case.
> 
> Regarding the cleanup, this was the thread [1] Tim was referring to.
> I  
> would set the noout flag, stop an OSD (so the device won't be busy  
> anymore), make sure that both ceph-osd@{OSD_ID} and  
> ceph-{FSID}@osd.{OSD_ID} then double check that everything you need
> is  
> still under /var/lib/ceph/{FSID}/osd.{OSD_ID}, like configs an  
> keyrings. Disable the ceph-osd@{OSD_ID} (as already pointed out),
> then  
> check if the orchestrator can start the OSD via systemd:
> 
> ceph orch daemon start osd.{OSD_ID}
> 
> or alternatively, try it manually:
> 
> systemctl reset-failed
> systemctl start ceph-{FSID}@osd.{OSD_ID}
> 
> Watch the log for that OSD to identify any issues. If it works,
> unset  
> the noout flag. You might want to ensure it also works after a
> reboot,  
> though.
> I don't think it should be necessary to redeploy the OSDs, but the  
> cleanup has to be proper.
> As a guidance you can check the cephadm tool's contents and look for 
> the "adopt" function. That migrates the contents of the pre-cephadm  
> daemons into the FSID specific directories.
> 
> Regards,
> Eugen
> 
> [1]  
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/K2R3MXRD3S2DSXCEGX5IPLCF5L3UUOQI/
> 
> Zitat von Dan O'Brien :
> 
> > OK... I've been in the Circle of Hell where systemd lives and I  
> > *THINK* I have convinced myself I'm OK. I *REALLY* don't want to  
> > trash and rebuild the OSDs.
> > 
> > In the manpage for systemd.unit, I found
> > 
> > UNIT GARBAGE COLLECTION
> > The system and service manager loads a unit's configuration  
> > automatically when a unit is referenced for the first time. It
> > will  
> > automatically unload the unit configuration and state again when
> > the  
> > unit is not needed anymore ("garbage collection").
> > 
> > I've disabled the systemd units (which removes the symlink from
> > the  
> > target) for the non-cephadm OSDs I created by mistake and I'm
> > PRETTY  
> > SURE if I wait long enough (or reboot) that I won't see them any  
> > more, since there won't be a unit for systemd to care about.
> > 
> > I *WILL* have to clean up /var/lib/ceph/osd eventually. I tried
> > just  
> > now, but it says "device busy." I think that's because there's
> > some  
> > OTHER systemd cruft that shows a mount:
> > [root@ceph02 ~]# systemctl --all | grep ceph | grep mount
> >   var-lib-ceph-osd-ceph\x2d11.mount loaded    active    
> > mounted   /var/lib/ceph/osd/ceph-11
> >   var-lib-ceph-osd-ceph\x2d25.mount    loaded    active    
> > mounted   /var/lib/ceph/osd/ceph-25
> >   var-lib-ceph-osd-ceph\x2d9.mount  loaded    active    
> > mounted   /var/lib/ceph/osd/ceph-9
> > 
> > When things settle down, I *MIGHT* put in a RFE to change the  
> > default for ceph-volume to --no-systemd to save someone else from  
> > this anguish.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> 
> 

[ceph-users] Prometheus and "404" error on console

[Tim Holloway]

Although I'm seeing this in Pacific, it appears to be a perennial issue
with no well-documented solution. The dashboard home screen is flooded
with popups saying "404 - Not Found

Could not reach Prometheus's API on
http://ceph1234.mydomain.com:9095/api/v1
"

If I was a slack-jawed PHB casually wandering into the operations
center and saw that, I'd probably doubt that Ceph was a good product
decision.

The "404" indicates that the Prometheus server is running and accepting
requests, but all the "404" says is that whatever requests it's
receiving are meaningless to it. Presumably either some handler needs
to be jacked into Prometheus or the sender needs to be notified to
desist.

Unfortunately, "404" doesn't provide any clues. Ideally, ceph should
log something more specific by default, but failing that, if anyone
knows how to shut it up (cleanly), I'd appreciate knowing!

   Tim
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Prometheus and "404" error on console

[Tim Holloway]

Since I use keepalived, I can affirm with virtual certainty that
keepalived could do stuff like that. Although it may involve using
special IP address that keepalived would aim at the preferred server
instance.

But that's not the problem here, as "404" means that the server is up,
but it sneers at what you are pushing at it.

If Prometheus could, for example, log WHAT is being pushed at it, it
would go a long way.

   Tim

On Mon, 2024-08-19 at 08:40 -0400, Daniel Brown wrote:
> 
> 
> I’ve seen similar. 
> 
> Have been wondering if it would be possible to either setup a
> LoadBalancer or something like “keeepalived” to provide a “VIP” which
> could move between nodes to support the dashboard (and prometheus,
> Grafana, etc).
> 
> I do see notes about HA Proxy in the docs, but haven’t gotten to
> trying that setup: 
> 
> https://docs.ceph.com/en/quincy/mgr/dashboard/#haproxy-example-configuration
> 
> 
> 
>  
> In my opinion, a VIP for the dashboard (etc.) could and maybe should
> be and out of the box config. 
> 
> 
> 
> 
> 
> > On Aug 19, 2024, at 8:23 AM, Tim Holloway 
> > wrote:
> > 
> > Although I'm seeing this in Pacific, it appears to be a perennial
> > issue
> > with no well-documented solution. The dashboard home screen is
> > flooded
> > with popups saying "404 - Not Found
> > 
> > Could not reach Prometheus's API on
> > http://ceph1234.mydomain.com:9095/api/v1
> > "
> > 
> > If I was a slack-jawed PHB casually wandering into the operations
> > center and saw that, I'd probably doubt that Ceph was a good
> > product
> > decision.
> > 
> > The "404" indicates that the Prometheus server is running and
> > accepting
> > requests, but all the "404" says is that whatever requests it's
> > receiving are meaningless to it. Presumably either some handler
> > needs
> > to be jacked into Prometheus or the sender needs to be notified to
> > desist.
> > 
> > Unfortunately, "404" doesn't provide any clues. Ideally, ceph
> > should
> > log something more specific by default, but failing that, if anyone
> > knows how to shut it up (cleanly), I'd appreciate knowing!
> > 
> >   Tim
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-ansible installation error

[Tim Holloway]

I believe that the original Ansible installation process is deprecated.
It was pretty messy, anyway, since it had to do a lot of grunt work.
Likewise the ceph-install program, which is in the Octopus docs, but
wasn't actually available in the release of Octopus I installed on my
servers.

The Ansible playbooks also assumed things about the environment that
didn't fit my setup, so I never used them.

The good news is that unless you're determined to install a pre-Octopus
release of Ceph or want the legacy setup, the cephadm application makes
life considerably simpler.

In fact, I rolled my own playbooks and all it contains are the
following steps:

1. Install the ceph repository that corresponds to the release of Ceph
you want.

2. Install the cephadm package from that repository. Optionally you can
install ceph-common if you don't want to be restricted to doing all
your work via "cephadm shell".

3. Copy your ceph config files to the targer machine.

Now when you first run cephadm it will download and install the
docker/podman container image that is shared between most of the Ceph
daemons and after that, you're pretty much good to go. Image
installation might not get done until you attempt something such as
creating an OSD on that machine, but it will be done when needed.

As a final note, even in the latest Ceph releases prefer you to the old
(legecy) "Manual procedure". IGNORE THAT Lock for the cephadm
equivalents. They're shy, but a little searching will show them.

In other words, you don't really need a ready-made playbook now. You
can roll your own like I did, but that's just a convenience.

   Tim

On Fri, 2024-08-30 at 17:53 +0300, Michel Niyoyita wrote:
> Dear team ,
> 
> 
> I configuring ceph cluster using ceph-ansible , ubuntu OS 20.04 . my
> previous production cluster have been configured using the same
> configuration and it is working perfectly , now I am trying to build
> the
> new cluster using the same configurations , but I am facing the
> following errors:
> 
> ansible-playbook site.yml
> Traceback (most recent call last):
>   File "/usr/bin/ansible-playbook", line 66, in 
>     from ansible.utils.display import Display, initialize_locale
> ImportError: cannot import name 'initialize_locale' from
> 'ansible.utils.display'
> (/usr/local/lib/python3.8/dist-packages/ansible/utils/display.py)
> 
> Can someone help to solve the issue?
> 
> Best regards
> 
> Michel
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: How to know is ceph is ready?

[Tim Holloway]

"Ceph is ready" covers a lot of territory. It's more like, "How can I
delay util Ceph is available for the particular service I need?

I've been taking a systemd-bsaed approach. Since I don't actually care
bout Ceph in the abstract, but I'm actually looking for the Ceph or
Ceph NFS shares, I create dependencies in my systemd setup such that
the dependent VM or container (quadlet) requires its ceph mounts to be
online before that service can start.

I'm still evaluating how well that works. I'll let you know after the
next hurricane hits. But so far, so good.

  Tim

On Fri, 2024-08-30 at 11:02 -0300, Alfredo Rezinovsky wrote:
> steps 2 to 4 are exactly my idea.
> 
> In step 1 All I will check is all active (o non inactive or unknown).
> Clean
> is not necessary, I can boot VMs with degraded.as long they are
> active.
> 
> El jue, 29 ago 2024 a las 8:03, Bogdan Adrian Velica
> ()
> escribió:
> 
> > Hi,
> > 
> > It's a hacky idea but create a script that checks if the Ceph RBD
> > pool is
> > fully "active+clean" to ensure it's ready before starting Proxmox
> > VMs.
> > Something like this...
> > 
> > 1. Bash Script:
> > - Write a script that checks if the Ceph RBD pool is in the
> > "active+clean"
> > state using ceph pg stat or ceph -s.
> > - The script should run until the pool is ready before exiting. A
> > loop or
> > something...
> > 
> > 2. Systemd Service for Ceph Readiness:
> > - Create a systemd service unit file (something like: ceph-
> > ready.service)
> > that runs the script at startup.
> > - Ensure the service waits until the Ceph cluster is ready before
> > proceeding.
> > 
> > 3. Systemd Service for Proxmox VMs:
> > - Modify an existing Proxmox service or create a new one to depend
> > on the
> > ceph-ready.service.
> > - Use After=ceph-ready.service in the unit file to ensure Proxmox
> > VMs
> > start only after Ceph is ready.
> > 
> > 4. Enable and Start the Services:
> > - Enable both systemd services to start at boot with systemctl
> > enable.
> > - Reboot to ensure that Proxmox waits for Ceph before starting VMs.
> > 
> > Just an idea...
> > 
> > Thank you,
> > Bogdan Velica
> > croit.io
> > 
> > On Thu, Aug 29, 2024 at 1:53 PM Alfredo Rezinovsky
> > 
> > wrote:
> > 
> > > I have a proxmox cluster using an external CEPH cluster.
> > > 
> > > Sometimes due to blackouts the servers need to restart. If
> > > proxmox starts
> > > before CEPH is ready the VMs fail to boot.
> > > 
> > > I want to add a dependency in proxmox to wait for ceph to be
> > > ready.
> > > 
> > > I can work with a HEALTH_WARN as long the RBD pool is usable.
> > > 
> > > ceph status exit status doesn´t helps
> > > 
> > > Should I grep for "pgs not active" in ceph status or for
> > > "inactive" pgs in
> > > ceph health or is there something more direct to know if
> > > everything is
> > > alright?
> > > 
> > > 
> > > 
> > > --
> > > Alfrenovsky
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > > 
> > 
> 
> -- 
> Alfrenovsky
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-ansible installation error

[Tim Holloway]

Those are reasonable objections, although some are now dated. In the 
context of Ceph some of those issues are also further addressed by Ceph 
itself. So let me present my take.


1. Networking. You can set up some gnarly virtual networks in both 
container and cloud systems, it's true. Docker has changed some of its 
original rules as well but more on that in a bit. In my non-ceph 
containers I've simply used host networking, which has its drawbacks but 
it's simple and it's all I require.


2. Logs in containerized ceph almost all go straight to the system 
journal. Specialized subsystems such as Prometheus can be configured in 
other ways, but everything's filed under 
/var/lib/ceph// so there's relatively little confusion.


3. I don't understand this, as I never stop all services just to play 
with firewalls. RHEL 8+ support firewall-cmd and you can can open ceph 
up with --add-service ceph, -add-service ceph-mon. Make them --permanent 
and do a --reload and it's all done.


4. Ceph knows exactly the names and locations of its containers (NOTE: a 
"package" is NOT a "container"). Within Ceph, almost all services 
actually employ the same container, just with different invocation 
options. You don't talk to "Docker*" directly, though, as systemd 
handles that.


5. I've never encountered this, so I can say nothing. But I run 
containers for about 6 different build base systems 24x7.


6. As I said, Ceph does almost everything via cephadm or ceph orch when 
running in containers, which actually means you need to learn less. 
Administration of ceph itself, is, again, done via systemd.


*Docker. As I've said elsewhere, Red Hat prefers Podman to Docker these 
days and even if you install Docker, there's a Podman transparency 
feature. Now if you really want networking headaches, run Podman 
containers rootless. I've learned how to account for the differences but 
Ceph, fortunately hasn't gone that route so far. Nor have they 
instituted private networks for Ceph internal controls.



On 9/1/24 15:54, Anthony D'Atri wrote:

* Docker networking is a hassle
* Not always clear how to get logs
* Not being able to update iptables without stopping all services
* Docker package management when the name changes at random
* Docker core leaks and kernel compatibility
* When someone isn’t already using containers, or has their own orchestration, 
going to containers steepens the learning curve.

Containers have advantages including decoupling the applications from the 
underlying OS


I would greatly like to know what the rationale is for avoiding containers

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-ansible installation error

[Tim Holloway]

I would greatly like to know what the rationale is for avoiding
containers.

Especially in large shops. From what I can tell, you need to use the
containerized Ceph if you want to run multiple Ceph filesystems on a
single host. The legacy installations only support dumping everything
directly under /var/lib/ceph, so you'd have to invest a lot of effort
into installing, maintaining and operating a second fsid under the
legacy architecture.

Plus, IBM Red Hat is a big fan of containers, so if you're a large
corporation that likes IBM hand-holding, they're throwing their support
in a direction contrary to the old install-directly approach,

And from an IBM viewpoint, supporting containers is generally going to
be easier than supporting software that's directly splattered all over
the OS. And much less overhead that spinning up an entire VM.

The only definite argument I've ever heard in my insular world against
containers was based on security. Yet the primary security issues
seemed to be more because people were pulling insecure containers from
Docker repositories. I'd expect Ceph to have safeguards. Plus Ceph
under RHEL 9 (and 8?) will run entirely and preferably under Podman,
which allegedly is more secure, and can in fact, run containers under
user accounts to allow additional security. I do that myself, although
I think the mechanisms could stand some extra polishing.

   Tim

On Sat, 2024-08-31 at 09:49 +0200, Janne Johansson wrote:
> Den fre 30 aug. 2024 kl 20:43 skrev Milan Kupcevic
> :
> > 
> > On 8/30/24 12:38, Tim Holloway wrote:
> > > I believe that the original Ansible installation process is
> > > deprecated.
> > 
> > This would be a bad news as I repeatedly hear from admins running
> > large
> > storage deployments that they prefer to stay away from containers.
> 
> You have other choices than Ansible or containers. It has always been
> possible to install using rpm/deb's manually, using any kind of
> scripts or frameworks. The point is that the ceph people no longer
> tries to make pre-made ansible scripts that work "everywhere",
> because
> they didn't.
> 
> This does not prevent you in any way from avoiding containers (if
> that
> is what you want) but it makes you responsible for figuring out the
> automation part if you need one.
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-ansible installation error

[Tim Holloway]

Sorry if that sounds trollish. It wasn't intended to be. Look at it this 
way.


There are two approaches to running an IT installation. One is the 
free-wheeling idependent aproach. The other is the stuffy corporate 
approach.


Free-wheeling shops run things like Ubuntu. Or even BSD (but that's 
another matter). They often roll their own solutions. Frequently they 
were set up by a resident genius who liked some particular platform. And 
more power to them. Whether they do or don't buy into containerization 
is basically going to be determined on whether their guiding genius(es) 
take an interest in such things.


They can ignore the points I just made. Said points were aimed at the 
stuffy business crowd. All that remains to be said on containers for 
that topic is that as far as I know Ceph never promoted setting up 
containerization manually.


Now Ubuntu tries very hard to support the Red Hat Enterprise products, 
but there are certain quirks to the Red Hat world that make it extra 
challenging. Literally from the very start. Kickstart and preseed are 
not directly compatible, and in general, the RPM-based distros have one 
crucial difference from the Debian world in that at no time does an RPM 
ever halt for interactive input. So the non-Red Hat folks do have to 
work a little harder. Ceph, I believe does have some serious Ubuntu 
support, but at its core it is a Red Hat/IBM product. And IBM Red Hat 
thus has significant control over its shaping and direction.


So, for the stuffy business crowd. Banks, insurance companies, Fortune 
corporations, as opposed to public institutions (where they're more 
likely to be free-wheeling).


These organizations generally pay for Red Hat contracts (bless them, Red 
Hat stock paid for my wife's retirement). They want the "neck to choke", 
they want someone to cry to (besides this forum!). They want access to 
the restricted Red Hat Q&A. They're generally going to install RHEL or 
at a minimum a CentOS-style equivalent. And they're going to go with 
what their paid support tells them to do.


As of Ceph Octopus, the older various methods of deploying Ceph have 
apparently been deprecated in favor of cephadm, and I expect that when 
you call for support, they're going to be hoping that's what your using. 
The cephadm utility collapses a LOT of previous manual work down into a 
central control program. It also (as far as I know, has no support for 
the legacy Ceph structure, being entirely based on "administered" (e.g. 
containerized) servers. Rather like virsh handles almost everything for 
KVM-based VMs, cephadm is the only program you have to install to 
ceph-enable a real or virtual machine.


The fact that "administered" daemons are container-based is actually 
incidental, as from the Ceph administrator's point of view, the daemons 
are black boxes. It does not support roll-your-own Ceph containers; it 
uses its own and everything is controlled either via cephadm or systemd. 
Being a container expert is neither required nor expected. And could 
even potentially get you into trouble (not that I'd ever have had that 
problem...).


The more critical point is that ceph has abandoned older 
install/maintenance processes. Ansible is no longer supported, and as I 
said, even though the Octopus docs give examples using ceph-install, 
that utility is not available from the repository install of Octopus. So 
at this point, it appears to be a choice between keeping legacy 
structure and doing everything the hard (manual) way or using cephadm. 
There is a serious possibility that eventually that choice may no longer 
exist.


Aside from making administration simpler overall, if you want to run 
multiple fsids on a single machine, the only simple way to do that is 
via cephadm. Incidentally, the Ceph config does still remain in 
/etc/ceph and is essentially unchanged, allowing for the items that now 
get stored in object store instead of in text configs.



A final note. The cephadm utility also can convert legacy resources into 
administered resources with a single command, thus easing migration. You 
can have both legacy and administered resources concurrently on the same 
machine so migrations can be gradual and outages are minimized.


Ultimately, the choice is yours, for now. But it wouldn't be a bad idea 
to set up a test cephadm-based system just to get familiar with the 
concept. Avoid Octopus, though. It had some serious teething issues, as 
I found to my pain.


  Tim

On 9/2/24 11:41, Anthony D'Atri wrote:

I should know to not feed the trolls, but here goes.  I was answering a 
question asked to the list, not arguing for or against containers.



2. Logs in containerized ceph almost all go straight to the system journal. Specialized 
subsystems such as Prometheus can be configured in other ways, but everything's filed under 
/var/lib/ceph// so there's relatively little confusion.

I see various other paths, which often aren’t /var/log. And don’t conflate 
“con

[ceph-users] Re: Discovery (port 8765) service not starting

[Tim Holloway]

While I generally don't recommend getting down and dirty with the
containers in Ceph, if you're going to build your own, well, that's
different.

When I have a container and the expected port isn't listening, the
first thing I do is see if it's really listening and internal-only or
truly not listening.

To do this, you can dial in to the container, like so (this is one of
my OSD's)

docker exec -it ceph-272fcd86-0831-11ee-a7df-9c5c8e84cf8f-osd-7
/bin/bash

The netstat command is not available in the stock Ceph containers, but
the "ss" command is, so use that to see if there is in fact a process
listening on that port.

If so, then your Dockerfile is probably not set to publish that port.
Are you using the stock Dockerfile to build from?

If not, then the process that listens apparently didn't come all the
way up.

Take a quick peek around the local /var/log directory just to see if
there's anything logged internally (probably not). Maybe do a "dmesg"
just for laughs.

Since this is (so I read it), the Prometheus container, look at the
/var/lib/ceph//prometheus. directory, which contains
the Prometheus config file (under "./etc) and see if there are
irregularities there.

   Tim

On Tue, 2024-09-03 at 13:59 +0100, Matthew Vernon wrote:
> On 03/09/2024 13:33, Eugen Block wrote:
> > Oh that's interesting :-D I have no explanation for that, except
> > maybe 
> > some flaw in your custom images? Or in the service specs? Not sure,
> > to 
> > be honest...
> 
> So obviously it _could_ be something in our images, but we're using 
> Ceph's published .debs (18.2.2) and not doing anything clever:
> 
> https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/refs/heads/master/images/ceph/Dockerfile.template
> 
> I've not done anything to specifically ask for the discovery endpoint
> to 
> be installed - as far as I can tell from the docs it should just get 
> started when you enable the prometheus endpoint (which does seem to
> be 
> working)...
> 
> Regards,
> 
> Matthew
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Discovery (port 8765) service not starting

[Tim Holloway]

Yeah. Although taming the Prometheus logs is on my list, I'm still
fuzzy on its details.

For your purposes, Docker and Podman can be considered as equivalent. I
also run under Podman, incidentally.

If the port isn't open inside the container, then blame Prometheus. I'd
consider bumping its logging levels - the exact opposite of what I'm
looking to do!

  Tim


On Tue, 2024-09-03 at 15:34 +0100, Matthew Vernon wrote:
> Hi,
> 
> On 03/09/2024 14:27, Tim Holloway wrote:
> 
> FWIW, I'm using podman not docker.
> 
> > The netstat command is not available in the stock Ceph containers,
> > but
> > the "ss" command is, so use that to see if there is in fact a
> > process
> > listening on that port.
> 
> I have done this, and there's nothing listening on port 8765
> per
> ss -l | grep 8765
> 
> > If not, then the process that listens apparently didn't come all
> > the
> > way up.
> 
> > Take a quick peek around the local /var/log directory just to see
> > if
> > there's anything logged internally (probably not). Maybe do a
> > "dmesg"
> > just for laughs.
> 
> Nothing obvious - and logs are available outside the container as
> well - 
> hence the paste earlier in this thread, 
> https://phabricator.wikimedia.org/P68521
> > Since this is (so I read it), the Prometheus container, look at the
> > /var/lib/ceph//prometheus. directory, which
> > contains
> > the Prometheus config file (under "./etc) and see if there are
> > irregularities there.
> 
> It's not, it's the mgr container (I've enabled the prometheus mgr 
> module, which makes an endpoint available from whence metrics can be 
> scraped, rather than the prometheus container which runs the service 
> that does the scraping).
> 
> Thanks,
> 
> Matthew
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Discovery (port 8765) service not starting

[Tim Holloway]

You may find this interesting.

I'm running Pacific from the Red Hat repo and Prometheus was given its
own discrete container image, not the generic Ceph one. Rather than
build custom Prometheus, Red Hat used the Prometheus project's own
containers.

In fact, it has 3: one for Prometheus, one for Prometheus node-exporter
and one for Prometheus alertmanager. Also Grafana is using its own
image as well.

Nothing on the host in question listens on Port 8765. Prometheus is
listening on ports in the 9000 range.

For whatever it's worth.

The name of the API you're interested in, on the other hand, sounds
familiar, but I'm too senile to recall from where.

   Tim

On Tue, 2024-09-03 at 11:48 -0400, Tim Holloway wrote:
> Yeah. Although taming the Prometheus logs is on my list, I'm still
> fuzzy on its details.
> 
> For your purposes, Docker and Podman can be considered as equivalent.
> I
> also run under Podman, incidentally.
> 
> If the port isn't open inside the container, then blame Prometheus.
> I'd
> consider bumping its logging levels - the exact opposite of what I'm
> looking to do!
> 
>   Tim
> 
> 
> On Tue, 2024-09-03 at 15:34 +0100, Matthew Vernon wrote:
> > Hi,
> > 
> > On 03/09/2024 14:27, Tim Holloway wrote:
> > 
> > FWIW, I'm using podman not docker.
> > 
> > > The netstat command is not available in the stock Ceph
> > > containers,
> > > but
> > > the "ss" command is, so use that to see if there is in fact a
> > > process
> > > listening on that port.
> > 
> > I have done this, and there's nothing listening on port 8765
> > per
> > ss -l | grep 8765
> > 
> > > If not, then the process that listens apparently didn't come all
> > > the
> > > way up.
> > 
> > > Take a quick peek around the local /var/log directory just to see
> > > if
> > > there's anything logged internally (probably not). Maybe do a
> > > "dmesg"
> > > just for laughs.
> > 
> > Nothing obvious - and logs are available outside the container as
> > well - 
> > hence the paste earlier in this thread, 
> > https://phabricator.wikimedia.org/P68521
> > > Since this is (so I read it), the Prometheus container, look at
> > > the
> > > /var/lib/ceph//prometheus. directory, which
> > > contains
> > > the Prometheus config file (under "./etc) and see if there are
> > > irregularities there.
> > 
> > It's not, it's the mgr container (I've enabled the prometheus mgr 
> > module, which makes an endpoint available from whence metrics can
> > be 
> > scraped, rather than the prometheus container which runs the
> > service 
> > that does the scraping).
> > 
> > Thanks,
> > 
> > Matthew
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Discovery (port 8765) service not starting

[Tim Holloway]

I've been monitoring my Ceph LAN segment for the last several hours and
absolutely no traffic has shown up on any server for port 8765.

Furthermore I did a quick review of Prometheus itself and it's only
claiming those 9000-series ports I mentioned previously.

So I conclude that this isn't literally a Prometheus problem, although
it could be something external that's supposed to plug in to Prometheus
that isn't part of my standard Ceph setup.

Since port 8765 isn't a Well-Known Port, that's about all I can say on
that matter. I'd only panic if Prometheus isn't collecting something
that's important to me.

  Tim

On Wed, 2024-09-04 at 16:22 +0100, Matthew Vernon wrote:
> Hi,
> 
> I tracked it down to 2 issues:
> 
> * our ipv6-only deployment (a bug fixed in 18.2.4, though that has
> buggy 
> .debs)
> 
> * Discovery service is only run on the active mgr
> 
> The latter point is surely a bug? Isn't the point of running a
> service 
> discovery endpoint that one could point e.g. an external Prometheus 
> scraper at the service discovery endpoint of any mgr and it would
> then 
> tell Prometheus where to scrape metrics from (i.e. the active mgr)?
> 
> Thanks,
> 
> Matthew
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Issue Replacing OSD with cephadm: Partition Path Not Accepted

[Tim Holloway]

One of my major regrets is that there isn't a "Ceph Lite" for setups
where you want a cluster with "only" a few terabytes and a half-dozen
servers. Ceph excels at really, really big storage and the tuning
parameters reflect that.

I, too ran into the issue where I couldn't allocate a disk partition to
Ceph. I didn't really want the extra potential overhead of LVM, but I
had a 4TB SSD and only needed about a quarter of that for an OSD with
the rest for local private use, so a partition seemed like a good
compromise. No such luck. It was either the entire 4TB or LVM. So it
ended up as LVM.

   Tim


On Wed, 2024-09-04 at 10:47 +, Eugen Block wrote:
> Hi,
> 
> apparently, I was wrong about specifying a partition in the path  
> option of the spec file. In my quick test it doesn't work either.  
> Creating a PV, VG, LV on that partition makes it work:
> 
> ceph orch daemon add osd soc9-ceph:data_devices=ceph-manual-vg/ceph-
> osd
> Created osd(s) 3 on host 'soc9-ceph'
> 
> But if you want an easy cluster management, especially OSDs, I'd  
> recommend to use the entire (raw) devices. It really makes (almost)  
> everything easier. Just recently a customer was quite pleased with
> the  
> process compared to prior ceph versions. They removed a failed disk  
> and only had to run 'ceph orch osd rm  --replace' and ceph did  
> the rest (given the existing service specs cover it). They literally 
> said:"wow, this changes our whole impression of ceph". They didn't  
> have many disk replacements in the past 5 years, this was the first  
> one since they adopted the cluster with cephadm.
> 
> Fiddling with partitions seems quite unnecessary, especially on
> larger  
> deployments.
> 
> Regards,
> Eugen
> 
> Zitat von Herbert Faleiros :
> 
> > On 03/09/2024 03:35, Robert Sander wrote:
> > > Hi,
> > 
> > Hello,
> > 
> > > On 9/2/24 20:24, Herbert Faleiros wrote:
> > > 
> > > > /usr/bin/docker: stderr ceph-volume lvm batch: error: /dev/sdb1
> > > > is a
> > > > partition, please pass LVs or raw block devices
> > > 
> > > A Ceph OSD nowadays needs a logical volume because it stores  
> > > crucial metadata in the LV tags. This helps to activate the OSD.
> > > IMHO you will have to redeploy the OSD to use LVM on the disk.
> > > It  
> > > does not need to be the whole disk if there is other data on it.
> > > It  
> > > should be sufficient to make /dev/sdb1 a PV of a new VG for the
> > > LV  
> > > of the OSD.
> > 
> > thank you for the suggestion. I understand the need for Ceph OSDs
> > to  
> > use LVM due to the metadata stored in LV tags. However, I’m facing
> > a  
> > challenge with the disk replacement process. Since I’ve already  
> > migrated the OSDs to use |ceph-volume|, I was hoping that
> > |cephadm|  
> > would handle the creation of LVM structures automatically.  
> > Unfortunately, it doesn’t seem to recreate these structures on its 
> > own when replacing a disk, and manually creating them isn’t ideal  
> > because |ceph-volume| uses its own specific naming conventions.
> > 
> > Do you have any recommendations on how to proceed with |cephadm|
> > in  
> > a way that it can handle the LVM setup automatically, or perhaps  
> > another method that aligns with the conventions used by |ceph-
> > volume|?
> > 
> > --
> > 
> > Herbert
> > 
> > 
> > > Regards
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Discovery (port 8765) service not starting

[Tim Holloway]

Now you've got me worried. As I said, there is absolutely no traffic
using port 8765 on my LAN.

Am I missing a service? Since my distro is based on stock Prometheus,
I'd have to assume that the port 8765 server would be part of the Ceph
generic container image and isn't being switched on for some reason.

   Tim

On Thu, 2024-09-05 at 15:05 +0100, Matthew Vernon wrote:
> On 05/09/2024 15:03, Matthew Vernon wrote:
> > Hi,
> > 
> > On 05/09/2024 12:49, Redouane Kachach wrote:
> > 
> > > The port 8765 is the "service discovery" (an internal server that
> > > runs in
> > > the mgr... you can change the port by changing the
> > > variable service_discovery_port of cephadm). Normally it is
> > > opened in the
> > > active mgr and the service is used by prometheus (server) to get
> > > the
> > > targets by using the http service discovery feature [1]. This
> > > feature has
> > > been there for a long time now and it's the default configuration
> > > used by
> > > Ceph monitoring stack. It should start automatically without any
> > > external
> > > intervention (or manual configuration). 
> > 
> > Right; it wasn't running because I have an IPv6 deployment (that
> > bug's 
> > fixed in 18.2.4 - https://tracker.ceph.com/issues/63448).
> 
> ...though I'm not sure that having only the active mgr run this
> endpoint 
> is correct, though? Isn't it more useful to be able to e.g. point my 
> Prometheus at any of the mgrs and have service discovery work, rather
> than needing Prometheus to know which mgr is active to know which sd
> to 
> talk to, which seems to rather defeat the point?
> 
> Thanks,
> 
> Matthew
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Discovery (port 8765) service not starting

[Tim Holloway]

Thank you, Redouane!

Some background.

I migrated to Ceph amidst a Perfect Storm. The Ceph docs, as I've often
complained, were/are a horrible mish-mash of deprecated instructions
and more modern information. So, among other things, I ended up with a
mess of resources, some legacy-based, some managed, and some OSDs that
managed to get defined as both.

On top of that, this was Ceph Octopus, which had the flaw that
scheduled changes to the system would hang until the system was in OK
status, but the way to make the system OK was to have those scheduled
changes actually executed.

I managed to clear that part out by migrating to Pacific. I originally
settled on Octopus because I wrongly had thought that no newer release
was supported standard under CentOS 7.

Over time I thought I'd managed to get the system clean, via a
combination of documented processes and occasional brute force, but it
looks like something may not have been configured/activated, since as I
said, I see no traffic targeting port 8765 not anyone listening there.

So if you could guide me on fixing that, I'd be grateful.

  Tim

On Fri, 2024-09-06 at 09:08 +0200, Redouane Kachach wrote:
> Hi Matthew,
> 
> That makes sense. The ipv6 BUG can lead to the issue you described.
> In the current implementation whenever a mgr failover takes place,
> prometheus configuration (when using the monitoring stack deployed by
> Ceph) is updated automatically to point to the new active mgr.
> Unfortunately it's not easy to have active services running in the
> standby mgr. At most, we can do some redirection as we do in the
> dashboard. So far we haven't had the need to do that. Next releases
> will come with the new mgmt-gateway service introduced in [1] and [2]
> which will make it easy to have a single entry point to the cluster
> handling HA transparently in the backend. This is still WIP but you
> can play with it if you want using the latest code from main. Support
> for OIDC based on oauth2-proxy is also being introduced as part of
> this effort by [3].
> 
> @ Timo Holloway, as I said the support [4] for service discovery has
> been there for a while (I'd say 2 years aprox) unless you are using
> an old Ceph version (where the prometheus config was static) you
> should see traffic in the port 8765.
> 
> [1] https://github.com/ceph/ceph/pull/57535
> [2] https://github.com/ceph/ceph/pull/58402
> [3] https://github.com/ceph/ceph/pull/58460
> [4] https://github.com/ceph/ceph/pull/46400
> 
> 
> 
> 
> On Thu, Sep 5, 2024 at 7:00 PM Tim Holloway 
> wrote:
> > Now you've got me worried. As I said, there is absolutely no
> > traffic
> > using port 8765 on my LAN.
> > 
> > Am I missing a service? Since my distro is based on stock
> > Prometheus,
> > I'd have to assume that the port 8765 server would be part of the
> > Ceph
> > generic container image and isn't being switched on for some
> > reason.
> > 
> >    Tim

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io