[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Janne Johansson]

Den ons 17 nov. 2021 kl 18:41 skrev Dave Hall :
>
> The real point here:  From what I'm reading in this mailing list it appears
> that most non-developers are currently afraid to risk an upgrade to Octopus
> or Pacific.  If this is an accurate perception then THIS IS THE ONLY
> PROBLEM.

You might also consider that Luminous had a bad streak somewhere in
the middle, so if people
are cautious about .0 / .1 releases, wait until .5-.9 and still get
burnt, that feeling gets stuck in your mind.

Kraken was experimental, so Hammer and Jewel clusters waited for L to
settle, then they got all kinds of
weird bugs in the middle of that release cycle anyway.

Jumping to a newish Mimic might not have felt like the best option to
escape Lum bugs.
Half of the problem of running into bugs like the ones in Lum is that
you often need to be able to move back out of them, before moving
forward again.

There is no guarantee that the developed-in-parallel M point release
.0/.1/.2 will have
corrective code that fixes the newly introduced errors in Lum, so
holding out for the
next Lum point will sometimes feel safer.

If you wonder why people wait for Oct to have ten or so releases
before upgrading to it,
meaning they are stuck in something that is unsupported by the time
Oct has "proven itself",
this would be one of the reasons. For new clusters, I would not mind
starting with as
late a release as possible.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: how to list ceph file size on ubuntu 20.04

[zxcs]

Thanks a ton!!! Very helps!Thanks,Xiong在 2021年11月17日，上午11:16，胡 玮文  写道：There is a rbytes mount option [1]. Besides, you can use “getfattr -n ceph.dir.rbytes /path/in/cephfs”[1]: https://docs.ceph.com/en/latest/man/8/mount.ceph/#advancedWeiwen Hu在 2021年11月17日，10:26，zxcs  写道：Hi,I want to list cephfs directory size on ubuntu 20.04, but when I use ls -alh [directory] ,it shows the number of files and directorys under this directory(it only count number not size) , i remember when i use ls -alh [directory] on ubuntu 16.04, it will shows the size of this directory (include it sub directory). i know du -sh can list the directory size, but it very slow as our ceph directory has tons of small files.Would anyone please help to shed some light here? Thanks a ton!Thanks,Xiong___ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-le...@ceph.io___ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Adding a RGW realm to a single cephadm-managed ceph cluster

[Eugen Block]

This lab environment runs on:

"ceph version 15.2.14-84-gb6e5642e260  
(b6e5642e2603d3e6bdbc158feff6a51722214e72) octopus (stable)": 3



Zitat von J-P Methot :

Yes, the single realm config works without issue. Same with the rgw  
container. It's with the second realm that everything stops working.  
Tell me, what version of Ceph are you using? We are on 15.2.13 on  
our test environment, as it seems to be the latest Octopus release  
for cephadm containers, despite 15.2.14 and 15.2.15 existing.


On 11/17/21 2:39 AM, Eugen Block wrote:

Hi,

I retried it with your steps, they worked for me.
The first non-default realm runs on the default port 80 on two  
RGWs, the second realm is on the same hosts on port 8081 as  
configured in the spec file. The period commit ran successfully,  
though, so maybe there's something wrong with your setup, it's hard  
to tell.


Does the single realm config work as expected? The rgw container  
starts successfully?



Zitat von J-P Methot :

I've rebuilt my rgw several times since our last emails and every  
time I get stuck more or less to the same point, so let me share  
with you all what I am doing step by step, along with the specs I  
use.


1. I create a default RGW  container and put it on my second  
monitor. Default configs are set


ceph orch apply rgw staging staging --placement="1 ceph-monitor2"

2. I create a default realm

radosgw-admin realm create --rgw-realm=staging

3. I create a second realm

radosgw-admin realm create --rgw-realm=test

4. I create a master zonegroup for my second realm

radosgw-admin zonegroup create --rgw-zonegroup=test  
--rgw-realm=test --realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef


5. I create a master zone for my second realm

radosgw-admin zone create --rgw-zonegroup=test --rgw-realm=test  
--realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef --master  
--rgw-zone=test


6. I try to commit my second realm (it fails, usually with a can't  
init error)


radosgw-admin period update --commit  
--realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef


7.I try to create my container with the following spec file

ceph orch apply -i multi-realm.spec

cat ceph orch apply -i multi-realm.spec

service_type: rgw
service_id: test.test
placement:
   label: rgw-aux
   count: 1
spec:
    rgw_realm: test
    rgw_zone: test
    ssl: false
    rgw_frontend_port: 


I've set Cephadm to output to a logfile and that logfile tells me:

Cannot find zone id=35ea43e5-e0d2-4ebe-81d6-9274231bd542 (name=test)

And that is even though the zone exists and is in the realm this  
gateway is set to serve. Is the commit supposed to work before or  
after the gateway is setup?



On 11/16/21 2:45 AM, Eugen Block wrote:

Hi,


Couldn't init storage provider (RADOS)


I usually see this when my rgw config is wrong, can you share  
your rgw spec(s)?



Zitat von J-P Methot :

After searching through logs and figuring out how cephadm works,  
I've figured out that when cephadm tries to create the new  
systemd service and launch the new RGW container, it fails with  
this error :


Couldn't init storage provider (RADOS)

Systemd then complains that the service doesn't exist :

ceph-04c5d4a4-8815-45fb-b97f-027252d1a...@rgw.test.test.ceph-monitor1.twltpl.service: Main process exited, code=exited,  
status=5/NOTINSTALLED


I can't seem to find further logs anywhere regarding why the  
storage provider fails to init. Googling indicate issues with  
the number of pgs in past versions, but I believe it shouldn't  
be an issue here, since the amount of PG auto-adjusts. How could  
I find out why this happens?


On 11/15/21 11:02 AM, J-P Methot wrote:
Yes, I'm trying to add a RGW container on a second port on the  
same server. For example, I do :


ceph orch apply rgw test test  
--placement="ceph-monitor1:[10.50.47.3:]"


and this results in :

ceph orch ls

NAME RUNNING  REFRESHED  AGE  
PLACEMENT  IMAGE  
NAME   IMAGE ID


rgw.test.test    0/1  2s ago 5s  
ceph-monitor1:[10.50.47.3:] docker.io/ceph/ceph:v15 



the image and container ID being unknown is making me scratch  
my head. A look in the log files show this:


2021-11-15 10:50:12,253 INFO Deploy daemon  
rgw.test.test.ceph-monitor1.rtoiwh ...
2021-11-15 10:50:12,254 DEBUG Running command: /usr/bin/docker  
run --rm --ipc=host --net=host --entrypoint stat -e  
CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e  
NODE_NAME=ceph-monitor1 docker.io/ceph/ceph:v15 -c %u %g  
/var/lib/ceph

2021-11-15 10:50:12,452 DEBUG stat: stdout 167 167
2021-11-15 10:50:12,525 DEBUG Running command: install -d  
-m0770 -o 167 -g 167  
/var/run/ceph/04c5d4a4-8815-45fb-b97f-027252d1aea5

2021-11-15 10:50:12,534 DEBUG Running command: systemctl daemon-reload
2021-11-15 10:50:12,869 DEBUG Running command: systemctl stop  
ceph-04c5d4a4-8815-45fb-b97f-027252d1a...@rgw.test.test.ceph-monitor1.rtoiwh 2021-11-15 10:50:12,879 DEBUG Running command: systemctl 

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Martin Verges]

docker itself is not the problem, it's super nice. It's just that adm/orch
is yet another deployment tool, and yet again not reliable enough. It's
easy to break, and adds additional errors like you can see at my
screenshot. I have a collection of them ;).

We are talking about a storage, meant to store data in a reliable way. Not
for days, but for years or longer. The oldest cluster we maintain runs
since around ~8y by now with completely replaced hardware, filestore to
bluestore migration, lot's of bugs and overall a hell of a ride. But it
never lost data and that's important. However when adding complexity, you
endangering that.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: martin.ver...@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx

Hans van den Bogert  schrieb am Mi., 17. Nov. 2021,
23:34:

> On 11/17/21 8:19 PM, Martin Verges wrote:
> > There are still alternative solutions without the need for useless
> > containers and added complexity. Stay away from that crap and you won't
> > have a hard time. 
>
> I don't have a problem with the containers *at all*. And with me
> probably a lot of users. But those who don't see the problem are silent
> in this thread.
>
> I love cephadm, finally spinning up a cluster and doing upgrades has
> become a lot less tedious.
>
> A positive sound was in due order here.
>
> Signing out again!
>
> Hans
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Annoying MDS_CLIENT_RECALL Warning

[胡 玮文]

Hi all,

We are consistently seeing the MDS_CLIENT_RECALL warning in our cluster, it 
seems harmless, but we cannot get HEALTH_OK, which is annoying.

The clients that are reported failing to respond to cache pressure are 
constantly changing, and most of the time we got 1-5 such clients out of ~20. 
All of the clients are kernel clients, running HWE kernel 5.11 of Ubuntu 20.04. 
The load is pretty low.

We are reading datasets that consist of millions of small files from cephfs, so 
we have tuned some config for performance. Some configs from "ceph config dump" 
that might be relevant:

WHO   LEVEL OPTION   VALUE
  mds basic mds_cache_memory_limit   51539607552
  mds advanced  mds_max_caps_per_client  8388608
  client  basic client_cache_size32768

We also manually pinned almost every directory to either rank 0 or rank 1.

Any thoughts about what causes the warning, or how can we get rid of it?

Thanks,
Weiwen Hu


# ceph -s
  cluster:
id: e88d509a-f6fc-11ea-b25d-a0423f3ac864
health: HEALTH_WARN
4 clients failing to respond to cache pressure

  services:
mon: 5 daemons, quorum gpu024,gpu006,gpu023,gpu013,gpu018 (age 7d)
mgr: gpu014.kwbqcf(active, since 2w), standbys: gpu024.bapbcz
mds: 2/2 daemons up, 2 hot standby
osd: 45 osds: 45 up (since 2h), 45 in (since 5d)
rgw: 2 daemons active (2 hosts, 1 zones)

  data:
volumes: 1/1 healthy
pools:   16 pools, 1713 pgs
objects: 265.84M objects, 55 TiB
usage:   115 TiB used, 93 TiB / 208 TiB avail
pgs: 1711 active+clean
 2active+clean+scrubbing+deep

  io:
client:   55 MiB/s rd, 5.2 MiB/s wr, 513 op/s rd, 14 op/s wr


# ceph fs status
cephfs - 23 clients
==
RANK  STATE   MDS  ACTIVITY DNSINOS   DIRS  
 CAPS
 0active  cephfs.gpu018.ovxvoz  Reqs:  241 /s  17.3M  17.3M  41.3k  
5054k
 1active  cephfs.gpu023.aetiph  Reqs:1 /s  13.1M  12.1M   864k  
 586k
1-s   standby-replay  cephfs.gpu006.ddpekw  Evts:2 /s  2517k  2393k   216k  
   0
0-s   standby-replay  cephfs.gpu024.rpfbnh  Evts:   17 /s  9587k  9587k   214k  
   0
  POOL  TYPE USED  AVAIL
   cephfs.cephfs.meta metadata   126G   350G
   cephfs.cephfs.data   data 102T  25.9T
 cephfs.cephfs.data_ssd data   0525G
cephfs.cephfs.data_mixeddata9.81T   350G
 cephfs.cephfs.data_ec  data 166G  41.4T
MDS version: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) 
pacific (stable)
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Recursive delete hangs on cephfs

[Sasha Litvak]

Gregory,
Thank you for your reply, I do understand that a number of serialized
lookups may take time.  However if 3.25 sec is OK,  11.2 seconds sounds
long, and I had once removed a large subdirectory which took over 20
minutes to complete.  I attempted to use nowsync mount option with kernel
5.15 and it seems to hide latency (i.e. it is almost immediately returns
prompt after recursive directory removal.  However, I am not sure whether
nowsync is safe to use with kernel >= 5.8.  I also have kernel 5.3 on one
of the client clusters and nowsync there is not supported, however all rm
operations happen reasonably fast.  So the second question is, does 5.3's
libceph behave differently on recursing rm compared to 5.4 or 5.8?


On Wed, Nov 17, 2021 at 9:52 AM Gregory Farnum  wrote:

> On Sat, Nov 13, 2021 at 5:25 PM Sasha Litvak
>  wrote:
> >
> > I continued looking into the issue and have no idea what hinders the
> > performance yet. However:
> >
> > 1. A client operating with kernel 5.3.0-42 (ubuntu 18.04) has no such
> > problems.  I delete a directory with hashed subdirs (00 - ff) and total
> > space taken by files ~707MB spread across those 256 in 3.25 s.
>
> Recursive rm first requires the client to get capabilities on the
> files in question, and the MDS to read that data off disk.
> Newly-created directories will be cached, but old ones might not be.
>
> So this might just be the consequence of having to do 256 serialized
> disk lookups on hard drives. 3.25 seconds seems plausible to me.
>
> The number of bytes isn't going to have any impact on how long it
> takes to delete from the client side — that deletion is just marking
> it in the MDS, and then the MDS does the object removals in the
> background.
> -Greg
>
> >
> > 2. A client operating with kernel 5.8.0-53 (ubuntu 20.04) processes a
> > similar directory with less space taken ~ 530 MB spread across 256
> subdirs
> > in 11.2 s.
> >
> > 3.Yet another client with kernel 5.4.156 has similar latency removing
> > directories as in line 2.
> >
> > In all scenarios, mounts are set with the same options, i.e.
> > noatime,secret-file,acl.
> >
> > Client 1 has luminous, client 2 has octopus, client 3 has nautilus.
>  While
> > they are all on the same LAN, ceph -s on 2 and 3 returns in ~ 800 ms and
> on
> > client in ~300 ms.
> >
> > Any ideas are appreciated,
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Nov 12, 2021 at 8:44 PM Sasha Litvak <
> alexander.v.lit...@gmail.com>
> > wrote:
> >
> > > The metadata pool is on the same type of drives as other pools; every
> node
> > > uses SATA SSDs.  They are all read / write mix DC types.  Intel and
> Seagate.
> > >
> > > On Fri, Nov 12, 2021 at 8:02 PM Anthony D'Atri <
> anthony.da...@gmail.com>
> > > wrote:
> > >
> > >> MDS RAM cache vs going to the metadata pool?  What type of drives is
> your
> > >> metadata pool on?
> > >>
> > >> > On Nov 12, 2021, at 5:30 PM, Sasha Litvak <
> alexander.v.lit...@gmail.com>
> > >> wrote:
> > >> >
> > >> > I am running Pacific 16.2.4 cluster and recently noticed that rm -rf
> > >> >  visibly hangs on the old directories.  Cluster is
> healthy,
> > >> has a
> > >> > light load, and any newly created directories deleted immediately
> (well
> > >> rm
> > >> > returns command prompt immediately).  The directories in question
> have
> > >> 10 -
> > >> > 20 small text files so nothing should be slow when removing them.
> > >> >
> > >> > I wonder if someone can please give me a hint on where to start
> > >> > troubleshooting as I see no "big bad bear" yet.
> > >> > ___
> > >> > ceph-users mailing list -- ceph-users@ceph.io
> > >> > To unsubscribe send an email to ceph-users-le...@ceph.io
> > >>
> > >>
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Dan van der Ster]

On Wed, Nov 17, 2021 at 6:10 PM Janne Johansson  wrote:
>
> > * I personally wouldn't want to run an LTS release based on ... what would
> > that be now.. Luminous + security patches??. IMO, the new releases really
> > are much more performant, much more scalable. N, O, and P are really much
> > much *much* better than previous releases. For example, I would not enable
> > snapshots on any cephfs except Pacific -- but do we really want to backport
> > the "stray dir splitting" improvement all the way back to mimic or L? -- to
> > me that seems extremely unwise and a waste of the developers' limited time.
>
> Then again, I am hit by the 13.2.7+ bug
> https://tracker.ceph.com/issues/43259 on one Mimic cluster.
>
> From what I can divine, it is a regression where some other fix or
> feature for rgw caused CopyObject to start failing for S3 clusters
> after 13.2.7.
> No fix or revert was done for mimic. Nor Nautilus. Then those two
> releases "timed out" and aren't considered worth the developers time
> anymore, so the ticket just got bumped to "fix later, and backport to
> Oct and Pac when that is done."

Minutes after you mentioned that bug, Casey pinged the issue and now
we should expect some movement on it. Thanks!

It's important to understand that the backport team doesn't have the
context or mailing list cycles to inherently understand the priority
of each issue on the backport list, so we users need to help make sure
the tracker priorities are accurate.
The priority of that issue is "Normal", and no user has asked to bump
it or pinged it recently, so it was easily skipped over.

I don't think users should be shy to nudge issues that seem to have
been missed for a few point releases, this is almost certainly an
honest misjudgement.

Cheers, Dan

P.S. I'm not sure, but maybe we can add some sort of vote feature to
the tracker to better understand the impact / urgency of the issues...
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 16.2.6 SMP NOPTI - OSD down - Node Exporter Tainted

[Marco Pizzolo]

Hello,

Not sure whether this is perhaps related:
https://bugs.launchpad.net/ubuntu/+source/linux-meta-gcp-5.11/+bug/1948471

Any insight would be appreciated

Thanks,
Marco



On Wed, Nov 17, 2021 at 9:18 AM Marco Pizzolo 
wrote:

> Good day everyone,
>
> This is a bit of a recurring theme for us on a new deployment performed at
> 16.2.6 on Ubuntu 20.04.3 with HWE stack.
>
> We have had good stability over the past 3 weeks or so copying data, and
> we now have about 230M objects (470TB of 1PB used) and we have had 1 OSD
> drop from each of the two OSD hosts currently in this cluster.  The third
> node is a monitor but not an OSD host at this time.  Size is set at 2 for
> now as we're migrating from an older Nautilus cluster.
>
> I'm seeing the following on one host:
>
> [905905.006041] BUG: kernel NULL pointer dereference, address:
> 00c0
> [905905.008536] #PF: supervisor read access in kernel mode
> [905905.02] #PF: error_code(0x) - not-present page
> [905905.013859] PGD 10f1b6067 P4D 10f1b6067 PUD 11bd2e067 PMD 0
> [905905.016678] Oops:  [#1] SMP NOPTI
> [905905.018982] CPU: 86 PID: 69590 Comm: node_exporter Tainted: G
>   OE 5.11.0-38-generic #42~20.04.1-Ubuntu
> [905905.020582] Hardware name: Supermicro SSG-6049P-E1CR60L+/X11DSC+, BIOS
> 3.3 02/21/2020
> [905905.022192] RIP: 0010:blk_mq_put_rq_ref+0xa/0x60
> [905905.023724] Code: 15 0f b6 d3 4c 89 e7 be 01 00 00 00 e8 cf fe ff ff
> 5b 41 5c 5d c3 0f 0b 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 8b 47 10
> <48> 8b 80 c0 00 00 00 48 89 e5 48 3b 78 40 74 1f 4c 8d 87 e8 00 00
> [905905.026794] RSP: 0018:b7713ca3fb08 EFLAGS: 00010202
> [905905.029492] RAX:  RBX: b7713ca3fb90 RCX:
> 0002
> [905905.031813] RDX: 0001 RSI: 0206 RDI:
> 9ea4f46ce400
> [905905.034098] RBP: b7713ca3fb40 R08:  R09:
> 0005
> [905905.036360] R10: 0825 R11: 000b R12:
> 9ea4f46ce400
> [905905.038564] R13: 9ea4f426ec00 R14:  R15:
> 0001
> [905905.040345] FS:  00c000507910() GS:9f033fb8()
> knlGS:
> [905905.042034] CS:  0010 DS:  ES:  CR0: 80050033
> [905905.043660] CR2: 00c0 CR3: 00010d2bc001 CR4:
> 007706e0
> [905905.045298] DR0:  DR1:  DR2:
> 
> [905905.046883] DR3:  DR6: fffe0ff0 DR7:
> 0400
> [905905.048424] PKRU: 5554
> [905905.049979] Call Trace:
> [905905.051907]  ? bt_iter+0x54/0x90
> [905905.053815]  blk_mq_queue_tag_busy_iter+0x18b/0x2d0
> [905905.055730]  ? blk_mq_hctx_mark_pending+0x70/0x70
> [905905.057711]  ? blk_mq_hctx_mark_pending+0x70/0x70
> [905905.059548]  blk_mq_in_flight+0x38/0x60
> [905905.061409]  diskstats_show+0x75/0x2b0
> [905905.063166]  seq_read_iter+0x2a3/0x450
> [905905.064871]  proc_reg_read_iter+0x5e/0x80
> [905905.066648]  new_sync_read+0x110/0x1a0
> [905905.068276]  vfs_read+0x154/0x1b0
> [905905.069879]  ksys_read+0x67/0xe0
> [905905.071436]  __x64_sys_read+0x1a/0x20
> [905905.072934]  do_syscall_64+0x38/0x90
> [905905.074409]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [905905.075841] RIP: 0033:0x4a5c20
> [905905.077230] Code: 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 49 c7 c2
> 00 00 00 00 49 c7 c0 00 00 00 00 49 c7 c1 00 00 00 00 48 8b 44 24 08 0f 05
> <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
> [905905.080065] RSP: 002b:00c00016e8b8 EFLAGS: 0212 ORIG_RAX:
> 
> [905905.081441] RAX: ffda RBX: 00c30a00 RCX:
> 004a5c20
> [905905.082878] RDX: 1000 RSI: 00c00062c000 RDI:
> 0006
> [905905.084299] RBP: 00c00016e908 R08:  R09:
> 
> [905905.085673] R10:  R11: 0212 R12:
> 0040
> [905905.087018] R13: 0040 R14: 00b6420c R15:
> 
> [905905.088305] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos
> jfs xfs cpuid binfmt_misc overlay cuse bonding rdma_ucm(OE) rdma_cm(OE)
> iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) nls_iso8859_1 dm_multipath
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr intel_rapl_common
> isst_if_common skx_edac nfit x86_pkg_temp_thermal coretemp kvm_intel kvm
> rapl ipmi_ssif intel_cstate mlx5_ib(OE) ib_uverbs(OE) ib_core(OE)
> efi_pstore input_leds joydev intel_pch_thermal mei_me mei ioatdma acpi_ipmi
> ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter mac_hid sch_fq_codel
> msr ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456
> async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
> libcrc32c raid0 multipath linear hid_generic usbhid hid ses enclosure
> scsi_transport_sas raid1 mlx5_core(OE) crct10dif_pclmul ast drm_vram_helper
> i2c_algo_bit crc32_pclmul drm_ttm_helper ttm ghash_clmulni_intel
> aesni_intel drm_kms_helper 

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Marc]

> > And it looks like I'll have to accept the move to containers even
> though I have serious concerns about operational maintainability due to
> the inherent opaqueness of container solutions.
> 
> There are still alternative solutions without the need for useless
> containers and added complexity. Stay away from that crap and you won't
> have a hard time. 
> 
   
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Martin Verges]

> And it looks like I'll have to accept the move to containers even though
I have serious concerns about operational maintainability due to the
inherent opaqueness of container solutions.

There are still alternative solutions without the need for useless
containers and added complexity. Stay away from that crap and you won't
have a hard time. 

We as croit started our own OBS Infrastructure to build packages for x86_64
and arm64. This should help us to maintain packages and avoid the useless
Ceph containers. I can post an update to the user ML when it's ready for
public service.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: martin.ver...@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx

Dave Hall  schrieb am Mi., 17. Nov. 2021, 20:05:

>
> On Wed, Nov 17, 2021 at 1:05 PM Martin Verges 
> wrote:
>
>> Hello Dave,
>>
>> > The potential to lose or lose access to millions of files/objects or
>> petabytes of data is enough to keep you up at night.
>> > Many of us out here have become critically dependent on Ceph storage,
>> and probably most of us can barely afford our production clusters, much
>> less a test cluster.
>>
>> Please remember, free software comes still with a price. You can not
>> expect someone to work on your individual problem while being cheap on your
>> highly critical data. If your data has value, then you should invest in
>> ensuring data safety. There are companies out, paying Ceph developers and
>> fixing bugs, so your problem will be gone as soon as you A) contribute code
>> yourself or B) pay someone to contribute code.
>>
>
> It's always tricky when one gets edgy.  I completely agree with your
> statements on free software.
>
> For the record, I don't actually have any Ceph problems right now.  It's
> been pretty smooth sailing since I first set the cluster up (Nautilus on
> Debian with Ceph-Ansible).  Some procedural confusion, but no outages in 18
> months  and expansion from 3 nodes to 12.
>
> So it's not about my pet bug or feature request.  What it is about is
> exactly the undeniable and unavoidable dilemmas of distributed open source
> development.  Ceph is wonderful, but it is incredibly complex all on it's
> own.  It wouldn't be easy even if all of the developers were sitting in the
> same building working for the same company.
>
> Further explanation:  Our Ceph cluster is entirely funded by research
> grants.  We can't just go out and buy a whole second cluster for data
> safety.  We can't go to management and ask for more systems.  We can't even
> get enough paid admins to do what we need to do.  But we also can't allow
> these limitations to impede useful research activities.  So unpaid overtime
> and shoe-string hardware budgets.
>
> We (myself and the researcher I'm supporting) chose Ceph because it is
> readily scalable and because it has redundancy and resiliency built in in
> the form of configurable failure domains, replication and EC pools.  I've
> looked at a lot of the distributed storage solutions out there.  Most,
> including the commercial offerings, don't even come close to Ceph on these
> points.
>
>
>> Don't get me wrong, every dev here should have the focus in providing
>> rock solid work and I believe they do, but in the end it's software, and
>> software never will be free of bugs. Ceph does quite a good job protecting
>> your data, and in my personal experience, if you don't do crazy stuff and
>> execute even crazier commands with "yes-i-really-mean-it", you usually
>> don't lose data.
>>
>
> I believe you that there are a lot of good devs out there doing good
> work.  Complexity is the biggest issue Ceph faces.  This complexity is
> necessary, but it can bite you.
>
> My honest perception of Pacific right now is that something dreadful could
> go wrong in the course of an upgrade to Pacific, even with a sensible
> cluster and a sensible cluster admin.  I wish I could pull some scrap
> hardware together and play out some scenarios, but I don't have the time or
> the hardware.
>
> To be completely straight, I am speaking up today because I want to see
> Ceph succeed and it seems like things are a bit rough right now.  In my
> decades in the business I've seen projects and even whole companies
> collapse because the developers lost touch with, or stopped listening to,
> the users.  I don't want that to happen to Ceph.
>
>
>>
>>
> > The real point here:  From what I'm reading in this mailing list it
>> appears that most non-developers are currently afraid to risk an upgrade to
>> Octopus or Pacific.  If this is an accurate perception then THIS IS THE
>> ONLY PROBLEM.
>>
>> Octopus is one of the best releases ever. Often our support engineers do
>> upgrade old unmaintained installations from some super old release to
>> Octopus to get them running again or have 

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Marc]

> 
> a good choice. It lacks RBD encryption and read leases. But for us
> upgrading from N to O or P is currently not
> 

what about just using osd encryption with N?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Martin Verges]

Hello Dave,

> The potential to lose or lose access to millions of files/objects or
petabytes of data is enough to keep you up at night.
> Many of us out here have become critically dependent on Ceph storage, and
probably most of us can barely afford our production clusters, much less a
test cluster.

Please remember, free software comes still with a price. You can not expect
someone to work on your individual problem while being cheap on your highly
critical data. If your data has value, then you should invest in ensuring
data safety. There are companies out, paying Ceph developers and fixing
bugs, so your problem will be gone as soon as you A) contribute code
yourself or B) pay someone to contribute code.

Don't get me wrong, every dev here should have the focus in providing rock
solid work and I believe they do, but in the end it's software, and
software never will be free of bugs. Ceph does quite a good job protecting
your data, and in my personal experience, if you don't do crazy stuff and
execute even crazier commands with "yes-i-really-mean-it", you usually
don't lose data.

> The real point here:  From what I'm reading in this mailing list it
appears that most non-developers are currently afraid to risk an upgrade to
Octopus or Pacific.  If this is an accurate perception then THIS IS THE
ONLY PROBLEM.

Octopus is one of the best releases ever. Often our support engineers do
upgrade old unmaintained installations from some super old release to
Octopus to get them running again or have propper tooling to fix the issue.
But I agree, we as croit are still afraid of pushing our users to Pacific,
as we encounter bugs in our tests. This however will change soon, as we are
close to a stable enough Pacific release as we believe.

--
Martin Verges
Managing director

Mobile: +49 174 9335695  | Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx


On Wed, 17 Nov 2021 at 18:41, Dave Hall  wrote:

> Sorry to be a bit edgy, but...
>
> So at least 5 customers that you know of have a test cluster, or do you
> have 5 test clusters?  So 5 test clusters out of how many total Ceph
> clusters worldwide.
>
> Answers like this miss the point.  Ceph is an amazing concept.  That it is
> Open Source makes it more amazing by 10x.  But storage is big, like
> glaciers and tectonic plates.  The potential to lose or lose access to
> millions of files/objects or petabytes of data is enough to keep you up at
> night.
>
> Many of us out here have become critically dependent on Ceph storage, and
> probably most of us can barely afford our production clusters, much less a
> test cluster.
>
> The best I could do right now today for a test cluster would be 3
> Virtualbox VMs with about 10GB of disk each.  Does anybody out there think
> I could find my way past some of the more gnarly O and P issues with this
> as my test cluster?
>
> The real point here:  From what I'm reading in this mailing list it appears
> that most non-developers are currently afraid to risk an upgrade to Octopus
> or Pacific.  If this is an accurate perception then THIS IS THE ONLY
> PROBLEM.
>
> Don't shame the users who are more concerned about stability than fresh
> paint.
>
> -Dave
>
> --
> Dave Hall
> Binghamton University
> kdh...@binghamton.edu
>
> On Wed, Nov 17, 2021 at 11:18 AM Stefan Kooman  wrote:
>
> > On 11/17/21 16:19, Marc wrote:
> > >> The CLT is discussing a more feasible alternative to LTS, namely to
> > >> publish an RC for each point release and involve the user community to
> > >> help test it.
> > >
> > > How many users even have the availability of a 'test cluster'?
> >
> > At least 5 (one physical 3 node). We installed a few of them with the
> > exact same version as when we started prod (luminous 12.2.4 IIRC) and
> > upgraded ever since. Especially for cases where old pieces of metadata
> > might cause issues in the long run (pre jewel blows up in pacific for
> > MDS case). Same for the osd OMAP conversion troubles in pacific.
> > Especially in these cases Ceph testing on real prod might have revealed
> > that. A VM enviroment would be ideal for this. As you could just
> > snapshot state and play back when needed. Ideally MDS / RGW / RBD
> > workloads on them to make sure all use cases are tested.
> >
> > But these cluster have not the same load as prod. Not the same data ...
> > so still stuff might break in special ways. But at least we try to avoid
> > that as much as possible.
> >
> > Gr. Stefan
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Peter Lieven]

Am 17.11.21 um 18:09 schrieb Janne Johansson:
>> * I personally wouldn't want to run an LTS release based on ... what would
>> that be now.. Luminous + security patches??. IMO, the new releases really
>> are much more performant, much more scalable. N, O, and P are really much
>> much *much* better than previous releases. For example, I would not enable
>> snapshots on any cephfs except Pacific -- but do we really want to backport
>> the "stray dir splitting" improvement all the way back to mimic or L? -- to
>> me that seems extremely unwise and a waste of the developers' limited time.
> Then again, I am hit by the 13.2.7+ bug
> https://tracker.ceph.com/issues/43259 on one Mimic cluster.
>
> From what I can divine, it is a regression where some other fix or
> feature for rgw caused CopyObject to start failing for S3 clusters
> after 13.2.7.
> No fix or revert was done for mimic. Nor Nautilus. Then those two
> releases "timed out" and aren't considered worth the developers time
> anymore, so the ticket just got bumped to "fix later, and backport to
> Oct and Pac when that is done."
>
> So, while I would have liked to simply upgrade to N,O,P to fix this,
> none of them has the fix currently, so upgrading the cluster will not
> help me, and the N->O bump will incur one of those "an hour or more
> per hdd OSD" conversions which are kind of sad to plan for, and as we
> know the Pac release is currently not a fun place to upgrade into
> either.
>
> In my dream world, an LTS would not have seen a feature break another,
> and if it wasn't a feature but some fix for something else then
> someone would have to make a decision if it is ok to cause one bug to
> fix another and revert the fix if it wasn't ok to break old features.
> As is it now, a third option was possible, to let it slip and not-fix
> it at all and just tag it for later because so much else has changed
> by now. It got "fixed" in master after almost two years but the
> backports to O+P has been stale for 4 months, which means I will have
> to make several jumps and then move to Q early on when it arrives in
> order to get back to the working setup I had at 13.2.6..
>
> This is at least my reason for wanting something like an LTS version.
>
>
Thanks for all your feedback. I hope we can all meet tomorrow in the Ceph Dev 
User meeting!


I have to say, that we use ceph only for RBD VM storage at the moment. For that 
application Nautilus seems to be

a good choice. It lacks RBD encryption and read leases. But for us upgrading 
from N to O or P is currently not

possible from an operational point of view. Mainly because of the waiting for 
readable issue when an OSD restarts.

This was the main reason to start with N and not O back 1 year ago.

So I can live with the lack of RBD encryption and just do it in the hypervisor.


N has no issue when a large numer of OSDs going up/down, rebalancing in the 
order of serveral tens of GB/s has nearly no

client impact if the wrong default for osd_op_queue_cut_off is corrected.


I do not know if it works as smooth under O or P once the laggy PG issue is 
fixed, but I'm afraid to test it.

I have a small dev environment, but that does not reflect production use 
patterns, I/O loads and size.


I'm seriously afraid to find out that it does not work as smooth after having 
upgraded to O or P with no way back.

The main reason is that we do not have e.g. an S3 gateway that is just slow for 
some time. I have several thousand VMs that

might have serious issues if there are I/O stalls or slow I/O for more than 
just a few (1-2) seconds.


We already had to find out that bug fixes where not backported even when N was 
still maintained, so we already use the

latest N release with very few additional fixes. If the people longing for an 
LTS release mainly are those who are

using Ceph as VM Storage, we could use this as a basis.


Thanks,

Peter


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Janne Johansson]

> * I personally wouldn't want to run an LTS release based on ... what would
> that be now.. Luminous + security patches??. IMO, the new releases really
> are much more performant, much more scalable. N, O, and P are really much
> much *much* better than previous releases. For example, I would not enable
> snapshots on any cephfs except Pacific -- but do we really want to backport
> the "stray dir splitting" improvement all the way back to mimic or L? -- to
> me that seems extremely unwise and a waste of the developers' limited time.

Then again, I am hit by the 13.2.7+ bug
https://tracker.ceph.com/issues/43259 on one Mimic cluster.

>From what I can divine, it is a regression where some other fix or
feature for rgw caused CopyObject to start failing for S3 clusters
after 13.2.7.
No fix or revert was done for mimic. Nor Nautilus. Then those two
releases "timed out" and aren't considered worth the developers time
anymore, so the ticket just got bumped to "fix later, and backport to
Oct and Pac when that is done."

So, while I would have liked to simply upgrade to N,O,P to fix this,
none of them has the fix currently, so upgrading the cluster will not
help me, and the N->O bump will incur one of those "an hour or more
per hdd OSD" conversions which are kind of sad to plan for, and as we
know the Pac release is currently not a fun place to upgrade into
either.

In my dream world, an LTS would not have seen a feature break another,
and if it wasn't a feature but some fix for something else then
someone would have to make a decision if it is ok to cause one bug to
fix another and revert the fix if it wasn't ok to break old features.
As is it now, a third option was possible, to let it slip and not-fix
it at all and just tag it for later because so much else has changed
by now. It got "fixed" in master after almost two years but the
backports to O+P has been stale for 4 months, which means I will have
to make several jumps and then move to Q early on when it arrives in
order to get back to the working setup I had at 13.2.6..

This is at least my reason for wanting something like an LTS version.


-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Multiple osd/disk

[Janne Johansson]

Den ons 17 nov. 2021 kl 17:16 skrev Szabo, Istvan (Agoda)
:
>
> Hi,
>
> I’m curious when you put more osd/nvme or ssd how you calculate the pg?
> Are you still consider each osd as a separate disk so in case of 4osd/nvme on 
> your disk actually you hold ~400pg?
> Or just have 4osd/nvme but you don’t go higher than ~100pg/disk so around 
> 25-30/osd?

If we have 4xOSD on one nvme, it gets ~100 pgs per OSD.
(which means more cores per disk to keep same "cores per OSD" ratio)

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs removing multiple snapshots

[Francois Legrand]

No, our osds are hdd (no ssd) and we have everything (data and metadata) 
on them (no nvme).


Le 17/11/2021 à 16:49, Arthur Outhenin-Chalandre a écrit :

Hi,

On 11/17/21 16:09, Francois Legrand wrote:
Now we are investingating this snapshot issue and I noticed that as 
long as we remove one snapshot alone, things seems to go well (only 
some pgs in "unknown state" but no global warning nor slow ops, osd 
down or crash). But if we remove several snapshots at the same time 
(I tryed with 2 for the moment), then we start to have some slow ops. 
I guess that if I remove 4 or 5 snapshots at the same time I will end 
with osds marked down and/or crash as we had just after the upgrade 
(I am not sure I want to try that with our production cluster).


Maybe you want to try to tweak `osd_snap_trim_sleep`. On 
Octopus/Pacific with hybrid OSDs the snapshots deletions seems pretty 
stable in our testing. Out of curiosity are your OSD on SSD? I suspect 
that the default setting of `osd_snap_trim_sleep` for SSD OSD could 
affect performance [1].


Cheers,

[1]: 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/FPRB2DW4N427U25LEHYICOKI4C37BKSO/



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Marc]

> 
> > Yeah, generally there is no much enthusiasm about supporting that
> among developers.
> 
> I guess its because none of them is administrating any large production
> installation 

Exactly!

> The actual implied upgrade period is every 2 years and every
> 4 years as an exception. For storage this is extremely fast, 

I am really surprised about the need to even having to mention this here. I am 
even wondering if the current 'development path' is able to keep binding 
organizations like CERN to use ceph with >3000 hdds.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Marc]

> features" per se -- one which comes to mind is the fix related to
> detecting
> network binding addrs, for example -- something that would reasonably
> have
> landed in and broken LTS clusters.)
> * I personally wouldn't want to run an LTS release based on ... what
> would
> that be now.. Luminous + security patches??. IMO, the new releases
> really
> are much more performant, much more scalable. N, O, and P are really
> much

Please do take into account that for some scalability and performance is not a 
reason to upgrade. If this is fine now.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Matthew Vernon]

On 17/11/2021 15:19, Marc wrote:

The CLT is discussing a more feasible alternative to LTS, namely to
publish an RC for each point release and involve the user community to
help test it.


How many users even have the availability of a 'test cluster'?


The Sanger has one (3 hosts), which was a real boon.

Regards,

Matthew
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Recursive delete hangs on cephfs

[Gregory Farnum]

On Sat, Nov 13, 2021 at 5:25 PM Sasha Litvak
 wrote:
>
> I continued looking into the issue and have no idea what hinders the
> performance yet. However:
>
> 1. A client operating with kernel 5.3.0-42 (ubuntu 18.04) has no such
> problems.  I delete a directory with hashed subdirs (00 - ff) and total
> space taken by files ~707MB spread across those 256 in 3.25 s.

Recursive rm first requires the client to get capabilities on the
files in question, and the MDS to read that data off disk.
Newly-created directories will be cached, but old ones might not be.

So this might just be the consequence of having to do 256 serialized
disk lookups on hard drives. 3.25 seconds seems plausible to me.

The number of bytes isn't going to have any impact on how long it
takes to delete from the client side — that deletion is just marking
it in the MDS, and then the MDS does the object removals in the
background.
-Greg

>
> 2. A client operating with kernel 5.8.0-53 (ubuntu 20.04) processes a
> similar directory with less space taken ~ 530 MB spread across 256 subdirs
> in 11.2 s.
>
> 3.Yet another client with kernel 5.4.156 has similar latency removing
> directories as in line 2.
>
> In all scenarios, mounts are set with the same options, i.e.
> noatime,secret-file,acl.
>
> Client 1 has luminous, client 2 has octopus, client 3 has nautilus.   While
> they are all on the same LAN, ceph -s on 2 and 3 returns in ~ 800 ms and on
> client in ~300 ms.
>
> Any ideas are appreciated,
>
>
>
>
>
>
>
>
>
> On Fri, Nov 12, 2021 at 8:44 PM Sasha Litvak 
> wrote:
>
> > The metadata pool is on the same type of drives as other pools; every node
> > uses SATA SSDs.  They are all read / write mix DC types.  Intel and Seagate.
> >
> > On Fri, Nov 12, 2021 at 8:02 PM Anthony D'Atri 
> > wrote:
> >
> >> MDS RAM cache vs going to the metadata pool?  What type of drives is your
> >> metadata pool on?
> >>
> >> > On Nov 12, 2021, at 5:30 PM, Sasha Litvak 
> >> wrote:
> >> >
> >> > I am running Pacific 16.2.4 cluster and recently noticed that rm -rf
> >> >  visibly hangs on the old directories.  Cluster is healthy,
> >> has a
> >> > light load, and any newly created directories deleted immediately (well
> >> rm
> >> > returns command prompt immediately).  The directories in question have
> >> 10 -
> >> > 20 small text files so nothing should be slow when removing them.
> >> >
> >> > I wonder if someone can please give me a hint on where to start
> >> > troubleshooting as I see no "big bad bear" yet.
> >> > ___
> >> > ceph-users mailing list -- ceph-users@ceph.io
> >> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs removing multiple snapshots

[Arthur Outhenin-Chalandre]

Hi,

On 11/17/21 16:09, Francois Legrand wrote:
Now we are investingating this snapshot issue and I noticed that as long 
as we remove one snapshot alone, things seems to go well (only some pgs 
in "unknown state" but no global warning nor slow ops, osd down or 
crash). But if we remove several snapshots at the same time (I tryed 
with 2 for the moment), then we start to have some slow ops. I guess 
that if I remove 4 or 5 snapshots at the same time I will end with osds 
marked down and/or crash as we had just after the upgrade (I am not sure 
I want to try that with our production cluster).


Maybe you want to try to tweak `osd_snap_trim_sleep`. On Octopus/Pacific 
with hybrid OSDs the snapshots deletions seems pretty stable in our 
testing. Out of curiosity are your OSD on SSD? I suspect that the 
default setting of `osd_snap_trim_sleep` for SSD OSD could affect 
performance [1].


Cheers,

[1]: 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/FPRB2DW4N427U25LEHYICOKI4C37BKSO/


--
Arthur Outhenin-Chalandre
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Marc]

> 
> The demand for LTS - at least in our case - does not stem from
> unprofessionalism or biased opinion.
> It's the desire to stay up to date on security patches as much as
> possible while maintaining a well tested and stable environment.

Is this not the definition of Long Term Stable? ;)

> Both Pacific and Octopus (we’re currently on Nautilus) have some
> problems within themselves that made us not upgrading our cluster.

Same here.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Marc]

> The CLT is discussing a more feasible alternative to LTS, namely to
> publish an RC for each point release and involve the user community to
> help test it.

How many users even have the availability of a 'test cluster'? 



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm / ceph orch : indefinite hang adding hosts to new cluster

[Lincoln Bryant]

Hi,

Yes, the hosts have internet access and other Ceph commands work successfully. 
Every host we have tried has worked for bootstrap, but adding another node to 
the cluster isn't working. We've also tried adding intentionally bad hosts and 
get expected failures (missing SSH key, etc).

Here's some check-host output for our mons:

[root@kvm-mon03 ~]# cephadm check-host
podman|docker (/usr/bin/podman) is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[root@kvm-mon02 ~]#  cephadm check-host
podman|docker (/usr/bin/podman) is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[root@kvm-mon01 ~]# cephadm check-host
podman (/usr/bin/podman) version 3.4.1 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK

Tailing the logs, journalctl simply reports:

 sshd[8135]: Accepted publickey for root from 192.168.7.13 port 41378 ssh2: RSA 
SHA256:JZxTh1Su9A+cqx14cIxzbP2W0vRHwgNcGQioLPCMFtk
 systemd-logind[2155]: New session 17 of user root.
 systemd[1]: Started Session 17 of user root.
 sshd[8135]: pam_unix(sshd:session): session opened for user root by (uid=0)

Very strange...

Maybe a manual installation will reveal issues?

--Lincoln

From: Eugen Block 
Sent: Wednesday, November 17, 2021 2:27 AM
To: ceph-users@ceph.io 
Subject: [ceph-users] Re: cephadm / ceph orch : indefinite hang adding hosts to 
new cluster

Hi,

> This is the only logging output we see:
> # cephadm shell -- ceph orch host add kvm-mon02 192.168.7.12
> Inferring fsid 826b9b36-4729-11ec-99f0-c81f66d05a38
> Using recent ceph image
> quay.io/ceph/ceph@sha256:a2c23b6942f7fbc1e15d8cfacd6655a681fe0e44f288e4a158db22030b8d58e3
>
> This command hangs indefinitely until killed via podman kill​.

the first thought coming to mind is, do the hosts have internet access
to download the container images? But if the bootstrap worked the
answer would be yes. And IIUC you tried different hosts for bootstrap
and all of them worked? Just for the record, a manual 'podman pull
...' on the second host works, too?

What does a 'cephadm check-host' on the second host report?
The syslog on the second node should usually reveal errors, have you
checked 'journalctl -f' during the attempt to add it?


Zitat von Lincoln Bryant :

> Greetings list,
>
> We have a new Ceph cluster we are trying to deploy on EL8 (CentOS
> Stream) using cephadm (+podman), targeting Pacific.
>
> We are successfully able to bootstrap the first host, but attempting
> to add any additional hosts hangs indefinitely. We have confirmed
> that we are able to SSH from the first host to subsequent hosts
> using the key generated by Ceph.
>
> This is the only logging output we see:
> # cephadm shell -- ceph orch host add kvm-mon02 192.168.7.12
> Inferring fsid 826b9b36-4729-11ec-99f0-c81f66d05a38
> Using recent ceph image
> quay.io/ceph/ceph@sha256:a2c23b6942f7fbc1e15d8cfacd6655a681fe0e44f288e4a158db22030b8d58e3
>
> This command hangs indefinitely until killed via podman kill​.
>
> Inspecting the host we're trying to add, we see that Ceph has
> launched a python process:
> root3604  0.0  0.0 164128  6316 ?S16:36   0:00
> |   \_ sshd: root@notty
> root3605  0.0  0.0  31976  8752 ?Ss   16:36   0:00
> |   \_ python3 -c import sys;exec(eval(sys.stdin.readline()))
>
> Inside of the mgr container, we see 2 SSH connections:
> ceph 186  0.0  0.0  44076  6676 ?S22:31   0:00
> \_ ssh -C -F /tmp/cephadm-conf-s0b8c90d -i
> /tmp/cephadm-identity-8ku7ib6b root@192.168.7.13 python3 -c "import
> sys;exec(eval(sys.stdin.readline()))"
> ceph 211  0.0  0.0  44076  6716 ?S22:36   0:00
> \_ ssh -C -F /tmp/cephadm-conf-s0b8c90d -i
> /tmp/cephadm-identity-8ku7ib6b root@192.168.7.12 python3 -c "import
> sys;exec(eval(sys.stdin.readline()))"
>
> where 192.168.1.13 is the IP of the first host in the cluster (which
> has succesfully bootstrapped and is running mgr, mon, and so on),
> and 196.168.1.12 is the host we are trying to unsuccessfully add.
>
> The mgr logs show no particularly interesting except for:
> debug 2021-11-16T22:39:03.570+ 7fb6e4914700  0 [progress WARNING
> root] complete: ev de058df7-b54a-4429-933a-99abe7796715 does not exist
> debug 2021-11-16T22:39:03.571+ 7fb6e4914700  0 [progress WARNING
> root] complete: ev 61fe4998-4ef4-4640-8a13-2b0928da737f does not exist
> debug 2021-11-16T22:39:03.571+ 7fb6e4914700  0 [progress WARNING
> root] complete: ev 2323b586-5262-497e-b318-42702c0dc3dc does not exist
> debug 2021-11-16T22:39:03.572+ 7fb6e4914700  0 [progress WARNING
> root] complete: ev f882757b-e03a-4798-8fae-4d55e2d1f531 does not exist
> debug 2021-11-16T22:39:03.572+ 7fb6e4914700  0 [progress WARNING
> root] complete: ev 3d35e91f-b475-4dbc-a040-65658e82fe67 does not exist
> debug 2021-11-16T22:39:03.572+ 

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Martin Verges]

Just as a friendly reminder:

1) No one prevents you from hiring developers to work on Ceph in a way you
like.
2) I personally dislike the current release cycle and would like change
that a bit.
3) There is a reason companies like our own prevent users from using latest
as "production", we tag them as nightly and not roll them out to most
environments until we or others found enough bugs or made sure that release
is stable enough for production. This often came with us ignoring releases
and still not setting pacific to default for new installations for a very
long time. So still every new croit deployment by default (without changing
it) is still on octopus instead of pacific and as you see on the changelog
for a good reason.

I would strongly suggest to not run something like a luminous or minic
release anymore, it's old but it also lacks a lot of debugging
functionality that makes it hard for support teams like ours to help users
fixing a cluster. Having 5-10y LTS is therefore something I would not
recommend and it's a requirement from the past in my personal opinion.
However a timespan of 3-5y is something the Ceph community should try to do
and we from croit would happily support that.

--
Martin Verges
Managing director

Mobile: +49 174 9335695  | Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx


On Wed, 17 Nov 2021 at 16:00, Marc  wrote:

> > >
> > > But since when do developers decide? Do you know any factory where
> > factory workers decide what product they are going to make and not the
> > product management???
> >
> > You might want to check out [1] and [2]. There are different
> > stakeholders with different interests. All these parties have the
> > possibility to influence the project, on all different levels and areas.
> > So it's not product management.
> >
> > IT is becoming such a refuge for undetected unprofessionals.
> >
> > So who do you target here? The ceph devs that try to make the product
> > better every day?
> >
>
> The target is, decision making based on "I do not want to need implement
> something, I want to move to a new version of python where there is a
> default library I can include" or "I need this functionality but there is
> no package available for centos7 so lets drop centos7 and only support
> centos8" or "but it works on my macos development environment" etc etc.
>
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephfs removing multiple snapshots

[Francois Legrand]

Hello,

We recently upgraded our ceph+cephfs cluster from nautilus to octopus.

After the upgrade, we noticed that removal of snapshots was causing a 
lot of problems (lot of slow ops, osd marked down, crashs etc...) so we 
suspended the snapshots for a while so the cluster get stable again for 
more than one week now. We had not these problems under nautilus.


Now we are investingating this snapshot issue and I noticed that as long 
as we remove one snapshot alone, things seems to go well (only some pgs 
in "unknown state" but no global warning nor slow ops, osd down or 
crash). But if we remove several snapshots at the same time (I tryed 
with 2 for the moment), then we start to have some slow ops. I guess 
that if I remove 4 or 5 snapshots at the same time I will end with osds 
marked down and/or crash as we had just after the upgrade (I am not sure 
I want to try that with our production cluster).


So my questions are does someone have noticed this king of problem, does 
the snapshot management have changed between nautilus and octopus, is 
there a way to solve it (apart from removing one snap at a time and 
waiting for the snaptrim to end before removing the next one) ?


We also change the bluefs_buffered_io from false to true (it was set to 
false a long time ago because of the bug 
https://tracker.ceph.com/issues/45337) because it seems that it can help 
(cf. 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/S4ZW7D5J5OAI76F44NNXMTKWNZYYYUJY/). 
Does the osds need to be restarted to make this change effective ?



Thanks.

F.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Marc]

> >
> > But since when do developers decide? Do you know any factory where
> factory workers decide what product they are going to make and not the
> product management???
> 
> You might want to check out [1] and [2]. There are different
> stakeholders with different interests. All these parties have the
> possibility to influence the project, on all different levels and areas.
> So it's not product management.
> 
> IT is becoming such a refuge for undetected unprofessionals.
> 
> So who do you target here? The ceph devs that try to make the product
> better every day?
> 

The target is, decision making based on "I do not want to need implement 
something, I want to move to a new version of python where there is a default 
library I can include" or "I need this functionality but there is no package 
available for centos7 so lets drop centos7 and only support centos8" or "but it 
works on my macos development environment" etc etc.



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Dan van der Ster]

My 2 cents:
* the best solution to ensure ceph's future rock solid stability is to
continually improve our upstream testing. We have excellent unit testing to
avoid regressions on specific bugs, and pretty adequate upgrade testing,
but I'd like to know if we're missing some high level major upgrade testing
that would have caught these issues and other unknown bugs.
* LTS alone wouldn't solve the root problem. Bugs can creep into LTS in the
exact same way that the recent pacific bugs have, if the testing coverage
is incomplete.
(I'm not convinced that all of the recent urgent bugs have come from "new
features" per se -- one which comes to mind is the fix related to detecting
network binding addrs, for example -- something that would reasonably have
landed in and broken LTS clusters.)
* I personally wouldn't want to run an LTS release based on ... what would
that be now.. Luminous + security patches??. IMO, the new releases really
are much more performant, much more scalable. N, O, and P are really much
much *much* better than previous releases. For example, I would not enable
snapshots on any cephfs except Pacific -- but do we really want to backport
the "stray dir splitting" improvement all the way back to mimic or L? -- to
me that seems extremely unwise and a waste of the developers' limited time.

So I would prioritize a short one off effort to review the upstream
testing, ensuring it is as complete and representative of our real user
environments as possible.
And *also* we can complement this with RC point releases, whereby we invite
the community members to participate in the testing and give feedback
before a point release would have broken.

Why RCs? Because our environments are so diverse, complete test coverage
will always be a challenge. So IMHO we need to help each other build
confidence in the latest stable releases. We should regularly share upgrade
stories on this list so it is very clear which use-cases are working or not
for each release.
We can also do things like pull broken releases from repos, clearly
document known issues in old releases, even add health warns to clusters
with known issues, ...

And we should be asking and understanding why some of us are still on
mimic (or nautilus, which I know personally why...)? What would convince us
to upgrade to O or P ?
That would be a good metric, IMHO -- let's say, 4 months from now, who is
not running pacific?? Why not??

Cheers, Dan


On Wed, Nov 17, 2021 at 2:40 PM Daniel Tönnißen  wrote:

> The demand for LTS - at least in our case - does not stem from
> unprofessionalism or biased opinion.
> It's the desire to stay up to date on security patches as much as possible
> while maintaining a well tested and stable environment.
>
> Both Pacific and Octopus (we’re currently on Nautilus) have some problems
> within themselves that made us not upgrading our cluster.
> The latest releases felt kind of rushed out rather than well tested.
>
> We (the LTS „faction“) still would like to keep up with security and minor
> patches on the cluster. We’re not talking about backporting new features.
>
> There’s a saying: „Never change a running system“
>
> --
> Mit freundlichen Grüßen aus Oberhausen
> *Daniel Tönnißen*
> (Systemadministrator)
> [image: Logo KAMP Netzwerkdienste GmbH]  [image:
> KAMP ist Zertifiziert nach ISO27001, DIN9001 und EC-S]
> 
> *KAMP Netzwerkdienste GmbH*
> Vestische Str. 89−91 | 46117 Oberhausen
> Fon: +49 (0) 208.89 402-50 <+492088940250>
> Fax: +49 (0) 208.89 402-40
> E-Mail: d...@kamp.de
> WWW: https://www.kamp.de
> Geschäftsführer: Heiner Lante | Michael Lante | Amtsgericht Duisburg | HRB
> Nr. 12154 | USt-IdNr.: DE120607556
>
> HINWEIS: Unsere Hinweise zum Umgang mit personenbezogenen Daten finden
> Sie in unserer Datenschutzerklärung unter
> https://www.kamp.de/datenschutz.html
>
> Diese Nachricht ist nur für den Adressaten bestimmt. Es ist nicht erlaubt,
> diese Nachricht zu kopieren oder Dritten zugänglich zu machen. Sollten Sie
> irrtümlich diese Nachricht erhalten haben, bitte ich um Ihre Mitteilung per
> E-Mail oder unter der oben angegebenen Telefonnummer.
>
>
>
> Am 17.11.2021 um 14:26 schrieb Dan van der Ster :
>
> The CLT is discussing a more feasible alternative to LTS, namely to
> publish an RC for each point release and involve the user community to
> help test it.
> This can be discussed at the user-dev meeting tomorrow.
> https://pad.ceph.com/p/ceph-user-dev-monthly-minutes
> (BTW I just restored that etherpad -- it had been spammed).
>
> Cheers, Dan
>
>
> On Wed, Nov 17, 2021 at 2:10 PM Eneko Lacunza  wrote:
>
>
>
> I think the desire for a LTS version comes from the perception that
> lately latest stable Ceph is not as stable as it has been before.
>
> So in that regard LTS version is a means, not the objective, at least
> for us.
>
> Cheers
>
> El 17/11/21 a las 14:01, Igor Fedotov escribió:
>
> First of all that's an open source - so 

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Marc]

Oh yes I have been telling gillette for years to stop producing so many 
different plastic model razors, but I still see racks full of them. I also have 
been telling BMW not to share so many parts between models, but they are still 
doing this. I also have been telling Microsoft about the many shit designs, and 
yet they are not listening. I have no idea where you got the illusion from that 
companies listen to customers??

> 
> Seems to me that it's the customers and not the factory management that
> determine what the factory workers produce.
> 
> -Dave
> 
> --
> Dave Hall
> Binghamton University
> kdh...@binghamton.edu 
> 
> 
> On Wed, Nov 17, 2021 at 7:41 AM Marc   > wrote:
> 
> 
> 
>   But since when do developers decide? Do you know any factory where
> factory workers decide what product they are going to make and not the
> product management??? IT is becoming such a refuge for undetected
> unprofessionals.
> 
> 
>   >
>   > Yeah, generally there is no much enthusiasm about supporting that
> among
>   > developers. But it would be nice to hear points from user side
> anyway...
>   >
>   > Igor
>   >
>   > On 11/17/2021 2:29 PM, Peter Lieven wrote:
>   > > Am 17.11.21 um 12:20 schrieb Igor Fedotov:
>   > >> Hi Peter,
>   > >>
>   > >> sure, why not...
>   > >
>   > > See [1]. I read it that it is not wanted by upstream
> developers. If we
>   > want it the community has to do it.
>   > >
>   > >
>   > > Nevertheless, I have put it on the list.
>   > >
>   > >
>   > >
>   > > Peter
>   > >
>   > >
>   > > [1]
>   >
> https://lists.ceph.io/hyperkitty/list/d...@ceph.io/thread/J3M3JWER7DS4CM3
>   > FNWLTG543X4VPJN7E/
>   > >
>   > >
>   > --
>   > Igor Fedotov
>   > Ceph Lead Developer
>   >
>   > Looking for help with your Ceph cluster? Contact us at
> https://croit.io
>   >
>   > croit GmbH, Freseniusstr. 31h, 81247 Munich
>   > CEO: Martin Verges - VAT-ID: DE310638492
>   > Com. register: Amtsgericht Munich HRB 231263
>   > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
>   >
>   > ___
>   > ceph-users mailing list -- ceph-users@ceph.io  us...@ceph.io>
>   > To unsubscribe send an email to ceph-users-le...@ceph.io
> 
>   ___
>   ceph-users mailing list -- ceph-users@ceph.io  us...@ceph.io>
>   To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: multiple active MDS servers is OK for production Ceph clusters OR Not

[Nathan Fish]

We have had excellent results with multi-MDS - *after* we pinned every
directory. directory migrations caused so much load that it was
frequently no faster than a single MDS. This was on Nautilus at the
time. The hard limit on strays is also per-MDS, so we ended up
splitting to more MDS' to buy some time there. From what I can tell,
snapshots don't get moved if you change a pin, at least not
immediately, so keep that in mind.

On Wed, Nov 17, 2021 at 8:12 AM Eugen Block  wrote:
>
> Hi,
>
> in this thread [1] Dan gives very helpful points to consider regarding
> multi-active MDS. Are you sure you need that?
> One of our customers has tested such a setup extensively with
> directory pinning because the MDS balancer couldn't handle the high
> client load. In order to better utilize the MDS servers (only one
> thread) they ran multiple daemons per server which also worked quite
> well. The only issue is the rolling upgrade where you need to reduce
> the max_mds to 1 which doesn't work here. But this is all still in
> Nautilus.
>
> [1]
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/4LNXN6U5DTB2BFPBDGDUKTEB4THD7HCH/
>
>
> Zitat von huxia...@horebdata.cn:
>
> > Dear Cephers,
> >
> > On reading a technical blog from Croit:
> > https://croit.io/blog/ceph-performance-test-and-optimization
> >
> > It says the following: "It should be noted that it is still debated
> > whether a configuration with multiple active MDS servers is OK for
> > production Ceph clusters."
> >
> > Just wonder whether multiple active MDS servers is OK for production
> > Ceph clusters OR Not?  Does any one has practical lessons ? Can some
> > one share more stories with us, whether success or failure
> >
> > Thanks a lot in advance,
> >
> > samuel
> >
> >
> >
> >
> >
> > huxia...@horebdata.cn
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] 16.2.6 SMP NOPTI - OSD down - Node Exporter Tainted

[Marco Pizzolo]

Good day everyone,

This is a bit of a recurring theme for us on a new deployment performed at
16.2.6 on Ubuntu 20.04.3 with HWE stack.

We have had good stability over the past 3 weeks or so copying data, and we
now have about 230M objects (470TB of 1PB used) and we have had 1 OSD drop
from each of the two OSD hosts currently in this cluster.  The third node
is a monitor but not an OSD host at this time.  Size is set at 2 for now as
we're migrating from an older Nautilus cluster.

I'm seeing the following on one host:

[905905.006041] BUG: kernel NULL pointer dereference, address:
00c0
[905905.008536] #PF: supervisor read access in kernel mode
[905905.02] #PF: error_code(0x) - not-present page
[905905.013859] PGD 10f1b6067 P4D 10f1b6067 PUD 11bd2e067 PMD 0
[905905.016678] Oops:  [#1] SMP NOPTI
[905905.018982] CPU: 86 PID: 69590 Comm: node_exporter Tainted: G
OE 5.11.0-38-generic #42~20.04.1-Ubuntu
[905905.020582] Hardware name: Supermicro SSG-6049P-E1CR60L+/X11DSC+, BIOS
3.3 02/21/2020
[905905.022192] RIP: 0010:blk_mq_put_rq_ref+0xa/0x60
[905905.023724] Code: 15 0f b6 d3 4c 89 e7 be 01 00 00 00 e8 cf fe ff ff 5b
41 5c 5d c3 0f 0b 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 8b 47 10
<48> 8b 80 c0 00 00 00 48 89 e5 48 3b 78 40 74 1f 4c 8d 87 e8 00 00
[905905.026794] RSP: 0018:b7713ca3fb08 EFLAGS: 00010202
[905905.029492] RAX:  RBX: b7713ca3fb90 RCX:
0002
[905905.031813] RDX: 0001 RSI: 0206 RDI:
9ea4f46ce400
[905905.034098] RBP: b7713ca3fb40 R08:  R09:
0005
[905905.036360] R10: 0825 R11: 000b R12:
9ea4f46ce400
[905905.038564] R13: 9ea4f426ec00 R14:  R15:
0001
[905905.040345] FS:  00c000507910() GS:9f033fb8()
knlGS:
[905905.042034] CS:  0010 DS:  ES:  CR0: 80050033
[905905.043660] CR2: 00c0 CR3: 00010d2bc001 CR4:
007706e0
[905905.045298] DR0:  DR1:  DR2:

[905905.046883] DR3:  DR6: fffe0ff0 DR7:
0400
[905905.048424] PKRU: 5554
[905905.049979] Call Trace:
[905905.051907]  ? bt_iter+0x54/0x90
[905905.053815]  blk_mq_queue_tag_busy_iter+0x18b/0x2d0
[905905.055730]  ? blk_mq_hctx_mark_pending+0x70/0x70
[905905.057711]  ? blk_mq_hctx_mark_pending+0x70/0x70
[905905.059548]  blk_mq_in_flight+0x38/0x60
[905905.061409]  diskstats_show+0x75/0x2b0
[905905.063166]  seq_read_iter+0x2a3/0x450
[905905.064871]  proc_reg_read_iter+0x5e/0x80
[905905.066648]  new_sync_read+0x110/0x1a0
[905905.068276]  vfs_read+0x154/0x1b0
[905905.069879]  ksys_read+0x67/0xe0
[905905.071436]  __x64_sys_read+0x1a/0x20
[905905.072934]  do_syscall_64+0x38/0x90
[905905.074409]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[905905.075841] RIP: 0033:0x4a5c20
[905905.077230] Code: 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 49 c7 c2 00
00 00 00 49 c7 c0 00 00 00 00 49 c7 c1 00 00 00 00 48 8b 44 24 08 0f 05
<48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
[905905.080065] RSP: 002b:00c00016e8b8 EFLAGS: 0212 ORIG_RAX:

[905905.081441] RAX: ffda RBX: 00c30a00 RCX:
004a5c20
[905905.082878] RDX: 1000 RSI: 00c00062c000 RDI:
0006
[905905.084299] RBP: 00c00016e908 R08:  R09:

[905905.085673] R10:  R11: 0212 R12:
0040
[905905.087018] R13: 0040 R14: 00b6420c R15:

[905905.088305] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos
jfs xfs cpuid binfmt_misc overlay cuse bonding rdma_ucm(OE) rdma_cm(OE)
iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) nls_iso8859_1 dm_multipath
scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr intel_rapl_common
isst_if_common skx_edac nfit x86_pkg_temp_thermal coretemp kvm_intel kvm
rapl ipmi_ssif intel_cstate mlx5_ib(OE) ib_uverbs(OE) ib_core(OE)
efi_pstore input_leds joydev intel_pch_thermal mei_me mei ioatdma acpi_ipmi
ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter mac_hid sch_fq_codel
msr ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456
async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
libcrc32c raid0 multipath linear hid_generic usbhid hid ses enclosure
scsi_transport_sas raid1 mlx5_core(OE) crct10dif_pclmul ast drm_vram_helper
i2c_algo_bit crc32_pclmul drm_ttm_helper ttm ghash_clmulni_intel
aesni_intel drm_kms_helper syscopyarea pci_hyperv_intf sysfillrect
[905905.088448]  crypto_simd mlxdevm(OE) sysimgblt cryptd fb_sys_fops
psample glue_helper cec mlxfw(OE) rc_core ixgbe tls drm mlx_compat(OE)
xfrm_algo megaraid_sas dca mdio vmd i2c_i801 xhci_pci ahci i2c_smbus
lpc_ich xhci_pci_renesas libahci wmi
[905905.102788] CR2: 00c0
[905905.104158] ---[ end trace 2e934962aff06160 ]---
[905905.155465] RIP: 

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Daniel Tönnißen]

The demand for LTS - at least in our case - does not stem from 
unprofessionalism or biased opinion.
It's the desire to stay up to date on security patches as much as possible 
while maintaining a well tested and stable environment.

Both Pacific and Octopus (we’re currently on Nautilus) have some problems 
within themselves that made us not upgrading our cluster.
The latest releases felt kind of rushed out rather than well tested.

We (the LTS „faction“) still would like to keep up with security and minor 
patches on the cluster. We’re not talking about backporting new features.

There’s a saying: „Never change a running system“

-- 
Mit freundlichen Grüßen aus Oberhausen
Daniel Tönnißen
(Systemadministrator)
  
KAMP Netzwerkdienste GmbH
Vestische Str. 89−91 | 46117 Oberhausen 
Fon:+49 (0) 208.89 402-50 
Fax: +49 (0) 208.89 402-40E-Mail:d...@kamp.de 
WWW: https://www.kamp.de
Geschäftsführer: Heiner Lante | Michael Lante | Amtsgericht Duisburg | HRB Nr. 
12154 | USt-IdNr.: DE120607556
HINWEIS: UNSERE HINWEISE ZUM UMGANG MIT PERSONENBEZOGENEN DATEN FINDEN SIE IN 
UNSERER DATENSCHUTZERKLÄRUNG UNTER HTTPS://WWW.KAMP.DE/DATENSCHUTZ.HTML 


DIESE NACHRICHT IST NUR FÜR DEN ADRESSATEN BESTIMMT. ES IST NICHT ERLAUBT, 
DIESE NACHRICHT ZU KOPIEREN ODER DRITTEN ZUGÄNGLICH ZU MACHEN. SOLLTEN SIE 
IRRTÜMLICH DIESE NACHRICHT ERHALTEN HABEN, BITTE ICH UM IHRE MITTEILUNG PER 
E-MAIL ODER UNTER DER OBEN ANGEGEBENEN TELEFONNUMMER.




> Am 17.11.2021 um 14:26 schrieb Dan van der Ster :
> 
> The CLT is discussing a more feasible alternative to LTS, namely to
> publish an RC for each point release and involve the user community to
> help test it.
> This can be discussed at the user-dev meeting tomorrow.
> https://pad.ceph.com/p/ceph-user-dev-monthly-minutes
> (BTW I just restored that etherpad -- it had been spammed).
> 
> Cheers, Dan
> 
> 
> On Wed, Nov 17, 2021 at 2:10 PM Eneko Lacunza  wrote:
>> 
>> 
>> I think the desire for a LTS version comes from the perception that
>> lately latest stable Ceph is not as stable as it has been before.
>> 
>> So in that regard LTS version is a means, not the objective, at least
>> for us.
>> 
>> Cheers
>> 
>> El 17/11/21 a las 14:01, Igor Fedotov escribió:
>>> First of all that's an open source - so developers tend to have higher
>>> influence to decision making.
>>> 
>>> And you can replace "among developers" to "among CLT" in my previous
>>> post...
>>> 
>>> Hopefully this position can be shifter if there is a wide "feature
>>> request" from the field hence please try to share your thoughts at the
>>> upcoming meeting.
>>> 
>>> 
>>> Igor
>>> 
>>> On 11/17/2021 3:40 PM, Marc wrote:
 But since when do developers decide? Do you know any factory where
 factory workers decide what product they are going to make and not
 the product management??? IT is becoming such a refuge for undetected
 unprofessionals.
 
 
> Yeah, generally there is no much enthusiasm about supporting that among
> developers. But it would be nice to hear points from user side
> anyway...
> 
> Igor
> 
> On 11/17/2021 2:29 PM, Peter Lieven wrote:
>> Am 17.11.21 um 12:20 schrieb Igor Fedotov:
>>> Hi Peter,
>>> 
>>> sure, why not...
>> See [1]. I read it that it is not wanted by upstream developers. If we
> want it the community has to do it.
>> 
>> Nevertheless, I have put it on the list.
>> 
>> 
>> 
>> Peter
>> 
>> 
>> [1]
> https://lists.ceph.io/hyperkitty/list/d...@ceph.io/thread/J3M3JWER7DS4CM3
> 
> FNWLTG543X4VPJN7E/
>> 
> --
> Igor Fedotov
> Ceph Lead Developer
> 
> Looking for help with your Ceph cluster? Contact us at https://croit.io
> 
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>>> 
>> 
>> Eneko Lacunza
>> Zuzendari teknikoa | Director técnico
>> Binovo IT Human Project
>> 
>> Tel. +34 943 569 206 | https://www.binovo.es
>> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
>> 
>> https://www.youtube.com/user/CANALBINOVO
>> https://www.linkedin.com/company/37269706/
>> 
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To 

[ceph-users] Re: [rgw multisite] adding lc policy to buckets in non-master zones result in 503 code

[Boris Behrens]

Is there an incompatibility between nautilus and octopus? master is
octopus, the other one is nautilus.

Am Mi., 17. Nov. 2021 um 11:12 Uhr schrieb Boris Behrens :

> Hi,
> we've set up a non replicated multisite environment.
> We have one realm, with multiple zonegroups and one zone per group.
>
> When I try to add a lifecycle policy to a bucket that is not located in
> the master zonegroup, I will receive 503 errorse from the RGW.
> s3cmd sometimes just hangs forever or returns with eiter S3Error: 503
> (Slow Down) or WARNING: Retrying failed request: /?lifecycle (503
> (ServiceUnavailable))
>
> I've checked the ceph-client logs and can see that the request gets
> forwarded to the master cluster and this responds with a 400 return code.
>
> log of the rgw
> 2021-11-17 10:06:07.239 7f922695e700 1 == starting new request
> req=0x7f92684116c0 = 2021-11-17 10:06:07.239 7f922695e700 0 req 6
> 0.000s sending request to master zonegroup 2021-11-17 10:06:07.344
> 7f922695e700 0 req 6 0.105s s3:put_lifecycle forward_request_to_master
> returned ret=-2202 2021-11-17 10:06:07.344 7f922695e700 1 == req done
> req=0x7f92684116c0 op status=-2202 http_status=503 latency=0.105s == 
> 2021-11-17
> 10:06:07.344 7f922695e700 1 beast: 0x7f92684116c0: xx:xx:xx:22::f0 - -
> [2021-11-17 10:06:07.0.34425s] "PUT /?lifecycle HTTP/1.1" 503 223 - - -
>
> log from the master rgw
> 2021-11-17T09:05:58.007+ 7f511d8ac700 1 civetweb: 0x55b7b7454000:
> fd00:2380:0:24::66 - - [17/Nov/2021:09:05:57 +] "PUT
> /myfancypants/?lifecycle=UUIDrgwx-zonegroup=UUID HTTP/1.1" 400 568
> - -
>
> Does someone ever came across this problem?
>
>

-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Dan van der Ster]

The CLT is discussing a more feasible alternative to LTS, namely to
publish an RC for each point release and involve the user community to
help test it.
This can be discussed at the user-dev meeting tomorrow.
https://pad.ceph.com/p/ceph-user-dev-monthly-minutes
(BTW I just restored that etherpad -- it had been spammed).

Cheers, Dan


On Wed, Nov 17, 2021 at 2:10 PM Eneko Lacunza  wrote:
>
>
> I think the desire for a LTS version comes from the perception that
> lately latest stable Ceph is not as stable as it has been before.
>
> So in that regard LTS version is a means, not the objective, at least
> for us.
>
> Cheers
>
> El 17/11/21 a las 14:01, Igor Fedotov escribió:
> > First of all that's an open source - so developers tend to have higher
> > influence to decision making.
> >
> > And you can replace "among developers" to "among CLT" in my previous
> > post...
> >
> > Hopefully this position can be shifter if there is a wide "feature
> > request" from the field hence please try to share your thoughts at the
> > upcoming meeting.
> >
> >
> > Igor
> >
> > On 11/17/2021 3:40 PM, Marc wrote:
> >> But since when do developers decide? Do you know any factory where
> >> factory workers decide what product they are going to make and not
> >> the product management??? IT is becoming such a refuge for undetected
> >> unprofessionals.
> >>
> >>
> >>> Yeah, generally there is no much enthusiasm about supporting that among
> >>> developers. But it would be nice to hear points from user side
> >>> anyway...
> >>>
> >>> Igor
> >>>
> >>> On 11/17/2021 2:29 PM, Peter Lieven wrote:
>  Am 17.11.21 um 12:20 schrieb Igor Fedotov:
> > Hi Peter,
> >
> > sure, why not...
>  See [1]. I read it that it is not wanted by upstream developers. If we
> >>> want it the community has to do it.
> 
>  Nevertheless, I have put it on the list.
> 
> 
> 
>  Peter
> 
> 
>  [1]
> >>> https://lists.ceph.io/hyperkitty/list/d...@ceph.io/thread/J3M3JWER7DS4CM3
> >>>
> >>> FNWLTG543X4VPJN7E/
> 
> >>> --
> >>> Igor Fedotov
> >>> Ceph Lead Developer
> >>>
> >>> Looking for help with your Ceph cluster? Contact us at https://croit.io
> >>>
> >>> croit GmbH, Freseniusstr. 31h, 81247 Munich
> >>> CEO: Martin Verges - VAT-ID: DE310638492
> >>> Com. register: Amtsgericht Munich HRB 231263
> >>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
> >>>
> >>> ___
> >>> ceph-users mailing list -- ceph-users@ceph.io
> >>> To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
> Eneko Lacunza
> Zuzendari teknikoa | Director técnico
> Binovo IT Human Project
>
> Tel. +34 943 569 206 | https://www.binovo.es
> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
>
> https://www.youtube.com/user/CANALBINOVO
> https://www.linkedin.com/company/37269706/
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Igor Fedotov]

First of all that's an open source - so developers tend to have higher 
influence to decision making.


And you can replace "among developers" to "among CLT" in my previous post...

Hopefully this position can be shifter if there is a wide "feature 
request" from the field hence please try to share your thoughts at the 
upcoming meeting.



Igor

On 11/17/2021 3:40 PM, Marc wrote:

But since when do developers decide? Do you know any factory where factory 
workers decide what product they are going to make and not the product 
management??? IT is becoming such a refuge for undetected unprofessionals.



Yeah, generally there is no much enthusiasm about supporting that among
developers. But it would be nice to hear points from user side anyway...

Igor

On 11/17/2021 2:29 PM, Peter Lieven wrote:

Am 17.11.21 um 12:20 schrieb Igor Fedotov:

Hi Peter,

sure, why not...

See [1]. I read it that it is not wanted by upstream developers. If we

want it the community has to do it.


Nevertheless, I have put it on the list.



Peter


[1]

https://lists.ceph.io/hyperkitty/list/d...@ceph.io/thread/J3M3JWER7DS4CM3
FNWLTG543X4VPJN7E/



--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Marc]

But since when do developers decide? Do you know any factory where factory 
workers decide what product they are going to make and not the product 
management??? IT is becoming such a refuge for undetected unprofessionals.


> 
> Yeah, generally there is no much enthusiasm about supporting that among
> developers. But it would be nice to hear points from user side anyway...
> 
> Igor
> 
> On 11/17/2021 2:29 PM, Peter Lieven wrote:
> > Am 17.11.21 um 12:20 schrieb Igor Fedotov:
> >> Hi Peter,
> >>
> >> sure, why not...
> >
> > See [1]. I read it that it is not wanted by upstream developers. If we
> want it the community has to do it.
> >
> >
> > Nevertheless, I have put it on the list.
> >
> >
> >
> > Peter
> >
> >
> > [1]
> https://lists.ceph.io/hyperkitty/list/d...@ceph.io/thread/J3M3JWER7DS4CM3
> FNWLTG543X4VPJN7E/
> >
> >
> --
> Igor Fedotov
> Ceph Lead Developer
> 
> Looking for help with your Ceph cluster? Contact us at https://croit.io
> 
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Igor Fedotov]

Yeah, generally there is no much enthusiasm about supporting that among 
developers. But it would be nice to hear points from user side anyway...


Igor

On 11/17/2021 2:29 PM, Peter Lieven wrote:

Am 17.11.21 um 12:20 schrieb Igor Fedotov:

Hi Peter,

sure, why not...


See [1]. I read it that it is not wanted by upstream developers. If we want it 
the community has to do it.


Nevertheless, I have put it on the list.



Peter


[1] 
https://lists.ceph.io/hyperkitty/list/d...@ceph.io/thread/J3M3JWER7DS4CM3FNWLTG543X4VPJN7E/



--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Peter Lieven]

Am 17.11.21 um 12:20 schrieb Igor Fedotov:
> Hi Peter,
>
> sure, why not...


See [1]. I read it that it is not wanted by upstream developers. If we want it 
the community has to do it.


Nevertheless, I have put it on the list.



Peter


[1] 
https://lists.ceph.io/hyperkitty/list/d...@ceph.io/thread/J3M3JWER7DS4CM3FNWLTG543X4VPJN7E/


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Igor Fedotov]

Hi Peter,

sure, why not...


Thanks,

Igor

On 11/17/2021 10:48 AM, Peter Lieven wrote:



Am 09.11.2021 um 00:01 schrieb Igor Fedotov :

Hi folks,

having a LTS release cycle could be a great topic for upcoming "Ceph User + Dev 
Monthly meeting".

The first one is scheduled  on November 18, 2021, 14:00-15:00 UTC

https://pad.ceph.com/p/ceph-user-dev-monthly-minutes

Any volunteers to extend the agenda and advocate the idea?

Hi Igor,

do you still think we can add the LTS topic to the agenda? I will attend 
tomorrow and can try to advocate it.

Best,
Peter


Thanks,
Igor


On 11/8/2021 3:21 PM, Frank Schilder wrote:
Hi all,

I followed this thread with great interest and would like to add my 
opinion/experience/wishes as well.

I believe the question packages versus containers needs a bit more context to 
be really meaningful. This was already mentioned several times with regards to 
documentation. I see the following three topics tightly connected (my 
opinion/answers included):

1. Distribution: Packages are compulsory, containers are optional.
2. Deployment: Ceph adm (yet another deployment framework) and ceph (the actual 
storage system) should be strictly different projects.
3. Release cycles: The release cadence is way too fast, I very much miss a ceph 
LTS branch with at least 10 years back-port support.

These are my short answers/wishes/expectations in this context. I will add 
below some more reasoning as optional reading (warning: wall of text ahead).


1. Distribution
-

I don't think the question is about packages versus containers, because even if 
a distribution should decide not to package ceph any more, other distributors 
certainly will and the user community just moves away from distributions 
without ceph packages. In addition, unless Rad Hat plans to move to a 
source-only container where I run the good old configure - make - make install, 
it will be package based any ways, so packages are there to stay.

Therefore, the way I understand this question is about ceph-adm versus other 
deployment methods. Here, I think the push to a container-based ceph-adm only 
deployment is unlikely to become the no. 1 choice for everyone for good reasons 
already mentioned in earlier messages. In addition, I also believe that 
development of a general deployment tool is currently not sustainable as was 
mentioned by another user. My reasons for this are given in the next section.


2. Deployment
-

In my opinion, it is really important to distinguish three components of any 
open-source project: development (release cycles), distribution and deployment. 
Following the good old philosophy that every tool does exactly one job and does 
it well, each of these components are separate projects, because they 
correspond to different tools.

This implies immediately that ceph documentation should not contain 
documentation about packaging and deployment tools. Each of these ought to be 
strictly separate. If I have a low-level problem with ceph and go to the ceph 
documentation, I do not want to see ceph-adm commands. Ceph documentation 
should be about ceph (the storage system) only. Such a mix-up is leading to 
problems and there were already ceph-user cases where people could not use the 
documentation for trouble shooting, because it showed ceph-adm commands but 
their cluster was not ceph-adm deployed.

In this context, I would prefer if there was a separate ceph-adm-users list so 
that ceph-users can focus on actual ceph problems again.

Now to the point that ceph-adm might be an un-sustainable project. Although at 
a first glance the idea of a generic deployment tool that solves all problems 
with a single command might look appealing, it is likely doomed to fail for a 
simple reason that was already indicated in an earlier message: ceph deployment 
is subject to a complexity paradox. Ceph has a very large configuration space 
and implementing and using a generic tool that covers and understands this 
configuration space is more complex than deploying any specific ceph cluster, 
each of which uses only a tiny subset of the entire configuration space.

In other words: deploying a specific ceph cluster is actually not that 
difficult.

Designing a - and dimensioning all components of a ceph cluster is difficult 
and none of the current deployment tools help here. There is not even a check 
for suitable hardware. In addition, technology is moving fast and adapting a 
generic tool to new developments in time seems a hopeless task. For example, 
when will ceph-adm natively support collocated lvm OSDs with dm_cache devices? 
Is it even worth trying to incorporate this?

My wish would be to keep the ceph project clean of any deployment tasks. In my 
opinion, the basic ceph tooling is already doing tasks that are the 
responsibility of a configuration management- and not a storage system (e.g. 
deploy unit files by default instead of as an option disabled by default).


3. Release cycles

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Marc]

> > having a LTS release cycle could be a great topic for upcoming "Ceph
> User + Dev Monthly meeting".
> >
> > The first one is scheduled  on November 18, 2021, 14:00-15:00 UTC
> >
> > https://pad.ceph.com/p/ceph-user-dev-monthly-minutes
> >
> > Any volunteers to extend the agenda and advocate the idea?
> 
> Hi Igor,
> 
> do you still think we can add the LTS topic to the agenda? I will attend
> tomorrow and can try to advocate it.
> 
> Best,
> Peter

I would like to have this also, I think almost everyone is interested in LTS 
version.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [rgw multisite] adding lc policy to buckets in non-master zones result in 503 code

[Boris Behrens]

Hi,
we've set up a non replicated multisite environment.
We have one realm, with multiple zonegroups and one zone per group.

When I try to add a lifecycle policy to a bucket that is not located in the
master zonegroup, I will receive 503 errorse from the RGW.
s3cmd sometimes just hangs forever or returns with eiter S3Error: 503 (Slow
Down) or WARNING: Retrying failed request: /?lifecycle (503
(ServiceUnavailable))

I've checked the ceph-client logs and can see that the request gets
forwarded to the master cluster and this responds with a 400 return code.

log of the rgw
2021-11-17 10:06:07.239 7f922695e700 1 == starting new request
req=0x7f92684116c0 = 2021-11-17 10:06:07.239 7f922695e700 0 req 6
0.000s sending request to master zonegroup 2021-11-17 10:06:07.344
7f922695e700 0 req 6 0.105s s3:put_lifecycle forward_request_to_master
returned ret=-2202 2021-11-17 10:06:07.344 7f922695e700 1 == req done
req=0x7f92684116c0 op status=-2202 http_status=503 latency=0.105s
== 2021-11-17
10:06:07.344 7f922695e700 1 beast: 0x7f92684116c0: xx:xx:xx:22::f0 - -
[2021-11-17 10:06:07.0.34425s] "PUT /?lifecycle HTTP/1.1" 503 223 - - -

log from the master rgw
2021-11-17T09:05:58.007+ 7f511d8ac700 1 civetweb: 0x55b7b7454000:
fd00:2380:0:24::66 - - [17/Nov/2021:09:05:57 +] "PUT
/myfancypants/?lifecycle=UUIDrgwx-zonegroup=UUID HTTP/1.1" 400 568
- -

Does someone ever came across this problem?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm / ceph orch : indefinite hang adding hosts to new cluster

[Eugen Block]

Hi,


This is the only logging output we see:
# cephadm shell -- ceph orch host add kvm-mon02 192.168.7.12
Inferring fsid 826b9b36-4729-11ec-99f0-c81f66d05a38
Using recent ceph image  
quay.io/ceph/ceph@sha256:a2c23b6942f7fbc1e15d8cfacd6655a681fe0e44f288e4a158db22030b8d58e3


This command hangs indefinitely until killed via podman kill​.


the first thought coming to mind is, do the hosts have internet access  
to download the container images? But if the bootstrap worked the  
answer would be yes. And IIUC you tried different hosts for bootstrap  
and all of them worked? Just for the record, a manual 'podman pull  
...' on the second host works, too?


What does a 'cephadm check-host' on the second host report?
The syslog on the second node should usually reveal errors, have you  
checked 'journalctl -f' during the attempt to add it?



Zitat von Lincoln Bryant :


Greetings list,

We have a new Ceph cluster we are trying to deploy on EL8 (CentOS  
Stream) using cephadm (+podman), targeting Pacific.


We are successfully able to bootstrap the first host, but attempting  
to add any additional hosts hangs indefinitely. We have confirmed  
that we are able to SSH from the first host to subsequent hosts  
using the key generated by Ceph.


This is the only logging output we see:
# cephadm shell -- ceph orch host add kvm-mon02 192.168.7.12
Inferring fsid 826b9b36-4729-11ec-99f0-c81f66d05a38
Using recent ceph image  
quay.io/ceph/ceph@sha256:a2c23b6942f7fbc1e15d8cfacd6655a681fe0e44f288e4a158db22030b8d58e3


This command hangs indefinitely until killed via podman kill​.

Inspecting the host we're trying to add, we see that Ceph has  
launched a python process:
root3604  0.0  0.0 164128  6316 ?S16:36   0:00   
|   \_ sshd: root@notty
root3605  0.0  0.0  31976  8752 ?Ss   16:36   0:00   
|   \_ python3 -c import sys;exec(eval(sys.stdin.readline()))


Inside of the mgr container, we see 2 SSH connections:
ceph 186  0.0  0.0  44076  6676 ?S22:31   0:00   
\_ ssh -C -F /tmp/cephadm-conf-s0b8c90d -i  
/tmp/cephadm-identity-8ku7ib6b root@192.168.7.13 python3 -c "import  
sys;exec(eval(sys.stdin.readline()))"
ceph 211  0.0  0.0  44076  6716 ?S22:36   0:00   
\_ ssh -C -F /tmp/cephadm-conf-s0b8c90d -i  
/tmp/cephadm-identity-8ku7ib6b root@192.168.7.12 python3 -c "import  
sys;exec(eval(sys.stdin.readline()))"


where 192.168.1.13 is the IP of the first host in the cluster (which  
has succesfully bootstrapped and is running mgr, mon, and so on),  
and 196.168.1.12 is the host we are trying to unsuccessfully add.


The mgr logs show no particularly interesting except for:
debug 2021-11-16T22:39:03.570+ 7fb6e4914700  0 [progress WARNING  
root] complete: ev de058df7-b54a-4429-933a-99abe7796715 does not exist
debug 2021-11-16T22:39:03.571+ 7fb6e4914700  0 [progress WARNING  
root] complete: ev 61fe4998-4ef4-4640-8a13-2b0928da737f does not exist
debug 2021-11-16T22:39:03.571+ 7fb6e4914700  0 [progress WARNING  
root] complete: ev 2323b586-5262-497e-b318-42702c0dc3dc does not exist
debug 2021-11-16T22:39:03.572+ 7fb6e4914700  0 [progress WARNING  
root] complete: ev f882757b-e03a-4798-8fae-4d55e2d1f531 does not exist
debug 2021-11-16T22:39:03.572+ 7fb6e4914700  0 [progress WARNING  
root] complete: ev 3d35e91f-b475-4dbc-a040-65658e82fe67 does not exist
debug 2021-11-16T22:39:03.572+ 7fb6e4914700  0 [progress WARNING  
root] complete: ev 694ed4e6-b741-4bf3-9f7f-b11c549aca87 does not exist
debug 2021-11-16T22:39:03.573+ 7fb6e4914700  0 [progress WARNING  
root] complete: ev 1ce718f4-a23b-4a27-8f7c-bc2610340403 does not exist


We've tried purging/reinstalling several times to no avail, and also  
tried swapping which host was used as the initial bootstrap mon and  
so on. I've also tried the option to disable the monitoring stack  
and that did not help either.


In any case, we are not sure how to proceed from here. Is there  
anything we can do to turn up logging verbosity, or other things to  
check? I've tried to find the ceph orch​ source code to try to  
understand what may be happening, but I'm not sure where to look.


Thanks,
Lincoln Bryant
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: multiple active MDS servers is OK for production Ceph clusters OR Not

[Eugen Block]

Hi,

in this thread [1] Dan gives very helpful points to consider regarding  
multi-active MDS. Are you sure you need that?
One of our customers has tested such a setup extensively with  
directory pinning because the MDS balancer couldn't handle the high  
client load. In order to better utilize the MDS servers (only one  
thread) they ran multiple daemons per server which also worked quite  
well. The only issue is the rolling upgrade where you need to reduce  
the max_mds to 1 which doesn't work here. But this is all still in  
Nautilus.


[1]  
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/4LNXN6U5DTB2BFPBDGDUKTEB4THD7HCH/



Zitat von huxia...@horebdata.cn:


Dear Cephers,

On reading a technical blog from Croit:  
https://croit.io/blog/ceph-performance-test-and-optimization


It says the following: "It should be noted that it is still debated  
whether a configuration with multiple active MDS servers is OK for  
production Ceph clusters."


Just wonder whether multiple active MDS servers is OK for production  
Ceph clusters OR Not?  Does any one has practical lessons ? Can some  
one share more stories with us, whether success or failure


Thanks a lot in advance,

samuel





huxia...@horebdata.cn
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io