[ceph-users] Re: "Incomplete" pg's

2022-03-07 Thread Eugen Block 

Hi,

IIUC the OSDs 3,4,5 have been removed while some PGs still refer to  
them, correct? Have the OSDs been replaced with the same IDs? If not  
(so there are currently no OSDs with IDs 3,4,5 in your osd tree) maybe  
marking them as lost [1] would resolve the stuck PG creation, although  
I doubt that this will do anything if there aren't any OSDs with these  
IDs anymore. I haven't had to mark an OSD lost yet myself, so I'm not  
sure of the consequences.
There's a similar thread [2] where the situation got resolved, not by  
marking the OSDs as lost but by using  
'osd_find_best_info_ignore_history_les' which I haven't used myself  
either. But maybe worth a shot?



[1] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/
[2]  
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/G6MJF7PGCCW5JTC6R6UV2EXT54YGU3LG/



Zitat von "Kyriazis, George" :

Ok, I saw that there is now a “ceph old force-create-pg” command.   
Not sure if it is a replacement of “ceph pg force_create_pg” or if  
it does something different.


I tried it, and it looked like it worked:

# ceph osd force-create-pg 1.353 --yes-i-really-mean-it
pg 1.353 now creating, ok
#

But the pg is still stuck in “incomplete” state.

Re-issuing the same command, I get:

# ceph osd force-create-pg 1.353 --yes-i-really-mean-it
pg 1.353 already creating
#

Which means that the request is queued up somewhere, however, the pg  
in question is still stuck in incomplete state:


# ceph pg ls | grep ^1\.353
1.3530 0  00 0
 0   0 0incomplete71m 
 0'054514:92[4,6,22]p4[4,6,22]p4   
2022-02-28T15:47:37.794357-0600  2022-02-02T07:53:15.339511-0600

#

How do I find out if it is stuck, or just plain queued behind some  
other request?


Thank you!

George

On Mar 7, 2022, at 12:09 PM, Kyriazis, George  
mailto:george.kyria...@intel.com>> wrote:


After some thought, I decided to try “ceph pg force_create_pg” on  
the incomplete pgs, as suggested by name online sources.


However, I got:

# ceph pg force_create_pg 1.353
no valid command found; 10 closest matches:
pg stat
pg getmap
pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...]
pg dump_json [all|summary|sum|pools|osds|pgs...]
pg dump_pools_json
pg ls-by-pool  [...]
pg ls-by-primary  
http://osd.id/>>>  
[] [...]
pg ls-by-osd  
http://osd.id/>>>  
[] [...]

pg ls [] [...]
pg dump_stuck [inactive|unclean|stale|undersized|degraded...]  
[]

Error EINVAL: invalid command
#

?

I am running pacific 16.2.7.

Thanks!

George


On Mar 4, 2022, at 7:51 AM, Kyriazis, George  
mailto:george.kyria...@intel.com>>  
wrote:


Thanks Janne,

(Inline)

On Mar 4, 2022, at 1:04 AM, Janne Johansson  
mailto:icepic...@gmail.com>>  
wrote:


Due to a mistake on my part, I accidentally destroyed more OSDs that  
I needed to, and I ended up with 2 pgs in “incomplete” state.


Doing “ceph pg query on one of the pgs that is incomplete, I get the  
following (somewhere in the output):


 "up": [
 12,
 6,
 20
 ],
 "acting": [
 12,
 6,
 20
 ],
 "avail_no_missing": [],
 "object_location_counts": [],
 "blocked_by": [
 3,
 4,
 5
 ],
 "up_primary": 12,
 "acting_primary": 12,
 "purged_snaps": []


I am assuming this means that OSDs 3,4,5 were the original ones  
(that are now destroyed), but I don’t understand why the output  
shows 12, 6, 20 as active.


I can't help with the cephfs part since we don't use that, but I think
the above output means "since 3,4,5 are gone, 12,6 and 20 are now
designated as the replacement OSDs to hold the PG", but since 3,4,5
are gone, none of them can backfill into 12,6,20, so 12,6,20 are
waiting for this PG to appear "somewhere" so they can recover.

I thought that if that was the case 3,4,5 should be listed as  
“active”, with 12,6,20 as “up”..


My corcern about cephfs is that, since it is a layer above the ceph  
base layer, there maybe the corrective action needs to start at  
cephfs, otherwise cephfs won’t be aware of any changes happening  
underneath.


Perhaps you can force pg creation, so that 12,6,20 gets an empty PG to
start the pool again, and then hope that the next rsync will fill in
any missing slots, but this part I am not so sure about since I don't
know what other data apart from file contents may exist in a cephfs
pool.

Is the worst-case (dropping the pool, recreating it and running a full
rsync again) a possible way out? If so, you can perhaps test and see
if you can bridge the gap of the missing PGs, but if resyncing is out,
then wait for suggestions from someone more qualified at cephfs stuff
than me. ;)

[ceph-users] Re: Cephadm is stable or not in product?

2022-03-07 Thread Martin Verges 
Some say it is, some say it's not.
Every time I try it, it's buggy as hell and I can destroy my test clusters
with ease. That's why I still avoid it. But as you can see in my signature,
I am biased ;).

--
Martin Verges
Managing director

Mobile: +49 174 9335695  | Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx


On Tue, 8 Mar 2022 at 05:18, norman.kern  wrote:

> Dear Ceph folks,
>
> Anyone is using cephadm in product(Version: Pacific)？ I found several bugs
> on it and
> I really doubt it.
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Cephadm is stable or not in product?

2022-03-07 Thread norman.kern 

Dear Ceph folks,

Anyone is using cephadm in product(Version: Pacific)？ I found several bugs on 
it and
I really doubt it.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] octopus (15.2.16) OSDs crash or don't answer heathbeats (and get marked as down)

2022-03-07 Thread Boris Behrens 
Hi,

we've had the problem with OSDs marked as offline since we updated to
octopus and hope the problem would be fixed with the latest patch. We have
this kind of problem only with octopus and there only with the big s3
cluster.
* Hosts are all Ubuntu 20,04 and we've set the txqueuelen to 10k
* Network interfaces are 20gbit (2x10 in a 802.3ad encap3+4 bond)
* We only use the frontend network.
* All disks are spinning, some have block.db devices.
* All disks are bluestore
* configs are mostly defaults
* we've set the OSDs to restart=always without a limit, because we had the
problem with unavailable PGs when two OSDs are marked as offline and the
share PGs.

But since we installed the latest patch we are experiencing more OSD downs
and even crashes.
I tried to remove as much duplicated lines as possible.

Is the numa error a problem?
Why do OSD daemons not respond to hearthbeats? I mean even when the disk is
totally loaded with IO, the system itself should answer heathbeats, or am I
missing something?

I really hope some of you could send me on the correct way to solve this
nasty problem.

This is how the latest crash looks like
Mar 07 17:44:15 s3db18 ceph-osd[4530]: 2022-03-07T17:44:15.099+
7f5f05d2a700 -1 osd.161 489755 set_numa_affinity unable to identify public
interface '' numa node: (2) No such file or directory
...
Mar 07 17:49:07 s3db18 ceph-osd[4530]: 2022-03-07T17:49:07.678+
7f5f05d2a700 -1 osd.161 489774 set_numa_affinity unable to identify public
interface '' numa node: (2) No such file or directory
Mar 07 17:53:07 s3db18 ceph-osd[4530]: *** Caught signal (Aborted) **
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  in thread 7f5ef1501700
thread_name:tp_osd_tp
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  ceph version 15.2.16
(d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  1: (()+0x143c0) [0x7f5f0d4623c0]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  2: (pthread_kill()+0x38)
[0x7f5f0d45ef08]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  3:
(ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*,
unsigned long)+0x471) [0x55a699a01201]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  4:
(ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, unsigned
long, unsigned long)+0x8e) [0x55a699a0199e]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  5:
(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3f0)
[0x55a699a224b0]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  6:
(ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a699a252c4]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  7: (()+0x8609) [0x7f5f0d456609]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  8: (clone()+0x43) [0x7f5f0cfc0163]
Mar 07 17:53:07 s3db18 ceph-osd[4530]: 2022-03-07T17:53:07.387+
7f5ef1501700 -1 *** Caught signal (Aborted) **
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  in thread 7f5ef1501700
thread_name:tp_osd_tp
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  ceph version 15.2.16
(d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  1: (()+0x143c0) [0x7f5f0d4623c0]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  2: (pthread_kill()+0x38)
[0x7f5f0d45ef08]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  3:
(ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*,
unsigned long)+0x471) [0x55a699a01201]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  4:
(ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, unsigned
long, unsigned long)+0x8e) [0x55a699a0199e]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  5:
(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3f0)
[0x55a699a224b0]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  6:
(ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a699a252c4]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  7: (()+0x8609) [0x7f5f0d456609]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  8: (clone()+0x43) [0x7f5f0cfc0163]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  NOTE: a copy of the executable, or
`objdump -rdS ` is needed to interpret this.
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  -5246> 2022-03-07T17:49:07.678+
7f5f05d2a700 -1 osd.161 489774 set_numa_affinity unable to identify public
interface '' numa node: (2) No such file or directory
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  0> 2022-03-07T17:53:07.387+
7f5ef1501700 -1 *** Caught signal (Aborted) **
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  in thread 7f5ef1501700
thread_name:tp_osd_tp
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  ceph version 15.2.16
(d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  1: (()+0x143c0) [0x7f5f0d4623c0]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  2: (pthread_kill()+0x38)
[0x7f5f0d45ef08]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  3:
(ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*,
unsigned long)+0x471) [0x55a699a01201]
Mar 07 17:53:07 s3db18 ceph-osd[4530]:  4:
(ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, unsigned
long, unsigned long)+0x8e) [0x55a699a0199e]
Mar 07 17:53:07 s3db18 ce

[ceph-users] Re: "Incomplete" pg's

2022-03-07 Thread Kyriazis, George 
Ok, I saw that there is now a “ceph old force-create-pg” command.  Not sure if 
it is a replacement of “ceph pg force_create_pg” or if it does something 
different.

I tried it, and it looked like it worked:

# ceph osd force-create-pg 1.353 --yes-i-really-mean-it
pg 1.353 now creating, ok
#

But the pg is still stuck in “incomplete” state.

Re-issuing the same command, I get:

# ceph osd force-create-pg 1.353 --yes-i-really-mean-it
pg 1.353 already creating
#

Which means that the request is queued up somewhere, however, the pg in 
question is still stuck in incomplete state:

# ceph pg ls | grep ^1\.353
1.3530 0  00 00 
  0 0incomplete71m 0'0
54514:92[4,6,22]p4[4,6,22]p4  2022-02-28T15:47:37.794357-0600  
2022-02-02T07:53:15.339511-0600
#

How do I find out if it is stuck, or just plain queued behind some other 
request?

Thank you!

George

On Mar 7, 2022, at 12:09 PM, Kyriazis, George 
mailto:george.kyria...@intel.com>> wrote:

After some thought, I decided to try “ceph pg force_create_pg” on the 
incomplete pgs, as suggested by name online sources.

However, I got:

# ceph pg force_create_pg 1.353
no valid command found; 10 closest matches:
pg stat
pg getmap
pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...]
pg dump_json [all|summary|sum|pools|osds|pgs...]
pg dump_pools_json
pg ls-by-pool  [...]
pg ls-by-primary http://osd.id/>>> 
[] [...]
pg ls-by-osd http://osd.id/>>> 
[] [...]
pg ls [] [...]
pg dump_stuck [inactive|unclean|stale|undersized|degraded...] []
Error EINVAL: invalid command
#

?

I am running pacific 16.2.7.

Thanks!

George


On Mar 4, 2022, at 7:51 AM, Kyriazis, George 
mailto:george.kyria...@intel.com>>
 wrote:

Thanks Janne,

(Inline)

On Mar 4, 2022, at 1:04 AM, Janne Johansson 
mailto:icepic...@gmail.com>> 
wrote:

Due to a mistake on my part, I accidentally destroyed more OSDs that I needed 
to, and I ended up with 2 pgs in “incomplete” state.

Doing “ceph pg query on one of the pgs that is incomplete, I get the following 
(somewhere in the output):

 "up": [
 12,
 6,
 20
 ],
 "acting": [
 12,
 6,
 20
 ],
 "avail_no_missing": [],
 "object_location_counts": [],
 "blocked_by": [
 3,
 4,
 5
 ],
 "up_primary": 12,
 "acting_primary": 12,
 "purged_snaps": []


I am assuming this means that OSDs 3,4,5 were the original ones (that are now 
destroyed), but I don’t understand why the output shows 12, 6, 20 as active.

I can't help with the cephfs part since we don't use that, but I think
the above output means "since 3,4,5 are gone, 12,6 and 20 are now
designated as the replacement OSDs to hold the PG", but since 3,4,5
are gone, none of them can backfill into 12,6,20, so 12,6,20 are
waiting for this PG to appear "somewhere" so they can recover.

I thought that if that was the case 3,4,5 should be listed as “active”, with 
12,6,20 as “up”..

My corcern about cephfs is that, since it is a layer above the ceph base layer, 
there maybe the corrective action needs to start at cephfs, otherwise cephfs 
won’t be aware of any changes happening underneath.

Perhaps you can force pg creation, so that 12,6,20 gets an empty PG to
start the pool again, and then hope that the next rsync will fill in
any missing slots, but this part I am not so sure about since I don't
know what other data apart from file contents may exist in a cephfs
pool.

Is the worst-case (dropping the pool, recreating it and running a full
rsync again) a possible way out? If so, you can perhaps test and see
if you can bridge the gap of the missing PGs, but if resyncing is out,
then wait for suggestions from someone more qualified at cephfs stuff
than me. ;)

I’ll wait a bit more for some other people to suggest something.  At this point 
I don’t have anything with high confidence that it will work.

Thanks!

George


--
May the most significant bit of your life be positive.

___
ceph-users mailing list -- 
ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Num objects: 18446744073709551603

2022-03-07 Thread Casey Bodley 
Eric fixed https://tracker.ceph.com/issues/51560 which caused an
underflow of rgw.none stats. i see that its backport tickets have
remained untouched for over 4 months now

On Mon, Mar 7, 2022 at 1:28 PM Paul Emmerich  wrote:
>
> Yeah, that happens occasionally, see https://tracker.ceph.com/issues/37942
>
> On Tue, Mar 1, 2022 at 1:00 PM Szabo, Istvan (Agoda) 
> wrote:
>
> > Hi,
> >
> > There is some bug which is related to this issue but I can't find it.
> > I have couple of buckets with this strange number where delete happens
> > often.
> > The workaround is to reshard the bucket to the same shard number that it
> > was before, BUT it comes back in 2-5 days.
> > What is weird if I reshard this bucket it can even speed up the cluster
> > operation not sure why.
> > Is there a way to make it permanent?
> >
> > FYI: I have multisite cluster, when I reshard it, I start the reshard on
> > the master zone and it will do it on the secondary zones HOWEVER I don't
> > stop rgw, because these buckets are not sync buckets, these buckets
> > individual buckets with individual objects in the secondary zone clusters.
> >
> > This is what I'm talking about (rgw.none num_objects):
> >
> > "rgw.none": {
> > "size": 0,
> > "size_actual": 0,
> > "size_utilized": 0,
> > "size_kb": 0,
> > "size_kb_actual": 0,
> > "size_kb_utilized": 0,
> > "num_objects": 18446744073709551603
> > },
> > "rgw.main": {
> > "size": 74813262451,
> > "size_actual": 74832678912,
> > "size_utilized": 74813262451,
> > "size_kb": 73059827,
> > "size_kb_actual": 73078788,
> > "size_kb_utilized": 73059827,
> > "num_objects": 8789
> > },
> > "rgw.multimeta": {
> > "size": 0,
> > "size_actual": 0,
> > "size_utilized": 0,
> > "size_kb": 0,
> > "size_kb_actual": 0,
> > "size_kb_utilized": 0,
> > "num_objects": 0
> > }
> >
> > Istvan Szabo
> > Senior Infrastructure Engineer
> > ---
> > Agoda Services Co., Ltd.
> > e: istvan.sz...@agoda.com
> > ---
> >
> >
> > 
> > This message is confidential and is for the sole use of the intended
> > recipient(s). It may also be privileged or otherwise protected by copyright
> > or other legal rules. If you have received it by mistake please let us know
> > by reply email and delete it from your system. It is prohibited to copy
> > this message or disclose its content to anyone. Any confidentiality or
> > privilege is not waived or lost by any mistaken delivery or unauthorized
> > disclosure of the message. All messages sent to and from Agoda may be
> > monitored to ensure compliance with company policies, to protect the
> > company's interests and to remove potential malware. Electronic messages
> > may be intercepted, amended, lost or deleted, or contain viruses.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: "Incomplete" pg's

2022-03-07 Thread Kyriazis, George 
After some thought, I decided to try “ceph pg force_create_pg” on the 
incomplete pgs, as suggested by name online sources.

However, I got:

# ceph pg force_create_pg 1.353
no valid command found; 10 closest matches:
pg stat
pg getmap
pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...]
pg dump_json [all|summary|sum|pools|osds|pgs...]
pg dump_pools_json
pg ls-by-pool  [...]
pg ls-by-primary http://osd.id>> [] [...]
pg ls-by-osd http://osd.id>> [] [...]
pg ls [] [...]
pg dump_stuck [inactive|unclean|stale|undersized|degraded...] []
Error EINVAL: invalid command
#

?

I am running pacific 16.2.7.

Thanks!

George


On Mar 4, 2022, at 7:51 AM, Kyriazis, George 
mailto:george.kyria...@intel.com>> wrote:

Thanks Janne,

(Inline)

On Mar 4, 2022, at 1:04 AM, Janne Johansson 
mailto:icepic...@gmail.com>> wrote:

Due to a mistake on my part, I accidentally destroyed more OSDs that I needed 
to, and I ended up with 2 pgs in “incomplete” state.

Doing “ceph pg query on one of the pgs that is incomplete, I get the following 
(somewhere in the output):

  "up": [
  12,
  6,
  20
  ],
  "acting": [
  12,
  6,
  20
  ],
  "avail_no_missing": [],
  "object_location_counts": [],
  "blocked_by": [
  3,
  4,
  5
  ],
  "up_primary": 12,
  "acting_primary": 12,
  "purged_snaps": []


I am assuming this means that OSDs 3,4,5 were the original ones (that are now 
destroyed), but I don’t understand why the output shows 12, 6, 20 as active.

I can't help with the cephfs part since we don't use that, but I think
the above output means "since 3,4,5 are gone, 12,6 and 20 are now
designated as the replacement OSDs to hold the PG", but since 3,4,5
are gone, none of them can backfill into 12,6,20, so 12,6,20 are
waiting for this PG to appear "somewhere" so they can recover.

I thought that if that was the case 3,4,5 should be listed as “active”, with 
12,6,20 as “up”..

My corcern about cephfs is that, since it is a layer above the ceph base layer, 
there maybe the corrective action needs to start at cephfs, otherwise cephfs 
won’t be aware of any changes happening underneath.

Perhaps you can force pg creation, so that 12,6,20 gets an empty PG to
start the pool again, and then hope that the next rsync will fill in
any missing slots, but this part I am not so sure about since I don't
know what other data apart from file contents may exist in a cephfs
pool.

Is the worst-case (dropping the pool, recreating it and running a full
rsync again) a possible way out? If so, you can perhaps test and see
if you can bridge the gap of the missing PGs, but if resyncing is out,
then wait for suggestions from someone more qualified at cephfs stuff
than me. ;)

I’ll wait a bit more for some other people to suggest something.  At this point 
I don’t have anything with high confidence that it will work.

Thanks!

George


--
May the most significant bit of your life be positive.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph-16.2.7 build fail

2022-03-07 Thread 杜承峻 
Hi all,

OS:centos8.5.2111

My steps are as follows：




./install-deps.sh




# add -pg for using gprofile

vim CMakeLists.txt 



SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pg")

SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -pg")

SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -pg")

SET(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -pg")




./do_cmake.sh  -DWITH_RDMA=ON  -DCMAKE_BUILD_TYPE=RelWithDebInfo




cd build

make -j8

make install




The error has occurred：

···

Installed /usr/local/lib/python3.8/site-packages/cephfs_top-0.0.1-py3.8.egg

Processing dependencies for cephfs-top==0.0.1

Searching for rados

Reading https://pypi.org/simple/rados/

Scanning index of all packages (this may take a while)

Couldn't find index page for 'rados' (maybe misspelled?)

Reading https://pypi.org/simple/

No local packages or working download links found for rados

error: Could not find suitable distribution for Requirement.parse('rados')

···




If I type 'ceph -v',I will get:



Traceback (most recent call last):

  File "/usr/local/bin/ceph", line 142, in

[ceph-users] Re: Ceph in kubernetes

2022-03-07 Thread ceph . novice 

Hi Hans.
 
any chance you could write up a blog, GitHub Gist, wiki,   to 
describe WHAT exactly you run and HOW... with (config) examples?!?
I wanted to also run the same kind of setup @ home, but hadn't the time to even 
start thinking / reading how to setup CEPH at home (OK, I had an CEPH 
"installation" bases on VirtualBox, Vagrant, Ansible, looong time ago on my 
"stand alone" Workstation, but ...)
 
Kind regards
 notna
 
 

Gesendet: Montag, 07. März 2022 um 10:43 Uhr
Von: "Hans van den Bogert" 
An: ceph-users@ceph.io
Betreff: [ceph-users] Re: Ceph in kubernetes
Just to add to warm fuzzy feeling, although just in a homelab, I'm using
rook for many years now, it's awesome. I trust* it with the family
photo's on a selfhosted nextcloud. All on K8s/Ceph/RGW.


Hans


* I have backups though ;)
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Retrieving cephx key from ceph-fuse

2022-03-07 Thread Dan van der Ster 
On Fri, Mar 4, 2022 at 2:07 PM Robert Vasek  wrote:
>
> Is there a way for an attacker with sufficient privileges to retrieve the
> key by somehow mining it off of the process memory of ceph-fuse which is
> now maintaining the volume mount?

Yes, one should assume that if they can gcore dump the ceph-fuse
process, they can extract the cephx key.
In normal deployment scenarios only a root user should be able to do
this, and they would normally have the key anyway.

Cheers, Dan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph in kubernetes

2022-03-07 Thread Hans van den Bogert 
Just to add to warm fuzzy feeling, although just in a homelab, I'm using 
rook for many years now, it's awesome. I trust* it with the family 
photo's on a selfhosted nextcloud. All on K8s/Ceph/RGW.



Hans


* I have backups though ;)

On 3/7/22 09:45, Bo Thorsen wrote:

Hi Nico and Janne,

Thank you very much for your quick answers. I did investigate rook 
already and had the feeling this might be the answer. But one of the 
things that is extremely hard is to find The Right Way (TM) to handle 
the setup I want. Getting the warm fuzzy feeling that I chose the proper 
way forward is difficult. You helped me do that, so thank you :)


Bo.

Den 07-03-2022 kl. 09:39 skrev Nico Schottelius:


Good morning Bo,

as Janne pointing out, rook is indeed a very good solution for k8s based
clusters.
We are even replacing our native Ceph clusters with rook, as it feels
smarter in a long term perspective than setting on cephadm.

Rook also allows to inject quite some parameters and while it is a bit
learning k8s in the beginning, it is sufficiently flexible to allow any
Ceph tuning that you'd have done without it.

Bo Thorsen  writes:


So here's my idea on how to solve this problem: Add a 1TB SSD in each
worker node and let ceph use these disks for storage.


This is probably best done using the auto capture of disks, which rook
supports by default.

Best regards,

Nico


--
Sustainable and modern Infrastructures by ungleich.ch

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Failed in ceph-osd -i ${osd_id} --mkfs -k /var/lib/ceph/osd/ceph-${osd_id}/keyring

2022-03-07 Thread Eugen Block 
I don't think that's an issue here unless you tried the same for osd.1  
but mixed up the error output from osd.3 and osd.1. But the error  
messages clearly state that there's no block device for osd.1 although  
you seem to try to create osd.3:


[...] bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label  
failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or  
directory


Can you clarify?

Zitat von huxia...@horebdata.cn:


thanks, Eugen.

I am suspecting this perhaps could be related to  
https://tracker.ceph.com/issues/42223





huxia...@horebdata.cn

From: Eugen Block
Date: 2022-03-05 09:52
To: huxiaoyu
CC: ceph-users
Subject: Re: [ceph-users] Failed in ceph-osd -i ${osd_id} --mkfs -k  
/var/lib/ceph/osd/ceph-${osd_id}/keyring

Hi, there must be some mixup of the OSD IDs. Your command seems to use
ID 3 but the log complains about ID 1. You should check your script
and workflow.

Zitat von huxia...@horebdata.cn:


Dear Ceph folks,

I encountered a strange behavior with Luminous 12.2.13, when running
the following
ceph-osd -i ${osd_id} --mkfs -k
/var/lib/ceph/osd/ceph-${osd_id}/keyring
to create OSD,

what could be the root cause? Any ideas or suggestions?

best regards,

Samuel


***Error log
messages as
below

+ ceph-osd -i 3 --mkkey
+ ceph-osd -i 3 --mkfs -k /var/lib/ceph/osd/ceph-3/keyring
--osd-uuid b9e67369-1d36-4d1d-9f81-1aa679d4c5bf
/var/lib/ceph-bin/cephosd.sh: line 451: 33918 Aborted
 (core dumped) ceph-osd -i "${OSD_ID}" --mkfs -k
"${OSD_DIR}"/keyring --osd-uuid "${OSD_UUID}" > ${log_file} 2>&1
+ '[' 134 -eq 0 ']'
+ echo osd mkfs fail
osd mkfs fail

   ${log_file}其中挂起的相关信息：
2022-03-05 05:53:40.450234 7f7368465f80 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb
2022-03-05 05:53:40.450356 7f7368465f80 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb
2022-03-05 05:53:40.451535 7f7368465f80 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb
2022-03-05 05:53:40.452372 7f7368465f80 -1
bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to
open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
2022-03-05 05:53:40.452390 7f7368465f80 -1
bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to
open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
2022-03-05 05:53:40.452395 7f7368465f80 -1
bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to
open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory
2022-03-05 05:53:40.460573 7f7368465f80 -1
bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In
function 'void AllocatorLevel02::_mark_allocated(uint64_t,
uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned
int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757:
FAILED assert(available >= allocated)
 ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e)
luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x559be313d662]
 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned
long)+0x685) [0x559be30ec865]
 3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
 4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
 5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
 6: (OSD::mkfs(CephContext*, ObjectStore*,
std::__cxx11::basic_string,
std::allocator > const&, uuid_d, int)+0x164) [0x559be2b1da24]
 7: (main()+0x11aa) [0x559be2a4424a]
 8: (__libc_start_main()+0xf0) [0x7f73656cb840]
 9: (_start()+0x29) [0x559be2ad3339]
 NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.
2022-03-05 05:53:40.489602 7f7368465f80 -1
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In
function 'void AllocatorLevel02::_mark_allocated(uint64_t,
uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned
int]' thread 7f7368465f80 time 2022-03-05 05:53:40.486908
/build/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757:
FAILED assert(available >= allocated)

 ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e)
luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x559be313d662]
 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned
long)+0x685) [0x559be30ec865]
 3: (BlueFS::mount()+0x3d1) [0x559be30b4bc1]
 4: (BlueStore::_open_db(bool)+0x16ac) [0x559be2fc157c]
 5: (BlueStore::mkfs()+0x1225) [0x559be2ff7215]
 6: (OSD::mkfs(CephContext*, ObjectStore*,
std::__cxx11::basic_string,
std::allocator > const&, uuid_d, int)+0x164) [0x559be2b1da24]
 7: (main()+0x11aa) [0x559be2a4424a]
 8: (__libc_start_main()+0xf0) [0x7f73656

[ceph-users] Re: Ceph in kubernetes

2022-03-07 Thread Janne Johansson 
Den mån 7 mars 2022 kl 09:45 skrev Bo Thorsen :
> Thank you very much for your quick answers. I did investigate rook
> already and had the feeling this might be the answer. But one of the
> things that is extremely hard is to find The Right Way (TM) to handle
> the setup I want. Getting the warm fuzzy feeling that I chose the proper
> way forward is difficult. You helped me do that, so thank you :)

Well, you will probably find me on the other side of the fence in
debates when it comes to containers and ceph, but since you are coming
from the container side and reaching out for ceph, I guess Rook is a
good choice to start with. There are things to learn about ceph and
things to learn about containerized ceph and lastly on containers
themselves, so if you are comfortable in your kubernetes env already,
then you are halfway done there.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: How often should I scrub the filesystem ?

2022-03-07 Thread Arnaud M 
Hello

Does anyone have some infos about filesystem scrubbing ? As it is a generic
thread (not specific to my cluster) I think the answers can help a lot of
admin :)

It would be awesome if someone can answer some of these questions. And
maybe complete the doc.

All the best

Arnaud

Le sam. 5 mars 2022 à 23:26, Arnaud M  a écrit :

> Hello to everyone :)
>
> Just some question about filesystem scrubbing
>
> In this documentation it is said that scrub will help admin check
> consistency of filesystem:
>
> https://docs.ceph.com/en/latest/cephfs/scrub/
>
> So my questions are:
>
> Is filesystem scrubbing mandatory ?
> How often should I scrub the whole filesystem (ie start at /)
> How often should I scrub ~mdsdir
> Should I set up a cronjob ?
> Is filesystem scrubbing considerated armless ? Even with recursive force
> repair ?
> Is there any chance for scrubbing to overload mds on a big file system (ie
> like find . -ls) ?
> What is the difference between "recursive repair" and "recursive force
> repair" ? Is "force" armless ?
> Is there any way to see at which file/folder is the scrub operation ? In
> fact any better way to see srub progress than "scrub status" which doesn't
> say much
>
> Sorry for all the questions, but there is not that much documentation
> about filesystem scrubbing. And I do think the answers will help a lot of
> cephfs administrators :)
>
> Thanks to all
>
> All the best
>
> Arnaud
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph in kubernetes

2022-03-07 Thread Anthony D'Atri 

Convergence by any other name:

Proxmox, VSAN, etc.

Sure, why not? Performance might not be stellar, but it sounds like you 
probably don’t need it to be.
Look into dedicating a couple of cores to Ceph if you can, to avoid contention 
with compute tasks.

How many nodes do you have?

> 
> I'm running a small kubernetes cluster in the office which is mostly used for 
> gitlab runners. But there are some internal services we would like to run in 
> it, which would require persistent storage.
> 
> So here's my idea on how to solve this problem: Add a 1TB SSD in each worker 
> node and let ceph use these disks for storage.
> 
> The nodes are all simple, they just have a single disk and does nothing other 
> than run kubernetes. So I can install sdb in each and have a completely 
> uniform setup which I hope will make installation simpler.
> 
> Does this setup make sense? I'm not worried about the amount of disk space in 
> the ceph cluster, it's going to be way more than I need. So it's more a 
> question of whether someone who truly understands ceph thinks this is a good 
> or a bad idea?
> 
> I would appreciate any thoughts you have about this idea for a setup.
> 
> Thank you,
> 
> Bo Thorsen.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io