[ceph-users] Any way can recover data?

2021-03-09 Thread Elians Wan 
we use  s3 api upload file to ceph cluster with non versioned bucket, and
we override many files then want to recover them, any way can recover?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] 2 Pgs (1x inconsistent, 1x unfound / degraded - unable to fix

2021-03-09 Thread Jeremi Avenant 
Good day

I'm currently decommissioning a cluster that runs EC3+1 (rack failure
domain - with 5 racks), however the cluster still has some production items
on it since I'm in the process of moving it to our new EC8+2 cluster.

Running Luminous 12.2.13 on Ubuntu 16 HWE, containerized with ceph-ansible
3.2.

I currently get the following error after we lost 1 OSD (195).

I'm forced to repair, scrub, deep scrub, restart OSDs etc, everything
mentioned in the troubleshooting docs & information from IRC but cannot for
the life of me get it to work.

What I'm seeing is, that pg 9.3dd (volume_images) has a status of 1 OSD
(osd is down) which I know, but the other OSD shows 316 (not queried). Also
pg 9.3dd has 4x functioning UP's but still reference the missing OSD under
acting.

Regards

OBJECT_UNFOUND 1/501815192 objects unfound (0.000%)
pg 9.3dd has 1 unfound objects
OSD_SCRUB_ERRORS 1 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 9.3d1 is active+clean+inconsistent, acting [347,316,307,249]
PG_DEGRADED Degraded data redundancy: 1219/2001837265 objects degraded
(0.000%), 1 pg degraded, 1 pg undersized
pg 9.3dd is stuck undersized for 55486.439002, current state
active+recovery_wait+forced_recovery+undersized+degraded+remapped, last
acting [355,2147483647,64,367]


ceph pg 9.3dd query
"up": [
355,
139,
64,
367
],
"acting": [
355,
2147483647,
64,
367

"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2021-03-08 16:07:51.239010",
"might_have_unfound": [
{
"osd": "64(2)",
"status": "already probed"
},
{
"osd": "139(1)",
"status": "already probed"
},
{
"osd": "195(1)",
"status": "osd is down"
},
{
"osd": "316(2)",
"status": "not queried"
},
{
"osd": "367(3)",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [
"139(1)"


ceph pg 9.3d1 query
{
"state": "active+clean+inconsistent",
"snap_trimq": "[]",
"snap_trimq_len": 0,
"epoch": 168488,
"up": [
347,
316,
307,
249
],
"acting": [
347,
316,
307,
249
],
"actingbackfill": [
"249(3)",
"307(2)",
"316(1)",
"347(0)"

-- 
CLUSTER STATS:

  cluster:
id: 1ea59fbe-46a4-474e-8225-a66b32ca86b7
health: HEALTH_ERR
1/490166525 objects unfound (0.000%)
1 scrub errors
Possible data damage: 1 pg inconsistent
Degraded data redundancy: 1259/1956055233 objects degraded
(0.000%), 1 pg degraded, 1 pg undersized

  services:
mon: 3 daemons, quorum B-04-11-cephctl,B-05-11-cephctl,B-03-11-cephctl
mgr: B-03-11-cephctl(active), standbys: B-04-11-cephctl, B-05-11-cephctl
mds: cephfs-1/1/1 up  {0=B-04-11-cephctl=up:active}, 2 up:standby
osd: 384 osds: 383 up, 383 in; 1 remapped pgs

  data:
pools:   11 pools, 13440 pgs
objects: 490.17M objects, 1.35PiB
usage:   1.88PiB used, 2.33PiB / 4.21PiB avail
pgs: 1259/1956055233 objects degraded (0.000%)
 1/490166525 objects unfound (0.000%)
 13332 active+clean
 96active+clean+scrubbing+deep
 10active+clean+scrubbing
 1 active+clean+inconsistent
 1
active+recovery_wait+forced_recovery+undersized+degraded+remapped



*Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist
Inter-University Institute for Data Intensive Astronomy
5th Floor, Department of Physics and Astronomy,
University of Cape Town

Tel: 021 959 4137 <0219592327>
Web: www.idia.ac.za 
E-mail (IDIA): jer...@idia.ac.za 
Rondebosch, Cape Town, 7600, South Africa
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ny way can recover data?

2021-03-09 Thread Elians Wan 
we use  s3 api upload file to ceph cluster with non versioned bucket, and
we override many files then want to recover them, any way can recover?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Cephfs metadata and MDS on same node

2021-03-09 Thread Jesper Lykkegaard Karlsen 
Dear Ceph’ers

I am about to upgrade MDS nodes for Cephfs in the Ceph cluster (erasure code 
8+3 ) I am administrating.

Since they will get plenty of memory and CPU cores, I was wondering if it would 
be a good idea to move metadata OSDs (NVMe's currently on OSD nodes together 
with cephfs_data ODS (HDD)) to the MDS nodes?

Configured as:

4 x MDS with each a metadata OSD and configured with 4 x replication

so each metadata OSD would have a complete copy of metadata.

I know MDS, stores al lot of metadata in RAM, but if metadata OSDs were on MDS 
nodes, would that not bring down latency?

Anyway, I am just asking for your opinion on this? Pros and cons or even better 
somebody who actually have tried this?

Best regards,
Jesper

--
Jesper Lykkegaard Karlsen
Scientific Computing
Centre for Structural Biology
Department of Molecular Biology and Genetics
Aarhus University
Gustav Wieds Vej 10
8000 Aarhus C

E-mail: je...@mbg.au.dk
Tlf:+45 50906203

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] OSD crashes create_aligned_in_mempool in 15.2.9 and 14.2.16

2021-03-09 Thread Andrej Filipcic 


Hi,

under heavy load our cluster is experiencing frequent OSD crashes. Is 
this a known bug or should I report it? Any workarounds? It looks to be 
highly correlated with memory tuning.


it happens with both nautilus 14.2.16 and octopus 15.2.9. I have forced 
the bitmap bluefs and bluestore allocator.


the cluster is ~60 nodes with 256GB ram and 25Gb NICs, ~1600 OSDs on 
100g network. Typically it is happening when the traffic is above 100GB/s.


Best regards,
Andrej


   -14> 2021-03-09T14:10:30.105+0100 7fc128e05700 10 monclient: tick
   -13> 2021-03-09T14:10:30.105+0100 7fc128e05700 10 monclient: 
_check_auth_rotating have uptodate secrets (they expire after 
2021-03-09T14:10:00.107344+0100)
   -12> 2021-03-09T14:10:30.210+0100 7fc119412700  5 osd.209 9539 
heartbeat osd_stat(store_statfs(0xe68762a/0x4000/0xe8d7fc0, 
data 0x24b195d923/0x24c97c, compress 0x0/0x0/0x0, omap 0xf5dca, meta 
0x3ff0a236), peers 
[6,8,11,21,22,23,24,25,26,27,28,29,34,35,37,38,65,86,87,90,120,128,129,135,136,140,150,153,154,160,184,188,192,193,203,208,210,217,229,233,242,248,254,25

6,275,277,282,290,311,313,324,326,331,339,348,369,409,411,413,466,477,532,535,538,539,542,544,546,548,552,554,556,558,561,576,580,600,601,604,612,614,624,625,631,657,689,695,704,717,738,739,740,766,790,810,812,833,839,890,891,895,903,909,916,926,927,946,960,965,991,1050,1055,1062,1064,1067,1069,1072,1073,1075,1077,1078,1079,1095,1100,1117,1125,1127,1141,1148,1149,1153,1155,1195,12
01,1202,1215,1229,1238,1253,1260,1283,1290,1298,1303,1329,1330,1349,1350,1388,1389,1422,1423,1430,1431,1434,1448,1455,1478,1479,1485,1488,1494,1497,1506,1516,1561,1564,1573,1574,1580] 
op hist [0,0,0,1,3,4,15,24,43,64,102,117])
   -11> 2021-03-09T14:10:30.468+0100 7fc137cc0700 10 monclient: 
handle_auth_request added challenge on 0x55c08becf800
   -10> 2021-03-09T14:10:30.543+0100 7fc1374bf700 10 monclient: 
handle_auth_request added challenge on 0x55c08becf400
    -9> 2021-03-09T14:10:30.712+0100 7fc1384c1700 10 monclient: 
handle_auth_request added challenge on 0x55c08becec00
    -8> 2021-03-09T14:10:31.029+0100 7fc137cc0700 10 monclient: 
handle_auth_request added challenge on 0x55c08becfc00
    -7> 2021-03-09T14:10:31.033+0100 7fc12ca33700  5 prioritycache 
tune_memory target: 7264711979 mapped: 1564606464 unmapped: 47874048 
heap: 1612480512 old mem: 5369698813 new mem: 5369698813

    -6> 2021-03-09T14:10:31.105+0100 7fc128e05700 10 monclient: tick
    -5> 2021-03-09T14:10:31.105+0100 7fc128e05700 10 monclient: 
_check_auth_rotating have uptodate secrets (they expire after 
2021-03-09T14:10:01.107451+0100)
    -4> 2021-03-09T14:10:31.574+0100 7fc1374bf700 10 monclient: 
handle_auth_request added challenge on 0x55c0b69a8000
    -3> 2021-03-09T14:10:32.036+0100 7fc12ca33700  5 prioritycache 
tune_memory target: 7264711979 mapped: 1708449792 unmapped: 46637056 
heap: 1755086848 old mem: 5369698813 new mem: 5369698813

    -2> 2021-03-09T14:10:32.106+0100 7fc128e05700 10 monclient: tick
    -1> 2021-03-09T14:10:32.106+0100 7fc128e05700 10 monclient: 
_check_auth_rotating have uptodate secrets (they expire after 
2021-03-09T14:10:02.107524+0100)
 0> 2021-03-09T14:10:32.661+0100 7fc1384c1700 -1 *** Caught signal 
(Aborted) **

 in thread 7fc1384c1700 thread_name:msgr-worker-0

 ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus 
(stable)

 1: (()+0x12b20) [0x7fc13cc20b20]
 2: (gsignal()+0x10f) [0x7fc13b8847ff]
 3: (abort()+0x127) [0x7fc13b86ec35]
 4: (()+0x9009b) [0x7fc13c23a09b]
 5: (()+0x9653c) [0x7fc13c24053c]
 6: (()+0x96597) [0x7fc13c240597]
 7: (()+0x967f8) [0x7fc13c2407f8]
 8: (ceph::buffer::v15_2_0::create_aligned_in_mempool(unsigned int, 
unsigned int, int)+0x229) [0x55c0669dce49]
 9: (ceph::buffer::v15_2_0::create_aligned(unsigned int, unsigned 
int)+0x26) [0x55c0669dcf46]
 10: (ceph::buffer::v15_2_0::create_small_page_aligned(unsigned 
int)+0x55) [0x55c0669dd8e5]

 11: (ProtocolV1::read_message_data_prepare()+0x368) [0x55c066b78ac8]
 12: (ProtocolV1::read_message_middle()+0x130) [0x55c066b78c90]
 13: (ProtocolV1::handle_message_front(char*, int)+0x2f4) [0x55c066b79624]
 14: (()+0xf72bed) [0x55c066b72bed]
 15: (AsyncConnection::process()+0x8a9) [0x55c066b6fa39]
 16: (EventCenter::process_events(unsigned int, 
std::chrono::duration 
>*)+0xcb7) [0x55c0669c41b7]

 17: (()+0xdc979c) [0x55c0669c979c]
 18: (()+0xc2ba3) [0x7fc13c26cba3]
 19: (()+0x814a) [0x7fc13cc1614a]
 20: (clone()+0x43) [0x7fc13b949f23]
 NOTE: a copy of the executable, or `objdump -rdS ` is 
needed to interpret this.



--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 rbd_rwl
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 immutable_obj_cache
   0/ 5 client
   1/ 5 osd
 

[ceph-users] Re: OSD crashes create_aligned_in_mempool in 15.2.9 and 14.2.16

2021-03-09 Thread Dan van der Ster 
Hi Andrej,

I wonder if this is another manifestation of the buggy gperftools-libs
v2.8 bug, e.g. https://tracker.ceph.com/issues/49618

If so, there is a fixed (downgraded) version in epel-testing now.

Cheers, Dan




On Tue, Mar 9, 2021 at 4:36 PM Andrej Filipcic  wrote:
>
>
> Hi,
>
> under heavy load our cluster is experiencing frequent OSD crashes. Is
> this a known bug or should I report it? Any workarounds? It looks to be
> highly correlated with memory tuning.
>
> it happens with both nautilus 14.2.16 and octopus 15.2.9. I have forced
> the bitmap bluefs and bluestore allocator.
>
> the cluster is ~60 nodes with 256GB ram and 25Gb NICs, ~1600 OSDs on
> 100g network. Typically it is happening when the traffic is above 100GB/s.
>
> Best regards,
> Andrej
>
>
> -14> 2021-03-09T14:10:30.105+0100 7fc128e05700 10 monclient: tick
> -13> 2021-03-09T14:10:30.105+0100 7fc128e05700 10 monclient:
> _check_auth_rotating have uptodate secrets (they expire after
> 2021-03-09T14:10:00.107344+0100)
> -12> 2021-03-09T14:10:30.210+0100 7fc119412700  5 osd.209 9539
> heartbeat osd_stat(store_statfs(0xe68762a/0x4000/0xe8d7fc0,
> data 0x24b195d923/0x24c97c, compress 0x0/0x0/0x0, omap 0xf5dca, meta
> 0x3ff0a236), peers
> [6,8,11,21,22,23,24,25,26,27,28,29,34,35,37,38,65,86,87,90,120,128,129,135,136,140,150,153,154,160,184,188,192,193,203,208,210,217,229,233,242,248,254,25
> 6,275,277,282,290,311,313,324,326,331,339,348,369,409,411,413,466,477,532,535,538,539,542,544,546,548,552,554,556,558,561,576,580,600,601,604,612,614,624,625,631,657,689,695,704,717,738,739,740,766,790,810,812,833,839,890,891,895,903,909,916,926,927,946,960,965,991,1050,1055,1062,1064,1067,1069,1072,1073,1075,1077,1078,1079,1095,1100,1117,1125,1127,1141,1148,1149,1153,1155,1195,12
> 01,1202,1215,1229,1238,1253,1260,1283,1290,1298,1303,1329,1330,1349,1350,1388,1389,1422,1423,1430,1431,1434,1448,1455,1478,1479,1485,1488,1494,1497,1506,1516,1561,1564,1573,1574,1580]
> op hist [0,0,0,1,3,4,15,24,43,64,102,117])
> -11> 2021-03-09T14:10:30.468+0100 7fc137cc0700 10 monclient:
> handle_auth_request added challenge on 0x55c08becf800
> -10> 2021-03-09T14:10:30.543+0100 7fc1374bf700 10 monclient:
> handle_auth_request added challenge on 0x55c08becf400
>  -9> 2021-03-09T14:10:30.712+0100 7fc1384c1700 10 monclient:
> handle_auth_request added challenge on 0x55c08becec00
>  -8> 2021-03-09T14:10:31.029+0100 7fc137cc0700 10 monclient:
> handle_auth_request added challenge on 0x55c08becfc00
>  -7> 2021-03-09T14:10:31.033+0100 7fc12ca33700  5 prioritycache
> tune_memory target: 7264711979 mapped: 1564606464 unmapped: 47874048
> heap: 1612480512 old mem: 5369698813 new mem: 5369698813
>  -6> 2021-03-09T14:10:31.105+0100 7fc128e05700 10 monclient: tick
>  -5> 2021-03-09T14:10:31.105+0100 7fc128e05700 10 monclient:
> _check_auth_rotating have uptodate secrets (they expire after
> 2021-03-09T14:10:01.107451+0100)
>  -4> 2021-03-09T14:10:31.574+0100 7fc1374bf700 10 monclient:
> handle_auth_request added challenge on 0x55c0b69a8000
>  -3> 2021-03-09T14:10:32.036+0100 7fc12ca33700  5 prioritycache
> tune_memory target: 7264711979 mapped: 1708449792 unmapped: 46637056
> heap: 1755086848 old mem: 5369698813 new mem: 5369698813
>  -2> 2021-03-09T14:10:32.106+0100 7fc128e05700 10 monclient: tick
>  -1> 2021-03-09T14:10:32.106+0100 7fc128e05700 10 monclient:
> _check_auth_rotating have uptodate secrets (they expire after
> 2021-03-09T14:10:02.107524+0100)
>   0> 2021-03-09T14:10:32.661+0100 7fc1384c1700 -1 *** Caught signal
> (Aborted) **
>   in thread 7fc1384c1700 thread_name:msgr-worker-0
>
>   ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus
> (stable)
>   1: (()+0x12b20) [0x7fc13cc20b20]
>   2: (gsignal()+0x10f) [0x7fc13b8847ff]
>   3: (abort()+0x127) [0x7fc13b86ec35]
>   4: (()+0x9009b) [0x7fc13c23a09b]
>   5: (()+0x9653c) [0x7fc13c24053c]
>   6: (()+0x96597) [0x7fc13c240597]
>   7: (()+0x967f8) [0x7fc13c2407f8]
>   8: (ceph::buffer::v15_2_0::create_aligned_in_mempool(unsigned int,
> unsigned int, int)+0x229) [0x55c0669dce49]
>   9: (ceph::buffer::v15_2_0::create_aligned(unsigned int, unsigned
> int)+0x26) [0x55c0669dcf46]
>   10: (ceph::buffer::v15_2_0::create_small_page_aligned(unsigned
> int)+0x55) [0x55c0669dd8e5]
>   11: (ProtocolV1::read_message_data_prepare()+0x368) [0x55c066b78ac8]
>   12: (ProtocolV1::read_message_middle()+0x130) [0x55c066b78c90]
>   13: (ProtocolV1::handle_message_front(char*, int)+0x2f4) [0x55c066b79624]
>   14: (()+0xf72bed) [0x55c066b72bed]
>   15: (AsyncConnection::process()+0x8a9) [0x55c066b6fa39]
>   16: (EventCenter::process_events(unsigned int,
> std::chrono::duration
>  >*)+0xcb7) [0x55c0669c41b7]
>   17: (()+0xdc979c) [0x55c0669c979c]
>   18: (()+0xc2ba3) [0x7fc13c26cba3]
>   19: (()+0x814a) [0x7fc13cc1614a]
>   20: (clone()+0x43) [0x7fc13b949f23]
>   NOTE: a copy of the executable, or `objdump -rdS ` is
> needed to interpret this.
>
>
> --- l

[ceph-users] Re: OSD crashes create_aligned_in_mempool in 15.2.9 and 14.2.16

2021-03-09 Thread Andrej Filipcic 



Hi,

I was checking that bug yesterday, yes, and it smells the same.

I will give a try to the epel one,

Thanks
Andrej

On 09/03/2021 16:44, Dan van der Ster wrote:

Hi Andrej,

I wonder if this is another manifestation of the buggy gperftools-libs
v2.8 bug, e.g. https://tracker.ceph.com/issues/49618

If so, there is a fixed (downgraded) version in epel-testing now.

Cheers, Dan




On Tue, Mar 9, 2021 at 4:36 PM Andrej Filipcic  wrote:


Hi,

under heavy load our cluster is experiencing frequent OSD crashes. Is
this a known bug or should I report it? Any workarounds? It looks to be
highly correlated with memory tuning.

it happens with both nautilus 14.2.16 and octopus 15.2.9. I have forced
the bitmap bluefs and bluestore allocator.

the cluster is ~60 nodes with 256GB ram and 25Gb NICs, ~1600 OSDs on
100g network. Typically it is happening when the traffic is above 100GB/s.

Best regards,
Andrej


 -14> 2021-03-09T14:10:30.105+0100 7fc128e05700 10 monclient: tick
 -13> 2021-03-09T14:10:30.105+0100 7fc128e05700 10 monclient:
_check_auth_rotating have uptodate secrets (they expire after
2021-03-09T14:10:00.107344+0100)
 -12> 2021-03-09T14:10:30.210+0100 7fc119412700  5 osd.209 9539
heartbeat osd_stat(store_statfs(0xe68762a/0x4000/0xe8d7fc0,
data 0x24b195d923/0x24c97c, compress 0x0/0x0/0x0, omap 0xf5dca, meta
0x3ff0a236), peers
[6,8,11,21,22,23,24,25,26,27,28,29,34,35,37,38,65,86,87,90,120,128,129,135,136,140,150,153,154,160,184,188,192,193,203,208,210,217,229,233,242,248,254,25
6,275,277,282,290,311,313,324,326,331,339,348,369,409,411,413,466,477,532,535,538,539,542,544,546,548,552,554,556,558,561,576,580,600,601,604,612,614,624,625,631,657,689,695,704,717,738,739,740,766,790,810,812,833,839,890,891,895,903,909,916,926,927,946,960,965,991,1050,1055,1062,1064,1067,1069,1072,1073,1075,1077,1078,1079,1095,1100,1117,1125,1127,1141,1148,1149,1153,1155,1195,12
01,1202,1215,1229,1238,1253,1260,1283,1290,1298,1303,1329,1330,1349,1350,1388,1389,1422,1423,1430,1431,1434,1448,1455,1478,1479,1485,1488,1494,1497,1506,1516,1561,1564,1573,1574,1580]
op hist [0,0,0,1,3,4,15,24,43,64,102,117])
 -11> 2021-03-09T14:10:30.468+0100 7fc137cc0700 10 monclient:
handle_auth_request added challenge on 0x55c08becf800
 -10> 2021-03-09T14:10:30.543+0100 7fc1374bf700 10 monclient:
handle_auth_request added challenge on 0x55c08becf400
  -9> 2021-03-09T14:10:30.712+0100 7fc1384c1700 10 monclient:
handle_auth_request added challenge on 0x55c08becec00
  -8> 2021-03-09T14:10:31.029+0100 7fc137cc0700 10 monclient:
handle_auth_request added challenge on 0x55c08becfc00
  -7> 2021-03-09T14:10:31.033+0100 7fc12ca33700  5 prioritycache
tune_memory target: 7264711979 mapped: 1564606464 unmapped: 47874048
heap: 1612480512 old mem: 5369698813 new mem: 5369698813
  -6> 2021-03-09T14:10:31.105+0100 7fc128e05700 10 monclient: tick
  -5> 2021-03-09T14:10:31.105+0100 7fc128e05700 10 monclient:
_check_auth_rotating have uptodate secrets (they expire after
2021-03-09T14:10:01.107451+0100)
  -4> 2021-03-09T14:10:31.574+0100 7fc1374bf700 10 monclient:
handle_auth_request added challenge on 0x55c0b69a8000
  -3> 2021-03-09T14:10:32.036+0100 7fc12ca33700  5 prioritycache
tune_memory target: 7264711979 mapped: 1708449792 unmapped: 46637056
heap: 1755086848 old mem: 5369698813 new mem: 5369698813
  -2> 2021-03-09T14:10:32.106+0100 7fc128e05700 10 monclient: tick
  -1> 2021-03-09T14:10:32.106+0100 7fc128e05700 10 monclient:
_check_auth_rotating have uptodate secrets (they expire after
2021-03-09T14:10:02.107524+0100)
   0> 2021-03-09T14:10:32.661+0100 7fc1384c1700 -1 *** Caught signal
(Aborted) **
   in thread 7fc1384c1700 thread_name:msgr-worker-0

   ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus
(stable)
   1: (()+0x12b20) [0x7fc13cc20b20]
   2: (gsignal()+0x10f) [0x7fc13b8847ff]
   3: (abort()+0x127) [0x7fc13b86ec35]
   4: (()+0x9009b) [0x7fc13c23a09b]
   5: (()+0x9653c) [0x7fc13c24053c]
   6: (()+0x96597) [0x7fc13c240597]
   7: (()+0x967f8) [0x7fc13c2407f8]
   8: (ceph::buffer::v15_2_0::create_aligned_in_mempool(unsigned int,
unsigned int, int)+0x229) [0x55c0669dce49]
   9: (ceph::buffer::v15_2_0::create_aligned(unsigned int, unsigned
int)+0x26) [0x55c0669dcf46]
   10: (ceph::buffer::v15_2_0::create_small_page_aligned(unsigned
int)+0x55) [0x55c0669dd8e5]
   11: (ProtocolV1::read_message_data_prepare()+0x368) [0x55c066b78ac8]
   12: (ProtocolV1::read_message_middle()+0x130) [0x55c066b78c90]
   13: (ProtocolV1::handle_message_front(char*, int)+0x2f4) [0x55c066b79624]
   14: (()+0xf72bed) [0x55c066b72bed]
   15: (AsyncConnection::process()+0x8a9) [0x55c066b6fa39]
   16: (EventCenter::process_events(unsigned int,
std::chrono::duration
  >*)+0xcb7) [0x55c0669c41b7]
   17: (()+0xdc979c) [0x55c0669c979c]
   18: (()+0xc2ba3) [0x7fc13c26cba3]
   19: (()+0x814a) [0x7fc13cc1614a]
   20: (clone()+0x43) [0x7fc13b949f23]
   NOTE: a copy of the executable, or `objdump -rd

[ceph-users] Re: Bluestore OSD crash with tcmalloc::allocate_full_cpp_throw_oom in multisite setup with PG_DAMAGED cluster error

2021-03-09 Thread David Orman 
For those who aren't on the bug tracker, this was brought up (and has
follow-up) here: https://tracker.ceph.com/issues/49618

On Thu, Mar 4, 2021 at 9:55 PM Szabo, Istvan (Agoda)
 wrote:
>
> Hi,
>
> I have a 3 DC multisite setup.
>
> The replication is directional like HKG->SGP->US so the bucket is replicated 
> from HKG to SGP and the same bucket is replicated further from SGP to US.
>
> The HKG > SGP connection is pretty fast 12.5millions objects (600GB) 
> transferred in 6.5 hours. Once the OSD crashed in SGP, it stopped the 
> complete chain replication and made PG_DAMAGED cluster error.
> The pg can be repaired but the sync never started back only bucket sync 
> disable/enable helped.
> I got OSD crash also in HKG BUT in ASH. ASH no any error, the replication 
> speed is 2 millions objects in 6.5 hours which is like 90GB of data.
>
> This is the crash of the osd:
>
> {
> "backtrace": [
> "(()+0x12b20) [0x7f597d3fbb20]",
> "(gsignal()+0x10f) [0x7f597c0667ff]",
> "(abort()+0x127) [0x7f597c050c35]",
> "(()+0x9009b) [0x7f597ca1c09b]",
> "(()+0x9653c) [0x7f597ca2253c]",
> "(()+0x96597) [0x7f597ca22597]",
> "(()+0x967f8) [0x7f597ca227f8]",
> "(()+0x19d24) [0x7f597e168d24]",
> "(tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x146) 
> [0x7f597e18b0d6]",
> "(rocksdb::Arena::AllocateNewBlock(unsigned long)+0x43) 
> [0x5632f08ccb93]",
> "(rocksdb::Arena::AllocateFallback(unsigned long, bool)+0x4b) 
> [0x5632f08ccc3b]",
> "(rocksdb::ConcurrentArena::AllocateAligned(unsigned long, unsigned 
> long, rocksdb::Logger*)+0xb4) [0x5632f07fae94]",
> "(()+0x1103e7e) [0x5632f085ae7e]",
> "(rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, 
> rocksdb::Slice const&, rocksdb::Slice const&, bool, 
> rocksdb::MemTablePostProcessInfo*)+0xcf) [0x5632f07f6f8f]",
> "(rocksdb::MemTableInserter::PutCFImpl(unsigned int, rocksdb::Slice 
> const&, rocksdb::Slice const&, rocksdb::ValueType)+0x452) [0x5632f08520e2]",
> "(rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice 
> const&, rocksdb::Slice const&)+0x17) [0x5632f0852e97]",
> "(rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) 
> const+0x480) [0x5632f084ac20]",
> 
> "(rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteThread::WriteGroup&, 
> unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, 
> bool, unsigned long, rocksdb::DB*, bool, bool, bool)+0x149) [0x5632f084ebe9]",
> "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, 
> rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, 
> bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1acd) 
> [0x5632f078a03d]",
> "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, 
> rocksdb::WriteBatch*)+0x21) [0x5632f078ac11]",
> "(RocksDBStore::submit_common(rocksdb::WriteOptions&, 
> std::shared_ptr)+0x8c) [0x5632f074180c]",
> 
> "(RocksDBStore::submit_transaction(std::shared_ptr)+0x87)
>  [0x5632f0742027]",
> "(BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)+0x426) 
> [0x5632f0226376]",
> "(BlueStore::_kv_sync_thread()+0x176f) [0x5632f024bc1f]",
> "(BlueStore::KVSyncThread::entry()+0x11) [0x5632f0273791]",
> "(()+0x814a) [0x7f597d3f114a]",
> "(clone()+0x43) [0x7f597c12bf23]"
> ],
> "ceph_version": "15.2.9",
> "crash_id": 
> "2021-03-04T14:55:45.094048Z_3d481fd3-7573-4cb7-9b22-20784b418e64",
> "entity_name": "osd.5",
> "os_id": "centos",
> "os_name": "CentOS Linux",
> "os_version": "8",
> "os_version_id": "8",
> "process_name": "ceph-osd",
> "stack_sig": 
> "9643c370a20c0d34f5e8965ae4461e2a7cf709ab4183929239bc263d0e1eef94",
> "timestamp": "2021-03-04T14:55:45.094048Z",
> "utsname_hostname": "hostname",
> "utsname_machine": "x86_64",
> "utsname_release": "4.18.0-240.10.1.el8_3.x86_64",
>"utsname_sysname": "Linux",
> "utsname_version": "#1 SMP Mon Jan 18 17:05:51 UTC 2021"
> }
>
> Any idea what I should tune?
>
> Thank you.
>
> 
> This message is confidential and is for the sole use of the intended 
> recipient(s). It may also be privileged or otherwise protected by copyright 
> or other legal rules. If you have received it by mistake please let us know 
> by reply email and delete it from your system. It is prohibited to copy this 
> message or disclose its content to anyone. Any confidentiality or privilege 
> is not waived or lost by any mistaken delivery or unauthorized disclosure of 
> the message. All messages sent to and from Agoda may be monitored to ensure 
> compliance with company policies, to protect the company's interests and to 
> remove potential malware. Electronic messages may be intercepted, amended, 
> lost or deleted, or contain viruses.
> ___
> ceph-use

[ceph-users] DocuBetter Meeting This Week -- 10 Mar 2021 1730 UTC

2021-03-09 Thread John Zachary Dover 
This week's meeting will focus on the ongoing rewrite of the cephadm
documentation and making certain that the documentation addresses the rough
edges in the Pacific release.

Meeting: https://bluejeans.com/908675367
Etherpad: https://pad.ceph.com/p/Ceph_Documentation
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSD crashes create_aligned_in_mempool in 15.2.9 and 14.2.16

2021-03-09 Thread Andrej Filipcic 


just confirming, crashes are gone with gperftools-libs-2.7-8.el8.x86_64.rpm

Cheers,
Andrej

On 09/03/2021 16:52, Andrej Filipcic wrote:


Hi,

I was checking that bug yesterday, yes, and it smells the same.

I will give a try to the epel one,

Thanks
Andrej

On 09/03/2021 16:44, Dan van der Ster wrote:

Hi Andrej,

I wonder if this is another manifestation of the buggy gperftools-libs
v2.8 bug, e.g. https://tracker.ceph.com/issues/49618

If so, there is a fixed (downgraded) version in epel-testing now.

Cheers, Dan




On Tue, Mar 9, 2021 at 4:36 PM Andrej Filipcic 
 wrote:


Hi,

under heavy load our cluster is experiencing frequent OSD crashes. Is
this a known bug or should I report it? Any workarounds? It looks to be
highly correlated with memory tuning.

it happens with both nautilus 14.2.16 and octopus 15.2.9. I have forced
the bitmap bluefs and bluestore allocator.

the cluster is ~60 nodes with 256GB ram and 25Gb NICs, ~1600 OSDs on
100g network. Typically it is happening when the traffic is above 
100GB/s.


Best regards,
Andrej


 -14> 2021-03-09T14:10:30.105+0100 7fc128e05700 10 monclient: tick
 -13> 2021-03-09T14:10:30.105+0100 7fc128e05700 10 monclient:
_check_auth_rotating have uptodate secrets (they expire after
2021-03-09T14:10:00.107344+0100)
 -12> 2021-03-09T14:10:30.210+0100 7fc119412700  5 osd.209 9539
heartbeat osd_stat(store_statfs(0xe68762a/0x4000/0xe8d7fc0,
data 0x24b195d923/0x24c97c, compress 0x0/0x0/0x0, omap 0xf5dca, 
meta

0x3ff0a236), peers
[6,8,11,21,22,23,24,25,26,27,28,29,34,35,37,38,65,86,87,90,120,128,129,135,136,140,150,153,154,160,184,188,192,193,203,208,210,217,229,233,242,248,254,25 

6,275,277,282,290,311,313,324,326,331,339,348,369,409,411,413,466,477,532,535,538,539,542,544,546,548,552,554,556,558,561,576,580,600,601,604,612,614,624,625,631,657,689,695,704,717,738,739,740,766,790,810,812,833,839,890,891,895,903,909,916,926,927,946,960,965,991,1050,1055,1062,1064,1067,1069,1072,1073,1075,1077,1078,1079,1095,1100,1117,1125,1127,1141,1148,1149,1153,1155,1195,12 

01,1202,1215,1229,1238,1253,1260,1283,1290,1298,1303,1329,1330,1349,1350,1388,1389,1422,1423,1430,1431,1434,1448,1455,1478,1479,1485,1488,1494,1497,1506,1516,1561,1564,1573,1574,1580] 


op hist [0,0,0,1,3,4,15,24,43,64,102,117])
 -11> 2021-03-09T14:10:30.468+0100 7fc137cc0700 10 monclient:
handle_auth_request added challenge on 0x55c08becf800
 -10> 2021-03-09T14:10:30.543+0100 7fc1374bf700 10 monclient:
handle_auth_request added challenge on 0x55c08becf400
  -9> 2021-03-09T14:10:30.712+0100 7fc1384c1700 10 monclient:
handle_auth_request added challenge on 0x55c08becec00
  -8> 2021-03-09T14:10:31.029+0100 7fc137cc0700 10 monclient:
handle_auth_request added challenge on 0x55c08becfc00
  -7> 2021-03-09T14:10:31.033+0100 7fc12ca33700  5 prioritycache
tune_memory target: 7264711979 mapped: 1564606464 unmapped: 47874048
heap: 1612480512 old mem: 5369698813 new mem: 5369698813
  -6> 2021-03-09T14:10:31.105+0100 7fc128e05700 10 monclient: tick
  -5> 2021-03-09T14:10:31.105+0100 7fc128e05700 10 monclient:
_check_auth_rotating have uptodate secrets (they expire after
2021-03-09T14:10:01.107451+0100)
  -4> 2021-03-09T14:10:31.574+0100 7fc1374bf700 10 monclient:
handle_auth_request added challenge on 0x55c0b69a8000
  -3> 2021-03-09T14:10:32.036+0100 7fc12ca33700  5 prioritycache
tune_memory target: 7264711979 mapped: 1708449792 unmapped: 46637056
heap: 1755086848 old mem: 5369698813 new mem: 5369698813
  -2> 2021-03-09T14:10:32.106+0100 7fc128e05700 10 monclient: tick
  -1> 2021-03-09T14:10:32.106+0100 7fc128e05700 10 monclient:
_check_auth_rotating have uptodate secrets (they expire after
2021-03-09T14:10:02.107524+0100)
   0> 2021-03-09T14:10:32.661+0100 7fc1384c1700 -1 *** Caught 
signal

(Aborted) **
   in thread 7fc1384c1700 thread_name:msgr-worker-0

   ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) 
octopus

(stable)
   1: (()+0x12b20) [0x7fc13cc20b20]
   2: (gsignal()+0x10f) [0x7fc13b8847ff]
   3: (abort()+0x127) [0x7fc13b86ec35]
   4: (()+0x9009b) [0x7fc13c23a09b]
   5: (()+0x9653c) [0x7fc13c24053c]
   6: (()+0x96597) [0x7fc13c240597]
   7: (()+0x967f8) [0x7fc13c2407f8]
   8: (ceph::buffer::v15_2_0::create_aligned_in_mempool(unsigned int,
unsigned int, int)+0x229) [0x55c0669dce49]
   9: (ceph::buffer::v15_2_0::create_aligned(unsigned int, unsigned
int)+0x26) [0x55c0669dcf46]
   10: (ceph::buffer::v15_2_0::create_small_page_aligned(unsigned
int)+0x55) [0x55c0669dd8e5]
   11: (ProtocolV1::read_message_data_prepare()+0x368) [0x55c066b78ac8]
   12: (ProtocolV1::read_message_middle()+0x130) [0x55c066b78c90]
   13: (ProtocolV1::handle_message_front(char*, int)+0x2f4) 
[0x55c066b79624]

   14: (()+0xf72bed) [0x55c066b72bed]
   15: (AsyncConnection::process()+0x8a9) [0x55c066b6fa39]
   16: (EventCenter::process_events(unsigned int,
std::chrono::duration
  >*)+0xcb7) [0x55c0669c41b7]
   17: (()+0xdc979c) [0x55c0669c979c]
   18:

[ceph-users] Replacing disk with xfs on it, documentation?

2021-03-09 Thread Drew Weaver 
Hello,

I haven't needed to replace a disk in awhile and it seems that I have misplaced 
my quick little guide on how to do it.

When searching the docs it is now recommending that you should use ceph-volume 
to create OSDs when doing that it creates LV:

Disk /dev/sde: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk 
/dev/mapper/ceph--34b5a0a9--f84a--416f--8b74--fb1e05161f80-osd--block--4581caf4--eef0--42e1--b237--c114dfde3d15:
 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Cool but all of my other OSDs look like this and appear to just be XFS.

Disk /dev/sdd: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
Disk identifier: D0195044-10B5-4113-8210-A5CFCB9213A2


# Start  EndSize  TypeName
1 2048   206847100M  Ceph OSDceph data
2   206848   78129397423.7T  unknown ceph block

35075 ?S< 0:00 [xfs-buf/sdd1]
  35076 ?S< 0:00 [xfs-data/sdd1]
  35077 ?S< 0:00 [xfs-conv/sdd1]
  35078 ?S< 0:00 [xfs-cil/sdd1]
  35079 ?S< 0:00 [xfs-reclaim/sdd]
  35080 ?S< 0:00 [xfs-log/sdd1]
  35082 ?S  0:00 [xfsaild/sdd1]

I can't seem to find the instructions in order to create an OSD like the 
original ones anymore.

I'm pretty sure that when this cluster was setup it was setup using the:

ceph-deploy osd prepare
ceph-deploy osd create
ceph-deploy osd activate

commands, but it seems as though the prepare and activate commands have since 
been removed from ceph-deploy, so I am really confused. =)

Does anyone by chance have the instructions for replacing a failed drive when 
it meets the above criteria?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Rados gateway basic pools missing

2021-03-09 Thread St-Germain, Sylvain (SSC/SPC) 
Hi everyone,

I just rebuild a (test) cluster using :

OS : Ubuntu 20.04.2 LTS
CEPH : ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus 
(stable)
3 nodes : monitor/storage


1.  The cluster looks good :

# ceph -s
cluster:
id: 9a89aa5a-1702-4f87-a99c-f94c9f2cdabd
health: HEALTH_OK

  services:
mon: 3 daemons, quorum dao-wkr-04,dao-wkr-05,dao-wkr-06 (age 7m)
mgr: dao-wkr-05(active, since 8m), standbys: dao-wkr-04, dao-wkr-06
mds: cephfs:1 {0=dao-wkr-04=up:active} 2 up:standby
osd: 9 osds: 9 up (since 7m), 9 in (since 4h)
rgw: 3 daemons active (dao-wkr-04.rgw0, dao-wkr-05.rgw0, dao-wkr-06.rgw0)

  task status:

  data:
pools:   7 pools, 121 pgs
objects: 234 objects, 16 KiB
usage:   9.0 GiB used, 2.0 TiB / 2.0 TiB avail
pgs: 121 active+clean


2.  except that the main pools for the radosgw are not there

# sudo ceph osd lspools

1 device_health_metrics
2 cephfs_data
3 cephfs_metadata
4 .rgw.root
5 default.rgw.log
6 default.rgw.control
7 default.rgw.meta


Missing : default.rgw.buckets.index  & default.rgw.buckets.data

What do you think ?
Thx !

Sylvain


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Rados gateway basic pools missing

2021-03-09 Thread St-Germain, Sylvain (SSC/SPC) 
Ok in the interface when I create a bucket the index in created automatically

1 device_health_metrics
2 cephfs_data
3 cephfs_metadata
4 .rgw.root
5 default.rgw.log
6 default.rgw.control
7 default.rgw.meta
8 default.rgw.buckets.index


* I think I just could not make an insertion using s3cmd

List command - connection problem
# s3cmd la 

!
An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
https://github.com/s3tools/s3cmd
  and have a look at the known issues list:

https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   s3tools-b...@lists.sourceforge.net


!

Invoked as: /usr/bin/s3cmd la
Problem: 
rc = main()
  File "/usr/bin/s3cmd", line 3001, in main
rc = cmd_func(args)
  File "/usr/bin/s3cmd", line 164, in cmd_all_buckets_list_all_content
response = s3.list_all_buckets()
  File "/usr/lib/python3/dist-packages/S3/S3.py", line 302, in list_all_buckets
response = self.send_request(request)
  File "/usr/lib/python3/dist-packages/S3/S3.py", line 1258, in send_request
conn = ConnMan.get(self.get_hostname(resource['bucket']))
  File "/usr/lib/python3/dist-packages/S3/ConnMan.py", line 253, in get
conn.c.connect()
  File "/usr/lib/python3.8/http/client.py", line 921, in connect
self.sock = self._create_connection(
  File "/usr/lib/python3.8/socket.py", line 808, in create_connection
raise err
  File "/usr/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

!
An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
https://github.com/s3tools/s3cmd
  and have a look at the known issues list:

https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   s3tools-b...@lists.sourceforge.net
!


-Message d'origine-
De : St-Germain, Sylvain (SSC/SPC)  
Envoyé : 9 mars 2021 17:19
À : ceph-users@ceph.io
Objet : [ceph-users] Rados gateway basic pools missing

Hi everyone,

I just rebuild a (test) cluster using :

OS : Ubuntu 20.04.2 LTS
CEPH : ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus 
(stable)
3 nodes : monitor/storage


1.  The cluster looks good :

# ceph -s
cluster:
id: 9a89aa5a-1702-4f87-a99c-f94c9f2cdabd
health: HEALTH_OK

  services:
mon: 3 daemons, quorum dao-wkr-04,dao-wkr-05,dao-wkr-06 (age 7m)
mgr: dao-wkr-05(active, since 8m), standbys: dao-wkr-04, dao-wkr-06
mds: cephfs:1 {0=dao-wkr-04=up:active} 2 up:standby
osd: 9 osds: 9 up (since 7m), 9 in (since 4h)
rgw: 3 daemons active (dao-wkr-04.rgw0, dao-wkr-05.rgw0, dao-wkr-06.rgw0)

  task status:

  data:
pools:   7 pools, 121 pgs
objects: 234 objects, 16 KiB
usage:   9.0 GiB used, 2.0 TiB / 2.0 TiB avail
pgs: 121 active+clean


2.  except that the main pools for the radosgw are not there

# sudo ceph osd lspools

1 device_health_metrics
2 cephfs_data
3 cephfs_metadata
4 .rgw.root
5 default.rgw.log
6 default.rgw.control
7 default.rgw.meta


Missing : default.rgw.buckets.index  & default.rgw.buckets.data

What do you think ?
Thx !

Sylvain


___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] node down pg with backfill_wait waiting for incomplete?

2021-03-09 Thread Marc 


I have a node down and pg's are remapping/backfilling. I have also a lot of 
pg's in backfill_wait. 

I was wondering if there is a specific order that this is being executed. Eg I 
have a large'garbage' pool ec21 that is stuck. I could resolve that by changing 
the min size. However I rather have this not remapped, and wait for this node 
to be back on line. 

Question is: is other remapping/backfilling waiting for this stuck to be fixed, 
or is backfilling/remapping so smart that it will do what ever it can?






- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. 
F1 Outsourcing Development Sp. z o.o.
Poland 

t:  +48 (0)12 4466 845
f:  +48 (0)12 4466 843
e:  m...@f1-outsourcing.eu


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] PG stuck at active+clean+remapped

2021-03-09 Thread Michael Fladischer 

Hi,

we have replaced some of our OSDs a while ago an while everything 
recovery as planned, one PG is still stuck at active+clean+remapped with 
no backfilling taking place.


Mpaaing the PG in question shows me that one OSD is missing:

$ ceph pg map 35.1fe
osdmap e1265760 pg 35.1fe (35.1fe) -> up 
[97,190,65,23,393,223,2147483647,354,132] acting 
[97,190,65,23,393,223,112,354,132]


It seems that osd.112 should be replaced with an other OSD and I suspect 
that CRUSH cannot find a suitable one.


Pool 35 is EC and has k=7 and m=2 and our Cluster has 9 OSD nodes. Is 
this just a case of CRUSH giving up to early as described in the 
troubleshooting PGs section[0] of the docs? Running the test as 
described there using `crushtool` gives several bad mapping rule results 
for "--num-rep 9".


If so, would it help to just add new OSDs to the existing hosts or would 
it be better to add a whole new OSD host?


Are there other options (upmap) to force this single PG to use a 
different set of OSDs for its "up" map?


[0] 
https://github.com/ceph/ceph/blob/master/doc/rados/troubleshooting/troubleshooting-pg.rst#crush-gives-up-too-soon


Thanks,
Michael
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph User Survey Now Available

2021-03-09 Thread Mike Perez 
Hi everyone,

We are approaching our deadline of April 2nd for the Ceph User Survey
to be filled out.

Thank you to everyone for the feedback so far. Please send further
feedback for this survey here:

https://pad.ceph.com/p/user-survey-2021-feedback

On Tue, Feb 16, 2021 at 2:20 PM Mike Perez  wrote:
>
> Hi everyone!
>
> Be sure to make your voice heard by taking the Ceph User Survey before
> April 2, 2021. This information will help guide the Ceph community’s
> investment in Ceph and the Ceph community's future development.
>
> https://ceph.io/user-survey/
>
> Thank you to the Ceph User Survey Working Group for designing this
> year's survey!
>
> * Anantha Adiga
> * Anthony D'Atri
> * Paul Mezzanini
> * Stefan Kooman
>
> https://tracker.ceph.com/projects/ceph/wiki/User_Survey_Working_Group
>
> We will provide the final results in the coming months after the
> survey has ended.
>
> --
> Mike Perez
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: February 2021 Tech Talk and Code Walk-through

2021-03-09 Thread Mike Perez 
Hi everyone,

In case you missed the Ceph Tech Talk and Code Walkthrough for
February, the recordings are now available to watch:

Jason Dillaman | Librbd Part 2: https://www.youtube.com/watch?v=nVjYVmqNClM
Sage Weil | What's New In the Pacific Release: https://youtu.be/PVtn53MbxTc

On Tue, Feb 16, 2021 at 11:14 PM Mike Perez  wrote:
>
> Hi everyone!
>
> I'm excited to announce two talks we have on the schedule for February 2021:
>
> Jason Dillaman will be giving part 2 to the librbd code walk-through.
>
> The stream starts on February 23rd at 18:00 UTC / 19:00 CET / 1:00 PM
> EST / 10:00 AM PST
>
> https://tracker.ceph.com/projects/ceph/wiki/Code_Walkthroughs
>
> Part 1: https://www.youtube.com/watch?v=L0x61HpREy4
>
> --
>
> What's New in the Pacific Release
>
> Hear Sage Weil give a live update on the development of the Pacific Release.
>
> The stream starts on February 25th at 17:00 UTC / 18:00 CET / 12 PM
> EST / 9 AM PST.
>
> https://ceph.io/ceph-tech-talks/
>
> All live streams will be recorded and
>
> --
> Mike Perez
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph pool with a whitespace as name

2021-03-09 Thread Boris Behrens 
Good morning ceph people,

I have a pool that got a whitespace as name. And I want to know what
creates the pool.
I already renamed it, but something recreates the pool.

Is there a way to find out what created the pool and what the content ist?

When I checked it's content I get
[root@s3db1 ~]# rados -p " " ls
.meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
.meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86469149.1766:_Feueqyht8adm09IVoqhR7Sh:1
.meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86450896.1831:_rU77PL7UJkTlOovtCWkvnfP:1
...
[root@s3db1 ~]# rados -p " " listxattr
.meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
ceph.objclass.version
user.rgw.acl
[root@s3db1 ~]# rados -p " " stat
.meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
 
/.meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
mtime 2021-03-10 06:27:43.00, size 317


-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Openstack rbd image Error deleting problem

2021-03-09 Thread Norman.Kern 
Hi Guys,

I have used Ceph rbd for Openstack for sometime, I met a problem while 
destroying a VM. The Openstack tried to

delete rbd image but failed. I have a test deleting a image by rbd command, it 
costs lots of time(image size 512G or more).

Anyone met the same problem with me? 

Thanks,

Norman
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Openstack rbd image Error deleting problem

2021-03-09 Thread Konstantin Shalygin 


> On 10 Mar 2021, at 09:50, Norman.Kern  wrote:
> 
> I have used Ceph rbd for Openstack for sometime, I met a problem while 
> destroying a VM. The Openstack tried to
> 
> delete rbd image but failed. I have a test deleting a image by rbd command, 
> it costs lots of time(image size 512G or more).
> 
> Anyone met the same problem with me? 

Object-map feature is enabled for your RBD's?



k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph pool with a whitespace as name

2021-03-09 Thread Boris Behrens 
Found it.
[root@s3db1 ~]# radosgw-admin zone get --rgw-zone=eu-central-1
{
"id": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad",
"name": "eu-central-1",
...SNIP...,
"metadata_heap": " ",
"realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef"
}


Am Mi., 10. März 2021 um 07:37 Uhr schrieb Boris Behrens :

> Good morning ceph people,
>
> I have a pool that got a whitespace as name. And I want to know what
> creates the pool.
> I already renamed it, but something recreates the pool.
>
> Is there a way to find out what created the pool and what the content ist?
>
> When I checked it's content I get
> [root@s3db1 ~]# rados -p " " ls
>
> .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
>
> .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86469149.1766:_Feueqyht8adm09IVoqhR7Sh:1
>
> .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86450896.1831:_rU77PL7UJkTlOovtCWkvnfP:1
> ...
> [root@s3db1 ~]# rados -p " " listxattr
> .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
> ceph.objclass.version
> user.rgw.acl
> [root@s3db1 ~]# rados -p " " stat
> .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
>  
> /.meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
> mtime 2021-03-10 06:27:43.00, size 317
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
>


-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Many OSD marked down after no beacon for XXX seconds, just becauseone MON's OS disk was blocked.

2021-03-09 Thread 912273...@qq.com 

Hello，everyone,

In the OS disk blocked  scene, the Mon service is still running ,but cant't 
work normally,
soon the mon will out of quorum, but some OSD were still markd down after  
mon_osd_report_timeout*2  seconds,
which will cause the cluster unavailable.
At this time， the OS is very slow，may be unable to operate or login，so it‘s 
impossible to kill the mon by hand or other service. 
This problem exist in L and N version.
Any suggestion？

 you can reproduce this problem as follow:
1、 block the OS disk use follow command.
# echo blocked > /sys/block/sdx/device/state
2、a few seconds later， the mon will out of quorum
3、after mon_osd_report_timeout*2 seconds，some osds will be markd down.
 （mon_osd_report_timeout default value 900s，you can set a smaller value）



912273...@qq.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph pool with a whitespace as name

2021-03-09 Thread Boris Behrens 
Ok,
i changed the value to
"metadata_heap": "",
but it is still used.

Any ideas how to stop this?

Am Mi., 10. März 2021 um 08:14 Uhr schrieb Boris Behrens :

> Found it.
> [root@s3db1 ~]# radosgw-admin zone get --rgw-zone=eu-central-1
> {
> "id": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad",
> "name": "eu-central-1",
> ...SNIP...,
> "metadata_heap": " ",
> "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef"
> }
>
>
> Am Mi., 10. März 2021 um 07:37 Uhr schrieb Boris Behrens :
>
>> Good morning ceph people,
>>
>> I have a pool that got a whitespace as name. And I want to know what
>> creates the pool.
>> I already renamed it, but something recreates the pool.
>>
>> Is there a way to find out what created the pool and what the content ist?
>>
>> When I checked it's content I get
>> [root@s3db1 ~]# rados -p " " ls
>>
>> .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
>>
>> .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86469149.1766:_Feueqyht8adm09IVoqhR7Sh:1
>>
>> .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86450896.1831:_rU77PL7UJkTlOovtCWkvnfP:1
>> ...
>> [root@s3db1 ~]# rados -p " " listxattr
>> .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
>> ceph.objclass.version
>> user.rgw.acl
>> [root@s3db1 ~]# rados -p " " stat
>> .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
>>  
>> /.meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1
>> mtime 2021-03-10 06:27:43.00, size 317
>>
>>
>> --
>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
>> groüen Saal.
>>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
>


-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io