[ceph-users] rados rm objects, still appear in rados ls

[Frank (lists)]

Hi,

On my cluster I tried to clear all objects from a pool. I used the 
command "rados -p bench ls | xargs rados -p bench rm". (rados -p bench 
cleanup doesn't clean everything, because there was a lot of other 
testing going on here).


Now 'rados -p bench ls' returns a list of objects, which don't exists: 
[root@ceph01 yum.repos.d]# rados -p bench stat 
benchmark_data_ceph01.example.com_1805226_object32453
 error stat-ing 
bench/benchmark_data_ceph01.example.com_1805226_object32453: (2) No such 
file or directory


I've tried scrub and deepscrub the pg the object is in, but the problem 
persists. What causes this?


I use Centos 7.5 with mimic 13.2.2


regards,

Frank de Bot

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Expected performane with Ceph iSCSI gateway

[Frank (lists)]

Hi,

I an test cluster (3 nodes, 24 osd's) I'm testing the ceph iscsi gateway 
(with http://docs.ceph.com/docs/master/rbd/iscsi-targets/). For a client 
I used a seperate server, everything runs Centos 7.5. The iscsi gateway 
are located on 2 of the existing nodes in the cluster.


How does iscsi perform compared to krbd? I've already did some 
benchmarking, but it didn't performed any near what krbd is doing. krbd 
easily saturates  the public netwerk, iscsi about 75%. Tmcu-runner is 
running during a benchmark at a load of 50 to 75% on the (owner)target



Regards,

Frank de Bot
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Frequent slow requests

[Frank (lists)]

Hi,

On a small cluster (3 nodes) I frequently have slow requests. When 
dumping the inflight ops from the hanging OSD, it seems it doesn't get a 
'response' for one of the subops. The events always look like:


    "events": [
    {
    "time": "2018-06-14 07:10:07.256196",
    "event": "initiated"
    },
    {
    "time": "2018-06-14 07:10:07.256671",
    "event": "queued_for_pg"
    },
    {
    "time": "2018-06-14 07:10:07.256745",
    "event": "reached_pg"
    },
    {
    "time": "2018-06-14 07:10:07.256826",
    "event": "started"
    },
    {
    "time": "2018-06-14 07:10:07.256924",
    "event": "waiting for subops from 18,20"
    },
    {
    "time": "2018-06-14 07:10:07.263769",
    "event": "op_commit"
    },
    {
    "time": "2018-06-14 07:10:07.263775",
    "event": "op_applied"
    },
    {
    "time": "2018-06-14 07:10:07.269989",
    "event": "sub_op_commit_rec from 18"
    }
             ]

The OSD id's are not the same. Looking at osd.20, the OSD process runs, 
it accepts requests ('ceph tell osd.20 bench' runs fine). When I restart 
the process for the OSD, the requests is completed.
I could not find any pattern on which OSD is too blame (always an other 
one) or one of the servers, it's also differs.


The cluster runs Ceph 7.5 with 'ceph version 12.2.5 
(cad919881333ac92274171586c827e01f554a70a) luminous (stable)'. It's just 
a testcluster with very little activity. What could be a cause of an 
(replica)OSD not replying?


Regards,

Frank de Bot

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] FreeBSD Initiator with Ceph iscsi

[Frank (lists)]

Jason Dillaman wrote:
Conceptually, I would assume it should just work if configured 
correctly w/ multipath (to properly configure the ALUA settings on the 
LUNs). I don't run FreeBSD, but any particular issue you are seeing?


When logged in to both targets,  the following message floods the log

WARNING: 192.168.5.109 (iqn.2018-06.lan.x.iscsi-gw:ceph-igw): underflow 
mismatch: target indicates 0, we calculated 512

(da1:iscsi6:0:0:0): READ(10). CDB: 28 00 0c 7f ff ff 00 00 01 00
(da1:iscsi6:0:0:0): CAM status: SCSI Status Error
(da1:iscsi6:0:0:0): SCSI status: Check Condition
(da1:iscsi6:0:0:0): SCSI sense: NOT READY asc:4,b (Logical unit not 
accessible, target port in standby state)

(da1:iscsi6:0:0:0): Error 6, Unretryable error
(da1:iscsi6:0:0:0): Invalidating pack

For both sessions the message are the same (besides numbering of devices)

When trying to read from either of the devices (da1 and da2 in my case), 
FreeBSD gives the error 'Device not configured'. When using gmultipath, 
manually created, because FreeBSD is not able to write a label to either 
of the devices, the created multipath is not functional because it 
markes both devices as FAIL






On Tue, Jun 26, 2018 at 6:06 PM Frank de Bot (lists) 
mailto:li...@searchy.net>> wrote:


Hi,

In my test setup I have a ceph iscsi gateway (configured as in
http://docs.ceph.com/docs/luminous/rbd/iscsi-overview/ )

I would like to use thie with a FreeBSD (11.1) initiator, but I
fail to
make a working setup in FreeBSD. Is it known if the FreeBSD initiator
(with gmultipath) can work with this gateway setup?


Regards,

Frank
___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Jason


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com