[ceph-users] multiple active connections to a single LUN

[Никитенко Виталий]

hi, install iscsi gateway as write 
http://docs.ceph.com/docs/master/rbd/iscsi-overview/
kernel version 4.16.1
ceph v12.2.10 Luminous
early i use tgt iscsi target i can connect 3 esxi server to one LUN as a shared 
drive and if one esxi was dropped the machines were transferred to others from 
the same storage
Now when I try to connect one LAN to 3 servers, I get an error on the ceph 
server

Dec 29 07:24:29 ceph-node1 tcmu-runner: tcmu_notify_lock_lost: 209 rbd / 
main_pool.store: Async lock drop. Old state 1
Dec 29 07:24:30 ceph-node1 tcmu-runner: alua_implicit_transition: 566 rbd / 
main_pool.store: Starting lock acquisition operation.
Dec 29 07:24:30 ceph-node1 tcmu-runner: tcmu_rbd_lock: 757 rbd / 
main_pool.store: Acquired exclusive lock.
Dec 29 07:24:35 ceph-node1 tcmu-runner: alua_implicit_transition: 566 rbd / 
main_pool.store: Starting lock acquisition operation.

such a scheme is possible in this configuration?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Balancing cluster with large disks - 10TB HHD

[jesper]

Hi. .. Just an update - This looks awesome.. and in a 8x5 company -
christmas is a good period to rebalance a cluster :-)

>> I'll try it out again - last I tried it complanied about older clients -
>> it should be better now.
> upmap is supported since kernel 4.13.
>
>> Second - should the reweights be set back to 1 then?
> Yes, also:
>
> 1. `ceph osd crush tunables optimal`

Done

> 2. All your buckets should be straw2, but in case `ceph osd crush
> set-all-straw-buckets-to-straw2`

Done

> 3. Your hosts imbalanced: elefant & capone have only eight 10TB's,
> another hosts - 12. So I recommend replace 8TB's spinners to 10TB or
> just shuffle it between hosts, like 2x8TB+10x10Tb.

Yes, we initially thought we could go with 3 osd-hosts .. but then found
out that EC-pools required more -- and then added.

> 4. Revert all your reweights.

Done

> 5. Balancer do his work: `ceph balancer mode upmap`, `ceph balancer on`.

So far - works awesome --
 sudo qms/server_documentation/ceph/ceph-osd-data-distribution hdd
hdd
x 
N   Min   MaxMedian   AvgStddev
x  72 50.82 55.65 52.88 52.916944 1.0002586

As compared to the best I got with reweighting:
$ sudo qms/server_documentation/ceph/ceph-osd-data-distribution hdd
hdd
x 
N   Min   MaxMedian   AvgStddev
x  72 45.36 54.98 52.63 52.131944 2.0746672


It took about 24 hours to rebalance -- and move quite some TB's around.

I would still like to have a log somewhere to grep and inspect what
balancer/upmap
actually does - when in my cluster. Or some ceph commands that deliveres
some monitoring capabilityes .. any suggestions?

-- 
Jesper

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] utilization of rbd volume

[Jason Dillaman]

There isn't anything built-in to Ceph to help with that. You could
either increase the logging level for the OSDs and grep out
"rbd_data." object accesses (assuming v2 image format), group by the
image id, and figure out which image is being hammered. You could also
create custom "iptables" LOG rules on the OSD nodes for each
client-node to see which client client is sending/receiving the most
data.

On Fri, Dec 28, 2018 at 11:43 AM Sinan Polat  wrote:
>
> Hi Jason,
>
> Thanks for your reply.
>
> Unfortunately we do not have access to the clients.
>
> We are running Red Hat Ceph 2.x which is based on Jewel, that means we cannot 
> pinpoint who or what is causing the load on the cluster, am I right?
>
> Thanks!
> Sinan
>
> > Op 28 dec. 2018 om 15:14 heeft Jason Dillaman  het 
> > volgende geschreven:
> >
> > With the current releases of Ceph, the only way to accomplish this is
> > by gathering the IO stats on each client node. However, with the
> > future Nautilus release, this data will now be available directly from
> > the OSDs.
> >
> >> On Fri, Dec 28, 2018 at 6:18 AM Sinan Polat  wrote:
> >>
> >> Hi all,
> >>
> >> We have a couple of hundreds RBD volumes/disks in our Ceph cluster, each 
> >> RBD disk is mounted by a different client. Currently we see quite high 
> >> IOPS happening on the cluster, but we don't know which client/RBD is 
> >> causing it.
> >>
> >> Is it somehow easily to see the utilization per RBD disk?
> >>
> >> Thanks!
> >> Sinan
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> > --
> > Jason
>


-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] utilization of rbd volume

[Sinan Polat]

Hi Jason,

Thanks for your reply.

Unfortunately we do not have access to the clients.

We are running Red Hat Ceph 2.x which is based on Jewel, that means we cannot 
pinpoint who or what is causing the load on the cluster, am I right?

Thanks!
Sinan

> Op 28 dec. 2018 om 15:14 heeft Jason Dillaman  het 
> volgende geschreven:
> 
> With the current releases of Ceph, the only way to accomplish this is
> by gathering the IO stats on each client node. However, with the
> future Nautilus release, this data will now be available directly from
> the OSDs.
> 
>> On Fri, Dec 28, 2018 at 6:18 AM Sinan Polat  wrote:
>> 
>> Hi all,
>> 
>> We have a couple of hundreds RBD volumes/disks in our Ceph cluster, each RBD 
>> disk is mounted by a different client. Currently we see quite high IOPS 
>> happening on the cluster, but we don't know which client/RBD is causing it.
>> 
>> Is it somehow easily to see the utilization per RBD disk?
>> 
>> Thanks!
>> Sinan
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> -- 
> Jason

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] EC pools grinding to a screeching halt on Luminous

[Mohamad Gebai]

Hi Marcus,

On 12/27/18 4:21 PM, Marcus Murwall wrote:
> Hey Mohamad
>
> I work with Florian on this issue.
> Just reinstalled the ceph cluster and triggered the error again.
> Looking at iostat -x 1 there is basically no activity at all against
> any of the osds.
> We get blocked ops all over the place but here are some output from
> one of the osds that had blocked requests:
> http://paste.openstack.org/show/738721/

Looking at the historic_slow_ops, the step in the pipeline that takes
the most time is sub_op_applied -> commit_sent. I couldn't say exactly
what these steps are from a high level view, but looking at the code,
commit_sent indicates that a message has been sent to the OSD's client
over the network. Can you look for network congestion (the fact that
there's nothing happening on the disks points in that direction too)?
Something like iftop might help. Is there anything suspicious in the logs?

Also, do you get the same throughput when benchmarking the replicated
compared to the EC pool?

Mohamad

>
>
> Regards
> Marcus
>
>> Mohamad Gebai 
>> 26 December 2018 at 18:27
>> What is happening on the individual nodes when you reach that point
>> (iostat -x 1 on the OSD nodes)? Also, what throughput do you get when
>> benchmarking the replicated pool?
>>
>> I guess one way to start would be by looking at ongoing operations at
>> the OSD level:
>>
>> ceph daemon osd.X dump_blocked_ops
>> ceph daemon osd.X dump_ops_in_flight
>> ceph daemon osd.X dump_historic_slow_ops
>>
>> (see ceph daemon osd.X help) for more commands.
>>
>> The first command will show currently blocked operations. The last
>> command shows recent slow operations. You can follow the flow of
>> individual operations, and you might find that the slow operations are
>> all associated with the same few PGs, or that they're spending too much
>> time waiting on something.
>>
>> Hope that helps.
>>
>> Mohamad
>>
>>
>> Florian Haas 
>> 26 December 2018 at 11:20
>> Hi everyone,
>>
>> We have a Luminous cluster (12.2.10) on Ubuntu Xenial, though we have
>> also observed the same behavior on 12.2.7 on Bionic (download.ceph.com
>> doesn't build Luminous packages for Bionic, and 12.2.7 is the latest
>> distro build).
>>
>> The primary use case for this cluster is radosgw. 6 OSD nodes, 22 OSDs
>> per node, of which 20 are SAS spinners and 2 are NVMe devices. Cluster
>> has been deployed with ceph-ansible stable-3.1, we're using
>> "objectstore: bluestore" and "osd_scenario: collocated".
>>
>> We're using a "class hdd" replicated CRUSH ruleset for all our pools,
>> except:
>>
>> - the bucket index pool, which uses a replicated "class nvme" rule, and
>> - the bucket data pool, which uses an EC (crush-device-class=hdd,
>> crush-failure-domain=host, k=3, m=2).
>>
>> We also have 3 pools that we have created in order to be able to do
>> benchmark runs while leaving the other pools untouched, so we have
>>
>> - bench-repl-hdd, replicated, size 3, using a CRUSH rule with "step take
>> default class hdd"
>> - bench-repl-nvme, replicated, size 3, using a CRUSH rule with "step
>> take default class nvme"
>> - bench-ec-hdd, EC, crush-device-class=hdd, crush-failure-domain=host,
>> k=3, m=2.
>>
>> Baseline benchmarks with "ceph tell osd.* bench" at the default block
>> size of 4M yield pretty exactly the throughput you'd expect from the
>> devices: approx. 185 MB/s from the SAS drives; the NVMe devices
>> currently pull only 650 MB/s on writes but that may well be due to
>> pending conditioning — this is new hardware.
>>
>> Now when we run "rados bench" against the replicated pools, we again get
>> exactly what we expect for a nominally performing but largely untuned
>> system.
>>
>> It's when we try running benchmarks against the EC pool that everything
>> appears to grind to a halt:
>>
>> http://paste.openstack.org/show/738187/
>>
>> After 19 seconds, that pool does not accept a single further object. We
>> simultaneously see slow request warnings creep up in the cluster, and
>> the only thing we can then do is kill the benchmark, and wait for the
>> slow requests to clear out.
>>
>> We've also seen the log messages discussed in
>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028972.html,
>> and they seem to correlate with the slow requests popping up, but from
>> Greg's reply in
>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028974.html
>> I'm assuming that that's benign and doesn't warrant further
>> investigation.
>>
>> Here's a few things we've tried, to no avail:
>>
>> - Make sure we use the latest Luminous release (we started out on Bionic
>> and 12.2.7, then reinstalled systems with Xenial so we could use
>> 12.2.10).
>> - Enable Bluestore buffered writes (bluestore_default_buffered_write =
>> true); buffered reads are on by default.
>> - Extend the BlueStore cache from 1G to 4G (bluestore_cache_size_hdd =
>> 4294967296; each OSD box has 128G RAM so should no

Re: [ceph-users] utilization of rbd volume

[Jason Dillaman]

With the current releases of Ceph, the only way to accomplish this is
by gathering the IO stats on each client node. However, with the
future Nautilus release, this data will now be available directly from
the OSDs.

On Fri, Dec 28, 2018 at 6:18 AM Sinan Polat  wrote:
>
> Hi all,
>
> We have a couple of hundreds RBD volumes/disks in our Ceph cluster, each RBD 
> disk is mounted by a different client. Currently we see quite high IOPS 
> happening on the cluster, but we don't know which client/RBD is causing it.
>
> Is it somehow easily to see the utilization per RBD disk?
>
> Thanks!
> Sinan
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] utilization of rbd volume

[Sinan Polat]

Hi all,

We have a couple of hundreds RBD volumes/disks in our Ceph cluster, each RBD
disk is mounted by a different client. Currently we see quite high IOPS
happening on the cluster, but we don't know which client/RBD is causing it.

Is it somehow easily to see the utilization per RBD disk?

Thanks!
Sinan___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] list admin issues

[Ilya Dryomov]

On Sat, Dec 22, 2018 at 7:18 PM Brian :  wrote:
>
> Sorry to drag this one up again.
>
> Just got the unsubscribed due to excessive bounces thing.
>
> 'Your membership in the mailing list ceph-users has been disabled due
> to excessive bounces The last bounce received from you was dated
> 21-Dec-2018.  You will not get any more messages from this list until
> you re-enable your membership.  You will receive 3 more reminders like
> this before your membership in the list is deleted.'
>
> can anyone check MTA logs to see what the bounce is?

Me too.  Happens regularly and only on ceph-users, not on sepia or
ceph-maintainers, etc.  David, Dan, could you or someone you know look
into this?

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com