Re: [ceph-users] low io with enterprise SSDs ceph luminous - can we expect more? [klartext]
Frank, Sorry for the confusion. I thought that turning off cache using hdparm -W 0 /dev/sdx takes effect right away and in case of non-raid controllers and Seagate or Micron SSDs I would see a difference starting fio benchmark right after executing hdparm. So I wonder it makes a difference whether cache turned off before OSD started or after. On Tue, Jan 21, 2020, 2:07 AM Frank Schilder wrote: > > So hdparam -W 0 /dev/sdx doesn't work or it makes no difference? > > I wrote "We found the raw throughput in fio benchmarks to be very > different for write-cache enabled and disabled, exactly as explained in the > performance article.", so yes, it makes a huge difference. > > > Also I am not sure I understand why it should happen before OSD have > been started. > > At least in my experience hdparam does it to hardware regardless. > > I'm not sure I understand this question. Ideally it happens at boot time > and if this doesn't work, at least sometimes before the OSD is started. Why > and how else would one want this to happen? > > Best regards, > > = > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] lists and gmail
It seems that people now split between new and old list servers. Regardless of either one of them, I am missing a number of messages that appear on archive pages but never seem to make to my inbox. And no they are not in my junk folder. I wonder if some of my questions are not getting a response because people don't receive them. Any other reason, like people choosing not to answer, is pretty acceptable. Does anyone else have difficulty communicating with user list using gmail account? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] low io with enterprise SSDs ceph luminous - can we expect more? [klartext]
So hdparam -W 0 /dev/sdx doesn't work or it makes no difference? Also I am
not sure I understand why it should happen before OSD have been started.
At least in my experience hdparam does it to hardware regardless.
On Mon, Jan 20, 2020, 2:25 AM Frank Schilder wrote:
> We are using Micron 5200 PRO, 1.92TB for RBD images on KVM and are very
> happy with the performance. We are using EC 6+2 pools, which really eat up
> IOPs. Still, we get enough performance out to run 20-50 VMs per disk, which
> results in good space utilisation as well since our default image size is
> 50GB and we take rolling snapshots. I was thinking about 4TB disks also,
> but am concerned that their IOPs/TB performance is too low for images on EC
> pools.
>
> We found the raw throughput in fio benchmarks to be very different for
> write-cache enabled and disabled, exactly as explained in the performance
> article. Changing write cache settings is a boot-time operation.
> Unfortunately, I couldn't find a reliable way to disable write cache at
> boot time (I was looking for tuned configs) and ended up adding this to a
> container startup script:
>
> if [[ "$1" == "osd_ceph_disk_activate" && -n "${OSD_DEVICE}" ]] ; then
> echo "Disabling write cache on ${OSD_DEVICE}"
> /usr/sbin/smartctl -s wcache=off "${OSD_DEVICE}"
> fi
>
> This works for both, SAS and SATA drives and ensures that write cache is
> disabled before an OSD daemon starts.
>
> Best regards,
>
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
>
> From: ceph-users on behalf of Eric K.
> Miller
> Sent: 19 January 2020 04:24:33
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] low io with enterprise SSDs ceph luminous - can
> we expect more? [klartext]
>
> Hi Vitaliy,
>
> Similar to Stefan, we have a bunch of Micron 5200's (3.84TB ECO SATA
> version) in a Ceph cluster (Nautilus) and performance seems less than
> optimal. I have followed all instructions on your site (thank you for your
> wonderful article btw!!), but I haven't seen much change.
>
> The only thing I could think of is that "maybe" disabling the write cache
> only takes place upon a reboot or power cycle? Is that necessary? Or is
> it a "live" change?
>
> I have tested with the cache disabled as well as enabled on all drives.
> We're using fio running in a QEMU/KVM VM in an OpenStack cluster, so not
> "raw" access to the Micron 5200's. OSD (Bluestore) nodes run CentOS 7
> using a 4.18.x kernel. Testing doesn't show any, or much, difference,
> enough that the variations could be considered "noise" in the results.
> Certainly no change that anyone could tell.
>
> Thought I'd check to see if you, or anyone else, might have any
> suggestions specific to the Micron 5200.
>
> We have some Micron 5300's inbound, but probably won't have them here for
> another few weeks due to Micron's manufacturing delays, so will be able to
> test these raw drives soon. I will report back after, but if you know
> anything about these, I'm all ears. :)
>
> Thank you!
>
> Eric
>
>
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Stefan Bauer
> Sent: Tuesday, January 14, 2020 10:28 AM
> To: undisclosed-recipients
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] low io with enterprise SSDs ceph luminous - can
> we expect more? [klartext]
>
>
> Thank you all,
>
>
>
> performance is indeed better now. Can now go back to sleep ;)
>
>
>
> KR
>
>
>
> Stefan
>
>
> -Ursprüngliche Nachricht-
> Von: Виталий Филиппов
> Gesendet: Dienstag 14 Januar 2020 10:28
> An: Wido den Hollander ; Stefan Bauer <
> stefan.ba...@cubewerk.de>
> CC: ceph-users@lists.ceph.com
> Betreff: Re: [ceph-users] low io with enterprise SSDs ceph luminous - can
> we expect more? [klartext]
>
> ...disable signatures and rbd cache. I didn't mention it in the email to
> not repeat myself. But I have it in the article :-)
> --
> With best regards,
> Vitaliy Filippov
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] SPAM in the ceph-users list
I am seeing more and more spam on this list. Recently a strain of messages announcing services and businesses in Bangalore for example. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] user and group acls on cephfs mounts
Figured out. Nothing ceph related. Someone created multiple ACL entries on a directory and ls -l had correct numbers but getfacl showed its real colors. Group write permissions were disabled at that level. On Tue, Nov 5, 2019 at 7:10 PM Yan, Zheng wrote: > On Wed, Nov 6, 2019 at 5:47 AM Alex Litvak > wrote: > > > > Hello Cephers, > > > > > > I am trying to understand how uid and gid are handled on the shared > cephfs mount. I am using 14.2.2 and cephfs kernel based client. > > I have 2 client vms with following uid gid > > > > vm1 user dev (uid=500) group dev (gid=500) > > vm2 user dev (uid=500) group dev (gid=500) > > > > > > vm1 user tomcat (uid=996) group tomcat (gid=995) > > vm2 user tomcat (uid=990) group tomcat (gid=990) > > > > ACLs only record IDs of users/groups. group tomcat is different on vm1/vm2 > > > > > on both machines user tomcat is added to a group dev. > > > > > > Directory /webcluster/data is a kernel cephfs mount and has permissions > visible on both clients as > > > > rwxrwxr-x dev dev /webcluster/data > > > > also > > > > rwxr-xr-x root root /webcluster > > > > So it is my understanding that on both vms I should be able to > successfully run > > > > touch /webcluster/data/foo as user tomcat. > > > > However, on vm2 I get permission denied when I attempt to write a file > in /webcluster/data. > > When I change uid and gid of tomcat on vm2 to match those on vm1, then I > successfully can write into /webcluster/data. > > > > As on both machines user tomcat is a member of group dev and group dev > is allowed to write in the directory, why do the uids of the group members > need to match across network? > > > > > > I tried research it on my own and failed to find a good explanation. > > > > > > Thank you for your help, > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CephFS kernel module lockups in Ubuntu linux-image-5.0.0-32-generic?
Also, search for this topic on the list. Ubuntu Disco with most recent Kernel 5.0.0-32 seems to be instable On Thu, Oct 24, 2019 at 10:45 AM Paul Emmerich wrote: > Could it be related to the broken backport as described in > https://tracker.ceph.com/issues/40102 ? > > (It did affect 4.19, not sure about 5.0) > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Thu, Oct 24, 2019 at 4:23 PM Christopher Wieringa > wrote: > > > > Hello all, > > > > > > > > I’ve been using the Ceph kernel modules in Ubuntu to load a CephFS > filesystem quite successfully for several months. Yesterday, I went > through a round of updates on my Ubuntu 18.04 machines, which loaded > linux-image-5.0.0-32-generic as the kernel. I’m noticing that while the > kernel boots and the CephFS filesystem is mounted successfully, any actual > file traffic between client/servers is causing the client to hard lock > without any kernel panics seen in the logs. Downgrading to > linux-image-5.0.0-31-generic fixes the issue. > > > > I’m planning on trying to figure out how to get a crash kernel running > and see if I can generate a log file of some sort, but just wondering if > anyone else mounting Ubuntu 18.04 and CephFS via kernel modules could > replicate this crashing behavior in their environment. I don’t have > experience in setting up crash kernels or even in filing Ubuntu kernel > bugs, so I thought I’d ask this list to see if someone else can replicate > this. > > > > My Setup: > > Ubuntu 18.04.3 LTS amd64 > > > > Kernel: linux-image-5.0.0-32-generic(installed through the > linux-generic-hwe-18.04 metapackage) > > > > Number of machines affected/replicated the lockup issues: >100 (three > generations of desktop computers, and multiple virtual machines as well) > > > > > > > > Also, any suggestions for how to get a log for this are appreciated. > > > > > > > > Chris Wieringa > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD crashed during the fio test
I updated firmware and kernel, running torture tests. So far no assert, but I still noticed this on the same osd as yesterday Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721 7f8d03150700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f8cd05d7700' had timed out after 60 Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721 7f8d03150700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f8cd0dd8700' had timed out after 60 Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721 7f8d03150700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f8cd2ddc700' had timed out after 60 Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721 7f8d03150700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f8cd35dd700' had timed out after 60 Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721 7f8d03150700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f8cd3dde700' had timed out after 60 The spike of latency on this OSD is 6 seconds at that time. Any ideas? On Tue, Oct 1, 2019 at 8:03 AM Sasha Litvak wrote: > It was hardware indeed. Dell server reported a disk being reset with > power on. Checking the usual suspects i.e. controller firmware, controller > event log (if I can get one), drive firmware. > I will report more when I get a better idea > > Thank you! > > On Tue, Oct 1, 2019 at 2:33 AM Brad Hubbard wrote: > >> Removed ceph-de...@vger.kernel.org and added d...@ceph.io >> >> On Tue, Oct 1, 2019 at 4:26 PM Alex Litvak >> wrote: >> > >> > Hellow everyone, >> > >> > Can you shed the line on the cause of the crash? Could actually client >> request trigger it? >> > >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 >> 22:52:58.867 7f093d71e700 -1 bdev(0x55b72c156000 >> /var/lib/ceph/osd/ceph-17/block) aio_submit retries 16 >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 >> 22:52:58.867 7f093d71e700 -1 bdev(0x55b72c156000 >> /var/lib/ceph/osd/ceph-17/block) aio submit got (11) Resource temporarily >> unavailable >> >> The KernelDevice::aio_submit function has tried to submit Io 16 times >> (a hard coded limit) and received an error each time causing it to >> assert. Can you check the status of the underlying device(s)? >> >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: >> > >> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc: >> > In fun >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: >> > >> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc: >> > 757: F >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: ceph version 14.2.2 >> (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable) >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 1: >> (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x14a) [0x55b71f668cf4] >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2: >> (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char >> const*, ...)+0) [0x55b71f668ec2] >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 3: >> (KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1] >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 4: >> (BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) [0x55b71fc29892] >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 5: >> (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b) >> [0x55b71fc496ab] >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 6: >> (BlueStore::queue_transactions(boost::intrusive_ptr&, >> std::vector> > std::allocator >&, >> boost::intrusive_ptr, ThreadPool::T >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 7: (non-virtual >> thunk to >> PrimaryLogPG::queue_transactions(std::vector> std::allocator >&, >> > boost::intrusive_ptr)+0x54) [0x55b71f9b1b84] >> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 8: >> (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t >> const&, eversion_t const&, std::unique_ptr> > std::default_delete >&&, eversion_t const&, eversion_t >> const&, s >> > Sep 30 22:52
Re: [ceph-users] Commit and Apply latency on nautilus
All,
Thank you for your suggestions. During the last night test, I had at least
one drive on one node doing a power-on reset by the controller. It caused
a couple of OSDs asserting / timing out on that node. I am testing and
updating the usual suspects on this node and after that on a whole cluster,
i.e. kernel, controller firmware, SSD firmware. All of these have updates
available. Dell mentioned a possible crush on bionic during high
throughput but none of it is clear and simple. I would like to eliminate
firmware/drivers, especially if there is a bug causing a crash under the
load. I will then proceed with Mokhtar's and Robert's suggestions.
If anyone has any more suggestions please share on this thread as it may
help someone else later on.
Best,
On Tue, Oct 1, 2019 at 2:56 PM Maged Mokhtar wrote:
> Some suggestions:
>
> monitor raw resources such as cpu %util raw disk %util/busy, raw disk iops.
>
> instead of running a mix of workloads at this stage, narrow it down first,
> for example using rbd rand writes and 4k block sizes, then change 1 param
> at a time for example change the block size. See how your cluster performs
> and what resources loads you get step by step. Latency from 4M will not be
> the same as 4k.
>
> i would also run fio tests on the raw Nytro 1551 devices including sync
> writes.
>
> I would not recommend you increase readahead for random io.
>
> I do not recommend making RAID0
>
> /Maged
>
>
> On 01/10/2019 02:12, Sasha Litvak wrote:
>
> At this point, I ran out of ideas. I changed nr_requests and readahead
> parameters to 128->1024 and 128->4096, tuned nodes to
> performance-throughput. However, I still get high latency during benchmark
> testing. I attempted to disable cache on ssd
>
> for i in {a..f}; do hdparm -W 0 -A 0 /dev/sd$i; done
>
> and I think it make things not better at all. I have H740 and H730
> controllers with drives in HBA mode.
>
> Other them converting them one by one to RAID0 I am not sure what else I
> can try.
>
> Any suggestions?
>
>
> On Mon, Sep 30, 2019 at 2:45 PM Paul Emmerich
> wrote:
>
>> BTW: commit and apply latency are the exact same thing since
>> BlueStore, so don't bother looking at both.
>>
>> In fact you should mostly be looking at the op_*_latency counters
>>
>>
>> Paul
>>
>> --
>> Paul Emmerich
>>
>> Looking for help with your Ceph cluster? Contact us at https://croit.io
>>
>> croit GmbH
>> Freseniusstr. 31h
>> 81247 München
>> www.croit.io
>> Tel: +49 89 1896585 90
>>
>> On Mon, Sep 30, 2019 at 8:46 PM Sasha Litvak
>> wrote:
>> >
>> > In my case, I am using premade Prometheus sourced dashboards in grafana.
>> >
>> > For individual latency, the query looks like that
>> >
>> > irate(ceph_osd_op_r_latency_sum{ceph_daemon=~"$osd"}[1m]) / on
>> (ceph_daemon) irate(ceph_osd_op_r_latency_count[1m])
>> > irate(ceph_osd_op_w_latency_sum{ceph_daemon=~"$osd"}[1m]) / on
>> (ceph_daemon) irate(ceph_osd_op_w_latency_count[1m])
>> >
>> > The other ones use
>> >
>> > ceph_osd_commit_latency_ms
>> > ceph_osd_apply_latency_ms
>> >
>> > and graph the distribution of it over time
>> >
>> > Also, average OSD op latency
>> >
>> > avg(rate(ceph_osd_op_r_latency_sum{cluster="$cluster"}[5m]) /
>> rate(ceph_osd_op_r_latency_count{cluster="$cluster"}[5m]) >= 0)
>> > avg(rate(ceph_osd_op_w_latency_sum{cluster="$cluster"}[5m]) /
>> rate(ceph_osd_op_w_latency_count{cluster="$cluster"}[5m]) >= 0)
>> >
>> > Average OSD apply + commit latency
>> > avg(ceph_osd_apply_latency_ms{cluster="$cluster"})
>> > avg(ceph_osd_commit_latency_ms{cluster="$cluster"})
>> >
>> >
>> > On Mon, Sep 30, 2019 at 11:13 AM Marc Roos
>> wrote:
>> >>
>> >>
>> >> What parameters are you exactly using? I want to do a similar test on
>> >> luminous, before I upgrade to Nautilus. I have quite a lot (74+)
>> >>
>> >> type_instance=Osd.opBeforeDequeueOpLat
>> >> type_instance=Osd.opBeforeQueueOpLat
>> >> type_instance=Osd.opLatency
>> >> type_instance=Osd.opPrepareLatency
>> >> type_instance=Osd.opProcessLatency
>> >> type_instance=Osd.opRLatency
>> >> type_instance=Osd.opRPrepareLatency
>> >> type_instance=Osd.opRProcessLatency
>> >> type_instance=Osd.opRwLatency
>> >> type_instance=Osd.opRwPre
Re: [ceph-users] OSD crashed during the fio test
It was hardware indeed. Dell server reported a disk being reset with power on. Checking the usual suspects i.e. controller firmware, controller event log (if I can get one), drive firmware. I will report more when I get a better idea Thank you! On Tue, Oct 1, 2019 at 2:33 AM Brad Hubbard wrote: > Removed ceph-de...@vger.kernel.org and added d...@ceph.io > > On Tue, Oct 1, 2019 at 4:26 PM Alex Litvak > wrote: > > > > Hellow everyone, > > > > Can you shed the line on the cause of the crash? Could actually client > request trigger it? > > > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 > 22:52:58.867 7f093d71e700 -1 bdev(0x55b72c156000 > /var/lib/ceph/osd/ceph-17/block) aio_submit retries 16 > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 > 22:52:58.867 7f093d71e700 -1 bdev(0x55b72c156000 > /var/lib/ceph/osd/ceph-17/block) aio submit got (11) Resource temporarily > unavailable > > The KernelDevice::aio_submit function has tried to submit Io 16 times > (a hard coded limit) and received an error each time causing it to > assert. Can you check the status of the underlying device(s)? > > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc: > > In fun > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc: > > 757: F > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: ceph version 14.2.2 > (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable) > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 1: > (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x14a) [0x55b71f668cf4] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2: > (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char > const*, ...)+0) [0x55b71f668ec2] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 3: > (KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 4: > (BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) [0x55b71fc29892] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 5: > (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b) > [0x55b71fc496ab] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 6: > (BlueStore::queue_transactions(boost::intrusive_ptr&, > std::vector > std::allocator >&, > boost::intrusive_ptr, ThreadPool::T > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 7: (non-virtual thunk > to PrimaryLogPG::queue_transactions(std::vector std::allocator >&, > > boost::intrusive_ptr)+0x54) [0x55b71f9b1b84] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 8: > (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t > const&, eversion_t const&, std::unique_ptr > std::default_delete >&&, eversion_t const&, eversion_t > const&, s > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 9: > (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, > PrimaryLogPG::OpContext*)+0xf12) [0x55b71f90e322] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 10: > (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfae) [0x55b71f969b7e] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 11: > (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3965) > [0x55b71f96de15] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 12: > (PrimaryLogPG::do_request(boost::intrusive_ptr&, > ThreadPool::TPHandle&)+0xbd4) [0x55b71f96f8a4] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 13: > (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, > ThreadPool::TPHandle&)+0x1a9) [0x55b71f7a9ea9] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 14: > (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, > ThreadPool::TPHandle&)+0x62) [0x55b71fa475d2] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 15: > (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x9f4) > [0x55b71f7c6ef4] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 16: > (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433) > [0x55b71fdc5ce3] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 17: > (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55b71fdc8d80] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 18: (()+0x7dd5) > [0x7f0971da9dd5] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 19: (clone()+0x6d) > [0x7f0970c7002d] > > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 > 22:52:58.879 7f093d71e700 -1 > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el
Re: [ceph-users] Commit and Apply latency on nautilus
At this point, I ran out of ideas. I changed nr_requests and readahead
parameters to 128->1024 and 128->4096, tuned nodes to
performance-throughput. However, I still get high latency during benchmark
testing. I attempted to disable cache on ssd
for i in {a..f}; do hdparm -W 0 -A 0 /dev/sd$i; done
and I think it make things not better at all. I have H740 and H730
controllers with drives in HBA mode.
Other them converting them one by one to RAID0 I am not sure what else I
can try.
Any suggestions?
On Mon, Sep 30, 2019 at 2:45 PM Paul Emmerich
wrote:
> BTW: commit and apply latency are the exact same thing since
> BlueStore, so don't bother looking at both.
>
> In fact you should mostly be looking at the op_*_latency counters
>
>
> Paul
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
> On Mon, Sep 30, 2019 at 8:46 PM Sasha Litvak
> wrote:
> >
> > In my case, I am using premade Prometheus sourced dashboards in grafana.
> >
> > For individual latency, the query looks like that
> >
> > irate(ceph_osd_op_r_latency_sum{ceph_daemon=~"$osd"}[1m]) / on
> (ceph_daemon) irate(ceph_osd_op_r_latency_count[1m])
> > irate(ceph_osd_op_w_latency_sum{ceph_daemon=~"$osd"}[1m]) / on
> (ceph_daemon) irate(ceph_osd_op_w_latency_count[1m])
> >
> > The other ones use
> >
> > ceph_osd_commit_latency_ms
> > ceph_osd_apply_latency_ms
> >
> > and graph the distribution of it over time
> >
> > Also, average OSD op latency
> >
> > avg(rate(ceph_osd_op_r_latency_sum{cluster="$cluster"}[5m]) /
> rate(ceph_osd_op_r_latency_count{cluster="$cluster"}[5m]) >= 0)
> > avg(rate(ceph_osd_op_w_latency_sum{cluster="$cluster"}[5m]) /
> rate(ceph_osd_op_w_latency_count{cluster="$cluster"}[5m]) >= 0)
> >
> > Average OSD apply + commit latency
> > avg(ceph_osd_apply_latency_ms{cluster="$cluster"})
> > avg(ceph_osd_commit_latency_ms{cluster="$cluster"})
> >
> >
> > On Mon, Sep 30, 2019 at 11:13 AM Marc Roos
> wrote:
> >>
> >>
> >> What parameters are you exactly using? I want to do a similar test on
> >> luminous, before I upgrade to Nautilus. I have quite a lot (74+)
> >>
> >> type_instance=Osd.opBeforeDequeueOpLat
> >> type_instance=Osd.opBeforeQueueOpLat
> >> type_instance=Osd.opLatency
> >> type_instance=Osd.opPrepareLatency
> >> type_instance=Osd.opProcessLatency
> >> type_instance=Osd.opRLatency
> >> type_instance=Osd.opRPrepareLatency
> >> type_instance=Osd.opRProcessLatency
> >> type_instance=Osd.opRwLatency
> >> type_instance=Osd.opRwPrepareLatency
> >> type_instance=Osd.opRwProcessLatency
> >> type_instance=Osd.opWLatency
> >> type_instance=Osd.opWPrepareLatency
> >> type_instance=Osd.opWProcessLatency
> >> type_instance=Osd.subopLatency
> >> type_instance=Osd.subopWLatency
> >> ...
> >> ...
> >>
> >>
> >>
> >>
> >>
> >> -Original Message-
> >> From: Alex Litvak [mailto:alexander.v.lit...@gmail.com]
> >> Sent: zondag 29 september 2019 13:06
> >> To: ceph-users@lists.ceph.com
> >> Cc: ceph-de...@vger.kernel.org
> >> Subject: [ceph-users] Commit and Apply latency on nautilus
> >>
> >> Hello everyone,
> >>
> >> I am running a number of parallel benchmark tests against the cluster
> >> that should be ready to go to production.
> >> I enabled prometheus to monitor various information and while cluster
> >> stays healthy through the tests with no errors or slow requests,
> >> I noticed an apply / commit latency jumping between 40 - 600 ms on
> >> multiple SSDs. At the same time op_read and op_write are on average
> >> below 0.25 ms in the worth case scenario.
> >>
> >> I am running nautilus 14.2.2, all bluestore, no separate NVME devices
> >> for WAL/DB, 6 SSDs per node(Dell PowerEdge R440) with all drives Seagate
> >> Nytro 1551, osd spread across 6 nodes, running in
> >> containers. Each node has plenty of RAM with utilization ~ 25 GB during
> >> the benchmark runs.
> >>
> >> Here are benchmarks being run from 6 client systems in parallel,
> >> repeating the test for each block size in <4k,16k,128k,4M>.
> >>
> >> On rbd mapped partition local to each client:
> &g
Re: [ceph-users] Commit and Apply latency on nautilus
In my case, I am using premade Prometheus sourced dashboards in grafana.
For individual latency, the query looks like that
irate(ceph_osd_op_r_latency_sum{ceph_daemon=~"$osd"}[1m]) / on
(ceph_daemon) irate(ceph_osd_op_r_latency_count[1m])
irate(ceph_osd_op_w_latency_sum{ceph_daemon=~"$osd"}[1m]) / on
(ceph_daemon) irate(ceph_osd_op_w_latency_count[1m])
The other ones use
ceph_osd_commit_latency_ms
ceph_osd_apply_latency_ms
and graph the distribution of it over time
Also, average OSD op latency
avg(rate(ceph_osd_op_r_latency_sum{cluster="$cluster"}[5m]) /
rate(ceph_osd_op_r_latency_count{cluster="$cluster"}[5m]) >= 0)
avg(rate(ceph_osd_op_w_latency_sum{cluster="$cluster"}[5m]) /
rate(ceph_osd_op_w_latency_count{cluster="$cluster"}[5m]) >= 0)
Average OSD apply + commit latency
avg(ceph_osd_apply_latency_ms{cluster="$cluster"})
avg(ceph_osd_commit_latency_ms{cluster="$cluster"})
On Mon, Sep 30, 2019 at 11:13 AM Marc Roos wrote:
>
> What parameters are you exactly using? I want to do a similar test on
> luminous, before I upgrade to Nautilus. I have quite a lot (74+)
>
> type_instance=Osd.opBeforeDequeueOpLat
> type_instance=Osd.opBeforeQueueOpLat
> type_instance=Osd.opLatency
> type_instance=Osd.opPrepareLatency
> type_instance=Osd.opProcessLatency
> type_instance=Osd.opRLatency
> type_instance=Osd.opRPrepareLatency
> type_instance=Osd.opRProcessLatency
> type_instance=Osd.opRwLatency
> type_instance=Osd.opRwPrepareLatency
> type_instance=Osd.opRwProcessLatency
> type_instance=Osd.opWLatency
> type_instance=Osd.opWPrepareLatency
> type_instance=Osd.opWProcessLatency
> type_instance=Osd.subopLatency
> type_instance=Osd.subopWLatency
> ...
> ...
>
>
>
>
>
> -Original Message-
> From: Alex Litvak [mailto:alexander.v.lit...@gmail.com]
> Sent: zondag 29 september 2019 13:06
> To: ceph-users@lists.ceph.com
> Cc: ceph-de...@vger.kernel.org
> Subject: [ceph-users] Commit and Apply latency on nautilus
>
> Hello everyone,
>
> I am running a number of parallel benchmark tests against the cluster
> that should be ready to go to production.
> I enabled prometheus to monitor various information and while cluster
> stays healthy through the tests with no errors or slow requests,
> I noticed an apply / commit latency jumping between 40 - 600 ms on
> multiple SSDs. At the same time op_read and op_write are on average
> below 0.25 ms in the worth case scenario.
>
> I am running nautilus 14.2.2, all bluestore, no separate NVME devices
> for WAL/DB, 6 SSDs per node(Dell PowerEdge R440) with all drives Seagate
> Nytro 1551, osd spread across 6 nodes, running in
> containers. Each node has plenty of RAM with utilization ~ 25 GB during
> the benchmark runs.
>
> Here are benchmarks being run from 6 client systems in parallel,
> repeating the test for each block size in <4k,16k,128k,4M>.
>
> On rbd mapped partition local to each client:
>
> fio --name=randrw --ioengine=libaio --iodepth=4 --rw=randrw
> --bs=<4k,16k,128k,4M> --direct=1 --size=2G --numjobs=8 --runtime=300
> --group_reporting --time_based --rwmixread=70
>
> On mounted cephfs volume with each client storing test file(s) in own
> sub-directory:
>
> fio --name=randrw --ioengine=libaio --iodepth=4 --rw=randrw
> --bs=<4k,16k,128k,4M> --direct=1 --size=2G --numjobs=8 --runtime=300
> --group_reporting --time_based --rwmixread=70
>
> dbench -t 30 30
>
> Could you please let me know if huge jump in applied and committed
> latency is justified in my case and whether I can do anything to improve
> / fix it. Below is some additional cluster info.
>
> Thank you,
>
> root@storage2n2-la:~# podman exec -it ceph-mon-storage2n2-la ceph osd df
> ID CLASS WEIGHT REWEIGHT SIZERAW USE DATAOMAPMETA AVAIL
> %USE VAR PGS STATUS
> 6 ssd 1.74609 1.0 1.7 TiB 93 GiB 92 GiB 240 MiB 784 MiB 1.7
> TiB 5.21 0.90 44 up
> 12 ssd 1.74609 1.0 1.7 TiB 98 GiB 97 GiB 118 MiB 906 MiB 1.7
> TiB 5.47 0.95 40 up
> 18 ssd 1.74609 1.0 1.7 TiB 102 GiB 101 GiB 123 MiB 901 MiB 1.6
> TiB 5.73 0.99 47 up
> 24 ssd 3.49219 1.0 3.5 TiB 222 GiB 221 GiB 134 MiB 890 MiB 3.3
> TiB 6.20 1.07 96 up
> 30 ssd 3.49219 1.0 3.5 TiB 213 GiB 212 GiB 151 MiB 873 MiB 3.3
> TiB 5.95 1.03 93 up
> 35 ssd 3.49219 1.0 3.5 TiB 203 GiB 202 GiB 301 MiB 723 MiB 3.3
> TiB 5.67 0.98 100 up
> 5 ssd 1.74609 1.0 1.7 TiB 103 GiB 102 GiB 123 MiB 901 MiB 1.6
> TiB 5.78 1.00 49 up
> 11 ssd 1.74609 1.0 1.7 TiB 109 GiB 108 GiB 63 MiB 961 MiB 1.6
> TiB 6.09 1.05 46 up
> 17 ssd 1.74609 1.0 1.7 TiB 104 GiB 103 GiB 205 MiB 819 MiB 1.6
> TiB 5.81 1.01 50 up
> 23 ssd 3.49219 1.0 3.5 TiB 210 GiB 209 GiB 168 MiB 856 MiB 3.3
> TiB 5.86 1.01 86 up
> 29 ssd 3.49219 1.0 3.5 TiB 204 GiB 203 GiB 272 MiB 752 MiB 3.3
> TiB 5.69 0.98 92 up
> 34 ssd 3.49219 1.0 3.5 TiB 198 GiB 197 GiB 295 MiB 729 MiB 3.3
> TiB 5.54 0.96 85 up
> 4 ssd 1.74609 1.0 1.7 TiB 119 GiB 1
Re: [ceph-users] Client admin socket for RBD
Tarek, Of course you are correct about the client nodes. I executed this command inside of container that runs mon. Or it can be done on the bare metal node that runs mon. You essentially quering mon configuration database. On Tue, Jun 25, 2019 at 8:53 AM Tarek Zegar wrote: > "config get" on a client.admin? There is no daemon for client.admin, I get > nothing. Can you please explain? > > > Tarek Zegar > Senior SDS Engineer > Email *tze...@us.ibm.com* > Mobile *630.974.7172* > > > > > [image: Inactive hide details for Sasha Litvak ---06/24/2019 07:48:46 > PM---ceph config get client.admin On Mon, Jun 24, 2019, 1:10 PM T]Sasha > Litvak ---06/24/2019 07:48:46 PM---ceph config get client.admin On Mon, Jun > 24, 2019, 1:10 PM Tarek Zegar wrote: > > From: Sasha Litvak > To: Tarek Zegar > Date: 06/24/2019 07:48 PM > Subject: [EXTERNAL] Re: Re: [ceph-users] Client admin socket for RBD > -- > > > > ceph config get client.admin > > On Mon, Jun 24, 2019, 1:10 PM Tarek Zegar <*tze...@us.ibm.com* > > wrote: > >Alex, > >Sorry real quick, what did you type to get that last bit of info? > >Tarek Zegar >Senior SDS Engineer > *Email **tze...@us.ibm.com* >Mobile *630.974.7172* > > > > >Alex Litvak ---06/24/2019 01:07:28 PM---Jason, Here you go: > >From: Alex Litvak <*alexander.v.lit...@gmail.com* >> >To: *ceph-users@lists.ceph.com* >Cc: ceph-users < >*public-ceph-users-idqoxfivofjgjs9i8mt...@plane.gmane.org* >> >Date: 06/24/2019 01:07 PM >Subject: [EXTERNAL] Re: [ceph-users] Client admin socket for RBD >Sent by: "ceph-users" <*ceph-users-boun...@lists.ceph.com* >> >-- > > > >Jason, > >Here you go: > >WHOMASK LEVELOPTION VALUE >RO >client advanced admin_socket > /var/run/ceph/$name.$pid.asok * >global advanced cluster_network *10.0.42.0/23* ><http://10.0.42.0/23> * >global advanced debug_asok 0/0 >global advanced debug_auth 0/0 >global advanced debug_bdev 0/0 >global advanced debug_bluefs0/0 >global advanced debug_bluestore 0/0 >global advanced debug_buffer0/0 >global advanced debug_civetweb 0/0 >global advanced debug_client0/0 >global advanced debug_compressor0/0 >global advanced debug_context 0/0 >global advanced debug_crush 0/0 >global advanced debug_crypto0/0 >global advanced debug_dpdk 0/0 >global advanced debug_eventtrace0/0 >global advanced debug_filer 0/0 >global advanced debug_filestore 0/0 >global advanced debug_finisher 0/0 >global advanced debug_fuse 0/0 >global advanced debug_heartbeatmap 0/0 >global advanced debug_javaclient0/0 >global advanced debug_journal 0/0 >global advanced debug_journaler 0/0 >global advanced debug_kinetic 0/0 >global advanced debug_kstore0/0 >global advanced debug_leveldb 0/0 >global advanced debug_lockdep 0/0 >global advanced debug_mds 0/0 >global advanced debug_mds_balancer 0/0 >global advanced debug_mds_locker0/0 >global advanced debug_mds_log 0/0 >global advanced debug_mds_log_expire0/0 >global advanced debug_mds_migrator 0/0 >global advanced debug_memdb 0/0 >global advanced debug_mgr 0/0 >global advanced debug_mgrc 0/0 >global advanced debug_mon 0/0 >global advanced debug_monc 0/00 >global advanced debug_ms0/0 >global advanced debug_none 0/0 >global advanced debug_objclass 0/0 >global advanced debug_objectcacher 0/0 >global advanced debug_objecter 0/0 >global advanced debug_optracker 0/0 >global advanced debug_osd 0/0 >global advanced de
Re: [ceph-users] Ceph release cadence
As a user, I woul like to add, I would like to see a real 2 year support for LTS releases. Hammer releases were sketchy at best in 2017. When luminous was released The outstanding bugs were auto closed, good buy and good readance. Also the decision to drop certain OS support created a barrier to upgrade and looking at jewel and luminous upgrade path where you cannot easily go back after upgrade is completed doesn't add the confidence. So making upgrades less radical may help production support to be more consistent and update process less dangerous. I would say 9 month is a good reference point but for me it is ready when it is really ready and tested. Keeping development release may be better for devs and early adopters. I don't believe production admins would go for intermediate one's as they being released now. This is only MHO and may be wrong On Saturday, September 9, 2017, Christian Theune wrote: > Hi, > > have been using Ceph for multiple years now. It’s unclear to me which of > your options fits best, but here are my preferences: > > * Updates are risky in a way that we tend to rather not do them every > year. Also, having seen jewel, we’ve been well off to avoid two > major issues what would have bitten us and will upgrade from hammer in > the next month or so. > > * Non-production releases are of not much value to me, as I have to keep > our dev/staging/prod clusters in sync to work on our stuff. > As you can never downgrade, there’s no value in it for me to try > non-production releases (without frying dev for everyone). > > * I’d prefer stability over new features. *Specifically* that new features > can be properly recombined with existing features (and each > other) without leading to surprises. (E.g. cache tiering breaking with > snapshots and then no way going back and a general notion of > “that combination wasn’t really well tested). > > * I’d prefer versions that I have to be maintained for production-critical > issues maybe 2 years, so I can have some time after a new > production release that overlaps with the new production release > receiving important bug fixes until I switch. > > Maybe this is close to what your "Drop the odd releases, and aim for a ~9 > month cadence.” would say. Waiting for a feature for a year is a pain, but > my personal goal for Ceph is that it first has to work properly, meaning: > not loose your data, not "stopping the show”, and not drawing you into a > corner you can’t get out. > > That’s my perspective as a user. As a fellow developer I feel your pain > about wanting to release faster and reducing maintenance load, so thanks > for asking! > > Hope this helps, > Christian > > > On Sep 6, 2017, at 5:23 PM, Sage Weil > > wrote: > > > > Hi everyone, > > > > Traditionally, we have done a major named "stable" release twice a year, > > and every other such release has been an "LTS" release, with fixes > > backported for 1-2 years. > > > > With kraken and luminous we missed our schedule by a lot: instead of > > releasing in October and April we released in January and August. > > > > A few observations: > > > > - Not a lot of people seem to run the "odd" releases (e.g., infernalis, > > kraken). This limits the value of actually making them. It also means > > that those who *do* run them are running riskier code (fewer users -> > more > > bugs). > > > > - The more recent requirement that upgrading clusters must make a stop at > > each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel > > -> lumninous) has been hugely helpful on the development side by reducing > > the amount of cross-version compatibility code to maintain and reducing > > the number of upgrade combinations to test. > > > > - When we try to do a time-based "train" release cadence, there always > > seems to be some "must-have" thing that delays the release a bit. This > > doesn't happen as much with the odd releases, but it definitely happens > > with the LTS releases. When the next LTS is a year away, it is hard to > > suck it up and wait that long. > > > > A couple of options: > > > > * Keep even/odd pattern, and continue being flexible with release dates > > > > + flexible > > - unpredictable > > - odd releases of dubious value > > > > * Keep even/odd pattern, but force a 'train' model with a more regular > > cadence > > > > + predictable schedule > > - some features will miss the target and be delayed a year > > > > * Drop the odd releases but change nothing else (i.e., 12-month release > > cadence) > > > > + eliminate the confusing odd releases with dubious value > > > > * Drop the odd releases, and aim for a ~9 month cadence. This splits the > > difference between the current even/odd pattern we've been doing. > > > > + eliminate the confusing odd releases with dubious value > > + waiting for the next release isn't quite as bad > > - required upgrades every 9 months instead of ever 12 months > > > > * Drop the odd releases, but relax the
Re: [ceph-users] Ceph release cadence
As a user, I woul like to add, I would like to see a real 2 year support for LTS releases. Hammer releases were sketchy at best in 2017. When luminous was released The outstanding bugs were auto closed, good buy and good readance. Also the decision to drop certain OS support created a barrier to upgrade and looking at jewel and luminous upgrade path where you cannot easily go back after upgrade is completed doesn't add the confidence. So making upgrades less radical may help production support to be more consistent and update process less dangerous. I would say 9 month is a good reference point but for me it is ready when it is really ready and tested. Keeping development release may be better for devs and early adopters. I don't believe production admins would go for intermediate one's as they being released now. This is only MHO and may be wrong. On Sep 9, 2017 15:32, "Christian Theune" wrote: > Hi, > > have been using Ceph for multiple years now. It’s unclear to me which of > your options fits best, but here are my preferences: > > * Updates are risky in a way that we tend to rather not do them every > year. Also, having seen jewel, we’ve been well off to avoid two > major issues what would have bitten us and will upgrade from hammer in > the next month or so. > > * Non-production releases are of not much value to me, as I have to keep > our dev/staging/prod clusters in sync to work on our stuff. > As you can never downgrade, there’s no value in it for me to try > non-production releases (without frying dev for everyone). > > * I’d prefer stability over new features. *Specifically* that new features > can be properly recombined with existing features (and each > other) without leading to surprises. (E.g. cache tiering breaking with > snapshots and then no way going back and a general notion of > “that combination wasn’t really well tested). > > * I’d prefer versions that I have to be maintained for production-critical > issues maybe 2 years, so I can have some time after a new > production release that overlaps with the new production release > receiving important bug fixes until I switch. > > Maybe this is close to what your "Drop the odd releases, and aim for a ~9 > month cadence.” would say. Waiting for a feature for a year is a pain, but > my personal goal for Ceph is that it first has to work properly, meaning: > not loose your data, not "stopping the show”, and not drawing you into a > corner you can’t get out. > > That’s my perspective as a user. As a fellow developer I feel your pain > about wanting to release faster and reducing maintenance load, so thanks > for asking! > > Hope this helps, > Christian > > > On Sep 6, 2017, at 5:23 PM, Sage Weil wrote: > > > > Hi everyone, > > > > Traditionally, we have done a major named "stable" release twice a year, > > and every other such release has been an "LTS" release, with fixes > > backported for 1-2 years. > > > > With kraken and luminous we missed our schedule by a lot: instead of > > releasing in October and April we released in January and August. > > > > A few observations: > > > > - Not a lot of people seem to run the "odd" releases (e.g., infernalis, > > kraken). This limits the value of actually making them. It also means > > that those who *do* run them are running riskier code (fewer users -> > more > > bugs). > > > > - The more recent requirement that upgrading clusters must make a stop at > > each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel > > -> lumninous) has been hugely helpful on the development side by reducing > > the amount of cross-version compatibility code to maintain and reducing > > the number of upgrade combinations to test. > > > > - When we try to do a time-based "train" release cadence, there always > > seems to be some "must-have" thing that delays the release a bit. This > > doesn't happen as much with the odd releases, but it definitely happens > > with the LTS releases. When the next LTS is a year away, it is hard to > > suck it up and wait that long. > > > > A couple of options: > > > > * Keep even/odd pattern, and continue being flexible with release dates > > > > + flexible > > - unpredictable > > - odd releases of dubious value > > > > * Keep even/odd pattern, but force a 'train' model with a more regular > > cadence > > > > + predictable schedule > > - some features will miss the target and be delayed a year > > > > * Drop the odd releases but change nothing else (i.e., 12-month release > > cadence) > > > > + eliminate the confusing odd releases with dubious value > > > > * Drop the odd releases, and aim for a ~9 month cadence. This splits the > > difference between the current even/odd pattern we've been doing. > > > > + eliminate the confusing odd releases with dubious value > > + waiting for the next release isn't quite as bad > > - required upgrades every 9 months instead of ever 12 months > > > > * Drop the odd releases, but relax the "must upgr
Re: [ceph-users] Mon Create currently at the state of probing
Do you have firewall on on new server by any chance? On Sun, Jun 18, 2017 at 8:18 PM, Jim Forde wrote: > I have an eight node ceph cluster running Jewel 10.2.5. > > One Ceph-Deploy node. Four OSD nodes and three Monitor nodes. > > Ceph-Deploy node is r710T > > OSD’s are r710a, r710b, r710c, and r710d. > > Mon’s are r710e, r710f, and r710g. > > Name resolution is in Hosts file on each node. > > > > Successfully removed Monitor r710e from cluster > > Upgraded ceph-deploy node r710T to Kraken 11.2.0 (ceph -v returns 11.2.0 > all other nodes are still 10.2.5) > > Ceph -s is HEALTH_OK 2 mons > > Rebuilt r710e with same OS (ubutnu 14.04 LTS) and same IP address. > > “Ceph-deploy install –release kraken r710e” is successful with ceph -v > returning 11.2.0 on node r710e > > “ceph-deploy admin r710e” is successful and puts the keyring in > /etc/ceph/ceph.client.admin.keyring > > “sudo chmod +r /etc/ceph/ceph.client.admin.keyring” > > > > Everything seems successful to this point. > > Then I run > > “ceph-deploy mon create r710e” and I get the following: > > > > [r710e][DEBUG ] ** > ** > > [r710e][INFO ] monitor: mon.r710e is currently at the state of probing > > [r710e][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon > /var/run/ceph/ceph-mon.r710e.asok mon_status > > [r710e][WARNIN] r710e is not defined in `mon initial members` > > [r710e][WARNIN] monitor r710e does not exist in monmap > > > > R710e is in the ‘mon initial members’. > > It is in the ceph.conf file correctly (it was running before and the > parameters have not changed) Public and Cluster networks are defined. > > It is the same physical server with the same (but freshly installed) OS > and same IP address. > > Looking at the local daemon mon_status on all three monitors I see. > > R710f and r710g see r710e as an “extra_probe_peers” > > R710e sees r710f and r710g as “extra_probe_peers” > > > > “ceph-deploy purge r710e” and “ceph-deploy purgedata r710e” with a reboot > of the 2 mon’s brings cluster back to HEALTH_OK > > > > Not sure what is going on. Is Ceph allergic to single node upgrades? > Afraid to push the upgrade on all mon’s. > > > > What I have done: > > Rebuilt r710e with different hardware. Rebuilt with different OS. Rebuilt > with different name and IP address. Same result. > > I have also restructured the NTP server. R710T is my NTP server on the > cluster. (HEALTH_OK prior to updating) I reset all Mon nodes to get time > from Ubuntu default NTP sources. Same error. > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Hammer update
I run centos 6.8 so no 0.94.10 packages for el6. On Mar 2, 2017 8:47 AM, "Abhishek L" wrote: Sasha Litvak writes: > Hello everyone, > > Hammer 0.94.10 update was announced in the blog a week ago. However, there are no packages available for either version of redhat. Can someone tell me what is going on? I see the packages at http://download.ceph.com/rpm-hammer/el7/x86_64/. Are you able to see the packages after following the instructions at http://docs.ceph.com/docs/master/install/get-packages/ ? Best, Abhishek -- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Hammer update
Hello everyone, Hammer 0.94.10 update was announced in the blog a week ago. However, there are no packages available for either version of redhat. Can someone tell me what is going on? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
