[ceph-users] Re: 16.2.11 pacific QE validation status
Dashboard lgtm!
Regards,
Nizam
On Fri, Jan 20, 2023, 22:09 Yuri Weinstein wrote:
> The overall progress on this release is looking much better and if we
> can approve it we can plan to publish it early next week.
>
> Still seeking approvals
>
> rados - Neha, Laura
> rook - Sébastien Han
> cephadm - Adam
> dashboard - Ernesto
> rgw - Casey
> rbd - Ilya (full rbd run in progress now)
> krbd - Ilya
> fs - Venky, Patrick
> upgrade/nautilus-x (pacific) - passed thx Adam Kraitman!
> upgrade/octopus-x (pacific) - almost passed, still running 1 job
> upgrade/pacific-p2p - Neha (same as in 16.2.8)
> powercycle - Brad (see new SELinux denials)
>
> On Tue, Jan 17, 2023 at 10:45 AM Yuri Weinstein
> wrote:
> >
> > OK I will rerun failed jobs filtering rhel in
> >
> > Thx!
> >
> > On Tue, Jan 17, 2023 at 10:43 AM Adam Kraitman
> wrote:
> > >
> > > Hey the satellite issue was fixed
> > >
> > > Thanks
> > >
> > > On Tue, Jan 17, 2023 at 7:43 PM Laura Flores
> wrote:
> > >>
> > >> This was my summary of rados failures. There was nothing new or amiss,
> > >> although it is important to note that runs were done with filtering
> out
> > >> rhel 8.
> > >>
> > >> I will leave it to Neha for final approval.
> > >>
> > >> Failures:
> > >> 1. https://tracker.ceph.com/issues/58258
> > >> 2. https://tracker.ceph.com/issues/58146
> > >> 3. https://tracker.ceph.com/issues/58458
> > >> 4. https://tracker.ceph.com/issues/57303
> > >> 5. https://tracker.ceph.com/issues/54071
> > >>
> > >> Details:
> > >> 1. rook: kubelet fails from connection refused - Ceph -
> Orchestrator
> > >> 2. test_cephadm.sh: Error: Error initializing source docker://
> > >> quay.ceph.io/ceph-ci/ceph:master - Ceph - Orchestrator
> > >> 3. qa/workunits/post-file.sh: postf...@drop.ceph.com: Permission
> denied
> > >> - Ceph
> > >> 4. rados/cephadm: Failed to fetch package version from
> > >>
> https://shaman.ceph.com/api/search/?status=ready=ceph=default=ubuntu%2F22.04%2Fx86_64=b34ca7d1c2becd6090874ccda56ef4cd8dc64bf7
> > >> - Ceph - Orchestrator
> > >> 5. rados/cephadm/osds: Invalid command: missing required parameter
> > >> hostname() - Ceph - Orchestrator
> > >>
> > >> On Tue, Jan 17, 2023 at 9:48 AM Yuri Weinstein
> wrote:
> > >>
> > >> > Please see the test results on the rebased RC 6.6 in this comment:
> > >> >
> > >> > https://tracker.ceph.com/issues/58257#note-2
> > >> >
> > >> > We're still having infrastructure issues making testing difficult.
> > >> > Therefore all reruns were done excluding the rhel 8 distro
> > >> > ('--filter-out rhel_8')
> > >> >
> > >> > Also, the upgrades failed and Adam is looking into this.
> > >> >
> > >> > Seeking new approvals
> > >> >
> > >> > rados - Neha, Laura
> > >> > rook - Sébastien Han
> > >> > cephadm - Adam
> > >> > dashboard - Ernesto
> > >> > rgw - Casey
> > >> > rbd - Ilya
> > >> > krbd - Ilya
> > >> > fs - Venky, Patrick
> > >> > upgrade/nautilus-x (pacific) - Adam Kraitman
> > >> > upgrade/octopus-x (pacific) - Adam Kraitman
> > >> > upgrade/pacific-p2p - Neha - Adam Kraitman
> > >> > powercycle - Brad
> > >> >
> > >> > Thx
> > >> >
> > >> > On Fri, Jan 6, 2023 at 8:37 AM Yuri Weinstein
> wrote:
> > >> > >
> > >> > > Happy New Year all!
> > >> > >
> > >> > > This release remains to be in "progress"/"on hold" status as we
> are
> > >> > > sorting all infrastructure-related issues.
> > >> > >
> > >> > > Unless I hear objections, I suggest doing a full rebase/retest QE
> > >> > > cycle (adding PRs merged lately) since it's taking much longer
> than
> > >> > > anticipated when sepia is back online.
> > >> > >
> > >> > > Objections?
> > >> > >
> > >> > > Thx
> > >> > > YuriW
> > >> > >
> > >> > > On Thu, Dec 15, 2022 at 9:14 AM Yuri Weinstein <
> ywein...@redhat.com>
> > >> > wrote:
> > >> > > >
> > >> > > > Details of this release are summarized here:
> > >> > > >
> > >> > > > https://tracker.ceph.com/issues/58257#note-1
> > >> > > > Release Notes - TBD
> > >> > > >
> > >> > > > Seeking approvals for:
> > >> > > >
> > >> > > > rados - Neha (https://github.com/ceph/ceph/pull/49431 is still
> being
> > >> > > > tested and will be merged soon)
> > >> > > > rook - Sébastien Han
> > >> > > > cephadm - Adam
> > >> > > > dashboard - Ernesto
> > >> > > > rgw - Casey (rwg will be rerun on the latest SHA1)
> > >> > > > rbd - Ilya, Deepika
> > >> > > > krbd - Ilya, Deepika
> > >> > > > fs - Venky, Patrick
> > >> > > > upgrade/nautilus-x (pacific) - Neha, Laura
> > >> > > > upgrade/octopus-x (pacific) - Neha, Laura
> > >> > > > upgrade/pacific-p2p - Neha - Neha, Laura
> > >> > > > powercycle - Brad
> > >> > > > ceph-volume - Guillaume, Adam K
> > >> > > >
> > >> > > > Thx
> > >> > > > YuriW
> > >> > ___
> > >> > Dev mailing list -- d...@ceph.io
> > >> > To unsubscribe send an email to dev-le...@ceph.io
> > >> >
> > >>
> > >>
> > >> --
> > >>
> > >> Laura Flores
> > >>
> > >> She/Her/Hers
> > >>
> > >> Software Engineer, Ceph Storage
>
[ceph-users] Re: Retrieve number of read/write operations for a particular file in Cephfs
Hello, On Mon, Jan 16, 2023 at 11:04 AM thanh son le wrote: > > Hi, > > I have been studying the document from Ceph and Rados but I could not find > any metrics to measure the number of read/write operations for each file. I > understand that Cephfs is the front-end, the file is going to be stored as > an object in the OSD and I have found that Ceph provides a Cache Tiering > feature which also requires the monitor for read/write operation for each > object. Could someone please give me guidance on how this is achieved? > Thanks. You can try to get the "perf dump" from ceph-fuse to measure. The osd write/reads are not fine-grained though. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 16.2.11 pacific QE validation status
On Fri, Jan 20, 2023 at 12:36 PM Laura Flores wrote:
> From my end, rados looks good. All failures are known. Leaving final
> approval to Neha.
>
> On Fri, Jan 20, 2023 at 12:03 PM Ernesto Puerta
> wrote:
>
>> CCing Nizam as Dashboard lead for review & approval.
>>
>> Kind Regards,
>> Ernesto
>>
>>
>> On Fri, Jan 20, 2023 at 6:42 PM Adam King wrote:
>>
>>> cephadm approved. Known failures.
>>>
>>> On Fri, Jan 20, 2023 at 11:39 AM Yuri Weinstein
>>> wrote:
>>>
The overall progress on this release is looking much better and if we
can approve it we can plan to publish it early next week.
Still seeking approvals
rados - Neha, Laura
>>>
No new issues were found in the recent runs done for the rhel jobs, rados
approved.
> rook - Sébastien Han
cephadm - Adam
dashboard - Ernesto
rgw - Casey
rbd - Ilya (full rbd run in progress now)
krbd - Ilya
fs - Venky, Patrick
upgrade/nautilus-x (pacific) - passed thx Adam Kraitman!
upgrade/octopus-x (pacific) - almost passed, still running 1 job
upgrade/pacific-p2p - Neha (same as in 16.2.8)
>>>
Agreed.
Thanks,
Neha
> powercycle - Brad (see new SELinux denials)
On Tue, Jan 17, 2023 at 10:45 AM Yuri Weinstein
wrote:
>
> OK I will rerun failed jobs filtering rhel in
>
> Thx!
>
> On Tue, Jan 17, 2023 at 10:43 AM Adam Kraitman
wrote:
> >
> > Hey the satellite issue was fixed
> >
> > Thanks
> >
> > On Tue, Jan 17, 2023 at 7:43 PM Laura Flores
wrote:
> >>
> >> This was my summary of rados failures. There was nothing new or
amiss,
> >> although it is important to note that runs were done with
filtering out
> >> rhel 8.
> >>
> >> I will leave it to Neha for final approval.
> >>
> >> Failures:
> >> 1. https://tracker.ceph.com/issues/58258
> >> 2. https://tracker.ceph.com/issues/58146
> >> 3. https://tracker.ceph.com/issues/58458
> >> 4. https://tracker.ceph.com/issues/57303
> >> 5. https://tracker.ceph.com/issues/54071
> >>
> >> Details:
> >> 1. rook: kubelet fails from connection refused - Ceph -
Orchestrator
> >> 2. test_cephadm.sh: Error: Error initializing source docker://
> >> quay.ceph.io/ceph-ci/ceph:master - Ceph - Orchestrator
> >> 3. qa/workunits/post-file.sh: postf...@drop.ceph.com:
Permission denied
> >> - Ceph
> >> 4. rados/cephadm: Failed to fetch package version from
> >>
https://shaman.ceph.com/api/search/?status=ready=ceph=default=ubuntu%2F22.04%2Fx86_64=b34ca7d1c2becd6090874ccda56ef4cd8dc64bf7
> >> - Ceph - Orchestrator
> >> 5. rados/cephadm/osds: Invalid command: missing required
parameter
> >> hostname() - Ceph - Orchestrator
> >>
> >> On Tue, Jan 17, 2023 at 9:48 AM Yuri Weinstein <
ywein...@redhat.com> wrote:
> >>
> >> > Please see the test results on the rebased RC 6.6 in this
comment:
> >> >
> >> > https://tracker.ceph.com/issues/58257#note-2
> >> >
> >> > We're still having infrastructure issues making testing
difficult.
> >> > Therefore all reruns were done excluding the rhel 8 distro
> >> > ('--filter-out rhel_8')
> >> >
> >> > Also, the upgrades failed and Adam is looking into this.
> >> >
> >> > Seeking new approvals
> >> >
> >> > rados - Neha, Laura
> >> > rook - Sébastien Han
> >> > cephadm - Adam
> >> > dashboard - Ernesto
> >> > rgw - Casey
> >> > rbd - Ilya
> >> > krbd - Ilya
> >> > fs - Venky, Patrick
> >> > upgrade/nautilus-x (pacific) - Adam Kraitman
> >> > upgrade/octopus-x (pacific) - Adam Kraitman
> >> > upgrade/pacific-p2p - Neha - Adam Kraitman
> >> > powercycle - Brad
> >> >
> >> > Thx
> >> >
> >> > On Fri, Jan 6, 2023 at 8:37 AM Yuri Weinstein <
ywein...@redhat.com> wrote:
> >> > >
> >> > > Happy New Year all!
> >> > >
> >> > > This release remains to be in "progress"/"on hold" status as
we are
> >> > > sorting all infrastructure-related issues.
> >> > >
> >> > > Unless I hear objections, I suggest doing a full rebase/retest
QE
> >> > > cycle (adding PRs merged lately) since it's taking much longer
than
> >> > > anticipated when sepia is back online.
> >> > >
> >> > > Objections?
> >> > >
> >> > > Thx
> >> > > YuriW
> >> > >
> >> > > On Thu, Dec 15, 2022 at 9:14 AM Yuri Weinstein <
ywein...@redhat.com>
> >> > wrote:
> >> > > >
> >> > > > Details of this release are summarized here:
> >> > > >
> >> > > > https://tracker.ceph.com/issues/58257#note-1
> >> > > > Release Notes - TBD
> >> > > >
> >> > > > Seeking approvals for:
> >> > > >
> >> > > > rados -
[ceph-users] Re: 16.2.11 pacific QE validation status
From my end, rados looks good. All failures are known. Leaving final
approval to Neha.
On Fri, Jan 20, 2023 at 12:03 PM Ernesto Puerta wrote:
> CCing Nizam as Dashboard lead for review & approval.
>
> Kind Regards,
> Ernesto
>
>
> On Fri, Jan 20, 2023 at 6:42 PM Adam King wrote:
>
>> cephadm approved. Known failures.
>>
>> On Fri, Jan 20, 2023 at 11:39 AM Yuri Weinstein
>> wrote:
>>
>>> The overall progress on this release is looking much better and if we
>>> can approve it we can plan to publish it early next week.
>>>
>>> Still seeking approvals
>>>
>>> rados - Neha, Laura
>>> rook - Sébastien Han
>>> cephadm - Adam
>>> dashboard - Ernesto
>>> rgw - Casey
>>> rbd - Ilya (full rbd run in progress now)
>>> krbd - Ilya
>>> fs - Venky, Patrick
>>> upgrade/nautilus-x (pacific) - passed thx Adam Kraitman!
>>> upgrade/octopus-x (pacific) - almost passed, still running 1 job
>>> upgrade/pacific-p2p - Neha (same as in 16.2.8)
>>> powercycle - Brad (see new SELinux denials)
>>>
>>> On Tue, Jan 17, 2023 at 10:45 AM Yuri Weinstein
>>> wrote:
>>> >
>>> > OK I will rerun failed jobs filtering rhel in
>>> >
>>> > Thx!
>>> >
>>> > On Tue, Jan 17, 2023 at 10:43 AM Adam Kraitman
>>> wrote:
>>> > >
>>> > > Hey the satellite issue was fixed
>>> > >
>>> > > Thanks
>>> > >
>>> > > On Tue, Jan 17, 2023 at 7:43 PM Laura Flores
>>> wrote:
>>> > >>
>>> > >> This was my summary of rados failures. There was nothing new or
>>> amiss,
>>> > >> although it is important to note that runs were done with filtering
>>> out
>>> > >> rhel 8.
>>> > >>
>>> > >> I will leave it to Neha for final approval.
>>> > >>
>>> > >> Failures:
>>> > >> 1. https://tracker.ceph.com/issues/58258
>>> > >> 2. https://tracker.ceph.com/issues/58146
>>> > >> 3. https://tracker.ceph.com/issues/58458
>>> > >> 4. https://tracker.ceph.com/issues/57303
>>> > >> 5. https://tracker.ceph.com/issues/54071
>>> > >>
>>> > >> Details:
>>> > >> 1. rook: kubelet fails from connection refused - Ceph -
>>> Orchestrator
>>> > >> 2. test_cephadm.sh: Error: Error initializing source docker://
>>> > >> quay.ceph.io/ceph-ci/ceph:master - Ceph - Orchestrator
>>> > >> 3. qa/workunits/post-file.sh: postf...@drop.ceph.com:
>>> Permission denied
>>> > >> - Ceph
>>> > >> 4. rados/cephadm: Failed to fetch package version from
>>> > >>
>>> https://shaman.ceph.com/api/search/?status=ready=ceph=default=ubuntu%2F22.04%2Fx86_64=b34ca7d1c2becd6090874ccda56ef4cd8dc64bf7
>>> > >> - Ceph - Orchestrator
>>> > >> 5. rados/cephadm/osds: Invalid command: missing required
>>> parameter
>>> > >> hostname() - Ceph - Orchestrator
>>> > >>
>>> > >> On Tue, Jan 17, 2023 at 9:48 AM Yuri Weinstein
>>> wrote:
>>> > >>
>>> > >> > Please see the test results on the rebased RC 6.6 in this comment:
>>> > >> >
>>> > >> > https://tracker.ceph.com/issues/58257#note-2
>>> > >> >
>>> > >> > We're still having infrastructure issues making testing difficult.
>>> > >> > Therefore all reruns were done excluding the rhel 8 distro
>>> > >> > ('--filter-out rhel_8')
>>> > >> >
>>> > >> > Also, the upgrades failed and Adam is looking into this.
>>> > >> >
>>> > >> > Seeking new approvals
>>> > >> >
>>> > >> > rados - Neha, Laura
>>> > >> > rook - Sébastien Han
>>> > >> > cephadm - Adam
>>> > >> > dashboard - Ernesto
>>> > >> > rgw - Casey
>>> > >> > rbd - Ilya
>>> > >> > krbd - Ilya
>>> > >> > fs - Venky, Patrick
>>> > >> > upgrade/nautilus-x (pacific) - Adam Kraitman
>>> > >> > upgrade/octopus-x (pacific) - Adam Kraitman
>>> > >> > upgrade/pacific-p2p - Neha - Adam Kraitman
>>> > >> > powercycle - Brad
>>> > >> >
>>> > >> > Thx
>>> > >> >
>>> > >> > On Fri, Jan 6, 2023 at 8:37 AM Yuri Weinstein <
>>> ywein...@redhat.com> wrote:
>>> > >> > >
>>> > >> > > Happy New Year all!
>>> > >> > >
>>> > >> > > This release remains to be in "progress"/"on hold" status as we
>>> are
>>> > >> > > sorting all infrastructure-related issues.
>>> > >> > >
>>> > >> > > Unless I hear objections, I suggest doing a full rebase/retest
>>> QE
>>> > >> > > cycle (adding PRs merged lately) since it's taking much longer
>>> than
>>> > >> > > anticipated when sepia is back online.
>>> > >> > >
>>> > >> > > Objections?
>>> > >> > >
>>> > >> > > Thx
>>> > >> > > YuriW
>>> > >> > >
>>> > >> > > On Thu, Dec 15, 2022 at 9:14 AM Yuri Weinstein <
>>> ywein...@redhat.com>
>>> > >> > wrote:
>>> > >> > > >
>>> > >> > > > Details of this release are summarized here:
>>> > >> > > >
>>> > >> > > > https://tracker.ceph.com/issues/58257#note-1
>>> > >> > > > Release Notes - TBD
>>> > >> > > >
>>> > >> > > > Seeking approvals for:
>>> > >> > > >
>>> > >> > > > rados - Neha (https://github.com/ceph/ceph/pull/49431 is
>>> still being
>>> > >> > > > tested and will be merged soon)
>>> > >> > > > rook - Sébastien Han
>>> > >> > > > cephadm - Adam
>>> > >> > > > dashboard - Ernesto
>>> > >> > > > rgw - Casey (rwg will be rerun on the latest SHA1)
>>> > >> > > > rbd - Ilya, Deepika
>>> > >> > > > krbd - Ilya, Deepika
>>> > >> >
[ceph-users] Re: trouble deploying custom config OSDs
Hello,
Thank you for the valuable info, and especially for the slack link (it's not
listed on the community page)
The ceph-volume command was issued in the following manner :
login to my 1st vps from which I performed the boostrap with cephadm
exec
> sudo cephadm shell
which gets me root shell inside the container and then ceph-volume [...] etc.
-
I nuked my environment to recreate the issues and paste them here so my new
vg/lv names are different
> root@dev0:/# ceph-volume raw prepare --bluestore --data /dev/sdb --block.db
> /dev/mapper/ssd0-ssd0_0
> Running command: /usr/bin/ceph-authtool --gen-print-key
> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd
> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
> 0011a2a8-084b-4d79-ab8f-2503dfc2c804
> stderr: 2023-01-20T11:50:23.495+ 7fdeebd02700 -1 auth: unable to find a
> keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or
> directory
> stderr: 2023-01-20T11:50:23.495+ 7fdeebd02700 -1
> AuthRegistry(0x7fdee4060d70) no keyring found at
> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
> stderr: 2023-01-20T11:50:23.495+ 7fdeebd02700 -1 auth: unable to find a
> keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or
> directory
> stderr: 2023-01-20T11:50:23.495+ 7fdeebd02700 -1
> AuthRegistry(0x7fdee4064440) no keyring found at
> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
> stderr: 2023-01-20T11:50:23.499+ 7fdeebd02700 -1 auth: unable to find a
> keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or
> directory
> stderr: 2023-01-20T11:50:23.499+ 7fdeebd02700 -1
> AuthRegistry(0x7fdeebd00ea0) no keyring found at
> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
> stderr: 2023-01-20T11:50:23.503+ 7fdee929d700 -1 monclient(hunting):
> handle_auth_bad_method server allowed_methods [2] but i only support [1]
> stderr: 2023-01-20T11:50:23.503+ 7fdeea29f700 -1 monclient(hunting):
> handle_auth_bad_method server allowed_methods [2] but i only support [1]
> stderr: 2023-01-20T11:50:23.503+ 7fdee9a9e700 -1 monclient(hunting):
> handle_auth_bad_method server allowed_methods [2] but i only support [1]
> stderr: 2023-01-20T11:50:23.503+ 7fdeebd02700 -1 monclient: authenticate
> NOTE: no keyring found; disabled cephx authentication
> stderr: [errno 13] RADOS permission denied (error connecting to the cluster)
> --> RuntimeError: Unable to create a new OSD id
After manually ln -s /etc/ceph/ceph.keyring /var/lib/ceph/bootstrap-osd/ i got
the the credentials from ceph auth ls and added them to the keyring file
respecting it's syntax
> [client.bootstrap-osd]
> key = AQA5vcdj/pClABAAt9hDro+HC73wrZysJSHyAg==
> caps mon = "allow profile bootstrap-osd"
Then it worked:
> root@dev0:/# ceph-volume raw prepare --bluestore --data /dev/sdb --block.db
> /dev/mapper/ssd0-ssd0_0
> Running command: /usr/bin/ceph-authtool --gen-print-key
> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd
> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
> 4d47af7e-cf8c-451a-8773-894854e3ce8a
> Running command: /usr/bin/ceph-authtool --gen-print-key
> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-3
> Running command: /usr/bin/chown -R ceph:ceph /dev/sdb
> Running command: /usr/bin/ln -s /dev/sdb /var/lib/ceph/osd/ceph-3/block
> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd
> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
> /var/lib/ceph/osd/ceph-3/activate.monmap
> stderr: got monmap epoch 3
> --> Creating keyring file for osd.3
> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/keyring
> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/
> Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ssd0-ssd0_0
> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
> Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore
> --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile -
> --bluestore-block-db-path /dev/mapper/ssd0-ssd0_0 --osd-data
> /var/lib/ceph/osd/ceph-3/ --osd-uuid 4d47af7e-cf8c-451a-8773-894854e3ce8a
> --setuser ceph --setgroup ceph
> stderr: 2023-01-20T11:50:57.723+ 7f0ee2e4b3c0 -1
> bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid-->
> ceph-volume raw clear prepare successful for: /dev/sdb
So it creates osd entries in /var/lib/ceph/osd
As for the mgr logs, I'm trying to figure out how to get them: I listed all
pods using podman ps on the host then podman logs of the node
that has mgr in the name. It's a lot to parse but I found something relevant:
> 2023-01-20T12:02:14.680+ 7f083900 -1 log_channel(cephadm) log [ERR] :
> Failed to apply osd.all-available-devices spec
> DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
> service_id:
[ceph-users] Re: 16.2.11 pacific QE validation status
CCing Nizam as Dashboard lead for review & approval.
Kind Regards,
Ernesto
On Fri, Jan 20, 2023 at 6:42 PM Adam King wrote:
> cephadm approved. Known failures.
>
> On Fri, Jan 20, 2023 at 11:39 AM Yuri Weinstein
> wrote:
>
>> The overall progress on this release is looking much better and if we
>> can approve it we can plan to publish it early next week.
>>
>> Still seeking approvals
>>
>> rados - Neha, Laura
>> rook - Sébastien Han
>> cephadm - Adam
>> dashboard - Ernesto
>> rgw - Casey
>> rbd - Ilya (full rbd run in progress now)
>> krbd - Ilya
>> fs - Venky, Patrick
>> upgrade/nautilus-x (pacific) - passed thx Adam Kraitman!
>> upgrade/octopus-x (pacific) - almost passed, still running 1 job
>> upgrade/pacific-p2p - Neha (same as in 16.2.8)
>> powercycle - Brad (see new SELinux denials)
>>
>> On Tue, Jan 17, 2023 at 10:45 AM Yuri Weinstein
>> wrote:
>> >
>> > OK I will rerun failed jobs filtering rhel in
>> >
>> > Thx!
>> >
>> > On Tue, Jan 17, 2023 at 10:43 AM Adam Kraitman
>> wrote:
>> > >
>> > > Hey the satellite issue was fixed
>> > >
>> > > Thanks
>> > >
>> > > On Tue, Jan 17, 2023 at 7:43 PM Laura Flores
>> wrote:
>> > >>
>> > >> This was my summary of rados failures. There was nothing new or
>> amiss,
>> > >> although it is important to note that runs were done with filtering
>> out
>> > >> rhel 8.
>> > >>
>> > >> I will leave it to Neha for final approval.
>> > >>
>> > >> Failures:
>> > >> 1. https://tracker.ceph.com/issues/58258
>> > >> 2. https://tracker.ceph.com/issues/58146
>> > >> 3. https://tracker.ceph.com/issues/58458
>> > >> 4. https://tracker.ceph.com/issues/57303
>> > >> 5. https://tracker.ceph.com/issues/54071
>> > >>
>> > >> Details:
>> > >> 1. rook: kubelet fails from connection refused - Ceph -
>> Orchestrator
>> > >> 2. test_cephadm.sh: Error: Error initializing source docker://
>> > >> quay.ceph.io/ceph-ci/ceph:master - Ceph - Orchestrator
>> > >> 3. qa/workunits/post-file.sh: postf...@drop.ceph.com:
>> Permission denied
>> > >> - Ceph
>> > >> 4. rados/cephadm: Failed to fetch package version from
>> > >>
>> https://shaman.ceph.com/api/search/?status=ready=ceph=default=ubuntu%2F22.04%2Fx86_64=b34ca7d1c2becd6090874ccda56ef4cd8dc64bf7
>> > >> - Ceph - Orchestrator
>> > >> 5. rados/cephadm/osds: Invalid command: missing required
>> parameter
>> > >> hostname() - Ceph - Orchestrator
>> > >>
>> > >> On Tue, Jan 17, 2023 at 9:48 AM Yuri Weinstein
>> wrote:
>> > >>
>> > >> > Please see the test results on the rebased RC 6.6 in this comment:
>> > >> >
>> > >> > https://tracker.ceph.com/issues/58257#note-2
>> > >> >
>> > >> > We're still having infrastructure issues making testing difficult.
>> > >> > Therefore all reruns were done excluding the rhel 8 distro
>> > >> > ('--filter-out rhel_8')
>> > >> >
>> > >> > Also, the upgrades failed and Adam is looking into this.
>> > >> >
>> > >> > Seeking new approvals
>> > >> >
>> > >> > rados - Neha, Laura
>> > >> > rook - Sébastien Han
>> > >> > cephadm - Adam
>> > >> > dashboard - Ernesto
>> > >> > rgw - Casey
>> > >> > rbd - Ilya
>> > >> > krbd - Ilya
>> > >> > fs - Venky, Patrick
>> > >> > upgrade/nautilus-x (pacific) - Adam Kraitman
>> > >> > upgrade/octopus-x (pacific) - Adam Kraitman
>> > >> > upgrade/pacific-p2p - Neha - Adam Kraitman
>> > >> > powercycle - Brad
>> > >> >
>> > >> > Thx
>> > >> >
>> > >> > On Fri, Jan 6, 2023 at 8:37 AM Yuri Weinstein
>> wrote:
>> > >> > >
>> > >> > > Happy New Year all!
>> > >> > >
>> > >> > > This release remains to be in "progress"/"on hold" status as we
>> are
>> > >> > > sorting all infrastructure-related issues.
>> > >> > >
>> > >> > > Unless I hear objections, I suggest doing a full rebase/retest QE
>> > >> > > cycle (adding PRs merged lately) since it's taking much longer
>> than
>> > >> > > anticipated when sepia is back online.
>> > >> > >
>> > >> > > Objections?
>> > >> > >
>> > >> > > Thx
>> > >> > > YuriW
>> > >> > >
>> > >> > > On Thu, Dec 15, 2022 at 9:14 AM Yuri Weinstein <
>> ywein...@redhat.com>
>> > >> > wrote:
>> > >> > > >
>> > >> > > > Details of this release are summarized here:
>> > >> > > >
>> > >> > > > https://tracker.ceph.com/issues/58257#note-1
>> > >> > > > Release Notes - TBD
>> > >> > > >
>> > >> > > > Seeking approvals for:
>> > >> > > >
>> > >> > > > rados - Neha (https://github.com/ceph/ceph/pull/49431 is
>> still being
>> > >> > > > tested and will be merged soon)
>> > >> > > > rook - Sébastien Han
>> > >> > > > cephadm - Adam
>> > >> > > > dashboard - Ernesto
>> > >> > > > rgw - Casey (rwg will be rerun on the latest SHA1)
>> > >> > > > rbd - Ilya, Deepika
>> > >> > > > krbd - Ilya, Deepika
>> > >> > > > fs - Venky, Patrick
>> > >> > > > upgrade/nautilus-x (pacific) - Neha, Laura
>> > >> > > > upgrade/octopus-x (pacific) - Neha, Laura
>> > >> > > > upgrade/pacific-p2p - Neha - Neha, Laura
>> > >> > > > powercycle - Brad
>> > >> > > > ceph-volume - Guillaume, Adam K
>> > >> > > >
>> > >> > > > Thx
>> > >> > > > YuriW
[ceph-users] Re: 16.2.11 pacific QE validation status
cephadm approved. Known failures.
On Fri, Jan 20, 2023 at 11:39 AM Yuri Weinstein wrote:
> The overall progress on this release is looking much better and if we
> can approve it we can plan to publish it early next week.
>
> Still seeking approvals
>
> rados - Neha, Laura
> rook - Sébastien Han
> cephadm - Adam
> dashboard - Ernesto
> rgw - Casey
> rbd - Ilya (full rbd run in progress now)
> krbd - Ilya
> fs - Venky, Patrick
> upgrade/nautilus-x (pacific) - passed thx Adam Kraitman!
> upgrade/octopus-x (pacific) - almost passed, still running 1 job
> upgrade/pacific-p2p - Neha (same as in 16.2.8)
> powercycle - Brad (see new SELinux denials)
>
> On Tue, Jan 17, 2023 at 10:45 AM Yuri Weinstein
> wrote:
> >
> > OK I will rerun failed jobs filtering rhel in
> >
> > Thx!
> >
> > On Tue, Jan 17, 2023 at 10:43 AM Adam Kraitman
> wrote:
> > >
> > > Hey the satellite issue was fixed
> > >
> > > Thanks
> > >
> > > On Tue, Jan 17, 2023 at 7:43 PM Laura Flores
> wrote:
> > >>
> > >> This was my summary of rados failures. There was nothing new or amiss,
> > >> although it is important to note that runs were done with filtering
> out
> > >> rhel 8.
> > >>
> > >> I will leave it to Neha for final approval.
> > >>
> > >> Failures:
> > >> 1. https://tracker.ceph.com/issues/58258
> > >> 2. https://tracker.ceph.com/issues/58146
> > >> 3. https://tracker.ceph.com/issues/58458
> > >> 4. https://tracker.ceph.com/issues/57303
> > >> 5. https://tracker.ceph.com/issues/54071
> > >>
> > >> Details:
> > >> 1. rook: kubelet fails from connection refused - Ceph -
> Orchestrator
> > >> 2. test_cephadm.sh: Error: Error initializing source docker://
> > >> quay.ceph.io/ceph-ci/ceph:master - Ceph - Orchestrator
> > >> 3. qa/workunits/post-file.sh: postf...@drop.ceph.com: Permission
> denied
> > >> - Ceph
> > >> 4. rados/cephadm: Failed to fetch package version from
> > >>
> https://shaman.ceph.com/api/search/?status=ready=ceph=default=ubuntu%2F22.04%2Fx86_64=b34ca7d1c2becd6090874ccda56ef4cd8dc64bf7
> > >> - Ceph - Orchestrator
> > >> 5. rados/cephadm/osds: Invalid command: missing required parameter
> > >> hostname() - Ceph - Orchestrator
> > >>
> > >> On Tue, Jan 17, 2023 at 9:48 AM Yuri Weinstein
> wrote:
> > >>
> > >> > Please see the test results on the rebased RC 6.6 in this comment:
> > >> >
> > >> > https://tracker.ceph.com/issues/58257#note-2
> > >> >
> > >> > We're still having infrastructure issues making testing difficult.
> > >> > Therefore all reruns were done excluding the rhel 8 distro
> > >> > ('--filter-out rhel_8')
> > >> >
> > >> > Also, the upgrades failed and Adam is looking into this.
> > >> >
> > >> > Seeking new approvals
> > >> >
> > >> > rados - Neha, Laura
> > >> > rook - Sébastien Han
> > >> > cephadm - Adam
> > >> > dashboard - Ernesto
> > >> > rgw - Casey
> > >> > rbd - Ilya
> > >> > krbd - Ilya
> > >> > fs - Venky, Patrick
> > >> > upgrade/nautilus-x (pacific) - Adam Kraitman
> > >> > upgrade/octopus-x (pacific) - Adam Kraitman
> > >> > upgrade/pacific-p2p - Neha - Adam Kraitman
> > >> > powercycle - Brad
> > >> >
> > >> > Thx
> > >> >
> > >> > On Fri, Jan 6, 2023 at 8:37 AM Yuri Weinstein
> wrote:
> > >> > >
> > >> > > Happy New Year all!
> > >> > >
> > >> > > This release remains to be in "progress"/"on hold" status as we
> are
> > >> > > sorting all infrastructure-related issues.
> > >> > >
> > >> > > Unless I hear objections, I suggest doing a full rebase/retest QE
> > >> > > cycle (adding PRs merged lately) since it's taking much longer
> than
> > >> > > anticipated when sepia is back online.
> > >> > >
> > >> > > Objections?
> > >> > >
> > >> > > Thx
> > >> > > YuriW
> > >> > >
> > >> > > On Thu, Dec 15, 2022 at 9:14 AM Yuri Weinstein <
> ywein...@redhat.com>
> > >> > wrote:
> > >> > > >
> > >> > > > Details of this release are summarized here:
> > >> > > >
> > >> > > > https://tracker.ceph.com/issues/58257#note-1
> > >> > > > Release Notes - TBD
> > >> > > >
> > >> > > > Seeking approvals for:
> > >> > > >
> > >> > > > rados - Neha (https://github.com/ceph/ceph/pull/49431 is still
> being
> > >> > > > tested and will be merged soon)
> > >> > > > rook - Sébastien Han
> > >> > > > cephadm - Adam
> > >> > > > dashboard - Ernesto
> > >> > > > rgw - Casey (rwg will be rerun on the latest SHA1)
> > >> > > > rbd - Ilya, Deepika
> > >> > > > krbd - Ilya, Deepika
> > >> > > > fs - Venky, Patrick
> > >> > > > upgrade/nautilus-x (pacific) - Neha, Laura
> > >> > > > upgrade/octopus-x (pacific) - Neha, Laura
> > >> > > > upgrade/pacific-p2p - Neha - Neha, Laura
> > >> > > > powercycle - Brad
> > >> > > > ceph-volume - Guillaume, Adam K
> > >> > > >
> > >> > > > Thx
> > >> > > > YuriW
> > >> > ___
> > >> > Dev mailing list -- d...@ceph.io
> > >> > To unsubscribe send an email to dev-le...@ceph.io
> > >> >
> > >>
> > >>
> > >> --
> > >>
> > >> Laura Flores
> > >>
> > >> She/Her/Hers
> > >>
> > >> Software Engineer, Ceph
[ceph-users] Re: 16.2.11 pacific QE validation status
On Fri, Jan 20, 2023 at 11:39 AM Yuri Weinstein wrote:
>
> The overall progress on this release is looking much better and if we
> can approve it we can plan to publish it early next week.
>
> Still seeking approvals
>
> rados - Neha, Laura
> rook - Sébastien Han
> cephadm - Adam
> dashboard - Ernesto
> rgw - Casey
+1 rgw still approved
> rbd - Ilya (full rbd run in progress now)
> krbd - Ilya
> fs - Venky, Patrick
> upgrade/nautilus-x (pacific) - passed thx Adam Kraitman!
> upgrade/octopus-x (pacific) - almost passed, still running 1 job
> upgrade/pacific-p2p - Neha (same as in 16.2.8)
> powercycle - Brad (see new SELinux denials)
>
> On Tue, Jan 17, 2023 at 10:45 AM Yuri Weinstein wrote:
> >
> > OK I will rerun failed jobs filtering rhel in
> >
> > Thx!
> >
> > On Tue, Jan 17, 2023 at 10:43 AM Adam Kraitman wrote:
> > >
> > > Hey the satellite issue was fixed
> > >
> > > Thanks
> > >
> > > On Tue, Jan 17, 2023 at 7:43 PM Laura Flores wrote:
> > >>
> > >> This was my summary of rados failures. There was nothing new or amiss,
> > >> although it is important to note that runs were done with filtering out
> > >> rhel 8.
> > >>
> > >> I will leave it to Neha for final approval.
> > >>
> > >> Failures:
> > >> 1. https://tracker.ceph.com/issues/58258
> > >> 2. https://tracker.ceph.com/issues/58146
> > >> 3. https://tracker.ceph.com/issues/58458
> > >> 4. https://tracker.ceph.com/issues/57303
> > >> 5. https://tracker.ceph.com/issues/54071
> > >>
> > >> Details:
> > >> 1. rook: kubelet fails from connection refused - Ceph - Orchestrator
> > >> 2. test_cephadm.sh: Error: Error initializing source docker://
> > >> quay.ceph.io/ceph-ci/ceph:master - Ceph - Orchestrator
> > >> 3. qa/workunits/post-file.sh: postf...@drop.ceph.com: Permission
> > >> denied
> > >> - Ceph
> > >> 4. rados/cephadm: Failed to fetch package version from
> > >> https://shaman.ceph.com/api/search/?status=ready=ceph=default=ubuntu%2F22.04%2Fx86_64=b34ca7d1c2becd6090874ccda56ef4cd8dc64bf7
> > >> - Ceph - Orchestrator
> > >> 5. rados/cephadm/osds: Invalid command: missing required parameter
> > >> hostname() - Ceph - Orchestrator
> > >>
> > >> On Tue, Jan 17, 2023 at 9:48 AM Yuri Weinstein
> > >> wrote:
> > >>
> > >> > Please see the test results on the rebased RC 6.6 in this comment:
> > >> >
> > >> > https://tracker.ceph.com/issues/58257#note-2
> > >> >
> > >> > We're still having infrastructure issues making testing difficult.
> > >> > Therefore all reruns were done excluding the rhel 8 distro
> > >> > ('--filter-out rhel_8')
> > >> >
> > >> > Also, the upgrades failed and Adam is looking into this.
> > >> >
> > >> > Seeking new approvals
> > >> >
> > >> > rados - Neha, Laura
> > >> > rook - Sébastien Han
> > >> > cephadm - Adam
> > >> > dashboard - Ernesto
> > >> > rgw - Casey
> > >> > rbd - Ilya
> > >> > krbd - Ilya
> > >> > fs - Venky, Patrick
> > >> > upgrade/nautilus-x (pacific) - Adam Kraitman
> > >> > upgrade/octopus-x (pacific) - Adam Kraitman
> > >> > upgrade/pacific-p2p - Neha - Adam Kraitman
> > >> > powercycle - Brad
> > >> >
> > >> > Thx
> > >> >
> > >> > On Fri, Jan 6, 2023 at 8:37 AM Yuri Weinstein
> > >> > wrote:
> > >> > >
> > >> > > Happy New Year all!
> > >> > >
> > >> > > This release remains to be in "progress"/"on hold" status as we are
> > >> > > sorting all infrastructure-related issues.
> > >> > >
> > >> > > Unless I hear objections, I suggest doing a full rebase/retest QE
> > >> > > cycle (adding PRs merged lately) since it's taking much longer than
> > >> > > anticipated when sepia is back online.
> > >> > >
> > >> > > Objections?
> > >> > >
> > >> > > Thx
> > >> > > YuriW
> > >> > >
> > >> > > On Thu, Dec 15, 2022 at 9:14 AM Yuri Weinstein
> > >> > wrote:
> > >> > > >
> > >> > > > Details of this release are summarized here:
> > >> > > >
> > >> > > > https://tracker.ceph.com/issues/58257#note-1
> > >> > > > Release Notes - TBD
> > >> > > >
> > >> > > > Seeking approvals for:
> > >> > > >
> > >> > > > rados - Neha (https://github.com/ceph/ceph/pull/49431 is still
> > >> > > > being
> > >> > > > tested and will be merged soon)
> > >> > > > rook - Sébastien Han
> > >> > > > cephadm - Adam
> > >> > > > dashboard - Ernesto
> > >> > > > rgw - Casey (rwg will be rerun on the latest SHA1)
> > >> > > > rbd - Ilya, Deepika
> > >> > > > krbd - Ilya, Deepika
> > >> > > > fs - Venky, Patrick
> > >> > > > upgrade/nautilus-x (pacific) - Neha, Laura
> > >> > > > upgrade/octopus-x (pacific) - Neha, Laura
> > >> > > > upgrade/pacific-p2p - Neha - Neha, Laura
> > >> > > > powercycle - Brad
> > >> > > > ceph-volume - Guillaume, Adam K
> > >> > > >
> > >> > > > Thx
> > >> > > > YuriW
> > >> > ___
> > >> > Dev mailing list -- d...@ceph.io
> > >> > To unsubscribe send an email to dev-le...@ceph.io
> > >> >
> > >>
> > >>
> > >> --
> > >>
> > >> Laura Flores
> > >>
> > >> She/Her/Hers
> > >>
> > >> Software Engineer, Ceph Storage
> > >>
> > >>
[ceph-users] Re: 16.2.11 pacific QE validation status
The overall progress on this release is looking much better and if we
can approve it we can plan to publish it early next week.
Still seeking approvals
rados - Neha, Laura
rook - Sébastien Han
cephadm - Adam
dashboard - Ernesto
rgw - Casey
rbd - Ilya (full rbd run in progress now)
krbd - Ilya
fs - Venky, Patrick
upgrade/nautilus-x (pacific) - passed thx Adam Kraitman!
upgrade/octopus-x (pacific) - almost passed, still running 1 job
upgrade/pacific-p2p - Neha (same as in 16.2.8)
powercycle - Brad (see new SELinux denials)
On Tue, Jan 17, 2023 at 10:45 AM Yuri Weinstein wrote:
>
> OK I will rerun failed jobs filtering rhel in
>
> Thx!
>
> On Tue, Jan 17, 2023 at 10:43 AM Adam Kraitman wrote:
> >
> > Hey the satellite issue was fixed
> >
> > Thanks
> >
> > On Tue, Jan 17, 2023 at 7:43 PM Laura Flores wrote:
> >>
> >> This was my summary of rados failures. There was nothing new or amiss,
> >> although it is important to note that runs were done with filtering out
> >> rhel 8.
> >>
> >> I will leave it to Neha for final approval.
> >>
> >> Failures:
> >> 1. https://tracker.ceph.com/issues/58258
> >> 2. https://tracker.ceph.com/issues/58146
> >> 3. https://tracker.ceph.com/issues/58458
> >> 4. https://tracker.ceph.com/issues/57303
> >> 5. https://tracker.ceph.com/issues/54071
> >>
> >> Details:
> >> 1. rook: kubelet fails from connection refused - Ceph - Orchestrator
> >> 2. test_cephadm.sh: Error: Error initializing source docker://
> >> quay.ceph.io/ceph-ci/ceph:master - Ceph - Orchestrator
> >> 3. qa/workunits/post-file.sh: postf...@drop.ceph.com: Permission denied
> >> - Ceph
> >> 4. rados/cephadm: Failed to fetch package version from
> >> https://shaman.ceph.com/api/search/?status=ready=ceph=default=ubuntu%2F22.04%2Fx86_64=b34ca7d1c2becd6090874ccda56ef4cd8dc64bf7
> >> - Ceph - Orchestrator
> >> 5. rados/cephadm/osds: Invalid command: missing required parameter
> >> hostname() - Ceph - Orchestrator
> >>
> >> On Tue, Jan 17, 2023 at 9:48 AM Yuri Weinstein wrote:
> >>
> >> > Please see the test results on the rebased RC 6.6 in this comment:
> >> >
> >> > https://tracker.ceph.com/issues/58257#note-2
> >> >
> >> > We're still having infrastructure issues making testing difficult.
> >> > Therefore all reruns were done excluding the rhel 8 distro
> >> > ('--filter-out rhel_8')
> >> >
> >> > Also, the upgrades failed and Adam is looking into this.
> >> >
> >> > Seeking new approvals
> >> >
> >> > rados - Neha, Laura
> >> > rook - Sébastien Han
> >> > cephadm - Adam
> >> > dashboard - Ernesto
> >> > rgw - Casey
> >> > rbd - Ilya
> >> > krbd - Ilya
> >> > fs - Venky, Patrick
> >> > upgrade/nautilus-x (pacific) - Adam Kraitman
> >> > upgrade/octopus-x (pacific) - Adam Kraitman
> >> > upgrade/pacific-p2p - Neha - Adam Kraitman
> >> > powercycle - Brad
> >> >
> >> > Thx
> >> >
> >> > On Fri, Jan 6, 2023 at 8:37 AM Yuri Weinstein
> >> > wrote:
> >> > >
> >> > > Happy New Year all!
> >> > >
> >> > > This release remains to be in "progress"/"on hold" status as we are
> >> > > sorting all infrastructure-related issues.
> >> > >
> >> > > Unless I hear objections, I suggest doing a full rebase/retest QE
> >> > > cycle (adding PRs merged lately) since it's taking much longer than
> >> > > anticipated when sepia is back online.
> >> > >
> >> > > Objections?
> >> > >
> >> > > Thx
> >> > > YuriW
> >> > >
> >> > > On Thu, Dec 15, 2022 at 9:14 AM Yuri Weinstein
> >> > wrote:
> >> > > >
> >> > > > Details of this release are summarized here:
> >> > > >
> >> > > > https://tracker.ceph.com/issues/58257#note-1
> >> > > > Release Notes - TBD
> >> > > >
> >> > > > Seeking approvals for:
> >> > > >
> >> > > > rados - Neha (https://github.com/ceph/ceph/pull/49431 is still being
> >> > > > tested and will be merged soon)
> >> > > > rook - Sébastien Han
> >> > > > cephadm - Adam
> >> > > > dashboard - Ernesto
> >> > > > rgw - Casey (rwg will be rerun on the latest SHA1)
> >> > > > rbd - Ilya, Deepika
> >> > > > krbd - Ilya, Deepika
> >> > > > fs - Venky, Patrick
> >> > > > upgrade/nautilus-x (pacific) - Neha, Laura
> >> > > > upgrade/octopus-x (pacific) - Neha, Laura
> >> > > > upgrade/pacific-p2p - Neha - Neha, Laura
> >> > > > powercycle - Brad
> >> > > > ceph-volume - Guillaume, Adam K
> >> > > >
> >> > > > Thx
> >> > > > YuriW
> >> > ___
> >> > Dev mailing list -- d...@ceph.io
> >> > To unsubscribe send an email to dev-le...@ceph.io
> >> >
> >>
> >>
> >> --
> >>
> >> Laura Flores
> >>
> >> She/Her/Hers
> >>
> >> Software Engineer, Ceph Storage
> >>
> >> Red Hat Inc.
> >>
> >> Chicago, IL
> >>
> >> lflo...@redhat.com
> >> M: +17087388804
> >> @RedHat Red Hat
> >> Red Hat
> >>
> >>
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
>
[ceph-users] RBD to fail fast/auto unmap in case of timeout
Hello, We would like to run a RAID1 between a local storage and a RBD device. This would allow us to sustain network failures or Ceph failures and also give better read performance as we would set it up with write-mostly on RBD in mdadm. Basically we would like to implement https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme-low-latency. RAID1 is working well but if there is timeouts, the RBD volume won't fail and mdadm will not catch the broken device. Also the writes then hangs waiting for the network/RBD to come back. If we force unmap the RBD device then it fails as expected and writes can continue on other RAID1 device. We tried setting the `osd_request_timeout` to a small value (3 or 2 seconds) but it only gives us timeout in kernel logs: ``` libceph: tid 25792 on osd39 timeout rbd: rbd0: write at objno 602 0~512 result -110 rbd: rbd0: write result -110 print_req_error: 15 callbacks suppressed blk_update_request: timeout error, dev rbd0, sector 4931584 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 libceph: tid 25794 on osd39 timeout rbd: rbd0: write at objno 602 512~512 result -110 rbd: rbd0: write result -110 blk_update_request: timeout error, dev rbd0, sector 4931585 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 ``` Is there something that we missed or is it currently impossible with kRBD to kind of "fail fast" in case of timeout and unmap/remove associated RBD devices? Or is there another client that can do what we want (ceph-nbd or with librbd)? We found this rook issue that is not really helpful but give insight https://github.com/rook/rook/issues/376. Thanks! -- Mathias Chapelain Storage Engineer Proton AG ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Pools and classes
Dear all
I have a ceph cluster where so far all OSDs have been rotational hdd disks
(actually there are some SSDs, used only for block.db and wal.db)
I now want to add some SSD disks to be used as OSD. My use case is:
1) for the existing pools keep using only hdd disks
2) create some new pools using only sdd disks
Let's start with 1 (I didn't have added yet the ssd disks in the cluster)
I have some replicated pools and some ec pools. The replicated pools are
using a replicated_ruleset rule [*].
I created a new "replicated_hdd" rule [**] using the command:
ceph osd crush rule create-replicated replicated_hdd default host hdd
I then changed the crush rule of a existing pool (that was using
'replicated_ruleset') using the command:
ceph osd pool set crush_rule replicated_hdd
This triggered the remapping of some pgs and therefore some data movement.
Is this normal/expected, since for the time being I have only hdd osds ?
Thanks, Massimo
[*]
rule replicated_ruleset {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
[**]
rule replicated_hdd {
id 7
type replicated
min_size 1
max_size 10
step take default class hdd
step chooseleaf firstn 0 type host
step emit
}
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
