[ceph-users] CfP Software Defined Storage devroom
FOSDEM is a free software event that offers open source communities a place to
meet, share ideas and collaborate. It is well known for being highly
developer-oriented and in the past brought together 8000+ participants from all
over the world. It's home is in the city of Brussels (Belgium).
FOSDEM 2021 will take place as an online event during the weekend of February
6./7. 2021. More details about the event can be found at http://fosdem.org/
** Call For Participation
The Software Defined Storage devroom will go into it's fifth round for talks
around Open Source Software Defined Storage projects, management tools
and real world deployments.
Presentation topics could include but are not limited too:
- Your work on a SDS project like Ceph, Gluster, OpenEBS, CORTX or Longhorn
- Your work on or with SDS related projects like OpenStack SWIFT or Container
Storage Interface
- Management tools for SDS deployments
- Monitoring tools for SDS clusters
** Important dates:
- Dec 27th 2020: submission deadline for talk proposals
- Dec 31st 2020: announcement of the final schedule
- Feb 6th 2021: Software Defined Storage dev room
Talk proposals will be reviewed by a steering committee:
- Niels de Vos (OpenShift Container Storage Developer - Red Hat)
- Jan Fajerski (Ceph Developer - SUSE)
- TBD
Use the FOSDEM 'pentabarf' tool to submit your proposal:
https://penta.fosdem.org/submission/FOSDEM21
- If necessary, create a Pentabarf account and activate it.
Please reuse your account from previous years if you have
already created it.
https://penta.fosdem.org/user/new_account/FOSDEM21
- In the "Person" section, provide First name, Last name
(in the "General" tab), Email (in the "Contact" tab)
and Bio ("Abstract" field in the "Description" tab).
- Submit a proposal by clicking on "Create event".
- If you plan to register your proposal in several tracks to increase your chances,
don't! Register your talk once, in the most accurate track.
- Presentations have to be pre-recorded before the event and will be streamed on
the event weekend.
- Important! Select the "Software Defined Storage devroom" track
(on the "General" tab).
- Provide the title of your talk ("Event title" in the "General" tab).
- Provide a description of the subject of the talk and the
intended audience (in the "Abstract" field of the "Description" tab)
- Provide a rough outline of the talk or goals of the session (a short
list of bullet points covering topics that will be discussed) in the
"Full description" field in the "Description" tab
- Provide an expected length of your talk in the "Duration" field.
We suggest a length between 15 and 45 minutes.
** For accepted talks
Once your proposal is accepted we will assign you a volunteer deputy who will
help you to produce the talk recording. The volunteer will also try to ensure
the recording is of good quality, help with uploading it to the system,
broadcasting it during the event and moderate the Q session after the
broadcast. Please note that as a presenter you're expected to be available
online during and especially after the broadcast of you talk. The schedule will
be available under
https://fosdem.org/2021/schedule/track/software_defined_storage/
Hope to hear from you soon! And please forward this announcement.
If you have any further questions, please write to the mailing list at
storage-devr...@lists.fosdem.org and we will try to answer as soon as
possible.
Thanks!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: How to use ceph-volume to create multiple OSDs per NVMe disk, and with fixed WAL/DB partition on another device?
On Fri, Nov 06, 2020 at 10:15:52AM -, victorh...@yahoo.com wrote: I'm building a new 4-node Proxmox/Ceph cluster, to hold disk images for our VMs. (Ceph version is 15.2.5). Each node has 6 x NVMe SSDs (4TB), and 1 x Optane drive (960GB). CPU is AMD Rome 7442, so there should be plenty of CPU capacity to spare. My aim is to create 4 x OSDs per NVMe SSD (to make more effective use of the NVMe performance) and use the Optane drive to store the WAL/DB partition for each OSD. (I.e. total of 24 x 35GB WAL/DB partitions). However, I am struggling to get the right ceph-volume command to achieve this. Thanks to a very kind Redditor, I was able to get close: /dev/nvme0n1 is an Optane device (900GB). /dev/nvme2n1 is an Intel NVMe SSD (4TB). ``` # ceph-volume lvm batch --osds-per-device 4 /dev/nvme2n1 --db-devices /dev/nvme0n1 Total OSDs: 4 Solid State VG: Targets: block.db Total size: 893.00 GB Total LVs: 16Size per LV: 223.25 GB Devices: /dev/nvme0n1 TypePathLV Size % of device [data] /dev/nvme2n1931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% [data] /dev/nvme2n1931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% [data] /dev/nvme2n1931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% [data] /dev/nvme2n1931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) ``` This does split up the NVMe disk into 4 OSDs, and creates WAL/DB partition on the Optane drive - however, it creates 4 x 223 GB partitions on the Optane (whereas I want 35GB partitions). Is there any way to specify the WAL/DB partition size in the above? And can it be done, such that you can run successive ceph-volume commands, to add the OSDs and WAL/DB partitions for each NVMe disk? Is there is particular reason you want to run ceph-volume multiple times? The batch subcommand can handle that in one go without the need to explicitly specify any sizes as another reply proposed (though that will work nicely). Something like this should get you there: ceph-volume lvm batch --osds-per-device 4 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --db-devices /dev/nvme0n1 This of course makes assumption regarding device names, adjust accordingly. Another option to size the volumes on the Optane drive would be to rely on the *slots arguments of the batch subcommand. See either ceph-volume lvm batch --help or https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/#implicit-sizing (Or if there's an easier way to achieve the above layout, please let me know). That being said - I also just saw this ceph-users thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/3Y6DEJCF7ZMXJL2NRLXUUEX76W7PPYXK/ It talks there about "osd op num shards" and "osd op num threads per shard" - is there some way to set those, to achieve similar performance to say, 4 x OSDs per NVMe drive, but with only 1 x NVMe? Has anybody done any testing/benchmarking on this they can share? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph Octopus and Snapshot Schedules
It sure is: https://docs.ceph.com/en/latest/releases/octopus/#rbd-block-storage On Tue, Oct 27, 2020 at 10:29:18AM -0400, Adam Boyhan wrote: That is exactly what I am thinking. My mistake, I should have specified RBD. Is snapshots scheduling/retention for RBD already in Octopus as well? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph Octopus and Snapshot Schedules
Maybe you are thinking of the rbd variety of this? https://docs.ceph.com/en/latest/rbd/rbd-mirroring/ On Tue, Oct 27, 2020 at 10:19:25AM -0400, Adam Boyhan wrote: I thought Octopus brought the new snapshot replication feature to the table? Was there issues with it? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph Octopus and Snapshot Schedules
snapshot scheduling for CephFS landed in master already, The octopus backport is pending still but on the agenda (https://github.com/ceph/ceph/pull/37142) Replication of those snapshots is being worked on and is planned for pacific, but rsync can help you out until then surely. On Thu, Oct 22, 2020 at 09:37:50AM -0400, Adam Boyhan wrote: Hey all. I was wondering if Ceph Octopus is capable of automating/managing snapshot creation/retention and then replication? Ive seen some notes about it, but can't seem to find anything solid. Open to suggestions as well. Appreciate any input! ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [doc] drivegroups advanced case
On Fri, Aug 21, 2020 at 06:52:19PM +, Tony Liu wrote: Hi, Regarding to the YAML in this section, https://ceph.readthedocs.io/en/latest/cephadm/drivegroups/#the-advanced-case Is the "rotational" supposed to be "1" meaning spinning HDD? Correct. This corresponds to /sys/block//queue/rotational. Thanks! Tony ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io -- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: can't get healthy cluster to trim osdmaps (13.2.8)
https://tracker.ceph.com/issues/44184
Looks similar, maybe you're also seeing other symptoms listed there? In any case
would be good to track this in one place.
On Mon, Mar 23, 2020 at 11:29:53AM +0100, Nikola Ciprich wrote:
OK, so after some debugging, I've pinned the problem down to
OSDMonitor::get_trim_to:
std::lock_guard l(creating_pgs_lock);
if (!creating_pgs.pgs.empty()) {
return 0;
}
apparently creating_pgs.pgs.empty() is not true, do I understand it
correctly that cluster thinks the list of creating pgs is not empty?
all pgs are in clean+active state, so maybe there's something malformed
in the db? How can I check?
I tried dumping list of creating_pgs according to
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030297.html
but to no avail
On Tue, Mar 17, 2020 at 12:25:29PM +0100, Nikola Ciprich wrote:
Hello dear cephers,
lately, there's been some discussion about slow requests hanging
in "wait for new map" status. At least in my case, it's being caused
by osdmaps not being properly trimmed. I tried all possible steps
to force osdmap pruning (restarting mons, restarting everyging,
poking crushmap), to no avail. Still all OSDs keep min osdmap version
1, while newest is 4734. Otherwise cluster is healthy, with no down
OSDs, network communication works flawlessly, all seems to be fine.
Just can't get old osdmaps to go away.. I's very small cluster and I've
moved all production traffic elsewhere, so I'm free to investigate
and debug, however I'm out of ideas on what to try or where to look.
Any ideas somebody please?
The cluster is running 13.2.8
I'd be very grateful for any tips
with best regards
nikola ciprich
--
-
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava
tel.: +420 591 166 214
fax:+420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: ser...@linuxbox.cz
-
--
-
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava
tel.: +420 591 166 214
fax:+420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: ser...@linuxbox.cz
-
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
--
Jan Fajerski
Senior Software Engineer Enterprise Storage
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph prometheus module no export content
On Thu, Feb 27, 2020 at 10:20:07AM +0800, 黄明友 wrote: hi,all I had enable the prometheus module on my ceph cluster , the ceph version is ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nauti lus (stable). when I eanble this modle, can get exported content from the prometheus module. but , when I restart all the ceph cluster's node , the module can't expor any content , curl http://host/metrics ; get blan; the tcp/9283 port is listen; I had disable and enable this module,but still no content export. You're probably querying a standby mgr daemon. Check ceph -s for the current active and query that one. For a prometheus server you always want to scrape all mgr daemons. Also see https://docs.ceph.com/docs/master/mgr/prometheus/ 黄明友 IT基础架构部经理 V.Photos 云摄影 移动电话: +86 13540630430 客服电话：400 - 806 - 5775 电子邮件: hmy@v.photos 官方网址: www.v.photos 上海 黄浦区中山东二路88号外滩SOHO3Q F栋 2层 北京 朝阳区光华路9号光华路SOHO二期南二门SOHO3Q 1层 广州 天河区林和中路136号天誉花园二期3Wcoffice 天誉青创社区 深圳 南山区蛇口网谷科技大厦二期A座102网谷双创街 1层 成都 成华区建设路世贸广场 7层 [cid:__aliyun158277000719672821] ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io -- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.6 ceph-volume bluestore _read_fsid unparsable uuid
On Tue, Jan 28, 2020 at 08:03:35PM +0100, bauen1 wrote: >Hi, > >I've run into the same issue while testing: > >ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) >nautilus (stable) > >debian bullseye > >Ceph was installed using ceph-ansible on a vm from the repo >http://download.ceph.com/debian-nautilus > >The output of `sudo sh -c 'CEPH_VOLUME_DEBUG=true ceph-volume >--cluster test lvm batch --bluestore /dev/vdb'` has been attached. Thx, I opened https://tracker.ceph.com/issues/43868. This looks like a bluestore/osd issue to me, though it might end up being ceph-volumes fault. > >Also worth noting might be that '/var/lib/ceph/osd/test-0/fsid' is >empty (but I don't know too much about the internals) > >- bauen1 > >On 1/28/20 4:54 PM, Dave Hall wrote: >>Jan, >> >>Unfortunately I'm under immense pressure right now to get some form >>of Ceph into production, so it's going to be Luminous for now, or >>maybe a live upgrade to Nautilus without recreating the OSDs (if >>that's possible). >> >>The good news is that in the next couple months I expect to add more >>hardware that should be nearly identical. I will gladly give it a >>go at that time and see if I can recreate. (Or, if I manage to >>thoroughly crash my current fledgling cluster, I'll give it another >>go on one node while I'm up all night recovering.) >> >>If you could tell me where to look I'd gladly read some code and see >>if I can find anything that way. Or if there's any sort of design >>document describing the deep internals I'd be glad to scan it to see >>if I've hit a corner case of some sort. Actually, I'd be interested >>in reading those documents anyway if I could. >> >>Thanks. >> >>-Dave >> >>Dave Hall >> >>On 1/28/2020 3:05 AM, Jan Fajerski wrote: >>>On Mon, Jan 27, 2020 at 03:23:55PM -0500, Dave Hall wrote: >>>>All, >>>> >>>>I've just spent a significant amount of time unsuccessfully chasing >>>>the _read_fsid unparsable uuid error on Debian 10 / Natilus 14.2.6. >>>>Since this is a brand new cluster, last night I gave up and moved back >>>>to Debian 9 / Luminous 12.2.11. In both cases I'm using the packages >>>>from Debian Backports with ceph-ansible as my deployment tool. >>>>Note that above I said 'the _read_fsid unparsable uuid' error. I've >>>>searched around a bit and found some previously reported issues, but I >>>>did not see any conclusive resolutions. >>>> >>>>I would like to get to Nautilus as quickly as possible, so I'd gladly >>>>provide additional information to help track down the cause of this >>>>symptom. I can confirm that, looking at the ceph-volume.log on the >>>>OSD host I see no difference between the ceph-volume lvm batch command >>>>generated by the ceph-ansible versions associated with these two Ceph >>>>releases: >>>> >>>> ceph-volume --cluster ceph lvm batch --bluestore --yes >>>> --block-db-size 133358734540 /dev/sdc /dev/sdd /dev/sde /dev/sdf >>>> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/nvme0n1 >>>> >>>>Note that I'm using --block-db-size to divide my NVMe into 12 segments >>>>as I have 4 empty drive bays on my OSD servers that I may eventually >>>>be able to fill. >>>> >>>>My OSD hardware is: >>>> >>>> Disk /dev/nvme0n1: 1.5 TiB, 1600321314816 bytes, 3125627568 sectors >>>> Disk /dev/sdc: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors >>>> Disk /dev/sdd: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors >>>> Disk /dev/sde: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors >>>> Disk /dev/sdf: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors >>>> Disk /dev/sdg: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors >>>> Disk /dev/sdh: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors >>>> Disk /dev/sdi: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors >>>> Disk /dev/sdj: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors >>>> >>>>I'd send the output of ceph-volume inventory on Luminous, but I'm >>>>getting -->: KeyError: 'human_readable_size'. >>>> >>>>Please let me know if I can provide any further information. >>>Mind re-running you ceph-volume command with debug output >>>enabled: >>>CEPH_VOLUME_DEBUG=true ceph-volume --cluster ceph lvm ba
[ceph-users] Re: Nautilus 14.2.6 ceph-volume bluestore _read_fsid unparsable uuid
On Mon, Jan 27, 2020 at 03:23:55PM -0500, Dave Hall wrote: >All, > >I've just spent a significant amount of time unsuccessfully chasing >the _read_fsid unparsable uuid error on Debian 10 / Natilus 14.2.6. >Since this is a brand new cluster, last night I gave up and moved back >to Debian 9 / Luminous 12.2.11. In both cases I'm using the packages >from Debian Backports with ceph-ansible as my deployment tool. > >Note that above I said 'the _read_fsid unparsable uuid' error. I've >searched around a bit and found some previously reported issues, but I >did not see any conclusive resolutions. > >I would like to get to Nautilus as quickly as possible, so I'd gladly >provide additional information to help track down the cause of this >symptom. I can confirm that, looking at the ceph-volume.log on the >OSD host I see no difference between the ceph-volume lvm batch command >generated by the ceph-ansible versions associated with these two Ceph >releases: > > ceph-volume --cluster ceph lvm batch --bluestore --yes > --block-db-size 133358734540 /dev/sdc /dev/sdd /dev/sde /dev/sdf > /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/nvme0n1 > >Note that I'm using --block-db-size to divide my NVMe into 12 segments >as I have 4 empty drive bays on my OSD servers that I may eventually >be able to fill. > >My OSD hardware is: > > Disk /dev/nvme0n1: 1.5 TiB, 1600321314816 bytes, 3125627568 sectors > Disk /dev/sdc: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors > Disk /dev/sdd: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors > Disk /dev/sde: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors > Disk /dev/sdf: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors > Disk /dev/sdg: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors > Disk /dev/sdh: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors > Disk /dev/sdi: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors > Disk /dev/sdj: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors > >I'd send the output of ceph-volume inventory on Luminous, but I'm >getting -->: KeyError: 'human_readable_size'. > >Please let me know if I can provide any further information. Mind re-running you ceph-volume command with debug output enabled: CEPH_VOLUME_DEBUG=true ceph-volume --cluster ceph lvm batch --bluestore ... Ideally you could also openen a bug report here https://tracker.ceph.com/projects/ceph-volume/issues/new Thanks! > >Thanks. > >-Dave > >-- >Dave Hall >Binghamton University > >___ >ceph-users mailing list -- ceph-users@ceph.io >To unsubscribe send an email to ceph-users-le...@ceph.io -- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph-volume lvm batch: strategy changed after filtering
On Fri, Jan 24, 2020 at 11:48:10AM +, Stolte, Felix wrote: >Hey guys, > >I’m struggling with the ceph-volume command in nautilus 14.2.6. I have 12 >disks on each server, 3 of them ssds (sda,sdb,sdc) and 9 spinning disks (sdd >.. sdl). Initial deploy witch ceph-volume batch works fine, one ssd is used >for wal and db for 3 spinning disks. But running the ‘ceph-volume lvm batch >--report /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg >/dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl’ after the initial deploy I get >'Aborting because strategy changed from bluestore.MixedType to >bluestore.SingleType after filtering' > >Looking at /var/log/ceph/ceph-volume.log I see: >[ceph_volume.devices.lvm.batch][INFO ] Ignoring devices already used by ceph: >/dev/sdd, /dev/sde, /dev/sdf, /dev/sdg, /dev/sdh, /dev/sdi, /dev/sdj, >/dev/sdk, /dev/sdl > >I do understand that before filtering I have MixedType (ssd + sata) and after >filtering (only ssds left) it is single type. So the error message is correct. > >But why does ceph-volume filter out the spinning disks with data partitions >but not the ssds with wal/db? Is this intended? Intended as in coded that way yes. See https://github.com/ceph/ceph/blob/master/src/ceph-volume/ceph_volume/util/device.py#L381 I'm not exactly sure why journal/wal/db was explicitly excluded here. I'm in the process of re-writing how batch creates OSDs to address inconsistencies like this. Is there maybe space on your SSD drives? In this case its certainly valid to not filter those, as one might want to co-locate additional volumes. What exactly are you trying to do? Nest, Jan > >Regards Felix > > >IT-Services >Telefon 02461 61-9243 >E-Mail: f.sto...@fz-juelich.de >- >- >Forschungszentrum Juelich GmbH >52425 Juelich >Sitz der Gesellschaft: Juelich >Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher >Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), >Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, >Prof. Dr. Sebastian M. Schmidt >- >- > > >_______ >ceph-users mailing list -- ceph-users@ceph.io >To unsubscribe send an email to ceph-users-le...@ceph.io -- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph-volume lvm filestore OSDs fail to start on reboot. Permission denied on journal partition
On Wed, Jan 22, 2020 at 12:00:28PM -0500, Wesley Dillingham wrote: > After upgrading to Nautilus 14.2.6 from Luminous 12.2.12 we are seeing > the following behavior on OSDs which were created with "ceph-volume lvm > create --filestore --osd-id --data --journal " > Upon restart of the server containing these OSDs they fail to start > with the following error in the logs: >2020-01-21 13:36:11.635 7fee633e8a80 -1 filestore(/var/lib/ceph/osd/ceph-199) >mo >unt(1928): failed to open journal /var/lib/ceph/osd/ceph-199/journal: (13) >Permi >ssion denied > > /var/lib/ceph/osd/ceph-199/journal symlinks to /dev/sdc5 in our case > and inspecting the ownership on /dev/sdc5 it is root:root, chowning > that to ceph:ceph causes the osd to start and come back up and in near > instantly. > As a note these OSDs we experience this with are OSDs which have > previously failed and been replaced using the above ceph-volume, longer > running OSDs in the same server created with ceph-disk or ceph-volume > simple (that have a corresponding .json in /etc/ceph/osd) start up fine > and get ceph:ceph on their journal partition. Bluestore OSDs also do > not have any issue. > My hope is that I can preemptively fix these OSDs before shutting them > down so that reboots happen seamlessly. Thanks for any insight. ceph-volume is supposed to take care of this via the ceph-volume@ systemd unit. This is a one shot unit, that should set things up and then start the osd. The unit name is a bit convoluted: ceph-volume@-, there should be symbolic link in /etc/systemd/system/multi-user.target.wants/ You can also check cat /var/log/ceph/ceph-volume-systemd.log for any errors. Feel free to open a tracker ticket on https://tracker.ceph.com/projects/ceph-volume > > Respectfully, > Wes Dillingham > [1]w...@wesdillingham.com > [2]LinkedIn > >References > > 1. mailto:w...@wesdillingham.com > 2. http://www.linkedin.com/in/wesleydillingham >___ >ceph-users mailing list -- ceph-users@ceph.io >To unsubscribe send an email to ceph-users-le...@ceph.io -- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph-volume sizing osds
On Fri, Dec 13, 2019 at 11:49:50PM +0100, Oscar Segarra wrote: > Hi, > I have recently started working with Ceph Nautilus release and I have > realized that you have to start working with LVM to create OSD instead > of the "old fashioned" ceph-disk. > In terms of performance and best practices, as far as I must use LVM I > can create volume groups that joins or extends two or more physical > disks. In this scenario (many disks for server) where ceph-volume is > manatory, It still remains the rule of one OSD for each physical > device? or I can reduce the number of OSDs? That is hard to answer generically. ceph-volume will not create multi-disk volumes. One disk, one OSD is still the recommendation afaik. However there are certainly scenarios where multi-disk OSDs are sensible. For example if you have a machine that has _a lot_ of disks attached but lacks CPU and RAM to run as many OSDs as disks, it can make sense to deploy one OSD on multiple disks. > Thanks in advance, > Óscar >___ >ceph-users mailing list >ceph-us...@lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Prometheus endpoint hanging with 13.2.7 release?
On Mon, Dec 09, 2019 at 05:01:04PM -0800, Paul Choi wrote:
> Hello,
> Anybody seeing the Prometheus endpoint hanging with the new 13.2.7
> release?
> With 13.2.6 the endpoint would respond with a payload of 15MB in less
> than 10 seconds.
I'd guess its not the prometheus module itself:
$ git diff v13.2.6 v13.2.7 src/pybind/mgr/prometheus
diff --git a/src/pybind/mgr/prometheus/module.py
b/src/pybind/mgr/prometheus/module.py
index 2d4472434a..3a398e0b0c 100644
--- a/src/pybind/mgr/prometheus/module.py
+++ b/src/pybind/mgr/prometheus/module.py
@@ -142,7 +142,8 @@ class Metric(object):
def promethize(path):
''' replace illegal metric name characters '''
-result = path.replace('.', '_').replace('+',
'_plus').replace('::', '_')
+result = path.replace('.', '_').replace(
+'+', '_plus').replace('::', '_').replace(' ', '_')
# Hyphens usually turn into underscores, unless they are
# trailing
@@ -720,7 +721,8 @@ class Module(MgrModule):
raise cherrypy.HTTPError(503, 'No MON connection')
# Make the cache timeout for collecting configurable
-self.collect_timeout = self.get_localized_config('scrape_interval',
5.0)
+self.collect_timeout = float(self.get_localized_config(
+'scrape_interval', 5.0))
server_addr = self.get_localized_config('server_addr', DEFAULT_ADDR)
server_port = self.get_localized_config('server_port', DEFAULT_PORT)
So the mgr would be a likely suspect. If you could open a tracker ticket,
ideally with mgr debug logs attached, this can be looked at.
> Now, if you restart ceph-mgr, the Prometheus endpoint responds quickly
> for the first run, then successive runs get slower and slower, until it
> takes several minutes.
> I have no customization for the mgr module. Except for the Prometheus
> module, the "status" module and "Zabbix" module are working fine.
> This is on Ubuntu 16 LTS:
> ii ceph-mgr 13.2.7-1xenial
> amd64manager for the ceph distributed storage
> system
> I'd love to know if there's a way to diagnose this issue - I tried
> upping the debug ms level but that doesn't seem to yield useful
> information.
> I don't know if this useful, but "prometheus self-test" is fine too.
> $ sudo ceph tell mgr.0 prometheus self-test
> Self-test OK
> pchoi@pchoi-desktop:~$ ceph mgr module ls
> {
> "enabled_modules": [
> "prometheus",
> "status",
> "zabbix"
> ],
> "disabled_modules": [
> {
> "name": "balancer",
> "can_run": true,
> "error_string": ""
> },
> {
> "name": "crash",
> "can_run": true,
> "error_string": ""
> },
> {
> "name": "dashboard",
> "can_run": true,
> "error_string": ""
> },
> {
> "name": "hello",
> "can_run": true,
> "error_string": ""
> },
> {
> "name": "influx",
> "can_run": false,
> "error_string": "influxdb python module not found"
> },
> {
> "name": "iostat",
> "can_run": true,
> "error_string": ""
> },
> {
> "name": "localpool",
> "can_run": true,
> "error_string": ""
> },
> {
> "name": "restful",
> "can_run": true,
> "error_string": ""
> },
> {
> "name": "selftest",
> "can_run": true,
> "error_string": ""
> },
> {
> "name": "smart",
> "can_run": true,
> "error_string": ""
> },
> {
> "name": "telegraf",
> "can_run": true,
> "error_string": ""
> },
> {
> "name": "telemetry",
> "can_run": true,
> "error_string": ""
> }
> ]
> }
> pchoi@pchoi-desktop:~$ ceph mgr services
> {
> "prometheus": "[1]http://woodenbox2:9283/;
> }
>
>References
>
> 1. http://woodenbox2:9283/
>___
>ceph-users mailing list -- ceph-users@ceph.io
>To unsubscribe send an email to ceph-users-le...@ceph.io
--
Jan Fajerski
Senior Software Engineer Enterprise Storage
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph-volume lvm create leaves half-built OSDs lying around
On Wed, Sep 11, 2019 at 11:17:47AM +0100, Matthew Vernon wrote: >Hi, > >We keep finding part-made OSDs (they appear not attached to any host, >and down and out; but still counting towards the number of OSDs); we >never saw this with ceph-disk. On investigation, this is because >ceph-volume lvm create makes the OSD (ID and auth at least) too early in >the process and is then unable to roll-back cleanly (because the >bootstrap-osd credential isn't allowed to remove OSDs). > >As an example (very truncated): > >Running command: /usr/bin/ceph --cluster ceph --name >client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring >-i - osd new 20cea174-4c1b-4330-ad33-505a03156c33 >Running command: vgcreate --force --yes >ceph-9d66ec60-c71b-49e0-8c1a-e74e98eafb0e /dev/sdbh > stderr: Device /dev/sdbh not found (or ignored by filtering). > Unable to add physical volume '/dev/sdbh' to volume group >'ceph-9d66ec60-c71b-49e0-8c1a-e74e98eafb0e'. >--> Was unable to complete a new OSD, will rollback changes >--> OSD will be fully purged from the cluster, because the ID was generated >Running command: ceph osd purge osd.828 --yes-i-really-mean-it > stderr: 2019-09-10 15:07:53.396528 7fbca2caf700 -1 auth: unable to find >a keyring on >/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >(2) No such file or directory > stderr: 2019-09-10 15:07:53.397318 7fbca2caf700 -1 monclient: >authenticate NOTE: no keyring found; disabled cephx authentication >2019-09-10 15:07:53.397334 7fbca2caf700 0 librados: client.admin >authentication error (95) Operation not supported > >This is annoying to have to clear up, and it seems to me could be >avoided by either: > >i) ceph-volume should (attempt to) set up the LVM volumes before >making the new OSD id >or >ii) allow the bootstrap-osd credential to purge OSDs > >i) seems like clearly the better answer...? Agreed. Would you mind opening a bug report on https://tracker.ceph.com/projects/ceph-volume. I have found other situation where a roll-back is working as it should, though not with as much impact as this. > >Regards, > >Matthew > >_______ >ceph-users mailing list >ceph-us...@lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH (HRB 247165, AG München) Geschäftsführer: Felix Imendörffer ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
