[ceph-users] Re: How to test PG mapping with reweight
I finally tracked down David Turner's offline reweight script [0] which in
theory does what I'm looking for, but he uses `crushtool -i
--reweight-item osd. ` which I confirmed changes the CRUSH
weight and not the reweight which is what I want.
When doing a `osdmp -i --dump=json`, I see weight in the osd
objects that corresponds with the reweight and not the CRUSH weight (CRUSH
weight is nowhere to be found in the dump). That seems to correlate with
[1]:
f->dump_float("weight", get_weightf(id));
where get_weightf() is [2]:
float get_weightf(int o) const {
return (float)get_weight(o) / (float)CEPH_OSD_IN;
}
and it looks like a property of the osdmap. That would mean that crushtool
would not be able to adjust reweight since it is not a CRUSH attribute,
right?
Looking at the code in osdmaptool, I don't see a way to set the reweight
for the osds.
Can someone point to the order of structs in the osdmap?
I could then read the binary osdmap in Python and make changes to the
binary map and then call osdmaptool to check the PG mappings. That sounds
better than trying to extend osdmaptool for Jewel.
[0]
http://lists.ceph.com/pipermail/ceph-large-ceph.com/attachments/20170110/11d6382f/attachment.py
[1]
https://github.com/ceph/ceph/blob/cdb8df13cb7c7242f95ebbf4e404ec0006504a9f/src/osd/OSDMap.cc#L3403
[2]
https://github.com/ceph/ceph/blob/f7376c0754ec700a0063bf14d0293cf31ad77798/src/osd/OSDMap.h#L768-L770
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Sat, Sep 7, 2019 at 11:10 AM Robert LeBlanc wrote:
> I've done some more digging into this. crushtool doesn't seem to apply
> reweights as the CRUSH map doesn't have the reweights in it. If I run `ceph
> osd getmap -o osdmap.o`, then I can run `osdmaptool --print osdmap.o` and I
> see all the osds with their reweights and if they are in the cluster or
> down. I can then run `osdmaptool --test-map-pg 5.1450 osdmap.o` and it
> gives me the OSDs that it would map to. This is exactly what I'm looking
> for. The only component missing is being able to modify the osdmap offline
> (since it is binary). There is an `--export-crush` and `--import-crush`,
> but that only pulls out the CRUSH map (which I can decompile fine), but
> again doesn't have the reweights or the up/down status.
>
> Any ideas how to modify the binary crushmap offline? It would be great to
> be able to just insert the final osdmap into the cluster rather than
> iterating through all 700+ osds reweighting them.
>
> Thanks,
> Robert LeBlanc
>
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>
>
> On Mon, Sep 2, 2019 at 10:29 AM Paul Emmerich
> wrote:
>
>> crushtool can simulate mappings for you, it's help/man page should
>> explain everything.
>>
>> Paul
>>
>> --
>> Paul Emmerich
>>
>> Looking for help with your Ceph cluster? Contact us at https://croit.io
>>
>> croit GmbH
>> Freseniusstr. 31h
>> 81247 München
>> www.croit.io
>> Tel: +49 89 1896585 90
>>
>> On Mon, Sep 2, 2019 at 7:08 PM Robert LeBlanc
>> wrote:
>> >
>> > I'd like to test how reweighting an OSD will change how the PGs map in
>> the cluster.
>> >
>> > I suspect that I'd dump the CRUSH map and PGs in the cluster that I'm
>> interested in then use osdmaptool. I'm not understanding how to use
>> osdmaptool to set the reweight, then query a PG or the entire set of PGs
>> that I'm interested in. I then suspect that if I'm okay with the new map
>> that I could inject it into the cluster instead of having to run reweight
>> on the OSD(s).
>> >
>> > This is a Jewel cluster and I'm trying to calculate OSD usage offline,
>> then inject a map that is more distributed instead of doing a reweight,
>> move the PGs which take a long time to just rinse and repeat over and over
>> again.
>> >
>> > Thanks,
>> > Robert LeBlanc
>> >
>> > Robert LeBlanc
>> > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>> > ___
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: How to test PG mapping with reweight
I've done some more digging into this. crushtool doesn't seem to apply reweights as the CRUSH map doesn't have the reweights in it. If I run `ceph osd getmap -o osdmap.o`, then I can run `osdmaptool --print osdmap.o` and I see all the osds with their reweights and if they are in the cluster or down. I can then run `osdmaptool --test-map-pg 5.1450 osdmap.o` and it gives me the OSDs that it would map to. This is exactly what I'm looking for. The only component missing is being able to modify the osdmap offline (since it is binary). There is an `--export-crush` and `--import-crush`, but that only pulls out the CRUSH map (which I can decompile fine), but again doesn't have the reweights or the up/down status. Any ideas how to modify the binary crushmap offline? It would be great to be able to just insert the final osdmap into the cluster rather than iterating through all 700+ osds reweighting them. Thanks, Robert LeBlanc Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Sep 2, 2019 at 10:29 AM Paul Emmerich wrote: > crushtool can simulate mappings for you, it's help/man page should > explain everything. > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Mon, Sep 2, 2019 at 7:08 PM Robert LeBlanc > wrote: > > > > I'd like to test how reweighting an OSD will change how the PGs map in > the cluster. > > > > I suspect that I'd dump the CRUSH map and PGs in the cluster that I'm > interested in then use osdmaptool. I'm not understanding how to use > osdmaptool to set the reweight, then query a PG or the entire set of PGs > that I'm interested in. I then suspect that if I'm okay with the new map > that I could inject it into the cluster instead of having to run reweight > on the OSD(s). > > > > This is a Jewel cluster and I'm trying to calculate OSD usage offline, > then inject a map that is more distributed instead of doing a reweight, > move the PGs which take a long time to just rinse and repeat over and over > again. > > > > Thanks, > > Robert LeBlanc > > > > Robert LeBlanc > > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] ceph-iscsi and tcmu-runner RPMs for CentOS?
Hi, In the Documentation on https://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli/ it is stated that you need at least CentOS 7.5 with at least kernel 4.16 and to install tcmu-runner and ceph-iscsi "from your Linux distribution's software repository". CentOS does not know about tcmu-runner nor ceph-iscsi. Where do I get these RPMs from? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] vfs_ceph and permissions
Hi ceph-users, I'm having problems with ACLs and vfs_ceph. I'm not sure this is the right list for those questions, please direct me elsewhere or just tell me to bugger off if I'm in the wrong place. I am running a Samba 4.7 domain controller connecting to a nautilus ceph cluster. I am using proxmox for the ceph cluster and Zentyal for managing the samba domain. I am not able to set the permissions for the share within Computer Management My smb.conf share settings block is: [data2] browseable = yes force create mode = 0660 force directory mode = 0660 valid users = @"Domain Users", @"Domain Admins", @"Domain Admins" read list = write list = @"Domain Users", @"Domain Admins" admin users = @"Domain Admins" vfs objects = acl_xattr full_audit full_audit:failure = connect opendir disconnect unlink mkdir rmdir open rename map acl inherit = yes store dos attributes = yes vfs objects = ceph path = /new/ kernel share modes = no ceph:config_file = /etc/ceph/ceph.conf ceph:user_id = admin (I know - I will shange the user from admin as soon as I have a working setup!) The settings are similar share (set up by the Zentyal UI) that allows me to set permissions correctly: [test] path = /home/samba/shares/test browseable = yes force create mode = 0660 force directory mode = 0660 valid users = @"Domain Admins", @"Domain Admins" read list = write list = @"Domain Admins" admin users = @"Domain Admins" vfs objects = acl_xattr full_audit recycle full_audit:failure = connect opendir disconnect unlink mkdir rmdir open rename When trying to, for instance, add full control permissions for a user in Computer Management I get "Failed to enumerate objects in the container. Access is denied." / "Unable to change permission changes [..] Access denied.". I am logged in as administrator when trying to change the permissions and this is (an extract) of the error in the samba.log [2019/09/07 15:57:14.656361, 2] ../source3/smbd/posix_acls.c:3008(set_canon_ace_list) set_canon_ace_list: sys_acl_set_file type file failed for file . (Operation not supported). [2019/09/07 15:57:14.656435, 3] ../source3/smbd/posix_acls.c:3093(convert_canon_ace_to_posix_perms) convert_canon_ace_to_posix_perms: Too many ACE entries for file . to convert to posix perms. [2019/09/07 15:57:14.656472, 3] ../source3/smbd/posix_acls.c:3986(set_nt_acl) set_nt_acl: failed to convert file acl to posix permissions for file .. [2019/09/07 15:57:14.656511, 3] ../source3/smbd/smb2_server.c:3139(smbd_smb2_request_error_ex) smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../source3/smbd/smb2_setinfo.c:132 [2019/09/07 15:57:18.042643, 3] ../source3/smbd/nttrans.c:2036(smbd_do_query_security_desc) smbd_do_query_security_desc: sd_size = 120. [2019/09/07 15:57:18.043643, 3] ../source3/smbd/nttrans.c:2036(smbd_do_query_security_desc) smbd_do_query_security_desc: sd_size = 76. [2019/09/07 15:57:18.045086, 2] ../source3/smbd/posix_acls.c:3008(set_canon_ace_list) set_canon_ace_list: sys_acl_set_file type file failed for file . (Operation not supported). [2019/09/07 15:57:18.045171, 3] ../source3/smbd/posix_acls.c:3093(convert_canon_ace_to_posix_perms) convert_canon_ace_to_posix_perms: Too many ACE entries for file . to convert to posix perms. [2019/09/07 15:57:18.045208, 3] ../source3/smbd/posix_acls.c:3986(set_nt_acl) set_nt_acl: failed to convert file acl to posix permissions for file .. [2019/09/07 15:57:18.045248, 3] ../source3/smbd/smb2_server.c:3139(smbd_smb2_request_error_ex) smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../source3/smbd/smb2_setinfo.c:132 [2019/09/07 15:57:19.016012, 3] ../source3/winbindd/winbindd_misc.c:395(winbindd_interface_version) [ 6733]: request interface version (version = 29) [2019/09/07 15:57:19.016349, 3] ../source3/winbindd/winbindd_misc.c:428(winbindd_priv_pipe_dir) Any pointers would be great! ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
