>> rm: cannot remove '/design/4695/8/6-50kb.jpg': No space left on device “No space left on device” issue typically in ceph FS might be caused if you have files > 1million(100000) in “single directory”. To mitigate this try increasing the "mds_bal_fragment_size_max" to a higher value, example 7 million (700000)
[mds] mds_bal_fragment_size_max = 700000 I am not going detail here, that said, there are many other tuning params including enabling dir frag, multiple MDS etc. It seems your CEPH version is 10.2.10(Jewel). Luminous(12.2.x) support better with multiple MDS and dir_frag etc. Jewel some of these options might be experimental features you might need todo bit reading related to cephFS. Just a note of advise based on my experience, cephFS is ideal for large files in MB/GB is not great for small files(in kbs). Also split your files into multiple sub-dirs to avoid the similar issue or any likewise. -- Deepak From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of ?????? Sent: Friday, February 09, 2018 2:35 AM To: ceph-users@lists.ceph.com Subject: [ceph-users] rm: cannot remove dir and files (cephfs) ceph version 10.2.10 ceph -s cluster 97f833aa-cc6f-41d5-bf82-bda5c09fd664 health HEALTH_OK monmap e3: 3 mons at {web23=192.168.65.24:6789/0,web25=192.168.65.55:6789/0,web26=192.168.65.56:6789/0} election epoch 1198, quorum 0,1,2 web23,web25,web26 fsmap e464: 1/1/1 up {0=web26=up:active}, 3 up:standby osdmap e325: 4 osds: 4 up, 4 in flags sortbitwise,require_jewel_osds pgmap v42475: 128 pgs, 3 pools, 274 GB data, 1710 kobjects 854 GB used, 2939 GB / 3793 GB avail 128 active+clean client io 181 kB/s wr, 0 op/s rd, 5 op/s wr ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 3793G 2941G 851G 22.46 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 822G 0 cephfs_metadata 1 17968k 0 822G 101458 cephfs_data 2 273G 24.99 822G 1643059 kernel mount cephfs grep design /etc/fstab 192.168.65.24:,192.168.65.55:,192.168.65.56:/ /design ceph rw,relatime,name=design,secret=.......,_netdev,noatime 0 0 I can not delete same files and dir. rm -rf /design/4* rm: cannot remove '/design/4695/8/6-50kb.jpg': No space left on device rm: cannot remove '/design/4695/8/9-300kb.png': No space left on device rm: cannot remove '/design/4695/8/0-300kb.png': No space left on device What ideas? help ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
