Ilya, Thank you for your suggestions!
`dmsg` (on the client node) only had `libceph: mon0 10.33.70.222:6789 socket error on write`. No further detail. But using the admin key (client.admin) for mounting CephFS solved my problem. I was able to write successfully! :-) $ sudo mount -t ceph 10.33.70.222:6789:/ /mnt/cephfs -o name=admin,secretfile=/etc/ceph/fsclient_secret // with the corresponding client.admin key $ sudo vim /mnt/cephfs/file4 $ sudo ls -l /mnt/cephfs total 1 -rw-r--r-- 1 root root 0 Jan 21 16:25 file1 -rw-r--r-- 1 root root 0 Jan 21 16:45 file2 -rw-r--r-- 1 root root 0 Jan 21 18:35 file3 -rw-r--r-- 1 root root 22 Jan 21 18:42 file4 Now, here is the difference between the two keys. client.testuser was obviously generated with the command `ceph fs authorize cephfs_test client.testuser / rw`, but something in there is obviously interfering with CephFS with a Cache Tier pool. Do I need to edit the `tag` or the `data` part? Now, I should mention the same type of key (like client.testuser) worked just fine when I was testing CephFS without a Cache Tier pool. client.admin key: XXXYYYYZZZ caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * client.testuser key: XXXYYYYZZZ caps: [mds] allow rw caps: [mon] allow r caps: [osd] allow rw tag cephfs data=cephfs_test *Mami Hayashida* *Research Computing Associate* Univ. of Kentucky ITS Research Computing Infrastructure On Tue, Jan 21, 2020 at 1:26 PM Ilya Dryomov <idryo...@gmail.com> wrote: > On Tue, Jan 21, 2020 at 6:02 PM Hayashida, Mami <mami.hayash...@uky.edu> > wrote: > > > > I am trying to set up a CephFS with a Cache Tier (for data) on a mini > test cluster, but a kernel-mount CephFS client is unable to write. Cache > tier setup alone seems to be working fine (I tested it with `rados put` and > `osd map` commands to verify on which OSDs the objects are placed) and > setting up CephFS without the cache-tiering also worked fine on the same > cluster with the same client, but combining the two fails. Here is what I > have tried: > > > > Ceph version: 14.2.6 > > > > Set up Cache Tier: > > $ ceph osd crush rule create-replicated highspeedpool default host ssd > > $ ceph osd crush rule create-replicated highcapacitypool default host hdd > > > > $ ceph osd pool create cephfs-data 256 256 highcapacitypool > > $ ceph osd pool create cephfs-metadata 128 128 highspeedpool > > $ ceph osd pool create cephfs-data-cache 256 256 highspeedpool > > > > $ ceph osd tier add cephfs-data cephfs-data-cache > > $ ceph osd tier cache-mode cephfs-data-cache writeback > > $ ceph osd tier set-overlay cephfs-data cephfs-data-cache > > > > $ ceph osd pool set cephfs-data-cache hit_set_type bloom > > > > ### > > All the cache tier configs set (hit_set_count, hit_set period, > target_max_bytes etc.) > > ### > > > > $ ceph-deploy mds create <mds node> > > $ ceph fs new cephfs_test cephfs-metadata cephfs-data > > > > $ ceph fs authorize cephfs_test client.testuser / rw > > $ ceph auth ls > > client.testuser > > key: XXXYYYYZZZZ > > caps: [mds] allow rw > > caps: [mon] allow r > > caps: [osd] allow rw tag cephfs data=cephfs_test > > > > ### Confirm the pool setting > > $ ceph osd pool ls detail > > pool 1 'cephfs-data' replicated size 3 min_size 2 crush_rule 2 > object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change > 63 lfor 53/53/53 flags hashpspool tiers 3 read_tier 3 write_tier 3 > stripe_width 0 application cephfs > > pool 2 'cephfs-metadata' replicated size 3 min_size 2 crush_rule 1 > object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change > 63 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 > recovery_priority 5 application cephfs > > pool 3 'cephfs-data-cache' replicated size 3 min_size 2 crush_rule 1 > object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change > 63 lfor 53/53/53 flags hashpspool,incomplete_clones tier_of 1 cache_mode > writeback target_bytes 80000000000 hit_set > bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 120s x2 > decay_rate 0 search_last_n 0 stripe_width 0 > > > > #### Set up the client side (kernel mount) > > $ sudo vim /etc/ceph/fsclient_secret > > $ sudo mkdir /mnt/cephfs > > $ sudo mount -t ceph <ceph MDS address>:6789:/ /mnt/cephfs -o > name=testuser,secretfile=/etc/ceph/fsclient_secret // no errors at this > point > > > > $ sudo vim /mnt/cephfs/file1 // Writing attempt fails > > > > "file1" E514: write error (file system full?) > > WARNING: Original file may be lost or damaged > > don't quit the editor until the file is successfully written! > > > > $ ls -l /mnt/cephfs > > total 0 > > -rw-r--r-- 1 root root 0 Jan 21 16:25 file1 > > > > Any help will be appreciated. > > Hi Mami, > > Is there anything in dmesg? > > What happens if you mount without involving testuser (i.e. using > client.admin and the admin key)? > > Thanks, > > Ilya >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com