Re: [ceph-users] CephFS emptying files or silently failing to mount?
Holy cow. Thank you for pointing out what should have been obvious. So glad these emails are kept on the web for future searchers like me ;) -bo On Tue, Jun 11, 2013 at 11:46 AM, Gregory Farnum wrote: > On Tue, Jun 11, 2013 at 9:39 AM, Bo wrote: > > howdy, y'all. > > > > we are testing ceph and all of its features. we love RBD! however cephFS, > > though clearly stated not production ready, has been stonewalling us. in > an > > attempt to get rolling quickly, we followed some guides on CephFS > > (http://goo.gl/BmVxG, http://goo.gl/1VtNk). > > > > when i mount CephFS, i can: > > 1) create a file with contents on clientA; > > 2) list the folder and see the file on clientB and clientC; > > 3) pull up the exact same xattr's on that file across all clients. > > > > however, as soon as i try to modify or view the contents of the file, > ubuntu > > 12.04 says "operation not permitted". only clientA, which made the file, > can > > try to view it's contents, but when clientA does so the file appears > empty. > > a remount of the share on any client makes no difference. for what it is > > worth, xattrs don't change upon remount. > > > > what in the world are we doing wrong? > > > > cephx is enabled. works great for RBD. our ceph auth command for cephFS > > user: > > ceph auth get-or-create client.cephfs mds 'allow *' mon 'allow r' osd > 'allow > > rwx pool=cephfs' > > > > we are allowing ceph to determine what pool to use, which i think > defaults > > to 0/"data". > > Yep, that's why. Your cephx key only lets the client touch the > "cephfs" pool, but the metadata server is telling the client to put > its file data in the "data" pool. The client buffers up writes, sends > all the metadata to the server (where it is successfully stored), but > when it tries to flush anything out it fails to do so. You'll need to > resolve that discrepancy, either by reconfiguring CephFS to store all > the file data in the "cephfs" pool (look up the newfs command in the > docs, or just set a new layout on the root dir), or adding the "data" > pool to the list that the clients can access. > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > -- "But God demonstrates His own love toward us, in that while we were yet sinners, Christ died for us. Much more then, having now been justified by His blood, we shall be saved from the wrath of God through Him." Romans 5:8-9 *All *have sinned, broken God's law, and deserve eternal torment. Jesus Christ, the Son of God, died for the sins of those that will believe, purchasing our salvation, and defeated death so that we all may spend eternity in heaven. Do you desire freedom from hell and be with God in His love for eternity? "If you confess with your mouth Jesus as Lord, and believe in your heart that God raised Him from the dead, you will be saved." Romans 10:9 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CephFS emptying files or silently failing to mount?
On Tue, Jun 11, 2013 at 9:39 AM, Bo wrote: > howdy, y'all. > > we are testing ceph and all of its features. we love RBD! however cephFS, > though clearly stated not production ready, has been stonewalling us. in an > attempt to get rolling quickly, we followed some guides on CephFS > (http://goo.gl/BmVxG, http://goo.gl/1VtNk). > > when i mount CephFS, i can: > 1) create a file with contents on clientA; > 2) list the folder and see the file on clientB and clientC; > 3) pull up the exact same xattr's on that file across all clients. > > however, as soon as i try to modify or view the contents of the file, ubuntu > 12.04 says "operation not permitted". only clientA, which made the file, can > try to view it's contents, but when clientA does so the file appears empty. > a remount of the share on any client makes no difference. for what it is > worth, xattrs don't change upon remount. > > what in the world are we doing wrong? > > cephx is enabled. works great for RBD. our ceph auth command for cephFS > user: > ceph auth get-or-create client.cephfs mds 'allow *' mon 'allow r' osd 'allow > rwx pool=cephfs' > > we are allowing ceph to determine what pool to use, which i think defaults > to 0/"data". Yep, that's why. Your cephx key only lets the client touch the "cephfs" pool, but the metadata server is telling the client to put its file data in the "data" pool. The client buffers up writes, sends all the metadata to the server (where it is successfully stored), but when it tries to flush anything out it fails to do so. You'll need to resolve that discrepancy, either by reconfiguring CephFS to store all the file data in the "cephfs" pool (look up the newfs command in the docs, or just set a new layout on the root dir), or adding the "data" pool to the list that the clients can access. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] CephFS emptying files or silently failing to mount?
howdy, y'all. we are testing ceph and all of its features. we love RBD! however cephFS, though clearly stated not production ready, has been stonewalling us. in an attempt to get rolling quickly, we followed some guides on CephFS ( http://goo.gl/BmVxG, http://goo.gl/1VtNk). when i mount CephFS, i can: 1) create a file with contents on clientA; 2) list the folder and see the file on clientB and clientC; 3) pull up the exact same xattr's on that file across all clients. however, as soon as i try to modify or view the contents of the file, ubuntu 12.04 says "operation not permitted". only clientA, which made the file, can try to view it's contents, but when clientA does so the file appears empty. a remount of the share on any client makes no difference. for what it is worth, xattrs don't change upon remount. what in the world are we doing wrong? cephx is enabled. works great for RBD. our ceph auth command for cephFS user: ceph auth get-or-create client.cephfs mds 'allow *' mon 'allow r' osd 'allow rwx pool=cephfs' we are allowing ceph to determine what pool to use, which i think defaults to 0/"data". mount command: mount -t ceph 10.0.10.53:6789:/ /mnt/test/ -o name=cephfs,secretfile=/etc/ceph/client.cephfs,noatime mount path exists; permissions are consistent across all clients. "/etc/ceph/client.cephfs" is populated with the output of "ceph auth print-key client.cephfs". all my clients are currently ubuntu 12.04 LTS within virtual machines. our company has not dedicated physical servers to this project. kernel for all clients: Linux os-comp2 3.5.0-30-generic #51~precise1-Ubuntu SMP Wed May 15 08:48:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux xattrs for test file (removed the hex code): # xattr -l testfile ceph.file.layout: chunk_bytes=4194304.stripe_count=1.object_size=4194304. ceph.layout: chunk_bytes=4194304.stripe_count=1.object_size=4194304. validating ceph actually created the file, in lue of http://tracker.ceph.com/issues/2753 : # cephfs testfile show_location location.file_offset: 0 location.object_offset:0 location.object_no:0 location.object_size: 4194304 location.object_name: 100. location.block_offset: 0 location.block_size: 4194304 location.osd: 4 -bo -- "But God demonstrates His own love toward us, in that while we were yet sinners, Christ died for us. Much more then, having now been justified by His blood, we shall be saved from the wrath of God through Him." Romans 5:8-9 *All *have sinned, broken God's law, and deserve eternal torment. Jesus Christ, the Son of God, died for the sins of those that will believe, purchasing our salvation, and defeated death so that we all may spend eternity in heaven. Do you desire freedom from hell and be with God in His love for eternity? "If you confess with your mouth Jesus as Lord, and believe in your heart that God raised Him from the dead, you will be saved." Romans 10:9 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
