Re: [ceph-users] CephFS writes = Permission denied
I thought that I had corrected that already and apparently I was wrong. The permissions set on MDS for the user mounting the filesystem needs to be "rw". Mine was set to "r'. ceph auth caps client.cephfs mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs_metadata,allow rwx pool=cephfs_data' Thanks! *Nate Curry* On Tue, Apr 12, 2016 at 3:56 PM, Gregory Farnum <gfar...@redhat.com> wrote: > On Tue, Apr 12, 2016 at 12:20 PM, Nate Curry <cu...@mosaicatm.com> wrote: > > I am seeing an issue with cephfs where I am unable to write changes to > the > > files system in anyway. I am running commands using sudo with a user > > account as well as the root user itself to modify ownership of files, > delete > > files, and create new files and all I get is "Permission denied". > > > > At first I thought maybe there was something wrong with the file system > and > > it was no longer read write but everything seems to check out. It is not > > mounted as read only, ceph is reporting HEALTH_OK, and there is nothing > in > > any of the logs that look like errors. I am able to unmount and remount > the > > filesystem without any issues. It also reboots and mounts no problem. > I am > > not sure what this could be caused by. Any ideas? > > Sounds like you've got your cephx permission caps set wrong. > http://docs.ceph.com/docs/master/cephfs/client-auth/ > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] CephFS writes = Permission denied
I am seeing an issue with cephfs where I am unable to write changes to the files system in anyway. I am running commands using sudo with a user account as well as the root user itself to modify ownership of files, delete files, and create new files and all I get is "Permission denied". At first I thought maybe there was something wrong with the file system and it was no longer read write but everything seems to check out. It is not mounted as read only, ceph is reporting HEALTH_OK, and there is nothing in any of the logs that look like errors. I am able to unmount and remount the filesystem without any issues. It also reboots and mounts no problem. I am not sure what this could be caused by. Any ideas? *Nate Curry* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Creating new user to mount cephfs
You are correct sir. I modified the user capabilities by adding the mds cap with the 'allow r' permission using the following command. *ceph auth caps client.cephfs mon 'allow r' mds 'allow r' osd 'allow rwx pool=cephfs_metadata,allow rwx pool=cephfs_data'* Thanks, *Nate Curry* On Thu, Apr 7, 2016 at 1:06 PM, Gregory Farnum <gfar...@redhat.com> wrote: > On Thu, Apr 7, 2016 at 10:02 AM, Nate Curry <cu...@mosaicatm.com> wrote: > > I am trying to get cephfs mounted as a user other than client.admin in > order > > to avoid putting the admin keyring on the client host. When I try to > > connect with that user and keyring I am getting the following error: > > > > mount error 13 = Permission denied > > > > I am following the exact instructions that Karan Singh put in Ceph > Cookbook > > but they do not seem to work. I checked the errata but didn't see any > > mention of there being any errors. It might be important to state that > I am > > using Jewel and am not sure if there would be a difference. I followed > the > > steps to create the osd pools for data and metadata, created the > filesystem, > > then created a user which has rwx access to both pools as follows: > > > > ceph auth get-or-create client.cephfs mon 'allow r' osd 'allow rwx > > pool=cephfs_metadata,allow rwx pool=cephfs_data' -o > > /etc/ceph/client.cephfs.keyring > > > > I then write out the keyring in /etc/ceph/ before I transfer it to my > cephfs > > client host: > > > > ceph-authtool -p -n client.cephfs /etc/ceph/client.cephfs.keyring > > > /etc/ceph/client.cephfs > > > > I then put the key in a file (cephfs.key) to be used to mount the > filesystem > > instead of specifying the key on the command like or during startup. > When I > > try to mount the filesystem as the root user using the following command > I > > get the above error. > > > > mount -t ceph mon2:6789:/ /data -o > > name=cephfs,secretfile=/etc/ceph/cephfs.key > > > > From what I can tell it looks like the user is created properly and ceph > > auth list shows that it has permissions to the pools: > > > > client.cephfs > > key: # > > caps: [mon] allow r > > caps: [osd] allow rwx pool=datastore_metadata,allow rwx > > pool=datastore_data > > > > > > Could someone tell me what else I would need to give the user permission > to > > in order to be able to mount the filesystem? > > You need an MDS stanza. See > http://docs.ceph.com/docs/master/cephfs/client-auth/ > -Greg > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Creating new user to mount cephfs
I am trying to get cephfs mounted as a user other than client.admin in order to avoid putting the admin keyring on the client host. When I try to connect with that user and keyring I am getting the following error: *mount error 13 = Permission denied* I am following the exact instructions that Karan Singh put in Ceph Cookbook but they do not seem to work. I checked the errata but didn't see any mention of there being any errors. It might be important to state that I am using Jewel and am not sure if there would be a difference. I followed the steps to create the osd pools for data and metadata, created the filesystem, then created a user which has rwx access to both pools as follows: *ceph auth get-or-create client.cephfs mon 'allow r' osd 'allow rwx pool=cephfs_metadata,allow rwx pool=cephfs_data' -o /etc/ceph/client.cephfs.keyring* I then write out the keyring in /etc/ceph/ before I transfer it to my cephfs client host: *ceph-authtool -p -n client.cephfs /etc/ceph/client.cephfs.keyring > /etc/ceph/client.cephfs* I then put the key in a file (cephfs.key) to be used to mount the filesystem instead of specifying the key on the command like or during startup. When I try to mount the filesystem as the root user using the following command I get the above error. *mount -t ceph mon2:6789:/ /data -o name=cephfs,secretfile=/etc/ceph/cephfs.key* >From what I can tell it looks like the user is created properly and ceph auth list shows that it has permissions to the pools: *client.cephfskey: #caps: [mon] allow rcaps: [osd] allow rwx pool=datastore_metadata,allow rwx pool=datastore_data* Could someone tell me what else I would need to give the user permission to in order to be able to mount the filesystem? Thanks, *Nate Curry* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSDs keep going down
That was it. I had recently rebuilt the OSD hosts and completely forgot to configure the firewall. Thanks, *Nate Curry* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] OSDs keep going down
I am having some issues with my newly setup cluster. I am able to get all
of my 32 OSDs to start after setting up udev rules for my journal
partitions but they keep going down. It did seem like half of them would
stay up at first but after I checked it this morning I found only 1/4 of
them were up when I ran "ceph osd tree". The systemd scripts are running
so it doesn't seem like that is the issue. I don't see anything glaring in
the log files, which may just reflect my experience level with ceph.
I tried to look for errors and knock out any that seemed obvious but I
can't seem to get that done either. The cluster was initially set to 64pgs
and I tried to update that to 1024 but it hasn't finished creating all of
them and it seems stuck with 270 stale+creating pgs. This is preventing me
from updating the number of pgps as it says it is busy creating pgs.
I am thinking that the downed OSDs are probably my problem as far as the
pgs getting created are concerned. I just don't can't seem to find the
reason why they are going down. Could someone help shine some light on
this for me?
[ceph@matm-cm1 ~]$ ceph status
cluster 5a463eb9-b918-4d97-b853-7a5ebd3c0ac2
health HEALTH_ERR
1006 pgs are stuck inactive for more than 300 seconds
1 pgs degraded
140 pgs down
736 pgs peering
1024 pgs stale
1006 pgs stuck inactive
18 pgs stuck unclean
1 pgs undersized
pool rbd pg_num 1024 > pgp_num 64
monmap e1: 3 mons at {matm-cm1=
192.168.41.153:6789/0,matm-cm2=192.168.41.154:6789/0,matm-cm3=192.168.41.155:6789/0
}
election epoch 8, quorum 0,1,2 matm-cm1,matm-cm2,matm-cm3
osdmap e417: 32 osds: 9 up, 9 in; 496 remapped pgs
flags sortbitwise
pgmap v1129: 1024 pgs, 1 pools, 0 bytes data, 0 objects
413 MB used, 16753 GB / 16754 GB avail
564 stale+remapped+peering
270 stale+creating
125 stale+down+remapped+peering
32 stale+peering
17 stale+active+remapped
15 stale+down+peering
1 stale+active+undersized+degraded+remapped
[ceph@matm-cm1 ~]$ ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 58.17578 root default
-2 14.54395 host matm-cs1
0 1.81799 osd.0down0 1.0
1 1.81799 osd.1down0 1.0
2 1.81799 osd.2down0 1.0
3 1.81799 osd.3down0 1.0
4 1.81799 osd.4down0 1.0
5 1.81799 osd.5down0 1.0
6 1.81799 osd.6down0 1.0
7 1.81799 osd.7down0 1.0
-3 14.54395 host matm-cs2
8 1.81799 osd.8 up 1.0 1.0
9 1.81799 osd.9 up 1.0 1.0
10 1.81799 osd.10 up 1.0 1.0
11 1.81799 osd.11 up 1.0 1.0
12 1.81799 osd.12 up 1.0 1.0
13 1.81799 osd.13 up 1.0 1.0
14 1.81799 osd.14 up 1.0 1.0
15 1.81799 osd.15 up 1.0 1.0
-4 14.54395 host matm-cs3
16 1.81799 osd.16 down0 1.0
17 1.81799 osd.17 down0 1.0
18 1.81799 osd.18 down0 1.0
19 1.81799 osd.19 down0 1.0
20 1.81799 osd.20 down0 1.0
21 1.81799 osd.21 down0 1.0
22 1.81799 osd.22 down0 1.0
23 1.81799 osd.23 down0 1.0
-5 14.54395 host matm-cs4
24 1.81799 osd.24 down0 1.0
31 1.81799 osd.31 down0 1.0
25 1.81799 osd.25 down0 1.0
27 1.81799 osd.27 down0 1.0
29 1.81799 osd.29 down0 1.0
28 1.81799 osd.28 down0 1.0
30 1.81799 osd.30 up 1.0 1.0
26 1.81799 osd.26 down0 1.00000
*Nate Curry*
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Monitor questions
Yes that was what I meant. Thanks. Was that in a production environment? Nate Curry On Jul 10, 2015 11:21 AM, Quentin Hartman qhart...@direwolfdigital.com wrote: You mean the hardware config? They are older Core2-based servers with 4GB of RAM. Nothing special. I have one running mon and rgw, one running mon and mds, and one run just a mon. QH On Fri, Jul 10, 2015 at 8:58 AM, Nate Curry cu...@mosaicatm.com wrote: What was your monitor node's configuration when you had multiple ceph daemons running on them? *Nate Curry* IT Manager ISSM *Mosaic ATM* mobile: 240.285.7341 office: 571.223.7036 x226 cu...@mosaicatm.com On Thu, Jul 9, 2015 at 5:36 PM, Quentin Hartman qhart...@direwolfdigital.com wrote: I have my mons sharing the ceph network, and while I currently do not run mds or rgw, I have run those on my mon hosts in the past with no perceptible ill effects. On Thu, Jul 9, 2015 at 3:20 PM, Nate Curry cu...@mosaicatm.com wrote: I have a question in regards to monitor nodes and network layout. Its my understanding that there should be two networks; a ceph only network for comms between the various ceph nodes, and a separate storage network where other systems will interface with the ceph nodes. Are the monitor nodes supposed to straddle both the ceph only network and the storage network or just in the ceph network? Another question is can I run multiple things on the monitor nodes? Like the RADOS GW and the MDS? Thanks, *Nate Curry* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Monitor questions
What was your monitor node's configuration when you had multiple ceph daemons running on them? *Nate Curry* IT Manager ISSM *Mosaic ATM* mobile: 240.285.7341 office: 571.223.7036 x226 cu...@mosaicatm.com On Thu, Jul 9, 2015 at 5:36 PM, Quentin Hartman qhart...@direwolfdigital.com wrote: I have my mons sharing the ceph network, and while I currently do not run mds or rgw, I have run those on my mon hosts in the past with no perceptible ill effects. On Thu, Jul 9, 2015 at 3:20 PM, Nate Curry cu...@mosaicatm.com wrote: I have a question in regards to monitor nodes and network layout. Its my understanding that there should be two networks; a ceph only network for comms between the various ceph nodes, and a separate storage network where other systems will interface with the ceph nodes. Are the monitor nodes supposed to straddle both the ceph only network and the storage network or just in the ceph network? Another question is can I run multiple things on the monitor nodes? Like the RADOS GW and the MDS? Thanks, *Nate Curry* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Monitor questions
I have a question in regards to monitor nodes and network layout. Its my understanding that there should be two networks; a ceph only network for comms between the various ceph nodes, and a separate storage network where other systems will interface with the ceph nodes. Are the monitor nodes supposed to straddle both the ceph only network and the storage network or just in the ceph network? Another question is can I run multiple things on the monitor nodes? Like the RADOS GW and the MDS? Thanks, *Nate Curry* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph Monitor Memory Sizing
I was reading the documentation on the website in regards to the recommended memory for the monitors. It says that there should be 1GB of RAM per daemon instance. Does the daemon instance refer to the number of OSDs? I am planning on setting up 4 hosts with 16 OSDs each initially. Would I need 64 GB of memory per monitor? I don't think that would scale well at some point so I am thinking that is not correct. Can I get some clarification? Thanks, *Nate Curry* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Journal Disk Size
Are you using the 4TB disks for the journal? *Nate Curry* IT Manager ISSM *Mosaic ATM* mobile: 240.285.7341 office: 571.223.7036 x226 cu...@mosaicatm.com On Thu, Jul 2, 2015 at 12:16 PM, Shane Gibson shane_gib...@symantec.com wrote: I'd def be happy to share what numbers I can get out of it. I'm still a neophyte w/ Ceph, and learning how to operate it, set it up ... etc... My limited performance testing to date has been with stock XFS ceph-disk built filesystem for the OSDs, basic PG/CRUSH map stuff - and using dd across RBD mounted volumes ... I'm learning how to scale it up, and start tweaking and tuning. If anyone on the list is interested in specific tests and can provide specific detailed instructions on configuration, test patterns, etc ... Im happy to run them if I can ... We're baking in automation around the Ceph deoployment from fresh build using the Open Crowbar deployment tooling, with a Ceph work load on it. RIght now, modifying the Ceph work load to work across multple L3 rack boundaries in the cluster. Physical servers are Dell R720xd platforms, with 12 spinning (4TB 7200 rpm) data disks, and 2x 10k 600 GB mirrired OS disks. Memory is 128 GB, and dual 6-core HT CPUs. ~~shane On 7/1/15, 5:24 PM, German Anders gand...@despegar.com wrote: I'm interested in such a configuration, can you share some perfomance test/numbers? Thanks in advance, Best regards, *German* 2015-07-01 21:16 GMT-03:00 Shane Gibson shane_gib...@symantec.com: It also depends a lot on the size of your cluster ... I have a test cluster I'm standing up right now with 60 nodes - a total of 600 OSDs each at 4 TB ... If I lose 4 TB - that's a very small fraction of the data. My replicas are going to be spread out across a lot of spindles, and replicating that missing 4 TB isn't much of an issue, across 3 racks each with 80 gbit/sec ToR uplinks to Spine. Each node has 20 gbit/sec to ToR in a bond. On the other hand ... if you only have 4 .. or 8 ... or 10 servers ... and a smaller number of OSDs - you have fewer spindles replicating that loss, and it might be more of an issue. It just depends on the size/scale of your environment. We're going to 8 TB drives - and that will ultimately be spread over a 100 or more physical servers w/ 10 OSD disks per server. This will be across 7 to 10 racks (same network topology) ... so an 8 TB drive loss isn't too big of an issue. Now that assumes that replication actually works well in that size cluster. We're still cessing out this part of the PoC engagement. ~~shane On 7/1/15, 5:05 PM, ceph-users on behalf of German Anders ceph-users-boun...@lists.ceph.com on behalf of gand...@despegar.com wrote: ask the other guys on the list, but for me to lose 4TB of data is to much, the cluster will still running fine, but in some point you need to recover that disk, and also if you lose one server with all the 4TB disk in that case yeah it will hurt the cluster, also take into account that with that kind of disk you will get no more than 100-110 iops per disk *German Anders* Storage System Engineer Leader *Despegar* | IT Team *office* +54 11 4894 3500 x3408 *mobile* +54 911 3493 7262 *mail* gand...@despegar.com 2015-07-01 20:54 GMT-03:00 Nate Curry cu...@mosaicatm.com: 4TB is too much to lose? Why would it matter if you lost one 4TB with the redundancy? Won't it auto recover from the disk failure? Nate Curry On Jul 1, 2015 6:12 PM, German Anders gand...@despegar.com wrote: I would probably go with less size osd disks, 4TB is to much to loss in case of a broken disk, so maybe more osd daemons with less size, maybe 1TB or 2TB size. 4:1 relationship is good enough, also i think that 200G disk for the journals would be ok, so you can save some money there, the osd's of course configured them as a JBOD, don't use any RAID under it, and use two different networks for public and cluster net. *German* 2015-07-01 18:49 GMT-03:00 Nate Curry cu...@mosaicatm.com: I would like to get some clarification on the size of the journal disks that I should get for my new Ceph cluster I am planning. I read about the journal settings on http://ceph.com/docs/master/rados/configuration/osd-config-ref/#journal-settings but that didn't really clarify it for me that or I just didn't get it. I found in the Learning Ceph Packt book it states that you should have one disk for journalling for every 4 OSDs. Using that as a reference I was planning on getting multiple systems with 8 x 6TB inline SAS drives for OSDs with two SSDs for journalling per host as well as 2 hot spares for the 6TB drives and 2 drives for the OS. I was thinking of 400GB SSD drives but am wondering if that is too much. Any informed opinions would be appreciated. Thanks, *Nate Curry* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph
[ceph-users] Ceph Journal Disk Size
I would like to get some clarification on the size of the journal disks that I should get for my new Ceph cluster I am planning. I read about the journal settings on http://ceph.com/docs/master/rados/configuration/osd-config-ref/#journal-settings but that didn't really clarify it for me that or I just didn't get it. I found in the Learning Ceph Packt book it states that you should have one disk for journalling for every 4 OSDs. Using that as a reference I was planning on getting multiple systems with 8 x 6TB inline SAS drives for OSDs with two SSDs for journalling per host as well as 2 hot spares for the 6TB drives and 2 drives for the OS. I was thinking of 400GB SSD drives but am wondering if that is too much. Any informed opinions would be appreciated. Thanks, *Nate Curry* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
