[ceph-users] Re: Problem : "1 pools have many more objects per pg than average"

[St-Germain, Sylvain (SSC/SPC)]

Ok case close, I found the solution :

1- Connect on the active mgr

2- Show config 
#ceph daemon mgr.`hostname -s` config show | grep mon_pg_warn_max_object_skew
"mon_pg_warn_max_object_skew": "10.00",

3- Change the value
#ceph config set mgr.`hostname -s` mon_pg_warn_max_object_skew 20

4- Confirm the change
#ceph daemon mgr.`hostname -s` config show | grep mon_pg_warn_max_object_skew
"mon_pg_warn_max_object_skew": "20.00",

5- Result
#ceph health detail
HEALTH_OK

Sylvain
-Message d'origine-
De : St-Germain, Sylvain (SSC/SPC) 
Envoyé : 23 janvier 2020 07:55
À : 'ceph-users@ceph.io' 
Objet : Problem : "1 pools have many more objects per pg than average"

/ Problem ///

 I've got a Warning on my cluster that I cannot remove :

"1 pools have many more objects per pg than average"

Does somebody has some insight ? I think it's normal to have this warning 
because I have just one pool in use, but how can I remove this warning ?

Thx !

/ INFORMATION ///

*** Here's some information about the cluster

# ceph health detail
HEALTH_WARN 1 pools have many more objects per pg than average 
MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
pool default.rgw.buckets.data objects per pg (50567) is more than 14.1249 
times cluster average (3580)

# sudo ceph-conf -D | grep mon_pg_warn_max_object_skew 
mon_pg_warn_max_object_skew = 10.00

# ceph daemon mon.dao-wkr-04 config show | grep mon_pg_warn_max_object_skew
"mon_pg_warn_max_object_skew": "10.00"

# ceph -v
ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)

# ceph df
RAW STORAGE:
CLASS   SIZEAVAIL   USED   RAW USED %RAW USED 
hdd 873 TiB 823 TiB 50 TiB   51 TiB  5.80 
TOTAL   873 TiB 823 TiB 50 TiB   51 TiB  5.80 
 
POOLS:
POOLID STORED  OBJECTS  
USED%USED MAX AVAIL 
.rgw.root   11 3.5 KiB  8   
1.5 MiB 0   249 TiB 
default.rgw.control 12 0 B  8   
0 B 0   249 TiB 
default.rgw.meta13  52 KiB  186 
34 MiB  0   249 TiB 
default.rgw.log 14 0 B  207 
0 B 0   249 TiB 
default.rgw.buckets.index   15 1.2 GiB  131 
1.2 GiB 0   249 TiB 
cephfs_data 29 915 MiB  202 
1.5 GiB 0   467 TiB 
cephfs_metadata 30 145 KiB  23  
2.1 MiB 0   249 TiB 
default.rgw.buckets.data31  30 TiB  12.95M  
50 TiB  6.32  467 TiB 


# ceph osd dump | grep default.rgw.buckets.data pool 31 
'default.rgw.buckets.data' erasure size 8 min_size 6 crush_rule 2 object_hash 
rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 9502 lfor 
0/2191/7577 flags hashpspool stripe_width 20480 target_size_ratio 0.4 
application rgw

/// SOLUTION TRIED ///

1- I try to increase the value of mon_pg_warn_max_object_skew parameter

#sudo ceph tell mon.* injectargs '--mon_pg_warn_max_object_skew 20'
#sudo ceph tell osd.* injectargs '--mon_pg_warn_max_object_skew 20'

+ And reboot the monitor

The parameter didn't change.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-volume lvm filestore OSDs fail to start on reboot. Permission denied on journal partition

[Wesley Dillingham]

Thanks all for responses. It seems that both original responders relied on
modification to the osd-prestart script. I can confirm that is working for
me too and am using that as a temporary solution.

Jan, It seems that the log you mentioned shows the unit attempting to do
the right thing here (chown the partition ceph:ceph) however it does not
seem to take. Will continue to look and file a bug/pr if I make any
progress.  Here are the logs related to an affected osd (219) and its
journal partition (/dev/sdc5). As a note this log output is from a server
which has the added chown in the osd prestart script.

[2020-01-23 13:03:27,230][systemd][INFO  ] raw systemd input received:
lvm-219-529ea347-b129-4b53-81cb-bb5f2d91f8ae
[2020-01-23 13:03:27,231][systemd][INFO  ] parsed sub-command: lvm, extra
data: 219-529ea347-b129-4b53-81cb-bb5f2d91f8ae
[2020-01-23 13:03:27,305][ceph_volume.process][INFO  ] Running command:
/usr/sbin/ceph-volume lvm trigger 219-529ea347-b129-4b53-81cb-bb5f2d91f8ae
[2020-01-23 13:03:28,235][ceph_volume.process][INFO  ] stderr Running
command: /bin/mount -t xfs -o rw,noatime,inode64
/dev/ceph-dd8d5283-fd1a-4114-9b53-6478dab7101c/osd-data-529ea347-b129-4b53-81cb-bb5f2d91f8ae
/var/lib/ceph/osd/ceph-219
[2020-01-23 13:03:28,472][ceph_volume.process][INFO  ] stderr Running
command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-219
[2020-01-23 13:27:19,566][ceph_volume.process][INFO  ] stderr Running
command: /bin/ln -snf /dev/sdc5 /var/lib/ceph/osd/ceph-219/journal
[2020-01-23 13:27:19,571][ceph_volume.process][INFO  ] stderr Running
command: /bin/chown -R ceph:ceph /dev/sdc5
[2020-01-23 13:27:19,576][ceph_volume.process][INFO  ] stderr Running
command: /bin/systemctl enable
ceph-volume@lvm-219-529ea347-b129-4b53-81cb-bb5f2d91f8ae
[2020-01-23 13:27:19,672][ceph_volume.process][INFO  ] stderr Running
command: /bin/systemctl enable --runtime ceph-osd@219
[2020-01-23 13:27:19,679][ceph_volume.process][INFO  ] stderr stderr:
Created symlink from
/run/systemd/system/ceph-osd.target.wants/ceph-osd@219.service to
/usr/lib/systemd/system/ceph-osd@.service.
[2020-01-23 13:27:19,765][ceph_volume.process][INFO  ] stderr Running
command: /bin/systemctl start ceph-osd@219
[2020-01-23 13:27:19,773][ceph_volume.process][INFO  ] stderr -->
ceph-volume lvm activate successful for osd ID: 219
[2020-01-23 13:27:19,784][systemd][INFO  ] successfully triggered
activation for: 219-529ea347-b129-4b53-81cb-bb5f2d91f8ae


Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Thu, Jan 23, 2020 at 4:31 AM Jan Fajerski  wrote:

> On Wed, Jan 22, 2020 at 12:00:28PM -0500, Wesley Dillingham wrote:
> >   After upgrading to Nautilus 14.2.6 from Luminous 12.2.12 we are seeing
> >   the following behavior on OSDs which were created with "ceph-volume lvm
> >   create --filestore --osd-id  --data  --journal "
> >   Upon restart of the server containing these OSDs they fail to start
> >   with the following error in the logs:
> >2020-01-21 13:36:11.635 7fee633e8a80 -1
> filestore(/var/lib/ceph/osd/ceph-199) mo
> >unt(1928): failed to open journal /var/lib/ceph/osd/ceph-199/journal:
> (13) Permi
> >ssion denied
> >
> >   /var/lib/ceph/osd/ceph-199/journal symlinks to /dev/sdc5 in our case
> >   and inspecting the ownership on /dev/sdc5 it is root:root, chowning
> >   that to ceph:ceph causes the osd to start and come back up and in near
> >   instantly.
> >   As a note these OSDs we experience this with are OSDs which have
> >   previously failed and been replaced using the above ceph-volume, longer
> >   running OSDs in the same server created with ceph-disk or ceph-volume
> >   simple (that have a corresponding .json in /etc/ceph/osd) start up fine
> >   and get ceph:ceph on their journal partition. Bluestore OSDs also do
> >   not have any issue.
> >   My hope is that I can preemptively fix these OSDs before shutting them
> >   down so that reboots happen seamlessly. Thanks for any insight.
> ceph-volume is supposed to take care of this via the ceph-volume@ systemd
> unit.
> This is a one shot unit, that should set things up and then start the osd.
> The unit name is a bit convoluted: ceph-volume@-, there
> should
> be symbolic link in /etc/systemd/system/multi-user.target.wants/
>
> You can also check cat /var/log/ceph/ceph-volume-systemd.log for any
> errors.
> Feel free to open a tracker ticket on
> https://tracker.ceph.com/projects/ceph-volume
>
> >
> >   Respectfully,
> >   Wes Dillingham
> >   [1]w...@wesdillingham.com
> >   [2]LinkedIn
> >
> >References
> >
> >   1. mailto:w...@wesdillingham.com
> >   2. http://www.linkedin.com/in/wesleydillingham
>
> >___
> >ceph-users mailing list -- ceph-users@ceph.io
> >To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> --
> Jan Fajerski
> Senior Software Engineer Enterprise Storage
> SUSE Software Solutions Germany GmbH
> Maxfeldstr. 5, 90409 Nürnberg, Germany
> 

[ceph-users] Re: ceph 14.2.6 problem with default args to rbd (--name)

[Jason Dillaman]

CCing correct mailing list ...

On Thu, Jan 23, 2020 at 3:19 PM Jason Dillaman  wrote:
>
> On Mon, Jan 20, 2020 at 8:26 AM Rainer Krienke  wrote:
> >
> > Hello,
> >
> > I am fighting with rbd and CEPH_ARGS in order to make typing easier on a
> > client. First I created a keyring on one of the ceph nodes:
> >
> > # ceph auth add client.rainer  mon 'profile rbd' osd 'profile rbd'
> > added key for client.rainer
> >
> > Then I added this keyring to /etc/ceph/ceph.keyring on a client machine.
> > # cat /etc/ceph/ceph.keyring:
> > [client.rainer]
> > key = xxyxyxyxyxyxyxyxyxyxyxyxyxyxy==
> >
> > Now on the client I try:
> >
> > # rbd ls
> > 2020-01-20 14:15:49.124 7fa9191f0700 -1 monclient(hunting):
> > handle_auth_bad_method server allowed_methods [2] but i only support [2]
> > 2020-01-20 14:15:49.124 7fa9189ef700 -1 monclient(hunting):
> > handle_auth_bad_method server allowed_methods [2] but i only support [2]
> > rbd: couldn't connect to the cluster!
> > rbd: listing images failed: (1) Operation not permitted
> >
> > Using the --name  arg things work good:
> >
> > # rbd ls --name client.rainer
> > 
> > ...
> >
> > Now to make typing easier I tried:
> > # export CEPH_ARGS="--name client.rainer --keyring=/etc/ceph/ceph.keyring"
> > # rbd ls
> > 2020-01-20 14:18:33.367 7f691177c700 -1 monclient(hunting):
> > handle_auth_bad_method server allowed_methods [2] but i only support [2]
> > rbd: couldn't connect to the cluster!
> > rbd: listing images failed: (1) Operation not permitted
> >
> > According to the documentation this should work, but it seems it
> > doesn't. Something I am doing wrong or is this a bug?
>
> Hmm, it looks like this was unintentionally broken globally throughout
> Ceph in commit 7f23142f5ccc5ac8153d32b2c9a8353593831967 back in the
> mimic release. Previously, the CEPH_ARGS environment overrides were
> combined with the command-line arguments near startup, but now they
> are not combined until after numerous early arguments are parsed (e.g.
> "name", "id", "cluster", "conf", etc). I'll open a tracker ticket
> against this issue.
>
> > Thanks
> > Rainer
> > --
> > Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
> > 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312
> > Web: http://userpages.uni-koblenz.de/~krienke
> > PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html
> > ___
> > ceph-users mailing list
> > ceph-us...@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
> --
> Jason



-- 
Jason
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap balancer

[Thomas Schneider]

Hi Dan,

I have opened this this bug report for balancer not working as expected.
https://tracker.ceph.com/issues/43586
Do you experience the same issue?

Regards
Thomas

Am 23.01.2020 um 16:05 schrieb Dan van der Ster:
> Hi Frank,
>
> No, it is basically balancing the num_pgs per TB (per osd).
>
> Cheers, Dan
>
>
> On Thu, Jan 23, 2020 at 3:53 PM Frank R  > wrote:
>
> Hi all,
>
> Does using the Upmap balancer require that all OSDs be the same size
> (per device class)?
>
> thx
> Frank
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> 
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS ghost usage/inodes

[Oskar Malnowicz]

Any other ideas ?

> Am 15.01.2020 um 15:50 schrieb Oskar Malnowicz 
> :
> 
> the situation is:
> 
> health: HEALTH_WARN
>   1 pools have many more objects per pg than average
> 
> $ ceph health detail
> MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
> pool cephfs_data objects per pg (315399) is more than 1227.23 times 
> cluster average (257)
> 
> $ ceph df
> RAW STORAGE:
> CLASS SIZEAVAIL   USEDRAW USED %RAW USED
> hdd   7.8 TiB 7.4 TiB 326 GiB  343 GiB  4.30
> TOTAL 7.8 TiB 7.4 TiB 326 GiB  343 GiB  4.30
> 
> POOLS:
> POOL   ID STORED  OBJECTS USED%USED 
> MAX AVAIL
> cephfs_data 6 2.2 TiB   2.52M 2.2 TiB 26.44   
> 3.0 TiB
> cephfs_metadata 7 9.7 MiB 379 9.7 MiB 0   
> 3.0 TiB
>   
> the stored value of the "cephfs_data" pool is 2.2TiB. This must be wrong. 
> When i execute "du -sh" from the MDS root "/" i get an usage:
> 
> $ du -sh
> 31G .
> 
> "df -h" shows:
> 
> $ df -h
> Filesystem   Size  Used Avail Use% Mounted on
> ip1,ip2,ip3:/5.2T  2.2T  3.0T  43% /storage/cephfs
> 
> It says that "Used" ist 2.2T but "du" shows 31G
> 
> the pg_num from the "cephfs_data" pool is now 8. Autoscale suggest me to set 
> this parameter to 512
> 
> $ ceph osd pool autoscale-status
> POOLSIZE TARGET SIZE RATE RAW CAPACITY  RATIO TARGET 
> RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE
> cephfs_metadata9994k  2.07959G 0. 
>   1.0  8off
> cephfs_data2221G  2.07959G 0.5582 
>   1.0  8512 off
> 
> after setting pg_num to 512 the situation is:
> 
> $ ceph health detail
> HEALTH_WARN 1 pools have many more objects per pg than average
> MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
> pool cephfs_data objects per pg (4928) is more than 100.571 times cluster 
> average (49)
> 
> $ ceph df
> RAW STORAGE:
> CLASS SIZEAVAIL   USEDRAW USED %RAW USED
> hdd   7.8 TiB 7.4 TiB 329 GiB  346 GiB  4.34
> TOTAL 7.8 TiB 7.4 TiB 329 GiB  346 GiB  4.34
> 
> POOLS:
> POOL  ID STORED  OBJECTS USED
> %USED MAX AVAIL
> cephfs_data6  30 GiB   2.52M  61 GiB  
> 0.99   3.0 TiB
> cephfs_metadata7 9.8 MiB 379  20 MiB  
>0   3.0 TiB
> 
> The "stored" value changed from 2.2TiB to 30GiB !!! This should be the 
> correct usage/size.
> 
> When i execute "du -sh" from the MDS root "/" i get again an usage:
> 
> $ du -sh
> 31G
> 
> and "df -h" shows again
> 
> $ df -h
> Filesystem   Size  Used Avail Use% Mounted on
> ip1,ip2,ip3:/5.2T  2.2T  3.0T  43% /storage/cephfs
> 
> It says that "Used" ist 2.2T but "du" shows 31G
> 
> Can anybody explain me whats the problem ?
> 
> 
> 
> 
> Am 14.01.20 um 11:15 schrieb Florian Pritz:
>> Hi,
>> 
>> When we tried putting some load on our test cephfs setup by restoring a
>> backup in artifactory, we eventually ran out of space (around 95% used
>> in `df` = 3.5TB) which caused artifactory to abort the restore and clean
>> up. However, while a simple `find` no longer shows the files, `df` still
>> claims that we have around 2.1TB of data on the cephfs. `df -i` also
>> shows 2.4M used inodes. When using `du -sh` on a top-level mountpoint, I
>> get 31G used, which is data that is still really here and which is
>> expected to be here.
>> 
>> Consequently, we also get the following warning:
>> 
>> 
>>> MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
>>> pool cephfs_data objects per pg (38711) is more than 231.802 times 
>>> cluster average (167)
>>> 
>> We are running ceph 14.2.5.
>> 
>> We have snapshots enabled on cephfs, but there are currently no active
>> snapshots listed by `ceph daemon mds.$hostname dump snaps --server` (see
>> below). I can't say for sure if we created snapshots during the backup
>> restore.
>> 
>> 
>>> {
>>> "last_snap": 39,
>>> "last_created": 38,
>>> "last_destroyed": 39,
>>> "pending_noop": [],
>>> "snaps": [],
>>> "need_to_purge": {},
>>> "pending_update": [],
>>> "pending_destroy": []
>>> }
>>> 
>> We only have a single CephFS.
>> 
>> We use the pool_namespace xattr for our various directory trees on the
>> cephfs.
>> 
>> `ceph df` shows:
>> 
>> 
>>> POOL ID STORED   OBJECTS   USED%USED MAX AVAIL
>>> cephfs_data  6  2.1 TiB  2.48M 2.1 TiB 24.97   3.1 TiB
>>> 
>> `ceph daemon mds.$hostname perf dump | grep stray` shows:
>> 
>> 
>>> "num_strays": 0,
>>> "num_strays_delayed": 0,
>>> "num_strays_enqueuing": 0,
>>> "strays_created": 5097138,
>>> "strays_enqueued": 5097138,
>>> 

[ceph-users] Re: Radosgw/Objecter behaviour for homeless session

[Biswajeet Patra]

+ bumping up
Any suggestions/thoughts on this issue ?

Regards,
Biswajeet

On Tue, Nov 26, 2019 at 11:28 AM Biswajeet Patra <
biswajeet.pa...@flipkart.com> wrote:

> Hi All,
> I have a query regarding objecter behaviour for homeless session. In
> situations when all OSDs containing copies (*let say replication 3*) of
> an object are down, the objecter assigns a homeless session (OSD=-1) to a
> client request. This request makes radosgw thread hang indefinitely as the
> data could never be served because all required OSDs are down. With
> multiple similar requests, all the radosgw threads gets exhausted and
> hanged indefinitely waiting for the OSDs to come up. This creates complete
> service unavailability as no rgw threads are present to process valid
> requests which could have been directed towards active PGs/OSDs.
> I think we should have behaviour in objecter or radosgw to terminate
> request and return early in case of a homeless session. Let me know your
> thoughts on this.
>
> Regards,
> Biswajeet
>

-- 



*-*


*This email and any files transmitted with it are confidential and 
intended solely for the use of the individual or entity to whom they are 
addressed. If you have received this email in error, please notify the 
system manager. This message contains confidential information and is 
intended only for the individual named. If you are not the named addressee, 
you should not disseminate, distribute or copy this email. Please notify 
the sender immediately by email if you have received this email by mistake 
and delete this email from your system. If you are not the intended 
recipient, you are notified that disclosing, copying, distributing or 
taking any action in reliance on the contents of this information is 
strictly prohibited.*

 

*Any views or opinions presented in this 
email are solely those of the author and do not necessarily represent those 
of the organization. Any information on shares, debentures or similar 
instruments, recommended product pricing, valuations and the like are for 
information purposes only. It is not meant to be an instruction or 
recommendation, as the case may be, to buy or to sell securities, products, 
services nor an offer to buy or sell securities, products or services 
unless specifically stated to be so on behalf of the Flipkart group. 
Employees of the Flipkart group of companies are expressly required not to 
make defamatory statements and not to infringe or authorise any 
infringement of copyright or any other legal right by email communications. 
Any such communication is contrary to organizational policy and outside the 
scope of the employment of the individual concerned. The organization will 
not accept any liability in respect of such communication, and the employee 
responsible will be personally liable for any damages or other liability 
arising.*

 

*Our organization accepts no liability for the 
content of this email, or for the consequences of any actions taken on the 
basis of the information *provided,* unless that information is 
subsequently confirmed in writing. If you are not the intended recipient, 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.*


_-_

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap balancer

[Dan van der Ster]

Hi Frank,

No, it is basically balancing the num_pgs per TB (per osd).

Cheers, Dan


On Thu, Jan 23, 2020 at 3:53 PM Frank R  wrote:

> Hi all,
>
> Does using the Upmap balancer require that all OSDs be the same size
> (per device class)?
>
> thx
> Frank
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS with cache-tier kernel-mount client unable to write (Nautilus)

[Ilya Dryomov]

On Thu, Jan 23, 2020 at 3:31 PM Hayashida, Mami  wrote:
>
> Thanks, Ilya.
>
> First, I was not sure whether to post my question on @ceph.io or 
> @lists.ceph.com (I subscribe to both) -- should I use @ceph.io in the future?

Yes.  I got the following when I replied to your previous email:

  As you may or may not be aware, most of the Ceph mailing lists have
  migrated to a new self-hosted instance of Mailman 3.  For the past few
  months, both this list and ceph-users@ceph.io have been enabled.

  As of January 22, 2020, mail sent to ceph-us...@lists.ceph.com will no
  longer be delivered.  This domain will remain so that permalinks to
  archives are preserved.

  Please use the new address ceph-users@ceph.io instead.

>
> Second, thanks for your advice on cache-tiering -- I was starting to feel 
> that way but always good to know what Ceph "experts" would say.
>
> Third, I tried enabling (and setting) the pool application commands you 
> outlined but got errors (Ceph is not allowing me to enable/set application on 
> the cache tier)
>
> $ ceph osd pool application enable cephfs-data-cache cephfs
> Error EINVAL: application must be enabled on base tier
>  $ ceph osd pool application set cephfs-data-cache cephfs data cephfs_test
> Error EINVAL: application metadata must be set on base tier
>
> Since at this point, it is highly unlikely that we will be utilizing 
> cache-tier on our production clusters, and there is a work around it (by 
> manually creating a CephFS client key), this is nothing serious or urgent; 
> but I thought I should let you guys know.

I haven't actually tried it.  There is probably a way to make it work
(recreating the pools and doing the application stuff before tiering or
something along those lines), but yeah, cache tiering is not without
sharp edges...

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] upmap balancer

[Frank R]

Hi all,

Does using the Upmap balancer require that all OSDs be the same size
(per device class)?

thx
Frank
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs : write error: Operation not permitted

[Ilya Dryomov]

On Wed, Jan 22, 2020 at 8:58 AM Yoann Moulin  wrote:
>
> Hello,
>
> On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook 
> stable-4.0, I have an issue with cephfs. I can create a folder, I can
> create empty files, but cannot write data on like I'm not allowed to write to 
> the cephfs_data pool.
>
> > $ ceph -s
> >   cluster:
> > id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09
> > health: HEALTH_OK
> >
> >   services:
> > mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h)
> > mgr: iccluster039(active, since 21h), standbys: iccluster041, 
> > iccluster042
> > mds: cephfs:3 
> > {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active}
> > osd: 24 osds: 24 up (since 22h), 24 in (since 22h)
> > rgw: 1 daemon active (iccluster043.rgw0)
> >
> >   data:
> > pools:   9 pools, 568 pgs
> > objects: 800 objects, 225 KiB
> > usage:   24 GiB used, 87 TiB / 87 TiB avail
> > pgs: 568 active+clean
>
> The 2 cephfs pools:
>
> > $ ceph osd pool ls detail | grep cephfs
> > pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash 
> > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 
> > 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application 
> > cephfs
> > pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 
> > object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 
> > flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 
> > pg_num_min 16 recovery_priority 5 application cephfs
>
> The status of the cephfs filesystem:
>
> > $ ceph fs status
> > cephfs - 1 clients
> > ==
> > +--++--+---+---+---+
> > | Rank | State  | MDS  |Activity   |  dns  |  inos |
> > +--++--+---+---+---+
> > |  0   | active | iccluster043 | Reqs:0 /s |   34  |   18  |
> > |  1   | active | iccluster041 | Reqs:0 /s |   12  |   16  |
> > |  2   | active | iccluster042 | Reqs:0 /s |   10  |   13  |
> > +--++--+---+---+---+
> > +-+--+---+---+
> > |   Pool  |   type   |  used | avail |
> > +-+--+---+---+
> > | cephfs_metadata | metadata | 4608k | 27.6T |
> > |   cephfs_data   |   data   |0  | 27.6T |
> > +-+--+---+---+
> > +-+
> > | Standby MDS |
> > +-+
> > +-+
> > MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) 
> > nautilus (stable)
>
>
> > # mkdir folder
> > # echo "foo" > bar
> > -bash: echo: write error: Operation not permitted
> > # ls -al
> > total 4
> > drwxrwxrwx  1 root root2 Jan 22 07:30 .
> > drwxr-xr-x 28 root root 4096 Jan 21 09:25 ..
> > -rw-r--r--  1 root root0 Jan 22 07:30 bar
> > drwxrwxrwx  1 root root1 Jan 21 16:49 folder
>
> > # df -hT .
> > Filesystem Type  Size  Used Avail Use% 
> > Mounted on
> > 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph   28T 0   28T   0% 
> > /cephfs
>
> I try 2 client config :
>
> > $ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw
> > [snip]
> > $ ceph auth  get client.fsadmin
> > exported keyring for client.fsadmin
> > [client.fsadmin]
> >   key = [snip]
> >   caps mds = "allow rw"
> >   caps mon = "allow r"
> >   caps osd = "allow rw tag cephfs data=cephfs"

For "allow rw tag cephfs data=cephfs" to work, having your pools tagged
with cephfs is not enough.  There should be a key-value pair in the tag
metadata with the name of the filesystem as well.  What is the output
of:

  ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name
| startswith("cephfs")) | .pool_name, .application_metadata'

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS with cache-tier kernel-mount client unable to write (Nautilus)

[Hayashida, Mami]

Thanks, Ilya.

First, I was not sure whether to post my question on @ceph.io or @
lists.ceph.com (I subscribe to both) -- should I use @ceph.io in the future?

Second, thanks for your advice on cache-tiering -- I was starting to feel
that way but always good to know what Ceph "experts" would say.

Third, I tried enabling (and setting) the pool application commands you
outlined but got errors (Ceph is not allowing me to enable/set application
on the cache tier)

$ ceph osd pool application enable cephfs-data-cache cephfs
Error EINVAL: application must be enabled on base tier
 $ ceph osd pool application set cephfs-data-cache cephfs data cephfs_test
Error EINVAL: application metadata must be set on base tier

Since at this point, it is highly unlikely that we will be utilizing
cache-tier on our production clusters, and there is a work around it (by
manually creating a CephFS client key), this is nothing serious or urgent;
but I thought I should let you guys know.

Again, thanks for your help!

Mami



On Thu, Jan 23, 2020 at 8:40 AM Ilya Dryomov  wrote:

> On Thu, Jan 23, 2020 at 2:36 PM Ilya Dryomov  wrote:
> >
> > On Wed, Jan 22, 2020 at 6:18 PM Hayashida, Mami 
> wrote:
> > >
> > > Thanks, Ilya.
> > >
> > > I just tried modifying the osd cap for client.testuser by getting rid
> of "tag cephfs data=cephfs_test" part and confirmed this key does work
> (i.e. lets the CephFS client read/write).  It now reads:
> > >
> > > [client.testuser]
> > > key = XXXZZZ
> > > caps mds = "allow rw"
> > > caps mon = "allow r"
> > > caps osd = "allow rw"  // previously "allow rw tag cephfs
> data=cephfs_test"
> > >
> > > I tried removing either "tag cephfs" or "data=cephfs_test" (and
> leaving the other), but neither worked.
> > >
> > > Now, here is my question: will not having the "allow rw tag cephfs
> data=" (under osd caps) result in a security/privacy
> loophole in a production cluster?   (I am still trying to assess whether
> having a Cache Tier behind CephFS is worth all the headaches...)
> >
> > It's probably not worth it.  Unless you have a specific tiered
> > workload in mind and your cache pool is large enough for it, I'd
> > recommend staying away from cache tiering.
> >
> > "allow rw" for osd is only marginally more restrictive than
> > client.admin's "allow *", allowing the user to read/write every object
> > in the cluster.  Scratch my reply about doing it by hand -- try the
> > following:
> >
> >   $ ceph osd pool application enable cephfs-data-cache cephfs
> >   $ ceph osd pool application set cephfs-data-cache cephfs data
> cephfs_test
> >   $ ceph fs authorize cephfs_test ...  (as before)
> >
> > You will see the same "allow rw tag cephfs data=cephfs_test" cap in
> > "ceph auth list" output, but it should allow accessing cephfs-data-cache.
>
> Dropping ceph-us...@lists.ceph.com and resending to ceph-users@ceph.io.
>
> Thanks,
>
> Ilya
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS with cache-tier kernel-mount client unable to write (Nautilus)

[Ilya Dryomov]

On Thu, Jan 23, 2020 at 2:36 PM Ilya Dryomov  wrote:
>
> On Wed, Jan 22, 2020 at 6:18 PM Hayashida, Mami  
> wrote:
> >
> > Thanks, Ilya.
> >
> > I just tried modifying the osd cap for client.testuser by getting rid of 
> > "tag cephfs data=cephfs_test" part and confirmed this key does work (i.e. 
> > lets the CephFS client read/write).  It now reads:
> >
> > [client.testuser]
> > key = XXXZZZ
> > caps mds = "allow rw"
> > caps mon = "allow r"
> > caps osd = "allow rw"  // previously "allow rw tag cephfs data=cephfs_test"
> >
> > I tried removing either "tag cephfs" or "data=cephfs_test" (and leaving the 
> > other), but neither worked.
> >
> > Now, here is my question: will not having the "allow rw tag cephfs 
> > data=" (under osd caps) result in a security/privacy 
> > loophole in a production cluster?   (I am still trying to assess whether 
> > having a Cache Tier behind CephFS is worth all the headaches...)
>
> It's probably not worth it.  Unless you have a specific tiered
> workload in mind and your cache pool is large enough for it, I'd
> recommend staying away from cache tiering.
>
> "allow rw" for osd is only marginally more restrictive than
> client.admin's "allow *", allowing the user to read/write every object
> in the cluster.  Scratch my reply about doing it by hand -- try the
> following:
>
>   $ ceph osd pool application enable cephfs-data-cache cephfs
>   $ ceph osd pool application set cephfs-data-cache cephfs data cephfs_test
>   $ ceph fs authorize cephfs_test ...  (as before)
>
> You will see the same "allow rw tag cephfs data=cephfs_test" cap in
> "ceph auth list" output, but it should allow accessing cephfs-data-cache.

Dropping ceph-us...@lists.ceph.com and resending to ceph-users@ceph.io.

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Google Summer of Code 2020

[Alastair Dewhurst - UKRI STFC]

Hello

I apologies if I have missed an announcement, but I can’t find anything 
regarding if Ceph will participate in the GSoC this year?  The only thing I 
could find was last years announcement (see below) and webpage.  My 
organisation has some ideas for Ceph projects, however we could submit them 
under a different banner organisation if necessary.

Thanks

Alastair



On 17 Jan 2019, at 08:12, Mike Perez 
mailto:mipe...@redhat.com>> wrote:

Hello everyone,

We're getting ready for the next round of Google Summer of Code and Outreachy.

Ali Maredia and I will help organize the Ceph project for each program.

We are looking to have project ideas for Google Summer of Code by
February 4th as our project application deadline is due by February
6th.

https://ceph.com/contribute/gsoc2019/
https://summerofcode.withgoogle.com/how-it-works/

Outreachy applicants in mid-February through the end of March will
begin selecting open source projects to work on and make small
contributions. On April 12th we hope to have some projects set for
Outreachy so we can start selecting applicants. End of May will begin
the internships.

https://www.outreachy.org/mentor/

You can submit project ideas for both programs on this etherpad.
https://pad.ceph.com/p/project-ideas

Stay tuned for more updates.

--
Mike Perez (thingee)
___
ceph-users mailing list
ceph-us...@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs : write error: Operation not permitted

[Yoann Moulin]

Le 23.01.20 à 12:50, Frank Schilder a écrit :

You should probably enable the application "cephfs" on the fs-pools.


it's already activated :

pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 
lfor 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs



In both your cases, the osd caps should read

caps osd = "allow rw tag cephfs data=cephfs_metadata, allow rw pool=cephfs_data"


Currently, I have this and it works:

caps: [osd] allow class-read object_prefix rbd_children, allow rw 
pool=cephfs_data

I opened a bug on tracker : https://tracker.ceph.com/issues/43761


This is independent of the replication type of cephfs_data.


Yup, this is what I understood.

Yoann



From: Yoann Moulin 
Sent: 23 January 2020 10:38:42
To: Frank Schilder; ceph-users
Subject: Re: [ceph-users] Re: cephfs : write error: Operation not permitted

Hi Frank,


for some reason, the command "ceph fs authorize" does not add the required 
permissions for a FS with data pools any more, older versions did. Now you need to add 
these caps by hand. It needs to look something like this:

caps osd = "allow rw tag cephfs pool=cephfs_data, allow rw pool=cephfs-data"

an easy way is:

- ceph auth export
- add the caps with an editor
- ceph auth import

I consider this a bug and thought it was fixed in newer versions already.

  >

Sorry, I had a typo. If you have separate meta- and data pools, the data pool 
is not added properly. The caps should look like:

caps osd = "allow rw tag cephfs pool=cephfs-meta-pool, allow rw 
pool=cephfs-data-pool"

If you don't have a separate data pool, it should work out of the box.


Thanks, for the feed back, I also got that information from dirtwash on the IRC 
channel that fix my issue. And yes for me it's also an issue.

But I have another cluster also on ubuntu 18.04 (4.15.0-74-generic) and 
Nautilus 14.2.6 installed one week before and it works. I did the same
command on both and I don't have the same behaviour.

there are few diff between the 2 clusters, not the same hw config (No SSD on 
the dslab2020 cluster) and cephfs_data on an 8+3 EC pool on Artemis
(see the end of artemis.txt). in attachement, I put the result of the commands 
I did on both cluster without the same behaviors at the end.

Best,

Yoann



From: Yoann Moulin 
Sent: 22 January 2020 08:58:29
To: ceph-users
Subject: [ceph-users] cephfs : write error: Operation not permitted

Hello,

On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook 
stable-4.0, I have an issue with cephfs. I can create a folder, I can
create empty files, but cannot write data on like I'm not allowed to write to 
the cephfs_data pool.


$ ceph -s
cluster:
  id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09
  health: HEALTH_OK

services:
  mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h)
  mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042
  mds: cephfs:3 
{0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active}
  osd: 24 osds: 24 up (since 22h), 24 in (since 22h)
  rgw: 1 daemon active (iccluster043.rgw0)

data:
  pools:   9 pools, 568 pgs
  objects: 800 objects, 225 KiB
  usage:   24 GiB used, 87 TiB / 87 TiB avail
  pgs: 568 active+clean


The 2 cephfs pools:


$ ceph osd pool ls detail | grep cephfs
pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 
flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs
pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool 
stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 
recovery_priority 5 application cephfs


The status of the cephfs filesystem:


$ ceph fs status
cephfs - 1 clients
==
+--++--+---+---+---+
| Rank | State  | MDS  |Activity   |  dns  |  inos |
+--++--+---+---+---+
|  0   | active | iccluster043 | Reqs:0 /s |   34  |   18  |
|  1   | active | iccluster041 | Reqs:0 /s |   12  |   16  |
|  2   | active | iccluster042 | Reqs:0 /s |   10  |   13  |
+--++--+---+---+---+
+-+--+---+---+
|   Pool  |   type   |  used | avail |
+-+--+---+---+
| cephfs_metadata | metadata | 4608k | 27.6T |
|   cephfs_data   |   data   |0  | 27.6T |
+-+--+---+---+
+-+
| Standby MDS |
+-+
+-+
MDS version: ceph version 14.2.6 

[ceph-users] Re: cephfs kernel mount option uid?

[Jeff Layton]

On Mon, 2020-01-20 at 16:12 +0100, Marc Roos wrote:
> Is it possible to mount a cephfs with a specific uid or gid? To make it 
> available to a 'non-root' user?
> 

It's not 100% clear what you're asking for here. CephFS is a (mostly)
POSIX filesystem that has permissions and ownership, so non-root users
can already access a CephFS mount.

Do you mean that you want to be able to do some sort of uid/gid
"shifting" on the fs? If so, then we will probably not be implementing
that in cephfs, as this is an area of active work upstream at the
generic vfs layer. For example:

https://lwn.net/Articles/687354/
https://lwn.net/Articles/806176/

Currently, there is no real solution for this in stock kernels, but you
could look at shiftfs as a possible solution for now and we may
eventually get uid/gid shifting as part of the generic VFS via the new
mount API.
-- 
Jeff Layton 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Problem : "1 pools have many more objects per pg than average"

[St-Germain, Sylvain (SSC/SPC)]

/ Problem ///

 I've got a Warning on my cluster that I cannot remove :

"1 pools have many more objects per pg than average"

Does somebody has some insight ? I think it's normal to have this warning 
because I have just one pool in use, but how can I remove this warning ?

Thx !

/ INFORMATION ///

*** Here's some information about the cluster

# ceph health detail
HEALTH_WARN 1 pools have many more objects per pg than average 
MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
pool default.rgw.buckets.data objects per pg (50567) is more than 14.1249 
times cluster average (3580)

# sudo ceph-conf -D | grep mon_pg_warn_max_object_skew 
mon_pg_warn_max_object_skew = 10.00

# ceph daemon mon.dao-wkr-04 config show | grep mon_pg_warn_max_object_skew
"mon_pg_warn_max_object_skew": "10.00"

# ceph -v
ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)

# ceph df
RAW STORAGE:
CLASS   SIZEAVAIL   USED   RAW USED %RAW USED 
hdd 873 TiB 823 TiB 50 TiB   51 TiB  5.80 
TOTAL   873 TiB 823 TiB 50 TiB   51 TiB  5.80 
 
POOLS:
POOLID STORED  OBJECTS  
USED%USED MAX AVAIL 
.rgw.root   11 3.5 KiB  8   
1.5 MiB 0   249 TiB 
default.rgw.control 12 0 B  8   
0 B 0   249 TiB 
default.rgw.meta13  52 KiB  186 
34 MiB  0   249 TiB 
default.rgw.log 14 0 B  207 
0 B 0   249 TiB 
default.rgw.buckets.index   15 1.2 GiB  131 
1.2 GiB 0   249 TiB 
cephfs_data 29 915 MiB  202 
1.5 GiB 0   467 TiB 
cephfs_metadata 30 145 KiB  23  
2.1 MiB 0   249 TiB 
default.rgw.buckets.data31  30 TiB  12.95M  
50 TiB  6.32  467 TiB 


# ceph osd dump | grep default.rgw.buckets.data pool 31 
'default.rgw.buckets.data' erasure size 8 min_size 6 crush_rule 2 object_hash 
rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 9502 lfor 
0/2191/7577 flags hashpspool stripe_width 20480 target_size_ratio 0.4 
application rgw

/// SOLUTION TRIED ///

1- I try to increase the value of mon_pg_warn_max_object_skew parameter

#sudo ceph tell mon.* injectargs '--mon_pg_warn_max_object_skew 20'
#sudo ceph tell osd.* injectargs '--mon_pg_warn_max_object_skew 20'

+ And reboot the monitor

The parameter didn't change.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs : write error: Operation not permitted

[Frank Schilder]

You should probably enable the application "cephfs" on the fs-pools.

In both your cases, the osd caps should read

caps osd = "allow rw tag cephfs data=cephfs_metadata, allow rw pool=cephfs_data"

This is independent of the replication type of cephfs_data.

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Yoann Moulin 
Sent: 23 January 2020 10:38:42
To: Frank Schilder; ceph-users
Subject: Re: [ceph-users] Re: cephfs : write error: Operation not permitted

Hi Frank,

>> for some reason, the command "ceph fs authorize" does not add the required 
>> permissions for a FS with data pools any more, older versions did. Now you 
>> need to add these caps by hand. It needs to look something like this:
>>
>> caps osd = "allow rw tag cephfs pool=cephfs_data, allow rw pool=cephfs-data"
>>
>> an easy way is:
>>
>> - ceph auth export
>> - add the caps with an editor
>> - ceph auth import
>>
>> I consider this a bug and thought it was fixed in newer versions already.
 >
> Sorry, I had a typo. If you have separate meta- and data pools, the data pool 
> is not added properly. The caps should look like:
>
> caps osd = "allow rw tag cephfs pool=cephfs-meta-pool, allow rw 
> pool=cephfs-data-pool"
>
> If you don't have a separate data pool, it should work out of the box.

Thanks, for the feed back, I also got that information from dirtwash on the IRC 
channel that fix my issue. And yes for me it's also an issue.

But I have another cluster also on ubuntu 18.04 (4.15.0-74-generic) and 
Nautilus 14.2.6 installed one week before and it works. I did the same
command on both and I don't have the same behaviour.

there are few diff between the 2 clusters, not the same hw config (No SSD on 
the dslab2020 cluster) and cephfs_data on an 8+3 EC pool on Artemis
(see the end of artemis.txt). in attachement, I put the result of the commands 
I did on both cluster without the same behaviors at the end.

Best,

Yoann

> 
> From: Yoann Moulin 
> Sent: 22 January 2020 08:58:29
> To: ceph-users
> Subject: [ceph-users] cephfs : write error: Operation not permitted
>
> Hello,
>
> On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook 
> stable-4.0, I have an issue with cephfs. I can create a folder, I can
> create empty files, but cannot write data on like I'm not allowed to write to 
> the cephfs_data pool.
>
>> $ ceph -s
>>cluster:
>>  id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09
>>  health: HEALTH_OK
>>
>>services:
>>  mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h)
>>  mgr: iccluster039(active, since 21h), standbys: iccluster041, 
>> iccluster042
>>  mds: cephfs:3 
>> {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active}
>>  osd: 24 osds: 24 up (since 22h), 24 in (since 22h)
>>  rgw: 1 daemon active (iccluster043.rgw0)
>>
>>data:
>>  pools:   9 pools, 568 pgs
>>  objects: 800 objects, 225 KiB
>>  usage:   24 GiB used, 87 TiB / 87 TiB avail
>>  pgs: 568 active+clean
>
> The 2 cephfs pools:
>
>> $ ceph osd pool ls detail | grep cephfs
>> pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash 
>> rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 
>> 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application 
>> cephfs
>> pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 
>> object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 
>> flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 
>> pg_num_min 16 recovery_priority 5 application cephfs
>
> The status of the cephfs filesystem:
>
>> $ ceph fs status
>> cephfs - 1 clients
>> ==
>> +--++--+---+---+---+
>> | Rank | State  | MDS  |Activity   |  dns  |  inos |
>> +--++--+---+---+---+
>> |  0   | active | iccluster043 | Reqs:0 /s |   34  |   18  |
>> |  1   | active | iccluster041 | Reqs:0 /s |   12  |   16  |
>> |  2   | active | iccluster042 | Reqs:0 /s |   10  |   13  |
>> +--++--+---+---+---+
>> +-+--+---+---+
>> |   Pool  |   type   |  used | avail |
>> +-+--+---+---+
>> | cephfs_metadata | metadata | 4608k | 27.6T |
>> |   cephfs_data   |   data   |0  | 27.6T |
>> +-+--+---+---+
>> +-+
>> | Standby MDS |
>> +-+
>> +-+
>> MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) 
>> nautilus (stable)
>
>
>> # mkdir folder
>> # echo "foo" > bar
>> -bash: echo: write error: Operation not permitted
>> # ls -al
>> total 4
>> drwxrwxrwx  1 root root2 Jan 22 07:30 .
>> drwxr-xr-x 28 root root 4096 Jan 21 09:25 ..
>> 

[ceph-users] Re: cephfs : write error: Operation not permitted

[Yoann Moulin]

Hi Frank,


for some reason, the command "ceph fs authorize" does not add the required 
permissions for a FS with data pools any more, older versions did. Now you need to add 
these caps by hand. It needs to look something like this:

caps osd = "allow rw tag cephfs pool=cephfs_data, allow rw pool=cephfs-data"

an easy way is:

- ceph auth export
- add the caps with an editor
- ceph auth import

I consider this a bug and thought it was fixed in newer versions already.

>

Sorry, I had a typo. If you have separate meta- and data pools, the data pool 
is not added properly. The caps should look like:

caps osd = "allow rw tag cephfs pool=cephfs-meta-pool, allow rw 
pool=cephfs-data-pool"

If you don't have a separate data pool, it should work out of the box.


Thanks, for the feed back, I also got that information from dirtwash on the IRC 
channel that fix my issue. And yes for me it's also an issue.

But I have another cluster also on ubuntu 18.04 (4.15.0-74-generic) and Nautilus 14.2.6 installed one week before and it works. I did the same 
command on both and I don't have the same behaviour.


there are few diff between the 2 clusters, not the same hw config (No SSD on the dslab2020 cluster) and cephfs_data on an 8+3 EC pool on Artemis 
(see the end of artemis.txt). in attachement, I put the result of the commands I did on both cluster without the same behaviors at the end.


Best,

Yoann



From: Yoann Moulin 
Sent: 22 January 2020 08:58:29
To: ceph-users
Subject: [ceph-users] cephfs : write error: Operation not permitted

Hello,

On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook 
stable-4.0, I have an issue with cephfs. I can create a folder, I can
create empty files, but cannot write data on like I'm not allowed to write to 
the cephfs_data pool.


$ ceph -s
   cluster:
 id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09
 health: HEALTH_OK

   services:
 mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h)
 mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042
 mds: cephfs:3 
{0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active}
 osd: 24 osds: 24 up (since 22h), 24 in (since 22h)
 rgw: 1 daemon active (iccluster043.rgw0)

   data:
 pools:   9 pools, 568 pgs
 objects: 800 objects, 225 KiB
 usage:   24 GiB used, 87 TiB / 87 TiB avail
 pgs: 568 active+clean


The 2 cephfs pools:


$ ceph osd pool ls detail | grep cephfs
pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 0/0/81 
flags hashpspool stripe_width 0 expected_num_objects 1 application cephfs
pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 flags hashpspool 
stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 pg_num_min 16 
recovery_priority 5 application cephfs


The status of the cephfs filesystem:


$ ceph fs status
cephfs - 1 clients
==
+--++--+---+---+---+
| Rank | State  | MDS  |Activity   |  dns  |  inos |
+--++--+---+---+---+
|  0   | active | iccluster043 | Reqs:0 /s |   34  |   18  |
|  1   | active | iccluster041 | Reqs:0 /s |   12  |   16  |
|  2   | active | iccluster042 | Reqs:0 /s |   10  |   13  |
+--++--+---+---+---+
+-+--+---+---+
|   Pool  |   type   |  used | avail |
+-+--+---+---+
| cephfs_metadata | metadata | 4608k | 27.6T |
|   cephfs_data   |   data   |0  | 27.6T |
+-+--+---+---+
+-+
| Standby MDS |
+-+
+-+
MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) 
nautilus (stable)




# mkdir folder
# echo "foo" > bar
-bash: echo: write error: Operation not permitted
# ls -al
total 4
drwxrwxrwx  1 root root2 Jan 22 07:30 .
drwxr-xr-x 28 root root 4096 Jan 21 09:25 ..
-rw-r--r--  1 root root0 Jan 22 07:30 bar
drwxrwxrwx  1 root root1 Jan 21 16:49 folder



# df -hT .
Filesystem Type  Size  Used Avail Use% 
Mounted on
10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph   28T 0   28T   0% 
/cephfs


I try 2 client config :


$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw
[snip]
$ ceph auth  get client.fsadmin
exported keyring for client.fsadmin
[client.fsadmin]
   key = [snip]
   caps mds = "allow rw"
   caps mon = "allow r"
   caps osd = "allow rw tag cephfs data=cephfs"



$ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw
[snip]
$ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw tag 
cephfs pool=cephfs_data "

[ceph-users] Re: ceph-volume lvm filestore OSDs fail to start on reboot. Permission denied on journal partition

[Jan Fajerski]

On Wed, Jan 22, 2020 at 12:00:28PM -0500, Wesley Dillingham wrote:
>   After upgrading to Nautilus 14.2.6 from Luminous 12.2.12 we are seeing
>   the following behavior on OSDs which were created with "ceph-volume lvm
>   create --filestore --osd-id  --data  --journal "
>   Upon restart of the server containing these OSDs they fail to start
>   with the following error in the logs:
>2020-01-21 13:36:11.635 7fee633e8a80 -1 filestore(/var/lib/ceph/osd/ceph-199) 
>mo
>unt(1928): failed to open journal /var/lib/ceph/osd/ceph-199/journal: (13) 
>Permi
>ssion denied
>
>   /var/lib/ceph/osd/ceph-199/journal symlinks to /dev/sdc5 in our case
>   and inspecting the ownership on /dev/sdc5 it is root:root, chowning
>   that to ceph:ceph causes the osd to start and come back up and in near
>   instantly.
>   As a note these OSDs we experience this with are OSDs which have
>   previously failed and been replaced using the above ceph-volume, longer
>   running OSDs in the same server created with ceph-disk or ceph-volume
>   simple (that have a corresponding .json in /etc/ceph/osd) start up fine
>   and get ceph:ceph on their journal partition. Bluestore OSDs also do
>   not have any issue.
>   My hope is that I can preemptively fix these OSDs before shutting them
>   down so that reboots happen seamlessly. Thanks for any insight.
ceph-volume is supposed to take care of this via the ceph-volume@ systemd unit. 
 
This is a one shot unit, that should set things up and then start the osd.
The unit name is a bit convoluted: ceph-volume@-, there 
should 
be symbolic link in /etc/systemd/system/multi-user.target.wants/

You can also check cat /var/log/ceph/ceph-volume-systemd.log for any errors.
Feel free to open a tracker ticket on 
https://tracker.ceph.com/projects/ceph-volume

>
>   Respectfully,
>   Wes Dillingham
>   [1]w...@wesdillingham.com
>   [2]LinkedIn
>
>References
>
>   1. mailto:w...@wesdillingham.com
>   2. http://www.linkedin.com/in/wesleydillingham

>___
>ceph-users mailing list -- ceph-users@ceph.io
>To unsubscribe send an email to ceph-users-le...@ceph.io


-- 
Jan Fajerski
Senior Software Engineer Enterprise Storage
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph at DevConf and FOSDEM

[Mike Perez]

Hi everyone,

I'm still looking for people to help at DevConf as our booth signup
schedule is looking still bare.

Also great news, we will have a BoF session at FOSDEM:

https://fosdem.org/2020/schedule/event/bof_opensource_storage/

We will also be sharing a booth with the CentOS project, but space will be
tight. If you are interested in helping at the Ceph booth for FOSDEM please
contact me directly and I'll provide the signup information. Thanks
everyone!

On Tue, Jan 21, 2020 at 2:54 AM Mike Perez  wrote:

> Hi Cephers,
>
> We will have a booth at DevConf this year!
>
> https://www.devconf.info/cz/
>
> If you're interested in helping at the booth at DevConf, please reply to
> me directly, and I can provide more details.
>
> I'm still working on getting a shared booth for FOSDEM with the CentOS
> table. Also, I will try to get us signed with a BoF session. If you're
> interested, please also let me know directly. Thanks!
>
> https://fosdem.org/2020/
>
> --
>
> Mike Perez
>
> he/him
>
> Ceph Community Manager
>
>
> M: +1-951-572-2633
>
> 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA
> @Thingee   Thingee
>  
> 
>


-- 

Mike Perez

he/him

Ceph Community Manager


M: +1-951-572-2633

494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA
@Thingee   Thingee
 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs : write error: Operation not permitted

[Frank Schilder]

Sorry, I had a typo. If you have separate meta- and data pools, the data pool 
is not added properly. The caps should look like:

caps osd = "allow rw tag cephfs pool=cephfs-meta-pool, allow rw 
pool=cephfs-data-pool"

If you don't have a separate data pool, it should work out of the box.

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Frank Schilder 
Sent: 23 January 2020 09:10:41
To: Yoann Moulin; ceph-users
Subject: [ceph-users] Re: cephfs : write error: Operation not permitted

Hi Yoann,

for some reason, the command "ceph fs authorize" does not add the required 
permissions for a FS with data pools any more, older versions did. Now you need 
to add these caps by hand. It needs to look something like this:

caps osd = "allow rw tag cephfs pool=cephfs_data, allow rw pool=cephfs-data"

an easy way is:

- ceph auth export
- add the caps with an editor
- ceph auth import

I consider this a bug and thought it was fixed in newer versions already.

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Yoann Moulin 
Sent: 22 January 2020 08:58:29
To: ceph-users
Subject: [ceph-users] cephfs : write error: Operation not permitted

Hello,

On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook 
stable-4.0, I have an issue with cephfs. I can create a folder, I can
create empty files, but cannot write data on like I'm not allowed to write to 
the cephfs_data pool.

> $ ceph -s
>   cluster:
> id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09
> health: HEALTH_OK
>
>   services:
> mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h)
> mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042
> mds: cephfs:3 
> {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active}
> osd: 24 osds: 24 up (since 22h), 24 in (since 22h)
> rgw: 1 daemon active (iccluster043.rgw0)
>
>   data:
> pools:   9 pools, 568 pgs
> objects: 800 objects, 225 KiB
> usage:   24 GiB used, 87 TiB / 87 TiB avail
> pgs: 568 active+clean

The 2 cephfs pools:

> $ ceph osd pool ls detail | grep cephfs
> pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash 
> rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 
> 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application 
> cephfs
> pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 
> object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 
> flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 
> pg_num_min 16 recovery_priority 5 application cephfs

The status of the cephfs filesystem:

> $ ceph fs status
> cephfs - 1 clients
> ==
> +--++--+---+---+---+
> | Rank | State  | MDS  |Activity   |  dns  |  inos |
> +--++--+---+---+---+
> |  0   | active | iccluster043 | Reqs:0 /s |   34  |   18  |
> |  1   | active | iccluster041 | Reqs:0 /s |   12  |   16  |
> |  2   | active | iccluster042 | Reqs:0 /s |   10  |   13  |
> +--++--+---+---+---+
> +-+--+---+---+
> |   Pool  |   type   |  used | avail |
> +-+--+---+---+
> | cephfs_metadata | metadata | 4608k | 27.6T |
> |   cephfs_data   |   data   |0  | 27.6T |
> +-+--+---+---+
> +-+
> | Standby MDS |
> +-+
> +-+
> MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) 
> nautilus (stable)


> # mkdir folder
> # echo "foo" > bar
> -bash: echo: write error: Operation not permitted
> # ls -al
> total 4
> drwxrwxrwx  1 root root2 Jan 22 07:30 .
> drwxr-xr-x 28 root root 4096 Jan 21 09:25 ..
> -rw-r--r--  1 root root0 Jan 22 07:30 bar
> drwxrwxrwx  1 root root1 Jan 21 16:49 folder

> # df -hT .
> Filesystem Type  Size  Used Avail Use% 
> Mounted on
> 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph   28T 0   28T   0% 
> /cephfs

I try 2 client config :

> $ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw
> [snip]
> $ ceph auth  get client.fsadmin
> exported keyring for client.fsadmin
> [client.fsadmin]
>   key = [snip]
>   caps mds = "allow rw"
>   caps mon = "allow r"
>   caps osd = "allow rw tag cephfs data=cephfs"

> $ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw
> [snip]
> $ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow 
> rw tag cephfs pool=cephfs_data "
> [snip]
> ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw 
> tag cephfs pool=cephfs_data "> updated caps for client.cephfsadmin
> $ ceph auth  get 

[ceph-users] Re: cephfs : write error: Operation not permitted

[Frank Schilder]

Hi Yoann,

for some reason, the command "ceph fs authorize" does not add the required 
permissions for a FS with data pools any more, older versions did. Now you need 
to add these caps by hand. It needs to look something like this:

caps osd = "allow rw tag cephfs pool=cephfs_data, allow rw pool=cephfs-data"

an easy way is:

- ceph auth export
- add the caps with an editor
- ceph auth import

I consider this a bug and thought it was fixed in newer versions already.

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Yoann Moulin 
Sent: 22 January 2020 08:58:29
To: ceph-users
Subject: [ceph-users] cephfs : write error: Operation not permitted

Hello,

On a fresh install (Nautilus 14.2.6) deploy with ceph-ansible playbook 
stable-4.0, I have an issue with cephfs. I can create a folder, I can
create empty files, but cannot write data on like I'm not allowed to write to 
the cephfs_data pool.

> $ ceph -s
>   cluster:
> id: fded5bb5-62c5-4a88-b62c-0986d7c7ac09
> health: HEALTH_OK
>
>   services:
> mon: 3 daemons, quorum iccluster039,iccluster041,iccluster042 (age 23h)
> mgr: iccluster039(active, since 21h), standbys: iccluster041, iccluster042
> mds: cephfs:3 
> {0=iccluster043=up:active,1=iccluster041=up:active,2=iccluster042=up:active}
> osd: 24 osds: 24 up (since 22h), 24 in (since 22h)
> rgw: 1 daemon active (iccluster043.rgw0)
>
>   data:
> pools:   9 pools, 568 pgs
> objects: 800 objects, 225 KiB
> usage:   24 GiB used, 87 TiB / 87 TiB avail
> pgs: 568 active+clean

The 2 cephfs pools:

> $ ceph osd pool ls detail | grep cephfs
> pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash 
> rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 83 lfor 
> 0/0/81 flags hashpspool stripe_width 0 expected_num_objects 1 application 
> cephfs
> pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 
> object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 48 
> flags hashpspool stripe_width 0 expected_num_objects 1 pg_autoscale_bias 4 
> pg_num_min 16 recovery_priority 5 application cephfs

The status of the cephfs filesystem:

> $ ceph fs status
> cephfs - 1 clients
> ==
> +--++--+---+---+---+
> | Rank | State  | MDS  |Activity   |  dns  |  inos |
> +--++--+---+---+---+
> |  0   | active | iccluster043 | Reqs:0 /s |   34  |   18  |
> |  1   | active | iccluster041 | Reqs:0 /s |   12  |   16  |
> |  2   | active | iccluster042 | Reqs:0 /s |   10  |   13  |
> +--++--+---+---+---+
> +-+--+---+---+
> |   Pool  |   type   |  used | avail |
> +-+--+---+---+
> | cephfs_metadata | metadata | 4608k | 27.6T |
> |   cephfs_data   |   data   |0  | 27.6T |
> +-+--+---+---+
> +-+
> | Standby MDS |
> +-+
> +-+
> MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) 
> nautilus (stable)


> # mkdir folder
> # echo "foo" > bar
> -bash: echo: write error: Operation not permitted
> # ls -al
> total 4
> drwxrwxrwx  1 root root2 Jan 22 07:30 .
> drwxr-xr-x 28 root root 4096 Jan 21 09:25 ..
> -rw-r--r--  1 root root0 Jan 22 07:30 bar
> drwxrwxrwx  1 root root1 Jan 21 16:49 folder

> # df -hT .
> Filesystem Type  Size  Used Avail Use% 
> Mounted on
> 10.90.38.15,10.90.38.17,10.90.38.18:/dslab2020 ceph   28T 0   28T   0% 
> /cephfs

I try 2 client config :

> $ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw
> [snip]
> $ ceph auth  get client.fsadmin
> exported keyring for client.fsadmin
> [client.fsadmin]
>   key = [snip]
>   caps mds = "allow rw"
>   caps mon = "allow r"
>   caps osd = "allow rw tag cephfs data=cephfs"

> $ ceph --cluster dslab2020 fs authorize cephfs client.cephfsadmin / rw
> [snip]
> $ ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow 
> rw tag cephfs pool=cephfs_data "
> [snip]
> ceph auth caps client.cephfsadmin mds "allow rw" mon "allow r" osd "allow rw 
> tag cephfs pool=cephfs_data "> updated caps for client.cephfsadmin
> $ ceph auth  get client.cephfsadmin
> exported keyring for client.cephfsadmin
> [client.cephfsadmin]
>   key = [snip]
>   caps mds = "allow rw"
>   caps mon = "allow r"
>   caps osd = "allow rw tag cephfs pool=cephfs_data "

I don't where to look to get more information about that issue. Anyone can help 
me? Thanks

Best regards,

--
Yoann Moulin
EPFL IC-IT
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___