[ceph-users] Reusing journal partitions when using ceph-deploy/ceph-disk --dmcrypt

[Alex Gorbachev]

Referencing
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-July/003293.html

When using --dmcrypt with ceph-deploy/ceph-disk, the journal device is not
allowed to be an existing partition. You have to specify the entire block
device, on which the tools create a partition equal to osd journal size
setting.

However, in the case when an HDD fails and OSD is deleted and then replaced
with another HDD, I have not been able to find a way to reuse the earlier
journal partition. Ceph-deploy creates a new one, which can lead into
unpleasant situations on the SSD used for journaling.

Is there a way anyone know of, to continue to use a specific partition as
journal with ceph-deploy?

Thanks in advance,
Alex
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cephfs quotas reporting

[Goncalo Borges]

Hi Again...

Once more, my environment:

   - ceph/cephfs in 10.2.2.
   - All infrastructure is in the same version (rados cluster, mons,
   mds and cephfs clients).
   - We mount cephfs using ceph-fuse.

I want to set up quotas to limit users from filling the filesystem and 
proactively avoid a situation where I have several simultaneous full or 
near full osds. However, I am not understanding how the reporting of 
space works once quotas are in place. My cephfs cluster provides a 
~100TB of space (~ 300 TB of raw space since I have a replication of 
3x). Check the following two cases:


   1./ In clients where the full filesystem hierarchy is available:

   - I have the following quota:

   # getfattr -n ceph.quota.max_bytes /coepp/cephfs
   getfattr: Removing leading '/' from absolute path names
   # file: coepp/cephfs
   ceph.quota.max_bytes="88"

   - I am mounting the client as

   # ceph-fuse --id mount_user -k
   /etc/ceph/ceph.client.mount_user.keyring -m :6789
   *--client-quota* --fuse_default_permissions=0
   --client_acl_type=posix_acl -r /cephfs /coepp/cephfs/

   - The results of two consecutive 'df' commands, executed after
   the mount operation, is the following. You can conclude that in
   the first command, the reported values are computed with respect
   to the quota but then fallback to the default when no quotas is
   in place.

   #  puppet agent -t; df -h ; df -h
   (...)
   ceph-fuse  81T   51T   30T  64% /coepp/cephfs
   (...)
   ceph-fuse 306T  153T  154T  50% /coepp/cephfs


2./ On another type of clients where I only mount a subdirectory of the 
filesystem (/coepp/cephfs/borg instead of coepp/cephfs):


   - I have the following quota:

   # getfattr -n ceph.quota.max_bytes /coepp/cephfs/borg
   getfattr: Removing leading '/' from absolute path name
   # file: coepp/cephfs/borg
   ceph.quota.max_bytes="10"

   - I mount the filesystem as:

   # ceph-fuse --id mount_user -k
   /etc/ceph/ceph.client.mount_user.keyring -m :6789
   --client-quota --fuse_default_permissions=0
   --client_acl_type=posix_acl -r /cephfs/borg /coepp/cephfs/borg


   - The reported space is

   # puppet agent -t; df -h ; df -h
   (...)
   ceph-fuse   9.1T  5.7T  3.5T  62% /coepp/cephfs/borg
   (...)
   ceph-fuse81T   51T   30T  64% /coepp/cephfs/borg


3./ Both clients are behaving in the same way where they start by 
reporting according to the implemented quota


51T of used space with respect to 81TB in total (case 1)
5.7T of used space with respect to 9.1T in total (case 2)

and then fallback to the values enforced in the previous level of the 
hierarchy.


153T of used space with respect to 306T in total (case 1)
51T of used space with respect to 81TB in total (case 2)

Am i doing something wrong here?

Cheers
Goncalo

--
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW  2006
T: +61 2 93511937

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw: how to prevent rgw user from creating a new bucket?

[Yang Joseph]

Thank you very much for your response.

I‘m confused about what this cap related to?

On 12/03/2016 12:13 AM, Yehuda Sadeh-Weinraub wrote:

On Fri, Dec 2, 2016 at 3:18 AM, Yang Joseph  wrote:

Hello,

I would like only to allow the user to read the object in a already existed
bucket, and not allow users
to create new bucket. It supposed to execute the following command:

$ radosgw-admin metadata put user:test3 < ...
   ...
 "caps": [
 {
 "type": "buckets",
 "perm": "read"
 }

But why user test3 can still create new bucket after I have set its caps to
"buckets=read"?



Because this cap is unrelated. iirc starting at jewel you can do:

$ radosgw-admin user modify --uid=test3 --max-buckets=-1

Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-fuse clients taking too long to update dir sizes

[Goncalo Borges]

Dear CephFSers.

We are running ceph/cephfs in 10.2.2. All infrastructure is in the same 
version (rados cluster, mons, mds and cephfs clients). We mount cephfs 
using ceph-fuse.


Last week I triggered some of my heavy users to delete data. In the 
following example, the user in question decreased his usage from ~4.5TB 
to ~ 600TB. However, some clients still did not update the usage 
(although several days have passed by) while others are ok.


From a point of view of the MDS, both types of client have healthy 
sessions. See detailed info after this email.


Trying to kick the session does not solve the issue. Probably only a 
remount but users are heavily using the filesystem and I do not want to 
break things for them now.



The only difference I can actually dig out between "good"/"bad" clients 
is that the user continues with active bash sessions in the "bad" client 
(from where he triggered the deletions)


   # lsof | grep user1 | grep ceph
   bash  15737   user1  cwd   DIR   0,24
   5285584388909 1099514070586 /coepp/cephfs/mel/user1
   vim   19233   user1  cwd   DIR   0,24 24521126
   1099514340633 /coepp/cephfs/mel/user1/Analysis/ssdilep/scripts
   vim   19233   user15u  REG 0,24 16384
   1099557935412
   /coepp/cephfs/mel/user1/Analysis/ssdilep/scripts/.histmgr.py.swp
   bash  24187   user1  cwd   DIR   0,24 826758558
   1099514314315 /coepp/cephfs/mel/user1/Analysis
   bash  24256   user1  cwd   DIR   0,24 147600
   1099514340621 /coepp/cephfs/mel/user1/Analysis/ssdilep/run
   bash  24327   user1  cwd   DIR   0,24 151068
   1099514340590 /coepp/cephfs/mel/user1/Analysis/ssdilep/algs
   bash  24394   user1  cwd   DIR   0,24 151068
   1099514340590 /coepp/cephfs/mel/user1/Analysis/ssdilep/algs
   bash  24461   user1  cwd   DIR   0,24 356436
   1099514340614 /coepp/cephfs/mel/user1/Analysis/ssdilep/samples
   bash  24528   user1  cwd   DIR   0,24 24521126
   1099514340633 /coepp/cephfs/mel/user1/Analysis/ssdilep/scripts
   bash  24601   user1  cwd   DIR   0,24 24521126
   1099514340633 /coepp/cephfs/mel/user1/Analysis/ssdilep/scripts

Is there a particular way to force the client to update these info? Do 
we actually know what it is taking so so long to update it?


Cheers

Goncalo

--- * ---


1) Reports from a client which shows "obsolete" file/directory sizes:

   # ll -h /coepp/cephfs/mel/ | grep user1
   *drwxr-xr-x 1 user1  coepp_mel 4.9T Oct  7 00:20 user1*

   # getfattr -d -m ceph /coepp/cephfs/mel/user1
   getfattr: Removing leading '/' from absolute path names
   # file: coepp/cephfs/mel/user1
   ceph.dir.entries="10"
   ceph.dir.files="1"
   *ceph.dir.rbytes="5285584388909"*
   ceph.dir.rctime="1480390891.09882864298"
   ceph.dir.rentries="161047"
   ceph.dir.rfiles="149669"
   ceph.dir.rsubdirs="11378"
   ceph.dir.subdirs="9"

   ---> Running following command in the client:
   # ceph daemon /var/run/ceph/ceph-client.mount_user.asok mds_sessions
   {
"id": 616794,
"sessions": [
{
"mds": 0,
"addr": ":6800\/1457",
"seq": 4884237,
"cap_gen": 0,
"cap_ttl": "2016-12-04 22:45:53.046697",
"last_cap_renew_request": "2016-12-04 22:44:53.046697",
"cap_renew_seq": 166765,
"num_caps": 1567318,
"state": "open"
}
],
"mdsmap_epoch": 5224
   }

   ---> Running the following command in the mds:
   # ceph daemon mds.rccephmds session ls
   (...)

   {
"id": 616794,
"num_leases": 0,
"num_caps": 21224,
"state": "open",
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.616794 :0\/68088301",
"client_metadata": {
"ceph_sha1": "45107e21c568dd033c2f0a3107dec8f0b0e58374",
"ceph_version": "ceph version 10.2.2
   (45107e21c568dd033c2f0a3107dec8f0b0e58374)",
"entity_id": "mount_user",
"hostname": "badclient.my.domain",
"mount_point": "\/coepp\/cephfs",
"root": "\/cephfs"
}
},

2) Reports from a client which shows "good" file/directory sizes:

   # ll -h /coepp/cephfs/mel/ | grep user1
   drwxr-xr-x 1 user1  coepp_mel 576G Oct  7 00:20 user1

   # getfattr -d -m ceph /coepp/cephfs/mel/user1
   getfattr: Removing leading '/' from absolute path names
   # file: coepp/cephfs/mel/user1
   ceph.dir.entries="10"
   ceph.dir.files="1"
   *ceph.dir.rbytes="617756983774"*
   ceph.dir.rctime="1480844101.09560671770"
   ceph.dir.rentries="96519"
   ceph.dir.rfiles="95091"
   ceph.dir.rsubdirs="1428"
   ceph.dir.subdirs="9"

   ---> Running following command in 

[ceph-users] First time deploying ceph on Amazon EC2

[Oleg Kolosov]

Hi
I'm first time deploying ceph. I've tried various manuals and read the ceph
documentation, but I keep failing on the same issue during create-initial.
For example, I've used this manual:
https://github.com/infn-bari-school/cloud-storage-tutorials/wiki/Ceph-cluster-installation-(jewel-on-Ubuntu-xenial)

I've tried running with Jewel release.

Each time I've reached the create-initial command, the following error
occurs:


[mon][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25
--cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon/keyring
auth get client.admin
[mon][ERROR ] "ceph auth get-or-create for keytype admin returned 1
[mon][DEBUG ] Traceback (most recent call last):
[mon][DEBUG ]   File "/usr/bin/ceph", line 948, in 
[mon][DEBUG ] retval = main()
[mon][DEBUG ]   File "/usr/bin/ceph", line 852, in main
[mon][DEBUG ] prefix='get_command_descriptions')
[mon][DEBUG ]   File "/usr/lib/python2.7/dist-packages/ceph_argparse.py",
line 1300, in json_command
[mon][DEBUG ] raise RuntimeError('"{0}": exception {1}'.format(argdict,
e))
[mon][DEBUG ] RuntimeError: "None": exception "['{"prefix":
"get_command_descriptions"}']": exception You cannot perform that operation
on a Rados object in state configuring.
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit
msg = self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 734, in format
return fmt.format(record)
  File "/usr/local/lib/python2.7/dist-packages/ceph_deploy/util/log.py",
line 56, in format
return logging.Formatter.format(self, record)
  File "/usr/lib/python2.7/logging/__init__.py", line 465, in format
record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/__init__.py", line 329, in getMessage
msg = msg % self.args


I suspect it's relate to the --name mon.   -   if it's the issue, how to
fix it?
[mon][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25
--cluster=ceph *--name mon.* --keyring=/var/lib/ceph/mon/ceph-mon/keyring
auth get client.admin


I'd really appreciate your help.

Thanks,
Oleg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph cluster having blocke requests very frequently

[Thomas Danan]

Hi Nick,

We have recently increase osd op threads from 2 (default value) to 16 because 
CPU usage on DN was very low.
We have the impression it has increased overall ceph cluster performances and 
reduced block ops occurrences.

I don’t think this is the end of our issue but it seems it helped to limit its 
impact.

Thomas

From: Nick Fisk [mailto:n...@fisk.me.uk]
Sent: mercredi 23 novembre 2016 14:09
To: Thomas Danan; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently

Hi Thomas,

I’m afraid I can’t offer anymore advice, they isn’t anything that I can see 
which could be the trigger. I know we spoke about downgrading the kernel, did 
you manage to try that?

Nick

From: Thomas Danan [mailto:thomas.da...@mycom-osi.com]
Sent: 23 November 2016 11:29
To: n...@fisk.me.uk; 'Peter Maloney' 
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently

Hi all,

Still not able to find any explanation to this issue.

I recently tested the network and I am seeing some retransmit being done in the 
output of iperf. But overall the bandwidth durin a test of 10sec is around 7 to 
8 Gbps.
I was not sure to understand if it was the test itself who was overloading the 
network or if my network switches that were having an issue.
Switches have been checked and they are showing no congestion issues or other 
errors.

I really don’t know what to check or test, any idea is more than welcomed …

Thomas

From: Thomas Danan
Sent: vendredi 18 novembre 2016 17:12
To: 'n...@fisk.me.uk'; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently

Hi Nick,

Here are some logs. The system is in IST TZ and I have filtered the logs to get 
only 2 last hours during which we can observe the issue.

In that particular case, issue is illustrated with the following OSDs

Primary:
ID:607
PID:2962227
HOST:10.137.81.18

Secondary1
ID:528
PID:3721728
HOST:10.137.78.194

Secondary2
ID:771
PID:2806795
HOST:10.137.81.25

In that specific example, first slow request message is detected at 16:18

2016-11-18 16:18:51.991185 7f13acd8a700  0 log_channel(cluster) log [WRN] : 7 
slow requests, 7 included below; oldest blocked for > 30.521107 secs
2016-11-18 16:18:51.991213 7f13acd8a700  0 log_channel(cluster) log [WRN] : 
slow request 30.521107 seconds old, received at 2016-11-18 16:18:21.469965: 
osd_op(client.2406870.1:140440919 rbd_data.616bf2ae8944a.002b85a7 
[set-alloc-hint object_size 4194304 write_size 4194304,write 1449984~524288] 
0.4e69d0de snapc 218=[218,1fb,1df] ondisk+write e212564) currently waiting for 
subops from 528,771

I see that it is about replicating a 4MB Object with snapc context but in my 
environment I have no snapshot (actually they were all deleted). Also I was 
said those messages were not necessary related to object replication to 
snapshot image.
Each time I have a slow request message it is formatted as described with 4MB 
Object and snapc context

Rados df is showing me that I have 4 cloned objects, I do not understand why.

15 minutes later seems ops are unblocked after initiating reconnect message

2016-11-18 16:34:38.120918 7f13acd8a700  0 log_channel(cluster) log [WRN] : 
slow request 960.264008 seconds old, received at 2016-11-18 16:18:37.856850: 
osd_op(client.2406634.1:104826541 rbd_data.636fe2ae8944a.00111eec 
[set-alloc-hint object_size 4194304 write_size 4194304,write 4112384~4096] 
0.f56e90de snapc 426=[426,3f9] ondisk+write e212564) currently waiting for 
subops from 528,771
2016-11-18 16:34:46.863383 7f137705a700  0 -- 10.137.81.18:6840/2962227 >> 
10.137.81.135:0/748393319 pipe(0x293bd000 sd=35 :6840 s=0 pgs=0 cs=0 l=0 
c=0x21405020).accept peer addr is really 10.137.81.135:0/748393319 (socket is 
10.137.81.135:26749/0)
2016-11-18 16:35:05.048420 7f138fea6700  0 -- 192.168.228.36:6841/2962227 >> 
192.168.228.28:6805/3721728 pipe(0x1271b000 sd=34 :50711 s=2 pgs=647 cs=5 l=0 
c=0x42798c0).fault, initiating reconnect

I do not manage to identify anything obvious in the logs.

Thanks for your help …

Thomas


From: Nick Fisk [mailto:n...@fisk.me.uk]
Sent: jeudi 17 novembre 2016 11:02
To: Thomas Danan; n...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently

Hi Thomas,

Do you have the OSD logs from around the time of that slow request (13:12 to 
13:29 period)?

Do you also see anything about OSD’s going down in the Mon ceph.log file around 
that time?

480 seconds is probably far too long for a disk to be busy for, I’m wondering 
if the OSD is either dying and respawning or if you are running out of some 
type of system resource….eg TCP connections or something like that, which means 
the OSD’s can’t communicate with each other.

From: