Re: [ceph-users] Samba vfs_ceph or kernel client

[David Disseldorp]

Hi Maged,

On Fri, 10 May 2019 18:32:15 +0200, Maged Mokhtar wrote:

> What is the recommended way for Samba gateway integration: using 
> vfs_ceph or mounting CephFS via kernel client ? i tested the kernel 
> solution in a ctdb setup and gave good performance, does it have any 
> limitations relative to vfs_ceph ?

At this stage kernel-backed and vfs_ceph-backed shares are pretty
similar feature wise. ATM kernel backed shares have the performance
advantage of page-cache + async vfs_default dispatch. vfs_ceph will
likely gain more features in future as cross-protocol share-mode locks
and leases can be supported without the requirement for a kernel
interface.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Snapshots in Mimic

[David Disseldorp]

Hi Kenneth,

On Tue, 31 Jul 2018 16:44:36 +0200, Kenneth Waegeman wrote:

> Hi all,
> 
> I updated an existing Luminous cluster to Mimic 13.2.1. All daemons were 
> updated, so I did ceph osd require-osd-release mimic, so everything 
> seems up to date.
> 
> I want to try the snapshots in Mimic, since this should be stable, so i ran:
> 
> [root@osd2801 alleee]# ceph fs set cephfs allow_new_snaps true
> enabled new snapshots
> 
> Now, when I try to create a snapshot, it is not working:
> 
> [root@osd2801 ~]# mkdir /mnt/bla/alleee/aaas
> [root@osd2801 ~]# mkdir /mnt/bla/alleee/aaas/.snap
> mkdir: cannot create directory ‘/mnt/bla/alleee/aaas/.snap’: File exists
> 
> I tried this using ceph-fuse and the kernel client, but always get the 
> same response.
> 
> Should I enable something else to get snapshots working ?

It looks as though you're just missing the snapshot name in the mkdir
request:
rapido1:/# cd /mnt/cephfs
rapido1:/mnt/cephfs# echo "before snap" >> data
rapido1:/mnt/cephfs# mkdir -p .snap/mysnapshot
rapido1:/mnt/cephfs# echo "after snap" >> data
rapido1:/mnt/cephfs# cat .snap/mysnapshot/data
before snap
rapido1:/mnt/cephfs# cat data
before snap
after snap

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] samba gateway experiences with cephfs ?

[David Disseldorp]

On Thu, 24 May 2018 15:13:09 +0200, Daniel Baumann wrote:

> On 05/24/2018 02:53 PM, David Disseldorp wrote:
> >> [ceph_test]
> >> path = /ceph-kernel
> >> guest ok = no
> >> delete readonly = yes
> >> oplocks = yes
> >> posix locking = no  
> 
> jftr, we use the following to disable all locking (on samba 4.8.2):
> 
>   oplocks = False
>   level2 oplocks = False
>   kernel oplocks = no

oplocks aren't locks per se - they allow the client to cache data
locally (leases in SMB2+), often allowing for improved application
performance. That said, if the same share path is accessible via NFS or
native CephFS then oplocks / leases should be disabled, until proper
vfs_ceph lease support is implemented via the delegation API.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] samba gateway experiences with cephfs ?

[David Disseldorp]

Hi Jake,

On Thu, 24 May 2018 13:17:16 +0100, Jake Grimmett wrote:

> Hi Daniel, David,
> 
> Many thanks for both of your advice.
> 
> Sorry not to reply to the list, but I'm subscribed to the digest and my
> mail client will not reply to individual threads - I've switched back to
> regular.

No worries, cc'ing the list in this response.

> As to this issue, I've turned off posix locking, which has improved
> write speeds - here are the old benchmarks plus new figures.
> 
> i.e. Using Helios LanTest 6.0.0 on Osx.
> 
> Create 300 Files
>  Cephfs (kernel) > samba (no Posix locks)
>average  3600 ms
>  Cephfs (kernel) > samba. average 5100 ms
>  Isilon   > CIFS  average 2600 ms
>  ZFS > samba   average  121 ms
> 
> Remove 300 files
>  Cephfs (kernel) > samba (no Posix locks)
>average  2200 ms
>  Cephfs (kernel) > samba. average 2100 ms
>  Isilon   > CIFS  average  900 ms
>  ZFS > samba   average  421 ms
> 
> Write 300MB to file
>  Cephfs (kernel) > samba (no Posix locks)
>average  53 MB/s
>  Cephfs (kernel) > samba. average 25 MB/s
>  Isilon   > CIFS  average  17.9 MB/s
>  ZFS > samba   average  64.4 MB/s
> 
> 
> Settings as follows:
> [global]
> (snip)
> smb2 leases = yes
> 
> 
> [ceph_test]
> path = /ceph-kernel
> guest ok = no
> delete readonly = yes
> oplocks = yes
> posix locking = no

Which version of Samba are you using here? If it's relatively recent
(4.6+), please rerun with asynchronous I/O enabled via:
[share]
aio read size = 1
aio write size = 1

...these settings are the default with Samba 4.8+. AIO won't help the
file creation / deletion benchmarks, but there should be a positive
affect on read/write performance.

> Disabling all locking (locking = no) gives some further speed improvements.
> 
> File locking hopefully will not be an issue...
> 
> We are not exporting this share via NFS. The shares will only be used by
> single clients (Windows or OSX Desktops) as a backup location.
> 
> Specifically, each machine has a separate smb mounted folder, to which
> they either use ChronoSync or Max SyncUp to write to.
> 
> One other point...
> Will CTDB work with "posix locking = no"?
> It would be great if CTDB works, as I'd like to have a several SMB heads
> to load-balance the clients

Yes, it shouldn't affect CTDB. Clustered FS POSIX locks are used by CTDB
for split-brain avoidance, and are separate to Samba's
client-lock <-> POSIX-lock mapping.
(https://wiki.samba.org/index.php/Configuring_the_CTDB_recovery_lock)
FYI, CTDB is now also capable of using RADOS objects for the recovery
lock:
https://ctdb.samba.org/manpages/ctdb_mutex_ceph_rados_helper.7.html

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] samba gateway experiences with cephfs ?

[David Disseldorp]

Hi Daniel and Jake,

On Mon, 21 May 2018 22:46:01 +0200, Daniel Baumann wrote:

> Hi
> 
> On 05/21/2018 05:38 PM, Jake Grimmett wrote:
> > Unfortunately we have a large number (~200) of Windows and Macs clients
> > which need CIFS/SMB  access to cephfs.  
> 
> we too, which is why we're (partially) exporting cephfs over samba too,
> 1.5y in production now.
> 
> for us, cephfs-over-samba is significantly slower than cephfs directly
> too, but it's not really an issue here (basically, if people use a
> windows client here, they're already on the slow track anyway).
> 
> we had to do two things to get it working reliably though:
> 
> a) disable all locking on samba (otherwise "opportunistic locking" on
> windows clients killed within hours all mds (kraken at that time))

Have you seen this on more recent versions? Please raise a bug if so -
client induced MDS outages would be a pretty serious issue.

If your share path is isolated from non-samba clients (as you mention
below), then allowing clients to cache reads / writes locally via
oplocks / SMB2+ leases should offer a significant performance
improvements.

> b) only allow writes to a specific space on cephfs, reserved to samba
> (with luminous; otherwise, we'd have problems with data consistency on
> cephfs with people writing the same files from linux->cephfs and
> samba->cephfs concurrently). my hunch is that samba caches writes and
> doesn't give them back appropriatly.

If you're sharing a kernel CephFS mount, then the Linux page cache will
be used for Samba share I/O, but Samba will fully abide by client sync
requests if "strict sync = yes" (default in Samba 4.7+).

> > Finally, is the vfs_ceph module for Samba useful? It doesn't seem to be
> > widely available pre-complied for for RHEL derivatives. Can anyone
> > comment on their experiences using vfs_ceph, or point me to a Centos 7.x
> > repo that has it?  
> 
> we use debian, with backported kernel and backported samba, which has
> vfs_ceph pre-compiled. however, we couldn't make vfs_ceph work at all -
> the snapshot patters just don't seem to match/align (and nothing we
> tried seem to work).

vfs_ceph doesn't support snapshots at this stage. I hope to work on this
feature in the near future, so that it's fully integrated with the
Explorer previous versions UI.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

[David Disseldorp]

Hi Maged,

On Mon, 12 Mar 2018 20:41:22 +0200, Maged Mokhtar wrote:

> I was thinking we would get the block request then loop down to all its
> osd requests and cancel those using the same  osd request cancel
> function. 

Until we can be certain of termination, I don't think it makes sense to
change the current behaviour of blocking the TMF ABORT response until
the cluster I/O completes.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

[David Disseldorp]

On Fri, 09 Mar 2018 11:23:02 +0200, Maged Mokhtar wrote:

> 2)I undertand that before switching the path, the initiator will send a 
> TMF ABORT can we pass this to down to the same abort_request() function 
> in osd_client that is used for osd_request_timeout expiry ? 

IIUC, the existing abort_request() codepath only cancels the I/O on the
client/gw side. A TMF ABORT successful response should only be sent if
we can guarantee that the I/O is terminated at all layers below, so I
think this would have to be implemented via an additional OSD epoch
barrier or similar.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

[David Disseldorp]

Hi shadowlin,

On Wed, 7 Mar 2018 23:24:42 +0800, shadow_lin wrote:

> Is it safe to use active/active multipath If use suse kernel with 
> target_core_rbd?
> Thanks.

A cross-gateway failover race-condition similar to what Mike described
is currently possible with active/active target_core_rbd. It's a corner
case that is dependent on a client assuming that unacknowledged I/O has
been implicitly terminated and can be resumed via an alternate path,
while the original gateway at the same time issues the original request
such that it reaches the Ceph cluster after differing I/O to the same
region via the alternate path.
It's not something that we've observed in the wild, but is nevertheless
a bug that is being worked on, with a resolution that should also be
usable for active/active tcmu-runner.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph iSCSI is a prank?

[David Disseldorp]

On Thu, 1 Mar 2018 09:11:21 -0500, Jason Dillaman wrote:

> It's very high on our priority list to get a solution merged in the
> upstream kernel. There was a proposal to use DLM to distribute the PGR
> state between target gateways (a la the SCST target) and it's quite
> possible that would have the least amount of upstream resistance since
> it would work for all backends and not just RBD. We, of course, would
> love to just use the Ceph cluster to distribute the state information
> instead of requiring a bolt-on DLM (with its STONITH error handling),
> but we'll take what we can get (merged).

I'm also very keen on having a proper upstream solution for this. My
preference is still to proceed with PR state backed by Ceph.

> I believe SUSE uses a custom downstream kernel that stores the PGR
> state in the Ceph cluster but requires two round-trips to the cluster
> for each IO (first to verify the PGR state and the second to perform
> the IO). The PetaSAN project is built on top of these custom kernel
> patches as well, I believe.

Maged from PetaSAN added support for rados-notify based PR state
retrieval. Sill, in the end the PR patch-set is too intrusive to make it
upstream, so we need to work on a proper upstreamable solution, with
tcmu-runner or otherwise.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to use vfs_ceph

[David Disseldorp]

On Fri, 22 Dec 2017 12:10:18 +0100, Felix Stolte wrote:

> I am using Samba 4.6.7 (shipped with Ubuntu 17.10). I've got it working 
> now by copying the ceph.client.admin.keyring to /etc/ceph (I'm very 
> unhappy with that).

The ceph:user_id smb.conf functionality was first shipped with
Samba 4.7.0 (via commit ec788bead311), so your version is likely
lacking this functionality.

> Which Samba Version & Linux Distribution are using? 

With SES5 and openSUSE 42.3 we ship Samba 4.6.9, but it includes a
backport of ec788bead311.

> Are you using quotas on subdirectories and are they applied when you 
> export the subdirectory via samba?

I've not used them personally, but given that Samba uses the standard
libcephfs API, max_bytes and max_files quotas should be enforced.
However, keep in mind that they don't in any way map to SMB quotas.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph+RBD+ISCSI = ESXI issue

[David Disseldorp]

Hi Nigel,

On Fri, 1 Dec 2017 13:32:43 +, nigel davies wrote:

> Ceph version 10.2.5
> 
> i have had an Ceph cluster going for a few months, with iscsi servers that
> are linked to Ceph by RBD.
> 
> All of an sudden i am starting the ESXI server will louse the isscsi data
> store (disk space goes to 0 B) and i only fix this by rebooting the ISCSI
> server
> 
> When checking syslogs on the iscsi server i get a loads of errors like
> 
> SENDING TMR_TASK_DOES_NOT_EXIST for ref_tag: 
> like 100+ lines
> 
> i looked at the logs and cant see anything saying hung io or an OSD come
> out and back in.
> 
> does any one have any susgestions on whats going on??

The TMR_TASK_DOES_NOT_EXIST error indicates that your initiator (ESXi)
is attempting to abort outstanding I/Os. ESXi is pretty latency
sensitive, so I'd guess that the abort-task requests are being sent by
the initiator after tripping a local I/O timeout. Your vmkernel logs
should shed a bit more light on this.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-ISCSI

[David Disseldorp]

On Wed, 11 Oct 2017 14:03:59 -0400, Jason Dillaman wrote:

> On Wed, Oct 11, 2017 at 1:10 PM, Samuel Soulard
>  wrote:
> > Hmmm, If you failover the identity of the LIO configuration including PGRs
> > (I believe they are files on disk), this would work no?  Using an 2 ISCSI
> > gateways which have shared storage to store the LIO configuration and PGR
> > data.  
> 
> Are you referring to the Active Persist Through Power Loss (APTPL)
> support in LIO where it writes the PR metadata to
> "/var/target/pr/aptpl_"? I suppose that would work for a
> Pacemaker failover if you had a shared file system mounted between all
> your gateways *and* the initiator requests APTPL mode(?).

I'm going off on a tangent here, but I can't seem to find where LIO
reads the /var/target/pr/aptpl_ PR state back off disk -
__core_scsi3_write_aptpl_to_file() seems to be the only function that
uses the path. Otherwise I would have thought the same, that the
propagating the file to backup gateways prior to failover would be
sufficient.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-ISCSI

[David Disseldorp]

Hi Jason,

Thanks for the detailed write-up...

On Wed, 11 Oct 2017 08:57:46 -0400, Jason Dillaman wrote:

> On Wed, Oct 11, 2017 at 6:38 AM, Jorge Pinilla López 
> wrote:
> 
> > As far as I am able to understand there are 2 ways of setting iscsi for
> > ceph
> >
> > 1- using kernel (lrbd) only able on SUSE, CentOS, fedora...
> >  
> 
> The target_core_rbd approach is only utilized by SUSE (and its derivatives
> like PetaSAN) as far as I know. This was the initial approach for Red
> Hat-derived kernels as well until the upstream kernel maintainers indicated
> that they really do not want a specialized target backend for just krbd.
> The next attempt was to re-use the existing target_core_iblock to interface
> with krbd via the kernel's block layer, but that hit similar upstream walls
> trying to get support for SCSI command passthrough to the block layer.
> 
> 
> > 2- using userspace (tcmu , ceph-iscsi-conf, ceph-iscsi-cli)
> >  
> 
> The TCMU approach is what upstream and Red Hat-derived kernels will support
> going forward.

SUSE is also in the process of migrating to the upstream tcmu approach,
for the reasons that you gave in (1).

...

> The TCMU approach also does not currently support SCSI persistent
> reservation groups (needed for Windows clustering) because that support
> isn't available in the upstream kernel. The SUSE kernel has an approach
> that utilizes two round-trips to the OSDs for each IO to simulate PGR
> support. Earlier this summer I believe SUSE started to look into how to get
> generic PGR support merged into the upstream kernel using corosync/dlm to
> synchronize the states between multiple nodes in the target. I am not sure
> of the current state of that work, but it would benefit all LIO targets
> when complete.

Zhu Lingshan (cc'ed) worked on a prototype for tcmu PR support. IIUC,
whether DLM or the underlying Ceph cluster gets used for PR state
storage is still under consideration.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] State of play for RDMA on Luminous

[David Disseldorp]

Hi Florian,

On Wed, 23 Aug 2017 10:26:45 +0200, Florian Haas wrote:

> - In case there is no such support in the kernel yet: What's the current
> status of RDMA support (and testing) with regard to
>   * libcephfs?
>   * the Samba Ceph VFS?

On the client side, the SMB3 added an SMB-Direct protocol extension for
RDMA. Quite a bit of work needs to be done to get this supported in
Samba: https://wiki.samba.org/index.php/Samba3/SMB2#SMB_Direct

...
>   * tcmu-runner?

Also on the client side, LIO offers support for iSCSI Extensions for
RDMA transport, which should be usable alongside a tcmu-runner
backstore: http://linux-iscsi.org/wiki/ISCSI_Extensions_for_RDMA
I'm not aware of any testing of this combination though.

Cheers, David


pgpnwbOwz5naj.pgp
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Does cephfs guarantee client cache consistency for file data?

[David Disseldorp]

Hi,

On Wed, 19 Apr 2017 08:19:50 +, 许雪寒 wrote:

> I’m new to cephfs. I wonder whether cephfs guarantee client cache consistency 
> for file content. For example, if client A read some data of file X, then 
> client B modified the X’s content in the range that A read, will A be 
> notified of the modification?

Yes, clients are granted fine-grained cache "capabilities" by the MDS.
These capabilities can be revoked to trigger a flush of cached content,
prior to satisfying a request from a separate client.

Jeff Layton did a nice write up on this:
https://jtlayton.wordpress.com/2016/09/01/cephfs-and-the-nfsv4-change-attribute/

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd iscsi gateway question

[David Disseldorp]

On Thu, 6 Apr 2017 14:27:01 +0100, Nick Fisk wrote:
...
> > I'm not to sure what you're referring to WRT the spiral of death, but we did
> > patch some LIO issues encountered when a command was aborted while
> > outstanding at the LIO backstore layer.
> > These specific fixes are carried in the mainline kernel, and can be tested
> > using the AbortTaskSimpleAsync libiscsi test.  
> 
> Awesome, glad this has finally been fixed. Death spiral was referring to when 
> using it with ESXi, both the initiator and target effectively hang forever 
> and if you didn't catch it soon enough, sometimes you end up having to kill 
> all vm's and reboot hosts.

Sounds like it could be the same thing. Stale iSCSI sessions remain
around which block subsequent login attempts.

> Do you know what kernel version these changes would have first gone into? I 
> thought I looked back into this last summer and it was still showing the same 
> behavior.

The fix I was referring to is:
commit 5e2c956b8aa24d4f33ff7afef92d409eed164746
Author: Nicholas Bellinger 
Date:   Wed May 25 12:25:04 2016 -0700

target: Fix missing complete during ABORT_TASK + CMD_T_FABRIC_STOP

It's carried in v4.8+ and was also flagged for 3.14+ stable inclusion,
so should be present in many distro kernels by now. That said, there
have been many other changes in this area.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd iscsi gateway question

[David Disseldorp]

Hi,

On Thu, 6 Apr 2017 13:31:00 +0100, Nick Fisk wrote:

> > I believe there
> > was a request to include it mainstream kernel but it did not happen,
> > probably waiting for TCMU solution which will be better/cleaner design.  

Indeed, we're proceeding with TCMU as a future upstream acceptable
implementation.

> Yes, should have mentioned this, if you are using the suse kernel, they have
> a fix for this spiral of death problem.

I'm not to sure what you're referring to WRT the spiral of death, but we
did patch some LIO issues encountered when a command was aborted while
outstanding at the LIO backstore layer.
These specific fixes are carried in the mainline kernel, and can be
tested using the AbortTaskSimpleAsync libiscsi test.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: Upgrade Woes on suse leap with OBS ceph.

[David Disseldorp]

Hi,

On Thu, 23 Feb 2017 21:07:41 -0800, Schlacta, Christ wrote:

> So hopefully when the suse ceph team get 11.2 released it should fix this,
> yes?

Please raise a bug at bugzilla.opensuse.org, so that we can track this
for the next openSUSE maintenance update.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [RFC] rbdmap unmap - unmap all, or only RBDMAPFILE listed images?

[David Disseldorp]

Hi,

I'm working on an rbdmap change https://github.com/ceph/ceph/pull/13361,
and would appreciate some input from existing users.

Currently "rbdmap map" maps any RBD images listed in the rbdmap config
file (RBDMAPFILE), whereas "rbdmap unmap" unmaps all mapped RBD images,
regardless of whether they're present in RBDMAPFILE or not.

In adding support to only unmap RBDMAPFILE listed images when called
with "rbdmap unmap", I'd like to know whether the existing unmap-all
functionality is currently used, or whether it could be dropped.

Feedback appreciated.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs quota

[David Disseldorp]

Hi Matthew,

On Fri, 16 Dec 2016 12:30:06 +, Matthew Vernon wrote:

> Hello,
> On 15/12/16 10:25, David Disseldorp wrote:
> 
> > Are you using the Linux kernel CephFS client (mount.ceph), or the
> > userspace ceph-fuse back end? Quota enforcement is performed by the
> > client, and is currently only supported by ceph-fuse.  
> 
> Is server enforcement of quotas planned?

Not that I'm aware of at this stage. John (cc'ed) mentioned at Vault that
stricter enforcement required improved infrastructure:
http://events.linuxfoundation.org/sites/events/files/slides/CephFS-Vault.pdf

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs quota

[David Disseldorp]

On Fri, 16 Dec 2016 12:48:39 +0530, gjprabu wrote:

> Now we are mounted client using ceph-fuse and still allowing me to put a data 
> above the limit(100MB). Below is quota details.
> 
> 
> 
> getfattr -n ceph.quota.max_bytes test
> 
> # file: test
> 
> ceph.quota.max_bytes="1"   
> 
> 
> 
> ceph-fuse  fuse.ceph-fuse  5.3T  485G  4.8T  10% /mnt/test
> 
> 
> 
> du -sh test
> 
> 1.9G test

Do the users performing the I/O have access to the /mnt/test path? Have
you tried setting the quota on a subdirectory within the filesystem?
Also, keep in mind that enforcement doesn't occur immediately. Normally
a short amount of time needs to elapse before the EDQUOT error is
returned to the writer that exceeded the quota limit.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs quota

[David Disseldorp]

Hi Prabu,

On Thu, 15 Dec 2016 13:11:50 +0530, gjprabu wrote:

>   We are using ceph version 10.2.4 (Jewel) and data's are mounted 
> with cephfs file system in linux. We are trying to set quota for directory 
> and files but its don't worked with below document. I have set 100mb for 
> directory quota but after reaching keep me allowing to put the data in that 
> location. Highly appreciate any one help on this issue.

Are you using the Linux kernel CephFS client (mount.ceph), or the
userspace ceph-fuse back end? Quota enforcement is performed by the
client, and is currently only supported by ceph-fuse.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Prevent cephfs clients from mount and browsing "/"

[David Disseldorp]

Hi Martin,

On Mon, 5 Dec 2016 13:27:01 +0100, Martin Palma wrote:

> Ok, just discovered that with the fuse client, we have to add the '-r
> /path' option, to treat that as root. So I assume the caps 'mds allow
> r' is only needed if we also what to be able to mount the directory
> with the kernel client. Right?

IIUC, this was recently fixed in the kernel client via:
commit ce2728aaa82bbebae7d20345324af3f0f49eeb20
Author: Yan, Zheng 
Date:   Wed Sep 14 14:53:05 2016 +0800

ceph: avoid accessing / when mounting a subpath

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] new Open Source Ceph based iSCSI SAN project

[David Disseldorp]

Hi Maged,

Thanks for the announcement - good luck with the project!
One comment...

On Mon, 17 Oct 2016 13:37:29 +0200, Maged Mokhtar wrote:

> if you are refering to clustering reservations through VAAI. We are using 
> upstream code from SUSE Enterprise Storage which adds clustered support for 
> VAAI (compare and write, write same) in the kernel as well as in ceph 
> (implemented as atomic  osd operations). We have tested VMware HA and 
> vMotion and they work fine. We have a guide you can download on this use 
> case.

Just so there's no ambiguity here, the vast majority of the clustered
compare-and-write and write-same implementation was done by Mike
Christie from Red Hat.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mount -t ceph

[David Disseldorp]

Hi Tom,

On Wed, 27 Apr 2016 20:17:51 +, Deneau, Tom wrote:

> I was using SLES 12, SP1 which has 3.12.49 
> 
> It did have a /usr/sbin/mount.ceph command but using it gave 
>   modprobe: FATAL: Module ceph not found.
>   failed to load ceph kernel module (1)

The SLES 12 SP1 kernel doesn't currently ship with CephFS support.

Regards, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ocfs2 for OSDs?

[David Disseldorp]

Hi Sage,

On Wed, 11 Sep 2013 09:18:13 -0700 (PDT)
Sage Weil s...@inktank.com wrote:

  REFLINKs (inode-based writeable snapshots)  
 
 This is the one item on this list I see that the ceph-osds could take real 
 advantage of; it would make object clones triggered by things like RBD 
 snapshots faster.  What is missing from this list that would be similarly 
 (or more) useful is a volume/fs snapshot feature.

I must be missing something here, but I don't see how this would offer
any advantage over Btrfs, which provides the same feature via
BTRFS_IOC_CLONE and BTRFS_IOC_CLONE_RANGE.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com