[ceph-users] Re: Forcing Posix Permissions On New CephFS Files

[Alexander E. Patrakov]

Hello Matthew,

You can inherit the group, but not the user, of the containing folder.
This can be achieved by making the folder setgid and then making sure
that the client systems have a proper umask. See the attached PDF for
a presentation that I conducted on this topic to my students in the
past.

On Thu, May 9, 2024 at 2:03 PM duluxoz  wrote:
>
> Hi All,
>
> I've gone and gotten myself into a "can't see the forest for the trees"
> state, so I'm hoping someone can take pity on me and answer a really dumb Q.
>
> So I've got a CephFS system happily bubbling along and a bunch of
> (linux) workstations connected to a number of common shares/folders. To
> take a single one of these folders as an example ("music") the
> sub-folders and files of that share all belong to root:music with
> permissions of 2770 (folders) and 0660 (files). The "music" folder is
> then connected to (as per the Ceph Doco: mount.ceph) via each
> workstation's fstab file - all good, all working, everyone's happy.
>
> What I'm trying to achieve is that when a new piece of music (a file) is
> uploaded to the Ceph Cluster the file inherits the music share's default
> ownership (root:music) and permissions (0660). What is happening at the
> moment is I'm getting permissions of 644 (and 755 for new folders).
>
> I've been looking for a way to do what I want but, as I said, I've gone
> and gotten myself thoroughly mixed-up.
>
> Could someone please point me in the right direction on how to achieve
> what I'm after - thanks
>
> Cheers
>
> Dulux-Oz
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io



-- 
Alexander Patrakov
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Forcing Posix Permissions On New CephFS Files

[Paul Mezzanini]

We use shared directories for projects in our hpc cluster and use acls to 
achieve this.

https://www.redhat.com/sysadmin/linux-access-control-lists

--

Paul Mezzanini
Platform Engineer III
Research Computing

Rochester Institute of Technology

 Sent from my phone, please excuse typos and brevity

From: duluxoz 
Sent: Thursday, May 9, 2024 2:03:02 AM
To: ceph-users@ceph.io 
Subject: [ceph-users] Forcing Posix Permissions On New CephFS Files

Hi All,

I've gone and gotten myself into a "can't see the forest for the trees"
state, so I'm hoping someone can take pity on me and answer a really dumb Q.

So I've got a CephFS system happily bubbling along and a bunch of
(linux) workstations connected to a number of common shares/folders. To
take a single one of these folders as an example ("music") the
sub-folders and files of that share all belong to root:music with
permissions of 2770 (folders) and 0660 (files). The "music" folder is
then connected to (as per the Ceph Doco: mount.ceph) via each
workstation's fstab file - all good, all working, everyone's happy.

What I'm trying to achieve is that when a new piece of music (a file) is
uploaded to the Ceph Cluster the file inherits the music share's default
ownership (root:music) and permissions (0660). What is happening at the
moment is I'm getting permissions of 644 (and 755 for new folders).

I've been looking for a way to do what I want but, as I said, I've gone
and gotten myself thoroughly mixed-up.

Could someone please point me in the right direction on how to achieve
what I'm after - thanks

Cheers

Dulux-Oz
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBD Mirroring with Journaling and Snapshot mechanism

[Ramana Krisna Venkatesh Raja]

On Tue, May 7, 2024 at 7:54 AM Eugen Block  wrote:
>
> Hi,
>
> I'm not the biggest rbd-mirror expert.
> As understand it, if you use one-way mirroring you can failover to the
> remote site, continue to work there but there's no failover back to
> primary site. You would need to stop client IO on DR, demote the image
> and then import the remote images back to primary site. Once
> everything is good you can promote the image on primary again. The
> rbd-mirror will then most likely be in a split-brain situation, which
> can be resolved by resyncing images from primary again. You can't do a
> resync on primary site because there's no rbd-mirror daemon running.
>
> Having two-way mirroring could help, I believe. Let's say you lose the
> primary site, you can (force) promote images on the remote site,
> continue working. Once the primary site is back up (but not primary
> yet), you can do the image resync from the remote (currently primary)
> site (because there's a rbd-mirror daemon running on the primary site
> as well). Once the primary site has all images promoted, you'll
> probably have to resync on the remote site again to get out of the
> split-brain.

Also, you need to demote the out-of-date images in the cluster that
came back, before issuing resync on them.
This is to resolve the split-brain.
See, https://docs.ceph.com/en/latest/rbd/rbd-mirroring/#force-image-resync

-Ramana

> But at least you won't need to export/import images.
>
> But you'll need to test this properly to find out if your requirements
> are met.
>
> Regards,
> Eugen
>
>
> Zitat von V A Prabha :
>
> > Dear Eugen,
> > We have a scenario of DC and DR replication, and planned to explore RBD
> > mirroring with both Journaling and Snapshot mechanism.
> > I have a 5 TB storage at Primary DC and 5 TB storage at DR site with
> > 2 different
> > ceph clusters configured.
> >
> > Please clarify the following queries
> >
> > 1. With One way mirroring, the failover works fine in both journaling and
> > snapshot mechanism and we are able to promote the workload from DR site. How
> > does Failback work? We wanted to move the contents from DR to DC but
> > it fails.
> > In journaling mechanism, it deletes the entire volume and recreates it 
> > afresh
> > which does not solve our problem.
> > 2. How does incremental replication work from DR to DC?
> > 3. Does Two-way mirroring help this situation. According to me, in
> > this method,
> > it is for 2 different clouds with 2 different storages and
> > replicating both the
> > clouds workloads? Does Failback work in this scenario ?
> > Please help us / guide us to deploy this solution
> >
> > Regards
> > V.A.Prabha
> >
> >
> > Thanks & Regards,
> > Ms V A Prabha / श्रीमती प्रभा वी ए
> > Joint Director / संयुक्त निदेशक
> > Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास
> > केन्द्र(सी-डैक)
> > Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं 
> > मंजिल,
> > “डी” ब्लॉक, (उत्तर और दक्षिण)
> > No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई
> > Taramani / तारामणि
> > Chennai / चेन्नई – 600113
> > Ph.No.:044-22542226/27
> > Fax No.: 044-22542294
> > 
> > [ C-DAC is on Social-Media too. Kindly follow us at:
> > Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
> >
> > This e-mail is for the sole use of the intended recipient(s) and may
> > contain confidential and privileged information. If you are not the
> > intended recipient, please contact the sender by reply e-mail and destroy
> > all copies and the original message. Any unauthorized review, use,
> > disclosure, dissemination, forwarding, printing or copying of this email
> > is strictly prohibited and appropriate legal action will be taken.
> > 
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBD Mirroring with Journaling and Snapshot mechanism

[Ramana Krisna Venkatesh Raja]

On Thu, May 2, 2024 at 2:56 AM V A Prabha  wrote:
>
> Dear Eugen,
> We have a scenario of DC and DR replication, and planned to explore RBD
> mirroring with both Journaling and Snapshot mechanism.
> I have a 5 TB storage at Primary DC and 5 TB storage at DR site with 2 
> different
> ceph clusters configured.
>
> Please clarify the following queries
>
> 1. With One way mirroring, the failover works fine in both journaling and
> snapshot mechanism and we are able to promote the workload from DR site. How
> does Failback work? We wanted to move the contents from DR to DC but it fails.

You'd need a RBD mirror daemon running in the DC cluster to replicate
the changes from DR to DC as Eugen said earlier. I suggest setting up
two-way mirroring with a RBD mirror daemon in each cluster for easy
fail over/fail back. The RBD mirror daemon in the other cluster that's
not replicating changes can just be left running. It won't be doing
any active mirroring work.

> In journaling mechanism, it deletes the entire volume and recreates it afresh
> which does not solve our problem.

Not sure about this. What commands did you run here?

> 2. How does incremental replication work from DR to DC?

The RBD mirror daemon in the DC cluster would use the same incremental
replication mechanism as that of the mirror daemon in the DR cluster
that replicated images before the failover.

> 3. Does Two-way mirroring help this situation. According to me, in this 
> method,
> it is for 2 different clouds with 2 different storages and replicating both 
> the
> clouds workloads? Does Failback work in this scenario ?
> Please help us / guide us to deploy this solution

Yes, two-way mirroring for easy failover/failback. Also keep in mind
that journal based mirroring involves writing to the primary image's
journal and to the image itself. Snapshot based mirroring is being
actively enhanced and doesn't have 2X writes in the primary cluster.
You'd have to find out the mirroring snapshot schedule that works for
your setup.
Snapshot based mirroring for propagating discards to secondary [1] and
for replicating clones [2] are being worked on.

Hope this helps.

Best,
Ramana

[1] https://tracker.ceph.com/issues/58852
[2] https://tracker.ceph.com/issues/61891


>
> Regards
> V.A.Prabha
>
>
> Thanks & Regards,
> Ms V A Prabha / श्रीमती प्रभा वी ए
> Joint Director / संयुक्त निदेशक
> Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास
> केन्द्र(सी-डैक)
> Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल,
> “डी” ब्लॉक, (उत्तर और दक्षिण)
> No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई
> Taramani / तारामणि
> Chennai / चेन्नई – 600113
> Ph.No.:044-22542226/27
> Fax No.: 044-22542294
> 
> [ C-DAC is on Social-Media too. Kindly follow us at:
> Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
>
> This e-mail is for the sole use of the intended recipient(s) and may
> contain confidential and privileged information. If you are not the
> intended recipient, please contact the sender by reply e-mail and destroy
> all copies and the original message. Any unauthorized review, use,
> disclosure, dissemination, forwarding, printing or copying of this email
> is strictly prohibited and appropriate legal action will be taken.
> 
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Call for Proposals: Cephalocon 2024

[Noah Lehman]

Hi Ceph community,

We are excited to share the CFP for Cephalocon 2024
, running
December 4-5 in Geneva, Switzerland.

The deadline to submit a proposal is August 11, 2024. You can submit a
proposal here .

Running 4-5 December 2024, Cephalocon is the premier yearly event that
brings together the global community of operators, developers, and
researchers to celebrate Ceph, the open source distributed storage system
designed to provide excellent performance, reliability, and scalability.
Join new and existing community members from around the world at CERN to
learn more about Ceph and the future of the project from the developers
writing the code and the operators deploying it.

*DATES TO REMEMBER:*

CFP Opens: 6 May, 2024
CFP Close: 11 August, 2024
Speaker Notifications: 9 September, 2024
Schedule Announced: 12 September, 2024
Slides due date: 2 December, 2024
Event Dates: 4 December, 2024 - 5 December, 2024

Best,

Noah
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS crash in interval_set: FAILED ceph_assert(p->first <= start)

[Dejan Lesjak]

Hi Xiubo,

Thanks. We'll check that.

Cheers,
Dejan

On 9. 05. 24 07:22, Xiubo Li wrote:


On 5/8/24 17:36, Dejan Lesjak wrote:

Hi Xiubo,

On 8. 05. 24 09:53, Xiubo Li wrote:

Hi Dejan,

This is a known issue and please see 
https://tracker.ceph.com/issues/61009.


For the workaround please see 
https://tracker.ceph.com/issues/61009#note-26.


Thank you for the links. Unfortunately I'm not sure I understand the 
workaround: the clients should be mounted without nowsync, however, 
the clients don't get to the point of mounting as mds is not available 
yet as it is doing replay.
Rebooting clients does not seem to help as they are still in clients 
list (from "ceph tell mds.1 client ls").



Hi Dejan,

We are disscussing the same issue in slack thread 
https://ceph-storage.slack.com/archives/C04LVQMHM9B/p1715189877518529.


Thanks

- Xiubo



Thanks,
Dejan


Thanks

- Xiubo

On 5/8/24 06:49, Dejan Lesjak wrote:

Hello,

We have cephfs with two active MDS. Currently rank 1 is repeatedly 
crashing with FAILED ceph_assert(p->first <= start) in md_log_replay 
thread. Is there any way to work around this and get to accesible 
file system or should we start with disaster recovery?

It seems similar to https://tracker.ceph.com/issues/61009
Crash info:

{
 "assert_condition": "p->first <= start",
 "assert_file": 
"/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el9/BUILD/ceph-18.2.2/src/include/interval_set.h",
 "assert_func": "void interval_set::erase(T, T, 
std::function) [with T = inodeno_t; C = std::map]",

 "assert_line": 568,
 "assert_msg": 
"/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el9/BUILD/ceph-18.2.2/src/include/interval_set.h: In function 'void interval_set::erase(T, T, std::function) [with T = inodeno_t; C = std::map]' thread 7fcdaaf8a640 time 2024-05-08T00:26:22.049974+0200\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el9/BUILD/ceph-18.2.2/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start)\n",

 "assert_thread_name": "md_log_replay",
 "backtrace": [
 "/lib64/libc.so.6(+0x54db0) [0x7fcdb7a54db0]",
 "/lib64/libc.so.6(+0xa154c) [0x7fcdb7aa154c]",
 "raise()",
 "abort()",
 "(ceph::__ceph_assert_fail(char const*, char const*, int, 
char const*)+0x188) [0x7fcdb83610ff]",
 "/usr/lib64/ceph/libceph-common.so.2(+0x161263) 
[0x7fcdb8361263]",

 "/usr/bin/ceph-mds(+0x1f3b0e) [0x55a5904a9b0e]",
 "/usr/bin/ceph-mds(+0x1f3b55) [0x55a5904a9b55]",
 "(EMetaBlob::replay(MDSRank*, LogSegment*, int, 
MDPeerUpdate*)+0x4b9d) [0x55a5906e1c8d]",

 "(EUpdate::replay(MDSRank*)+0x5d) [0x55a5906eacbd]",
 "(MDLog::_replay_thread()+0x7a1) [0x55a590694af1]",
 "/usr/bin/ceph-mds(+0x1460f1) [0x55a5903fc0f1]",
 "/lib64/libc.so.6(+0x9f802) [0x7fcdb7a9f802]",
 "/lib64/libc.so.6(+0x3f450) [0x7fcdb7a3f450]"
 ],
 "ceph_version": "18.2.2",
 "crash_id": 
"2024-05-07T22:26:22.050652Z_8be89ffb-bb87-4832-9339-57f8bd29f766",

 "entity_name": "mds.spod19",
 "os_id": "almalinux",
 "os_name": "AlmaLinux",
 "os_version": "9.3 (Shamrock Pampas Cat)",
 "os_version_id": "9.3",
 "process_name": "ceph-mds",
 "stack_sig": 
"3d0a2ca9b3c7678bf69efc20fff42b588c63f8be1832e1e0c28c99bafc082c15",

 "timestamp": "2024-05-07T22:26:22.050652Z",
 "utsname_hostname": "spod19.ijs.si",
 "utsname_machine": "x86_64",
 "utsname_release": "5.14.0-362.8.1.el9_3.x86_64",
 "utsname_sysname": "Linux",
 "utsname_version": "#1 SMP PREEMPT_DYNAMIC Tue Nov 7 14:54:22 
EST 2023"

}


Cheers,
Dejan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io








___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io