[ceph-users] Re: Cephfs over internet

[Marc]

> Hi all,
> Due to so many reasons (political, heating problems, lack of space
> aso.) we have to
> plan for our ceph cluster to be hosted externaly.
> The planned version to setup is reef.
> Reading up on documentation we found that it was possible to run in
> secure mode.
> 
> Our ceph.conf file will state both v1 and v2 addresses for mons:
> mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0]
> [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0]
> [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0]
> 
> Then changing the following configuration options to only secure:
> ms_cluster_mode = secure
> ms_service_mode = secure
> ms_client_mode = secure
> ms_mon_cluster_mode = secure
> ms_mon_service_mode = secure
> ms_mon_client_mode = secure
> 
> Then I remounted cephfs on the clients on our test cluster,
> but still the fs would mount on ports 6789.
> I thought that the above secure config change would "force"
> the mount on port 3300 and v2.
> Mounting with option ms_mode=secure, did the trick.
> Is that the way cephfs is working that you explicit have to
> specify secure mode? I thought that cephfs clients would
> use the secure mode with these settings, but maybe I am wrong?
> 
> Of cause we also plan to limit the firewalls on servers so only
> the specific subnet will be able to connect and mount cephfs.
> 
>  From my understanding from the documenation this would be the
> way to set this up with ceph exposed to internet.
> 
> Is there something that we are missing or something that would
> make the setup more secure?
> 

What about a tunnel, and have a local ip range route through it? I am not sure 
what happens if someone is brute forcing your monitors.



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs over internet

[Malcolm Haak]

Yeah, you really want to do this over a vpn.

Performance is going to be average at best. It would probably be
faster to re-export it as NFS/SMB and push that across the internet.

On Mon, May 20, 2024 at 11:37 PM Marc  wrote:
>
> > Hi all,
> > Due to so many reasons (political, heating problems, lack of space
> > aso.) we have to
> > plan for our ceph cluster to be hosted externaly.
> > The planned version to setup is reef.
> > Reading up on documentation we found that it was possible to run in
> > secure mode.
> >
> > Our ceph.conf file will state both v1 and v2 addresses for mons:
> > mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0]
> > [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0]
> > [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0]
> >
> > Then changing the following configuration options to only secure:
> > ms_cluster_mode = secure
> > ms_service_mode = secure
> > ms_client_mode = secure
> > ms_mon_cluster_mode = secure
> > ms_mon_service_mode = secure
> > ms_mon_client_mode = secure
> >
> > Then I remounted cephfs on the clients on our test cluster,
> > but still the fs would mount on ports 6789.
> > I thought that the above secure config change would "force"
> > the mount on port 3300 and v2.
> > Mounting with option ms_mode=secure, did the trick.
> > Is that the way cephfs is working that you explicit have to
> > specify secure mode? I thought that cephfs clients would
> > use the secure mode with these settings, but maybe I am wrong?
> >
> > Of cause we also plan to limit the firewalls on servers so only
> > the specific subnet will be able to connect and mount cephfs.
> >
> >  From my understanding from the documenation this would be the
> > way to set this up with ceph exposed to internet.
> >
> > Is there something that we are missing or something that would
> > make the setup more secure?
> >
>
> What about a tunnel, and have a local ip range route through it? I am not 
> sure what happens if someone is brute forcing your monitors.
>
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs over internet

[Marcus]

Thanks for your answers!
I read somewhere that a vpn would really have an impact on performance, 
so it was not recommended, and I found v2 protocol.

But vpn feels like the solution and you have to accept the lower speed.

Thanks again!

On tis, maj 21 2024 at 17:07:48 +1000, Malcolm Haak 
 wrote:

Yeah, you really want to do this over a vpn.

Performance is going to be average at best. It would probably be
faster to re-export it as NFS/SMB and push that across the internet.

On Mon, May 20, 2024 at 11:37 PM Marc > wrote:


 > Hi all,
 > Due to so many reasons (political, heating problems, lack of space
 > aso.) we have to
 > plan for our ceph cluster to be hosted externaly.
 > The planned version to setup is reef.
 > Reading up on documentation we found that it was possible to run 
in

 > secure mode.
 >
 > Our ceph.conf file will state both v1 and v2 addresses for mons:
 > mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0]
 > [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0]
 > [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0]
 >
 > Then changing the following configuration options to only secure:
 > ms_cluster_mode = secure
 > ms_service_mode = secure
 > ms_client_mode = secure
 > ms_mon_cluster_mode = secure
 > ms_mon_service_mode = secure
 > ms_mon_client_mode = secure
 >
 > Then I remounted cephfs on the clients on our test cluster,
 > but still the fs would mount on ports 6789.
 > I thought that the above secure config change would "force"
 > the mount on port 3300 and v2.
 > Mounting with option ms_mode=secure, did the trick.
 > Is that the way cephfs is working that you explicit have to
 > specify secure mode? I thought that cephfs clients would
 > use the secure mode with these settings, but maybe I am wrong?
 >
 > Of cause we also plan to limit the firewalls on servers so only
 > the specific subnet will be able to connect and mount cephfs.
 >
 >  From my understanding from the documenation this would be the
 > way to set this up with ceph exposed to internet.
 >
 > Is there something that we are missing or something that would
 > make the setup more secure?
 >

 What about a tunnel, and have a local ip range route through it? I 
am not sure what happens if someone is brute forcing your monitors.




 ___
 ceph-users mailing list -- ceph-users@ceph.io 

 To unsubscribe send an email to ceph-users-le...@ceph.io 


___
ceph-users mailing list -- ceph-users@ceph.io 

To unsubscribe send an email to ceph-users-le...@ceph.io 



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs over internet

[Paul Mezzanini]

We did a proof of concept moving some compute into "the cloud" and exported our 
cephfs shares using wireguard as the tunnel.   The performance impact on our 
storage was completely latency and bandwidth dependent with no noticeable 
impact from the tunnel itself.  

-paul

--

Paul Mezzanini
Platform Engineer III
Research Computing
Rochester Institute of Technology




From: Marcus 
Sent: Tuesday, May 21, 2024 7:39 AM
To: ceph-users
Subject: [ceph-users] Re: Cephfs over internet

Thanks for your answers!
I read somewhere that a vpn would really have an impact on performance,
so it was not recommended, and I found v2 protocol.
But vpn feels like the solution and you have to accept the lower speed.

Thanks again!

On tis, maj 21 2024 at 17:07:48 +1000, Malcolm Haak
 wrote:
> Yeah, you really want to do this over a vpn.
>
> Performance is going to be average at best. It would probably be
> faster to re-export it as NFS/SMB and push that across the internet.
>
> On Mon, May 20, 2024 at 11:37 PM Marc  <mailto:m...@f1-outsourcing.eu>> wrote:
>>
>>  > Hi all,
>>  > Due to so many reasons (political, heating problems, lack of space
>>  > aso.) we have to
>>  > plan for our ceph cluster to be hosted externaly.
>>  > The planned version to setup is reef.
>>  > Reading up on documentation we found that it was possible to run
>> in
>>  > secure mode.
>>  >
>>  > Our ceph.conf file will state both v1 and v2 addresses for mons:
>>  > mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0]
>>  > [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0]
>>  > [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0]
>>  >
>>  > Then changing the following configuration options to only secure:
>>  > ms_cluster_mode = secure
>>  > ms_service_mode = secure
>>  > ms_client_mode = secure
>>  > ms_mon_cluster_mode = secure
>>  > ms_mon_service_mode = secure
>>  > ms_mon_client_mode = secure
>>  >
>>  > Then I remounted cephfs on the clients on our test cluster,
>>  > but still the fs would mount on ports 6789.
>>  > I thought that the above secure config change would "force"
>>  > the mount on port 3300 and v2.
>>  > Mounting with option ms_mode=secure, did the trick.
>>  > Is that the way cephfs is working that you explicit have to
>>  > specify secure mode? I thought that cephfs clients would
>>  > use the secure mode with these settings, but maybe I am wrong?
>>  >
>>  > Of cause we also plan to limit the firewalls on servers so only
>>  > the specific subnet will be able to connect and mount cephfs.
>>  >
>>  >  From my understanding from the documenation this would be the
>>  > way to set this up with ceph exposed to internet.
>>  >
>>  > Is there something that we are missing or something that would
>>  > make the setup more secure?
>>  >
>>
>>  What about a tunnel, and have a local ip range route through it? I
>> am not sure what happens if someone is brute forcing your monitors.
>>
>>
>>
>>  ___
>>  ceph-users mailing list -- ceph-users@ceph.io
>> <mailto:ceph-users@ceph.io>
>>  To unsubscribe send an email to ceph-users-le...@ceph.io
>> <mailto:ceph-users-le...@ceph.io>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> <mailto:ceph-users@ceph.io>
> To unsubscribe send an email to ceph-users-le...@ceph.io
> <mailto:ceph-users-le...@ceph.io>

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs over internet

[Burkhard Linke]

Hi,

On 5/21/24 13:39, Marcus wrote:

Thanks for your answers!
I read somewhere that a vpn would really have an impact on 
performance, so it was not recommended, and I found v2 protocol.

But vpn feels like the solution and you have to accept the lower speed.



Also keep in mind that clients have to be able to access all nodes of 
the cluster. You are not only exposing the ceph mons, but all ceph services.



Using VPN/tunnel is the minimal acceptable solution. Using a NFS/SMB 
gateway (and expose this gateway via tunnel/VPN + kerberos) should IMHO 
be a better solution security wise.



Regards,

Burkhard

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs over internet

[adam.ther]

Hello,

You will want to do this over WireGuard tech from experience, IOPS will 
be brutal, like 200 IOPS.


Wireguard has a few benefits but notably:

   - Higher rate of transfer per CPU load.

   - State of the the art protocols. As opposed to some of the more
   legacy systems.

   - Extremely fast re-connections, re-keys on IKE can be brutal.

Further fine tuning is possible too but i'm, not at that level of 
knowledge. For flat files of several gigs you're going to get a decent 
experience, but if you try to run VMs off this... you're in for a bad time.


Regards,

Adam

On 5/21/24 01:07, Malcolm Haak wrote:

Yeah, you really want to do this over a vpn.

Performance is going to be average at best. It would probably be
faster to re-export it as NFS/SMB and push that across the internet.

On Mon, May 20, 2024 at 11:37 PM Marc  wrote:

Hi all,
Due to so many reasons (political, heating problems, lack of space
aso.) we have to
plan for our ceph cluster to be hosted externaly.
The planned version to setup is reef.
Reading up on documentation we found that it was possible to run in
secure mode.

Our ceph.conf file will state both v1 and v2 addresses for mons:
mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0]
[v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0]
[v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0]

Then changing the following configuration options to only secure:
ms_cluster_mode = secure
ms_service_mode = secure
ms_client_mode = secure
ms_mon_cluster_mode = secure
ms_mon_service_mode = secure
ms_mon_client_mode = secure

Then I remounted cephfs on the clients on our test cluster,
but still the fs would mount on ports 6789.
I thought that the above secure config change would "force"
the mount on port 3300 and v2.
Mounting with option ms_mode=secure, did the trick.
Is that the way cephfs is working that you explicit have to
specify secure mode? I thought that cephfs clients would
use the secure mode with these settings, but maybe I am wrong?

Of cause we also plan to limit the firewalls on servers so only
the specific subnet will be able to connect and mount cephfs.

  From my understanding from the documenation this would be the
way to set this up with ceph exposed to internet.

Is there something that we are missing or something that would
make the setup more secure?


What about a tunnel, and have a local ip range route through it? I am not sure 
what happens if someone is brute forcing your monitors.



___
ceph-users mailing list --ceph-users@ceph.io
To unsubscribe send an email toceph-users-le...@ceph.io

___
ceph-users mailing list --ceph-users@ceph.io
To unsubscribe send an email toceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io