Re: [Gluster-users] Gluster inside containers

[Zach Lanich]

It's good to hear the support is coming though. Thanks!


Best Regards,

Zach Lanich
Owner/Lead Developer
weCreate LLC
www.WeCreate.com
814.580.6636

> On Aug 17, 2016, at 8:54 AM, Kaushal M  wrote:
> 
> On Wed, Aug 17, 2016 at 5:18 PM, Humble Devassy Chirammal
>  wrote:
>> Hi Zach,
>> 
>>> 
>> Option 1. 3 Gluster nodes, one large volume, divided up into subdirs (1 for
>> each website), mounting the respective subdirs into their containers & using
>> ACLs & LXD’s u/g id maps (mixed feelings about security here)
>>> 
>> 
>> Which version of GlusterFS is in use here ? because gluster sub directory
>> support patch is available in upstream, however  I dont think its in a good
>> state to consume. Yeah, if the subdirectory mount is performed we have to
>> take enough care to make sure the isolation of the mounts between multiple
>> user, ie security is a concern here.
> 
> A correction here. Sub-directory mount support hasn't been merged yet.
> It's still a patch under review.
> 
>> 
>>> 
>> Option 2. 3 Gluster nodes, website-specifc bricks on each, creating
>> website-specific volumes, then mounting those respective volumes into their
>> containers. Example:
>>gnode-1
>>- /data/website1/brick1
>>- /data/website2/brick1
>>gnode-2
>>- /data/website1/brick2
>>- /data/website2/brick2
>>gnode-3
>>- /data/website1/brick3
>>- /data/website2/brick3
>>> 
>> 
>> Yes, this looks to be an ideal or more consumable approach to me.
>> 
>>> 
>> 
>> Option 3. 3 Gluster nodes, every website get’s their own mini “Gluster
>> Cluster” via LXD containers on the Gluster nodes. Example:
>>gnode-1
>>- gcontainer-website1
>>  - /data/brick1
>>- gcontainer-website2
>>  - /data/brick1
>>gnode-2
>>- gcontainer-website1
>>  - /data/brick2
>>- gcontainer-website2
>>  - /data/brick2
>>gnode-3
>>- gcontainer-website1
>>  - /data/brick3
>>- gcontainer-website2
>>  - /data/brick3
>>> 
>> 
>> This is very difficult or complex to achieve and maintain.
>> 
>> In short,  I would vote for option 2.
>> 
>> Also for safer side,  you may need take snapshot of the volumes or configure
>> a backup for these volumes to avoid single point of failure.
>> 
>> Please let me know if you need any details.
>> 
>> --Humble
>> 
>> 
>> 
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster inside containers

[Personal]

Thanks Humble. Re: The single point of failure, would there be a single point 
of failure in a 4 or 6 node Distributed Replicated setup? I still have to wrap 
my head around exactly how many nodes I need for H/A & linear scalability over 
time. 

PS good to hear subdirectory mount support is coming.


Best Regards,

Zach Lanich
Business Owner, Entrepreneur, Creative
Owner/Lead Developer
weCreate LLC
www.WeCreate.com

> On Aug 17, 2016, at 7:48 AM, Humble Devassy Chirammal 
>  wrote:
> 
> Hi Zach, 
> 
> >
> Option 1. 3 Gluster nodes, one large volume, divided up into subdirs (1 for 
> each website), mounting the respective subdirs into their containers & using 
> ACLs & LXD’s u/g id maps (mixed feelings about security here)
> >
> 
> Which version of GlusterFS is in use here ? because gluster sub directory 
> support patch is available in upstream, however  I dont think its in a good 
> state to consume. Yeah, if the subdirectory mount is performed we have to 
> take enough care to make sure the isolation of the mounts between multiple 
> user, ie security is a concern here.
> 
> >
> Option 2. 3 Gluster nodes, website-specifc bricks on each, creating 
> website-specific volumes, then mounting those respective volumes into their 
> containers. Example:
> gnode-1
> - /data/website1/brick1
> - /data/website2/brick1
> gnode-2
> - /data/website1/brick2
> - /data/website2/brick2
> gnode-3
> - /data/website1/brick3
> - /data/website2/brick3
> >
> 
> Yes, this looks to be an ideal or more consumable approach to me.
> 
> >
> 
> Option 3. 3 Gluster nodes, every website get’s their own mini “Gluster 
> Cluster” via LXD containers on the Gluster nodes. Example:
> gnode-1
> - gcontainer-website1
>   - /data/brick1
> - gcontainer-website2
>   - /data/brick1
> gnode-2
> - gcontainer-website1
>   - /data/brick2
> - gcontainer-website2
>   - /data/brick2
> gnode-3
> - gcontainer-website1
>   - /data/brick3
> - gcontainer-website2
>   - /data/brick3
> >
> 
> This is very difficult or complex to achieve and maintain. 
> 
> In short,  I would vote for option 2. 
> 
> Also for safer side,  you may need take snapshot of the volumes or configure 
> a backup for these volumes to avoid single point of failure. 
> 
> Please let me know if you need any details.
> 
> --Humble
> 
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster inside containers

[Kaushal M]

On Wed, Aug 17, 2016 at 5:18 PM, Humble Devassy Chirammal
 wrote:
> Hi Zach,
>
>>
> Option 1. 3 Gluster nodes, one large volume, divided up into subdirs (1 for
> each website), mounting the respective subdirs into their containers & using
> ACLs & LXD’s u/g id maps (mixed feelings about security here)
>>
>
> Which version of GlusterFS is in use here ? because gluster sub directory
> support patch is available in upstream, however  I dont think its in a good
> state to consume. Yeah, if the subdirectory mount is performed we have to
> take enough care to make sure the isolation of the mounts between multiple
> user, ie security is a concern here.

A correction here. Sub-directory mount support hasn't been merged yet.
It's still a patch under review.

>
>>
> Option 2. 3 Gluster nodes, website-specifc bricks on each, creating
> website-specific volumes, then mounting those respective volumes into their
> containers. Example:
> gnode-1
> - /data/website1/brick1
> - /data/website2/brick1
> gnode-2
> - /data/website1/brick2
> - /data/website2/brick2
> gnode-3
> - /data/website1/brick3
> - /data/website2/brick3
>>
>
> Yes, this looks to be an ideal or more consumable approach to me.
>
>>
>
> Option 3. 3 Gluster nodes, every website get’s their own mini “Gluster
> Cluster” via LXD containers on the Gluster nodes. Example:
> gnode-1
> - gcontainer-website1
>   - /data/brick1
> - gcontainer-website2
>   - /data/brick1
> gnode-2
> - gcontainer-website1
>   - /data/brick2
> - gcontainer-website2
>   - /data/brick2
> gnode-3
> - gcontainer-website1
>   - /data/brick3
> - gcontainer-website2
>   - /data/brick3
>>
>
> This is very difficult or complex to achieve and maintain.
>
> In short,  I would vote for option 2.
>
> Also for safer side,  you may need take snapshot of the volumes or configure
> a backup for these volumes to avoid single point of failure.
>
> Please let me know if you need any details.
>
> --Humble
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster inside containers

[Humble Devassy Chirammal]

Hi Zach,

>
*Option 1*. 3 Gluster nodes, one large volume, divided up into subdirs (1
for each website), mounting the respective subdirs into their containers &
using ACLs & LXD’s u/g id maps (mixed feelings about security here)
>

Which version of GlusterFS is in use here ? because gluster sub directory
support patch is available in upstream, however  I dont think its in a good
state to consume. Yeah, if the subdirectory mount is performed we have to
take enough care to make sure the isolation of the mounts between multiple
user, ie security is a concern here.

>
*Option 2*. 3 Gluster nodes, website-specifc bricks on each, creating
website-specific volumes, then mounting those respective volumes into their
containers. Example:
gnode-1
- /data/website1/brick1
- /data/website2/brick1
gnode-2
- /data/website1/brick2
- /data/website2/brick2
gnode-3
- /data/website1/brick3
- /data/website2/brick3
>

Yes, this looks to be an ideal or more consumable approach to me.

>

*Option 3*. 3 Gluster nodes, every website get’s their own mini “Gluster
Cluster” via LXD containers on the Gluster nodes. Example:
gnode-1
- gcontainer-website1
  - /data/brick1
- gcontainer-website2
  - /data/brick1
gnode-2
- gcontainer-website1
  - /data/brick2
- gcontainer-website2
  - /data/brick2
gnode-3
- gcontainer-website1
  - /data/brick3
- gcontainer-website2
  - /data/brick3
>

This is very difficult or complex to achieve and maintain.

In short,  I would vote for option 2.

Also for safer side,  you may need take snapshot of the volumes or
configure a backup for these volumes to avoid single point of failure.

Please let me know if you need any details.

--Humble
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster inside containers

[Zach Lanich]

Hey guys, has anyone had a few mins to look at the aforementioned decision 
dilemmas I’m faced with? :) I also have a couple follow-up questions:

1. Is it possible to change a Replicated (replica3, 3 node) setup to a 
Distributed Replicated (replica 2, 4 node setup)?

2. I’m leaning toward Option #2 in some form as I feel volumes would be better 
separation than subdirectories (correct me if I’m wrong), so is there a good 
way to manage access to separate Gluster volumes? I can’t have the containers 
being able to mount w/e volume they want. One option is to mount the correct 
volume from the top down using lxc device add, but if possible, I might avoid 
that as it sort of breaks the rule of isolation for the containers. Do you 
agree?

3. Is it feasible to resize a set of bricks being used for a Gluster volume, 
should I want to add more HDD space on the already existing nodes? Or am I just 
going about this the wrong way? Would I just create more bricks on those nodes 
and add them to the Gluster volume?

Best Regards,

Zach Lanich
Business Owner, Entrepreneur, Creative
Owner/CTO
weCreate LLC
www.WeCreate.com

> On Aug 16, 2016, at 1:13 PM, Atin Mukherjee  wrote:
> 
> Adding Luis, Humble, Ashiq to comment as they have done some extensive work 
> on this area.
> 
> On Tuesday 16 August 2016, Zach Lanich  > wrote:
> Hey guys, I’m having a real hard time figuring out how to handle my Gluster 
> situation for the web hosting setup I’m working on. Here’s the rundown of 
> what I’m trying to accomplish:
> 
> - Load-balanced web nodes (2 nodes right now), each with multiple LXD 
> containers in them (1 container per website)
> - Gluster vols mounted into the containers (I probably need site-specific 
> volumes, not mounting the same volume into all of them)
> 
> Here are 3 scenarios I’ve come up with for a replica 3 (possibly w/ arbiter):
> 
> Option 1. 3 Gluster nodes, one large volume, divided up into subdirs (1 for 
> each website), mounting the respective subdirs into their containers & using 
> ACLs & LXD’s u/g id maps (mixed feelings about security here)
> 
> Option 2. 3 Gluster nodes, website-specifc bricks on each, creating 
> website-specific volumes, then mounting those respective volumes into their 
> containers. Example:
> gnode-1
> - /data/website1/brick1
> - /data/website2/brick1
> gnode-2
> - /data/website1/brick2
> - /data/website2/brick2
> gnode-3
> - /data/website1/brick3
> - /data/website2/brick3
> 
> Option 3. 3 Gluster nodes, every website get’s their own mini “Gluster 
> Cluster” via LXD containers on the Gluster nodes. Example:
> gnode-1
> - gcontainer-website1
>   - /data/brick1
> - gcontainer-website2
>   - /data/brick1
> gnode-2
> - gcontainer-website1
>   - /data/brick2
> - gcontainer-website2
>   - /data/brick2
> gnode-3
> - gcontainer-website1
>   - /data/brick3
> - gcontainer-website2
>   - /data/brick3
> 
> Where I need help:
> 
> - I don’t know which method is best (or if all 3 are technically possible, 
> though I feel they are)
> 
> My concerns/frustrations:
> 
> - Security
>   - Option 1 - Gives me mixed feelings about putting all customers’ website 
> files on one large volume and mounting subdirs of that volume into the LXD 
> containers, giving the containers R/W to that sub dir using ACLs on the host. 
> Mounting via "lxc device add” supposedly is secure itself, but I’m just not 
> sure here.
> 
> - Performance 
>   - Option 2 - Not sure if Gluster will suffer in any way by using it with 
> say 50 volumes? (one for each customer website)
>   - Option 3 - Not sure if I’m incurring any significant overhead running 
> multiple instances of the Gluster Daemons, etc by creating an isolated 
> Gluster cluster for every customer website. LXD itself is very lightweight, 
> but would this be any worse than running say 50x the FOPs through a single 
> more powerful Gluster cluster?
> 
> - Networking
>   - Option 3 - If all these mini Gluster clusters will be in their own 
> containers, it seems I will have some majorly annoying networking to do. I 
> force a couple ways to do this (and please let me know if you see alt ways):
> - a. Send all Gluster traffic to the Gluster nodes, then use iptables & 
> port forwarding to send traffic to the correct container - Seems like a 
> nightmare. I think I’d have to use different sets ports for every website’s 
> Gluster cluster.
> - b. Bridge the containers to their host’s internal network and assign 
> the containers unique IPs on the host’s network - Much more realistic, but 
> not 100% sure I can do this atm as I’m on Digital Ocean. I know there’s 
> private networking, but I’m not 100% sure I can assign IPs on that network as 
> DO seems to assign the Droplets private IPs automatically. I foresee IP 
> collisions here. If I have to move to a diff provider to do 

Re: [Gluster-users] Gluster inside containers

[Atin Mukherjee]

Adding Luis, Humble, Ashiq to comment as they have done some extensive work
on this area.

On Tuesday 16 August 2016, Zach Lanich  wrote:

> Hey guys, I’m having a real hard time figuring out how to handle my
> Gluster situation for the web hosting setup I’m working on. Here’s the
> rundown of what I’m trying to accomplish:
>
> - Load-balanced web nodes (2 nodes right now), each with multiple LXD
> containers in them (1 container per website)
> - Gluster vols mounted into the containers (I probably need site-specific
> volumes, not mounting the same volume into all of them)
>
> Here are 3 scenarios I’ve come up with for a replica 3 (possibly w/
> arbiter):
>
> *Option 1*. 3 Gluster nodes, one large volume, divided up into subdirs (1
> for each website), mounting the respective subdirs into their containers &
> using ACLs & LXD’s u/g id maps (mixed feelings about security here)
>
> *Option 2*. 3 Gluster nodes, website-specifc bricks on each, creating
> website-specific volumes, then mounting those respective volumes into their
> containers. Example:
> gnode-1
> - /data/website1/brick1
> - /data/website2/brick1
> gnode-2
> - /data/website1/brick2
> - /data/website2/brick2
> gnode-3
> - /data/website1/brick3
> - /data/website2/brick3
>
> *Option 3*. 3 Gluster nodes, every website get’s their own mini “Gluster
> Cluster” via LXD containers on the Gluster nodes. Example:
> gnode-1
> - gcontainer-website1
>   - /data/brick1
> - gcontainer-website2
>   - /data/brick1
> gnode-2
> - gcontainer-website1
>   - /data/brick2
> - gcontainer-website2
>   - /data/brick2
> gnode-3
> - gcontainer-website1
>   - /data/brick3
> - gcontainer-website2
>   - /data/brick3
>
> *Where I need help:*
>
> - I don’t know which method is best (or if all 3 are technically possible,
> though I feel they are)
>
> *My concerns/frustrations:*
>
> - *Security*
>   - Option 1 - Gives me mixed feelings about putting all customers’
> website files on one large volume and mounting subdirs of that volume into
> the LXD containers, giving the containers R/W to that sub dir using ACLs on
> the host. Mounting via "lxc device add” supposedly is secure itself, but
> I’m just not sure here.
>
> - *Performance *
>   - Option 2 - Not sure if Gluster will suffer in any way by using it with
> say 50 volumes? (one for each customer website)
>   - Option 3 - Not sure if I’m incurring any significant overhead running
> multiple instances of the Gluster Daemons, etc by creating an isolated
> Gluster cluster for every customer website. LXD itself is very lightweight,
> but would this be any worse than running say 50x the FOPs through a single
> more powerful Gluster cluster?
>
> - *Networking*
>   - Option 3 - If all these mini Gluster clusters will be in their own
> containers, it seems I will have some majorly annoying networking to do. I
> force a couple ways to do this (and please let me know if you see alt ways):
> - a. Send all Gluster traffic to the Gluster nodes, then use iptables
> & port forwarding to send traffic to the correct container - Seems like a
> nightmare. I think I’d have to use different sets ports for every website’s
> Gluster cluster.
> - b. Bridge the containers to their host’s internal network and assign
> the containers unique IPs on the host’s network - Much more realistic, but
> not 100% sure I can do this atm as I’m on Digital Ocean. I know there’s
> private networking, but I’m not 100% sure I can assign IPs on that network
> as DO seems to assign the Droplets private IPs automatically. I foresee IP
> collisions here. If I have to move to a diff provider to do this, then so
> be it, but I like the SSDs :)
>
> I’d appreciate help on this as I’ma bit in over my head, but extremely
> eager to figure this out and make it happen. I’m not 100% aware of the
> Security/Performance/Networking implications are for the above decisions
> and I need an expert so I don’t go too far off in left field.
>
> Best Regards,
>
> Zach Lanich
> *Business Owner, Entrepreneur, Creative*
> *Owner/CTO*
> weCreate LLC
> *www.WeCreate.com *
>
>

-- 
--Atin
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster inside containers

[Zach Lanich]

Hey guys, I’m having a real hard time figuring out how to handle my Gluster 
situation for the web hosting setup I’m working on. Here’s the rundown of what 
I’m trying to accomplish:

- Load-balanced web nodes (2 nodes right now), each with multiple LXD 
containers in them (1 container per website)
- Gluster vols mounted into the containers (I probably need site-specific 
volumes, not mounting the same volume into all of them)

Here are 3 scenarios I’ve come up with for a replica 3 (possibly w/ arbiter):

Option 1. 3 Gluster nodes, one large volume, divided up into subdirs (1 for 
each website), mounting the respective subdirs into their containers & using 
ACLs & LXD’s u/g id maps (mixed feelings about security here)

Option 2. 3 Gluster nodes, website-specifc bricks on each, creating 
website-specific volumes, then mounting those respective volumes into their 
containers. Example:
gnode-1
- /data/website1/brick1
- /data/website2/brick1
gnode-2
- /data/website1/brick2
- /data/website2/brick2
gnode-3
- /data/website1/brick3
- /data/website2/brick3

Option 3. 3 Gluster nodes, every website get’s their own mini “Gluster Cluster” 
via LXD containers on the Gluster nodes. Example:
gnode-1
- gcontainer-website1
  - /data/brick1
- gcontainer-website2
  - /data/brick1
gnode-2
- gcontainer-website1
  - /data/brick2
- gcontainer-website2
  - /data/brick2
gnode-3
- gcontainer-website1
  - /data/brick3
- gcontainer-website2
  - /data/brick3

Where I need help:

- I don’t know which method is best (or if all 3 are technically possible, 
though I feel they are)

My concerns/frustrations:

- Security
  - Option 1 - Gives me mixed feelings about putting all customers’ website 
files on one large volume and mounting subdirs of that volume into the LXD 
containers, giving the containers R/W to that sub dir using ACLs on the host. 
Mounting via "lxc device add” supposedly is secure itself, but I’m just not 
sure here.

- Performance 
  - Option 2 - Not sure if Gluster will suffer in any way by using it with say 
50 volumes? (one for each customer website)
  - Option 3 - Not sure if I’m incurring any significant overhead running 
multiple instances of the Gluster Daemons, etc by creating an isolated Gluster 
cluster for every customer website. LXD itself is very lightweight, but would 
this be any worse than running say 50x the FOPs through a single more powerful 
Gluster cluster?

- Networking
  - Option 3 - If all these mini Gluster clusters will be in their own 
containers, it seems I will have some majorly annoying networking to do. I 
force a couple ways to do this (and please let me know if you see alt ways):
- a. Send all Gluster traffic to the Gluster nodes, then use iptables & 
port forwarding to send traffic to the correct container - Seems like a 
nightmare. I think I’d have to use different sets ports for every website’s 
Gluster cluster.
- b. Bridge the containers to their host’s internal network and assign the 
containers unique IPs on the host’s network - Much more realistic, but not 100% 
sure I can do this atm as I’m on Digital Ocean. I know there’s private 
networking, but I’m not 100% sure I can assign IPs on that network as DO seems 
to assign the Droplets private IPs automatically. I foresee IP collisions here. 
If I have to move to a diff provider to do this, then so be it, but I like the 
SSDs :)

I’d appreciate help on this as I’ma bit in over my head, but extremely eager to 
figure this out and make it happen. I’m not 100% aware of the 
Security/Performance/Networking implications are for the above decisions and I 
need an expert so I don’t go too far off in left field.

Best Regards,

Zach Lanich
Business Owner, Entrepreneur, Creative
Owner/CTO
weCreate LLC
www.WeCreate.com

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users