Re: [ceph-users] private network - VLAN vs separate switch
Thanks for all the help. We will follow the more careful approach! -Sreenath On 11/26/14, Kyle Bader wrote: >> Thanks for all the help. Can the moving over from VLAN to separate >> switches be done on a live cluster? Or does there need to be a down >> time? > > You can do it on a life cluster. The more cavalier approach would be > to quickly switch the link over one server at a time, which might > cause a short io stall. The more careful approach would be to 'ceph > osd set noup' mark all the osds on a node down, move the link, 'ceph > osd unset noup', and then wait for their peers to mark them back up > before proceeding to the next host. > > -- > > Kyle > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] private network - VLAN vs separate switch
> Thanks for all the help. Can the moving over from VLAN to separate > switches be done on a live cluster? Or does there need to be a down > time? You can do it on a life cluster. The more cavalier approach would be to quickly switch the link over one server at a time, which might cause a short io stall. The more careful approach would be to 'ceph osd set noup' mark all the osds on a node down, move the link, 'ceph osd unset noup', and then wait for their peers to mark them back up before proceeding to the next host. -- Kyle ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] private network - VLAN vs separate switch
Thanks for all the help. Can the moving over from VLAN to separate switches be done on a live cluster? Or does there need to be a down time? -Sreenath On 11/26/14, Kyle Bader wrote: >> For a large network (say 100 servers and 2500 disks), are there any >> strong advantages to using separate switch and physical network >> instead of VLAN? > > Physical isolation will ensure that congestion on one does not affect > the other. On the flip side, asymmetric network failures tend to be > more difficult to troubleshoot eg. backend failure with functional > front end. That said, in a pinch you can switch to using the front end > network for both until you can repair the backend. > >> Also, how difficult it would be to switch from a VLAN to using >> separate switches later? > > Should be relatively straight forward. Simply configure the > VLAN/subnets on the new physical switches and move links over one by > one. Once all the links are moved over you can remove the VLAN and > subnets that are now on the new kit from the original hardware. > > -- > > Kyle > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] private network - VLAN vs separate switch
> For a large network (say 100 servers and 2500 disks), are there any > strong advantages to using separate switch and physical network > instead of VLAN? Physical isolation will ensure that congestion on one does not affect the other. On the flip side, asymmetric network failures tend to be more difficult to troubleshoot eg. backend failure with functional front end. That said, in a pinch you can switch to using the front end network for both until you can repair the backend. > Also, how difficult it would be to switch from a VLAN to using > separate switches later? Should be relatively straight forward. Simply configure the VLAN/subnets on the new physical switches and move links over one by one. Once all the links are moved over you can remove the VLAN and subnets that are now on the new kit from the original hardware. -- Kyle ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] private network - VLAN vs separate switch
Hi, From my humble opinion if you have enough money, separate switches are always a better choice. Regrads, I 2014-11-25 20:47 GMT+01:00 Craig Lewis : > It's mostly about bandwidth. With VLANs, the public and cluster networks > are going to be sharing the inter-switch links. > > For a cluster that size, I don't see much advantage to the VLANs. You'll > save a few ports by having the inter-switch links shared, at the expense of > contention on those links. > > If you're trying to save ports, I'd go with a single network. Adding a > cluster network later is relatively straight forward. Just monitor the > bandwidth on the inter-switch links, and plan to expand when you saturate > those links. > > > That said, I am using VLANs, but my cluster is much smaller. I only have > 5 nodes and a single switch. I'm planning to transition to a dedicated > cluster switch when I need the extra ports. I don't anticipate the > transition being difficult. I'll continue to use the same VLAN on the > dedicated switch, just to make the migration less complicated. > > > On Tue, Nov 25, 2014 at 3:11 AM, Sreenath BH wrote: > >> Hi >> For a large network (say 100 servers and 2500 disks), are there any >> strong advantages to using separate switch and physical network >> instead of VLAN? >> >> Also, how difficult it would be to switch from a VLAN to using >> separate switches later? >> >> -Sreenath >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- Iban Cabrillo Bartolome Instituto de Fisica de Cantabria (IFCA) Santander, Spain Tel: +34942200969 PGP PUBLIC KEY: http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC Bertrand Russell: *"El problema con el mundo es que los estúpidos están seguros de todo y los inteligentes están llenos de dudas*" ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] private network - VLAN vs separate switch
It's mostly about bandwidth. With VLANs, the public and cluster networks are going to be sharing the inter-switch links. For a cluster that size, I don't see much advantage to the VLANs. You'll save a few ports by having the inter-switch links shared, at the expense of contention on those links. If you're trying to save ports, I'd go with a single network. Adding a cluster network later is relatively straight forward. Just monitor the bandwidth on the inter-switch links, and plan to expand when you saturate those links. That said, I am using VLANs, but my cluster is much smaller. I only have 5 nodes and a single switch. I'm planning to transition to a dedicated cluster switch when I need the extra ports. I don't anticipate the transition being difficult. I'll continue to use the same VLAN on the dedicated switch, just to make the migration less complicated. On Tue, Nov 25, 2014 at 3:11 AM, Sreenath BH wrote: > Hi > For a large network (say 100 servers and 2500 disks), are there any > strong advantages to using separate switch and physical network > instead of VLAN? > > Also, how difficult it would be to switch from a VLAN to using > separate switches later? > > -Sreenath > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] private network - VLAN vs separate switch
Hi For a large network (say 100 servers and 2500 disks), are there any strong advantages to using separate switch and physical network instead of VLAN? Also, how difficult it would be to switch from a VLAN to using separate switches later? -Sreenath ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
