psc wrote: > Thank you all for the answers. Would you guys please share with me some > good brands of those > 200+ 1GB Ethernet switches? I think I'll leave our current clusters > alone , but the new cluster I > will design for about 500 to 1000 nodes --- I don't think that we will > go much above since for big jobs > our scientists using outside resources. We do all our calculations and > analysis on the nodes and only the final produce > we sent to the frontend , also we don't run jobs across the nodes , so I > don't need to get too much creative with the network > beside being sure that I can expand the cluster without having the > switches as a limitation (our current situation) > > thank you again! > > > Henning Fehrmann wrote: >> Hi >> >> On Wed, Sep 09, 2009 at 03:23:30PM -0400, psc wrote: >> >>> I wonder what would be the sensible biggest cluster possible based on >>> 1GB Ethernet network . >>> >> Hmmm, may I cheat and use a 10Gb core switch? >> >> If you setup a cluster with few thousand nodes you have to ask yourself >> whether this network should be non-blocking or not. >> >> For a non blocking network you need the right core-switch technology. >> Unfortunately, there are not many vendors out there which provide >> non-blocking Ethernet based core switches but I am aware of at least >> two. One provides or will provide 144 10Gb Ethernet ports. Another one >> sells switches with more than 1000 1 GB ports. >> You could buy edge-switches with 4 10Gb uplinks and 48 1GB ports. If >> you just use 40 of them you end up with a 1440 non-blocking 1Gb ports. >> >> It might be also possible to cross connect two of these core-switches >> with the help of some smaller switches so that one ends up with 288 >> 10Gb ports and, in principle, one might connect 2880 nodes in a >> non-blocking way, but we did not have the possibility to test it >> successfully yet. One of problems is that the internal hash table can >> not store that many mac addresses. Anyway, one probably needs to change >> the mac addresses of the nodes to avoid an overflow of the hash tables. >> An overflow might cause arp storms. >> >> Once this works one runs into some smaller problems. One of them is the arp >> cache of the nodes. It should be adjusted to hold as many mac addresses >> as you have nodes in the cluster. >> >> >> >>> And especially how would you connect those 1GB >>> switches together -- now we have (on one of our four clusters) Two 48 >>> ports gigabit switches connected together with 6 patch cables and I just >>> ran out of ports for expansion and wonder where to go from here as we >>> already have four clusters and it would be great to stop adding cluster >>> and start expending them beyond number of outlets on the switch/s .... >>> NFS and 1GB Ethernet works great for us and we want to stick with it , >>> but we would love to find a way how to overcome the current "switch >>> limitation". >>> >> With NFS you can nicely test the setup. Use one NFS server and let all >> nodes write different files into it and look what happens. >> >> Cheers, >> Henning >> > > _______________________________________________ > Beowulf mailing list, [email protected] sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf
It looks like Allied Telesis makes chassis switches now too. -- Geoffrey D. Jacobs _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
