Re: [j-nsp] qfabric 1536k mac address?
On Friday, January 03, 2014 09:09:53 AM giovanni rana wrote: > Well, thanks a lot eugeniu, got it now, it's not easy to > change your mind about L2 mechanism because they haven't > been changed for years within standard switches. Tell me what you think when you come across Cisco's EVC solution :-). Mark. signature.asc Description: This is a digitally signed message part. ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] qfabric 1536k mac address?
Well, thanks a lot eugeniu, got it now, it's not easy to change your mind about L2 mechanism because they haven't been changed for years within standard switches. Date: Thu, 2 Jan 2014 23:45:29 +0200 Subject: Re: [j-nsp] qfabric 1536k mac address? From: eu...@imacandi.net To: superburri...@hotmail.com CC: juniper-nsp@puck.nether.net On Thu, Jan 2, 2014 at 11:18 PM, giovanni rana wrote: Ahahah that was just an example ;)) And of course i can put another standard L2 switch between the hosts and the qfx3500 which aggregates all the hosts and goes to the qfx3500 with a single 10ge port, if I don't mind about oversubscr. ratio, but this is not about design, it's about understanding how and where those 1536k Mac addressees are indexed, or it is just marketing? please take the "The QFabric Architecture - Juniper Networks" document, go to page 21, figure 17. i got 2 nodes now, node A and node B. i got also 131072 VMs with unique MAC addresses in my single /15 subnet because i like it big, VMs' are equally splitted between the nodeA and nodeB, so 65536 VMs are attached on nodeA and 65536 on nodeB. Let's forget clusters, ESXi limitations and so on. the table on Qfabric nodeB is made like in the figure 17:MAC,VLAN; NEXTHOPmac1,VLAN1; QFabric nodeA (remote VM attached on nodeA)mac2,VLAN1; QFabric nodeA mac3,VLAN1; QFabric nodeA ..mac65536,VLAN1;QFabric nodeAmac65537,VLAN1; port P (locally attached VM) mac65538,VLAN1; port P mac65539,VLAN1; port P...mac128000,VLAN1;port Pwhat's next? the qfx3500 CAM is exhausted now, how can i handle mac128001mac131072? adding nodes or changing the number of VLAN's does not make me understand how can i overcome the 128K limitation, or better, i'm missing how this "information hiding" principle works. from the document, it seems that all the entries from mac1 to mac65536 which are all pointing to nodeA can be aggregated/merged all together, because they can point to a remote unique node instead to a local port, but this doesn't mean they take less entries (or less "space") in the mac address table compared to normal entries. It doesn't get exhausted. This is something you fail to understand: The control node keeps track of which MAC where it belongs and the switch will forward the packet to the spine if it does not know about the MAC locally. You only need to ensure that you don't go over 128.000 MAC addresses locally on the QFX switch, that's all. Think of it like routing: when you don't have a more specific route to a destination you send all your packets to the default gateway and the default gateway will try to figure out where to send the packets. Look again at the link I gave you previously and it's clearly explained there how it works. Eugniu ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] qfabric 1536k mac address?
On Thu, Jan 2, 2014 at 11:18 PM, giovanni rana wrote: > Ahahah that was just an example ;)) And of course i can put another > standard L2 switch between the hosts and the qfx3500 which aggregates all > the hosts and goes to the qfx3500 with a single 10ge port, if I don't mind > about oversubscr. ratio, but this is not about design, it's about > understanding how and where those 1536k Mac addressees are indexed, or it > is just marketing? > > please take the "The QFabric Architecture - Juniper Networks" document, > go to page 21, figure 17. i got 2 nodes now, node A and node B. i got also > 131072 VMs with unique MAC addresses in my single /15 subnet because i like > it big, VMs' are equally splitted between the nodeA and nodeB, so 65536 VMs > are attached on nodeA and 65536 on nodeB. Let's forget clusters, ESXi > limitations and so on. > > the table on Qfabric nodeB is made like in the figure 17: > MAC,VLAN; NEXTHOP > mac1,VLAN1; QFabric nodeA (remote VM attached on nodeA) > mac2,VLAN1; QFabric nodeA > mac3,VLAN1; QFabric nodeA > . > . > mac65536,VLAN1;QFabric nodeA > mac65537,VLAN1; port P (locally attached VM) > mac65538,VLAN1; port P > mac65539,VLAN1; port P > . > . > . > mac128000,VLAN1;port P > what's next? the qfx3500 CAM is exhausted now, how can i handle > mac128001mac131072? > > adding nodes or changing the number of VLAN's does not make me understand > how can i overcome the 128K limitation, or better, i'm missing how this > "information hiding" principle works. from the document, it seems that all > the entries from mac1 to mac65536 which are all pointing to nodeA can be > aggregated/merged all together, because they can point to a remote unique > node instead to a local port, but this doesn't mean they take less entries > (or less "space") in the mac address table compared to normal entries. > > It doesn't get exhausted. This is something you fail to understand: The control node keeps track of which MAC where it belongs and the switch will forward the packet to the spine if it does not know about the MAC locally. You only need to ensure that you don't go over 128.000 MAC addresses locally on the QFX switch, that's all. Think of it like routing: when you don't have a more specific route to a destination you send all your packets to the default gateway and the default gateway will try to figure out where to send the packets. Look again at the link I gave you previously and it's clearly explained there how it works. Eugniu ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] qfabric 1536k mac address?
Ahahah that was just an example ;)) And of course i can put another standard L2 switch between the hosts and the qfx3500 which aggregates all the hosts and goes to the qfx3500 with a single 10ge port, if I don't mind about oversubscr. ratio, but this is not about design, it's about understanding how and where those 1536k Mac addressees are indexed, or it is just marketing? please take the "The QFabric Architecture - Juniper Networks" document, go to page 21, figure 17. i got 2 nodes now, node A and node B. i got also 131072 VMs with unique MAC addresses in my single /15 subnet because i like it big, VMs' are equally splitted between the nodeA and nodeB, so 65536 VMs are attached on nodeA and 65536 on nodeB. Let's forget clusters, ESXi limitations and so on. the table on Qfabric nodeB is made like in the figure 17:MAC,VLAN; NEXTHOPmac1,VLAN1; QFabric nodeA (remote VM attached on nodeA)mac2,VLAN1; QFabric nodeAmac3,VLAN1; QFabric nodeA ..mac65536,VLAN1;QFabric nodeAmac65537,VLAN1; port P (locally attached VM)mac65538,VLAN1; port P mac65539,VLAN1; port P...mac128000,VLAN1;port Pwhat's next? the qfx3500 CAM is exhausted now, how can i handle mac128001mac131072? adding nodes or changing the number of VLAN's does not make me understand how can i overcome the 128K limitation, or better, i'm missing how this "information hiding" principle works. from the document, it seems that all the entries from mac1 to mac65536 which are all pointing to nodeA can be aggregated/merged all together, because they can point to a remote unique node instead to a local port, but this doesn't mean they take less entries (or less "space") in the mac address table compared to normal entries. thanks! Date: Thu, 2 Jan 2014 20:27:54 +0200 Subject: Re: [j-nsp] qfabric 1536k mac address? From: eu...@imacandi.net To: superburri...@hotmail.com CC: bd...@comlinx.com.au; juniper-nsp@puck.nether.net On Thu, Jan 2, 2014 at 8:10 PM, giovanni rana wrote: I do, but how big is my DC is not relevant, like being flat or non flat does not matter...Since the data sheet clearly says 1.536.000 Mac addresses are supported, I need to understand if we are talking of unique Mac addresses or that's a number made by using mac-vlan pairs (and so Overlapping MACs) as described in "The QFabric Architecture - Juniper Networks" PDF. Let's make it simple, qfx3500 has a 128k Mac table. I've attached some big esxi clusters on my qfabric and now I got 129k VMs each one with unique Mac address, will I face any problem? Of course you will face a problem as one single switch can only handle 128k MAC addresses. The whole QFabric system can handle 1.5M MAC addresses but only if you have a full buildout of leaves and spines, not just a single switch. And just for fun, if you do the math, you won't be able to cram anywhere near 128K ESXi VMs on QFX3500 switch as you have usable 48 10GbE ports (assuming the maximum). You can have a maximum 512 VMs per host or 4000 VMs per 32 host cluster under vSphere 5.5. This gives you 2 clusters available (1 of 32 hosts and another of 16 hosts). This 8000 VMs maximum with two VMware clusters. Even if you would go with 48 independent ESXi hosts and each fully loaded with 512 VMs you would get a maximum of 24576 MAC addresses from them + 48 from the hosts. I think you need to rethink your scenario or ask what you really want to ask. Eugeniu ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] qfabric 1536k mac address?
On Thu, Jan 2, 2014 at 8:10 PM, giovanni rana wrote: > I do, but how big is my DC is not relevant, like being flat or non flat > does not matter...Since the data sheet clearly says 1.536.000 Mac addresses > are supported, I need to understand if we are talking of unique Mac > addresses or that's a number made by using mac-vlan pairs (and so > Overlapping MACs) as described in "The QFabric Architecture - Juniper > Networks" PDF. > Let's make it simple, qfx3500 has a 128k Mac table. I've attached some big > esxi clusters on my qfabric and now I got 129k VMs each one with unique Mac > address, will I face any problem? > > Of course you will face a problem as one single switch can only handle 128k MAC addresses. The whole QFabric system can handle 1.5M MAC addresses but only if you have a full buildout of leaves and spines, not just a single switch. And just for fun, if you do the math, you won't be able to cram anywhere near 128K ESXi VMs on QFX3500 switch as you have usable 48 10GbE ports (assuming the maximum). You can have a maximum 512 VMs per host or 4000 VMs per 32 host cluster under vSphere 5.5. This gives you 2 clusters available (1 of 32 hosts and another of 16 hosts). This 8000 VMs maximum with two VMware clusters. Even if you would go with 48 independent ESXi hosts and each fully loaded with 512 VMs you would get a maximum of 24576 MAC addresses from them + 48 from the hosts. I think you need to rethink your scenario or ask what you really want to ask. Eugeniu ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] qfabric 1536k mac address?
I do, but how big is my DC is not relevant, like being flat or non flat does not matter...Since the data sheet clearly says 1.536.000 Mac addresses are supported, I need to understand if we are talking of unique Mac addresses or that's a number made by using mac-vlan pairs (and so Overlapping MACs) as described in "The QFabric Architecture - Juniper Networks" PDF. Let's make it simple, qfx3500 has a 128k Mac table. I've attached some big esxi clusters on my qfabric and now I got 129k VMs each one with unique Mac address, will I face any problem? Date: Thu, 2 Jan 2014 19:10:06 +0200 Subject: Re: [j-nsp] qfabric 1536k mac address? From: eu...@imacandi.net To: superburri...@hotmail.com CC: bd...@comlinx.com.au; juniper-nsp@puck.nether.net On Thu, Jan 2, 2014 at 4:20 PM, giovanni rana wrote: Even in the case you mentioned the node shall be able to keep a table where there's an index Made by 1536k entries. I can understand that some memory can be saved by using a vpls style approach, but if I got 1536k VMs with unique and Mac addresses I'm still able to manage them via qfabric? Public docs does not clarify enough this aspects. Thanks for your answer! You do realise that you are talking about approximately 1,5000,000 MAC addresses in your network in a flat any-to-any topology and not about 1,500 or 15,000 MAC addresses ? This is how QFabric does MAC learning and forwarding: http://blog.ipspace.net/2011/09/qfabric-part-3-forwarding.html Regards,Eugeniu ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] qfabric 1536k mac address?
If the QFabric controllers can do anything it's scale up the number of MACs that can be learned across the fabric. Their solution was novel before SDN and I guess it's to their credit that they are not trying to market the QFabric that way now ;) rgds, --r On Jan 2, 2014, at 9:10 AM, Eugeniu Patrascu wrote: > On Thu, Jan 2, 2014 at 4:20 PM, giovanni rana > wrote: > >> Even in the case you mentioned the node shall be able to keep a table >> where there's an index Made by 1536k entries. I can understand that some >> memory can be saved by using a vpls style approach, but if I got 1536k VMs >> with unique and Mac addresses I'm still able to manage them via qfabric? >> Public docs does not clarify enough this aspects. Thanks for your answer! >> >> > You do realise that you are talking about approximately 1,5000,000 MAC > addresses in your network in a flat any-to-any topology and not about 1,500 > or 15,000 MAC addresses ? > > This is how QFabric does MAC learning and forwarding: > http://blog.ipspace.net/2011/09/qfabric-part-3-forwarding.html > > Regards, > Eugeniu > ___ > juniper-nsp mailing list juniper-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] qfabric 1536k mac address?
On Thu, Jan 2, 2014 at 4:20 PM, giovanni rana wrote: > Even in the case you mentioned the node shall be able to keep a table > where there's an index Made by 1536k entries. I can understand that some > memory can be saved by using a vpls style approach, but if I got 1536k VMs > with unique and Mac addresses I'm still able to manage them via qfabric? > Public docs does not clarify enough this aspects. Thanks for your answer! > > You do realise that you are talking about approximately 1,5000,000 MAC addresses in your network in a flat any-to-any topology and not about 1,500 or 15,000 MAC addresses ? This is how QFabric does MAC learning and forwarding: http://blog.ipspace.net/2011/09/qfabric-part-3-forwarding.html Regards, Eugeniu ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] qfabric 1536k mac address?
Even in the case you mentioned the node shall be able to keep a table where there's an index Made by 1536k entries. I can understand that some memory can be saved by using a vpls style approach, but if I got 1536k VMs with unique and Mac addresses I'm still able to manage them via qfabric? Public docs does not clarify enough this aspects. Thanks for your answer! > Subject: Re: [j-nsp] qfabric 1536k mac address? > From: bd...@comlinx.com.au > Date: Thu, 2 Jan 2014 18:39:51 +1000 > CC: juniper-nsp@puck.nether.net > To: superburri...@hotmail.com > > At a guess I'd say it's because in Q-Fabric, the "MAC-learning" is performed > by a similar to a VPLS instance per VLAN rather than traditional per-switch > CAM tables. > > I imagine this allows for better scalability/optimisation. > > The switch CAM would then only need to worry about L2-adjacent QF nodes and > adjacent hosts. > > On 2 Jan 2014, at 5:28 pm, giovanni rana wrote: > > > Hi, during the evaluation of a new data center architecture, how can > > qfabric support so many Mac addresses? The qfx3500 node supports only 128k > > Mac tables, how can they get 1536k as declared in the data sheet? Is it > > because of an optimized use of mac-vlan pairs? > > Thanks for any answer and happy 2014! > > > > ___ > > juniper-nsp mailing list juniper-nsp@puck.nether.net > > https://puck.nether.net/mailman/listinfo/juniper-nsp > > > ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] qfabric 1536k mac address?
At a guess I'd say it's because in Q-Fabric, the "MAC-learning" is performed by a similar to a VPLS instance per VLAN rather than traditional per-switch CAM tables. I imagine this allows for better scalability/optimisation. The switch CAM would then only need to worry about L2-adjacent QF nodes and adjacent hosts. On 2 Jan 2014, at 5:28 pm, giovanni rana wrote: > Hi, during the evaluation of a new data center architecture, how can qfabric > support so many Mac addresses? The qfx3500 node supports only 128k Mac > tables, how can they get 1536k as declared in the data sheet? Is it because > of an optimized use of mac-vlan pairs? > Thanks for any answer and happy 2014! > > ___ > juniper-nsp mailing list juniper-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/juniper-nsp > ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
