Wired Intelligent Edge

Hi everyone,

We have two Spine & Leaf fabrics located in two different sites. The sites are connected via Border Leader Leafs (two VSX stacks with four links). The setup itself is working fine, but due to the way VNIs, VLANs, and EVPN are implemented on Aruba switches, I cannot configure VNIs on the Border Leader Leafs without also configuring VLANs.
For example:

vlan 31
    vsx-sync
    name TEST
evpn
    dyn-vxlan-tunnel-bridging-mode ibgp-ebgp
    vlan 31
        rd 10.100.x.2:10031
        route-target export 65101:10031
        route-target import 65101:10031
interface vxlan 1
    source ip 10.100.x.2
    no shutdown
    vni 10030
        vlan 31

So on the Border Leader Leafs I cannot configure more than 4096 VNIs, because I cannot have more than 4096 VLANs on a single switch.

Now we are onboarding some "large" customers. These customers can have up to 3000 of their own VLANs. They get dedicated leaf switches. Within a single site everything works fine: the customers use their own VNI pools (for example 20000–25000 and 30000–35000), and their VLANs exist only on their own leafs.

The problem appears when such a "large" customer needs connectivity in both sites. In that case, I would need to configure more than 6000 VLANs on the Border Leader Leafs, which is not possible.

When I designed the two-site setup, I followed this documentation:
https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/HTML/vxlan/Content/Chp_EV-VX-MF/ex_intro.htm

So the question is: how do you transport more than 6000 VNIs (VLANs) between two sites in an Aruba EVPN Multisite design?

Any examples or references would be highly appreciated. I assume I'm not the first one running into this limitation

You could just ensure to have routing between the VTEP's on both sides. You don't need to terminate all VLAN's in the border switch. 

Hi everyone,

We have two Spine & Leaf fabrics located in two different sites. The sites are connected via Border Leader Leafs (two VSX stacks with four links). The setup itself is working fine, but due to the way VNIs, VLANs, and EVPN are implemented on Aruba switches, I cannot configure VNIs on the Border Leader Leafs without also configuring VLANs.
For example:

vlan 31
    vsx-sync
    name TEST
evpn
    dyn-vxlan-tunnel-bridging-mode ibgp-ebgp
    vlan 31
        rd 10.100.x.2:10031
        route-target export 65101:10031
        route-target import 65101:10031
interface vxlan 1
    source ip 10.100.x.2
    no shutdown
    vni 10030
        vlan 31

So on the Border Leader Leafs I cannot configure more than 4096 VNIs, because I cannot have more than 4096 VLANs on a single switch.

Now we are onboarding some "large" customers. These customers can have up to 3000 of their own VLANs. They get dedicated leaf switches. Within a single site everything works fine: the customers use their own VNI pools (for example 20000–25000 and 30000–35000), and their VLANs exist only on their own leafs.

The problem appears when such a "large" customer needs connectivity in both sites. In that case, I would need to configure more than 6000 VLANs on the Border Leader Leafs, which is not possible.

When I designed the two-site setup, I followed this documentation:
https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/HTML/vxlan/Content/Chp_EV-VX-MF/ex_intro.htm

So the question is: how do you transport more than 6000 VNIs (VLANs) between two sites in an Aruba EVPN Multisite design?

Any examples or references would be highly appreciated. I assume I'm not the first one running into this limitation

Could you please clarify what you mean?
Within each site (and fabric) we are running iBGP, and eBGP is used between the fabrics. Naturally, the regular leaf switches (VTEPs) in different fabrics have routes to each other's Lo0 and Lo1.

You could just ensure to have routing between the VTEP's on both sides. You don't need to terminate all VLAN's in the border switch. 

Hi everyone,

We have two Spine & Leaf fabrics located in two different sites. The sites are connected via Border Leader Leafs (two VSX stacks with four links). The setup itself is working fine, but due to the way VNIs, VLANs, and EVPN are implemented on Aruba switches, I cannot configure VNIs on the Border Leader Leafs without also configuring VLANs.
For example:

vlan 31
    vsx-sync
    name TEST
evpn
    dyn-vxlan-tunnel-bridging-mode ibgp-ebgp
    vlan 31
        rd 10.100.x.2:10031
        route-target export 65101:10031
        route-target import 65101:10031
interface vxlan 1
    source ip 10.100.x.2
    no shutdown
    vni 10030
        vlan 31

So on the Border Leader Leafs I cannot configure more than 4096 VNIs, because I cannot have more than 4096 VLANs on a single switch.

Now we are onboarding some "large" customers. These customers can have up to 3000 of their own VLANs. They get dedicated leaf switches. Within a single site everything works fine: the customers use their own VNI pools (for example 20000–25000 and 30000–35000), and their VLANs exist only on their own leafs.

The problem appears when such a "large" customer needs connectivity in both sites. In that case, I would need to configure more than 6000 VLANs on the Border Leader Leafs, which is not possible.

When I designed the two-site setup, I followed this documentation:
https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/HTML/vxlan/Content/Chp_EV-VX-MF/ex_intro.htm

So the question is: how do you transport more than 6000 VNIs (VLANs) between two sites in an Aruba EVPN Multisite design?

Any examples or references would be highly appreciated. I assume I'm not the first one running into this limitation

Just like you don't have same VID for different customers on one regular leaf, if you don't have anything ingress or egress on the VLAN you don't need the VLAN and VNI on the border leaf. Just ensure the routing is correct.

Think of it a bit like a spine - how does the VXLAN go from leaf to leaf when no vlan / vni is on the spine?  

Could you please clarify what you mean?
Within each site (and fabric) we are running iBGP, and eBGP is used between the fabrics. Naturally, the regular leaf switches (VTEPs) in different fabrics have routes to each other's Lo0 and Lo1.

You could just ensure to have routing between the VTEP's on both sides. You don't need to terminate all VLAN's in the border switch. 

Hi everyone,

We have two Spine & Leaf fabrics located in two different sites. The sites are connected via Border Leader Leafs (two VSX stacks with four links). The setup itself is working fine, but due to the way VNIs, VLANs, and EVPN are implemented on Aruba switches, I cannot configure VNIs on the Border Leader Leafs without also configuring VLANs.
For example:

vlan 31
    vsx-sync
    name TEST
evpn
    dyn-vxlan-tunnel-bridging-mode ibgp-ebgp
    vlan 31
        rd 10.100.x.2:10031
        route-target export 65101:10031
        route-target import 65101:10031
interface vxlan 1
    source ip 10.100.x.2
    no shutdown
    vni 10030
        vlan 31

So on the Border Leader Leafs I cannot configure more than 4096 VNIs, because I cannot have more than 4096 VLANs on a single switch.

Now we are onboarding some "large" customers. These customers can have up to 3000 of their own VLANs. They get dedicated leaf switches. Within a single site everything works fine: the customers use their own VNI pools (for example 20000–25000 and 30000–35000), and their VLANs exist only on their own leafs.

The problem appears when such a "large" customer needs connectivity in both sites. In that case, I would need to configure more than 6000 VLANs on the Border Leader Leafs, which is not possible.

When I designed the two-site setup, I followed this documentation:
https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/HTML/vxlan/Content/Chp_EV-VX-MF/ex_intro.htm

So the question is: how do you transport more than 6000 VNIs (VLANs) between two sites in an Aruba EVPN Multisite design?

Any examples or references would be highly appreciated. I assume I'm not the first one running into this limitation

Wait, I'm a bit confused.
I also used to think that VNIs and VLANs do not need to be configured on the Border Leader Leafs. However, in the documentation
https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/HTML/vxlan/Content/Chp_EV-VX-MF/ex_fab1-config.htm
under "Step 8.6 – Route-target adjustment for L2VNI", it states:

"To extend a VLAN between fabrics, the VLAN and VNI must be configured on the border VTEP even if the said VLAN has no local member ports on the VTEP to downstream servers or switches."

Without configuring the VLAN and VNI on the border VTEP, VLAN connectivity between the sites does not work. I have tested this.

Is it possible to interconnect the sites in some other way that is not described in this document?

Just like you don't have same VID for different customers on one regular leaf, if you don't have anything ingress or egress on the VLAN you don't need the VLAN and VNI on the border leaf. Just ensure the routing is correct.

Think of it a bit like a spine - how does the VXLAN go from leaf to leaf when no vlan / vni is on the spine?  

Could you please clarify what you mean?
Within each site (and fabric) we are running iBGP, and eBGP is used between the fabrics. Naturally, the regular leaf switches (VTEPs) in different fabrics have routes to each other's Lo0 and Lo1.

You could just ensure to have routing between the VTEP's on both sides. You don't need to terminate all VLAN's in the border switch. 

Hi everyone,

We have two Spine & Leaf fabrics located in two different sites. The sites are connected via Border Leader Leafs (two VSX stacks with four links). The setup itself is working fine, but due to the way VNIs, VLANs, and EVPN are implemented on Aruba switches, I cannot configure VNIs on the Border Leader Leafs without also configuring VLANs.
For example:

vlan 31
    vsx-sync
    name TEST
evpn
    dyn-vxlan-tunnel-bridging-mode ibgp-ebgp
    vlan 31
        rd 10.100.x.2:10031
        route-target export 65101:10031
        route-target import 65101:10031
interface vxlan 1
    source ip 10.100.x.2
    no shutdown
    vni 10030
        vlan 31

So on the Border Leader Leafs I cannot configure more than 4096 VNIs, because I cannot have more than 4096 VLANs on a single switch.

Now we are onboarding some "large" customers. These customers can have up to 3000 of their own VLANs. They get dedicated leaf switches. Within a single site everything works fine: the customers use their own VNI pools (for example 20000–25000 and 30000–35000), and their VLANs exist only on their own leafs.

The problem appears when such a "large" customer needs connectivity in both sites. In that case, I would need to configure more than 6000 VLANs on the Border Leader Leafs, which is not possible.

When I designed the two-site setup, I followed this documentation:
https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/HTML/vxlan/Content/Chp_EV-VX-MF/ex_intro.htm

So the question is: how do you transport more than 6000 VNIs (VLANs) between two sites in an Aruba EVPN Multisite design?

Any examples or references would be highly appreciated. I assume I'm not the first one running into this limitation

Well , your usecase does not match the example in the document. You just have to figure out how to best get your routing of the VNIs between the sites.

Wait, I'm a bit confused.
I also used to think that VNIs and VLANs do not need to be configured on the Border Leader Leafs. However, in the documentation
https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/HTML/vxlan/Content/Chp_EV-VX-MF/ex_fab1-config.htm
under "Step 8.6 – Route-target adjustment for L2VNI", it states:

"To extend a VLAN between fabrics, the VLAN and VNI must be configured on the border VTEP even if the said VLAN has no local member ports on the VTEP to downstream servers or switches."

Without configuring the VLAN and VNI on the border VTEP, VLAN connectivity between the sites does not work. I have tested this.

Is it possible to interconnect the sites in some other way that is not described in this document?

Just like you don't have same VID for different customers on one regular leaf, if you don't have anything ingress or egress on the VLAN you don't need the VLAN and VNI on the border leaf. Just ensure the routing is correct.

Think of it a bit like a spine - how does the VXLAN go from leaf to leaf when no vlan / vni is on the spine?  

Could you please clarify what you mean?
Within each site (and fabric) we are running iBGP, and eBGP is used between the fabrics. Naturally, the regular leaf switches (VTEPs) in different fabrics have routes to each other's Lo0 and Lo1.

You could just ensure to have routing between the VTEP's on both sides. You don't need to terminate all VLAN's in the border switch. 

Hi everyone,

We have two Spine & Leaf fabrics located in two different sites. The sites are connected via Border Leader Leafs (two VSX stacks with four links). The setup itself is working fine, but due to the way VNIs, VLANs, and EVPN are implemented on Aruba switches, I cannot configure VNIs on the Border Leader Leafs without also configuring VLANs.
For example:

vlan 31
    vsx-sync
    name TEST
evpn
    dyn-vxlan-tunnel-bridging-mode ibgp-ebgp
    vlan 31
        rd 10.100.x.2:10031
        route-target export 65101:10031
        route-target import 65101:10031
interface vxlan 1
    source ip 10.100.x.2
    no shutdown
    vni 10030
        vlan 31

So on the Border Leader Leafs I cannot configure more than 4096 VNIs, because I cannot have more than 4096 VLANs on a single switch.

Now we are onboarding some "large" customers. These customers can have up to 3000 of their own VLANs. They get dedicated leaf switches. Within a single site everything works fine: the customers use their own VNI pools (for example 20000–25000 and 30000–35000), and their VLANs exist only on their own leafs.

The problem appears when such a "large" customer needs connectivity in both sites. In that case, I would need to configure more than 6000 VLANs on the Border Leader Leafs, which is not possible.

When I designed the two-site setup, I followed this documentation:
https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/HTML/vxlan/Content/Chp_EV-VX-MF/ex_intro.htm

So the question is: how do you transport more than 6000 VNIs (VLANs) between two sites in an Aruba EVPN Multisite design?

Any examples or references would be highly appreciated. I assume I'm not the first one running into this limitation

That is exactly what my question in the previous post was about.
What other options or best practices exist for interconnecting two sites in order to work around this 4096 VNI limitation?

Well , your usecase does not match the example in the document. You just have to figure out how to best get your routing of the VNIs between the sites.

Wait, I'm a bit confused.
I also used to think that VNIs and VLANs do not need to be configured on the Border Leader Leafs. However, in the documentation
https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/HTML/vxlan/Content/Chp_EV-VX-MF/ex_fab1-config.htm
under "Step 8.6 – Route-target adjustment for L2VNI", it states:

"To extend a VLAN between fabrics, the VLAN and VNI must be configured on the border VTEP even if the said VLAN has no local member ports on the VTEP to downstream servers or switches."

Without configuring the VLAN and VNI on the border VTEP, VLAN connectivity between the sites does not work. I have tested this.

Is it possible to interconnect the sites in some other way that is not described in this document?

Just like you don't have same VID for different customers on one regular leaf, if you don't have anything ingress or egress on the VLAN you don't need the VLAN and VNI on the border leaf. Just ensure the routing is correct.

Think of it a bit like a spine - how does the VXLAN go from leaf to leaf when no vlan / vni is on the spine?  

Could you please clarify what you mean?
Within each site (and fabric) we are running iBGP, and eBGP is used between the fabrics. Naturally, the regular leaf switches (VTEPs) in different fabrics have routes to each other's Lo0 and Lo1.

You could just ensure to have routing between the VTEP's on both sides. You don't need to terminate all VLAN's in the border switch. 

Hi everyone,

We have two Spine & Leaf fabrics located in two different sites. The sites are connected via Border Leader Leafs (two VSX stacks with four links). The setup itself is working fine, but due to the way VNIs, VLANs, and EVPN are implemented on Aruba switches, I cannot configure VNIs on the Border Leader Leafs without also configuring VLANs.
For example:

vlan 31
    vsx-sync
    name TEST
evpn
    dyn-vxlan-tunnel-bridging-mode ibgp-ebgp
    vlan 31
        rd 10.100.x.2:10031
        route-target export 65101:10031
        route-target import 65101:10031
interface vxlan 1
    source ip 10.100.x.2
    no shutdown
    vni 10030
        vlan 31

So on the Border Leader Leafs I cannot configure more than 4096 VNIs, because I cannot have more than 4096 VLANs on a single switch.

Now we are onboarding some "large" customers. These customers can have up to 3000 of their own VLANs. They get dedicated leaf switches. Within a single site everything works fine: the customers use their own VNI pools (for example 20000–25000 and 30000–35000), and their VLANs exist only on their own leafs.

The problem appears when such a "large" customer needs connectivity in both sites. In that case, I would need to configure more than 6000 VLANs on the Border Leader Leafs, which is not possible.

When I designed the two-site setup, I followed this documentation:
https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/HTML/vxlan/Content/Chp_EV-VX-MF/ex_intro.htm

So the question is: how do you transport more than 6000 VNIs (VLANs) between two sites in an Aruba EVPN Multisite design?

Any examples or references would be highly appreciated. I assume I'm not the first one running into this limitation