Wired Intelligent Edge

Hello Airheads Community,

I'm currently working on a network setup involving two pairs of Aruba 8100 switches configured as Virtual Switching Extension (VSX) and connected with 2Gbps Metro Ethernet interconnect links. The architecture is designed to stretch Layer 2 over these interconnects using VXLAN technology. Each site hosts servers connected to each switch pair, with server NICs bonded through LACP and Multi-Chassis Link Aggregation (MCLAG) configured on the switches.

Configuration Overview:

• Two pairs of Aruba 8100 switches in VSX.
• Server connections with LACP Dynamic to the switches.
• MCLAG configured on the switches.
• 2Gbps Metro Ethernet interconnect links.
• VXLAN technology for Layer 2 stretching.

Primary Objectives:

1. Implement VXLAN over the interconnected links to avoid Spanning Tree Protocol (SPT) usage and maintain a loop-free topology (VXLAN static configured on both links).
2. Achieve full redundancy, ensuring traffic reroutes through an alternate link if one fails.
3. Bundle two interconnected links to enhance capacity to 4Gbps.

Encountered Issues:

1. Connectivity Loss with LACP: When servers have their NICs bonded through LACP, we lose connectivity. Pings from one server to another (left to right) fail.
2. Successful Ping with Single Interface: If we configure a single interface on each server (without NIC teaming) and no Link Aggregation Group (LAG) on the switches, pinging works.
3. Challenges with VSX and Link Capacity Doubling: Unable to configure the interconnected links to double the capacity (4Gbps) due to VSX limitations.
4. VXLAN and MCLAG/LAG Incompatibility: After configuring LAG and MCLAG, VXLAN seems unable to handle the traffic effectively.
5. Lack of Redundancy with Single Interface: With no server teaming and VXLAN operational, the redundancy fails. If the top link is shut down, traffic does not reroute via the bottom link/VXLAN.

I'm seeking insights or solutions to address these issues, particularly in achieving seamless connectivity with LACP, effective VXLAN implementation alongside MCLAG/LAG, and ensuring full redundancy and capacity enhancement of the interconnected links.

Any advice, experiences, or recommendations from the community would be greatly appreciated.

Good details and well explained.

How have you troubleshot this so far?

Does VSX appear to be established and the inter-switch links show in tact and operational?

Are you using Active Gateway in this configuration?

If you established LACP between a Win Server and two (or more) interfaces on a single 8100 switch do you have a problem or does it only arise when the LAG is across two switches?

How is the winservers dual link config look like ?

Just LACP dynamic, teaming of a couple of NICs

Hello,

Thank you for the response.

Active Gateway is predominantly associated with Layer 3 configurations ( as per my understanding), particularly in scenarios involving core switching functionalities. In our current setup, we've integrated the topology into the client's network at Layer 2, utilizing static VXLAN for the purpose. This decision was driven by the specific requirements of our deployment, which did not necessitate Layer 3 capabilities at the edge of our network.

During our installation today, as outlined in the provided network diagram, we connected an Arista switch to the right-side Aruba pair using LAG LACP and MC-LAG configurations, respectively. This setup was then tested by connecting a laptop to the Arista switch and pinging a real Active Directory server, which was linked via LAG to the left side of our topology. The connection between these points was facilitated by Metro Ethernet links. Initially, the ping tests to the AD server were successful, indicating a healthy connection through the top link.

However, an issue arose when the top link was disconnected; the ping tests failed to continue, suggesting a failure in traffic rerouting through the remaining bottom link. This led me to consider several potential explanations:

1. The WAN link failure might not have triggered a corresponding change in the VSX LAG state, causing the primary switch to retain traffic handling responsibilities without a viable delivery path.
2. Traffic might be persistently directed towards the failed WAN link by the primary switch, which fails to reroute it through the secondary switch as expected.
3. Considering the LAG setup on the Arista switch, I expected traffic from the laptop to utilize both links towards the Aruba pair. The primary switch's continued operation might be influencing traffic to favor it, raising questions about the load balancing behavior in this configuration.

I'm currently delving into these areas to pinpoint the exact cause of the issue and would greatly appreciate any further insights or suggestions you might have. Specifically, I'm interested in understanding if there are nuances in the MC-LAG or VSX configurations that could be influencing this behavior, or if there are recommended practices for such deployments that I might have overlooked.

Any advice or shared experiences with similar topologies would be incredibly valuable as I work towards resolving this challenge.

We use MCLAGs to connect our 8320 VSX core to our (non-VSX) distribution layer, and we use a point to point routing VLAN on that connection.

After much trial and error, the model we came up with looks like this.

interface lag xxx multi-chassis
    vsx-sync vlans
    no shutdown
    no routing
    vlan trunk native xxx
    vlan trunk allowed x y z
    lacp mode active
    lacp fallback

and

interface vlan xxx
    vsx active-forwarding
    ip address a.b.c.d/29
    ip ospf 1 area 0.0.0.0

Hope there's a nugget in there you can use.

Hi,

I have not used static VXLAN, but some things to check

• MTU Size on both of your Metro Ethernet connections is big enough
• You have loopbacks for OSPF
• You have a additional loopback for VXLAN source with the same IP address for both switches in a VSX peer (also in OSPF)
• The VTEP peer points to the VXLAN source IP 

For verification of the MCLAG you need to add a device on one side to see if the forwarding goes good internally in the VSX.

Your drawing looks like you do som testing using EVE-NG and the CX Simulator. You can configure it but you can't get any traffic through. VXLAN and VSX is not supported in the Simulator.