Wired Intelligent Edge

We have an EVPN VXLAN setup where our top of Rack switches are  2x8325 in a VSX pair.

We have A Paloalto Firewall cluster that is Active/Passive. Both firewalls are connected to one VSX pair. Now to gain some redundancy  we'd like to move one of the FW's to another rack. 

At this point we have the FW's connected trough LACP. The firewall has tagged sub-interfaces and the switches have vlan interfaces. We use a /29 subnet where the firewall has 1 address on the tagged sub interface  and the switches both have an address in their vlan interface. Routes are exchanged trough eBGP. The firewall load balances the traffic using ECMP and we have active-forwarding in the VSX pair to prevent unneccesary ISL traffic.

When I configure this on a second rack for the standby firewall I wonder how the failover would work. The same vlan can off course be added to a second VSX pair and there is room to provide those two extra switches with an IP adres as well. But do I need to have 4 BGP sessions to all firewalls? (probably yes) And how do I ensure that the traffic is routed by the directly connected switch?  Active forwarding does not work between two diferent VSX stacks I'm sure.

I could  do a setup  where I have 4 BGP sessions and use local preff to have preference to the first rack. This would work well in de 'normal' situation but in a failover situation it would lead to suboptimal routing until I manually fix the preferences.

So I came up with the following and hope to get some other people's insights on this aproach.

Stretch the Vlan trought the eVPN. Remove the active forwarding and add an active gateway. Adjust the next hop for all routes advertised by the siwtches to the firewall, to the active gateway adres using a route-map in the BGP export. There are still 4 BGP sessions, but they all use the same next hop. A firewall failover using gracefull restart should be as fast as the MAC learning  over eVPN.

Is this a viable aproach or is there a beter way to design this?

The complexity relies behind the fact that the second FW is standby and not attached to the same VSX pair.

So I would suggest to either:

• indeed consider 4 eBGP sessions
• move from eBGP to OSFP
• use active-active FW instead of act/standby (knowing that only one can be active next-hop at a time for the EVPN-VXLAN fabric)
• in addition to distribute FW among 2 racks, distribute VSX pairs among 2 racks: keep design simple and solved by physical distribution.

So it is a few years later but maybe this will help someone.

 I have built the setup that I described above. It does work but is has a few downsides. The major one being that 4 top of rack switches have a peering with the Paloalto and they will all advertise thos route with the same cost/metric. Some of the traffic will be going to the wrong TOP of rack switches using the  L3 vni  and than to the proper switches om L2 vni. Initially I fixed this by advertising my routes with a different cost to both racks. That works but after a failover the traffice will keep going to the TOR that has preference and use a suboptimal path. (The active gateway fixes this for traffic from the paloalto to the switches, but not the other way arround)

So for a new pair of firewalls I built the same thing with a few differences:

I got rid of active gateway and went instead went back to active forwarding. That makes the configuration less complex. I dit add "redistribute local-mac" to the evpn configuration because after a failover the Palo will initially use the MAC addres for the old TOR's as a next hop until BGP is all established and setup. This improved my failover time from 3 to 1 second.

To deal with the issue of the suboptimal routing a just made an ACL on both TOR's LAG interface blocking BGP traffic from the paloalto to the other stack. Also I made the peer-groups passive on the Aruba side.

So after a failover the mac move will be enough to make everything work very fast initially. Then the BGP wil make sure that all the routing becomes optimal for the paloalto in a second or two. After the gracefull restart timers expire the old TOR switches will remove the old routes and all trafffic is flowing the optimal path optimal again.

Pro's

• No different peer-groups neccesary on Paloalto
• No AS-path prepend or other route modifers on Paloalto
• Traffic automatically  flows the opimal route after a failover
• No need for next-hop rewrites on Aruba

Con's:

• Need an ACL on aruba
• two configured BGP peers will always be down