Wired Intelligent Edge

I have 3 datacenters connected with some L2 VPLS circuit. There is a pair of Aruba CX switches in each of datacenters  conneted to an operator CPE with LACP link aggregation.  In one site there is something wrong which I can't diagnose. Operator says that their configuration in the same int each site.

 The problem: 
A stack of 6300F in DC3 has a bgp session with each of 8325 in other DCs. One session does not get up. I can ping all 8325 switches, even with large packers so it is not MTU. but anything more complicated than ICMP is problematic. I did a packet capture on the 6300 and it show a lot of duplicates. I have two vlans on interfaces to CPE one native and one tagged. problemm occurs over tagged vlan.  I tried to disable links in dc3, it did not change anything, so it is not an optics. If I use native vlan problem does not occur but I need tagged frames for CoS tagging.

Any ideas?

The interface configuration is pretty basic for now.

6300
interface lag 51
    description to VPLS
    no shutdown
    no routing
    vlan trunk native 3002
    vlan trunk allowed 10,3002
    lacp mode active
    qos trust dscp
    spanning-tree bpdu-filter
    exit

8325

interface lag 47 multi-chassis
    description to VPLS
    no shutdown
    no routing
    vlan trunk native 3002
    vlan trunk allowed 10, 3002
    lacp mode active
    lacp fallback
    qos trust dscp
    spanning-tree bpdu-filter
    exit

Diagram

Hi Tommy,

• Did you check the VSX status "show vsx status"?
• Does each VSX set have his own unique system-mac? both members int the VSX must be equal.
• Are you LACP connection ok? "show lacp interface"
• What firmware level are you running?
• Anything in the system logging?
• Note that 8325 don't have large mtu enabled by default.

For urgent matter please open an Aruba TAC case to investigate.

------------------------------
Marcel Koedijk | MVP Expert 2023 | ACEP | ACMP | ACCP | ACDP | Ekahau ECSE | Not an HPE Employee | Opinions are my own
------------------------------

Everything seems to be in a mint condition.  All the 8325 have dozens of bgp session established. There are three locations with a LACP uplink. I am waiting  for the second operator to convert to LACP in the problematic location. 

 > Did you check the VSX status "show vsx status"?

Yes, everything OK  

> Does each VSX set have his own unique system-mac? both members int the VSX must be equal.

Both pairs have their own system MAC, they have several LAG connected to them. 

 > Are you LACP connection ok? "show lacp interface"

Seem OK

DC2-CORE1# show lacp interfaces   1/1/47

State abbreviations :
A - Active        P - Passive      F - Aggregable I - Individual
S - Short-timeout L - Long-timeout N - InSync     O - OutofSync
C - Collecting    D - Distributing
X - State m/c expired              E - Default neighbor state

Aggregate-name : lag47(multi-chassis)
-------------------------------------------------
                       Actor             Partner
-------------------------------------------------
Port-id            | 47                 | 1
Port-priority      | 1                  | 1
Key                | 47                 | 259
State              | ALFNCD             | ASFNCD
System-ID          | 02:00:ac:1f:ff:02  | 00:1a:d0:53:bc:4c
System-priority    | 65534              | 32768

DC2-CORE2# show lacp interfaces   1/1/47

State abbreviations :
A - Active        P - Passive      F - Aggregable I - Individual
S - Short-timeout L - Long-timeout N - InSync     O - OutofSync
C - Collecting    D - Distributing
X - State m/c expired              E - Default neighbor state

Aggregate-name : lag47(multi-chassis)
-------------------------------------------------
                       Actor             Partner
-------------------------------------------------
Port-id            | 1047               | 2
Port-priority      | 1                  | 1
Key                | 47                 | 259
State              | ALFNCD             | ASFNCD
System-ID          | 02:00:ac:1f:ff:02  | 00:1a:d0:53:bc:4c
System-priority    | 65534              | 32768

> What firmware level are you running? 

GL.10.11.0001 on 8325 and FL.10.10.0002 on 6300

> Anything in the system logging?

Ony Peer up/down on 6300 and one of 3325 

2024-01-15T09:37:47.005500+01:00 DC2-CORE1 hpe-routing[8421]: Event|2902|LOG_INFO|AMM|1/1|172.28.4.3: Peer down. error-code: Hold Timer Expired, error-sub-code: Unspecific. vrf-name: prod
2024-01-15T09:37:08.185050+01:00 DC2-CORE1 hpe-routing[8421]: Event|2901|LOG_INFO|AMM|1/1|172.28.4.3: Peer up. vrf-name: prod
2024-01-15T09:36:52.748354+01:00 DC2-CORE1 hpe-routing[8421]: Event|2902|LOG_INFO|AMM|1/1|172.28.4.3: Peer down. error-code: Hold Timer Expired, error-sub-code: Unspecific. vrf-name: prod
2024-01-15T09:35:56.380130+01:00 DC2-CORE1 hpe-routing[8421]: Event|2901|LOG_INFO|AMM|1/1|172.28.4.3: Peer up. vrf-name: prod

2024-01-15T10:11:09.228441+01:00 DC4-6300 hpe-routing[5395]: Event|2902|LOG_INFO|CDTR|1|172.28.4.5: Peer down. error-code: Hold Timer Expired, error-sub-code: Unspecific. vrf-name: default
2024-01-15T10:10:33.246289+01:00 DC4-6300 hpe-routing[5395]: Event|2901|LOG_INFO|CDTR|1|172.28.4.5: Peer up. vrf-name: default
2024-01-15T10:06:01.565081+01:00 DC4-6300 hpe-routing[5395]: Event|2902|LOG_INFO|CDTR|1|172.28.4.5: Peer down. error-code: Unrecognized error code, error-sub-code: Unrecognized error subcode. vrf-name: default
2024-01-15T10:05:59.070562+01:00 DC4-6300 hpe-routing[5395]: Event|2901|LOG_INFO|CDTR|1|172.28.4.5: Peer up. vrf-name: default
2024-01-15T10:05:43.211262+01:00 DC4-6300 hpe-routing[5395]: Event|2902|LOG_INFO|CDTR|1|172.28.4.5: Peer down. error-code: Hold Timer Expired, error-sub-code: Unspecific. vrf-name: default
2024-01-15T10:05:04.350350+01:00 DC4-6300 hpe-routing[5395]: Event|2901|LOG_INFO|CDTR|1|172.28.4.5: Peer up. vrf-name: default
2024-01-15T10:04:48.489774+01:00 DC4-6300 hpe-routing[5395]: Event|2902|LOG_INFO|CDTR|1|172.28.4.5: Peer down. error-code: Hold Timer Expired, error-sub-code: Unspecific. vrf-name: default
2024-01-15T10:02:24.670560+01:00 DC4-6300 hpe-routing[5395]: Event|2901|LOG_INFO|CDTR|1|172.28.4.5: Peer up. vrf-name: default
2024-01-15T10:02:07.996848+01:00 DC4-6300 hpe-routing[5395]: Event|2902|LOG_INFO|CDTR|1|172.28.4.5: Peer down. error-code: Hold Timer Expired, error-sub-code: Unspecific. vrf-name: default
2024-01-15T10:01:37.998367+01:00 DC4-6300 hpe-routing[5395]: Event|2901|LOG_INFO|CDTR|1|172.28.4.5: Peer up. vrf-name: default
2024-01-15T10:01:22.569072+01:00 DC4-6300 hpe-routing[5395]: Event|2902|LOG_INFO|CDTR|1|172.28.4.5: Peer down. error-code: Hold Timer Expired, error-sub-code: Unspecific. vrf-name: default

> Note that 8325 don't have large mtu enabled by default.

Oprrator provides L2 MTU of 1550 to fit tagged frames, so I did not touch the MTU on interfaces to WAN. 

Hi Tommy,

It's not clear to me it it's the LACP or the BGP which does not go up.

Verify also your LAG with LACP is Aruba - CPE for all loactions. In your setup you cant run LAG Aruba - Aruba.

Verify the physical interfaces is of same type and speed, which is a requirement for running LACP.

On all arubas run show lacp interfaces to verify the LAG is up and running. If you find which is not running you need your service provider debug on the CPEs.

Debugging BGP - then BGP information is needed to be able to come with some recomandations.

iBGP / eBGP, SVI interfaces, loopbacks etc...

> It's not clear to me it it's the LACP or the BGP which does not go up.

LAG is OK. One of four  BGP sessions over this LAG interface is unstable.  It establishes from time to time and then is dropped. See log excerpt above. 

> Verify also your LAG with LACP is Aruba - CPE for all loactions. In your setup you cant run LAG Aruba - Aruba.

Everything checked. Those switches have several LAG to various things. 

> Verify the physical interfaces is of same type and speed, which is a requirement for running LACP.

LAG would not go up with different interface speed. 

On all arubas run show lacp interfaces to verify the LAG is up and running. If you find which is not running you need your service provider debug on the CPEs.

Debugging BGP - then BGP information is needed to be able to come with some recomandations.

iBGP / eBGP, SVI interfaces, loopbacks etc...

DC4-6300

# show run   interface loopback
interface loopback 0
    ip address 172.26.0.11/32
    ip ospf 1 area 0.0.0.0
    exit

# show running-config interface vl 10
interface vlan 10
    description VPLS-Tagged
    ip address 172.28.4.3/24
    l3-counters
    exit
 # show running-config interface lag 51
interface lag 51
    description VPLS
    no shutdown
    no routing
    vlan trunk native 3002
    vlan trunk allowed 10,3002
    lacp mode active
    spanning-tree bpdu-filter
    exit

# show running-config bgp
router bgp 64700
    bgp router-id 172.26.0.11
    bgp log-neighbor-changes

    neighbor 172.28.4.5 description TO-DC2-CORE1 Not working
    neighbor 172.28.4.5 timers 5 30
    neighbor 172.28.4.5 fall-over
    neighbor 172.28.4.6 remote-as 65001
    neighbor 172.28.4.6 description TO-DC2-CORE2 Working
    neighbor 172.28.4.6 timers 5 30
    neighbor 172.28.4.6 fall-over
    address-family ipv4 unicast
        neighbor 172.28.4.5 activate
        neighbor 172.28.4.6 activate

8325 

Those switches have dozens of neighbors working well. 

DC2-CORE1# show  running-config interface lag 47
interface lag 47 multi-chassis
    description VPLS Orange
    no shutdown
    no routing
    vlan trunk native 
    vlan trunk allowed 3002,3012
    vlan translate 10 3012
    lacp mode active
    lacp fallback
    qos trust dscp
    spanning-tree bpdu-filter
    exit

DC2-CORE2# show  running-config interface lag 47
interface lag 47 multi-chassis
    description VPLS 
    no shutdown
    no routing
    vlan trunk native 3002
    vlan trunk allowed 3002,3012
    vlan translate 10 3012
    lacp mode active
    lacp fallback
    qos trust dscp
    spanning-tree bpdu-filter
    exit

DC2-CORE1# show running-config interface vl 3012
interface vlan 3012
    description DC2-C1 VPLS  Tagged
    vrf attach prod
    ip address 172.28.4.5/24
    l3-counters
    exit

DC2-CORE2# show running-config interface vl 3012
interface vlan 3012
    description DC2-C2 VPLS Tagged
    vrf attach prod
    ip address 172.28.4.6/24
    l3-counters
    exit

DC2-CORE1# show running-config bgp
router bgp 65001
    bgp router-id 172.31.2.3
    bgp log-neighbor-changes
    vrf prod
        neighbor 172.28.4.3 remote-as 64700
        neighbor 172.28.4.3 timers 5 30
        neighbor 172.28.4.3 fall-over
        address-family ipv4 unicast
            neighbor 172.28.4.3 activate

DC2-CORE2# show running-config bgp
router bgp 65001
    bgp router-id 172.31.2.4
    bgp log-neighbor-changes
    vrf prod
        neighbor 172.28.4.3 remote-as 64700
        neighbor 172.28.4.3 timers 5 30
        neighbor 172.28.4.3 fall-over
        address-family ipv4 unicast
            neighbor 172.28.4.3 activate