Wired Intelligent Edge

 View Only
  • 1.  Spanning tree issues

    Posted Sep 19, 2025 09:02 AM

    Been trying for months to get to the bottom of it with no luck

    Core - stack of Comware based 5900AF-48XG-4QSFP+ JC772A

    Satellite - various cabinets with 2920/2930F/2930M/6300M/6100/2540 

    all connected on 10Gb links

    From time to time (sometimes multiple times a day, sometimes once every few days - no pattern) the spanning tree will throw a fit & various switches will be reported by HP IMC as not responding & then a minute later all will get back to normal.

    Sometimes 1 cabinet, sometimes 2/3 or 10+

    Today I had one instance & the log on 6300 shows

    2025-09-19T08:11:36.726102+00:00 6300 intfd[1079]: Event|403|LOG_INFO|UKWN|1|Link status for interface 1/1/15 is up at 100 Mbps
    2025-09-19T08:11:36.690864+00:00 6300 hpe-mstpd[3612]: Event|2011|LOG_INFO|CDTR|1|Topology Change received on port 1/1/19 for CIST from source: 38:10:f0:4e:fc:f0
    2025-09-19T08:11:36.644696+00:00 6300 intfd[1079]: Event|403|LOG_INFO|UKWN|1|Link status for interface 1/1/48 is up at 100 Mbps
    2025-09-19T08:11:36.005182+00:00 6300 poe-hald[3595]: Event|7902|LOG_INFO|CDTR|1|Powered device power delivery on interface 1/1/45
    2025-09-19T08:11:36.000967+00:00 6300 poe-hald[3595]: Event|7902|LOG_INFO|CDTR|1|Powered device power delivery on interface 1/1/46
    2025-09-19T08:11:35.779562+00:00 6300 lldpd[3564]: Event|104|LOG_INFO|CDTR|1|LLDP neighbor LEC-SH added on 1/1/26
    2025-09-19T08:11:35.529032+00:00 6300 poe-hald[3595]: Event|7902|LOG_INFO|CDTR|1|Powered device power delivery on interface 1/1/40
    2025-09-19T08:11:35.528514+00:00 6300 intfd[1079]: Event|403|LOG_INFO|UKWN|1|Link status for interface 1/1/31 is up at 1 Gbps
    2025-09-19T08:11:35.525232+00:00 6300 poe-hald[3595]: Event|7902|LOG_INFO|CDTR|1|Powered device power delivery on interface 1/1/39
    2025-09-19T08:11:35.522398+00:00 6300 poe-hald[3595]: Event|7902|LOG_INFO|CDTR|1|Powered device power delivery on interface 1/1/43
    
    2025-09-19T08:11:35.511910+00:00 6300 hpe-mstpd[3612]: Event|2011|LOG_INFO|CDTR|1|Topology Change received on port 1/1/52 for CIST from source: e8:f7:24:51:34:9c
    2025-09-19T08:11:35.501666+00:00 6300 hpe-mstpd[3612]: Event|2014|LOG_INFO|CDTR|1|Port 1/1/19 blocked on CIST
    2025-09-19T08:11:35.498481+00:00 6300 hpe-mstpd[3612]: Event|2011|LOG_INFO|CDTR|1|Topology Change received on port 1/1/19 for CIST from source: 38:10:f0:4e:fc:f0
    2025-09-19T08:11:35.489890+00:00 6300 hpe-mstpd[3612]: Event|2015|LOG_INFO|CDTR|1|Port 1/1/19 unblocked on CIST
    
    2025-09-19T08:11:35.489657+00:00 6300 hpe-mstpd[3612]: Event|2006|LOG_INFO|CDTR|1|CST  - Root changed from 32768: 38:10:f0:4e:fc:c0 to 0: e8:f7:24:51:34:64
    
    2025-09-19T08:11:35.438065+00:00 6300 intfd[1079]: Event|403|LOG_INFO|UKWN|1|Link status for interface 1/1/1 is up at 100 Mbps
    2025-09-19T08:11:35.390894+00:00 6300 lldpd[3564]: Event|113|LOG_INFO|CDTR|1|PVID mismatch on 1/1/52 pvid = 2, Neighbor e8:f7:24:51:34:64 port_id = Ten-GigabitEthernet1/0/16 pvid = 1
    

    On Comware I have

    stp global enable
    stp mode rstp
    stp instance 0 root primary

    on Procurve/ArubaCX I have

    spanning-tree mode mstp
    spanning-tree force-version rstp-operation
    spanning-tree priority 15
    spanning-tree enable

    Would anybody have any idea what could be causing it?

    Thanks



    ------------------------------
    spgsitsupport
    ------------------------------


  • 2.  RE: Spanning tree issues

    Posted Sep 19, 2025 12:41 PM

    Have you tried to pick one STP and make it identical everywhere because i think i can see here stp mode rstp (comware) with mstp + rstp-opreation (cx/procurve)

    From what i see, try to run MSTP on all devices, one MST region and make the 5900 stack with CIST. So some examples would be:

    On COMWARE you can run:
    stp global enable
    stp mode mstp
    stp region-configuration
     region-name FABRIC
     revision-level 1
     instance 1 vlan 10 20
     instance 2 vlan 30 40
     active region-configuration
    #
    stp instance 0 root primary     ; this would be the CIST root
    stp instance 1 root primary
    stp instance 2 root primary

    On ARUBA CX you can run:
    spanning-tree mode mstp
    spanning-tree config-name FABRIC
    spanning-tree config-revision 1
    spanning-tree instance 1 vlan 10,20
    spanning-tree instance 2 vlan 30,40
    spanning-tree
    !
    spanning-tree priority 0         ; keep in mind this should be on the intended root only

    Aruba (Procurve) you can run:
    spanning-tree
    spanning-tree mode mstp
    spanning-tree config-name "FABRIC"
    spanning-tree config-revision 1
    spanning-tree instance 1 vlan 10,20
    spanning-tree instance 2 vlan 30,40
    ; non-root closets:
    spanning-tree priority 32768
    ; root/secondary as needed:
    ; spanning-tree priority 0   (root) / 4096 (secondary)

    Also, fix every uplinks native VLAN/PVID.  Your LLDP line told you exactly where to start: CX 6300 1/1/52 PVID = 2 and Neighbor (Comware Ten-Gig 1/0/16) PVID = 1

    So on Aruba CX you can have as an example:
    interface 1/1/52
     no shutdown
     vlan trunk native 1
     vlan trunk allowed 1,10,20,30,40
     spanning-tree port-type normal

    On Comware for example:
    interface Ten-GigabitEthernet1/0/16
     port link-mode bridge
     port link-type trunk
     port trunk permit vlan 1 10 20 30 40
     port trunk pvid vlan 1
     stp edged-port disable

    This should be crosschecked on all inter-switch links. In addition, lock down the edge ports to stop TCN Storms as an example:

    On CX and ProCruve:
    ; Globally (CX):
    spanning-tree bpdu-guard
    ; Per edge port:
    interface 1/1/x
     spanning-tree port-type admin-edge
     spanning-tree bpdu-guard

    On Comware:
    interface GigabitEthernet1/0/x
     stp edged-port enable
     bpdu-protection enable

    Also a Rate-limit damage from TCNs such as in Comware 
    stp tc-protection enable
    stp tc-protection threshold 10 interval 10

    and on CX:
    spanning-tree tc-protection
    spanning-tree tc-protection interval 10

    I think, if not mistaken on some procurves you have spanning-tree tcn-guard



    ------------------------------
    Shpat | ACEP | ACMP | ACCP | ACDP
    Just an Aruba enthusiast and contributor by cases
    If you find my comment helpful, KUDOS are appreciated.
    ------------------------------



  • 3.  RE: Spanning tree issues

    Posted Sep 19, 2025 01:11 PM

    uplinks native VLAN/PVID should not matter to anything, but it was just one off mismatch

    nothing is using VLAN 1

    If I am not wrong with my config everything was/is set to rtsp (which is what I wanted to use)

    I do not have edge ports on Comware 7 stack (all are uplinks to other switches/routers)



    ------------------------------
    spgsitsupport
    ------------------------------



  • 4.  RE: Spanning tree issues

    Posted Sep 20, 2025 01:48 AM

    Your comware core should be the root. Try to add root-guard on all downlinks to other switches so root will not move.

    Best, Gorazd



    ------------------------------
    Gorazd Kikelj
    MVP Guru 2025
    ------------------------------



  • 5.  RE: Spanning tree issues

    Posted Sep 21, 2025 10:02 AM
    Edited by spgsitsupport Sep 21, 2025 10:04 AM

    stp root-protection on all 5900 downlinks

    or

    spanning-tree <interface> root-guard

    on all satellite switch uplinks?

    or both?

     Just wonder, if STP causes so many issues on such simple network setup (single 10Gb fibre connection from core to each cab), what is the point of having it at all?

    Thanks



    ------------------------------
    spgsitsupport
    ------------------------------



  • 6.  RE: Spanning tree issues

    Posted Sep 22, 2025 02:43 AM
    Edited by spgsitsupport Sep 22, 2025 02:44 AM

    If this is true:

    Cisco Portfast, BPDU Guard, and Root Guard - FlackBoxRoot Guard works by blocking ports that receive a Bridge Protocol Data Unit (BPDU) that indicates a superior root bridge, preventing an unexpected or unauthorized switch from becoming the Spanning Tree Protocol (STP) root. It's applied to downstream or edge ports where the root bridge should never appear, effectively creating a perimeter for the STP root. If a superior BPDU is received, the port enters a root-inconsistent state, stopping traffic until the superior BPDU is no longer sent. 
     
    then applying root guard to the downlink ports will not change much what I am seeing now
    The port will be blocked and communication to this port will not happen (making that cabinet and all that is attached to it, unreachable and isolated). At least for a period of time
    I would rather figure what is causing it, and "fix" that instead



    ------------------------------
    spgsitsupport
    ------------------------------



  • 7.  RE: Spanning tree issues

    Posted Sep 23, 2025 03:32 AM

    There are several steps in this process.

    1. select your primary root switch (stp instance <inst.list> root primary)
    2. select your secondary root swithc (stp instance <inst.list> root secondary)
    3. set priority on primary root switch (stp instance <inst.list> priority 0)
    4. set priority on secondary root switch (stp instance <inst.list> priority 1)
    5. protect your downlink ports with stp root-protection (root guard) 

      Enabling root guard
      About root guard
      Configure root guard on a designated port.
      The root bridge and secondary root bridge of a spanning tree should be located in the same MST
      region. Especially for the CIST, the root bridge and secondary root bridge are put in a high-bandwidth
      core region during network design. However, due to possible configuration errors or malicious
      attacks in the network, the legal root bridge might receive a configuration BPDU with a higher priority.
      Another device supersedes the current legal root bridge, causing an undesired change of the
      network topology. The traffic that should go over high-speed links is switched to low-speed links,
      resulting in network congestion.
      To prevent this situation, MSTP provides the root guard feature. If root guard is enabled on a port of
      a root bridge, this port plays the role of designated port on all MSTIs. After this port receives a
      configuration BPDU with a higher priority from an MSTI, it performs the following operations:
      • Immediately sets that port to the listening state in the MSTI.
      • Does not forward the received configuration BPDU.
      This is equivalent to disconnecting the link connected to this port in the MSTI. If the port receives no
      BPDUs with a higher priority within twice the forwarding delay, it reverts to its original state.
      Restrictions and guidelines
      On a port, the loop guard feature and the root guard feature are mutually exclusive.
      Procedure
      1. Enter system view.
      system-view
      2. Enter interface view.
      interface interface-type interface-number
      3. Enable the root guard feature.
      stp root-protection
      By default, root guard is disabled.

    Best, Gorazd



    ------------------------------
    Gorazd Kikelj
    MVP Guru 2025
    ------------------------------



  • 8.  RE: Spanning tree issues

    Posted Sep 23, 2025 02:33 PM

    I got primary already set (config above)

    I do not have secondary (do not need it)

    All satellites are equal , all on 10Gb links direct to core

    I really do not understand how and why I would get stp issues with RANDOM switches

    Sometimes Cab 1 maybe Cab 12 or Cab 23

    I do not have new things randomly connecting to ports that could be causing it (none of the not used ports is patched!)

    All connected devices are pretty static, they are what they are and they do not move on daily basis

    So logically if something is bad, it would be bad always 

    But no, there is no pattern of any kind, total randomness

    And the root (core) is never disturbed (inaccessible etc)

    It is only random satellites

    Ofcourse I can apply root guard on downlinks ports, but if somehow one of the satellites goes wonky, then this config does not change anything (satellite will still be inaccessible with all that is connected to it for likely the same amount of time - usually ~ 60-90 seconds)

    Thanks



    ------------------------------
    spgsitsupport
    ------------------------------



  • 9.  RE: Spanning tree issues

    Posted Sep 24, 2025 04:30 AM

    Hi.

    root-guard will prevent root moving between switches. 

    Add loop-protect, admin-edge and possibly bpdu-guard as explained by shpat post on all edge ports. Also check stp priority on all edge and intermediate (if any) switches.

    You should have all edge ports configured as such so they wont respond to bpdu-s. There are no bpdu expected to be originated to edge ports hence is safe to discard all bpdu frames.

    Best, Gorazd



    ------------------------------
    Gorazd Kikelj
    MVP Guru 2025
    ------------------------------