Wired Intelligent Edge

Hello,

We just purchased new 5400R Aruba branded procurve switches for our high school and middle school this year. We also had new SM fiber intalled to all closets as well. 

All switch closets connect back to our core switch, again a 5412R, at 10GB using HPE 10-Gbe SFP+ 10KM LR 13010 Transceiver, Mfg Part# J9151A. 

The switches are all running firmware KB.16.01.0007 and Boot ROM KB.16.01.0006.

We are experience packet loss and excessive Errors Rx anywhere from 10-100 per minute. We are not experience drops Tx, Alignment Rx, or Collision tx. 2 days ago we had one of the closet connections go completly offline. The tranceivers would not re-light. 

If we install 1GB J4895C SFP 1310 trancievers in the same fiber pathing using the same patch cables into the same ports on the switches they will run happily without issue. 

School has not started yet and these switches are under almost no load at this point... hardly any traffic passing through them. I have reached out to our vendor to see if they can help me with getting HPE support but thought i would ask here as well. 

Thanks,

~jake

Hi, in which HPE Modules are installed your (Genuine, right?) HPE X132 10G SFP+ LC LR Transceivers (J9151A)?

Supported HPE Modules for the usage of J9151A SFP+ Transceivers are:

• J9536A 20-Port Gig-T PoE+ / 2-Port SFP+ v2 zl Module
• J9538A 8-Port 10-GbE SFP+ v2 zl Module
• J9548A 20-Port Gig-T / 2-Port SFP+ v2 zl Module
• J9990A 20-Port PoE+ / 4-Port SFP+ v3 zl2 Module
• J9993A 8-Port 1G/10GbE SFP+ v3 zl2 Module

Any other HP Module is not supported for the usage of J9151A.

The 10GBASE-LR (10G Long Range, up to 10 km length) standard requires single mode Fiber Optic cables (working at 1310 nm) with a core/cladding of 9/125 um, is that fulfilled by actual FO inter-Switch/Building cabling?

Parnassus,

Thanks for your reply. 

Yes they are Genuine HPE trancievers. They are all installed in HPE Module J9990A in all of the switches. 

Our fiber is all 9/125 um single mode fiber as well as patch cables certified for 1310.

All runs are under under 1KM.

Thanks,

~jake

Great, that's reassuring.

Were you able to isolate a particular 10G-Base LR uplink where this issue shows up or any of yours 10G-Base LR uplinks shows this issue?

Software versions (excluded the case this is a new bug, which I really don't know) should be OK if all involved Switches are using the same KB.16.01.0007 or newer (on the same Branch there is the KB.16.01.0008 recently released) even if upgrading to the latest Branch KB.16.02 should not be mandatory regarding the Transceiver's support (As example the KB.16.02.0010 <-- a side note: I don't know if it is a typo or not but I noticed that KB.16.02 software branch, exactly as happen with the KB.15.18 software branch, states to use the BootROM KB.15.01.0001. BootROM KB.16.01.0006 is used instead on the KB.16.01 software branch).

Can you provide Ports status (show interface brief or show interface config) for all involved 10G SFP+ ports and Transceivers status (show tech transceivers) for all involved Transceivers?

We are seeing this issue on links to 3 or 4 switches atm. 1 is paticualrly bad so far averaging 8-10% packet loss. This is the same one that I tested putting the HPE 1GB trancievers in and then tested and it dropped no packets and had no errors. 

All of the switches are for sure running the exact same version of firmware I double checked that. I also tried testing with flow control on and off, since its about the only thing you can change on 10gb link, and it made no difference. 

Below is the port info for the really bad edge switch in question and the core switch they all connect too. I also included the detail port counters in case its of use.

Edge Switch

show interfaces brief A21

Status and Counters - Port Status

| Intrusion MDI Flow Bcast
Port Type | Alert Enabled Status Mode Mode Ctrl Limit
------------ ---------- + --------- ------- ------ ---------- ---- ---- -----
A21 SFP+LR | No Yes Up 10GigFD NA off 0

show tech transceivers

transceivers

Transceiver Technical Information:
Port # | Type | Prod # | Serial # | Part #
-------+-----------+------------+------------------+----------
A21 | SFP+LR | J9151A | Private Info Erased| 1990-3883

Status and Counters - Port Counters - Port A21

Name :
MAC Address : Private Info Erased
Link Status : Up
Totals (Since boot or last clear) :
Bytes Rx : 104,493,639 Bytes Tx : 71,316,404
Unicast Rx : 282,124 Unicast Tx : 296,783
Bcast/Mcast Rx : 35,203 Bcast/Mcast Tx : 2368
Errors (Since boot or last clear) :
FCS Rx : 77,095 Drops Tx : 0
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 657 Excessive Colln : 0
Total Rx Errors : 77,752 Deferred Tx : 0

Core Switch

show interfaces brief e2

Status and Counters - Port Status

| Intrusion MDI Flow Bcast
Port Type | Alert Enabled Status Mode Mode Ctrl Limit
------------ ---------- + --------- ------- ------ ---------- ---- ---- -----
E2 SFP+LR | No Yes Up 10GigFD NA off 0

show tech transceivers

transceivers

Transceiver Technical Information:
Port # | Type | Prod # | Serial # | Part #
-------+-----------+------------+------------------+----------
C2 | SFP+LR | J9151A | Private Info Erased| 1990-3883
C3 | SFP+LR | J9151A | Private Info Erased| 1990-3883
C4 | SFP+LR | J9151A | Private Info Erased| 1990-3883
C5 | SFP+LR | J9151A | Private Info Erased| 1990-3883
C6 | SFP+LR | J9151A | Private Info Erased| 1990-3883
E2 | SFP+LR | J9151A | Private Info Erased| 1990-3883
E3 | SFP+LR | J9151A | Private Info Erased| 1990-3883
E6 | SFP+LR | J9151A |Private Info Erased | 1990-3883
E7 | SFP+LR | J9151A | Private Info Erased| 1990-3883
E8 | SFP+LR | J9151A | Private Info Erased| 1990-3883

Status and Counters - Port Counters - Port E2

Name : FHTMS Library Trunk 2
MAC Address : Private Info Erased
Link Status : Up
Totals (Since boot or last clear) :
Bytes Rx : 80,772,325 Bytes Tx : 198,874,653
Unicast Rx : 328,852 Unicast Tx : 397,444
Bcast/Mcast Rx : 2697 Bcast/Mcast Tx : 42,067
Errors (Since boot or last clear) :
FCS Rx : 0 Drops Tx : 0
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 0 Excessive Colln : 0
Total Rx Errors : 0 Deferred Tx : 0

Thanks,

~jake

Even if you didn't explicitly wrote that your network topology is a star with the "core" connected to many other "edge" switches using single link F.O. uplinks, I presume it is (I shouldn't be wrong).

Said so is...the E2<->A21 link (E2 Port on the "core" Switch connected straight to the A21 Port of one of the "edge" Switch) the problematic one?

If so how is it possible that the FCS Rx errors (and Giant Rx errors too) on the "edge" side haven't any corresponding Tx errors of the same types on the "core" side? That's strange...

What's about Fiber Optic Patch Cables? Are they all Single Mode LC (UPC) / LC (UPC) from the SFP+ port up to the Optical Patch Panel on the Rack or what? Have you tried to exchange them?

Yes it is a star topology with all of the edge switches connected back using single FO uplinks using the 10GB trancievers. 

You are also right in regard to E2 is the core switch and A21 is the edge switch port. And yes that A21 is the problematic switch in question. 

In regards to why the errors are not matching up on the core side I am not certain either. That is why im reaching out for help. 

the cables we are using are these https://www.neweggbusiness.com/product/product.aspx?item=9siv00c20k8711

The fiber path is:

Core switch (E2) -> fiber panel ( from HS under parking lot to MS) -> fiber patch (jumper) -> Edge closet fiber panel -> Edge switch (A21)

We had our fiber contractor come back and test, with the excat same patch cables (all 3), from the HS to the MS and it came up fine on the fluke. Also as I said if i use the 1GB J4895C SFP 1310 trancievers in the exact same ports on both the core switch and edge switch using the same patch panels we have no packet loss and also have not tx or rx errors. 

Thanks,

~jake

Hi,

have you tried "Diagnostic Optical Monitoring"

You can see it with "show interface tranceiver E2 detail"

More info:

http://h22208.www2.hpe.com/eginfolib/networking/docs/switches/RA/15-18/5998-8161_ra_2620_mcg/content/ch11s05.html

hth

Alex

I ran out of ideas...a last question: are the Aruba 5400R zl2 Switches equipped only with v3 zl2 Modules or there is a mix of v2+v3 Modules an so you're operating all the Switches forcing the v3 Modules to work in compatibility mode to v2 Modules (and so forcing the v3 Modules to behave like v2 Modules)?

The command show module will help to understand which Module's versions each Switch is equipped with.

I ask because would be interesting to see if a configuration that impose to the 5400R zl2 to not run its v3 Modules in compatibility mode (with the no allow-v2-modules command which will concurrently disable any v2 Module installed and will enable only v3 Modules to operate) may/may not have an impact on the issue or not.

Mmmm...cables...those FCS Errors...cables, it's a pity that Time Domain Reflectometry (TDR), introduced on v3 Modules to detect cable faults, doesn't work on 10G links...

Parnassus,

The Edge switches are all v3 modules.

show modules

Status and Counters - Module Information

Chassis: 5412Rzl2 J9851A Serial Number:Private Info Erased

Allow V2 Modules: Yes

Core Mod
Slot Module Description Serial Number Status Dump Ver
----- -------------------------------------- -------------- -------- ----- ---
MM1 HP J9827A Management Module 5400Rzl2 Private Info Erased Active YES 1

A HP J9990A 20p PoE+ / 4p SFP+ v3 zl2... Private Info Erased Up YES 3
B HP J9986A 24p 10/100/1000BASE-T PoE... Private Info Erased Up YES 3
C HP J9987A 24p 10/100/1000BASE-T v3 ... Private Info Erased Up YES 3
D HP J9987A 24p 10/100/1000BASE-T v3 ... Private Info Erased Up YES 3

The core switch was purchased last year its the only one not brand new and has a mix of both. 

show modules

Status and Counters - Module Information

Chassis: 5412Rzl2 J9851A Serial Number:Private Info Erased

Allow V2 Modules: Yes

Core Mod
Slot Module Description Serial Number Status Dump Ver
----- -------------------------------------- -------------- -------- ----- ---
MM1 HP J9827A Management Module 5400Rzl2 Private Info Erased Active YES 1

A HP J9546A 8p 10GBase-T v2 zl Module Private Info Erased Up YES 2
B HP J9536A 20p GT PoE+/2p SFP+ v2 zl... Private Info Erased Up YES 2
C HP J9993A 8p 1G/10GbE SFP+ v3 zl2 Mod Private Info Erased Up YES 3
D HP J9550A 24p Gig-T v2 zl Module Private Info Erased Up YES 2
E HP J9993A 8p 1G/10GbE SFP+ v3 zl2 Mod Private Info Erased Up YES 3
F HP J9550A 24p Gig-T v2 zl Module Private Info Erased Up YES 2

Unfortunatly I can't disable the v2 modules on the core switch because things like our 20GB trunks to our server core and our firewall and content fillter are using multiple ports on the v2 modules. 

I hear you on patches.. that was were i started it had to be the fiber. We just had new 6 strand fiber run to each of the edge closets tested and certified by our contractor. I have tried using different paths and different patch cables and the same results happen 1GB fine, 10GB dropped packets and errors. Now having said that I have not tried a different brand of cable if you strongly feel that it could still be the patch cables I am willing to buy 3 new ones of a different brand to try?

In regard to the TDR does it work with 1GB tranceivers. I have on order, for another poject at our sports field press box, a pair of J4859C 1GB 1310 trancievers which should arrive today. I could certainly light those up to test if they suppor that. The only issue again is do ALL of the modules in the switch need to be in V3? I can put them into the V3 J9993A module on the core but Allow V2 modules is on for that switch and cant be disabled. 

Thanks,

~jake

---

Jkane wrote: Now having said that I have not tried a different brand of cable if you strongly feel that it could still be the patch cables I am willing to buy 3 new ones of a different brand to try?

Giving that the cost of 3 new Patch Cables (LC (UPC) / LC (UPC) 1310 nm single mode) is not so expensive, I think it's worth a try.

---

Jkane wrote: The only issue again is do ALL of the modules in the switch need to be in V3? I can put them into the V3 J9993A module on the core but Allow V2 modules is on for that switch and cant be disabled.

Are you asking about that regarding the usage of TDR or are you asking that generally speaking?

For TDR feature usage, as you may have read yet on 5400R zl2 ArubaOS-Switch Management and Configuration Guide for KB.16.01, it works only on v3 zl2 Modules that provide 100 Base-TX and 1000 Base-T ports (like the J9990A but I suspect only on its first 20 copper ports, while the J9993A is not listed as supported by that feature since it has SFP+ ports only) so I doubt that TDR feature will be of any help when tried on Optical Transceivers...but, absolutely, generally speaking (so not TDR related) it's not necessary nor mandatory to work with either v2 zl2 Modules or v3 zl2 Modules...you can have a 5400R zl2 and work with both in a mix...it's just a matter of having the 5400R zl2 switch that works with v3 zl2 Modules' 6th generation ASICSs at their full potential (which can/can't be related to your issue) instead of made those v3 zl2 Modules working in compatibility mode (demoted to v2 zl2 Modules' feature set).

So, for convenience, you can safely try to perform a no allow-v2-modules on Edge switches (read the manual about that command for its usage guidelines) and leave the actual settings (allow-v2-modules enabled) on the Core one...but I don't want to encourage you to introduce another difference since, as written, I'm not sure it will have for sure a benefical impact on your issue specifically...was only a side note.

Are your 10G Fiber Optics uplinks all distributed from both J9993A Module C and Module E on the Core or just from the Module E only? In other terms how are distributed the uplinks from the Core switch?

If the cables tracks are really OK...probably it's a bug, but it appears really strange.

parnassus,

I agree that cables are not to expensive and am happy to try that. Do you have something that you would recommend for LC/LC SM cables?

I guess i was asking about TDR just for my own education and to see if it was in anyway helpfull to my situation, which it appears not to be.

I did try doing the no allow-v2-modules on the edge switch and it made no difference at all.

Both module C and E do both provide fiber uplinks to edge closets. Just for ease of use i tagged out module C for our HS vlan and E for our MS vlan accross the parking lot. I have tried putting the tranciever for the bad edge closet in a free sfp+ port in the C module and tagging it out as MS, it made no difference.

Thanks,

~jake

I'm not clear from all of that if you've swapped transceivers & patch leads between a "bad" link and a "good" link to see if the problem follows the transceiver, patch leads or the run.

I have in the past had to use attenuators on short runs. Pretty rare though.
You can probably tell whether it's excessive power by getting a 5m fibre patch lead and winding it tightly around a stick, then substituting for the live patch lead in the "bad" run. If the level of errors diminishes, that would indicate it could be worth buying attenuators and seeing if they help.

Vince,

Hi thanks for responding. Yes we can swap the fiber paths between the 1GB trancievers and the 10GB trancievers and the issues follow the 10GB tranceivers,  packet loss and errors. 

The intresting thing is that this would be one of our longest runs, the 10GB trancievers are working (0k) on some of the shorter runs. 

Base on Parnassus's suggestion I am going to buy a few of the cables he suggested just to see if it makes any difference. I will order a longer patch to test idea. Would a the 10GB trancievers run with that 

Here is the show interface transceiver detail output

Transceiver in A21
Interface Index : 21
Type : SFP+LR
Model : J9151A
Connector Type : LC
Wavelength : 1310nm
Transfer Distance : 10.0km (9um),
Diagnostic Support : DOM
Serial Number :Private Info Erased

Status
Temperature : 41.109C
Voltage : 3.2729V
Tx Bias : 36.716mA
Tx Power : 0.5589mW, -2.527dBm
Rx Power : 0.3268mW, -4.857dBm

Time Stamp : Mon Aug 29 08:02:22 2016

Transceiver in A23
Interface Index : 23
Type : 1000LX
Model : J4859C
Connector Type : LC
Wavelength : 1310nm
Transfer Distance : 10.0km (9um), 550m (50um), 270m (62.5um),
Diagnostic Support : DOM
Serial Number :Private Info Erased

Status
Temperature : 36.546C
Voltage : 3.3021V
Tx Bias : 19.260mA
Tx Power : 0.2380mW, -6.234dBm
Rx Power : 0.1444mW, -8.404dBm

It does look like the 10GB tranciever is running at more power but I really have no idea what those numbers should be. 

I also do have a case open with HPE and worked with level 1 Friday afternoon and they have transfered me to Level 2 and im expecting a call this AM.

Thanks,

~jake

Judging by (optical) Tx/Rx Power Levels reported by the J9151 Transceiver's DOM (Diagnostic Optical Monitoring), used on one end of your longest run (is it just about 1 km long right?), reported here:

Transceiver in A21
Interface Index : 21
Type : SFP+LR
Model : J9151A
Connector Type : LC
Wavelength : 1310nm
Transfer Distance : 10.0km (9um),
Diagnostic Support : DOM
Serial Number :Private Info Erased

Status
Temperature : 41.109C
Voltage : 3.2729V
Tx Bias : 36.716mA
Tx Power : 0.5589mW, -2.527dBm
Rx Power : 0.3268mW, -4.857dBm

Time Stamp : Mon Aug 29 08:02:22 2016

That Transceiver's operating conditions look, AFAIK, quite normal (Temperature too).

A Tx Power of about 0,5 mW and a Rx Power of about 0,3 mW (where I suppose m stands for milli as in milli-Watt) mean respectively 500 uW and 300 uW (where u stands for micro-Watt), those values appear in the range of "normal operation" considered the lenght of your fiber optic longest run in which these transceivers are used.

A sensible difference between the optical Tx Power and the optical Rx Power measured on each end means that transceivers are not working on a very short run (let's say below 20/30 meters or more?)...that difference is the normal path loss...moreover that difference should be almost the same once measured on both links (Transceiver 1 Tx <--> Rx Transceiver 2 and Transceiver 1 Rx <--> Tx Transceiver 2).

The lower Tx/Rx Power values (expressed in Watt) - the higher (absolute) Tx/Rx Power levels values (expressed in dBm) - the worst the Transceiver working conditions are.

So if you see a Power level sensibly higher than, let's say, generally 6 or 7 dBm (>> |-7| dBm) - corresponding to a very very low Power wattage - and depending if you're evaluating the Tx or the Rx levels (those levels are differents [*]...so Rx normal operating levels can be slightly higher than Tx normal operating levels)...then that value means there is something wrong with the Tx or the Rx optic (dirty?) or with the whole fiber link (if both values are bad)...and that condition should trigger/generate a sort of Transceiver Low Power Alarm/Warning.

[*] probably Warning/Alarm's for signal Low level thresholds can be found at about -7/-10 dBm for optical Tx Power and -11/-14 dBm for optical Rx Power. Warning/Alarm's for signal High level thresholds instead can be found in the range of about +2 to -1 dBm for both Rx and Tx.

---

Yes we can swap the fiber paths between the 1GB trancievers and the 10GB trancievers and the issues follow the 10GB tranceivers,  packet loss and errors. 

 

---

I'm still not clear - have you taken the 10Gb transceivers off both ends of the "faulty" link and put them on a "good" link, and taken the 10Gb transceivers off both ends of a "good" link and put them on the "faulty" link?

That is always the very first thing I would do, because you need to know whether the errors follow the transceivers.

If the error stays on the same link, ie, the "good" transceivers" are now "bad", then you do the same thing with the patch leads at both ends.

If the error stays on the same link, then you repatch the link using known-good transceivers and known-good patch leads to different ports on a different switch module or even better an entirely different chassis, otherwise blame the structured cabling.

---

Jkane wrote: Do you have something that you would recommend for LC/LC SM cables?

Hi Jake, I thought, just as example, something similar to those ones advertised here (note that KK code stands for LC/UPC on both duplex connectors' ends).

Abe,

Here is the output from details

Edge Switch

show interfaces transceiver a21 detail

Transceiver in A21
Interface Index : 21
Type : SFP+LR
Model : J9151A
Connector Type : LC
Wavelength : 1310nm
Transfer Distance : 10.0km (9um),
Diagnostic Support : DOM
Serial Number :Private Info Erased

Status
Temperature : 38.691C
Voltage : 3.2729V
Tx Bias : 36.334mA
Tx Power : 0.5589mW, -2.527dBm
Rx Power : 0.3769mW, -4.238dBm

Time Stamp : Fri Aug 26 06:19:13 2016

Core Switch

show interface transceiver E2 detail

Transceiver in E2
Interface Index : 130
Type : SFP+LR
Model : J9151A
Connector Type : LC
Wavelength : 1310nm
Transfer Distance : 10.0km (9um),
Diagnostic Support : DOM
Serial Number :Private Info Erased

Status
Temperature : 36.074C
Voltage : 3.3441V
Tx Bias : 35.250mA
Tx Power : 0.5716mW, -2.429dBm
Rx Power : 0.3530mW, -4.522dBm

Time Stamp : Fri Aug 26 06:20:18 2016

thanks,

~jake

Can you please check if disabling tcp-push-preserve stop your Rx Drop?

If you still have Rx Drop after disabling tcp-push-preserve, please upgrade to KB.16.02.0012 or KB.16.01.0008 and then disable tcp-push-preserve and check if it resolve your issue.

The command is: "no tcp-push-preserve"

Hi SteveHP,

That's interesting...is the KB.16.02.0012 already published?

I checked few moments ago but the very latest Software Release was the KB.16.02.0010 (Published on August, 19th 2016).

Do you think that a (congested) ingress packet queue of the receiving port could be the root source of the issue?

To check the tcp-push-preserve feature status (globally) is also available the show tcp-push-preserve command.

Probably the tcp-push-preserve feature is yet enabled by default on these Switches since its introduction with the KB.15.15.0008 release (was the 2014).

Side note: interestingly - as commands - the [no] tcp-push-preserve and the show tcp-push-preserve are erroneously reported on the HPE ArubaOS-Switch Feature and Command Index 16.02 as consultable on the ArubaOS-Switch Management and Configuration Guide (MCG)...instead they are consultable exactly on the ArubaOS-Switch Advanced Traffic Management Guide (ATMG) at page 270.