Wired Intelligent Edge

Hi all!

My company uses HP 2530 8G, 24G and 48G swithes and we monitor port counters via SNMP. Now and then we are experiencing port counters freezing at those switches. When the counters stop to increment, one of the Total Bytes port counter at the particular switch is always near 2^32 value:

swa# show interface

 Status and Counters - Port Counters

                                     
  Port  Total Bytes    Total Frames  
  ----- -------------- --------------
  1     3,535,555,073  1,293,926,141 
  2     0              0             
  3     4,276,546,335  1,527,795,391 
  4     0              0             
  5     2,460,528,005  263,988,780   
  6     0              0             
  7     0              0             
  8     0              0             
  9     0              0             
  10    459,308,314    1,794,836,185 

At 2530-48G switches we can observe freezing a block of port counters - port1..24 or port 25..28. The counters are dead and doesn't help resetting them via CLI nor Menu. The only way to get counters alive is to reboot the switch, which is really not desirable in production environment. We have tried different firmwares, from 15.12.0014 to 15.16.05 without any change.

Is it a HW or SW bug? Is anyone else experiencing this syndrome?

Petr

I've got the same issue. the counters were working and suddenly stopped. The web interface and snmp show no changes over many hours of known traffic which most certainly should have increased them.

I can't reboot now but I'm betting that will fix it but what I want to know is why they stopped globally and is this going to happen on a regular basis?

After some investigations I have found solution: query 64 bit counters.

Now it works for more than 4 months without any problem.

In our case we use MRTG and we changed its configuration to:

community@switch:::::2

where "2" means SNMP version 2 requests. This allow you to poll 64 bit traffic counters in the switch.

Side effect is that not only MRTG counters do not freeze, but also web interface counters and CLI counters stay alive!

Petr

What version of code?  I wasn't able to immediately reproduce.  Will check to see if this was fixed at some point.

Sorry, I missed the versions you tried and couldn't reproduce on any of those.  Any chance you could attach the config?  

Today I found another one with dead counters on ports 25-48. It has 255 days uptime.

Here is config (ports 37,39,41,43,45  are attached to Motorola wifi APs) :

net300# show version
Image stamp:
 /ws/swbuildm/rel_orlando_qaoff/code/build/lakes(swbuildm_rel_orlando_qaoff_rel_
orlando)
                Nov 19 2014 15:11:53
                YA.15.16.0005
                330
Boot Image:     Primary


net300# show config

Startup configuration: 33

; J9775A Configuration Editor; Created on release #YA.15.16.0005
; Ver #06:04.9c.63.ff.37.27:12
hostname "net300"
timesync sntp
sntp unicast
sntp server priority 1 172.16.0.2
time daylight-time-rule western-europe
time timezone 60
ip default-gateway 172.16.0.1
interface 37
   speed-duplex auto-100
   exit
interface 39
   speed-duplex auto-100
   exit
interface 41
   speed-duplex auto-100
   exit
interface 43
   speed-duplex auto-100
   exit
interface 45
   speed-duplex auto-100
   exit
snmp-server community "xxx" operator
snmp-server community "yyy" unrestricted
snmp-server contact "ProCurve Switch 2530-48G" location "Designstudio"
vlan 1
   name "DEFAULT_VLAN"
   no untagged 46
   untagged 1-45,47-52
   ip address 172.16.4.20 255.255.0.0
   exit
vlan 202
   name "symbol"
   tagged 48
   no ip address
   exit
vlan 888
   name "video"
   tagged 18,48
   no ip address
   exit
vlan 999
   name "guest"
   tagged 37,39,41,43,45,48
   no ip address
   exit
vlan 1012
   name "facility"
   untagged 46
   tagged 48
   no ip address
   exit
vlan 1014
   name "wifi_wh"
   tagged 37,39,41,43,45,48
   no ip address
   exit
spanning-tree
spanning-tree 37 admin-edge-port
spanning-tree 39 admin-edge-port
spanning-tree 41 admin-edge-port
spanning-tree 43 admin-edge-port
spanning-tree 45 admin-edge-port
spanning-tree instance ist 37 path-cost 200000
spanning-tree instance ist 39 path-cost 200000
spanning-tree instance ist 41 path-cost 200000
spanning-tree instance ist 43 path-cost 200000
spanning-tree instance ist 45 path-cost 200000
no tftp server
no dhcp config-file-update

Hmm, I was not able to reproduce on that switch with your config.  A few questions:

• What speeds are the "frozen" ports linked at?  100?  1000?
• What management system are you using to collect SNMP stats?
• Would you be willing to run some diagnostic commands?  I'd like to read the hardware couters directly to see if they are "stuck".  If so I will send you a follow up message to your forum "inbox".  

Thanks.

-ports are connected at mixed speeds - 10,100,1000 mbps

-for statistics collecting we use mrtg (http://oss.oetiker.ch/mrtg/)

-yes, I am keen on solving this issue

Petr

I have to admit that using 64-bit counters didn't help. The issue occurs occasionally, switches had different uptimes - from 30 to 226 days.

I am still waiting, if anybody can help. Rebooting is not solution in production environment.

I had no luck reproducing it.  Time for a support ticket.  I sent you a message for some follow up info, sorry this has dragged out.

Will report back to the thread if we find a solution (or fix).  

Hello,

There is any solution for this problem?.

I have the same issue (HP J9772A 2530-48G-PoEP Switch, Software revision YA.15.15.0010). Port 1-24 works well and 25-48 have freezing counters

Thanks

In progress.  We have reproduced it in the lab and are investigating.  Will post back when a solution is available.  Thanks for your patience, this is a tricky one.  

We have the same issue on multiple 2530 Switches 48G and 48nonG (Software: YA.15.17.0009)which is extremly annoying...
On some of them only a few ports are affected every other month or so..., on a new one we've put into production a few weeks ago all the ports are affected once a week :(
And on some of them it works without any issues for months now...

I've noticed that everytime this happened each of the switches did an sntp time update immediately prior to the counters freezing:

"I 01/31/16 07:26:27 00413 SNTP: Updated time by 4 seconds from server at
192.168.10.40. Previous time was Sun Jan 31 07:26:23 2016. Current
time is Sun Jan 31 07:26:27 2016."

I really hope this gets solved soon.

For everyone as annoyed as I'am by this bug:

Disable SNTP updates/time sync, then the counters won't freeze anymore...

This workaround should be sufficient for most people until HP decides to finally fix this...

Firmware 16.01.0007 finally fixed this:

"CR_0000183578 Symptom: Interface packet counters do not work correctly.

Scenario: When the time is changed on the switch, either by SNTP or manually via the CLI,
there is a potential for the interface packet counters to stop incrementing. They may potentially
start incrementing again, but the counters might not be accurate. Rebooting the switch correctly
resets the interface counters.

Workaround: Avoid updating the switch time if the interface counters’ accuracy is needed, or
reboot the switch to reset the counters."