Comware

On July 3rd, my PCM received autoupdate package #5 and since then I get bogus high water marks. I called NetHelp and was told to remove all traffic settings, delete the switches, let them rediscover, and add back all the traffic devices. I've done this before and was pretty sure I got the security settings all done globally but not having much joy now. Now the traffic monitor reports all segments as "not reporting" and there are security related errors in the event log.

It should not be this difficult. I can use the CLI Wizard and it takes the defaults just fine but but if I try to add Authorized Managers using Device Manager, it fails.

Hmmm... time for another phonecall I guess...

Keep us informed on what they say... Would be interested in finding out if this is a general bug with Update 5 or whether it is soecific to your kit....

Cheers!

Well... I decided to hold off phoning just yet and worked through the security errors. For some reason, the write community string changed in the global preferences. All the other switches that I have in PCM without traffic monitoring, I did not delete/rediscover and they were all set to "Use PCM Defaults" too yet they worked. What finally caught my eye was the number of asterisks masking the write community string didn't match the number of chars in the string. Go figure!

So, the Test Communication Parms wizard passes now with flying colours but still no devices reporting. Do I delete all the traffic settings now and start over or sit on my hands and wait?

OK, got the circulation going in my hands again... :p

Removed the traffic monitor settings and added them back in and now the segments are all responding. Time will tell if the bogus values are gone, never to return... so far so good.

Oh well, I had a few ports that were discovered at the wrong speed previously so rebooting the switches and rediscovering them sorted that at least. I can't wait for the the next release that purports to fix this speed thing.

Dang! whilst typing this out, I glanced up at my other monitor and see a bunch of pegged needles and high water marks. See attached. Time to call NetHelp I guess.

Curious whether anyone else is seeing this with update 5?

Escalated to division.

I can say that I have the exact same problem. Just updated to update 5 and the traffic monitor goes bananas! Even on ports not connected. But only on the subnet that the PCM is on, not on any of our other subnets.

Add me to the list too! Same symptoms even after re-discovery, re-establishing "stats" and "sampling" settings, and a server restart.

I don't see a relationship with the subnet proximity to the PCM server however. Only a select portion of my network switches are included in the free-for-all.

Mike

FYI my incident number is 3215393241 in case you want to call in and be added to the list.

A division engineer was able to duplicate this issue in his test lab late yesterday. No timeframe for a fix and the only proposed work-around is to completely un-install and re-install PCM with Update 4.

I cannot find any upadates for PCM+ 2.1 at http://www.hp.com/rnd/software/network_management.htm although they are still available through the FTP site.

I will it as-is and fly blind for now and see what Monday brings.

Oh, I forgot to mention that I did NOT enable the setting "Download and install automatically", opting only for "Notify if updates are available" but yet somehow automagically this update installed itself. :(

I don't want to make a big stink about it because I would have installed it anyway if notified and would have ended up in the same situation I am in now. What does bother me a little though is that I was unaware of the automagic update and based partly on the bogus reporting, I scheduled an emergency service window to reboot the entire mesh. It was due for a reboot anyway as one of the switches was ignoring too many pings and SNMP gets and WebAgent on another was not responding.

There is still no word on this issue.

A bug was introduced in autoupdate package #5 (AU5) that led to the symptoms you reported. There were many changes made to the traffic data collector in AU5, almost all related to improving collection scalability. Normally changes of this magnitude would not be included in an autoupdate but it was felt that the potential benefits in terms of scalability were worth the heightened risk, as the code had been under test within the development lab for about 6 weeks prior to the release of AU5.

In a nutshell, our testing failed to anticipate a particular pattern of IP addressing which anyone who has seen this bug has in their environment. When the collector receives sFlow samples from devices that meet the pattern it gets confused when computing hash keys from the incoming sFlow datagrams, and essentially multiplexes the data from multiple ports (across the devices) into the same set of internal data counters. This multiplexing leads to the erroneous values you observed in Traffic Monitor. Note that *only* sFlow data collection is affected by this bug - AU5 will correctly report statistics and sampled XRMON data.

This problem has been remedied and another autoupdate that fixes the problem, AU6, is on the way this week. We are truly sorry for any issues that this bug has caused for you. Please rest assured that we have augmented our testing procedures to ensure that nothing of this sort will occur again, and that changes proposed for inclusion in autoupdate will undergo additional scrutiny before they are accepted from this point forward.

Regards,

SVB

Thanks Steve, for the update. Is it a known issue whereby setting for notify actually installs the update? Do you want to persue this? On the same incident number or open a new one? I just changed the frequency from once a week to once a day.

Les,

It is not a known issue whereby setting for notify actually installs an autoupdate. We will definitely pursue it, and in fact have been trying to replicate the behavior internally since you first mentioned the issue with no success thus far. Let's just use the same incident number; I will make sure our support engineers are aware that the autoupdate facet of the incident is still open.

AU6 was posted today, and should fix the erroneous traffic numbers you were seeing with AU5.

Regards,

SVB

I have updated to update 6 now. But now all I get i "Awaiting Connection" in Traffic Monitor. I tried to delete the devices and discover them again, but I get the same result.
Is there some configuration that I have missed?

Per,

Go to the Windows Service control panel and restart the HP ProCurve Traffic Launch Service. The message you're seeing indicates that for some reason the socket connection between the traffic data collector and PCM server processes has not been established - the PCM server is awaiting that connection. After you restart you should see that message disappear, and if you look at the Processes tab in Windows Task Manager you should see Trafficd.exe - the traffic data collector - listed.

Regards,

SVB

OK, I dragged my heels to see what PCM was going to do with the update and all it did was notify me and make a liar out of me. It now has me doubting myself... as I move the mouse around, I see that the focus need not be right on the radio button but anywhere on the line so I wonder if I inadvertently switched between Download and Notify with the slip of the mouse.

Lets not persue this any further at this time. I will watch it closely and if does it again, re-open this case. The bogus high watemarks are gone.

Thanks

OK, I re-opened the incident. Despite PCM being set just to notify, it downloaded and installed Update 7.

Speaking of update 7, it may be too early to tell but so far traffic is reporting more points and for a longer time. Generally my graphs look like a bad dentist worked on them and they seem to diminish what they will report on within hours of being rebooted. Eventually the top talkers reported will even include insignificant traffic like DHCP and DNS requests.

The release notes make no mention of traffic reporting changes.

Hi

I wonder what kind of data collection methode you use, is it sampling or stats ?

Good Luck !!!

I use only Sampler.

So far on day two, my graphs still show more data than toothless gaps and all Top 5 connections are plausible without miniscule any ping or DHCP or DNS stuff included.

I'm guessing something was improved but not mentioned in the release notes. Maybe I should "Disable automatic updates" entirely and stay frozen on this build which appears to be the best it has been in a very long time.

NAW! I still need the port speed change autodetect fix. What was I thinking?

Les,

There were two bugs fixed which likely influenced the traffic data you were seeing before autoupdate 7 (AU7). I checked the release notes after seeing your posting, and the text for these bugs had been inadvertantly omitted. The release notes have now been updated to include this text.

On the topic of autoupdate, since you first opened the incident about AU5 automatically downloading and installing we were unable to reproduce it. With AU7 we have successfully reproduced and diagnosed the issue; it will be fixed in AU8 which, somewhat ironically, will be automatically downloaded and installed due to the same bug. But after AU8 this mechanism should behave properly.

Regards,

SVB

Good to hear that they were documented fixes. I would have worried had they been coincidental, that they could revert. I was going in circles with Division a while back on those issues and eventually gave up on it so am very pleased to see the improvement now.

Thanks

Les -

The AU7 fix list is updated (corrected), as of today.

See ftp://ftp.hp.com/pub/networking/software/PCM21-Update7-RelNotes-59908860.pdf , page 8.

Regards,
Ralph