Comware

Hi,

Are all upgrades for IRF on HPE 5930 switches performed using the In-Service Software Upgrade (ISSU) method? We are trying to build a resilient network which uses IRF but which can be upgraded without have to reboot all members of the IRF. Is this possible or are there some upgrades which cannot be done using ISSU?

      Regards,

          Gerry

Gerry,

It depends.  You need to consult the release notes on the version that you want to upgrade to.  Some you can and some you cannot. 

All upgrades can be staged on the switch and then reboot them when you are able to if ISSU is not supported.  It is a nice idea to be able to upgrade with no downtime but there are a lot of cases where you don't have that choice.,

Regards,

David

On my 4 member stack with 5900AF I did not manage to do upgrade with ISSU. It failed miserably splitting IRF into two parts with same IP (just disaster really)

Seb

Seb,

Did you look at the release notes to see the ISSU compatibility matrix to see if what you were trying to do was supported?  The information is there. 

I'm sorry your IRF stack split with the same IP addresses.  I hope you were able to recover it quickly before too many issues occurred.

Regards,

David

Isn't this the issue:

When using more than 2 units in the IRF system, ISSU assumes the peer devices are connected to all switches in the IRF system. For example: if you have a server IRF system with 4 switches, server are assumed to be connected to each of the 4 switches. This is highly unlikely, and therefore I recommend to stick to 2 units in the IRF system for any deployment which requires ISSU. When a customer can have a maintenance window and accepts downtime, more than 2 switches in the IRF can be used of course."

4 member IRF stack seems to need full reboot

<HPE5900-SR1>display version comp-matrix file ipe flash:/5900_5920-CMW710-R2432P01.ipe
Verifying the file flash:/5900_5920-cmw710-r2432p01.ipe on slot 1..........Done.
Boot image: 5900_5920-cmw710-boot-r2432p01.bin
  Version:
  7.1.045

System image: 5900_5920-cmw710-system-r2432p01.bin
  Version:
  7.1.045-Release 2432P01
  Version compatibility list:
  7.1.045-Feature 2421
  7.1.045-Feature 2424
  7.1.045-Feature 2426
  7.1.045-Feature 2427
  7.1.045-Feature 2428
  7.1.045-Release 2416
  7.1.045-Release 2418P01
  7.1.045-Release 2418P06
  7.1.045-Release 2422
  7.1.045-Release 2422P01
  7.1.045-Release 2422P02
  7.1.045-Release 2423
  7.1.045-Release 2432P01
  Version dependency boot list:
  7.1.045

  Slot                        Upgrade Way
  1                           Reboot
  2                           Reboot
  3                           Reboot
  4                           Reboot

That is just plain awful and confusing, as there is mention of multi member ISSU upgrade here:

http://www.h3c.com.hk/Technical_Support___Documents/Technical_Documents/Switches/H3C_S12500_Series_Switches/Configuration/Operation_Manual/H3C_S12500_CG-Release7128-6W710/01/201301/772597_1285_0.htm#_Toc341179349

Hi Spgsitsupport!

So no, there is no assumption whatsoever on the part of ISSU.  The only thing that it does is check to see if there is a possibility of doing a compatible upgrade in the ISSU sense.  Engineering tests this prior to release to come up with the matrix.

It is up to the customer to ensure that their servers are dual-homed to reduce the possibility of a network outage and to minimize that downtime.  If an inventory of all connections to the stack are done and the links are split among the stack members there is a chance that there will be a reduced outage..  THe members still need to reboot though.

The latest versions of Comware 7 on some platforms allow up to 8 members.  That's a lot of equipment but some sites need that connectivity.

Regards,

David

---

VoIP-Buddy wrote: It is up to the customer to ensure that their servers are dual-homed to reduce the possibility of a network outage and to minimize that downtime.  If an inventory of all connections to the stack are done and the links are split among the stack members there is a chance that there will be a reduced outage..  THe members still need to reboot though.

Just dual-homed or, better, multi-homed [*] to keep the outage window as low as possible (or to let the necessary outage time to not impact IRF Stack's multi-homed connected hosts at all)?

Just curious about such scenario...

[*] As example when the IRF Stack is made of more than, let's say, four or more members.

As I said, last time I tried it with 4 members it failed in spectacular way.  Followed instructions, upgraded member, rebooted, swapped master & then I had it split, with both parts using the same IP

Total disaster

Seb

Ofcourse each of my servers is dual connected, which did not help in that case

Hi,  We have a 4 x FlexFabric 5930 IRF stack., slots 1 to 4, connected in ring topology with fully connected BFD MAD. The odd slots, 1 & 3 are in building A and the even slots, 2 & 4 are in building B. The ring is slots 1, 3, 4, 2 and back to 1. Servers and storage in building A are dual homed and connected to slots 1 & 3. Likewise, servers and storage are dually connected to slots 2 & 4.

 Our local HPE support informed us that ISSU updates are not supported for more than a 2 slot IRF stack. This seems strange guiven that a stack can have something like 15 members. Is this HPE official policy?

 In any event, we have sucessfully performed an ISSU compatible upgrade by following the manual. After the "issu run switchover" we performed "issu commit slot N" on each of the remaining slots in turn, waiting until each one had rejoined the stack before continuing on to the next slot. The was no server or storage downtime, as each is dually connected, only one slot was down at any given time and the slots are connected in a ring. This is what one would expect, nothwithstanding HPE's disclaimer above.

  Our real issue is how to prform an imcompatible upgrade. The issue here is that after a stack slot that has been upgraded it cannot rejoin the current IRF stack and MAD will shutdown all its ports. Any idea how a ring stack can be upgraded without downtime?

    Regards,

      Gerry

@GerryOBrien

Did you do the ISSU on 4 member stack as per:

https://community.arubanetworks.com/aruba/attachments/aruba/NorwegianForum/16/1/ISSU%20Docs.pdf

or

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_sf-EN_US000005131

Seb

Starting on sheet 22 (of 68) there are three "Examples of using issu commands for ISSU on a four-member IRF fabric", respectively:

• Feature upgrade to a compatible version,
• Feature upgrade to an incompatible version (upgrading one subordinate member first),
• Feature upgrade to an incompatible version (upgrading multiple subordinate members first).

Hi Parnassus,

  Thanks you for the links. These are the most up-to-date doumentation we have seen for ISSU upgrades. We are now quite comfortable peforming a compatible upgrade on a four member IRF stack. Our problem is that we believe we will get a split-brain at some point in perforing an incompatible upgrade. Our proposed solution is below.

  Regards,
      Gerry



Our network is a 4 switch IRF stack connected in a ring and also with fully connected BFD MAD. Switches 1 & 2 are in one building and 2 & 4 in another. All servers and storage are dually connected to either switches 1 & 2 or else 3 & 4.Basically, our procedure is to upgrade all switches and reboot switches 3 & 4 first, splitting the IRF in two. After switches 3 & 4 have restarted they are running the upgraded software but all their non-MAD ports are disabled but we have  connectivity through switches 1 & 2. We then temporarily disable the MAD on switches 1 & 2 by shutting down the MAD Vlan. We then restore the MAD on switches 3 & 4 and, after a brief delay, reboot switches 1 & 2. When switches 1 & 2 have restarted all four switches are running the upgraded software so switches 1 & 2 join the IRF of switches 3 & 4. 

This manual  was written by somebody does know very littl;e or way too much.

It is completly inaccurate.

To start it references .bin file where the system upgrade comes ar .ipe

It does not assume how the connection to the stack is happening (IP or serial console)

It is is by IP?, in which case I will most likely loose connection during:

# Perform a main/backup feature process switchover.
<Sysname> issu run switchover

Or am I connect by serial, and if then to WHICH member in the stack?

Also:

# Upgrade the feature on subordinate member 2. After the upgrade, the subordinate member will leave
the original IRF fabric and form a new IRF fabric by itself.

If I remember that is exactly what happened last time, new IRF fabric with 1 member got created, but (as expected logically) with IP of the original IRF stack (could not be otherwise!)

So I had two identical IPs on the network, rather unworkable!

So the instructions are FAR from clear how to doi it properly with 4 members & an incompatible version (which I assume what I am facing)

Hi Spgsitsupport!

So you have to be careful here... Comware is Comware until it is not... 

The first document is taken out of context.  That is a document for ISSU for a multi-slot switch like a 12500, 12900, 11900, etc.  The ISSU procedure for that class of switch is different to some extent than the ISSU process for a TOR switch like the 5930.  The second document is closer to the reality of this situation.

They both look like official HPN documentation but not specifically for the 5930.

Regards,

David

...and, worth to mention, the latest available HPE FlexFabric 5930 Switch Series IRF Configuration Guide (Part Number 5998-7767R valid for R242x, document version 6W100-20151220) doesn't contain a single reference to "ISSU"...neither the related IRF Command Reference reports any reference to "ISSU"...so will interesting to drill down a little bit why it is so for the 5930.

All I want to know how to upgrade 4 switch IRF stack, surely that is not too much to ask from company that charges big bucs for the units?

The only way I managed to upgrade it, was to do FULL reboot.

Which I find unacceptable (on this occassion I had full shutdown planned anyway, so was not a problem, but in a future it will certainly be a big problem)

Also at which point replication of updated system .bin file is supposed to happen from master to other members?

It did NOT happen before system reboot (so I did the extraction on each member by hand, in case it all went pear shape)

Seb

---

spgsitsupport wrote: Ofcourse each of my servers is dual connected, which did not help in that case

---

And that's the reason because I asked: if your Server were multi-homed instead of only dual-homed (so each Server host connected to the IRF Stack had four uplinks equally distributed to all four members of the IRF Stack...instead of only two links as I understand your Servers had) things could have been better (in terms of impact against the necessary outage time) or essentially not?

I mean: managing (or suffering) an IRF split brain is a thing, managing a Compatible ISSU with a sub group of 2 IRF members (of the grand total of 4) that reboot while Server hosts still have two links up (and two others down) to remaining still active IRF members is another thing...

Hope to be not Off Topic with my question.

I physically can not have each server connected to each switch as theey are different kinds:

2x JG336A
2 xJC772A

Fibre & copper, 2 of each.

ISSU upgrade still should NOT destroy the IRF during upgrade & spit it!