Wired Intelligent Edge

Hi everyone,

We're currently evaluating an upgrade from AOS-CX 10.13.1130 to 10.16.1051 for our production environment and would appreciate hearing from others who have already made the same move.

Our environment consists of two CX10000 VSX clusters forming the core of our EVPN/VXLAN data center fabric. This is mission-critical infrastructure, so stability and reliability are our primary concerns.

I'm interested in both positive and negative experiences, especially regarding:

• Overall stability
• Known bugs or unexpected behavior
• VSX synchronization
• EVPN/VXLAN reliability
• Upgrade experience from 10.13.x (or other previous releases)
• How long you've been running 10.16.1051 in production

If you've upgraded from 10.13.x to 10.16.1051, I'd especially like to hear whether the upgrade was uneventful or if you encountered any issues during or after the migration.

If you're running this release successfully, I'd also be interested in the approximate size of your deployment (number of CX10000 switches or overall fabric size).

I'm particularly interested in issues that only became apparent after days or weeks of production uptime, not just during initial deployment or lab testing.

Any feedback, whether you've had a completely stable experience or encountered specific issues, would be greatly appreciated.

Thanks in advance!

10.13 to 10.16 is a supported one-step upgrade per the 10K release notes, and it's LSR to LSR, so the path is right. Things to check before the window: the Pensando PSM compatibility matrix (and note downgrade requires stripping unsupported PSM config first), the 10K-specific Known Issues for 1051, and whether a newer 10.16 MR shipped.

Plan extra time per member, the DPU firmware applies during boot and a VSX pair upgrades sequentially. Raise linkup-delay-timer toward 600s for an EVPN table that size and exclude your underlay LAGs. Single-homed ports drop for the full member reboot, so walk your orphans first.

No long-uptime 10K war stories from me, but 1051 is six maintenance drops into the branch, which is when I start trusting one. Early builds churned (1005 got pulled).

------------------------------
Dustin Burns

Lead Mobility Engineer @Worldcom Exchange, Inc.

ACCX 1271| ACMX 509| ACSP | ACDA | MVP Guru 2022-2023
If my post was useful accept solution and/or give kudos
------------------------------

Sorry to hear that, losing a member mid upgrade with no window to fall back on is about as bad as it gets. Before you kick off the next attempt, get a full backup and image/boot set inventory on the new secondary, and confirm it's running the exact same base build as production before it ever joins the VSX pair, so the upgrade isn't doing a bigger jump than you think.

Ask TAC for the RCA on the dead unit before you touch the second one: if it's a DPU firmware issue specific to that image path, they may tell you to stage the secondary at 10.16 standalone first and rejoin it, instead of letting the automated sequence do it live. Also pull a fresh health check on the primary DPU before you start, so you've got a known good baseline if something goes sideways again.

------------------------------
Dustin Burns

Lead Mobility Engineer @Worldcom Exchange, Inc.

ACCX 1271| ACMX 509| ACSP | ACDA | MVP Guru 2022-2023
If my post was useful accept solution and/or give kudos
------------------------------

Original Message:
Sent: Aug 03, 2026 11:18 AM
From: MR-90c854
Subject: CX10000 AOS-CX 10.16.1051 with VSX/EVPN-VXLAN – Production Experience?

Update to my initial post:

To clarify the context and the urgency behind this thread: The reason I asked is because our first upgrade attempt failed severely during a live production run.

Since our environment is a 24/7 mission-critical infrastructure, we do not use maintenance windows. We have to rely entirely on the VSX cluster's built-in redundancy to perform live upgrades during full production.

During the automated VSX upgrade sequence from 10.13.1130 to 10.16.1051, the secondary CX10000 went completely dark a few minutes after the process started. It became entirely unresponsive, lost all console/management access, and never booted back up.

This left our entire production fabric running on a single, non-redundant primary switch while we scrambled to open an emergency TAC case. Ultimately, the secondary unit was declared dead and required an RMA/hardware replacement.

We now have the replacement hardware in place and are facing our next upgrade attempt. Because we must do this in production again, we are highly alert.