Wireless Access

Head-scratcher here.

A clustered pair of 7205 controllers have been faithfully doing their job for a few years, connecting back to a remote MM through a Site-to-site VPN.  They are connected using IP-and-PSK for their authentication, and there haven't been issues with that.

A few days ago, one of the controllers lost its connection to the MM.  It shows as "down" in the "show switches" list, but "Up" in Airwave (indeed, the datapath sessions table shows it is communicating with Airwave and also sending some info to the MM (port 6633 (openflow) and another odd port (8822? not 8211)).  The device shows as "Update Required" from MM side, and "Master Unreachable/Last Snapshot" from the controller side.  I turned on disaster-recovery and then off again, to try to get it to pull down the new config - but it is stuck at ConfigID -1.

I've checked the logs, and the most relevant things I see are about heartbeat timeouts and the IKE tunnel going down due to expiration.  In the logs I see that it did establish a tunnel this morning to the MM but a bit later it went down.

I checked the keys are correct, and even reentered them in the MM exactly how they are from the MD.

Any other ideas?  There haven't been any material changes to the devices or their config hierarchy node over the last week, and the other controller in the cluster has been connected fine this whole time.

------------------------------
- ryh
------------------------------

Follow-up: Had the customer reboot the MM (MD was rebooted yesterday but problem persisted) and the same issue exists.

I’m wondering if a packet capture of the datapath on the controller/MM would help at all?

---------------------------------
ryh
---------------------------------



Original Message:
Sent: Jan 12, 2023
From: ryh
Subject: IPsec tunnel between one MD and MM/MC is failing to form

Head-scratcher here.

A clustered pair of 7205 controllers have been faithfully doing their job for a few years, connecting back to a remote MM through a Site-to-site VPN.  They are connected using IP-and-PSK for their authentication, and there haven't been issues with that.

A few days ago, one of the controllers lost its connection to the MM.  It shows as "down" in the "show switches" list, but "Up" in Airwave (indeed, the datapath sessions table shows it is communicating with Airwave and also sending some info to the MM (port 6633 (openflow) and another odd port (8822? not 8211)).  The device shows as "Update Required" from MM side, and "Master Unreachable/Last Snapshot" from the controller side.  I turned on disaster-recovery and then off again, to try to get it to pull down the new config - but it is stuck at ConfigID -1.

I've checked the logs, and the most relevant things I see are about heartbeat timeouts and the IKE tunnel going down due to expiration.  In the logs I see that it did establish a tunnel this morning to the MM but a bit later it went down.

I checked the keys are correct, and even reentered them in the MM exactly how they are from the MD.

Any other ideas?  There haven't been any material changes to the devices or their config hierarchy node over the last week, and the other controller in the cluster has been connected fine this whole time.

------------------------------
- ryh
------------------------------

Resolved.

The resolution was to have the far-end firewall clear the session of that traffic, and then the tunnel came up. Nothing on the Aruba side - just the intermediate firewalls were somehow interfering with the IPSec formation. Weird!

---------------------------------
ryh
---------------------------------



Original Message:
Sent: Jan 12, 2023
From: ryh
Subject: RE: IPsec tunnel between one MD and MM/MC is failing to form

Follow-up: Had the customer reboot the MM (MD was rebooted yesterday but problem persisted) and the same issue exists.

I’m wondering if a packet capture of the datapath on the controller/MM would help at all?

---------------------------------
ryh
---------------------------------



Original Message:
Sent: Jan 12, 2023
From: ryh
Subject: IPsec tunnel between one MD and MM/MC is failing to form

Head-scratcher here.

A clustered pair of 7205 controllers have been faithfully doing their job for a few years, connecting back to a remote MM through a Site-to-site VPN.  They are connected using IP-and-PSK for their authentication, and there haven't been issues with that.

A few days ago, one of the controllers lost its connection to the MM.  It shows as "down" in the "show switches" list, but "Up" in Airwave (indeed, the datapath sessions table shows it is communicating with Airwave and also sending some info to the MM (port 6633 (openflow) and another odd port (8822? not 8211)).  The device shows as "Update Required" from MM side, and "Master Unreachable/Last Snapshot" from the controller side.  I turned on disaster-recovery and then off again, to try to get it to pull down the new config - but it is stuck at ConfigID -1.

I've checked the logs, and the most relevant things I see are about heartbeat timeouts and the IKE tunnel going down due to expiration.  In the logs I see that it did establish a tunnel this morning to the MM but a bit later it went down.

I checked the keys are correct, and even reentered them in the MM exactly how they are from the MD.

Any other ideas?  There haven't been any material changes to the devices or their config hierarchy node over the last week, and the other controller in the cluster has been connected fine this whole time.

------------------------------
- ryh
------------------------------

Please take a look at the end of the thread here:  https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=31471

------------------------------
Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

HPE Design and Deploy Guides: https://community.arubanetworks.com/support/migrated-knowledge-base?attachments=&communitykey=dcc83c62-1a3a-4dd8-94dc-92968ea6fff1&pageindex=0&pagesize=12&search=&sort=most_recent&viewtype=card
------------------------------

Original Message:
Sent: Jan 13, 2023 01:58 PM
From: ryh
Subject: IPsec tunnel between one MD and MM/MC is failing to form

Resolved.

The resolution was to have the far-end firewall clear the session of that traffic, and then the tunnel came up. Nothing on the Aruba side - just the intermediate firewalls were somehow interfering with the IPSec formation. Weird!

---------------------------------
ryh

Original Message:
Sent: Jan 12, 2023
From: ryh
Subject: RE: IPsec tunnel between one MD and MM/MC is failing to form

Follow-up: Had the customer reboot the MM (MD was rebooted yesterday but problem persisted) and the same issue exists.

I'm wondering if a packet capture of the datapath on the controller/MM would help at all?

---------------------------------
ryh

Original Message:
Sent: Jan 12, 2023
From: ryh
Subject: IPsec tunnel between one MD and MM/MC is failing to form

Head-scratcher here.

A clustered pair of 7205 controllers have been faithfully doing their job for a few years, connecting back to a remote MM through a Site-to-site VPN.  They are connected using IP-and-PSK for their authentication, and there haven't been issues with that.

A few days ago, one of the controllers lost its connection to the MM.  It shows as "down" in the "show switches" list, but "Up" in Airwave (indeed, the datapath sessions table shows it is communicating with Airwave and also sending some info to the MM (port 6633 (openflow) and another odd port (8822? not 8211)).  The device shows as "Update Required" from MM side, and "Master Unreachable/Last Snapshot" from the controller side.  I turned on disaster-recovery and then off again, to try to get it to pull down the new config - but it is stuck at ConfigID -1.

I've checked the logs, and the most relevant things I see are about heartbeat timeouts and the IKE tunnel going down due to expiration.  In the logs I see that it did establish a tunnel this morning to the MM but a bit later it went down.

I checked the keys are correct, and even reentered them in the MM exactly how they are from the MD.

Any other ideas?  There haven't been any material changes to the devices or their config hierarchy node over the last week, and the other controller in the cluster has been connected fine this whole time.

------------------------------
- ryh
------------------------------