SD-WAN

 View Only

  • 1.  VPNC Microbranch connection issues

    Posted Feb 13, 2024 04:44 PM
    Edited by ryh Feb 13, 2024 04:50 PM

    Greetings, Wizards!

    I have been battling an issue with microbranch-VPNC communications for a customer.  What appeared at first to be a end-device issue (ip-phone on a wired port, CL2 to the DataCenter VPNC), now appears to be the microbranch connection to the VPNC itself.  It seemed to be just one of the microbranch APs, but I have replicated the issue on my own (303H).

    What I see in the logs is the tunnel not establishing, or trying to re-establish often.  The wired client is put into a "deny all" role since the VPNC is not responding for the auth-request (show ap debug auth-trace-buf shows "server out-of-service").  The AP "show ata state" information reveals something like the following:

    ub_303h# show ata state-transition-history
     
    Tunnel State Transition History
    -------------------------------
    Timestamp            Peer IP            UUID                                  Current State   Event              Next State
    ---------            -------            ----                                  -------------   -----              ----------
    2024-02-13 20:53:21  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  INIT            TUN_RECV           CONNECTING
    2024-02-13 20:53:27  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      TUN_RECV           CONNECTING
    2024-02-13 20:53:30  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      PROBE_OK           CONNECTED
    2024-02-13 20:53:56  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTED       TUN_REKEY          REKEYING
    2024-02-13 20:54:27  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  REKEYING        HB_TIMEOUT         CONNECTING
    2024-02-13 20:54:28  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      HB_TIMEOUT         INIT
    2024-02-13 20:57:00  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  INIT            TUN_RECV           CONNECTING
    2024-02-13 20:58:12  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      TUN_RECV           CONNECTING
    2024-02-13 20:59:55  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      TUN_RECV           CONNECTING
    2024-02-13 21:01:58  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      PROBE_TIMEOUT      INIT
    2024-02-13 21:06:58  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  INIT            TUN_RECV_TIMEOUT   SURVIVING
    2024-02-13 21:07:58  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       TUN_RECV           CONNECTING
    2024-02-13 21:08:12  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      TUN_RECV           CONNECTING
    2024-02-13 21:08:14  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      TUN_RECV           CONNECTING
    2024-02-13 21:08:16  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      TUN_RECV           CONNECTING
    2024-02-13 21:09:32  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      TUN_RECV           CONNECTING
    2024-02-13 21:11:35  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      PROBE_TIMEOUT      INIT
    2024-02-13 21:11:35  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  INIT            TUN_RECV_TIMEOUT   SURVIVING
    2024-02-13 21:11:52  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       TUN_RECV           CONNECTING
    2024-02-13 21:13:55  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      PROBE_TIMEOUT      INIT
    2024-02-13 21:18:56  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  INIT            TUN_RECV_TIMEOUT   SURVIVING
    2024-02-13 21:20:01  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       SUR_TUN_ERR        SURVIVING
    2024-02-13 21:21:07  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       SUR_TUN_ERR        SURVIVING
    2024-02-13 21:22:14  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       SUR_TUN_ERR        SURVIVING
    2024-02-13 21:22:45  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       TUN_RECV           CONNECTING
    2024-02-13 21:24:45  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  CONNECTING      PROBE_TIMEOUT      INIT
    2024-02-13 21:24:45  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  INIT            TUN_RECV_TIMEOUT   SURVIVING
    2024-02-13 21:25:53  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       SUR_TUN_ERR        SURVIVING
    2024-02-13 21:27:02  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       SUR_TUN_ERR        SURVIVING
    2024-02-13 21:28:12  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       SUR_TUN_ERR        SURVIVING
    2024-02-13 21:29:23  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       SUR_TUN_ERR        SURVIVING
    2024-02-13 21:30:36  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       SUR_TUN_ERR        SURVIVING
    2024-02-13 21:31:49  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       SUR_TUN_ERR        SURVIVING
    2024-02-13 21:33:03  x.x.x.x     317de5bb-646c-4f30-adef-744b204dfe92  SURVIVING       SUR_TUN_ERR        SURVIVING
    ub_303h#
    However, when it is finally established it may stay good for a long (30-90 minutes) period of time, or it may break before the client device can complete DHCP.  When the connection is up, pings reliably pass through the tunnel at a stable latency (~100ms) and no drops over 100 pings.
    Some of the microbranch APs are up and stay up.  Some do not.
    In my lab (no internet) I am able to set up a microbranch-to-VPNC setup with no issues and it is completely stable.  Aruba TAC has been no help in the many requests I've made to have them look at it.
    Any suggestions?  It appears to be an IKE-related issue, but could it be a routing issue or packet drop at the VPNC side?



    ------------------------------
    ryh
    ------------------------------



  • 2.  RE: VPNC Microbranch connection issues

    Posted Feb 13, 2024 04:46 PM

    Also to note, this is a Central-managed 303H on version 10.5.0.1 (others have been tried as well), connecting to a VPNC on 10.4.1.0

    -ryh



    ------------------------------
    ryh
    ------------------------------



  • 3.  RE: VPNC Microbranch connection issues

    Posted Feb 13, 2024 04:57 PM

    has there been any packet captures on on the AP or VPNC or the firewall side? that would quickly narrow down the issue.



    ------------------------------
    If my post was useful accept solution and/or give kudos.
    Any opinions expressed here are solely my own and not necessarily that of HPE or Aruba.
    ------------------------------

    Original Message


  • 4.  RE: VPNC Microbranch connection issues

    Posted Feb 14, 2024 10:39 AM
    Edited by ryh Feb 14, 2024 10:41 AM

    At the AP side, all appears to be fine (it works to a local VPNC, similarly configured).  The customer has shown me their firewall's port-forwarding rules for external-to-internal IP translation for this controller (Ext-IP:udp4500 <-> Int-IP:udp4500) and it appears to be all correct and is what is received by the AP's "show ata endpoint" configuration.  They see the UDP4500 traffic in the firewall from these microbranch devices, and I see what I believe is the tunnel trying to negotiate.   I have the wired client (IP phone's) pcaps by inserting a wire pcap monitoring device and observing the traffic there, but this just shows problems when the tunnels go down. 

    In the "show datapath session" I see the traffic hitting the VPNC, and when established it can remain good for minutes or weeks.  I've asked if they do any SSL Inspection on their firewall, which they say they do not.  We've tried rebooting the VPNC, upgrading it, nothing seems to work.  I suppose we could try with just a public IP exposed on the box with no firewall in between and see if that helps?

    What would be the appropriate debug/investigative steps to look at why the IPsec tunnel is not forming?  Is there a "controlpath" pcap I could obtain on the VPNC for the AP's ike/isakmp/IPsec negotiation?

    I've had Aruba troubleshoot other VPNC issues that I would think would depend on this being solid, and they have always come up empty-handed and say my config "looks" good.  Perhaps if I could find a smoking gun in a process failure or debug log, then that would help them.



    ------------------------------
    ryh
    ------------------------------

    Original Message


  • 5.  RE: VPNC Microbranch connection issues

    Posted Feb 15, 2024 01:14 PM

    I will try taking a "packet-capture controlpath udp 4500" to see what is happening.  Initial investigation shows it just keeps trying until it works.



    ------------------------------
    ryh
    ------------------------------

    Original Message


  • 6.  RE: VPNC Microbranch connection issues
    Best Answer

    Posted Mar 11, 2024 05:54 PM
    Edited by ryh Mar 11, 2024 07:37 PM

    This issue is resolved with AOS 10.5.1.0 code.  The months of TAC cases dealing with IPSec, Microbranch stability, and Aruba VIA VPN issues amount to bugs in the code.

    If you are experiencing any similar issues with IPsec and VPNCs, make sure you upgrade all the way up to (or past) 10.5.1.0 (for both AP and VPNC code).  Even 10.5.0.1 exhibited these problems.

    Good luck out there.

    -ryh



    ------------------------------
    ryh
    ------------------------------



  • 7.  RE: VPNC Microbranch connection issues

    Posted Mar 15, 2024 02:17 PM
    Edited by Bdub324 Mar 15, 2024 02:42 PM

    Howdy! Sounds like a similar scenario I've been battling, I encountered a whole slew of issues. From memory leaks on 10.5.X.X to OAP routes not getting instated into the Microbranch routing table. 

    I found a document at one point outlining the compatibility matrix of Microbranch OS10 deployed APs versus IAP-VPN deployed APs and their respective compatibility in regards to what version the VPNC is on. For the life of me, I can't seem to find it again, would anyone have any clue where this document is or if I'm crazy and simply imagining it?

    Environment appears stable now with Microbranch and VPNC on 10.5.1.0. 


    Original Message


Related Content