Wired Intelligent Edge

 View Only
Back to discussions
Expand all | Collapse all

ArubaCX: High CPU utilization

This thread has been viewed 102 times

  • 1.  ArubaCX: High CPU utilization

    Posted Jan 17, 2021 11:19 AM
      |   view attached
    Dear Sir,

    Today we have upgraded 2 unit core switch firmware from FL.10.05.0030 to FL.10.05.0040. Right after the upgrade, i noticed the CPU of primary core spike frequently while the secondary cpu looks normal. You may find the cpu utilization graph  by HPE IMC in the attachment. 

    I have run the top cpu on the primary switch as shown below. 

    **********************************************************************

    ZA-CX6405-CS1# top cpu

    top - 23:13:28 up 10:53,  2 users,  load average: 5.43, 5.48, 5.46

    Tasks: 279 total,   2 running, 223 sleeping,   0 stopped,   0 zombie

    %Cpu(s): 18.8 us, 22.9 sy,  0.0 ni, 58.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

    KiB Mem : 16030564 total, 11237628 free,  1446556 used,  3346380 buff/cache

    KiB Swap:        0 total,        0 free,        0 used. 14077612 avail Mem

     

      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

    10836 root      20   0   75700    484    412 R 108.3  0.0 639:51.31 /lib/systemd/systemd-socket-proxyd 169.2+

      799 root      20   0   53484  29056   7036 S  25.0  0.2  71:05.95 /usr/sbin/ovsdb-server --remote=ptcp:664+

     6788 azlan     20   0    3344   2140   1644 R  25.0  0.0   0:00.06 /usr/bin/top -b -n 2 -c -o %CPU -w 110 -+

     1669 root      20   0  647188  28004  11624 S   8.3  0.2  17:45.16 /usr/bin/ndmd --pidfile -vSYSLOG:INFO

     1673 root      20   0  276144  50496   7384 S   8.3  0.3   2:40.39 python /usr/bin/hpe_sshd

        1 root      20   0    7480   6336   3044 S   0.0  0.0   0:04.46 /sbin/init

        2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 [kthreadd]

        4 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 [kworker/0:0H]

        6 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 [mm_percpu_wq]

        7 root      20   0       0      0      0 S   0.0  0.0   0:01.96 [ksoftirqd/0]

        8 root      20   0       0      0      0 I   0.0  0.0   0:50.51 [rcu_sched]

        9 root      20   0       0      0      0 I   0.0  0.0   0:00.00 [rcu_bh]

       10 root      rt   0       0      0      0 S   0.0  0.0   0:00.03 [migration/0]

       11 root      rt   0       0      0      0 S   0.0  0.0   0:00.08 [watchdog/0]

       12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 [cpuhp/0]

       13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 [cpuhp/1]

       14 root      rt   0       0      0      0 S   0.0  0.0   0:00.06 [watchdog/1]

       15 root      rt   0       0      0      0 S   0.0  0.0   0:00.02 [migration/1]

       16 root      20   0       0      0      0 S   0.0  0.0   0:02.19 [ksoftirqd/1]

       18 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 [kworker/1:0H]

    Primary core switch
    ZA-CX6405-CS1# sh system
    Hostname : ZA-CX6405-CS1
    System Description : FL.10.05.0040
    System Contact :
    System Location :

    Vendor : Aruba
    Product Name : R0X24A 6405 Chassis
    Chassis Serial Nbr : SG9AKM70BW
    Base MAC Address : f860f0-612000
    ArubaOS-CX Version : FL.10.05.0040

    Time Zone : Asia/Kuala_Lumpur

    Up Time : 11 hours, 51 minutes
    CPU Util (%) : 35
    Memory Usage (%) : 12

    Secondary Core Switch
    ZA-CX6405-CS2# sh system
    Hostname : ZA-CX6405-CS2
    System Description : FL.10.05.0040
    System Contact :
    System Location :

    Vendor : Aruba
    Product Name : R0X24A 6405 Chassis
    Chassis Serial Nbr : SG9AKM70L8
    Base MAC Address : f860f0-616400
    ArubaOS-CX Version : FL.10.05.0040

    Time Zone : Asia/Kuala_Lumpur

    Up Time : 11 hours, 54 minutes
    CPU Util (%) : 0
    Memory Usage (%) : 13

    *************************************************************************************
    May i know the detail of the process  that use 108% CPU as shown above. Or could it be some kind of a bug?

    Thank you



    ------------------------------
    Saiful Azlan Anuar
    ------------------------------


  • 2.  RE: ArubaCX: High CPU utilization

    Posted Jan 17, 2021 12:04 PM
    Edited by parnassus Jan 17, 2021 12:26 PM
    Hi! what's the output of show system resource-utilization command executed on both VSX nodes at various times?

    I'm observing similar results (either on the VSX Secondary and VSX Primary, after about 60 days from upgrade to 10.05.0021 for our two Aruba 8320...thus on a scenario not exactly comparable with your one)...but I never noticed anything alarming from the ArubaOS-CX standpoint on both VSX nodes.

    The CPU usage was 30/35% on average exactly when I checked with the top cpu command but then dropped to lower levels:

    vsx-2# top
    cpu memory
    vsx-2# top cpu
    top - 18:39:31 up 58 days, 7:43, 1 user, load average: 1.18, 1.53, 1.53
    Tasks: 241 total, 2 running, 170 sleeping, 0 stopped, 0 zombie
    %Cpu(s): 28.6 us, 25.4 sy, 0.0 ni, 44.4 id, 0.0 wa, 0.0 hi, 1.6 si, 0.0 st
    KiB Mem : 15964776 total, 9668040 free, 1564156 used, 4732580 buff/cache
    KiB Swap: 0 total, 0 free, 0 used. 11832548 avail Mem

    PID   USER  PR NI VIRT     RES    SHR    S  %CPU  %MEM   TIME+    COMMAND
    24836 root  20 0  82244    920    832    R  100.0 0.0    1:02.48  /lib/systemd/systemd-socket-proxyd 169.2+
    3591  root  20 0  1087732  318416 94132  S  62.5  2.0    31596:48 /usr/sbin/ops-switchd --no-chdir --pidfi+
    13833 admin 20 0  17736    2420   1976   R  37.5  0.0    0:00.14  /usr/bin/top -b -n 2 -c -o %CPU -w 110 -+

    and quite the same on VSX Primary (same process same %CPU value).

    The point is that...in both cases...the top cpu command, after just a few seconds from its first execution, once re-run changed completely (that's expected) and indeed the /lib/systemd/systemd-socket-proxyd process disappeared entirely...both VSX nodes' CPU usage concurrently dropped at 5-10%.

    ------------------------------
    Davide Poletto
    ------------------------------



  • 3.  RE: ArubaCX: High CPU utilization

    Posted Jan 17, 2021 11:08 PM
    Hi Davide,

     I have run a command below at different time each at around 5 minute lapse at both node. The result  is compiled  in the attachment. Also, a cpu utilization graph both node from HPE IMC today for your reference.

    1. show clock
    2. show system resource-utilization
    3. Top cpu

    Thank you.

    ------------------------------
    Saiful Azlan Anuar
    ------------------------------

    Attachment(s)

    txt
    cpu-uti-core2.txt   38 KB 1 version
    txt
    cpu-uti-core1.txt   38 KB 1 version
    Original Message


  • 4.  RE: ArubaCX: High CPU utilization

    Posted Jan 18, 2021 05:05 AM
    I see.

    Despite this trend (maximum CPU usage on VSX Primary looks high on the 1 Day time range, mid term it not):
    VSX Primary CPU 1 Day

    and the VSX Secondary trends of Max/Min/Mid terms CPU usage look quite the same (again over 1 Day time range):
    VSX Secondary CPU 1 Day

    If I look at latest 90 days time range the reports are quite reasonable and the medium term CPU usage values on both nodes slightly dropped after the upgrade we did (the steps you see on 22nd November were due to ArubaOS-CX upgrade to 10.05.0021):
    VSX Primary CPU 90 Days

    VSX Secondary CPU 90 Days

    I've no clue if what you're observing is something alarming or not...I mean...it looks like watching at top cpu and HPE IMC CPU graphs make things worst than perceived by looking at NAE graphs.

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 5.  RE: ArubaCX: High CPU utilization

    Posted Jan 18, 2021 09:53 AM
    Hi Davide,

    May i know how you built this nice cpu NAE graph? I dont remember if i have share the switch access to you. Just curious :-)

    Thanks

    ------------------------------
    Saiful Azlan Anuar
    ------------------------------

    Original Message


  • 6.  RE: ArubaCX: High CPU utilization

    Posted Jan 18, 2021 11:13 AM
    Hi Davide,

    I have upload the NAE cpu utilization on both node. As per my understanding  from the graph, the cpu on primary node increase  after the firmware upgrade activity (around 12 noon of 17-January-2021).

    Thank you.

    ------------------------------
    Saiful Azlan Anuar
    ------------------------------

    Original Message


  • 7.  RE: ArubaCX: High CPU utilization

    Posted Jan 19, 2021 06:03 AM
    I think, it is also a good idea to open a issue to TAC

    ------------------------------
    PowerArubaSW : Powershell Module to use Aruba Switch API for Vlan, VlanPorts, LACP, LLDP...

    PowerArubaCP: Powershell Module to use ClearPass API (create NAD, Guest...)

    PowerArubaCX: Powershell Module to use ArubaCX API (get interface/vlan/ports info)..

    ACEP / ACMX #107 / ACDX #1281
    ------------------------------

    Original Message


  • 8.  RE: ArubaCX: High CPU utilization

    Posted Jan 19, 2021 07:25 AM
    Next time try a simple paste of the snipped image...it's better IMHO.

    By the way the Medium Term CPU usage (%) grew of about 25% on one VSX node (from an average slightly below 10% to an average post-update of a little bit higher than 30%) and basically nothing on the other VSX node...I don't know if this behaviour could be seen as normal/expected or not.

    Better, at this point, to engage the Aruba TAC by opening a Support Ticket and reporting your detailed findings.

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 9.  RE: ArubaCX: High CPU utilization

    Posted Jan 19, 2021 10:57 AM
    Hi Davide,

    Thank you very much for your analysis. Actually, we have already open a TAC case due to the issue of the vsx-sync not functioning. We upgraded the firmware as suggested by TAC. And then i noticed the CPU start to become high compare to previous firmware. I will report this finding to TAC.  Appreciate your assistance.

    Thank

    ------------------------------
    Saiful Azlan Anuar
    ------------------------------

    Original Message


  • 10.  RE: ArubaCX: High CPU utilization

    Posted Jan 19, 2021 02:49 PM
    Interesting by the final feedback from TAC

    ------------------------------
    PowerArubaSW : Powershell Module to use Aruba Switch API for Vlan, VlanPorts, LACP, LLDP...

    PowerArubaCP: Powershell Module to use ClearPass API (create NAD, Guest...)

    PowerArubaCX: Powershell Module to use ArubaCX API (get interface/vlan/ports info)..

    ACEP / ACMX #107 / ACDX #1281
    ------------------------------

    Original Message


  • 11.  RE: ArubaCX: High CPU utilization

    Posted Jan 19, 2021 06:30 PM
    Hi! keep us updated.

    I noticed both 10.05.0040 and 10.06.0010 already addressed the VSX related Bug ID 92243 "The VSX status is seen as non-operational and
    configuration sync does not work." - a bug with consequences on VSX Synchronization - but, probably, despite the reference to VSX Synchronization, that bug is not the same hitting your VSX implementation.

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


Related Content