Comware

 View Only
  • 1.  High CPU Usage

    Posted Nov 18, 2013 10:52 AM

    Hey Everyone,

     

    I recently replaced a pair of A5800-24Gs (JC100A) with a set of A5800AF-48Gs (JG225A). These switches are serving core switch functionality in my DR site. I had to replace the switches for increased growth in our DR site. The AFs are bonded together using IRF and I have MAD bfd enabled.

     

    For some reason, I seem to have high cpu usage and I believe it's causing some issues on my network. It looks like my latency is higher than it was with the JC100As. I have another site setup with a pair of AFs and it's only reporting 5-10% useage on both members as opposed to the new ones that have a 25-30% cpu use. 

     

    Any ideas on how I can start troubleshooting?



  • 2.  RE: High CPU Usage

    Posted Nov 20, 2013 05:58 AM

    Hi,

     

    This can be checked using the hidden commands, that will show you which process is taking up the cpu resources:

     

    <switch>sys
    System View: return to User View with Ctrl+Z.
    [switch]_hidecmd
    Now you enter a hidden command view for developer's testing, some commands may
    affect operation by wrong use, please carefully use it with our engineer's
    direction.
    [switch-hidecmd]dis cpu task
    ===== Current CPU usage info =====
    CPU Usage Stat. Cycle: 26 (Second)
    CPU Usage            : 12%
    CPU Usage Stat. Time : 2000-07-16  10:14:00
    CPU Usage Stat. Tick : 0x1a7f2(CPU Tick High) 0xca7dd6c2(CPU Tick Low)
    Actual Stat. Cycle   : 0x0(CPU Tick High) 0x6a14fa50(CPU Tick Low)

    TaskName        CPU        Runtime(CPU Tick High/CPU Tick Low)
    VIDL            88%               0/5e52f721
    TICK             0%               0/  837d32
    STMR             0%               0/   f6cbc
    DSTK             0%               0/   1d088
    SYSM             0%               0/   2c6a4
    L2X0             2%               0/ 28f63d8
    bC.0             2%               0/ 2ab7f44
    bLK0             0%               0/  f13733
    DQFD             0%               0/  116f6c
    DQIT             1%               0/ 158ce25
    LPDT             0%               0/     2f6
    STAT             0%               0/   51bf9
    FMCK             0%               0/  114eeb
    T_DM             0%               0/   4be51
    mIPC             0%               0/   4c67b
    ...

     

    Best regards,Peter



  • 3.  RE: High CPU Usage

    Posted Nov 20, 2013 08:40 AM

    Hey Peter! Thanks so much for the response. I really appreciated it. While messing around a bit with the CLI, I also found out you can get the same results without going into the developer mode by typing the following:

     

    [sys]disp cpu-usage 1 verbose

     

    ===== CPU usage info (no: 0 idx: 52) =====
    CPU Usage Stat. Cycle: 60 (Second)
    CPU Usage : 27%
    CPU Usage Stat. Time : 2013-11-20 09:24:31
    CPU Usage Stat. Tick : 0x12aa(CPU Tick High) 0xa49357f3(CPU Tick Low)
    Actual Stat. Cycle : 0x0(CPU Tick High) 0xee6cddfe(CPU Tick Low)

    TaskName CPU Runtime(CPU Tick High/CPU Tick Low)
    VIDL 73% 0/aef3337e
    TICK 0% 0/ 8a8562
    STMR 0% 0/ 18b4c3
    DSTK 0% 0/ 12d2f2
    SYSM 0% 0/ 6f926
    MSGR 0% 0/ 5f2d45
    bDPC 0% 0/ 2f9b4
    L2X1 4% 0/ b6caf5f
    bC.1 3% 0/ 8f1c339

     

    This doesn't seem to give me as much information as I would have expected. Here's a summary of my overall concern with these switches: I have 2 cores at different data centers. Both cores have a pair of 5800AF-48Gs setup in an IRF configuration. There is one major difference between the two cores, one core runs 1211P01 and the other core system is running 1808P11. The 1808 core is currently registering 25-30% cpu load and the pings are very volitile by comparison to the 1211 core. The 1808 core recently replaced a set of 5800-24Gs that were running 1211P01 as well. The ping metrics have substantially changed and I am concerned it relates to the CPU overhead or even something thats misconfigured on the switch. This graph here illustrates the differences I am seeing in performance on the switch by comparison to the statistics I was pulling from the 5800-24G running 1211P01. Smokeping (the application forming the graph) is clearly showing that there is a significant "drift" being shown. The green dot represents the "average" ms but the darker shaded area is the spread of the pings. As you can see the spread varies quite a bit more. 

     

    Does anyone have any input on why this might be happening? Is it standard for a 5800AF-48G to be running 30% CPU with an IRF configruation? This switch really isn't doing much more than the 5800-24G that it replaced. There is an extra IRF link added to the mix and I've configred BFD MAD. 



  • 4.  RE: High CPU Usage

    Posted Nov 20, 2013 11:20 AM

    Hi,

     

    in my experience it is quite normal to see 10-30% CPU load with IRF (many control plane protocols must be hot-synced between the members). The difference does seem very large between the 2 releases IMO.

     

    Can you post the full process and cpu usage list ?

    Generally speaking, ICMP has a very low prio on the CPU, so other processes will typically be served ahead of any ICMP reply.

     

    Best regards,Peter



  • 5.  RE: High CPU Usage

    Posted Nov 20, 2013 02:18 PM

    Sure no prob. 

     

    ===== CPU usage info (no: 0 idx: 38) =====
    CPU Usage Stat. Cycle: 60 (Second)
    CPU Usage : 26%
    CPU Usage Stat. Time : 2013-11-20 15:17:26
    CPU Usage Stat. Tick : 0x13f3(CPU Tick High) 0x508b4ff2(CPU Tick Low)
    Actual Stat. Cycle : 0x0(CPU Tick High) 0xee6b2834(CPU Tick Low)

    TaskName CPU Runtime(CPU Tick High/CPU Tick Low)
    VIDL 74% 0/b199ef4f
    TICK 0% 0/ 8b8114
    STMR 0% 0/ 19bbba
    DSTK 0% 0/ 12eb47
    SYSM 0% 0/ 6d89d
    MSGR 0% 0/ 6266cc
    bDPC 0% 0/ 1dc2c
    L2X1 4% 0/ bbd9893
    bC.1 3% 0/ 90cadf2
    bLK1 1% 0/ 31bf775
    L2X0 4% 0/ b9bf3ed
    bC.0 3% 0/ 9065e9a
    bLK0 1% 0/ 2dcc72c
    DQFD 0% 0/ 1e07440
    DQIT 1% 0/ 31d14d0
    LPDT 0% 0/ 421
    STAT 0% 0/ 10fcb0
    FMCK 0% 0/ 19fc3b
    T_DM 0% 0/ 51ef6
    mIPC 0% 0/ 4fe1a
    T_VA 0% 0/ 8adc
    DARP 0% 0/ 1b402e
    T_ST 0% 0/ 5911e
    T_SI 0% 0/ 6b962
    TPBR 0% 0/ 7d474
    BGRT 0% 0/ a8382
    MARP 0% 0/ 88a17
    VMAC 0% 0/ 8014
    SCAR 0% 0/ 3b52db
    RTMC 0% 0/ 16ee32
    bRX1 0% 0/ 2dfca9
    bRX2 0% 0/ 2cb12b
    STKT 0% 0/ 1724cb
    IUCT 0% 0/ 2cf7da
    tEXP 0% 0/ 9f5d
    bIP0 0% 0/ ff2e
    bIP1 0% 0/ 191bf
    NSET 0% 0/ 246053
    NSTM 0% 0/ 1e7c23e
    IPCB 0% 0/ 8f2a67
    IPCD 0% 0/ 1bfc
    RPCQ 0% 0/ 3f67cb
    RPCD 0% 0/ 294bb
    VP 0% 0/ 4ae
    STM 0% 0/ 844eb2
    STMH 0% 0/ 20f066
    PMOT 0% 0/ 1f09b2
    EXCP 0% 0/ db124
    DRVR 0% 0/ 30a57
    IPCM 0% 0/ cb07
    INFO 0% 0/ cde8
    HAMT 0% 0/ 2887
    HACT 0% 0/ 47865
    DEV 0% 0/ cf1ea
    SOCK 0% 0/ 4f7be
    ADJ4 0% 0/ 3e69d
    SFLW 0% 0/ 21329
    ACL 0% 0/ 23ac8
    mac 0% 0/ b1d6
    LAGG 0% 0/ 11608
    MSTP 0% 0/ 1865b
    GARP 0% 0/ e919b
    CLST 0% 0/ 2f935
    NDP 0% 0/ 316b6
    NTDP 0% 0/ e585
    HABP 0% 0/ d229
    CFD 0% 0/ 111e69
    RRPP 0% 0/ 426ca
    LLDP 0% 0/ 650ca2
    DCBX 0% 0/ 232eb
    DLDP 0% 0/ 34bda
    SMLK 0% 0/ 1ae2d
    ARP 0% 0/ 2ca9ff
    IP 0% 0/ 3c1a73
    FSLH 0% 0/ 985f
    FSLR 0% 0/ 6dd4e
    NTPT 0% 0/ 468f8
    VTYD 0% 0/ 95c13
    DHCC 0% 0/ 135e1
    ND 0% 0/ 11cc23
    TRAP 0% 0/ 709
    CWMP 0% 0/ 2484b
    ACM 0% 0/ 1af3c4
    LS 0% 0/ 65ebc
    RDSO 0% 0/ 4d3c4
    RDS 0% 0/ 7dc2
    SC 0% 0/ 181ff
    NAT 0% 0/ 8e0b
    BFD 0% 0/ 361119
    MFIB 0% 0/ 1573ca
    STND 0% 0/ 27de2
    ROUT 0% 0/ 572d0a
    TNLM 0% 0/ c30c
    IFNT 0% 0/ f863
    GMAC 0% 0/ 1817a
    DNS 0% 0/ 6ef7f
    vt0 0% 0/ 4f96c