Network Management

Hello, I have Mellanox ConnectX-2930 Servers with Redhat 2930F operating system connected to Hp Aruba 254F (JL254A) Swithcs over 10Gbe. In the tests I did with iperf3 between the 2 servers, there is a connection speed as follows. What kind of adjustment needs to be made in the 10gbe ports on the Hp Aruba 2930F. As if something is blocking the packets? I need your support on this issue.

[root@mellanox-10gbe ~]# iperf3 -c 10.155.155.245
Connecting to host 10.251.11.245, port 5201
[  5] local 10.42.36.19 port 58248 connected to 10.155.155.245 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  84.8 KBytes   694 Kbits/sec   13   1.41 KBytes
[  5]   1.00-2.00   sec   117 KBytes   962 Kbits/sec   17   2.83 KBytes
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec   12   1.41 KBytes
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    4   1.41 KBytes
[  5]   4.00-5.00   sec  80.6 KBytes   661 Kbits/sec   20   2.83 KBytes
[  5]   5.00-6.00   sec  79.2 KBytes   648 Kbits/sec   16   2.83 KBytes
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec   16   1.41 KBytes
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
[  5]   8.00-9.00   sec  79.2 KBytes   649 Kbits/sec   23   1.41 KBytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    5   1.41 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   441 KBytes   361 Kbits/sec  127             sender
[  5]   0.00-10.04  sec   386 KBytes   315 Kbits/sec                  receiver

I suspected that this was due to the settings on the Hp Aruba 2930F switch 10gbe port, I figured it out by disabling the switch with the Dac cable between the two servers with Mellanox ConnectX-4 Cards. When the switch is disabled, I get the following result.

I get the following result when I connect two servers directly from 10gbe port with dac cable.

[  5]   0.00-1.00   sec  1.15 GBytes  9.84 Gbits/sec    0   2.43 MBytes

[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    0   2.57 MBytes

[  5]   2.00-3.00   sec  1.15 GBytes  9.91 Gbits/sec    0   2.57 MBytes

[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec    0   2.75 MBytes

[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    0   2.75 MBytes

[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec    0   2.75 MBytes

[  5]   6.00-7.00   sec  1.15 GBytes  9.91 Gbits/sec    0   2.75 MBytes

[  5]   7.00-8.00   sec  1.15 GBytes  9.89 Gbits/sec    0   2.75 MBytes

[  5]   8.00-9.00   sec  1.15 GBytes  9.91 Gbits/sec    0   2.75 MBytes

[  5]   9.00-10.00  sec  1.02 GBytes  8.72 Gbits/sec    0   2.75 MBytes

What setting should be done on Hp Aruba 2930F switch or 10gbe port? 

Thankyou.

Hello, as it can be seen in the network topology I attached in the attachment, I have a total of 5 servers on 2 switches, which are connected to each other with 10Gbe connection without any problems. However, as it appears in the first post in the speed test with iper3 over the servers, it is very slow and even does not copy. How should I make a setting on the ports connected to 10GbE on these switches so that the mutual speed problem is solved. If I remove these servers from the hp aruba switch 2930f and connect them directly to each other with dac cablous, the speed problem disappears, but I have to use them with HP switches.

It's not clear if you're performing IPERF testing between a pair of servers connected to the same Aruba 2930F switch or between a pair of servers connected to two different Aruba 2930F switches (and thus necessarily traversing the 10G fiber optic interconnection existing between the two Switches). Also you should provide some more configuration and diagnostic status (not only the client/server IPERF test outputs) regarding the switch involved ports (edge ports to Servers and interlink ports between neighboring switches) and, if routing is involved - as it is given the VLAN memberships of your servers - explain what is the device performing the IP routing between VLANs? as example, you simply didn't answered to that question...so it's hard to tell what's going on just reading what you're telling us...it could be a configuration issue or anything else...implying that involved OSes (RedHat/Debian are properly configured on each server...was the MTU set to Jumbo on both the OS Ethernet ports and Switch ports? as example...)

Hello, I made a mistake in the connections of the switches in the first drawing.  The correct connection will be as in this picture.

Iperf test was done both between servers connected to the same Aruba switch and between servers connected to 2 different Aruba switches. 

Jumbo Packet has been activated on the server side ethernet cards and switches to which the servers are connected on a vlan basis. MTU is set to 8000. 

However, as shown in the picture, Jumbo Packet is not activated on the uplink ports on the 10GbE Fiber connections connected to the INTERNET Switch. Does this affect the iperf test? What command should we use to activate Jumbo Packet on Hp Switches and change MTU on port basis. Vlan-based Jumbo Packet is activated, but with which command we change the Port-based MTU. I wonder if I have entered the correct command to confirm this, can you share how to make such a configuration in you? 

Hi!

So you wrote:

"Iperf test was done both between servers connected to the same Aruba switch and between servers connected to 2 different Aruba switches."

and what?

If Servers placements and their IP addressing is what you reported (VLAN 611 for Server 1, Server 2, Server 3 and Server 4 plus VLAN 422 for Server 4 and Server 5) then...

(a) Scenario: Server 1 and Server 2 connected to the same Switch AND to the same VLAN segment (no IP routing involved by the IPERF test between them) -> results?

(b) Scenario: Server 1 (or Server 2) and Server 3 (or Server 4) connected to different Switches BUT belonging to same VLAN segment (no IP routing involved as above BUT upstream/downstream traffic will necessarily traverse the network) -> results?

(c) Scenario: Server 1 (or Server 2) and Server 5 connected to different Switches AND belonging to different VLAN segments (thus IP routing is involved AND upstream/downstream traffic will necessarily traverse the network) -> results?

Given the result matrix one can understand WHEN the IPER issue happens and start its correlations.

Since you're testing IPERF (performance) between locally connected servers (locally = LAN) you shouldn't care about Internet connectivity, focus only on the switches through which the data flows due to your IPERF tests.

P.S. MTU should be set to 9000...