SD-WAN

Hi,

I am using hp 2920-24G Switch with the following version:

Aug 29 2016 12:09:14
WB.16.02.0012
485
Boot Image: Primary

Boot ROM Version: WB.16.03
Active Boot ROM: Primary

I using a floodlight controller and an aggregate OpenFlow instance. I was successfully able to create a meter with id =1.

HP-2920-24G(of-inst-aggregate)# sh openflow inst agg meter

OpenFlow Instance Meters

Meter ID : 1
Flow Count : 0
Input Packet Count : 0
Input Byte Count : 0
Duration : 152

Packet/Byte
Band Type Rate Count
--------- ---------------- --------------------
Drop 30000 kbps 0

I want to create a flow entry that would direct all packets coming from in-port 2 into meter 1 then out to port 3. I use the following floodlight REST call:

curl -X POST -d '{"switch":"00:02:48:0f:cf:0c:b1:c0", "name":"flow-mod-1","table": "100" ,"cookie":"0", "priority":"1000", "in_port":"2","active":"true","instruction_goto_meter":"1","actions":"output=3"}' http://127.0.0.1:8080/wm/staticflowpusher/json

A flow entry is created however the meter instruction is ignored.

Flow 2
Match
Incoming Port : 2 Ethernet Type : Any
Source MAC : Any Destination MAC : Any
Source MAC Mask : 000000-000000
Destination MAC Mask : 000000-000000
VLAN ID : Any VLAN Priority : Any
Source IP Address : Any
Destination IP Address : Any
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Source Port Range : NA
Destination Port Range : NA
TCP Flags : NA
TCP Mask : NA
Attributes
Priority : 1000 Duration : 5 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 0
Flow Table ID : 100 Controller ID : 1
Cookie : 0xa000004039d1ad
Hardware Index: 17
Instructions
Apply Actions
Output : 3

If I send the same command but without the output action:

curl -X POST -d '{"switch":"00:02:48:0f:cf:0c:b1:c0", "name":"flow-mod-2","table": "100" ,"cookie":"0", "priority":"1000", "in_port":"2","active":"true","instruction_goto_meter":"1"}' http://127.0.0.1:8080/wm/staticflowpusher/json

The meter instruction is set correclty:

Flow 2
Match
Incoming Port : 2 Ethernet Type : Any
Source MAC : Any Destination MAC : Any
Source MAC Mask : 000000-000000
Destination MAC Mask : 000000-000000
VLAN ID : Any VLAN Priority : Any
Source IP Address : Any
Destination IP Address : Any
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Source Port Range : NA
Destination Port Range : NA
TCP Flags : NA
TCP Mask : NA
Attributes
Priority : 1000 Duration : 48 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 4
Flow Table ID : 100 Controller ID : 1
Cookie : 0xa000004039d1ad
Hardware Index: 0
Instructions
Meter ID : 1

OpenFlow Instance Meters

Meter ID : 1
Flow Count : 1
Input Packet Count : 0
Input Byte Count : 0
Duration : 430

Packet/Byte
Band Type Rate Count
--------- ---------------- --------------------
Drop 30000 kbps 0

Thus, I am struggling to add a flow entry that would set the meter and then an output port action. If you think it is a problem related to floodlight and not the switch let me know and I can look into the issue in the floodlight forums.

Thank you very much,

Enas

Hello Enas,

The HPE Aruba 2920 does support an OpenFlow rule with both METER and APPLY {OUTPUT} instructions.

Would it possible for you to share the OpenFlow packet (as a .pcap) of the flow-mod sent from the FloodLight controller to the switch. That can help us conclude if the flow-mod had the meter instruction in it or not.

You could use the latest version of WireShark which does support OpenFlow dissection and look for the instructions in the flow-mod.

Thanks!

Abhay

Thanks Abhay for your reply, 

It turns out it was a bug in the floodlight static flow pusher API. Using a floodlight module instead I was able to add a flow entry that sets the meter ID and then an output port action. 

However, an unexpected behavior was observed when testing the meter. The meter I installed is as the following :

Band: Drop and Rate: 30000kbps.

When I run an Iperf TCP test between the two hosts without the meter I get around 980mbps, when I use the meter I get a max of 10mbps, what I was expecting is to get a throughput of around 30mbps.

Any thoughts?

Hello Enas,

There are 2 types of meters that can be created on the switch. One that meters based on packets per second and the other based on the number of kilo bits per second.

This is determined based on the 'flags' field sent in the OFPT_METER_MOD add message.

/* Meter configuration flags */
enum ofp_meter_flags {
OFPMF_KBPS = 1 << 0, /* Rate value in kb/s (kilo-bit per second). */
OFPMF_PKTPS = 1 << 1, /* Rate value in packet/sec. */
OFPMF_BURST = 1 << 2, /* Do burst size. */
OFPMF_STATS = 1 << 3, /* Collect statistics. */
};

Can you confirm the meter is of type kbps and not pps? If it of type pps, it would explain why you don't see the egress rate to be 30mbps.

Thanks!

Abhay

Thanks for your reply,

Yes I have set the flag to 1 which creates a meter with type kbps.

HP-2920-24G(of-inst-aggregate)# sh openflow inst agg meter

OpenFlow Instance Meters

Meter ID : 1
Flow Count : 1
Input Packet Count : 8945
Input Byte Count : 13474167
Duration : 131

Packet/Byte
Band Type Rate Count
--------- ---------------- --------------------
Drop 30000 kbps 2114574

The posts in this thread are rising a similar issue, however I am not sure of the two cases are related:

/t5/SDN-Development/HP3800-Metering-drop-policy/m-p/6849870/highlight/true#M695

Hello Enas,

Are there any other flows in the OpenFlow instance that "some" of the packets in the iperf test stream could match on? Can you confirm all the packets of the test stream are hitting the rule with the meter attached?

Is it possible for you to dump all the flows on the switch when you run this test?

I tried a small test by creating a similar meter and attaching to a flow on Table 100 that matches on TCP packets and executes the a simple output to a physical port action. There were no other flows in the OpenFlow instance other than the miss flow.I pumped an ingress TCP stream of 1Gbps and verified the egress stream is rate limited to 30mbps.

Please note the meter rates are in bits per second and not bytes per second. I hope your calculations do take this into account.

Thanks!

Abhay

Hi Abhay,

May I know how are you generating the TCP stream ? Are you using iPerf ?

The thing is that when I tried the same test but with using UDP instead of TCP in iPerf (and created a flow entry that would match UDP traffic instead of TCP into the 30mbps meter) I get around 29mbps throughput, which is correct.

I have a feeling that TCP backoff mechanism when observing drops is affecting the throughput in the test. 

Hello Enas,

No, I wasn't using iPerf. I was using a traffic generator to generate unintelligent TCP traffic streams.

Does iPerf reduce the traffic load when it encounters TCP backoffs? 

Thanks!

Abhay

Hi Enas

With iperf you will have a fully state full tcp stream. This would imply that windowing would work like in the real world. Since you have a drop the tcp window would be aggressively reduced and then slowly scale back up till a drop is detected, and this would keep on happening. Since this is a tcp iperf it will by default attempt to use max mtu frames if it the default mtu of 1500 bytes(roughly 12000 bits), it should not take too much effort from iperf to step over the 30Mbps.

When using drops on tcp you should never see the rate limit throughput being reached, it will all ways be less and exponentially so. If you have your bucket rate closer to 1Gbps you should see less off a discrepancy but there will still be one.

Now with regards to iperf, not sure if you are using iperf2 or iperf3. But with a bit of command output we should be able to see more. I am hoping the interval resolution is high enough else we will have to look at a packet capture.

Add the following option to the iperf commands “-i 0.1” on both the client and server. You should be able to use the default 10 second interval. If the stream ramps up fast enough.

On iperf2 this will drop the interval to 0.5 seconds, on iperf3 this will drop the interval to 0.1 seconds for reporting.

If you can supply me the output from both client and server as shown below it would be great. It might be an option to go to iperf3 to increase the resolution if you are not on it.

Example with iperf2. I do not have a switch on hand so just using the loopback for sample output

Server:

gpr@ubuntu:~$ iperf -s -i 0.1

WARNING: interval too small, increasing from 0.10 to 0.5 seconds.

------------------------------------------------------------

Server listening on TCP port 5001

TCP window size: 85.3 KByte (default)

------------------------------------------------------------

[ 4] local 127.0.0.1 port 5001 connected with 127.0.0.1 port 50176

[ ID] Interval       Transfer     Bandwidth

[ 4] 0.0- 0.5 sec 2.74 GBytes 47.0 Gbits/sec

[ 4] 0.5- 1.0 sec 2.43 GBytes 41.7 Gbits/sec

[ 4] 1.0- 1.5 sec 2.65 GBytes 45.5 Gbits/sec

… Snipped …

[ 4] 9.5-10.0 sec 2.45 GBytes 42.2 Gbits/sec

[ 4] 0.0-10.0 sec 51.1 GBytes 43.9 Gbits/sec

Client:

gpr@ubuntu:~$ iperf -c 127.0.0.1 -i 0.1

WARNING: interval too small, increasing from 0.10 to 0.5 seconds.

------------------------------------------------------------

Client connecting to 127.0.0.1, TCP port 5001

TCP window size: 2.50 MByte (default)

------------------------------------------------------------

[ 3] local 127.0.0.1 port 50176 connected with 127.0.0.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[ 3] 0.0- 0.5 sec 2.74 GBytes 47.1 Gbits/sec

[ 3] 0.5- 1.0 sec 2.42 GBytes 41.6 Gbits/sec

[ 3] 1.0- 1.5 sec 2.65 GBytes 45.5 Gbits/sec

… Snipped …

[ 3] 9.5-10.0 sec 2.45 GBytes 42.2 Gbits/sec

[ 3] 0.0-10.0 sec 51.1 GBytes 43.9 Gbits/sec

Hope this helps.

Kind Regards

Gerhard Roets

HPE SDN CoE

Hi Gerhard,

I am using iPerf2, here is the requested output: (Still using a 30mbps meter)

iperf -s -i 0.1

WARNING: interval too small, increasing from 0.10 to 0.5 seconds.

------------------------------------------------------------

Server listening on TCP port 5001

TCP window size: 85.3 KByte (default)

------------------------------------------------------------

[  4] local 10.0.0.4 port 5001 connected with 10.0.0.3 port 49565

[ ID] Interval       Transfer     Bandwidth

[  4]  0.0- 0.5 sec   908 KBytes  14.9 Mbits/sec

[  4]  0.5- 1.0 sec   460 KBytes  7.53 Mbits/sec

[  4]  1.0- 1.5 sec   455 KBytes  7.46 Mbits/sec

[  4]  1.5- 2.0 sec   629 KBytes  10.3 Mbits/sec

[  4]  2.0- 2.5 sec   915 KBytes  15.0 Mbits/sec

[  4]  2.5- 3.0 sec   457 KBytes  7.48 Mbits/sec

[  4]  3.0- 3.5 sec   452 KBytes  7.41 Mbits/sec

[  4]  3.5- 4.0 sec   457 KBytes  7.48 Mbits/sec

[  4]  4.0- 4.5 sec   913 KBytes  15.0 Mbits/sec

[  4]  4.5- 5.0 sec   457 KBytes  7.48 Mbits/sec

[  4]  5.0- 5.5 sec   457 KBytes  7.48 Mbits/sec

[  4]  5.5- 6.0 sec   457 KBytes  7.48 Mbits/sec

[  4]  6.0- 6.5 sec   912 KBytes  14.9 Mbits/sec

[  4]  6.5- 7.0 sec   457 KBytes  7.48 Mbits/sec

[  4]  7.0- 7.5 sec   457 KBytes  7.48 Mbits/sec

[  4]  7.5- 8.0 sec   455 KBytes  7.46 Mbits/sec

[  4]  8.0- 8.5 sec   913 KBytes  15.0 Mbits/sec

[  4]  8.5- 9.0 sec   457 KBytes  7.48 Mbits/sec

[  4]  9.0- 9.5 sec   455 KBytes  7.46 Mbits/sec

[  4]  9.5-10.0 sec   457 KBytes  7.48 Mbits/sec

[  4]  0.0-10.1 sec  11.5 MBytes  9.58 Mbits/sec

********************************************************************

iperf -c 10.0.0.4 -i 0.1

WARNING: interval too small, increasing from 0.10 to 0.5 seconds.

------------------------------------------------------------

Client connecting to 10.0.0.4, TCP port 5001

TCP window size:  129 KByte (default)

------------------------------------------------------------

[  4] local 10.0.0.3 port 49565 connected with 10.0.0.4 port 5001

[ ID] Interval       Transfer     Bandwidth

[  4]  0.0- 0.5 sec  1.25 MBytes  21.0 Mbits/sec

[  4]  0.5- 1.0 sec   128 KBytes  2.10 Mbits/sec

[  4]  1.0- 1.5 sec   512 KBytes  8.39 Mbits/sec

[  4]  1.5- 2.0 sec   640 KBytes  10.5 Mbits/sec

[  4]  2.0- 2.5 sec   896 KBytes  14.7 Mbits/sec

[  4]  2.5- 3.0 sec   384 KBytes  6.29 Mbits/sec

[  4]  3.0- 3.5 sec   512 KBytes  8.39 Mbits/sec

[  4]  3.5- 4.0 sec   384 KBytes  6.29 Mbits/sec

[  4]  4.0- 4.5 sec  1.00 MBytes  16.8 Mbits/sec

[  4]  4.5- 5.0 sec   384 KBytes  6.29 Mbits/sec

[  4]  5.0- 5.5 sec   512 KBytes  8.39 Mbits/sec

[  4]  5.5- 6.0 sec   384 KBytes  6.29 Mbits/sec

[  4]  6.0- 6.5 sec   896 KBytes  14.7 Mbits/sec

[  4]  6.5- 7.0 sec   512 KBytes  8.39 Mbits/sec

[  4]  7.0- 7.5 sec   512 KBytes  8.39 Mbits/sec

[  4]  7.5- 8.0 sec   384 KBytes  6.29 Mbits/sec

[  4]  8.0- 8.5 sec   896 KBytes  14.7 Mbits/sec

[  4]  8.5- 9.0 sec   512 KBytes  8.39 Mbits/sec

[  4]  9.0- 9.5 sec   384 KBytes  6.29 Mbits/sec

[  4]  9.5-10.0 sec   512 KBytes  8.39 Mbits/sec

[  4]  0.0-10.1 sec  11.5 MBytes  9.58 Mbits/sec

*************************************************************

I have noticed that the default window size is small, I reran the test with a larger window size:

local 10.0.0.4 port 5001 connected with 10.0.0.3 port 49700

[  4]  0.0- 0.5 sec   908 KBytes  14.9 Mbits/sec

[  4]  0.5- 1.0 sec   457 KBytes  7.48 Mbits/sec

[  4]  1.0- 1.5 sec   455 KBytes  7.46 Mbits/sec

[  4]  1.5- 2.0 sec   455 KBytes  7.46 Mbits/sec

[  4]  2.0- 2.5 sec   912 KBytes  14.9 Mbits/sec

[  4]  2.5- 3.0 sec   455 KBytes  7.46 Mbits/sec

[  4]  3.0- 3.5 sec   457 KBytes  7.48 Mbits/sec

[  4]  3.5- 4.0 sec   615 KBytes  10.1 Mbits/sec

[  4]  4.0- 4.5 sec   913 KBytes  15.0 Mbits/sec

[  4]  4.5- 5.0 sec   457 KBytes  7.48 Mbits/sec

[  4]  5.0- 5.5 sec   779 KBytes  12.8 Mbits/sec

[  4]  5.5- 6.0 sec   458 KBytes  7.51 Mbits/sec

[  4]  6.0- 6.5 sec   913 KBytes  15.0 Mbits/sec

[  4]  6.5- 7.0 sec   455 KBytes  7.46 Mbits/sec

[  4]  7.0- 7.5 sec   457 KBytes  7.48 Mbits/sec

[  4]  7.5- 8.0 sec   455 KBytes  7.46 Mbits/sec

[  4]  8.0- 8.5 sec   912 KBytes  14.9 Mbits/sec

[  4]  8.5- 9.0 sec   455 KBytes  7.46 Mbits/sec

[  4]  9.0- 9.5 sec   457 KBytes  7.48 Mbits/sec

[  4]  9.5-10.0 sec   455 KBytes  7.46 Mbits/sec

[  4] 10.0-10.5 sec   911 KBytes  14.9 Mbits/sec

[  4] 10.5-11.0 sec   455 KBytes  7.46 Mbits/sec

[  4] 11.0-11.5 sec   457 KBytes  7.48 Mbits/sec

[  4] 11.5-12.0 sec   457 KBytes  7.48 Mbits/sec

[  4]  0.0-12.1 sec  14.1 MBytes  9.82 Mbits/sec

***********************************************************************

iperf -c 10.0.0.4 -i 0.1 -w 2.5m

WARNING: interval too small, increasing from 0.10 to 0.5 seconds.

------------------------------------------------------------

Client connecting to 10.0.0.4, TCP port 5001

TCP window size: 2.50 MByte (WARNING: requested 2.50 MByte)

------------------------------------------------------------

[  4] local 10.0.0.3 port 49700 connected with 10.0.0.4 port 5001

[ ID] Interval       Transfer     Bandwidth

[  4]  0.0- 0.5 sec  3.38 MBytes  56.6 Mbits/sec

[  4]  0.5- 1.0 sec   384 KBytes  6.29 Mbits/sec

[  4]  1.0- 1.5 sec   512 KBytes  8.39 Mbits/sec

[  4]  1.5- 2.0 sec   384 KBytes  6.29 Mbits/sec

[  4]  2.0- 2.5 sec   896 KBytes  14.7 Mbits/sec

[  4]  2.5- 3.0 sec   512 KBytes  8.39 Mbits/sec

[  4]  3.0- 3.5 sec   512 KBytes  8.39 Mbits/sec

[  4]  3.5- 4.0 sec   512 KBytes  8.39 Mbits/sec

[  4]  4.0- 4.5 sec   896 KBytes  14.7 Mbits/sec

[  4]  4.5- 5.0 sec   512 KBytes  8.39 Mbits/sec

[  4]  5.0- 5.5 sec   768 KBytes  12.6 Mbits/sec

[  4]  5.5- 6.0 sec   512 KBytes  8.39 Mbits/sec

[  4]  6.0- 6.5 sec   896 KBytes  14.7 Mbits/sec

[  4]  6.5- 7.0 sec   384 KBytes  6.29 Mbits/sec

[  4]  7.0- 7.5 sec   512 KBytes  8.39 Mbits/sec

[  4]  7.5- 8.0 sec   512 KBytes  8.39 Mbits/sec

[  4]  8.0- 8.5 sec   896 KBytes  14.7 Mbits/sec

[  4]  8.5- 9.0 sec   384 KBytes  6.29 Mbits/sec

[  4]  9.0- 9.5 sec   512 KBytes  8.39 Mbits/sec

[  4]  9.5-10.0 sec   384 KBytes  6.29 Mbits/sec

[  4]  0.0-10.0 sec  14.1 MBytes  11.8 Mbits/sec

Not much of a difference.

Hi Enas

Based on the information you provided I can see that you have a classical saw tooth performance pattern.

From the Server you see a typical throughput of

7.5Mbps and Once in a while 15 Mbps. You can also see there is a kind of “rhythm” to it this is the pattern I am talking about.

It is similar from the client side point of view.

This is typical of TCP sessions where there is drops, this is understandable because of how tcp windowing works. The switch simply drops packets when the bucket is full. This causes tcp to react in the way that you see.

One thing you can do is decrease the window size, not increase. So if you make the window size as small as possible you might see different results. In the end though this is an artificial fix as you cannot control all clients individually like this.

The solution for this is to start randomly dropping packets from multiple streams once you detect getting close to the threshold and thus forcing some back off, but this assumes there is multiple streams.

The reason why UDP shows it more correct is that UDP does not have Windowing and TCP Slow start. There are various articles around this that you can find on the web with your web search engine of choice just search for “TCP Slow start”, “TCP Congestion Control”, and “TCP Synchronization”.

Hope this helps

Gerhard Roets

HPE SDN CoE