Wired Intelligent Edge

Hi, we are using a Spectralink DECT system that uses PTP (Precision Time Protocol) to sync its base stations. All base stations are connected to HP ProVision-based switches (2510, 2530) with a 5406zl2 core switch. The time-critical PTP packets are marked with DSCP EF (Expedited Forwarding), which is mapped to priority 7 in the switches. The same DSCP is used for VoIP traffic (Skype for Business).

According to Spectralink, the jitter between base stations is "too high", sometimes it goes up to 1200 ns, usually it is around 300 ns. Sometimes it is even above 400 ns between two stations on the same switch, and sometimes there are also huge differences for stations on the same switch (like, station on port 8 has 500 ns jitter to a remote station, but station on port 10 of the same switch has 800 ns to the same remote counterpart).

Another DECT vendor says that ProCurve switches are just perfect for syncing DECT stations with PTP, while Spectralink says that our jitter is way too high and no wonder we have issues (should be < 400 ns).

These are the relevant lines from our config files:

qos dscp-map cs5 priority 7 name "Lync-40"
qos dscp-map ef name "Lync-46"
qos type-of-service diff-services
vlan 30
name "DECT"
ip igmp
exit

Is there anything I can do from the configuration? Or are the Spectralink bases just too picky?

Hi, really interesting.

Does your DECT Network belong to a separate high priority dedicated VLAN (let's say you defined a IP-DECT dedicated VLAN) traversing all involved Switches? yep, I see, it does (VLAN 30).

That seems to be an essential requirement (a mandatory one) for similar IP DECT deployments (as example, the one offered by Siemens - now Unify - with its HiPath Cordless IP V1 system) and there is also a quite common recommandation about the maximum supported number of cascaded Switches between the BSIP Sync Master and BSIP Sync Salves (so it's important where the Sync Master BSIP is located/defined); also Layer 3 Routing is not supported between the DECT Server (represented by a BSIP Base Station operating in IWU mode or by a bare metal Server which operates as Server IWU) and BSIP Base Stations operating in BSIP only mode...but this, I think, you know yet.

How is your Synchronization Topology (where is the BSIP Sync Master with respect to the BSIP Sync Slaves)?

How is your Network Topology in terms of Trunk Uplinks between the Switches?

Do all your involved Switches manage high traffic situations?

As example the Unify HiPath Cordless IP V1 (which is basically an Ikon GmbH based IP-DECT system) reports a maximum of 700 ns for Jitter (Variation of Delay), stating that higher values will definitely lead to re-synchronizations (so IP-DECT calls go KO).

Regarding the LAN Synchronization (IEEE 1588 PTP) I've to admit that the Siemens solution uses a method that was developed using the IEEE 1588 as a base: as someone at Siemens told me four years ago, to overcome the issue represented by Low End non IEEE 1588 compliant Switches or by High End IEEE 1588 compliant Switches that show high Jitter values even with low traffic situations, they developed a method that let the "...Master BS to generate UDP messages with time stamps and send them to the Slave BS(s). The Slave BS(s) are then able to read the time stamps and so to deduct the flying time needed between Slave and Master. If this is successful the different time bases in the different DECT IP Base stations are kept in sync...during the development was recognized that Jitter variations are most critical, if the Jitter is constant the developed methodology can adapt this Jitter, but if the Jitter varies very much, this methodology is not able to adapt the Jitter to a fix value. The LAN synchronization will not work in this case."

Indeed LAN Synchronization, in contrast to the DECT Synchornization (Over the Air), requires a PSR (Project Specific Release) approved by Siemens/Unify.

Don't know much about Spectralink IP Dect solution and its requirements (apart of what is reported in their Spectralink DECT/IP-DECT Server 400/6500/2500/8000 Synchronization and Deployment Guide here) so I can't judge if their requirements are too strict or not (at some point they state that "The total forwarding jitter added by switches on any path through the deployment network should be less than one microsecond and preferably less than 100 nanoseconds." but I haven't any means to understand if this requirment is too strict or not even with HP ProCurve Switches)...but, hey, first:

• Do HP ProCurve Switches support IEEE 1588 Version 2 (PTPv2) or IEEE 802.1AS? <- I found NO reference for that (on the contrary the HPE FlexNetwork family seems to support it) in any QuickSpecs or Configuration Manuals.
• What are the HP ProVision 6th Generation ASICs specs (related to HPE 5400R zl2) regarding the delay time (in nano-seconds or fraction of micro-seconds) added to process/manage PTP traffic? <- really don't know but, personally, I expect at least that an High End Switch to perform significantly better than a Low End one!

Edit: consider, I'm reading it now on a IEEE 802.3AS six years old presentation, that one of the 802.3AS golas was to keep the time at any two LAN-attached stations accurate to within +/- 500ns (assuming 7 newtork hops) and that 802.3AS goals were not to:

• Improve the latency of media packets
• Improve the delivery jitter of media packets
• Improve latency or delivery jitter of 802.1AS packets
• Provide clock which all media must be synchronized to
• Solve time synchronization over 802.3 only (or .11 only)

Also, be careful when looking for IEEE 1588 PTP support on various HPE Switches QuickSpecs...I found now that HPE reports it incorrectly as RFC 1855 PTP (gosh!) so when doing a research specifically for 1588 I initially found nothing even if the Switch appears to support it (see here for HPE FlexNetwork 5900 Switch Series QuickSpecs, just as example).

Thanks for the comprehensive reply!

Spectralink specifically say that they do NOT support such thing as 'PTP-compliant switches' ;)

In our topology, we have few cascaded switches, but it's not more than 5 switches between any base stations. All switches are interconnected with 1 G links (copper or fiber), and all involved ports have zero failures/drops.

Could it be related to VoIP traffic that shares the same priority?

---

@svhelden wrote:

In our topology, we have few cascaded switches, but it's not more than 5 switches between any base stations. All switches are interconnected with 1 G links (copper or fiber), and all involved ports have zero failures/drops.

---

I've read that only the Store and Forward Latency of an Ethernet Switch for the maximum size Ethernet frame (1500 bytes) at 100 Mbps is about 120 μs...for comparison...the minimum size frame (64 bytes) at Gigabit speeds has a latency of just 0.5 μs (but 0.5 μs are still 500 ns, aren't?)...this without considering Switch Fabric Latency, Wireline Latency (really trascurable in LAN) and Queueing Latency (sensible to Network load).

So when I re-read the Spectralink 100 ns critical threshold...well...I start thinking their one is very strict (is it really possible to stay below the 100 ns threshold within the same Switch?).

---

@svhelden wrote:

Spectralink specifically say that they do NOT support such thing as 'PTP-compliant switches' ;)

Yep, also the Siemens HiPath Cordless IP V1's Product Manager (at time, so 4 years ago) stated something very similar...they didn't rely on IEEE 1588 Switch capabilities to permit the LAN Synchronization...but actual documentation seems to say the contrary (see below).

The recommandation about the maximum supported number of cascaded Switches admitted between the BSIP Sync Master and any BSIP Sync Salve is only 3 for the HiPath Cordless IP solution (your is higher but you have Spectralink...on the other hand it's also clear that you still have issue within the same Switch and not only between them...but...it's important to understand where is located - in the network topology - the BSIP Sync Master with respect to all BSIP Sync Slaves <- Star Synchronization Topology only).

And just a configuration example:

As you see the example above starts from a very simplified topology (basically no cascaded Switches).

---

@svhelden wrote:

Could it be related to VoIP traffic that shares the same priority?

---

Probably: Siemens recommends to separate VoIP VLAN from DECT Network VLAN and give to this one the highest priority (see above).

Don't forget how much high traffic situations - within a Switch or between them - can impact the Jitter value!

Note that Siemens placed all the BSIP(s) Base Stations (IWU too) within the control of just a single Switch! <-- that's a simply too ideal scenario in my opinion.

Real world cases, like your one, are a little bit more complex (or a little bit less simple).

Thanks parnassus!

From your comments, it seems that there is nothing much I can do configuration-wise. Maybe I could try giving VoIP traffic a lower priority, so that PTP packets are still treated with higher priority than these.

I'm also still waiting for information from Spectralink, maybe our issue is not related to Jitter at all. Their first response to our ticket was basically 'You have high jitter, of course it cannot work', but in the meantime they are also considering other issues..

The recommandation about the maximum supported number of cascaded Switches admitted between the BSIP Sync Master and any BSIP Sync Salve is only3 for the HiPath Cordless IP solution (your is higher but you have Spectralink..

Spectralink does not support a specific number, but their documentation says that they tested it with 5 switches, and with that we would comply.

it's important to understand where is located - in the network topology - the BSIP Sync Master

We tested with multiple stations as master, without notable difference. 

Anyway, thank you very much for the profound reply! I will post my future findings here ...

One more note, I noticed that older HP switches had an option "ip igmp high-priority-forward", which seems to be gone from newer models. Is there any replacement for this command, or is the functionality gone completely?

---

@svhelden wrote:

One more note, I noticed that older HP switches had an option "ip igmp high-priority-forward", which seems to be gone from newer models. Is there any replacement for this command, or is the functionality gone completely?

---

Strange.

On latest HPE ArubaOS-Switch Software Multicast and Routing Guide for K/KA/KB.16.01 (May 2016) there is a clear reference (sheet 30) about IGMP traffic normal/high-priority forwarding feature with or without IP addressing configured on VLAN but then, looking at the command ip igmp options, there isn't (sheet 39)!

Considering your reported QoS settings:

qos dscp-map cs5 priority 7 name "Lync-40"
qos dscp-map ef name "Lync-46"
qos type-of-service diff-services
vlan 30
name "DECT"
ip igmp

I'm curious enough to ask you to show yours QoS DSCP mappings (with show qos vlan-priority and show qos dscp-map)...could you? I never stop learning.

I'm curious enough to ask you to show yours QoS DSCP mappings

Sure.

SHOW RUN

...
qos dscp-map cs5 priority 5 name "Skype-40"
qos dscp-map ef name "PTP-46"
qos type-of-service diff-services
...
vlan 30
name "DECT"
...
ip igmp
exit
...

SHOW QOS DSCP

DSCP Policies

DSCP CodePoint DSCP Value 802.1p tag DSCP Policy name
-------------- ---------- ----------- --------------------------------
000000 0 No-override cs0
000001 1 No-override
000010 2 No-override
000011 3 No-override
000100 4 No-override
000101 5 No-override
000110 6 No-override
000111 7 No-override
001000 8 No-override cs1
001001 9 No-override
001010 10 No-override af11
001011 11 No-override
001100 12 No-override af12
001101 13 No-override
001110 14 No-override af13
001111 15 No-override
010000 16 No-override cs2
010001 17 No-override
010010 18 No-override af21
010011 19 No-override
010100 20 No-override af22
010101 21 No-override
010110 22 No-override af23
010111 23 No-override
011000 24 No-override cs3
011001 25 No-override
011010 26 No-override af31
011011 27 No-override
011100 28 No-override af32
011101 29 No-override
011110 30 No-override af33
011111 31 No-override
100000 32 No-override cs4
100001 33 No-override
100010 34 No-override af41
100011 35 No-override
100100 36 No-override af42
100101 37 No-override
100110 38 No-override af43
100111 39 No-override
101000 40 5 Skype-40
101001 41 No-override
101010 42 No-override
101011 43 No-override
101100 44 No-override
101101 45 No-override
101110 46 7 PTP-46
101111 47 No-override
110000 48 No-override cs6
110001 49 No-override
110010 50 No-override
110011 51 No-override
110100 52 No-override
110101 53 No-override
110110 54 No-override
110111 55 No-override
111000 56 No-override cs7
111001 57 No-override
111010 58 No-override
111011 59 No-override
111100 60 No-override
111101 61 No-override
111110 62 No-override
111111 63 No-override

SHOW QOS VLAN

VLAN-based Prioritization

VLAN ID Apply rule | DSCP Priority
------- ----------- + ------ -----------
1 No-override | No-override
30 No-override | No-override

Hey all, actually I´m experiencing similar problems with my Spectralink system, but with a Comware based Star-Topology (5900AF core, 5130 Access)...I tried to understand the PTP configuration in the Comware manuals...but actually anything I configured there had only one effect, that the stations lose sync immediately :-)

Any ideas?

Am I the only one, considering the Dect network requirements a sign of a badly designed system? 

I think you're not alone...but...are you asking that...with PTP related issues/restrictions [*] in mind or, let's say, totally in general about the whole technology (maybe compared to "pure DECT" or to "pure VoIP over WLAN")?

Just curious.

[*] we should keep in mind that synchronization between DECT IP Base Stations (which is the prerequisite for seamless handover withing the same DECT IP cluster) - normally on small deployments - happens "Over the Air" (also called "DECT based synchronization") [**] and this is more true if there aren't design limitations on the number of installable base stations (to grant the right overlapping) and about their placement on the entire covered site (to ensure the requested overlapping).

[**] for synchronization "Over the Air" (by means of DECT - so on range of 1.8/1.9 GHz - radio signals), the DECT IP Base Stations that synchronize each others must be able to exchange their management information (Beacons) flawlessly so those base stations must be located in areas where their radio cells overlap respecting the requested RSSI (Received Signal Strength Indication) not only between them and moving DECT mobile terminals but also between them too.

---

@svhelden wrote:

We tested with multiple stations as master, without notable difference.

Don't you see any difference if you assign the Sync Master role to a Base Station connected to the same Switch that connects most of the others (Sync Slave) Base Stations?

Don't you see any difference if you assign the Sync Master role to a Base Station connected to the same Switch that connects most of the others (Sync Slave) Base Stations?

The stations are distributed quite evenly across all switches (it does not make sense to have all DECT base stations in the same corner ;) )

As expected, average jitter is lower for stations on the same switch as the master, and higher for stations on 'distant' switches. But still there is a high varation within one switch.

"In our topology, we have few cascaded switches"

Can you shorten the topology to a star, so all nodes have at most two hops to each other?

Daisy chaining or cascading switches will add latency and jitter at each hop, so reducing hops may help. I tend to try and reduce the hop count as much as possible.

Could you add more trunk links (lacp, etc) between the switches, and tune trunking to better utilise the links?

"Could it be related to VoIP traffic that shares the same priority?"

Probably. I would prefer to separate realtime, voice and video into different queues. As Skype for business often gets used with video conferencing, I'd suggest separating the traffic, as video conferencing will cause problems with purely realtime and voice traffic.

QoS on HP Procurve switches is based (AFAIK) on DWRR (Deficit Weighted Round Robin), and the highest priority queue 8 (ip prec 7) by default has a large percentage of time allococated. Each switch ASIC may have a slightly different queue style and default config, so I would review the queue differences between the models as well.

Maybe if you split realtime and voice/video into different queues (PTP queue 8, Voip queue 7), and if you adjust the queue config, that may help. You won't know unless you test it. By adding more time to Queue 7, you'll have to reduce time elsewhere down the queue list.

Have you captured the traffic and checked QoS DSCP marking is used/applied at each hop correctly, and whether IP PREC is used between switches on the trunks?

HP Procurve QoS Queueing operation depends upon IP PREC, and if the traffic has a DSCP marking, every switch in the path must also have a DSCP-MAP to IP PREC, so the queue works correctly on that traffic. End to end QoS requires all hops to have the same DSCP-MAP's, if you want the packet to classify and queue correctly. I'd be checking source and destination switches ports for vlan ip precedence to confirm (only kept on vlan tagged links, so won't show on untagged ports), so check the egress and ingress trunks.

There are some better queue statistics commands in later models and versions. Show int queues <int> on 5406R shows TX queue stats by default for example. This is more difficult on older switch models or ealier software versions.

IP Precedence 7 is probably wrong. Just because it is a higher number than "6" doesn't mean it will get you better performance than "6".

Generally 7 is for quiet protocols where reliability is important (so no drops) but not latency.

I just read that you are expecting sub-microsecond latency. Provision switching latency is 10X higher than this.

You need ultra-low latency switches. Try Arista, or IBM.

I just read that you are expecting sub-microsecond latency. Provision switching latency is 10X higher than this.

Absolute latency doesn't matter. It's about Jitter (latency variation).

You can't get jitter that is an order of magnitude lower than the latency is. 

Put it this way - if your ruler is graduated in centimetres, you can't generate measurements of sub-millimetre precision.

@Sorry Vince I induced you in a error when I wrote:

---

@parnassus wrote:

I've read that only the Store and Forward Latency of an Ethernet Switch for the maximum size Ethernet frame (1500 bytes) at 100 Mbps is about 120 μs...for comparison...the minimum size frame (64 bytes) at Gigabit speeds has a latency of just 0.5 μs (but 0.5 μs are still 500 ns, aren't?)...this without considering Switch Fabric Latency, Wireline Latency (really trascurable in LAN) and Queueing Latency (sensible to Network load).

So when I re-read the Spectralink 100 ns critical threshold...well...I start thinking their one is very strict (is it really possible to stay below the 100 ns threshold within the same Switch?).

because I created a non-existent relationship between the overall traffic forwarding latency of a Switch's port and the Jitter its traffic can suffer of.

Forwarding Latency, on its own, has nothing to do with the Jitter (Variation of Delay)...overall traffic load can do. So it's perfectly possible to have a Switch capable of an average forwarding time of less than 3 μs (3000 ns) and still having its Jitter in the region of hundreds nano-seconds.

The point here is to understand how much a specific system (Spectralink DECT-IP) can tolerate Jitter for PTP traffic it generates...an average range of +/- 100 ns is very low, +/- 700 ns is seven times more.