Wired Intelligent Edge

Bear with me please, I'm a "Jack of all IT", broad of knowledge, but shallow. And I'm new to AOS-CX.

Most of our sites use 3810M switches, 802.1x port authentication and VLAN switching via freeradius3. We have a new site, and this site uses 6200M switches, same freeradius3 servers. But there are significant network reliability issues for the users, to the point where they're all now using wifi (Aruba 615, connected to the same 6200M switches).

User hardware, OS, NIC drivers, cabling faults: all ruled out as a cause. It turns out that the reliability issues are only seen by users connected to ports with 802.1x authentication enabled. Dig a little deeper, and the 6200M port statistics show RX giants errors. There's no MTU settings changed for the switch stack, VLAN, or individual ports.

Set up a test laptop, running 6 ping tests as follows -
ping -t           8.8.8.8
ping -t -f -l 970 8.8.8.8
ping -t    -l 972 8.8.8.8
ping -t -f -l 972 8.8.8.8
ping -t -f -l 972 subnet-gateway
ping -t -f -l 972 stack-ip

The first two test always work, and the rest work if authentication for the port is disabled, but fail if authentication is enabled. I can toggle between authentication enabled/disabled with aaa authentication port-access dot1x authenticator enable and aaa authentication port-access dot1x authenticator disabled, no other changes, and the last four ping tests change from fail to pass pretty much instantly (and vise versa).

Why would 802.1x port authentication be having such an impact? Note that I'm using the same VLAN for ports without authentication and for ports that are authenticated. The freeradius post-auth configuration contains only the following -
update reply {
    Tunnel-Type := VLAN
    Tunnel-Medium-Type := IEEE-802
    Tunnel-Private-Group-Id = "11"
}

An example switch port setting is -
interface 4/1/2
    description test laptop NDTR5035
    no shutdown
    no routing
    vlan access 11
    aaa authentication port-access dot1x authenticator
        cached-reauth

The same setup is working at other sites, where we use Aruba 3810M switches (of course the config statements differ but so far as radius/auth is concerned they're pretty much out of the box).

So what on earth is going on? How can I debug this? How can I fix this?!

OS version: ArubaOS-CX ML.10.13.1120

are you using EAP-TLS authentication? and is your freeRadius in the same campus or across the WAN? 

I don't know about freeRadius, but see if it has a way to perform/enable eap-tls fragmentation . 

Bear with me please, I'm a "Jack of all IT", broad of knowledge, but shallow. And I'm new to AOS-CX.

Most of our sites use 3810M switches, 802.1x port authentication and VLAN switching via freeradius3. We have a new site, and this site uses 6200M switches, same freeradius3 servers. But there are significant network reliability issues for the users, to the point where they're all now using wifi (Aruba 615, connected to the same 6200M switches).

User hardware, OS, NIC drivers, cabling faults: all ruled out as a cause. It turns out that the reliability issues are only seen by users connected to ports with 802.1x authentication enabled. Dig a little deeper, and the 6200M port statistics show RX giants errors. There's no MTU settings changed for the switch stack, VLAN, or individual ports.

Set up a test laptop, running 6 ping tests as follows -
ping -t           8.8.8.8
ping -t -f -l 970 8.8.8.8
ping -t    -l 972 8.8.8.8
ping -t -f -l 972 8.8.8.8
ping -t -f -l 972 subnet-gateway
ping -t -f -l 972 stack-ip

The first two test always work, and the rest work if authentication for the port is disabled, but fail if authentication is enabled. I can toggle between authentication enabled/disabled with aaa authentication port-access dot1x authenticator enable and aaa authentication port-access dot1x authenticator disabled, no other changes, and the last four ping tests change from fail to pass pretty much instantly (and vise versa).

Why would 802.1x port authentication be having such an impact? Note that I'm using the same VLAN for ports without authentication and for ports that are authenticated. The freeradius post-auth configuration contains only the following -
update reply {
    Tunnel-Type := VLAN
    Tunnel-Medium-Type := IEEE-802
    Tunnel-Private-Group-Id = "11"
}

An example switch port setting is -
interface 4/1/2
    description test laptop NDTR5035
    no shutdown
    no routing
    vlan access 11
    aaa authentication port-access dot1x authenticator
        cached-reauth

The same setup is working at other sites, where we use Aruba 3810M switches (of course the config statements differ but so far as radius/auth is concerned they're pretty much out of the box).

So what on earth is going on? How can I debug this? How can I fix this?!

OS version: ArubaOS-CX ML.10.13.1120

We're using PEAP-MS-CHAP v2, but I think I know what you're referring to, and that's the packet size issue with EAP wanting the entire certificate in one packet. Correct me if I'm wrong, but that prevents authentication? In this case authentication is actually successful, it's the resulting connection that has issues. There is a radius variable, framed-MTU, but I'm not specifically setting that (for any sites), and I don't know if AOS-CX pays any attention to it.

The radius servers... one is local, one is remote (there are two for failover). We do also have EAP-TLS for printers, and MAC authentication for some small infrastructure such as door security controllers. I'll see if I can test the issue against those things this weekend (I have to be careful as this is now a live site, though I'm awaiting delivery of a spare switch for me to use at my base location for dev/test.

are you using EAP-TLS authentication? and is your freeRadius in the same campus or across the WAN? 

I don't know about freeRadius, but see if it has a way to perform/enable eap-tls fragmentation . 

Bear with me please, I'm a "Jack of all IT", broad of knowledge, but shallow. And I'm new to AOS-CX.

Most of our sites use 3810M switches, 802.1x port authentication and VLAN switching via freeradius3. We have a new site, and this site uses 6200M switches, same freeradius3 servers. But there are significant network reliability issues for the users, to the point where they're all now using wifi (Aruba 615, connected to the same 6200M switches).

User hardware, OS, NIC drivers, cabling faults: all ruled out as a cause. It turns out that the reliability issues are only seen by users connected to ports with 802.1x authentication enabled. Dig a little deeper, and the 6200M port statistics show RX giants errors. There's no MTU settings changed for the switch stack, VLAN, or individual ports.

Set up a test laptop, running 6 ping tests as follows -
ping -t           8.8.8.8
ping -t -f -l 970 8.8.8.8
ping -t    -l 972 8.8.8.8
ping -t -f -l 972 8.8.8.8
ping -t -f -l 972 subnet-gateway
ping -t -f -l 972 stack-ip

The first two test always work, and the rest work if authentication for the port is disabled, but fail if authentication is enabled. I can toggle between authentication enabled/disabled with aaa authentication port-access dot1x authenticator enable and aaa authentication port-access dot1x authenticator disabled, no other changes, and the last four ping tests change from fail to pass pretty much instantly (and vise versa).

Why would 802.1x port authentication be having such an impact? Note that I'm using the same VLAN for ports without authentication and for ports that are authenticated. The freeradius post-auth configuration contains only the following -
update reply {
    Tunnel-Type := VLAN
    Tunnel-Medium-Type := IEEE-802
    Tunnel-Private-Group-Id = "11"
}

An example switch port setting is -
interface 4/1/2
    description test laptop NDTR5035
    no shutdown
    no routing
    vlan access 11
    aaa authentication port-access dot1x authenticator
        cached-reauth

The same setup is working at other sites, where we use Aruba 3810M switches (of course the config statements differ but so far as radius/auth is concerned they're pretty much out of the box).

So what on earth is going on? How can I debug this? How can I fix this?!

OS version: ArubaOS-CX ML.10.13.1120

Can you explain the reliability issues? What do you notice?

AAA doesn't have direct impact on the MTU (unless you lower the MTU using a VSA). There is not reason to set the framed MTU value.

The Giants you see must have another reason. Are you using Aruba controllers/gateways in the setup. If yes, AP's will do MTU discovery which can results in giants on the switch interface where APs are connected.

Regarding authentication. If the clients are doing doing certificate based authentication, that could result in fragmentation issues. But you're using EAP-PEAP MSCHAP so that is username/password based. 

Regarding EAP-TLS. It's recommended to lower the eap fragment size using the configuration. This will fragment the client certificate on the EAP layer. I believe freeradius does the same by default with the MTU set to 1024. 

aaa authentication port-access dot1x authenticator    eap-tls-fragment towards-server 1024

We're using PEAP-MS-CHAP v2, but I think I know what you're referring to, and that's the packet size issue with EAP wanting the entire certificate in one packet. Correct me if I'm wrong, but that prevents authentication? In this case authentication is actually successful, it's the resulting connection that has issues. There is a radius variable, framed-MTU, but I'm not specifically setting that (for any sites), and I don't know if AOS-CX pays any attention to it.

The radius servers... one is local, one is remote (there are two for failover). We do also have EAP-TLS for printers, and MAC authentication for some small infrastructure such as door security controllers. I'll see if I can test the issue against those things this weekend (I have to be careful as this is now a live site, though I'm awaiting delivery of a spare switch for me to use at my base location for dev/test.

are you using EAP-TLS authentication? and is your freeRadius in the same campus or across the WAN? 

I don't know about freeRadius, but see if it has a way to perform/enable eap-tls fragmentation . 

Bear with me please, I'm a "Jack of all IT", broad of knowledge, but shallow. And I'm new to AOS-CX.

Most of our sites use 3810M switches, 802.1x port authentication and VLAN switching via freeradius3. We have a new site, and this site uses 6200M switches, same freeradius3 servers. But there are significant network reliability issues for the users, to the point where they're all now using wifi (Aruba 615, connected to the same 6200M switches).

User hardware, OS, NIC drivers, cabling faults: all ruled out as a cause. It turns out that the reliability issues are only seen by users connected to ports with 802.1x authentication enabled. Dig a little deeper, and the 6200M port statistics show RX giants errors. There's no MTU settings changed for the switch stack, VLAN, or individual ports.

Set up a test laptop, running 6 ping tests as follows -
ping -t           8.8.8.8
ping -t -f -l 970 8.8.8.8
ping -t    -l 972 8.8.8.8
ping -t -f -l 972 8.8.8.8
ping -t -f -l 972 subnet-gateway
ping -t -f -l 972 stack-ip

The first two test always work, and the rest work if authentication for the port is disabled, but fail if authentication is enabled. I can toggle between authentication enabled/disabled with aaa authentication port-access dot1x authenticator enable and aaa authentication port-access dot1x authenticator disabled, no other changes, and the last four ping tests change from fail to pass pretty much instantly (and vise versa).

Why would 802.1x port authentication be having such an impact? Note that I'm using the same VLAN for ports without authentication and for ports that are authenticated. The freeradius post-auth configuration contains only the following -
update reply {
    Tunnel-Type := VLAN
    Tunnel-Medium-Type := IEEE-802
    Tunnel-Private-Group-Id = "11"
}

An example switch port setting is -
interface 4/1/2
    description test laptop NDTR5035
    no shutdown
    no routing
    vlan access 11
    aaa authentication port-access dot1x authenticator
        cached-reauth

The same setup is working at other sites, where we use Aruba 3810M switches (of course the config statements differ but so far as radius/auth is concerned they're pretty much out of the box).

So what on earth is going on? How can I debug this? How can I fix this?!

OS version: ArubaOS-CX ML.10.13.1120

Hi, authentication is successful, there isn't a problem between the switch and the radius servers, but there's a problem with the clients that are authenticated and everything else.

Disable authentication on the port and the client can ping other devices with a payload in of 972 bytes or greater.

Enable authentication on the port, the client authenticates, the client can not ping other devices when the payload is 972 bytes or greater.

The giants count for the port is static when the port is configured to not use authentication, then immediately begins to increment when authentication is enabled.

So far as I can tell framed-MTU only relates to communication between the switch and the radius server, so no, it won' t have an impact on traffic through the authenticated port (I was clutching at straws).

Can you explain the reliability issues? What do you notice?

AAA doesn't have direct impact on the MTU (unless you lower the MTU using a VSA). There is not reason to set the framed MTU value.

The Giants you see must have another reason. Are you using Aruba controllers/gateways in the setup. If yes, AP's will do MTU discovery which can results in giants on the switch interface where APs are connected.

Regarding authentication. If the clients are doing doing certificate based authentication, that could result in fragmentation issues. But you're using EAP-PEAP MSCHAP so that is username/password based. 

Regarding EAP-TLS. It's recommended to lower the eap fragment size using the configuration. This will fragment the client certificate on the EAP layer. I believe freeradius does the same by default with the MTU set to 1024. 

aaa authentication port-access dot1x authenticator    eap-tls-fragment towards-server 1024

We're using PEAP-MS-CHAP v2, but I think I know what you're referring to, and that's the packet size issue with EAP wanting the entire certificate in one packet. Correct me if I'm wrong, but that prevents authentication? In this case authentication is actually successful, it's the resulting connection that has issues. There is a radius variable, framed-MTU, but I'm not specifically setting that (for any sites), and I don't know if AOS-CX pays any attention to it.

The radius servers... one is local, one is remote (there are two for failover). We do also have EAP-TLS for printers, and MAC authentication for some small infrastructure such as door security controllers. I'll see if I can test the issue against those things this weekend (I have to be careful as this is now a live site, though I'm awaiting delivery of a spare switch for me to use at my base location for dev/test.

are you using EAP-TLS authentication? and is your freeRadius in the same campus or across the WAN? 

I don't know about freeRadius, but see if it has a way to perform/enable eap-tls fragmentation . 

Bear with me please, I'm a "Jack of all IT", broad of knowledge, but shallow. And I'm new to AOS-CX.

Most of our sites use 3810M switches, 802.1x port authentication and VLAN switching via freeradius3. We have a new site, and this site uses 6200M switches, same freeradius3 servers. But there are significant network reliability issues for the users, to the point where they're all now using wifi (Aruba 615, connected to the same 6200M switches).

User hardware, OS, NIC drivers, cabling faults: all ruled out as a cause. It turns out that the reliability issues are only seen by users connected to ports with 802.1x authentication enabled. Dig a little deeper, and the 6200M port statistics show RX giants errors. There's no MTU settings changed for the switch stack, VLAN, or individual ports.

Set up a test laptop, running 6 ping tests as follows -
ping -t           8.8.8.8
ping -t -f -l 970 8.8.8.8
ping -t    -l 972 8.8.8.8
ping -t -f -l 972 8.8.8.8
ping -t -f -l 972 subnet-gateway
ping -t -f -l 972 stack-ip

The first two test always work, and the rest work if authentication for the port is disabled, but fail if authentication is enabled. I can toggle between authentication enabled/disabled with aaa authentication port-access dot1x authenticator enable and aaa authentication port-access dot1x authenticator disabled, no other changes, and the last four ping tests change from fail to pass pretty much instantly (and vise versa).

Why would 802.1x port authentication be having such an impact? Note that I'm using the same VLAN for ports without authentication and for ports that are authenticated. The freeradius post-auth configuration contains only the following -
update reply {
    Tunnel-Type := VLAN
    Tunnel-Medium-Type := IEEE-802
    Tunnel-Private-Group-Id = "11"
}

An example switch port setting is -
interface 4/1/2
    description test laptop NDTR5035
    no shutdown
    no routing
    vlan access 11
    aaa authentication port-access dot1x authenticator
        cached-reauth

The same setup is working at other sites, where we use Aruba 3810M switches (of course the config statements differ but so far as radius/auth is concerned they're pretty much out of the box).

So what on earth is going on? How can I debug this? How can I fix this?!

OS version: ArubaOS-CX ML.10.13.1120

Please share the output of the following command.

show port-access clients interface <client-interface> detail

Hi, authentication is successful, there isn't a problem between the switch and the radius servers, but there's a problem with the clients that are authenticated and everything else.

Disable authentication on the port and the client can ping other devices with a payload in of 972 bytes or greater.

Enable authentication on the port, the client authenticates, the client can not ping other devices when the payload is 972 bytes or greater.

The giants count for the port is static when the port is configured to not use authentication, then immediately begins to increment when authentication is enabled.

So far as I can tell framed-MTU only relates to communication between the switch and the radius server, so no, it won' t have an impact on traffic through the authenticated port (I was clutching at straws).

Can you explain the reliability issues? What do you notice?

AAA doesn't have direct impact on the MTU (unless you lower the MTU using a VSA). There is not reason to set the framed MTU value.

The Giants you see must have another reason. Are you using Aruba controllers/gateways in the setup. If yes, AP's will do MTU discovery which can results in giants on the switch interface where APs are connected.

Regarding authentication. If the clients are doing doing certificate based authentication, that could result in fragmentation issues. But you're using EAP-PEAP MSCHAP so that is username/password based. 

Regarding EAP-TLS. It's recommended to lower the eap fragment size using the configuration. This will fragment the client certificate on the EAP layer. I believe freeradius does the same by default with the MTU set to 1024. 

aaa authentication port-access dot1x authenticator    eap-tls-fragment towards-server 1024

We're using PEAP-MS-CHAP v2, but I think I know what you're referring to, and that's the packet size issue with EAP wanting the entire certificate in one packet. Correct me if I'm wrong, but that prevents authentication? In this case authentication is actually successful, it's the resulting connection that has issues. There is a radius variable, framed-MTU, but I'm not specifically setting that (for any sites), and I don't know if AOS-CX pays any attention to it.

The radius servers... one is local, one is remote (there are two for failover). We do also have EAP-TLS for printers, and MAC authentication for some small infrastructure such as door security controllers. I'll see if I can test the issue against those things this weekend (I have to be careful as this is now a live site, though I'm awaiting delivery of a spare switch for me to use at my base location for dev/test.

are you using EAP-TLS authentication? and is your freeRadius in the same campus or across the WAN? 

I don't know about freeRadius, but see if it has a way to perform/enable eap-tls fragmentation . 

Bear with me please, I'm a "Jack of all IT", broad of knowledge, but shallow. And I'm new to AOS-CX.

Most of our sites use 3810M switches, 802.1x port authentication and VLAN switching via freeradius3. We have a new site, and this site uses 6200M switches, same freeradius3 servers. But there are significant network reliability issues for the users, to the point where they're all now using wifi (Aruba 615, connected to the same 6200M switches).

User hardware, OS, NIC drivers, cabling faults: all ruled out as a cause. It turns out that the reliability issues are only seen by users connected to ports with 802.1x authentication enabled. Dig a little deeper, and the 6200M port statistics show RX giants errors. There's no MTU settings changed for the switch stack, VLAN, or individual ports.

Set up a test laptop, running 6 ping tests as follows -
ping -t           8.8.8.8
ping -t -f -l 970 8.8.8.8
ping -t    -l 972 8.8.8.8
ping -t -f -l 972 8.8.8.8
ping -t -f -l 972 subnet-gateway
ping -t -f -l 972 stack-ip

The first two test always work, and the rest work if authentication for the port is disabled, but fail if authentication is enabled. I can toggle between authentication enabled/disabled with aaa authentication port-access dot1x authenticator enable and aaa authentication port-access dot1x authenticator disabled, no other changes, and the last four ping tests change from fail to pass pretty much instantly (and vise versa).

Why would 802.1x port authentication be having such an impact? Note that I'm using the same VLAN for ports without authentication and for ports that are authenticated. The freeradius post-auth configuration contains only the following -
update reply {
    Tunnel-Type := VLAN
    Tunnel-Medium-Type := IEEE-802
    Tunnel-Private-Group-Id = "11"
}

An example switch port setting is -
interface 4/1/2
    description test laptop NDTR5035
    no shutdown
    no routing
    vlan access 11
    aaa authentication port-access dot1x authenticator
        cached-reauth

The same setup is working at other sites, where we use Aruba 3810M switches (of course the config statements differ but so far as radius/auth is concerned they're pretty much out of the box).

So what on earth is going on? How can I debug this? How can I fix this?!

OS version: ArubaOS-CX ML.10.13.1120

And also 

show interface <client-interface>

Please share the output of the following command.

show port-access clients interface <client-interface> detail

Hi, authentication is successful, there isn't a problem between the switch and the radius servers, but there's a problem with the clients that are authenticated and everything else.

Disable authentication on the port and the client can ping other devices with a payload in of 972 bytes or greater.

Enable authentication on the port, the client authenticates, the client can not ping other devices when the payload is 972 bytes or greater.

The giants count for the port is static when the port is configured to not use authentication, then immediately begins to increment when authentication is enabled.

So far as I can tell framed-MTU only relates to communication between the switch and the radius server, so no, it won' t have an impact on traffic through the authenticated port (I was clutching at straws).

Can you explain the reliability issues? What do you notice?

AAA doesn't have direct impact on the MTU (unless you lower the MTU using a VSA). There is not reason to set the framed MTU value.

The Giants you see must have another reason. Are you using Aruba controllers/gateways in the setup. If yes, AP's will do MTU discovery which can results in giants on the switch interface where APs are connected.

Regarding authentication. If the clients are doing doing certificate based authentication, that could result in fragmentation issues. But you're using EAP-PEAP MSCHAP so that is username/password based. 

Regarding EAP-TLS. It's recommended to lower the eap fragment size using the configuration. This will fragment the client certificate on the EAP layer. I believe freeradius does the same by default with the MTU set to 1024. 

aaa authentication port-access dot1x authenticator    eap-tls-fragment towards-server 1024

We're using PEAP-MS-CHAP v2, but I think I know what you're referring to, and that's the packet size issue with EAP wanting the entire certificate in one packet. Correct me if I'm wrong, but that prevents authentication? In this case authentication is actually successful, it's the resulting connection that has issues. There is a radius variable, framed-MTU, but I'm not specifically setting that (for any sites), and I don't know if AOS-CX pays any attention to it.

The radius servers... one is local, one is remote (there are two for failover). We do also have EAP-TLS for printers, and MAC authentication for some small infrastructure such as door security controllers. I'll see if I can test the issue against those things this weekend (I have to be careful as this is now a live site, though I'm awaiting delivery of a spare switch for me to use at my base location for dev/test.

are you using EAP-TLS authentication? and is your freeRadius in the same campus or across the WAN? 

I don't know about freeRadius, but see if it has a way to perform/enable eap-tls fragmentation . 

Bear with me please, I'm a "Jack of all IT", broad of knowledge, but shallow. And I'm new to AOS-CX.

Most of our sites use 3810M switches, 802.1x port authentication and VLAN switching via freeradius3. We have a new site, and this site uses 6200M switches, same freeradius3 servers. But there are significant network reliability issues for the users, to the point where they're all now using wifi (Aruba 615, connected to the same 6200M switches).

User hardware, OS, NIC drivers, cabling faults: all ruled out as a cause. It turns out that the reliability issues are only seen by users connected to ports with 802.1x authentication enabled. Dig a little deeper, and the 6200M port statistics show RX giants errors. There's no MTU settings changed for the switch stack, VLAN, or individual ports.

Set up a test laptop, running 6 ping tests as follows -
ping -t           8.8.8.8
ping -t -f -l 970 8.8.8.8
ping -t    -l 972 8.8.8.8
ping -t -f -l 972 8.8.8.8
ping -t -f -l 972 subnet-gateway
ping -t -f -l 972 stack-ip

The first two test always work, and the rest work if authentication for the port is disabled, but fail if authentication is enabled. I can toggle between authentication enabled/disabled with aaa authentication port-access dot1x authenticator enable and aaa authentication port-access dot1x authenticator disabled, no other changes, and the last four ping tests change from fail to pass pretty much instantly (and vise versa).

Why would 802.1x port authentication be having such an impact? Note that I'm using the same VLAN for ports without authentication and for ports that are authenticated. The freeradius post-auth configuration contains only the following -
update reply {
    Tunnel-Type := VLAN
    Tunnel-Medium-Type := IEEE-802
    Tunnel-Private-Group-Id = "11"
}

An example switch port setting is -
interface 4/1/2
    description test laptop NDTR5035
    no shutdown
    no routing
    vlan access 11
    aaa authentication port-access dot1x authenticator
        cached-reauth

The same setup is working at other sites, where we use Aruba 3810M switches (of course the config statements differ but so far as radius/auth is concerned they're pretty much out of the box).

So what on earth is going on? How can I debug this? How can I fix this?!

OS version: ArubaOS-CX ML.10.13.1120

That's the answer.

Running radiusd -X shows clearly that attribute Framed-MTU += 994 is being sent to the switch as part of the access-accept message, and I should have spotted that sooner. I had thought that this attribute was applied to the comms between radius and switch - that's wrong, it is applied to the port being authenticated.

I don't have Framed-MTU set anywhere in my radius configs, so it must be baked into the code. I don't know why. Fortunately I can override it by adding Framed-MTU := 1500 within update_reply in the radius post-auth config. Now I see that the port MTU is set to 1500, as it is on ports not configured to use 802.1x authentication.

I checked back against my 3810M switches and they definitely were ignoring the radius attribute, so that's why they've been running for years without issue and only our new 6200M switches were seeing the issue.

Thanks folks! I was really starting to worry about this. Now I can relax and enjoy the rest of my weekend, I hope you all have good weekend too :-)

And also 

show interface <client-interface>

Please share the output of the following command.

show port-access clients interface <client-interface> detail

Hi, authentication is successful, there isn't a problem between the switch and the radius servers, but there's a problem with the clients that are authenticated and everything else.

Disable authentication on the port and the client can ping other devices with a payload in of 972 bytes or greater.

Enable authentication on the port, the client authenticates, the client can not ping other devices when the payload is 972 bytes or greater.

The giants count for the port is static when the port is configured to not use authentication, then immediately begins to increment when authentication is enabled.

So far as I can tell framed-MTU only relates to communication between the switch and the radius server, so no, it won' t have an impact on traffic through the authenticated port (I was clutching at straws).

Can you explain the reliability issues? What do you notice?

AAA doesn't have direct impact on the MTU (unless you lower the MTU using a VSA). There is not reason to set the framed MTU value.

The Giants you see must have another reason. Are you using Aruba controllers/gateways in the setup. If yes, AP's will do MTU discovery which can results in giants on the switch interface where APs are connected.

Regarding authentication. If the clients are doing doing certificate based authentication, that could result in fragmentation issues. But you're using EAP-PEAP MSCHAP so that is username/password based. 

Regarding EAP-TLS. It's recommended to lower the eap fragment size using the configuration. This will fragment the client certificate on the EAP layer. I believe freeradius does the same by default with the MTU set to 1024. 

aaa authentication port-access dot1x authenticator    eap-tls-fragment towards-server 1024

We're using PEAP-MS-CHAP v2, but I think I know what you're referring to, and that's the packet size issue with EAP wanting the entire certificate in one packet. Correct me if I'm wrong, but that prevents authentication? In this case authentication is actually successful, it's the resulting connection that has issues. There is a radius variable, framed-MTU, but I'm not specifically setting that (for any sites), and I don't know if AOS-CX pays any attention to it.

The radius servers... one is local, one is remote (there are two for failover). We do also have EAP-TLS for printers, and MAC authentication for some small infrastructure such as door security controllers. I'll see if I can test the issue against those things this weekend (I have to be careful as this is now a live site, though I'm awaiting delivery of a spare switch for me to use at my base location for dev/test.

are you using EAP-TLS authentication? and is your freeRadius in the same campus or across the WAN? 

I don't know about freeRadius, but see if it has a way to perform/enable eap-tls fragmentation . 

Bear with me please, I'm a "Jack of all IT", broad of knowledge, but shallow. And I'm new to AOS-CX.

Most of our sites use 3810M switches, 802.1x port authentication and VLAN switching via freeradius3. We have a new site, and this site uses 6200M switches, same freeradius3 servers. But there are significant network reliability issues for the users, to the point where they're all now using wifi (Aruba 615, connected to the same 6200M switches).

User hardware, OS, NIC drivers, cabling faults: all ruled out as a cause. It turns out that the reliability issues are only seen by users connected to ports with 802.1x authentication enabled. Dig a little deeper, and the 6200M port statistics show RX giants errors. There's no MTU settings changed for the switch stack, VLAN, or individual ports.

Set up a test laptop, running 6 ping tests as follows -
ping -t           8.8.8.8
ping -t -f -l 970 8.8.8.8
ping -t    -l 972 8.8.8.8
ping -t -f -l 972 8.8.8.8
ping -t -f -l 972 subnet-gateway
ping -t -f -l 972 stack-ip

The first two test always work, and the rest work if authentication for the port is disabled, but fail if authentication is enabled. I can toggle between authentication enabled/disabled with aaa authentication port-access dot1x authenticator enable and aaa authentication port-access dot1x authenticator disabled, no other changes, and the last four ping tests change from fail to pass pretty much instantly (and vise versa).

Why would 802.1x port authentication be having such an impact? Note that I'm using the same VLAN for ports without authentication and for ports that are authenticated. The freeradius post-auth configuration contains only the following -
update reply {
    Tunnel-Type := VLAN
    Tunnel-Medium-Type := IEEE-802
    Tunnel-Private-Group-Id = "11"
}

An example switch port setting is -
interface 4/1/2
    description test laptop NDTR5035
    no shutdown
    no routing
    vlan access 11
    aaa authentication port-access dot1x authenticator
        cached-reauth

The same setup is working at other sites, where we use Aruba 3810M switches (of course the config statements differ but so far as radius/auth is concerned they're pretty much out of the box).

So what on earth is going on? How can I debug this? How can I fix this?!

OS version: ArubaOS-CX ML.10.13.1120

Please share the output of the following command.

show port-access clients interface <client-interface> detail

Hi, authentication is successful, there isn't a problem between the switch and the radius servers, but there's a problem with the clients that are authenticated and everything else.

Disable authentication on the port and the client can ping other devices with a payload in of 972 bytes or greater.

Enable authentication on the port, the client authenticates, the client can not ping other devices when the payload is 972 bytes or greater.

The giants count for the port is static when the port is configured to not use authentication, then immediately begins to increment when authentication is enabled.

So far as I can tell framed-MTU only relates to communication between the switch and the radius server, so no, it won' t have an impact on traffic through the authenticated port (I was clutching at straws).

Can you explain the reliability issues? What do you notice?

AAA doesn't have direct impact on the MTU (unless you lower the MTU using a VSA). There is not reason to set the framed MTU value.

The Giants you see must have another reason. Are you using Aruba controllers/gateways in the setup. If yes, AP's will do MTU discovery which can results in giants on the switch interface where APs are connected.

Regarding authentication. If the clients are doing doing certificate based authentication, that could result in fragmentation issues. But you're using EAP-PEAP MSCHAP so that is username/password based. 

Regarding EAP-TLS. It's recommended to lower the eap fragment size using the configuration. This will fragment the client certificate on the EAP layer. I believe freeradius does the same by default with the MTU set to 1024. 

aaa authentication port-access dot1x authenticator    eap-tls-fragment towards-server 1024

We're using PEAP-MS-CHAP v2, but I think I know what you're referring to, and that's the packet size issue with EAP wanting the entire certificate in one packet. Correct me if I'm wrong, but that prevents authentication? In this case authentication is actually successful, it's the resulting connection that has issues. There is a radius variable, framed-MTU, but I'm not specifically setting that (for any sites), and I don't know if AOS-CX pays any attention to it.

The radius servers... one is local, one is remote (there are two for failover). We do also have EAP-TLS for printers, and MAC authentication for some small infrastructure such as door security controllers. I'll see if I can test the issue against those things this weekend (I have to be careful as this is now a live site, though I'm awaiting delivery of a spare switch for me to use at my base location for dev/test.

are you using EAP-TLS authentication? and is your freeRadius in the same campus or across the WAN? 

I don't know about freeRadius, but see if it has a way to perform/enable eap-tls fragmentation . 

Bear with me please, I'm a "Jack of all IT", broad of knowledge, but shallow. And I'm new to AOS-CX.

Most of our sites use 3810M switches, 802.1x port authentication and VLAN switching via freeradius3. We have a new site, and this site uses 6200M switches, same freeradius3 servers. But there are significant network reliability issues for the users, to the point where they're all now using wifi (Aruba 615, connected to the same 6200M switches).

User hardware, OS, NIC drivers, cabling faults: all ruled out as a cause. It turns out that the reliability issues are only seen by users connected to ports with 802.1x authentication enabled. Dig a little deeper, and the 6200M port statistics show RX giants errors. There's no MTU settings changed for the switch stack, VLAN, or individual ports.

Set up a test laptop, running 6 ping tests as follows -
ping -t           8.8.8.8
ping -t -f -l 970 8.8.8.8
ping -t    -l 972 8.8.8.8
ping -t -f -l 972 8.8.8.8
ping -t -f -l 972 subnet-gateway
ping -t -f -l 972 stack-ip

The first two test always work, and the rest work if authentication for the port is disabled, but fail if authentication is enabled. I can toggle between authentication enabled/disabled with aaa authentication port-access dot1x authenticator enable and aaa authentication port-access dot1x authenticator disabled, no other changes, and the last four ping tests change from fail to pass pretty much instantly (and vise versa).

Why would 802.1x port authentication be having such an impact? Note that I'm using the same VLAN for ports without authentication and for ports that are authenticated. The freeradius post-auth configuration contains only the following -
update reply {
    Tunnel-Type := VLAN
    Tunnel-Medium-Type := IEEE-802
    Tunnel-Private-Group-Id = "11"
}

An example switch port setting is -
interface 4/1/2
    description test laptop NDTR5035
    no shutdown
    no routing
    vlan access 11
    aaa authentication port-access dot1x authenticator
        cached-reauth

The same setup is working at other sites, where we use Aruba 3810M switches (of course the config statements differ but so far as radius/auth is concerned they're pretty much out of the box).

So what on earth is going on? How can I debug this? How can I fix this?!

OS version: ArubaOS-CX ML.10.13.1120