Wired Intelligent Edge

Are you using the same AS number at all the devices? If yes, then you need to make a full mesh network or use route reflectors. This because of the iBGP rules.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

I forgot to reply about iBGO peering. On all switches I have very simple config for now an each switch has something like this:


router bgp 65001
! underlay neighbors
    address-family l2vpn evpn
        ! underlay neighbors activate and options
    exit-address-family
!
    vrf prod
    ! overlay external neighbors
        address-family ipv4 unicast
                      ! overlay external  neighbors activate and options
            redistribute connected
            redistribute static
        exit-address-family

I've attached full configs in the reply 7

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

Okay, that's looking quite good in my eyes. 

Regarding the prefix behind the Mikrotik router (10.180.10.0/24), this is present on all the 4 VSX devices either as eBGP route or as BGP-EVPN route (type 5) depending on the switch. And the default gateway towards the firewall is present as well on all 4 Aruba cores. 

This leaves me with the question, what issue you still have. How do you test whether communication from a network attached to the core with hosts in net 10.180.10.0/24 works or not? Do you have a machine connected to for instance DC2-S2? Or do you use a ping (send us the exact command) from the CLI of DC2-S2? What is the source and the destination?

You also need to make sure that the other end knows about the networks behind your Aruba cores. Did you check? Are prefixes like 10.100.1.0/24, 10.200.1.0/24 and 10.40.1.0/24 present? Otherwise the forwarding towards the network works but the reverse path is unknown/wrong.

Another thing which may be connected to post 11 by vincent.giles. It's usually good practice to have an "IGP continuity" peering between the two VSX nodes per VRF to exchange routes in case BGP/EVPN peerings brake for a reason on one node. This is more important in a leave-spine topology than with your full-mesh network but still something you perhaps want to have. Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.

Regards, 

Thomas

I forgot to reply about iBGO peering. On all switches I have very simple config for now an each switch has something like this:


router bgp 65001
! underlay neighbors
    address-family l2vpn evpn
        ! underlay neighbors activate and options
    exit-address-family
!
    vrf prod
    ! overlay external neighbors
        address-family ipv4 unicast
                      ! overlay external  neighbors activate and options
            redistribute connected
            redistribute static
        exit-address-family

I've attached full configs in the reply 7

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

Okay, that's looking quite good in my eyes. 

Regarding the prefix behind the Mikrotik router (10.180.10.0/24), this is present on all the 4 VSX devices either as eBGP route or as BGP-EVPN route (type 5) depending on the switch. And the default gateway towards the firewall is present as well on all 4 Aruba cores. 

This leaves me with the question, what issue you still have. How do you test whether communication from a network attached to the core with hosts in net 10.180.10.0/24 works or not? Do you have a machine connected to for instance DC2-S2? Or do you use a ping (send us the exact command) from the CLI of DC2-S2? What is the source and the destination?

You also need to make sure that the other end knows about the networks behind your Aruba cores. Did you check? Are prefixes like 10.100.1.0/24, 10.200.1.0/24 and 10.40.1.0/24 present? Otherwise the forwarding towards the network works but the reverse path is unknown/wrong.

Another thing which may be connected to post 11 by vincent.giles. It's usually good practice to have an "IGP continuity" peering between the two VSX nodes per VRF to exchange routes in case BGP/EVPN peerings brake for a reason on one node. This is more important in a leave-spine topology than with your full-mesh network but still something you perhaps want to have. Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.

Regards, 

Thomas

I forgot to reply about iBGO peering. On all switches I have very simple config for now an each switch has something like this:


router bgp 65001
! underlay neighbors
    address-family l2vpn evpn
        ! underlay neighbors activate and options
    exit-address-family
!
    vrf prod
    ! overlay external neighbors
        address-family ipv4 unicast
                      ! overlay external  neighbors activate and options
            redistribute connected
            redistribute static
        exit-address-family

I've attached full configs in the reply 7

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

> Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.

Do I understand  it correctly that to achieve this I need to add a secondary transit vlan on the VSX ISL link  and SVIs  in the production VRF?

If I use ospf it does not need to know anything about  the other dc, right?

I am not sure how to use BGP for that. Just and iBGP peering in the same AS as the Underlay using the additional addresses?

PS. The lack of proper quoting and threading in this forum is at least inconvenient. 

Okay, that's looking quite good in my eyes. 

Regarding the prefix behind the Mikrotik router (10.180.10.0/24), this is present on all the 4 VSX devices either as eBGP route or as BGP-EVPN route (type 5) depending on the switch. And the default gateway towards the firewall is present as well on all 4 Aruba cores. 

This leaves me with the question, what issue you still have. How do you test whether communication from a network attached to the core with hosts in net 10.180.10.0/24 works or not? Do you have a machine connected to for instance DC2-S2? Or do you use a ping (send us the exact command) from the CLI of DC2-S2? What is the source and the destination?

You also need to make sure that the other end knows about the networks behind your Aruba cores. Did you check? Are prefixes like 10.100.1.0/24, 10.200.1.0/24 and 10.40.1.0/24 present? Otherwise the forwarding towards the network works but the reverse path is unknown/wrong.

Another thing which may be connected to post 11 by vincent.giles. It's usually good practice to have an "IGP continuity" peering between the two VSX nodes per VRF to exchange routes in case BGP/EVPN peerings brake for a reason on one node. This is more important in a leave-spine topology than with your full-mesh network but still something you perhaps want to have. Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.

Regards, 

Thomas

I forgot to reply about iBGO peering. On all switches I have very simple config for now an each switch has something like this:


router bgp 65001
! underlay neighbors
    address-family l2vpn evpn
        ! underlay neighbors activate and options
    exit-address-family
!
    vrf prod
    ! overlay external neighbors
        address-family ipv4 unicast
                      ! overlay external  neighbors activate and options
            redistribute connected
            redistribute static
        exit-address-family

I've attached full configs in the reply 7

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

> Do I understand  it correctly that to achieve this I need to add a secondary transit vlan on the VSX ISL link  and SVIs  in the production VRF?

Yes, you need a transit VLAN per VRF in use for L3 routing between the two VSX nodes in a clean design. However, you may also use one of the VLAN interfaces with an IP on both nodes (you typically have 1 or more vlan interfaces in the VRF). Technically this does the same job but is a bit of a misuse.So the recommendation clearly is a separate transit VLAN. 

> If I use ospf it does not need to know anything about  the other dc, right?

Nope, only between the two VSX node in the same cluster. The inter-DC connectivity is handled by BGP-EVPN (with ipv4 address-family activated)

> I am not sure how to use BGP for that. Just and iBGP peering in the same AS as the Underlay using the additional addresses?

A single Aruba switch/router only can have one AS number assigned to it. In your case it is 65001. This automatically results in an iBGP type peering between the two VSX node if you configure a neighborship in the VRF "prod". But there is nothing against it. So just configure a transit VLAN, activate a neighborship within the VRF on it and add it as network. If everything is "normal", there shouldn't be any route in your routing table coming from that peering. However, if you for instance shutdown OSPF on one of the VSX nodes and therefore lose the connectivity to the other VTEPs you still have "a way out" by routing the traffic to the other VSX node. 

> Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.

Do I understand  it correctly that to achieve this I need to add a secondary transit vlan on the VSX ISL link  and SVIs  in the production VRF?

If I use ospf it does not need to know anything about  the other dc, right?

I am not sure how to use BGP for that. Just and iBGP peering in the same AS as the Underlay using the additional addresses?

PS. The lack of proper quoting and threading in this forum is at least inconvenient. 

Okay, that's looking quite good in my eyes. 

Regarding the prefix behind the Mikrotik router (10.180.10.0/24), this is present on all the 4 VSX devices either as eBGP route or as BGP-EVPN route (type 5) depending on the switch. And the default gateway towards the firewall is present as well on all 4 Aruba cores. 

This leaves me with the question, what issue you still have. How do you test whether communication from a network attached to the core with hosts in net 10.180.10.0/24 works or not? Do you have a machine connected to for instance DC2-S2? Or do you use a ping (send us the exact command) from the CLI of DC2-S2? What is the source and the destination?

You also need to make sure that the other end knows about the networks behind your Aruba cores. Did you check? Are prefixes like 10.100.1.0/24, 10.200.1.0/24 and 10.40.1.0/24 present? Otherwise the forwarding towards the network works but the reverse path is unknown/wrong.

Another thing which may be connected to post 11 by vincent.giles. It's usually good practice to have an "IGP continuity" peering between the two VSX nodes per VRF to exchange routes in case BGP/EVPN peerings brake for a reason on one node. This is more important in a leave-spine topology than with your full-mesh network but still something you perhaps want to have. Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.

Regards, 

Thomas

I forgot to reply about iBGO peering. On all switches I have very simple config for now an each switch has something like this:


router bgp 65001
! underlay neighbors
    address-family l2vpn evpn
        ! underlay neighbors activate and options
    exit-address-family
!
    vrf prod
    ! overlay external neighbors
        address-family ipv4 unicast
                      ! overlay external  neighbors activate and options
            redistribute connected
            redistribute static
        exit-address-family

I've attached full configs in the reply 7

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

> Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.

Do I understand  it correctly that to achieve this I need to add a secondary transit vlan on the VSX ISL link  and SVIs  in the production VRF?

If I use ospf it does not need to know anything about  the other dc, right?

I am not sure how to use BGP for that. Just and iBGP peering in the same AS as the Underlay using the additional addresses?

PS. The lack of proper quoting and threading in this forum is at least inconvenient. 

Okay, that's looking quite good in my eyes. 

Regarding the prefix behind the Mikrotik router (10.180.10.0/24), this is present on all the 4 VSX devices either as eBGP route or as BGP-EVPN route (type 5) depending on the switch. And the default gateway towards the firewall is present as well on all 4 Aruba cores. 

This leaves me with the question, what issue you still have. How do you test whether communication from a network attached to the core with hosts in net 10.180.10.0/24 works or not? Do you have a machine connected to for instance DC2-S2? Or do you use a ping (send us the exact command) from the CLI of DC2-S2? What is the source and the destination?

You also need to make sure that the other end knows about the networks behind your Aruba cores. Did you check? Are prefixes like 10.100.1.0/24, 10.200.1.0/24 and 10.40.1.0/24 present? Otherwise the forwarding towards the network works but the reverse path is unknown/wrong.

Another thing which may be connected to post 11 by vincent.giles. It's usually good practice to have an "IGP continuity" peering between the two VSX nodes per VRF to exchange routes in case BGP/EVPN peerings brake for a reason on one node. This is more important in a leave-spine topology than with your full-mesh network but still something you perhaps want to have. Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.

Regards, 

Thomas

I forgot to reply about iBGO peering. On all switches I have very simple config for now an each switch has something like this:


router bgp 65001
! underlay neighbors
    address-family l2vpn evpn
        ! underlay neighbors activate and options
    exit-address-family
!
    vrf prod
    ! overlay external neighbors
        address-family ipv4 unicast
                      ! overlay external  neighbors activate and options
            redistribute connected
            redistribute static
        exit-address-family

I've attached full configs in the reply 7

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

> The delay between interface shutdown and ping stopping  seems to be equal to the BGP hello time (which I set to 30s between Aurba and Mikrotik)

Make sure you have bgp fast-external-fallover and neighbor fall-over active in order to optimize switchover times to redundant links

What I don't like to see when looking at your outputs:

• In normal situation with both links up to DC3-R1:
  • DC1-CORE2 points the route to 10.180.10.0/24 towards DC2 (logical VTEP 10.31.100.2). I would expect this to be DC1-CORE1. A corrective measure would be an iBGP peering between DC1-CORE1 and DC1-CORE2
  • DC2-CORE2 points the route to 10.180.10.0/24 towards DC2 (logical VTEP 10.31.100.1). I would expect this to be DC2-CORE1. A corrective measure would be an iBGP peering between DC2-CORE1 and DC2-CORE2
• When 1 link shutdown
  • Are you sure you shut down the link between DC1-CORE1 to DC3? The output of Mikrotik still shows the address of DC1-CORE1 (172.30.100.66) as the next-hop for your 10.30.x/10.40.x routes. 
  • Does DC1-Core2 really show no route to 10.180.10.0/24 or is this a typo / copy&paste mistake?

Does the situation change (to the positive) after all BGP timers expired?

>  Probably  one VM hits primary switch in its DC and the other hits the secondary

Yes, if using standard vSwitch on VMware with 2 redundant links (both in "active" state) to the physical network each VM (not the portgroup!) has a "preferred" uplink NIC and the other is standby. This results in a behavior that same VMs use the first uplink and other use the secondary. You can influence it by setting one to "standby" in the portgroup settings. This may be useful for testing in order to have a more deterministic behavior. 

> Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.

Do I understand  it correctly that to achieve this I need to add a secondary transit vlan on the VSX ISL link  and SVIs  in the production VRF?

If I use ospf it does not need to know anything about  the other dc, right?

I am not sure how to use BGP for that. Just and iBGP peering in the same AS as the Underlay using the additional addresses?

PS. The lack of proper quoting and threading in this forum is at least inconvenient. 

Okay, that's looking quite good in my eyes. 

Regarding the prefix behind the Mikrotik router (10.180.10.0/24), this is present on all the 4 VSX devices either as eBGP route or as BGP-EVPN route (type 5) depending on the switch. And the default gateway towards the firewall is present as well on all 4 Aruba cores. 

This leaves me with the question, what issue you still have. How do you test whether communication from a network attached to the core with hosts in net 10.180.10.0/24 works or not? Do you have a machine connected to for instance DC2-S2? Or do you use a ping (send us the exact command) from the CLI of DC2-S2? What is the source and the destination?

You also need to make sure that the other end knows about the networks behind your Aruba cores. Did you check? Are prefixes like 10.100.1.0/24, 10.200.1.0/24 and 10.40.1.0/24 present? Otherwise the forwarding towards the network works but the reverse path is unknown/wrong.

Another thing which may be connected to post 11 by vincent.giles. It's usually good practice to have an "IGP continuity" peering between the two VSX nodes per VRF to exchange routes in case BGP/EVPN peerings brake for a reason on one node. This is more important in a leave-spine topology than with your full-mesh network but still something you perhaps want to have. Either activate OSPF also within the VRFs just between the two VSX nodes (as systems connected with MCLAG/LACP don't possibly know about the fact that L3 routing on one of the VSX nodes is not ready) in order to still have an exit on layer3 if EVPN goes down for whatever reason. The other option is to use BGP.

Regards, 

Thomas

I forgot to reply about iBGO peering. On all switches I have very simple config for now an each switch has something like this:


router bgp 65001
! underlay neighbors
    address-family l2vpn evpn
        ! underlay neighbors activate and options
    exit-address-family
!
    vrf prod
    ! overlay external neighbors
        address-family ipv4 unicast
                      ! overlay external  neighbors activate and options
            redistribute connected
            redistribute static
        exit-address-family

I've attached full configs in the reply 7

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

I think I've foudd the problem. OSPF cost on interfaces connecting DCs where different. After fixing that and clearing bgp session everything started to work as expected

interface 1/1/49
    description dci-1
    no shutdown
    mtu 9198
    ip address 172.30.255.128/31
    l3-counters
    ip ospf 1 area 0.0.0.0
    no ip ospf passive
    ip ospf cost 50 ! <-----here
    ip ospf network point-to-point
interface 1/1/50
    description dci-2
    no shutdown
    mtu 9198
    ip address 172.30.255.130/31
    l3-counters
    ip ospf 1 area 0.0.0.0
    no ip ospf passive
    ip ospf cost 100  ! <-----here
    ip ospf network point-to-point

Hi 

Thanks for the additional information. 

Any reason why you have different route-target import/export statements in DC1 and DC2?

For better understanding, could you please mention the peering VLAN / peering addresses to the external routers (e.g. firewall and DC3-R1) in your diagram? Could you also name the routes (or some of them) you are missing from the external routers?

Could you please send us the extract of "show ip route vrf prod" of each of the core devices?

If using EVPN you don't need a separate iBGP peering per VRF. This will be done though the peering used in the global bgp config. Just make sure you activate the ipv4 address-family per VRF and redistribute "connected" which you have. So in my eyes your bgp peerings look fine, given that you use a different BGP AS for the external peering with DC3-R1 and/or the firewall.

Regards, 

Thomas

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.

Thank you for the reply. 

I've attached output from  show bgp l2vpn evpn as files for clarity. How do I configure this  iBGP peering inside the VRF? Only between switches in VSX pair in each DC or between all of them. I would need a bunch of interconnects and IP addresses in the overlay network.  Or can I use  those propagated by evpn?

I've updated the diagram in original post because I posted wrong loopback addresses and I also added an internet uplink to the setup. The internet uplink behavior  is even stranger. The internet firewall have static routes for private address ranges pointing to an active gateway on shared vlan configured on DC1-S1 and DC1 S2.  All switches can ping to the internet except DC2-S1. It probably is connected to the topic because of how physical and vrrp addresses look from DC1-S1 and DC1-S2. 

For the interested I have attached also, somewhat cleaned configs.

Getting some show ip route and show bgp l2evpn evpn output would help on DC1-S1 and DC1-S2 to better understand.

As it is likely that the EVPN type-5 route on DC1-S2 includes the VSX logical VTEP anycast IP as the next-hop (172.31.101.1), this NH IP being hosted on DC1-S2, the packet is dropped. So, to your point, you would need an iBGP peering inside the tenant VRF (ala VRF-lite) to get routes properly learnt inside the said VRF between the VSX primary and secondary of the same VSX cluster.

Hope this helps.