EVPN – Basic Layer Two Extension Lab, Virtual Lab Example

Introduction

This is a basic introduction to EVPN and why it’s a better alternative to other Data Center Interconnect (DCI) technologies such as VPLS/MPLS. Here I’m going to repeat some information from Tim Gregory’s posting.

####insert EVPN stuff here, additional detailed info added later ####

Brief description of EVPN and it’s benefits

####insert EVPN stuff here, additional detailed info added later ####

VMware/vMX Lab Example

This lab example is based off of Tim Gregory’s lab example linked below.

Tim Gregory’s EVPN Basic Example

Now Tim’s article is great, but most people don’t have a bunch of Juniper MX routers laying around, so I’ve created the same configuration virtually using Juniper vMXs running on VMware ESXi. I also added some client linux VMs to log into to and test to connectivity between sites. In addtion, having the linux VMs allow for a excellent way to demonstrate data center technologies such as VMware vMotion between data centers combined with EVPN.

Historically speaking, you normally can’t stretch a VMware vSphere between two or more WAN separated data centers. With the advent of VMware vSphere 6.0 you can vMotion between layer 3 separated data centers but it requires changes to your VMware environment and some technical limitations still exist. Just using EVPN doesn’t require any changes a production VMware vSphere environment. It works for all versions of VMware vSphere since it’s hides the networking complexity from VMware.

Lab Diagram

Lab Components used

1. VMWare ESXi 6.0 Update 2
2. VMware vCenter Server Appliance 6.0.0
3. Virtual Distributed Switches
4. Six Juniper vMX routers running 16.1R1.7 code
5. vMX Demo license
6. Three Linux Desktop VMs

vMX router configurations

The link below is a zip file containing the example configurations for the vMX routers used. Also, the account information is below.

root password: lab123

admin account name: admin

admin account password: lab123

evpn-lab-evpn-basic-layer-two-extension

General Lab Configuration Notes

Looking at the lab drawing above, notice the connections between the routers. Since we are using VMware, we need to make connections between the routers using VMware port groups. In VMware we give each port group a unique VLAN number and we create one port group between each pair of devices. This guarantees each network segment between the devices are isolated and no other devices are seen.

Also, we need to remember to make sure in the “Polices” section of the port group that the “Promiscuous Mode” option is set to “Accept”. If this isn’t set, then you won’t see any traffic between the vMX routers.

Security Setting Example for Port Groups

Also, another note you need to know, is the vNICs associated between the FPC VM and the actual interfaces inside the vMX router are randomly assigned after the first three vNICs. What does this mean? Remember the first two vNICs are for management and the internal connection to VCP VM. The other possible eight vNICs can be used for connections to other devices. But you need to  match the MAC address of the vNIC to the MAC address of the vMX interface.

Here’s an example of the R3-P1 FPC VM in the two pictures below. Notice the vNICs aren’t listed in same order that they are on the vMX router. In this example, vNIC 4 is associated with ge-0/0/2 inside the vMX router. ge-0/0/2 is the third ge interface in the vMX but it’s the fourth vNIC inside the VM.

VM vNIC Example

Luckily we can track down the correct interface with a quick “show interface” command on the vMX and compare the hardware address inside the vMX to the MAC address inside VMware.

Show Interface Example

Also, as a personal preference, I normally name my port groups with the VLAN number first and then what segment it’s supposed to represent. For example, VLAN444-R1-PE1-to-R3-P1 port group uses VLAN 444 and is used between router R1-PE1 and router R3-P1.

Technical limitations with VMware vSwitch and Work Arounds

Now Tim Gregory’s lab example uses physical EX4200 switches to connect VLANs to the MX routers. Let’s keep it virtual. I’m using VMware’s virtual distributed switch as my switch. This switch is distributed between all of my ESXi hosts. It simplifies my virtualization environment and keeps the vdswitch consistent between all of my hosts. But the example lab environment needs to have the same VLAN on all three EX4200 switches. In VMware, if I create three different port groups on the same vdswitch and give them the same VLAN ID, the VMs will see each other at layer two because they on the same vdswitch. That’s a big problem in the lab example when we convert it to VMware. That’s because we want the linux VMs to only connect to their vMX PE router ge interfaces so we can truly show EVPN 2 layer connectivity and mac  address mobility.

One option I have is to create three different vdswitches but it complicates my VMware environment because I have to associate different ESXi host NICs to each vdswitch I create. Another simpler option, is to create port groups as I normally do, each with a unique VLAN ID, them use the vMX PE routers to untag the VLAN ID when the traffic comes in from the virtual LAN segment, and put the traffic in the correct VLAN inside the PE routers. This is what i’m doing in this lab example.

For example, for the port group connecting the linux hosts to the R1-PE router has a VLAN ID of 461 but within the vMX I don’t care about the VLAN tag and I drop the traffic into VLAN 458 via the bridging-domain subsection of the routing-instance.

Here’s the configuration of the port group in VMware.

 

Example Port Group in VMware

Here’s an excerpt of the R1-PE1 router’s interfaces configuration. In this example, ge-0/0/0 and ge-0/0/2 are setup as switching ports. These are the ports I connect the linux desktop VMs to the vMX.

R1-PE1 ge Interface Example

Here’s an excerpt of the R1-PE1 router’s routing instances configuration. In this example, ge-0/0/0 and ge-0/0/2 are in the bridge domain and are associated to VLAN 458. It’s almost the same setup on R2-PE2 and R5-PE3 PE routers.

R1-PE1 routing-instance Example

Important Note: Routing traffic between more than one ESXi host.

Another thing to note is for port groups to pass VLAN traffic between two or more ESXi hosts you need to make sure the VLANS also exist on the physical switch connected to the ESXi hosts. Also, the switch ports that the ESXi NICs are connected to are configured as Trunk ports and that the required VLANs are allowed on the ports.

Additional Troubleshooting and Verification Commands

Tim Gregory’s article has lots of good commands to verify that your configuration is working. Beyond what he’s providing here’s some additional things I usually use.

show bridge mac-table

I use the “show bridge mac-table” to verify all of the layer 2 mac addresses coming from both the local linux VMs and the remote  linux VMs connected to the other PE vMX routers.

In the example below, you can see one linux VM attached to ge-0/0/0.0 and another one connected to ge-0/0/2.0. You’ll also notice two other mac addresses with no logical interface listed. These are from the EVPN layer two LAN segments and are from the other two PE routers. These MAC addresses are from the other two linux desktops connected to those PE routers.

“show bridge mac-table” Example

On the the terminal of the linux desktop attached to R1-PE1, if I run the “ifconfig | grep HW” command, I can see the local MAC address of the linux machine and I can verify that it’s the same MAC address that is shown on the vMX router on port ge-0/0/0.0.

“ifconfig | grep HW” Example

I can use a combination of these two commands on all of the PE routers and the linux machines I can track, at layer 2 on the PEs, which linux machines are connected and where.

 

I this helps you out in your own labs. 🙂