Create great junos labs with Logical System and Tunnels. Works in vMX too!

Here’s a great article from 2012 on Junos logical systems and logical tunnel interfaces. MX routers support 15 logical systems per router and when you combine this with logical tunnel interfaces, you can make extremely complex lab networks with one or more MX routers. Here’s the article below along with some additional information about logical tunnel interfaces.

JUNOS Logical-System: A Great Way To Learn Junos & Prepare For Certification

Configuring Logical Tunnel Interfaces

Also, you don’t have an MX router laying around? No problem! This is fully supported on Juniper’s vMX.

Below is an example to enable tunnel services within a vMX router.

chassis {

    fpc 0 {

        pic 0 {

            tunnel-services {

                bandwidth 10g;


            interface-type ge;          

            number-of-ports 8;




    network-services enhanced-ip;


Here’s an example topology I created with logical systems and logical tunnel interfaces between the logical systems. In this example, there are two vMX routers with five logical systems each. This is like having ten routers total. Within the MXs, the logical systems are interconnected via logical tunnel interfaces which saves the ge interfaces for just interconnecting the vMXs together. That’s important due to the current limitation of 8 useable revenue interfaces on vMX routers.


Have fun making great labs with this!

EVPN – Basic Layer Two Extension Lab, Virtual Lab Example


This is a basic introduction to EVPN and why it’s a better alternative to other Data Center Interconnect (DCI) technologies such as VPLS/MPLS. Here I’m going to repeat some information from Tim Gregory’s posting.

####insert EVPN stuff here, additional detailed info added later ####

Brief description of EVPN and it’s benefits

####insert EVPN stuff here, additional detailed info added later ####

VMware/vMX Lab Example

This lab example is based off of Tim Gregory’s lab example linked below.

Tim Gregory’s EVPN Basic Example

Now Tim’s article is great, but most people don’t have a bunch of Juniper MX routers laying around, so I’ve created the same configuration virtually using Juniper vMXs running on VMware ESXi. I also added some client linux VMs to log into to and test to connectivity between sites. In addtion, having the linux VMs allow for a excellent way to demonstrate data center technologies such as VMware vMotion between data centers combined with EVPN.

Historically speaking, you normally can’t stretch a VMware vSphere between two or more WAN separated data centers. With the advent of VMware vSphere 6.0 you can vMotion between layer 3 separated data centers but it requires changes to your VMware environment and some technical limitations still exist. Just using EVPN doesn’t require any changes a production VMware vSphere environment. It works for all versions of VMware vSphere since it’s hides the networking complexity from VMware.

Lab Diagram

Lab Components used

  1. VMWare ESXi 6.0 Update 2
  2. VMware vCenter Server Appliance 6.0.0
  3. Virtual Distributed Switches
  4. Six Juniper vMX routers running 16.1R1.7 code
  5. vMX Demo license
  6. Three Linux Desktop VMs

vMX router configurations

The link below is a zip file containing the example configurations for the vMX routers used. Also, the account information is below.

root password: lab123

admin account name: admin

admin account password: lab123


General Lab Configuration Notes

Looking at the lab drawing above, notice the connections between the routers. Since we are using VMware, we need to make connections between the routers using VMware port groups. In VMware we give each port group a unique VLAN number and we create one port group between each pair of devices. This guarantees each network segment between the devices are isolated and no other devices are seen.

Also, we need to remember to make sure in the “Polices” section of the port group that the “Promiscuous Mode” option is set to “Accept”. If this isn’t set, then you won’t see any traffic between the vMX routers.


Security Setting Example for Port Groups

Also, another note you need to know, is the vNICs associated between the FPC VM and the actual interfaces inside the vMX router are randomly assigned after the first three vNICs. What does this mean? Remember the first two vNICs are for management and the internal connection to VCP VM. The other possible eight vNICs can be used for connections to other devices. But you need to  match the MAC address of the vNIC to the MAC address of the vMX interface.

Here’s an example of the R3-P1 FPC VM in the two pictures below. Notice the vNICs aren’t listed in same order that they are on the vMX router. In this example, vNIC 4 is associated with ge-0/0/2 inside the vMX router. ge-0/0/2 is the third ge interface in the vMX but it’s the fourth vNIC inside the VM.


VM vNIC Example

Luckily we can track down the correct interface with a quick “show interface” command on the vMX and compare the hardware address inside the vMX to the MAC address inside VMware.


Show Interface Example

Also, as a personal preference, I normally name my port groups with the VLAN number first and then what segment it’s supposed to represent. For example, VLAN444-R1-PE1-to-R3-P1 port group uses VLAN 444 and is used between router R1-PE1 and router R3-P1.

Technical limitations with VMware vSwitch and Work Arounds

Now Tim Gregory’s lab example uses physical EX4200 switches to connect VLANs to the MX routers. Let’s keep it virtual. I’m using VMware’s virtual distributed switch as my switch. This switch is distributed between all of my ESXi hosts. It simplifies my virtualization environment and keeps the vdswitch consistent between all of my hosts. But the example lab environment needs to have the same VLAN on all three EX4200 switches. In VMware, if I create three different port groups on the same vdswitch and give them the same VLAN ID, the VMs will see each other at layer two because they on the same vdswitch. That’s a big problem in the lab example when we convert it to VMware. That’s because we want the linux VMs to only connect to their vMX PE router ge interfaces so we can truly show EVPN 2 layer connectivity and mac  address mobility.

One option I have is to create three different vdswitches but it complicates my VMware environment because I have to associate different ESXi host NICs to each vdswitch I create. Another simpler option, is to create port groups as I normally do, each with a unique VLAN ID, them use the vMX PE routers to untag the VLAN ID when the traffic comes in from the virtual LAN segment, and put the traffic in the correct VLAN inside the PE routers. This is what i’m doing in this lab example.

For example, for the port group connecting the linux hosts to the R1-PE router has a VLAN ID of 461 but within the vMX I don’t care about the VLAN tag and I drop the traffic into VLAN 458 via the bridging-domain subsection of the routing-instance.

Here’s the configuration of the port group in VMware.


Example Port Group in VMware

Here’s an excerpt of the R1-PE1 router’s interfaces configuration. In this example, ge-0/0/0 and ge-0/0/2 are setup as switching ports. These are the ports I connect the linux desktop VMs to the vMX.


R1-PE1 ge Interface Example

Here’s an excerpt of the R1-PE1 router’s routing instances configuration. In this example, ge-0/0/0 and ge-0/0/2 are in the bridge domain and are associated to VLAN 458. It’s almost the same setup on R2-PE2 and R5-PE3 PE routers.

evpn-r1-pe1-routing-instance-configR1-PE1 routing-instance Example

Important Note: Routing traffic between more than one ESXi host.

Another thing to note is for port groups to pass VLAN traffic between two or more ESXi hosts you need to make sure the VLANS also exist on the physical switch connected to the ESXi hosts. Also, the switch ports that the ESXi NICs are connected to are configured as Trunk ports and that the required VLANs are allowed on the ports.

Additional Troubleshooting and Verification Commands

Tim Gregory’s article has lots of good commands to verify that your configuration is working. Beyond what he’s providing here’s some additional things I usually use.

show bridge mac-table

I use the “show bridge mac-table” to verify all of the layer 2 mac addresses coming from both the local linux VMs and the remote  linux VMs connected to the other PE vMX routers.

In the example below, you can see one linux VM attached to ge-0/0/0.0 and another one connected to ge-0/0/2.0. You’ll also notice two other mac addresses with no logical interface listed. These are from the EVPN layer two LAN segments and are from the other two PE routers. These MAC addresses are from the other two linux desktops connected to those PE routers.


“show bridge mac-table” Example

On the the terminal of the linux desktop attached to R1-PE1, if I run the “ifconfig | grep HW” command, I can see the local MAC address of the linux machine and I can verify that it’s the same MAC address that is shown on the vMX router on port ge-0/0/0.0.


“ifconfig | grep HW” Example

I can use a combination of these two commands on all of the PE routers and the linux machines I can track, at layer 2 on the PEs, which linux machines are connected and where.


I this helps you out in your own labs. 🙂

See Bandwidth License Useage in Juniper vMX


Juniper vMX features are managed via a software license. There are three levels of licensing along with the overall bandwidth license. Also, licenses are additive; if you add more than one license to a vMX router then it will add all of the licenses together. This is a great way to manage to growth as you continue to expand the usage of your vMX router.

The following shows you how to verify the software licenses installed and the total bandwidth of the vMX router.


Determine the bandwidth license usage in Juniper vMX and configured bandwidth in the packet forwarding engine (PFE).

Components used

  1. VMWare ESXi 6.0 Update 2
  2. Juniper vMX 16.1R7

These instructions also work with vMX deployed on KVM or other virtualization platforms. The commands are only dependant on junos.

Verify Commands

The “show system license” command displays the licenses installed and their current usage. In the example below the system has two license for a total of 100 mb of bandwidth.

 show system license   
show system license example output:

Use the “show pfe statistics traffic bandwidth” to show the actual bandwidth being used. This shows the configured bandwidth based off of the licenses along with the current and average bandwidth being used.

show pfe statistics traffic bandwidth
show pfe statistics traffic bandwidth output:


If you combine the same command with the pipe command , the “refresh” command and a value in seconds,  then you can set up a nice little bandwidth monitor. This is very handy if you are troubleshooting and you think you’re out of bandwidth, licensing has kicked in, and you’re seeing some impact to traffic outside normal troubleshooting.

show pfe statistics traffic bandwidth | refresh 1
show pfe statistics traffic bandwidth | refresh 1 output:


I hope these commands help you out!

Setting up Juniper vMX on vSphere vDS

Juniper has a great “Getting Started” guide on setting up vMX on VMware vSphere. In the “Installing vMX with OVA files” section of the guide, it only uses the standard vswitch in the setup.

Installing vMX on VMWare Guide

So how do we setup vMX on a Virtual Distributed switch? Unfortunately, the “Getting Started” guide won’t help you. So I’ve created this guide to help you out. This guide assumes you’ve already setup a VDS already in your VMware environment. Below are two links that go over the basics of setting up the VMware vSphere Distributed Switch if you haven’t done that yet.

Part 1: Getting Starting with the VMware vSphere Distributed Switch

Part 2: Getting Starting with the VMware vSphere Distributed Switch

This guide was created using VMware vSphere 6.0 Update 2 via the web console. While the official documentation for vMX supports VMware 5.5, 6.0 works fine for lab environments.

So rather than repeat the entire “Getting Started” guide, I’m just going to go over the differences starting from the “Setting Up the Network” section of the guide.

Once you complete this section you can continue on the “Getting Starting” guide at the “Deploying the VCP VM” section.

  1. After logging into the web interface, click on the Networking section.


2. Click on the root of the virtual distributed switch, then the “Getting Started” tab, and then the “Create a new port group” section to start creating a port group.

VM port groups are a way that we can create logical rules around the virtual ports that are made available to VMs. It’s common to create a port group for each VLAN and network subnet that you want to present to your VMs. Port groups also can be used to create 802.1Q trunks.


As shown in the drawing below we are first creating the management port group. This is just a personal preference, but I normally name the port groups after the VLANS that they are part of. In the case of the management port group, you can just make one and associate all of your VMX’s first interface to the same VLAN. You don’t need to make a separate management port group for each router.


3. As stated earlier, give the management port group a descriptive name. In this example shown below, I’m using “VLAN350-Juniper-Mgmt”. Click next.


4. Change the VLAN type to “VLAN”, give it a VLAN ID, and check the “Customize default policies configuration”. In this example, I gave it the VLAN ID of 350. Click next.


5. This step is critical.  In the security settings, “Promiscuous Mode”, “MAC Address Changes”, and “Forged transmits” need to be set to “Accept”. If you don’t do this you won’t be able to get into the management interface of the vMXs that you create later.


Next step we need to create the internal bridge between the VFP virtual machine and the VCP virtual machine, as shown in the illustration below.

This is just like the internal connection of a real hardware MX router that separates the control plane from the data forwarding plane. In this case, the Control Plane is the VCP virtual machine and the VFP virtual machine is the data forwarding plane. The VCP only has two connections. One is the management interface we created earlier and the other is the next interface we are going to create. This  interface has to be unique for each pair of VCP/VFP virtual machines.

The VFP virtual machine can have up to ten interfaces. The last eight of them will be revenue ports for routing and bridging traffic but the first two interfaces are the management port and the internal port to connect to the VCP virtual machine.

FYI, the ten interface limit is a limitation of VMWare. Linux KVM can scale to 23 port per vMX.


6. We start the process the same way we created the other port group. In this case, I’ve given it the name of “VLAN351-R1-br-int”.


7. Change the VLAN type to “VLAN”, give it a VLAN ID, and check the “Customize default policies configuration”. In this example, I gave it the VLAN ID of 351. Click next.


8. Just as in the previous example, this step is critical.  In the security settings, “Promiscuous Mode”, “MAC Address Changes”, and “Forged transmits” need to be set to “Accept”. If you don’t do this VCP and the VFP virtual machines won’t be able to communicate with each other and you’ll have no interfaces because the VFP won’t register to the VCP via the internal bridge we are building with this port group.


Final notes, before you continue on to the “Getting Starting” guide at the “Deploying the VCP VM” section.

  • The Management port group can be the same for all of the vMXs. But the internal port group MUST be unique for each pair of VCP/VFP virtual machines. If two vMXs have the same port group as their internal bridge, neither will come up properly.
  • When you create the other eight ports on the VFP virtual machine, you can use the same process as described above. “Promiscuous Mode”, “MAC Address Changes”, and “Forged transmits” need to be set to “Accept”.
  • Also, if you want to play with 802.1Q trunking you can setup a port group as  a trunk instead of a VLAN. Then you can play around with trunking vMXs together like you would traditional switches. There’s a lot of flexibility.
  • This is just a personal preference, but I also create a port group and name it “disabled” and go into the “Miscellaneous” section of the port group and set “Block all ports” to “Yes” as shown below. Then any port that I’m not actively using on the VFP virtual machine I normally put in this port group.


I hope this helps you. Please feel free to leave me comments, questions, or corrections.