Create great junos labs with Logical System and Tunnels. Works in vMX too!

Here’s a great article from 2012 on Junos logical systems and logical tunnel interfaces. MX routers support 15 logical systems per router and when you combine this with logical tunnel interfaces, you can make extremely complex lab networks with one or more MX routers. Here’s the article below along with some additional information about logical tunnel interfaces.

JUNOS Logical-System: A Great Way To Learn Junos & Prepare For Certification

Configuring Logical Tunnel Interfaces

Also, you don’t have an MX router laying around? No problem! This is fully supported on Juniper’s vMX.

Below is an example to enable tunnel services within a vMX router.

chassis {

    fpc 0 {

        pic 0 {

            tunnel-services {

                bandwidth 10g;


            interface-type ge;          

            number-of-ports 8;




    network-services enhanced-ip;


Here’s an example topology I created with logical systems and logical tunnel interfaces between the logical systems. In this example, there are two vMX routers with five logical systems each. This is like having ten routers total. Within the MXs, the logical systems are interconnected via logical tunnel interfaces which saves the ge interfaces for just interconnecting the vMXs together. That’s important due to the current limitation of 8 useable revenue interfaces on vMX routers.


Have fun making great labs with this!

EVPN – Basic Layer Two Extension Lab, Virtual Lab Example


This is a basic introduction to EVPN and why it’s a better alternative to other Data Center Interconnect (DCI) technologies such as VPLS/MPLS. Here I’m going to repeat some information from Tim Gregory’s posting.

####insert EVPN stuff here, additional detailed info added later ####

Brief description of EVPN and it’s benefits

####insert EVPN stuff here, additional detailed info added later ####

VMware/vMX Lab Example

This lab example is based off of Tim Gregory’s lab example linked below.

Tim Gregory’s EVPN Basic Example

Now Tim’s article is great, but most people don’t have a bunch of Juniper MX routers laying around, so I’ve created the same configuration virtually using Juniper vMXs running on VMware ESXi. I also added some client linux VMs to log into to and test to connectivity between sites. In addtion, having the linux VMs allow for a excellent way to demonstrate data center technologies such as VMware vMotion between data centers combined with EVPN.

Historically speaking, you normally can’t stretch a VMware vSphere between two or more WAN separated data centers. With the advent of VMware vSphere 6.0 you can vMotion between layer 3 separated data centers but it requires changes to your VMware environment and some technical limitations still exist. Just using EVPN doesn’t require any changes a production VMware vSphere environment. It works for all versions of VMware vSphere since it’s hides the networking complexity from VMware.

Lab Diagram

Lab Components used

  1. VMWare ESXi 6.0 Update 2
  2. VMware vCenter Server Appliance 6.0.0
  3. Virtual Distributed Switches
  4. Six Juniper vMX routers running 16.1R1.7 code
  5. vMX Demo license
  6. Three Linux Desktop VMs

vMX router configurations

The link below is a zip file containing the example configurations for the vMX routers used. Also, the account information is below.

root password: lab123

admin account name: admin

admin account password: lab123


General Lab Configuration Notes

Looking at the lab drawing above, notice the connections between the routers. Since we are using VMware, we need to make connections between the routers using VMware port groups. In VMware we give each port group a unique VLAN number and we create one port group between each pair of devices. This guarantees each network segment between the devices are isolated and no other devices are seen.

Also, we need to remember to make sure in the “Polices” section of the port group that the “Promiscuous Mode” option is set to “Accept”. If this isn’t set, then you won’t see any traffic between the vMX routers.


Security Setting Example for Port Groups

Also, another note you need to know, is the vNICs associated between the FPC VM and the actual interfaces inside the vMX router are randomly assigned after the first three vNICs. What does this mean? Remember the first two vNICs are for management and the internal connection to VCP VM. The other possible eight vNICs can be used for connections to other devices. But you need to  match the MAC address of the vNIC to the MAC address of the vMX interface.

Here’s an example of the R3-P1 FPC VM in the two pictures below. Notice the vNICs aren’t listed in same order that they are on the vMX router. In this example, vNIC 4 is associated with ge-0/0/2 inside the vMX router. ge-0/0/2 is the third ge interface in the vMX but it’s the fourth vNIC inside the VM.


VM vNIC Example

Luckily we can track down the correct interface with a quick “show interface” command on the vMX and compare the hardware address inside the vMX to the MAC address inside VMware.


Show Interface Example

Also, as a personal preference, I normally name my port groups with the VLAN number first and then what segment it’s supposed to represent. For example, VLAN444-R1-PE1-to-R3-P1 port group uses VLAN 444 and is used between router R1-PE1 and router R3-P1.

Technical limitations with VMware vSwitch and Work Arounds

Now Tim Gregory’s lab example uses physical EX4200 switches to connect VLANs to the MX routers. Let’s keep it virtual. I’m using VMware’s virtual distributed switch as my switch. This switch is distributed between all of my ESXi hosts. It simplifies my virtualization environment and keeps the vdswitch consistent between all of my hosts. But the example lab environment needs to have the same VLAN on all three EX4200 switches. In VMware, if I create three different port groups on the same vdswitch and give them the same VLAN ID, the VMs will see each other at layer two because they on the same vdswitch. That’s a big problem in the lab example when we convert it to VMware. That’s because we want the linux VMs to only connect to their vMX PE router ge interfaces so we can truly show EVPN 2 layer connectivity and mac  address mobility.

One option I have is to create three different vdswitches but it complicates my VMware environment because I have to associate different ESXi host NICs to each vdswitch I create. Another simpler option, is to create port groups as I normally do, each with a unique VLAN ID, them use the vMX PE routers to untag the VLAN ID when the traffic comes in from the virtual LAN segment, and put the traffic in the correct VLAN inside the PE routers. This is what i’m doing in this lab example.

For example, for the port group connecting the linux hosts to the R1-PE router has a VLAN ID of 461 but within the vMX I don’t care about the VLAN tag and I drop the traffic into VLAN 458 via the bridging-domain subsection of the routing-instance.

Here’s the configuration of the port group in VMware.


Example Port Group in VMware

Here’s an excerpt of the R1-PE1 router’s interfaces configuration. In this example, ge-0/0/0 and ge-0/0/2 are setup as switching ports. These are the ports I connect the linux desktop VMs to the vMX.


R1-PE1 ge Interface Example

Here’s an excerpt of the R1-PE1 router’s routing instances configuration. In this example, ge-0/0/0 and ge-0/0/2 are in the bridge domain and are associated to VLAN 458. It’s almost the same setup on R2-PE2 and R5-PE3 PE routers.

evpn-r1-pe1-routing-instance-configR1-PE1 routing-instance Example

Important Note: Routing traffic between more than one ESXi host.

Another thing to note is for port groups to pass VLAN traffic between two or more ESXi hosts you need to make sure the VLANS also exist on the physical switch connected to the ESXi hosts. Also, the switch ports that the ESXi NICs are connected to are configured as Trunk ports and that the required VLANs are allowed on the ports.

Additional Troubleshooting and Verification Commands

Tim Gregory’s article has lots of good commands to verify that your configuration is working. Beyond what he’s providing here’s some additional things I usually use.

show bridge mac-table

I use the “show bridge mac-table” to verify all of the layer 2 mac addresses coming from both the local linux VMs and the remote  linux VMs connected to the other PE vMX routers.

In the example below, you can see one linux VM attached to ge-0/0/0.0 and another one connected to ge-0/0/2.0. You’ll also notice two other mac addresses with no logical interface listed. These are from the EVPN layer two LAN segments and are from the other two PE routers. These MAC addresses are from the other two linux desktops connected to those PE routers.


“show bridge mac-table” Example

On the the terminal of the linux desktop attached to R1-PE1, if I run the “ifconfig | grep HW” command, I can see the local MAC address of the linux machine and I can verify that it’s the same MAC address that is shown on the vMX router on port ge-0/0/0.0.


“ifconfig | grep HW” Example

I can use a combination of these two commands on all of the PE routers and the linux machines I can track, at layer 2 on the PEs, which linux machines are connected and where.


I this helps you out in your own labs. 🙂

See Bandwidth License Useage in Juniper vMX


Juniper vMX features are managed via a software license. There are three levels of licensing along with the overall bandwidth license. Also, licenses are additive; if you add more than one license to a vMX router then it will add all of the licenses together. This is a great way to manage to growth as you continue to expand the usage of your vMX router.

The following shows you how to verify the software licenses installed and the total bandwidth of the vMX router.


Determine the bandwidth license usage in Juniper vMX and configured bandwidth in the packet forwarding engine (PFE).

Components used

  1. VMWare ESXi 6.0 Update 2
  2. Juniper vMX 16.1R7

These instructions also work with vMX deployed on KVM or other virtualization platforms. The commands are only dependant on junos.

Verify Commands

The “show system license” command displays the licenses installed and their current usage. In the example below the system has two license for a total of 100 mb of bandwidth.

 show system license   
show system license example output:

Use the “show pfe statistics traffic bandwidth” to show the actual bandwidth being used. This shows the configured bandwidth based off of the licenses along with the current and average bandwidth being used.

show pfe statistics traffic bandwidth
show pfe statistics traffic bandwidth output:


If you combine the same command with the pipe command , the “refresh” command and a value in seconds,  then you can set up a nice little bandwidth monitor. This is very handy if you are troubleshooting and you think you’re out of bandwidth, licensing has kicked in, and you’re seeing some impact to traffic outside normal troubleshooting.

show pfe statistics traffic bandwidth | refresh 1
show pfe statistics traffic bandwidth | refresh 1 output:


I hope these commands help you out!

VMware vSphere 6, Update 2 Home Lab

I’ve been running various VMware ESXi home configurations for many years. Until recently, I’ve been using Apple Mac Minis and kept hoping Apple would get around to updating them to run additional memory in their next release. Alas, no such luck. I used VirtuallyGhetto’s guide back in the day to set them up. This is a great setup. It’s painless to configure, quiet, and has low power requirements. Here’s that guide linked below.

VirtuallyGhetto’s Mac Mini ESXi guide

Unfortunately, these days, I need something with a lot more memory and processing power. But I still want something small, easy to setup, quiet, and easy on the power bill.

After some research, I decided to use Shuttle PC’s SH170R6. It supports 64 GB of RAM, the latest Intel Skylake i7 four core 6700k processor,  a built in Intel i219LM network adapter, and an integrated video port. The power supply is only 300 watts and the CPU is a 95 watt processor. The old mac minis use 85 watts each. So my power will be more than my old mac minis but the processing power is much more and has four times as much RAM so I think it’s an excellent tradeoff. The setup is fully recognized without any additional software drivers with VMWare ESXi 6, update 2. Unfortunately, because of the newer Skylake chipset, you can’t run older versions of VMWare on this platform.

Shuttle SH170R6 on Newegg

Also, after doing some additional research, I discovered it’s got an M.2 port so I decided to use it to load ESXi onto a small M.2 drive it as opposed to booting ESXi off of an external usb drive.

The labs I’m doing  these days are focused on networking so I added an additional four port Intel I350-T4 PCI-E network adapter. This isn’t the cheapest card available but it’s recognized natively without drivers and supports IO-SRV. I’m building a lot network function virtualization labs and want to experiment with the support of IO-SRV in VMware vSphere. These ports will be trunked to my three Juniper EX2200-Cs that are configured into a single virtual chassis. I’m using the built in Intel adapter as the VMware management interface.

Intel I350-4T on Newegg

Here’s some photos of the external layout and form factor of the Shuttle SH170R6.




Here’s what it looks like inside without the drive cage installed. It’s got plenty of room to install the components. A nice touch is most of the cables are already plugged in from the factory. It makes for a very clean setup.


Another cool feature is the Integrated Cooling Engine 2 (ICE 2) Heat pipe technology by Shuttle. It comes with the case and does an excellent job of keeping the CPU cool.


Here’s what it looks like outside of the case. It’s a very simple design.


So for storage, I’m using an ADATA M.2 128 GB card and 960 GB SSD. The M.2 card is for ESXi and the SSD is for VM storage. The ADATA is much bigger than what’s needed for the ESXi OS but it was only 50 bucks on newegg.

Also, to mount the SSD in the case I picked up this Sabrent adapter. It’s only 1o bucks and it allows you mount two 2.5 inch SSDs in an 3.5 drive bay. I might expand the internal storage in the future. It’s nice to have options.

Sabrent 3.5-Inch to x2 SSD / 2.5-Inch Internal Hard Drive Mounting Kit


Here’s the SSD drive mounted in the Sabrent adapter.


Here’s the M.2 drive mounted in the case. It sits flat in the case so if you have a longer PCI-E card it’ll have good clearance and not interfere.


I got G.Skill Ripjaws RAM for this system.

G.SKILL Ripjaws V Series 64GB (4 x 16GB) 288-Pin DDR4 SDRAM on Newegg


The Shuttle comes with thermal paste so it makes it a snap to get the cpu mounted with the heat pipe heat sink.


Here’s what it looks like with the CPU and heat pipe heat sink mounted. Nothing to see here!


Here’s all of the components installed except drive cage.


Here’s the layout once the drive cage is installed.


Here’s what two Shuttles look like installed in my lab. In this setup, I’ve got 128 GB of RAM and 8 i7 cores to play with. You can also see my old mac minis in the center.


I’m really happy with this setup. It’s dead quiet, even under full CPU load. These days, my two QNAP network storage arrays are the loudest things in my rack setup. The three Juniper EX2200-C POE switches are fanless so they are silent as well.

The full lab along with the Juniper switches allow for excellent, close to real world, virtualization and networking scenarios. The Juniper EX2200s support full Junos OS features such as VLAN trunking, virtual chassis, LAG, VLANs, private VLANs, PoE, QoS, and layer 3 routing protocols.