Key takeaways

  • A VLAN separates Layer 2 broadcast domains. A VRF separates Layer 3 routing tables. A global design needs both, because a VLAN alone does not stop routes from leaking and a VRF alone does not stop broadcasts.
  • The 802.1Q tag has a 12-bit VLAN ID, so a single VLAN space is small. VXLAN (RFC 7348) carries a 24-bit VNI, which allows up to 16 million segments in one administrative domain.
  • VXLAN is only the data plane. Without a control plane, unknown destinations are flooded. EVPN (RFC 7432, with RFC 8365 for overlays) distributes MAC and IP reachability in BGP instead, so flooding is the exception.
  • EVPN route type 5 (RFC 9136) advertises IP prefixes into a tenant VRF, which is how inter-subnet routing and datacenter interconnect are built without stretching more Layer 2 than needed.
  • Stretching a VLAN between datacenters is a business decision as much as a technical one. Every stretched segment shares fate, broadcast noise and failure domains. Route where you can, bridge only where an application needs it.
  • The underlay must be sized for the overlay: VXLAN adds headers, and VTEPs must not fragment (RFC 7348), so raise the physical MTU everywhere on the path.

Why separate networks at all

A company that operates in many countries usually has several reasons to keep traffic apart. Regulators expect payment systems, patient records or trading systems to be isolated from office networks. Business units are acquired and keep their own address plans. Customers of a hosting or coworking operation must never see each other. A security team wants a breach in one place to stay in that place.

The usual first answer is a firewall between zones. That works, but it does not scale to hundreds of segments, and it does nothing about the physical question: how do you give each segment the same reach across many sites without building a separate set of cables for each one? The answer is virtualization at Layer 2 and Layer 3 on shared hardware. This article walks through the standards that make that possible. It is written as an engineering overview; the exact design for any organization depends on its sites, carriers and applications.

Layer 2 first: 802.1Q VLANs and QinQ

A VLAN splits one physical switch network into several broadcast domains. IEEE 802.1Q does this by inserting a four-byte tag after the source MAC address of an Ethernet frame. The tag's EtherType is 0x8100 and it carries a 12‑bit VLAN ID, which gives 4,096 values. Two of them, 0 and 4095, are reserved, so a single switched domain has 4,094 usable VLANs. That is plenty for one building. It is not plenty for a global provider with many tenants, each with their own VLAN numbering.

IEEE 802.1ad, commonly called QinQ, addresses that by stacking a second tag. The outer tag is the service tag (S-tag, EtherType 0x88A8) and the inner tag is the customer tag (C-tag, 0x8100). A carrier assigns one outer tag per customer, and the customer's own VLAN IDs travel untouched inside it. This is how a metro Ethernet circuit can carry a customer's VLANs between two offices without the carrier knowing or caring what they are. Some older gear used 0x9100 for the outer tag, so check what both ends expect before you order the circuit.

Plain VLANs and QinQ share a limit: they are bridged. Broadcast, unknown-unicast and multicast traffic (BUM) reaches every port in the domain, and the spanning tree protocols that keep bridged networks loop-free get harder to reason about as the domain grows. Wikipedia's summary of 802.1ad makes the same point: QinQ alone is not a complete end-to-end solution, because MAC learning scale and spanning tree still need other mechanisms. That is the reason large networks move the boundary of Layer 2 down to the rack and route everything above it.

Layer 3 separation: VRFs

A virtual routing and forwarding instance (VRF) gives a router several independent routing tables. RFC 4364 describes the model for provider-edge routers: each attachment circuit is tied by configuration to a VRF, and the destination lookup for packets from that circuit uses only that VRF's table. Two VRFs can hold the same 10.0.0.0/8 address plan without conflict, because they never see each other's routes.

Two BGP concepts from RFC 4364 are worth knowing because they come back in EVPN. A route distinguisher (RD) is an 8‑byte value prepended to a prefix so that identical prefixes from different VPNs stay distinct in BGP. It does not say who owns the route. A route target (RT) is an extended community that decides which VRFs import a route: a route enters a VRF only if one of its RTs matches one of the VRF's import targets. In plain terms, the RD keeps routes from colliding and the RT decides who may see them.

A VRF without a Layer 2 design is incomplete. Hosts in one subnet still need a switch, and a switch with one VLAN per site still needs a way to reach the next site. That is where overlays come in.

VXLAN: Layer 2 over Layer 3

RFC 7348 defines the Virtual eXtensible LAN. It wraps an Ethernet frame in a VXLAN header, an outer UDP header, an outer IP header and an outer Ethernet header, and carries the result across any routed network. The tunnel endpoints are called VTEPs. A VTEP can sit in a hypervisor, a server or a physical switch, and the virtual machine or server behind it never sees the encapsulation.

The VXLAN header carries a 24‑bit VXLAN Network Identifier (VNI). The RFC says this allows up to 16 million segments to coexist within the same administrative domain, against 4,094 usable VLANs. The UDP destination port is 4789, assigned by IANA, and the source port is derived from the inner frame so that ordinary equal-cost multipath hashing can spread flows over many links. That last detail is why VXLAN fits a spine-and-leaf fabric well: the fabric sees ordinary IP and UDP.

Two practical points follow from the RFC. First, the encapsulation adds the sizes of the four outer headers (roughly 50 bytes for IPv4 with no outer VLAN tag, by adding 14 + 20 + 8 + 8). The RFC says VTEPs must not fragment VXLAN packets and recommends that physical network MTUs be set high enough to accommodate the larger frame. In practice that means jumbo frames on the underlay, set consistently on every device in the path, including the carrier circuit. Second, the base RFC floods BUM traffic, using IP multicast groups mapped to VNIs. Many operators do not want to run multicast across a global backbone, and flooding scales poorly. A control plane fixes both.

EVPN: a control plane for the overlay

Ethernet VPN (EVPN) uses BGP to carry Layer 2 and Layer 3 reachability. RFC 7432 defines it as a BGP address family (AFI 25, SAFI 70) with four route types in the original document. Type 1 is the Ethernet auto-discovery route, used for fast convergence and aliasing. Type 2 advertises a MAC address, optionally with an IP address. Type 3 is the inclusive multicast route that sets up BUM handling. Type 4 is the Ethernet segment route, used to elect a designated forwarder when a host is attached to more than one switch.

The effect is that a VTEP learns where a MAC lives from BGP instead of from a flood. RFC 7432 also says a PE that knows the MAC-to-IP binding should perform ARP proxy, which reduces ARP and neighbor-discovery flooding across the core. Multihoming works at the edge too: a server can connect to two leaf switches, and EVPN lets both forward for it (all-active) while the designated-forwarder election keeps a single copy of each broadcast frame. The Ethernet segment identifier is a 10‑octet value that names the shared attachment.

RFC 8365 then describes how EVPN works as a network virtualization overlay with VXLAN, NVGRE or MPLS over GRE. For VXLAN, the 24‑bit VNI is carried in the label field of the MAC/IP, auto-discovery and multicast routes, and usually maps one-to-one to a tenant VLAN. The RFC also covers service interfaces. With one broadcast domain per EVPN instance, each VNI has its own RD and RT, and the RT can be derived from the VNI. With many broadcast domains per instance, a tenant-level setup needs no per-VNI provisioning but imports routes some switches do not need. For multihoming without MPLS labels, VXLAN uses local bias: the receiving switch filters flooded frames by the tunnel source address of the ingress switch.

Routing between segments: IRB and route type 5

Stretching Layer 2 is half the story. Most traffic needs to cross subnets, and that is routing. In an EVPN fabric the leaf switch usually provides an integrated routing and bridging (IRB) interface, so the default gateway is local on every leaf. Each tenant gets an IP-VRF, and the VNIs belonging to that tenant are bridged inside it and routed across it.

RFC 9136 adds route type 5, the IP prefix route. Instead of advertising every host as a MAC/IP route, a switch or gateway can advertise whole prefixes into the IP-VRF. The RFC describes this for tenant systems such as firewalls, load balancers and NAT devices that do not run routing protocols themselves, and it describes an interface-less model where VRFs connect directly across the overlay with no extra broadcast domain between them. In a multi-site design, type 5 routes are how a datacenter exports 'these prefixes live here' without exporting every MAC address.

Datacenter interconnect: stretch with care

RFC 8365 describes two ways to connect EVPN domains. With gateways, a border device terminates the EVPN routes from one datacenter, translates the VNIs if the domains use different numbers, and re-originates them into the next. Leaf switches then keep state only for their own site, and failures in one site are less likely to disturb another. With ASBR-style interconnect, border routers relay routes like inter-AS option B and keep no MAC forwarding state. Locally significant VNIs make the second model possible without a gateway. Which one fits depends on the transport: dark fiber or wavelength services, a carrier's MPLS network, or an encrypted tunnel across the public internet.

The more important decision is how much to stretch. A stretched VLAN has one broadcast domain across two buildings. A misbehaving device, a loop or a storm in one place affects both. Latency also matters: an application that expects a local database will behave differently when the next hop is 80 milliseconds away. Good practice is to treat Layer 2 stretch as an exception with a named owner and an expiry date, for example during a migration, and to give most applications routed connectivity with a stable address provided by anycast gateways or load balancers. EVPN makes it easy to stretch a segment. That is a reason to be disciplined, not a reason to do it everywhere.

Running it: operations that decide success

  • Underlay first. Use a simple routed fabric (BGP or an IGP), equal-cost paths, and a consistent MTU. Verify the largest packet end to end before enabling any overlay.
  • One naming convention. Map VLAN, VNI, VRF and tenant in a single table, so an engineer can tell at a glance that VLAN 210 in Sao Paulo, VNI 10210 in the fabric and the 'payments' VRF are the same segment.
  • Route-target discipline. Use import and export targets deliberately. Leaking routes between VRFs should be a reviewed, logged exception, usually through a firewall.
  • Observability. Collect BGP EVPN route counts, VTEP reachability, MAC mobility events and interface errors. A sudden surge of MAC moves is a loop warning.
  • Security between segments. Separation is not the same as protection. Put a stateful firewall or policy engine at the points where VRFs meet, and record the rules.
  • Failure drills. Pull a link, a leaf and a border device in a maintenance window. Measure convergence rather than trusting a datasheet.

What this looks like for an organization with several regions

A typical outline is as follows. Each region has a pair of datacenters with a spine-and-leaf fabric, running EVPN with VXLAN. Each business unit or compliance zone is a tenant with its own VRF, its own VNIs and its own policy. Regions are linked by routed backbones with the same VRF names, and most inter-region traffic is routed using type 5 prefixes. A very small number of segments are bridged across regions for specific applications. Carrier circuits that must carry customer VLANs use QinQ at the handoff, or a routed handoff with encryption on top. Our team designs and operates fabrics of this kind, and we start every project with a segment inventory, because the inventory tells you which boundaries really need to exist.

None of this requires a particular vendor. VXLAN and EVPN are IETF standards, implemented by many switch and router platforms and by open-source network operating systems, so the design can be written against the RFCs and checked against whichever hardware you buy. Related reading: MPLS, SD-WAN or EVPN: Choosing a Backbone for a Multi-Site Company.

Sources

This article is general technical information, not a design for any specific network. Vendors implement these standards differently, so confirm behavior, limits and licensing on your own equipment. The 4,094 usable VLAN IDs and the approximate 50‑byte VXLAN overhead are derived from the IEEE and RFC header definitions, not quoted from the sources listed. Last reviewed .

Keep reading

Need help with this?

Cloud cost review and management, without compromising on quality or security.

Cloud Management