From the blog.

Managing Digital Racket
The more I tune out, the less I miss it. But that has presented me with some complex choices for a nuanced approach to curb
Complexity – My Friend, My Enemy
Over my years of network engineering, I've learned that the fewer features you can implement while still achieving a business goal, the better. Why? Fewer
Enterprise WAN - Last Mile

Enterprise WAN Design & Last Mile Considerations

1,468 Words. Plan about 9 minute(s) to read this.

When building your local area network (LAN), inevitably you’ll need to connect it to a wide area network (WAN). A WAN differs from a LAN.

  1. You probably own the LAN entirely. Every switch, router, and cable is likely to be under your direct control.
  2. You probably do NOT own the WAN. You might own a WAN router such as a Cisco ISR, but from there, the WAN is owned by a service provider, aka a carrier such as Verizon, AT&T, Level 3, Comcast, or CenturyLink. Internet circuits provided by your local ISP are WAN circuits as well. The net effect of you not owning your WAN is that you lease WAN infrastructure on a contractual basis.

What is really at the other end of that WAN cable?

Let’s consider WAN circuits for a moment. What are they, really? When talking about in-building cabling, a WAN circuit is a cable that connects your equipment to the carrier’s equipment. That cable could be used to signal something you’re familiar with, like straightforward Ethernet. That cable could be used to signal in a way that you’re less familiar with, like T1 (in the US) or E1 — channelized signals that come under the general heading of time division multiplexing (TDM). The cable might be single- or multi-mode fiber optic (probably an Ethernet hand-off) or twisted-pair copper (increasingly Ethernet, but just as likely to be TDM).

That cable is handed off to you by the carrier from the demarcation point (demarc). The demarc likely appears as a jack in the wall to you. From there, where does it go? What’s behind the jack? The answer is that it depends. In general, that demarc point runs back to carrier owned equipment nearby. If your business is located in a shared office complex, the carrier might have equipment right on the campus to connect your circuit. If you’re in your own building or in a residential neighborhood, the cable might run out to a utility pole.

Eventually, your WAN circuit will land at a central office. Now, it’s at this point that we have to make a key observation. The connection between your office and the central office is what we call the last mile. The last mile, at least in the US, has one big problem: it’s often owned by a single organization. That owner is likely to be the incumbent local exchange carrier (ILEC).

Enterprise WAN - Last Mile

You might be buying a WAN circuit from, say, CenturyLink. But for CenturyLink to get that circuit to you, at some point, they have to deal with whoever owns the last mile, often the incumbent local exchange carrier (ILEC). You won’t have to deal with that in the sense that you won’t get a bill from the last mile owner. But the fact that the last mile is very possibly owned by a single entity presents a concern when designing WAN infrastructure for your business.

Considerations for last mile design.

One instinct most network engineers will have is to use diverse carriers to improve WAN resilience. The idea is that if one of your carriers has a problem, the second one will not. Add dynamic routing, and voila: business continuity. The logic is that at least one of the two circuits will be up at any given time. A common enterprise business strategy is to lease a private (expensive) WAN to connect all of their remote offices together, and use  public (cheap) Internet circuits secured with VPN as a failover.

Another strategy used by those companies able to justify the expense is to create two private WANs, each running in parallel. The comparatively cheap “Internet plus VPN” circuit is used as a tertiary connection. And yet another strategy is to use multiple private WANs — one for one group of offices, one for another, and perhaps a third or fourth for others. HQ sites connect to all the carriers. Remote sites only connect to a single carrier. This is a viable solution for businesses with hub-and-spoke data flows, i.e. remote offices that talk to HQ, but don’t need to talk directly to one another very much. This approach reduces the failure domain such that only a limited number of offices are impacted when a particular carrier is having a bad day.

All of these approaches spread their traffic across multiple carriers, expecting to be able to tolerate a single failure in the WAN. However, there is still a more fundamental single point of failure to ponder: the last mile.

  1. If all of the carriers you lease WAN infrastructure from use the ILEC to connect your business to their network, then an ILEC problem could knock out all carrier connections. This is true if the service is public Internet or private WAN.
  2. A single event could damage multiple physical lines in a shared conduit. This is the “Billy Backhoe” problem, or the “car crashed into a utility pole” problem. If the carriers you chose share a common route, one bad thing happening in meatspace can take both connections out of service.

image

For enterprise WAN designers, the solution is to do your very best to insure the following.

  1. Different carriers should enter and exit your building at different places. This is a tough request, especially if you don’t own the building or if the build costs are high. But if you can manage it, this reduces the chance that Billy Backhoe takes out both carriers. If all the conduit for communications lines enters the building in one place, it represents a point of risk.
  2. Run some cables aerial, and others underground. Again, this is another tough one to manage if you inherited the physical plant — you’ve got what you’ve got and augmenting what you’ve got might be prohibitively expensive. But especially in the case of new construction, this is a step to consider that goes hand in hand with the step above.
  3. Ask your carriers to provide you with a map illustrating how the last mile circuit gets from your building to the CO. If all carriers are taking the exact same route, ask for a diverse path. There’s a decent chance that there are multiple ways to get from your building to the CO. The idea is to have the paths be separated as much as possible to reduce the chance of something taking out both lines. Now, getting this map will take some time, but trust me — the information exists. The ILEC most certainly has it, and as a customer, you should be able to see it. At the same time, expect it to take a while before you’re seeing the diagram. I’ve never worked with an ILEC or carrier that moved all that quickly on a request like this.
  4. Look for unique cable plants in the area that offer true last mile diversity. For example, a cable operator might be able to provide a physically distinct connection from some other carrier that relies on the ILEC. In addition, metropolitan areas often have fiber network operators who might be in (or near) your location.
  5. Another option might be to use wireless technology as an alternate WAN connection. 4G/LTE router interfaces are easy to come by these days, although to get decent performance from them, you might need an outside antenna. But if you can pull it off, Billy Backhoe can’t kill it. Other types of wireless WAN providers might exist in your area, but using them is highly dependent on having line of sight to their towers. In addition, you might have to erect an antenna on the roof, something building owners don’t always grant permission to do. As a side note, if you have offices within several miles of each other, you might be able to connect them yourself via line-of-sight wireless. A wireless specialist in your area should be able to perform a feasibility study at a reasonable price.

To the future.

If you’ve conquered last mile diversity, the next challenge is managing routing across the infrastructure. For hybrid WANs, this is, frankly, a pain in the back side. The core issue is that routing protocols worry primarily about best path, but don’t have a sufficiently complex metric that reflects a true “best” at any given point in time. This is where software defined WAN solutions start to add value. Therefore, enterprises that are building out diverse WANs designed for business resilience might be interested in SD-WAN as a technology that can make better use of the links they are leasing.

That said, no amount of SD-WAN is going to remedy a last mile problem that takes out all of your WAN circuits. Implementing the best physically redundant design possible is still necessary.