A 100G colocation built for microseconds: PTP time, kernel-bypass and no surprises
A systematic hedge fund in New York and New Jersey, a leader in its segment.
- Scale
- 2 colocation sites, 100 Gb/s links, sub-100 ns clock accuracy, microsecond-class software path
- Engagement
- Design, build and operation of two colocation footprints
The challenge
Trading strategies lived and died on latency and on knowing exactly when each packet arrived. The firm’s existing setup used general-purpose servers and best-effort time, so timestamps could not be trusted and delays were uneven.
For a systematic trading firm, the network is part of the strategy. The firm's existing setup used general-purpose servers and best-effort time, so timestamps could not be trusted and delays were uneven. The brief had three parts: make the path from the exchange handoff to the application as short and as consistent as possible, make time trustworthy across every machine, and keep the evidence to prove what happened. Consistency matters as much as speed, because the tail of the latency distribution is where money is lost.
The physical design is deliberately simple: fixed-length cross-connects, as few hops as possible, and cut-through 100 Gb/s QSFP28 switches that begin forwarding before a frame has fully arrived. On the servers, the kernel network stack is bypassed with user-space packet processing (DPDK or vendor user-space libraries), interrupts are pinned, cores are isolated for the trading process and memory is kept local to the network card's NUMA node. Time comes from a GNSS-disciplined grandmaster clock distributed over IEEE 1588 PTP with hardware timestamps on the network cards and boundary clocks in the switches, which is what makes timestamps comparable between machines.
The migration ran beside production. The new footprint was built and measured in parallel while the old one kept trading, and strategies moved one at a time after a week of shadow running in which both paths received the same market data and their timestamps were compared. Each move happened over a weekend, with a rollback that meant re-pointing a cross-connect. Less latency-critical traffic, such as research data and file transfers, was moved to a separate path over BGP-routed links with failover to the second site, so a bulk copy could never queue behind an order.
Measurement came before claims. Hardware capture on the switch mirror ports gave nanosecond timestamps for every frame, and we compared PTP offsets against a second, independent GNSS receiver to verify the clock itself. Tests included microbursts at line rate, deliberate PTP grandmaster failover, a cable pull on the primary path and an interrupt-affinity change that we confirmed showed up as a regression on the dashboard within minutes. Reports showed the latency histogram, with the 99th and 99.99th percentiles, not the average.
The outcome is a repeatable, measured path in which the median time from wire to application is in the low microseconds, clocks agree to well within a microsecond, and every packet can be replayed with a trustworthy timestamp. Copies of market-data and order traffic are timestamped in hardware and stored for replay and for best-execution evidence. The firm now watches its latency distribution on a dashboard and notices a regression the day it appears, not the quarter after.
What our team delivered
- Cut-through 100 Gb/s switching (QSFP28) with short, fixed-length cross-connects to the exchange handoffs
- IEEE 1588 Precision Time Protocol with hardware timestamping, a GNSS-disciplined grandmaster and boundary clocks, holding hosts within about 100 ns of the reference
- Kernel-bypass networking on tuned servers (DPDK or vendor user-space stacks), CPU isolation, pinned interrupts and NUMA-aware memory
- Hardware packet capture with nanosecond timestamps for every market-data feed and order, for replay and best-execution evidence
- Redundant paths and a second site with BGP-based failover for non-latency-critical flows
- Monitoring that measures latency distribution, not averages
Result
A repeatable, measured path in which the median time from wire to application is in the low microseconds, clocks agree to well within a microsecond, and every packet can be replayed with a trustworthy timestamp. The modeled build was commissioned in stages: cross-connect and fiber lengths were fixed and documented, clock offset between sites was measured continuously against the GNSS reference, and latency was tracked as a distribution (median, 99th and 99.99th percentile) rather than an average. Changes to the trading path went through a staging replica first, and a replayed capture reproduced any disputed order sequence with its original timestamps.
Round-trip times between seven cities
Round-trip time in milliseconds on direct paths between seven sites, close to what today's fibre routes deliver (light in glass covers about 100 km per millisecond one way, and real routes are longer than the map line). The darker the cell, the longer the round trip. Illustrative figures.
A fabric with no single point of failure
Every leaf switch has a 100G QSFP28 uplink to every spine, servers attach at 25G or 10G through LACP pairs, and ECMP spreads flows across all spines. If a spine fails, the flows simply rehash onto the rest.
The right module for every reach
From 10G SFP+ to 400G QSFP-DD, the module is chosen by distance and fibre: DAC or AOC inside the rack, short-reach optics on OM4 multimode in the hall, long-reach on OS2 single-mode between buildings and coherent 400ZR between datacenters. One 100G port can also break out into four 25G.
Failover in under a second
ISP A fails at 300 ms. BFD (RFC 5880) with a 50 ms interval and multiplier 3 declares the session down 150 ms later, BGP withdraws the routes, and the forwarding table repoints to ISP B. Traffic is back at full rate in well under a second; without BFD the router would wait for the 90 s BGP hold timer. Illustrative figures.
Facing something similar?
Tell us about your environment. We will tell you plainly what we would do and what it would take.