The new Facebook DC Topology, F16, introduced at the 2019 OCP Global Summit. Hardware, protocols, and distributed systems, require evaluation, development, and validation before deployment. This is a significant effort when done in a Facebook Data Center, and happens late in the cycle. Facebook Network Hardware and the Broadcom Switch organization jointly created an alternative: collapsing the F16 network into a single switch. A DC Network can be conveniently recreated in the lab at Facebook, at vendor’s facilities, and at ODMs. We called it F16InABox.
F16InABox recreates all F16 stages (RSW, FSW, SSW) inside a 12.8Tbps Broadcom Tomahawk 3 switch with ECMP groups, oversubscription ratios, and hashing logic between these stages matching F16. The system has about 4.8T for server endpoints (about 2 racks equivalent) where the logical placement of a server can be rack local, POD local, or cross-POD with respect to any other server. The multi-stage/multi-path DC network queuing behaviors appear in F16inabox without having to use a control plane like BGP.
F16InABox can self-generate background traffic without requiring external traffic generators, and uses timestamping to accurately measure end-to-end delay for every packet.
We used F16inabox to evaluate very fast distributed systems consensus and consistency primitives, and will discuss other uses like - PTP performance evaluation - Storage over the network - NIC feature evaluation and validation - ODM integration testing - Latency reduction mechanisms (AQM)