Skip to content

Networking

What experienced engineers forget about networking before an interview, grouped by subtopic. For TLS see security.md; for load balancers, CDNs, and real-time delivery see system.md; for sockets and epoll see os.md.

End to end

Loading a URL

  • DNS resolves the name → TCP handshake (1 RTT) → TLS 1.3 handshake (1 RTT) → HTTP request → response → the browser parses and renders (frontend.md).
  • A cold HTTPS request over TCP costs about 3 RTTs before the first response byte; HTTP/3 saves one, and resumption with 0-RTT saves more.
  • Every later request reuses the connection, which is why keep-alive and connection pools matter.

Connection failure signatures

Symptom Usual cause Check with
Name doesn't resolve, or resolves to an old IP NXDOMAIN, stale cache, TTL not expired dig (answer, TTL)
Connection refused RST: host reachable, nothing listening on the port ss -tlnp on the server
Connect timeout SYN dropped: firewall, security group, or dead host; Linux retries the SYN for ~131 s (~127 s before Linux 6.5), so set connect timeouts traceroute/mtr, tcpdump for SYNs
Connection reset by peer peer crashed, or a middlebox dropped the flow after its idle timeout tcpdump/Wireshark around the RST
502 from a proxy upstream refused, reset, or sent an invalid response proxy error log, ss -tanp states
504 from a proxy, or just slow upstream didn't answer in time curl -w '%{time_connect} %{time_appconnect} %{time_starttransfer}' per phase

TCP

Handshake and connection queues

Teardown and TIME_WAIT

  • The side that closes first enters TIME_WAIT for 2×MSL (60 s on Linux) so delayed packets can't corrupt a new connection on the same port pair.
  • Many short-lived outbound connections exhaust ephemeral ports (~28k by default) — reuse connections (keep-alive, pooling) instead of tuning the kernel.
  • Many connections in CLOSE_WAIT mean the application never called close() — a bug in your code, not the network.

Flow and congestion control

  • Flow control (receive window) protects the receiver; congestion control (congestion window) protects the network.
  • Slow start begins at 10 segments (~14 KB) and roughly doubles per RTT — why the first 14 KB of a page matter and why new connections are slow.
  • Throughput ≤ window / RTT (bandwidth-delay product) — high-latency links need large windows.
  • CUBIC (Linux default) backs off on packet loss; BBR models bandwidth and RTT instead, doing better on lossy links.

Retransmission and loss recovery

  • The retransmission timeout (RTO) derives from smoothed RTT; Linux floors it at 200 ms (initial 1 s) and doubles it on each retry.
  • One lost packet on a quiet connection can therefore add ≥ 200 ms — a classic cause of p99 spikes.
  • Fast retransmit: 3 duplicate ACKs resend the missing segment without waiting for the RTO; SACK tells the sender exactly which ranges arrived.
  • A lost last segment produces no duplicate ACKs; Linux's tail loss probe resends it after ~2 RTTs instead of waiting for the RTO.

Keepalive and half-open connections

  • TCP keepalive probes an idle connection to detect a dead peer (Linux: first probe after 2 hours by default); HTTP keep-alive only means reusing a connection — unrelated.
  • A peer that crashes or loses the network without sending FIN/RST leaves a half-open connection, invisible until you write or probe.
  • Keep the client's idle timeout shorter than the server's or load balancer's, or requests land on connections the other side just closed (sporadic 502s and resets).

Nagle and delayed ACKs

  • Nagle's algorithm holds small writes until earlier data is acknowledged; combined with the receiver's delayed ACK it adds ~40 ms stalls on Linux (~200 ms on Windows) for request-response traffic.
  • Set TCP_NODELAY for latency-sensitive protocols (most RPC libraries do).

Head-of-line blocking

  • TCP delivers bytes in order, so one lost packet stalls everything behind it — even data for unrelated HTTP/2 streams.
  • HTTP/1.1 blocks at the request level, HTTP/2 fixes that but not TCP's, and QUIC fixes both: streams are delivered independently, so a lost packet stalls only the streams whose data it carried.

HTTP

HTTP/1.1

  • Persistent connections (keep-alive), but one outstanding request per connection in practice (pipelining is unused).
  • Browsers open about 6 connections per host to parallelize.

HTTP/2

HTTP/3 and QUIC

  • UDP has no handshake, ordering, or retransmission, so QUIC implements them in user space; DNS, VoIP, and games use UDP directly and handle loss themselves.
  • QUIC runs over UDP with TLS 1.3 built in: 1-RTT setup (0-RTT on resumption) and independent stream delivery, so no cross-stream head-of-line blocking; loss detection and congestion control stay per connection.
  • Connection migration: a connection ID survives an IP change (Wi-Fi → cellular).
  • Clients discover it via the Alt-Svc header or a DNS HTTPS record; many networks block UDP, so browsers fall back to TCP.

WebSocket handshake

Client IP behind proxies

  • L7 proxies append the peer's address to X-Forwarded-For; clients can forge earlier entries, so trust only those your own proxies added — read from the right, skipping known proxy hops.
  • L4 load balancers either preserve the source IP or pass it in a PROXY protocol header, which the backend must be configured to expect.
  • Rate limits, geo-blocking, and audit logs depend on it: taking the leftmost entry lets any client spoof its IP.

DNS

Resolution path

Record types

  • A/AAAA (IPv4/IPv6), CNAME (alias — not allowed at the zone apex), MX, NS, SRV.
  • TXT carries SPF, DKIM, DMARC, and domain verification; CAA limits which CAs may issue certificates; HTTPS/SVCB advertise protocol hints such as HTTP/3.

TTL trade-offs

  • Low TTL = faster failover and migration, more queries; high TTL = fewer queries, slow changes.
  • Before a migration, lower the TTL at least one old-TTL in advance — resolvers keep the old record until it expires; some clients (JVMs, apps with their own cache) ignore TTLs anyway.

Transport and privacy

  • UDP port 53 by default, TCP for large responses: an answer bigger than the client's advertised EDNS buffer size (commonly 1232 bytes, to avoid fragmentation) is truncated and retried over TCP.
  • DoT/DoH encrypt queries to the resolver; DNSSEC signs records for authenticity but doesn't encrypt them.

IP and routing

Addressing and CIDR

Routing tables

  • The kernel picks the longest prefix match: a /32 route beats a /24, which beats the default route 0.0.0.0/0 (the gateway).
  • On the local link, ARP (IPv6: neighbor discovery) resolves the next hop's MAC address; MACs change at every hop, while IPs stay end to end unless NAT rewrites them.
  • ip route get <addr> shows the route, interface, and source address the kernel would use.

NAT

  • NAT rewrites source IP and port and tracks each flow in a connection-tracking table — unsolicited inbound connections are dropped unless a port forward or static mapping exists.
  • One public IP offers at most ~64k source ports per destination IP:port (AWS NAT gateway: 55k), so heavy traffic to one endpoint exhausts it; a full conntrack table drops new connections.

MTU and fragmentation

  • Ethernet MTU 1500 → TCP MSS 1460 over IPv4 (1440 over IPv6); tunnels (VPN, VXLAN) shrink it.
  • Path MTU discovery needs ICMP "fragmentation needed" messages; firewalls that drop ICMP cause black holes — small requests work, large ones hang.

Anycast and BGP

  • BGP exchanges routes between autonomous systems; a bad announcement or withdrawal can take a whole company offline (Facebook, October 2021).
  • Anycast announces one IP from many locations and BGP delivers each client to the nearest by routing policy (usually, not always, the lowest-latency) — CDNs, public DNS resolvers, DDoS absorption.

IPv6

  • 128-bit addresses, no NAT needed; hosts configure themselves with SLAAC, and a subnet is normally a /64.
  • Dual-stack clients prefer IPv6 but race both families (Happy Eyeballs) so a broken path doesn't stall connections.