Frank.

← Back to Blog

January 20, 2024

Alright, backend engineers, gather 'round! Forget your ORMs and message queues for a moment. We're about to embark on a journey deep into the silicon veins of the network, to witness the birth and life of a TCP segment. This isn't just about packets; it's about understanding the unsung hero that ensures your carefully crafted API responses and database queries actually make it across the digital abyss reliably.

Imagine your application, running sweetly in user space, decides to send some data. Perhaps it's a JSON payload from a send() call on a socket. This data, currently lounging on the heap (or maybe the stack if it's small enough and local), is about to get a first-class ticket to its destination. But it can't travel naked. It needs a vehicle, a chaperone, a contract – it needs to become a TCP segment.

The Genesis: From Application Data to Kernel Trust

When your application calls send(), it's a system call. This is a crucial context switch: the CPU flips from user mode to kernel mode. Your application's data is copied from its user-space memory buffers into kernel-space buffers (think sk_buff in Linux). This copy is vital; the kernel can't trust user-space pointers to remain valid.

Now, within the kernel's TCP/IP stack, the transformation begins. The data needs to be wrapped in a TCP header. This isn't just any wrapper; it's a meticulously crafted 20-byte (minimum) structure that dictates the segment's entire journey.

Anatomy of a Traveler: The 20-Byte (Minimum) TCP Header

This header is like the segment's passport and shipping manifest, all rolled into one. Let's dissect it, field by field, understanding how the kernel populates and uses them.

1. Source Port (16 bits) and Destination Port (16 bits): The Digital Addresses

  • Story: Think of your server application. When it started, it bind()-ed to a specific port (e.g., 443 for HTTPS). This port number is its unique address within the server (identified by its IP address). The Source Port is this "return address." The Destination Port is the port number of the application on the receiving machine it wants to talk to.
  • Under the Hood: The kernel maintains a table of active ports and the processes associated with them. When your segment is being crafted, the kernel plucks the source port associated with the sending socket and looks up the destination port agreed upon during the connection setup (the TCP handshake). These 16-bit numbers allow up to 65,535 possible ports – a bustling city of potential communication endpoints on any machine.

2. Sequence Number (32 bits): The "Page Number" of Your Data Stream

  • Story: TCP guarantees ordered delivery. But networks are chaotic; packets can take different routes, get delayed, or arrive out of order. Imagine sending a multi-volume epic novel one page at a time. Without page numbers, reassembling it would be a nightmare! The Sequence Number is that page number. It tells the receiver the byte position in the overall data stream where this segment's payload begins.
  • Under the Hood: For a new connection, the Initial Sequence Number (ISN) is chosen pseudo-randomly by the kernel (a security measure to prevent connection hijacking). For subsequent segments, the kernel meticulously increments this number by the number of bytes sent in the previous segment's payload. This 32-bit field allows for a massive 4GB of byte numbering before it wraps around – usually more than enough for most connections.

3. Acknowledgment Number (ACK Number) (32 bits): The "Got It, Send Next!"

  • Story: This is the receiver's way of saying, "I've successfully received all data up to byte X, and I'm now expecting the segment that starts with byte X+1." It's the backbone of TCP's reliability. If the sender doesn't get an ACK for a segment, it knows something went wrong and resends.
  • Under the Hood: When the kernel on the receiving end gets a segment, it checks the Sequence Number. If it's the one it's expecting, it processes the data, then prepares an ACK segment to send back. The ACK number it puts in this response is the next sequence number it expects. This field is only valid if the ACK flag (see below) is set.

4. Data Offset (Header Length) (4 bits): "My Header is This Big"

  • Story: The TCP header is at least 20 bytes, but it can be larger if TCP options (like MSS or SACK) are included. This 4-bit field tells the receiver how many 32-bit words (4-byte chunks) are in the header. So, a value of 5 here means 5 * 4 = 20 bytes. If it's 6, it means 24 bytes.
  • Under the Hood: The kernel calculates this based on whether any TCP options are being added to this specific segment. The receiver's kernel uses this to know exactly where the actual application payload begins in the segment.

5. Reserved (3 bits): "For Future Use, Do Not Touch!"

  • Story: These bits are set aside for future protocol enhancements. For now, they must be zero.
  • Under the Hood: The kernel simply sets these to zero.

6. Flags (9 bits): The Control Panel Switches

  • Story: These are single-bit switches that signal important events or states. Think of them as urgent instructions to the receiving TCP stack.
    • NS (1 bit): ECN-nonce concealment protection (experimental).
    • CWR (1 bit): Congestion Window Reduced – the sender indicates it has reduced its sending rate.
    • ECE (1 bit): ECN-Echo – indicates the TCP peer is ECN capable and that a congestion event was received.
    • URG (1 bit): Urgent Pointer field is significant. Tells the receiver that some "urgent" data exists. The kernel on the receiving side might signal the application (e.g., with SIGURG).
    • ACK (1 bit): Acknowledgment Number field is valid. Almost all segments after the initial SYN will have this set.
    • PSH (1 bit): Push Function. Tells the receiver to push the data to the application layer immediately, rather than waiting for its internal buffers to fill. The sending kernel might set this if the application used a flag like MSG_EOR or when its send buffer is flushed.
    • RST (1 bit): Reset the connection. An abrupt termination. "Something is irrevocably broken, let's tear down this connection NOW." The kernel sends this if it receives a segment for a non-existent connection or if a fatal error occurs. No cleanup, just boom, gone. Registers holding connection state are cleared, kernel memory for buffers is reclaimed.
    • SYN (1 bit): Synchronize sequence numbers. Used to initiate a connection (the first packet in the three-way handshake). "Hi, I'd like to connect, my starting sequence number is X." The kernel allocates resources (memory for connection state tables, buffers) when it sends or receives a SYN.
    • FIN (1 bit): Finish. Gracefully terminates the connection. "I have no more data to send." The kernel begins the connection teardown sequence, but will continue to receive data until the other side also sends a FIN.
  • Under the Hood: The kernel's TCP state machine dictates which flags are set. For instance, during connection setup, it sends a SYN. When data is acknowledged, it sends an ACK.

7. Window Size (16 bits): "I Can Handle This Much Data Right Now"

  • Story: This is TCP's flow control mechanism. The receiver uses this field to tell the sender how much data (in bytes) it's currently willing to accept. Think of it as the amount of free space in the receiver's kernel buffer for this specific connection. If the receiver is busy, it might advertise a small window, or even a zero window, telling the sender to pause.
  • Under the Hood: The receiving kernel constantly monitors its buffer space for this connection. It updates this value in the segments it sends back. The sending kernel respects this value, ensuring it doesn't overwhelm the receiver. This prevents data loss due to buffer overflows on the receiver side.

8. Checksum (16 bits): "Is This Header (and Payload) Intact?"

  • Story: Networks can be noisy. Bits can flip! The checksum is a mathematical calculation performed over the TCP header, the TCP payload, and parts of the IP header (a "pseudo-header"). The sender calculates it, and the receiver recalculates it. If they don't match, the segment is considered corrupted and is discarded.
  • Under the Hood: This calculation is often offloaded to the network interface card (NIC) for performance reasons. If not, the CPU in the kernel performs the calculation. This is a crucial step for data integrity.

9. Urgent Pointer (16 bits): "Emergency Data This Way!"

  • Story: If the URG flag is set, this pointer indicates an offset from the current sequence number where "urgent" data ends. This mechanism is rarely used by modern applications but exists for scenarios where out-of-band data needs to be processed quickly.
  • Under the Hood: If the kernel sees the URG flag, it uses this pointer to identify the urgent data. It might then notify the receiving application, potentially via a signal like SIGURG on POSIX systems, allowing the application to read the urgent data separately.

Adding the Goods: The Payload and MSS

Once the header is meticulously prepared by the kernel, it's time for the actual application data (that was copied from user-space into kernel buffers) to be appended as the payload.

But how much data can one segment carry? This is governed by the Maximum Segment Size (MSS).

  • Story: Think of MSS as the maximum cargo capacity of our segment, not including the TCP and IP headers. It's like saying, "Each of my delivery trucks can carry at most X kilograms of goods." This value is typically negotiated during the TCP handshake (using a TCP option). Both sides will suggest an MSS, and usually, the smaller of the two (or a value derived from the path MTU) is chosen.
  • Under the Hood: The MSS is crucial. The kernel needs to ensure that a TCP segment, when wrapped in an IP header, doesn't exceed the Maximum Transmission Unit (MTU) of the underlying network path. The MTU is the largest packet size a specific network link can handle (e.g., Ethernet typically has an MTU of 1500 bytes). If a segment is too large for a link, it would need to be fragmented at the IP layer, which is inefficient and something TCP tries to avoid by respecting the MSS.
    • For example, if the MTU is 1500 bytes, and IP header is 20 bytes, and TCP header is 20 bytes, the MSS would be 1500 - 20 - 20 = 1460 bytes.
    • The kernel's TCP layer segments the application data stream into chunks no larger than the MSS. Each chunk gets its own TCP header.

A Note on Jumbo Frames: For specialized environments, like high-speed local networks or storage area networks, you might encounter Jumbo Frames. These are Ethernet frames that can carry up to 9000 bytes of payload, significantly larger than the standard ~1500 bytes. If the entire path supports them, using jumbo frames can mean a larger MSS, fewer segments, less header overhead, and potentially better throughput as the kernel has to process fewer individual packets. However, this requires consistent configuration across all network devices in the path.

The Kernel's Send-Off: Down the Stack and Out the Wire

Our TCP segment, now fully formed with its header and payload, resides in a kernel buffer.

  1. It's passed from the TCP layer down to the IP layer within the kernel. Here, it gets another header – the IP header (with source/destination IP addresses, etc.), becoming an IP datagram.
  2. The kernel then consults its routing table to determine which network interface to send this datagram through.
  3. Finally, the datagram is handed off to the network interface card (NIC) driver. The driver prepares it for transmission onto the physical network. This often involves Direct Memory Access (DMA), where the NIC pulls the packet data directly from the kernel's memory buffers without further CPU intervention, loading it into its own transmit registers or buffers before sending it out as electrical signals or light pulses.

The Receiver's Welcome: Validation and Reassembly

When the segment arrives at the destination machine's NIC, the reverse process begins:

  1. The NIC driver DMAs the incoming frame into kernel memory.
  2. The kernel's IP layer strips the IP header and passes the TCP segment up to the TCP layer.
  3. The TCP layer is where the magic we've discussed happens:
    • Checksum verification: Is the segment corrupted? If so, discard.
    • Port numbers: Is there an application listening on this destination port? If not, the kernel might send an RST back.
    • Sequence Number: Is this the segment we were expecting? If it's an old duplicate, discard. If it's out of order, the kernel might buffer it (if it has space, guided by its receive window) hoping the missing pieces arrive soon. If it's the correct next piece, great!
    • ACKs: The kernel prepares an acknowledgment segment to send back to the sender, updating the Acknowledgment Number and its own Window Size.
    • Reassembly: The kernel places the payload data into the correct order in a per-connection receive buffer.
    • Notifying the Application: Once data is available in the receive buffer, the kernel notifies the waiting user-space application (e.g., a recv() call unblocks, or a select()/poll()/epoll() indicates readability). The application can then copy the data from the kernel's socket buffer into its own user-space buffers.

Why All This TCP Segmentation Fuss? The Grand Design

TCP segmentation isn't just busywork for your kernel. It's fundamental for:

  • Reliability: Sequence numbers and acknowledgments ensure (or at least try very hard to ensure) that all data arrives, and arrives in order. Lost segments are retransmitted.
  • Flow Control: The window size prevents a fast sender from overwhelming a slow receiver or a congested network.
  • Congestion Control (Beyond this Scope, but related): TCP has sophisticated algorithms (using signals like packet loss and round-trip times) to slow down when it detects network congestion, helping the overall internet function.
  • Fairness: By breaking data into smaller segments, TCP allows multiple connections to share network resources more equitably than if one application tried to blast a massive, monolithic block of data.
  • Navigating Network Limits: The MSS ensures segments fit within the MTUs of diverse network links.

So, the next time your backend service flawlessly delivers gigabytes of data, take a moment to appreciate the billions of tiny, well-orchestrated TCP segments, each with its meticulously crafted 20-byte header, managed by the tireless kernel, making it all possible. Each field in that header tells a story of careful engineering designed to make an unreliable internet a surprisingly dependable place for our applications to communicate.