Serialization

When two applications talk over a network, they don’t exchange messages, they exchange bytes. But those bytes aren’t random: they follow a strict structure defined by protocols and data formats. In this post, we’ll dive into how structured data is serialized into byte streams, how headers give those streams meaning, and how receivers deserialize them back into usable data.
Recall how send()returns the number of bytes sent out. But let me tell you that it may be less than what you actually wanted it to send. This happens even in normal conditions when there is no error (-1) or end-of-file. The catch here is send() might not send all the bytes you asked it to. For example, if you want it to send 512 bytes, but it returns 412; the remaining 100 bytes are still in your little buffer waiting to be sent out. You have to continuously check the length of the string sent and compare with the actual len of buffer defined to send out the complete data. Fortunately, it can be easily done by looping over send()function until we get an error (-1)
the accompanying code can be found at eop.

Now we have gracefully handled the partial send()problem. It is easy to send text data across the network, as you see, but what if you want to send some binary data like ints or floats? Well at this point you may think of some possibilities:

1. convert the number into text and send the text (IRC).
2. send the raw data with a pointer attached.

Each method has its own advantages and drawbacks, and the task of finding them is left as an exercise for the reader. Before I begin this section, I want to let you know that there are libraries specifically designed for this task. But rolling your own and keeping it portable and error-free is no easy feat. Fear not! (Were you afraid there for a second? No? Not even a little bit?) There is something we can do: we can pack (or “serialize”, or one of a thousand million other names) the data into a known binary format that the receiver can unpack on the remote side.

#include <stdint.h>  
  
uint32_t htonf(float f)  
{
    uint32_t p;
    uint32_t sign;

    if (f < 0) { sign = 1; f = -f; }  
    else { sign = 0; }
          
    p = ((((uint32_t)f)&0x7fff)<<16) | (sign<<31);     // whole part and sign  
    p |= (uint32_t)(((f - (int)f) * 65536.0f))&0xffff; // fraction  

    return p;
}
  
float ntohf(uint32_t p)
{
    float f = ((p>>16)&0x7fff);   // whole part  
    f += (p&0xffff) / 65536.0f;   // fraction  

    if (((p>>31)&0x1) == 0x1) { f = -f; }   // sign bit set  

    return f;
}

The above code is sort of a naive implementation that stores a float in a 32-bit number. The high bit (31) is used to store the sign of the number (“1” means negative), and the next seven bits (30-16) are used to store the whole number portion of the float. Finally, the remaining bits (15-0) are used to store the fractional portion of the number.
Usage is fairly straightforward:

#include <stdio.h>  
  
int main(void)
{
    float f = 3.1415926, f2;
    uint32_t netf;

    netf = htonf(f);  // convert to "network" form  
    f2 = ntohf(netf); // convert back to test  

    printf("Original: %f\n", f);        // 3.141593  
    printf(" Network: 0x%08X\n", netf); // 0x0003243F  
    printf("Unpacked: %f\n", f2);       // 3.141586  

    return 0;
}

The standard for storing floating point numbers is used by most modern day computers internally for doing floating point math, so in those cases, strictly speaking, conversion wouldn’t need to be done. There are some pointers if you want to implement your custom serialization logic. Caution: I haven’t tried them yet but they look respectable. All in all, be aware while unpacking data sent over a network — a malicious user might send badly constructed packets in an effort to attack your system.
Now the data has been packed and sent over the network. What happens at the receiver’s end when a packet of data arrives? A common mistake is to assume that a read somehow corresponds to a write from the peer. This is not possible because a byte stream does not preserve any boundaries.
If the packets are of variable length, how does the receiver know when one packet ends and other begins? (lot of questions already) This brings the concept of data encapsulation.

Data Encapsulation

Recall how each layer in a network protocol wraps its underlying layer in a hierarchical fashion i.e., Ethernet contains IP, IP contains UDP or TCP, UDP or TCP contains application protocols.
When a packet is created, it is wrapped (“encapsulated”) in a header by the first (say FTP/SFTP) protocol, which is(header + data) again wrapped by the second (TCP) protocol, and so on… When another computer receives the packet, the hardware strips the Ethernet header, the kernel strips the IP and TCP headers, the SFTP program strips the SFTP header, and it finally has the data. Header is some binary metadata associated with a packet.

Each application running over TCP or UDP distinguishes itself from other applications using the service by reserving and using a 16-bit port number. An application protocol determines message length and has two structural levels:

1. Message splitting in the byte stream
2. Internal message structure (deserialization)

A simple binary protocol

The first step is to split the byte stream into messages. For now, both the request and response messages are just strings.

The variable-length makes it difficult to parse the message stream. For example, one person named “Vishal” might say, “Hi”, and another person named “Sid” might say, “Hey everyone, what’s going on?” If we simply stream the raw data V i s h a l H i S i d H e y e v e r y o n e ..there’s no way for the receiver to distinguish where one message ends and another begins. To solve this, we use data encapsulation, wrapping each message in a structured packet that includes metadata. This helps both sender and receiver agree on how to interpret the data. It’s the basis of a protocol.

In this case, let’s assume the user name is a fixed length of 8 characters, padded with \0. And then let’s assume the data is variable length, up to a maximum of 128 characters. Let’s have a look a sample packet structure that we might use in this situation:
• len (1 byte, unsigned) — The total length of the packet, counting the 8-byte name and data.
• name (8 bytes)—The user’s name, NUL-padded if shorter.
• chatdata (upto 128 bytes)—Actual message text
Using the above packet definition, the first packet would consist of the following information (in hex and ASCII):

0A       56 69 73 68 61 6C 00 00      48 69  
(length)  V i  s  h  a  l (padding)   H  i

And the second is similar:

24       53 69 64 00 00 00 00 00      48 65 79 20 65 76 ...  
(length)  S  i  d    (padding)        H  e  y     E  v ...

Why This Works

The first byte tells the receiver how many total bytes to expect. The receiver reads the length, then reads that many bytes to get the full packet.
This prevents message mix-ups and enables the client to reconstruct each message cleanly.

#include <sys/types.h>  
#include <sys/socket.h>  
  
int sendall(int s, char *buf, int *len)
{
    int total = 0;            // how many bytes are sent  
    int bytesleft = *len;     // how many are left to send  
    int n;

    while(total < *len) {
        n = send(s, buf+total, bytesleft, 0);
        if (n == -1) { break; }
        total += n;
        bytesleft -= n;
    }

    *len = total;        // return number actually sent here  

    return n==-1?-1:0;   // return -1 on failure, 0 on success
}

More on protocols

Why read_full() and write_all()?

read_full:
When reading from a socket, a single read call might return fewer bytes than requested even when more data is coming. This happens because data is pushed by the remote peer and stored in a kernel buffer, so read simply copies whatever is available at the time. To ensure that you receive the full amount of data you expect, you must repeatedly call read until either the desired number of bytes is collected, an error occurs, or the end of the stream is reached. It’s also important to handle cases where read is interrupted by a signal (EINTR) by retrying the operation.

write_all:
When writing data to a socket, the write call may not write all the bytes in one go. This can occur because the kernel buffer may be full or because the write operation was interrupted by a signal. To ensure that the entire data payload is sent, you need to loop and call write repeatedly until all bytes are successfully written. This approach guarantees that no portion of the message is lost due to partial writes.

References:

Beej guide to network programming
[a link to my brain lol]