Beyond basic socket operations, there are several things a programmer should know and use when developing distributed systems. This article covers advanced network programming concepts such as synchronous I/O multiplexing, threads and extensible message format specification.

Most programming languages offer basic socket operations that allow programmers to implement basic client/server applications. These basic socket operations include :

• Creating a socket (client and server)
• Binding a socket (server only)
• Listening to a socket (server only)
• Reading from a socket (client and server)
• Writing to a socket (client and server)
• Connecting to remote socket (client side only)

Beyond these basic operations, several techniques can be used to provide additional features for improving performance and extensibility of the whole distributed system. These techniques are described in the following sections.

The read operation on sockets is a blocking operation, which means that the program reading from the socket will be blocked until data is available. This can cause issues when the a program has several socket connections at the same time; the program can be waiting for data on some socket but some other socket may have data available.

To prevent this situation, the programmer can use synchronous I/O multiplexing on sockets. This allows the program to know if there is data available on a socket by checking a flag that is set by the Kernel if data for that socket is received from the network. This way, the program can cycle through opened sockets and process incoming data as it comes in an asynchronous way.

Functions for performing multiplexed I/O are as follows :

Threads are lightweight processes that share the same memory and data structures as the process that created them. Threads can be used to enhance program performance by adding time multiplexing capability that mimics parallel processing of tasks. The use of threads provide several advantages over process forking. First, threads allow the program to specify scheduling policy and priority. Second, context switching between threads takes less cpu time. This means that two threads performing two tasks are faster that two processes performing two tasks in a time multiplexed cpu.

From network programming perspective, threads provide an excellent way for improving server performance by handling client requests in separate threads. This gives the impression that there are many serves running in parallel to accommodate simultaneous client connections. If a single process/thread was used, clients will have to queue while the server is sequentially processing client requests.

Functions for manipulating Posix threads are as follows :

It is important that all entities involved in a protocol exchange must know how to layout protocol messages as payload of lower transport protocols. This is referred to as encoding. Also, these entities need to know how to extract and interpret the encoded payload and convert it into a usable data structure.

There are several alternatives for message structure specification and encoding rules. Since messages are essentially data structures, the programmer needs first to decide what fields to use in the data structure, then an encoding scheme must be selected. Below are a some of the available options.

In its simplest form, a message can be a text string that contains command code, arguments, result indication etc.. This is the case for old protocols such as HTTP, POP, SMTP, FTP. These protocols are simple and easy to debug since protocol exchanges are human readable. However, text message protocols are not flexible. It can be very difficult to find an encoding scheme if message structures are hierarchical and complex. Generally, text messages, once built to represent a certain data structure, are sent as text strings over the network.

Many protocols (TCP, IP and DNS) use data structures to represent messages. Specifying and communicating protocols messages as data structures and sending the data structures in their binary representation alleviates the designer and programmer from the complexity of encoding messages into strings.

It is important to keep in mind that numeric data are stored in memory using different layouts or byte ordering depending on the architecture of the host. This is referred to as endianness. It is important when communicating data structures that contain numeric data over the network to compensate for different endianness between the involved hosts. This is generally done by converting from host byte order to network byte order when sending the data structure, then converting from network byte order to host byte order at the receiving end.

Declaring a C structure whose fields appear in the same order as the packet fields, and whose field data types match those of the packet fields, will not necessarily produce a compatible set of bits on the wire due to the alignment requirements of individual fields within that structure. For example, if a char is followed by a short, the Windows/Intel x86/Microsoft Visual C++ compiler will position the short at an address two bytes after the address of the char, not one byte after it. It may also append padding bytes to the end of a structure so that its size is a multiple of 2, 4 or 8, for efficiency, which can result in more, or fewer, bytes sent across the wire in a packet than the other side expects. Note that when a packet contains a series of structures (either different types, or multiple occurrences of the same type), the location of each structure (not just the first) is affected by data alignment rules. For example, using Microsoft Visual C++ on the Intel x86 platform, each structure is positioned at an address that is a multiple of the largest simple data type (1, 2, 4 or 8 bytes) appearing in the structure.

When using complex data types such as C structures and unions in the implementation of an RPC or packet-based network protocol, verify that the memory layout of the structure or union on the communicating systems follows the same data alignment rules by using a debugger or network monitor program to inspect the data that is sent and received. If the size or layout does not match, the application might send or receive an incorrect number of bytes or it might reference field contents at the wrong offset in memory and retrieve incorrect data. More on data alignement.

In protocols where extensibility is important, the use of Type-Lenght-Values (TLVs) or Atrribute-Value-Pairs (AVPs) is a must. Message specification based on TLVs or AVPs can be easily extended without breaking compatibility with older versions of the protocol.

TLVs are data structures that contain three fields: Type, Length and Value. The Type field is an integer that indicates the type of the payload or value carried by the TLV. The Length field indicates the size of the payload. 'Value' is a variable length field that carried the payload. Attribute Value Pairs are TLVs but without the Length field, and its use assumes that each type as a fixed and pre-defined length that end points of the protocol already know.

To extend a protocol using TLV or AVP based messages, new TLVs can be added seamlessly to data structures representing the protocol messages. Entities that understand how to process the new TLV will process it, and entities that do not know the Type of the TLV will ignore it. Protocols using TLVs and AVPs include Diameter, RADIUS.

ASN.1 allows to describe complex data structures independently of any particular programming language. The ASN.1 compiler would take these ASN.1 specifications and produce a set of target language (C, C++, Java) files which would contain the native type definitions for these abstractly specified structures, and also generate a code which would perform the conversions of these structures into/from a series of bytes (serialization/deserialization), (presumably, these routines would be useful if the structure is going to be transferred over the network or written to an external media). ASN.1 is a powerful framework that ensures compatibility across programming languages and alleviates the need for managing byte ordering.

See an example : ASN1 DER encoding and decoding example in C

Labels: coding, howto