This is a short technical paper on the Usenet News network’s structure and function.
The Usenet News network is a globally distributed news network built on the Network News Transfer Protocol (NNTP) which provides public access to a bulletin board system where anyone accessing the network can post or reply to messages. The Usenet News network or Usenet news (for short) is one of the oldest and most significant communication networks still being used for communication today.
Usenet was designed to be a “broadly decentralized bulletin board system” [1] (p.11) where news groups could be formed, and messages posted to them asynchronously to form discussions. It functions as a publicly accessible social network that allowed people to easily view and reply to messages from discussions on their local news server.
The Usenet News network was created in the mid-1980s, and since then Usenet’s infrastructure and use has changed significantly, but its core concept of messages being organized into groups and distributed between servers has remained unchanged [1]. Unfortunately, the networks popularity as a social space has declined due to a range of factors including the prevalence of spam and the popularity of other new social media services such as Twitter and Facebook.
The network has become a haven for file sharing with binary file distribution making up over 90% of Usenet News traffic [2]–[4]. This transformation may be attributed to the networks unique structure which is well suited to the sharing of information – the NNTP protocol creates a decentralized network, with a high level of anonymity that is only regulated by server administrators. Server administrators are responsible for maintaining news servers, which involves managing and regulating a news server’s content.
The Usenet news network’s structure and function has been shaped by the people and technology which underpin it but it is firstly a logical network, originally resting on top of many different physical networks. In 1987 these networks included “but [were] not limited to, UUCP (Unix to Unix Copy), the internet, an Ethernet, the BLICN network, an NSC Hyper channel and a BERKNET” [5]. Today the internet is responsible for supporting the Usenet News network [5].
Two aspects that are important for understanding the network are its topological structure and the distribution algorithm guiding messages propagation through the network.
Network topology
Horton et al. [5] describe the Usenet network topology as resembling a directed graph, where each node in the graph represents a news server and edges are transmission paths between two news servers. These edges or links between news servers are bidirectional with messages from news groups able to be exchanged backwards and forwards between news groups. The Usenet News network is also made up of sub-networks or hierarchies such as comp, sci, aus, and alt. The topological structure of these sub-networks function as connected graphs, where each node is connected to every other node in the sub-network, theoretically making the entire network connected.
Message structure
Messages posted to news servers on the Usenet News network by users are required to contain certain header information, the message format requires the following header fields to be included for a message to be acceptable: From, Date, Newsgroups, Subject, Message-ID, and Path [5].
- The from field of a messages contains an e-mail address for the person who posted the associated message.
- The date field contains a record of the date a message was originally posted to the Usenet News network.
- The newsgroups field contains a list of news groups a message was originally posted too.
- The subject field should contain a string describing “what the message is about” (Horton, et al. 1984).
- The message-id field contains a unique string for identifying a message, usually containing the hosts name and a string of numbers.
- A messages path field contains a list of news servers it has visited while propagating the network, starting with the news server it was originally posted on.
Other message header fields are also included in messages; Request for Comments (RFCs) 1036 and 5536 contain a full list with descriptions of the various fields and the information they should contain. In 2009 RFC 1036 was superseded by RFC 5536 which is an updated standard to reflect current practice.
Distribution algorithm
Messages posted to news servers on the network propagate indirectly from one server to the next, which is unlike other internet services such as e-mail networks where articles are routed and sent directly, traveling through the smallest number of servers possible. On the Usenet News network, news servers must subscribe to other news servers (known as feeds) to receive and send messages. When a news server receives and accepts a message from another server that message will then be forwarded to a list of subscribing news servers [6]. Message distribution can be summarized as follows (Figure 1):
- A message is posted to one or more news groups by a user on their local news server. The news server will then either accept or rejects it locally.
- Articles accepted locally will then be forwarded to all the news servers subscribing neighbours.
- The next receiving news server then examines the message to confirm that they want it and if they do, it is accepted locally, and in turn forwards the message onto all subscribing news servers.
This process will continue until the article reaches all news servers on the network or has propagated as far as possible. A critical aspect of the propagation algorithm is the prevention of loops within the network. In the above example it is possible for a news server to send an article to its neighbour and in turn have the neighbour send it back to that news server (since the network is bidirectional) and for this process to continue infinitely. The problem is solved by news servers keeping records of all messages they have seen, and discarding those that they have already seen, this process is further optimized by using a messages path field. These path strings provide a record of Usenet’s topological map, news servers also rely on these message paths to discover new servers on the network.
The path field
All messages on the Usenet News network are required to contain a path field, which contains a list of servers that a message has traversed to reach its current destination. Entries are added by a news server as it is forwarded on to another subscribing news server. A message’s path is used to prevent news servers from sending a message to neighbours who have already seen it, lessening unnecessary resource usage and traffic between news servers and to establish path for reaching new news servers [5].
Normally the right most entry in the path will be an entry from the news server that the message originates from, it is also permitted that an entry be added before this, which specifies the name of the sender (used for compatibility reasons). Path entries are not required to represent the real physical address of news servers; as a result, several different entries could be identified in a messages path, such as the following:
- Aliases: These entries usually appear as the abbreviated names of news server addresses, and do not represent real server addresses. Aliases can also be added to represent the sender of a message at the beginning of a path.
- Real addresses: The actual physical address of a news server, which can be reached on the network.
- Feeders: These entries are news servers which form part of a larger internal organization or group of news servers for a single domain. Feeding servers tend to be dedicated to handling either outbound or inbound messages, their purpose is usual indicated in the server’s path entry.
References
- D. Fisher, ‘Studying social information spaces’, in From Usenet to CoWebs, Springer, 2003, pp. 3–19.
- T. C. Turner, M. A. Smith, D. Fisher, and H. T. Welser, ‘Picturing Usenet: Mapping computer-mediated collective action’, J. Comput.-Mediat. Commun., vol. 10, no. 4, p. JCMC1048, 2005.
- J. Kim, F. Schneider, B. Ager, and A. Feldmann, ‘Today’s usenet usage: NNTP traffic characterization’, in 2010 INFOCOM IEEE Conference on Computer Communications Workshops, 2010, pp. 1–6.
- G. Fellows, ‘Newsgroups reborn–The binary posting renaissance’, Digit. Investig., vol. 3, no. 2, pp. 73–78, 2006.
- M. R. Horton, ‘Standard for interchange of USENET messages’, 1983.
- C. Lueg, From Usenet to CoWebs: interacting with social information spaces. Springer Science \& Business Media, 2003.