EMule: Encyclopedia II - EMule - Network and file sharing issues

EMule - Network and file sharing issues

EMule - Basic concepts

All files are given a hash value that depends on the file's contents, but not its filename. This allows each user to find all sources to a particular file no matter what file name each user has given the file.

Files are broken into "chunks" of 9.28 MB each (the last chunk may be smaller if the file size is not exactly divisible by the chunk size). A file's hash value is computed by first computing the MD4 checksum of each chunk, and then computing the MD4 checksum of the string of those chunks' checksums concatenated with each other.

Like the file hash, each user in the network gets a unique and permanent user hash. This user identification is highly secured by a public / private key handshake to prevent misuse.

After determining sources for a file (i.e. other clients who are sharing that file), a client contacts each of those sources to request the file. Each client maintains a "download queue" of other clients who are requesting files for download. Once a client reaches the top of such a queue, the clients contact each other and transfer begins. Clients usually request the chunk that fewest other sources have to aid distribution of the file through the network; otherwise rare chunks may generate a bottleneck. If a client already has most of a chunk, however, it may choose to request the rest of that chunk first because only complete chunks can be shared.

Before sending data eMule attempts to compress the data with [zlib] set at maximum compression. Then it chooses the smaller set of data to transmit. For already compressed files this is likely to be the raw file, while uncompressed files are likely to result in zlib compressed data to be transmitted.

EMule - Classic server based eD2k

(eD2k :EDonkey network)

The key to this network is the eD2k server. Each client must be connected to a server to enter the network. When connecting your client to a server, the server checks to see if other clients can freely connect to your client. If yes, the server assigns your client a so-called high ID. If communication is blocked, the server assigns your client a low ID. After the ID is assigned, eMule will send a list of all shared files to the server. The server adds the filenames and hash values you sent to its database.

Once connected to the network, the client can search for keywords in filenames. A search can either be local or global. If it’s a local search (searches only the server you are connected to), searches are quicker but will have fewer results. If the search is a global search (searches all the servers within the network), it will take longer but have more results. Each server looks up the keyword in its local database and returns any file names (with the hash value) that matches the keyword.

Note that there are fake servers in the network that do not return any results for certain type of searches (e.g. mp3 files)

Downloads can be added by eMule’s search function or a special eD2k link format offered on many websites. Once they are in the Download list, eMule first queries the local (connected) server then all other servers in the network for sources to that particular download. The server looks up the file’s hash value in its database and returns the clients it knows for having it. Sources are other clients who have at least downloaded one entire part (9.28 MB) of the file matching the hash.

EMule - Kad serverless network

(based on Kademlia protocol)

The only thing needed to connect to this network is the IP and port of any eMule client already connected. This is called a Boot Strap. Once a client is in the network, the client then requests for other clients to determine if it can be contacted freely. This process is very similar to the HighID/LowID check on the servers. If you can be freely contacted, you are assigned an ID (similar to a HighID) and given an open status. If you are not freely contacted, you are given a firewalled status. From version v.44a on, the Kad network supports a Buddy for firewalled users. Buddies are other Kad clients who have status open and work as a relay for connections, that the firewalled user cannot manage.

In this network it does not matter what you search for. Be it a search for filenames, for sources of a download or for other users, all work pretty much the same. There are no servers to keep track of clients and the files they share so it has to be done by each participating client in the network – in essence, every client is also a small server. Since every client is identified by a unique hash value, the idea of Kademlia is to associate a certain “responsibility” based on this hash. Each client in the Kad network works as a server for certain keywords or sources. The clients hash determines the specific keywords or sources. So the goal of any kind of search is to find those clients that have the responsibility for the current search topic. This is accomplished by a complex calculation of the possible distance to the target client by asking other clients for the shortest route to it.

EMule - Basic rules of communication behavior

eMule and other programs on eDonkey network support the network in two basic ways:

1) Users are forced to share files: Shared files larger than 9500 KB are divided and downloaded in parts, so that a recipient starts to share the parts of the file already downloaded even before the download of the whole file is complete. The recipient is forced to share in this way until the download is complete.

2) Users are forced to share their bandwidth, thus increasing network's data stream: The official version of eMule forces the users to set their maximum download speed to no more than four times their maximum upload speed. This limitation does not apply if the maximum upload speed is set to 10K per second or more.

Although there are many mods of eMule, it is hard to find one that doesn't enforce the second feature, which is a part of official eMule policy. Furthermore, attempting to circumvent the uploading feature can possibly slow a user's download, as he will get worse credits from uploaders. Also, many known "leecher" clients are often banned from the downloading through various means by eMule's mods (banning is not supported in official version).

eMule includes a feature that limits download and upload speeds, so that they do not exceed the limits set by the user.

eMule versions before 0.47a cannot share files larger than about 4 GB. This was an inherent limitation of the ed2k protocol. With the release of 0.47a, this limit was raised to 256GB, but only for clients supporting it (currently just eMule and aMule).

EMule - Summary

Both networks have totally different concepts for achieving the same: Searching for files and finding sources to a file. The main goal of the Kad network is to be independent of servers and improve scalability. Servers can only handle a certain amount of users and should a large server go down the network is severely handicapped. Kad network is self-organising and tunes itself for best possible performance depending on the number of users and their connection qualities. Therefore, it is more resistant to a large-scale network loss. Some servers also limit the number of files that can be shared by any particular user. This limitation doesn't exist in Kad network, therefore this network is more suitable for users that share big collections of files.

Adapted from the Wikipedia article "Network and file sharing issues", under the G.N U Free Docmentation License. Please also see http://en.wikipedia.org/wiki