Wget: Encyclopedia II - Wget - Features

Wget - Features

Wget - Robustness

Wget has been designed for robustness over slow or unstable network connections. If a download does not complete due to a network problem, Wget will automatically try to continue the download from where it left off, and repeat this until the whole file has been retrieved. It was one of the first clients to make use of the then-new Range HTTP header to support this feature.

Wget - Recursive download

Wget can extract resources linked from HTML pages and download them in sequence, repeating the process recursively until all the visible pages have been downloaded or a maximum recursion depth specified by the user has been reached. The downloaded pages are saved in a directory structure resembling that on the remote server. This "recursive download" enables partial or complete mirroring of web sites via HTTP. Links in downloaded HTML pages can be adjusted to point to locally downloaded material for offline viewing. When performing this kind of automatic mirroring of web sites, Wget supports the Robots Exclusion Standard.

Recursive download works with FTP as well, where Wget issues the LIST command to find which additional files to download, repeating this process for directories and files under the one specified in the top URL. Shell-like wildcards are supported when the download of FTP URLs is requested.

When downloading recursively over either HTTP or FTP, Wget can be instructed to inspect the timestamps of local and remote files, and download only the remote files newer than the corresponding local ones. This allows easy mirroring of HTTP and FTP sites, but is considered inefficient and more error-prone when compared to programs designed for mirroring from the ground up, such as rsync. On the other hand, Wget does not require special server-side software for mirroring, so the comparison is at least somewhat flawed.

Wget - Non-interactiveness

Wget is non-interactive in the sense that, once started, it does not require user interaction and does not need to control a TTY, being able to log its progress to a separate file for later inspection. That way the user can start Wget and log off, leaving the program unattended. By contrast, most graphical and curses-based web browsers require the user to remain logged in and to manually restart failed downloads, which can be a great hindrance when transferring a lot of data.

Wget - Portability

Written in a highly portable style of C with minimal dependencies on third-party libraries, Wget requires little more than a C compiler and a BSD-like interface to TCP/IP networking. Designed as a Unix program invoked from the Unix shell, Wget has been ported to numerous Unix-like environments and systems, such as Cygwin and Mac OS X, as well as to Microsoft Windows.

Wget - Other

• Wget supports download through proxies, which are widely deployed to provide web access inside company firewalls and to cache and quickly deliver frequently accessed content.
• It makes use of persistent HTTP connections where available.
• IPv6 is supported on systems that include the appropriate interfaces.
• SSL/TLS is supported for encrypted downloads using the OpenSSL library.
• Files larger than 2 GB are supported on 32-bit systems that include the appropriate interfaces.
• Download speed may be limited to avoid using up all of the available bandwidth.

Adapted from the Wikipedia article "Features", under the G.N U Free Docmentation License. Please also see http://en.wikipedia.org/wiki