What is wget?
Wget is a free GNU command-line utility tool used to download files from the internet. It retrieves files using HTTP, HTTPS, and FTP protocols.
It serves as a tool to sustain unstable and slow network connections. If a network problem occurs during a download, this helpful software can resume retrieving the files without starting from scratch.
Another important aspect is its capability of recursive downloads, with which it mirrors websites. It transfers parts of a website by following links and directory structure, thus creating local versions of webpages.
The wget command is also highly flexible and can be used in terminals, scripts, and cron jobs. During the download, the user does not have to be active nor logged in. As wget is non-interactive, it can independently run in the background.
Read this article to learn how to use some of the most common wget commands.
How to Check if wget is Installed?
Most likely, the wget package is already on your system as it now comes pre-installed on most Linux distributions.
To check, open the terminal window and type in:
wget
If you have the wget software, the output tells you that the wget command is missing a URL, as shown in the image below:
wget Command Not Found
If the output displays wget command not found you need to download and install the tool manually. Below you will find the installation instructions for Ubuntu/Debian, CentOS, and Windows.
How to Install wget on Ubuntu/Debian?
To install wget on Ubuntu or Debian releases, use the command:
sudo apt-get install wget
How to Install wget on CentOS/Fedora?
To install wget on CentOS or Fedora, type the following command:
sudo yum install wget
How to Install wget on Windows?
To install and configure wget for Windows:
- Download wget for Windows and install the package.
- Add the wget bin path to environment variables (optional). Configuring this removes the need for full paths, and makes it a lot easier to run wget from the command prompt:
- Open the Start menu and search for “environment.”
- Select Edit the system environment variables.
- Select the Advanced tab and click the Environment Variables button.
- Select the Path variable under System Variables.
- Click Edit.
- In the Variable value field add the path to the wget bin directory preceded by a semicolon (;). If installed in the default path, add
C:Program Files (x86)GnuWin32bin.
- Open the command prompt (cmd.exe) and start running
wgetcommands.
Introduction to wget Syntax
The wget syntax has the following pattern:
wget [option][URL]
Each [option] has its long and short form which are conveniently interchangeable. This attribute specifies what to do with the URL that follows.
[URL] is the address of the file or directory you wish to download.
wget Command Examples
Download File from Web
To download a file from the web use:
wget [URL]
For example, to install Tomcat 9, first you need to download the package with wget using the command:
wget http://apache.cs.utah.edu/tomcat/tomcat-9/v9.0.20/bin/apache-tomcat-9.0.20.tar.gz
Download File and Save Under Specific Name
To download a file and save it under a specified name run:
wget -O [file_name] [URL]
The wget command allows you to rename files prior to downloading them on your computer.
For instance, you may want to install Terraform. To download the package and rename it terraform.zip use the following command:
wget -O terraform.zip https://releases.hashicorp.com/terraform/0.12.2/terraform_0.12.2_linux_amd64.zip
Download File to Specific Directory
By default wget downloads a file in the directory the user is in. To save the file in a different location, add the -P option:
wget -P [wanted_directory] [URL]
For example, while installing Git on Ubuntu, you can download the package in the /temp directory with the command:
wget -P /temp https://github.com/git/git/archive/master.zip
Set Download Speed
You can set the download speed when downloading a big file, so it does not use the full available bandwidth. The download speed is defined in kilobytes (k) and megabytes (m). Use the command:
wget --limit-rate [wanted_speed] [URL]
For example, if you are installing NVIDIA TESLA drivers on Linux and want to limit the download speed to 1 megabyte, would use the command:
wget --limit-rate 1m http://us.download.nvidia.com/tesla/396.37/nvidia-diag-driver-local-repo-ubuntu1710-396.37_1.0-1_amd64.deb
Continue Download After Interruption
Instead of having to start from scratch, wget can resume downloading where it stopped before the interruption. This is a useful feature if there is a lost of connection while downloading a file.
wget -c [URL]
For instance, you may want to install a Mumble Server on Linux and suddenly lose internet connection while downloading the installation file. To continue downloading, type in the command:
wget -c https://github.com/mumble-voip/mumble/releases/download/1.2.19/murmur-static_x86-1.2.19.tar.bz2
Download Multiple Files
wget allows downloading multiple files at the same time using the command:
wget -i [file_name]
To do so, follow the steps outlined below:
1. First, create and open a file under the name MultipleDownloads.txt (or a name of your choice), using a text editor. In this case, we used Nano:
nano MultipleDownloads.txt
2. Once in the editor, add the URLs of the packages you want to download, one per line.
3. Save and exit the file.
4. Run the following wget command in the terminal window:
wget -i MultipleDownloads.txt
This prompts wget to download from each URL in the text file.
Download Web page (Mirror Web page)
With wget you can download an entire website from the internet, using the -m option. It prompts wget to create a mirror of the specified webpage. The basic command for doing so is:
wget -m [URL]
For example:
wget -m <a href="https://phoenixnap.com/" target="_blank" rel="noreferrer noopener">https://phoenixnap.com</a>
Download via FTP
To download via FTP, type in the username and password of the FTP server, followed by the ftp address:
wget --ftp-user=[ftp_username] --ftp-password=[ftp_password] ftp://...
For instance:
wget --ftp-user=sofiftp --ftp-password=TopSecretPassword ftp://123.456.7890
Download in Background
You can download in the background, a practical feature when dealing with a large file:
wget -b [URL]
You can check the status of the download with the command:
tail -f wget -log
To download the RPM package manager in the background, type:
wget -b http://some_website/sample_file.rpm
Increase Retry Attempts
You can set how many times wget attempts to download a file after being interrupted by a bad network with:
wget --tries=[number_of_tries] [URL]
By default, the number of retry attempts is set to 20.
You can also set the number to infinity with the values 0 or inf, as in the following example:
wget --tries=inf http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo
Skip Certificate Check
By default, wget checks whether the server has a valid SSL/TLS certificate. If it does not identify an authentic certificate, it refuses to download.
The --no-check-certificate option is used to avoid certificate authorities checking for a server certificate. However, utilize it only if you are sure of the website’s credibility or are not worried about security issues it may cause.
wget --no-check-certificate [URL]
If http://enteratonerisk.com has an untrusted certificate, but will not harm the system, you can download it with:
wget --no-check-certificate http://enteratonerisk.com
Change User Agent
When downloading a webpage, wget essentially emulates a browser. In some cases, the output might say you don’t have permission to access the server, or that the connection is forbidden. This may be due to a website blocking client browsers that have a specific “User-Agent.”
“User-Agent” is a header field that the browser sends to the server it wants to access. Therefore, to download from a server that is refusing to connect, try to modify the user agent.
Find a database of all user agents online, search for the one you need and run the command:
wget --user-agent="User Agent Here" "[URL]"
or
wget -U "User Agent Here" "[URL]"
For example, to emulate Chrome (version 74), you would change the user agent with the command:
wget --user-agent=" Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "https://phoenixnap.com"
Note: Make sure you always download from a trusted source because wget can instruct a download of a script from a malicious source. We outline this and many other dangerous commands in our article 14 Dangerous Linux Terminal Commands.
Conclusion
This article sums up why wget is such a powerful tool for downloading files over the internet. It also serves as a good reference for beginners with its list of 12 essential wget commands and examples.
Credit: phoenixnap.com/kb/wget-command-with-examples