Testing Web Sites with the curl and wget Utilities

Testing a Web site's performance using a Web browser alone is sometimes insufficient to get a good idea of the source of slow Web server performance. Many useful HTTP error codes are often not displayed by browsers, making troubleshooting difficult. A much better combination of tools is to use telnet to test your site's TCP port 80 response time in conjunction with data from the Linux curl and wget HTTP utilities.

Rapid TCP response times, but slow curl and wget response times, usually point not to a network issue, but to slowness in the configuration of the Web server or any supporting application or database servers it may use to generate the Web page.

Using curl

The curl utility acts like a text-based Web browser in which you can select to see either the header or complete body of a Web page's HTML code displayed on your screen.

A good start is to use the curl command with the -I flag to view just the Web page's header and HTTP status code. By not using the -I flag you will see all the Web page's HTML code displayed on the screen. Either method can provide a good idea of your server's performance:

     [root@ bigboy tmp]# curl -I www.linuxhomenetworking.com
     HTTP/1.1 200 OK
     Date: Tue, 19 Oct 2004 05:11:22 GMT
     Server: Apache/2.0.51 (Fedora)
     Accept-Ranges: bytes
     Vary: Accept-Encoding,User-Agent
     Connection: close
     Content-Type: text/html; charset=UTF-8

     [root@bigboy tmp]#

In this case the Web server appears to be working correctly because it returns a 200 OK code. Please refer to Chapter 20, "The Apache Web Server," for a more complete listing of possibilities.

Using wget

You can use wget to recursively download a Web site's pages, including the entire directory structure of the Web site, to a local directory.

By not using recursion and activating the time stamping feature (the -N switch), you view not only the HTML content of the Web site's index page in your local directory, but also the download speed, file size, and precise start and stop times for the download. This can be very helpful in providing a simple way to obtain snapshots of your server's performance:

     [root@zippy tmp]# wget -N www.linuxhomenetworking.com
     --23:07:22--  http://www.linuxhomenetworking.com/
                => `index.html'
     Resolving www.linuxhomenetworking.com... done.
     Connecting to www.linuxhomenetworking.com[65.115.71.34]:80...
     connected.
     HTTP request sent, awaiting response... 200 OK
     Length: unspecified [text/html]
     Last-modified header missing -- time-stamps turned off.
     --23:07:22-- http://www.linuxhomenetworking.com/
                => `index.html'
     Connecting to www.linuxhomenetworking.com[65.115.71.34]:80...
     connected.
     HTTP request sent, awaiting response... 200 OK
     Length: unspecified [text/html]

         [   <=>                                         ] 122,150
     279.36K/s

     23:07:22 (279.36 KB/s) - `index.html' saved [122150]

     [root@zippy tmp]#