Sit 'n' click to pull down page after page is a real pain, watching paint dry is way more fun. That's where wget solves the problem. I pulled down around 90 pages recently, wget did it in a little over 3 minutes. Can you beat that with the mouse? I don't think so!
If you have never used wget, here's a quick over view.
First check you have wget installed. In an XTerm, do:
You should get something like:
If its not, you know the routine.
aptitude install wget
wget has several switches (sometimes also called options), the ones I mostly use are:
wget -r -l2 -k -t5 -p http://www.oddwebsite.com
-r Turn on recursive retrieves.
-l Maximum recursion level. Default is 5.
-k After convert links to allow local viewing.
Affects visible hyperlinks, or parts linking to
external content, ie images, style sheets, etc.
-t Number of retries. Zero for infinite retries.
Default is 20. Fatal errors are not retried.
"connection refused'' or "not found'' (404).
-p Download all files to properly display page.
images, sounds, stylesheets.
>wget -r -l1 -t1 -nd -N -U mozilla -np -A.mp3 -erobots=off -i mp3_sites.txt
-r Turn on recursion.
-l1 Go down one level below entry level.
-t1 Number of retries is 1.
-nd Don't create directory for downloaded files.
-N Turn on time-stamping.
-U mozilla. Identify as wget as mozilla to the
Some sites require browser identity.
If you get that problem, use -U switch.
-np No parent content. Don't include parent level
-A Get all .mp3 files. Similar to "globbing".
-e Execute command.
Ignore robots.txt file.
Read URLs from mp3_sites.txt file.
If '-' used, URLs are read from command line.
wget is such a powerful utility. Mouse and browser are good, but the command line wins again. Check out the man page, for examples and other goodies.