Extracting complete list of URL's from a Web Site
Submitted on February 24, 2003, Updated on September 20, 2006
This article describes how to extract a list of all web resources (URL's) from a Web Site using the Clever Internet Suite Internet components.
Currently the newer version of the Clever Internet Suite 6.2 has been released. Please see the Notes at the end of this article to update the URL Extractor with this new version 6.2
There is big demand for applications that allow look up through the web server (URL) and collect list of all web resources available from that web server. Most famous programs with such functionality probably would be Teleport Pro and Flash Get.
The general idea however is very simple - download a web page, parse it and extract all links and urls.
In this article we will discuss very simple and generic algorithm based on recursive downloading and parsing web pages in asynchronous mode.
As we mentioned above the main steps of algorithm are:
The first step can be implemented with Clever Downloader component. TclDownloader component in addition to base functionality provided by another popular libraries (such as Indy, IPWorks and so on) allows you to download web page / URL in asynchronous mode without interfering with main application process. After downloading process completed the OnProcessCompleted component event occurs. In order to implement recursive downloading we also need to use OnIsBusyChanged component event. OnIsBusyChanged event will protect us from stack overflow during crawling through server URL's.
Here is the code for the first step of our algorithm:
Main loop within the ProcessNextURL method is searching for next URL in URL list and the important thing is that URL should looks like html page (in this example we just check for page extension). We simplified this method but if you need more advanced analysis you can easily modify it according your needs.
After downloading completed the OnIsBusyChanged event occurs. Here is a code for this event:
You can extract URLs from downloaded web page with any method you like the most. In our example we used simple page parsing. Full source code can be downloaded at urlextractor.zip.
When parsing URL content please pay attention to the following issues:
In order to check if link belongs to the web site on which this web page is hosted you can simply compare host part of both URL's (for example http://www.site.com and http://www.site.com/index.asp). Clever Internet Suite has TclURLParser class specially designed for this purpose.
Given example is far from perfect and might reqire additional alterations. You can always enhance this code to desired functionality:
Clever Internet Suite 6.2 Notes
In the newer version 6.2 we have improved the URL parsing algorithm and applied some code modifications. Please check it before compiling our samples:
Please declare the clUriUtils unit within your uses list and re-compile your program. You can get all class and method declarations for the Clever Internet Suite library at inetsuitehdr.zip
With Html Parser component you can replace your code as follows: