WinInet: implementing resuming feature

	WinInet: implementing resuming feature

Submitted on July 29, 2003

Abstract

This article describes the most efficient way to continue downloading a file from a point where it stoped and organize the mutithreaded downloading feature using the WinInet library.

When you download a file or some other resource from the web server, sometimes connection errors may occur. These lead to connections being lost and forces you to reconnect and start the downloading process again. From here it would be a great idea if you could continue the downloading of the file from the point of interruption.

In order to implement this functionality you need a method which allows you to position the remote resource pointer at some random place inside the resource. Lets go and consider the WinInet library and its interfaces which provides users with the possibility of organizing the downloading process from the point where it stops (named the resuming feature).

Set a resource position

The Microsoft WinInet library provides users with the InternetSetFilePointer function which sets a file position for the InternetReadFile function. You can call up InternetSetFilePointer the next time when you are about to continue downloading in order to move to the last error place. The sample code below demonstrates how to use this function for the resuming issue:

var
   hOpen, hConnect, hResource: HINTERNET;
   DataProceed: array[0..8191] of Byte;
   numread: DWORD;
begin
   hOpen := InternetOpen('WinInet resuming sample', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
   hConnect := InternetConnect(hOpen, PChar(host), INTERNET_DEFAULT_HTTP_PORT, nil, nil, INTERNET_SERVICE_HTTP, 0, 0);
   hResource := HttpOpenRequest(hConnect, 'GET', PChar(resource), nil, nil, nil, 0, 0);
   HttpSendRequest(hResource, nil, 0, nil, 0);

   InternetSetFilePointer(hResource, Position, nil, FILE_BEGIN, 0)
   releat
      InternetReadFile(hResource, @DataProceed, SizeOf(DataProceed), numread);
      ...
   until (numread <= 0);

   InternetCloseHandle(hConnect);
   InternetCloseHandle(hOpen);
end;

Random resource access

There are some serious restrictions in using of the InternetSetFilePointer function. Among them the demand that the WinInet cache should be enabled. As a result you have two copies of the resource - one of them is in the local file to which you store the data and the other one within the Internet Temporary Files. In most cases this is not a problem. So if you are about to implement the resource resuming feature only, you can freely use the InternetSetFilePointer function without any problems.

Another thing you will not be able to peform with this mechanism is: simultaneous resource access from different threads. This method is used in multithreaded downloading mode. The multipart multithreaded downloading allows you to improve the performance, by fully using the TCP channel.

Lets consider it in more detail. Suppose you are about to start two downloading threads. The first thread starts the downloading from beginning of the file while the second thread starts from the middle of the same file. The second thread should set the file position but the InternetSetFilePointer function is unable to perform it because the whole of the previous file content is not yet cached in the WinInet cache.

To work around this problem lets use another approach - range request specifing.

With using the HttpAddRequestHeaders WinInet function it is possible to set-up the additional request header which contains the range of data in bytes you are about to retrieve from server. The sample below demonstrates using of HttpAddRequestHeaders:

procedure TMyThread.SetResourcePos(Resource: HINTERNET; ResourcePos, BytesToProceed: Integer);
var
   s: string;
begin
   s := Format('Range: bytes=%d-%d', [ResourcePos, ResourcePos + BytesToProceed]);
   HttpAddRequestHeaders(Resource, PChar(s), Length(s), HTTP_ADDREQ_FLAG_ADD_IF_NEW);
end;

The sample code which allows to start the downloading with random data range is shown below:

var
   hOpen, hConnect, hResource: HINTERNET;
   s: string;
   DataProceed: array[0..8191] of Byte;
   numread: DWORD;
begin
   hOpen := InternetOpen('WinInet resuming sample', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
   hConnect := InternetConnect(hOpen, PChar(host), INTERNET_DEFAULT_HTTP_PORT, nil, nil, INTERNET_SERVICE_HTTP, 0, 0);
   hResource := HttpOpenRequest(hConnect, 'GET', PChar(resource), nil, nil, nil, 0, 0);

   s := Format('Range: bytes=%d-', [Position]);
   HttpAddRequestHeaders(hResource, PChar(s), Length(s), HTTP_ADDREQ_FLAG_ADD_IF_NEW);
   HttpSendRequest(hResource, nil, 0, nil, 0);
   repeat
      ZeroMemory(@DataProceed, SizeOf(DataProceed));
      InternetReadFile(hResource, @DataProceed, SizeOf(DataProceed), numread);
      ...
   until (numread <= 0);

   InternetCloseHandle(hConnect);
   InternetCloseHandle(hOpen);
end;

As you can see an example above works much faster than example with the InternetSetFilePointer.

Conclusion

All the algorithms we considered in this paper are operable only when the web site from which the downloadable file belongs supports the random resource access. If not, the "range" request being sent to the web server will be ignored and the whole amount of data will be sent to the client. Before using the range request header or an algorithm with InternetSetFilePointer call, please check whether the web server being connected to supports the random access feature.

In order to perform this check in the easiest way you can use the following code:

AllowsRandomAccess := Integer(InternetSetFilePointer(hResource, 0, nil, FILE_END, 0)) > 0;

Also please note that InternetSetFilePointer cannot be used reliably if the content length is unknown. So you should also perform this check as follows:

var
   reslen: Integer;
   buflen, tmp: DWORD;
begin
   buflen := SizeOf(reslen);
   tmp := 0;
   reslen = 0;
   HttpQueryInfo(hResource, HTTP_QUERY_CONTENT_LENGTH or HTTP_QUERY_FLAG_NUMBER,
      @reslen, buflen, tmp);
   AllowsRandomAccess := (reslen > 0);
end;

Retrieving of the Content-Lenght resouce information is very important for range requests too when they are used for multipart multithreaded downloading mode. If you are simply organizing the resuming of the download after an error has occured you can simple specify the first range value and continue to retrieve the data from error point up to the end.

The full source code of samples being used in this article can be downloaded at:

Please note that WinInet error handling was not implemented in these examples in order to keep them as simple as possible. No warning messages will appear in case of invalid URL. You can easily implement error handling if such issue is important to you.

While writing this article we used the following resources:

http://msdn.microsoft.com - Microsoft MSDN Library
http://www.w3.org/Protocols/rfc2616/rfc2616.html - Hypertext Transfer Protocol -- HTTP/1.1 (RFC 2616)

This code is constantly being improved and your comments and suggestions are always welcome.

With best regards,
Sergey Shirokov
Clever Components team.