ReadMe for CS425 Assignment no 1

Application Name: mGet/v1.2.5 (Multi Threaded Wget)

Executable:
	Please run genmake to create appropriate makefile. If you do not want unnecessary messages change the genmake to make DEBUG= (no value).

Motivation:
	The basic motivation for mGet is the problems that are encountered in  downloading large files over a slow connection. There are many applications called Download Managers who counter this by breaking down the file into segments of smaller size and download them seperately from the server and join them together on the client side to form the final downloaded file.

Implementation:
	The program uses the GET command following the HTTP/1.1 (RFC2068) protocol to first query the server for the requested file using the RANGE header, when the response is either with a status code of 200 or 206(Partial Content) the file is retrieved or the appropriate error code is given following which the application terminates. If the status code is 200 then the server doesnot support partial content then the program generates only one thread which tries to retrieve the file. If the status code is 206, then the appropriate number of threads are created each of which opens a socket and retrieves the appropriate portions of the file. The retrieved data is stored temporarily in a 1024 byte buffer, when this fills up the data is written into the file on disk. Pthread mutexes and conditionals are used to synchronize file access and to make the parent thread to wait for the child threads. In case the connection breaks or times out the threads keep trying to retrieve the file until it is retrieved. The program supports proxy firewalls (Base64 encoding is used to give password to the proxy server).

Usage: <file_name> refers to the URL
	Some sample usage:
	$ mget [just run mget to get the usage]
	$ mget -f<file_name> [default number of threads=1]
	$ mget -n10 -f<file_name> -o<out_file_name>
	$ mget -n10 -H<http_proxy> -f<file_name>
	$ mget -n10 -F -f<file_name> [there is an environment variable ftp_proxy][getting value from environment variable
					does not work in SunOS due to a problem in their getopt() implementation]
	$ mget --use_http_proxy=<http_proxy> -f<file_name> -t<timeout value>
	$ mget -f<file_name> -R<referrer URL>
etc................	

The proxy options are optional and generally the -p option is not required since most servers use the default port. The maximum number of threads supported are 10 (max. 10 segments can be downloaded at a time.) On specifying the proxy options proxy authentication is asked for and the password is masked and encoded using the standard Base64 encoding (RFC1341). (Unlike wget, which shows the username password as a part of process, mGet kee[s this as internal variables])

Files:
	1)mget.h :header file
	2)mget.c :main C file
	3)mgetutil.c :some additional parsing, error reporting and encoding functions.
	4)genmake:run this file to generate appropriate makefile.
	5)README :this file
Samples:
	We have tried out the following runs and the files have downloaded properly:
mget -n3 -f www.iitk.ac.in
mget -n10 -f http://bms.cse.iitk.ac.in/sw/Zlib/zlib.tar.gz
mget -n3 -f www.google.com -R proxy.iitk.ac.in -T 3128
mget -n10 -f http://www.starwars.tierranet.com/lyworld/desktop/bubbles.zip -Rproxy -T3128
mget -n3 -f http://www.google.com/images/title_homepage4.gif -R proxy -T 3128

Assumptions:
	Some of the impicit assumptions made are:
1) The Range field in the http request is in "bytes". This is the most common but not necessary, so the program won't work on sites who donot specify Content-Range in bytes. (This can be easily changed.)
2) mGet can download files only from HTTP servers. (It follows HTTP/1.1 )
3) The HTTP headers are assumed to be within 1024 bytes in length. It is very rare to find longer headers (I haven't seen any).
4) The file is downloaded in the directory from which mGet is run. So any existing file with the same name is overwritten.
5) In case the complete_file_url doesnot specify a file (like www.iitk.ac.in or www.iitk.ac.in/) then the downloaded file is named 'index.html'.
6) The file to be downloaded must be given as complete_url (relative urls cannot be specified).

Group Members:

Bera, Debajyoti (98117)
Chakraborty, Arindam (98071)
Dutta, Chinmoy (98115)
Kumar, Ashutosh (98091)


BUG FIXES and CHANGES:
----------------------
Early August, 2001: added check to verify the number of segments; it does not make sense to download
                    small files in large segments. number of segments is reset, if required and told
		    to the end-user.
		    showing the download details is improved; it now shows the percentage of the segments
		    in a single line updating them frequently (the last method was horrible).
		    
17 August, 2001:    added the option to save the file in a different name. previously the name was 
                    taken from the URL; however it caused inconvenience and now one can specify the name.
		    changed the default number of segments to 1. no need of specifying numnber of segments
		    if only one segment would do (this however makes no sense except for testing as
		    if you donot need segmentsm then better use wget etc.)
		    Fixed Bug: number of segments =1 if file size is very small (earlier became 0 and caused fault)

1 September, 2001: Added timeout-support. Default timeout is 60 seconds (suitable for IIT-K Network). 
                   it can also an be given as option.  
		   now ftp:// links can also be given.
		
3 September, 2001: Added GNU style long option support for proxy name. Proxy can also be now given as environment variables. 

15 September, 2001: Removed a number of confusing bugs: timeout/retry broken pipe error, percentage>100% error.
                    :) Added easy way to change version
		    Changed the way the length and 206 support is calculated (need to verify it will cause less damage).
		    Changed Version to 1.3.0 ... seems stable enough.
		    Added #ifdef DEBUG ,,, no more rubbish degug info on the output.

21 September, 2001: Fixed a small but serious bug regarding finding the size of the incoming file.
                    Added a makefile generator - support of SunOS (to be tested ) and Linux.
		    Fixed a bug due to no envirinment variable.
                    Fixed a bug causing problem in SunOS (empty string). 
		    In case of redirection, (I faced this problem while trying to
		    download a file from freshmeat !!!), report the user of the
		    new address (enhancements later).
		    Added support to specify referrer. (needed in some cases to download from Yahoo Briefcase)
		    Now if server doesnot support resume, timeout is increased.
		    Fixed a bug for resuming when server dows not support resume.

25 September, 2001: Now can download files whose size is not known.		    

3 October, 2001   : Made a small change so that ftp:// URLs can also be downloaded in threads
                    Please NOTE that: this does not mean multithreaded ftp, only that getting files from ftp
		    via proxy server would help.

11 October, 2001  : Fixed a small but important bug, which caused corrupted download due to incorrect range request.		    
