lrzip v0.13

Long Range ZIP or Lzma RZIP

This is a compression program optimised for large files. The larger the file
and the more memory you have, the better the compression advantage this will
provide, especially once the files are larger than 100MB. The advantage can
be chosen to be either size (much smaller than bzip2) or speed (much faster
than bzip2). Decompression is much always faster than bzip2.

Lrzip uses an extended version of rzip which scales according to memory size.
The data is then either:
1. Compressed by lzma (default) which gives excellent compression
at approximately half the speed of bzip2 compression
2. Compressed by lzo which on most machines compresses faster than disk
writing making it as fast (or even faster) than simply copying a large file
3. Leaving it uncompressed and rzip prepared. This form improves substantially
any compression performed on the resulting file in both size and speed (due to
the nature of rzip preparation merging similar compressible blocks of data and
creating a smaller file).
4. Compressed by bzip2 as an rzip-like compression format.

The major disadvantages are:
1. It only works on single files. To get the best performance out of the
compression it is best to tarball all your files together.
2. It requires a lot of memory to get the best performance out of, and is not
really usable (for compression) with less than 256MB. Decompression requires
very little ram.
3. Does not work on stdin/stdout.
4. lzma is not 64bit ready (but lzo and bzip2 is, so read the FAQS).


Example on a 1GB ram P4 3GHz: 

A tarball of a fully compiled kernel tree:
		Size		Compression 	Decompression
base file:	646963200
gzip		218071923	1:27.27		0:45.39	
bzip2		192484690	4:41.62		1:41.20
bzip2 -1	215555795	3:24.08		1:21.45
bzip2 -9	192484690	4:53.18		1:31.40
lzma		112229937	11:48.07	0:56.38
lzma -9		97704505	27:18.77	?
lrzip		91544208	10:11.28	0:57.88
lrzip -l	191415649	0:30.19		0:50.69
lrzip -L 9	86595597	12:17.46	1:06.18
lrzip -n	389125460	0:31.02		0:58.9

Summary: 	Ratio		Value(Ratio/Time)
gzip		2.97		2.048
bzip2		3.36		0.717
bzip2 -1	3.00		0.883
bzip2 -9	3.36		0.688
lzma		5.76		0.488
lzma -9		6.62		0.242
lrzip		7.07		0.694
lrzip -l	3.38		6.760 *
lrzip -9	7.47 *		0.608
lrzip -n	1.66		3.222


Requires:
liblzo2-dev
libbz2-dev


To build/install:
./configure
make
make install


FAQS.

Q. I want the absolute maximum compression I can possibly get, what do I do?
A. The major limitation is ram, and the more ram you have the more compression
you will be able to achieve. The default values chosen by lrzip are used to 
not cause heavy swapping usage on the machine during compression. If you wish
to leave your machine unattended to do as much compression as possible, and
are happy to let it heavily hit swap you can do the following you can do the
following:
Make sure you have more swapspace than you have ram.
lrzip -w XXX -L 9 filename
in the XXX spot put 100MB less than you have total ram, eg -w 9 for a machine
with 1GB ram (gives a window of 900MB).

Q. Can I use your tool for even more compression than lzma offers?
A. Yes, the rzip preparation of files makes them more compressible by every
other compression technique I have tried. Using the -n option will generate
a .lrz file smaller than the original which should e more compressible, and
since it is smaller it will compress faster than it otherwise would have.

Q. How about 64bit?
A. It builds fine, but lzma compression/decompression is not 64bit safe as
lzma is primarily developed on windows machines and they don't have a 64bit
OS yet... lzo and bzip2 compression, on the other hand, works fine. Yes I know
it's counterproductive for a compression format designed for massive files to
not fully support 64bit.

Q. Other operating systems?
A. Patches are welcome. As far as I can tell, so far it will only build and
work on linux. The code shouldn't be hard to convert to *bsd.

Q. Can it be made to work on stdin/stdout?
A. The rzip design basically works in a way that makes this virtually
impossible.

Q. I have another compression format that is even better than lzma, can you
use that?
A. You can use it yourself on rzip prepared files (see above). Alternatively
if the source code is compatible with the GPL license it can be added to the
lrzip source code. Libraries with functions similar to compress() and
decompress() functions of zlib would make the process most painless.

Q. What's this "Progress percentage pausing during lzma compression" message?
A. While I'm a big fan of progress percentage being visible, unfortunately
lzma compression can't currently be tracked when handing over 100+MB chunks
over to the lzma library. Therefore you'll see progress percentage until
each chunk is handed over to the lzma library. lzo or no compression doesn't
have this problem and shows progress continuously.

Q. I Have truckloads of ram so I can compress files much better, but can my
generated file be decompressed on machines with less ram?
A. Yes. Ram requirements go up only by the -L compression option and are
never anywhere near as large as the compression requirements.

Q. Any plans to turn this into a complete archiver?
A. Not really. The compression format relies on being fed large files, and
tar does a good job of this already.

Q. I've changed the compression level with -L in combination with -l and the
file size doesn't vary?
A. That's right, -l only has one compression level.

Q. Help? I'm a newbie and have no idea how to turn my directory into a
tarball!
A. Here is a walkthrough for a directory called myfiles
to compress:
	tar cf myfiles.tar myfiles
	lrzip myfiles.tar
this will create a file called myfiles.tar.lrz
to extract:
	lrzip -d myfiles.tar.lrz
	tar xf myfiles.tar
will create and extract everything into a directory called myfiles

Q. Why are you including bzip2 compression?
A. To maintain a similar compression format to the original rzip (although the
other modes are more useful). Also it works on 64bit whereas lzma does not.

Q. What about multimedia?
A. Most multimedia is already in a heavily compressed "lossy" format which by
its very nature has very little redundancy. This means that there is not
much that can actually be compressed. If your video/audio/picture is in a
high bitrate, there will be more redundancy than a low bitrate one making it
more suitable to compression. None of the compression techniques in lrzip are
optimised for this sort of data. However, the nature of rzip preparation
means that you'll still get better compression than most normal compression
algorithms give you if you have very large files. ISO images of dvds for
example are best compressed directly instead of individual .VOB files.

Q. Is this multithreaded?
A. Short answer, no.
The main compression advantage of lrzip is that it uses most of the
available ram during compression. To compress with multiple threads or
processes would require just as much ram per thread. This means that any speed
advantage of multithreading would compromise the compression. In lzo 
compression, where speed is everything, the speed is not remotely limited by
cpu performance so adding threading to this would be unhelpful, and would
adversely affect compression.


Links:
rzip:
http://rzip.samba.org/
lzo:
http://www.oberhumer.com/opensource/lzo/
lzma:
http://www.7-zip.org/


Thanks to Andrew Tridgell for rzip. Thanks to Markus Oberhumer for lzo.
Thanks to Igor Pavlov for lzma. Thanks to Christian Leber for lzma
compat layer and everyone else who coded along the way.


Con Kolivas <kernel@kolivas.org>
Mon Oct 30 2006
