Data compression algorithms and tools

From Archiveteam
Jump to navigation Jump to search

This list contains the most popular data compression algorithms and tools. All of them are free and open source, an important detail if you want to preserve data for a long time from now and to be able to decompress the data in the future.

General purpose compression



    • Not strong but fast and very widely supported.
    • Pre-installed on pretty much every Linux computer.



Aside from 7z, there's also

  • xz
    • Based on LZMA SDK
    • Commonly included by default in Linux distros
  • lzip
    • Claims to be stronger yet faster than bzip2.
    • Not as widely supported as xz, bzip2, and gzip.
    • Well defined file format and emphasis on file integrity
    • lziprecover can correct some bit-flip errors and merge damaged copies.


  • Available by default in any Windows version available today, but if you need cross-platform, use 7-zip.


  • Very efficient in both time and compression ratio.
  • First-class support for custom dictionaries, which is particularly useful when compressing many small data units (e.g. WARC file with many HTML pages from one particular website). Using a trained dictionary for the compression massively improves the compression ratio in such scenarios.

Heavy duty compression

These programs often use large amounts of memory to get the best possible compression ratio.


"This is a compression program optimised for large files" -lrzip readme

lrzip is fantastic for archiving - the compression ratio improves as the size of the input file grows - albeit a terribly slow compressor. lrzip really shines when compressing large sets of redundant information - but distant, and otherwise unconnected. General purpose compression algorithms would never see this, given their tiny compression window.

lrzip benchmarks


  • Uses deduplication, journaling, and several different compression algorithms (LZ77, BWT, and PAQ context mixing)
  • Supported by lrzip
  • EXTREMELY slow


Uses the PAQ6 compression algorithm. Excellent compression ratio (better than 7z), but a bit slow.

You can install it in Ubuntu with: sudo apt-get install kgb

How to:

  • kgb -m file.kgb originalfile
  • m is a number from 0 to 9 (lowest compression ratio from higher; higher use 1616 MB of RAM, a lot of CPU and time)

External links