Difference between revisions of "Data compression algorithms and tools"

From Archiveteam
Jump to navigation Jump to search
m (Reverted edits by Megalanya1 (talk) to last revision by Start)
(more information)
Line 28: Line 28:


=== lrzip ===
=== lrzip ===
"This is a compression program optimised for large files" -[http://ck.kolivas.org/apps/lrzip/README lrzip readme ]
"This is a compression program optimised for large files" -[http://ck.kolivas.org/apps/lrzip/README.md lrzip readme ]


lrzip is fantastic for archiving - the compression ratio '''improves''' as the size of the input file grows - albeit a  '''''terribly''''' slow compressor. lrzip really shines when compressing large sets of redundant information - but distant, and otherwise unconnected. General purpose compression algorithms would never see this, given their tiny compression window.  
lrzip is fantastic for archiving - the compression ratio '''improves''' as the size of the input file grows - albeit a  '''''terribly''''' slow compressor. lrzip really shines when compressing large sets of redundant information - but distant, and otherwise unconnected. General purpose compression algorithms would never see this, given their tiny compression window.  
Line 37: Line 37:
* http://mattmahoney.net/dc/zpaq.html
* http://mattmahoney.net/dc/zpaq.html
* Uses deduplication, journaling, and several different compression algorithms (LZ77, BWT, and PAQ context mixing)
* Uses deduplication, journaling, and several different compression algorithms (LZ77, BWT, and PAQ context mixing)
* Supported by lrzip
* EXTREMELY slow


=== KGB ===
=== KGB ===

Revision as of 13:36, 23 June 2021

This list contains the most popular data compression algorithms and tools. All of them are free and open source, an important detail if you want to preserve data for a long time from now and to be able to decompress the data in the future.

General purpose compression

7z

Bzip2

Gzip

LZMA/LZMA2

Aside from 7z, there's also

  • xz
    • Based on LZMA SDK
    • Commonly included by default in Linux distros
  • lzip
    • Not as widely used as xz
    • Well defined file format and emphasis on file integrity
    • lziprecover can correct some bit-flip errors and merge damaged copies.

Zip

  • Available by default in any Windows version available today, but if you need cross-platform, use 7-zip.

Heavy duty compression

These programs often use large amounts of memory to get the best possible compression ratio.

lrzip

"This is a compression program optimised for large files" -lrzip readme

lrzip is fantastic for archiving - the compression ratio improves as the size of the input file grows - albeit a terribly slow compressor. lrzip really shines when compressing large sets of redundant information - but distant, and otherwise unconnected. General purpose compression algorithms would never see this, given their tiny compression window.

lrzip benchmarks

ZPAQ

  • http://mattmahoney.net/dc/zpaq.html
  • Uses deduplication, journaling, and several different compression algorithms (LZ77, BWT, and PAQ context mixing)
  • Supported by lrzip
  • EXTREMELY slow

KGB

Uses the PAQ6 compression algorithm. Excellent compression ratio (better than 7z), but a bit slow.

You can install it in Ubuntu with: sudo apt-get install kgb

How to:

  • kgb -m file.kgb originalfile
  • m is a number from 0 to 9 (lowest compression ratio from higher; higher use 1616 MB of RAM, a lot of CPU and time)

External links