Jump to content

7z: Difference between revisions

From Wikipedia, the free encyclopedia
Content deleted Content added
mNo edit summary
rm list of mostly nonnotable software
 
(556 intermediate revisions by more than 100 users not shown)
Line 1: Line 1:
{{short description|Family of archive file formats used by 7-Zip}}
In [[computing]], '''7z''' is an archive format which allows [[Data compression|data compression]] using a number of compression [[algorithm]]s. The format was initially implemented by the [[7-Zip]] archiver but the file format is public and 7-Zip's implementation of it is publicly available under the [[GNU LGPL]] licence.
{{use dmy dates|date=July 2020}}
{{other uses|7Z (disambiguation)}}
{{Infobox file format
| name = 7z file format
| icon = 7zip archive icon.svg
| extension = .7z
| mime = application/x-7z-compressed
| uniform type = org.7-zip.7-zip-archive
| owner = [[Igor Pavlov (programmer)|Igor Pavlov]]<ref name="dobb">{{cite web |url=http://www.ddj.com/architect/184405338 |title=A Few Questions for Igor Pavlov |date=2003-04-30 |publisher=[[Dr. Dobb's]] Data Compression Newsletter |access-date=2009-12-26 |archive-date=28 October 2008 |archive-url=https://web.archive.org/web/20081028091651/http://www.ddj.com/architect/184405338 |url-status=live }}</ref>
| released = {{Start date and age|1999}}<ref name="History">{{Cite web |url=http://www.7-zip.org/history.txt |title=History of 7-zip changes |access-date=10 June 2010 |archive-date=27 February 2015 |archive-url=https://web.archive.org/web/20150227213935/http://www.7-zip.org/history.txt |url-status=live }}</ref>
| creatorcode =
| genre = [[Data compression]]
| magic = '7', 'z', 0xBC, 0xAF, 0x27, 0x1C
| max_size = 2<sup>64</sup> bytes (roughly 18 [[exabytes]])
| containerfor =
| containedby =
| extendedfrom =
| extendedto =
| open = Yes: [[GNU Lesser General Public License]] / [[Public domain]]
| url = {{URL|7-zip.org}}


}}
The main features of the 7z format are:
* Open, modular architecture (which allows any compression, conversion, or encryption method to be used)
* High compression ratio (depending on which compression method is used)
* Strong [[AES]]-256 encryption
* Support for large files (up to approximately 16 [[exabyte]]s)
* [[Unicode]] file names
* Compression of archive headers
* allows compact archives: As with e.g. RAR and TGZ, one compressed data stream contains several files, as opposed to e.g. ZIP, where the archive file contains a separate compressed data stream for each archived file.


'''7z''' is a compressed [[archive file format]] that supports several different [[data compression]], [[encryption]] and pre-processing algorithms. The 7z format initially appeared as implemented by the [[7-Zip]] archiver. The 7-Zip program is publicly available under the terms of the [[GNU Lesser General Public License]]. The LZMA SDK 4.62 was placed in the [[public domain]] in December 2008. The latest stable version of 7-Zip and [[Lempel-Ziv-Markov chain algorithm|LZMA]] SDK is version 24.08.<ref name="History"/>
The format's open architecture allows additional compression methods to be added to the standard. The following are defined in the current version:
* [[LZMA]] - Improved and optimized version of [[LZ77]] algorithm, using [[Markov chain|Markov chains]]/[[entropy coding]] and [[Patricia trie|Patricia tries]].
* [[PPMD]] - Dmitry Shkarin's 2002 PPMdH (PPMII/cPPMII) with small changes: PPMII is an improved version of the 1984 [[PPM compression algorithm]] (prediction by partial matching).
* BCJ - Converter for 32-bit x86 executables, see [[LZMA]]. Compresses target addresses of near jumps and calls.
* BCJ2 - Converter for 32-bit x86 executables, see [[LZMA]]. Jump, call and conditional jump targets addresses are compressed separately.
* [[Bzip2]] - Standard [[Burrows-Wheeler transform]] algorithm. Bzip2 uses (faster) Huffman coding, while Bzip is using (stronger) entropy coding.
* [[DEFLATE (algorithm)|DEFLATE]] - Standard [[LZ77]]-based algorithm


The 7z file format specification is distributed with 7-Zip's source code since 2015. The specification can be found in plain text format in the 'doc' sub-directory of the source code distribution.<ref>LZMA SDK, "DOC" directory, 7zFormat.txt</ref>
7z also supports encryption with the [[AES]] algorithm with a 256-bit key. The key is generated from a user-supplied passphrase using an algorithm based on the [[SHA|SHA-256]] hash algorithm (with a large number of iterations, to make a brute-force search for the passphrase more difficult).


== Features and enhancements ==
The MIME type of 7z is <code>application/x-7z-compressed</code>.
The 7z format provides the following main features:
* [[Open format|Open]], modular architecture that allows any compression, conversion, or encryption method to be stacked.
* High [[data compression ratio|compression ratio]]s (depending on the compression method used).
* [[Advanced Encryption Standard|AES]]-256 bit [[encryption]].
* Zip 2.0 (Legacy) Encryption
* Large file support (up to approximately 16 [[exbibyte]]s, or 2<sup>64</sup> bytes).
* [[Unicode]] file names.
* Support for [[solid compression]], where multiple files of similar type are compressed within a single stream, in order to exploit the combined redundancy inherent in similar files.
* Compression and encryption of archive [[header (computing)|headers]].
* Support for multi-part archives : e.g. xxx.7z.001, xxx.7z.002, ... (see the context menu items ''Split File...'' to create them and ''Combine Files...'' to re-assemble an archive from a set of multi-part component files).
* Support for custom codec plugin DLLs.


The format's [[open architecture]] allows additional future compression methods to be added to the standard.
Other compressors which use PPMII: [[RAR]]


=== Compression methods ===
Compressors which encode branch targets to improve program file compression ratio:
The following compression methods are currently defined:
* UPX - for x86 16- and 32-bit binaries.
* 7-Zip - for x86 32-bit binaries, ARM, ARMThumb, PPC and IA64.
* RAR - for x86 32-bit and for IA64 Itanium. Also has preprocessors for true color data, audio data and tables.


* [[LZMA]]&nbsp;– A variation of the [[LZ77 and LZ78|LZ77]] algorithm, using a sliding dictionary up to 4 GB in length for duplicate string elimination. The LZ stage is followed by [[entropy coding]] using a [[Markov chain]]-based [[range encoding|range coder]] and [[binary tree]]s.
* [[Lempel–Ziv–Markov chain algorithm#LZMA2 format|LZMA2]]&nbsp;– modified version of LZMA providing better multithreading support and less expansion of incompressible data.<ref name="lzma2_source_code">{{cite web
| url = http://jpf91.github.io/lzmad/enwiki/api/lzma_lzma.html
| title = lzma_.lzma
| work = liblzma bindings
| first = Lasse
| last = Collin
| access-date = 2010-01-03
| quote = Compared to LZMA1, LZMA2 adds support for LZMA_SYNC_FLUSH, uncompressed chunks (smaller expansion when trying to compress uncompressible data), possibility to change lc/lp/pb in the middle of encoding, and some other internal improvements.
| archive-url= https://web.archive.org/web/20100208075245/https://www.google.com/codesearch/p?hl=en| archive-date= 8 February 2010 | url-status= live}}</ref>
* [[Bzip2]]&nbsp;– The standard [[Burrows–Wheeler transform]] algorithm. Bzip2 uses two reversible transformations; BWT, then [[Move to front]] with [[Huffman coding]] for symbol reduction (the actual compression element).
<!-- [[Bzip]] used (stronger, but patented) [[arithmetic coding]]. No point mentioning this since 7z doesn't use it! -->
* [[Prediction by Partial Matching|PPMd]]&nbsp;– Dmitry Shkarin's 2002 PPMdH (PPMII (Prediction by Partial matching with Information Inheritance) and cPPMII (complicated PPMII)) with small changes: PPMII is an improved version of the 1984 [[PPM compression algorithm]] (prediction by partial matching).
* [[DEFLATE]]&nbsp;– Standard algorithm based on 32 kB [[LZ77 and LZ78|LZ77]] and [[Huffman coding]]. Deflate is found in several file formats including [[ZIP (file format)|ZIP]], [[gzip]], [[Portable Network Graphics|PNG]] and [[PDF]]. 7-Zip contains a from-scratch DEFLATE encoder that frequently beats the ''de facto'' standard [[zlib]] version in compression size, but at the expense of CPU usage.

A suite of recompression tools called AdvanceCOMP contains a copy of the DEFLATE encoder from the 7-Zip implementation; these utilities can often be used to further compress the size of existing [[gzip]], [[ZIP (file format)|ZIP]], [[Portable Network Graphics|PNG]], or [[Multiple-image Network Graphics|MNG]] files.

=== Pre-processing filters ===
The LZMA SDK comes with the [[BCJ (algorithm)|BCJ]] and [[BCJ2]] preprocessors included, so that later stages are able to achieve greater compression: For [[x86]], [[ARM architecture|ARM]], [[PowerPC]] (PPC), IA-64 [[Itanium]], and [[ARM Thumb]] processors, jump targets are 'normalized' <ref name="lzma2_source_code" /> before compression by changing relative position into absolute values. For x86, this means that near jumps, calls and conditional jumps (but not short jumps and conditional jumps) are converted from the machine language "jump 1655 bytes backwards" style notation to normalized "jump to address 5554" style notation; all jumps to 5554, perhaps a common subroutine, are thus encoded identically, making them more compressible.

*[[BCJ (algorithm)|BCJ]]&nbsp;– Converter for 32-bit x86 executables. Normalise target addresses of near jumps and calls from relative distances to absolute destinations.
*BCJ2– Pre-processor for 32-bit x86 executables. BCJ2 is an improvement on BCJ, adding additional x86 jump/call instruction processing. Near jump, near call, conditional near jump targets are split out and compressed separately in another stream.
*[[Delta encoding]]&nbsp;– delta filter, basic preprocessor for multimedia data.

Similar executable pre-processing technology is included in other software; the [[RAR (file format)|RAR]] compressor features displacement compression for 32-bit x86 executables and IA-64 executables, and the [[UPX]] runtime executable file compressor includes support for working with 16-bit values within [[DOS]] binary files.

=== Encryption ===
The 7z format supports [[encryption]] with the [[Advanced Encryption Standard|AES]] algorithm with a 256-bit key. The key is generated from a user-supplied [[passphrase]] using an algorithm based on the [[SHA-256]] hash function. The SHA-256 is executed 2<sup>19</sup> (524288) times,<ref>{{Cite web |url=https://sourceforge.net/projects/sevenzip/?source=directory |title=7-zip source code |access-date=23 March 2018 |archive-date=22 March 2019 |archive-url=https://web.archive.org/web/20190322095659/https://sourceforge.net/projects/sevenzip/?source=directory |url-status=live }}</ref> which causes a significant delay on slow PCs before compression or extraction starts. This technique is called [[key stretching]] and is used to make a [[brute-force search]] for the passphrase more difficult. Current GPU-based, and custom hardware attacks limit the effectiveness of this particular method of key stretching,<ref name="percival2009">[[Colin Percival]].
[http://www.tarsnap.com/scrypt.html scrypt] {{Webarchive|url=https://web.archive.org/web/20190528073159/https://www.tarsnap.com/scrypt.html |date=28 May 2019 }}.
As presented in
[http://www.tarsnap.com/scrypt/scrypt.pdf "Stronger Key Derivation via Sequential Memory-Hard Functions"] {{Webarchive|url=https://web.archive.org/web/20190414144147/http://www.tarsnap.com/scrypt/scrypt.pdf |date=14 April 2019 }}.
presented at BSDCan'09, May 2009.</ref> so it is still important to choose a strong password.
The 7z format provides the option to encrypt the filenames of a 7z archive.

=== Limitations ===
The 7z format does not store [[filesystem permissions]] (such as [[UNIX]] owner/group permissions or [[NTFS]] [[Access control list|ACL]]s), and hence can be inappropriate for backup/archival purposes. A workaround on UNIX-like systems for this is to convert data to a [[Tar (file format)|tar bitstream]] before compressing with 7z. But GNU tar (common in many UNIX environments) can also compress with the LZMA2 algorithm ("[[XZ Utils|xz]]") natively, without the use of 7z, using the "-J" switch. The resulting file extension is ".tar.xz" or ".txz" and not ".tar.7z". This method of compression has been adopted with many distributions for packaging, such as Arch, Debian (deb), Fedora (rpm) and Slackware. (The older "lzma" format is less efficient.)<ref>{{Cite web|url=https://www.gnu.org/software/tar/manual/html_section/Compression.html|title=GNU tar 1.34: 8.1 Using Less Space through Compression|access-date=17 March 2015|archive-date=2 April 2015|archive-url=https://web.archive.org/web/20150402103628/https://www.gnu.org/software/tar/manual/html_section/Compression.html|url-status=live}}</ref> On the other hand, it is important to note, that tar does not save the filesystem encoding, which means that tar compressed filenames can become unreadable if decompressed on a different computer.

The 7z format does not allow extraction of some "broken files"—that is (for example) if one has the first segment of a series of 7z files, 7z cannot give the start of the files within the archive—it must wait until all segments are downloaded. The 7z format also lacks recovery records, making it vulnerable to [[data degradation]] unless used in conjunction with external solutions, like [[Parchive|parchives]], or within [[File system|filesystems]] with robust [[Error correction code|error-correction]]. By way of comparison, [[zip (file format)|zip]] files also lack a recovery feature while the [[RAR_(file_format)|rar]] format has one.


== See also ==
== See also ==
*[[7-Zip]]
*[[Comparison of archive formats]]
*[[List of archive formats]]
*[[List of archive formats]]
*[[Open file format]]

==References==
{{Reflist}}

==Further reading==
* {{cite book |title=Data compression: the complete reference |last=Salomon |first=David |year=2007 |publisher=Springer |isbn=978-1-84628-602-5 |page=241}}


== External links ==
== External links ==
<!-- Per [[WP:ELMINOFFICIAL]], choose one official website only -->
* [http://www.7-zip.org/ 7-Zip homepage]
* {{Official Website|www.7-zip.org}}
* [http://p7zip.sourceforge.net/ 7-Zip for Unix]
* {{sourceforge|sevenzip}}
* [http://datacompression.info/Miscellaneous/PPMII_DCC02.pdf Dimtry Shkarin (Institute for Dynamics of Geospheres, Moscow, Russia) 2002: PPM: One Step to Practicality]


[[Category:Archive formats]]
{{Archive formats}}


[[Category:Computer-related introductions in 1999]]
[[de:7z]]
[[Category:Archive formats]]
[[es:7z]]
[[Category:Russian inventions]]
[[pl:7z]]

Latest revision as of 14:19, 29 December 2024

7z file format
Filename extension
.7z
Internet media type
application/x-7z-compressed
Uniform Type Identifier (UTI)org.7-zip.7-zip-archive
Magic number'7', 'z', 0xBC, 0xAF, 0x27, 0x1C
Size limitation264 bytes (roughly 18 exabytes)
Developed byIgor Pavlov[1]
Initial release1999; 26 years ago (1999)[2]
Type of formatData compression
Open format?Yes: GNU Lesser General Public License / Public domain
Website7-zip.org

7z is a compressed archive file format that supports several different data compression, encryption and pre-processing algorithms. The 7z format initially appeared as implemented by the 7-Zip archiver. The 7-Zip program is publicly available under the terms of the GNU Lesser General Public License. The LZMA SDK 4.62 was placed in the public domain in December 2008. The latest stable version of 7-Zip and LZMA SDK is version 24.08.[2]

The 7z file format specification is distributed with 7-Zip's source code since 2015. The specification can be found in plain text format in the 'doc' sub-directory of the source code distribution.[3]

Features and enhancements

[edit]

The 7z format provides the following main features:

  • Open, modular architecture that allows any compression, conversion, or encryption method to be stacked.
  • High compression ratios (depending on the compression method used).
  • AES-256 bit encryption.
  • Zip 2.0 (Legacy) Encryption
  • Large file support (up to approximately 16 exbibytes, or 264 bytes).
  • Unicode file names.
  • Support for solid compression, where multiple files of similar type are compressed within a single stream, in order to exploit the combined redundancy inherent in similar files.
  • Compression and encryption of archive headers.
  • Support for multi-part archives : e.g. xxx.7z.001, xxx.7z.002, ... (see the context menu items Split File... to create them and Combine Files... to re-assemble an archive from a set of multi-part component files).
  • Support for custom codec plugin DLLs.

The format's open architecture allows additional future compression methods to be added to the standard.

Compression methods

[edit]

The following compression methods are currently defined:

  • LZMA – A variation of the LZ77 algorithm, using a sliding dictionary up to 4 GB in length for duplicate string elimination. The LZ stage is followed by entropy coding using a Markov chain-based range coder and binary trees.
  • LZMA2 – modified version of LZMA providing better multithreading support and less expansion of incompressible data.[4]
  • Bzip2 – The standard Burrows–Wheeler transform algorithm. Bzip2 uses two reversible transformations; BWT, then Move to front with Huffman coding for symbol reduction (the actual compression element).
  • PPMd – Dmitry Shkarin's 2002 PPMdH (PPMII (Prediction by Partial matching with Information Inheritance) and cPPMII (complicated PPMII)) with small changes: PPMII is an improved version of the 1984 PPM compression algorithm (prediction by partial matching).
  • DEFLATE – Standard algorithm based on 32 kB LZ77 and Huffman coding. Deflate is found in several file formats including ZIP, gzip, PNG and PDF. 7-Zip contains a from-scratch DEFLATE encoder that frequently beats the de facto standard zlib version in compression size, but at the expense of CPU usage.

A suite of recompression tools called AdvanceCOMP contains a copy of the DEFLATE encoder from the 7-Zip implementation; these utilities can often be used to further compress the size of existing gzip, ZIP, PNG, or MNG files.

Pre-processing filters

[edit]

The LZMA SDK comes with the BCJ and BCJ2 preprocessors included, so that later stages are able to achieve greater compression: For x86, ARM, PowerPC (PPC), IA-64 Itanium, and ARM Thumb processors, jump targets are 'normalized' [4] before compression by changing relative position into absolute values. For x86, this means that near jumps, calls and conditional jumps (but not short jumps and conditional jumps) are converted from the machine language "jump 1655 bytes backwards" style notation to normalized "jump to address 5554" style notation; all jumps to 5554, perhaps a common subroutine, are thus encoded identically, making them more compressible.

  • BCJ – Converter for 32-bit x86 executables. Normalise target addresses of near jumps and calls from relative distances to absolute destinations.
  • BCJ2– Pre-processor for 32-bit x86 executables. BCJ2 is an improvement on BCJ, adding additional x86 jump/call instruction processing. Near jump, near call, conditional near jump targets are split out and compressed separately in another stream.
  • Delta encoding – delta filter, basic preprocessor for multimedia data.

Similar executable pre-processing technology is included in other software; the RAR compressor features displacement compression for 32-bit x86 executables and IA-64 executables, and the UPX runtime executable file compressor includes support for working with 16-bit values within DOS binary files.

Encryption

[edit]

The 7z format supports encryption with the AES algorithm with a 256-bit key. The key is generated from a user-supplied passphrase using an algorithm based on the SHA-256 hash function. The SHA-256 is executed 219 (524288) times,[5] which causes a significant delay on slow PCs before compression or extraction starts. This technique is called key stretching and is used to make a brute-force search for the passphrase more difficult. Current GPU-based, and custom hardware attacks limit the effectiveness of this particular method of key stretching,[6] so it is still important to choose a strong password. The 7z format provides the option to encrypt the filenames of a 7z archive.

Limitations

[edit]

The 7z format does not store filesystem permissions (such as UNIX owner/group permissions or NTFS ACLs), and hence can be inappropriate for backup/archival purposes. A workaround on UNIX-like systems for this is to convert data to a tar bitstream before compressing with 7z. But GNU tar (common in many UNIX environments) can also compress with the LZMA2 algorithm ("xz") natively, without the use of 7z, using the "-J" switch. The resulting file extension is ".tar.xz" or ".txz" and not ".tar.7z". This method of compression has been adopted with many distributions for packaging, such as Arch, Debian (deb), Fedora (rpm) and Slackware. (The older "lzma" format is less efficient.)[7] On the other hand, it is important to note, that tar does not save the filesystem encoding, which means that tar compressed filenames can become unreadable if decompressed on a different computer.

The 7z format does not allow extraction of some "broken files"—that is (for example) if one has the first segment of a series of 7z files, 7z cannot give the start of the files within the archive—it must wait until all segments are downloaded. The 7z format also lacks recovery records, making it vulnerable to data degradation unless used in conjunction with external solutions, like parchives, or within filesystems with robust error-correction. By way of comparison, zip files also lack a recovery feature while the rar format has one.

See also

[edit]

References

[edit]
  1. ^ "A Few Questions for Igor Pavlov". Dr. Dobb's Data Compression Newsletter. 30 April 2003. Archived from the original on 28 October 2008. Retrieved 26 December 2009.
  2. ^ a b "History of 7-zip changes". Archived from the original on 27 February 2015. Retrieved 10 June 2010.
  3. ^ LZMA SDK, "DOC" directory, 7zFormat.txt
  4. ^ a b Collin, Lasse. "lzma_.lzma". liblzma bindings. Archived from the original on 8 February 2010. Retrieved 3 January 2010. Compared to LZMA1, LZMA2 adds support for LZMA_SYNC_FLUSH, uncompressed chunks (smaller expansion when trying to compress uncompressible data), possibility to change lc/lp/pb in the middle of encoding, and some other internal improvements.
  5. ^ "7-zip source code". Archived from the original on 22 March 2019. Retrieved 23 March 2018.
  6. ^ Colin Percival. scrypt Archived 28 May 2019 at the Wayback Machine. As presented in "Stronger Key Derivation via Sequential Memory-Hard Functions" Archived 14 April 2019 at the Wayback Machine. presented at BSDCan'09, May 2009.
  7. ^ "GNU tar 1.34: 8.1 Using Less Space through Compression". Archived from the original on 2 April 2015. Retrieved 17 March 2015.

Further reading

[edit]
  • Salomon, David (2007). Data compression: the complete reference. Springer. p. 241. ISBN 978-1-84628-602-5.
[edit]