956 lines
		
	
	
		
			36 KiB
		
	
	
	
		
			Plaintext
		
	
	
	
	
	
		
		
			
		
	
	
			956 lines
		
	
	
		
			36 KiB
		
	
	
	
		
			Plaintext
		
	
	
	
	
	
|   | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Network Working Group                                         P. Deutsch | |||
|  | Request for Comments: 1951                           Aladdin Enterprises | |||
|  | Category: Informational                                         May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |         DEFLATE Compressed Data Format Specification version 1.3 | |||
|  | 
 | |||
|  | Status of This Memo | |||
|  | 
 | |||
|  |    This memo provides information for the Internet community.  This memo | |||
|  |    does not specify an Internet standard of any kind.  Distribution of | |||
|  |    this memo is unlimited. | |||
|  | 
 | |||
|  | IESG Note: | |||
|  | 
 | |||
|  |    The IESG takes no position on the validity of any Intellectual | |||
|  |    Property Rights statements contained in this document. | |||
|  | 
 | |||
|  | Notices | |||
|  | 
 | |||
|  |    Copyright (c) 1996 L. Peter Deutsch | |||
|  | 
 | |||
|  |    Permission is granted to copy and distribute this document for any | |||
|  |    purpose and without charge, including translations into other | |||
|  |    languages and incorporation into compilations, provided that the | |||
|  |    copyright notice and this notice are preserved, and that any | |||
|  |    substantive changes or deletions from the original are clearly | |||
|  |    marked. | |||
|  | 
 | |||
|  |    A pointer to the latest version of this and related documentation in | |||
|  |    HTML format can be found at the URL | |||
|  |    <ftp://ftp.uu.net/graphics/png/documents/zlib/zdoc-index.html>. | |||
|  | 
 | |||
|  | Abstract | |||
|  | 
 | |||
|  |    This specification defines a lossless compressed data format that | |||
|  |    compresses data using a combination of the LZ77 algorithm and Huffman | |||
|  |    coding, with efficiency comparable to the best currently available | |||
|  |    general-purpose compression methods.  The data can be produced or | |||
|  |    consumed, even for an arbitrarily long sequentially presented input | |||
|  |    data stream, using only an a priori bounded amount of intermediate | |||
|  |    storage.  The format can be implemented readily in a manner not | |||
|  |    covered by patents. | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                      [Page 1] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  | Table of Contents | |||
|  | 
 | |||
|  |    1. Introduction ................................................... 2 | |||
|  |       1.1. Purpose ................................................... 2 | |||
|  |       1.2. Intended audience ......................................... 3 | |||
|  |       1.3. Scope ..................................................... 3 | |||
|  |       1.4. Compliance ................................................ 3 | |||
|  |       1.5.  Definitions of terms and conventions used ................ 3 | |||
|  |       1.6. Changes from previous versions ............................ 4 | |||
|  |    2. Compressed representation overview ............................. 4 | |||
|  |    3. Detailed specification ......................................... 5 | |||
|  |       3.1. Overall conventions ....................................... 5 | |||
|  |           3.1.1. Packing into bytes .................................. 5 | |||
|  |       3.2. Compressed block format ................................... 6 | |||
|  |           3.2.1. Synopsis of prefix and Huffman coding ............... 6 | |||
|  |           3.2.2. Use of Huffman coding in the "deflate" format ....... 7 | |||
|  |           3.2.3. Details of block format ............................. 9 | |||
|  |           3.2.4. Non-compressed blocks (BTYPE=00) ................... 11 | |||
|  |           3.2.5. Compressed blocks (length and distance codes) ...... 11 | |||
|  |           3.2.6. Compression with fixed Huffman codes (BTYPE=01) .... 12 | |||
|  |           3.2.7. Compression with dynamic Huffman codes (BTYPE=10) .. 13 | |||
|  |       3.3. Compliance ............................................... 14 | |||
|  |    4. Compression algorithm details ................................. 14 | |||
|  |    5. References .................................................... 16 | |||
|  |    6. Security Considerations ....................................... 16 | |||
|  |    7. Source code ................................................... 16 | |||
|  |    8. Acknowledgements .............................................. 16 | |||
|  |    9. Author's Address .............................................. 17 | |||
|  | 
 | |||
|  | 1. Introduction | |||
|  | 
 | |||
|  |    1.1. Purpose | |||
|  | 
 | |||
|  |       The purpose of this specification is to define a lossless | |||
|  |       compressed data format that: | |||
|  |           * Is independent of CPU type, operating system, file system, | |||
|  |             and character set, and hence can be used for interchange; | |||
|  |           * Can be produced or consumed, even for an arbitrarily long | |||
|  |             sequentially presented input data stream, using only an a | |||
|  |             priori bounded amount of intermediate storage, and hence | |||
|  |             can be used in data communications or similar structures | |||
|  |             such as Unix filters; | |||
|  |           * Compresses data with efficiency comparable to the best | |||
|  |             currently available general-purpose compression methods, | |||
|  |             and in particular considerably better than the "compress" | |||
|  |             program; | |||
|  |           * Can be implemented readily in a manner not covered by | |||
|  |             patents, and hence can be practiced freely; | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                      [Page 2] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |           * Is compatible with the file format produced by the current | |||
|  |             widely used gzip utility, in that conforming decompressors | |||
|  |             will be able to read data produced by the existing gzip | |||
|  |             compressor. | |||
|  | 
 | |||
|  |       The data format defined by this specification does not attempt to: | |||
|  | 
 | |||
|  |           * Allow random access to compressed data; | |||
|  |           * Compress specialized data (e.g., raster graphics) as well | |||
|  |             as the best currently available specialized algorithms. | |||
|  | 
 | |||
|  |       A simple counting argument shows that no lossless compression | |||
|  |       algorithm can compress every possible input data set.  For the | |||
|  |       format defined here, the worst case expansion is 5 bytes per 32K- | |||
|  |       byte block, i.e., a size increase of 0.015% for large data sets. | |||
|  |       English text usually compresses by a factor of 2.5 to 3; | |||
|  |       executable files usually compress somewhat less; graphical data | |||
|  |       such as raster images may compress much more. | |||
|  | 
 | |||
|  |    1.2. Intended audience | |||
|  | 
 | |||
|  |       This specification is intended for use by implementors of software | |||
|  |       to compress data into "deflate" format and/or decompress data from | |||
|  |       "deflate" format. | |||
|  | 
 | |||
|  |       The text of the specification assumes a basic background in | |||
|  |       programming at the level of bits and other primitive data | |||
|  |       representations.  Familiarity with the technique of Huffman coding | |||
|  |       is helpful but not required. | |||
|  | 
 | |||
|  |    1.3. Scope | |||
|  | 
 | |||
|  |       The specification specifies a method for representing a sequence | |||
|  |       of bytes as a (usually shorter) sequence of bits, and a method for | |||
|  |       packing the latter bit sequence into bytes. | |||
|  | 
 | |||
|  |    1.4. Compliance | |||
|  | 
 | |||
|  |       Unless otherwise indicated below, a compliant decompressor must be | |||
|  |       able to accept and decompress any data set that conforms to all | |||
|  |       the specifications presented here; a compliant compressor must | |||
|  |       produce data sets that conform to all the specifications presented | |||
|  |       here. | |||
|  | 
 | |||
|  |    1.5.  Definitions of terms and conventions used | |||
|  | 
 | |||
|  |       Byte: 8 bits stored or transmitted as a unit (same as an octet). | |||
|  |       For this specification, a byte is exactly 8 bits, even on machines | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                      [Page 3] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |       which store a character on a number of bits different from eight. | |||
|  |       See below, for the numbering of bits within a byte. | |||
|  | 
 | |||
|  |       String: a sequence of arbitrary bytes. | |||
|  | 
 | |||
|  |    1.6. Changes from previous versions | |||
|  | 
 | |||
|  |       There have been no technical changes to the deflate format since | |||
|  |       version 1.1 of this specification.  In version 1.2, some | |||
|  |       terminology was changed.  Version 1.3 is a conversion of the | |||
|  |       specification to RFC style. | |||
|  | 
 | |||
|  | 2. Compressed representation overview | |||
|  | 
 | |||
|  |    A compressed data set consists of a series of blocks, corresponding | |||
|  |    to successive blocks of input data.  The block sizes are arbitrary, | |||
|  |    except that non-compressible blocks are limited to 65,535 bytes. | |||
|  | 
 | |||
|  |    Each block is compressed using a combination of the LZ77 algorithm | |||
|  |    and Huffman coding. The Huffman trees for each block are independent | |||
|  |    of those for previous or subsequent blocks; the LZ77 algorithm may | |||
|  |    use a reference to a duplicated string occurring in a previous block, | |||
|  |    up to 32K input bytes before. | |||
|  | 
 | |||
|  |    Each block consists of two parts: a pair of Huffman code trees that | |||
|  |    describe the representation of the compressed data part, and a | |||
|  |    compressed data part.  (The Huffman trees themselves are compressed | |||
|  |    using Huffman encoding.)  The compressed data consists of a series of | |||
|  |    elements of two types: literal bytes (of strings that have not been | |||
|  |    detected as duplicated within the previous 32K input bytes), and | |||
|  |    pointers to duplicated strings, where a pointer is represented as a | |||
|  |    pair <length, backward distance>.  The representation used in the | |||
|  |    "deflate" format limits distances to 32K bytes and lengths to 258 | |||
|  |    bytes, but does not limit the size of a block, except for | |||
|  |    uncompressible blocks, which are limited as noted above. | |||
|  | 
 | |||
|  |    Each type of value (literals, distances, and lengths) in the | |||
|  |    compressed data is represented using a Huffman code, using one code | |||
|  |    tree for literals and lengths and a separate code tree for distances. | |||
|  |    The code trees for each block appear in a compact form just before | |||
|  |    the compressed data for that block. | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                      [Page 4] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  | 3. Detailed specification | |||
|  | 
 | |||
|  |    3.1. Overall conventions In the diagrams below, a box like this: | |||
|  | 
 | |||
|  |          +---+ | |||
|  |          |   | <-- the vertical bars might be missing | |||
|  |          +---+ | |||
|  | 
 | |||
|  |       represents one byte; a box like this: | |||
|  | 
 | |||
|  |          +==============+ | |||
|  |          |              | | |||
|  |          +==============+ | |||
|  | 
 | |||
|  |       represents a variable number of bytes. | |||
|  | 
 | |||
|  |       Bytes stored within a computer do not have a "bit order", since | |||
|  |       they are always treated as a unit.  However, a byte considered as | |||
|  |       an integer between 0 and 255 does have a most- and least- | |||
|  |       significant bit, and since we write numbers with the most- | |||
|  |       significant digit on the left, we also write bytes with the most- | |||
|  |       significant bit on the left.  In the diagrams below, we number the | |||
|  |       bits of a byte so that bit 0 is the least-significant bit, i.e., | |||
|  |       the bits are numbered: | |||
|  | 
 | |||
|  |          +--------+ | |||
|  |          |76543210| | |||
|  |          +--------+ | |||
|  | 
 | |||
|  |       Within a computer, a number may occupy multiple bytes.  All | |||
|  |       multi-byte numbers in the format described here are stored with | |||
|  |       the least-significant byte first (at the lower memory address). | |||
|  |       For example, the decimal number 520 is stored as: | |||
|  | 
 | |||
|  |              0        1 | |||
|  |          +--------+--------+ | |||
|  |          |00001000|00000010| | |||
|  |          +--------+--------+ | |||
|  |           ^        ^ | |||
|  |           |        | | |||
|  |           |        + more significant byte = 2 x 256 | |||
|  |           + less significant byte = 8 | |||
|  | 
 | |||
|  |       3.1.1. Packing into bytes | |||
|  | 
 | |||
|  |          This document does not address the issue of the order in which | |||
|  |          bits of a byte are transmitted on a bit-sequential medium, | |||
|  |          since the final data format described here is byte- rather than | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                      [Page 5] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |          bit-oriented.  However, we describe the compressed block format | |||
|  |          in below, as a sequence of data elements of various bit | |||
|  |          lengths, not a sequence of bytes.  We must therefore specify | |||
|  |          how to pack these data elements into bytes to form the final | |||
|  |          compressed byte sequence: | |||
|  | 
 | |||
|  |              * Data elements are packed into bytes in order of | |||
|  |                increasing bit number within the byte, i.e., starting | |||
|  |                with the least-significant bit of the byte. | |||
|  |              * Data elements other than Huffman codes are packed | |||
|  |                starting with the least-significant bit of the data | |||
|  |                element. | |||
|  |              * Huffman codes are packed starting with the most- | |||
|  |                significant bit of the code. | |||
|  | 
 | |||
|  |          In other words, if one were to print out the compressed data as | |||
|  |          a sequence of bytes, starting with the first byte at the | |||
|  |          *right* margin and proceeding to the *left*, with the most- | |||
|  |          significant bit of each byte on the left as usual, one would be | |||
|  |          able to parse the result from right to left, with fixed-width | |||
|  |          elements in the correct MSB-to-LSB order and Huffman codes in | |||
|  |          bit-reversed order (i.e., with the first bit of the code in the | |||
|  |          relative LSB position). | |||
|  | 
 | |||
|  |    3.2. Compressed block format | |||
|  | 
 | |||
|  |       3.2.1. Synopsis of prefix and Huffman coding | |||
|  | 
 | |||
|  |          Prefix coding represents symbols from an a priori known | |||
|  |          alphabet by bit sequences (codes), one code for each symbol, in | |||
|  |          a manner such that different symbols may be represented by bit | |||
|  |          sequences of different lengths, but a parser can always parse | |||
|  |          an encoded string unambiguously symbol-by-symbol. | |||
|  | 
 | |||
|  |          We define a prefix code in terms of a binary tree in which the | |||
|  |          two edges descending from each non-leaf node are labeled 0 and | |||
|  |          1 and in which the leaf nodes correspond one-for-one with (are | |||
|  |          labeled with) the symbols of the alphabet; then the code for a | |||
|  |          symbol is the sequence of 0's and 1's on the edges leading from | |||
|  |          the root to the leaf labeled with that symbol.  For example: | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                      [Page 6] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |                           /\              Symbol    Code | |||
|  |                          0  1             ------    ---- | |||
|  |                         /    \                A      00 | |||
|  |                        /\     B               B       1 | |||
|  |                       0  1                    C     011 | |||
|  |                      /    \                   D     010 | |||
|  |                     A     /\ | |||
|  |                          0  1 | |||
|  |                         /    \ | |||
|  |                        D      C | |||
|  | 
 | |||
|  |          A parser can decode the next symbol from an encoded input | |||
|  |          stream by walking down the tree from the root, at each step | |||
|  |          choosing the edge corresponding to the next input bit. | |||
|  | 
 | |||
|  |          Given an alphabet with known symbol frequencies, the Huffman | |||
|  |          algorithm allows the construction of an optimal prefix code | |||
|  |          (one which represents strings with those symbol frequencies | |||
|  |          using the fewest bits of any possible prefix codes for that | |||
|  |          alphabet).  Such a code is called a Huffman code.  (See | |||
|  |          reference [1] in Chapter 5, references for additional | |||
|  |          information on Huffman codes.) | |||
|  | 
 | |||
|  |          Note that in the "deflate" format, the Huffman codes for the | |||
|  |          various alphabets must not exceed certain maximum code lengths. | |||
|  |          This constraint complicates the algorithm for computing code | |||
|  |          lengths from symbol frequencies.  Again, see Chapter 5, | |||
|  |          references for details. | |||
|  | 
 | |||
|  |       3.2.2. Use of Huffman coding in the "deflate" format | |||
|  | 
 | |||
|  |          The Huffman codes used for each alphabet in the "deflate" | |||
|  |          format have two additional rules: | |||
|  | 
 | |||
|  |              * All codes of a given bit length have lexicographically | |||
|  |                consecutive values, in the same order as the symbols | |||
|  |                they represent; | |||
|  | 
 | |||
|  |              * Shorter codes lexicographically precede longer codes. | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                      [Page 7] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |          We could recode the example above to follow this rule as | |||
|  |          follows, assuming that the order of the alphabet is ABCD: | |||
|  | 
 | |||
|  |             Symbol  Code | |||
|  |             ------  ---- | |||
|  |             A       10 | |||
|  |             B       0 | |||
|  |             C       110 | |||
|  |             D       111 | |||
|  | 
 | |||
|  |          I.e., 0 precedes 10 which precedes 11x, and 110 and 111 are | |||
|  |          lexicographically consecutive. | |||
|  | 
 | |||
|  |          Given this rule, we can define the Huffman code for an alphabet | |||
|  |          just by giving the bit lengths of the codes for each symbol of | |||
|  |          the alphabet in order; this is sufficient to determine the | |||
|  |          actual codes.  In our example, the code is completely defined | |||
|  |          by the sequence of bit lengths (2, 1, 3, 3).  The following | |||
|  |          algorithm generates the codes as integers, intended to be read | |||
|  |          from most- to least-significant bit.  The code lengths are | |||
|  |          initially in tree[I].Len; the codes are produced in | |||
|  |          tree[I].Code. | |||
|  | 
 | |||
|  |          1)  Count the number of codes for each code length.  Let | |||
|  |              bl_count[N] be the number of codes of length N, N >= 1. | |||
|  | 
 | |||
|  |          2)  Find the numerical value of the smallest code for each | |||
|  |              code length: | |||
|  | 
 | |||
|  |                 code = 0; | |||
|  |                 bl_count[0] = 0; | |||
|  |                 for (bits = 1; bits <= MAX_BITS; bits++) { | |||
|  |                     code = (code + bl_count[bits-1]) << 1; | |||
|  |                     next_code[bits] = code; | |||
|  |                 } | |||
|  | 
 | |||
|  |          3)  Assign numerical values to all codes, using consecutive | |||
|  |              values for all codes of the same length with the base | |||
|  |              values determined at step 2. Codes that are never used | |||
|  |              (which have a bit length of zero) must not be assigned a | |||
|  |              value. | |||
|  | 
 | |||
|  |                 for (n = 0;  n <= max_code; n++) { | |||
|  |                     len = tree[n].Len; | |||
|  |                     if (len != 0) { | |||
|  |                         tree[n].Code = next_code[len]; | |||
|  |                         next_code[len]++; | |||
|  |                     } | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                      [Page 8] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |                 } | |||
|  | 
 | |||
|  |          Example: | |||
|  | 
 | |||
|  |          Consider the alphabet ABCDEFGH, with bit lengths (3, 3, 3, 3, | |||
|  |          3, 2, 4, 4).  After step 1, we have: | |||
|  | 
 | |||
|  |             N      bl_count[N] | |||
|  |             -      ----------- | |||
|  |             2      1 | |||
|  |             3      5 | |||
|  |             4      2 | |||
|  | 
 | |||
|  |          Step 2 computes the following next_code values: | |||
|  | 
 | |||
|  |             N      next_code[N] | |||
|  |             -      ------------ | |||
|  |             1      0 | |||
|  |             2      0 | |||
|  |             3      2 | |||
|  |             4      14 | |||
|  | 
 | |||
|  |          Step 3 produces the following code values: | |||
|  | 
 | |||
|  |             Symbol Length   Code | |||
|  |             ------ ------   ---- | |||
|  |             A       3        010 | |||
|  |             B       3        011 | |||
|  |             C       3        100 | |||
|  |             D       3        101 | |||
|  |             E       3        110 | |||
|  |             F       2         00 | |||
|  |             G       4       1110 | |||
|  |             H       4       1111 | |||
|  | 
 | |||
|  |       3.2.3. Details of block format | |||
|  | 
 | |||
|  |          Each block of compressed data begins with 3 header bits | |||
|  |          containing the following data: | |||
|  | 
 | |||
|  |             first bit       BFINAL | |||
|  |             next 2 bits     BTYPE | |||
|  | 
 | |||
|  |          Note that the header bits do not necessarily begin on a byte | |||
|  |          boundary, since a block does not necessarily occupy an integral | |||
|  |          number of bytes. | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                      [Page 9] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |          BFINAL is set if and only if this is the last block of the data | |||
|  |          set. | |||
|  | 
 | |||
|  |          BTYPE specifies how the data are compressed, as follows: | |||
|  | 
 | |||
|  |             00 - no compression | |||
|  |             01 - compressed with fixed Huffman codes | |||
|  |             10 - compressed with dynamic Huffman codes | |||
|  |             11 - reserved (error) | |||
|  | 
 | |||
|  |          The only difference between the two compressed cases is how the | |||
|  |          Huffman codes for the literal/length and distance alphabets are | |||
|  |          defined. | |||
|  | 
 | |||
|  |          In all cases, the decoding algorithm for the actual data is as | |||
|  |          follows: | |||
|  | 
 | |||
|  |             do | |||
|  |                read block header from input stream. | |||
|  |                if stored with no compression | |||
|  |                   skip any remaining bits in current partially | |||
|  |                      processed byte | |||
|  |                   read LEN and NLEN (see next section) | |||
|  |                   copy LEN bytes of data to output | |||
|  |                otherwise | |||
|  |                   if compressed with dynamic Huffman codes | |||
|  |                      read representation of code trees (see | |||
|  |                         subsection below) | |||
|  |                   loop (until end of block code recognized) | |||
|  |                      decode literal/length value from input stream | |||
|  |                      if value < 256 | |||
|  |                         copy value (literal byte) to output stream | |||
|  |                      otherwise | |||
|  |                         if value = end of block (256) | |||
|  |                            break from loop | |||
|  |                         otherwise (value = 257..285) | |||
|  |                            decode distance from input stream | |||
|  | 
 | |||
|  |                            move backwards distance bytes in the output | |||
|  |                            stream, and copy length bytes from this | |||
|  |                            position to the output stream. | |||
|  |                   end loop | |||
|  |             while not last block | |||
|  | 
 | |||
|  |          Note that a duplicated string reference may refer to a string | |||
|  |          in a previous block; i.e., the backward distance may cross one | |||
|  |          or more block boundaries.  However a distance cannot refer past | |||
|  |          the beginning of the output stream.  (An application using a | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                     [Page 10] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |          preset dictionary might discard part of the output stream; a | |||
|  |          distance can refer to that part of the output stream anyway) | |||
|  |          Note also that the referenced string may overlap the current | |||
|  |          position; for example, if the last 2 bytes decoded have values | |||
|  |          X and Y, a string reference with <length = 5, distance = 2> | |||
|  |          adds X,Y,X,Y,X to the output stream. | |||
|  | 
 | |||
|  |          We now specify each compression method in turn. | |||
|  | 
 | |||
|  |       3.2.4. Non-compressed blocks (BTYPE=00) | |||
|  | 
 | |||
|  |          Any bits of input up to the next byte boundary are ignored. | |||
|  |          The rest of the block consists of the following information: | |||
|  | 
 | |||
|  |               0   1   2   3   4... | |||
|  |             +---+---+---+---+================================+ | |||
|  |             |  LEN  | NLEN  |... LEN bytes of literal data...| | |||
|  |             +---+---+---+---+================================+ | |||
|  | 
 | |||
|  |          LEN is the number of data bytes in the block.  NLEN is the | |||
|  |          one's complement of LEN. | |||
|  | 
 | |||
|  |       3.2.5. Compressed blocks (length and distance codes) | |||
|  | 
 | |||
|  |          As noted above, encoded data blocks in the "deflate" format | |||
|  |          consist of sequences of symbols drawn from three conceptually | |||
|  |          distinct alphabets: either literal bytes, from the alphabet of | |||
|  |          byte values (0..255), or <length, backward distance> pairs, | |||
|  |          where the length is drawn from (3..258) and the distance is | |||
|  |          drawn from (1..32,768).  In fact, the literal and length | |||
|  |          alphabets are merged into a single alphabet (0..285), where | |||
|  |          values 0..255 represent literal bytes, the value 256 indicates | |||
|  |          end-of-block, and values 257..285 represent length codes | |||
|  |          (possibly in conjunction with extra bits following the symbol | |||
|  |          code) as follows: | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                     [Page 11] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |                  Extra               Extra               Extra | |||
|  |             Code Bits Length(s) Code Bits Lengths   Code Bits Length(s) | |||
|  |             ---- ---- ------     ---- ---- -------   ---- ---- ------- | |||
|  |              257   0     3       267   1   15,16     277   4   67-82 | |||
|  |              258   0     4       268   1   17,18     278   4   83-98 | |||
|  |              259   0     5       269   2   19-22     279   4   99-114 | |||
|  |              260   0     6       270   2   23-26     280   4  115-130 | |||
|  |              261   0     7       271   2   27-30     281   5  131-162 | |||
|  |              262   0     8       272   2   31-34     282   5  163-194 | |||
|  |              263   0     9       273   3   35-42     283   5  195-226 | |||
|  |              264   0    10       274   3   43-50     284   5  227-257 | |||
|  |              265   1  11,12      275   3   51-58     285   0    258 | |||
|  |              266   1  13,14      276   3   59-66 | |||
|  | 
 | |||
|  |          The extra bits should be interpreted as a machine integer | |||
|  |          stored with the most-significant bit first, e.g., bits 1110 | |||
|  |          represent the value 14. | |||
|  | 
 | |||
|  |                   Extra           Extra               Extra | |||
|  |              Code Bits Dist  Code Bits   Dist     Code Bits Distance | |||
|  |              ---- ---- ----  ---- ----  ------    ---- ---- -------- | |||
|  |                0   0    1     10   4     33-48    20    9   1025-1536 | |||
|  |                1   0    2     11   4     49-64    21    9   1537-2048 | |||
|  |                2   0    3     12   5     65-96    22   10   2049-3072 | |||
|  |                3   0    4     13   5     97-128   23   10   3073-4096 | |||
|  |                4   1   5,6    14   6    129-192   24   11   4097-6144 | |||
|  |                5   1   7,8    15   6    193-256   25   11   6145-8192 | |||
|  |                6   2   9-12   16   7    257-384   26   12  8193-12288 | |||
|  |                7   2  13-16   17   7    385-512   27   12 12289-16384 | |||
|  |                8   3  17-24   18   8    513-768   28   13 16385-24576 | |||
|  |                9   3  25-32   19   8   769-1024   29   13 24577-32768 | |||
|  | 
 | |||
|  |       3.2.6. Compression with fixed Huffman codes (BTYPE=01) | |||
|  | 
 | |||
|  |          The Huffman codes for the two alphabets are fixed, and are not | |||
|  |          represented explicitly in the data.  The Huffman code lengths | |||
|  |          for the literal/length alphabet are: | |||
|  | 
 | |||
|  |                    Lit Value    Bits        Codes | |||
|  |                    ---------    ----        ----- | |||
|  |                      0 - 143     8          00110000 through | |||
|  |                                             10111111 | |||
|  |                    144 - 255     9          110010000 through | |||
|  |                                             111111111 | |||
|  |                    256 - 279     7          0000000 through | |||
|  |                                             0010111 | |||
|  |                    280 - 287     8          11000000 through | |||
|  |                                             11000111 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                     [Page 12] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |          The code lengths are sufficient to generate the actual codes, | |||
|  |          as described above; we show the codes in the table for added | |||
|  |          clarity.  Literal/length values 286-287 will never actually | |||
|  |          occur in the compressed data, but participate in the code | |||
|  |          construction. | |||
|  | 
 | |||
|  |          Distance codes 0-31 are represented by (fixed-length) 5-bit | |||
|  |          codes, with possible additional bits as shown in the table | |||
|  |          shown in Paragraph 3.2.5, above.  Note that distance codes 30- | |||
|  |          31 will never actually occur in the compressed data. | |||
|  | 
 | |||
|  |       3.2.7. Compression with dynamic Huffman codes (BTYPE=10) | |||
|  | 
 | |||
|  |          The Huffman codes for the two alphabets appear in the block | |||
|  |          immediately after the header bits and before the actual | |||
|  |          compressed data, first the literal/length code and then the | |||
|  |          distance code.  Each code is defined by a sequence of code | |||
|  |          lengths, as discussed in Paragraph 3.2.2, above.  For even | |||
|  |          greater compactness, the code length sequences themselves are | |||
|  |          compressed using a Huffman code.  The alphabet for code lengths | |||
|  |          is as follows: | |||
|  | 
 | |||
|  |                0 - 15: Represent code lengths of 0 - 15 | |||
|  |                    16: Copy the previous code length 3 - 6 times. | |||
|  |                        The next 2 bits indicate repeat length | |||
|  |                              (0 = 3, ... , 3 = 6) | |||
|  |                           Example:  Codes 8, 16 (+2 bits 11), | |||
|  |                                     16 (+2 bits 10) will expand to | |||
|  |                                     12 code lengths of 8 (1 + 6 + 5) | |||
|  |                    17: Repeat a code length of 0 for 3 - 10 times. | |||
|  |                        (3 bits of length) | |||
|  |                    18: Repeat a code length of 0 for 11 - 138 times | |||
|  |                        (7 bits of length) | |||
|  | 
 | |||
|  |          A code length of 0 indicates that the corresponding symbol in | |||
|  |          the literal/length or distance alphabet will not occur in the | |||
|  |          block, and should not participate in the Huffman code | |||
|  |          construction algorithm given earlier.  If only one distance | |||
|  |          code is used, it is encoded using one bit, not zero bits; in | |||
|  |          this case there is a single code length of one, with one unused | |||
|  |          code.  One distance code of zero bits means that there are no | |||
|  |          distance codes used at all (the data is all literals). | |||
|  | 
 | |||
|  |          We can now define the format of the block: | |||
|  | 
 | |||
|  |                5 Bits: HLIT, # of Literal/Length codes - 257 (257 - 286) | |||
|  |                5 Bits: HDIST, # of Distance codes - 1        (1 - 32) | |||
|  |                4 Bits: HCLEN, # of Code Length codes - 4     (4 - 19) | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                     [Page 13] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |                (HCLEN + 4) x 3 bits: code lengths for the code length | |||
|  |                   alphabet given just above, in the order: 16, 17, 18, | |||
|  |                   0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15 | |||
|  | 
 | |||
|  |                   These code lengths are interpreted as 3-bit integers | |||
|  |                   (0-7); as above, a code length of 0 means the | |||
|  |                   corresponding symbol (literal/length or distance code | |||
|  |                   length) is not used. | |||
|  | 
 | |||
|  |                HLIT + 257 code lengths for the literal/length alphabet, | |||
|  |                   encoded using the code length Huffman code | |||
|  | 
 | |||
|  |                HDIST + 1 code lengths for the distance alphabet, | |||
|  |                   encoded using the code length Huffman code | |||
|  | 
 | |||
|  |                The actual compressed data of the block, | |||
|  |                   encoded using the literal/length and distance Huffman | |||
|  |                   codes | |||
|  | 
 | |||
|  |                The literal/length symbol 256 (end of data), | |||
|  |                   encoded using the literal/length Huffman code | |||
|  | 
 | |||
|  |          The code length repeat codes can cross from HLIT + 257 to the | |||
|  |          HDIST + 1 code lengths.  In other words, all code lengths form | |||
|  |          a single sequence of HLIT + HDIST + 258 values. | |||
|  | 
 | |||
|  |    3.3. Compliance | |||
|  | 
 | |||
|  |       A compressor may limit further the ranges of values specified in | |||
|  |       the previous section and still be compliant; for example, it may | |||
|  |       limit the range of backward pointers to some value smaller than | |||
|  |       32K.  Similarly, a compressor may limit the size of blocks so that | |||
|  |       a compressible block fits in memory. | |||
|  | 
 | |||
|  |       A compliant decompressor must accept the full range of possible | |||
|  |       values defined in the previous section, and must accept blocks of | |||
|  |       arbitrary size. | |||
|  | 
 | |||
|  | 4. Compression algorithm details | |||
|  | 
 | |||
|  |    While it is the intent of this document to define the "deflate" | |||
|  |    compressed data format without reference to any particular | |||
|  |    compression algorithm, the format is related to the compressed | |||
|  |    formats produced by LZ77 (Lempel-Ziv 1977, see reference [2] below); | |||
|  |    since many variations of LZ77 are patented, it is strongly | |||
|  |    recommended that the implementor of a compressor follow the general | |||
|  |    algorithm presented here, which is known not to be patented per se. | |||
|  |    The material in this section is not part of the definition of the | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                     [Page 14] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  |    specification per se, and a compressor need not follow it in order to | |||
|  |    be compliant. | |||
|  | 
 | |||
|  |    The compressor terminates a block when it determines that starting a | |||
|  |    new block with fresh trees would be useful, or when the block size | |||
|  |    fills up the compressor's block buffer. | |||
|  | 
 | |||
|  |    The compressor uses a chained hash table to find duplicated strings, | |||
|  |    using a hash function that operates on 3-byte sequences.  At any | |||
|  |    given point during compression, let XYZ be the next 3 input bytes to | |||
|  |    be examined (not necessarily all different, of course).  First, the | |||
|  |    compressor examines the hash chain for XYZ.  If the chain is empty, | |||
|  |    the compressor simply writes out X as a literal byte and advances one | |||
|  |    byte in the input.  If the hash chain is not empty, indicating that | |||
|  |    the sequence XYZ (or, if we are unlucky, some other 3 bytes with the | |||
|  |    same hash function value) has occurred recently, the compressor | |||
|  |    compares all strings on the XYZ hash chain with the actual input data | |||
|  |    sequence starting at the current point, and selects the longest | |||
|  |    match. | |||
|  | 
 | |||
|  |    The compressor searches the hash chains starting with the most recent | |||
|  |    strings, to favor small distances and thus take advantage of the | |||
|  |    Huffman encoding.  The hash chains are singly linked. There are no | |||
|  |    deletions from the hash chains; the algorithm simply discards matches | |||
|  |    that are too old.  To avoid a worst-case situation, very long hash | |||
|  |    chains are arbitrarily truncated at a certain length, determined by a | |||
|  |    run-time parameter. | |||
|  | 
 | |||
|  |    To improve overall compression, the compressor optionally defers the | |||
|  |    selection of matches ("lazy matching"): after a match of length N has | |||
|  |    been found, the compressor searches for a longer match starting at | |||
|  |    the next input byte.  If it finds a longer match, it truncates the | |||
|  |    previous match to a length of one (thus producing a single literal | |||
|  |    byte) and then emits the longer match.  Otherwise, it emits the | |||
|  |    original match, and, as described above, advances N bytes before | |||
|  |    continuing. | |||
|  | 
 | |||
|  |    Run-time parameters also control this "lazy match" procedure.  If | |||
|  |    compression ratio is most important, the compressor attempts a | |||
|  |    complete second search regardless of the length of the first match. | |||
|  |    In the normal case, if the current match is "long enough", the | |||
|  |    compressor reduces the search for a longer match, thus speeding up | |||
|  |    the process.  If speed is most important, the compressor inserts new | |||
|  |    strings in the hash table only when no match was found, or when the | |||
|  |    match is not "too long".  This degrades the compression ratio but | |||
|  |    saves time since there are both fewer insertions and fewer searches. | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                     [Page 15] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  | 5. References | |||
|  | 
 | |||
|  |    [1] Huffman, D. A., "A Method for the Construction of Minimum | |||
|  |        Redundancy Codes", Proceedings of the Institute of Radio | |||
|  |        Engineers, September 1952, Volume 40, Number 9, pp. 1098-1101. | |||
|  | 
 | |||
|  |    [2] Ziv J., Lempel A., "A Universal Algorithm for Sequential Data | |||
|  |        Compression", IEEE Transactions on Information Theory, Vol. 23, | |||
|  |        No. 3, pp. 337-343. | |||
|  | 
 | |||
|  |    [3] Gailly, J.-L., and Adler, M., ZLIB documentation and sources, | |||
|  |        available in ftp://ftp.uu.net/pub/archiving/zip/doc/ | |||
|  | 
 | |||
|  |    [4] Gailly, J.-L., and Adler, M., GZIP documentation and sources, | |||
|  |        available as gzip-*.tar in ftp://prep.ai.mit.edu/pub/gnu/ | |||
|  | 
 | |||
|  |    [5] Schwartz, E. S., and Kallick, B. "Generating a canonical prefix | |||
|  |        encoding." Comm. ACM, 7,3 (Mar. 1964), pp. 166-169. | |||
|  | 
 | |||
|  |    [6] Hirschberg and Lelewer, "Efficient decoding of prefix codes," | |||
|  |        Comm. ACM, 33,4, April 1990, pp. 449-459. | |||
|  | 
 | |||
|  | 6. Security Considerations | |||
|  | 
 | |||
|  |    Any data compression method involves the reduction of redundancy in | |||
|  |    the data.  Consequently, any corruption of the data is likely to have | |||
|  |    severe effects and be difficult to correct.  Uncompressed text, on | |||
|  |    the other hand, will probably still be readable despite the presence | |||
|  |    of some corrupted bytes. | |||
|  | 
 | |||
|  |    It is recommended that systems using this data format provide some | |||
|  |    means of validating the integrity of the compressed data.  See | |||
|  |    reference [3], for example. | |||
|  | 
 | |||
|  | 7. Source code | |||
|  | 
 | |||
|  |    Source code for a C language implementation of a "deflate" compliant | |||
|  |    compressor and decompressor is available within the zlib package at | |||
|  |    ftp://ftp.uu.net/pub/archiving/zip/zlib/. | |||
|  | 
 | |||
|  | 8. Acknowledgements | |||
|  | 
 | |||
|  |    Trademarks cited in this document are the property of their | |||
|  |    respective owners. | |||
|  | 
 | |||
|  |    Phil Katz designed the deflate format.  Jean-Loup Gailly and Mark | |||
|  |    Adler wrote the related software described in this specification. | |||
|  |    Glenn Randers-Pehrson converted this document to RFC and HTML format. | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                     [Page 16] | |||
|  |  | |||
|  | RFC 1951      DEFLATE Compressed Data Format Specification      May 1996 | |||
|  | 
 | |||
|  | 
 | |||
|  | 9. Author's Address | |||
|  | 
 | |||
|  |    L. Peter Deutsch | |||
|  |    Aladdin Enterprises | |||
|  |    203 Santa Margarita Ave. | |||
|  |    Menlo Park, CA 94025 | |||
|  | 
 | |||
|  |    Phone: (415) 322-0103 (AM only) | |||
|  |    FAX:   (415) 322-1734 | |||
|  |    EMail: <ghost@aladdin.com> | |||
|  | 
 | |||
|  |    Questions about the technical content of this specification can be | |||
|  |    sent by email to: | |||
|  | 
 | |||
|  |    Jean-Loup Gailly <gzip@prep.ai.mit.edu> and | |||
|  |    Mark Adler <madler@alumni.caltech.edu> | |||
|  | 
 | |||
|  |    Editorial comments on this specification can be sent by email to: | |||
|  | 
 | |||
|  |    L. Peter Deutsch <ghost@aladdin.com> and | |||
|  |    Glenn Randers-Pehrson <randeg@alumni.rpi.edu> | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | 
 | |||
|  | Deutsch                      Informational                     [Page 17] | |||
|  |  |