otomatalabs/furnace
8
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

update zlib to 1.3.1

Browse source 
fixes macOS 15 compilation
This commit is contained in:
tildearrow 2025-07-27 17:51:52 -05:00
parent 98030de8c7
commit a4da787c1b
169 changed files with 18553 additions and 11917 deletions
Download patch file Download diff file Expand all files Collapse all files

5
extern/zlib/examples/README.examples vendored
View file

@ -34,6 +34,10 @@ gzlog.h
and deflateSetDictionary()
- illustrates use of a gzip header extra field
gznorm.c
normalize a gzip file by combining members into a single member
- demonstrates how to concatenate deflate streams using Z_BLOCK
zlib_how.html
painfully comprehensive description of zpipe.c (see below)
- describes in excruciating detail the use of deflate() and inflate()
@ -44,6 +48,7 @@ zpipe.c
- deeply commented in zlib_how.html (see above)
zran.c
zran.h
index a zlib or gzip stream and randomly access it
- illustrates the use of Z_BLOCK, inflatePrime(), and
inflateSetDictionary() to provide random access

735
extern/zlib/examples/enough.c vendored
View file

@ -1,7 +1,7 @@
/* enough.c -- determine the maximum size of inflate's Huffman code tables over
* all possible valid and complete Huffman codes, subject to a length limit.
* Copyright (C) 2007, 2008, 2012 Mark Adler
* Version 1.4 18 August 2012 Mark Adler
* all possible valid and complete prefix codes, subject to a length limit.
* Copyright (C) 2007, 2008, 2012, 2018 Mark Adler
* Version 1.5 5 August 2018 Mark Adler
*/
/* Version history:
@ -17,101 +17,107 @@
1.4 18 Aug 2012 Avoid shifts more than bits in type (caused endless loop!)
Clean up comparisons of different types
Clean up code indentation
1.5 5 Aug 2018 Clean up code style, formatting, and comments
Show all the codes for the maximum, and only the maximum
*/
/*
Examine all possible Huffman codes for a given number of symbols and a
maximum code length in bits to determine the maximum table size for zilb's
inflate. Only complete Huffman codes are counted.
Examine all possible prefix codes for a given number of symbols and a
maximum code length in bits to determine the maximum table size for zlib's
inflate. Only complete prefix codes are counted.
Two codes are considered distinct if the vectors of the number of codes per
length are not identical. So permutations of the symbol assignments result
length are not identical. So permutations of the symbol assignments result
in the same code for the counting, as do permutations of the assignments of
the bit values to the codes (i.e. only canonical codes are counted).
We build a code from shorter to longer lengths, determining how many symbols
are coded at each length. At each step, we have how many symbols remain to
are coded at each length. At each step, we have how many symbols remain to
be coded, what the last code length used was, and how many bit patterns of
that length remain unused. Then we add one to the code length and double the
number of unused patterns to graduate to the next code length. We then
number of unused patterns to graduate to the next code length. We then
assign all portions of the remaining symbols to that code length that
preserve the properties of a correct and eventually complete code. Those
preserve the properties of a correct and eventually complete code. Those
properties are: we cannot use more bit patterns than are available; and when
all the symbols are used, there are exactly zero possible bit patterns
remaining.
all the symbols are used, there are exactly zero possible bit patterns left
unused.
The inflate Huffman decoding algorithm uses two-level lookup tables for
speed. There is a single first-level table to decode codes up to root bits
in length (root == 9 in the current inflate implementation). The table
has 1 << root entries and is indexed by the next root bits of input. Codes
shorter than root bits have replicated table entries, so that the correct
entry is pointed to regardless of the bits that follow the short code. If
the code is longer than root bits, then the table entry points to a second-
level table. The size of that table is determined by the longest code with
that root-bit prefix. If that longest code has length len, then the table
has size 1 << (len - root), to index the remaining bits in that set of
codes. Each subsequent root-bit prefix then has its own sub-table. The
total number of table entries required by the code is calculated
incrementally as the number of codes at each bit length is populated. When
all of the codes are shorter than root bits, then root is reduced to the
longest code length, resulting in a single, smaller, one-level table.
speed. There is a single first-level table to decode codes up to root bits
in length (root == 9 for literal/length codes and root == 6 for distance
codes, in the current inflate implementation). The base table has 1 << root
entries and is indexed by the next root bits of input. Codes shorter than
root bits have replicated table entries, so that the correct entry is
pointed to regardless of the bits that follow the short code. If the code is
longer than root bits, then the table entry points to a second-level table.
The size of that table is determined by the longest code with that root-bit
prefix. If that longest code has length len, then the table has size 1 <<
(len - root), to index the remaining bits in that set of codes. Each
subsequent root-bit prefix then has its own sub-table. The total number of
table entries required by the code is calculated incrementally as the number
of codes at each bit length is populated. When all of the codes are shorter
than root bits, then root is reduced to the longest code length, resulting
in a single, smaller, one-level table.
The inflate algorithm also provides for small values of root (relative to
the log2 of the number of symbols), where the shortest code has more bits
than root. In that case, root is increased to the length of the shortest
code. This program, by design, does not handle that case, so it is verified
that the number of symbols is less than 2^(root + 1).
than root. In that case, root is increased to the length of the shortest
code. This program, by design, does not handle that case, so it is verified
that the number of symbols is less than 1 << (root + 1).
In order to speed up the examination (by about ten orders of magnitude for
the default arguments), the intermediate states in the build-up of a code
are remembered and previously visited branches are pruned. The memory
are remembered and previously visited branches are pruned. The memory
required for this will increase rapidly with the total number of symbols and
the maximum code length in bits. However this is a very small price to pay
the maximum code length in bits. However this is a very small price to pay
for the vast speedup.
First, all of the possible Huffman codes are counted, and reachable
First, all of the possible prefix codes are counted, and reachable
intermediate states are noted by a non-zero count in a saved-results array.
Second, the intermediate states that lead to (root + 1) bit or longer codes
are used to look at all sub-codes from those junctures for their inflate
memory usage. (The amount of memory used is not affected by the number of
memory usage. (The amount of memory used is not affected by the number of
codes of root bits or less in length.) Third, the visited states in the
construction of those sub-codes and the associated calculation of the table
size is recalled in order to avoid recalculating from the same juncture.
Beginning the code examination at (root + 1) bit codes, which is enabled by
identifying the reachable nodes, accounts for about six of the orders of
magnitude of improvement for the default arguments. About another four
orders of magnitude come from not revisiting previous states. Out of
approximately 2x10^16 possible Huffman codes, only about 2x10^6 sub-codes
magnitude of improvement for the default arguments. About another four
orders of magnitude come from not revisiting previous states. Out of
approximately 2x10^16 possible prefix codes, only about 2x10^6 sub-codes
need to be examined to cover all of the possible table memory usage cases
for the default arguments of 286 symbols limited to 15-bit codes.
Note that an unsigned long long type is used for counting. It is quite easy
to exceed the capacity of an eight-byte integer with a large number of
symbols and a large maximum code length, so multiple-precision arithmetic
would need to replace the unsigned long long arithmetic in that case. This
program will abort if an overflow occurs. The big_t type identifies where
the counting takes place.
Note that the uintmax_t type is used for counting. It is quite easy to
exceed the capacity of an eight-byte integer with a large number of symbols
and a large maximum code length, so multiple-precision arithmetic would need
to replace the integer arithmetic in that case. This program will abort if
an overflow occurs. The big_t type identifies where the counting takes
place.
An unsigned long long type is also used for calculating the number of
possible codes remaining at the maximum length. This limits the maximum
code length to the number of bits in a long long minus the number of bits
needed to represent the symbols in a flat code. The code_t type identifies
where the bit pattern counting takes place.
The uintmax_t type is also used for calculating the number of possible codes
remaining at the maximum length. This limits the maximum code length to the
number of bits in a long long minus the number of bits needed to represent
the symbols in a flat code. The code_t type identifies where the bit-pattern
counting takes place.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>
#include <stdint.h>
#include <assert.h>
#define local static
/* special data types */
typedef unsigned long long big_t; /* type for code counting */
typedef unsigned long long code_t; /* type for bit pattern counting */
struct tab { /* type for been here check */
size_t len; /* length of bit vector in char's */
char *vec; /* allocated bit vector */
// Special data types.
typedef uintmax_t big_t; // type for code counting
#define PRIbig "ju" // printf format for big_t
typedef uintmax_t code_t; // type for bit pattern counting
struct tab { // type for been-here check
size_t len; // allocated length of bit vector in octets
char *vec; // allocated bit vector
};
/* The array for saving results, num[], is indexed with this triplet:
@ -126,447 +132,466 @@ struct tab { /* type for been here check */
left: 2..syms - 1, but only the evens (so syms == 8 -> 2, 4, 6)
len: 1..max - 1 (max == maximum code length in bits)
syms == 2 is not saved since that immediately leads to a single code. left
syms == 2 is not saved since that immediately leads to a single code. left
must be even, since it represents the number of available bit patterns at
the current length, which is double the number at the previous length.
left ends at syms-1 since left == syms immediately results in a single code.
the current length, which is double the number at the previous length. left
ends at syms-1 since left == syms immediately results in a single code.
(left > sym is not allowed since that would result in an incomplete code.)
len is less than max, since the code completes immediately when len == max.
The offset into the array is calculated for the three indices with the
first one (syms) being outermost, and the last one (len) being innermost.
We build the array with length max-1 lists for the len index, with syms-3
of those for each symbol. There are totsym-2 of those, with each one
varying in length as a function of sym. See the calculation of index in
count() for the index, and the calculation of size in main() for the size
of the array.
The offset into the array is calculated for the three indices with the first
one (syms) being outermost, and the last one (len) being innermost. We build
the array with length max-1 lists for the len index, with syms-3 of those
for each symbol. There are totsym-2 of those, with each one varying in
length as a function of sym. See the calculation of index in map() for the
index, and the calculation of size in main() for the size of the array.
For the deflate example of 286 symbols limited to 15-bit codes, the array
has 284,284 entries, taking up 2.17 MB for an 8-byte big_t. More than
half of the space allocated for saved results is actually used -- not all
possible triplets are reached in the generation of valid Huffman codes.
has 284,284 entries, taking up 2.17 MB for an 8-byte big_t. More than half
of the space allocated for saved results is actually used -- not all
possible triplets are reached in the generation of valid prefix codes.
*/
/* The array for tracking visited states, done[], is itself indexed identically
to the num[] array as described above for the (syms, left, len) triplet.
Each element in the array is further indexed by the (mem, rem) doublet,
where mem is the amount of inflate table space used so far, and rem is the
remaining unused entries in the current inflate sub-table. Each indexed
remaining unused entries in the current inflate sub-table. Each indexed
element is simply one bit indicating whether the state has been visited or
not. Since the ranges for mem and rem are not known a priori, each bit
not. Since the ranges for mem and rem are not known a priori, each bit
vector is of a variable size, and grows as needed to accommodate the visited
states. mem and rem are used to calculate a single index in a triangular
array. Since the range of mem is expected in the default case to be about
states. mem and rem are used to calculate a single index in a triangular
array. Since the range of mem is expected in the default case to be about
ten times larger than the range of rem, the array is skewed to reduce the
memory usage, with eight times the range for mem than for rem. See the
calculations for offset and bit in beenhere() for the details.
memory usage, with eight times the range for mem than for rem. See the
calculations for offset and bit in been_here() for the details.
For the deflate example of 286 symbols limited to 15-bit codes, the bit
vectors grow to total approximately 21 MB, in addition to the 4.3 MB done[]
array itself.
vectors grow to total 5.5 MB, in addition to the 4.3 MB done array itself.
*/
/* Globals to avoid propagating constants or constant pointers recursively */
local int max; /* maximum allowed bit length for the codes */
local int root; /* size of base code table in bits */
local int large; /* largest code table so far */
local size_t size; /* number of elements in num and done */
local int *code; /* number of symbols assigned to each bit length */
local big_t *num; /* saved results array for code counting */
local struct tab *done; /* states already evaluated array */
// Type for a variable-length, allocated string.
typedef struct {
char *str; // pointer to allocated string
size_t size; // size of allocation
size_t len; // length of string, not including terminating zero
} string_t;
/* Index function for num[] and done[] */
#define INDEX(i,j,k) (((size_t)((i-1)>>1)*((i-2)>>1)+(j>>1)-1)*(max-1)+k-1)
/* Free allocated space. Uses globals code, num, and done. */
local void cleanup(void)
{
size_t n;
if (done != NULL) {
for (n = 0; n < size; n++)
if (done[n].len)
free(done[n].vec);
free(done);
}
if (num != NULL)
free(num);
if (code != NULL)
free(code);
// Clear a string_t.
local void string_clear(string_t *s) {
s->str[0] = 0;
s->len = 0;
}
/* Return the number of possible Huffman codes using bit patterns of lengths
len through max inclusive, coding syms symbols, with left bit patterns of
length len unused -- return -1 if there is an overflow in the counting.
Keep a record of previous results in num to prevent repeating the same
calculation. Uses the globals max and num. */
local big_t count(int syms, int len, int left)
{
big_t sum; /* number of possible codes from this juncture */
big_t got; /* value returned from count() */
int least; /* least number of syms to use at this juncture */
int most; /* most number of syms to use at this juncture */
int use; /* number of bit patterns to use in next call */
size_t index; /* index of this case in *num */
// Initialize a string_t.
local void string_init(string_t *s) {
s->size = 16;
s->str = malloc(s->size);
assert(s->str != NULL && "out of memory");
string_clear(s);
}
/* see if only one possible code */
// Release the allocation of a string_t.
local void string_free(string_t *s) {
free(s->str);
s->str = NULL;
s->size = 0;
s->len = 0;
}
// Save the results of printf with fmt and the subsequent argument list to s.
// Each call appends to s. The allocated space for s is increased as needed.
local void string_printf(string_t *s, char *fmt, ...) {
va_list ap;
va_start(ap, fmt);
size_t len = s->len;
int ret = vsnprintf(s->str + len, s->size - len, fmt, ap);
assert(ret >= 0 && "out of memory");
s->len += ret;
if (s->size < s->len + 1) {
do {
s->size <<= 1;
assert(s->size != 0 && "overflow");
} while (s->size < s->len + 1);
s->str = realloc(s->str, s->size);
assert(s->str != NULL && "out of memory");
vsnprintf(s->str + len, s->size - len, fmt, ap);
}
va_end(ap);
}
// Globals to avoid propagating constants or constant pointers recursively.
struct {
int max; // maximum allowed bit length for the codes
int root; // size of base code table in bits
int large; // largest code table so far
size_t size; // number of elements in num and done
big_t tot; // total number of codes with maximum tables size
string_t out; // display of subcodes for maximum tables size
int *code; // number of symbols assigned to each bit length
big_t *num; // saved results array for code counting
struct tab *done; // states already evaluated array
} g;
// Index function for num[] and done[].
local inline size_t map(int syms, int left, int len) {
return ((size_t)((syms - 1) >> 1) * ((syms - 2) >> 1) +
(left >> 1) - 1) * (g.max - 1) +
len - 1;
}
// Free allocated space in globals.
local void cleanup(void) {
if (g.done != NULL) {
for (size_t n = 0; n < g.size; n++)
if (g.done[n].len)
free(g.done[n].vec);
g.size = 0;
free(g.done); g.done = NULL;
}
free(g.num); g.num = NULL;
free(g.code); g.code = NULL;
string_free(&g.out);
}
// Return the number of possible prefix codes using bit patterns of lengths len
// through max inclusive, coding syms symbols, with left bit patterns of length
// len unused -- return -1 if there is an overflow in the counting. Keep a
// record of previous results in num to prevent repeating the same calculation.
local big_t count(int syms, int left, int len) {
// see if only one possible code
if (syms == left)
return 1;
/* note and verify the expected state */
assert(syms > left && left > 0 && len < max);
// note and verify the expected state
assert(syms > left && left > 0 && len < g.max);
/* see if we've done this one already */
index = INDEX(syms, left, len);
got = num[index];
// see if we've done this one already
size_t index = map(syms, left, len);
big_t got = g.num[index];
if (got)
return got; /* we have -- return the saved result */
return got; // we have -- return the saved result
/* we need to use at least this many bit patterns so that the code won't be
incomplete at the next length (more bit patterns than symbols) */
least = (left << 1) - syms;
// we need to use at least this many bit patterns so that the code won't be
// incomplete at the next length (more bit patterns than symbols)
int least = (left << 1) - syms;
if (least < 0)
least = 0;
/* we can use at most this many bit patterns, lest there not be enough
available for the remaining symbols at the maximum length (if there were
no limit to the code length, this would become: most = left - 1) */
most = (((code_t)left << (max - len)) - syms) /
(((code_t)1 << (max - len)) - 1);
// we can use at most this many bit patterns, lest there not be enough
// available for the remaining symbols at the maximum length (if there were
// no limit to the code length, this would become: most = left - 1)
int most = (((code_t)left << (g.max - len)) - syms) /
(((code_t)1 << (g.max - len)) - 1);
/* count all possible codes from this juncture and add them up */
sum = 0;
for (use = least; use <= most; use++) {
got = count(syms - use, len + 1, (left - use) << 1);
// count all possible codes from this juncture and add them up
big_t sum = 0;
for (int use = least; use <= most; use++) {
got = count(syms - use, (left - use) << 1, len + 1);
sum += got;
if (got == (big_t)0 - 1 || sum < got) /* overflow */
return (big_t)0 - 1;
if (got == (big_t)-1 || sum < got) // overflow
return (big_t)-1;
}
/* verify that all recursive calls are productive */
// verify that all recursive calls are productive
assert(sum != 0);
/* save the result and return it */
num[index] = sum;
// save the result and return it
g.num[index] = sum;
return sum;
}
/* Return true if we've been here before, set to true if not. Set a bit in a
bit vector to indicate visiting this state. Each (syms,len,left) state
has a variable size bit vector indexed by (mem,rem). The bit vector is
lengthened if needed to allow setting the (mem,rem) bit. */
local int beenhere(int syms, int len, int left, int mem, int rem)
{
size_t index; /* index for this state's bit vector */
size_t offset; /* offset in this state's bit vector */
int bit; /* mask for this state's bit */
size_t length; /* length of the bit vector in bytes */
char *vector; /* new or enlarged bit vector */
/* point to vector for (syms,left,len), bit in vector for (mem,rem) */
index = INDEX(syms, left, len);
mem -= 1 << root;
offset = (mem >> 3) + rem;
// Return true if we've been here before, set to true if not. Set a bit in a
// bit vector to indicate visiting this state. Each (syms,len,left) state has a
// variable size bit vector indexed by (mem,rem). The bit vector is lengthened
// as needed to allow setting the (mem,rem) bit.
local int been_here(int syms, int left, int len, int mem, int rem) {
// point to vector for (syms,left,len), bit in vector for (mem,rem)
size_t index = map(syms, left, len);
mem -= 1 << g.root; // mem always includes the root table
mem >>= 1; // mem and rem are always even
rem >>= 1;
size_t offset = (mem >> 3) + rem;
offset = ((offset * (offset + 1)) >> 1) + rem;
bit = 1 << (mem & 7);
int bit = 1 << (mem & 7);
/* see if we've been here */
length = done[index].len;
if (offset < length && (done[index].vec[offset] & bit) != 0)
return 1; /* done this! */
// see if we've been here
size_t length = g.done[index].len;
if (offset < length && (g.done[index].vec[offset] & bit) != 0)
return 1; // done this!
/* we haven't been here before -- set the bit to show we have now */
// we haven't been here before -- set the bit to show we have now
/* see if we need to lengthen the vector in order to set the bit */
// see if we need to lengthen the vector in order to set the bit
if (length <= offset) {
/* if we have one already, enlarge it, zero out the appended space */
// if we have one already, enlarge it, zero out the appended space
char *vector;
if (length) {
do {
length <<= 1;
} while (length <= offset);
vector = realloc(done[index].vec, length);
if (vector != NULL)
memset(vector + done[index].len, 0, length - done[index].len);
vector = realloc(g.done[index].vec, length);
assert(vector != NULL && "out of memory");
memset(vector + g.done[index].len, 0, length - g.done[index].len);
}
/* otherwise we need to make a new vector and zero it out */
// otherwise we need to make a new vector and zero it out
else {
length = 1 << (len - root);
length = 16;
while (length <= offset)
length <<= 1;
vector = calloc(length, sizeof(char));
vector = calloc(length, 1);
assert(vector != NULL && "out of memory");
}
/* in either case, bail if we can't get the memory */
if (vector == NULL) {
fputs("abort: unable to allocate enough memory\n", stderr);
cleanup();
exit(1);
}
/* install the new vector */
done[index].len = length;
done[index].vec = vector;
// install the new vector
g.done[index].len = length;
g.done[index].vec = vector;
}
/* set the bit */
done[index].vec[offset] |= bit;
// set the bit
g.done[index].vec[offset] |= bit;
return 0;
}
/* Examine all possible codes from the given node (syms, len, left). Compute
the amount of memory required to build inflate's decoding tables, where the
number of code structures used so far is mem, and the number remaining in
the current sub-table is rem. Uses the globals max, code, root, large, and
done. */
local void examine(int syms, int len, int left, int mem, int rem)
{
int least; /* least number of syms to use at this juncture */
int most; /* most number of syms to use at this juncture */
int use; /* number of bit patterns to use in next call */
/* see if we have a complete code */
// Examine all possible codes from the given node (syms, len, left). Compute
// the amount of memory required to build inflate's decoding tables, where the
// number of code structures used so far is mem, and the number remaining in
// the current sub-table is rem.
local void examine(int syms, int left, int len, int mem, int rem) {
// see if we have a complete code
if (syms == left) {
/* set the last code entry */
code[len] = left;
// set the last code entry
g.code[len] = left;
/* complete computation of memory used by this code */
// complete computation of memory used by this code
while (rem < left) {
left -= rem;
rem = 1 << (len - root);
rem = 1 << (len - g.root);
mem += rem;
}
assert(rem == left);
/* if this is a new maximum, show the entries used and the sub-code */
if (mem > large) {
large = mem;
printf("max %d: ", mem);
for (use = root + 1; use <= max; use++)
if (code[use])
printf("%d[%d] ", code[use], use);
putchar('\n');
fflush(stdout);
// if this is at the maximum, show the sub-code
if (mem >= g.large) {
// if this is a new maximum, update the maximum and clear out the
// printed sub-codes from the previous maximum
if (mem > g.large) {
g.large = mem;
string_clear(&g.out);
}
// compute the starting state for this sub-code
syms = 0;
left = 1 << g.max;
for (int bits = g.max; bits > g.root; bits--) {
syms += g.code[bits];
left -= g.code[bits];
assert((left & 1) == 0);
left >>= 1;
}
// print the starting state and the resulting sub-code to g.out
string_printf(&g.out, "<%u, %u, %u>:",
syms, g.root + 1, ((1 << g.root) - left) << 1);
for (int bits = g.root + 1; bits <= g.max; bits++)
if (g.code[bits])
string_printf(&g.out, " %d[%d]", g.code[bits], bits);
string_printf(&g.out, "\n");
}
/* remove entries as we drop back down in the recursion */
code[len] = 0;
// remove entries as we drop back down in the recursion
g.code[len] = 0;
return;
}
/* prune the tree if we can */
if (beenhere(syms, len, left, mem, rem))
// prune the tree if we can
if (been_here(syms, left, len, mem, rem))
return;
/* we need to use at least this many bit patterns so that the code won't be
incomplete at the next length (more bit patterns than symbols) */
least = (left << 1) - syms;
// we need to use at least this many bit patterns so that the code won't be
// incomplete at the next length (more bit patterns than symbols)
int least = (left << 1) - syms;
if (least < 0)
least = 0;
/* we can use at most this many bit patterns, lest there not be enough
available for the remaining symbols at the maximum length (if there were
no limit to the code length, this would become: most = left - 1) */
most = (((code_t)left << (max - len)) - syms) /
(((code_t)1 << (max - len)) - 1);
// we can use at most this many bit patterns, lest there not be enough
// available for the remaining symbols at the maximum length (if there were
// no limit to the code length, this would become: most = left - 1)
int most = (((code_t)left << (g.max - len)) - syms) /
(((code_t)1 << (g.max - len)) - 1);
/* occupy least table spaces, creating new sub-tables as needed */
use = least;
// occupy least table spaces, creating new sub-tables as needed
int use = least;
while (rem < use) {
use -= rem;
rem = 1 << (len - root);
rem = 1 << (len - g.root);
mem += rem;
}
rem -= use;
/* examine codes from here, updating table space as we go */
// examine codes from here, updating table space as we go
for (use = least; use <= most; use++) {
code[len] = use;
examine(syms - use, len + 1, (left - use) << 1,
mem + (rem ? 1 << (len - root) : 0), rem << 1);
g.code[len] = use;
examine(syms - use, (left - use) << 1, len + 1,
mem + (rem ? 1 << (len - g.root) : 0), rem << 1);
if (rem == 0) {
rem = 1 << (len - root);
rem = 1 << (len - g.root);
mem += rem;
}
rem--;
}
/* remove entries as we drop back down in the recursion */
code[len] = 0;
// remove entries as we drop back down in the recursion
g.code[len] = 0;
}
/* Look at all sub-codes starting with root + 1 bits. Look at only the valid
intermediate code states (syms, left, len). For each completed code,
calculate the amount of memory required by inflate to build the decoding
tables. Find the maximum amount of memory required and show the code that
requires that maximum. Uses the globals max, root, and num. */
local void enough(int syms)
{
int n; /* number of remaing symbols for this node */
int left; /* number of unused bit patterns at this length */
size_t index; /* index of this case in *num */
// Look at all sub-codes starting with root + 1 bits. Look at only the valid
// intermediate code states (syms, left, len). For each completed code,
// calculate the amount of memory required by inflate to build the decoding
// tables. Find the maximum amount of memory required and show the codes that
// require that maximum.
local void enough(int syms) {
// clear code
for (int n = 0; n <= g.max; n++)
g.code[n] = 0;
/* clear code */
for (n = 0; n <= max; n++)
code[n] = 0;
// look at all (root + 1) bit and longer codes
string_clear(&g.out); // empty saved results
g.large = 1 << g.root; // base table
if (g.root < g.max) // otherwise, there's only a base table
for (int n = 3; n <= syms; n++)
for (int left = 2; left < n; left += 2) {
// look at all reachable (root + 1) bit nodes, and the
// resulting codes (complete at root + 2 or more)
size_t index = map(n, left, g.root + 1);
if (g.root + 1 < g.max && g.num[index]) // reachable node
examine(n, left, g.root + 1, 1 << g.root, 0);
/* look at all (root + 1) bit and longer codes */
large = 1 << root; /* base table */
if (root < max) /* otherwise, there's only a base table */
for (n = 3; n <= syms; n++)
for (left = 2; left < n; left += 2)
{
/* look at all reachable (root + 1) bit nodes, and the
resulting codes (complete at root + 2 or more) */
index = INDEX(n, left, root + 1);
if (root + 1 < max && num[index]) /* reachable node */
examine(n, root + 1, left, 1 << root, 0);
/* also look at root bit codes with completions at root + 1
bits (not saved in num, since complete), just in case */
if (num[index - 1] && n <= left << 1)
examine((n - left) << 1, root + 1, (n - left) << 1,
1 << root, 0);
// also look at root bit codes with completions at root + 1
// bits (not saved in num, since complete), just in case
if (g.num[index - 1] && n <= left << 1)
examine((n - left) << 1, (n - left) << 1, g.root + 1,
1 << g.root, 0);
}
/* done */
printf("done: maximum of %d table entries\n", large);
// done
printf("maximum of %d table entries for root = %d\n", g.large, g.root);
fputs(g.out.str, stdout);
}
/*
Examine and show the total number of possible Huffman codes for a given
maximum number of symbols, initial root table size, and maximum code length
in bits -- those are the command arguments in that order. The default
values are 286, 9, and 15 respectively, for the deflate literal/length code.
The possible codes are counted for each number of coded symbols from two to
the maximum. The counts for each of those and the total number of codes are
shown. The maximum number of inflate table entires is then calculated
across all possible codes. Each new maximum number of table entries and the
associated sub-code (starting at root + 1 == 10 bits) is shown.
// Examine and show the total number of possible prefix codes for a given
// maximum number of symbols, initial root table size, and maximum code length
// in bits -- those are the command arguments in that order. The default values
// are 286, 9, and 15 respectively, for the deflate literal/length code. The
// possible codes are counted for each number of coded symbols from two to the
// maximum. The counts for each of those and the total number of codes are
// shown. The maximum number of inflate table entries is then calculated across
// all possible codes. Each new maximum number of table entries and the
// associated sub-code (starting at root + 1 == 10 bits) is shown.
//
// To count and examine prefix codes that are not length-limited, provide a
// maximum length equal to the number of symbols minus one.
//
// For the deflate literal/length code, use "enough". For the deflate distance
// code, use "enough 30 6".
int main(int argc, char **argv) {
// set up globals for cleanup()
g.code = NULL;
g.num = NULL;
g.done = NULL;
string_init(&g.out);
To count and examine Huffman codes that are not length-limited, provide a
maximum length equal to the number of symbols minus one.
For the deflate literal/length code, use "enough". For the deflate distance
code, use "enough 30 6".
This uses the %llu printf format to print big_t numbers, which assumes that
big_t is an unsigned long long. If the big_t type is changed (for example
to a multiple precision type), the method of printing will also need to be
updated.
*/
int main(int argc, char **argv)
{
int syms; /* total number of symbols to code */
int n; /* number of symbols to code for this run */
big_t got; /* return value of count() */
big_t sum; /* accumulated number of codes over n */
code_t word; /* for counting bits in code_t */
/* set up globals for cleanup() */
code = NULL;
num = NULL;
done = NULL;
/* get arguments -- default to the deflate literal/length code */
syms = 286;
root = 9;
max = 15;
// get arguments -- default to the deflate literal/length code
int syms = 286;
g.root = 9;
g.max = 15;
if (argc > 1) {
syms = atoi(argv[1]);
if (argc > 2) {
root = atoi(argv[2]);
g.root = atoi(argv[2]);
if (argc > 3)
max = atoi(argv[3]);
g.max = atoi(argv[3]);
}
}
if (argc > 4 || syms < 2 || root < 1 || max < 1) {
if (argc > 4 || syms < 2 || g.root < 1 || g.max < 1) {
fputs("invalid arguments, need: [sym >= 2 [root >= 1 [max >= 1]]]\n",
stderr);
return 1;
}
/* if not restricting the code length, the longest is syms - 1 */
if (max > syms - 1)
max = syms - 1;
// if not restricting the code length, the longest is syms - 1
if (g.max > syms - 1)
g.max = syms - 1;
/* determine the number of bits in a code_t */
for (n = 0, word = 1; word; n++, word <<= 1)
;
// determine the number of bits in a code_t
int bits = 0;
for (code_t word = 1; word; word <<= 1)
bits++;
/* make sure that the calculation of most will not overflow */
if (max > n || (code_t)(syms - 2) >= (((code_t)0 - 1) >> (max - 1))) {
// make sure that the calculation of most will not overflow
if (g.max > bits || (code_t)(syms - 2) >= ((code_t)-1 >> (g.max - 1))) {
fputs("abort: code length too long for internal types\n", stderr);
return 1;
}
/* reject impossible code requests */
if ((code_t)(syms - 1) > ((code_t)1 << max) - 1) {
// reject impossible code requests
if ((code_t)(syms - 1) > ((code_t)1 << g.max) - 1) {
fprintf(stderr, "%d symbols cannot be coded in %d bits\n",
syms, max);
syms, g.max);
return 1;
}
/* allocate code vector */
code = calloc(max + 1, sizeof(int));
if (code == NULL) {
fputs("abort: unable to allocate enough memory\n", stderr);
return 1;
}
// allocate code vector
g.code = calloc(g.max + 1, sizeof(int));
assert(g.code != NULL && "out of memory");
/* determine size of saved results array, checking for overflows,
allocate and clear the array (set all to zero with calloc()) */
if (syms == 2) /* iff max == 1 */
num = NULL; /* won't be saving any results */
// determine size of saved results array, checking for overflows,
// allocate and clear the array (set all to zero with calloc())
if (syms == 2) // iff max == 1
g.num = NULL; // won't be saving any results
else {
size = syms >> 1;
if (size > ((size_t)0 - 1) / (n = (syms - 1) >> 1) ||
(size *= n, size > ((size_t)0 - 1) / (n = max - 1)) ||
(size *= n, size > ((size_t)0 - 1) / sizeof(big_t)) ||
(num = calloc(size, sizeof(big_t))) == NULL) {
fputs("abort: unable to allocate enough memory\n", stderr);
cleanup();
return 1;
}
g.size = syms >> 1;
int n = (syms - 1) >> 1;
assert(g.size <= (size_t)-1 / n && "overflow");
g.size *= n;
n = g.max - 1;
assert(g.size <= (size_t)-1 / n && "overflow");
g.size *= n;
g.num = calloc(g.size, sizeof(big_t));
assert(g.num != NULL && "out of memory");
}
/* count possible codes for all numbers of symbols, add up counts */
sum = 0;
for (n = 2; n <= syms; n++) {
got = count(n, 1, 2);
// count possible codes for all numbers of symbols, add up counts
big_t sum = 0;
for (int n = 2; n <= syms; n++) {
big_t got = count(n, 2, 1);
sum += got;
if (got == (big_t)0 - 1 || sum < got) { /* overflow */
fputs("abort: can't count that high!\n", stderr);
cleanup();
return 1;
}
printf("%llu %d-codes\n", got, n);
assert(got != (big_t)-1 && sum >= got && "overflow");
}
printf("%llu total codes for 2 to %d symbols", sum, syms);
if (max < syms - 1)
printf(" (%d-bit length limit)\n", max);
printf("%"PRIbig" total codes for 2 to %d symbols", sum, syms);
if (g.max < syms - 1)
printf(" (%d-bit length limit)\n", g.max);
else
puts(" (no length limit)");
/* allocate and clear done array for beenhere() */
// allocate and clear done array for been_here()
if (syms == 2)
done = NULL;
else if (size > ((size_t)0 - 1) / sizeof(struct tab) ||
(done = calloc(size, sizeof(struct tab))) == NULL) {
fputs("abort: unable to allocate enough memory\n", stderr);
cleanup();
return 1;
g.done = NULL;
else {
g.done = calloc(g.size, sizeof(struct tab));
assert(g.done != NULL && "out of memory");
}
/* find and show maximum inflate table usage */
if (root > max) /* reduce root to max length */
root = max;
if ((code_t)syms < ((code_t)1 << (root + 1)))
// find and show maximum inflate table usage
if (g.root > g.max) // reduce root to max length
g.root = g.max;
if ((code_t)syms < ((code_t)1 << (g.root + 1)))
enough(syms);
else
puts("cannot handle minimum code lengths > root");
fputs("cannot handle minimum code lengths > root", stderr);
/* done */
// done
cleanup();
return 0;
}

6
extern/zlib/examples/fitblk.c vendored
View file

@ -17,7 +17,7 @@
data in order to determine how much of that input will compress to
nearly the requested output block size. The first pass generates
enough deflate blocks to produce output to fill the requested
output size plus a specfied excess amount (see the EXCESS define
output size plus a specified excess amount (see the EXCESS define
below). The last deflate block may go quite a bit past that, but
is discarded. The second pass decompresses and recompresses just
the compressed data that fit in the requested plus excess sized
@ -109,7 +109,7 @@ local int recompress(z_streamp inf, z_streamp def)
if (ret == Z_MEM_ERROR)
return ret;
/* compress what was decompresed until done or no room */
/* compress what was decompressed until done or no room */
def->avail_in = RAWLEN - inf->avail_out;
def->next_in = raw;
if (inf->avail_out != 0)
@ -198,7 +198,7 @@ int main(int argc, char **argv)
if (ret == Z_MEM_ERROR)
quit("out of memory");
/* set up for next reocmpression */
/* set up for next recompression */
ret = inflateReset(&inf);
assert(ret != Z_STREAM_ERROR);
ret = deflateReset(&def);

2
extern/zlib/examples/gun.c vendored
View file

@ -43,7 +43,7 @@
gun will also decompress files made by Unix compress, which uses LZW
compression. These files are automatically detected by virtue of their
magic header bytes. Since the end of Unix compress stream is marked by the
end-of-file, they cannot be concantenated. If a Unix compress stream is
end-of-file, they cannot be concatenated. If a Unix compress stream is
encountered in an input file, it is the last stream in that file.
Like gunzip and uncompress, the file attributes of the original compressed

6
extern/zlib/examples/gzappend.c vendored
View file

@ -33,7 +33,7 @@
* - Add L to constants in lseek() calls
* - Remove some debugging information in error messages
* - Use new data_type definition for zlib 1.2.1
* - Simplfy and unify file operations
* - Simplify and unify file operations
* - Finish off gzip file in gztack()
* - Use deflatePrime() instead of adding empty blocks
* - Keep gzip file clean on appended file read errors
@ -54,7 +54,7 @@
block boundary to facilitate locating and modifying the last block bit at
the start of the final deflate block. Also whether using Z_BLOCK or not,
another required feature of zlib 1.2.x is that inflate() now provides the
number of unusued bits in the last input byte used. gzappend will not work
number of unused bits in the last input byte used. gzappend will not work
with versions of zlib earlier than 1.2.1.
gzappend first decompresses the gzip file internally, discarding all but
@ -137,7 +137,7 @@ local void rotate(unsigned char *list, unsigned len, unsigned rot)
/* do simple left shift by one */
if (rot == 1) {
tmp = *list;
memcpy(list, list + 1, len - 1);
memmove(list, list + 1, len - 1);
*last = tmp;
return;
}

10
extern/zlib/examples/gzlog.c vendored
View file

@ -1,8 +1,8 @@
/*
* gzlog.c
* Copyright (C) 2004, 2008, 2012, 2016 Mark Adler, all rights reserved
* Copyright (C) 2004, 2008, 2012, 2016, 2019 Mark Adler, all rights reserved
* For conditions of distribution and use, see copyright notice in gzlog.h
* version 2.2, 14 Aug 2012
* version 2.3, 25 May 2019
*/
/*
@ -212,8 +212,8 @@
to the appropriate recovery below. If there is no foo.add file, provide
a zero data length to the recovery. In that case, the append recovery
restores the foo.gz to the previous compressed + uncompressed data state.
For the the compress recovery, a missing foo.add file results in foo.gz
being restored to the previous compressed-only data state.
For the compress recovery, a missing foo.add file results in foo.gz being
restored to the previous compressed-only data state.
- Append recovery:
- Pick up append at + step above
- Compress recovery:
@ -756,12 +756,14 @@ local int log_recover(struct log *log, int op)
return -2;
}
if ((fd = open(log->path, O_RDONLY, 0)) < 0) {
free(data);
log_log(log, op, ".add file read failure");
return -1;
}
ret = (size_t)read(fd, data, len) != len;
close(fd);
if (ret) {
free(data);
log_log(log, op, ".add file read failure");
return -1;
}

2
extern/zlib/examples/gzlog.h vendored
View file

@ -40,7 +40,7 @@
its new size at that time. After each write operation, the log file is a
valid gzip file that can decompressed to recover what was written.
The gzlog operations can be interupted at any point due to an application or
The gzlog operations can be interrupted at any point due to an application or
system crash, and the log file will be recovered the next time the log is
opened with gzlog_open().
*/

470
extern/zlib/examples/gznorm.c vendored Normal file
View file

@ -0,0 +1,470 @@
/* gznorm.c -- normalize a gzip stream
* Copyright (C) 2018 Mark Adler
* For conditions of distribution and use, see copyright notice in zlib.h
* Version 1.0 7 Oct 2018 Mark Adler */
// gznorm takes a gzip stream, potentially containing multiple members, and
// converts it to a gzip stream with a single member. In addition the gzip
// header is normalized, removing the file name and time stamp, and setting the
// other header contents (XFL, OS) to fixed values. gznorm does not recompress
// the data, so it is fast, but no advantage is gained from the history that
// could be available across member boundaries.
#include <stdio.h> // fread, fwrite, putc, fflush, ferror, fprintf,
// vsnprintf, stdout, stderr, NULL, FILE
#include <stdlib.h> // malloc, free
#include <string.h> // strerror
#include <errno.h> // errno
#include <stdarg.h> // va_list, va_start, va_end
#include "zlib.h" // inflateInit2, inflate, inflateReset, inflateEnd,
// z_stream, z_off_t, crc32_combine, Z_NULL, Z_BLOCK,
// Z_OK, Z_STREAM_END, Z_BUF_ERROR, Z_DATA_ERROR,
// Z_MEM_ERROR
#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(__CYGWIN__)
# include <fcntl.h>
# include <io.h>
# define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY)
#else
# define SET_BINARY_MODE(file)
#endif
#define local static
// printf to an allocated string. Return the string, or NULL if the printf or
// allocation fails.
local char *aprintf(char *fmt, ...) {
// Get the length of the result of the printf.
va_list args;
va_start(args, fmt);
int len = vsnprintf(NULL, 0, fmt, args);
va_end(args);
if (len < 0)
return NULL;
// Allocate the required space and printf to it.
char *str = malloc(len + 1);
if (str == NULL)
return NULL;
va_start(args, fmt);
vsnprintf(str, len + 1, fmt, args);
va_end(args);
return str;
}
// Return with an error, putting an allocated error message in *err. Doing an
// inflateEnd() on an already ended state, or one with state set to Z_NULL, is
// permitted.
#define BYE(...) \
do { \
inflateEnd(&strm); \
*err = aprintf(__VA_ARGS__); \
return 1; \
} while (0)
// Chunk size for buffered reads and for decompression. Twice this many bytes
// will be allocated on the stack by gzip_normalize(). Must fit in an unsigned.
#define CHUNK 16384
// Read a gzip stream from in and write an equivalent normalized gzip stream to
// out. If given no input, an empty gzip stream will be written. If successful,
// 0 is returned, and *err is set to NULL. On error, 1 is returned, where the
// details of the error are returned in *err, a pointer to an allocated string.
//
// The input may be a stream with multiple gzip members, which is converted to
// a single gzip member on the output. Each gzip member is decompressed at the
// level of deflate blocks. This enables clearing the last-block bit, shifting
// the compressed data to concatenate to the previous member's compressed data,
// which can end at an arbitrary bit boundary, and identifying stored blocks in
// order to resynchronize those to byte boundaries. The deflate compressed data
// is terminated with a 10-bit empty fixed block. If any members on the input
// end with a 10-bit empty fixed block, then that block is excised from the
// stream. This avoids appending empty fixed blocks for every normalization,
// and assures that gzip_normalize applied a second time will not change the
// input. The pad bits after stored block headers and after the final deflate
// block are all forced to zeros.
local int gzip_normalize(FILE *in, FILE *out, char **err) {
// initialize the inflate engine to process a gzip member
z_stream strm;
strm.zalloc = Z_NULL;
strm.zfree = Z_NULL;
strm.opaque = Z_NULL;
strm.avail_in = 0;
strm.next_in = Z_NULL;
if (inflateInit2(&strm, 15 + 16) != Z_OK)
BYE("out of memory");
// State while processing the input gzip stream.
enum { // BETWEEN -> HEAD -> BLOCK -> TAIL -> BETWEEN -> ...
BETWEEN, // between gzip members (must end in this state)
HEAD, // reading a gzip header
BLOCK, // reading deflate blocks
TAIL // reading a gzip trailer
} state = BETWEEN; // current component being processed
unsigned long crc = 0; // accumulated CRC of uncompressed data
unsigned long len = 0; // accumulated length of uncompressed data
unsigned long buf = 0; // deflate stream bit buffer of num bits
int num = 0; // number of bits in buf (at bottom)
// Write a canonical gzip header (no mod time, file name, comment, extra
// block, or extra flags, and OS is marked as unknown).
fwrite("\x1f\x8b\x08\0\0\0\0\0\0\xff", 1, 10, out);
// Process the gzip stream from in until reaching the end of the input,
// encountering invalid input, or experiencing an i/o error.
int more; // true if not at the end of the input
do {
// State inside this loop.
unsigned char *put; // next input buffer location to process
int prev; // number of bits from previous block in
// the bit buffer, or -1 if not at the
// start of a block
unsigned long long memb; // uncompressed length of member
size_t tail; // number of trailer bytes read (0..8)
unsigned long part; // accumulated trailer component
// Get the next chunk of input from in.
unsigned char dat[CHUNK];
strm.avail_in = fread(dat, 1, CHUNK, in);
if (strm.avail_in == 0)
break;
more = strm.avail_in == CHUNK;
strm.next_in = put = dat;
// Run that chunk of input through the inflate engine to exhaustion.
do {
// At this point it is assured that strm.avail_in > 0.
// Inflate until the end of a gzip component (header, deflate
// block, trailer) is reached, or until all of the chunk is
// consumed. The resulting decompressed data is discarded, though
// the total size of the decompressed data in each member is
// tracked, for the calculation of the total CRC.
do {
// inflate and handle any errors
unsigned char scrap[CHUNK];
strm.avail_out = CHUNK;
strm.next_out = scrap;
int ret = inflate(&strm, Z_BLOCK);
if (ret == Z_MEM_ERROR)
BYE("out of memory");
if (ret == Z_DATA_ERROR)
BYE("input invalid: %s", strm.msg);
if (ret != Z_OK && ret != Z_BUF_ERROR && ret != Z_STREAM_END)
BYE("internal error");
// Update the number of uncompressed bytes generated in this
// member. The actual count (not modulo 2^32) is required to
// correctly compute the total CRC.
unsigned got = CHUNK - strm.avail_out;
memb += got;
if (memb < got)
BYE("overflow error");
// Continue to process this chunk until it is consumed, or
// until the end of a component (header, deflate block, or
// trailer) is reached.
} while (strm.avail_out == 0 && (strm.data_type & 0x80) == 0);
// Since strm.avail_in was > 0 for the inflate call, some input was
// just consumed. It is therefore assured that put < strm.next_in.
// Disposition the consumed component or part of a component.
switch (state) {
case BETWEEN:
state = HEAD;
// Fall through to HEAD when some or all of the header is
// processed.
case HEAD:
// Discard the header.
if (strm.data_type & 0x80) {
// End of header reached -- deflate blocks follow.
put = strm.next_in;
prev = num;
memb = 0;
state = BLOCK;
}
break;
case BLOCK:
// Copy the deflate stream to the output, but with the
// last-block-bit cleared. Re-synchronize stored block
// headers to the output byte boundaries. The bytes at
// put..strm.next_in-1 is the compressed data that has been
// processed and is ready to be copied to the output.
// At this point, it is assured that new compressed data is
// available, i.e., put < strm.next_in. If prev is -1, then
// that compressed data starts in the middle of a deflate
// block. If prev is not -1, then the bits in the bit
// buffer, possibly combined with the bits in *put, contain
// the three-bit header of the new deflate block. In that
// case, prev is the number of bits from the previous block
// that remain in the bit buffer. Since num is the number
// of bits in the bit buffer, we have that num - prev is
// the number of bits from the new block currently in the
// bit buffer.
// If strm.data_type & 0xc0 is 0x80, then the last byte of
// the available compressed data includes the last bits of
// the end of a deflate block. In that case, that last byte
// also has strm.data_type & 0x1f bits of the next deflate
// block, in the range 0..7. If strm.data_type & 0xc0 is
// 0xc0, then the last byte of the compressed data is the
// end of the deflate stream, followed by strm.data_type &
// 0x1f pad bits, also in the range 0..7.
// Set bits to the number of bits not yet consumed from the
// last byte. If we are at the end of the block, bits is
// either the number of bits in the last byte belonging to
// the next block, or the number of pad bits after the
// final block. In either of those cases, bits is in the
// range 0..7.
; // (required due to C syntax oddity)
int bits = strm.data_type & 0x1f;
if (prev != -1) {
// We are at the start of a new block. Clear the last
// block bit, and check for special cases. If it is a
// stored block, then emit the header and pad to the
// next byte boundary. If it is a final, empty fixed
// block, then excise it.
// Some or all of the three header bits for this block
// may already be in the bit buffer. Load any remaining
// header bits into the bit buffer.
if (num - prev < 3) {
buf += (unsigned long)*put++ << num;
num += 8;
}
// Set last to have a 1 in the position of the last
// block bit in the bit buffer.
unsigned long last = (unsigned long)1 << prev;
if (((buf >> prev) & 7) == 3) {
// This is a final fixed block. Load at least ten
// bits from this block, including the header, into
// the bit buffer. We already have at least three,
// so at most one more byte needs to be loaded.
if (num - prev < 10) {
if (put == strm.next_in)
// Need to go get and process more input.
// We'll end up back here to finish this.
break;
buf += (unsigned long)*put++ << num;
num += 8;
}
if (((buf >> prev) & 0x3ff) == 3) {
// That final fixed block is empty. Delete it
// to avoid adding an empty block every time a
// gzip stream is normalized.
num = prev;
buf &= last - 1; // zero the pad bits
}
}
else if (((buf >> prev) & 6) == 0) {
// This is a stored block. Flush to the next
// byte boundary after the three-bit header.
num = (prev + 10) & ~7;
buf &= last - 1; // zero the pad bits
}
// Clear the last block bit.
buf &= ~last;
// Write out complete bytes in the bit buffer.
while (num >= 8) {
putc(buf, out);
buf >>= 8;
num -= 8;
}
// If no more bytes left to process, then we have
// consumed the byte that had bits from the next block.
if (put == strm.next_in)
bits = 0;
}
// We are done handling the deflate block header. Now copy
// all or almost all of the remaining compressed data that
// has been processed so far. Don't copy one byte at the
// end if it contains bits from the next deflate block or
// pad bits at the end of a deflate block.
// mix is 1 if we are at the end of a deflate block, and if
// some of the bits in the last byte follow this block. mix
// is 0 if we are in the middle of a deflate block, if the
// deflate block ended on a byte boundary, or if all of the
// compressed data processed so far has been consumed.
int mix = (strm.data_type & 0x80) && bits;
// Copy all of the processed compressed data to the output,
// except for the last byte if it contains bits from the
// next deflate block or pad bits at the end of the deflate
// stream. Copy the data after shifting in num bits from
// buf in front of it, leaving num bits from the end of the
// compressed data in buf when done.
unsigned char *end = strm.next_in - mix;
if (put < end) {
if (num)
// Insert num bits from buf before the data being
// copied.
do {
buf += (unsigned)(*put++) << num;
putc(buf, out);
buf >>= 8;
} while (put < end);
else {
// No shifting needed -- write directly.
fwrite(put, 1, end - put, out);
put = end;
}
}
// Process the last processed byte if it wasn't written.
if (mix) {
// Load the last byte into the bit buffer.
buf += (unsigned)(*put++) << num;
num += 8;
if (strm.data_type & 0x40) {
// We are at the end of the deflate stream and
// there are bits pad bits. Discard the pad bits
// and write a byte to the output, if available.
// Leave the num bits left over in buf to prepend
// to the next deflate stream.
num -= bits;
if (num >= 8) {
putc(buf, out);
num -= 8;
buf >>= 8;
}
// Force the pad bits in the bit buffer to zeros.
buf &= ((unsigned long)1 << num) - 1;
// Don't need to set prev here since going to TAIL.
}
else
// At the end of an internal deflate block. Leave
// the last byte in the bit buffer to examine on
// the next entry to BLOCK, when more bits from the
// next block will be available.
prev = num - bits; // number of bits in buffer
// from current block
}
// Don't have a byte left over, so we are in the middle of
// a deflate block, or the deflate block ended on a byte
// boundary. Set prev appropriately for the next entry into
// BLOCK.
else if (strm.data_type & 0x80)
// The block ended on a byte boundary, so no header
// bits are in the bit buffer.
prev = num;
else
// In the middle of a deflate block, so no header here.
prev = -1;
// Check for the end of the deflate stream.
if ((strm.data_type & 0xc0) == 0xc0) {
// That ends the deflate stream on the input side, the
// pad bits were discarded, and any remaining bits from
// the last block in the stream are saved in the bit
// buffer to prepend to the next stream. Process the
// gzip trailer next.
tail = 0;
part = 0;
state = TAIL;
}
break;
case TAIL:
// Accumulate available trailer bytes to update the total
// CRC and the total uncompressed length.
do {
part = (part >> 8) + ((unsigned long)(*put++) << 24);
tail++;
if (tail == 4) {
// Update the total CRC.
z_off_t len2 = memb;
if (len2 < 0 || (unsigned long long)len2 != memb)
BYE("overflow error");
crc = crc ? crc32_combine(crc, part, len2) : part;
part = 0;
}
else if (tail == 8) {
// Update the total uncompressed length. (It's ok
// if this sum is done modulo 2^32.)
len += part;
// At the end of a member. Set up to inflate an
// immediately following gzip member. (If we made
// it this far, then the trailer was valid.)
if (inflateReset(&strm) != Z_OK)
BYE("internal error");
state = BETWEEN;
break;
}
} while (put < strm.next_in);
break;
}
// Process the input buffer until completely consumed.
} while (strm.avail_in > 0);
// Process input until end of file, invalid input, or i/o error.
} while (more);
// Done with the inflate engine.
inflateEnd(&strm);
// Verify the validity of the input.
if (state != BETWEEN)
BYE("input invalid: incomplete gzip stream");
// Write the remaining deflate stream bits, followed by a terminating
// deflate fixed block.
buf += (unsigned long)3 << num;
putc(buf, out);
putc(buf >> 8, out);
if (num > 6)
putc(0, out);
// Write the gzip trailer, which is the CRC and the uncompressed length
// modulo 2^32, both in little-endian order.
putc(crc, out);
putc(crc >> 8, out);
putc(crc >> 16, out);
putc(crc >> 24, out);
putc(len, out);
putc(len >> 8, out);
putc(len >> 16, out);
putc(len >> 24, out);
fflush(out);
// Check for any i/o errors.
if (ferror(in) || ferror(out))
BYE("i/o error: %s", strerror(errno));
// All good!
*err = NULL;
return 0;
}
// Normalize the gzip stream on stdin, writing the result to stdout.
int main(void) {
// Avoid end-of-line conversions on evil operating systems.
SET_BINARY_MODE(stdin);
SET_BINARY_MODE(stdout);
// Normalize from stdin to stdout, returning 1 on error, 0 if ok.
char *err;
int ret = gzip_normalize(stdin, stdout, &err);
if (ret)
fprintf(stderr, "gznorm error: %s\n", err);
free(err);
return ret;
}

26
extern/zlib/examples/zlib_how.html vendored
View file

@ -1,10 +1,10 @@
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/loose.dtd">
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>zlib Usage Example</title>
<!-- Copyright (c) 2004, 2005 Mark Adler. -->
<!-- Copyright (c) 2004-2023 Mark Adler. -->
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000FF" vlink="#00A000">
<h2 align="center"> zlib Usage Example </h2>
@ -17,7 +17,7 @@ from an input file to an output file using <tt>deflate()</tt> and <tt>inflate()<
annotations are interspersed between lines of the code. So please read between the lines.
We hope this helps explain some of the intricacies of <em>zlib</em>.
<p>
Without further adieu, here is the program <a href="zpipe.c"><tt>zpipe.c</tt></a>:
Without further ado, here is the program <a href="zpipe.c"><tt>zpipe.c</tt></a>:
<pre><b>
/* zpipe.c: example of proper use of zlib's inflate() and deflate()
Not copyrighted -- provided to the public domain
@ -155,13 +155,11 @@ before we fall out of the loop at the bottom.
</b></pre>
We start off by reading data from the input file. The number of bytes read is put directly
into <tt>avail_in</tt>, and a pointer to those bytes is put into <tt>next_in</tt>. We also
check to see if end-of-file on the input has been reached. If we are at the end of file, then <tt>flush</tt> is set to the
check to see if end-of-file on the input has been reached using feof().
If we are at the end of file, then <tt>flush</tt> is set to the
<em>zlib</em> constant <tt>Z_FINISH</tt>, which is later passed to <tt>deflate()</tt> to
indicate that this is the last chunk of input data to compress. We need to use <tt>feof()</tt>
to check for end-of-file as opposed to seeing if fewer than <tt>CHUNK</tt> bytes have been read. The
reason is that if the input file length is an exact multiple of <tt>CHUNK</tt>, we will miss
the fact that we got to the end-of-file, and not know to tell <tt>deflate()</tt> to finish
up the compressed stream. If we are not yet at the end of the input, then the <em>zlib</em>
indicate that this is the last chunk of input data to compress.
If we are not yet at the end of the input, then the <em>zlib</em>
constant <tt>Z_NO_FLUSH</tt> will be passed to <tt>deflate</tt> to indicate that we are still
in the middle of the uncompressed data.
<p>
@ -540,6 +538,12 @@ int main(int argc, char **argv)
}
</b></pre>
<hr>
<i>Copyright (c) 2004, 2005 by Mark Adler<br>Last modified 11 December 2005</i>
<i>Last modified 24 January 2023<br>
Copyright &#169; 2004-2023 Mark Adler</i><br>
<a rel="license" href="http://creativecommons.org/licenses/by-nd/4.0/">
<img alt="Creative Commons License" style="border-width:0"
src="https://i.creativecommons.org/l/by-nd/4.0/88x31.png"></a>
<a rel="license" href="http://creativecommons.org/licenses/by-nd/4.0/">
Creative Commons Attribution-NoDerivatives 4.0 International License</a>.
</body>
</html>

730
extern/zlib/examples/zran.c vendored
View file

@ -1,383 +1,503 @@
/* zran.c -- example of zlib/gzip stream indexing and random access
* Copyright (C) 2005, 2012 Mark Adler
/* zran.c -- example of deflate stream indexing and random access
* Copyright (C) 2005, 2012, 2018, 2023 Mark Adler
* For conditions of distribution and use, see copyright notice in zlib.h
Version 1.1 29 Sep 2012 Mark Adler */
* Version 1.4 13 Apr 2023 Mark Adler */
/* Version History:
1.0 29 May 2005 First version
1.1 29 Sep 2012 Fix memory reallocation error
1.2 14 Oct 2018 Handle gzip streams with multiple members
Add a header file to facilitate usage in applications
1.3 18 Feb 2023 Permit raw deflate streams as well as zlib and gzip
Permit crossing gzip member boundaries when extracting
Support a size_t size when extracting (was an int)
Do a binary search over the index for an access point
Expose the access point type to enable save and load
1.4 13 Apr 2023 Add a NOPRIME define to not use inflatePrime()
*/
/* Illustrate the use of Z_BLOCK, inflatePrime(), and inflateSetDictionary()
for random access of a compressed file. A file containing a zlib or gzip
stream is provided on the command line. The compressed stream is decoded in
its entirety, and an index built with access points about every SPAN bytes
in the uncompressed output. The compressed file is left open, and can then
be read randomly, having to decompress on the average SPAN/2 uncompressed
bytes before getting to the desired block of data.
An access point can be created at the start of any deflate block, by saving
the starting file offset and bit of that block, and the 32K bytes of
uncompressed data that precede that block. Also the uncompressed offset of
that block is saved to provide a referece for locating a desired starting
point in the uncompressed stream. build_index() works by decompressing the
input zlib or gzip stream a block at a time, and at the end of each block
deciding if enough uncompressed data has gone by to justify the creation of
a new access point. If so, that point is saved in a data structure that
grows as needed to accommodate the points.
To use the index, an offset in the uncompressed data is provided, for which
the latest access point at or preceding that offset is located in the index.
The input file is positioned to the specified location in the index, and if
necessary the first few bits of the compressed data is read from the file.
inflate is initialized with those bits and the 32K of uncompressed data, and
the decompression then proceeds until the desired offset in the file is
reached. Then the decompression continues to read the desired uncompressed
data from the file.
Another approach would be to generate the index on demand. In that case,
requests for random access reads from the compressed data would try to use
the index, but if a read far enough past the end of the index is required,
then further index entries would be generated and added.
There is some fair bit of overhead to starting inflation for the random
access, mainly copying the 32K byte dictionary. So if small pieces of the
file are being accessed, it would make sense to implement a cache to hold
some lookahead and avoid many calls to extract() for small lengths.
Another way to build an index would be to use inflateCopy(). That would
not be constrained to have access points at block boundaries, but requires
more memory per access point, and also cannot be saved to file due to the
use of pointers in the state. The approach here allows for storage of the
index in a file.
*/
// Illustrate the use of Z_BLOCK, inflatePrime(), and inflateSetDictionary()
// for random access of a compressed file. A file containing a raw deflate
// stream is provided on the command line. The compressed stream is decoded in
// its entirety, and an index built with access points about every SPAN bytes
// in the uncompressed output. The compressed file is left open, and can then
// be read randomly, having to decompress on the average SPAN/2 uncompressed
// bytes before getting to the desired block of data.
//
// An access point can be created at the start of any deflate block, by saving
// the starting file offset and bit of that block, and the 32K bytes of
// uncompressed data that precede that block. Also the uncompressed offset of
// that block is saved to provide a reference for locating a desired starting
// point in the uncompressed stream. deflate_index_build() decompresses the
// input raw deflate stream a block at a time, and at the end of each block
// decides if enough uncompressed data has gone by to justify the creation of a
// new access point. If so, that point is saved in a data structure that grows
// as needed to accommodate the points.
//
// To use the index, an offset in the uncompressed data is provided, for which
// the latest access point at or preceding that offset is located in the index.
// The input file is positioned to the specified location in the index, and if
// necessary the first few bits of the compressed data is read from the file.
// inflate is initialized with those bits and the 32K of uncompressed data, and
// decompression then proceeds until the desired offset in the file is reached.
// Then decompression continues to read the requested uncompressed data from
// the file.
//
// There is some fair bit of overhead to starting inflation for the random
// access, mainly copying the 32K byte dictionary. If small pieces of the file
// are being accessed, it would make sense to implement a cache to hold some
// lookahead to avoid many calls to deflate_index_extract() for small lengths.
//
// Another way to build an index would be to use inflateCopy(). That would not
// be constrained to have access points at block boundaries, but would require
// more memory per access point, and could not be saved to a file due to the
// use of pointers in the state. The approach here allows for storage of the
// index in a file.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <limits.h>
#include "zlib.h"
#include "zran.h"
#define local static
#define WINSIZE 32768U // sliding window size
#define CHUNK 16384 // file input buffer size
#define SPAN 1048576L /* desired distance between access points */
#define WINSIZE 32768U /* sliding window size */
#define CHUNK 16384 /* file input buffer size */
/* access point entry */
struct point {
off_t out; /* corresponding offset in uncompressed data */
off_t in; /* offset in input file of first full byte */
int bits; /* number of bits (1-7) from byte at in - 1, or 0 */
unsigned char window[WINSIZE]; /* preceding 32K of uncompressed data */
};
/* access point list */
struct access {
int have; /* number of list entries filled in */
int size; /* number of list entries allocated */
struct point *list; /* allocated list */
};
/* Deallocate an index built by build_index() */
local void free_index(struct access *index)
{
// See comments in zran.h.
void deflate_index_free(struct deflate_index *index) {
if (index != NULL) {
free(index->list);
free(index);
}
}
/* Add an entry to the access point list. If out of memory, deallocate the
existing list and return NULL. */
local struct access *addpoint(struct access *index, int bits,
off_t in, off_t out, unsigned left, unsigned char *window)
{
struct point *next;
/* if list is empty, create it (start with eight points) */
// Add an access point to the list. If out of memory, deallocate the existing
// list and return NULL. index->mode is temporarily the allocated number of
// access points, until it is time for deflate_index_build() to return. Then
// index->mode is set to the mode of inflation.
static struct deflate_index *add_point(struct deflate_index *index, int bits,
off_t in, off_t out, unsigned left,
unsigned char *window) {
if (index == NULL) {
index = malloc(sizeof(struct access));
if (index == NULL) return NULL;
index->list = malloc(sizeof(struct point) << 3);
// The list is empty. Create it, starting with eight access points.
index = malloc(sizeof(struct deflate_index));
if (index == NULL)
return NULL;
index->have = 0;
index->mode = 8;
index->list = malloc(sizeof(point_t) * index->mode);
if (index->list == NULL) {
free(index);
return NULL;
}
index->size = 8;
index->have = 0;
}
/* if list is full, make it bigger */
else if (index->have == index->size) {
index->size <<= 1;
next = realloc(index->list, sizeof(struct point) * index->size);
else if (index->have == index->mode) {
// The list is full. Make it bigger.
index->mode <<= 1;
point_t *next = realloc(index->list, sizeof(point_t) * index->mode);
if (next == NULL) {
free_index(index);
deflate_index_free(index);
return NULL;
}
index->list = next;
}
/* fill in entry and increment how many we have */
next = index->list + index->have;
next->bits = bits;
next->in = in;
// Fill in the access point and increment how many we have.
point_t *next = (point_t *)(index->list) + index->have++;
if (index->have < 0) {
// Overflowed the int!
deflate_index_free(index);
return NULL;
}
next->out = out;
next->in = in;
next->bits = bits;
if (left)
memcpy(next->window, window + WINSIZE - left, left);
if (left < WINSIZE)
memcpy(next->window + left, window, WINSIZE - left);
index->have++;
/* return list, possibly reallocated */
// Return the index, which may have been newly allocated or destroyed.
return index;
}
/* Make one entire pass through the compressed stream and build an index, with
access points about every span bytes of uncompressed output -- span is
chosen to balance the speed of random access against the memory requirements
of the list, about 32K bytes per access point. Note that data after the end
of the first zlib or gzip stream in the file is ignored. build_index()
returns the number of access points on success (>= 1), Z_MEM_ERROR for out
of memory, Z_DATA_ERROR for an error in the input file, or Z_ERRNO for a
file read error. On success, *built points to the resulting index. */
local int build_index(FILE *in, off_t span, struct access **built)
{
int ret;
off_t totin, totout; /* our own total counters to avoid 4GB limit */
off_t last; /* totout value of last access point */
struct access *index; /* access points being generated */
z_stream strm;
unsigned char input[CHUNK];
unsigned char window[WINSIZE];
// Decompression modes. These are the inflateInit2() windowBits parameter.
#define RAW -15
#define ZLIB 15
#define GZIP 31
/* initialize inflate */
strm.zalloc = Z_NULL;
strm.zfree = Z_NULL;
strm.opaque = Z_NULL;
strm.avail_in = 0;
strm.next_in = Z_NULL;
ret = inflateInit2(&strm, 47); /* automatic zlib or gzip decoding */
if (ret != Z_OK)
return ret;
// See comments in zran.h.
int deflate_index_build(FILE *in, off_t span, struct deflate_index **built) {
// Set up inflation state.
z_stream strm = {0}; // inflate engine (gets fired up later)
unsigned char buf[CHUNK]; // input buffer
unsigned char win[WINSIZE] = {0}; // output sliding window
off_t totin = 0; // total bytes read from input
off_t totout = 0; // total bytes uncompressed
int mode = 0; // mode: RAW, ZLIB, or GZIP (0 => not set yet)
/* inflate the input, maintain a sliding window, and build an index -- this
also validates the integrity of the compressed data using the check
information at the end of the gzip or zlib stream */
totin = totout = last = 0;
index = NULL; /* will be allocated by first addpoint() */
strm.avail_out = 0;
// Decompress from in, generating access points along the way.
int ret; // the return value from zlib, or Z_ERRNO
off_t last; // last access point uncompressed offset
struct deflate_index *index = NULL; // list of access points
do {
/* get some compressed data from input file */
strm.avail_in = fread(input, 1, CHUNK, in);
if (ferror(in)) {
ret = Z_ERRNO;
goto build_index_error;
}
// Assure available input, at least until reaching EOF.
if (strm.avail_in == 0) {
ret = Z_DATA_ERROR;
goto build_index_error;
}
strm.next_in = input;
/* process all of that, or until end of stream */
do {
/* reset sliding window if necessary */
if (strm.avail_out == 0) {
strm.avail_out = WINSIZE;
strm.next_out = window;
}
/* inflate until out of input, output, or at end of block --
update the total input and output counters */
strm.avail_in = fread(buf, 1, sizeof(buf), in);
totin += strm.avail_in;
totout += strm.avail_out;
ret = inflate(&strm, Z_BLOCK); /* return at end of block */
totin -= strm.avail_in;
totout -= strm.avail_out;
if (ret == Z_NEED_DICT)
ret = Z_DATA_ERROR;
if (ret == Z_MEM_ERROR || ret == Z_DATA_ERROR)
goto build_index_error;
if (ret == Z_STREAM_END)
strm.next_in = buf;
if (strm.avail_in < sizeof(buf) && ferror(in)) {
ret = Z_ERRNO;
break;
/* if at end of block, consider adding an index entry (note that if
data_type indicates an end-of-block, then all of the
uncompressed data from that block has been delivered, and none
of the compressed data after that block has been consumed,
except for up to seven bits) -- the totout == 0 provides an
entry point after the zlib or gzip header, and assures that the
index always has at least one access point; we avoid creating an
access point after the last block by checking bit 6 of data_type
*/
if ((strm.data_type & 128) && !(strm.data_type & 64) &&
(totout == 0 || totout - last > span)) {
index = addpoint(index, strm.data_type & 7, totin,
totout, strm.avail_out, window);
if (index == NULL) {
ret = Z_MEM_ERROR;
goto build_index_error;
}
last = totout;
}
} while (strm.avail_in != 0);
} while (ret != Z_STREAM_END);
/* clean up and return index (release unused entries in list) */
(void)inflateEnd(&strm);
index->list = realloc(index->list, sizeof(struct point) * index->have);
index->size = index->have;
if (mode == 0) {
// At the start of the input -- determine the type. Assume raw
// if it is neither zlib nor gzip. This could in theory result
// in a false positive for zlib, but in practice the fill bits
// after a stored block are always zeros, so a raw stream won't
// start with an 8 in the low nybble.
mode = strm.avail_in == 0 ? RAW : // empty -- will fail
(strm.next_in[0] & 0xf) == 8 ? ZLIB :
strm.next_in[0] == 0x1f ? GZIP :
/* else */ RAW;
ret = inflateInit2(&strm, mode);
if (ret != Z_OK)
break;
}
}
// Assure available output. This rotates the output through, for use as
// a sliding window on the uncompressed data.
if (strm.avail_out == 0) {
strm.avail_out = sizeof(win);
strm.next_out = win;
}
if (mode == RAW && index == NULL)
// We skip the inflate() call at the start of raw deflate data in
// order generate an access point there. Set data_type to imitate
// the end of a header.
strm.data_type = 0x80;
else {
// Inflate and update the number of uncompressed bytes.
unsigned before = strm.avail_out;
ret = inflate(&strm, Z_BLOCK);
totout += before - strm.avail_out;
}
if ((strm.data_type & 0xc0) == 0x80 &&
(index == NULL || totout - last >= span)) {
// We are at the end of a header or a non-last deflate block, so we
// can add an access point here. Furthermore, we are either at the
// very start for the first access point, or there has been span or
// more uncompressed bytes since the last access point, so we want
// to add an access point here.
index = add_point(index, strm.data_type & 7, totin - strm.avail_in,
totout, strm.avail_out, win);
if (index == NULL) {
ret = Z_MEM_ERROR;
break;
}
last = totout;
}
if (ret == Z_STREAM_END && mode == GZIP &&
(strm.avail_in || ungetc(getc(in), in) != EOF))
// There is more input after the end of a gzip member. Reset the
// inflate state to read another gzip member. On success, this will
// set ret to Z_OK to continue decompressing.
ret = inflateReset2(&strm, GZIP);
// Keep going until Z_STREAM_END or error. If the compressed data ends
// prematurely without a file read error, Z_BUF_ERROR is returned.
} while (ret == Z_OK);
inflateEnd(&strm);
if (ret != Z_STREAM_END) {
// An error was encountered. Discard the index and return a negative
// error code.
deflate_index_free(index);
return ret == Z_NEED_DICT ? Z_DATA_ERROR : ret;
}
// Shrink the index to only the occupied access points and return it.
index->mode = mode;
index->length = totout;
point_t *list = realloc(index->list, sizeof(point_t) * index->have);
if (list == NULL) {
// Seems like a realloc() to make something smaller should always work,
// but just in case.
deflate_index_free(index);
return Z_MEM_ERROR;
}
index->list = list;
*built = index;
return index->size;
/* return error */
build_index_error:
(void)inflateEnd(&strm);
if (index != NULL)
free_index(index);
return ret;
return index->have;
}
/* Use the index to read len bytes from offset into buf, return bytes read or
negative for error (Z_DATA_ERROR or Z_MEM_ERROR). If data is requested past
the end of the uncompressed data, then extract() will return a value less
than len, indicating how much as actually read into buf. This function
should not return a data error unless the file was modified since the index
was generated. extract() may also return Z_ERRNO if there is an error on
reading or seeking the input file. */
local int extract(FILE *in, struct access *index, off_t offset,
unsigned char *buf, int len)
{
int ret, skip;
z_stream strm;
struct point *here;
unsigned char input[CHUNK];
unsigned char discard[WINSIZE];
#ifdef NOPRIME
// Support zlib versions before 1.2.3 (July 2005), or incomplete zlib clones
// that do not have inflatePrime().
/* proceed only if something reasonable to do */
if (len < 0)
# define INFLATEPRIME inflatePreface
// Append the low bits bits of value to in[] at bit position *have, updating
// *have. value must be zero above its low bits bits. bits must be positive.
// This assumes that any bits above the *have bits in the last byte are zeros.
// That assumption is preserved on return, as any bits above *have + bits in
// the last byte written will be set to zeros.
static inline void append_bits(unsigned value, int bits,
unsigned char *in, int *have) {
in += *have >> 3; // where the first bits from value will go
int k = *have & 7; // the number of bits already there
*have += bits;
if (k)
*in |= value << k; // write value above the low k bits
else
*in = value;
k = 8 - k; // the number of bits just appended
while (bits > k) {
value >>= k; // drop the bits appended
bits -= k;
k = 8; // now at a byte boundary
*++in = value;
}
}
// Insert enough bits in the form of empty deflate blocks in front of the
// low bits bits of value, in order to bring the sequence to a byte boundary.
// Then feed that to inflate(). This does what inflatePrime() does, except that
// a negative value of bits is not supported. bits must be in 0..16. If the
// arguments are invalid, Z_STREAM_ERROR is returned. Otherwise the return
// value from inflate() is returned.
static int inflatePreface(z_stream *strm, int bits, int value) {
// Check input.
if (strm == Z_NULL || bits < 0 || bits > 16)
return Z_STREAM_ERROR;
if (bits == 0)
return Z_OK;
value &= (2 << (bits - 1)) - 1;
// An empty dynamic block with an odd number of bits (95). The high bit of
// the last byte is unused.
static const unsigned char dyn[] = {
4, 0xe0, 0x81, 8, 0, 0, 0, 0, 0x20, 0xa8, 0xab, 0x1f
};
const int dynlen = 95; // number of bits in the block
// Build an input buffer for inflate that is a multiple of eight bits in
// length, and that ends with the low bits bits of value.
unsigned char in[(dynlen + 3 * 10 + 16 + 7) / 8];
int have = 0;
if (bits & 1) {
// Insert an empty dynamic block to get to an odd number of bits, so
// when bits bits from value are appended, we are at an even number of
// bits.
memcpy(in, dyn, sizeof(dyn));
have = dynlen;
}
while ((have + bits) & 7)
// Insert empty fixed blocks until appending bits bits would put us on
// a byte boundary. This will insert at most three fixed blocks.
append_bits(2, 10, in, &have);
// Append the bits bits from value, which takes us to a byte boundary.
append_bits(value, bits, in, &have);
// Deliver the input to inflate(). There is no output space provided, but
// inflate() can't get stuck waiting on output not ingesting all of the
// provided input. The reason is that there will be at most 16 bits of
// input from value after the empty deflate blocks (which themselves
// generate no output). At least ten bits are needed to generate the first
// output byte from a fixed block. The last two bytes of the buffer have to
// be ingested in order to get ten bits, which is the most that value can
// occupy.
strm->avail_in = have >> 3;
strm->next_in = in;
strm->avail_out = 0;
strm->next_out = in; // not used, but can't be NULL
return inflate(strm, Z_NO_FLUSH);
}
#else
# define INFLATEPRIME inflatePrime
#endif
// See comments in zran.h.
ptrdiff_t deflate_index_extract(FILE *in, struct deflate_index *index,
off_t offset, unsigned char *buf, size_t len) {
// Do a quick sanity check on the index.
if (index == NULL || index->have < 1 || index->list[0].out != 0)
return Z_STREAM_ERROR;
// If nothing to extract, return zero bytes extracted.
if (len == 0 || offset < 0 || offset >= index->length)
return 0;
/* find where in stream to start */
here = index->list;
ret = index->have;
while (--ret && here[1].out <= offset)
here++;
// Find the access point closest to but not after offset.
int lo = -1, hi = index->have;
point_t *point = index->list;
while (hi - lo > 1) {
int mid = (lo + hi) >> 1;
if (offset < point[mid].out)
hi = mid;
else
lo = mid;
}
point += lo;
/* initialize file and inflate state to start there */
strm.zalloc = Z_NULL;
strm.zfree = Z_NULL;
strm.opaque = Z_NULL;
strm.avail_in = 0;
strm.next_in = Z_NULL;
ret = inflateInit2(&strm, -15); /* raw inflate */
// Initialize the input file and prime the inflate engine to start there.
int ret = fseeko(in, point->in - (point->bits ? 1 : 0), SEEK_SET);
if (ret == -1)
return Z_ERRNO;
int ch = 0;
if (point->bits && (ch = getc(in)) == EOF)
return ferror(in) ? Z_ERRNO : Z_BUF_ERROR;
z_stream strm = {0};
ret = inflateInit2(&strm, RAW);
if (ret != Z_OK)
return ret;
ret = fseeko(in, here->in - (here->bits ? 1 : 0), SEEK_SET);
if (ret == -1)
goto extract_ret;
if (here->bits) {
ret = getc(in);
if (ret == -1) {
ret = ferror(in) ? Z_ERRNO : Z_DATA_ERROR;
goto extract_ret;
}
(void)inflatePrime(&strm, here->bits, ret >> (8 - here->bits));
}
(void)inflateSetDictionary(&strm, here->window, WINSIZE);
if (point->bits)
INFLATEPRIME(&strm, point->bits, ch >> (8 - point->bits));
inflateSetDictionary(&strm, point->window, WINSIZE);
/* skip uncompressed bytes until offset reached, then satisfy request */
offset -= here->out;
strm.avail_in = 0;
skip = 1; /* while skipping to offset */
// Skip uncompressed bytes until offset reached, then satisfy request.
unsigned char input[CHUNK];
unsigned char discard[WINSIZE];
offset -= point->out; // number of bytes to skip to get to offset
size_t left = len; // number of bytes left to read after offset
do {
/* define where to put uncompressed data, and how much */
if (offset == 0 && skip) { /* at offset now */
strm.avail_out = len;
strm.next_out = buf;
skip = 0; /* only do this once */
}
if (offset > WINSIZE) { /* skip WINSIZE bytes */
strm.avail_out = WINSIZE;
if (offset) {
// Discard up to offset uncompressed bytes.
strm.avail_out = offset < WINSIZE ? (unsigned)offset : WINSIZE;
strm.next_out = discard;
offset -= WINSIZE;
}
else if (offset != 0) { /* last skip */
strm.avail_out = (unsigned)offset;
strm.next_out = discard;
offset = 0;
else {
// Uncompress up to left bytes into buf.
strm.avail_out = left < UINT_MAX ? (unsigned)left : UINT_MAX;
strm.next_out = buf + len - left;
}
/* uncompress until avail_out filled, or end of stream */
do {
if (strm.avail_in == 0) {
strm.avail_in = fread(input, 1, CHUNK, in);
if (ferror(in)) {
ret = Z_ERRNO;
goto extract_ret;
}
if (strm.avail_in == 0) {
ret = Z_DATA_ERROR;
goto extract_ret;
}
strm.next_in = input;
}
ret = inflate(&strm, Z_NO_FLUSH); /* normal inflate */
if (ret == Z_NEED_DICT)
ret = Z_DATA_ERROR;
if (ret == Z_MEM_ERROR || ret == Z_DATA_ERROR)
goto extract_ret;
if (ret == Z_STREAM_END)
// Uncompress, setting got to the number of bytes uncompressed.
if (strm.avail_in == 0) {
// Assure available input.
strm.avail_in = fread(input, 1, CHUNK, in);
if (strm.avail_in < CHUNK && ferror(in)) {
ret = Z_ERRNO;
break;
} while (strm.avail_out != 0);
}
strm.next_in = input;
}
unsigned got = strm.avail_out;
ret = inflate(&strm, Z_NO_FLUSH);
got -= strm.avail_out;
/* if reach end of stream, then don't keep trying to get more */
if (ret == Z_STREAM_END)
break;
// Update the appropriate count.
if (offset)
offset -= got;
else
left -= got;
/* do until offset reached and requested data read, or stream ends */
} while (skip);
// If we're at the end of a gzip member and there's more to read,
// continue to the next gzip member.
if (ret == Z_STREAM_END && index->mode == GZIP) {
// Discard the gzip trailer.
unsigned drop = 8; // length of gzip trailer
if (strm.avail_in >= drop) {
strm.avail_in -= drop;
strm.next_in += drop;
}
else {
// Read and discard the remainder of the gzip trailer.
drop -= strm.avail_in;
strm.avail_in = 0;
do {
if (getc(in) == EOF)
// The input does not have a complete trailer.
return ferror(in) ? Z_ERRNO : Z_BUF_ERROR;
} while (--drop);
}
/* compute number of uncompressed bytes read after offset */
ret = skip ? 0 : len - strm.avail_out;
if (strm.avail_in || ungetc(getc(in), in) != EOF) {
// There's more after the gzip trailer. Use inflate to skip the
// gzip header and resume the raw inflate there.
inflateReset2(&strm, GZIP);
do {
if (strm.avail_in == 0) {
strm.avail_in = fread(input, 1, CHUNK, in);
if (strm.avail_in < CHUNK && ferror(in)) {
ret = Z_ERRNO;
break;
}
strm.next_in = input;
}
strm.avail_out = WINSIZE;
strm.next_out = discard;
ret = inflate(&strm, Z_BLOCK); // stop at end of header
} while (ret == Z_OK && (strm.data_type & 0x80) == 0);
if (ret != Z_OK)
break;
inflateReset2(&strm, RAW);
}
}
/* clean up and return bytes read or error */
extract_ret:
(void)inflateEnd(&strm);
return ret;
// Continue until we have the requested data, the deflate data has
// ended, or an error is encountered.
} while (ret == Z_OK && left);
inflateEnd(&strm);
// Return the number of uncompressed bytes read into buf, or the error.
return ret == Z_OK || ret == Z_STREAM_END ? len - left : ret;
}
/* Demonstrate the use of build_index() and extract() by processing the file
provided on the command line, and the extracting 16K from about 2/3rds of
the way through the uncompressed output, and writing that to stdout. */
int main(int argc, char **argv)
{
int len;
off_t offset;
FILE *in;
struct access *index = NULL;
unsigned char buf[CHUNK];
#ifdef TEST
/* open input file */
if (argc != 2) {
fprintf(stderr, "usage: zran file.gz\n");
#define SPAN 1048576L // desired distance between access points
#define LEN 16384 // number of bytes to extract
// Demonstrate the use of deflate_index_build() and deflate_index_extract() by
// processing the file provided on the command line, and extracting LEN bytes
// from 2/3rds of the way through the uncompressed output, writing that to
// stdout. An offset can be provided as the second argument, in which case the
// data is extracted from there instead.
int main(int argc, char **argv) {
// Open the input file.
if (argc < 2 || argc > 3) {
fprintf(stderr, "usage: zran file.raw [offset]\n");
return 1;
}
in = fopen(argv[1], "rb");
FILE *in = fopen(argv[1], "rb");
if (in == NULL) {
fprintf(stderr, "zran: could not open %s for reading\n", argv[1]);
return 1;
}
/* build index */
len = build_index(in, SPAN, &index);
// Get optional offset.
off_t offset = -1;
if (argc == 3) {
char *end;
offset = strtoll(argv[2], &end, 10);
if (*end || offset < 0) {
fprintf(stderr, "zran: %s is not a valid offset\n", argv[2]);
return 1;
}
}
// Build index.
struct deflate_index *index = NULL;
int len = deflate_index_build(in, SPAN, &index);
if (len < 0) {
fclose(in);
switch (len) {
case Z_MEM_ERROR:
fprintf(stderr, "zran: out of memory\n");
break;
case Z_BUF_ERROR:
fprintf(stderr, "zran: %s ended prematurely\n", argv[1]);
break;
case Z_DATA_ERROR:
fprintf(stderr, "zran: compressed data error in %s\n", argv[1]);
break;
@ -391,19 +511,23 @@ int main(int argc, char **argv)
}
fprintf(stderr, "zran: built index with %d access points\n", len);
/* use index by reading some bytes from an arbitrary offset */
offset = (index->list[index->have - 1].out << 1) / 3;
len = extract(in, index, offset, buf, CHUNK);
if (len < 0)
// Use index by reading some bytes from an arbitrary offset.
unsigned char buf[LEN];
if (offset == -1)
offset = ((index->length + 1) << 1) / 3;
ptrdiff_t got = deflate_index_extract(in, index, offset, buf, LEN);
if (got < 0)
fprintf(stderr, "zran: extraction failed: %s error\n",
len == Z_MEM_ERROR ? "out of memory" : "input corrupted");
got == Z_MEM_ERROR ? "out of memory" : "input corrupted");
else {
fwrite(buf, 1, len, stdout);
fprintf(stderr, "zran: extracted %d bytes at %llu\n", len, offset);
fwrite(buf, 1, got, stdout);
fprintf(stderr, "zran: extracted %ld bytes at %lld\n", got, offset);
}
/* clean up and exit */
free_index(index);
// Clean up and exit.
deflate_index_free(index);
fclose(in);
return 0;
}
#endif

51
extern/zlib/examples/zran.h vendored Normal file
View file

@ -0,0 +1,51 @@
/* zran.h -- example of deflated stream indexing and random access
* Copyright (C) 2005, 2012, 2018, 2023 Mark Adler
* For conditions of distribution and use, see copyright notice in zlib.h
* Version 1.3 18 Feb 2023 Mark Adler */
#include <stdio.h>
#include "zlib.h"
// Access point.
typedef struct point {
off_t out; // offset in uncompressed data
off_t in; // offset in compressed file of first full byte
int bits; // 0, or number of bits (1-7) from byte at in-1
unsigned char window[32768]; // preceding 32K of uncompressed data
} point_t;
// Access point list.
struct deflate_index {
int have; // number of access points in list
int mode; // -15 for raw, 15 for zlib, or 31 for gzip
off_t length; // total length of uncompressed data
point_t *list; // allocated list of access points
};
// Make one pass through a zlib, gzip, or raw deflate compressed stream and
// build an index, with access points about every span bytes of uncompressed
// output. gzip files with multiple members are fully indexed. span should be
// chosen to balance the speed of random access against the memory requirements
// of the list, which is about 32K bytes per access point. The return value is
// the number of access points on success (>= 1), Z_MEM_ERROR for out of
// memory, Z_BUF_ERROR for a premature end of input, Z_DATA_ERROR for a format
// or verification error in the input file, or Z_ERRNO for a file read error.
// On success, *built points to the resulting index.
int deflate_index_build(FILE *in, off_t span, struct deflate_index **built);
// Use the index to read len bytes from offset into buf. Return the number of
// bytes read or a negative error code. If data is requested past the end of
// the uncompressed data, then deflate_index_extract() will return a value less
// than len, indicating how much was actually read into buf. If given a valid
// index, this function should not return an error unless the file was modified
// somehow since the index was generated, given that deflate_index_build() had
// validated all of the input. If nevertheless there is a failure, Z_BUF_ERROR
// is returned if the compressed data ends prematurely, Z_DATA_ERROR if the
// deflate compressed data is not valid, Z_MEM_ERROR if out of memory,
// Z_STREAM_ERROR if the index is not valid, or Z_ERRNO if there is an error
// reading or seeking on the input file.
ptrdiff_t deflate_index_extract(FILE *in, struct deflate_index *index,
off_t offset, unsigned char *buf, size_t len);
// Deallocate an index built by deflate_index_build().
void deflate_index_free(struct deflate_index *index);