InformationTheory.Shannon.Kolmogorov.SufficientStatistic
Kolmogorov sufficient statistics and two-part descriptions #
Cover–Thomas (2nd ed.). A finite model S ∋ x describes x in two parts:
the model itself, and the index of x inside S. The model part is measured by
the prefix complexity modelComplexity S of a canonical code for S, and the
index part by Nat.clog 2 S.card bits.
Main definitions #
modelCode/modelComplexity— the canonical code of a finite model and its prefix complexity.twoPartLength— the length of the two-part description built from a model.mdlComplexity— the shortest two-part description length ofx.structureFunction— the least index part achievable within a model budget.IsSufficientStatistic— a model whose two-part description is within an additive constant ofprefixComplexity x.
Main statements #
prefixComplexity_le_twoPartLength— every two-part description bounds the prefix complexity, up to an additive constant.mdlComplexity_sub_prefixComplexity_le— the shortest two-part description length and the prefix complexity agree up to an additive constant.
Implementation notes #
The index part of twoPartLength carries the coefficient 4, not 1, and the
two factors of two have different sources. One is a property of the machine
prefixUniversalEval: prefix complexity and payload complexity are rigidly tied
by the identity prefixComplexity x = 2 * payloadComplexity x + 1, so a bound
proved in the payload world doubles on the way back. The other is a property of
the packing used here, which delimits the index with selfDelimit and therefore
spends two bits per index bit. The textbook inequality
K(x) ≤ K(S) + log |S| + O(1), with coefficient 1 on the index part, is a
statement about an additively universal prefix machine and is not claimed here;
the coefficient on the model part modelComplexity S is exactly 1 because both
sides pass through the same identity.
References #
Cover–Thomas (2nd ed.).
Tags #
Kolmogorov complexity, sufficient statistic, minimum description length
Models and two-part descriptions #
InformationTheory.Kolmogorov.modelCode
sourceThe canonical code of a finite model: the Encodable code of its sorted
element list. Sorted lists are used rather than Finset itself because the
decoder that reads a model back has to be primitive recursive, and List ℕ is
Primcodable while Finset ℕ is not.
@audit:ok
Equations
- InformationTheory.Kolmogorov.modelCode S = Encodable.encode (S.sort fun (x1 x2 : ℕ) => x1 ≤ x2)
Instances For
Used by
InformationTheory.Kolmogorov.modelComplexity
sourceThe description length of a finite model: the prefix complexity of its
canonical code. Using prefix complexity rather than the literal length of
modelCode keeps the quantity insensitive to the redundancy of the canonical
code.
@audit:ok
Equations
Instances For
Used by
InformationTheory.Kolmogorov.twoPartLength
sourceThe length of the two-part description built from the model S: the model
part plus the index part. The coefficient 4 on the index part is built into the
definition rather than recovered by the argument (see the implementation notes);
the coefficient on the model part is exactly 1.
@audit:ok
Equations
Instances For
Used by
InformationTheory.Kolmogorov.mdlComplexity
sourceThe shortest two-part description length of x, minimized over all finite
models containing x. The singleton model always competes, so the infimum is
attained (mdlComplexity_spec).
@audit:ok
Equations
Instances For
Used by
InformationTheory.Kolmogorov.structureFunction
sourceThe structure function of x: the least index part ⌈log₂ |S|⌉ over models
S ∋ x whose description length is within the budget k. The value is ℕ∞
because the constraint set is empty for small budgets, where the value has to be
⊤ rather than 0 (structureFunction_zero); over ℕ the empty infimum would
collapse to 0 and contradict structureFunction_antitone.
@audit:ok
Equations
Instances For
Used by
InformationTheory.Kolmogorov.IsSufficientStatistic
sourceA model S is a sufficient statistic for x at slack c when it contains
x and its two-part description is no longer than prefixComplexity x + c. The
slack is an explicit argument so that minimality statements can quantify over it.
@audit:ok
Equations
Instances For
Used by
InformationTheory.Kolmogorov.natLen_le_clog_card
sourceUsed by
InformationTheory.Kolmogorov.exists_index_of_mem
sourceUsed by
InformationTheory.Kolmogorov.structureFunction_antitone
sourceUsed by
InformationTheory.Kolmogorov.structureFunction_zero
sourceUsed by
InformationTheory.Kolmogorov.structureFunction_eq_zero_of_singleton_budget
sourceUsed by
InformationTheory.Kolmogorov.mdlComplexity_set_nonempty
sourceUsed by
InformationTheory.Kolmogorov.mdlComplexity_spec
sourceUsed by
InformationTheory.Kolmogorov.mdlComplexity_le_of_mem
sourceUsed by
The bit codec is primitive recursive #
InformationTheory.Kolmogorov.encodeNat_zero
sourceUsed by
InformationTheory.Kolmogorov.encodeNat_cast_posNum
sourceUsed by
InformationTheory.Kolmogorov.encodePosNum_eq_cons
sourceUsed by
InformationTheory.Kolmogorov.encodeNat_eq_cons
sourceUsed by
InformationTheory.Kolmogorov.encodePosNum_decodePosNum_concat
sourceUsed by
InformationTheory.Kolmogorov.encodeNat_decodeNat_concat
sourceUsed by
InformationTheory.Kolmogorov.bitStep
sourceOne step of the little-endian bit expansion: peel off the lowest bit of the remaining value and append it to the accumulated bit string.
Equations
Instances For
Used by
InformationTheory.Kolmogorov.bitStep_iterate
sourceUsed by
InformationTheory.Kolmogorov.encodeNat_eq_iterate
sourceUsed by
InformationTheory.Kolmogorov.encodeNat_primrec
sourceUsed by
InformationTheory.Kolmogorov.decodeNat_of_ne_nil
sourceUsed by
InformationTheory.Kolmogorov.decodeNat_cons
sourceUsed by
InformationTheory.Kolmogorov.decodeNat_eq_rec
sourceUsed by
InformationTheory.Kolmogorov.decodeNat_primrec
sourceUsed by
InformationTheory.Kolmogorov.parseUnary_eq_rec
sourceUsed by
InformationTheory.Kolmogorov.parseUnary_primrec
sourceUsed by
The payload decoder is partial recursive #
InformationTheory.Kolmogorov.payloadDispatch
sourceThe payload decoder presented as a code together with its input: the literal
mode uses the identity code, and the interpret mode the code named by the unary
index. The empty payload has no dispatch, which is how decodePayload diverges
on it.
Equations
- One or more equations did not get rendered due to their size.
- InformationTheory.Kolmogorov.payloadDispatch [] = none
- InformationTheory.Kolmogorov.payloadDispatch (false :: bs) = some (Nat.Partrec.Code.id, Computability.decodeNat bs)
Instances For
Used by
InformationTheory.Kolmogorov.payloadDispatch_primrec
sourceUsed by
InformationTheory.Kolmogorov.payloadDispatch_computable
sourceUsed by
InformationTheory.Kolmogorov.decodePayload_eq_dispatch
sourceUsed by
InformationTheory.Kolmogorov.decodePayload_partrec
sourceUsed by
Self-simulation of the machine #
InformationTheory.Kolmogorov.payloadComplexity_le_of_eval
sourceUsed by
InformationTheory.Kolmogorov.payload_invariance
sourceUsed by
InformationTheory.Kolmogorov.packIndex
sourceThe index part packed into N: the self-delimited prefix of the bit string
of N.
Equations
- One or more equations did not get rendered due to their size.
Instances For
Used by
InformationTheory.Kolmogorov.packPayload
sourceThe payload part packed into N: what follows the self-delimited index,
with the terminating sentinel bit removed.
Equations
- One or more equations did not get rendered due to their size.
Instances For
Used by
InformationTheory.Kolmogorov.packBits
sourceThe bit string packing an index i together with a payload d: the index in
self-delimited form, then the payload, then a sentinel bit making the string a
canonical binary expansion.
Equations
Instances For
Used by
InformationTheory.Kolmogorov.packBits_length
sourceUsed by
InformationTheory.Kolmogorov.parseUnary_packBits
sourceUsed by
InformationTheory.Kolmogorov.encodeNat_decodeNat_packBits
sourceUsed by
InformationTheory.Kolmogorov.packIndex_packBits
sourceUsed by
InformationTheory.Kolmogorov.packPayload_packBits
sourceUsed by
InformationTheory.Kolmogorov.twoPartUnpack
sourceThe unpacking machine: read the index and the payload out of N, run the
payload through the machine's own payload decoder, and feed the result together
with the index to A.
Equations
- One or more equations did not get rendered due to their size.
Instances For
Used by
InformationTheory.Kolmogorov.packIndex_primrec
sourceUsed by
InformationTheory.Kolmogorov.packPayload_primrec
sourceUsed by
InformationTheory.Kolmogorov.twoPartUnpack_partrec
sourceUsed by
InformationTheory.Kolmogorov.payloadComplexity_two_part_le
sourceFeeding a partial recursive A a value y and an index i costs, in payload
complexity, no more than the payload complexity of y plus the self-delimited
index, up to a constant depending only on A. The hypothesis Partrec₂ A is what
puts A on the machine at all: it is discharged into a Code and never supplies
the bound.
@audit:ok
Used by
Two-part descriptions bound the prefix complexity #
InformationTheory.Kolmogorov.listIndexDecoder
sourceThe two-part decoder: read y as a list of naturals and return its i-th
entry.
Equations
Instances For
Used by
InformationTheory.Kolmogorov.listIndexDecoder_partrec
sourceUsed by
InformationTheory.Kolmogorov.singletonCode_partrec
sourceUsed by
InformationTheory.Kolmogorov.modelComplexity_singleton_le
sourceUsed by
InformationTheory.Kolmogorov.exists_isSufficientStatistic_singleton
sourceUsed by
InformationTheory.Kolmogorov.prefixComplexity_le_twoPartLength
sourceEvery two-part description of x bounds its prefix complexity, up to an
additive constant independent of x and of the model. The index part carries the
machine-specific coefficient 4 built into twoPartLength, so this is not the
textbook inequality K(x) ≤ K(S) + log |S| + O(1); the model part does carry
coefficient 1.
@audit:ok
Used by
InformationTheory.Kolmogorov.mdlComplexity_sub_prefixComplexity_le
sourceThe shortest two-part description length agrees with the prefix complexity up
to an additive constant. As with prefixComplexity_le_twoPartLength, the index
part of twoPartLength carries the machine-specific coefficient 4, so this is
a statement about that quantity and not the textbook minimum description length
principle.
@audit:ok