InformationTheory

InformationTheory.Shannon.Kolmogorov.SufficientStatistic

source

Kolmogorov sufficient statistics and two-part descriptions #

Cover–Thomas (2nd ed.). A finite model S ∋ x describes x in two parts: the model itself, and the index of x inside S. The model part is measured by the prefix complexity modelComplexity S of a canonical code for S, and the index part by Nat.clog 2 S.card bits.

Main definitions #

Main statements #

Implementation notes #

The index part of twoPartLength carries the coefficient 4, not 1, and the two factors of two have different sources. One is a property of the machine prefixUniversalEval: prefix complexity and payload complexity are rigidly tied by the identity prefixComplexity x = 2 * payloadComplexity x + 1, so a bound proved in the payload world doubles on the way back. The other is a property of the packing used here, which delimits the index with selfDelimit and therefore spends two bits per index bit. The textbook inequality K(x) ≤ K(S) + log |S| + O(1), with coefficient 1 on the index part, is a statement about an additively universal prefix machine and is not claimed here; the coefficient on the model part modelComplexity S is exactly 1 because both sides pass through the same identity.

References #

Cover–Thomas (2nd ed.).

Tags #

Kolmogorov complexity, sufficient statistic, minimum description length

Models and two-part descriptions #

noncomputable def

InformationTheory.Kolmogorov.modelCode

source
(S : Finset ) :

The canonical code of a finite model: the Encodable code of its sorted element list. Sorted lists are used rather than Finset itself because the decoder that reads a model back has to be primitive recursive, and List is Primcodable while Finset is not. @audit:ok

Equations
Instances For
    Used by
      noncomputable def

      InformationTheory.Kolmogorov.modelComplexity

      source
      (S : Finset ) :

      The description length of a finite model: the prefix complexity of its canonical code. Using prefix complexity rather than the literal length of modelCode keeps the quantity insensitive to the redundancy of the canonical code. @audit:ok

      Equations
      Instances For
        Used by
          noncomputable def

          InformationTheory.Kolmogorov.twoPartLength

          source
          (S : Finset ) :

          The length of the two-part description built from the model S: the model part plus the index part. The coefficient 4 on the index part is built into the definition rather than recovered by the argument (see the implementation notes); the coefficient on the model part is exactly 1. @audit:ok

          Equations
          Instances For
            Used by
              noncomputable def

              InformationTheory.Kolmogorov.mdlComplexity

              source
              (x : ) :

              The shortest two-part description length of x, minimized over all finite models containing x. The singleton model always competes, so the infimum is attained (mdlComplexity_spec). @audit:ok

              Equations
              Instances For
                Used by
                  noncomputable def

                  InformationTheory.Kolmogorov.structureFunction

                  source
                  (x k : ) :

                  The structure function of x: the least index part ⌈log₂ |S|⌉ over models S ∋ x whose description length is within the budget k. The value is ℕ∞ because the constraint set is empty for small budgets, where the value has to be rather than 0 (structureFunction_zero); over the empty infimum would collapse to 0 and contradict structureFunction_antitone. @audit:ok

                  Equations
                  Instances For
                    Used by
                      def

                      InformationTheory.Kolmogorov.IsSufficientStatistic

                      source
                      (c x : ) (S : Finset ) :

                      A model S is a sufficient statistic for x at slack c when it contains x and its two-part description is no longer than prefixComplexity x + c. The slack is an explicit argument so that minimality statements can quantify over it. @audit:ok

                      Equations
                      Instances For
                        Used by
                          theorem

                          InformationTheory.Kolmogorov.natLen_le_clog_card

                          source
                          {S : Finset } {i : } (h : i < S.card) :
                          Used by
                            theorem

                            InformationTheory.Kolmogorov.exists_index_of_mem

                            source
                            {S : Finset } {x : } (h : x S) :
                            i < S.card, (S.sort fun (x1 x2 : ) => x1 x2)[i]? = some x
                            Used by
                              theorem

                              InformationTheory.Kolmogorov.structureFunction_antitone

                              source
                              (x : ) {k k' : } (h : k k') :
                              Used by
                                theorem

                                InformationTheory.Kolmogorov.structureFunction_zero

                                source
                                Used by
                                  theorem

                                  InformationTheory.Kolmogorov.structureFunction_eq_zero_of_singleton_budget

                                  source
                                  {x k : } (h : modelComplexity {x} k) :
                                  Used by
                                    theorem

                                    InformationTheory.Kolmogorov.mdlComplexity_set_nonempty

                                    source
                                    (x : ) :
                                    {l : | ∃ (S : Finset ), x S twoPartLength S = l}.Nonempty
                                    Used by
                                      theorem

                                      InformationTheory.Kolmogorov.mdlComplexity_spec

                                      source
                                      (x : ) :
                                      Used by
                                        theorem

                                        InformationTheory.Kolmogorov.mdlComplexity_le_of_mem

                                        source
                                        {x : } {S : Finset } (h : x S) :
                                        Used by

                                          The bit codec is primitive recursive #

                                          theorem

                                          InformationTheory.Kolmogorov.encodeNat_zero

                                          source
                                          Used by
                                            theorem

                                            InformationTheory.Kolmogorov.encodeNat_cast_posNum

                                            source
                                            Used by
                                              theorem

                                              InformationTheory.Kolmogorov.encodePosNum_eq_cons

                                              source
                                              Used by
                                                theorem

                                                InformationTheory.Kolmogorov.encodeNat_eq_cons

                                                source
                                                {n : } (hn : n 0) :
                                                Used by
                                                  theorem

                                                  InformationTheory.Kolmogorov.encodePosNum_decodePosNum_concat

                                                  source
                                                  Used by
                                                    theorem

                                                    InformationTheory.Kolmogorov.encodeNat_decodeNat_concat

                                                    source
                                                    Used by
                                                      def

                                                      InformationTheory.Kolmogorov.bitStep

                                                      source

                                                      One step of the little-endian bit expansion: peel off the lowest bit of the remaining value and append it to the accumulated bit string.

                                                      Equations
                                                      Instances For
                                                        Used by
                                                          theorem

                                                          InformationTheory.Kolmogorov.bitStep_iterate

                                                          source
                                                          (k n : ) (acc : List Bool) :
                                                          n k(bitStep^[k] (n, acc)).2 = acc ++ Computability.encodeNat n
                                                          Used by
                                                            theorem

                                                            InformationTheory.Kolmogorov.encodeNat_eq_iterate

                                                            source
                                                            Used by
                                                              theorem

                                                              InformationTheory.Kolmogorov.encodeNat_primrec

                                                              source
                                                              Used by
                                                                theorem

                                                                InformationTheory.Kolmogorov.decodeNat_of_ne_nil

                                                                source
                                                                Used by
                                                                  theorem

                                                                  InformationTheory.Kolmogorov.decodeNat_cons

                                                                  source
                                                                  (b : Bool) (l : List Bool) :
                                                                  Computability.decodeNat (b :: l) = if l = [] then bif b then 1 else 2 else 2 * Computability.decodeNat l + bif b then 1 else 0
                                                                  Used by
                                                                    theorem

                                                                    InformationTheory.Kolmogorov.decodeNat_eq_rec

                                                                    source
                                                                    (l : List Bool) :
                                                                    Computability.decodeNat l = List.recOn l 0 fun (b : Bool) (rest : List Bool) (IH : ) => if rest = [] then bif b then 1 else 2 else 2 * IH + bif b then 1 else 0
                                                                    Used by
                                                                      theorem

                                                                      InformationTheory.Kolmogorov.decodeNat_primrec

                                                                      source
                                                                      Used by
                                                                        theorem

                                                                        InformationTheory.Kolmogorov.parseUnary_eq_rec

                                                                        source
                                                                        (l : List Bool) :
                                                                        parseUnary l = List.recOn l (0, []) fun (b : Bool) (rest : List Bool) (IH : × List Bool) => bif b then (IH.1 + 1, IH.2) else (0, rest)
                                                                        Used by
                                                                          theorem

                                                                          InformationTheory.Kolmogorov.parseUnary_primrec

                                                                          source
                                                                          Used by

                                                                            The payload decoder is partial recursive #

                                                                            noncomputable def

                                                                            InformationTheory.Kolmogorov.payloadDispatch

                                                                            source

                                                                            The payload decoder presented as a code together with its input: the literal mode uses the identity code, and the interpret mode the code named by the unary index. The empty payload has no dispatch, which is how decodePayload diverges on it.

                                                                            Equations
                                                                            Instances For
                                                                              Used by
                                                                                theorem

                                                                                InformationTheory.Kolmogorov.payloadDispatch_primrec

                                                                                source
                                                                                Used by
                                                                                  theorem

                                                                                  InformationTheory.Kolmogorov.payloadDispatch_computable

                                                                                  source
                                                                                  Used by
                                                                                    theorem

                                                                                    InformationTheory.Kolmogorov.decodePayload_eq_dispatch

                                                                                    source
                                                                                    (d : List Bool) :
                                                                                    decodePayload d = (↑(payloadDispatch d)).bind fun (p : Nat.Partrec.Code × ) => p.1.eval p.2
                                                                                    Used by
                                                                                      theorem

                                                                                      InformationTheory.Kolmogorov.decodePayload_partrec

                                                                                      source
                                                                                      Used by

                                                                                        Self-simulation of the machine #

                                                                                        theorem

                                                                                        InformationTheory.Kolmogorov.payloadComplexity_le_of_eval

                                                                                        source
                                                                                        (c : Nat.Partrec.Code) :
                                                                                        ∃ (b : ), ∀ (x : ) (q : List Bool), x c.eval (Nat.pair (Computability.decodeNat q) 0)payloadComplexity x q.length + b
                                                                                        Used by
                                                                                          theorem

                                                                                          InformationTheory.Kolmogorov.payload_invariance

                                                                                          source
                                                                                          (A : Part ) (hA : Partrec₂ A) :
                                                                                          ∃ (b : ), ∀ (x : ) (q : List Bool), x A (Computability.decodeNat q) 0payloadComplexity x q.length + b
                                                                                          Used by
                                                                                            def

                                                                                            InformationTheory.Kolmogorov.packIndex

                                                                                            source
                                                                                            (N : ) :

                                                                                            The index part packed into N: the self-delimited prefix of the bit string of N.

                                                                                            Equations
                                                                                            • One or more equations did not get rendered due to their size.
                                                                                            Instances For
                                                                                              Used by
                                                                                                def

                                                                                                InformationTheory.Kolmogorov.packPayload

                                                                                                source
                                                                                                (N : ) :

                                                                                                The payload part packed into N: what follows the self-delimited index, with the terminating sentinel bit removed.

                                                                                                Equations
                                                                                                • One or more equations did not get rendered due to their size.
                                                                                                Instances For
                                                                                                  Used by
                                                                                                    def

                                                                                                    InformationTheory.Kolmogorov.packBits

                                                                                                    source
                                                                                                    (i : ) (d : List Bool) :

                                                                                                    The bit string packing an index i together with a payload d: the index in self-delimited form, then the payload, then a sentinel bit making the string a canonical binary expansion.

                                                                                                    Equations
                                                                                                    Instances For
                                                                                                      Used by
                                                                                                        theorem

                                                                                                        InformationTheory.Kolmogorov.packBits_length

                                                                                                        source
                                                                                                        (i : ) (d : List Bool) :
                                                                                                        (packBits i d).length = 2 * natLen i + d.length + 2
                                                                                                        Used by
                                                                                                          theorem

                                                                                                          InformationTheory.Kolmogorov.parseUnary_packBits

                                                                                                          source
                                                                                                          Used by
                                                                                                            theorem

                                                                                                            InformationTheory.Kolmogorov.encodeNat_decodeNat_packBits

                                                                                                            source
                                                                                                            Used by
                                                                                                              theorem

                                                                                                              InformationTheory.Kolmogorov.packIndex_packBits

                                                                                                              source
                                                                                                              Used by
                                                                                                                theorem

                                                                                                                InformationTheory.Kolmogorov.packPayload_packBits

                                                                                                                source
                                                                                                                Used by
                                                                                                                  noncomputable def

                                                                                                                  InformationTheory.Kolmogorov.twoPartUnpack

                                                                                                                  source
                                                                                                                  (A : Part ) (N : ) :

                                                                                                                  The unpacking machine: read the index and the payload out of N, run the payload through the machine's own payload decoder, and feed the result together with the index to A.

                                                                                                                  Equations
                                                                                                                  • One or more equations did not get rendered due to their size.
                                                                                                                  Instances For
                                                                                                                    Used by
                                                                                                                      theorem

                                                                                                                      InformationTheory.Kolmogorov.packIndex_primrec

                                                                                                                      source
                                                                                                                      Used by
                                                                                                                        theorem

                                                                                                                        InformationTheory.Kolmogorov.packPayload_primrec

                                                                                                                        source
                                                                                                                        Used by
                                                                                                                          theorem

                                                                                                                          InformationTheory.Kolmogorov.twoPartUnpack_partrec

                                                                                                                          source
                                                                                                                          {A : Part } (hA : Partrec₂ A) :
                                                                                                                          Used by
                                                                                                                            theorem

                                                                                                                            InformationTheory.Kolmogorov.payloadComplexity_two_part_le

                                                                                                                            source
                                                                                                                            (A : Part ) (hA : Partrec₂ A) :
                                                                                                                            ∃ (c : ), ∀ (x y i : ), x A y ipayloadComplexity x payloadComplexity y + 2 * natLen i + c

                                                                                                                            Feeding a partial recursive A a value y and an index i costs, in payload complexity, no more than the payload complexity of y plus the self-delimited index, up to a constant depending only on A. The hypothesis Partrec₂ A is what puts A on the machine at all: it is discharged into a Code and never supplies the bound. @audit:ok

                                                                                                                            Used by

                                                                                                                              Two-part descriptions bound the prefix complexity #

                                                                                                                              def

                                                                                                                              InformationTheory.Kolmogorov.listIndexDecoder

                                                                                                                              source
                                                                                                                              (y i : ) :

                                                                                                                              The two-part decoder: read y as a list of naturals and return its i-th entry.

                                                                                                                              Equations
                                                                                                                              Instances For
                                                                                                                                Used by
                                                                                                                                  theorem

                                                                                                                                  InformationTheory.Kolmogorov.listIndexDecoder_partrec

                                                                                                                                  source
                                                                                                                                  :
                                                                                                                                  Partrec₂ fun (y i : ) => (listIndexDecoder y i)
                                                                                                                                  Used by
                                                                                                                                    theorem

                                                                                                                                    InformationTheory.Kolmogorov.singletonCode_partrec

                                                                                                                                    source
                                                                                                                                    Used by
                                                                                                                                      theorem

                                                                                                                                      InformationTheory.Kolmogorov.modelComplexity_singleton_le

                                                                                                                                      source
                                                                                                                                      :
                                                                                                                                      ∃ (c : ), ∀ (x : ), modelComplexity {x} prefixComplexity x + c
                                                                                                                                      Used by
                                                                                                                                        theorem

                                                                                                                                        InformationTheory.Kolmogorov.exists_isSufficientStatistic_singleton

                                                                                                                                        source
                                                                                                                                        :
                                                                                                                                        ∃ (c : ), ∀ (x : ), IsSufficientStatistic c x {x}
                                                                                                                                        Used by
                                                                                                                                          theorem

                                                                                                                                          InformationTheory.Kolmogorov.prefixComplexity_le_twoPartLength

                                                                                                                                          source
                                                                                                                                          :
                                                                                                                                          ∃ (c : ), ∀ (x : ) (S : Finset ), x SprefixComplexity x twoPartLength S + c

                                                                                                                                          Every two-part description of x bounds its prefix complexity, up to an additive constant independent of x and of the model. The index part carries the machine-specific coefficient 4 built into twoPartLength, so this is not the textbook inequality K(x) ≤ K(S) + log |S| + O(1); the model part does carry coefficient 1. @audit:ok

                                                                                                                                          Used by
                                                                                                                                            theorem

                                                                                                                                            InformationTheory.Kolmogorov.mdlComplexity_sub_prefixComplexity_le

                                                                                                                                            source

                                                                                                                                            The shortest two-part description length agrees with the prefix complexity up to an additive constant. As with prefixComplexity_le_twoPartLength, the index part of twoPartLength carries the machine-specific coefficient 4, so this is a statement about that quantity and not the textbook minimum description length principle. @audit:ok

                                                                                                                                            Used by