InformationTheory

InformationTheory.Shannon.LZ78.GreedyLongestPrefix

source

LZ78 longest-prefix greedy parsing — distinct-phrase invariant #

A one-symbol-per-step parse (always feeding the dictionary the singleton [s], so every phrase consumes exactly one symbol and count = n) makes the distinct-phrase invariant false (the same singleton recurs), so the sharp counting bound c(n) · log c(n) ≤ K·n cannot hold.

This file establishes the longest-prefix-match greedy parse together with the central invariant: the list of emitted phrase strings (the LZ78 dictionary entries) is Nodup.

Design (Mathlib-shape-driven) #

The dominant lemma for the distinct invariant is List.Nodup.concat : a ∉ l → l.Nodup → (l.concat a).Nodup (Mathlib/Data/List/Nodup.lean). To make its hypothesis a ∉ l available by construction rather than recovered from a "longest match" argument, the worker grows the current prefix symbol-by-symbol from the input and emits the prefix the moment it leaves the dictionary:

  • maintain dict : List (List α) of already-emitted phrase strings;
  • walk the input building a candidate prefix acc; while acc is still in dict, keep extending; the first time acc ∉ dict, emit acc and reset.

The emitted string is then definitionally ∉ dict, so Nodup.concat applies directly with no longest-match reconstruction. This is exactly LZ78's behavior (match the longest dictionary prefix, append one new symbol), recorded in the cheapest shape for the invariant proof.

We track only the phrase strings here — that is the object the counting bound reasons about. The back-pointer LZ78Parsing / inRange structure of LZ78/Basic.lean is left untouched.

File layout #

  • §1. Longest-prefix worker — lz78PhraseStringsAux / lz78PhraseStrings: the greedy parse, returning the list of emitted phrase strings.
  • §2. Distinct-phrase invariant — lz78PhraseStrings_nodup: the list of emitted phrase strings is Nodup.
  • §3. Length conservation — lz78PhraseStrings_total_length: the total number of symbols across emitted phrases (plus the unfinished tail) equals the input length, supplying the counting denominator n.

§1. Longest-prefix worker #

def

InformationTheory.Shannon.lz78PhraseStringsAux

source
{α : Type u_1} [DecidableEq α] :
List (List α)List αList αList (List α)

The longest-prefix greedy worker returning the list of emitted phrase strings (the LZ78 dictionary entries) in order.

  • fuel : ℕ bounds recursion depth (instantiated to input.length + 1).
  • dict : List (List α) — phrase strings emitted so far, in order.
  • cur : List α — the candidate prefix being grown (still ∈ dict or empty); acc plus the next symbol forms the next test.
  • input : List α — the remaining un-consumed suffix.

Invariant maintained for the Nodup proof: dict.Nodup and cur ∈ dict (or cur = []). At each symbol s, form cur ++ [s]; if it is already a dictionary entry, keep growing; otherwise emit it (it is ∉ dict by the guard) and reset cur := [].

Equations
Instances For
    Used by
      def

      InformationTheory.Shannon.lz78PhraseStrings

      source
      {α : Type u_1} [DecidableEq α] (input : List α) :
      List (List α)

      lz78PhraseStrings input — the longest-prefix greedy parse of input, returning the ordered list of emitted phrase strings.

      Equations
      Instances For
        Used by

          §2. Distinct-phrase invariant #

          theorem

          InformationTheory.Shannon.lz78PhraseStringsAux_nodup

          source
          {α : Type u_1} [DecidableEq α] (fuel : ) (dict : List (List α)) (cur input : List α) :
          dict.Nodup(lz78PhraseStringsAux fuel dict cur input).Nodup

          The worker preserves Nodup: if the running dictionary is already Nodup, the dictionary returned by the worker is Nodup. The emitted string is ∉ dict by the if-guard, so List.Nodup.concat applies.

          Used by
            theorem

            InformationTheory.Shannon.lz78PhraseStrings_nodup

            source
            {α : Type u_1} [DecidableEq α] (input : List α) :

            The list of emitted phrase strings of the longest-prefix greedy parse is Nodup.

            Used by

              §3. Length conservation #

              theorem

              InformationTheory.Shannon.foldr_length_append_singleton

              source
              {α : Type u_1} (l : List (List α)) (w : List α) :
              List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (l ++ [w]) = List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 l + w.length

              The additive foldr-length over an append: the total phrase length of l ++ [w] is the total over l plus w.length. (The accumulator fun w acc => w.length + acc is additive, so foldr_append would leave a non-zero seed; this dedicated lemma keeps the seed at 0.)

              Used by
                theorem

                InformationTheory.Shannon.foldrLength_take_succ

                source
                {α : Type u_1} (L : List (List α)) (j : ) (h : j < L.length) :
                List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (List.take (j + 1) L) = List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (List.take j L) + L[j].length
                Used by
                  theorem

                  InformationTheory.Shannon.foldrLength_take_mono

                  source
                  {α : Type u_1} (L : List (List α)) {i j : } (hij : i j) :
                  List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (List.take i L) List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (List.take j L)
                  Used by
                    theorem

                    InformationTheory.Shannon.foldrLength_take_ge_of_forall_ne_nil

                    source
                    {α : Type u_1} (L : List (List α)) (hne : wL, w []) {j : } (hj : j L.length) :
                    j List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (List.take j L)
                    Used by
                      theorem

                      InformationTheory.Shannon.lz78PhraseStringsAux_total_length

                      source
                      {α : Type u_1} [DecidableEq α] (fuel : ) (dict : List (List α)) (cur input : List α) :
                      input.length < fuelList.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (lz78PhraseStringsAux fuel dict cur input) List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 dict + cur.length + input.length

                      Worker form of the length budget: the total symbol count across the phrases the worker emits is at most the symbols it holds, namely the accumulated dict length plus cur.length plus input.length. No length is created, since the worker only re-emits dict entries and pieces carved out of cur ++ input.

                      Used by
                        theorem

                        InformationTheory.Shannon.lz78PhraseStrings_total_length_le

                        source
                        {α : Type u_1} [DecidableEq α] (input : List α) :
                        List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (lz78PhraseStrings input) input.length

                        The total number of symbols across all emitted phrase strings is at most the input length. (Each emitted phrase consumes input symbols; the unfinished tail accounts for the slack, so this is , not =.)

                        Used by
                          theorem

                          InformationTheory.Shannon.lz78PhraseStringsAux_flatten_conserve

                          source
                          {α : Type u_1} [DecidableEq α] (fuel : ) (dict : List (List α)) (cur input : List α) :
                          input.length < fuel∃ (tail : List α), (lz78PhraseStringsAux fuel dict cur input).flatten ++ tail = dict.flatten ++ cur ++ input

                          The worker conserves the flattened string: the concatenation (List.flatten) of all emitted phrase strings, followed by an unfinished tail, reproduces the symbols seen so far dict.flatten ++ cur ++ input. This is the reconstruction invariant: the phrases tile a prefix of the input by concatenation at their cumulative lengths, with the leftover tail being the un-emitted suffix (the -slack of lz78PhraseStrings_total_length_le). Proved by structural induction on fuel, tracking the cur accumulator across the keep-growing / emit branches.

                          @audit:ok

                          Used by
                            theorem

                            InformationTheory.Shannon.lz78PhraseStrings_flatten_prefix

                            source
                            {α : Type u_1} [DecidableEq α] (input : List α) :
                            ∃ (tail : List α), (lz78PhraseStrings input).flatten ++ tail = input

                            The concatenation of all emitted phrase strings, followed by an unfinished tail, equals the input. The cumulative phrase lengths therefore furnish an absolute-position tiling of a prefix of the input, with the tail accounting for the -slack. Instantiates lz78PhraseStringsAux_flatten_conserve at the top-level fuel.

                            @audit:ok

                            Used by
                              theorem

                              InformationTheory.Shannon.lz78PhraseStringsAux_tail_mem

                              source
                              {α : Type u_1} [DecidableEq α] (fuel : ) (dict : List (List α)) (cur input : List α) :
                              input.length < fuelcur dict cur = []∃ (tail : List α), (lz78PhraseStringsAux fuel dict cur input).flatten ++ tail = dict.flatten ++ cur ++ input (tail lz78PhraseStringsAux fuel dict cur input tail = [])

                              The worker conserves the flattened string and bounds the tail: the unfinished tail at termination is the final candidate prefix cur, which the greedy invariant keeps as a dictionary entry (or empty). So the tail is a member of the returned phrase list (or []), bounding its length by the longest phrase. Proved by induction on fuel, threading the invariant cur ∈ dict ∨ cur = [] (preserved on both the keep-growing and emit branches).

                              @audit:ok (fuel-induction establishing the tail ∈ dict ∪ {[]} invariant).

                              Used by
                                theorem

                                InformationTheory.Shannon.lz78PhraseStrings_flatten_tail_mem

                                source
                                {α : Type u_1} [DecidableEq α] (input : List α) :
                                ∃ (tail : List α), (lz78PhraseStrings input).flatten ++ tail = input (tail lz78PhraseStrings input tail = [])

                                The unfinished tail of the parse is a phrase string (a member of lz78PhraseStrings input) or empty. Hence its length is at most the longest phrase length, which bounds the un-emitted trailing tail input.length - e.

                                Used by
                                  theorem

                                  InformationTheory.Shannon.length_flatten_eq_foldr_length

                                  source
                                  {α : Type u_1} (L : List (List α)) :
                                  L.flatten.length = List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 L

                                  The flatten length equals the additive foldr total, bridging the reconstruction-invariant List.flatten length to the cumulative-length foldr accumulator used by the tiling.

                                  Used by

                                    Slice / content correspondence #

                                    theorem

                                    InformationTheory.Shannon.flatten_drop_take_getElem

                                    source
                                    {α : Type u_1} (L : List (List α)) (j : ) (hj : j < L.length) :
                                    List.take L[j].length (List.drop (List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (List.take j L)) L.flatten) = L[j]

                                    Flatten slice content correspondence (pure list fact). For a list of lists L, dropping the cumulative length of the first j sublists from the flatten and taking the j-th sublist length recovers L[j] exactly: (L.flatten.drop (cumLen j)).take (L[j].length) = L[j], where cumLen j = (L.take j).foldr (·.length + ·) 0.

                                    This is the content half of the absolute-position tiling: the tiling carries phrase lengths/positions, and this lemma certifies that the slice at those positions reproduces the j-th phrase string.

                                    Used by
                                      theorem

                                      InformationTheory.Shannon.getElem?_eq_some_drop_take_of_flatten_prefix

                                      source
                                      {α : Type u_1} (L : List (List α)) {input tail : List α} (htail : L.flatten ++ tail = input) (idx : ) (hidx : idx < L.length) :
                                      L[idx]? = some (List.take (List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (List.take (idx + 1) L) - List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (List.take idx L)) (List.drop (List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 (List.take idx L)) input))
                                      Used by

                                        §3b. Parent-extension invariant #

                                        theorem

                                        InformationTheory.Shannon.lz78PhraseStringsAux_dropLast_earlier

                                        source
                                        {α : Type u_1} [DecidableEq α] (fuel : ) (dict : List (List α)) (cur input : List α) :
                                        cur dict cur = [](∀ (j : ) (h : j < dict.length), dict[j].dropLast List.take j dict dict[j].dropLast = [])∀ (j : ) (h : j < (lz78PhraseStringsAux fuel dict cur input).length), (lz78PhraseStringsAux fuel dict cur input)[j].dropLast List.take j (lz78PhraseStringsAux fuel dict cur input) (lz78PhraseStringsAux fuel dict cur input)[j].dropLast = []

                                        The worker maintains the parent-extension invariant: every emitted phrase string is (an earlier dictionary entry) ++ [symbol] or [symbol]. Stated positionally: for the worker output D, each D[j].dropLast either equals [] or appears strictly earlier in D (i.e. in D.take j).

                                        This is the dictionary structure behind the LZ78 phrase-count bound: at each step the worker emits cur ++ [s] where cur is a current dictionary entry (or []), and dictionaries only grow by appending, so cur sits strictly before the emitted phrase. Threaded with the running invariant cur ∈ dict ∨ cur = [].

                                        Used by
                                          theorem

                                          InformationTheory.Shannon.lz78PhraseStrings_dropLast_earlier

                                          source
                                          {α : Type u_1} [DecidableEq α] (input : List α) (j : ) (h : j < (lz78PhraseStrings input).length) :

                                          For the longest-prefix greedy parse, each emitted phrase's dropLast is either [] or an earlier emitted phrase.

                                          The property is specific to the LZ78 dictionary structure, not a general fact about lists of lists: it fails for L = [[a, b]], where L[0].dropLast = [a] ∉ L.take 0 = [] and ≠ []. What makes it hold is that the dictionary grows only by appending cur ++ [s] with cur an earlier entry, so the emit branch has (cur ++ [s]).dropLast = cur ∈ dict by List.dropLast_concat.

                                          @audit:ok (non-vacuous; the fuel-induction discharges its base and nil cases through the threaded running invariant hdict, not by fuel exhaustion, and the top level supplies the sufficient fuel input.length + 1, so the quantified parse is not the empty list).

                                          Used by

                                            §4. Phrase count bound #

                                            theorem

                                            InformationTheory.Shannon.lz78PhraseStringsAux_forall_ne_nil

                                            source
                                            {α : Type u_1} [DecidableEq α] (fuel : ) (dict : List (List α)) (cur input : List α) :
                                            (∀ wdict, w [])wlz78PhraseStringsAux fuel dict cur input, w []

                                            The worker emits only non-empty phrases: if every dictionary entry is non-empty, every entry of the worker output is non-empty. The emitted string is cur ++ [s], which is always non-empty.

                                            Used by
                                              theorem

                                              InformationTheory.Shannon.lz78PhraseStrings_forall_ne_nil

                                              source
                                              {α : Type u_1} [DecidableEq α] (input w : List α) :
                                              w lz78PhraseStrings inputw []

                                              All emitted phrase strings are non-empty.

                                              Used by
                                                theorem

                                                InformationTheory.Shannon.length_le_foldr_length_of_ne_nil

                                                source
                                                {α : Type u_1} (l : List (List α)) (h : wl, w []) :
                                                l.length List.foldr (fun (w : List α) (acc : ) => w.length + acc) 0 l

                                                When every phrase is non-empty, the count is bounded by the total length: the number of phrases is at most the sum of their lengths, since each length is ≥ 1.

                                                Used by
                                                  theorem

                                                  InformationTheory.Shannon.length_le_foldr_max_of_mem

                                                  source
                                                  {α : Type u_1} (l : List (List α)) (w : List α) (h : w l) :
                                                  w.length List.foldr (fun (w : List α) (acc : ) => max w.length acc) 0 l

                                                  Every entry of a list of strings has length at most the longest entry length (the Lmax accumulator used to bound the leading-boundary and trailing-tail symbol lengths in the tiling).

                                                  Used by
                                                    theorem

                                                    InformationTheory.Shannon.lz78PhraseStrings_count_le

                                                    source
                                                    {α : Type u_1} [DecidableEq α] (input : List α) :

                                                    The number of distinct phrases emitted by the longest-prefix greedy parse is at most the input length. Combined with lz78PhraseStrings_nodup (the strings are distinct), this is the count c(n) ≤ n feeding the Cover–Thomas counting bound c(n) · log c(n) ≤ K·n.

                                                    Used by

                                                      §5. Absolute-position tiling of the parse #

                                                      theorem

                                                      InformationTheory.Shannon.lz78_parse_tiling_positions

                                                      source
                                                      {α : Type u_1} [DecidableEq α] (input : List α) (k : ) :
                                                      ∃ (b : ) (c : ) (e : ) (bAbsorbed : ) (Lmax : ) (N : Fin (c + 1)), N 0 = b N (Fin.last c) = e e input.length (∀ (j : Fin c), N j.castSucc + 1 N j.succ) (∀ (j : Fin c), k < N j.castSucc) c + bAbsorbed = (lz78PhraseStrings input).length bAbsorbed k + 1 input.length - e Lmax b k + Lmax ∀ (j : Fin c), (lz78PhraseStrings input)[bAbsorbed + j]? = some (List.take (N j.succ - N j.castSucc) (List.drop (N j.castSucc) input))

                                                      Deterministic absolute-position tiling from the greedy parse. For an input list and a Markov order k, the greedy longest-prefix parse furnishes an absolute-position partition of the input prefix it covers: a leading boundary length b, a phrase count c, a covered length e ≤ input.length, the count bAbsorbed of leading phrases absorbed into [0, b) (those whose cumulative end is ≤ k), and the cumulative-position function N : Fin (c + 1) → ℕ.

                                                      The partition [b, e) is strictly monotone (each phrase consumes ≥ 1 symbol), every phrase start N j.castSucc exceeds k (the leading ≤ k phrases are absorbed), the count is anchored to the parse via c + bAbsorbed = (parse).length, and bAbsorbed ≤ k + 1 (the cumulative length increases by ≥ 1 each step, so at most k + 1 phrases fit below position k).

                                                      The boundary-length bounds (for the downstream limsup discharge): with Lmax the longest phrase length, the leading boundary b ≤ k + Lmax (the last absorbed phrase extends one phrase past the ≤ k cumulative prefix) and the trailing tail input.length - e ≤ Lmax (the un-emitted tail is one dictionary phrase or empty, lz78PhraseStrings_flatten_tail_mem).

                                                      This is the pure list-combinatorial heart of the LZ78 threading tiling: it carries the phrase lengths only (the downstream threading reads phrase content directly off the process, never the parse strings' content).

                                                      @audit:ok (pure list-combinatorial core, no hypothesis bundling; non-vacuous because c := parseCount - bAbsorbed with bAbsorbed = Nat.find the least index whose cumulative length exceeds k, so c > 0 whenever parseCount > k+1, and the boundary bounds come from lz78PhraseStrings_flatten_tail_mem plus the last-absorbed-phrase argument rather than from a vacuous tiling).

                                                      Used by