InformationTheory

InformationTheory.Shannon.SMB.AlgoetCover.MarkovLikelihoodRatio

source

Likelihood ratio of the k-Markov approximation #

The upward likelihood ratio comparing the k-Markov approximation of a stationary process against its true block law, and the almost-sure bound on that ratio used by the Algoet–Cover proof of the Shannon–McMillan–Breiman theorem.

Main definitions #

  • markovFactor — the k-Markov conditional-kernel mass at the last index of a Fin (n+1)-tuple: the full prefix is used while n ≤ k, the trailing window of k symbols afterwards.
  • qkSingleton — the k-Markov joint mass of a path, the product of markovFactors along it.
  • condQk / condQkState — conditional forms of that mass, carrying the total-mass bounds ∑ ≤ 1.
  • MRatioUp — the upward ratio exp (n · blockLogAvg − negLogQk), which is a.s. the likelihood ratio qₖ(Xⁿ) / Pₙ(Xⁿ) (MRatioUp_eq_ofReal_exp).

Main statements #

Likelihood ratio and Borel–Cantelli #

noncomputable def

InformationTheory.Shannon.markovFactor

source
{Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] (p : StationaryProcess μ α) (k n : ) (y : Fin (n + 1)α) :

The k-Markov conditional-kernel mass at the last index of a Fin (n+1)-tuple. For n ≤ k: uses the full prefix Fin.init y : Fin n → α and the kernel condDistrib (obs n) (blockRV n) μ. For n > k: uses the last k symbols of the prefix (a window indexed n-k+j) and the kernel condDistrib (obs k) (blockRV k) μ.

Equations
  • One or more equations did not get rendered due to their size.
Instances For
    Used by
      noncomputable def

      InformationTheory.Shannon.qkSingleton

      source
      {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] (p : StationaryProcess μ α) (k n : ) :
      (Fin nα)ENNReal

      The k-Markov joint mass of a path y : Fin n → α, defined recursively as the product of markovFactors along the path. When evaluated at y = blockRV n ω, this equals (a.s.) exp(-negLogQk μ p k n ω).

      Equations
      Instances For
        Used by
          theorem

          InformationTheory.Shannon.markovFactor_sum_eq_one

          source
          {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (p : StationaryProcess μ α) (k n : ) (z : Fin nα) :
          a : α, markovFactor μ p k n (Fin.snoc z a) = 1

          Per-state, markovFactor μ p k n is a genuine probability distribution over the next symbol: summing over all continuations a : α of a fixed prefix z gives exactly 1. This holds because markovFactor is a condDistrib kernel singleton mass and condDistrib is an IsMarkovKernel (so kernel z univ = 1). It is the enabler for the correct-direction conditional log-sum (the per-step factor of the k-Markov measure is a bona-fide sub-distribution that telescopes to qkSingleton).

          Used by
            noncomputable def

            InformationTheory.Shannon.condQk

            source
            {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] (p : StationaryProcess μ α) (k start : ) (z : Fin startα) ( : ) :
            (Fin α)ENNReal

            The k-Markov conditional mass of a length- continuation w : Fin ℓ → α extending a fixed prefix z : Fin start → α, defined as the product of markovFactors at the absolute positions start, start+1, …, start+ℓ-1. The tuple fed to each markovFactor (start+m) is the combined prefix Fin.append z (init…w) recast to Fin ((start+m)+1) → α. At ℓ = 0 this is 1; the recursion peels the last symbol of w via Fin.init, matching qkSingleton's chain-rule structure so the product telescopes to qkSingleton (start+ℓ) / qkSingleton start.

            Equations
            Instances For
              Used by
                theorem

                InformationTheory.Shannon.condQk_sum_le_one

                source
                {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (p : StationaryProcess μ α) (k start : ) (z : Fin startα) ( : ) :
                w : Fin α, condQk μ p k start z w 1

                Conditional product sub-distribution from a fixed prefix: for any prefix z : Fin start → α, the k-Markov conditional masses of all length- continuations sum to at most 1. The non-empty-start generalization of sum_qkSingleton_le_one (which is the start = 0 case): same induction on , each step reindexing via Fin.snocEquiv and collapsing the inner symbol sum with markovFactor_sum_eq_one (which holds for an arbitrary prefix, hence works from a non-empty start). This is the per-fixed-context sub-distribution the conditional Ziv (k-state, length) grouping instantiates.

                Used by
                  theorem

                  InformationTheory.Shannon.markovFactor_sum_subset_le_one

                  source
                  {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (p : StationaryProcess μ α) (k n : ) (z : Fin nα) (T : Finset α) :
                  aT, markovFactor μ p k n (Fin.snoc z a) 1

                  Per-state sub-distribution: summing markovFactor μ p k n over any finite subset T of continuations is at most 1 (subset sum ≤ full sum = 1). This is the building block that the conditional Ziv grouping instantiates: a restricted set of continuations carries at most the full conditional probability mass.

                  Used by
                    theorem

                    InformationTheory.Shannon.markovFactor_eq_of_window_eq

                    source
                    {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] (p : StationaryProcess μ α) (k : ) {n₁ n₂ : } (h₁ : k < n₁) (h₂ : k < n₂) (y₁ : Fin (n₁ + 1)α) (y₂ : Fin (n₂ + 1)α) (hwin : ∀ (j : Fin k), y₁ n₁ - k + j, = y₂ n₂ - k + j, ) (hlast : y₁ (Fin.last n₁) = y₂ (Fin.last n₂)) :
                    markovFactor μ p k n₁ y₁ = markovFactor μ p k n₂ y₂

                    Position invariance of markovFactor (the n > k branch). For n₁, n₂ > k, markovFactor μ p k n depends only on the last k + 1 symbols of its argument (the k-symbol window plus the last symbol), not on the absolute position n. This is the foundation of the conditional Ziv (k-state, length) grouping: the conditional mass of a phrase depends only on its trailing k-state, so phrases sharing a k-state may be grouped regardless of where they occur. The n > k branch of markovFactor uses the fixed kernel condDistrib (obs k) (blockRV k) μ, whose argument is the window and whose singleton set is the last symbol; both agree across n₁, n₂ under the window/last hypotheses, so the two factors are equal.

                    Used by
                      noncomputable def

                      InformationTheory.Shannon.condQkState

                      source
                      {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] (p : StationaryProcess μ α) (k : ) (s : Fin kα) ( : ) :
                      (Fin α)ENNReal

                      The k-Markov conditional masses started from a fixed k-state s : Fin k → α. This is condQk specialized to start = k: the per-k-state conditional product sub-distribution that the (k-state, length) Ziv grouping instantiates.

                      Equations
                      Instances For
                        Used by
                          theorem

                          InformationTheory.Shannon.condQkState_sum_le_one

                          source
                          {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (p : StationaryProcess μ α) (k : ) (s : Fin kα) ( : ) :
                          w : Fin α, condQkState μ p k s w 1

                          The per-k-state conditional masses sum to at most 1: the start = k specialization of condQk_sum_le_one.

                          Used by
                            theorem

                            InformationTheory.Shannon.sum_qkSingleton_le_one

                            source
                            {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (p : StationaryProcess μ α) (k n : ) :
                            y : Fin nα, qkSingleton μ p k n y 1

                            ∑_y qkSingleton k n y ≤ 1: the inductive product is bounded by 1 because each inner sum ∑_a (condDistrib ...){a} = 1 by IsMarkovKernel.

                            Used by
                              theorem

                              InformationTheory.Shannon.markovFactor_blockRV_pos_ae

                              source
                              {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (p : StationaryProcess μ α) (k : ) :
                              ∀ᵐ (ω : Ω) μ, ∀ (i : ), 0 < (markovFactor μ p k i (p.blockRV (i + 1) ω)).toReal

                              A.s. positivity of every per-position markovFactor evaluated at the block random variable: for a probability-preserving stationary process, a.s. (in ω) the real-valued Markov factor markovFactor μ p k i (blockRV (i+1) ω) is strictly positive at every position i. This is the a.s. positivity input that the LZ78 threading tiling discharges (the per-phrase hposfac precondition of negLogQk_phrase_threading). It follows from cond_singleton_pos_ae (the conditional kernel singleton mass is a.s. positive): for i ≤ k directly, and for k < i via the measure-preserving shift T^[i-k].

                              @audit:ok

                              Used by
                                theorem

                                InformationTheory.Shannon.qkSingleton_blockRV_eq_ofReal_exp_negLogQk

                                source
                                {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (p : StationaryProcess μ α) (k n : ) :
                                ∀ᵐ (ω : Ω) μ, qkSingleton μ p k n (p.blockRV n ω) = ENNReal.ofReal (Real.exp (-negLogQk μ p k n ω))

                                A.s., qkSingleton μ p k n (blockRV n ω) equals ofReal (exp (-negLogQk μ p k n ω)). This is the numerator half of MRatioUp_eq_ofReal_exp.

                                Used by
                                  theorem

                                  InformationTheory.Shannon.MRatioUp_eq_ofReal_exp

                                  source
                                  {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (p : StationaryProcess μ α) (k n : ) :
                                  ∀ᵐ (ω : Ω) μ, qkSingleton μ p k n (p.blockRV n ω) / (MeasureTheory.Measure.map (p.blockRV n) μ) {p.blockRV n ω} = ENNReal.ofReal (Real.exp (n * blockLogAvg μ p n ω - negLogQk μ p k n ω))

                                  A.s. equivalence between the MRatioUp ratio form and the exp-of-difference form used by downstream lemmas (MRatioUp_le_sq_eventually, blockLogAvg_le_negLogQk_plus_error).

                                  Used by
                                    noncomputable def

                                    InformationTheory.Shannon.MRatioUp

                                    source

                                    Upward likelihood ratio: exp(n · blockLogAvg - negLogQk) lifted to ENNReal.

                                    Equations
                                    Instances For
                                      Used by
                                        theorem

                                        InformationTheory.Shannon.integral_MRatioUp_le_one

                                        source
                                        {Ω : Type u_1} [MeasurableSpace Ω] {α : Type u_2} [Fintype α] [Nonempty α] [MeasurableSpace α] [MeasurableSingletonClass α] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (p : StationaryProcess μ α) (k n : ) :
                                        ∫⁻ (ω : Ω), MRatioUp μ p k n ω μ 1

                                        Markov inequality input: the upward ratio integrates to at most 1.

                                        Proof:

                                        1. Bridge MRatioUp to the ratio form qkSingleton k n (blockRV n ω) / P_n {blockRV n ω} a.s. via MRatioUp_eq_ofReal_exp.
                                        2. Push forward through blockRV n using lintegral_map, then lintegral_fintype over the finite alphabet: ∑_y qkSingleton k n y / P_n {y} * P_n {y}.
                                        3. (a / b) * b ≤ a (unconditional in ENNReal): bound the sum by ∑_y qkSingleton k n y.
                                        4. Apply sum_qkSingleton_le_one.
                                        Used by
                                          theorem

                                          InformationTheory.Shannon.MRatioUp_le_sq_eventually

                                          source

                                          Borel–Cantelli consequence: the upward ratio is eventually bounded by a.s.

                                          Used by