InformationTheory.Shannon.SMB.AlgoetCover.MarkovLikelihoodRatio
Likelihood ratio of the k-Markov approximation #
The upward likelihood ratio comparing the k-Markov approximation of a stationary
process against its true block law, and the almost-sure bound on that ratio used by
the Algoet–Cover proof of the Shannon–McMillan–Breiman theorem.
Main definitions #
markovFactor— thek-Markov conditional-kernel mass at the last index of aFin (n+1)-tuple: the full prefix is used whilen ≤ k, the trailing window ofksymbols afterwards.qkSingleton— thek-Markov joint mass of a path, the product ofmarkovFactors along it.condQk/condQkState— conditional forms of that mass, carrying the total-mass bounds∑ ≤ 1.MRatioUp— the upward ratioexp (n · blockLogAvg − negLogQk), which is a.s. the likelihood ratioqₖ(Xⁿ) / Pₙ(Xⁿ)(MRatioUp_eq_ofReal_exp).
Main statements #
sum_qkSingleton_le_one— thek-Markov masses of the paths of lengthnsum to at most1, soqₖis a sub-probability on each block.integral_MRatioUp_le_one—∫⁻ MRatioUp ∂μ ≤ 1, the input to Markov's inequality.MRatioUp_le_sq_eventually— a.s.MRatioUp k n ≤ n²for all largen, by Markov's inequality plus the first Borel–Cantelli lemma over the summable∑ 1/n².
Likelihood ratio and Borel–Cantelli #
InformationTheory.Shannon.markovFactor
sourceThe k-Markov conditional-kernel mass at the last index of a Fin (n+1)-tuple.
For n ≤ k: uses the full prefix Fin.init y : Fin n → α and the kernel
condDistrib (obs n) (blockRV n) μ. For n > k: uses the last k symbols of
the prefix (a window indexed n-k+j) and the kernel
condDistrib (obs k) (blockRV k) μ.
Equations
- One or more equations did not get rendered due to their size.
Instances For
Used by
InformationTheory.Shannon.qkSingleton
sourceThe k-Markov joint mass of a path y : Fin n → α, defined recursively as
the product of markovFactors along the path. When evaluated at y = blockRV n ω,
this equals (a.s.) exp(-negLogQk μ p k n ω).
Equations
- InformationTheory.Shannon.qkSingleton μ p k 0 x_2 = 1
- InformationTheory.Shannon.qkSingleton μ p k n.succ y = InformationTheory.Shannon.qkSingleton μ p k n (Fin.init y) * InformationTheory.Shannon.markovFactor μ p k n y
Instances For
Used by
InformationTheory.Shannon.markovFactor_sum_eq_one
sourcePer-state, markovFactor μ p k n is a genuine probability distribution over the
next symbol: summing over all continuations a : α of a fixed prefix z gives
exactly 1. This holds because markovFactor is a condDistrib kernel singleton
mass and condDistrib is an IsMarkovKernel (so kernel z univ = 1). It is the
enabler for the correct-direction conditional log-sum (the per-step factor of the
k-Markov measure is a bona-fide sub-distribution that telescopes to qkSingleton).
Used by
InformationTheory.Shannon.condQk
sourceThe k-Markov conditional mass of a length-ℓ continuation w : Fin ℓ → α
extending a fixed prefix z : Fin start → α, defined as the product of
markovFactors at the absolute positions start, start+1, …, start+ℓ-1. The tuple
fed to each markovFactor (start+m) is the combined prefix Fin.append z (init…w)
recast to Fin ((start+m)+1) → α. At ℓ = 0 this is 1; the recursion peels the
last symbol of w via Fin.init, matching qkSingleton's chain-rule structure so
the product telescopes to qkSingleton (start+ℓ) / qkSingleton start.
Equations
- One or more equations did not get rendered due to their size.
- InformationTheory.Shannon.condQk μ p k start z 0 x_2 = 1
Instances For
Used by
InformationTheory.Shannon.condQk_sum_le_one
sourceConditional product sub-distribution from a fixed prefix: for any prefix
z : Fin start → α, the k-Markov conditional masses of all length-ℓ
continuations sum to at most 1. The non-empty-start generalization of
sum_qkSingleton_le_one (which is the start = 0 case): same induction on ℓ,
each step reindexing via Fin.snocEquiv and collapsing the inner symbol sum with
markovFactor_sum_eq_one (which holds for an arbitrary prefix, hence works from a
non-empty start). This is the per-fixed-context sub-distribution the conditional Ziv
(k-state, length) grouping instantiates.
Used by
InformationTheory.Shannon.markovFactor_sum_subset_le_one
sourcePer-state sub-distribution: summing markovFactor μ p k n over any finite subset
T of continuations is at most 1 (subset sum ≤ full sum = 1). This is the
building block that the conditional Ziv grouping instantiates: a restricted set of
continuations carries at most the full conditional probability mass.
Used by
InformationTheory.Shannon.markovFactor_eq_of_window_eq
sourcePosition invariance of markovFactor (the n > k branch). For n₁, n₂ > k,
markovFactor μ p k n depends only on the last k + 1 symbols of its argument (the
k-symbol window plus the last symbol), not on the absolute position n. This is the
foundation of the conditional Ziv (k-state, length) grouping: the conditional mass of
a phrase depends only on its trailing k-state, so phrases sharing a k-state may be
grouped regardless of where they occur. The n > k branch of markovFactor uses the
fixed kernel condDistrib (obs k) (blockRV k) μ, whose argument is the window and whose
singleton set is the last symbol; both agree across n₁, n₂ under the window/last
hypotheses, so the two factors are equal.
Used by
InformationTheory.Shannon.condQkState
sourceThe k-Markov conditional masses started from a fixed k-state s : Fin k → α.
This is condQk specialized to start = k: the per-k-state conditional product
sub-distribution that the (k-state, length) Ziv grouping instantiates.
Equations
- InformationTheory.Shannon.condQkState μ p k s = InformationTheory.Shannon.condQk μ p k k s
Instances For
Used by
InformationTheory.Shannon.condQkState_sum_le_one
sourceThe per-k-state conditional masses sum to at most 1: the start = k
specialization of condQk_sum_le_one.
Used by
InformationTheory.Shannon.sum_qkSingleton_le_one
source∑_y qkSingleton k n y ≤ 1: the inductive product is bounded by 1 because each
inner sum ∑_a (condDistrib ...){a} = 1 by IsMarkovKernel.
Used by
InformationTheory.Shannon.markovFactor_blockRV_pos_ae
sourceA.s. positivity of every per-position markovFactor evaluated at the block
random variable: for a probability-preserving stationary process, a.s. (in ω) the
real-valued Markov factor markovFactor μ p k i (blockRV (i+1) ω) is strictly
positive at every position i. This is the a.s. positivity input that the LZ78
threading tiling discharges (the per-phrase hposfac precondition of
negLogQk_phrase_threading). It follows from cond_singleton_pos_ae (the
conditional kernel singleton mass is a.s. positive): for i ≤ k directly, and for
k < i via the measure-preserving shift T^[i-k].
@audit:ok
Used by
InformationTheory.Shannon.qkSingleton_blockRV_eq_ofReal_exp_negLogQk
sourceA.s., qkSingleton μ p k n (blockRV n ω) equals
ofReal (exp (-negLogQk μ p k n ω)). This is the numerator half of
MRatioUp_eq_ofReal_exp.
Used by
InformationTheory.Shannon.MRatioUp_eq_ofReal_exp
sourceA.s. equivalence between the MRatioUp ratio form and the
exp-of-difference form used by downstream lemmas (MRatioUp_le_sq_eventually,
blockLogAvg_le_negLogQk_plus_error).
Used by
InformationTheory.Shannon.MRatioUp
sourceUpward likelihood ratio: exp(n · blockLogAvg - negLogQk) lifted to ENNReal.
Equations
- InformationTheory.Shannon.MRatioUp μ p k n ω = ENNReal.ofReal (Real.exp (↑n * InformationTheory.Shannon.blockLogAvg μ p n ω - InformationTheory.Shannon.negLogQk μ p k n ω))
Instances For
Used by
InformationTheory.Shannon.integral_MRatioUp_le_one
sourceMarkov inequality input: the upward ratio integrates to at most 1.
Proof:
- Bridge
MRatioUpto the ratio formqkSingleton k n (blockRV n ω) / P_n {blockRV n ω}a.s. viaMRatioUp_eq_ofReal_exp. - Push forward through
blockRV nusinglintegral_map, thenlintegral_fintypeover the finite alphabet:∑_y qkSingleton k n y / P_n {y} * P_n {y}. (a / b) * b ≤ a(unconditional in ENNReal): bound the sum by∑_y qkSingleton k n y.- Apply
sum_qkSingleton_le_one.
Used by
InformationTheory.Shannon.MRatioUp_le_sq_eventually
sourceBorel–Cantelli consequence: the upward ratio is eventually bounded by n² a.s.