InformationTheory.Shannon.ChannelCoding.MIDecomp
Continuous-channel mutual-information chain rule (generic body) #
This file establishes the continuous-channel MI chain rule
I(X;Y) = h(Y) − h(Y|X), with h(Y|X) realized as the integral of fibrewise
differential entropies. The identity is not AWGN-specific: it holds for any
Markov channel W : Channel ℝ ℝ and input law p.
This is the AWGN-independent generic core, living upstream of the AWGN converse
chain (AWGN.Converse) so that the per-letter MI bridge
(awgn_per_letter_mi_bridge_genuine) can reuse it without an import cycle.
InformationTheory/Shannon/AWGN/ContChannelMIDecomp.lean imports this file and
re-exports these declarations under their unchanged fully-qualified names
(InformationTheory.Shannon.ChannelCoding.*), so downstream consumers (ParallelGaussian,
AWGN/CapacityConverseMaxent.lean) are unaffected.
Approach #
The MI chain identity is a density-level identity opened from the klDiv definition
of mutualInfoOfChannel:
I = ∫_z llr (p⊗ₘW) (p.prod q) z ∂(p⊗ₘW) -- toReal_klDiv_of_measure_eq
= ∫_z [log f_{Wx}(z.2) − log f_q(z.2)] ∂(p⊗ₘW) -- Bayes density split (★)
= ∫_x ∫_y log f_{Wx}(y) ∂(W x) ∂p -- integral_compProd
− ∫_y log f_q(y) ∂q -- snd marginal of (p⊗ₘW)
= −∫_x h(W x) ∂p + h(Y).
The KL→integral expansion, the Fubini split (integral_compProd), the output
marginal identification (outputDistribution = (p⊗ₘW).snd) and the
differential-entropy density form (differentialEntropy_eq_integral_density)
assemble the chain. The single step (★) — the Bayes density split of the joint
log-likelihood ratio into fibre/output log densities — is the
conditional-rnDeriv-to-fibre identification provided by the linchpin
rnDeriv_compProd_fibre (withDensity route), assembled by llr_compProd_prod_split.
InformationTheory.Shannon.ChannelCoding.integral_snd_outputDistribution
sourceMarginal identification. For a bounded-density observable
g : ℝ → ℝ, the joint integral of g ∘ snd against p ⊗ₘ W equals the integral
of g against the output marginal outputDistribution p W = (p ⊗ₘ W).snd.
Used by
InformationTheory.Shannon.ChannelCoding.integral_log_rnDeriv_eq_neg_diffEntropy
sourceGeneral log-density entropy identification. For any μ : Measure ℝ
with μ ≪ volume and measurable density f := μ.rnDeriv volume, the integral of
log f against μ is −differentialEntropy μ. This is the generalization of
integral_log_density_fibre to an arbitrary ≪ volume measure (the proof never used
that μ was a channel fibre); it is reused for both the fibre term (μ := W x) and
the output term (μ := outputDistribution p W) in the assembly below.
@audit:ok
Used by
InformationTheory.Shannon.ChannelCoding.integral_log_density_fibre
sourceFibre differential-entropy identification. For an ≪ volume
fibre W x with measurable density f := (W x).rnDeriv volume, the inner integral
of log f against W x is −differentialEntropy (W x). Thin wrapper around the
general integral_log_rnDeriv_eq_neg_diffEntropy.
Used by
InformationTheory.Shannon.ChannelCoding.integral_log_proxy_fibre
sourceProxy form of the fibre differential-entropy identification (Route B).
Same conclusion as integral_log_density_fibre, but stated with a measurable
PDF proxy g in place of the (non-jointly-measurable) measure-form rnDeriv.
The proxy↔rnDeriv bridge is absorbed inside the integral via integral_congr_ae
fed by the per-fibre a.e. agreement hg_ae, so no joint measurability is ever
needed. This is the step that lets the body keep its fibre term in proxy form.
Used by
InformationTheory.Shannon.ChannelCoding.rnDeriv_compProd_fibre
sourceLinchpin: fibre form of the compProd Radon-Nikodym derivative (Mathlib TODO,
Composition/RadonNikodym.lean:28-29). For finite μ, κ, η with μ ⊗ₘ κ ≪ μ ⊗ₘ η,
the conditional rnDeriv ∂(μ⊗ₘκ)/∂(μ⊗ₘη) is (μ⊗ₘη)-a.e. the fibrewise kernel
rnDeriv Kernel.rnDeriv κ η p.1 p.2.
Proof (withDensity route): μ⊗ₘκ ≪ μ⊗ₘη gives κ a ≪ η a a.e. (kernel_of_compProd),
so κ =ᵐ[μ] η.withDensity (κ.rnDeriv η) (withDensity_rnDeriv_eq); hence
μ⊗ₘκ = μ⊗ₘ(η.withDensity (κ.rnDeriv η)) = (μ⊗ₘη).withDensity (fun p ↦ κ.rnDeriv η p.1 p.2)
(compProd_congr + compProd_withDensity); finish with Measure.rnDeriv_withDensity.
Used by
InformationTheory.Shannon.ChannelCoding.log_rnDeriv_split
sourcePer-measure log-density split (Bayes step). For ν ≪ q ≪ volume
all σ-finite, the log of the relative density dν/dq splits as the difference of
the two volume-log-densities:
log (dν/dq y) = log (dν/dvol y) − log (dq/dvol y), ν-a.e. Built from the rnDeriv
chain rule dν/dq · dq/dvol =ᵐ dν/dvol plus positivity (q ≪ vol ⇒ dq/dvol > 0
on ν since ν ≪ q).
Used by
InformationTheory.Shannon.ChannelCoding.llr_compProd_prod_split
source★ General Bayes density split of the joint llr (modulo named ac
hyps). For input law p, Markov channel W, output q := outputDistribution p W,
with each fibre ≪ q ≪ volume and joint ≪ p.prod q, the log-likelihood ratio of
the joint against the product factorizes into fibre/output log-densities. This is
the body of the residual hypothesis h_llr_split. Combines the linchpin
rnDeriv_compProd_fibre (at η := Kernel.const ℝ q) with rnDeriv_eq_rnDeriv_measure
and the per-fibre log_rnDeriv_split.
Used by
InformationTheory.Shannon.ChannelCoding.mutualInfoOfChannel_toReal_eq_diffEntropy_sub
source★ Continuous-channel MI chain rule body (AWGN-independent).
(mutualInfoOfChannel p W).toReal = h(Y) − ∫ h(Y|X=x) dp(x), the density-level
analogue of the discrete mutualInfo_eq_entropy_add_entropy_sub_jointEntropy.
Assembled from the local helpers. The proof opens
mutualInfoOfChannel = klDiv (p⊗ₘW) (p.prod q)
(q := outputDistribution p W) via toReal_klDiv_of_measure_eq (both factors are
probability measures, so the univ-mass condition is automatic), rewrites the joint
log-likelihood ratio by the Bayes density split llr_compProd_prod_split, splits the
resulting integral with integral_sub, identifies the fibre term with
integral_compProd + integral_log_proxy_fibre (each fibre ↦ −h(W x)), and
identifies the output term with integral_snd_outputDistribution +
integral_log_rnDeriv_eq_neg_diffEntropy (↦ −h(q)).
@audit:ok
Used by
Generic (output type β + reference measure ref) MI chain rule #
The 1-D body mutualInfoOfChannel_toReal_eq_diffEntropy_sub above is specialized to
Channel ℝ ℝ with the Lebesgue reference volume : Measure ℝ. The block AWGN converse
needs the n-dimensional output form (β := Fin n → ℝ, ref := volume), so we
re-derive the same chain identity for an arbitrary input type α, output type β, and
reference measure ref : Measure β, stated in log-density-integral form
(∫ log (rnDeriv · ref)) rather than differentialEntropy, so the consumer is free to
identify each integral with whatever entropy notion it uses (jointDifferentialEntropyPi
for the AWGN block). The proof mirrors the 1-D one step for step; only the helper lemmas
are re-stated generically.
InformationTheory.Shannon.ChannelCoding.integral_snd_outputDistribution_gen
sourceGeneric marginal identification. For a ref-integrable observable g : β → ℝ,
the joint integral of g ∘ snd against p ⊗ₘ W equals the integral of g against the
output marginal outputDistribution p W = (p ⊗ₘ W).snd. Generic in α, β.
Used by
InformationTheory.Shannon.ChannelCoding.log_rnDeriv_split_gen
sourceGeneric per-measure log-density split (Bayes step). For ν ≪ q ≪ ref all
σ-finite (with ν.HaveLebesgueDecomposition q, q.HaveLebesgueDecomposition ref),
log (dν/dq y) = log (dν/dref y) − log (dq/dref y), ν-a.e. Generic reference ref.
Used by
InformationTheory.Shannon.ChannelCoding.llr_compProd_prod_split_gen
source★ Generic Bayes density split of the joint llr. For input law p, Markov
channel W : Channel α β, output q := outputDistribution p W, with each fibre
≪ q ≪ ref and joint ≪ p.prod q, the log-likelihood ratio of the joint against the
product factorizes into fibre/output log-densities. Generic in α, β, ref.
Used by
InformationTheory.Shannon.ChannelCoding.mutualInfoOfChannel_toReal_eq_log_density_sub
source★ Generic continuous-channel MI chain rule body (output type β, reference
ref), in log-density-integral form:
I.toReal = (∫ x, ∫ y, log(d(W x)/d ref y) ∂(W x) ∂p) − (∫ y, log(dq/d ref y) ∂q),
i.e. I = (−h(Y|X)) − (−h(Y)) = h(Y) − h(Y|X) once each integral is identified with the
relevant neg-entropy by integral_log_rnDeriv_self_eq_neg. Mirrors the 1-D body
mutualInfoOfChannel_toReal_eq_diffEntropy_sub. The hypothesis bundle is
all-regularity (AC chain hWx_q/hq_ref/h_joint_ac, measurable proxy g +
a.e.-identity hg_ae, compProd-level integrabilities, fibre/output integral-equality
h_fibre_self/h_out_self); the core (KL→llr→Bayes split→Fubini) lives in the body
via llr_compProd_prod_split_gen.
@audit:ok