InformationTheory

InformationTheory.Shannon.ChannelCoding.MIDecomp

source

Continuous-channel mutual-information chain rule (generic body) #

This file establishes the continuous-channel MI chain rule I(X;Y) = h(Y) − h(Y|X), with h(Y|X) realized as the integral of fibrewise differential entropies. The identity is not AWGN-specific: it holds for any Markov channel W : Channel ℝ ℝ and input law p.

This is the AWGN-independent generic core, living upstream of the AWGN converse chain (AWGN.Converse) so that the per-letter MI bridge (awgn_per_letter_mi_bridge_genuine) can reuse it without an import cycle. InformationTheory/Shannon/AWGN/ContChannelMIDecomp.lean imports this file and re-exports these declarations under their unchanged fully-qualified names (InformationTheory.Shannon.ChannelCoding.*), so downstream consumers (ParallelGaussian, AWGN/CapacityConverseMaxent.lean) are unaffected.

Approach #

The MI chain identity is a density-level identity opened from the klDiv definition of mutualInfoOfChannel:

I = ∫_z llr (p⊗ₘW) (p.prod q) z ∂(p⊗ₘW)          -- toReal_klDiv_of_measure_eq
  = ∫_z [log f_{Wx}(z.2) − log f_q(z.2)] ∂(p⊗ₘW)  -- Bayes density split  (★)
  = ∫_x ∫_y log f_{Wx}(y) ∂(W x) ∂p               -- integral_compProd
      − ∫_y log f_q(y) ∂q                          -- snd marginal of (p⊗ₘW)
  = −∫_x h(W x) ∂p + h(Y).

The KL→integral expansion, the Fubini split (integral_compProd), the output marginal identification (outputDistribution = (p⊗ₘW).snd) and the differential-entropy density form (differentialEntropy_eq_integral_density) assemble the chain. The single step (★) — the Bayes density split of the joint log-likelihood ratio into fibre/output log densities — is the conditional-rnDeriv-to-fibre identification provided by the linchpin rnDeriv_compProd_fibre (withDensity route), assembled by llr_compProd_prod_split.

theorem

InformationTheory.Shannon.ChannelCoding.integral_snd_outputDistribution

source

Marginal identification. For a bounded-density observable g : ℝ → ℝ, the joint integral of g ∘ snd against p ⊗ₘ W equals the integral of g against the output marginal outputDistribution p W = (p ⊗ₘ W).snd.

Used by
    theorem

    InformationTheory.Shannon.ChannelCoding.integral_log_rnDeriv_eq_neg_diffEntropy

    source

    General log-density entropy identification. For any μ : Measure ℝ with μ ≪ volume and measurable density f := μ.rnDeriv volume, the integral of log f against μ is −differentialEntropy μ. This is the generalization of integral_log_density_fibre to an arbitrary ≪ volume measure (the proof never used that μ was a channel fibre); it is reused for both the fibre term (μ := W x) and the output term (μ := outputDistribution p W) in the assembly below. @audit:ok

    Used by
      theorem

      InformationTheory.Shannon.ChannelCoding.integral_log_density_fibre

      source

      Fibre differential-entropy identification. For an ≪ volume fibre W x with measurable density f := (W x).rnDeriv volume, the inner integral of log f against W x is −differentialEntropy (W x). Thin wrapper around the general integral_log_rnDeriv_eq_neg_diffEntropy.

      Used by
        theorem

        InformationTheory.Shannon.ChannelCoding.integral_log_proxy_fibre

        source
        {W : Channel } [ProbabilityTheory.IsMarkovKernel W] (x : ) (hWx : (W x).AbsolutelyContinuous MeasureTheory.volume) {g : × ENNReal} (hg_ae : (fun (y : ) => (W x).rnDeriv MeasureTheory.volume y) =ᵐ[W x] fun (y : ) => g (x, y)) :

        Proxy form of the fibre differential-entropy identification (Route B). Same conclusion as integral_log_density_fibre, but stated with a measurable PDF proxy g in place of the (non-jointly-measurable) measure-form rnDeriv. The proxy↔rnDeriv bridge is absorbed inside the integral via integral_congr_ae fed by the per-fibre a.e. agreement hg_ae, so no joint measurability is ever needed. This is the step that lets the body keep its fibre term in proxy form.

        Used by
          theorem

          InformationTheory.Shannon.ChannelCoding.rnDeriv_compProd_fibre

          source

          Linchpin: fibre form of the compProd Radon-Nikodym derivative (Mathlib TODO, Composition/RadonNikodym.lean:28-29). For finite μ, κ, η with μ ⊗ₘ κ ≪ μ ⊗ₘ η, the conditional rnDeriv ∂(μ⊗ₘκ)/∂(μ⊗ₘη) is (μ⊗ₘη)-a.e. the fibrewise kernel rnDeriv Kernel.rnDeriv κ η p.1 p.2.

          Proof (withDensity route): μ⊗ₘκ ≪ μ⊗ₘη gives κ a ≪ η a a.e. (kernel_of_compProd), so κ =ᵐ[μ] η.withDensity (κ.rnDeriv η) (withDensity_rnDeriv_eq); hence μ⊗ₘκ = μ⊗ₘ(η.withDensity (κ.rnDeriv η)) = (μ⊗ₘη).withDensity (fun p ↦ κ.rnDeriv η p.1 p.2) (compProd_congr + compProd_withDensity); finish with Measure.rnDeriv_withDensity.

          Used by
            theorem

            InformationTheory.Shannon.ChannelCoding.log_rnDeriv_split

            source

            Per-measure log-density split (Bayes step). For ν ≪ q ≪ volume all σ-finite, the log of the relative density dν/dq splits as the difference of the two volume-log-densities: log (dν/dq y) = log (dν/dvol y) − log (dq/dvol y), ν-a.e. Built from the rnDeriv chain rule dν/dq · dq/dvol =ᵐ dν/dvol plus positivity (q ≪ voldq/dvol > 0 on ν since ν ≪ q).

            Used by
              theorem

              InformationTheory.Shannon.ChannelCoding.llr_compProd_prod_split

              source
              {p : MeasureTheory.Measure } [MeasureTheory.IsProbabilityMeasure p] {W : Channel } [ProbabilityTheory.IsMarkovKernel W] (q : MeasureTheory.Measure ) [MeasureTheory.IsProbabilityMeasure q] (hWx_q : ∀ (x : ), (W x).AbsolutelyContinuous q) (hq_vol : q.AbsolutelyContinuous MeasureTheory.volume) (h_joint_ac : (p.compProd W).AbsolutelyContinuous (p.prod q)) (g : × ENNReal) (hg_meas : Measurable g) (hg_ae : ∀ (x : ), (fun (y : ) => (W x).rnDeriv MeasureTheory.volume y) =ᵐ[W x] fun (y : ) => g (x, y)) :

              ★ General Bayes density split of the joint llr (modulo named ac hyps). For input law p, Markov channel W, output q := outputDistribution p W, with each fibre ≪ q ≪ volume and joint ≪ p.prod q, the log-likelihood ratio of the joint against the product factorizes into fibre/output log-densities. This is the body of the residual hypothesis h_llr_split. Combines the linchpin rnDeriv_compProd_fibre (at η := Kernel.const ℝ q) with rnDeriv_eq_rnDeriv_measure and the per-fibre log_rnDeriv_split.

              Used by
                theorem

                InformationTheory.Shannon.ChannelCoding.mutualInfoOfChannel_toReal_eq_diffEntropy_sub

                source

                ★ Continuous-channel MI chain rule body (AWGN-independent).

                (mutualInfoOfChannel p W).toReal = h(Y) − ∫ h(Y|X=x) dp(x), the density-level analogue of the discrete mutualInfo_eq_entropy_add_entropy_sub_jointEntropy.

                Assembled from the local helpers. The proof opens mutualInfoOfChannel = klDiv (p⊗ₘW) (p.prod q) (q := outputDistribution p W) via toReal_klDiv_of_measure_eq (both factors are probability measures, so the univ-mass condition is automatic), rewrites the joint log-likelihood ratio by the Bayes density split llr_compProd_prod_split, splits the resulting integral with integral_sub, identifies the fibre term with integral_compProd + integral_log_proxy_fibre (each fibre ↦ −h(W x)), and identifies the output term with integral_snd_outputDistribution + integral_log_rnDeriv_eq_neg_diffEntropy (↦ −h(q)). @audit:ok

                Used by

                  Generic (output type β + reference measure ref) MI chain rule #

                  The 1-D body mutualInfoOfChannel_toReal_eq_diffEntropy_sub above is specialized to Channel ℝ ℝ with the Lebesgue reference volume : Measure ℝ. The block AWGN converse needs the n-dimensional output form (β := Fin n → ℝ, ref := volume), so we re-derive the same chain identity for an arbitrary input type α, output type β, and reference measure ref : Measure β, stated in log-density-integral form (∫ log (rnDeriv · ref)) rather than differentialEntropy, so the consumer is free to identify each integral with whatever entropy notion it uses (jointDifferentialEntropyPi for the AWGN block). The proof mirrors the 1-D one step for step; only the helper lemmas are re-stated generically.

                  theorem

                  InformationTheory.Shannon.ChannelCoding.integral_snd_outputDistribution_gen

                  source
                  {α : Type u_1} { : MeasurableSpace α} {β : Type u_2} { : MeasurableSpace β} {p : MeasureTheory.Measure α} {W : Channel α β} (g : β) (hg : MeasureTheory.Integrable g (outputDistribution p W)) :
                  (z : α × β), g z.2 p.compProd W = (y : β), g y outputDistribution p W

                  Generic marginal identification. For a ref-integrable observable g : β → ℝ, the joint integral of g ∘ snd against p ⊗ₘ W equals the integral of g against the output marginal outputDistribution p W = (p ⊗ₘ W).snd. Generic in α, β.

                  Used by
                    theorem

                    InformationTheory.Shannon.ChannelCoding.log_rnDeriv_split_gen

                    source

                    Generic per-measure log-density split (Bayes step). For ν ≪ q ≪ ref all σ-finite (with ν.HaveLebesgueDecomposition q, q.HaveLebesgueDecomposition ref), log (dν/dq y) = log (dν/dref y) − log (dq/dref y), ν-a.e. Generic reference ref.

                    Used by
                      theorem

                      InformationTheory.Shannon.ChannelCoding.llr_compProd_prod_split_gen

                      source
                      {α : Type u_1} { : MeasurableSpace α} {β : Type u_2} { : MeasurableSpace β} [MeasurableSpace.CountableOrCountablyGenerated α β] {p : MeasureTheory.Measure α} [MeasureTheory.IsProbabilityMeasure p] {W : Channel α β} [ProbabilityTheory.IsMarkovKernel W] (q ref : MeasureTheory.Measure β) [MeasureTheory.IsProbabilityMeasure q] [MeasureTheory.SigmaFinite ref] [q.HaveLebesgueDecomposition ref] (hWx_q : ∀ (x : α), (W x).AbsolutelyContinuous q) (hq_ref : q.AbsolutelyContinuous ref) (h_joint_ac : (p.compProd W).AbsolutelyContinuous (p.prod q)) (g : α × βENNReal) (hg_meas : Measurable g) (hg_ae : ∀ (x : α), (fun (y : β) => (W x).rnDeriv ref y) =ᵐ[W x] fun (y : β) => g (x, y)) :
                      (fun (z : α × β) => MeasureTheory.llr (p.compProd W) (p.prod q) z) =ᵐ[p.compProd W] fun (z : α × β) => Real.log (g z).toReal - Real.log (q.rnDeriv ref z.2).toReal

                      ★ Generic Bayes density split of the joint llr. For input law p, Markov channel W : Channel α β, output q := outputDistribution p W, with each fibre ≪ q ≪ ref and joint ≪ p.prod q, the log-likelihood ratio of the joint against the product factorizes into fibre/output log-densities. Generic in α, β, ref.

                      Used by
                        theorem

                        InformationTheory.Shannon.ChannelCoding.mutualInfoOfChannel_toReal_eq_log_density_sub

                        source
                        {α : Type u_1} { : MeasurableSpace α} {β : Type u_2} { : MeasurableSpace β} [MeasurableSpace.CountableOrCountablyGenerated α β] {p : MeasureTheory.Measure α} [MeasureTheory.IsProbabilityMeasure p] {W : Channel α β} [ProbabilityTheory.IsMarkovKernel W] (ref : MeasureTheory.Measure β) [MeasureTheory.SigmaFinite ref] [(outputDistribution p W).HaveLebesgueDecomposition ref] (hWx_q : ∀ (x : α), (W x).AbsolutelyContinuous (outputDistribution p W)) (hq_ref : (outputDistribution p W).AbsolutelyContinuous ref) (h_joint_ac : (p.compProd W).AbsolutelyContinuous (p.prod (outputDistribution p W))) (g : α × βENNReal) (hg_meas : Measurable g) (hg_ae : ∀ (x : α), (fun (y : β) => (W x).rnDeriv ref y) =ᵐ[W x] fun (y : β) => g (x, y)) (h_int_fibre : MeasureTheory.Integrable (fun (z : α × β) => Real.log (g z).toReal) (p.compProd W)) (h_int_out : MeasureTheory.Integrable (fun (z : α × β) => Real.log ((outputDistribution p W).rnDeriv ref z.2).toReal) (p.compProd W)) (h_fibre_self : ∀ (x : α), (y : β), Real.log (g (x, y)).toReal W x = (y : β), Real.log ((W x).rnDeriv ref y).toReal W x) (h_out_self : MeasureTheory.Integrable (fun (y : β) => Real.log ((outputDistribution p W).rnDeriv ref y).toReal) (outputDistribution p W)) :
                        (mutualInfoOfChannel p W).toReal = (x : α), (y : β), Real.log ((W x).rnDeriv ref y).toReal W x p - (y : β), Real.log ((outputDistribution p W).rnDeriv ref y).toReal outputDistribution p W

                        ★ Generic continuous-channel MI chain rule body (output type β, reference ref), in log-density-integral form:

                        I.toReal = (∫ x, ∫ y, log(d(W x)/d ref y) ∂(W x) ∂p) − (∫ y, log(dq/d ref y) ∂q),

                        i.e. I = (−h(Y|X)) − (−h(Y)) = h(Y) − h(Y|X) once each integral is identified with the relevant neg-entropy by integral_log_rnDeriv_self_eq_neg. Mirrors the 1-D body mutualInfoOfChannel_toReal_eq_diffEntropy_sub. The hypothesis bundle is all-regularity (AC chain hWx_q/hq_ref/h_joint_ac, measurable proxy g + a.e.-identity hg_ae, compProd-level integrabilities, fibre/output integral-equality h_fibre_self/h_out_self); the core (KL→llr→Bayes split→Fubini) lives in the body via llr_compProd_prod_split_gen. @audit:ok

                        Used by