InformationTheory

InformationTheory.Shannon.ChannelCoding.CodeToAmbient

source

From a block code to its ambient law #

Model-independent infrastructure for the operational converses: a block code together with a message law and a per-letter product channel determines an ambient probability measure on message × output block, and the structural hypotheses of the single-letter converses (Markov factorization, memorylessness, per-letter joint laws) are read off that measure. None of the statements below mention a particular channel model — the encoder enters only through the factorization hypotheses κ m = Wcode (g m) and κ m = ∏ⱼ W (x m j) — so a multi-user converse instantiates them by supplying its own encoder and channel.

Main statements #

  • isMarkovChain_of_compProd_encoder: an ambient ν ⊗ₘ κ whose kernel factors through a deterministic encoder g carries the Markov chain M → g M → Y.
  • isMarkovChain_of_compProd_pi: for a per-letter product kernel ∏ⱼ W (x m j), the output letter Yᵢ is conditionally independent of any variable that does not read the i-th output coordinate, given the input letter Xᵢ. The conditioner is the input letter itself; the variable decoupled from Yᵢ is an arbitrary measurable function of the whole ambient point, constrained only by invariance under re-randomizing coordinate i, so it may read the message as well as the other letters.
  • isMemorylessChannel_of_compProd_pi: if the kernel is the per-letter product ∏ⱼ W (x m j) of a channel applied to a deterministic codeword, the ambient is a memoryless channel.
  • compProd_pi_map_pair_eq_of_update_invariant: pair the i-th output letter with any map G that is invariant under updating the i-th output coordinate and that retracts onto the input letter; the joint law of that pair is ((ν ⊗ₘ κ).map G) ⊗ₘ W.comap g, the composition product of G's own law with the channel read along the retraction. The invariance is what lets a padded auxiliary variable sit in the first component.
  • compProd_pi_map_pair_eq: its special case where G is the input letter x · i itself, so the joint law of the i-th input-output pair is the channel joint (ν.map fun m ↦ x m i) ⊗ₘ W.
  • compProd_comap_map_prodMap: a composition product with a comapped kernel is the composition product of the pushed-forward measure with the kernel itself.
  • compProd_map_prodMap: the two-sided form of the previous item — a composition product transported by a pair of maps, one on the base and one on the fiber, provided the kernel's dependence on the base factors through the base map.
  • pi_map_comp_of_injective: a finite product measure reindexed along an injection is the product measure of the reindexed family.
  • piBlockKernel, pi_map_unzip_eq_compProd: a product of laws each of which is its own first marginal followed by a kernel Q, read as a pair of blocks, is the composition product of the product of the first marginals with the blockwise product of Q.
  • mutualInfo_map_comp, condDistrib_map_comp, condMutualInfo_map_comp: information quantities and conditional distributions are invariant under a shared pushforward of all their arguments.
  • le_log_of_ceil_exp_le: ⌈exp x⌉₊ ≤ M implies x ≤ log M, turning a message count into a rate bound.
  • le_toReal_of_inv_mul_le: an averaged bound m⁻¹ · S ≤ J in ℝ≥0∞ together with m · r ≤ S.toReal gives r ≤ J.toReal, which is how a per-letter bound becomes a rate bound.

The uniform message law #

instance

InformationTheory.Shannon.uniformCount_isProbabilityMeasure

source

The uniform probability law (card X)⁻¹ • count on a nonempty finite type.

Used by

    Markov factorization of the ambient #

    theorem

    InformationTheory.Shannon.isMarkovChain_of_compProd_encoder

    source
    {M : Type u_1} {Z : Type u_2} {Y : Type u_3} [MeasurableSpace M] [StandardBorelSpace M] [Nonempty M] [MeasurableSpace Z] [MeasurableSpace Y] [StandardBorelSpace Y] [Nonempty Y] (ν : MeasureTheory.Measure M) [MeasureTheory.IsProbabilityMeasure ν] (g : MZ) (hg : Measurable g) (κ : ProbabilityTheory.Kernel M Y) [ProbabilityTheory.IsMarkovKernel κ] (Wcode : ProbabilityTheory.Kernel Z Y) [ProbabilityTheory.IsMarkovKernel Wcode] ( : ∀ (m : M), κ m = Wcode (g m)) :
    IsMarkovChain (ν.compProd κ) Prod.fst (fun (ω : M × Y) => g ω.1) Prod.snd

    Abstract Markov-chain factorization M → g M → Y for an ambient ν ⊗ₘ κ in which the message-to-output kernel κ factors through a deterministic encoder g : M → Z and a codeword kernel Wcode : Z → Y (i.e. κ m = Wcode (g m)). This is the general shape behind the message-to-codeword-to-output chain of a multi-user converse; it needs no product/pi structure, only the factorization . @audit:ok

    Used by

      Marginalizing and reindexing a product measure #

      theorem

      InformationTheory.Shannon.lintegral_pi_reRandomize

      source
      {γ : Type u_1} [MeasurableSpace γ] {k : } (ζ : Fin kMeasureTheory.Measure γ) [∀ (j : Fin k), MeasureTheory.IsProbabilityMeasure (ζ j)] (i : Fin k) (F : (Fin kγ)ENNReal) (hF : Measurable F) :
      ∫⁻ (y : Fin kγ), F y MeasureTheory.Measure.pi ζ = ∫⁻ (y : Fin kγ), ∫⁻ (b : γ), F (Function.update y i b) ζ i MeasureTheory.Measure.pi ζ

      Re-randomizing a single coordinate of a product of probability measures leaves the Measure.pi-integral unchanged. Used to peel the i-th output letter off the block channel ∏ⱼ W (xⱼ) in the memoryless-channel derivation. @audit:ok

      Used by
        theorem

        InformationTheory.Shannon.lintegral_pi_eval

        source
        {γ : Type u_1} [MeasurableSpace γ] {k : } (ζ : Fin kMeasureTheory.Measure γ) [∀ (j : Fin k), MeasureTheory.IsProbabilityMeasure (ζ j)] (i : Fin k) (g : γENNReal) (hg : Measurable g) :
        ∫⁻ (y : Fin kγ), g (y i) MeasureTheory.Measure.pi ζ = ∫⁻ (b : γ), g b ζ i

        Marginalization of a product of probability measures at a single coordinate. @audit:ok

        Used by
          theorem

          InformationTheory.Shannon.pi_map_comp_of_injective

          source
          {γ : Type u_1} [MeasurableSpace γ] {k m : } (ν : Fin mMeasureTheory.Measure γ) [∀ (j : Fin m), MeasureTheory.IsProbabilityMeasure (ν j)] (e : Fin kFin m) (he : Function.Injective e) :
          MeasureTheory.Measure.map (fun (y : Fin mγ) (j : Fin k) => y (e j)) (MeasureTheory.Measure.pi ν) = MeasureTheory.Measure.pi fun (j : Fin k) => ν (e j)
          Used by

            Per-letter conditional independence, memorylessness and the joint law #

            theorem

            InformationTheory.Shannon.isMarkovChain_of_compProd_pi

            source
            {M : Type u_1} {A : Type u_2} {B : Type u_3} {C : Type u_4} [MeasurableSpace M] [StandardBorelSpace M] [Nonempty M] [MeasurableSpace A] [StandardBorelSpace A] [Nonempty A] [MeasurableSpace B] [StandardBorelSpace B] [Nonempty B] [MeasurableSpace C] [StandardBorelSpace C] [Nonempty C] {k : } (ν : MeasureTheory.Measure M) [MeasureTheory.IsProbabilityMeasure ν] (x : MFin kA) (hx : Measurable x) (W : ProbabilityTheory.Kernel A B) [ProbabilityTheory.IsMarkovKernel W] (κ : ProbabilityTheory.Kernel M (Fin kB)) [ProbabilityTheory.IsMarkovKernel κ] ( : ∀ (m : M), κ m = MeasureTheory.Measure.pi fun (j : Fin k) => W (x m j)) (i : Fin k) (F : M × (Fin kB)C) (hF : Measurable F) (hFupd : ∀ (m : M) (y : Fin kB) (b : B), F (m, Function.update y i b) = F (m, y)) :
            IsMarkovChain (ν.compProd κ) F (fun (ω : M × (Fin kB)) => x ω.1 i) fun (ω : M × (Fin kB)) => ω.2 i

            Per-letter conditional independence for a product-channel ambient: if the message-to-output kernel factors as the per-letter product κ m = ∏ⱼ W (x m j), then at every letter i the output ω.2 i is conditionally independent of F given the input letter x ω.1 i, for any measurable F that does not read output coordinate i. Not reading that coordinate is what hFupd says — re-randomizing it leaves F unchanged — so F may read the message, hence every input letter, together with all the other output letters. @audit:ok

            Used by
              theorem

              InformationTheory.Shannon.isMemorylessChannel_of_compProd_pi

              source
              {M : Type u_1} {A : Type u_2} {B : Type u_3} [MeasurableSpace M] [StandardBorelSpace M] [Nonempty M] [MeasurableSpace A] [StandardBorelSpace A] [Nonempty A] [MeasurableSpace B] [StandardBorelSpace B] [Nonempty B] {k : } (ν : MeasureTheory.Measure M) [MeasureTheory.IsProbabilityMeasure ν] (x : MFin kA) (hx : Measurable x) (W : ProbabilityTheory.Kernel A B) [ProbabilityTheory.IsMarkovKernel W] (κ : ProbabilityTheory.Kernel M (Fin kB)) [ProbabilityTheory.IsMarkovKernel κ] ( : ∀ (m : M), κ m = MeasureTheory.Measure.pi fun (j : Fin k) => W (x m j)) :
              ChannelCodingConverseGeneral.IsMemorylessChannel (ν.compProd κ) (fun (i : Fin k) (ω : M × (Fin kB)) => x ω.1 i) fun (i : Fin k) (ω : M × (Fin kB)) => ω.2 i

              A product-channel ambient is a memoryless channel: if the message-to-output kernel factors as the per-letter product κ m = ∏ⱼ W (x m j) of a channel W applied to a deterministic codeword x m, then ν ⊗ₘ κ is a memoryless channel with per-letter inputs x ω.1 i and per-letter outputs ω.2 i. This is isMarkovChain_of_compProd_pi read at the conditioner that collects every input and output letter other than the i-th. @audit:ok

              Used by
                theorem

                InformationTheory.Shannon.compProd_comap_map_prodMap

                source
                {A : Type u_1} {A' : Type u_2} {B : Type u_3} [MeasurableSpace A] [MeasurableSpace A'] [MeasurableSpace B] (μ : MeasureTheory.Measure A) [MeasureTheory.SFinite μ] (κ : ProbabilityTheory.Kernel A' B) [ProbabilityTheory.IsMarkovKernel κ] {g : AA'} (hg : Measurable g) :
                MeasureTheory.Measure.map (fun (z : A × B) => (g z.1, z.2)) (μ.compProd (κ.comap g hg)) = (MeasureTheory.Measure.map g μ).compProd κ

                @audit:ok

                Used by
                  theorem

                  InformationTheory.Shannon.compProd_map_prodMap

                  source
                  {Z : Type u_1} {Z' : Type u_2} {B : Type u_3} {B' : Type u_4} [MeasurableSpace Z] [MeasurableSpace Z'] [MeasurableSpace B] [MeasurableSpace B'] (ρ : MeasureTheory.Measure Z) [MeasureTheory.SFinite ρ] (κ : ProbabilityTheory.Kernel Z B) [ProbabilityTheory.IsMarkovKernel κ] {f : ZZ'} (hf : Measurable f) {g : BB'} (hg : Measurable g) (κ' : ProbabilityTheory.Kernel Z' B') [ProbabilityTheory.IsMarkovKernel κ'] ( : κ.map g = κ'.comap f hf) :
                  Used by
                    theorem

                    InformationTheory.Shannon.compProd_pi_map_pair_eq_of_update_invariant

                    source
                    {M : Type u_1} {A : Type u_2} {B : Type u_3} {C : Type u_4} [MeasurableSpace M] [MeasurableSpace A] [MeasurableSpace B] [MeasurableSpace C] {k : } (ν : MeasureTheory.Measure M) [MeasureTheory.IsProbabilityMeasure ν] (x : MFin kA) (W : ProbabilityTheory.Kernel A B) [ProbabilityTheory.IsMarkovKernel W] (κ : ProbabilityTheory.Kernel M (Fin kB)) [ProbabilityTheory.IsMarkovKernel κ] ( : ∀ (m : M), κ m = MeasureTheory.Measure.pi fun (j : Fin k) => W (x m j)) (i : Fin k) (G : M × (Fin kB)C) (hG : Measurable G) (hGupd : ∀ (m : M) (y : Fin kB) (b : B), G (m, Function.update y i b) = G (m, y)) (g : CA) (hg : Measurable g) (hgG : ∀ (ω : M × (Fin kB)), g (G ω) = x ω.1 i) :
                    MeasureTheory.Measure.map (fun (ω : M × (Fin kB)) => (G ω, ω.2 i)) (ν.compProd κ) = (MeasureTheory.Measure.map G (ν.compProd κ)).compProd (W.comap g hg)

                    @audit:ok

                    Used by
                      theorem

                      InformationTheory.Shannon.compProd_pi_map_pair_eq

                      source
                      {M : Type u_1} {A : Type u_2} {B : Type u_3} [MeasurableSpace M] [MeasurableSpace A] [MeasurableSpace B] {k : } (ν : MeasureTheory.Measure M) [MeasureTheory.IsProbabilityMeasure ν] (x : MFin kA) (hx : Measurable x) (W : ProbabilityTheory.Kernel A B) [ProbabilityTheory.IsMarkovKernel W] (κ : ProbabilityTheory.Kernel M (Fin kB)) [ProbabilityTheory.IsMarkovKernel κ] ( : ∀ (m : M), κ m = MeasureTheory.Measure.pi fun (j : Fin k) => W (x m j)) (i : Fin k) :
                      MeasureTheory.Measure.map (fun (ω : M × (Fin kB)) => (x ω.1 i, ω.2 i)) (ν.compProd κ) = (MeasureTheory.Measure.map (fun (m : M) => x m i) ν).compProd W

                      Per-letter joint pushforward of a product-channel compProd: for an ambient ν ⊗ₘ κ whose message-to-output kernel factors as the per-letter product κ m = ∏ⱼ W (x m j), the joint law of the i-th input-output pair (x ω.1 i, ω.2 i) is the channel joint (ν.map fun m ↦ x m i) ⊗ₘ W. Stated separately from isMarkovChain_of_compProd_pi, which establishes the same identity internally, because it is what identifies a per-letter information quantity of the ambient with the corresponding channel quantity.

                      Used by

                        Unzipping a product of two-stage laws #

                        theorem

                        InformationTheory.Shannon.measure_singleton_eq_mul_of_append

                        source
                        Used by
                          noncomputable def

                          InformationTheory.Shannon.piBlockKernel

                          source

                          The blockwise product u ↦ ∏ⱼ Q (u j) of a kernel Q out of a finite alphabet.

                          Mathlib has no Kernel.pi; over a countable alphabet with measurable singletons the family of product measures is a kernel for free, which is what Kernel.ofFunOfCountable supplies here.

                          Equations
                          Instances For
                            Used by
                              instance

                              InformationTheory.Shannon.piBlockKernel_isMarkovKernel

                              source
                              Used by
                                theorem

                                InformationTheory.Shannon.pi_map_unzip_eq_compProd

                                source
                                {A : Type u_1} {B : Type u_2} [Fintype A] [MeasurableSpace A] [MeasurableSingletonClass A] [Fintype B] [MeasurableSpace B] [MeasurableSingletonClass B] {k : } (Q : ProbabilityTheory.Kernel A B) [ProbabilityTheory.IsMarkovKernel Q] (ρ : Fin kMeasureTheory.Measure (A × B)) [∀ (j : Fin k), MeasureTheory.IsProbabilityMeasure (ρ j)] ( : ∀ (j : Fin k), ρ j = (MeasureTheory.Measure.map Prod.fst (ρ j)).bind fun (a : A) => MeasureTheory.Measure.map (fun (b : B) => (a, b)) (Q a)) :
                                MeasureTheory.Measure.map (fun (y : Fin kA × B) => (fun (j : Fin k) => (y j).1, fun (j : Fin k) => (y j).2)) (MeasureTheory.Measure.pi ρ) = (MeasureTheory.Measure.pi fun (j : Fin k) => MeasureTheory.Measure.map Prod.fst (ρ j)).compProd (piBlockKernel Q)
                                Used by

                                  Information transport under a shared pushforward #

                                  theorem

                                  InformationTheory.Shannon.mutualInfo_map_comp

                                  source
                                  {Ω : Type u_1} {Ω' : Type u_2} {A : Type u_3} {B : Type u_4} [MeasurableSpace Ω] [MeasurableSpace Ω'] [MeasurableSpace A] [MeasurableSpace B] (μ : MeasureTheory.Measure Ω) (T : ΩΩ') (hT : Measurable T) (f : Ω'A) (hf : Measurable f) (g : Ω'B) (hg : Measurable g) :
                                  mutualInfo (MeasureTheory.Measure.map T μ) f g = mutualInfo μ (fun (ω : Ω) => f (T ω)) fun (ω : Ω) => g (T ω)

                                  Mutual information is invariant under a shared pushforward of both random variables: I(f; g) = I(f ∘ T; g ∘ T) when the pair law on μ.map T matches the pair law of the composed variables on μ.

                                  Used by
                                    theorem

                                    InformationTheory.Shannon.condDistrib_map_comp

                                    source
                                    {Ω : Type u_1} {Ω' : Type u_2} {A : Type u_3} {C : Type u_4} [MeasurableSpace Ω] [MeasurableSpace Ω'] [MeasurableSpace A] [StandardBorelSpace A] [Nonempty A] [MeasurableSpace C] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (T : ΩΩ') (hT : Measurable T) (f : Ω'A) (hf : Measurable f) (h : Ω'C) (hh : Measurable h) :
                                    (ProbabilityTheory.condDistrib f h (MeasureTheory.Measure.map T μ)) =ᵐ[MeasureTheory.Measure.map h (MeasureTheory.Measure.map T μ)] (ProbabilityTheory.condDistrib (fun (ω : Ω) => f (T ω)) (fun (ω : Ω) => h (T ω)) μ)

                                    condDistrib is stable under a shared pushforward of the conditioning and conditioned variables: condDistrib f h (μ.map T) =ᵐ condDistrib (f ∘ T) (h ∘ T) μ on the conditioning marginal (μ.map T).map h.

                                    Used by
                                      theorem

                                      InformationTheory.Shannon.condMutualInfo_map_comp

                                      source
                                      {Ω : Type u_1} {Ω' : Type u_2} {A : Type u_3} {B : Type u_4} {C : Type u_5} [MeasurableSpace Ω] [MeasurableSpace Ω'] [MeasurableSpace A] [StandardBorelSpace A] [Nonempty A] [MeasurableSpace B] [StandardBorelSpace B] [Nonempty B] [MeasurableSpace C] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (T : ΩΩ') (hT : Measurable T) (f : Ω'A) (hf : Measurable f) (g : Ω'B) (hg : Measurable g) (h : Ω'C) (hh : Measurable h) :
                                      condMutualInfo (MeasureTheory.Measure.map T μ) f g h = condMutualInfo μ (fun (ω : Ω) => f (T ω)) (fun (ω : Ω) => g (T ω)) fun (ω : Ω) => h (T ω)

                                      Conditional mutual information is invariant under a shared pushforward of all three random variables: I(f; g | h) = I(f ∘ T; g ∘ T | h ∘ T).

                                      Used by
                                        theorem

                                        InformationTheory.Shannon.isMarkovChain_map_comp

                                        source
                                        {Ω : Type u_1} {Ω' : Type u_2} {A : Type u_3} {B : Type u_4} {C : Type u_5} [MeasurableSpace Ω] [MeasurableSpace Ω'] [MeasurableSpace A] [StandardBorelSpace A] [Nonempty A] [MeasurableSpace B] [StandardBorelSpace B] [Nonempty B] [MeasurableSpace C] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (T : ΩΩ') (hT : Measurable T) (ρ : MeasureTheory.Measure Ω') [MeasureTheory.IsFiniteMeasure ρ] ( : ρ = MeasureTheory.Measure.map T μ) (f : Ω'A) (hf : Measurable f) (g : Ω'C) (hg : Measurable g) (h : Ω'B) (hh : Measurable h) (hchain : IsMarkovChain μ (fun (ω : Ω) => f (T ω)) (fun (ω : Ω) => g (T ω)) fun (ω : Ω) => h (T ω)) :
                                        IsMarkovChain ρ f g h
                                        Used by
                                          theorem

                                          InformationTheory.Shannon.condMutualInfo_map_comp'

                                          source
                                          {Ω : Type u_1} {Ω' : Type u_2} {A : Type u_3} {B : Type u_4} {C : Type u_5} [MeasurableSpace Ω] [MeasurableSpace Ω'] [MeasurableSpace A] [StandardBorelSpace A] [Nonempty A] [MeasurableSpace B] [StandardBorelSpace B] [Nonempty B] [MeasurableSpace C] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] (T : ΩΩ') (hT : Measurable T) (ρ : MeasureTheory.Measure Ω') [MeasureTheory.IsFiniteMeasure ρ] ( : ρ = MeasureTheory.Measure.map T μ) (f : Ω'A) (hf : Measurable f) (g : Ω'B) (hg : Measurable g) (h : Ω'C) (hh : Measurable h) :
                                          condMutualInfo ρ f g h = condMutualInfo μ (fun (ω : Ω) => f (T ω)) (fun (ω : Ω) => g (T ω)) fun (ω : Ω) => h (T ω)

                                          condMutualInfo_map_comp phrased against any measure ρ propositionally equal to μ.map T. The equation hypothesis is substituted (transporting its IsFiniteMeasure instance), which sidesteps the ill-typed motive of rewriting the measure argument of condMutualInfo directly.

                                          Used by

                                            From a message count to a rate #

                                            theorem

                                            InformationTheory.Shannon.le_log_of_ceil_exp_le

                                            source
                                            {x : } {M : } (hM : Real.exp x⌉₊ M) :
                                            x Real.log M

                                            If ⌈exp x⌉₊ ≤ M then x ≤ log M, converting a message count into a rate bound. From exp x ≤ ⌈exp x⌉₊ ≤ M, taking logs (both sides positive) gives x = log (exp x) ≤ log M.

                                            Used by

                                              From an averaged bound to a real inequality #

                                              theorem

                                              InformationTheory.Shannon.le_toReal_of_inv_mul_le

                                              source
                                              {S J : ENNReal} {m : } (hm : 0 < m) (hSJ : (↑m)⁻¹ * S J) (hJ : J ) {r : } (hr : m * r S.toReal) :
                                              Used by