InformationTheory

InformationTheory.Shannon.MultivariateDiffEntropy

source

Multivariate differential entropy and subadditivity #

Common foundation for AWGN / Parallel-Gaussian output-entropy upper bounds.

Main definitions #

Main statements #

Implementation notes #

Subadditivity follows from KL ≥ 0 + the bridge (klDiv(joint ‖ ∏ marginals)).toReal = ∑ h(marginalᵢ) − h(joint). The Bayes density split is established via Mathlib's prod_withDensity₀ + rnDeriv_mul_rnDeriv. The 2-variable *_of_llr_split variants, which take that split as an explicit hypothesis instead, are retained for backward compatibility and carry @audit:superseded-by(...).

pi_withDensity (joint density = ∏ marginal densities on Fin n → ℝ) is absent from Mathlib, so it is built in-tree as pi_withDensity_fin by measurePreserving_piFinSuccAbove induction. The generic withDensity_map_equiv (change-of-variables under a measurable equivalence) is also absent in Mathlib's non-rnDeriv form and is supplied here.

Definitions (Mathlib-shape-driven, mirror the 1-D differentialEntropy) #

noncomputable def

InformationTheory.Shannon.jointDifferentialEntropy

source

The 2-variable joint differential entropy. Defined -∫ negMulLog (dμ/dvol) on Measure (ℝ × ℝ), identical in shape to the 1-D differentialEntropy, so the existing 1-D density lemmas apply through volume_eq_prod (which holds by rfl).

Equations
Instances For
    Used by
      noncomputable def

      InformationTheory.Shannon.jointDifferentialEntropyPi

      source
      {n : } (μ : MeasureTheory.Measure (Fin n)) :

      The n-variable joint differential entropy on Measure (Fin n → ℝ) (the parallel-Gaussian consumer form). Fin n → ℝ is chosen over EuclideanSpace so that the product-Lebesgue API (volume_pi, Measure.pi) applies directly.

      Equations
      Instances For
        Used by

          Reusable core: ∫ log(dμ/dν) ∂μ = -∫ negMulLog(dμ/dν) ∂ν #

          theorem

          InformationTheory.Shannon.integral_log_rnDeriv_self_eq_neg

          source

          For μ ≪ ν, ∫ x, log((μ.rnDeriv ν x).toReal) ∂μ = -∫ x, negMulLog((μ.rnDeriv ν x).toReal) ∂ν.

          The RHS is the (joint/1-D) differential entropy when ν is the relevant Lebesgue measure.

          Used by

            Generic withDensity change-of-variables under a measurable equivalence #

            theorem

            InformationTheory.Shannon.withDensity_map_equiv

            source
            {α : Type u_1} {β : Type u_2} [MeasurableSpace α] [MeasurableSpace β] {μ : MeasureTheory.Measure α} (e : α ≃ᵐ β) {g : αENNReal} (hg : Measurable g) :

            A generic withDensity_map (Mathlib absent, rnDeriv-version de-specialized). Pushforward of a withDensity measure along a measurable equivalence e: (μ.withDensity g).map e = (μ.map e).withDensity (g ∘ e.symm). Mathlib only ships the rnDeriv-specialized MeasurableEmbedding.map_withDensity_rnDeriv; the generic form below de-specializes its 5-line proof, replacing the final rnDeriv_map congruence by the trivial e.symm_apply_apply cancellation. @audit:ok

            Used by

              2-variable bridge + subadditivity #

              theorem

              InformationTheory.Shannon.klDiv_prod_marginals_toReal_eq_sum_sub_joint_of_llr_split

              source
              {μ : MeasureTheory.Measure ( × )} [MeasureTheory.IsProbabilityMeasure μ] (h_fst_ac : (MeasureTheory.Measure.map Prod.fst μ).AbsolutelyContinuous MeasureTheory.volume) (h_snd_ac : (MeasureTheory.Measure.map Prod.snd μ).AbsolutelyContinuous MeasureTheory.volume) (h_joint_ac : μ.AbsolutelyContinuous ((MeasureTheory.Measure.map Prod.fst μ).prod (MeasureTheory.Measure.map Prod.snd μ))) (h_llr_split : (fun (z : × ) => MeasureTheory.llr μ ((MeasureTheory.Measure.map Prod.fst μ).prod (MeasureTheory.Measure.map Prod.snd μ)) z) =ᵐ[μ] fun (z : × ) => Real.log (μ.rnDeriv MeasureTheory.volume z).toReal - Real.log ((MeasureTheory.Measure.map Prod.fst μ).rnDeriv MeasureTheory.volume z.1).toReal - Real.log ((MeasureTheory.Measure.map Prod.snd μ).rnDeriv MeasureTheory.volume z.2).toReal) (h_int_fst : MeasureTheory.Integrable (fun (z : × ) => Real.log ((MeasureTheory.Measure.map Prod.fst μ).rnDeriv MeasureTheory.volume z.1).toReal) μ) (h_int_snd : MeasureTheory.Integrable (fun (z : × ) => Real.log ((MeasureTheory.Measure.map Prod.snd μ).rnDeriv MeasureTheory.volume z.2).toReal) μ) (h_int_joint : MeasureTheory.Integrable (fun (z : × ) => Real.log (μ.rnDeriv MeasureTheory.volume z).toReal) μ) (h_int_fst_marg : MeasureTheory.Integrable (fun (x : ) => Real.log ((MeasureTheory.Measure.map Prod.fst μ).rnDeriv MeasureTheory.volume x).toReal) (MeasureTheory.Measure.map Prod.fst μ)) (h_int_snd_marg : MeasureTheory.Integrable (fun (y : ) => Real.log ((MeasureTheory.Measure.map Prod.snd μ).rnDeriv MeasureTheory.volume y).toReal) (MeasureTheory.Measure.map Prod.snd μ)) :

              2-variable subadditivity bridge: (klDiv(joint ‖ μ_X ⊗ μ_Y)).toReal = h(μ_X) + h(μ_Y) − h(joint).

              Hypotheses: absolute continuity + Bayes llr split h_llr_split + integrability. Superseded by klDiv_prod_marginals_toReal_eq_sum_sub_joint, which internalizes the split.

              @audit:superseded-by(klDiv_prod_marginals_toReal_eq_sum_sub_joint)

              Used by
                theorem

                InformationTheory.Shannon.jointDifferentialEntropy_le_sum_of_llr_split

                source
                {μ : MeasureTheory.Measure ( × )} [MeasureTheory.IsProbabilityMeasure μ] (h_fst_ac : (MeasureTheory.Measure.map Prod.fst μ).AbsolutelyContinuous MeasureTheory.volume) (h_snd_ac : (MeasureTheory.Measure.map Prod.snd μ).AbsolutelyContinuous MeasureTheory.volume) (h_joint_ac : μ.AbsolutelyContinuous ((MeasureTheory.Measure.map Prod.fst μ).prod (MeasureTheory.Measure.map Prod.snd μ))) (h_llr_split : (fun (z : × ) => MeasureTheory.llr μ ((MeasureTheory.Measure.map Prod.fst μ).prod (MeasureTheory.Measure.map Prod.snd μ)) z) =ᵐ[μ] fun (z : × ) => Real.log (μ.rnDeriv MeasureTheory.volume z).toReal - Real.log ((MeasureTheory.Measure.map Prod.fst μ).rnDeriv MeasureTheory.volume z.1).toReal - Real.log ((MeasureTheory.Measure.map Prod.snd μ).rnDeriv MeasureTheory.volume z.2).toReal) (h_int_fst : MeasureTheory.Integrable (fun (z : × ) => Real.log ((MeasureTheory.Measure.map Prod.fst μ).rnDeriv MeasureTheory.volume z.1).toReal) μ) (h_int_snd : MeasureTheory.Integrable (fun (z : × ) => Real.log ((MeasureTheory.Measure.map Prod.snd μ).rnDeriv MeasureTheory.volume z.2).toReal) μ) (h_int_joint : MeasureTheory.Integrable (fun (z : × ) => Real.log (μ.rnDeriv MeasureTheory.volume z).toReal) μ) (h_int_fst_marg : MeasureTheory.Integrable (fun (x : ) => Real.log ((MeasureTheory.Measure.map Prod.fst μ).rnDeriv MeasureTheory.volume x).toReal) (MeasureTheory.Measure.map Prod.fst μ)) (h_int_snd_marg : MeasureTheory.Integrable (fun (y : ) => Real.log ((MeasureTheory.Measure.map Prod.snd μ).rnDeriv MeasureTheory.volume y).toReal) (MeasureTheory.Measure.map Prod.snd μ)) :

                2-variable differential-entropy subadditivity h(X,Y) ≤ h(X) + h(Y).

                Requires an explicit h_llr_split hypothesis for the Bayes density split. Superseded by jointDifferentialEntropy_le_sum, which internalizes the split.

                @audit:superseded-by(jointDifferentialEntropy_le_sum)

                Used by

                  n-variable bridge + subadditivity #

                  theorem

                  InformationTheory.Shannon.pi_withDensity_fin

                  source
                  {n : } (ν : Fin nMeasureTheory.Measure ) [∀ (i : Fin n), MeasureTheory.SigmaFinite (ν i)] {f : Fin nENNReal} (hf : ∀ (i : Fin n), Measurable (f i)) [∀ (i : Fin n), MeasureTheory.SigmaFinite ((ν i).withDensity (f i))] :
                  (MeasureTheory.Measure.pi fun (i : Fin n) => (ν i).withDensity (f i)) = (MeasureTheory.Measure.pi ν).withDensity fun (z : Fin n) => i : Fin n, f i (z i)

                  pi_withDensity (Mathlib absent, built by piFinSuccAbove induction). The product measure of withDensity factors is the withDensity of the product measure with the product density z ↦ ∏ᵢ fᵢ (z i). Specialized to Fin n → ℝ (all factors on ), the form the n-variable density split requires. @audit:ok

                  Used by
                    theorem

                    InformationTheory.Shannon.pi_marginals_eq_volume_withDensity

                    source
                    {n : } {μ : MeasureTheory.Measure (Fin n)} [MeasureTheory.IsProbabilityMeasure μ] [∀ (i : Fin n), MeasureTheory.IsProbabilityMeasure (MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ)] (h_marg_ac : ∀ (i : Fin n), (MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ).AbsolutelyContinuous MeasureTheory.volume) :
                    (MeasureTheory.Measure.pi fun (i : Fin n) => MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ) = MeasureTheory.volume.withDensity fun (z : Fin n) => i : Fin n, (MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ).rnDeriv MeasureTheory.volume (z i)

                    n-variable product-marginals factorization: Measure.pi (μ.map (· i)) expressed as a withDensity on Lebesgue measure with product density z ↦ ∏ᵢ (μ.map (· i)).rnDeriv volume (z i). @audit:ok

                    Used by
                      theorem

                      InformationTheory.Shannon.llr_split_from_density_factorize_pi

                      source
                      {n : } {μ : MeasureTheory.Measure (Fin n)} [MeasureTheory.IsProbabilityMeasure μ] [∀ (i : Fin n), MeasureTheory.IsProbabilityMeasure (MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ)] (h_marg_ac : ∀ (i : Fin n), (MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ).AbsolutelyContinuous MeasureTheory.volume) (hμ_ac : μ.AbsolutelyContinuous MeasureTheory.volume) (h_joint_ac : μ.AbsolutelyContinuous (MeasureTheory.Measure.pi fun (i : Fin n) => MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ)) :
                      (fun (z : Fin n) => MeasureTheory.llr μ (MeasureTheory.Measure.pi fun (i : Fin n) => MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ) z) =ᵐ[μ] fun (z : Fin n) => Real.log (μ.rnDeriv MeasureTheory.volume z).toReal - i : Fin n, Real.log ((MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ).rnDeriv MeasureTheory.volume (z i)).toReal

                      n-variable LLR split (a.e.[μ]): the log-likelihood ratio of μ against the product of its marginals equals log(joint density) − ∑ᵢ log(marginalᵢ density) almost-everywhere wrt μ. The n-variable analogue of llr_split_from_density_factorize. @audit:ok

                      Used by
                        theorem

                        InformationTheory.Shannon.klDiv_pi_marginals_toReal_eq_sum_sub_joint

                        source
                        {n : } {μ : MeasureTheory.Measure (Fin n)} [MeasureTheory.IsProbabilityMeasure μ] [∀ (i : Fin n), MeasureTheory.IsProbabilityMeasure (MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ)] (h_marg_ac : ∀ (i : Fin n), (MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ).AbsolutelyContinuous MeasureTheory.volume) (hμ_ac : μ.AbsolutelyContinuous MeasureTheory.volume) (h_joint_ac : μ.AbsolutelyContinuous (MeasureTheory.Measure.pi fun (i : Fin n) => MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ)) (h_int_joint : MeasureTheory.Integrable (fun (z : Fin n) => Real.log (μ.rnDeriv MeasureTheory.volume z).toReal) μ) (h_int_marg : ∀ (i : Fin n), MeasureTheory.Integrable (fun (z : Fin n) => Real.log ((MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ).rnDeriv MeasureTheory.volume (z i)).toReal) μ) :
                        (klDiv μ (MeasureTheory.Measure.pi fun (i : Fin n) => MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ)).toReal = i : Fin n, differentialEntropy (MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ) - jointDifferentialEntropyPi μ

                        n-variable subadditivity bridge: (klDiv(joint ‖ ∏ᵢ μᵢ)).toReal = ∑ᵢ h(μᵢ) − h(joint), where μᵢ := μ.map (· i).

                        Regularity hypotheses: absolute continuity + Bochner integrability of log-density observables. @audit:ok

                        Used by
                          theorem

                          InformationTheory.Shannon.jointDifferentialEntropyPi_le_sum

                          source
                          {n : } {μ : MeasureTheory.Measure (Fin n)} [MeasureTheory.IsProbabilityMeasure μ] [∀ (i : Fin n), MeasureTheory.IsProbabilityMeasure (MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ)] (h_marg_ac : ∀ (i : Fin n), (MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ).AbsolutelyContinuous MeasureTheory.volume) (hμ_ac : μ.AbsolutelyContinuous MeasureTheory.volume) (h_joint_ac : μ.AbsolutelyContinuous (MeasureTheory.Measure.pi fun (i : Fin n) => MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ)) (h_int_joint : MeasureTheory.Integrable (fun (z : Fin n) => Real.log (μ.rnDeriv MeasureTheory.volume z).toReal) μ) (h_int_marg : ∀ (i : Fin n), MeasureTheory.Integrable (fun (z : Fin n) => Real.log ((MeasureTheory.Measure.map (fun (z : Fin n) => z i) μ).rnDeriv MeasureTheory.volume (z i)).toReal) μ) :

                          n-variable differential-entropy subadditivity h(Yⁿ) ≤ ∑ᵢ h(Yᵢ) (the parallel-Gaussian consumer form). KL ≥ 0 + the bridge, by linarith. @audit:ok

                          Used by

                            2-variable Bayes density split #

                            The family below supplies the h_llr_split hypothesis of klDiv_prod_marginals_toReal_eq_sum_sub_joint_of_llr_split and jointDifferentialEntropy_le_sum_of_llr_split internally, via Mathlib's prod_withDensity + rnDeriv_mul_rnDeriv.

                            theorem

                            InformationTheory.Shannon.prod_marginals_eq_volume_withDensity

                            source

                            Product of marginals expressed as a withDensity on Lebesgue measure.

                            For a joint probability measure μ on ℝ × ℝ with marginals μX, μY both absolutely continuous wrt the Lebesgue measure, the product μX × μY factors through the Lebesgue measure on ℝ × ℝ as (μX).prod (μY) = volume.withDensity (z ↦ μX.rnDeriv volume z.1 * μY.rnDeriv volume z.2).

                            Used by
                              theorem

                              InformationTheory.Shannon.llr_split_from_density_factorize

                              source

                              Log-likelihood ratio split for the 2-variable joint (a.e.[μ]).

                              The LLR of μ against the product of its marginals equals log(joint density) − log(marginal_X density on z.1) − log(marginal_Y density on z.2) almost-everywhere wrt μ.

                              Used by
                                theorem

                                InformationTheory.Shannon.klDiv_prod_marginals_toReal_eq_sum_sub_joint

                                source

                                2-variable subadditivity bridge without explicit h_llr_split: (klDiv(joint ‖ μ_X ⊗ μ_Y)).toReal = h(μ_X) + h(μ_Y) − h(joint).

                                The Bayes density split is produced internally by llr_split_from_density_factorize.

                                Used by
                                  theorem

                                  InformationTheory.Shannon.jointDifferentialEntropy_le_sum

                                  source

                                  2-variable differential-entropy subadditivity h(X,Y) ≤ h(X) + h(Y).

                                  The Bayes density split is internalized via llr_split_from_density_factorize; no explicit h_llr_split argument required.

                                  Used by