InformationTheory

InformationTheory.Shannon.Cramer.Cramer

source

Cramér's theorem #

The Legendre transform of a real-valued function and the Cramér rate function (I(a) = Λ^*(a) for Λ = cgf X μ), together with the upper bound of Cramér's large deviation theorem (Cover–Thomas, upper half) in per-n Chernoff, log, and limsup forms. The lower bound is developed downstream in CramerGeneralLower.lean.

Main definitions #

  • legendre Λ a — the Legendre transform Λ^*(a) := sup_λ (λ·a − Λ(λ)).
  • cramerRate X μ a — the Cramér rate function I(a) := (cgf X μ)^*(a).

Main statements #

References #

  • T. M. Cover and J. A. Thomas, Elements of Information Theory (2nd ed.), Wiley, 2006.

legendre and cramerRate: definitions and basic properties #

noncomputable def

InformationTheory.Shannon.Cramer.legendre

source
(Λ : ) (a : ) :

The Legendre transform of Λ : ℝ → ℝ at a: Λ^*(a) := sup_λ (λ·a − Λ(λ)).

Mathlib has no general Legendre-transform / convex-conjugate API, so this is the textbook sSup form. When the image set is not BddAbove, Mathlib returns sSup = 0, so the basic properties below take an explicit BddAbove hypothesis.

Equations
Instances For
    Used by
      noncomputable def

      InformationTheory.Shannon.Cramer.cramerRate

      source
      {Ω : Type u_1} [MeasurableSpace Ω] (X : Ω) (μ : MeasureTheory.Measure Ω) (a : ) :

      Cramér rate function I(a) := Λ^*(a) for Λ := cgf X μ.

      Equations
      Instances For
        Used by
          theorem

          InformationTheory.Shannon.Cramer.legendre_apply_le

          source
          (Λ : ) (a : ) (h_bdd : BddAbove ((fun (lam : ) => lam * a - Λ lam) '' Set.univ)) (lam : ) :
          lam * a - Λ lam legendre Λ a

          Each linear-minus-Λ value is ≤ the Legendre transform.

          Used by
            theorem

            InformationTheory.Shannon.Cramer.legendre_nonneg

            source
            (Λ : ) (hΛ0 : Λ 0 = 0) (a : ) (h_bdd : BddAbove ((fun (lam : ) => lam * a - Λ lam) '' Set.univ)) :
            0 legendre Λ a

            If Λ 0 = 0 (e.g. Λ = cgf X μ for a probability measure), the Legendre transform is non-negative whenever the supremum exists.

            Used by
              theorem

              InformationTheory.Shannon.Cramer.cramerRate_apply_le

              source
              {Ω : Type u_1} [MeasurableSpace Ω] (X : Ω) (μ : MeasureTheory.Measure Ω) (a : ) (h_bdd : BddAbove ((fun (lam : ) => lam * a - ProbabilityTheory.cgf X μ lam) '' Set.univ)) (lam : ) :
              lam * a - ProbabilityTheory.cgf X μ lam cramerRate X μ a

              Cramér rate-function version of legendre_apply_le.

              Used by
                theorem

                InformationTheory.Shannon.Cramer.cramerRate_nonneg

                source
                {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsProbabilityMeasure μ] (X : Ω) (a : ) (h_bdd : BddAbove ((fun (lam : ) => lam * a - ProbabilityTheory.cgf X μ lam) '' Set.univ)) :
                0 cramerRate X μ a

                Cramér rate function is non-negative whenever the Legendre supremum exists for a probability measure (then cgf X μ 0 = 0).

                Used by

                  cgf sum and bounded-RV integrability helpers #

                  theorem

                  InformationTheory.Shannon.Cramer.integrable_exp_mul_of_bounded

                  source
                  {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsFiniteMeasure μ] {Y : Ω} (hY_meas : Measurable Y) (h_bdd : ∃ (M : ), ∀ (ω : Ω), |Y ω| M) (t : ) :
                  MeasureTheory.Integrable (fun (ω : Ω) => Real.exp (t * Y ω)) μ

                  For a bounded real random variable on a finite measure space, the exponential moment exp (t * Y) is integrable for every t. This is the hypothesis-eliminator the Cramér chain uses to remove Integrable premises from the main statements.

                  Used by
                    theorem

                    InformationTheory.Shannon.Cramer.cgf_sum_eq_nsmul

                    source
                    {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} {X : Ω} (h_indep : ProbabilityTheory.iIndepFun X μ) (h_meas : ∀ (i : ), Measurable (X i)) (h_ident : ∀ (i : ), ProbabilityTheory.IdentDistrib (X i) (X 0) μ μ) (h_int : ∀ (t : ) (i : ), MeasureTheory.Integrable (fun (ω : Ω) => Real.exp (t * X i ω)) μ) (t : ) (n : ) :
                    ProbabilityTheory.cgf (∑ iFinset.range n, X i) μ t = n * ProbabilityTheory.cgf (X 0) μ t

                    i.i.d. CGF sum formula: for an i.i.d. family X : ℕ → Ω → ℝ (with X i identically distributed to X 0 and the exponential moments integrable for every i), cgf (∑ i ∈ range n, X i) μ t = n · cgf (X 0) μ t.

                    Used by

                      Cramér upper bound: per-n Chernoff bound (i.i.d. strengthening) #

                      theorem

                      InformationTheory.Shannon.Cramer.chernoff_bound_n_iid

                      source
                      {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsProbabilityMeasure μ] {X : Ω} (h_indep : ProbabilityTheory.iIndepFun X μ) (h_meas : ∀ (i : ), Measurable (X i)) (h_ident : ∀ (i : ), ProbabilityTheory.IdentDistrib (X i) (X 0) μ μ) (h_bdd : ∃ (M : ), ∀ (i : ) (ω : Ω), |X i ω| M) (a : ) (n : ) (lam : ) (hlam : 0 lam) :
                      μ.real {ω : Ω | a * n iFinset.range n, X i ω} Real.exp (-n * (lam * a - ProbabilityTheory.cgf (X 0) μ lam))

                      Per-n Chernoff bound for the upper tail of an i.i.d. sum of bounded real random variables: tilting by any lam ≥ 0, the upper-tail probability decays exponentially with rate at least lam * a − Λ(lam).

                      Used by
                        theorem

                        InformationTheory.Shannon.Cramer.cramer_log_bound_n_iid

                        source
                        {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsProbabilityMeasure μ] {X : Ω} (h_indep : ProbabilityTheory.iIndepFun X μ) (h_meas : ∀ (i : ), Measurable (X i)) (h_ident : ∀ (i : ), ProbabilityTheory.IdentDistrib (X i) (X 0) μ μ) (h_bdd : ∃ (M : ), ∀ (i : ) (ω : Ω), |X i ω| M) (a : ) {n : } (hn : 0 < n) (h_pos : 0 < μ.real {ω : Ω | a * n iFinset.range n, X i ω}) (lam : ) (hlam : 0 lam) :
                        1 / n * Real.log (μ.real {ω : Ω | a * n iFinset.range n, X i ω}) -(lam * a - ProbabilityTheory.cgf (X 0) μ lam)

                        Per-n Cramér upper bound, log form: for each n ≥ 1 with positive tail probability, (1/n) · log P[a·n ≤ Sₙ] ≤ -(lam · a − Λ(lam)) for every lam ≥ 0.

                        Used by

                          Cramér upper bound: limsup form #

                          theorem

                          InformationTheory.Shannon.Cramer.cramer_upper

                          source
                          {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsProbabilityMeasure μ] {X : Ω} (h_indep : ProbabilityTheory.iIndepFun X μ) (h_meas : ∀ (i : ), Measurable (X i)) (h_ident : ∀ (i : ), ProbabilityTheory.IdentDistrib (X i) (X 0) μ μ) (h_bdd : ∃ (M : ), ∀ (i : ) (ω : Ω), |X i ω| M) (a lam : ) (hlam : 0 lam) (h_pos : ∀ᶠ (n : ) in Filter.atTop, 0 < μ.real {ω : Ω | a * n iFinset.range n, X i ω}) (h_cobdd : Filter.IsCoboundedUnder (fun (x1 x2 : ) => x1 x2) Filter.atTop fun (n : ) => 1 / n * Real.log (μ.real {ω : Ω | a * n iFinset.range n, X i ω})) :
                          Filter.limsup (fun (n : ) => 1 / n * Real.log (μ.real {ω : Ω | a * n iFinset.range n, X i ω})) Filter.atTop -(lam * a - ProbabilityTheory.cgf (X 0) μ lam)

                          Cramér's theorem (upper bound, limsup form): for each lam ≥ 0, the upper-tail probability of the i.i.d. sample sum decays at exponential rate at least lam * a − Λ(lam):

                          limsup_n (1/n) log P[a·n ≤ Sₙ] ≤ -(lam · a − Λ(lam)).

                          Two technical hypotheses make the result clean:

                          • h_pos — the tail probability is eventually positive (e.g. when a ≤ ess sup X), so that log is finite.
                          • h_cobdd — the resulting log-rate sequence is cobounded below in the limsup sense. This holds whenever the sequence does not blow up to -∞, e.g. when the tail probabilities admit any sub-exponential lower bound.
                          Used by
                            theorem

                            InformationTheory.Shannon.Cramer.cramer_upper_legendre

                            source
                            {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsProbabilityMeasure μ] {X : Ω} (h_indep : ProbabilityTheory.iIndepFun X μ) (h_meas : ∀ (i : ), Measurable (X i)) (h_ident : ∀ (i : ), ProbabilityTheory.IdentDistrib (X i) (X 0) μ μ) (h_bdd : ∃ (M : ), ∀ (i : ) (ω : Ω), |X i ω| M) (a lam : ) (hlam : 0 lam) (hlam_opt : lam * a - ProbabilityTheory.cgf (X 0) μ lam = cramerRate (X 0) μ a) (h_pos : ∀ᶠ (n : ) in Filter.atTop, 0 < μ.real {ω : Ω | a * n iFinset.range n, X i ω}) (h_cobdd : Filter.IsCoboundedUnder (fun (x1 x2 : ) => x1 x2) Filter.atTop fun (n : ) => 1 / n * Real.log (μ.real {ω : Ω | a * n iFinset.range n, X i ω})) :
                            Filter.limsup (fun (n : ) => 1 / n * Real.log (μ.real {ω : Ω | a * n iFinset.range n, X i ω})) Filter.atTop -cramerRate (X 0) μ a

                            Cramér's theorem (upper bound, Legendre form): if the Legendre transform of Λ = cgf (X 0) μ at a is attained by some non-negative lam (hlam_opt), then

                            limsup_n (1/n) log P[a·n ≤ Sₙ] ≤ -cramerRate (X 0) μ a.

                            hlam_opt is a regularity precondition (it holds for a ≥ 𝔼[X] by convexity of the Legendre transform), not part of the proof core.

                            See also cramer_upper.

                            Used by

                              Cramér lower bound: tilted change-of-measure identities #

                              The lower bound uses an exponential change-of-measure to the tilted measure μ_lam := μ.tilted (lam * X ·), where lam is chosen so that Λ'(lam) = a. Under the tilted measure the mean of X equals a, a tilted-side law of large numbers concentrates the sample mean near a, and converting back to μ via the Radon–Nikodym derivative of tilted picks up an exp(-n · (lam · a − Λ(lam))) = exp(-n · cramerRate) factor.

                              This file publishes the change-of-measure building blocks — klDiv_tilted_eq (the KL-of-tilted identity) and integral_tilted_eq_deriv_cgf — that the downstream lower bound consumes.

                              theorem

                              InformationTheory.Shannon.Cramer.mem_interior_integrableExpSet_of_bounded

                              source
                              {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsFiniteMeasure μ] {Y : Ω} (hY_meas : Measurable Y) (h_bdd : ∃ (M : ), ∀ (ω : Ω), |Y ω| M) (t : ) :

                              Universal integrability for bounded RVs: a bounded random variable has every t ∈ ℝ in its integrableExpSet, hence the whole real line lies in the interior. This kills the interior (integrableExpSet X μ) hypothesis of integral_tilted_mul_self / variance_tilted_mul in the bounded-RV setting.

                              Used by
                                theorem

                                InformationTheory.Shannon.Cramer.isProbabilityMeasure_tilted_of_bounded

                                source
                                {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsProbabilityMeasure μ] {Y : Ω} (hY_meas : Measurable Y) (h_bdd : ∃ (M : ), ∀ (ω : Ω), |Y ω| M) (lam : ) :
                                MeasureTheory.IsProbabilityMeasure (μ.tilted fun (ω : Ω) => lam * Y ω)

                                The tilted measure of a bounded random variable is a probability measure.

                                Used by
                                  theorem

                                  InformationTheory.Shannon.Cramer.integral_tilted_eq_deriv_cgf

                                  source
                                  {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsProbabilityMeasure μ] {Y : Ω} (hY_meas : Measurable Y) (h_bdd : ∃ (M : ), ∀ (ω : Ω), |Y ω| M) (lam : ) :
                                  ( (ω : Ω), Y ω μ.tilted fun (ω : Ω) => lam * Y ω) = deriv (ProbabilityTheory.cgf Y μ) lam

                                  For a bounded random variable Y, the expectation of Y under μ.tilted (lam * Y ·) equals the first derivative of cgf Y μ at lam.

                                  Used by
                                    theorem

                                    InformationTheory.Shannon.Cramer.klDiv_tilted_eq

                                    source
                                    {Ω : Type u_1} [MeasurableSpace Ω] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsProbabilityMeasure μ] (X : Ω) (hX_meas : Measurable X) (h_bdd : ∃ (M : ), ∀ (ω : Ω), |X ω| M) (lam : ) :
                                    ( (ω : Ω), Real.log ((μ.tilted fun (ω' : Ω) => lam * X ω').rnDeriv μ ω).toReal μ.tilted fun (ω' : Ω) => lam * X ω') = (lam * (ω : Ω), X ω μ.tilted fun (ω' : Ω) => lam * X ω') - ProbabilityTheory.cgf X μ lam

                                    KL-of-tilted identity.

                                    For a bounded real random variable X : Ω → ℝ on a probability measure μ, the (integral form of the) Kullback-Leibler divergence between μ.tilted (lam * X ·) and μ admits the closed form

                                    ∫ ω, log (rnDeriv (μ.tilted (lam * X ·)) μ ω).toReal ∂(μ.tilted (lam * X ·)) = lam * (μ.tilted (lam * X ·))[X] - cgf X μ lam.

                                    We work directly with the ∫ log (rnDeriv) representation rather than the ℝ≥0∞-valued klDiv to keep the algebraic identity transparent.

                                    Used by