InformationTheory

InformationTheory.Shannon.ChannelCoding.StrongConverseAsymptotic

source

Channel coding asymptotic strong converse (Wolfowitz) #

Builds on the single-shot Verdú-Han lower bound channelCoding_average_success_le (StrongConverse.lean) to prove the Wolfowitz strong converse: for a memoryless channel W over finite alphabets, if the rate log (M n) / n eventually exceeds capacity W + δ, then the average error probability tends to 1.

The argument substitutes the i.i.d. reference Q := q*^n (with q* the capacity-achieving output) and threshold n·(C + δ/2) into the single-shot bound, then drives both the exponential term and the high-information-density tail term to 0.

Main statements #

References #

  • T. M. Cover and J. A. Thomas, Elements of Information Theory (2nd ed.), Wiley, 2006. (strong converse).
  • J. Wolfowitz, Coding Theorems of Information Theory, Springer, 1978.

Capacity saddle point: keystone bridge + envelope directional derivative #

theorem

InformationTheory.Shannon.ChannelCoding.sum_channel_real_singleton_eq_one

source
{α : Type u_1} {β : Type u_2} [MeasurableSpace α] [Fintype β] [MeasurableSpace β] [MeasurableSingletonClass β] (W : Channel α β) [ProbabilityTheory.IsMarkovKernel W] (x : α) :
b : β, (W x).real {b} = 1

For a Markov channel each fiber W x is a probability measure, so its singleton masses sum to 1 over the finite output alphabet.

Used by
    theorem

    InformationTheory.Shannon.ChannelCoding.klDivPmf_crossEntropy

    source
    {β : Type u_2} [Fintype β] {P Q : β} (hP_nn : ∀ (b : β), 0 P b) (hP_sum : b : β, P b = 1) (hQ_sum : b : β, Q b = 1) (hQ_pos : ∀ (b : β), 0 < Q b) :
    CsiszarProjection.klDivPmf P Q = b : β, P b * Real.log (P b) - b : β, P b * Real.log (Q b)

    Cross-entropy form of klDivPmf: for a sub-probability-free pmf P (only required non-negative and summing to 1) and a full-support pmf Q, klDivPmf P Q equals the cross-entropy minus the entropy of P. Unlike klDivPmf_eq_log_diff_sum, P may vanish.

    Used by
      theorem

      InformationTheory.Shannon.ChannelCoding.mutualInfoOfChannel_toReal_eq_outputEntropy_sub

      source
      {α : Type u_1} {β : Type u_2} [Fintype α] [MeasurableSpace α] [MeasurableSingletonClass α] [Fintype β] [Nonempty β] [MeasurableSpace β] [MeasurableSingletonClass β] [Nonempty α] {p : α} (hp : p stdSimplex α) (W : Channel α β) [ProbabilityTheory.IsMarkovKernel W] :
      (mutualInfoOfChannel (pmfToMeasure p) W).toReal = b : β, (∑ x : α, p x * (W x).real {b}).negMulLog - x : α, p x * b : β, ((W x).real {b}).negMulLog

      Keystone bridge (I = H(Y) − H(Y|X)): for p ∈ stdSimplex, the channel mutual information equals the output entropy minus the weighted conditional entropy of the fibers.

      Used by
        theorem

        InformationTheory.Shannon.ChannelCoding.mutualInfo_segment_hasDerivAt

        source
        {α : Type u_1} {β : Type u_2} [Fintype α] [DecidableEq α] [MeasurableSpace α] [MeasurableSingletonClass α] [Fintype β] [Nonempty β] [MeasurableSpace β] [MeasurableSingletonClass β] (W : Channel α β) [ProbabilityTheory.IsMarkovKernel W] {p : α} (hp : p stdSimplex α) (hq_pos : ∀ (b : β), 0 < (outputDistribution (pmfToMeasure p) W).real {b}) (a : α) :
        HasDerivWithinAt (fun (t : ) => (mutualInfoOfChannel (pmfToMeasure ((1 - t) p + t Pi.single a 1)) W).toReal) ((CsiszarProjection.klDivPmf (fun (b : β) => (W a).real {b}) fun (b : β) => (outputDistribution (pmfToMeasure p) W).real {b}) - (mutualInfoOfChannel (pmfToMeasure p) W).toReal) (Set.Ici 0) 0

        The one-sided (right) directional derivative of t ↦ I(p_t; W).toReal at t = 0 along the segment p_t := (1 - t) • p + t • δ_a towards the Dirac input at a equals D(W(a)‖q*) − I(p; W) (the envelope/Danskin cancellation: the moving reference q_{p_t} contributes nothing because ∑_b (dq/dt)(b) = 0).

        Stated as a HasDerivWithinAt over Set.Ici 0 (right derivative), NOT a two-sided HasDerivAt. The two-sided form is FALSE for boundary achievers: when p a = 0, for t < 0 the segment leaves the simplex (p_t a = t < 0), pmfToMeasure clamps the negative coordinate via ENNReal.ofReal to 0, giving the non-probability measure (1 - t) • pmfToMeasure p, so I(p_t; W).toReal no longer follows the smooth simplex functional and develops a corner at 0 (left derivative ≠ right derivative). Concrete refutation: α = β = Bool, p = δ_false, a = true, any channel with W false full support and W true ≠ W false; the right derivative is D(W true‖W false) > 0, but the left branch is the non-probability functional t ↦ (klDiv ((1-t) • J) ((1-t)² • (J)) ).toReal (here the input-deterministic joint J equals the product measure), whose derivative at 0 does not match. The one-sided form is also exactly what the downstream first-order optimality argument consumes (cf. csiszar_first_order_condition, which uses the 𝓝[>] 0 slope). @audit:ok

        Used by
          theorem

          InformationTheory.Shannon.ChannelCoding.klDiv_channel_le_capacity

          source
          {α : Type u_1} {β : Type u_2} [Fintype α] [DecidableEq α] [MeasurableSpace α] [MeasurableSingletonClass α] [Fintype β] [Nonempty β] [MeasurableSpace β] [MeasurableSingletonClass β] (W : Channel α β) [ProbabilityTheory.IsMarkovKernel W] {p : α} (hp : p stdSimplex α) (hp_max : IsMaxOn (fun (p : α) => (mutualInfoOfChannel (pmfToMeasure p) W).toReal) (stdSimplex α) p) (hq_pos : ∀ (b : β), 0 < (outputDistribution (pmfToMeasure p) W).real {b}) (a : α) :
          (CsiszarProjection.klDivPmf (fun (b : β) => (W a).real {b}) fun (b : β) => (outputDistribution (pmfToMeasure p) W).real {b}) capacity W

          Capacity saddle point: for a capacity-achieving input p with full-support output q* := outputDistribution (pmfToMeasure p) W, every input symbol a satisfies D(W(a)‖q*) ≤ capacity W. Carved out as a shared lemma for reuse across the channel-coding converse family. @audit:ok

          Used by

            Non-i.i.d. Chebyshev concentration of the information density #

            theorem

            InformationTheory.Shannon.ChannelCoding.integral_logRatio_eq_klDivPmf

            source
            {α : Type u_1} {β : Type u_2} [MeasurableSpace α] [Fintype β] [MeasurableSpace β] [MeasurableSingletonClass β] (W : Channel α β) [ProbabilityTheory.IsMarkovKernel W] (a : α) {qf : β} (hqf_sum : b : β, qf b = 1) (hqf_pos : ∀ (b : β), 0 < qf b) :
            (b : β), Real.log ((W a).real {b}) - Real.log (qf b) W a = CsiszarProjection.klDivPmf (fun (b : β) => (W a).real {b}) qf

            The expectation of the per-letter log-likelihood ratio log (W a)(·) − log q*(·) under the channel fiber W a is the discrete KL divergence D(W a ‖ q*).

            Used by
              noncomputable def

              InformationTheory.Shannon.ChannelCoding.llrUnifBound

              source
              {α : Type u_1} {β : Type u_2} [Fintype α] [MeasurableSpace α] [Fintype β] [MeasurableSpace β] (W : Channel α β) (p : α) :

              Uniform bound on the per-letter log-likelihood ratio log (W a)(b) − log q*(b) across the finite input/output alphabets; an n-independent variance bound for the information density.

              Equations
              • One or more equations did not get rendered due to their size.
              Instances For
                Used by
                  theorem

                  InformationTheory.Shannon.ChannelCoding.highLLRSet_real_le

                  source
                  {α : Type u_1} {β : Type u_2} [Fintype α] [DecidableEq α] [MeasurableSpace α] [MeasurableSingletonClass α] [Fintype β] [Nonempty β] [MeasurableSpace β] [MeasurableSingletonClass β] (W : Channel α β) [ProbabilityTheory.IsMarkovKernel W] {δ : } ( : 0 < δ) {p : α} (hp : p stdSimplex α) (hp_max : IsMaxOn (fun (p : α) => (mutualInfoOfChannel (pmfToMeasure p) W).toReal) (stdSimplex α) p) (hq_pos : ∀ (b : β), 0 < (outputDistribution (pmfToMeasure p) W).real {b}) {n : } (hn : 0 < n) {M : } (c : Code M n α β) (m : Fin M) :
                  (MeasureTheory.Measure.pi fun (i : Fin n) => W (c.encoder m i)).real (highLLRSet W c (MeasureTheory.Measure.pi fun (x : Fin n) => outputDistribution (pmfToMeasure p) W) (n * (capacity W + δ / 2)) m) llrUnifBound W p ^ 2 / (δ / 2) ^ 2 * (1 / n)

                  Per-codeword Chebyshev bound: for block length n ≥ 1, the channel-output mass of the high-LLR set for codeword m is at most B² / (δ/2)² · (1/n), where B := llrUnifBound W p is an n-independent uniform bound on the per-letter log-likelihood ratios. Uses the saddle point klDiv_channel_le_capacity for the uniform mean bound.

                  Used by
                    theorem

                    InformationTheory.Shannon.ChannelCoding.channelCoding_highLLR_tendsto_zero

                    source
                    {α : Type u_1} {β : Type u_2} [Fintype α] [DecidableEq α] [MeasurableSpace α] [MeasurableSingletonClass α] [Fintype β] [Nonempty β] [MeasurableSpace β] [MeasurableSingletonClass β] (W : Channel α β) [ProbabilityTheory.IsMarkovKernel W] {δ : } ( : 0 < δ) {p : α} (hp : p stdSimplex α) (hp_max : IsMaxOn (fun (p : α) => (mutualInfoOfChannel (pmfToMeasure p) W).toReal) (stdSimplex α) p) (hq_pos : ∀ (b : β), 0 < (outputDistribution (pmfToMeasure p) W).real {b}) (M : ) (c : (n : ) → Code (M n) n α β) :
                    Filter.Tendsto (fun (n : ) => 1 / (M n) * m : Fin (M n), (MeasureTheory.Measure.pi fun (i : Fin n) => W ((c n).encoder m i)).real (highLLRSet W (c n) (MeasureTheory.Measure.pi fun (x : Fin n) => outputDistribution (pmfToMeasure p) W) (n * (capacity W + δ / 2)) m)) Filter.atTop (nhds 0)

                    The average high-LLR tail mass vanishes as the block length grows, using the non-i.i.d. Chebyshev concentration (meas_ge_le_variance_div_sq + variance_sum_pi) with the i.i.d. reference q*^n and threshold n·(capacity W + δ/2). Depends on the saddle point klDiv_channel_le_capacity for the uniform per-codeword mean bound.

                    Used by

                      The strong converse and its operational form #

                      theorem

                      InformationTheory.Shannon.ChannelCoding.channelCoding_strong_converse_asymptotic

                      source
                      {α : Type u_1} {β : Type u_2} [Fintype α] [DecidableEq α] [MeasurableSpace α] [MeasurableSingletonClass α] [Fintype β] [Nonempty β] [MeasurableSpace β] [MeasurableSingletonClass β] (W : Channel α β) [ProbabilityTheory.IsMarkovKernel W] (M : ) (hM : ∀ (n : ), 0 < M n) (c : (n : ) → Code (M n) n α β) {δ : } ( : 0 < δ) (p : α) (hp : p stdSimplex α) (hp_max : IsMaxOn (fun (p : α) => (mutualInfoOfChannel (pmfToMeasure p) W).toReal) (stdSimplex α) p) (hq_pos : ∀ (b : β), 0 < (outputDistribution (pmfToMeasure p) W).real {b}) (hrate : ∀ᶠ (n : ) in Filter.atTop, capacity W + δ Real.log (M n) / n) :

                      Wolfowitz strong converse (asymptotic): for a memoryless channel W over finite alphabets, if the rate log (M n) / n eventually exceeds capacity W + δ (with δ > 0), then the average error probability tends to 1.

                      The capacity-achieving input p (existing by exists_capacity_achiever) is received explicitly together with the regularity precondition hq_pos (full-support output, so the log-likelihood ratios are well-defined); both are preconditions, not load-bearing hypotheses. @audit:ok

                      Used by
                        theorem

                        InformationTheory.Shannon.ChannelCoding.channelCoding_operational_rate_le_capacity

                        source
                        {α : Type u_1} {β : Type u_2} [Fintype α] [DecidableEq α] [MeasurableSpace α] [MeasurableSingletonClass α] [Fintype β] [Nonempty β] [MeasurableSpace β] [MeasurableSingletonClass β] [Nonempty α] (W : Channel α β) [ProbabilityTheory.IsMarkovKernel W] (p : α) (hp : p stdSimplex α) (hp_max : IsMaxOn (fun (p : α) => (mutualInfoOfChannel (pmfToMeasure p) W).toReal) (stdSimplex α) p) (hq_pos : ∀ (b : β), 0 < (outputDistribution (pmfToMeasure p) W).real {b}) {R : } (hach : ∀ (ε : ), 0 < ε∃ (N : ), ∀ (n : ), N n∃ (M : ) (c : Code M n α β), Real.exp (n * R) M (c.averageErrorProb W).toReal < ε) :

                        An operationally achievable rate is at most the capacity: if for every target error ε > 0 all large enough block lengths carry a code with at least exp (n R) messages and average error below ε, then R ≤ capacity W.

                        This is the contrapositive of channelCoding_strong_converse_asymptotic: were capacity W < R, the codes would have rate eventually above capacity W + (R - capacity W), forcing their average error to tend to 1 and contradicting the ε-small errors. The capacity achiever p and the full-support output hq_pos are the preconditions of the strong converse, passed through unchanged. @audit:ok

                        Used by