Tight Lower Bound on Differential Entropy for Mixed Gaussian Distributions
DOI:
https://doi.org/10.26636/jtit.2024.2.1444Keywords:
differential entropy, lower bound, mixture random variable, multimodal GaussianAbstract
In this paper, a tight lower bound for the differential entropy of the Gaussian mixture model is presented. First, the probability model of mixed Gaussian distribution that is created by mixing both discrete and continuous random variables is investigated in order to represent symmetric bimodal Gaussian distribution using the hyperbolic cosine function, on which a tighter upper bound is set. Then, this tight upper bound is used to derive a tight lower bound for the differential entropy of the Gaussian mixture model introduced. The proposed lower bound allows to maintain its tightness over the entire range of the model's parameters and shows more tightness when compared with other bounds that lose their tightness over certain parameter ranges. The presented results are then extended to introduce a more general tight lower bound for asymmetric bimodal Gaussian distribution, in which the two modes have a symmetric mean but differ in terms of their weights.
Downloads
References
J. Jose et al., "An Image Quality Enhancement Scheme Employing Adolescent Identity Search Algorithm in the NSST Domain for Multimodal Medical Image Fusion", Biomedical Signal Processing and Control, vol. 66, art. no. 102480, 2021.
View in Google Scholar
J. Xu and Z. Cai, "Gaussian Mixture Deep Dynamic Latent Variable Model with Application to Soft Sensing for Multi-mode Industrial Processes", Applied Soft Computing, vol. 114, art. no. 108092, 2022.
View in Google Scholar
J. Skvara and I. Nezbeda, "Thermodynamics and Structure of Supercooled Water. II.", Journal of Molecular Liquids, vol. 367, art. no. 120508, 2022.
View in Google Scholar
K. Moshksar and A.K. Khandani, "Arbitrarily Tight Bounds on Differential Entropy of Gaussian Mixtures", IEEE Transactions on Information Theory, vol. 62, no. 6, pp. 3340-3354, 2016.
View in Google Scholar
M. Wen, X. Cheng, and L. Yang, Index Modulation for 5G Wireless Communications, Springer, 154 p., 2017.
View in Google Scholar
V. Bhatia and B. Mulgrew, "Non-parametric Likelihood Based Channel Estimator for Gaussian Mixture Noise", Signal Processing, vol. 87, no. 11, pp. 2569-2586, 2007.
View in Google Scholar
D. Peel and G.J. McLachlan, Finite Mixture Models, John Wiley & Sons, 427 p., 2000.
View in Google Scholar
G.J. McLachlan, S.X. Lee, and S.I. Rathnayake, "Finite Mixture Models", Annual Review of Statistics and its Application, vol. 6, pp. 335-378, 2019.
View in Google Scholar
T.M. Cover and J.A. Thomas, Elements of Information Theory, John Wiley & Sons, 748 p., 2005.
View in Google Scholar
M.F. Huber, T. Bailey, H. Durrant-Whyte, and U.D. Hanebeck, "On Entropy Approximation for Gaussian Mixture Random Vectors", 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, Seoul, South Korea, 2008.
View in Google Scholar
J. Melbourne, S. Talukdar, S. Bhaban, and M.V. Salapaka, "Error Bounds on a Mixed Entropy Inequality", 2018 IEEE International Symposium on Information Theory (ISIT), Vail, USA, 2018.
View in Google Scholar
A. Kolchinsky and B.D. Tracey, "Estimating Mixture Entropy with Pairwise Distances", Entropy, vol. 19, no. 7, art. no. 361, 2017.
View in Google Scholar
L.V. Michalowicz, J.M. Nichols, and F. Bucholtz, "Calculation of Differential Entropy for a Mixed Gaussian Distribution", Entropy, vol. 10, no. 3, pp. 200-206, 2008.
View in Google Scholar
J. Sandor, "On Some Inequalities for the Identric, Logarithmic and Related Means", Aequationes Mathematicae, vol. 40, pp. 261-270, 1990.
View in Google Scholar
K.B. Stolarsky, "Holder Means, Lehmer Means, andx-1 log coshx", Journal of Mathematical Analysis and Applications, vol. 202, no. 3, pp. 810-818, 1996.
View in Google Scholar
W.J. Weltner, P. Schuster, and K. Weltner, Mathematics for Engineers and Scientists, Stanley Thornes, 520 p., 1986 (ISBN: 9780859501200).
View in Google Scholar
M. Raginsky, "On the Information Capacity of Gaussian Channels under Small Peak Power Constraints", 46th Annual Allerton Conference on Communication, Control, and Computing, Monticello, USA, 2008.
View in Google Scholar
I.S. Gradshteyn and I.M. Ryzhik, Table of Integrals, Series, and Products, Elsevier, 1160 p., 1980.
View in Google Scholar
J. Melbourne, S. Talukdar, S. Bhaban, M. Madiman, and M.V. Salapaka, "The Differential Entropy of Mixtures: New Bounds and Applications", Entropy, vol. 68, no. 4, pp. 2123-2146, 2022.
View in Google Scholar
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Abdelrahman Marconi, Ahmed H. Elghandour, Ashraf D. Elbayoumy, Amr Abdelaziz
This work is licensed under a Creative Commons Attribution 4.0 International License.
