High-level and Low-level Feature Set for Image Caption Generation with Optimized Convolutional Neural Network

Authors

DOI:

https://doi.org/10.26636/jtit.2022.164222

Keywords:

CNN, image caption, proposed contrast, sharpness, SMO-SCME algorithm

Abstract

Automatic creation of image descriptions, i.e. captioning of images, is an important topic in artificial intelligence (AI) that bridges the gap between computer vision (CV) and natural language processing (NLP). Currently, neural networks are becoming increasingly popular in captioning images and researchers are looking for more efficient models for CV and sequence-sequence systems. This study focuses on a new image caption generation model that is divided into two stages. Initially, low-level features, such as contrast, sharpness, color and their high-level counterparts, such as motion and facial impact score, are extracted. Then, an optimized convolutional neural network (CNN) is harnessed to generate the captions from images. To enhance the accuracy of the process, the weights of CNN are optimally tuned via spider monkey optimization with sine chaotic map evaluation (SMO-SCME). The development of the proposed method is evaluated with a diversity of metrics.

Downloads

Download data is not yet available.

References

[1] Z. Deng, Z. Jiang, R. Lan, W. Huang, and X. Luo, “Image captioning using DenseNet network and adaptive attention”, Signal Processing: Image Communication, vol. 85, 2020.
View in Google Scholar

[2] J. Su, J. Tang, Z. Lu, X. Han, and H. Zhang, “A neural image captioning model with caption-to-images semantic constructor”, Neurocomputing, vol. 367, pp. 144–151, 2019.
View in Google Scholar

[3] S. Bang and H. Kim, “Context-based information generation for managing UAV-acquired data using image captioning”, Automation in Construction, vol. 112, 2020.
View in Google Scholar

[4] H. Wang, H. Wang, and K. Xu, “Evolutionary recurrent neural network for image captioning”, Neurocomputing, vol. 401, pp. 249–256, 2020.
View in Google Scholar

[5] R. Li, H. Liang, Y. Shi, F. Feng, and X. Wang, “Dual-CNN: A convolutional language decoder for paragraph image captioning”, Neurocomputing, vol. 396, pp. 92–101, 2020.
View in Google Scholar

[6] J. Guan and E. Wang, “Repeated review-based image captioning for image evidence review”, Signal Processing: Image Communication, vol. 63, pp. 141–148, 2018.
View in Google Scholar

[7] A. Singh, T. D. Singh, and S. Bandyopadhyay, “An encoder-decoder-based framework for Hindi image caption generation”, Multimed. Tools Appl., vol. 80, pp. 35721–35740, 2021.
View in Google Scholar

[8] Ph. Kinghorn, L. Zhang, and L. Shao, “A region-based image caption generator with refined descriptions”, Neurocomputing, vol. 272, pp. 416–424, 2018.
View in Google Scholar

[9] Q. Liu, Y. Chen, J. Wang, and S. Zhang, “Multi-view pedestrian captioning with an attention topic CNN model”, Computers in Industry, vol. 97, pp. 47–53, 2018.
View in Google Scholar

[10] G. Christie, A. Laddha, A. Agrawal, S. Antol, and D. Batra, “Resolving vision and language ambiguities together: Joint segmentation & prepositional attachment resolution in captioned scenes”, Computer Vision and Image Understanding, vol. 163, pp. 101–112, 2017.
View in Google Scholar

[11] F. Xiao, X. Gong, Y. Zhang, Y. Shen, and X. Gao, “DAA: Dual LSTMs with adaptive attention for image captioning”, Neurocomputing, vol. 364, pp. 322–329, 2019.
View in Google Scholar

[12] G. Huang and H. Hu, “c-RNN: A fine-grained language model for image captioning”, Neural Process. Lett., 2018.
View in Google Scholar

[13] C. Wu, Y. Wei, X. Chu, F. Su, and L. Wang, “Modeling visual and word-conditional semantic attention for image captioning”, Signal Processing: Image Communication, vol. 67, pp. 100–107, 2018.
View in Google Scholar

[14] J. Yang, Y. Sun, J. Liang, B. Ren, and S. Lai, “Image captioning by incorporating affective concepts learned from both visual and textual components”, Neurocomputing, 2018.
View in Google Scholar

[15] Y. H. Tan and C. S. Chan, “Phrase-based image caption generator with hierarchical LSTM network”, Neurocomputing, 2018.
View in Google Scholar

[16] A. Yuan, X. Li, and X. Lu, “3G structure for image caption generation”, Neurocomputing, 2018.
View in Google Scholar

[17] Ch. Fan, Z. Zhang, and D. J. Crandall, “DeepDiary: Lifelogging image captioning and summarization”, Journal of Visual Communication and Image Representation, vol. 55, pp. 40–55, 2018.
View in Google Scholar

[18] X. Chen, M. Zhang, Z. Wang, L. Zuo, and Y. Yang, “Leveraging unpaired out-of-domain data for image captioning”, Pattern Recognition Letters, In press, accepted manuscript, 2018.
View in Google Scholar

[19] Z. Ye et al., “A novel automatic image caption generation using bidirectional long-short term memory framework”, Multimed. Tools Appl., vol. 80, pp. 25557–25582, 2021.
View in Google Scholar

[20] H. Zhang et al., “Novel model to integrate word embeddings and syntactic trees for automatic caption generation from images”, Soft Comput., vol. 24, pp. 1377–1397, 2020.
View in Google Scholar

[21] C. Sur, “AACR: Feature fusion effects of algebraic amalgamation composed representation on (De)Compositional network for caption generation for images”, SN Comput. Sci., vol. 1, Article no. 229, 2020.
View in Google Scholar

[22] S. Cao, G. An, Z. Zheng, and Q. Ruan, “Interactions guided generative adversarial network for unsupervised image captioning”, Neurocomputing, vol. 417, pp. 419–431, 2020.
View in Google Scholar

[23] Y. Wei, L. Wang, and C. Wu, “Multi-attention generative adversarial network for image captioning”, Neurocomputing, vol. 387, pp. 91–99, 2019.
View in Google Scholar

[24] M. Yang et al., “An ensemble of generation- and retrieval-based image captioning with dual generator generative adversarial network”, IEEE Transactions on Image Processing, vol. 29, pp. 9627–9640, 2020.
View in Google Scholar

[25] D. Zhao, Z. Chang, and S. Guo, “A multimodal fusion approach for image captioning”, Neurocomputing, vol. 329, pp. 476–485, 2019.
View in Google Scholar

[26] S. Ding, S. Qu, and S. Wan, “Image caption generation with high-level image features”, Pattern Recognition Letters, vol. 123, pp. 89–95, 2019.
View in Google Scholar

[27] S. R. Kodituwakku, “Comparison of color features for image retrieval”, Indian Journal of Computer Science and Engineering, vol. 1, no. 3, pp. 207–211, 2010 [Online]. Available: http://www.ijcse.com/docs/IJCSE10-01-03-06.pdf.
View in Google Scholar

[28] [Online]. Available: https://photography.tutsplus.com/tutorials/what-is-image-sharpening--cms-26627.
View in Google Scholar

[29] T. Bouwmans et al., “On the role and the importance of features for background modeling and foreground detection”, Computer Science Review, vol. 28, pp. 26–91, 2018 (ISSN: 15740137).
View in Google Scholar

[30] [Online]. Available: https://en.wikipedia.org/wiki/Motion_analysis.
View in Google Scholar

[31] S. Harish, G. Hazrati, and J. C. Bansal, “Spider monkey optimization algorithm”, 2019.
View in Google Scholar

[32] B. R. Rajakumar, “Impact of static and adaptive mutation techniques on genetic algorithm”, International Journal of Hybrid Intelligent Systems, vol. 10, no. 1, pp. 11–22, 2013.
View in Google Scholar

[33] B. R. Rajakumar, “Static and adaptive mutation techniques for genetic algorithm: A systematic comparative analysis”, International Journal of Computational Science and Engineering, vol. 8, no. 2, pp. 180–193, 2013.
View in Google Scholar

[34] S. M. Swamy, B. R. Rajakumar, and I. R. Valarmathi, “Design of hybrid wind and photovoltaic power system using opposition-based genetic algorithm with Cauchy mutation”, in Proc. IET Chennai Fourth International Conference on Sustainable Energy and Intelligent Systems (SEISCON 2013), 2013.
View in Google Scholar

[35] A. George and B. R. Rajakumar, “APOGA: An adaptive population pool size-based genetic algorithm”, AASRI Procedia, vol. 4, pp. 288–296, 2013.
View in Google Scholar

[36] B. R. Rajakumar and A. George, “A new adaptive mutation technique for genetic algorithm”, in Proc. IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), 2012, pp. 1–7.
View in Google Scholar

[37] M. B. Wagh and N. Gomathi, “Improved GWO-CS algorithm-based optimal routing strategy in VANET”, Journal of Networking and Communication Systems, vol. 2, no. 1, pp. 34–42, 2019.
View in Google Scholar

[38] S. Halbhavi, S. F. Kodad, S. K. Ambekar, and D. Manjunath, “Enhanced invasive weed optimization algorithm with chaos theory for weightage-based combined economic emission dispatch”, Journal of Computational Mechanics, Power System and Control, vol. 2, no. 3, pp. 19–27, 2019.
View in Google Scholar

[39] A. N. Jadhav and N. Gomathi, “DIGWO: Hybridization of dragonfly algorithm with improved grey wolf optimization algorithm for data clustering”, Multimedia Research, vol. 2, no. 3, pp. 1–11, 2019.
View in Google Scholar

[40] [Online]. Available: https://www.kaggle.com/ming666/flicker8k-dataset.
View in Google Scholar

[41] S. Ding et al., “Image caption generation with high-level image features”, Pattern Recognition Letters, vol. 123, pp. 89–95, 2019.
View in Google Scholar

Downloads

Submitted

2023-05-07

Published

2022-12-30

Issue

No. 4 (2022)

Section

ARTICLES FROM THIS ISSUE

License

Copyright (c) 2022 Journal of Telecommunications and Information Technology

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

[1]
M. Kalla, A. Jain, A. Sharma, and R. Padate, “High-level and Low-level Feature Set for Image Caption Generation with Optimized Convolutional Neural Network”, JTIT, vol. 90, no. 4, pp. 67–75, Dec. 2022, doi: 10.26636/jtit.2022.164222.