Logo PTI Logo FedCSIS

Proceedings of the 21st Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 47

Improving Multimodal Plant Disease Captioning through Adaptive Query Token Selection

, , ,

DOI: http://dx.doi.org/10.15439/2026F1013

Citation: Rabson Zimba, , ,

Full text

Abstract. Recent research has focused on vision-language models (VLMs) that demonstrated high performance, offering significant improvement across a broad range of applications, including agriculture in the era of rapid climate change. However, an important yet underexplored component of VLMs is the role of the query tokens in cross-modal alignment. Most existing VLMs adopt fixed query token configuration, and it is unclear whether this is optimal for crop disease diagnostic tasks. This study systematically evaluates the role of query tokens in VLM-based crop disease captioning, employing a systematic analysis to identify trade-offs and optimal configurations across semantic alignment, lexical accuracy and computational efficiency. Experiments were conducted on the LeafNet dataset using a multimodal framework integrating SigLIP and Qwen2.5 VL. The findings provide clear evidence that both captioning performance and computational cost vary considerably with query-token capacity, showing that no single fixed setup offers the best balance across all objectives. This study highlights the value of adaptive design in multimodal learning and position query token optimisation as a promising direction for future research.

References

  1. Saleem, M.H., Potgieter, J., and Arif, K.M.: ‘Automation in Agriculture by Machine and Deep Learning Techniques: A Review of Recent Developments’, Precision Agriculture, 2021, 22, (6), pp. 2053–2091. DOI: 10.1007/s11119-021-09806-x
  2. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J.: ‘Learning transferable visual models from natural language supervision’, in Editor (Ed.)^(Eds.): ‘Book Learning transferable visual models from natural language supervision’ (PmLR, 2021, edn.), pp. 8748–8763.
  3. Awais, M., Naseer, M., Khan, S., Anwer, R.M., Cholakkal, H., Shah, M., Yang, M.H., and Khan, F.S.: ‘Foundation Models Defining a New Era in Vision: A Survey and Outlook’, IEEE Trans Pattern Anal Mach Intell, 2025, 47, (4), pp. 2245–2264. DOI: 10.1109/TPAMI.2024.3506283
  4. Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., and Hoi, S.: ‘Instructblip: Towards general-purpose vision-language models with instruction tuning’, Advances in neural information processing systems, 2023, 36, pp. 49250–49267.
  5. Quoc, K.N., Dao, P.D., and Quach, L.-D.: ‘LeafNet: A Large-Scale Dataset and Comprehensive Benchmark for Foundational Vision-Language Understanding of Plant Diseases’, arXiv preprint https://arxiv.org/abs/2602.13662, 2026
  6. Dong, J., Yao, Y., Fuentes, A., Jeong, Y., Yoon, S., and Park, D.S.: ‘Visual information guided multi-modal model for plant disease anomaly detection’, Smart Agricultural Technology, 2024, 9, pp. 100568. DOI: 10.1016/j.atech.2024.100568
  7. Ghosh, D., Ghosh, S., Jana, N.D., and Mallipeddi, R.: ‘Intelligent neural architecture search via Taguchi design and language model-based differential evolution for agricultural image recognition’, Expert Systems with Applications, 2026, 309, pp. 131135. DOI: 10.1016/j.eswa.2026.131135
  8. Bie, Y., Xu, G., and Wang, Y.: ‘Adapting Vision-Language Models for Precision Agriculture: A Study on Crop Segmentation based on UAV Remote Sensing Data’, in Editor (Ed.)^(Eds.): ‘Book Adapting Vision-Language Models for Precision Agriculture: A Study on Crop Segmentation based on UAV Remote Sensing Data’ (2025, edn.), pp. 1–6. DOI: 10.1109/Agro-Geoinformatics66479.2025.11136213
  9. Upadhyay, A., Chandel, N.S., Singh, K.P., Chakraborty, S.K., Nandede, B.M., Kumar, M., Subeesh, A., Upendar, K., Salem, A., and Elbeltagi, A.: ‘Deep learning and computer vision in plant disease detection: a comprehensive review of techniques, models, and trends in precision agriculture’, Artificial Intelligence Review, 2025, 58, (3), pp. 92. DOI: 10.1007/s10462-024-11100-x
  10. Xu, C., Zhao, L., Wen, H., Zhang, Y., Wang, L., and Zhang, L.: ‘Intelligent management of crop diseases and pests in multiscale and multimodal complex scenarios: Technologies, applications, and prospects’, Computers and Electronics in Agriculture, 2026, 244, pp. 111443. DOI: 10.1016/j.compag.2026.111443
  11. Shao, K., Keda, T., Zhang, K., Feng, S., Cai, M., Shang, Y., You, H., Qin, C., Sui, Y., and Wang, H.: ‘A Survey of Token Compression for Efficient Multimodal Large Language Models’, Transactions on Machine Learning Research, 2026
  12. Jin, Y., Li, J., Gu, T., Liu, Y., Zhao, B., Lai, J., Gan, Z., Wang, Y., Wang, C., Tan, X., and Ma, L.: ‘Efficient multimodal large language models: a survey’, Visual Intelligence, 2025, 3, (1), pp. 27. DOI: 10.1007/s44267-025-00099-6
  13. Cai, K., Zhang, J., Yang, J., Fan, Y., Xie, P., Wang, J., and Wang, K.: ‘Flashvlm: Text-guided visual token selection for large multimodal models’, arXiv preprint https://arxiv.org/abs/2512.20561, 2025
  14. Wen, F., Guo, Y., Wang, J., Xiao, J., Zhou, Y., Shen, Y., Li, C., Jia, Q., and Zhang, Z.: ‘Improve MLLM Benchmark Efficiency Through Interview’, in Editor (Ed.)^(Eds.): ‘Book Improve MLLM Benchmark Efficiency Through Interview’ (2026, edn.), pp. 19892–19896. DOI: 10.1109/ICASSP55912.2026.11464672
  15. Chen, L., Zhao, X., Ding, K., Feng, W., Miao, C., Wang, Z., Guo, W., Wang, Y., Zheng, K., and Zhang, B.: ‘Beyond next-token alignment: Distilling multimodal large language models via token interactions’, arXiv preprint https://arxiv.org/abs/2602.09483, 2026
  16. K. M, S., Kumar, H., and Charangatt, J.V.: ‘Disease Classification and Report Generation from Chest X-Ray Images Using SigLIP and LLaVA’, in Editor (Ed.)^(Eds.): ‘Book Disease Classification and Report Generation from Chest X-Ray Images Using SigLIP and LLaVA’ (2025, edn.), pp. 1514–1518. DOI: 10.1109/TENCON66050.2025.11375249
  17. Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L.: ‘Sigmoid loss for language image pre-training’, in Editor (Ed.)^(Eds.): ‘Book Sigmoid loss for language image pre-training’ (2023, edn.), pp. 11975–11986.
  18. Goldfeld, Z., and Polyanskiy, Y.: ‘The Information Bottleneck Problem and its Applications in Machine Learning’, IEEE Journal on Selected Areas in Information Theory, 2020, 1, (1), pp. 19–38. DOI: 10.1109/jsait.2020.2991561
  19. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.: ‘Bleu: a method for automatic evaluation of machine translation’, in Editor (Ed.)^(Eds.): ‘Book Bleu: a method for automatic evaluation of machine translation’ (2002, edn.), pp. 311–318.
  20. Hrga, I., and Ivasic-Kos, M.: ‘Measuring the Sensitivity of Image Captioning Metrics to Caption Perturbations’, in Editor (Ed.)^(Eds.): ‘Book Measuring the Sensitivity of Image Captioning Metrics to Caption Perturbations’ (Springer Nature Singapore, 2024, edn.), pp. 1053–1063.
  21. Chen, J., Wu, P., Zhang, X., Xu, R., and Liang, J.: ‘Add-Vit: CNN-Transformer Hybrid Architecture for Small Data Paradigm Processing’, Neural Processing Letters, 2024, 56, (3), pp. 198. DOI: 10.1007/s11063-024-11643-8