Logo PTI Logo FedCSIS

Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 43

Keypoint-based metric for evaluating image super-resolution quality

, , , , , ,

DOI: http://dx.doi.org/10.15439/2025F8791

Citation: Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS), M. Bolanowski, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 43, pages 255–263 ()

Full text

Abstract. Recent advances in single- and multi-image super- resolution have revealed the limitations of classical image sim- ilarity metrics (like peak signal-to-noise ratio), as they often fail to align with human perception when evaluating the visual quality of super-resolved outputs. In this paper, we explore how to exploit keypoint-based metrics to evaluate super-resolution image quality. Specifically, we explore two correlated metrics: (i) a multi- scale index proposal measure capturing salience of keypoints, and (ii) a repeatability metric quantifying how consistently the corresponding keypoints are identified in super-resolved and ground-truth images. Experiments on several simulated and real-world datasets show that the repeatability correlates with subjective judgments, and multi-scale index proposal can be helpful for difficult datasets when other metrics are insufficient.

References

  1. P. Benecki, M. Kawulok, D. Kostrzewa, and L. Skonieczny, “Evaluating super-resolution reconstruction of satellite images,” Acta Astronautica, vol. 153, pp. 15–25, 2018. https://dx.doi.org/10.1016/j.actaastro.2018.07.035
  2. A. B. Laguna, E. Riba, D. Ponsa, and K. Mikolajczyk, “Key.Net: Keypoint detection by handcrafted and learned CNN filters,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019. https://dx.doi.org/10.1109/ICCV.2019.00593 pp. 5835–5843.
  3. K. Mikolajczyk and C. Schmid, “A performance evaluation of local descriptors,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 10, pp. 1615–1630, 2005. https://dx.doi.org/10.1109/TPAMI.2005.188
  4. M. Kawulok, P. Benecki, S. Piechaczek, K. Hrynczenko, D. Kostrzewa, and J. Nalepa, “Deep learning for multiple-image super-resolution,” IEEE Geoscience and Remote Sensing Letters, vol. 17, no. 6, pp. 1062–1066, 2020. https://dx.doi.org/10.1109/LGRS.2019.2940483
  5. X. Lin, X. Liu, H. Yang, X. He, and H. Chen, “Perception- and fidelity-aware reduced-reference super-resolution image quality assessment,” IEEE Transactions on Broadcasting, vol. 71, no. 1, pp. 323–333, 2025. https://dx.doi.org/10.1109/TBC.2024.3475820
  6. J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE TIP, vol. 19, no. 11, pp. 2861–2873, 2010. https://dx.doi.org/10.1109/TIP.2010.2050625
  7. C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016. https://dx.doi.org/10.1109/TPAMI.2015.2439281
  8. C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in Computer Vision – ECCV 2016, B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Springer International Publishing, 2016. https://dx.doi.org/10.1007/978-3-319-46475-6_25 pp. 391–407.
  9. J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. https://dx.doi.org/10.1109/CVPR.2016.182 pp. 1646–1654.
  10. B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced deep residual networks for single image super-resolution,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017. https://dx.doi.org/10.1109/CVPRW.2017.151 pp. 1132–1140.
  11. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. https://dx.doi.org/10.1109/CVPR.2017.19 pp. 105–114.
  12. X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. C. Loy, “ESRGAN: Enhanced super-resolution generative adversarial networks,” in Computer Vision – ECCV 2018 Workshops, L. Leal-Taixé and S. Roth, Eds. Cham: Springer International Publishing, 2019. https://dx.doi.org/10.1007/978-3-030-11021-5_5 pp. 63–79.
  13. M. Zyrek and M. Kawulok, “Task-driven single-image super-resolution reconstruction of document scan,” in Proceedings of the 19th Conference on Computer Science and Intelligence Systems (FedCSIS), ser. Annals of Computer Science and Information Systems, M. Bolanowski, M. Ganzha, L. Maciaszek, M. Paprzycki, and D. Ślęzak, Eds., vol. 39. IEEE, 2024. https://dx.doi.org/10.15439/2024F7855 p. 259–264.
  14. Z. Bairi, K. B. Bey, O. Ben-Ahmed, A. Amamra, and A. Bradai, “Dual-path image reconstruction: Bridging vision transformer and perceptual compressive sensing networks,” in Proceedings of the 18th Conference on Computer Science and Intelligence Systems, ser. Annals of Computer Science and Information Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, and D. Ślęzak, Eds., vol. 35. PTI, 2023. https://dx.doi.org/10.15439/2023F978 pp. 347–354.
  15. C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4713–4726, 2023. https://dx.doi.org/10.1109/TPAMI.2022.3204461
  16. R. Schultz and R. Stevenson, “Improved definition video frame enhancement,” in 1995 International Conference on Acoustics, Speech, and Signal Processing, vol. 4, 1995. https://dx.doi.org/10.1109/ICASSP.1995.479905 pp. 2169–2172 vol.4.
  17. S. Farsiu, M. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Transactions on Image Processing, vol. 13, no. 10, pp. 1327–1344, 2004. https://dx.doi.org/10.1109/TIP.2004.834669
  18. M. Märtens, D. Izzo, A. Krzic, and D. Cox, “Super-resolution of proba-v images using convolutional neural networks,” Astrodynamics, vol. 3, no. 4, pp. 387–402, 2019. https://dx.doi.org/10.1007/s42064-019-0059-8
  19. M. Deudon, A. Kalaitzis, I. Goytom, M. R. Arefin, Z. Lin, K. Sankaran, V. Michalski, S. E. Kahou, J. Cornebise, and Y. Bengio, “HighRes-net: Recursive fusion for multi-frame super-resolution of satellite imagery,” arXiv preprint https://arxiv.org/abs/2002.06460, 2020. https://dx.doi.org/10.48550/arXiv.2002.06460
  20. A. Bordone Molini, D. Valsesia, G. Fracastoro, and E. Magli, “DeepSUM: Deep neural network for super-resolution of unregistered multitemporal images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 5, pp. 3644–3656, 2020. https://dx.doi.org/10.1109/TGRS.2019.2959248
  21. F. Salvetti, V. Mazzia, A. Khaliq, and M. Chiaberge, “Multi-image super resolution of remotely sensed images using residual attention deep neural networks,” Remote Sensing, vol. 12, no. 14, 2020. https://dx.doi.org/10.3390/rs12142207
  22. J. Li, Q. Lv, W. Zhang, B. Zhu, G. Zhang, and Z. Tan, “Multi-attention multi-image super-resolution transformer (mast) for remote sensing,” Remote Sensing, vol. 15, no. 17, 2023. https://dx.doi.org/10.3390/rs15174183
  23. T. An, X. Zhang, C. Huo, B. Xue, L. Wang, and C. Pan, “TR-MISR: Multiimage super-resolution based on feature fusion with transformers,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 1373–1388, 2022. https://dx.doi.org/10.1109/JS-TARS.2022.3143532
  24. T. Tarasiewicz and M. Kawulok, “A graph neural network for heterogeneous multi-image super-resolution,” Pattern Recognition Letters, vol. 189, pp. 214–220, 2025. https://dx.doi.org/10.1016/j.patrec.2025.01.028
  25. T. Tarasiewicz and M. Kawulok, “A graph attention network for real-world multi-image super-resolution,” Information Fusion, p. 103325, 2025. https://dx.doi.org/10.1016/j.inffus.2025.103325
  26. L. Shu, Q. Zhu, Y. He, W. Chen, and J. Yan, “A survey of superresolution image quality assessment,” Neurocomputing, vol. 621, p. 129279, 2025. https://dx.doi.org/10.1016/j.neucom.2024.129279
  27. H. Zhang, S. Su, Y. Zhu, J. Sun, and Y. Zhang, “Boosting no-reference super-resolution image quality assessment with knowledge distillation and extension,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023. doi: 10.1109/ICASSP49357.2023.10095465 pp. 1–5.
  28. Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. https://dx.doi.org/10.1109/CVPR.2018.00652 pp. 6228–6237.
  29. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. https://dx.doi.org/10.1109/CVPR.2018.00068 pp. 586–595.
  30. K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, “Image quality assessment: Unifying structure and texture similarity,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 5, pp. 2567–2581, 2022. https://dx.doi.org/10.1109/TPAMI.2020.3045810
  31. K. Zhang, T. Zhao, W. Chen, Y. Niu, J. Hu, and W. Lin, “Perception-driven similarity-clarity tradeoff for image super-resolution quality assessment,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 7, pp. 5897–5907, 2024. https://dx.doi.org/10.1109/TCSVT.2023.3341626
  32. L. Zhang, L. Zhang, X. Mou, and D. Zhang, “Fsim: A feature similarity index for image quality assessment,” IEEE Transactions on Image Processing, vol. 20, no. 8, pp. 2378–2386, 2011. https://dx.doi.org/10.1109/TIP.2011.2109730
  33. A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2013. https://dx.doi.org/10.1109/LSP.2012.2227726
  34. Q. Jiang, Z. Liu, K. Gu, F. Shao, X. Zhang, H. Liu, and W. Lin, “Single image super-resolution quality assessment: A real-world dataset, subjective studies, and an objective metric,” IEEE Transactions on Image Processing, vol. 31, pp. 2279–2294, 2022. https://dx.doi.org/10.1109/TIP.2022.3154588
  35. C. Ma, C.-Y. Yang, X. Yang, and M.-H. Yang, “Learning a noreference quality metric for single-image super-resolution,” Computer Vision and Image Understanding, vol. 158, pp. 1–16, 2017. https://dx.doi.org/10.1016/j.cviu.2016.12.009
  36. T. Zhao, Y. Lin, Y. Xu, W. Chen, and Z. Wang, “Learning-based quality assessment for image super-resolution,” IEEE Transactions on Multimedia, vol. 24, pp. 3570–3581, 2022. https://dx.doi.org/10.1109/TMM.2021.3102401
  37. P. Kowaleczko, T. Tarasiewicz, M. Ziaja, D. Kostrzewa, J. Nalepa, P. Rokita, and M. Kawulok, “A real-world benchmark for sentinel-2 multi-image super-resolution,” Scientific Data, vol. 10, no. 1, p. 644, 2023. https://dx.doi.org/10.1038/s41597-023-02538-9
  38. D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2, 2001. https://dx.doi.org/10.1109/ICCV.2001.937655 pp. 416–423 vol.2.
  39. C. G. Harris and M. J. Stephens, “A combined corner and edge detector,” in Alvey Vision Conference, 1988.