Performance analysis of full-reference objective image and video quality assessment metrics

  • Boban P. Bondžulić Univerzitet odbrane u Beogradu, Vojna akademija, Katedra telekomunikacija i informatike
  • Boban Z. Pavlović Univerzitet odbrane u Beogradu, Vojna akademija, Katedra telekomunikacija i informatike
  • Vladimir S. Petrović Univerzitet u Novom Sadu, Fakultet tehničkih nauka
DOI: https://doi.org/10.5937/vojtehg66-12708
Keywords: JPEG compression, H.264 and H.265 video compression, objective image and video quality assessment,

Abstract


This paper presents the performance evaluation of image and video quality assessment metrics on two publicly available datasets with subjective quality ratings. In addition to the performance analysis at the global level – at the level of complete datasets, the paper presents the objective measures performance evaluation on subsets of signals inside them. The image dataset contains five subsets created by using different types of JPEG compression, while the video dataset contains six subsets of sequences – four created by compression of original sequences, and two subsets are with video signal transmission characteristic degradations. To determine the success of objective measures, i.e. comparison of subjective and objective quality scores, there were used measures accepted by the International Telecommunication Union – ITU (linear correlation coefficient, rank-order correlation coefficient, mean absolute error, root mean squared error and outlier ratio). It was shown that objective quality measures can reach a high level of agreement with the results of subjective tests on subsets of datasets. Objective measures performances depend on the type of degradation which significantly affects the performance at the complete dataset level. The difference in performances is more pronounced on video sequences due to considerable visual differences in sequences created by using compression, packet losses and additive Gaussian noise. Therefore, we can say that a universal objective measure, i.e. measure that is useful for different types of signal degradation, for different degradation levels, and for different applications currently does not exist.

References

Bondzulic, B., & Petrovic, V. 2011. Edge-based objective evaluation of image quality. In: IEEE International Conference on Image Processing (ICIP), Brussels, Belgium, pp.3305-3308. September 11-14. Available at: http://dx.doi.org/10.1109/ICIP.2011.6116378.

Bondžulić, B. 2016. Procena kvaliteta slike i videa kroz očuvanje informacija o gradijentu. Novi Sad: University in Novi Sad. Ph.D. thesis (in Serbian).

Bovik, A.C. 2010. Perceptual video processing: Seeing the future. Proceedings of the IEEE, 98(11), pp.1799-1803. Available at: http://dx.doi.org/10.1109/JPROC.2010.2068371.

Bovik, A.C. 2013. Automatic prediction of perceptual image and video quality. Proceedings of the IEEE, 101(9), pp.2008-2024. Available at: http://dx.doi.org/10.1109/JPROC.2013.2257632.

Chandler, D.M., & Hemami, S.S. 2007. VSNR: A wavelet-based visual signal-to-noise ratio for natural images. IEEE Transactions on Image Processing, 16(9), pp.2284-2298. pmid:17784602. Available at: http://dx.doi.org/10.1109/TIP.2007.901820.

De Simone, F., Goldmann, L., Baroncini, V., & Ebrahimi, T. 2009. Subjective evaluation of JPEG XR image compression. In: Proc. of SPIE, 7443, San Diego, CA, pp.1-12 74430L. August 02. Available at: http://dx.doi.org/10.1117/12.830714.

-International Telecommunication Union. 2004. ITU TUTORIAL: Objective perceptual assessment of video quality: Full reference television. Geneva, Switzerland.

-International Telecommunication Union. 2008. ITU-T Recommendation P.910: Subjective video quality assessment methods for multimedia applications. Geneva, Switzerland.

-International Telecommunication Union. 2012. ITU-R Recommendation BT.500-13: Methodology for the subjective assessment of the quality of television pictures. Geneva, Switzerland.

-International Telecommunication Union. 2016. ITU-T Recommendation P.913: Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment. Geneva, Switzerland.

Larson, E.C., & Chandler, D.M. 2010. Most apparent distortion: Full reference image quality assessment and the role of strategy. Journal of Electronic Imaging, 19(1), pp.1-21, 011006. Available at: http://dx.doi.org/10.1117/1.3267105.

Narwaria, M., Lin, W., & Liu, A. 2012. Low-complexity video quality assessment using temporal quality variations. IEEE Transactions on Multimedia, 14(3), pp.525-535. Available at: http://dx.doi.org/10.1109/TMM.2012.2190589.

Pinson, M.H., & Wolf, S. 2004. A new standardized method for objectively measuring video quality. IEEE Transactions on Broadcasting, 50(3), pp.312-322. Available at: http://dx.doi.org/10.1109/TBC.2004.834028.

Seshadrinathan, K., & Bovik, A.C. 2010. Motion tuned spatio-temporal quality assessment of natural videos. IEEE Transactions on Image Processing, 19(2), pp.335-350, pmid:19846374. Available at: http://dx.doi.org/10.1109/TIP.2009.2034992.

Sheikh, H.R., & Bovik, A.C. 2006. Image information and visual quality. IEEE Transactions on Image Processing, 15(2), pp.430-444, pmid:16479813. Available at: http://dx.doi.org/10.1109/TIP.2005.859378.

Vu, P.V., & Chandler, D.M. 2014. ViS3: An algorithm for video quality assessment via analysis of spatial and spatiotemporal slices. Journal of Electronic Imaging, 23(1), pp.1-24, 01316. Available at: http://dx.doi.org/10.1117/1.JEI.23.1.0130.

Wang, Z., & Bovik, A.C. 2002. A universal image quality index. IEEE Signal Processing Letters, 9(3), pp.81-84. Available at: http://dx.doi.org/10.1109/97.995823.

Wang, Z., & Bovik, A.C. 2011. Reduced- and no-reference image quality assessment. IEEE Signal Processing Magazine, 28(6), pp.29-40. Available at: http://dx.doi.org/10.1109/MSP.2011.942471.

Wang, Z., Bovik, A.C., & Lu, L. 2002. Why is image quality assessment so difficult? In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Orlando, FL, pp.3313-3316. May 13-17. Available at: http://dx.doi.org/10.1109/ICASSP.2002.5745362.

Wang, Z., Bovik, A.C., Sheikh, H.R., & Simoncelli, E.P. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), pp.600-612, pmid:15376593. Available at: http://dx.doi.org/10.1109/TIP.2003.819861.

Wang, Z., Simoncelli, E.P., & Bovik, A.C. 2003. Multi-scale structural similarity for image quality assessment. In: Conference Record of the 37th Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, pp.1398-1402. November 9-12. Available at: http://dx.doi.org/10.1109/ACSSC.2003.1292216.

Published
2018/03/16
Issue
Vol. 66 No. 2 (2018): April – June
Section
Original Scientific Papers

Proposed Creative Commons Copyright Notices

Proposed Policy for Military Technical Courier (Journals That Offer Open Access)

Authors who publish with this journal agree to the following terms:
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.

  1. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  2. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).