Comparison of quality metric - TID2008 database

The plots below compare quality predictions of the selected metrics agains the TID2008 database [Ponomarenko N, Lukin V, Zelensky A, et al. TID2008-a database for evaluation of full-reference visual quality assessment metrics. Advances of Modern Radioelectronics. 2009;10:30-45].

The prediction error is reported as Spearman's and Kendall's correlation coeffcients (values closer to 1 or -1 indicate higher correlation), root-mean-square error (RMSE, the lower value means the lower prediction error) and chi2red statistic (RMSE scaled in the units of measurement standard deviation). Additionally, Spearman's rho and RMSE are reported for each distortion type separatelly. The solid line is the best fit of the logistic function. A single logistic function has been fitted for all distortion types, but only the LIVE database was used for fitting.

The original score values from the TID2008 database were rescaled to the 0-100 range for better consitency across the datasets. The mapping function log(1-SSIM) was used for both SSIM and MS-SSIM before fitting the logistic function as it provided better fits for these two metrics.

HDR-VDP-2

Spearman's rho = -0.8186
Kendall's tau = -0.6116
RMSE = 13.59
chi2red = 76.68
Logistic function: MOS=100/(1+exp(4.745*(x+0.1755)))
DistortionrhoRMSE
Additive noise in color -0.8228 7.69
Contrast change -0.4195 17.24
Image denoising -0.9245 10.72
Gaussian blur -0.8742 10.65
Additive Gaussian noise -0.8446 11.27
High frequency noise -0.895 17.74
Impulse noise -0.5914 22.56
JPEG2000 compression -0.9562 8.535
JPEG2000 transmission errors -0.611 12.47
JPEG compression -0.9191 7.717
JPEG transmission errors -0.8008 11.98
Local block-wise distortions -0.6584 23.69
Masked noise -0.7154 7.91
Mean shift (intensity shift) -0.6787 14.8
Non eccentricity pattern noise -0.7324 10.36
Quantization noise -0.8002 10.35
Spatially correlated noise -0.8692 10.75
MS-SSIM

Spearman's rho = -0.8086
Kendall's tau = -0.6112
RMSE = 13.25
chi2red = 67.14
Logistic function: MOS=100/(1+exp(0.8517*(x+2.819)))
DistortionrhoRMSE
Additive noise in color 0.8051 9.128
Contrast change 0.6252 27.93
Image denoising 0.9613 5.777
Gaussian blur 0.9547 11.55
Additive Gaussian noise 0.8003 12.85
High frequency noise 0.8685 14.64
Impulse noise 0.7005 12.38
JPEG2000 compression 0.9706 9.318
JPEG2000 transmission errors 0.8585 12.04
JPEG compression 0.935 7.45
JPEG transmission errors 0.8747 10.61
Local block-wise distortions 0.493 12.37
Masked noise 0.8096 12.67
Mean shift (intensity shift) 0.7702 18.66
Non eccentricity pattern noise 0.7507 11.85
Quantization noise 0.8455 8.681
Spatially correlated noise 0.8177 11.96
DCT-PSNR

Spearman's rho = 0.7183
Kendall's tau = 0.5253
RMSE = 16.61
chi2red = 116.8
Logistic function: MOS=100/(1+exp(-0.0948*(x-24.06)))
DistortionrhoRMSE
Additive noise in color 0.8183 14.36
Contrast change 0.4927 19.8
Image denoising 0.9535 7.805
Gaussian blur 0.9272 14.51
Additive Gaussian noise 0.848 16.96
High frequency noise 0.8783 22.68
Impulse noise 0.7363 12.93
JPEG2000 compression 0.9301 12.82
JPEG2000 transmission errors 0.7869 9.056
JPEG compression 0.9481 8.319
JPEG transmission errors 0.9121 13.43
Local block-wise distortions 0.8308 32.97
Masked noise 0.8189 11.84
Mean shift (intensity shift) 0.4203 18.33
Non eccentricity pattern noise 0.7942 21.09
Quantization noise 0.7843 13.81
Spatially correlated noise 0.7766 12.44
SSIM

Spearman's rho = -0.6052
Kendall's tau = -0.436
RMSE = 18.37
chi2red = 148.4
Logistic function: MOS=100/(1+exp(1.197*(x+1.62)))
DistortionrhoRMSE
Additive noise in color 0.8065 13.94
Contrast change 0.4698 17.5
Image denoising 0.9288 8.169
Gaussian blur 0.9387 9.789
Additive Gaussian noise 0.789 23.39
High frequency noise 0.8379 30.07
Impulse noise 0.7557 14.7
JPEG2000 compression 0.8888 14.75
JPEG2000 transmission errors 0.8395 10.78
JPEG compression 0.8994 9.589
JPEG transmission errors 0.8216 13.58
Local block-wise distortions 0.8809 42.94
Masked noise 0.8151 13.66
Mean shift (intensity shift) 0.7193 15.92
Non eccentricity pattern noise 0.6935 16.71
Quantization noise 0.7862 11.04
Spatially correlated noise 0.8147 10.66
PSNR

Spearman's rho = 0.5239
Kendall's tau = 0.369
RMSE = 18.96
chi2red = 138.7
Logistic function: MOS=100/(1+exp(-0.1657*(x-26.19)))
DistortionrhoRMSE
Additive noise in color 0.9067 22.62
Contrast change 0.6787 33.11
Image denoising 0.9409 6.583
Gaussian blur 0.8653 9.368
Additive Gaussian noise 0.9104 14.92
High frequency noise 0.9323 25.34
Impulse noise 0.9177 11.89
JPEG2000 compression 0.8255 16.5
JPEG2000 transmission errors 0.7769 8.67
JPEG compression 0.899 10.11
JPEG transmission errors 0.7646 12.19
Local block-wise distortions 0.5966 6.833
Masked noise 0.8488 22.63
Mean shift (intensity shift) 0.7032 43.27
Non eccentricity pattern noise 0.5842 15.86
Quantization noise 0.8712 6.91
Spatially correlated noise 0.9233 6.357