Comparison of quality metric - TID2008 database

The plots below compare quality predictions of the selected metrics agains the TID2008 database [Ponomarenko N, Lukin V, Zelensky A, et al. TID2008-a database for evaluation of full-reference visual quality assessment metrics. Advances of Modern Radioelectronics. 2009;10:30-45].

The prediction error is reported as Spearman's and Kendall's correlation coeffcients (values closer to 1 or -1 indicate higher correlation), root-mean-square error (RMSE, the lower value means the lower prediction error) and chi2red statistic (RMSE scaled in the units of measurement standard deviation). Additionally, Spearman's rho and RMSE are reported for each distortion type separatelly. The solid line is the best fit of the logistic function. A single logistic function has been fitted for all distortion types, but only the LIVE database was used for fitting.

The original score values from the TID2008 database were rescaled to the 0-100 range for better consitency across the datasets. The mapping function log(1-SSIM) was used for both SSIM and MS-SSIM before fitting the logistic function as it provided better fits for these two metrics.

HDR-VDP-2-1

Spearman's rho = -0.8695
Kendall's tau = -0.6747
RMSE = 11.75
chi2red = 56.11
Logistic function: MOS=100/(1+exp(4.446*(x+0.8994)))
DistortionrhoRMSE
Additive noise in color -0.8444 5.431
Contrast change -0.5584 12.91
Image denoising -0.9499 8.848
Gaussian blur -0.8541 12.26
Additive Gaussian noise -0.8422 9.966
High frequency noise -0.899 13.06
Impulse noise -0.6696 13.69
JPEG2000 compression -0.9671 6.703
JPEG2000 transmission errors -0.7733 11.95
JPEG compression -0.9331 6.217
JPEG transmission errors -0.8288 12.95
Local block-wise distortions -0.6497 22.53
Masked noise -0.7 7.416
Mean shift (intensity shift) -0.6829 8.339
Non eccentricity pattern noise -0.7898 14.51
Quantization noise -0.8305 10.54
Spatially correlated noise -0.8504 10.81
HDR-VDP-2

Spearman's rho = -0.8186
Kendall's tau = -0.6116
RMSE = 13.59
chi2red = 76.68
Logistic function: MOS=100/(1+exp(4.745*(x+0.1755)))
DistortionrhoRMSE
Additive noise in color -0.8228 7.69
Contrast change -0.4195 17.24
Image denoising -0.9245 10.72
Gaussian blur -0.8742 10.65
Additive Gaussian noise -0.8446 11.27
High frequency noise -0.895 17.74
Impulse noise -0.5914 22.56
JPEG2000 compression -0.9562 8.535
JPEG2000 transmission errors -0.611 12.47
JPEG compression -0.9191 7.717
JPEG transmission errors -0.8008 11.98
Local block-wise distortions -0.6584 23.69
Masked noise -0.7154 7.91
Mean shift (intensity shift) -0.6787 14.8
Non eccentricity pattern noise -0.7324 10.36
Quantization noise -0.8002 10.35
Spatially correlated noise -0.8692 10.75
MS-SSIM

Spearman's rho = 0.8086
Kendall's tau = 0.6112
RMSE = 12.45
chi2red = 63.41
Logistic function: MOS=100/(1+exp(-11.19*(x-0.9037)))
DistortionrhoRMSE
Additive noise in color 0.8051 4.173
Contrast change 0.6252 26.54
Image denoising 0.9613 7.083
Gaussian blur 0.9547 11.39
Additive Gaussian noise 0.8003 8.816
High frequency noise 0.8685 13.17
Impulse noise 0.7005 11.98
JPEG2000 compression 0.9706 8.771
JPEG2000 transmission errors 0.8585 14.64
JPEG compression 0.935 8.898
JPEG transmission errors 0.8747 10.36
Local block-wise distortions 0.493 11.56
Masked noise 0.8096 6.269
Mean shift (intensity shift) 0.7702 15.83
Non eccentricity pattern noise 0.7507 10.13
Quantization noise 0.8455 11.43
Spatially correlated noise 0.8177 14.25
DCT-PSNR

Spearman's rho = 0.7183
Kendall's tau = 0.5253
RMSE = 16.61
chi2red = 116.8
Logistic function: MOS=100/(1+exp(-0.0948*(x-24.06)))
DistortionrhoRMSE
Additive noise in color 0.8183 14.36
Contrast change 0.4927 19.8
Image denoising 0.9535 7.805
Gaussian blur 0.9272 14.51
Additive Gaussian noise 0.848 16.96
High frequency noise 0.8783 22.68
Impulse noise 0.7363 12.93
JPEG2000 compression 0.9301 12.82
JPEG2000 transmission errors 0.7869 9.056
JPEG compression 0.9481 8.319
JPEG transmission errors 0.9121 13.43
Local block-wise distortions 0.8308 32.97
Masked noise 0.8189 11.84
Mean shift (intensity shift) 0.4203 18.33
Non eccentricity pattern noise 0.7942 21.09
Quantization noise 0.7843 13.81
Spatially correlated noise 0.7766 12.44
SSIM

Spearman's rho = 0.6052
Kendall's tau = 0.436
RMSE = 15.56
chi2red = 127
Logistic function: MOS=100/(1+exp(-4.816*(x-0.7533)))
DistortionrhoRMSE
Additive noise in color 0.8065 10.32
Contrast change 0.4698 13.95
Image denoising 0.9288 8.523
Gaussian blur 0.9387 7.853
Additive Gaussian noise 0.789 22.24
High frequency noise 0.8379 31.86
Impulse noise 0.7557 13.44
JPEG2000 compression 0.8888 15.67
JPEG2000 transmission errors 0.8395 10.38
JPEG compression 0.8994 10.82
JPEG transmission errors 0.8216 10.33
Local block-wise distortions 0.8809 28.71
Masked noise 0.8151 10.21
Mean shift (intensity shift) 0.7193 8.839
Non eccentricity pattern noise 0.6935 12.52
Quantization noise 0.7862 11.76
Spatially correlated noise 0.8147 10.63
PSNR

Spearman's rho = 0.5239
Kendall's tau = 0.369
RMSE = 18.96
chi2red = 138.7
Logistic function: MOS=100/(1+exp(-0.1657*(x-26.19)))
DistortionrhoRMSE
Additive noise in color 0.9067 22.62
Contrast change 0.6787 33.11
Image denoising 0.9409 6.583
Gaussian blur 0.8653 9.368
Additive Gaussian noise 0.9104 14.92
High frequency noise 0.9323 25.34
Impulse noise 0.9177 11.89
JPEG2000 compression 0.8255 16.5
JPEG2000 transmission errors 0.7769 8.67
JPEG compression 0.899 10.11
JPEG transmission errors 0.7646 12.19
Local block-wise distortions 0.5966 6.833
Masked noise 0.8488 22.63
Mean shift (intensity shift) 0.7032 43.27
Non eccentricity pattern noise 0.5842 15.86
Quantization noise 0.8712 6.91
Spatially correlated noise 0.9233 6.357