Comparison of quality metric - TID2008 database
The plots below compare quality predictions of the selected metrics
agains the TID2008 database [Ponomarenko N, Lukin V, Zelensky A, et
al. TID2008-a database for evaluation of full-reference visual quality
assessment metrics. Advances of Modern
Radioelectronics. 2009;10:30-45].
The prediction error is reported as Spearman's and Kendall's
correlation coeffcients (values closer to 1 or -1 indicate higher
correlation), root-mean-square error (RMSE, the lower value means the
lower prediction error) and chi2red statistic
(RMSE scaled in the units of measurement standard
deviation). Additionally, Spearman's rho and RMSE are reported for
each distortion type separatelly. The solid line is the best fit of
the logistic function. A single logistic function has been fitted for
all distortion types, but only the LIVE database was used for
fitting.
The original score values from the TID2008 database were rescaled
to the 0-100 range for better consitency across the datasets. The
mapping function log(1-SSIM) was used for both SSIM and MS-SSIM before
fitting the logistic function as it provided better fits for these two
metrics.
HDR-VDP-2
Spearman's rho = -0.8186
Kendall's tau = -0.6116
RMSE = 13.59
chi
2red = 76.68
Logistic function: MOS=100/(1+exp(4.745*(x+0.1755)))
| Distortion | rho | RMSE |
|---|
| Additive noise in color | -0.8228 | 7.69 |
| Contrast change | -0.4195 | 17.24 |
| Image denoising | -0.9245 | 10.72 |
| Gaussian blur | -0.8742 | 10.65 |
| Additive Gaussian noise | -0.8446 | 11.27 |
| High frequency noise | -0.895 | 17.74 |
| Impulse noise | -0.5914 | 22.56 |
| JPEG2000 compression | -0.9562 | 8.535 |
| JPEG2000 transmission errors | -0.611 | 12.47 |
| JPEG compression | -0.9191 | 7.717 |
| JPEG transmission errors | -0.8008 | 11.98 |
| Local block-wise distortions | -0.6584 | 23.69 |
| Masked noise | -0.7154 | 7.91 |
| Mean shift (intensity shift) | -0.6787 | 14.8 |
| Non eccentricity pattern noise | -0.7324 | 10.36 |
| Quantization noise | -0.8002 | 10.35 |
| Spatially correlated noise | -0.8692 | 10.75 |
MS-SSIM
Spearman's rho = -0.8086
Kendall's tau = -0.6112
RMSE = 13.25
chi
2red = 67.14
Logistic function: MOS=100/(1+exp(0.8517*(x+2.819)))
| Distortion | rho | RMSE |
|---|
| Additive noise in color | 0.8051 | 9.128 |
| Contrast change | 0.6252 | 27.93 |
| Image denoising | 0.9613 | 5.777 |
| Gaussian blur | 0.9547 | 11.55 |
| Additive Gaussian noise | 0.8003 | 12.85 |
| High frequency noise | 0.8685 | 14.64 |
| Impulse noise | 0.7005 | 12.38 |
| JPEG2000 compression | 0.9706 | 9.318 |
| JPEG2000 transmission errors | 0.8585 | 12.04 |
| JPEG compression | 0.935 | 7.45 |
| JPEG transmission errors | 0.8747 | 10.61 |
| Local block-wise distortions | 0.493 | 12.37 |
| Masked noise | 0.8096 | 12.67 |
| Mean shift (intensity shift) | 0.7702 | 18.66 |
| Non eccentricity pattern noise | 0.7507 | 11.85 |
| Quantization noise | 0.8455 | 8.681 |
| Spatially correlated noise | 0.8177 | 11.96 |
DCT-PSNR
Spearman's rho = 0.7183
Kendall's tau = 0.5253
RMSE = 16.61
chi
2red = 116.8
Logistic function: MOS=100/(1+exp(-0.0948*(x-24.06)))
| Distortion | rho | RMSE |
|---|
| Additive noise in color | 0.8183 | 14.36 |
| Contrast change | 0.4927 | 19.8 |
| Image denoising | 0.9535 | 7.805 |
| Gaussian blur | 0.9272 | 14.51 |
| Additive Gaussian noise | 0.848 | 16.96 |
| High frequency noise | 0.8783 | 22.68 |
| Impulse noise | 0.7363 | 12.93 |
| JPEG2000 compression | 0.9301 | 12.82 |
| JPEG2000 transmission errors | 0.7869 | 9.056 |
| JPEG compression | 0.9481 | 8.319 |
| JPEG transmission errors | 0.9121 | 13.43 |
| Local block-wise distortions | 0.8308 | 32.97 |
| Masked noise | 0.8189 | 11.84 |
| Mean shift (intensity shift) | 0.4203 | 18.33 |
| Non eccentricity pattern noise | 0.7942 | 21.09 |
| Quantization noise | 0.7843 | 13.81 |
| Spatially correlated noise | 0.7766 | 12.44 |
SSIM
Spearman's rho = -0.6052
Kendall's tau = -0.436
RMSE = 18.37
chi
2red = 148.4
Logistic function: MOS=100/(1+exp(1.197*(x+1.62)))
| Distortion | rho | RMSE |
|---|
| Additive noise in color | 0.8065 | 13.94 |
| Contrast change | 0.4698 | 17.5 |
| Image denoising | 0.9288 | 8.169 |
| Gaussian blur | 0.9387 | 9.789 |
| Additive Gaussian noise | 0.789 | 23.39 |
| High frequency noise | 0.8379 | 30.07 |
| Impulse noise | 0.7557 | 14.7 |
| JPEG2000 compression | 0.8888 | 14.75 |
| JPEG2000 transmission errors | 0.8395 | 10.78 |
| JPEG compression | 0.8994 | 9.589 |
| JPEG transmission errors | 0.8216 | 13.58 |
| Local block-wise distortions | 0.8809 | 42.94 |
| Masked noise | 0.8151 | 13.66 |
| Mean shift (intensity shift) | 0.7193 | 15.92 |
| Non eccentricity pattern noise | 0.6935 | 16.71 |
| Quantization noise | 0.7862 | 11.04 |
| Spatially correlated noise | 0.8147 | 10.66 |
PSNR
Spearman's rho = 0.5239
Kendall's tau = 0.369
RMSE = 18.96
chi
2red = 138.7
Logistic function: MOS=100/(1+exp(-0.1657*(x-26.19)))
| Distortion | rho | RMSE |
|---|
| Additive noise in color | 0.9067 | 22.62 |
| Contrast change | 0.6787 | 33.11 |
| Image denoising | 0.9409 | 6.583 |
| Gaussian blur | 0.8653 | 9.368 |
| Additive Gaussian noise | 0.9104 | 14.92 |
| High frequency noise | 0.9323 | 25.34 |
| Impulse noise | 0.9177 | 11.89 |
| JPEG2000 compression | 0.8255 | 16.5 |
| JPEG2000 transmission errors | 0.7769 | 8.67 |
| JPEG compression | 0.899 | 10.11 |
| JPEG transmission errors | 0.7646 | 12.19 |
| Local block-wise distortions | 0.5966 | 6.833 |
| Masked noise | 0.8488 | 22.63 |
| Mean shift (intensity shift) | 0.7032 | 43.27 |
| Non eccentricity pattern noise | 0.5842 | 15.86 |
| Quantization noise | 0.8712 | 6.91 |
| Spatially correlated noise | 0.9233 | 6.357 |