Comparison of quality metrics for the LIVE database
The plots below compare quality predictions of the selected metrics
agains the LIVE image quality database [Sheikh HR, Sabir MF, Bovik
AC. A Statistical Evaluation of Recent Full Reference Image Quality
Assessment Algorithms. IEEE Transactions on Image
Processing. 2006;15(11):3440-3451].
The prediction error is reported as Spearman's and Kendall's
correlation coeffcients (values closer to 1 or -1 indicate higher
correlation), root-mean-square error (RMSE, the lower value means the
lower prediction error) and chi2red statistic
(RMSE scaled in the units of measurement standard
deviation). Additionally, Spearman's rho and RMSE are reported for
each distortion type separatelly. The solid line is the best fit of
the logistic function. A single logistic function has been fitted for
all distortion types, but only the LIVE database was used for
fitting.
The original score values from the LIVE database were inverted
(MOS=100-DMOS) for more intuitive interpretation and better consitency
across the datasets. The mapping function log(1-SSIM) was used for
both SSIM and MS-SSIM before fitting the logistic function as it
provided better fits for these two metrics.
HDR-VDP-2
Spearman's rho = -0.9544
Kendall's tau = -0.8077
RMSE = 8.5
chi
2red = 0.5869
Logistic function: MOS=100/(1+exp(4.745*(x+0.1755)))
| Distortion | rho | RMSE |
|---|
| Fast-fading wireless | -0.9483 | 8.588 |
| Gaussian blur | -0.9554 | 6.709 |
| JPEG2000 compression | -0.9486 | 7.976 |
| JPEG compression | -0.9744 | 8.56 |
| White noise | -0.9671 | 10.34 |
MS-SSIM
Spearman's rho = -0.9486
Kendall's tau = -0.8008
RMSE = 9.07
chi
2red = 0.6948
Logistic function: MOS=100/(1+exp(0.8517*(x+2.819)))
| Distortion | rho | RMSE |
|---|
| Fast-fading wireless | 0.9429 | 9.914 |
| Gaussian blur | 0.9543 | 8.202 |
| JPEG2000 compression | 0.9632 | 6.9 |
| JPEG compression | 0.982 | 9.783 |
| White noise | 0.9756 | 10.34 |
DCT-PSNR
Spearman's rho = 0.9291
Kendall's tau = 0.7629
RMSE = 11.63
chi
2red = 1.251
Logistic function: MOS=100/(1+exp(-0.0948*(x-24.06)))
| Distortion | rho | RMSE |
|---|
| Fast-fading wireless | 0.9485 | 10.72 |
| Gaussian blur | 0.888 | 14.73 |
| JPEG2000 compression | 0.9589 | 7.121 |
| JPEG compression | 0.9832 | 10.4 |
| White noise | 0.9667 | 14.23 |
SSIM
Spearman's rho = -0.9065
Kendall's tau = -0.7252
RMSE = 12.11
chi
2red = 1.193
Logistic function: MOS=100/(1+exp(1.197*(x+1.62)))
| Distortion | rho | RMSE |
|---|
| Fast-fading wireless | 0.9415 | 9.96 |
| Gaussian blur | 0.895 | 10.38 |
| JPEG2000 compression | 0.9354 | 8.834 |
| JPEG compression | 0.9517 | 13.05 |
| White noise | 0.9625 | 16.95 |
PSNR
Spearman's rho = 0.8746
Kendall's tau = 0.6828
RMSE = 13.66
chi
2red = 1.427
Logistic function: MOS=100/(1+exp(-0.1657*(x-26.19)))
| Distortion | rho | RMSE |
|---|
| Fast-fading wireless | 0.8874 | 13.54 |
| Gaussian blur | 0.7876 | 11.97 |
| JPEG2000 compression | 0.8886 | 12.28 |
| JPEG compression | 0.9069 | 15.56 |
| White noise | 0.9856 | 14.61 |