[IEEE 2013 International Conference on Communications and Information Technology (ICCIT) - Beirut, Lebanon (2013.06.19-2013.06.21)] 2013 Third International Conference on Communications

Reduced-Reference Quality Assessment ofComputer-Generated Images based on RVM.

Joseph ConstantinLaboratoire de Physique Appliquée,Dpt de Mathématiques Appliquées,PR2N, UL, Faculté des sciences 2,Fanar, BP 90656, Jdeidet, LibanEmail: [email protected]

Samuel Delepoulle, André Bigandand Christophe Renaud

LISIC, ULCOCalais Cedex, 62228, France

Email: [email protected]

Abstract—Reduced-reference image quality assessment needsno prior knowledge of reference image but only a minimal knowl-edge about processed images. A new reduced-reference imagequality measure, based on Relevance Vector Machine (RVM),using a supervised learning framework and synthetic imagesis proposed. This new metric is compared with experimentalpsycho-visual data. A recently performed psycho-visual exper-iment provides psycho-visual scores on some synthetic images,and comprehensive testing demonstrates the good consistencybetween these scores and the quality measures we obtain. Theproposed measure has been too compared with close methodslike RBF, MLP and SVM and gives satisfactory performance.Index Terms—Computer graphics; computer-generated im-

ages; Reduced-reference image quality metric; Relevance vectormachine; Supervised learning.

I. INTRODUCTIONImage quality measures are very important to characterize

images visual quality. They are of great interest in imagecompression (JPEG models) and in image synthesis. In thelast decade numerous methods have been proposed but theresulting models are limited in practice and they are still ininvestigation. At the moment, the classical model to character-ize image quality remains psycho-visual experiments (Humanin the loop experiment [1]).This is particularly the case in image synthesis using globalillumination methods. The main goal of global illuminationmethods is to produce synthetic images with photorealisticquality. For this purpose photon propagations and light interac-tions with matter have to be accurately simulated. Stochasticmethods were proposed for more than 20 years in order toreach these goals. They are generally based on the Path Tracingmethod proposed by Kajiya [2] where stochastic paths aregenerated from the camera point of view towards the 3Dscene. Because paths are randomly chosen the light gatheringcan change greatly from one path to another generatinghigh frequency color variations through the image [3]. TheMonte Carlo theory however ensures that this process willconverge to the correct image when the number of samples(the paths) grows. But no information is available about thenumber of samples that are really required for the image

being considered as visually satisfactory. Indeed the final useof these images is to be seen by human observers who aregenerally very sensitive to any image artefact. The humanvisual system (HVS) is endowed with powerful performancesbut is a very complex process! Consequently and due to thehigh computational cost of global illumination algorithms,perception-driven approaches were proposed. The main ideaof such approaches is to replace the human observer bya vision model. By mimicking HVS such techniques canprovide important improvements for rendering. They can beused for driving rendering algorithms to visually satisfactoryimages and to focus on visually important features [4]–[6].Most HVS models provide interesting results but are complexand still incomplete due to the internal system complexityand its partial knowledge. They generally require relativelylong computation times and are often difficult to use and toparameterize.

II. RELATED WORKS

Image quality assessment models are usually classified intothree models families in the literature. These three families are(see [7] for a more complete presentation):

• Full reference models that use the original version of theimage for the quality assessment of the processed version(as well-known PSNR and SSIM). These models are themost used methods to evaluate images quality. They areeasy to compute in real time and correlated with humansubjective appreciation, so they could be interesting forour purpose. Unfortunately, it is obvious to understandthat image synthesis is proceeded without the final wantedimage at the beginning of the process, so we have toinvestigate other kind of metric.

• No-reference models that evaluate the quality of imageswithout access to reference images (some recent papers[8], [9] proposed no-reference quality assessment ofJPEG images; although the authors obtained good results,these reported quality measures have their limitations).Due to the image synthesis environment, we tried to

978-1-4673-5307-6/13/$31.00 ©2013 IEEE978-1-4673-5307-6/13/$31.00 ©2013 IEEE

The 3rd International Conference on Communications and Information Technology (ICCIT-2013): Wireless Communicationsand Signal Processing, Beirut


320

propose a new method in that sense with some success[10]. A complete framework has to be yet defined.

• Reduced-reference models: the processed image is ana-lyzed using some relevant information to calculate thequality of the result image. This model seems to beparticularly interesting for our study as we show in thispaper.

A reference image quality measure is often expensive toobtain (or not available). So, this paper focuses on the useof a new perceptual index to replace psycho-visual index inthe perception-driven model. We propose a novel reduced-reference image quality, based on Relevance Vector Machine(RVM, to modelize uncertainty brought by noises affectingthe image synthesis). Relevance Vector Machine has beenstudied by Tipping ( [11]). In this paper, Tipping introducedthe principles of Bayesian inference in a machine learningcontext, with a particular emphasis on the importance ofmarginalisation for dealing with uncertainty. This RVM isapplicable for regression and classification and provides sparseBayesian models interesting in image processing.In this paper we will focus on gray-scale images (in fact ’L’component of ’Lab’ color images, since noise only affects ’L’component, [12]) and use RVM for image quality evaluation.The paper is organized as follows: section 2 describes theexperimental setup we use, section 3 briefly describes theRVM, section 4 introduces image quality evaluation usingRVM. Finally, the paper is summarized with some conclusionsin section 5.

III. EXPERIMENTAL SETUP

Unbiased Global Illumination (G.I.) methods use randomlychosen paths for sampling the illumination of visible objects.This process generates stochastic noise which is perceptibleby any human observer. Image denoising techniques, used aposteriori, are largely present in the literature [13]–[15]. Noisemodels and noise estimation from images are however moredifficult. Anyway these models are based on theoretical modelsof noise like additive white noise. However in G.I. algorithms,noise is not additive and arises from an unknown randomdistribution function. To our knowledge there is no existingmodel able to detect and to quantify stochastic visible noisein an image. We detail in the following the different steps ofour approach for solving this problem by using a new imagequality measure.

A. Overview

Our goal is to mimic the human visual detection of noiseby way of a reduced-reference image quality measure. So it isnecessary to provide to the proposed model some examples ofwhat human judgment consider to be noisy images or noiselessones. After validation on all these examples the method willgenerate a model that will then be used on images that haveto be analyzed.

Fig. 1. Two used (reference) scenes.

B. Data acquisition1) The images dataset: The model is built on data corre-

sponding to images of globally illuminated scenes. We used(as a first approach) a Path Tracing with next event algorithm[3] which computes several images from the same point ofview by adding successively N new samples1 equally foreach pixel. For each scene and each point of view we thushave several images available, the first ones being stronglynoisy and the last ones being converged. The images werecomputed at 512 ! 512 resolution, the number of additionalsamples between two successive images was N = 100 and12 scenes were used. The largest number of samples perpixel was set at 10.000 which appeared to be sufficient forgenerating visually converged images. Figure 1 presents 2 ofthese scenes that were used during the validation stage ofthe model. These scenes highlight different illuminations andvarious geometrical and textures complexities.

C. Psycho-visual scores acquisitionBecause we have to evaluate the noise level present in

each generated image, some experiments were necessary inorder to specify the noise threshold that is used as stoppingcriterion in images synthesis. But considering the entire imagefor noise thresholding has two main drawbacks: on one hand,it requires evaluation methods to work on very large data sets;this has been experimentally shown to reduce their learningefficiency. On the other hand the noise is generally not equallyperceived by human observers through any part of an image;noise thresholds are thus different for each location in eachimage and the use of a global threshold would reduce theefficiency of the approach by requiring the same number ofsamples to be computed for each pixel of the image. We thusdefined a very simple protocol in which pairs of images arepresented to the observer. One of this image is called thereference image and has been computed with Nr = 10.000samples per pixel. The second image so called the test imageis built as a stack of images, from very noisy ones above toconverged ones below: by calling Ni the number of samplesin the stack’s image i, with i = 100 at the top of the stackand i = max at its bottom, we thus ensure the property"i # [100,max[, Ni < Ni+100 $ Nr. Each of these images

1In the following we will call sample a stochastic path between the viewpoint and a light source.



321

are opaque and virtually cut into non-overlapping blocks ofsize 128! 128 over the entire image. For the used 512! 512test images we thus get 16 different blocks (clockwise sortedfrom 0 to 15, top image of figure 3) for each of the stack’simages. During the experiments the observer is asked tomodify the quality of the noisy image by pointing the areaswhere differences are perceived between the current imageand its reference one. Each point-and-click operation thencauses the selection and the display of the correspondingi + 100 level block thus visually reducing noise in thisimage’s subpart. This operation is done until the observerconsiders that the two images are visually identical. Note thatfor reducing experiment artefacts this operation is reversiblemeaning that an observer is able to go down or up into theimages stack. The pair of images that is presented to theobserver is chosen randomly but we ensure that each pair willbe presented two times. Obviously the block grid is not visibleand all the observers worked in the same conditions (samedisplay with identical luminance tuning, same illuminationconditions, ...). The results were recorded for 33 differentobservers and we computed the average number of samplesN that are required for each subimage to be perceived asidentical to the reference one by 95% of the observers. We gotexperimentally N # [1441, 6631] with often large differencesbetween subimages of the same image (see figure 3). So nowwe present attempts of automatic image quality index wepropose.

D. Previous work: automatic stopping criterion based on SVMIn a previous work, we first applied SVMs (Support Vector

Machine or RVM) to the perception of noise towards anautomatic stopping criterion [16]. Let us recall some basicproperties of SVMs. SVM makes predictions based on thefunction y(x) defined over the input space, and ’learning isthe process of inferring this function (often parameters of thefunction). Popular set of candidates for y(x) is that of theform:

y(x;w) =M!

i=1

[wi.!(x)] =N

!

i=1

[wi.K(x, xi) + w0] (1)

where !(x) are basic functions, M the number of supportvectors, wi are weights (adjustable parameters). SVMs arebased on basis functions K(x, xi), named kernel functions.The key feature of the SVM is that, in classification case,its target function attempts to minimize a measure of erroron the training set while maximising the ’margin’ betweenthe two classes (in the feature space defined by the kernel).Learning methods like RBF (Radial Basis Function) neuralnetwork or multi-layer neural networks (MLP) are particularcases of SVM (they are built with different kernels). So theywill too be considered for comparisons in the following.However, SVMs are limited to small sizes of images, due tothe great number of kernels (neurons) required (the numberof support vectors required grows linearly with the size of thetraining size). Predictions made by SVMs are not probabilistic:this is crucial in classification where posterior probabilities of

class memebership are necessary to adapt to varying class pri-ors and asymetric misclassification costs (SVMs leran texturebut not noise!). It is necessary to estimate the error/margintrade-off parameter C and the kernel function K(x, xi) mustsatisfy Mercer’s condition. Tipping [17] have shown that RVMare interesting in sparse Bayesian learning. Particularly, thenumber of kernels of RVMs drastically decreases comparedto SVMs. The obtained sparsity is interesting to investigatethe reduced-reference image quality model we propose now.

IV. RELEVANCE VECTOR MACHINEA. DefinitionThe major motivation of RVM implementation is to remove

the above limitations of SVMs. We now recall some resultsabout RVM.Given a data set of input-target pairs {xn, tn}N

n=1. The targetsare samples from the model with additive noise:

tn = y(xn;w) + "n (2)

where "n are independent samples from some noise processassumed to be mean-zero Gaussian with variance #2. Due tothe assumption of independence of the tn, the likeliwood ofthe complete data set can be written as:

p(T | W,# 2) ="

2$#2#!N/2

exp

$

%1

2

%

|| t % !.W ||

#

&2'

(3)where T = (t1, t2, ....tN )T , W = (w0, w1, ...., wN )T ,! = [!(x1),!(x2), ...!(xN )] and !(xN ) =[1,K(xn, x1),K(xn, x2), ...,K(xn, xn)]T .A classical zero-mean Gaussian prior distribution over W isdefined as follows:

p(W | %) =N(

i=0

N (wi | 0,%!1i ) (4)

where N is a Gaussian distribution over wi. This priordistribution is favourable to weak weights models.

B. Sparse Bayesian LearningHaving defined the prior, Bayesian inference proceeds by

computing the posterior over all unknowns given the data(from Bayes’rule, [17]):

p(T | W,%,# 2)

=p(T | W,# 2)p(W | %)

p(T | %,# 2)

= (2$)!(N+1)/2 |!

|!1/2 exp

$

%1

2(W % µ)T

!1!

(W % µ)

'

(5)

where the posterior covariance and mean are respectively:!

= (#!2!T ! + A)!1

µ = #!2!

!T T(6)

where A = diag(%0,%1, ....,%N ).



322

C. RVM constructionThe aim of the RVM consists in maximalization of the

marginal probability, that makes it possible the eliminationof useless parameters. This property is illustrated in Fig.2,using images of size 512x512, each image is divided in16 blocks. During learning phase, the input of the networkconsists in the difference between the processed image (withNi odd number of samples, while test images are thoseobtained with Ni even number of samples) and a noisy image(first image created during the synthesis process, with 100samples). This procedure makes it possible the detection ofnoise in images, and do not use reference image (which isobviously unavailable). Processed images (60 different imageswere used) are examples obtained with different noise levels(different number of sample Ni) and unnoisy images. Theoutput of the network is " % 1" if the image is considered’noisy’ or "+1" if image is considered non-affected with noise(human judgement). The network parameters are optimizedusing classical cross-validation (Fig.2-b).

(A) Complete structure of the proposed RVM

(B) Kernel width optimization for minimum relevant vectors

Fig. 2. RVM design.

V. THE DESIGN OF THE RVM IMAGE QUALITYEVALUATION

We propose to use the previous defined RVM to extract anew noise level index.

A. Proposed schemeThe noise level measure scheme is divided into two steps. In

the first one, we perform images learning procedure applied toa block of the processed image I (obtained with Ni samples).Then, in a second step, we use this network to test the resultingRVM.

B. AlgorithmThe implementation of image quality evaluation based on

RVM is given by the following algorithm:

Algorithm 1 RVM Image quality measureRequire: a M ! N gray-level image I, divided in 16 blocks

and presented to neural network of Figure 21: Initialize the values %i and #2

2: if Optimisation condition unsatisfied then3: Find values of µ and

)

4: Process values of & and estimate new values of % and#2

5: else6: Suppress weights associated to values µ = 07: end if8: Keep the vectors of the RVM associated wih values µ &= 09: Display result (image is noisy or not)

C. Experimental results with a synthetic imageIn order to test the performance of the proposed technique,

some results obtained with the synthetic image named "Bar",figure 3, are shown is this presentation (other images weretested and same behaviors were observed, so they are notpresented here due to the lack of space). This image iscomposed of homogeneous and noisy blocks and is interestingto present some results. We also present the results obtainedwith RBF, MLP and SVM (’C’ parameter optimized to 512)networks with the same image. The first figure represents the"Bar" image, the second one represents the learning averagequadratic error, the third one the test average quadratic errorand the fourth one the number of relevant vectors, supportvectors or hidden neurons for the four implemented methods.These curves exhibit the advantage of using RVM for this task,where RVM performances outperform the other equivalentnetworks. Particularly, learning average quadratic error is verylow, according to other methods. RVM makes it possible tolearn images uncertainty (as other Bayesian methods), wherasSVM, RBF and MLP are known to better learn shapes inimages. Another interesting key point is the low number ofrelevant vectors, which makes it possible to consider the wholeimage as input space (features space is not ). So we have nowto generalize and confirm these interesting results to a greatvariety of computer-generated images.

VI. CONCLUSIONThe central idea of this paper was to introduce the ap-

plication of RVM, to take into account uncertainty (noise)present at the image synthesis stage, and this idea seems to



323

(A) Reference Image

(B) Learning average quadratic error

(C) Test average quadratic error

(D) Number of relevant vectors, support vectors or hidden neurons forthe four implemented methods

Fig. 3. Original image and compared evolutions of different networks.

be very promising. This technique uses dramatically fewerbasis functions than a comparable SVM while offering anumber of additional advantages. These include the benefitsof probabilistic predictions (noise detection), automatic es-timation of ’nuisance’ parameters and the use of arbitrarybasis functions (non-’Mercer’ kernels). The biggest advantagefor our application is sparse Bayesian learning that makes itpossible to treat complete images (rather than image blocksof small size). In particular, more extensive investigationson the effect of parameters influencing the RVM are underinvestigation (to establish a link between these parametersand level and type of noise) towards an automatic noise

quantification and will be presented in the future. It is alsonecessary to establish a more complete framework with a hugenumber of images to generalize this new concept.

REFERENCES[1] O. Faugeras, “Digital color image processing within the framework of

a human visual model,” IEEE Trans. on ASSP, vol. 27, pp. 380–393,1979.

[2] J. Kajiya, “The Rendering Equation,” ACM Computer Graphics, vol. 20,no. 4, pp. 143–150, Août 1986.

[3] P. Shirley, C. Wang, and K. Zimmerman, “Monte Carlo techniques fordirect lighting calculations,” ACM Transactions on Graphics, vol. 15,no. 1, pp. 1–36, 1996.

[4] D.P.Mitchell, “Generating antialiased images at low sampling densities,”in Proceedings of SIGGRAPH’87. New York, NY, USA: ACM Press,1987, pp. 65–72.

[5] J. Farrugia and B. Péroche, “A Progressive Rendering Algorithm Usingan Adaptive Perceptually Based Image Metric.” Comput. Graph. Forum,vol. 23, no. 3, pp. 605–614, 2004.

[6] P. Longhurst, K. Debattista, and A. Chalmers, “A GPU based SaliencyMap for High-Fidelity Selective Rendering,” in AFRIGRAPH 20064th International Conference on Computer Graphics, Virtual Reality,Visualisation and Interaction in Africa. ACM SIGGRAPH, January2006, pp. 21–29.

[7] A. Lahoudou, E. Viennet, and A. Beghdadi, “Selecting low-level fea-tures for image quality assessment by statistical methods,” Journal ofComputing and Information Technology, vol. 2, pp. 183–189, 2010.

[8] R. Ferzli and L. Karam, “No-reference objective wavelet based noiseimmune image sharpness metric,” in International Conference on ImageProcessing, 2005.

[9] J. Zhang, S. Ong, and M. Thinh, “Kurtosis-based no-reference qualityassessment of JPEG2000 images,” Signal Processing: Image Communi-cation, vol. 26, pp. 13–23, 2011.

[10] S. Delepoulle, A. Bigand, and C. Renaud, “An Interval Type-2 FuzzySets No-reference Computer-Generated Images Quality Metric And ItsApplication To Denoising,” in Intelligent Systems IS’12 IEEE Congress(Sofia, Bulgaria), 2012.

[11] M.E.Tipping, “The Relevance Vector Machine,” Advances in NeuralInformation Processing Systems, vol. 12, pp. 652–658, 2000.

[12] M.Carnet, P. L. Callet, and D.Barba, “Objective quality assessment ofcolor images based on a generic perceptual reduced reference,” ImageCommunication, vol. 23(4), pp. 239–256, 2008.

[13] P. Heinonen and Y. Neuvo, “FIR-median hybrid filters,” IEEE Trans.ASSP, vol. 35, no. 6, pp. 832–833, 1987.

[14] K. Arakawa, “Median filters based on fuzzy rules and its application toimage restoration,” Fuzzy Sets and Systems, vol. 77, pp. 3–13., 1996.

[15] A. Bigand and O. Colot, “Fuzzy filter based on interval-valued fuzzysets for image filtering,” Fuzzy Sets and Systems, vol. 161, pp. 96–117,2010.

[16] N. Takouachet, S. Delepoulle, and C. Renaud, “A perceptual stoppingcondition for global illumination computations,” in Proc. Spring Con-ference on Computer Graphics 2007, Budmerice, Slovakia, April 2007,pp. 61–68.

[17] M.E.Tipping, “Sparse Bayesian Learning and the Relevance VectorMachine,” Journal of Machine Learning, vol. 1, pp. 211–244, 2001.



324

Documents

[IEEE 2013 International Conference on Communications and Information Technology (ICCIT) - Beirut, Lebanon (2013.06.19-2013.06.21)] 2013 Third International Conference on Communications