Image processing pipeline based artefacts

The third link in the image creation chain is the image processing pipeline as defined in section 2.2. In modern mobile phone cameras, the significance of the pipeline is incontrovertible. Due to small sensor size, extremely small pixel size, and very demanding lens requirements, the quality of the RAW images from the sensor is quite often very poor. Images are dark, noisy, distorted, and colors of images are not correct. In practice, the image processing pipeline recreates the image by denoising, color corrections, gamma correction, tone mapping, and several other algorithms. An example of differences between the original RAW image from sensor and the processed image is shown in Figure 14.

(a) (b)

Figure 14 Image processing pipeline example: a) RAW image from sensor and b) Processed image

It may even be said that severe overexposure, underexposure and defocus are the only artefacts the image processing pipeline cannot correct. However, the quality of the final image depends highly on the algorithms and severe errors may occur when the algorithms or parameterization of the algorithms are not correct. This section declares the most problematic artefacts the image processing pipeline may cause.

3.3.3.1 Compression

Usually all images and video streams are compressed before they are used, stored or broadcast. The majority of compression algorithms are lossy i.e. they remove information from the original image or stream. Obviously, compression artefacts are related to the compression method actually used. Artefacts of the most common compression methods used in mobile phone cameras are discussed in this section.

A block based compression method is used in many image and video compression algorithms, for example JPEG, MPEG-1, MPEG-2, and H.26x compressions use the method. In the case of the widely used JPEG compression, the block size is 8x8 pixels and a local discrete cosine transform (DCT) is executed in each block. Since the compression is based on blocks, a discontinuity between blocks is possible and can cause blocking artefacts. (Keelan 2002; Wang and Bovik 2006; ITU-T T.81 1992)

JPEG2000 compression is based on a wavelet compression, which transforms the whole image and does not suffer from blocking. However, the wavelet based compression may cause a ringing artefact, which causes faulty luminance or color

highlights near high intensity edges in a quite similar way to over sharpening.

(Wang and Bovik 2006)

In general, compression tends to filter out high frequencies especially in chromatic channels because the human vision system is less sensitive to those. Too high compression rate may lead to the blurring artefact where high frequencies, small objects and texture, are filtered out.

In case of video compression, the artefacts can be more visible, as the compression is not made inside one video frame but between frames. This may cause also temporal artefacts. Video artefacts are discussed more in section 3.4.

3.3.3.2 Color inaccuracy

Color accuracy in image processing is based on two methods; estimating the light temperature of ambient light and color correction according to this estimation.

Obviously, both methods can cause color errors in the final image.

If the ambient light is estimated falsely, wrong color correction factors are used and, for example, a scene captured in sunlight may turn bluish, if it is corrected using fluorescent correction factors. On the other hand, the color correction factors can be inaccurate, if the camera system is not calibrated correctly to each type of ambient light or the interpolation between color correction factors does not work correctly.

Moreover, the algorithm itself can fail to reproduce the colors of the scene. A good example is an old but widely used method called gray world which assumes that the mean color of the whole image is always gray and estimates the ambient color correction according to that assumption. The method works well until the scene includes a dominant color. In such a case, the colors of the images are biased according to the dominant color.

3.3.3.3 Sharpening artefacts

Sharpening is a method where the intensity of the edges in the image is artificially amplified by increasing the contrast of the edges. For example, the border between light gray and dark gray is amplified by darkening the dark gray area near the edge and lightening the light gray area correspondingly. The sharpening can be used to increase the perceptual sharpness of images, but it may easily generate various artefacts, too.

If the edges are amplified too much, the sharpening comes visible and causes a ringing artefact, a halo around edges. Correspondingly the dark side of the edges

may turn too dark causing visible dark lines. Wrongly parametrized sharpening starts to highlight particles in the image that are too small, it amplifies the noise and also may exaggerate textures or even filter high frequencies. (Caponigro) All in all, the image may turn unnatural looking.

Figure 15 illustrates the sharpening artefacts both in the edges and in texture areas.

Figure 15 Sharpening artefacts 3.3.3.4 Noise removal artefacts

The main artefact related to noise removal is generic blurring. When the noise particles are removed efficiently, also edge areas tend to be smoothed. On the other hand, if the image contains small detailed natural texture, for example sand, the characteristics of the texture are quite similar to the noise generated by the camera system. It is very difficult for the noise removal algorithm to separate natural texture and artefactual noise. This is problematic especially in the texture parts of the picture where noise removal may cause texture loss (Artmann and Wueller 2012).

Equally, when denoising is too efficient, it may cause over smoothing in uniform areas of the images. This may lead to unnatural images which appear oil painted

(da Silva et al. 2013). Some block based denoising algorithms like block-matching and 3D filtering (BM3D) may cause also blockiness in the images (Dabov et al.

2006). An example of a poor image quality after too aggressive BM3D noise removal is shown in Figure 16.

(a) (b)

Figure 16 Noise removal artefacts, blurring and blockiness: a) Original scene and b) Aggressive denoising

3.3.3.5 Demosaicing

Since demosaicing interpolates the colored pixels of the Bayer filter to single colored pixel values, it may affect several quality features of the camera system;

noise, colors and resolution. In addition, more specific errors like maze pattern artefact, moiré and zippering may occur. It has to be also remembered that the Bayer type sensor has two times more the green pixels than red or blue pixels, thus the resolution of green color channel is two times better than the other ones. The demosaicing algorithms have to allow this imbalance.

There have been a lot of research and suggestions for algorithms to be used in demosaicing. For example, nearest neighbor replication, bilinear interpolation or cubic spline interpolation can be used to calculate the colors of a pixel (Menon et al. 2006). Since demosaicing is always based on interpolation, it generates only an estimate of the missing two color components of a certain pixel. The estimation always generates noise in the image and the accuracy of the estimation defines the level of blurriness caused by demosaicing as well as the color accuracy of the final pixel value.

Inefficient demosaicing may cause a maze type pattern in the image, if the original Bayer filter structure is not filtered out properly. Finally some demosaicing types,

like plane-wise interpolation may distort object boundaries by generating zipper shaped edges (Hirakawa and Parks 2005).

3.3.3.6 Over processed images

Finally, an image can be processed too much. The reason for an over processed image may be poor quality of the RAW image or too aggressive parametrized image processing algorithms. Even if the algorithms like denoising, sharpening and tone mapping do not create any artefacts in final images, the final image may look unnatural. Obviously, the naturalness of the image is very perceptual quality feature and it is difficult to measure.