Path tracing is a rendering technique to generate photorealistic images by physically em-ulating the light traversing inthree-dimensional(3D) scenes. Even though new hardware for path tracing has started to appear for consumers, path tracing in real-time is still a challenging problem. In order to generate path traced images in real-time the number of paths traced must be limited which generates highly noisy images. But because path tracing itself is an ’embarrassingly parallel’ problem one possible solution for generating more samples would be to increase the computational power with multiple devices and do the path tracing in server cluster. Even though distributing the path tracing for mul-tiple devices, it is still very hard to generate fully converged images (mulmul-tiple thousand paths per pixel) in real-time. Moreover, in path tracing to half the error of the output the number of samples must be quadrupled. This means also that the computational power must be increased in the same relation. However, fast reconstruction methods have been developed also to enhance the final image and there is a point where it is more efficient to denoise the path traced image instead of increasing the number of samples. There-fore, best way still seems to be is to limit the number of paths traced and utilize pre- and post-processing methods to reconstruct the final image.
Most real-time solutions for path tracing have concentrated on only using onesample per pixel (spp) and using temporal accumulation of samples and post-processing to denoise the highly noisy image. For example, in other work the solution has been to use the temporal accumulation of 5 samples per pixel and using wavelet-based denoising on the output [1]. But new consumer level Graphics Processing Units (GPUs) have started to appear with dedicated hardware for ray traversal [2]. This is an important prerequisite for generating better quality images more quickly which in turn enables more samples per pixel in real-time implementations also. This is an interesting direction to decrease the noise for the generated image which helps to simplify the denoising method.
The evolution of modern Machine Vision and learning based methods have also been an interesting topic for classifying, generating and reconstructing images. Many Convo-lutional Neural Network (CNN) based solutions have surpassed the manual and model based methods in many image classification and reconstructing problems. They have also been successfully implemented for reconstructing path traced images but have usu-ally lacked the computational speed for real-time implementations. These problems are
Figure 1.1. Results of convolutional neural network denoised path traced inputs with different samples per pixel inputs. The left part of the images are the different samples per pixel input images used in this thesis, the middle parts are the 4096 spp fully converged reference images and the right part is a fast convolutional neural network used to denoise the low sample per pixel input.
being solved with the increasingly faster dedicated inference hardware for machine learn-ing and development of new faster network architectures and layers. Still the speed of the methods lags compared to many of the analytical models created for the same problems.
Moreover, interesting direction has been found to combine the learning-based methods with the model-based methods.
In this thesis we study the use of convolutional neural networks for denoising the results of multiple samples per pixel path traced images in real-time. Path tracing with 8, 16, 32 and 64 samples were considered. The main idea of the thesis is to test the real-time performance of machine learning reconstruction for path tracing and to study the effect of this post-processing in relation to increasing the number of samples in rendering. This requires limiting the number of parameters and complexity inside the network to handle the demanding requirements for real-time path tracing. For convolutional neural networks this means trading off the receptive field when decreasing the depth of the network. In this work the depth of the networks is decreased to only two hidden layers compared to for example ten hidden layers used in other work [3]. To increase the receptive field, different methods are tested; different size of convolutional kernels, pooling and dilated convolutions. Also, pruning and separable convolutions are utilized for trying to improve the performance of the networks. When designing the fast networks, few different non-linear activation functions are tested for the small reconstruction networks. After this, 3
different small networks are evaluated and compared with a fast analytical reconstruction method also with varying receptive field and a slower network in 3 different scenes. The denoising results achieved for a small CNN is illustrated in figure 1.1.
In Chapter 2 firstly the basics of Convolutional Neural Networks (CNN) are presented and how they can be implemented for image reconstruction in real-time. In Chapter 3 path tracing is introduced and the constraints of rendering path traced images in real-time are described. In the next Chapter 4 the different designed networks are introduced and tested with different hyperparameters and optimizations. After this in Chapter 5 the designed networks for path tracing reconstruction with CNNs are evaluated and compared with analytical real-time denoiser. Next in Chapter 6 there is discussion of future work related to the thesis. In the final Chapter 7 there are conclusions.