The first actual reconstruction algorithm introduced in this thesis is cross bilateral blur and its variants which has been optimized better runtime. Bilateral blur is an extension of the basic the Gaussian blur. The difference is that bilateral blur tries to preserve the edges of the content. The problem with bilateral blur is that it is not fast enough with big enough blur kernels for real-time path tracing.
One of the fundamental ways to blur an image is to use Gaussian blur. Gaussian
blur decides the weights of the neighboring samples based only on the spatial distance of the sample to the blurred pixel. The formula for one sample pixel’s weight is
w(x,y) =e−(x−x)
2+(y−y)2
2σ2 , (3.1)
wherexis the blurred pixel coordinate on the x-axis,xis the sample pixel coordinate on the x-axis, y and y are the same variables on the y-axis, and σ is the wanted standard deviation of the Gaussian kernel.
At spatial distances further than 3σfrom the blurred pixel, the Gaussian weight of the sample pixels are already more than thousand times lower compared to the center. Therefore, practical real-time implementations can limit sampling area which saves the memory bandwidth, without noticeable difference in resulting quality.
The basic version of Bilateral blur[TM98]extends this formula by introducing the color space distance to it
wb(x,y) =e−
(x−x)2+(y−y)2 2σ2
d
−|I(x,y)−I(x,y)|
2σ2
r , (3.2)
where I(x,y) is the color value at the blurred pixel. Note that there are separate standard deviation factorsσfor the distance and the color value.
Bilateral blur can be extended to use other information than just the spatial and color space distance. This is called cross bilateral filtering[ED04; Pet+04]. More-over, the color space distance varies a lot in path tracing noise and therefore it is not very useful information in the path tracing reconstruction case. So, for example, the distance from the camera to the first intersection and surface normals on that point typically contain useful information about the possible edges in the 3D scene [Dam+10]. The weightwb(x,y)from Equation 3.2 must be multiplied with the weights from these buffers
wc(x,y) =wb(x,y)·wz(x,y)·wn(x,y), (3.3) where wn(x,y)is the weight from the normal buffer andwz(x,y)is the weight from distance to the first intersect buffer specifically the depth buffer.
One good way to compute the weight from the normal buffer is
wn(x,y) =max(0,n(x,y)·n(x,y))σn, (3.4)
i = 0 i = 1 i = 2
Figure 3.2 An illustration of sampling pattern on three first iterations of 1D À Trous filter with kernel window size of 5. The light purple pixel is the target pixel which also sampled from the input and where the bilateral blur result is stored in the output.
wheren(x,y)is the normal vector, in other words, three values∈| −1≤ p≤1 of the closest surface in front of the pixelx,y [Sch+17].
The weight from the depth buffer can, for example, be wz(x,y) =e− |Z(x,y)−Z(x,y)|
σz|∇Z(x,y)·[x−x,y−y]|+ε, (3.5) whereZ(x,y)is the depth buffer value at the pixelx,y,∇Z(x,y)is the gradient of the depth, andεis used to avoid division by zero[Sch+17].
Multidimensional Gaussian blur can be optimized by separating it to separate passes per axis, which reduces the number of expensive memory access from(nm) to(m×n)wherenis the blur kernel diameter andmis the count of dimensions.
In contrast, Bilateral blur cannot be separated because blur on one axis affects many pixels on the other axis and one distance cannot be used in the spaces that are used for finding the edges. Therefore, fast timings require some other approximation of the Bilateral blur. One option is to use so calledadaptive manifoldswhere the work can be shared between the neighboring pixels, but it requires deciding how many mani-fods are needed, which affects the quality and the runtime greatly[Bau+15; GO12].
Therefore, currently some sparse versions of the bilateral blur like theÀ Trous Fil-ter are typically used[Dam+10; Imm17; Mar+17; Sch+17]. The fast timings are achieved by not sampling every intermediate pixel.
3.2.1 À Trous Filter
The idea of the À Trous filter [Bur81] is to run multiple passes over the image, which all blur different frequencies. Therefore, À Trous filter is also called a
dis-(a)1 spp input (b)1 iteration (c)2 iterations
(d)3 iterations (e)4 iterations (f)5 iterations
Figure 3.3 An example of how every iteration of the À Trous filter blurs lower frequencies of the 1 spp input frame. The kernel size is5×5and no variance or temporal data is used. The 5th iteration starts to show typical artifacts of basic À Trous filter around the area where the back wall and ceiling meet.
crete wavelet transform[Fow05]. Every pass uses the same bilateral blur window, but what changes between the iteration is the amount pixels in between the sampling locations. More specifically, every iteration makes the bilateral filter more sparse, as can be seen in Figure 3.2. Sparse filtering makes it possible to blur with a bigger blur window without actually sampling all of the values in the area. Also, À Trous can be extended to use the feature buffers for edge stopping[Dam+10]. In the Figure 3.3, À Trous filter is applied to a 1 spp frame using albedo, normals, and world position feature buffers as guidance. Sometimes the typical À Trous artifacts visible in the Figure 3.3 can be avoided if iterations are applied in reverse order[QWH12].
(a)1 spp input (b)BMFR [P2] (c)SVGF [Sch+17]
Figure 3.4 The same scene as in Figure 3.3 reconstructed with BMFR and SVGF algorithms. BMFR algorithm produces a halo around the light source that could be reduced by giving BMFR a feature buffer containing a similar pattern as the luminance in the reference such as 1-bit buffer indicating if the material is emitting light. SVGF produces some luminance bumps on the walls and À Trous style artifacts both of which could be reduced by tweaking the parameters.
3.2.2 Spatiotemporal Variance-Guided Filtering
In path tracing reconstruction one interesting edge stopping feature buffer is the per pixel variance. With the variance buffer the certainty of the sample luminance can be used to increase the sample’s weight. If there is just one sample, variance cannot be estimated. Therefore,Spatiotemporal Variance-Guided Filtering(SVGF)[Sch+17]
uses temporal data for the variance estimation and in case of occlusion the variance is estimated spatially in screen space. In addition, SVGF updates variance estimation on every iteration of the À Trous filter. An example of the quality of SVGF can be seen in Figure 3.4c. Notice how the use of temporal data and the variance estimation removes edge artifacts and improves the contact shadow quality of the smaller box compared to the original À Trous filter shown in Figure 3.3.