This section introduces a few key concepts which can be used as basic building blocks with most of the real-time reconstruction algorithms described later in this chapter.
3.1.1 Feature Buffers
A path tracer can store information about the 3D scene and the reconstruction al-gorithms can use this information for guiding their reconstruction process. Most importantly this feature information is often completelynoise-free. Examples of fea-ture buffers are all G-buffer channels, specifically, surface normals, positions in the
3D world, surface roughness, material albedo, etc. Some examples of these buffers can be seen in Figure 3.1. For faster runtime in contemporary real-time applications feature buffers and primary rays are typically computed with rasterization hardware [Bar18; Mar+17;P2; Sch+17].
In contrast, photograph denoisers must rely completely on noisy 2D bitmap data.
This could also be the case if the path tracer has motion blur or depth of field sim-ulation, which generate noise also to the feature buffers. Some options for working without any noise-free data is to fit polynomials to the data or find similar areas from the image and use them to arrive at a noise-free estimate[DFE07]. However, at the time of writing, motion blur and depth of field are out of reach of real-time path tracing and they are generated with post-processing estimation techniques[GMN14;
YWY10].
3.1.2 Motion Vectors
In path tracing, the exact parameters of the simulated camera are known and the world positions or depths of the first intersections can be stored. With this infor-mation, previous frames and other viewpoints can be projected to the current cam-era location and orientation. Motion vectors can also support simple animations as long as there is a way to find out where the point was in screen space in the previous frame. The position can be computed, for instance, if the animation is constructed from a set of basic matrix operations like translations, rotations and scalings. What makes the camera parametersexactis that the camera’s parameters like the position are known down to the accuracy of the used data type. Similar information can be extracted from just a video stream[SB91]. However, this requires a lot of memory traffic and from the video it is hard to acquire the information as accurately and without noise.
Reprojection gives us per pixel motion vectors, which denote where in the screen space the world space position of a pixel was in the previous frame. With 1 spp frames, sampling history data based on motion vectors and computing the exponen-tial moving average can give results that are similar to 10 spp frames[Mak+19]. This requires thresholds, for example in sample’s normal and position temporal change for realizing if the point was occluded on the previous frame. Otherwise there are so called ghosting artifacts where the foreground data is mixed with the background.
Reprojection can also be used in the spatial domain if multiple views are generated, for example for a stereo HMD or for a light field display.
Motion vectors can be computed for different components of lighting separately [Zim+15], which preserves effects such as reflections. However, this complicates the motion vector and luminance computations, since components need to be stored separately and, therefore, it is difficult to use the technique in real-time with contem-porary hardware.
There are at least two drawbacks with the use of reprojected data. Firstly, the quality varies across the screen, since reprojection cannot be done on areas that were occluded on the previous frames. To be more precise, if the reconstruction algo-rithm uses reprojected and accumulated frames it must support varying quality in-puts. Secondly, using temporal previous frame data introduces temporal lag to the illumination changes. Depending on the parameters, the lag can, for example, be 10 frames long[P2]. A lag of 10 frames can be invisible to the user in some cases, but for example a light source flashing on every other frame would appear to be constant and half as bright as it really is.
There is a solution which removes the temporal lag[SPD18]. The idea is that one path tracing sample in every block of pixels is path-traced with the same random seed as in the previous frame. Using the same seed means that, if the illumination conditions are the same, the sample generates the same result as in the previous frame.
If the result is different it means that the illumination has changed and, in that case, the temporal data can be discarded. Basically, the algorithm falls back to the first frame 1 spp quality in areas where there are changes. Interestingly this technique also removes ghosting from reflecting surfaces, because also they fall back to 1 spp quality when the camera is moving. However, current real-time reconstruction algorithms are not good enough with just 1 spp input and there will be artifacts. The severity and the type of the artifacts is determined by the used reconstruction algorithm.
Another problem is that generating the same sample as in the previous frame requires altering the sub-pixel offsets per pixel, which is not supported by the fastest primary ray computation method of hardware accelerated rasterization.
Reprojection can also be used after the reconstruction algorithm. An extra re-projection step makes the results more temporally stable [P2; SPD18]. In addi-tion, more temporal stability can be achieved with Temporal Anti-Aliasing (TAA) [All+17; Kar14;P2; Sch+17]. TAA uses temporal reprojection without discarding
the occluded data and instead it clamps the history sample’s luminance to the current frame’s neighboring pixels’ luminance.
3.1.3 Component Separation
Monte Carlo integration is the sum over incoming light directions. Therefore, it is possible to reconstruct the samples in separate groups, without introducing bias to the results.
One idea is to compute the filtering parameters for two groups, each containing half of the samples, separately and then do so calledcross filtering[RKZ12]. In cross filtering the parameters computed for the first half are used to reconstruct the second half and parameters computed for the second half are used to reconstruct the first half. The final result is the average of the two reconstructed images. The idea of cross filtering is to reduce over fitting of the filtering parameters. Currently, path tracing two different full resolution sets of samples is unfeasible[All+17; Bar18;P2;
P5; Sch+17], but this could be one interesting direction in the near future, since it can produce good results in an offline context[Bit+16].
Another idea for reaching better quality is to filter the direct illumination and indirect illumination separately[Mar+17; Sch+17]or diffuse and specular compo-nent separately[Bak+17]. In those cases, there can be separate reconstruction algo-rithms specifically tuned for their inputs. For instance, the direct illumination can be generated with noise-free shadow mapping techniques and then there is no need to reconstruct it[Mar+17]. However, in some work[P2; SPD18]mainly for faster execution reasons, separate reconstruction was not found beneficial and both of the components are reconstructed at once.