Finally, the uvgRTP is compared against other implementations in terms of provided features, HEVC goodput and average latency. The focus is going to be in modern video codecs and secu-rity features. Both encrypted and unencrypted HEVC goodput of uvgRTP is benchmarked and compared against LIVE555 and FFmpeg.
6.1 Features
As previously mentioned, there are several RTP implementations available. Table 14 lists the most famous of them and some of their features.
RTPv2 VVC HEVC SRTP ZRTP License Language
uvgRTP Yes Yes Yes Yes Yes BSD 2-Clause C++
FFmpeg Yes No Yes Yes No LGPLv2.1 C
Gstreamer Yes No Yes Yes Yes LGPLv2.1 C/Rust
Live555 No No No* No No LGPLv3 C++
ccRTP Yes No No Yes Yes GPLv2 C++
JRTPLIB Yes No No Yes No MIT C++
libre Yes No No No No BSD C
oRTP Yes No No No No GPLv3 C
libjitsi Yes No No No No Apache-2.0 Java
Restcomm Media Core
Yes No No No No AGPLv3 Java
Table 14: Comparison of RTP libraries
As can been seen, almost all of the libraries support RFC 3350 (RTP v2) and RFC 3711 (SRTP).
LIVE555 claims to support RFC 7798 but it is required by the specification that the latest RTP specification is supported which LIVE555 not does support. The only libraries that come close to matching uvgRTP’s feature set are FFmpeg and Gstreamer. None of the listed libraries support the latest VVC video codec. Some of the libraries are also licensed under GPL/LGPL which is not suitable license for many proprietary projects that would have a need for real-time streaming so-lution.
6.2 Test setup
All tests were done on two Linux machines that were in the same local network. Figure 49 shows the test setup.
Figure 49: Test setup
The goodput and latency tests were performed in a local network over a 10 Gbit/s link. The com-puter that acted as the sender was equipped with an Intel Core i7-4770 CPU and was running Linux kernel version 4.15.0. The computer that acted as the receiver was running a Linux kernel version 5.0.0 and was equipped with an AMD Threadripper 2990WX. Both computers had a 10 Gbit network interface card.
The goodput tests were performed by streaming a 4K HEVC video sequence which had a bitrate of 202 Mbit/s and 600 frames. Send goodput was calculated by how long it took the sender to send the entire HEVC file and receive goodput was calculated how long it took to receive the HEVC file. Receiver clock was started when the first RTP packet was received and stopped when the last was received. These time values were then used to calculate the goodput. The test was repeated 100 times for each configuration and an average of the runs was taken as the final result. Each configuration was tested for each FPS value in the range [100, 1000], with a step size of 100 FPS. In Figure 49, the arrows with solid lines depict the goodput benchmark test.
Latency tests were performed by sending at a constant rate of 30 FPS and calculating the round-trip latency for each HEVC NAL unit. Each NAL unit latency was measured separately by starting a clock, sending an HEVC NAL unit and waiting for it to be received, after which the clock was stopped, and time difference was noted down. This test was repeated 100 times and average of the runs was taken as the final result. In Figure 49, arrows with both dashed and solid lines depict the latency benchmark test.
LIVE555’s latency tests did not yield values that could be used directly when reporting the num-bers. The problem is that outside control for FPS calculations is not acceptable as it blocks LIVE555’s event loop and prevents it from running optimally, increasing the measured latency.
On the other hand, instructing LIVE555 to stream at a constant 30 FPS increases the measured latency as after the frame is given to LIVE555, it is not sent immediately but buffered until LIVE555 determines that the frame should be sent to keep the constant FPS. As the timer is started when the frame is given to LIVE555, but because the frame is not sent immediately, there is an increase
in the latency number that incorrectly contributes to the final latency values. To get the best esti-mate for LIVE555’s latency values, it was required that at least 98 % of the frames were received so sensible estimates could be calculated. Allowing a 2 % packet loss gives LIVE555’s benchmark the ability to stream very close to its maximum speed, i.e., minimum latency, while still yielding good estimates for its latency values. 100 % packet delivery requirement is too strict for LIVE555’s benchmark to handle when the upper limit for streaming performance, i.e., low latency, must be found. FFmpeg and uvgRTP do not need these amendments as they do not have any internal sleep mechanisms and their latency benchmarks yield accurate values.
In the configuration runs, configurations with SCC mean that System Call Clustering was enabled with 512 packets and SCL means that the optimized start code lookup was enabled. UDP socket buffers were set to 40 MB and the socket backlog was set to 10,000 entries.
Out of the listed libraries, LIVE555 and FFmpeg were selected and benchmarked against uvgRTP. Gstreamer had the best feature parity with uvgRTP but its tightly-knit filter-based archi-tecture made it very hard to implement the required testbench for it. Furthermore, implementing SRTP testbench for FFmpeg in C++ was very hard and it was omitted. It is show, however, that uvgRTP’s performance when streaming encrypted HEVC can outperform FFmpeg when it is streaming unencrypted HEVC. From this can be extrapolated that FFmpeg’s SRTP performance would be worse than uvgRTP’s SRTP performance. Thus, uvgRTP’s unencrypted HEVC goodput and latency performance is compared against FFmpeg and LIVE555 and its encrypted HEVC performance is compared against its unencrypted configuration alternatives.
6.3 Unencrypted HEVC Goodput Evaluation
The first performance test was to test the goodput of sending and receiving unencrypted HEVC in local network. Figure 50 shows the goodput each tested configuration can achieve as a function of FPS.
Figure 50: Goodput of uvgRTP, LIVE555, and FFmpeg
As Figure 50 depicts, all uvgRTP configurations outperform both LIVE555 and FFmpeg. SCL and SCC are roughly equivalent optimizations in terms of goodput improvement. SCC does incur a slightly larger frame loss due to its bursty nature which decreases the goodput. Figure 50 also shows that uvgRTP is hitting a limit in speed when 600 MB/s is reached. When FPS reaches
1000, the data rate becomes too high which results in excessive packet loss which, effectively decreasing goodput.
LIVE555 suffered large frame losses after FPS was over 500 which caused its goodput to drop down significantly. Increasing the socket buffer sizes to 80 MB and the socket backlog to 20,000 entries showed an improvement to the results.
FFmpeg’s poor performance is explained by its low CPU usage, averaging at 65 %. The CPU usage does not increase even if FPS control code is removed from the benchmarking code and when FFmpeg is called as fast as possible. This indicates there is some internal functionality which limits the CPU usage and thus reduces the goodput of the library.
Figure 51 shows the frame loss for each configuration as a function of FPS.
Figure 51: Frame loss of uvgRTP, LIVE555, and FFmpeg
As could already be expected from looking at Figure 50, the frame loss for uvgRTP at 1000 FPS is quite significant. This is either explained by some network limit or by packet dispatcher’s inability to process packets faster than the rate that 600 MB/s requires. Implementing SCC for the receiver could improve goodput and lower packet loss as the receiver could be able to process the incom-ing packets faster.
LIVE555 suffers the largest frame loss when the streaming speed crosses 400 FPS boundary.
Beyond that, the stream is unplayable, and this frame loss greatly contributed to the low goodput values show in Figure 50. Possible fix for this would be to increase the socket buffer and backlog sizes to allow the receiver more time to process the incoming packets.
FFmpeg’s frame loss grows to ~2 % at 200 fps and stays there for the rest of the benchmark. A possible fix for this also would be to increase the socket buffer size and backlog to allow it more time to process the incoming data.
6.4 Unencrypted HEVC Latency evaluation
Figure 52 shows the round-trip latency values for each configuration of uvgRTP, LIVE555, and FFmpeg. The scale is logarithmic because the differences between uvgRTP and FFmpeg were very different.
Figure 52: Round-trip latencies of uvgRTP, LIVE555, and FFmpeg
The figure clearly shows that all uvgRTP configurations can attain a lower average latency than LIVE555 and FFmpeg. This could be explained by the relatively simple architecture of uvgRTP where the call stack if short and there are no memory copies done during the whole send process.
LIVE555 is also able to achieve a good round-trip latency even though the benchmarks only yielded estimates for its latency values.
Table 15 tabulates the latency values for each uvgRTP configuration, LIVE555, and FFmpeg as well as how many discarded runs each configuration run had. For uvgRTP and FFmpeg, 100 % of frame delivery was required and for LIVE555, the requirement was 98 % for reasons explained in Section 6.2.
Sender Intra (ms) Inter (ms) Average (ms) Discarded
uvgRTP (unoptimized) 15.7 7.7 7.8 7 %
uvgRTP + SCL 14.8 7.2 7.3 3 %
uvgRTP + SCC 14.8 7.0 7.1 40 %
uvgRTP + SCL + SCC 14.3 6.8 6.9 44 %
LIVE555 20.3 18.0 18.0 89 %
FFmpeg 108.3 80.6 81.0 52 %
Table 15: Latency values for uvgRTP, LIVE555, and FFmpeg
Table 15 clearly shows that uvgRTP is able to outperform both FFmpeg and LIVE555 even with-out any optimizations. Furthermore, uvgRTP with SCL and SCC optimizations can improve its own latency slightly. SCC optimization seems to result in higher packet loss and thus to higher frame rejection rate. This could be explained by the bursty nature of its operating mode which the network might not always handle gracefully.
LIVE555 is able to reach a good round-trip latency even if they are slightly over-optimistic esti-mates. The real average latency for LIVE555 would be slightly higher than reported in the table as allowing a 2 % frame loss results in more intra frames being dropped as they are much larger than inter frames which in turns decreases the average latency. The value for inter frame latency can be considered to be accurate and as inter-frame latency and average latency are nearly identical for all tested configurations, the estimated average latency can be considered to be very close to its real value with high confidence. These findings are further corroborated by the goodput benchmark results as goodput and average are directly proportional, and LIVE555 is somewhat close to uvgRTP’s unencrypted HEVC streaming performance when optimizations are disabled.
FFmpeg’s round-trip latency would most likely not be acceptable for an application requiring real-time streaming. The high latency could be explained by its low CPU usage, a reason why its goodput is so low compared to uvgRTP and FFmpeg.
6.5 Encrypted HEVC Goodput Evaluation
The encrypted HEVC goodput benchmarks were performed exactly the same as with unen-crypted HEVC but this time SRTP was enabled. LIVE555 was excluded because it did not support SRTP and despite efforts, attempts to get SRTP working with FFmpeg from C++ code were not successful. Thus, uvgRTP’s performance when streaming encrypted HEVC is compared to its performance when streaming unencrypted HEVC. Figure 53 shows the goodput values of all uvgRTP configurations as a function of FPS.
Figure 53: uvgRTP goodput, all configurations
As expected, enabling SRTP degraded the performance quite significantly. The configuration with SCC and SCL enabled is around 100 MB/s short of what uvgRTP with no optimizations enabled can deliver for unencrypted HEVC. This is due to extra copy operation being done to construct the packet from separate buffers into one so it can be fed to the ciphering routine. The ciphering also takes its own time, further degrading the goodput. Figure 54 shows the frame loss for various uvgRTP configuration as a function of FPS.
Figure 54: uvgRTP frame loss, all configurations
Figure 54 shows that the frame loss for all SRTP configurations is very moderate. This is explained by the SRTP’s effect of slowing down the speed which decreases packet loss. SCL + SCC is also very viable configuration when SRTP is used as it provides a noticeable speed-up while causing no additional frame loss.
6.6 Encrypted HEVC Latency Evaluation
The encrypted HEVC latency benchmarks were performed for uvgRTP and results were com-pared against uvgRTP’s unencrypted HEVC streaming performance. Figure 55 shows the results.
Figure 55: uvgRTP latency values, all configurations
As Figure 55 clearly shows, the enabling of SRTP causes large spikes to the latency values of all three measurements. The average round-trip latency is still, however, below 15 milliseconds mak-ing the performance of encrypted uvgRTP streammak-ing very suitable for a plethora of real-time streaming applications. Table 16 tabulates the latency values for encrypted HEVC streaming.
Sender Intra (ms) Inter (ms) Average (ms) Discarded
uvgRTP (unoptimized SRTP) 23.4 12.3 12.4 13 %
uvgRTP + SRTP + SCL 22.5 11.6 11.8 12 %
uvgRTP + SRTP + SCC 23.7 12.2 12.4 68 %
uvgRTP + SRTP + SCL + SCC 22.7 11.3 11.5 66 %
Table 16: Latency values for different uvgRTP configurations
Comparing the values in Tables 15 and 16, it can be clearly seen that enabling SRTP nearly doubles the latency for all three measurements. Taking into consideration, however, that uvgRTP’s latency when streaming encrypted HEVC is very close to LIVE555’s latency when streaming unencrypted HEVC, these values are acceptable. Sacrificing negligible amounts in av-erage latency and goodput gives an application a possibility to use E2EE which is in many cases the more attractive alternative.
6.7 Discussion
As Figures 50-55 and Tables 15 and 16 show, uvgRTP is a very viable alternative to current solutions. It follows the latest RTP specification, implements the RTP payload formats for AVC, HEVC, and VVC, and supports both SRTP and ZRTP, providing transparent E2EE support for the application. It outperforms both LIVE555 and FFmpeg when streaming unencrypted HEVC.
Furthermore, uvgRTP’s encrypted HEVC performance can outperform FFmpeg’s unencrypted HEVC performance. It also has a very permissive license so it can be used with proprietary soft-ware freely. For applications that do not care about conformity with other RTP implementations, the generic interface it provides is a very attractive feature for anyone who wishes to send data using Real-time Transport Protocol.