Challenges in benchmarking video streaming systems

Live OTT video delivery is a challenging workload because of (i) the massive scale of delivery during popular events, and (ii) the fact that events implicitly synchronize users as they tend to connect simultaneously to watch the same popular game, same popular event. To reinforce that challenge, video delivery protocols (DASH / HLS) ensure that every video player discovers new video segments in a timely manner, resulting in more or less synchronized requests for content. This last point is further reinforced by low latency variants of DASH and HLS protocols that add an explicit clock (UTCTiming) or anticipated requests to rub network latency out. This creates burstiness on an already high-volume network traffic. Putting systems under load to evaluate their performance, and the effect of the burstiness of the workload is critical to gain confidence before going into production. Yet, tools of the trade such as wrk/ab lack the ability to reproduce player behavior regarding (i) the timings of requests, (ii) reactions to errors or even minor delays in delivery (with aggressive retry), or (iii) low latency anticipated requests. In addition, to ensure proper benchmarking of systems that may perform manifest manipulation, on-the-fly video drm-ization or packaging, it is critical to deduce requests URLs for real segments from HLS and DASH manifests rather than relying on fixed URLs and content. In this talk, we’ll uncover some fo the practices at Broadpeak, including showing the benchmarking tools we’ve developed to support video-system benchmarking and explain how these have been instrumental in building efficient systems, without having to systematically rely on experimentation with production traffic. The benchmarking tools we’ve developed combine both (i) the power of a full-blown video player, (ii) and very high throughput for handling high-efficiency video servers benchmarking without requiring a full-rack of clients. Finally, we’ll discuss the challenges and opportunities for making players less aggressive to CDN servers by taming requests bursts.