Written by Jens Berger | March 31, 2021
The study’s credibility was based on the following:
- Maximum transparency: SRG applied the holistic, integrative and QoE-focused European Telecommunications Standards Institute (ETSI) scoring methodology and the mobile network benchmarking recommendations in ETSI TR 103 559
- Minimal complexity: The SRG results used the network performance score (NPS) point scale, which allows readers to understand and comprehend the results from high-level competitive rankings and scores down to single measurements based on defined key performance indicators (KPI)
- Full independence: The SRG approach ensures full network infrastructure supplier independence and uses sophisticated models to reflect the true end-user experience in today’s mobile networks. The methodology is technology and vendor neutral, delivering aggregated QoE metrics and scores as well all relevant KPIs
- International comparability: The transparent methodology applied by SRG and Rohde & Schwarz also enables benchmarks and neutral comparisons with other global mobile operator networks by using a uniform and transparent approach
SRG had many reasons for working with Rohde & Schwarz in their latest public study, including the leading solutions from R&S for testing mobile networks and improving 5G network quality and performance. Most important is the company’s native support for the ETSI scoring method across the entire mobile network testing product portfolio, since this holistic approach through the entire processing chain enables efficient data collection, data processing and reporting. Find out more in the SRG video or below.
Why is end-user testing important? We evaluate the certain very interesting conclusions from the SRG study
Overview:
- Interestingly, the operator with the “slowest” average network bitrate came in first for QoE in the SRG benchmarking study
- The operators with the best 5G NR network coverage were not able to benefit from the best user experience
- In general, data latency is crucial for interactive, real-time and ultra-reliability low latency communications (URLLC) use cases
SRG study conclusion: Interestingly, the operator with the “slowest average” network bitrate came in first for QoE in the SRG benchmarking study
How could the slowest operator win in this category? The R&S SmartAnalytics software suite can reveal the reasons for any observed degradation and compare results for the aggregated NPS score, down to individual measurements and KPIs. This structured approach makes it easy to understand in detail, why and where one operator performs better than the other. More importantly, the software provides direct and qualified suggestions for where and how to improve networks. The operator benefits from a guided optimization process, enabling network quality and performance improvements that are highly targeted and end-user focused. The structured approach also reduces complexity for everybody working in optimization and quality management teams, while also enabling C-level management to set clear quality improvement targets. Thus, it is a sustainable, fully transparent and global quality management tool.
SRG study conclusion: Operators with the best 5G NR network coverage often deliver a poorer user experience
End-user opinions about the perceived quality and performance of a service or product usually are not based on a single factor but a sequence and set of multiple factors over a certain time for all of their most important uses. Claims of having the fastest network or the best coverage in the city center or any other hotspot is usually just based on a technical KPI and says very little about the end-user experience. A normal user does not see “bitrate” they just know how long it takes to finish a task, the presentation quality and whether they are receiving the information they want. The process depends on more than just bitrates. With this in mind, SRG decided to use the R&S NPS methodology, which runs a set of typical use case scenarios for a human smartphone user, including browsing popular sites, streaming video, posting to social media and making phone calls, all while testing bitrate and network capacity. The methodology involves choosing KPIs that reflect perception and weighting these according to their importance in daily use.
R&S is also at the cutting edge of measurement methods. This includes the application of real-picture based video quality measurements for smartphones (in line with ITU-T J.343.1) or speech quality in line with perceptual objective listening quality analysis (POLQA) (ITU-T P.863). R&S MNT co-owns the intellectual property rights for this application, which gives the company unparalleled insight into its algorithm.
A transparent measurement study needs more than just the best measurement technologies and the most accurate results, governing principles that ensure fair treatment of all networks are also important, these include:
- Use of the latest flagship smartphones
- Measuring the same tasks in parallel at the same position and synchronized across all selected operators (benchmarking)
- Well defined and balanced test sequences based on true end-user behavior (voice calls, video viewing, browsing, posting to social media etc.)
- Defined geographical regions
- Having all devices equipped with the same test software (QualiPoc) and controlled by state-of-the-art and specialized test equipment (SmartBenchmarker)
- Having in place best industry practices for network quality benchmarking (ETSI TR 103 559)
All this is arranged to professionally assess mobile network quality, make qualified statements about actual perceived end-user quality and draw meaningful conclusions about the competition.
SRG study conclusion: In general, data latency is crucial for interactive, real-time and possible URLLC use cases
5G will enable a new set of services, not just for people but also industrial and automotive applications. One or possibly THE most important factor with 5G is low latency, which is very important in automotive or industrial applications as well as in everyday smartphone usage. Video chats, remote meetings or mobile gaming sessions are normal nowadays and all need to be low latency. The often-cited ultra-short latencies (URLLC) are essential for real-time control in production or automotive applications, but the requirements for many applications are less demanding but still need more than what legacy mobile technologies can provide. To understand how 5G networks help keep latency low, Rohde & Schwarz created a dedicated interactivity test that combines testing data latency, packet delay variation, ratio of disqualified packets and checks for sustainable given bit rates in a single test. The test generates a full spectrum of results for assessing interactivity and covers a wide range of latency measurements. SRG added this interactivity test to their measurement study to create the network performance score and gain additional insight into transport times. Measuring network latency and reliability is like testing the future today.
A free preview of the SRG report or the full fee-based version is available at the SRG website.
