Coverage and performance testing of 5G private networks

Are we trying to reinvent the wheel?

Manufacturing is already very automated and many ask: “Aren´t existing manufacturing processes good enough?” The current demand for transforming manufacturing processes we now see mainly comes from the business side. There are limits to the cost reductions possible using standard approaches. Many consultancies base their business models on delivering cost-cutting projects to customers, where they examine the entire production chain to identify potential for cutting operating costs. Their measures are struggling to produce the expected reductions and cost pressures continue to rise, making innovative approaches necessary.

5G technologies address this challenge at its source for faster and safer operations, while creating new capabilities in industrial processes. This approach allows factories to connect various facility components over 5G networks, including manufacturing robots, AGVs or other transport vehicles. They can be managed more easily and production lines can be quickly refitted to shorten lead times. 5G can also replace existing WLAN networks in production facilities to overcome WLAN limitations for seamless mobility, stronger security, greater efficiency in the manufacturing process and decreased operating costs.

Coverage and performance testing

Keeping in mind all these requirements, a network must provide the best from the entire 5G triangle:

1) mMTC – helps networks handle many connections in parallel when there is a high number of connected devices
2) eMBB – enhanced mobile broadband helps when high broadband and high throughput are required (e.g. remote video inspection via 5G at Lufthansa in Hamburg)
3) URLLC – ultra reliability ultra-low latency services (e.g. low-latency based robot communication)

As our series has already discussed, private networks are implemented in five phases. After rollout preparation and site acceptance testing, the third phase focuses on coverage and performance testing.

Solutions from Rohde & Schwarz mobile network testing have been in use since the creation of the very first 5G engineering use cases and pre-commercial trials of 5G private networks. Our 5G scanners (R&S®TSMA6 and R&S®TSMA6B) in combination with our drive test software R&S®ROMES4 have given us the lead in 5G network testing. The setup enables passive testing. Simply by providing the wanted bands for scanning, the setup automatically identifies on-air carriers and their operators, decodes master information blocks (MIB) and system information blocks (SIB). The information makes it easy to find broadcast details for every on-air operator or measure each beam and cell.

However, to completely understand the quality of a 5G network used to connect robots or AGVs, active measurements with a 5G device are needed. These measurements let you view performance from the user perspective and verify the uplink access behaviour, mobility procedures, layer 3 communications between the device and network, as well as the QoE and QoS at the application level.

The Rohde & Schwarz answer to private network testing challenges

Private networks are mostly standalone and implemented in a dedicated band, currently usually in the Sub6 GHz band. They have their own MNC and MCC coding and specific SIMs that are used to connect to the network. Below we present the Rohde & Schwarz setup used for testing private networks. It consists of an R&S®TSMA6B scanner with R&S®ROMES4 software. The setup is complemented by an RM500Q industry module for active testing and a tablet for remote connection to control the scanner and software. A device running QualiPoc can be connected in parallel to measure networks from a smartphone perspective.

Passive (scanner) measurements in private networks

Industry environments (such as production lines with many robots) have very harsh multi-path radio propagation with high interference levels from various machines and robots. Our scanners can properly test many of these challenges in advance:

• Automatic channel detection (ACD) can detect all on-air channels with their bandwidths. MIB and SIB decoding provide insights into the broadcasted information (such as TDD patterns).
• Coverage and beam verification where beams are measured with one scanner for comparing the actual radiated beam pattern with the planned one.
• Time synchronization in TDD networks is important to prevent interference between symbols in an unsynchronized network. Time of arrival (ToA) measurements can help detect broken or unsynchronized components.
• Power spectrum measurements provide insights into the spectrum and can help identify potential interferers in the used band. They show the quality of network filters, since private network spectra are often close to public ones.

Active measurements of an industry based module in private networks

Passive measurements are sufficient for the issues above. However, connection related details (such as uplink performance, radio access control, mobility control or application layer performance) require an active device with a SIM card. We have integrated the industry-based module Quectel RM500 into our measurement solutions for such scenarios. The Qualcomm x55 based module is on a development board with a USB-C connection and a SIM card slot. This is an unmodified, commercially available module, connected with a USB-C cable to a scanner (above picture). It can measure networks from the user perspective, such as a robot or an AGV.

By complementing the passive measurement setup with a Quectel module, we can control the connected module with R&S®ROMES4 software and configure various tests to verify all major aspects of the 5G triangle:

• Network connectivity of many connected devices is tested with a ping test
• URLLC of 5G private networks is tested with an interactivity test
• eMBB (Network throughput) is tested with an iperf3 test

Most private networks cannot be accessed from outside. The tests above are best performed using a server we have installed in a private network. This server handles all active measurements (ping, Interactivity testing and iperf3). This allows us to perfectly mimic the real-life scenario of robots or AGVs communicating via 5G to the network in an industrial environment.

What have we found?

We have been very active in measuring various private networks that have been installed in agricultural segment, automotive production and the chemical industry, as well as those designed as test-beds at universities or exhibition centers (e.g. the Hannover fairground). Most private networks come from well-known suppliers, such as Nokia or Ericsson. However, we also see OpenRAN growing more popular.

Ping testing verifies a connection and how quickly the network responds. The test sends a 32-byte packet to the server within the private network. The packet sizes can be adjusted with R&S®ROMES4 as needed. Our recent measurements have revealed very good ping values, some even below 10ms. This is still far from the URLLC promise of 1ms latency for 5G, but it is a good sign for the future.

Interactivity tests use predefined patterns against a server installed in the private network. Ping testing is good enough to test end-to-end connectivity. Understanding network performance closely resembling real-life scenarios (such as Industry 4.0 process automation) is possible with our interactivity test that we developed in house based on TWAMP. R&S®ROMES4 provides a long range of various patterns, each of them with a specific:

• number of packets sent per second
• packet size
• packet delay budget
• test duration

We engineer various patterns and offer them for testing appropriate use cases and latency in 5G networks. Here is an example of one measurement campaign that measured various patterns:

The roundtrip times (RTT) for such patterns here are a bit higher than those for pure ping tests. This was expected, since the length of such patterns corresponds with real traffic patterns and should be seen as a more realistic measurement than ping tests.

Every pattern in the table had an extensive metric (RTT, packet delay variation etc.). Users can also zoom into the details and find out the latency per packet, as seen in the chart below. This helps identify network locations with high packet delay for troubleshooting and optimizing them.

Iperf3 measurements were usually within the range of 800Mbps – 1Gbps (tested from the module against a server). The measurement below used the UDP protocol with 10 parallel streams. UDP is the main protocol for real-time applications since it avoids additional, uncontrollable traffic with acknowledgements and retransmissions. We have generally seen UDP protocols deliver higher throughput than TCP ones. Please note here that we used an empty network with no additional users, which needs to be taken into account when evaluating the overall quality of the eMBB.

Here is an example of a TCP vs. UDP iperf3 measurement:

For troubleshooting potential interactivity issues (e.g. high packet delay variation) or throughput problems, R&S®ROMES4 can examine network performance at the TTI level. The tool can store the data from the TTI level to provide very detailed insights into potential issues from multiple network layers. R&S®ROMES4 customers use this feature for troubleshooting throughput drops, network access issues, high block error rates, etc.

Here is an example of the 5G TTI view with detailed insights into RACH failures with a clear overview of retransmissions, modulations, TA, RACH requests etc.

For troubleshooting potential mobility issues, R&S®ROMES4 can present very detailed mobility analysis based on the layer 3 communication between the UE and the network. The innovative approach in the tool analyzes the layer 3 message flow to identify reconfigurations, handovers and potential network issues, while identifying the main KPIs.

The entire radio ressource management procedure for 5G is explained in a very detailed manner in our recently released education note, which is accessible here: Smart analysis and optimization of 5G radio procedures | Rohde & Schwarz.

Conclusion

Private networks are becoming more important in various industries. At the moment, Germany is a leader thanks to the government’s quick reaction in assigning part of the frequency spectrum to private networks. We expect other European countries to follow soon, primarily France and the UK. Relying on public 5G networks for manufacturing processes is not an option and many manufacturers are choosing to implement private 5G networks instead. Only they can meet demanding requirements for high reliability, high performance and, most importantly, high security. If manufacturers want to control the entire automotive production with a 5G network, they will choose a private network and not a public one. The industry still seems to be in the trial phase and different Industries are playing around with private 5G networks and testing various use cases. Our experience with recent measurement campaigns shows that we still have a long way to go to meet the targets for latency and throughput in high traffic networks, but the future looks bright and we are on the right path.

Learn more about 5G private network testing and watch our on-demand Webinar .