What Mobile Testing Tools Can’t Test
There is a large ecosystem of mobile testing tools available to test almost every aspect of a mobile app or site—functional testing, load testing, compatibility testing, and more. But there is one thing they cannot test: the full impact of the wireless network on your application.
Testing a mobile app under wired or ideal network conditions is not a real test—in reality, applications suffer the majority of their issues because of variation in their network connection. For instance, the same application has to operate under a range of circumstances: when connected to a strong Wi-Fi network in their homes, a weak Wi-Fi network in an airport, or an overloaded Wi-Fi network in a busy café. Often they are subjected to periods of being completely offline. These big, sudden changes in network conditions have a serious effect on any mobile application that relies on wireless communication with a server or with other mobile devices.
5 TYPES OF NETWORK VARIABILITY IMPACTING MOBILE APPS
Researchers from Infosys Labs performed a detailed study of the main types of network variability that need to be taken into account and tested in a mobile app. Below are the top five types they identified.
1. Signal loss: A user might step out of the range of a Wi-Fi network and suddenly lose connection to the server.
2. Data loss over network: Some data might fail to reach its destination due to network congestion.
3. Bandwidth fluctuation: When multiple users are sharing the same wireless network, the bandwidth available to an individual device constantly changes.
4. Enterprise-carrier network switching: Employees who enter their offices will typically switch from their mobile carrier’s data connection to the enterprise Wi-Fi.
5. Jitter: Jitter occurs due to variation in network delays—this happens when the time it takes for data packets to reach their destinations changes rapidly.
WHY IS IT IMPORTANT TO TEST THESE NETWORK VARIABILITY CONDITIONS?
According to the Infosys study, the five issues above, if not handled properly by the mobile application, can result in:
- Unpredictable application behavior such as freezing, inappropriate error messages, inconsistent behavior, and incorrect display of the user interface
- Database corruption, in extreme cases when there is packet loss during a transaction and the client and server are not in sync
- Poor performance and loss of potentially critical data due to sub-optimal network conditions
NETWORK VARIABILITY IS DIFFICULT TO TEST
Why is it so difficult to test for network variability and see how your application reacts? First, most mobile testing tools used by application developers focus on the software aspect of a mobile app (functionality, software performance, and load testing) and are not able to test the network layer.
Second, it is very difficult to reconstruct wireless network variability conditions in a lab, because they depend on unique combinations of network parameters that exist uniquely in the user’s environment. While test engineers can physically test in realistic conditions, it is time consuming, only represents one set of operating conditions, and most importantly, is not easily replicable should they encounter an application glitch. For instance, if you test a new wireless medical monitoring device in a hospital setting and the application suddenly resets during the test, was that due to an application glitch, an issue with the wireless connection, or interference from something else in the hospital? These types of partial tests, while seemingly attractive for their real world benefits, are mostly for show. They exercise only a few of the thousands of needed network conditions.
To ensure your mobile application is fully tested, you need to complement your standard tests with insights into how your application behaves under radically varying wireless network conditions.
MOBILE NETWORK TESTING
Ixia is the leader in wired and wireless network testing, with a strong focus on mobility. Our products are used to precision-test network functions on a variety of mobile devices, verify that wireless network equipment is working as promised, and test large enterprise networks to ensure that enterprise applications are keeping users happy. Ixia’s IxVeriWave™ portfolio of Wi-Fi test products test mobile devices and apps across a large number of real-world network configurations and conditions in the lab. The world’s leading wireless local area network (WLAN) infrastructure and mobile device manufacturers, service providers, system integrators, and enterprises use IxVeriWave to measure and optimize performance, reliability, and scalability throughout the product/service lifecycle.
Testing with off-the-shelf wireless access points or testing in the field to represent different physical locations quickly proves unwieldy—with no efficient, automated means of modeling, measuring, or replicating the many possible combinations of network conditions. IxVeriWave provides a “golden access point” capability that emulates the widest variety of access point capabilities and the networks behind them. Multipurpose traffic generation modules emulate hundreds of access points and settings in thousands of combinations.
IxVeriWave allows testers to push mobile devices to their limit and see the response of the application under varying throughput and signal strength, roaming, interoperability, and failover.
WITH IXVERIWAVE, YOU CAN:
- Test your app with hundreds of WLAN access point technologies and thousands of combinations of access point settings
- Generate wireless signal in real time and perform comprehensive functional and performance testing of RF components
- Test your app in different scenarios of connectivity, forwarding rates, aggregation, and media utilization
- Emulate distance of device from access points to test how your app functions under weak or variable signal and during roaming
- Perform repeatable, automated testing in a highly controlled RF environment
- Measure quality of experience (QoE) metrics for video, voice, web, unified communications, and other types of applications
- Capture expert analysis of WLAN network protocols, data transferred, packets dropped, etc.
Learn more about how to test mobile apps and devices in different network conditions with Ixia technology.