mLearning Testing Tools & Methods

As active practitioners and consultants for enterprise mobile learning initiatives, a frequent question we get asked is “What kinds of tools can you recommend to help us test our mLearning content before we release it to our mobile workers?”  It is a great high level question and the answer is both complicated and multifaceted as each deployment environment comes with its own unique requirements that inject their own set of delivery complexities.  If you’re fortunate enough to have a highly structured and standardized delivery environment where everyone has the same BlackBerry Bold/9000 wireless handheld or your users have two or three different brands of Windows Mobile devices, your testing efforts can prove to be straight forward.  But the bigger your audience is, the greater your challenge can become.  And this may mean that full and comprehensive testing can take you as long as the content authoring effort itself – at least for simple content!

Building & Testing for a Broad Audience

Depending on the “device diversity” of your environment, you may need to create mLearning content for a wide audience who carry an even broader range of mobile devices ranging from basic feature phones to a selection of today’s hottest smartphone devices; in order to do so, you’re going to need to get your hands dirty too.  More to the point, you’ll need to test and verify the functionality, effectiveness and overall user experience of mobile learning content by trying it yourself on real devices under real world conditions whenever possible.  And if you want to ensure the best possible experience for every class of mobile learner, you’ll need to build a collection of working mobile devices, simulators/emulators and testing tools to span the potential reach of your target audience.  Leave “no stone unturned” by testing the full end-to-end experience from distribution/delivery to installation/loading to access/playback to reporting/analysis.  We commonly see teams making assumptions that because something worked fine on one device – even one from the same device OEM – it should work on others and that’s not always the case.  These variations are often the result of myriad factors like different processor speeds, available device memory, device OS versions, encryption settings, etc.

From our experience, there’s no 100% substitute for actually using a physical, operational handset to perform all your testing but this method may not be practical and/or affordable for some teams/content developers.  On the good news front, the longer you’ve been in the mobile learning field, the more likely you are to have an expanding office drawer full of recently retired but still functional mobile devices; all they generally lack is a SIM card module to activate them on a particular carrier network and anyone with a little skill and patience can quickly get comfortable “swapping cards” from one device to another to perform their structured testing protocol (it’s always a good idea to develop one) for any new mobile learning course they plan to deploy. 

Those teams that don’t have full device drawers or enough representative physical devices can also try using available device simulators and emulators  which can generally be downloaded (for free!) from the device manufacturer’s support site or other common web locations.  The price is right but, remember, the experience will not the exactly the same as the real thing especially where possible concerns about access security, content encryption and download speeds are concerned given simulated phones attached to broadband Internet connections are not a perfect equivalent.  After many years of using available simulators and emulators, we’re now comfortable and quite familiar with where they work and where they don’t too; they can certainly be used for most of your initial testing and content verification exercises.  Finally, the best simulators/emulators are from the device OEMs themselves including RIM/BlackBerry, Google/Android, Nokia/Symbian and Microsoft/WinMo.  If your team is developing native Apple iPhone and iPod touch applications, a fully functional simulator can also be accessed using Apple’s Xcode IDE too.  Don’t forget your testing environment may also need to be expanded to include appropriate devices for testing 1-way and 2-way SMS messaging across multiple carriers as well as voice/IVR-delivered services on entry level phones.   Finally, it may also be practical to have a few other non-phone mobile devices on hand like a Netbook computer or an Ultra-Mobile Personal Computer (“UMPC”) to test content delivery to alternative mobile devices wherever appropriate.

The image below is of my office desk and it presents several of the devices and tools we keep on-hand to ensure that all mobile content prepared for wide distribution works as intended across the broadest array of smartphones (old and new), basic feature phones, netbook computers, specialized ultra-mobile personal computers (“UMPCs”) and even spanning carriers and wireless delivery methods (GSM vs. CDMA).   This picture demonstrates how we use a combination of both virtual and physical devices and we’ve certainly learned from experience that there are minor yet myriad differences between a real and simulated playback experience – in short, the only 100% verification test must be performed under the same target delivery conditions using a physical device across an actual wireless network. 

So, what’s in your drawer? My physical mobile devices and virtual tools for testing include:

a. Windows-based RIM BlackBerry Simulators (for all devices & carrier-specific) – we have about 10 of these we use regularly.
b. Mac-based Apple Xcode-based Simulator (for all iPhone & iPod touch device for testing apps)
c. Android G1 and G2 Emulators (for all 1.x, 2.x devices)
d. Windows Mobile Emulator (for WinMo 5, 6 and 7 using VMware Windows partition)
e. Nokia Symbian/S60 Sims (using VMware Windows partition)
f. Windows -based Netbook (for Netbook applet testing); we also have an Android Netbook
g. Sony basic feature phone (for voice and SMS testing on T-Mobile)
h. RIM BlackBerry 9000 smartphone (media support and encryption on ATT)
i. RIM BlackBerry 8703 smartphone (limited media support)
j. Vulcan Flipstart Windows-based UMPC device (1024 x 768 display)
k. RIM BlackBerry Storm2 (full media support and encryption on Verizon)
l. Nokia 5800 smartphone (Symbian 60/v5 testing on ATT)
m. Android G1 smartphone (on T-Mobile)
n. Jitterbug basic feature phone (voice/IVR and SMS testing on MNVO)
o. OQO Windows-based UMPC device (800 x 480 display)
p. Apple iPhone 3G (on ATT)
q. RIM BlackBerry 8310 Curve (no Wi-Fi, no encryption on ATT)
r. RIM BlackBerry 8330 Curve (with Wi-Fi, unlocked for GSM carriers)
s. RIM BlackBerry 7210 phone (very limited media support)
t. RIM BlackBerry 8800 World smartphone (on ATT) 

Next time I’ll explore some of the fee-based mobile testing solutions and alternative platforms you can consider instead of investing in all the physical devices and long-term carrier contracts required to replicate our current methods.