In previous blog posts we’ve discussed the results of our initial user testing. In this blog post we’ll talk about the process and challenges of getting the app into the hands of our testers. We won’t be offering loaner devices when our app is officially released, but instead a visitor will be able to download our app onto their device. As you’ll read in this post, we realized that not offering loaner devices will most likely serve us well as we move forward. Up until very recently distributing test versions of an app had a limitation of up to 100 devices. This was great in theory but those 100 devices were over the course of a whole year and surprisingly we’d already used about 20 of those device tokens just on our own internal testing. Reaching that 100 mark wouldn’t take much.
Of course the other important factor was making the process of having a visitor test our app quick and easy. Distributing a test app to someone’s device was a several step process. We were using TestFlight internally for distribution of our app which requires a tester to create a TestFlight account and ask to join our app distribution list. This then registers their device UUID—used to uniquely identify an Apple device. But it doesn’t end there, each time a new device is added to a distribution list a new version of the app and various other Apple certificates need to generated...painful at best.
So to try and circumvent the process we made the decision to use our own devices and hand them out to visitors to test with. This seemed like the easiest approach. We’d no longer have to worry about hitting that 100 device limit and we’d totally bypass the pain of installing the app on a new device. This also gave us full control over our testing environment. We’d know what iOS version was being run and set any device settings accordingly—turning notifications on etc…
We had seven iPod Touches (5th generation) left over from the very first test. We thought we’d created a perfect testing environment…
That’s when we starting learning about the limitations of an iPod Touch. In our first round of internal testing we had some of our team using their own devices (mainly iPhones) and various other team members using the iPod Touches. Interestingly, we noticed that the Bluetooth signal kept dropping out on the iPod Touches, but not once did that happen on the iPhones. This was a key feature of our app to detect visitor location using Bluetooth LE and iBeacon technology. It wasn’t acceptable to have the Bluetooth connection drop out intermittently during member testing. In addition we also saw that the wifi signal dropped out on occasion which we rarely saw with the iPhones. Again this was problematic—no wifi meant no notifications arriving to tell a visitor the question was answered. In fact questions couldn’t be asked in the first place as no means of network connectivity could be established without wifi on the iPod Touches.
We realized at this point that there was a fundamental problem testing our app with the iPod Touches. With an iPhone if the device auto locks (this is usually managed in settings with a default of five minutes) notifications will still arrive but with an iPod Touch if the device auto locks the wifi drops out and notifications will never arrive.
So controlling the environment needed a little more control. We had to set the iPod Touches to never auto lock which would keep a steady-ish wifi signal. We also had to add code to start and restart the iBeacon detection manager in an effort to keep Bluetooth enabled. We were working around the problems, but it was starting to feel a little too far from a real world scenario. We were forcing behaviors that didn’t mimic those used by real app users.
One final interesting point about the iPod Touches was also around receiving a notification. After a couple of rounds of member testing we had feedback asking for a vibration when a response to their question arrived on the device, this way they wouldn’t keep looking at the device to see if a response had arrived. Sure no problem—and thats when we found out that an iPod Touch doesn’t have a vibration mechanism—who knew! Yes we could add a subtle sound (and for testing purposes we did) but long term that sort of interruption in a museum setting is never desirable and a vibration is definitely preferred.
Along comes September 2014—Apple Worldwide Developers Conference (WWDC). During that conference Apple announced that they were partnering with TestFlight to allow even easier distribution of test apps. We’d now have up to a 1000 device limit for testing and the process of installing the test version on someone’s device was greatly streamlined. At the end of October 2014 Apple had officially integrated with TestFlight. We quickly made a decision that to really be testing our app we needed to have our visitors install it on their device. In a couple of weeks we are going to invite visitors to do just that and are excited to see more real world results!