Data Reliability and Our App Customer Service

We have been excited to see the wonderful reception that Popplet has been receiving lately. Recently, we have been:

• Mentioned at a leading educational technology conference
• Listed as Apple staff iPad app favorite, and
• Featured in blogs on everything from social enterprise development to business productivity!

With so many new users coming on board – as well as our current users finding new ways to make use of Popplet daily – protecting your data and responding to your questions through our app customer service has been our top priority. We keep track of all requests for enhancements to our web and iPad apps, and we will prioritize these ideas for future upgrades, but right now our focus is on fortifying our data servers in the wake of our recent growing pains.

As we launched this blog yesterday, our server systems had a grumpy Monday morning wake up. Thanks to Popplet users (Poppleteers?) for tweeting us the minute you noticed some login problems:

• @DA_Chris_Mason
• @pupmp
• @rseneta
• @Hayz_10_10

We do realize how frustrating problems are when you can’t access your data. For teachers on a Monday morning, it can impact on how you approach the whole school week, and for productivity and business minded users, it is incredibly annoying and anger-generating to lose an hour or more of your work.

Here’s what happened:

• To safeguard your data, Popplet uses a cluster of data storage servers. The cluster is designed so that if any individual server fails, the other servers will step in and allow you to seamlessly access your data without a hitch.
• On Monday morning, one of the data storage servers suffered an unexpected outage, but due to a misconfiguration, the cluster did not route around that downed server and instead became unresponsive (it became a bottleneck instead of a detour).
• We have since reconfigured the cluster in our Popplet system so that in these cases, it will route your access requests around any server that isn’t working at the time.
• For most users, this meant that they could not access their Popplet account for up to three hours on Monday, as when they tried to login, they were sent around in circles. For an unlucky few hardworkers who were already in their Popplet account, this meant the server didn’t save about an hour’s work.
• We have since installed further safeguards and during one of our quiet times (when the least number of our users needed to access their Popplet accounts), we ran some tests switching off one server at a time to make sure that accounts are routed correctly to the back up or primary data storage source as necessary. We started testing this overnight and the configuration operates as it should, but we will also be running more robust tests in the very near future.

To let everyone know what was happening, here’s what we did through our app customer service:

• Responded to individual tweets by direct message or publicly to acknowledge the problem
• Escalated the issue priority to our tech leader to address the server issues immediately
• Emailed anyone who asked us what was happening and remained in contact with them until the problem was solved
• Tweeted to let our 1,000+ followers know what was going on and when the system was restored.

Is there something more that we can do to alert you in future? In the comment section below, or on our Facebook page, please share your ideas about how to best communicate with you about using Popplet, and about the app customer service standards you expect from us. Don’t forget to subscribe to our blog Poppletrocks! or follow us on twitter to make sure you know everything that is happening with Popplet.