"We work in technology. Technology breaks."
February 26, 2010
By Jeff Freund
One of my favorite quotes ever from a customer was during the negotiation of SLA's a few years back. Their CIO's simple statement of "We work in technology. Technology breaks." was a very powerful statement - this acknowledges that things will always go wrong with technology, and rather than holding us to a measure of perfection, they were more interested in having rigorous expectations about what happens when things do go wrong.
A standard practice that we follow at Clickability is the creation of Post-Mortem reports for anytime that something does indeed go wrong. Whenever something negatively impacts our services, we create a report that detail the following:
- Incident summary - what went wrong and what was effected
- Incident time line - when and for how long the problem existing
- What happened - details of what let to the problem, be it hardware failure, software issues, a process breakdown, human error or a combination of such.
- What was done to fix it - how did we restore the services to normal operations
- How we will prevent this in the future - details of the immeediate and future infrastructure changes, application updates, process improvements, monitoring additions, or any other means by which we believe we can prevent the issue from happening again.
These reports are not only distributed internally, but also sent to customers and posted in summary on our public status portal.
It is this sort of transparency that establishes trust, which is clearly evident from the following email that the support team received this week:
As I've written before, I greatly appreciate these post-mortems. So many other companies try to sweep problems under the rug, whereas the honesty of taking responsibility for a problem and owning up to it let me know that I can rely on you in the future.
This illustrates why this transparency is so critical in a business like ours - it is what builds and maintains long term relationships. We work in a business that involves technology and people - there will always be problems of one sort or another. It is how we respond to such problems that ultimately defines who we truly are to our customers.