A few months ago I took the challenge to teach deep, technical troubleshooting to anyone who’s interested. So far in my team, only the folks with some coding background worked on complex technical stuff. Nobody ever taught us to troubleshoot, we kinda learned it as a side effect of coding, and trying to fix our own broken code.
Even though we work in customer support, technical skills are quite handy. WordPress can be customised in infinite ways, which is both its beauty and its curse. There are more possible configurations of our products than atoms in known Universe. It’s actually surprising that most of what our customers report are relatively known issues, with a relatively known way to solve. Yet about every one in a twenty is a unique and complex case, where all routine procedures break down, and you need to take the whole thing apart and work your way from ground up.
So far these 5% of requests would take a significant portion of our time and effort, and usually require someone code-savvy to bring them to a close. Now my job was to equip more people with the right tools to tackle such problems, regardless of their background and what they already knew.
Nobody actually teaches troubleshooting. There’s no training materials on that. No college has a class on troubleshooting in their curriculum, there’s even no online course about it. All the people who know how to do it, learned it incidentally while trying to do something else.
But I know it well that almost everyone can learn the basics of some coding language, and I’ve successfully helped a few people do it. Why wouldn’t anyone be able to learn how to debug the code?
So how do you teach something that nobody ever tried to teach?
First, you need to write down what kind of things there are actually to be taught. This is much harder than I thought it would be. All these things that are obvious and come without much thought to you, you suddenly realise are completely alien to almost everyone else.
I used to wonder why people don’t sometimes check quite obvious things when trying to solve a problem. Only recently I realised, these things are not obvious at all to non-technical folks. To me it’s natural to check the logs as the first thing, and compare the timing in logs with the timing of other events that happened on the site. Most people, even when they open the log file, don’t really see all the tiny hints about cause-effect chains that the logs can reveal. They don’t even know where to look and what for.
Discovering these unknown unknowns was hard, and for the most part, quite ridiculous. I thought I’d record a screencast of me working through a complex problem and use it to teach my process and tools. I got 25 minutes of video where I randomly check different fragments of code, output random variables to log, and murmur to myself “I have no idea what’s going on, but let’s try this and see what we got here”. 5 minutes after I gave up on recording, the answer came to me intuitively, when my brain connected some seemingly unrelated facts. How could I ever teach someone to do THIS?
Yet after weeks of working through complex problems, and writing down what tools, techniques, and chains of reasoning we used, my team came up with a rough list of what it is that we’re doing differently than most folks. The list itself was 6 pages long, not including materials on any of these topics. Writing even a short note on each seemed like a daunting task.
Fast forward another few weeks, we knew at least what sort of training materials we already had. As it turned out, we did have internal docs of some sort about many topics from our list. Some other topics, like WordPress actions and filters, or how HTTP works, are described pretty well online. I ended up with a lot of links, and about an equal number of gaps, the things no one had ever written about in the context of our work.
It would be a full-time job for at least one person over the course of a month to fill in the gaps. It would take another to build a well-rounded training program, with all needed information, exercises and tests. We didn’t have two man/months to spare. Since I’ve taken over this project, we’ve gone from 8 people to 4, and were barely keeping up with day-to-day work.
And then we realised, no one would actually have the time to complete such a training. Everyone had their day-to-day responsibilities as well, and working through all the modules would take them weeks or perhaps even months. By the time they encountered their first problem with WordPress Cron, they’d already forget what it is, let alone all the tips on how to troubleshoot it. We had to come up with something else.
It wasn’t what I imagined as a proper training
Here is a list of all the things we know we do, here is another one with all the materials we have about these things. The first list is most probably incomplete, the second one obviously is, I highlighted the gaps in red. Rate yourself on each topic on the scale 1-5, then pick 3 areas out of those where you scored low. This will be the core focus of your training. We’ll assign you the tickets on your chosen topics, be around to help whenever you get stuck, but in the end it is your responsibility to get these cases solved. After 4 weeks, let us know what you’ve learned, and write at least one article to share this knowledge with peers.
I was anxious of how half-baked it sounded when I was announcing this. It went against all my intuitions of how a proper training should look like. I thought we should at least have our documents in check. Once we at least organised all materials into a field guide, and added a few exercises that people could practice on, we would be kinda ready to point people there, so that the can learn what this whole troubleshooting thing is all about.
As it turns out, what we did was the best possible thing
I’m glad we didn’t wait until we’re “ready”, cause we would never be. We can only accept one or two students at once, but they’re getting what they needed the most. If we followed the mass-schooling script of teaching everything to everyone trough lectures, exercises and tests, I’m sure we could have much more folks go through it at a time. But I doubt they would learn more in result.
Complex problems are complex because they’re unique. You’re looking for a black cat, in a dark cellar, that isn’t even there. You don’t know what the outcome will be, because you’ve never encountered something like this before. If you knew, you could apply one of the well-known procedures and your job would be done.
Sometimes such routine procedures will seem like magic to someone who’s never tried them. But there’s no discovery involved there anymore, just a simple if-then. The first time I came across a site broken due to auto increment not set in the database, it took me a few hours to figure what the heck was happening, then another one to clean up the mess. Now I can identify such problems and fix them casually while chatting with a few customers at once.
Of course, documenting the routines is super helpful for everyone involved. It’s counterproductive and wasteful to reinventing the wheel, and we could do a much better job writing down everything that’s routine for us. Especially the newcomers on the Woo Happiness team could benefit from a standard catalog of known problems, their causes, and techniques to fix them.
Yet the troubleshooting process itself can’t be written down as another routine to follow. There’s no if-then decision tree that will take you from all possible starting points to the perfect solution of each problem. The essence of troubleshooting lies in being able to use your judgement and work your way through an issue that has no known answer yet. Most likely, there will be some familiar cases that this one resembles the most, and some tools and techniques that make most sense to apply. You’ll need to keep an eye on the big picture, adapt those tools and techniques to the situation at hand, and evaluate the progress, even if you have no idea what you’re looking for.
We could write all the articles on all the topics we’d ever come across, make them accessible to everyone on the team, and do video walkthroughs to help them memorise it. And quite hopefully, we’ll get there one day. Still, this would be no guarantee that the students can solve a similar-but-different case. Routine problems still make up 95% of the requests we get, and we’d be all better of if everyone knew exactly how to deal with them, but there will always be unknown unknowns. In these cases, only understanding why we’re using this particular technique, and what limitations it has in this particular context can help us avoid slipping into a Cargo Cult.
Nothing prepares for a troubleshooting quest like doing it already, only narrowed-down to certain areas. So what that at the end of the training you won’t know all the things written in our six-page long list? Neither do I, we all pretty much make it up as we go.
There is no such thing as teaching someone
There’s only setting up the environment for the student to walk through. But it is the student’s job to find their own way. It’s their job to understand what kind of a thing they’re actually looking for, to find all the best tools for the job, to make a guess and evaluate the results. Of course, they can, and will, ask for help, but the decisions and choices need always be their own.
If you guide someone by hand and tell them where to look, they will always rely on you telling them if they’re doing it right. Without a deep understanding of the Essence of whatever they’re doing, whether that’s troubleshooting, meditating, playing guitar, or building planes, they’ll be only able to repeat someone else’s routines, unsure what they mean, and why they are working. In the long run, people would go through the motions that had lost their true meaning centuries ago. Many things might have made sense when they were first established, but if you don’t know what their meaning originally was, you won’t be able to tell when it’s there no more.
Our training doesn’t come in a pre-packaged format. It’s not an Ultimate Solution to All The Things. And it certainly won’t guarantee that as you finish it, you’ll know what to do with every problem you encounter at work. But it will show you what sort of questions to ask, how to probe things, and how to cut through all the irrelevant details right into the core of the problem.
Perhaps this is the only sort of a training there is.