This past Friday, May 21st, 2021, was exactly 34 years since my first day at Apple Computer as a tester. Before that I was a dev, but I have been a tester ever since. This has put me in a retrospective mood. The industry has gone through changes, yes, but it seems to me there […]
I want to explore the application of an important lesson from land navigation to our engineering work: always have a backstop.
Backstops in Navigation
When navigating in the wilderness, it is important to identify backstops for your journey. Fundamentally, a backstop is any kind of kind of boundary condition that keeps us within our target area and lets us know if we’ve gone off course. A backstop can be a natural landmark (such as a hill, river, or lake), a man-made landmark (such as a road, trail, or building), a distance, a specific number of steps, or a time duration. Backstops can be absolute ("if we come to this, we know we’ve gone too far"), or they can simply serve as a checkpoint for pulling out our maps and figuring out the next leg of the journey (“we’ve reached the trail intersection, let’s determine which way we want to go from here”).
We can also use backstops to introduce deliberate error, which is called an "offset" in navigation lingo. Applying an offset is a technique that can be useful for ensuring that we make it to our target. For example, we may have gone off-trail and need to make it back to the car. We can find our position on the map and plot a course directly to the car. But what happens if we come to the road, and there is no car in sight? Which direction do we head on the road – north or south? Instead of entering into an ambiguous situation, we can aim for a backstop that is offset from our vehicle in a known direction (i.e., we can pick a point that is to the north of where we parked the car). Then, when we hit the target backstop, we know the direction we need to move in to find our car.
Note: GPS has made land navigation somewhat easier, but it’s still important to use backstops for knowing when we’ve gone too far for one important reason: we aren’t constantly looking at our GPS system (especially if we need to preserve battery power on longer trips). In the time between GPS checks, we can still go significantly off course. Having a backstop can prevent this from happening.
I was recently reminded of the importance of having backstops while hiking in the mountains with my cousin and his friends. Now, these lovely folks were on a vacation, so there was no time pressure for them. I, however, had left my wife at home with a two year old and a two month old. I knew that I needed to be back by a certain time, so the first backstops I put into place were a maximum distance and a maximum time spent on the hike in. When either of those backstops were hit, I needed to turn around and head back to the car even if it meant splitting from the group early.
The goal of this particular hike was reaching a trio of waterfalls. We made it to the lower and mid falls. I wanted to catch a view of the upper falls, so I convinced the group to continue on the trail to see if we could find a nice view and a spot for lunch.
I looked at my map and selected a backstop: a creek that crossed the trail after the trail turned north. At this point, I knew we had made it past any potential views of the upper falls, and we were simply heading deeper and deeper into the mountains.
We reached the creek, and I suggested that we head back toward the cars and find a shady spot to eat lunch. The rest of the group was still convinced that if they just kept hiking along the trail they’d come to a view of the waterfall. I broke out the map and showed them the terrain, and how the trail wound away from our target. Others in the group broke out their maps to check my claim. After some more discussion, we all finally turned around. I still wonder how far they would have hiked before deciding they had gone too far!
Application to Engineering Projects
I suspect that most (all?) engineers will experience a project with a significant overrun of the budget or schedule. Sometimes this is due to an error in scoping, where we significantly underestimated the effort needed to complete a project. In my experience, however, the most frequent cause is a small handful of efforts that end up unexpectedly derailing the project. It seems that when these overruns occur, we become trapped by the sunk cost fallacy. Our natural tendency is to throw more time, people, and money at the problem, rather than changing course.
Just like in the wilderness, we can use backstops on engineering projects to identify problematic scenarios before we’ve invested too much time or money into them. This is especially important when we are working on efforts with significant unknown factors.
Examples of using backstops on engineering projects include:
If we can’t fix the bugs in this module in the next 2 days, we will throw it out and implement it from scratch
Spend no more than X days investigating this, then update the team on your findings. At this point, if there isn’t a clear path to implementing the solution in another two weeks, we will investigate approach Y. If we think the implementation is feasible, we will set a further backstop of two weeks. At that point, if it is not completed, we will switch to investigating approach Y, no matter how close you think you are.
We need to create a prototype solution that demonstrates we are able to meet the critical specs X and Y with this approach. If the prototype cannot meet those specs, we will not proceed to full implementation in our system.
If a feature does not pass reliability/certification/acceptance testing X, it must be spun off into a separate project to keep the main product release on schedule (or dropped all together).
If the feature does not meet these performance specifications by Y, we are switching to our backup plan.
If we cannot guarantee a supply for X units over Y time, we need to qualify alternative parts (or remove the feature from the product).
It’s important to plan our backstops and our responses in advance. At the start of an effort, we are not yet clouded by sunk costs. Once the effort is underway, we become like the hiking group that was so focused on seeing the upper falls that they wanted to push on, even though they were now heading away from the falls. We will ignore the warning signs and continue plodding along our chosen path. We already made it this far, why not keep going?
Aside: We are so used to post-mortems and looking for valuable lessons after something has gone wrong. What if you instead had a pre-mortem to figure out what might go wrong? This would be an ideal time to identify backstops, create fallback plans, and put risk mitigations in place.
We can draw another lesson from my recent hiking experience: it is important that the entire group understands the backstops and the course(s) of action that will be taken if you run into them. In my case, I only picked a backstop for myself. I did not inform the group. Had I done this before we started looking for the upper falls, I expect that there would have been no debate about continuing on once we hit the creek. Communication is essential. If the entire team isn’t on board, then we can expect difficulties in changing course once we hit a backstop.
Whether you’re in the wilderness or working on an engineering project, blindly continuing on a path is an excellent technique for landing yourself in an unpleasant situation. It’s important to know when we need to stop and regroup, analyze the situation, and decide where to go next. Make sure you’re doing that on your projects.