Introduction
Five years ago I started farming. I began farming because I enjoy eating well and wanted greater control over that process. Around the same time managing people became a much larger part of my day. I went from task leader and Project Manager to Engineering Manager. I have learned, time and again, lessons from one role that inform the other. Here are three of those lessons.
Plan for bad weather and hope for good.
No matter how well prepared you are, you are subject to the weather. Did you expect to plant a few acres of oats today? Not if it’s raining. But waiting a few days won’t ruin your year on the farm.
From another perspective: Did you expect to deploy your shiny, new AWS managed database instance but, mid-deployment, that service went down in your region?
The weather taught me to plan for all contingencies. If you are deploying software, contingency means having a rollback plan. If you are backing up company data, it means having a verified and comprehensive backup strategy. Remember, it’s only the end of the world if you didn’t plan for it. Our rule of thumb is that there should be a contingency plan in place for any action over 4 man hours. Whether it’s one line or a page - it should be a succinctly written record of what to do when a failure occurs. Our teams know what success looks like. What happens when things go wrong? Before rolling out changes, we ask ourselves, what could go wrong and what we could do about it. Using this strategy we developed a library of playbooks to deal with different failure modes.
Even in the worst-case scenario, your playbook should hold you over until you can deploy the right solution. Which brings us to quick fixes, AKA “technical debt.”
Regularly address “quick fixes” and technical debt.
Something went wrong in planning, architecture, or execution. It needs to be fixed right now.
This calls for a quick fix because time is of the essence. Quick fixes are the most visible example of “technical debt.” Technical debt is a term used for the accumulated costs of moving from quick fixes to doing things the “right way.”
One rainy day, your livestock walks through your electric fence as if it isn’t there. So you check your fence energizer. Determining that it is working, you start walking the length of your fence, looking for a problem. When you get the furthest away from your fencing materials, you find a tree has broken off a fencepost. You decide that you can fix the fence right now by taking a branch from that tree and making a hasty fencepost with it.
The technical debt of this decision is immediately clear. You made this decision because you didn’t want to walk back and forth in bad weather to make the fix the right way. A classic cost/convenience tradeoff.
But what’s the lifecycle on that fix? Will you really fix it on the next nice day? Or is your fence held together with a long series of “good enough” fixes?
Is your software held together the same way?
Technical debt can slip through the cracks. On the farm I carry a notepad to record these “quick fixes.” Each month, I look over our infrastructure: fences, waterers, barn doors, and equipment. The result is a comprehensive list of our technical debt. I know exactly what needs maintenance that I didn’t expect, what quick fixes need fixing, and where to schedule that work so that it doesn’t disrupt the day-to-day operations. Is there something huge that needs fixing immediately and should stop all other work? Rarely. Scheduling a couple hours over each week is enough to keep all the cruft from accumulating.
The same is true in software. Our team schedules time into each sprint to catalog, schedule, and repair things. Or “quick fixes” that have marched on for months or years.
With the right tooling and planning, neither software nor fences need to live as an unreliable patchwork.
When to buy tools versus when to make do.
Few people jump into farming who don’t have it in their recent family tree. The myriad of tools and equipment alone make it difficult. So you start small with a few hand tools, a sledgehammer, some wire, and you are making infrastructure. You build fences, animal waterers, and before you know it, you built the whole thing with “spit and bailing wire,” as the saying goes. You don’t start with a $500,000 tractor to work 10 acres.
It is exactly the same in software.
Jumping into the “gold standard” in any industry is costly. No newcomer should begin their journey with a $500,000 tractor or a $50,000 SaaS/PaaS bill. You only understand your true needs once you have a mature process. How do you know when you need that new, expensive tool? Easy, by answering the question, where does it hurt?
Years ago I started building lots of electric fences so I could rotate my livestock around on the property. When I wanted to test if my fence was working well, I would grab the hot wire and get a small shock. Unpleasant, but telling. I would know immediately that things were working.
Over the years I built larger fences with more powerful fencers. If you test these with a grab, the shock would cause a grown man to cry out. So I tested it with a grounded wire and watched the spark jump to know how well the fence worked. While that works, it is hard to quantify a spark.
Low and behold, a fellow farmer said to me “a good fence tester is only 40 dollars.” The proverbial lightbulb turned on. When you are spending so much time and effort starting out, every dollar counts. If you bought every last tool your team “might need,” you’d have a huge bill and a shed (or server) full of expensive tools that were used once. So you begin as simply as possible. But once you really know the pain you want to avoid, you won’t believe you went so long without spending that $40.
I see this in infrastructure and tooling regularly. People don’t stop and ask themselves “Are we still doing this the right way? Is this the easiest way to do this?” They have been doing it one way for so long, they don’t stop to ask themselves where it hurts.
Is it painful to release new features? Deploy them? Roll back the deployment if there’s a problem?
Once you identify that pain point, you will realize that you don’t have to grab that electric fence anymore. There are better solutions.
Once you know where it hurts, the right tool makes all the difference.
I’d love to hear from you:
- How does your team plan for contingencies?
- How do you address technical debt?
- What pain point pushed you to a better solution?
This is the start of a series on Engineering Management where I tie together two things that I enjoy - delivering great software and working on the farm.