I used to think that writing a fat suite of automated regression tests was the way to hold the line on software quality release over release. But after 12 years of pursuing that goal at various companies, I’ve given up. It was always doomed to fail.

In part, it’s because I’ve always had to automate tests through a UI. When I did straight record-and-playback automation, the tests were enormously fragile. Even when I designed the tests as reusable modules, and even when I worked with a keyword-driven framework, the tests were still pretty fragile. My automation teams always ended up spending more time maintaining the test suite than building new tests. It’s tedious and expensive to keep UI-level test automation running.

But the bigger reason is that I’ve made a fundamental shift in how I think about software quality. Namely, you can’t test in quality – you have to build it in. Once code reaches the test team, it’s garbage in, garbage out. The test team can’t polish a turd.

Writing an enormous pile of automated tests through the UI? Turdpolishing.

I’ve worked in some places where turdpolishing was the best that could be done. Company leadership couldn’t bear the thought of spending the time and money necessary to pay down years of technical debt, and hoped that building out a big pile of automated tests would hold the line on quality well enough. I’ve led the effort at a couple companies to do just that. We never developed the breadth and depth of coverage necessary to prevent every critical bug from reaching customers, but the automation did find some bugs and that made company leadership feel better. So I guess the automation had some value.

But if you want to deliver real value, you have to improve the quality of the code that reaches your test team. Even if the software you’re building is sitting on a mountain of technical debt, better new code can be delivered to the test team starting today. I’m a big believer in unit testing. If your software development team writes meaningful unit tests for all new code that cover 60, 70, 80 percent of the code, you will see initial code quality skyrocket. Other practices such as continuous integration, pair programming, test-driven development, and even good old code reviews can really help, too.

But whatever you do, don’t expect your software test team to be a magic filter through which working software passes. You will always be disappointed.

Three-point estimation is a critical skill you should build right now

Maybe you’re one of the lucky ones.

Maybe you work in a shop that executes scrum well. Your team keeps the backlog well groomed, understands its velocity, and delivers working software sprint after sprint. Or maybe you work in an open research-and-development shop where the journey is as important as the result, and projects are always open-ended.

Madison Bank clockIf, like me, you have to deliver software on deadline using a less-than-perfectly executed methodology, this post is for you. It’s going to give you a technique for knowing where you stand against your deadlines – a very useful skill that will reduce the chaos you experience and help your boss steer projects more successfully, both of which are good for you. It is a method for implementing continuous reestimation, which I wrote about in this post. Read it to know why this will make you and your boss more effective.

Teams that have worked for me and have estimated using this technique have seen their projects come in within estimate about 80% of the time. The other 20% of the time, they learned important things about their work that let them estimate more accurately the next time.

Here’s the technique:

  1. Break down your work into tasks.
  2. Estimate the tasks using a three-point formula.
  3. As you work, reestimate every day.

Break down your work into tasks

You might resist this step because in your core you know you will miss some tasks. It’s okay. This process honors that you don’t know what you don’t know. Just outline the tasks as best you can.

But think small, because you’ll be more accurate when you estimate more small things than fewer big things. Break things down as small as you reasonably can.

Estimate the tasks using a three-point formula

For each task, then, think of the following:

  • If everything goes right on that task, how long will it take? This is the best case, B.
  • If everything goes wrong on that task, how long will it take? This is the worst case, W.
  • If a reasonable amount of things go wrong on that task, how long will it take? This is the likely case, L.

Think for a minute or two about each task, but any more than that is probably overthinking it.

Important: Calculate your estimates in hours. Not days, not weeks, hours. There’s too much slush in days and weeks. Besides, in an 8-hour day you probably work just 6 hours on project tasks. You spend the other two hours in meetings, having hallway conversations, using the restroom, and being interrupted.

Here is where some magic comes in. Just go with me on this. For each task, drop B, W, and L into this formula and calculate the answer.

\displaystyle Estimate=\frac{B+4L+W}{6}

You’re weighting the likely case, but honoring the best and worst cases – and in one deft motion you take fear out of your estimates. What makes estimates bad is either unexpected things happening or overpadding for unexpected things happening. Thinking best/likely/worst helps you be more objective. And when you add up these three-point estimates for all your tasks, you get a reasonable cushion for the things that could go wrong.

Divide the number of hours by the number of hours per day or week you will work on the project. Lay that number of days out on a calendar; tell your boss that’s when you’ll be done.

As you work, reestimate every day

You are going to discover tasks you didn’t think of at first. And some tasks are going to take longer than you originally thought. When this happens, estimate the new tasks and/or reestimate the task that’s taking longer. Recalculate your end date and break the news to your boss.

Your boss ought to kiss you right then and there, by the way, because you gave him or her information early enough to do something about it – adjust scope, add resources, move the date, or tell you that unfortunately you are looking at some late nights. Hopefully the answer will be a, b, or c more often than d.

And every time you have to reestimate, you’ve learned something about your work that helps you estimate more accurately in the future.

If you want to ship software, stay in touch with how much you suck

My colleague Matt Block recently posted on his blog a link to an article about how a software shop’s business model affects how well agile scrum works for them. It breaks business models down into emergent, essentially meaning that the company builds product to meet goals such as selling ads or driving traffic, and convergent, essentially meaning that the company builds product that directly serves a target market. The article argues that agile is made for emergent and is a poor fit for emergent. That’s just a sketch of the article; go read it to get the full flavor.

I’ve always worked for companies following convergent business models. We’ve made our money by selling the software we created, which made it always important to deliver a certain scope by a certain time. When those companies implemented agile scrum, they could never fully adapt a key principle of it: when it’s time to ship, you ship whatever is built. In a convergent world, scope is king; you ship when everything specified is built.

I e-mailed my brother, Rick Grey, a link to this article. It’s great to have a brother who does the same thing I do for a living as we can talk endlessly about it. I thought we’d have a conversation about how to scope an agile project, but instead he had a brilliant insight: What if agile is good for convergent-model companies because it tells you sooner how much your project is off track? He gave me permission to share his e-mailed reply, which I’ve edited.

What if the companies we’ve worked for and all the other convergent-model teams of the world are doing agile just fine? By “just fine” I mean “as good as they do waterfall,” which may not be “just fine,” but we’ll get to that in a minute. Meanwhile, consider:

Long waterfall project:

    • No one pays real attention to progress (there’s always next month to catch up)
    • Engineers go dark, checking out huge sections of the codebase and not merging them back for long periods
    • Engineers (who are notoriously poor estimators) claim 50% done when it’s really about 25% – and then, as the code-complete milestone nears, they (usually innocently) claim 90% done when it’s really 70%
    • A couple of days before the code-complete milestone, engineering finally acknowledges they won’t hit the milestone and delays delivery to QA – “but we’re 95% done, for sure”
    • Under the pressure of already having missed a deadline, developers quietly take shortcuts to make it possible to hit the new QA delivery date
    • Weeks and months of unmerged changes come crashing in, creating conflicts and compile/deploy problems, further delaying delivery to QA
    • QA, now staring with a multiple-week handicap on an already-too-aggressive schedule, quietly takes its own shortcuts
    • QA finds hairy showstopper bugs and so the ship date gets moved
    • Management is livid, so QA goes into confirmatory testing mode just to get it out the door

Agile project of the same size:

    • Much of the above happens at a smaller scale, one iteration at a time
    • You fail to deliver everything planned starting with the first sprint
    • Instead of spending 80% of the project thinking you don’t suck as an organization and the last 20% realizing that you do, agile lets you feel like you suck every step of the way
    • Takeaway for management: “agile sucks” and/or “we suck at agile”

I assert that most teams are bad at delivering under a convergent business model.  The hallmark pathologies of software delivery under a convergent model are too numerous and powerful for most teams to overcome, but their struggles are masked by waterfall until the end. Agile surfaces the problems every iteration. You feel like a loser by week 4 instead of week 40.

But this is actually a win. You get better project visibility and a tighter feedback loop, meaning you’ve got a better chance to make adjustments earlier to get the most out of your team you have. Embrace the feedback loop as a chance to make things better, and learn not to view it as proof of how much you (collectively) suck.

I will add that agile also helps you keep resetting expectations within your organization, because it makes it standard practice to keep reestimating what it will take to finish everything. This is just what I was talking about in my last post (read it here).

The power of continuous reestimation

Not long ago a software developer I work with suggested that building software could not, and so should not, be managed to a delivery date. “You can’t know everything you’re going to encounter while you build it,” he said, “and so it just takes the time it takes to get it done.”

I said that I agreed: you don’t know what you don’t know, and the work takes the time it takes. But I couldn’t agree that projects should be open-ended, finishing when they finish.

Capitalism interferes. Pesky, pesky capitalism. Most of us make software that achieves some corporate goal about making piles of money. Because the company has plans for that money, it needs to know when it will start rolling in. The money can’t start rolling in until you ship.

Or maybe you’re making a prototype that will be demoed at a trade show in hopes of attracting investors, or you’re fixing a batch of bugs that have angered a key customer who is threatening to walk, or you have to deliver some fixes to comply with some new regulation that’s about to go into effect. Point is, your company has good reasons to ask when you’ll ship.

So you need to set expectations about the size and duration of our work. But fortunately, these guesses don’t have to lock you in and enslave you to your keyboard. That’s because every day you can check your progress, adjust for the things you’ve learned, and see if you need to set new expectations.

You’re driving to a destination

Getting your work done is a lot like taking a road trip. I love road trips! You never know what you’re going to encounter. Road construction, bad weather, mechanical failure, and even fun roadside attractions can all slow you down. You have to steer your way through all of these things.

The road to MadisonWhen you’re on the road, it’s pretty easy to know at any time about when you’ll get where you’re going. It involves answering these three questions:

  • How will you know when you get where you’re going? On the road, you just recognize your destination. But I’m going to fit this metaphor to your work, where it isn’t always as clear what it means to arrive.
  • How far do you have left to go? Watch the signs; they’ll tell you.
  • How fast are you going? Just check your speedometer.

Every time you see a sign, divide the number of miles to go by your average speed, and you have the best case for how long it will take to arrive. You can’t control what might go wrong on the road ahead, of course. But after something on the road does slow you down, you can just recalculate and get a new arrival time.

You can manage your own work in much the same way. But it means you have to build some skill at estimating.

“Uh, two weeks”

Here’s how not to estimate. My first post-college job was as a technical writer in a software company. My first assignment was to convert a thick product manual from an old publishing platform to a new one. It was hardly a mission-critical project, but I was green.

When the boss asked when I’d be done, I had no idea, so I shrugged and said, “Uh, two weeks.” Then the task turned out to be enormous and tedious. I had to put in a ton of extra time to finish in two weeks.

When my boss asked “how long” on my next assignment, I guessed a number – and then doubled it to account for the unknown. Success! I worked no extra time and finished my project before the deadline.

Except that I hadn’t learned anything of value. My “guess” estimating technique became “guess and pad like crazy.” And I still had the problem that if I didn’t pad enough, my only recourse was to work tons of extra hours. I had no way of knowing on any given day whether I was on track to finish when I said I would. I just had to work hard and hope.

It’s okay that you don’t know what you don’t know

If I could tell my 21-year-old self something, it would be this: Ask the boss if you can get back to him in two days. Then use your two days to do this:

  • Think about what done looks like. What must you do to finish the work? How will you know when you’ve gotten through it all? In this case, I had about the easiest possible situation: “Done” meant running out of pages to convert. You may have a number of requirements to code, or a batch of test charters to execute, or a set of stories to complete – there are any number of ways to chunk the work.
  • Do some of the work. Pay attention to how many chunks you complete in your two days. Then note how many chunks you have left. There’s your speedometer and your miles-to-go sign. Divide to get your rate: chunks per hour or chunks per day.
  • Calculate how long the rest of the work will take. Divide your rate into the number of remaining chunks. Lay that number out on a calendar to find your finish date. Tell that to the boss.

And then as you work, recalculate every day based on how much you got done that day. If you need to put in a few extra hours to stay on track, just do it. But when the amount of extra effort needed to stay on track starts to edge toward heroic, it’s time to negotiate.

That’s right, negotiate. You are the master of your own work, after all; nobody knows the reality of it better than you. Your estimate is what you believe it takes to reach Done, and you need to stand up for it. This is true, by the way, even when your boss has given you a due date.

So go to the boss and ask which is more important: the due date or this body of work? In other words, can you change what it means to be done, removing enough work to hit the date? Or can you move the date out to get all the work done? Or can you do a little of both?

Sometimes the answer to this question doesn’t go far enough to make the remaining work match the due date. So ask yourself if you can give some of the work to someone else, enough to hit the due date. If so, ask for people to help you.

What you’re doing is giving the boss and all of your project’s stakeholders as much runway as possible to understand the probability that you’ll deliver the originally defined Done on time. Should that probability drop, your negotiations give them the opportunity to change course, change expectations, or both.

Sometimes, you’ll get no answers to all of these questions. Sometimes, you’ll get yes answers to some or all of them and it still won’t be enough. But asking these questions is your best chance of avoiding working heroic hours.

An estimate should not be a commitment

If no is always the answer, or your bosses just expect you to rise heroically to every occasion, or your initial estimate is interpreted as a contract or an iron-clad promise, you just might be an in environment with dysfunctional beliefs about accountability. Consider finding another place to work.

Your estimate is only your best (and, hopefully, educated) view into when you will finish. It assumes that you don’t know what you don’t know. When you learn more, your estimate is obviously going to change. You’re better off working someplace that has at least an elementary understanding of this.

And now you understand it. Estimate based on what you do know. Go ahead and pad a little (a little!) for the unknown.

But then steer your work to successful completion.

I thank my brother, Rick Grey, who tests software for his supper, for contributing several key concepts to this post.

Obamacare, healthcare.gov, and how government software gets made

I was not surprised when I heard that the Obamacare Web site, healthcare.gov, crashed and burned right out of the gate.

But I was disappointed. Regardless of what I think of the Affordable Care Act, it’s the law. I wanted its implementation, including healthcare.gov, to go well.

obamacare2Still, I wasn’t surprised because I know how government software gets made. Several years ago I worked in middle management for a company that built a government Web application related to health-care customer service. I was in charge of testing it to make sure it worked. It is probably not going out on a limb to say that the people who built healthcare.gov experienced many of the same kinds of things I experienced on that project.

Let me be plain up front: I was a poor fit for government software development. I was too free-wheeling and entrepreneurial for the control-and-compliance environment that government contracting encourages. I find it difficult to write about the experience without showing my frustrations with its realities. But I think I understand those realities well and objectively.

The government doesn’t know how to do anything. They hire it all out, and then they manage and administer the process. As a result, on this project they relied heavily on compliance with “best practices,” as if those practices contained some sort of magic that delivered quality software. They don’t, of course; the government was shocked when Version 1.0 of our software had typical quality problems right out of the gate. Those practices served primarily to leave an audit trail the government could follow.

In the end, the project was a success. Despite Version 1.0′s glitches, which we quickly fixed, the software was immediately put to use and led to productivity improvements over an older, green-screen system. I spoke with many of the software’s users, and despite a few grumbles most of them liked using it.

But this was one mighty expensive piece of software to build, from winning the contract to defining what the software should do to building and maintaining the software. Here’s why.

The bid process

I was hired after we won the contract, but I heard stories about the bid process. We had no experience building software on this scale, but we wanted into the lucrative cost-plus government contracting business for its guaranteed profit margins. So we offered a lowball bid aimed at getting the government’s attention, not at what it actually was going to take to build the software. And then to our surprise we won the business. After the elation wore off, we were left with an “oh shit” feeling – we needed to actually build the software for that amount. How the heck would we pull that off?

We finished Version 1.0 on my watch, but I don’t know whether we delivered it within budget. It seemed to me, however, that the bid process encouraged underbidding and overspending.

The requirements process

When you make something for the government, they want to know exactly what they’re getting, in excruciating detail. So we started by writing the biggest, thickest requirements document I’ve ever seen. We weren’t building this software from scratch – we bought what was then the leading customer-relationship-management software platform and used it’s software-development toolkit to heavily customize it for our needs. But we had to write highly detailed specifications anyway.

obamacare1Requirements gathering was more about navigating choppy political waters and brokering compromise than about specifying usable, stable, and scalable software. To develop the requirements, we flew in representatives from every company that would use the software and put them into a big room to hash out how it would work. But all of these companies were themselves government contractors. Their people all knew each other – and, frequently, competed against each other for contracts. Some of them competed against us trying to win this contract. The room was thick with mistrust and agenda,

The building process

The government lives in constant fear of being screwed by its contractors. It goes back to Abraham Lincoln’s time, when rampant fraud among suppliers threatened the Civil War effort. (Seriously. Gunpowder cut with sawdust. Uniforms that dissolved in the rain. Read about it here.)

So not only did the government hire us to build the software, they hired another firm to watch us do it. This is called independent verification and validation, or IV&V. Their job was to make sure that we followed software-development “best practices” and that we built what we said we were going to build. But making matters worse, the company that won the IV&V contract, I’m told, also had bid on the project to build the software in the first place. It always seemed clear to me that they wanted to show us to be fools so that they could take over the project. They ran us ragged over every last minor detail.

The level of perfectionism in terms of “best practice” adherence was intense. Yet when we delivered the software, it had several usability challenges and outright bugs. Worse, it struggled to keep up with the load users placed on it. If you’ve ever built software, you know that these are typical challenges with Version 1.0 of anything. But the government was shocked, dismayed, and appalled. We spent the next several months issuing update releases to make it perform as it needed to. Of course, IV&V ran roughshod over us the whole way – but they were in the hot seat too because their “best practices” had failed to prevent these problems.

The process overhead

Process is tricky to apply well. Too little leads to chaos, too much adds needless cost and delay. I’m not anti-process – rather, I’ve built a career on bringing just the right level of process into a software development environment to make it effective. But most of the process we had to follow involved documenting our work to prove to the government that we had actually done it. This frequently hindered our ability to deliver software cost-effectively, and sometimes stood in the way of quality.

obamacare3We bought a well-known software product that stored requirements and linked them to the code and the test cases so we could prove that we built and tested each requirement. This involved tracing every requirement to every line of code and every test case, an enormous task in and of itself. I personally created a traceability report each quarter and sent it to the government. All of this required a lot of work from skilled technical people, but in my judgment did not materially help us better build or test the software.

Our test cases were contractually required to be documented in such detail that a trained monkey could execute them. They were at the level of “Step 1. Type your username into the Login box. Expected result: Your username appears in the Login box. Step 2. Type your password into the Password box. Expected result: A row of asterisks appears in the Password box.” A test case that took fifteen minutes to execute could have taken two hours to write and could have been a dozen pages long. We had hundreds of test cases. Many test cases were not appropriate to be added to the regression test suite and be executed every release, so we spent a lot of time writing them to execute them a small handful of times.

It was supposed to be against the rules to write a bug report that had no associated test case. Testers would often stumble upon a bug by accident or find one while doing ad-hoc testing – and then find themselves in a conundrum. Writing the test case that led to the bug and tracing it back to requirements took time we frequently lacked at that point in the game. When the bug was serious enough, everybody looked the other way when it wasn’t associated with a test case. I wonder whether any of the testers avoided writing test cases by falsely associating the bug with an existing test case.

We did get one big break. We lobbied for, and to our astonishment successfully won, an exception to a standard practice: we did not have to print screen shots of the results of every test step. Other projects for which we had contracts had to do this. As you can imagine, managing all that paper slowed progress considerably. Those projects collected those screen shots into boxes, which were sent to offsite storage.

The mounting costs

All of these process steps meant spending more money, mostly in the form of human effort. There were other ways in which the government’s way of making software added costs to the project. Here’s a short, incomplete list:

  • Frequent, ongoing training about compliance with standards, which, amusingly, is where I learned about the Civil War fraud.
  • Entering time worked on three separate software systems – one for the project-management tool, one for government accounting, and one my employer used to manage time off. I spent an hour a week entering time.
  • A prohibition on open-source software. The government wanted all software used to be “supported,” meaning that there had to be a phone number to call for help. So we spent money on commercial tools that sometimes weren’t as capable as open-source versions. In a couple cases, the only tool or component available for a task was open source, and we couldn’t build the application without it. We did get the government to bend the rule for us in those cases, but it took heavily documented justifications and layers of approvals to make it happen.
  • Strict separation of duties to protect the government against a rogue contract employee from sabotaging the system. This meant, for example, that I couldn’t restart the computers we used for testing when they needed it, I knew how to do it, but I was not allowed. I had to write a request for an infrastructure engineer to do it, and then wait sometimes for days for it to reach the top of his priority list.

As you can see, there was nothing easy or inexpensive about this project. Yet we got it done and the software worked. It’s still in use today. We showed that it’s possible – just slow and expensive – to build software the government’s way.

So I have great empathy for those who built healthcare.gov. No doubt about it: the site failed, and they built it. But they must feel tremendous pressure right now as they scramble to both handle the heat they’re getting from the government and to rush fixes to the site so that it works well enough. But if their experience building that site was anything like my experience building government software, it’s hardly shocking that it launched with challenges.

Giving testers less to do

I hear legends of companies who hire nothing but programmers in their test departments, and rely almost entirely on code-based tests in their software development methodologies. I’ve never seen such a shop in person. Out here in the Midwest, most testing involves humans directly exercising user interfaces.

Eric Jacobson recently wondered aloud on his blog whether testers are simply too busy finding bugs through the UI to move into more technical or programmatic testing. My typical experience has been that Development doesn’t deliver software to QA that is solid enough that testers didn’t have to spend the bulk of their time making sure the UI and the immediate interface with the database are working.

For years my schtick was to join a company when it is transitioning from small to mid-sized, when it was feeling crushed by quality problems caused by mounting technical and defect debt. They had focused on getting to market fast but had grown a tangled mess of code. My response was always to grow the QA team, primarily by hiring automation engineers to build out large automated regression suites.

After doing that at a couple companies, I lost interest in the strategy. It was expensive, it took too much time, and it never moved the quality needle enough. It just seems absurd to me now to prop up years of thin development practices with more post-development testing, especially given that automated tests in QA generally work through the UI and are therefore brittle and slow.


The view from my deck after a particularly heavy rain. You’d better believe the sump pump was running.

It’s like when my home’s crawl space used to flood after each heavy rain. A company that specialized in drying out crawl spaces recommended $6,000 in a French drain, multiple sump pumps, and encapsulation to move the water out and keep the moisture from seeping up into the house. But a buddy of mine who builds houses said, “You’ve got a negative grade around your foundation. Buy $300 in topsoil and a couple cases of beer. Invite all your friends over and issue them shovels. Fix the grading and you’ll keep the water from getting in.” I went with the topsoil and the friends. I also put in one sump pump, just in case. It runs pretty much only when the rain is torrential.

In case my admittedly imperfect metaphor isn’t obvious, the graded topsoil is the unit testing, and the sump pump is the lightweight QA automation solution. Let’s try preventing the bugs from getting in as much as we can, shall we? But let’s still check for the odd and extreme cases that are bound to get by.

This is the hierarchy of testing similar to the one Mike Kelly recommends in this blog post. (He’s building on the work of Brian Marick, by the way, who gives a framework for testing in this blog post.) Mike recommends building on the order of thousands of automated unit and component tests, hundreds of automated business-logic tests, and tens of UI-level automated tests. The unit tests run fast and frequently. The inherently slow UI automated tests run far less often.

That’s what I resolved to try after I lost my will to build huge automated regression suites in QA. I deliberately took a QA leadership role with a company transitioning out of its startup phase. The product wasn’t yet too large and too saddled with technical and defect debt; I felt like we could make up the lost ground. After some encouragement from me, the fellow who ran engineering began insisting that developers write automated unit tests for all new code. We started building each release into an environment where developers could perform rudimentary testing on it themselves with realistic data. Their goal was to make sure all the happy paths work, so that when my testers get in there they are not immediately stymied by obvious critical bugs.

Before we started to make this transition, I quietly started tracking a simple little metric. I counted the defects QA found, plus the defects created in the release that were found in production, and divided by the number of development hours in the release. The metric was a little mushy because I was working with estimated and not actual hours, and because I was having to make judgment calls about which defects in production were caused by the release and which were latent bugs. But it’s hard to ignore the order-of-magnitude improvement we got on this metric. We were tracking to about .3 defects per development hour in the couple releases before we made these changes, and within two releases we dropped to about .05-.09 defects per development hour and held steady.

This had incredible impact. Initial quality went way up in QA, meaning initial quality went way up in production. Just adding these two steps was like flipping a switch not only on many of the quality challenges we faced, but also on the amount of chaos and churn we experienced as an overall engineering team.

A side benefit was that developers seemed happier. It wasn’t that writing tests made them happy – it didn’t. They would rather have built more new stuff. But delivering better code into QA meant that they spent less time in the fix-test cycle and were interrupted by way fewer production crises. They told me that finally they could focus.

The reason why I titled this post as I did – and it’s meant to be tongue in cheek, by the way – is because my strategy means hiring more developers and fewer testers. But the benefit to testers is that they get to do far more interesting work, going deeper, thinking more creatively, and exploring more technical kinds of testing.

I can’t imagine ever moving to an all-code testing strategy. All automated testing can do is repeat series of actions. Skilled human testers can cope with complexity and adapt to change, gain and synthesize knowledge and apply it to their testing, know from experience where the product is likely to be broken, and explore the system creatively. The kinds of products I’ve always delivered and am likely to keep delivering will always need that.

“Software engineering” might be an oxymoron

The S Bridge at BlaineI’ve said it to my test teams many times: Making software isn’t quite engineering. Building a bridge – now that’s engineering. You determine how long the bridge needs to be, how much load it needs to carry, and what kind of bridge to build (steel truss, concrete arch, etc.), and from there it’s mostly mathematics and physics. Just run the calculations and you’re good.

We have bridge-building down. With a couple of notable exceptions, such as the Tacoma Narrows bridge which heaved and twisted and finally collapsed (video here), new bridges seldom fail. Old bridges fail sometimes, but it’s reliably due to accident or neglect.

My apologies to any civil engineers who stumble upon this post. I’m sure you’re cringing that I’m overlooking many subtleties of your discipline.

1859 General Dean Suspension BridgeThere’s nothing subtle, however, about how often software fails. Our users aren’t happy about it, but they aren’t surprised by it, either.

For any thing you ask a software developer to build, there will be a whole bunch of valid ways to do it, each with its own unique ways of creating failures. This is especially true when when that developer enhances existing software that he or she didn’t make in the first place. It’s tough to predict exactly how the enhancements will affect the rest of the software. The more lines of legacy code, the more time and analysis it takes to think that through.

If a developer had unlimited time and money, it might be possible to deliver perfect software. Ah, a developer can dream! But here’s where bridge-building and making software have an important thing in common: time and money are never unlimited.

I sympathize with the folks who call software a craft. People who make software use tools and knowledge in its design and construction. These are hallmarks of craft.

Canadian River BridgeAnother way that software is like craft is that it’s difficult to fully separate the design from the making. Even when one person designs the software and another writes the code, the coder has to make a bunch of lower-level design decisions along the way.

The software craftsmanship movement meets corporate resistance because revenue and profit ride on what we build. Our companies need to sell features to meet revenue projections, or deliver bug fixes to retain customers. That’s why timed delivery is so important: if you wait too long to deliver, the opportunity to grow or retain revenue begins to shrink.

Feeling pressure to deliver, yet knowing that if we deliver junk we’ll be in an even worse pickle, we tend to manage software-development projects like engineering projects. I think we feel like we have better control when we manage them that way. But that feeling of control can’t mask it: no matter how tightly you plan a software project, no matter how you shape your development and delivery processes to mitigate risk, no matter how much you try to predict the troubles you’ll encounter, you will discover things along the way can seriously derail those plans. It happens in two-week scrum sprints just as it does in ten-month waterfall projects. Discovery is simply endemic to software development.

Old US 36As a software project manager, I try to build in buffers for the unknown. I also steer projects daily based on what we discover, adjusting plans and communicating impacts to whomever needs to know. I try to make sure our development practices deliver the best possible code to test, and then I try to arrange testing to find the worst bugs first so that near the hoped-for end, only minor bugs remain. Despite all that, important bugs still sometimes reach the user.

We ship when the software is good enough. What “good enough” means varies from context to context, but it is unfailingly short of perfect. Shipping at good enough means you succeeded.

If I delivered bridges that way, I’d never drive over one I built.