Categories
Quality

Programming is the number one quality assurance activity

By Jim Grey (about)

Programming is the number one quality assurance activity.

Or maybe it’s design. Or maybe it’s writing good user stories. Or maybe it’s having good ideas for things to build in the first place. Or maybe it’s paying down technical debt.

Test

But it sure as hell isn’t testing.

When I talk to engineering leaders at small software development shops and they find out I make my living in testing, many of them admit that they don’t have any testers yet. They’re sheepish about it. It’s as if I’ll be offended!

I tell them to rock on, and to delay hiring that first tester for as long as they can.

And then I ask them about the quality challenges they have. Too many bugs? Won’t scale? Bogs down under load? Don’t fall prey to the gut reaction “oh my god we need to test,” as if testers are a magic filter through which perfect software passes.

Because if you respond by hiring testers, you’re likely to end up with testing theater. Your testers will do testery things and find bugs, sometimes even good ones, bugs that let you sleep better at night.

But they can’t fix your quality problems. Only your developers can do that. Instead of letting your developers do whatever it is they do and hope a tester can find everything that’s wrong, challenge your developers to get better.

I ask them these questions:

Do your developers have the skills needed to build the software you’re asking of them? If not, help them build those skills or, gulp, replace them with developers who have them.

Are you following good development practices? Test-driven development, pairing, and code reviews. Not only do they promote solid code, they help create a culture of quality among your developers.

Is your team writing lots of automated unit tests and and acceptance tests? (By acceptance tests, I mean thin functional tests at the API or controller level.) Do they run on every commit? This traps basic problems as soon as they’re introduced.

Do you have a well-functioning delivery methodology, at the right scale for your organization? If you’re two developers, that might be a kanban board drawn on a whiteboard. If you’re ten developers, you might use a tool like Pivotal Tracker and have a couple defined but light rules and ceremonies. If you’re 100 developers, it might be some scaled agile methodology like SAFe, backed up with a tool like JIRA, guided by a program management office. Whatever it is, are you following it and is work flowing smoothly through the delivery pipeline?

Are you giving your developers the time they need to do thoughtful work? To design resilient software that performs? To architect it for long-term growth?

Do all of these things before you hire your first tester. Your developers will give you a level of quality that makes it hard for testers to find low-level bugs. Testers will then have time to find more interesting and valuable bugs because basic stuff is seldom broken. This makes testing a lot less expensive, by the way. You need way fewer testers when you deliver software to them where core functionality works.

And then your testers, instead of feverishly testing the basics, can contribute at a higher level. Testers bring a different mindset to your teams, one of critical thinking about how the software is deployed and used. When you harness that mindset, you can improve your designs and your architecture before you write that first line of code.

Categories
Process Quality The Business of Software

“Software engineering” might be an oxymoron

By Jim Grey (about)

I’ve said it to my test teams many times: Making software isn’t quite engineering. Building a bridge – now that’s engineering. You determine how long the bridge needs to be, how much load it needs to carry, and what kind of bridge to build (steel truss, concrete arch, etc.), and from there it’s mostly mathematics and physics. Just run the calculations and you’re good.

We have bridge-building down. With a couple of notable exceptions, such as the Tacoma Narrows bridge which heaved and twisted and finally collapsed (video here), new bridges seldom fail. Old bridges fail sometimes, but it’s reliably due to accident or neglect.

The S Bridge at Blaine

My apologies to any civil engineers who stumble upon this post. I’m sure you’re cringing that I’m overlooking many subtleties of your discipline.

There’s nothing subtle, however, about how often software fails. Our users aren’t happy about it, but they aren’t surprised by it, either.

For any thing you ask a software developer to build, there will be a whole bunch of valid ways to do it, each with its own unique ways of creating failures. This is especially true when when that developer enhances existing software that he or she didn’t make in the first place. It’s tough to predict exactly how the enhancements will affect the rest of the software. The more lines of legacy code, the more time and analysis it takes to think that through.

If a developer had unlimited time and money, it might be possible to deliver perfect software. Ah, a developer can dream! But here’s where bridge-building and making software have an important thing in common: time and money are never unlimited.

I sympathize with the folks who call software a craft. People who make software use tools and knowledge in its design and construction. These are hallmarks of craft.

Another way that software is like craft is that it’s difficult to fully separate the design from the making. Even when one person designs the software and another writes the code, the coder has to make a bunch of lower-level design decisions along the way.

The software craftsmanship movement meets corporate resistance because revenue and profit ride on what we build. Our companies need to sell features to meet revenue projections, or deliver bug fixes to retain customers. That’s why timed delivery is so important: if you wait too long to deliver, the opportunity to grow or retain revenue begins to shrink.

Feeling pressure to deliver, yet knowing that if we deliver junk we’ll be in an even worse pickle, we tend to manage software-development projects like engineering projects. I think we feel like we have better control when we manage them that way. But that feeling of control can’t mask it: no matter how tightly you plan a software project, no matter how you shape your development and delivery processes to mitigate risk, no matter how much you try to predict the troubles you’ll encounter, you will discover things along the way can seriously derail those plans. It happens in two-week scrum sprints just as it does in ten-month waterfall projects. Discovery is simply endemic to software development.

As a software project manager, I try to build in buffers for the unknown. I also steer projects daily based on what we discover, adjusting plans and communicating impacts to whomever needs to know. I try to make sure our development practices deliver the best possible code to test, and then I try to arrange testing to find the worst bugs first so that near the hoped-for end, only minor bugs remain. Despite all that, important bugs still sometimes reach the user.

We ship when the software is good enough. What “good enough” means varies from context to context, but it is unfailingly short of perfect. Shipping at good enough means you succeeded.

If I delivered bridges that way, I’d never drive over one I built.

Categories
Process

Even a mediocre plan will work if everybody follows it

By Jim Grey (about)

I have been astonished in my life by how few problems are truly unsolvable. I have also noticed that, most of the time, when a problem ends up not being solved it is for one of two reasons: people deny the problem, or they won’t work together on the solution.

There are any number of ways to make software. All of them involve programming, of course; it’s the core activity. But there are other jobs to perform, and the bigger the software being developed, the more people you need to perform them. It becomes necessary to organize the people and form work processes. The ways to do that range from loose and chaotic to structured but flexible to ponderous and formal. They all have their strengths and weaknesses; they all can work. I’ve made software in all of these situations.

Perfect gravel road
Even a gravel road will get you there.

One of my past employers sequestered its programmers in a room to hack code together as fast as they could. When they were done, a couple people would play with the product for a week or two in hopes of finding some bugs, which the programmers might or might not fix. Then we shipped it to five customers, who upon installing it invariably found that it failed spectacularly. When we fixed all the problems they found, we chose five more customers. We repeated this process until the software stopped failing, and then announced to the rest of our customers that it was ready.

At the other end of the scale, another past employer had such a heavy software development process that following it felt like slipping into a straitjacket. There were reams of documentation to write and keep with layers of approval at each step. I led the test team. We had to take a screen shot of every test step, print it, and put it in a folder. At the end of a project, we had boxes and boxes and boxes of printouts, hundreds of pounds worth, that we would send to an offsite storage facility. This was in case our only customer, which happened to be the U.S. government, wanted to audit us.

Both companies are still in business.

A third past employer carefully hired smart people and trusted them to know what to do. Because they hired well, this worked for a long time. But as the company grew, this approach became more and more difficult to manage. Product quality became a problem. A big part of the software was an accounting package, and in one fateful release the general ledger simply would not balance. This affected every customer, and to make a long story short, it took the company almost a year to fix it. Several customers quit us.

Too often it takes great pain to drive important change. This pain made us face that we needed more structure to deliver successfully. So we hired a consultant to guide us toward a better software development process. He showed us a way with just enough checks and balances that we could have greater confidence in our work without being too bogged down. We also hired a consultant to help the management team (of which I was a part) work more collaboratively. She taught us how to lead the company through this transition.

It worked. In the very first product release under the new process, quality problems fell off dramatically and we delivered on time for the first time anybody could remember.

The more important hire was the second consultant. The new methodology was good, but it wasn’t magic, and it might not have worked for us if we had not done a good job helping the team through the changes. You see, a few people didn’t understand why the old way couldn’t still work and still others thought our new process wasn’t going far enough. Our task was to overcome the resistance and get everybody singing from the same sheet of music, and it was the hardest thing we did. But we pulled it off.

It’s a rare organization that faces its failings and works to change. It’s a rarer organization still that drives change collaboratively.