Software Engineering in Large Organizations: Requirements

Figuring out what the heck it is you are going to do is the most important part of any project, isn’t it? It’s the case whether you are fixing a toilet (which I outsource) or building the next version of a software package. As a team lead I always wanted everyone on the team to be able to describe in couple of sentences what it is that we are trying to achieve, and their role in it.

Requirements are invented things. This everyone knows, but at small companies people tend to forget to write them down, and at big companies people tend to forget that they can be changed. Either way, it’s important to keep in mind that requirements must be clearly articulated otherwise they will surely not be met. In other words, we need to write down requirements so that we will be able to determine whether we have met them. Once they’re written down, requirements documents can start to seem like stone tablets. Requirements do not exist for their own sake, but to realize a larger vision. If we can’t explain why a requirement is a requirement, then something has gone wrong. It’s common to question requirements later on in a project – perhaps they are too hard to implement. That’s okay to some extent, but in many cases people are just a bit too sloppy about thinking about (and writing down) requirements at the start of a project.

Where do requirements come from? We already said they are invented, but how do we invent them? Big companies draw from many sources:

  • User surveys
  • Instrumentation
  • Senior management
  • Customer advisory boards
  • Partner feedback

The “Building Windows 8” blog is a wonderful public example of such artifacts. The B8 blog gives a true sense of how requirements are developed for a “big league” project. Basically the idea is to be like a five year-old: keep asking why questions until you get to axioms.

Also notable is the Russian doll-like nesting of requirements inside of requirements. Even a much smaller project like Solver Foundation cannot simply have one requirement document. Requirements are typically written down in increasingly narrow form: from a vision document to scenarios to themes to features to specifications. A specification, or “spec”, is the fundamental unit of requirements at a larger software shop. It describes a unit of work specified by a single PM, implemented by a single developer, and tested by a single tester during a single product release (or milestone).

Specs should:

  • provide justification for the feature
  • state goals
  • state non-goals
  • define user scenarios
  • imply a self-contained unit of work
  • specify integration points
  • describe performance goals
  • be written for the engineering team
  • be self-contained

They should not:

  • describe how a feature is built
  • be a list of APIs
  • be written in “business speak”
  • be written for management

Sad to say, it is often the case that a spec ends up beginning with an amateurish mishmash of MBA gobbledygook and ending with a hastily cut-and-pasted set of API signatures, with comments from ten different people in the margins. Few things are less useful and more depressing than a spec of this kind. Be clear, don’t try to impress, and justify your reasoning. Write for someone who is smart but is not intimately familiar with your product and team history. After all, those are the kinds of people who will be using the thing you’re trying to build.



Software Engineering in Large Organizations: Software Lifecycle

(This is part two in a series.)

The software lifecycle is the same wherever you go. It’s one of these things that you are taught in school that really is the way it’s described. The steps are basically to figure out what you need to do (requirements), how it will work (design), do it (implementation), and verify you’ve done it right (testing). Then you’ve got to get the finished product to the people you promised to get it to (deployment). When you lay it out sequentially it feels very much like a waterfall development model, but of course all the same steps happen in agile as well. There’s been a big move towards agile over the past ten years and big companies are no exception. You do encounter a lot of “faux agile” as well (fauxgile?) – a team I was once a part of had one hour “scrum meetings” with 15 or 20 managers with laptops sitting down in a room. I digress.

At a big operation, each stage is documented according to organizational or team standards. This is a good idea, and it’s vital if you want to be able to share institutional knowledge among past and future team members, seed the localization and user documentation teams with good information, and form the genesis of patent applications.

Accountability for different stages in the software life cycle is divided among the team. Teams are large enough to permit specialization. At Microsoft (and other places besides), the three primary job descriptions are “program manager”, “developer”, and “tester” (or QA). Program managers are accountable for requirements, developers for implementation, and QA for testing. All three disciplines are involved in all stages, but each discipline takes its turn in the spotlight as concepts move from vague notions to concrete implementation. Different outfits divide these responsibilities in different ways. The concept of a “program manager” (as opposed to project manager) was essentially invented at Microsoft and is not universal. Some teams combine dev and QA responsibilities. Other teams include operations (accountable for deployment) in the core engineering team.

This separation of powers feels like the division that exists between legislative (PM), executive (dev), and judicial (QA) in government. As in government, tension sometimes exists between the three branches. Some amount of this is natural and healthy because after all, software engineering is an activity that is undertaken with limited resources under changing conditions. Tradeoffs are necessary, and figuring out how and when to make these changes naturally leads to difference of opinion. One difference between engineering teams and governments is that in an engineering team there is a fourth party sitting above all the others: management. Management, if it is to be useful, should step in when necessary to remind all three disciplines of their common mission and purpose, and to make the judgment calls that are necessary to keep them on track. They’re in a good position to do that when the mission is clearly defined, they can articulate it, and when they can relate it to the day-to-day work that their team is being asked to do. (Knuth: “the psychological profiling of a programmer is mostly the ability to shift levels of abstraction, from low level to high level. To see something in the small and to see something in the large.”)

I don’t know about you, but I’d rather be a president than a legislator or a judge. I always liked being a dev. It’s common for the devs to feel like they are special – I remember a conversation early on in my Microsoft career where a more senior dev told me that devs were special because they were the only ones that could perform the other two job functions. I have found that it is not really true – a great PM could be a dev or a tester, and a great tester could be a PM or a dev.  This makes sense because in order to do your job well, you need to understand how the work you do fits into the larger story. It is common for Microsoft employees to change from one discipline to another in the course of their careers.

A “triad” of PM, dev, and tester form a basic unit that can take a portion of a product (a feature) from start to finish. It’s become more common in certain divisions at Microsoft to make this partnership more formal by calling this triad a “feature crew”. They meet regularly from the inception of the project to the very end, reviewing each other’s work and tracking its progress together. Opinions vary on whether formal feature crews are a good idea or simply bureaucracy, but I liked them. Camaraderie develops between the triad, which is enjoyable and effective.

Next time (whenever that is) I will talk a bit about the first stage of the process: requirements.


Software Engineering in Large Organizations

This afternoon I gave a talk at the University of Iowa ACM conference, where I spoke about software engineering in large organizations. It’s a topic I enjoy speaking and writing about, and I was particularly enthusiastic because the audience was mostly undergrads and grads in the CS department. I tried to resist the temptation to simply tell “war stories” for 75 minutes and I almost succeeded.

The premise behind the talk is that in a healthy organization, team success and professional development go hand-in-hand, but in practice the reality often differs from the ideal. A key to success and professional fulfillment is to build hard and soft skills that allow you to achieve team goals as well as individual growth in the face of these realities.

Team success and individual professional development are clearly beneficial to everyone involved, and not at all impossible to achieve simultaneously. An organization that makes long term investments in talented people working on clearly defined goals, extending the opportunity to accept and conquer big challenges is likely to succeed on both counts. (Not incidentally, it’s impossible to pull this off without creating a positive, encouraging, lively work environment.) Unhappily, it’s often the case that organizations collectively apply tactical thinking to ill-defined or changing goals, leading to poorly managed projects with periods of tedium followed by “death marches”, caffeine and cold pizza.

Employees of large organizations are not unique in facing these challenges, but the impact may be more acute because simple math says that they are likely to have less control over their professional environment. The serenity prayer comes to mind. Nevertheless, large organizations provide tremendous advantages. Big companies draw outstanding talent and are able to provide them all the tools they need to do their job. There’s a lot going on – the diversity of interesting, relevant projects at Microsoft continued to inspire and amaze me year after year. The same is true at Nielsen, and other companies.  Big companies tend to have formal processes in place for employee evaluation and development. They can be great places to learn new skills, be they technical or interpersonal.

In order to understand why software development at a big company is different from a startup, you need to think about what the job requires. Software development is a creative activity requiring engineering discipline. There are huge differences in the skill level of coders, even out in the professional world. Ubercoders do exist. But that is not the key factor determining success. I like to think of software engineers in terms of the following axes:

No matter where you work, you want to be a pro – in the upper right hand corner. Engineering discipline and creativity can absolutely coexist – and the traces of both are plainly evident in technology that truly inspires, be it the iPad, the Kinect, or whatever. Engineering discipline is simply more important (in a relative sense) than in smaller organizations, and for sound reasons. Large companies have different considerations. Big companies have big teams working together towards common goals. They all must march and work together. The cost of failure is often higher. Typically the team needs to support a large past body of work, such as a previous version. Mistakes can have consequences that last years. The list goes on.

A potentially uncomfortable but eminently fair example is with Windows Vista and Windows 8. I need to be careful here: I have never been a part of the Windows team and I don’t know anything that you don’t, and even if I did I would not reveal anything about a company that I no longer work for but still root for. Furthermore, Windows 8 has not shipped, and opinions may differ as to whether it will be enough of a success to stave off Apple, etc. But – I am sure that when it ships, Windows 8 will be a precise embodiment of a vision that was laid out years ago; that the choices that were made will be able to be justified with data; that it will ship on time and with high quality; that it will be something that the team who built it will be exceedingly proud of.  It is a matter of public record that this was not the case with Vista.  Why the difference? I will tell you what is not the case. It is not the case that Microsoft fired all the Vista people and hired new programmers (although there is new leadership). It is not the case that the existing team became significantly better programmers, testers, and designers. It is not the case that there were was a lack creative, interesting ideas during Vista development. (Au contraire, mon frere.) What happened was that the team realized a collective understanding of the importance of engineering discipline in all phases of the software cycle. Change is hard and not without cost, as is described in the recent Business Insider article on Microsoft (check it out). But I am sure that the costs will be repaid many times over by the work that has resulted from these changes. I am sure that many of those who stuck with the changes are better software engineers too.

This was the message behind the first portion of the talk. In the remainder of the talk I walked through the software lifecycle, giving my best account of how things work in a big organization and doing my best to explain why, with an aim towards identifying skills and techniques that improve one’s game. As time permits, in future posts I will share some of my thoughts from the remainder of the talk.


Shipbuilders, Sailors, and Passengers

I sometimes trot out a boat analogy when I am asked about a career in data science, which is tenuous since I grew up in Iowa.

Cutty Sark

Building solvers is like shipbuilding. Shipbuilding is an ancient discipline which in more recent times has grown into a sophisticated engineering task. Building ships takes time to do right, and it takes a lot of practice to learn – it helps if you’re an apprentice first. It’s not an art, but it’s not quite a science either. For all of the technology, for all of the engineering, as a shipbuilder there are certain things you just don’t do, because that’s just the way the that you were taught. People long before you have tried it a different way and it just didn’t work. You don’t need to be young to be a shipbuilder – in fact in some ways it might be a little bit better if you are a bit older and wiser. Some people think the very idea of building a boat is a terrible bore (or just too damn hard), but for others it’s captivating. That’s pretty much all they want to do.

Using a solver – modeling – is like sailing. Sailors care very deeply about boats, of course, but that’s not all they care about. If you have ever taken a sailing lesson, one of the first things you will hear about is the weather. Your instructor will tell you to pay attention to the way the wind plays off the water, the trees near the shore, flags on buildings, and so on. A sailor needs to understand how their boat interacts with the wind and the water. When the forces of nature and the sailboat are in harmony, the experience is almost magical. If you’re sailing anywhere interesting then you will also need to know about any hazards along the way, like rocks, mermaids or whatever. Even experienced sailors can’t figure this stuff without some help – navigational aids, or the advice of locals is often helpful. Some people sail for fun, and others sail because it’s their job, but either way there’s a goal in mind. What matters is not the general characteristics of boats, or wind, or water, but the specific conditions that come into play on their voyages. Shipbuilders and sailors evaluate boats differently. Shipbuilders have no idea how their boat is going to be used, so they have to think about the entire range of conditions they may face. They may even design for the worst possible storm. Sailors only care about the voyages that they themselves are on. But since people sail for a reason, the destination and the conditions are often out of the ordinary, and may stress the boat in ways that shipbuilders, or even Hans Mittelmann, may not have anticipated.

Just because you can sail doesn’t mean that you can build a boat. Sailors want boats, not shipbuilders, but you can’t have a boat without a shipbuilder. Good shipbuilders are hard to find. That all seems obvious. The funny thing is that it’s hard for many of us to think of it the other way round. Just because you can build a boat doesn’t mean that you know how to sail. Some can, it’s true…but you can’t bank on it. I certainly wouldn’t try to turn a bunch of shipbuilders into sailors without some sort of training or evaluation. Maybe because we’re so in awe of the few that know how to build ships that we think that they simply must know how to sail. I think things are starting to change, but it seems to me that most operations research graduate students are trained to be shipbuilders. This is not a bad thing. The thing is that once they’re trained to be shipbuilders, they are often hired to be sailors. For my part, I was trained to be a shipbuilder as a CS grad at the University of Iowa. When I joined Solver Foundation at Microsoft several years ago, I was (thankfully) hired on to be a shipbuilder, writing our interior point solvers. The more I got familiar with Solver Foundation and its customers, and especially after I took over leadership of the team, I began to realize that many of our customers were asking us to teach them to sail, or just to sail the damn boat for them. My head was filled with the alphas, mus, and sigmas of the shipyard but they are not always all that useful out on the water. I needed to learn to sail. Now I find that I like sailing more than I do shipbuilding. Go figure. Management is sometimes not as familiar with nautical terminology so these distinctions are sometimes lost. Management often seems more interested in cars than boats.

I’ve neglected to mention the most important group. People who ride on boats – passengers (or voyagers if you are more romantic) – are like people who use models. Some people take trips for fun, like a trip around Puget Sound, or a cruise to Alaska. Other people take trips to get from one place to another, like from Seattle to Victoria. Some voyages don’t involve passengers at all – we’re moving freight.  The important difference is that in each of the cases, the fact that sailors and shipbuilders are involved at all is incidental – a passenger is paying for an experience, or for a service. While you and I are boat enthusiasts – can’t get enough! – most passengers couldn’t give a crap about the type of fabric used to build the sail, or the horsepower on the engines, or how narrow the strait is. The experience is what matters. Sometimes there is only a dim awareness that a boat is involved at all – all they know is that when they go down the ramp, they are in a new place. Solvers help determine how an Amazon package gets to your door. That’s amazing.

There are only so many shipbuilders in the world. There are many more sailors, but even sailors are overwhelmingly outnumbered by passengers. It’s not even close. But I don’t think that means that any one of these groups is any more important, or any more noble or intelligent. Everyone is coming at this from their own perspective, and you have to respect that. Shipbuilders have mastered a craft, and that requires a lot of dedication. Modelers are able to adjust to conditions and get people where they need to go. Passengers have their own lives, and often a particular trip is only a line or two in the larger narrative that is their lives. The sailor shouldn’t mock the guy playing shuffleboard – he’s probably earned the right to kick back a little. The passenger shouldn’t look down on the sailor, either. They will keep you from drowning.

I wonder – which would you rather do? Build boats, sail, or go on a voyage? Where is the most money? Does it matter? Will we ever be able to eliminate the need for boats, voyages, or trips? Beats me.

How to fix bugs (and influence people)

I work on a large team with a lot of dependencies and fairly tight deadlines.  I wanted to share some of a few techniques for fixing bugs quickly.  Maybe this is all common knowledge but I’ll share it anyway.

  1. Trade bugs. It’s not very efficient if you and another dev to be modifying the same code at the same time.  Swap bugs with your teammates to define clear ownership.  Sooner is better than later.
  2. Fix areas, not bugs. Don’t just fix one string on a page, fix all the strings on the page.  If a validation check was missing in one API, it is probably missing in other APIs too.  Proactively look for related bugs assigned to other people and take them on.
  3. Kick off a build every night before you leave. I have known people who scheduled jobs to do this automatically.
  4. Pipeline your work. There is a lot of idle time spent waiting for builds to finish, deploying, etc.  Fill that time with something useful by switching over to another bug.  My pattern is usually to repro the bug (stepping through the code), code the solution, then do final testing.  So I like to work on three bugs (actually three sets of bugs, see above) at once:
    debug  code  test
          debug  code  test
                 debug code  test

    Sometimes there is a fourth step – “think” – for more difficult bugs.

  5. Write tools. Sometimes it is a big pain in the ass to work through the product to repro and fix bugs.  Sometimes building simple tools can really help your productivity, for example a simple winforms app that mimicks a more complicated client application.  I have written command line apps that talked to webservices in the past.  Once you build the tool, let your teammates know.
  6. Unit test and refactor. Try to think about how you can build a strong foundation of unit tests that are easy to add to.  Unit tests prevent regressions.  Also, once you have confidence in your unit tests you can fearlessly refactor code if you notice lots of bugs in a certain area.
  7. Schedule uninterrupted time for yourself. It’s hard to work effectively in 15 minute chunks.
  8. Don’t break the f***ing build. That slows down the whole team.  If you notice a blocking issue, jump on it immediately even if it is not your area.  You will learn something in the process.