Software Engineering in Large Organizations: Software Lifecycle

(This is part two in a series.)

The software lifecycle is the same wherever you go. It’s one of these things that you are taught in school that really is the way it’s described. The steps are basically to figure out what you need to do (requirements), how it will work (design), do it (implementation), and verify you’ve done it right (testing). Then you’ve got to get the finished product to the people you promised to get it to (deployment). When you lay it out sequentially it feels very much like a waterfall development model, but of course all the same steps happen in agile as well. There’s been a big move towards agile over the past ten years and big companies are no exception. You do encounter a lot of “faux agile” as well (fauxgile?) – a team I was once a part of had one hour “scrum meetings” with 15 or 20 managers with laptops sitting down in a room. I digress.

At a big operation, each stage is documented according to organizational or team standards. This is a good idea, and it’s vital if you want to be able to share institutional knowledge among past and future team members, seed the localization and user documentation teams with good information, and form the genesis of patent applications.

Accountability for different stages in the software life cycle is divided among the team. Teams are large enough to permit specialization. At Microsoft (and other places besides), the three primary job descriptions are “program manager”, “developer”, and “tester” (or QA). Program managers are accountable for requirements, developers for implementation, and QA for testing. All three disciplines are involved in all stages, but each discipline takes its turn in the spotlight as concepts move from vague notions to concrete implementation. Different outfits divide these responsibilities in different ways. The concept of a “program manager” (as opposed to project manager) was essentially invented at Microsoft and is not universal. Some teams combine dev and QA responsibilities. Other teams include operations (accountable for deployment) in the core engineering team.

This separation of powers feels like the division that exists between legislative (PM), executive (dev), and judicial (QA) in government. As in government, tension sometimes exists between the three branches. Some amount of this is natural and healthy because after all, software engineering is an activity that is undertaken with limited resources under changing conditions. Tradeoffs are necessary, and figuring out how and when to make these changes naturally leads to difference of opinion. One difference between engineering teams and governments is that in an engineering team there is a fourth party sitting above all the others: management. Management, if it is to be useful, should step in when necessary to remind all three disciplines of their common mission and purpose, and to make the judgment calls that are necessary to keep them on track. They’re in a good position to do that when the mission is clearly defined, they can articulate it, and when they can relate it to the day-to-day work that their team is being asked to do. (Knuth: “the psychological profiling of a programmer is mostly the ability to shift levels of abstraction, from low level to high level. To see something in the small and to see something in the large.”)

I don’t know about you, but I’d rather be a president than a legislator or a judge. I always liked being a dev. It’s common for the devs to feel like they are special – I remember a conversation early on in my Microsoft career where a more senior dev told me that devs were special because they were the only ones that could perform the other two job functions. I have found that it is not really true – a great PM could be a dev or a tester, and a great tester could be a PM or a dev.  This makes sense because in order to do your job well, you need to understand how the work you do fits into the larger story. It is common for Microsoft employees to change from one discipline to another in the course of their careers.

A “triad” of PM, dev, and tester form a basic unit that can take a portion of a product (a feature) from start to finish. It’s become more common in certain divisions at Microsoft to make this partnership more formal by calling this triad a “feature crew”. They meet regularly from the inception of the project to the very end, reviewing each other’s work and tracking its progress together. Opinions vary on whether formal feature crews are a good idea or simply bureaucracy, but I liked them. Camaraderie develops between the triad, which is enjoyable and effective.

Next time (whenever that is) I will talk a bit about the first stage of the process: requirements.

 

Software Engineering in Large Organizations

This afternoon I gave a talk at the University of Iowa ACM conference, where I spoke about software engineering in large organizations. It’s a topic I enjoy speaking and writing about, and I was particularly enthusiastic because the audience was mostly undergrads and grads in the CS department. I tried to resist the temptation to simply tell “war stories” for 75 minutes and I almost succeeded.

The premise behind the talk is that in a healthy organization, team success and professional development go hand-in-hand, but in practice the reality often differs from the ideal. A key to success and professional fulfillment is to build hard and soft skills that allow you to achieve team goals as well as individual growth in the face of these realities.

Team success and individual professional development are clearly beneficial to everyone involved, and not at all impossible to achieve simultaneously. An organization that makes long term investments in talented people working on clearly defined goals, extending the opportunity to accept and conquer big challenges is likely to succeed on both counts. (Not incidentally, it’s impossible to pull this off without creating a positive, encouraging, lively work environment.) Unhappily, it’s often the case that organizations collectively apply tactical thinking to ill-defined or changing goals, leading to poorly managed projects with periods of tedium followed by “death marches”, caffeine and cold pizza.

Employees of large organizations are not unique in facing these challenges, but the impact may be more acute because simple math says that they are likely to have less control over their professional environment. The serenity prayer comes to mind. Nevertheless, large organizations provide tremendous advantages. Big companies draw outstanding talent and are able to provide them all the tools they need to do their job. There’s a lot going on – the diversity of interesting, relevant projects at Microsoft continued to inspire and amaze me year after year. The same is true at Nielsen, and other companies.  Big companies tend to have formal processes in place for employee evaluation and development. They can be great places to learn new skills, be they technical or interpersonal.

In order to understand why software development at a big company is different from a startup, you need to think about what the job requires. Software development is a creative activity requiring engineering discipline. There are huge differences in the skill level of coders, even out in the professional world. Ubercoders do exist. But that is not the key factor determining success. I like to think of software engineers in terms of the following axes:

No matter where you work, you want to be a pro – in the upper right hand corner. Engineering discipline and creativity can absolutely coexist – and the traces of both are plainly evident in technology that truly inspires, be it the iPad, the Kinect, or whatever. Engineering discipline is simply more important (in a relative sense) than in smaller organizations, and for sound reasons. Large companies have different considerations. Big companies have big teams working together towards common goals. They all must march and work together. The cost of failure is often higher. Typically the team needs to support a large past body of work, such as a previous version. Mistakes can have consequences that last years. The list goes on.

A potentially uncomfortable but eminently fair example is with Windows Vista and Windows 8. I need to be careful here: I have never been a part of the Windows team and I don’t know anything that you don’t, and even if I did I would not reveal anything about a company that I no longer work for but still root for. Furthermore, Windows 8 has not shipped, and opinions may differ as to whether it will be enough of a success to stave off Apple, etc. But – I am sure that when it ships, Windows 8 will be a precise embodiment of a vision that was laid out years ago; that the choices that were made will be able to be justified with data; that it will ship on time and with high quality; that it will be something that the team who built it will be exceedingly proud of.  It is a matter of public record that this was not the case with Vista.  Why the difference? I will tell you what is not the case. It is not the case that Microsoft fired all the Vista people and hired new programmers (although there is new leadership). It is not the case that the existing team became significantly better programmers, testers, and designers. It is not the case that there were was a lack creative, interesting ideas during Vista development. (Au contraire, mon frere.) What happened was that the team realized a collective understanding of the importance of engineering discipline in all phases of the software cycle. Change is hard and not without cost, as is described in the recent Business Insider article on Microsoft (check it out). But I am sure that the costs will be repaid many times over by the work that has resulted from these changes. I am sure that many of those who stuck with the changes are better software engineers too.

This was the message behind the first portion of the talk. In the remainder of the talk I walked through the software lifecycle, giving my best account of how things work in a big organization and doing my best to explain why, with an aim towards identifying skills and techniques that improve one’s game. As time permits, in future posts I will share some of my thoughts from the remainder of the talk.