malvasia bianca

At 4:00pm on the friday of GDC, I attended The Iterative Level Design Process of Bioware’s Mass Effect 2. I went because I loved Mass Effect and because I’m always happy to see the word “iterative” used, but the talk turned out to be an excellent final experience from the conference for a completely unexpected reason: it was all about how they were in the middle of an apparently quite successful lean transition!

Mass Effect was a fabulous game, but it had problems. Elevators are one example; also, the world where you found Liara was once intended to be the same length as the other key worlds, but they ran into late issues that they couldn’t solve in time. There was costly, unplanned rework; insufficient cross-department communication; and performance issues that they couldn’t iron out in the final product. One factor contributing to several of the problems was that levels were rarely playable: this made it difficult to evaluate the game and its mechanics, it made QA testing quite difficult, it lead to late revision instead of early revision. So they wanted to do better in Mass Effect 2.

To fix these problems, they broke down the work on each level in the sequel into six phases; each phase was designed to answer a specific question, you were supposed to avoid any work that didn’t bear on that question, and the end result of each phase was a deliverable that bears specifically on that question. In addition, they had three ground rules: they game should always be playable, the current stage should always be a foundation for future work, and the game should always have acceptable performance.

The stages:

• Stage 0: Narrative Overview. This stage is answering the question “What is the story?”, and the deliverable is a document describing the narrative, the characters, the 2D layout, the art themes, and the cutscenes. This stage is different from the others (hence the numbering) in that it largely involves the writers rather than the full team and in that there’s nothing playable at the end of it.
• Stage 1: Narrative Playable. The question: “Is the pacing and spacing good?”; the deliverable is the first playable. At this stage, the level only contains its box-level geometry, placeholder set pieces, and “box level” dialogs (i.e. saying “X will tell Y to do A, B, C” rather than the actual words that X will use); there are pop-up cutscenes, and prototype level mechanics. But it’s still enough to get a feel for what’s going on, and an awful lot gets changed at this stage. In fact, this can even be enough to get a global overview rather than a local overview: over Christmas, team members took the game home, even though many of the levels were at stage 1, and discovered that, while the endgame felt fine in isolation, it didn’t work well in the context of the whole story. Which led to significant rethinking, but it’s much less costly to do that rethinking at this stage than later on, or to not do that rethinking at all.
• Stage 2: White Box. The question: “Can you see the fun?”; the deliverable is a level with representatitive collision. You have first-past modeling on the geometry, and a first pass at the dialog. There’s basic placement of cover for the combat scenes, and animatronic cutscenes. And there’s placeholder music: without that, levels felt blah, leading to criticisms that, in retrospect, weren’t well-founded.
• Stage 3: Orange Box. The question: “Is it fun?”; the deliverable is a level with actual collision. The textures aren’t all there, yet, but it’s working; the dialogs are ready for voice acting and are cinematically blocked out; there’s full cover placement for combat; and they’ve done the basic motion capture for the cut scenes.
• Stage 4: Hardening. The question: “Could this be shipped?”: they’d rather not ship the level like that, but if the rest of the game was polished and a deadline was coming up, they could ship the game with a level in this stage. The deliverable is a “finished level”: the level art is textured and lit, the voice-over dialogue has been recorded, and unfortunately I didn’t managed to take notes about what the other aspects of the level should look like. But you get the idea, I hope.
• Stage 5: Finishing. The question: “Can you feel the awesome?”; the deliverable is the final shipping level.

After this, they talked more about some of the lean principles. They avoid muda (waste) by only doing work that they’re willing to iterate upon. They avoid muri (overburdening) by time-boxing and load balancing. (But they don’t load balance across teams.) They avoid mura (variation) by having established deliverables at each phase. They do kaizen (continuous improvement) with level reviews at each phase, with peer reviews (once per level, at the phase in which that department’s work should be final), and they have level design mindshare meetings weekly.

In general, they use a scrum-like process, but not completely; in particular, the individual components (art, cutscenes, modeling, etc.) didn’t always proceed in lock-step, so they felt that sticking too rigorously to sprint goals slowed them down. This is one difference from a pure software development context: in software development, it’s frequently reasonable to have enough cross-training that everybody can swarm on unfinished tasks, whereas a writer is unlikely to be able to help with cinematic rendering. They do have final reviews by the product owner, preceded by at-desk previews; this is a creative signoff, and makes sure that the next steps are clear.

What worked:

• Pairing level art with level design.
• Colocated teams.
• Time boxing.
• Going deep with some levels: rather than doing stage 1 on all levels before going to stage 2 on any, they finished a few levels much earlier than others. Among other things, this helped make sure that the process worked, and improved the quality of their estimates.
• Testing out the combat systems in game.
• Evaluating levels in sections, as necessary.
• Being agile with their Agile process.
• Stopping to assess the big picture.
• Having different goals in different phases. For example, each stage’s question let QA know what bugs can be legitimately filed at a given stage; incidentally, one of the stage exit criteria is that they can’t have any bugs at major severity or higher.

What didn’t work:

• Sometimes, they skipped a step; it came back to bite them.
• Not every level can be a “special snowflake”.
• Level teams that got too big.
• Creatures need to be “representative” early: it’s hard to build a space around a creature if you don’t know what it looks like, how it will fight, how big it is.

That’s my notes from the presentation. What I found most exciting about the talk (other than its heavy lean focus) was the intelligent questions that they used to differentiate stages. In a lot of the agile software literature, you’re told that you not only can but should ship your software frequently (weekly, daily, hourly), leading to a feeling that there’s a lack of texture in the development process. But that simply won’t work for a long narrative project like this. You can imagine breaking the game up into smaller episodes (c.f. their point about going deep on some levels), but nothing that gets you close to a weekly ship cycle. And I think it’s not a coincidence that they went beyond agile to lean in their inspirations: Toyota also has people working on a project with significantly different skills (somebody designing the shape of the body of a car won’t be able to contribute meaningfully to refining the engine), and Toyota is unlikely to have a genuinely shippable car at many of the stages in their design process.

So what they did was keep the core of that “constantly shippable” idea (indeed, they heartily embraced the core, getting huge amounts of mileage out of it), in that the game was always playable and performant; but they explicitly iterated up to a releasable version instead of pretending that the game could always be releasable. And they turned this into a further strength by combining it with the notion of mura: the focused questions for the stages of the iteration turned into a very powerful tool for the elimination of waste.

Of course, their questions, while applicable to many projects, won’t be applicable to all situations. In particular, this game is a sequel, which reduces the scope of the experiments that you’ll want to perform. And they were very up-front about this: when asked if this process would have worked on the original Mass Effect, they answered that yes, they felt that the process would have worked, but the questions would be different.

Which raises the issue of how you come up with the questions that will fit your situation? Further combining my obsessions, I’m wondering if Christopher Alexander’s The Nature of Order might give us some ideas. The second volume is all about growing your structures by amplifying centers, which seems to me to be quite consistent with this sort of phased process; I’ve just started reading the third volume, and I’m cautiously optimistic that some of the worked out examples there of how designs got fleshed out might give some insight into how to structure the specifics of this sort of iteration, how to give up on the notion of a product being always releasable without slipping back into Big Design Up Front.

That last paragraph is probably pretty unlikely; I really don’t have enough of a feel for how, or even if, Alexander’s techniques work in practice to know if they can be turned into concrete guidance like this. Something to dream about, though; certainly this talk was a fascinating talk and a wonderful end to the conference.