The talk that did the most to crystallize these thoughts was the Harmonix one. The speaker started the talk off by addressing the issue that everybody at the company has an idea of how the games should work, should evolve; rather than treating this as an imposition (how dare those programmers tell the designers what the game should be like!), they apparently treat this as an opportunity, a resource. This doesn’t mean that they let everybody go off and stick their own ideas into the game—indeed, most of the talk was about addressing that tension, about how to get a coherent game in the face of so many cooks. But what it made me realize is: I’m totally one of those people who has opinions about everything, and who gets unhappy if his opinions get brushed off. (I’m quite happy to have my opinions argued against and discarded after due consideration, though!) So I’d better spend my time working at companies like Harmonix.

And that has nothing to do with the video game industry: on any but the smallest project, there will be multiple voices that want to be heard, that (in my opinion) deserve to be heard. One of the reasons why I’ve stayed with my current job for so long (coming up on six years!) is that I’ve never (or at least rarely) felt that my voice wasn’t heard: people have given my ideas due consideration, and a reasonable number of them have been adopted by the project. (And a reasonable number have been discarded, too. That’s okay. Incidentally, none of what I’m saying here should be taken as arguing against the agile idea that there’s a Customer who ultimately decides what goes into the project: I have nothing against somebody other than myself having final authority over certain decisions, I just want those decisions to be made after appropriate Consultation.)

This also shed some light on the sort of organizational role that I’d prefer. For the last four and a half years, if I’m counting correctly, I’ve been managing people, but I’ve maintained a job label that puts me in Sun’s engineering track rather than it’s manager track. And this isn’t just a conceit: I will immodestly suggest that I know the details of our product’s code as well as anybody, and I’ve touched the code quite regularly over the course of those years. (And continue to do so: some of the back story behind my last post is my thinking that I should increase my technical efforts in certain areas.)

This puts me in an uncomfortable position career-wise: I like what I’m doing now, with a foot in the programming world and a foot in the managerial world, and I’m all-too-well aware that the vast majority of jobs out there make you choose the one or the other. And I won’t rule out the possibility that I’ll eventually leave the programming world entirely—in particular, I’m getting more and more curious about how organizational change works, which fits better in the managerial world. (This is one of the ways in which my current job continues to hold my interest: over the last half-year or year, I’ve gotten to talk to more people in other parts of the organization, and started thinking about how we might do things differently.)

But, for the time being, I think the main aspect that I like about being a manager is that it makes it a bit easier for my voice to be heard, about the organization of the team as well as the organization of the code. And this brings me back to the thoughts at the start of this blog post: I’m pretty sure that I’d be happy in any sort of position where my thoughts were heard, irrespective of whatever formal power my position within the organization gave me. So I shouldn’t be thinking about being a manager versus being an individual contributor: from the former, I should take the idea that I want to be heard, and from the latter I should take the idea that I want to be in touch with the beauty of the details of what I’m working on. And wherever that leads me is okay.

Back to GDC and the video game industry: in the GDC Confab, Michael asked me if I was thinking of entering the gaming industry; my answer was that I wasn’t sure, and one big worry I had was the industry’s belief that it’s appropriate to force people to work extremely long hours. (I didn’t have anything to say at the time about the “But What I Really Want to Do Is Make Games” panel, but one of the comments that’s stuck with me was a panelist saying, basically, games journalists are already used to working ridiculous hours, so that’s a way in which they fit right into the game development world!) I don’t think that’s a good way to produce software in general—I’m pretty sure I can’t maintain disciplined, creative thoughts on one project for more than 40 hours a week—but even if it were, I’m not so obsessed with my career that I’m willing to let it dominate all other aspects of my life, my family in particular. I’ve recently heard the statistic claimed that the average amount of time people spend in the game development industry as a whole is five years, while I’ve already been at a single job for longer than that; the humane working conditions are a big part of that, and any industry would have a hard time learning and growing if it churns through its workers as quickly as the games industry apparently does.

It is certainly the case, however, that a lot of what I saw at GDC excited me. People were talking about all sorts of interesting things, things that I don’t necessarily get a chance to think about so much in my current job. And its artistic nature is a big draw for me: I’d like to work on something beautiful, something that nourishes my soul. Though, to be sure, code can be beautiful and nourish my soul, even if the beauty is hidden within the product: it may well be the case that I’d get more out of revealing the beauty hidden within some of our code at work than working on something more overtly artistic! Also, working on games would be a change of pace, which I like; I’m sure the fact that my role at work has changed significantly every year or two is one of the big reasons why I’m still there. (And has been changing again over the last year, the last few months, even seeing the seeds of new ideas over the last couple of weeks.)

If I had to say what sort of position appeals to me most right now, here’s a stab at what my list would look like:

• I’d be getting my hands dirty with software
• but would be able to think and talk about aspects of the product beyond just the code
• (and have people listen to me!)
• including the structure of the organization.
• (Said organization would ideally be small,
• and in particular I’d be on a small team.)
• That structure would be an agile one
• or, better/broader yet, a lean one,
• including working at a Sustainable Pace.
• (Working close to home would be good, too.)
• The product would be a work of art
• (a game would be nice),
• working on it would nourish my soul.
• And I’d be learning something new,
• both technologically
• and at a domain level.

Which is a lot to ask for. (But if there are any companies in Mountain View using agile methods to write soul-nourishing games in Erlang, please get in touch!) In the mean time, I’m doing pretty well on that checklist right now…

This session’s simulation was about a factory that had decided to enter the lucrative “fancy pinwheel” market. She started out by dividing us into four groups (cutters, assemblers, testers, managers), and plunked us down in a room without a lot of information. We muddled around for a while, gradually figuring out our environment: there was some information about the customer, about the market, about suppliers that took quite some effort to piece together. (And it was quite a bit more work to make good use of that information; even at the end of the exercise, there were important pieces of information that I was unaware of and that, as far as I could tell, we weren’t using at all.) While doing this muddling, the groups changed (to some extent), and eventually we managed to produce something useful; but we neither produced as much as we could have nor reached that productive state as quickly as we could have.

Thoughts:

• The simulation was well-paced: everything went by a bit too fast, quickly enough that we didn’t react nearly well enough to the information (we would probably have done a lot better if we’d had twice or even half again as much time between stages), but not so quickly that we were just lost. Which felt realistic to me, though for all I know a lot of real-world change situations feel like they’re going by a lot faster! I was also impressed at how well the task of pinwheel construction served as a problem for us to work on.
• There were (at least) three different kinds of groups in play: the pre-existing groups that we were assigned into, groups that formed out of their members’ common interests, and groups that managers tried to form. These didn’t overlap particularly well: e.g. when management decided to form an R&D group, they didn’t realize that there were already people who’d started doing that without direction!
• I was surprised at the power of the pre-existing groups: those were created out of nothing by handing us labels (whereas in a real-world situation they would be much much stronger), yet they had a strong life throughout the exercise. I strengthened them in an effort to wrap my head around the situation, by asking people to display their group identifier in a more prominent fashion; in retrospect, I wonder if I hurt us more than I helped us by doing that?
• People outside of groups were useful, too: just wandering around noticing stuff and seeing if information should be moved from place to place had its benefits. In fact, that probably should have happened more, and the results should have been broadcast more publicly: in times of change, it may be more useful to spread new learning than it is to work off of your old instincts. (There’s no point in running more quickly in the wrong direction.)
• There are least two ways in which you should give extra care to information in a change situation. In general, in such circumstances, people are grasping for information, and they fill in gaps by “horribilizing”. (I.e. filling in gaps with the worst possible interpretation.) So go out of your way to not hide information: even if people don’t need to know something, that doesn’t mean you should prevent them from learning that, and you probably want to go out of your way to overcommunicate. (Side note: according to Esther, people have to hear something 30-40 times for it to sink in.)
• But if you’re immersed in a sea of information, you’ll just stay in chaos: you need vision and guidance to get to the new status quo together. So make sure to highlight (30-40 times!) the information that helps reinforce the new vision. In this particular situation, I think there were a couple of pieces of information (scheduling for a constraint, and our best guess at the desired design) that should have been placed in a prominent location.
• In particular, thinking in Theory of Constraints terms, we didn’t exploit the constraint effectively. This exercise gave me a much more concrete appreciation of ToC than I’d had before: I’ve been struggling to apply it at work (I’ve been thinking about this for years and I still don’t know where our constraint really is), but this exercise made that focusing step much more vivid to me.

She also passed out a lovely article about change involving her dog. I haven’t yet found a copy online, but there’s an earlier version available in this blog post.

Edit: Here’s the full article mentioned in the last paragraph.

• Visualizing the Python commit history.
• Leadership, responsibility, and sausage.
• Solving sudoku games via package management.
• Japan, computers, appliances. (Via Niels ‘t Hooft.)
• Breakpoints as a checklist.
• Programmers, insecurity, source control.

• I linked to a movie of strandbeests (amazing wind-powered sculptures that walk along beaches) before; the creator has done a TED talk with more background.

  (Via Giles Bowkett.)

• How to draw hands.

  (Via Drawn!)

• This year’s rice paddy art.
• I am a sucker for dialogues; having them be about the video game business is even better.
• Minus, a thoroughly delightful webcomic. I should go through them all again… (Via Dan Bruno.)
• I laughed out loud at the second menu. These ones are amusing, too.
• Abandoned cities. Provides some interesting context for the second part of Shenmue II. (Insert obligitory lament on the nonexistence of Shenmue III…)
• Transcriptions and discussion of Zelda (specifically Ocarina) music.
• How adding tools for user-generated content can significantly shorten the developers’ content-generation time. Constant refactoring, taken up a level.

• Pachelbel’s Canon and Korean breakdancers. And now I know what a gayageum is…

  (Via How the World Works, which also has several other nice renditions.)

• Brecht and MGS2.

This seemed like a sensible enough idea to me: in particular, it’s all to the good to have an reminder to lift your head a bit and step away from the details in order to get a broader overview of what you want to be doing. And, at home, it’s worked out in a fairly straightforward fashion.

It’s worked out well at work, too, but with one surprise: doing a weekly review takes forever! Well, not forever, but about an hour on average: put another way, even though I have GTD and the whole agile toolbag to help organize both my general priorities and my specific next actions, I still have to spend around two percent of my working time making sure that I’m not going off the rails. (Hmm, what percentage of my working time is spent on planning and organization in all its forms?)

It’s definitely worth it, though: much better to spend time to learn each week how I’m starting to go off the rails than to save time in the short term by doing the wrong thing! (Ending up with a bad product and, probably, spending more time later picking up the pieces.)

So what’s going on there? Part of the difference between work and home is that I just have more moving pieces at work than at home: I’m interacting with more people, and I have more projects. Come to think of it, maybe I have about the same number of projects at both places, but I’m generally happy for the ones at home to be done when they’re done, while the ones at work have more pressure behind them.

Part of the weekly review at both places is a sweep through my e-mail folders: actions/waiting/scheduled/conversations. That definitely takes more time at work than at home: I save a lot more e-mails, and I’m more likely to have to spend time thinking about whether or not I’m comfortable with, say, letting an e-mail thread in ‘conversations’ rest, or whether there’s a covert action / project / someday lurking in it.

But probably the biggest difference is that, at work, I accumulate new potential tasks at a much higher rate than I do at home. Each week, new high-priority items will come along; I’ll typically shuffle them into the projects / action item lists somehow, but every week I need to take a hard look and ask myself if I can really expect to make progress on all of these projects. And the answer is almost invariably no, at which point I have to move something from projects to someday / maybe, and communicate that decision to other people.

Looking back, it seems that I’ve been doing GTD for more than half a year now. I remain convinced that it’s a great system: well-thought-out parts, simple ideas, and I’ve found it personally quite effective. Though I still have a ways to go to implement it fully: in particular, I’m seeing more and more that I need to regularize my filing system at home. But I have a project for that (together with its next action or two) in my projects file, so I’m confident that I’ll accomplish it!

Specifically, Sun’s internal tools only accept resumes in either text format or variants of Microsoft Word. Lots of people apparently don’t have a resume natively available in one of those formats, which means that their resume gets cut-and-pasted from another format into text, with the result that it looks like crap and is a pain in the butt to read. (In particular, resumes are typically full of bullet points and indentation, and neither of those reliably survives that journey.) I literally spend most of five minutes going through a typical resume changing the formatting so it doesn’t get in the way of my reading the resume; it isn’t a complete waste of time, because I’m doing a first pass at skimming the resume while reformatting it, but it also isn’t much fun.

So: what is a resume writer to do? It’s been ages since I’ve updated my resume (and no, I am not looking for a job: I just think it’s wise to update your resume every year or so, since I can’t reliably remember what I was doing much farther back than that); the last time I applied for jobs, I used a LaTeX file which I converted into PDF. (Which took a surprising amount of care to get looking right, if I remember correctly: some sort of font problem in the conversion.) Which means that hiring managers at Sun probably wouldn’t like me!

So what are good resume formats these days? Based on my experiences, it’s essential to have a good-looking text version: text is easy to e-mail, it’s a fallback that will always be available. You also want a version that’s nicely formatted; presumably PDF is the format of choice there. And you may want to put your resume on your personal web page, so HTML is probably a good third option.

But, of course, you only want one source representation. I used LaTeX for this in the past, and I’m still not convinced it’s a crazy idea: I think there are probably decent tools to go from LaTeX to HTML or text. Having said that, there’s also nothing about LaTeX that makes it uniquely well suited to the task. A resume is a lightly formatted extremely hierarchical document; any sort of markup language that lets you easily express that hierarchy while giving a reasonable amount of control over formatting should do the trick.

In particular, HTML should probably do the trick. You’d want to take a bit of care over the CSS that you use to style it with, but I don’t think resumes put any excessive demands on styling. You’d especially want to take care when converting it to PDF; PrinceXML seems to be getting a fair amount of buzz these days, so I’d be tempted to play around with that, despite its closed-source nature. Though my first line of attack would just be to provide a print-specific version of my CSS file; among other things, that would improve the way it looks to people who are printing out the resume from my web page. Were I to chose to put my resume there; not sure what I feel about that yet.

What’s the best way to convert HTML to text in a way that works well for resumes? I could have fun with XSLT, but that’s probably overkill. Honestly, maybe just loading the web page with Lynx would be good enough; I’d have to try it and see, once I get around to actually updating my resume. If not, there must be hundreds of other options.

One other tip for job applicants: when you attach your resume to something, the original file name of the resume will be available to the person receiving the resume, and it will probably be given as a default option for that person to save the resume under. So realize that, when you name your resume, you aren’t the main client of that name, the hiring managers are. In particular, if you want to give your hiring manager warm fuzzies, don’t call it “resume.pdf” or “David C Alternate Resume.pdf”: call it “DavidCarlton.pdf” or “DavidCarltonResume.pdf”. The details aren’t important—different hiring managers have different conventions about the names they’d use to save resumes under—but make sure that your full name is there and that there isn’t other extraneous garbage in the name. If you need to store metadata like that in your local copy, put it in the directory hierarchy, not in the filename.

One-on-one meetings provide managers an opportunity to ascertain status, solve problems, and provide positive and corrective feedback.

Which I was a bit dubious about: daily standups provide a mechanism for me to get status from my team members every morning and for them to raise problems that I can solve, and having one-on-ones just to give “positive and corrective feedback” didn’t sit well with me. Still, I respected the authors, and thought that it was a good book in general, so, a year and a half or so ago, I decided to give it a try. (Incidentally, I still respect the authors and still think it’s a good book!)

I planned to have them once a month; they happened for about three months, and then petered out, and nobody seemed to miss them.

A few months later, though, I realized that I have one-on-ones with my boss every week and rather enjoy them: what’s the difference here? Part of the difference is that he doesn’t have a daily standup for his team, so there really is more room for getting weekly status updates. But I didn’t think that was the whole story: in particular, what I most liked about the one-on-ones was getting a chance to talk about matters that didn’t seem to fit into other interactions we had. (Sometimes issues that one of us had, sometimes just talking at random about this and that.)

Looking back at my team’s earlier one-on-ones with this in mind, then, it seemed like there were two differences. One is that I’d tried to have a goal for each one-on-one; another is that they only happened once a month, instead of once a week. I suspect that these differences are related: the less frequent something is, the more of an occasion it is, hence the more pressure there is to make something of it.

Having said that, I didn’t feel like there was much of a reason to have one-on-ones with my team members every week. But every two weeks, with no set agenda (except for rare exceptions), seemed like a good fit. That’s frequent enough for them to be routine and to fit in on a repeating schedule in my calendar, and it means that I rarely have anything specific that I want to talk about in the meetings, which reinforces their informal nature.

We’ve been doing this for several months now; I like the results. We usually find something to chat about; and, if we really don’t have anything to say, it’s fine if it only lasts five minutes! Not infrequently, it turns out that we really do have something to talk to that we wouldn’t have talked about without the excuse of a one-on-one, which means that issues are less likely to build up. (Don’t get me wrong, it’s not like we had huge buried minefields before.) On a few occasions, something big has come up; on one occasion, it’s been useful that I had scheduled opportunities to talk to team members individually, so I could sound people out on an issue that we’d had a hard time dealing with collectively without blowing it up out of proportion.

So: I am a fan of one-on-ones, but not for the reasons given in the book. Don’t get me wrong, if you aren’t getting status updates regularly through other mechanisms, then by all means talk about that in your one-on-ones, but there’s also real value in just having a regular opportunity to chat. And schedule them more frequently than you think would be necessary, frequent enough that they’re not an event and that there’s never any pressure to find something suitably weighty to chat about.

• At work, I think we’re doing a reasonable job of adding new features. Though I’m sure there’s room for improvement.
• I also think we’re doing a reasonable job of fixing bugs: acceptance test failures are way down, and we’re even successfully attacking sporadic bugs and old, thorny bugs.
• Not so sure about code maintainability: there’s even some evidence to suggest that our code maintainability has, in some areas, gotten worse recently.
• Code maintainability is harder to measure than features and bugs. And there’s less external pressure to get it right, so not surprising that it’s fallen by the wayside. Because of our successes in other areas, and because we’re doing a better job of planning this release than the last one, though, we have some time to attack it.
• I wish I were better at helping various teams that I’m part of improve our processes.
• One-on-ones are a good idea, even (especially?) if you don’t know what you’ll get out of them. And the more frequently you have them, the lower the pressure, which is a big help to everybody.
• The book I’m currently reading at work is Matthew May’s The Elegant Solution. Which is reminding me of some aspects of lean that I hadn’t been focusing on, especially on the “respect for people” side.
• Having the team all focused on the same, small-granularity tasks is wonderful in terms of making concrete progress in ways where our work reinforces each other and matches business value. Not necessarily so great in terms of letting people, say, focus on what they do best or define their own job.
• One thing that May talks about is the power of opposing goals (make a car faster and get better mileage and lighter and cheaper by these specific amounts), and the evils of satisficing. Simultaneous satisfying all your goals sounds wonderful if you can do it; I wish I knew how. I suspect that Toyota has some very useful techniques to this end.
• Alexia Bowers gave a good examples of meetin opposing goals, if I’m remembering the podcast correctly.
• The book before last that I read was a guide to the ToC thinking tools. (See also It’s Not Luck.) Do these actually work? My brain is strangely resistant to even giving them a try.
• I think I’m getting better at not talking in meetings, about chiming in and then letting other people argue for a while. Gratifying that, not infrequently, other people make the arguments that I would have made were I talking.
• I stayed home on Friday, because Miranda was sick, and called into two meetings. Both of which were very frustrating. I think part of it was that I missed some of the cues of the flow of the meeting, and part of it was that I wasn’t very good at explaining, or even seeing, how we were talking past each other. (I did think of an evaporating cloud to explain one of the conflicts after the fact, for better or for worse.)
• I wish I spent more time talking to people in other parts of Sun.
• What do I want to do when I grow up?
• After taking a break for a few years, I’ve gone to one conference each of the last two years, and gotten a lot out of each of them. I should continue this going forward. (And possibly even ramp it up a bit, since if there are further Agile Open Californias, I’m not going to stop going to them.) Where should I go next year?
• What communities do I want to be part of? What does it mean to be part of those communities?
• What teams am I currently part of? Do those teams behave how I think a team should behave? If not, how should I behave?

I could probably ramble on in this vein for quite some time; time to go to bed. Happy Armistice Day, all.

From the first volume, Systems Thinking, p. 110:

I once was called in to consult on a troubled project that was missing all of its goals, much to the puzzlement of its project manager, Simon. As part of my visit, I attended a code review meeting that (against my advice) Simon also attended. Herb, whose code was being reviewed, took a lot of personal abuse from Simon to the point where his eyes started watering. I called for a “health break,” and during the break, Simon came up to me and asked, “Does Herb have something in his eye?”

“Why do you ask?” I replied.

“Well, I noticed that there was water coming out of his eye.”

From the third volume, Congruent Action, pp. 60–61:

If you are acting incongruently, you are likely to trigger incongruent reactions in others. Rather than blaming them for incongruence, ask yourself, “What could I be doing to contribute to their behavior?” Here’s how it worked for Parson, one of my students:

“I was telling one of my project managers that I had to see a plan to get her project back on schedule. As she handed me a folder containing her revised project plan, I became aware that the papers in her hand were rattling. That caught my attention, and I thought, ‘How strange that the papers should rattle like that.’ Looking for an explanation, I noticed she was trembling, that her face was ashen, and finally that her eyes were wet.

“My first thought was, ‘Oh, she’s sick!’ but then I remembered the idea from class that she might be reacting to me. That seemed ridiculous, as I thought I was merely talking to her normally, but I decided to check it out.

“The first thing I noticed about myself was that I was gripping the edge of my desk as if it were a safety rail between me and the Grand Canyon. I thought I should loosen my grip, but then I realized I would probably fall on my face toward her if I did. All this while I continued talking to her about the project plan, until I noticed that I was actually shouting and banging my other fist on the desk. That’s when I had my big AHA!”

From the second volume, First-Order Measurement, p. 109, defending the definition that “Quality is value to some person”:

Why is this definition so troublesome to some people? There are people in the world whose strongest desire is to find perfection, the one right way. Many of these people choose to work with computers, and particularly with software. They don’t like the idea that quality is relative – to people, to place, to time – but it is.

And again from Congruent Action, p. 196 this time:

Avoiding blamers works if you don’t have to deal with them again, but this is exactly the opposite of engaging. If your work requires that you engage the blamer, you need to learn the method of the aikido masters when dealing with all forms of attack, including blame:

When someone hits you, he is extending his ki toward you and it starts to flow when he thinks he will hit you — even before his body moves. His action is directed by his mind. You don’t need to deal with his body at all if you can redirect his mind and the flow of his ki. That’s the secret; lead his mind away from you and the body will folow.

In aikido, you lead the mind away by physical means, though these are always as gentle as possible. The idea is not to further upset the person or draw more attention to yourself, as stiff resistance or a punch would do. In fact, skilled aikidoists often disable their attackers without even touching them, which of course is what you want to do with someone who is blaming you.

Blaming can be handled in the same way, first by yielding, but in such a way that the blame is unable to harm you. Done properly, this surprising move engages the blamer’s mind, so that you can easily change its direction. For example, suppose that an employee blames you for his failure to develop a specified function. “You didn’t tell me that this function was part of the spec,” he screams. Although you remember telling him, you don’t try to deny the allegation, which only focuses his blame more firmly on you. Instead, you may say, “If you didn’t know it was in the spec, I can certainly understand why you didn’t develop it.”

Saying this, you have agreed with his anger without accepting his blame. Next, you redirect the energy of this blame (the ki, in aikido terms) into something more productive. You have aligned with his energy, so you can push from behind rather than resisting it from the front. You might say, “What’s the best way for you to be informed of functions to implement?” This makes you collaborators, rather than opponents, by turning the energy toward preventing the problem in the future, rather than belaboring the unchangeable past. Even better, you have not become a blamer yourself.

Sometimes, rejection over e-mail makes sense. I typically put candidates through up to three stages of filters. (Not counting the initial resume screen.) The first stage is a sanity check over the phone: is there some obvious reason why this situation is a misfit that I didn’t figure out from the resume? Almost everybody makes it through this stage; in the few places where a candidate doesn’t make it through this stage, it’s because it’s clear to both of us that their goals aren’t a fit for this position, so we agree that it’s not a match, and it’s simply not a matter of me rejecting them. (One could argue that I should make my phone interview stricter and reject more candidates at this stage; for better or for worse, I’m not doing that yet, and it’s tangential to the issues in this post.) The third stage is a half-day team interview; in this situation, my team discusses the candidate as a group after the interview is over (usually the next day), so I’m simply not in a position to deliver an immediate verdict to the candidate.

The second stage is the tricky one. This is a one hour in-person interview, where I ask some questions and have the candidate do a bit of programming. Sometimes, after the end of the interview, I’m genuinely not sure whether or not I want to bring the candidate back. More frequently, however, I am sure, and the answer is “no”. I can’t remember having second thoughts about my initial no reaction at this stage, so why not deliver the news then?

I think my actions started to bother me because of some blog posts that I’ve read recently, but I just looked at the usual suspects, and I couldn’t find any relevant posts. Certainly one reason why I’m thinking about this, though, is that I’m in the middle of rereading Gerald Weinberg’s Congruent Action. I can’t claim that I’m acting congruently in this case: I don’t believe that it makes job candidates any happier to have to wait and wonder for a day or two. Nobody likes being rejected, but you might as well get the news as soon as it’s available. The only reason why I’m delaying is to avoid in-person awkwardness; it’s just a sign of lack of courage on my part, with no obvious benefit to anybody.

So today I rejected two candidates in person. And, actually, it turned out rather well: both were disappointed, but both took it well, and both asked for advice about what they could have done better. In retrospect, it seems that I haven’t been saving myself any grief: all I’ve been doing is elimimating a potential learning opportunity for the job candidates. (Which raises the question: what learning opportunities might my actions in this regard be costing me?) Nice to have immediate positive reinforcement like that: I will try to continue to behave this way in the future. (And I should spend some time thinking about how to politely reject people in person.)

To be sure, I don’t claim to have any great pearls of wisdom on what the candidates could have done better. But, in time-hallowed blogger tradition, I won’t let that stop me from sharing my thoughts on the subject. The interesting thing about this case was that both candidates failed for the same reason: neither was completely hopeless, their initial attempts at a solution were both reasonable, but both floundered significantly when debugging. (Learning about people’s debugging skills is actually my goal in the programming question: I’m always a little disappointed when people solve the problem without making a mistake, because I know that, no matter whom I hire, they’ll make programming mistakes with some regularity, and I want to see how they deal with that.)

And, in both cases, it seemed like they didn’t know what they were looking for when debugging, or indeed when testing the program before discovering that debugging is warranted. (I tell them to implement a function and do enough testing to convince themselves that it was correct.) I suspect that both of them could be helped by taking a more scientific approach to debugging.

Sometimes, scientists are just gathering data: poking around, seeing if they see anything interesting. But science really progresses when people are making and testing hypotheses. Make a prediction that is concrete enough to be testable, to be falsifiable, and then test it. If the test results match your prediction, that’s always fun; if not, though, you’ve still learned something concrete, and can use that to make further hypotheses.

And this works for debugging, too. At the very basic level: if you think something is wrong with your code, you should know what you expect to have happened, so you can tell whether or not something really did go wrong! But it helps to make concrete predictions at a more refined level than that: “we know that something went wrong overall because we saw X instead of Y. If the problem isn’t in this part of the code, then expression E should have value V, whereas if it is there, then E should have a different value”. If you do this a few times, you should be able to zero in on the problem quite quickly, much faster than you could have by just looking at the code and hoping for inspiration.

This helped crystallize one thing that I think is wrong with printf debugging. Many people’s first reaction, when confronted with misbehaving code, is to sprinkle printouts throughout the code and hope that enlightenment results. This can be great, if you have specific hypotheses about what the output of those messages should be. I think that many people, though, don’t really have specific hypotheses in mind, just a vague feeling of values that they’re interested in. When this happens, printf debugging doesn’t lead to answers very quickly: the messages give you a lot of data, data that can easily lead you down all sorts of unproductive paths, data that can fool you into thinking that you’re learning something when you don’t really know what that data represents. (I suspect that debugging with a debugger is less likely to lead to this problem, if only because it takes more effort to generate lots of values, so you’re more likely to spend time thinking about what values you want to generate.)

Maybe I’m unfairly characterizing printf debugging, but I will stand by the value of concrete hypotheses when debugging. Which raises the question: if it’s so good when debugging, can we use the same ideas when writing new code, code which we don’t yet have reason to believe is incorrect? The answer is yes, of course: that’s exactly what test-driven development is all about.

People sometimes get annoyed when I say job interviews are a weak, often useless, way to select new employees. I’ve had executives, middle managers, engineers, scientists, lawyers, a fire chief, and a minister respond with anecdotes that “prove” how skilled they are at using interviews to pick which job candidates will succeed and which will fail, even if others are lousy interviewers. Their confidence clashes with literally hundreds of studies, going back to before World War I, showing that there is rarely much agreement about who should be hired or who will perform best (and worst) when several interviewers talk to the same job candidate. These studies conclude that the typical “selection interview” is a bad method for deciding which employees to hire. A much better way to pick good employees is to just see if they can do the job, or at least crucial parts of the job – to give them “job sample tests.”

Most companies interview candidates something like this: An untrained interviewer leads a job candidate through an unstructured, unplanned conversation. No record is kept of what questions were asked or answered, and the person who ultimately makes the decision to hire the person – or not – sometimes has only a dim understanding of the job skills needed. Despite these flaws, the interviewer has great confidence that he or she can distinguish between good and bad candidates. Unfortunately, research shows that job interviewing is a lot like driving, where 90 percent of adult drivers report that they have “above average” skills. The truth is that the typical interviewer learns little useful information for predicting job performance beyond what is available no the applicant’s job application and resume.

Another reminder of how much I doubtless have to learn about hiring. (And a reminder that there might be much for me to learn from empirical studies about this and other managerial topics.)

Despite which I insist on continuing to try to hire. Here are my current open reqs. (The reqs should basically be identical; I only link to both of them because I might fill one of them one of these days, at which point that link will go stale.) If you know anybody who lives in the Bay Area or is planning to move here, who has good OO programming skills, and who likes the idea of streaming out 320Gbps of data, please send them my way. Consider this an open invitation: I expect to be hiring off and on for the indefinite future.

I’d gotten my previous take on the subject from others’ books and from a presentation by Ron Jeffries; back when (a previous incarnation of) my team used to estimate regularly, we followed Ron’s 1-point, 2-point, 3-point idea. (Well, we did until we dropped 3 and added 1/2, but the result is almost the same thing.) Mike Cohn, however, uses much larger numbers: 1/2, 1, 2, 3, 5, 8, 13, 20, 40, 100.

That’s for story points; he recommends estimating tasks within stories in terms of hours. (I can’t remember if Ron talked about estimating tasks at all.) And he made a good point about why you should use artificial points instead of real time units for your story estimations: if you use real time for both, you’ll be tempted to expect, say, the time estimated for the tasks making up a story to add up to the time for the entire story. Which makes sense, except that you estimate a story before you’ve broken it up into tasks (you don’t do the latter until somebody has decided that you’ll work on it), so when you do the task estimation, you’ll have thought much more about what’s involved in implementing the story. And you can’t convert between “hours that you’ve thought a lot about” and “hours that you haven’t thought much about”, which you’d be sorely tempted to do if you use hours in both situations.

Mike came about his expertise on the subject honestly, by the way: he was VP of engineering at a company that had adopted Scrum, and that had a fair number of teams working on not-very-long-lived projects. So teams had to estimate stuff, and he imposed the rule that, each project, you had to do something different when you estimated. There was enough discussion going around that people had an idea of what teams with accurate estimates had done in the past, but the rule meant that they couldn’t just stop and declare victory, they had to keep on trying to find ways to improve. A nice example of evolutionary process improvement.

Anyways, after the talk, I asked him about his versus Ron’s recommended range of estimation values. Part of his answer was that maybe the right thing for somebody working in the trenches is different for the right thing for a VP of engineering. More generally, people are going to ask the team how long it takes to implement a feature that’s larger than a simple story; they need a way to answer that. Which is a good point – I don’t have that clear a view on how Ron recommends estimating features larger than a single story. (I should ask, shouldn’t I?)

There’s still a tension there that I’m not entirely comfortable with. Unless you go with long iterations (and Mike prefers two-week iterations, which already doesn’t seem long enough), I don’t see how you can fit stories that vary anywhere near a hundredfold in length into a single iteration. Now, stories at the extremes (especially the large end) are bad, but still, a 1- to 13-point range (or whatever) seems too wide to me to fit within an iteration. But a story that can’t be done within a single iteration isn’t really a story, is it?

So maybe there are there levels needed: features, stories, tasks. Each with their own (non-convertible, as above) estimations. But that’s too much estimation. Given that, I’d actually be tempted to drop the task estimation instead of the feature estimation: isn’t it kind of pointless to spend time how many hours a task will take? Just implement the damn thing! In the previous incarnation of my team, we did break down stories into tasks (we should get back to doing that, it was useful), but we didn’t estimate individual tasks, and I never felt the lack. Maybe I was missing something, but it still seems funny to me.

Actually, though, it’s entirely possible that we were subtly shifting things by a level (and making them too long, to boot.) Because the truth is that a lot of our stories were technical: we weren’t clever enough (and weren’t working with a Customer representative to give us a nudge) to break work up into small, customer-visible units. So maybe what we called stories were really tasks? I don’t think that’s quite accurate, but there’s enough truth to that to make me nervous; something to think about more.

Since Stuart brought it up (see his blog post on the talk if you want another take), I might as well talk about another question I had. Mike presented some very interesting examples (you can see his slides, by the way) of studies that showed that, when people were given extra, irrelevant information, their estimates for tasks increased. (My favorite example was when group A and group B were given exactly the same text, but in one case on a single piece of paper while in another case spread over seven pieces of paper.) To which I asked: that’s neat, but which estimate is more accurate?

I freely admit that I asked this solely out of methodological purity: even though the studies didn’t give any evidence about the relative accuracy, I know which way I’d bet. (Well, one of the studies sort of did give evidence: they gave three teams the same tasks, but told team A nothing about expectations, team B that the customer hoped it could be done in 500 hours and team C that the customer hoped it could be done in 50 hours. All teams insisted that the hopes had nothing to do with their estimates, but team A ended up with an estimate of 456 hours, team B with 555 hours, and team C with 99 hours! Scary, that: a trap that I fall into all too often myself.)

But, the more I think about it, the less sure I am which team’s estimate is the most accurate. Take, for example, the study where team A was told to estimate requirements 1-4, team B was told to estimate requirements 1-5, and team C was told to estimate requirements 1-4 but were also given the future requirement 5 for purely informational purposes. In this case, A and B both estimated 4 hours (even though B was told to estimate strictly more work than A) while C estimated 8 hours (even though they were told to estimate the same work as A)! Looking at that, I don’t see at all why I should believe that A is the most accurate – they give the same answer as B, which is within the margin of error but clearly odd. What seems more likely to me is that A and B are estimating in terms of “hours we haven’t thought much about” while C is estimating in terms of “hours we’ve thought more about”, which we learned earlier can’t be converted to each other!

Anyways: good talk, a good reminder that we should get back to estimating once matters get a bit more under control, and I ended up with a book and enough sets of his planning poker cards that we can use them in our future team meetings. If you have a chance to hear him, I definitely recommend it.

A company that cannot standardize will struggle to learn from experience and is not truly engaged in lean thinking. Indeed, any company that simply tries new things without standardizing along the way is “randomly wandering through a maze,” repeating the same errors, relying on little more than undocumented hearsay and a wide range of opinions among its employees only to eventually discover that “it has been here before.”

A little more context:

Though based on science, the real world practice of engineering is an art form that relies on tacit knowledge gained through experience and judgment in considering multiple variables that interact in complex ways. As a result, a best solution cannot necessarily be predicted in advance. It is learned over time through experience and is guided by the spirit of kaizen, which postulates that there is always an opportunity to learn more and that learning is an ongoing process. This spirit of engineering kaizen is driven by the never-ending pursuit of technical excellence that underlies consistent checklist utilization, validation, and improvement.

A company that cannot standardize will struggle to learn from experience and is not truly engaged in lean thinking. Indeed, any company that simply tries new things without standardizing along the way is “randomly wandering through a maze,” repeating the same errors, relying on little more than undocumented hearsay and a wide range of opinions among its employees only to eventually discover that “it has been here before.” Toyota uses a systematic and scientific approach to product development. It tests, evaluates, standardizes, improves, and retests, scrupulously following the Plan-Do-Check-Act cycle that was introduced to the company decades ago by Deming. It then standardizes “today’s” best practice. As it accumulates new information and new experiences, these are used to modify current shared standards and reborn as a future “today’s” best practice.

Go experimentation, both trying it and taking the results seriously. We came to a similar conclusion at work last week; we’ll see how our experiment of experimenting turns out.

(I should read some Deming, shouldn’t I?)

Toyota does very little “information broadcasting” to the masses. Instead, it is up to the individual engineer to know what he or she is responsible for, to pull what is needed, and to know where to get it.

Here’s the full context (pp. 95-96; italics in original):

Pulling Knowledge Through the [Product Development] System

In lean manufacturing, pull production eliminates overproduction by having downstream activities signal their needs (demand) to upstream activities. Kanban cards usually signal (control) production in a pull system. In product development, knowledge and information are the materials that are required by the downstream activity. The speed at which technology delivers information in automotive product development is overwhelming. However, not all information is equal to all people. The lean [Product Development] System uses “pull” to sort through this mass of data to get the right information to the right engineer at the right time. Knowledge is the fundamental element (material) in product development.

Toyota does very little “information broadcasting” to the masses. Instead, it is up to the individual engineer to know what he or she is responsible for, to pull what is needed, and to know where to get it. Individual engineers are expected to locate and extract needed information, whether this be design data residing in the data collector, a product performance experience, or a perspective from a senior executive. This policy holds true for everyone, from the most junior design and release engineer to the chief engineer. The key underlying principle that makes this work is that everyone has access to both the design data and the [Chief Engineer].

For an example from the opposite end of the program hierarchy, all engineers are responsible for creating benchmarks for their respective components. They are expected to gather relevant information and understand the latest technological developments, industry trends, and supplier and competitor products that affect their designs. Once the execution phase begins, manufacturing engineers pull design data from data collectors as they need it to start working on die or fixture designs. All engineers pull requirements from checklists, which are updated at the end of each program.

The supplier mentioned earlier (a company that had an unacceptable management cycle time) illustrates how the [Product Development] system links processes. This seat maker through value stream mapping identified thaht they were batch dumping information onto the next process (design sent hundreds of drawings to purchasing and ordered hundreds of parts prototyping, to build the hundreds of different variations of prototype seats, etc.) After moving to a staggered release system where a subset of seat designs were released on a preplanned schedule, weekly reviews of progress were set up and the supplier set up status boards at each functional area within the value chain. A key purpose of the status boards (referred to as “pull boards”) was to signal the need for information from other functions. Once the status board was in place, it was easy to spot when key information was needed. When key information was delayed, it was identified within a week rather than months later. The example clearly shows that in a lean [Product Development] process, a key enabler for pull knowledge systems is reducing management cycle time.

At first, I was thinking of “don’t broadcast information” in terms of “I don’t like being lectured at”, which made me happy. But I do like a general low level of chatter among team members – e.g. at our daily standup this morning, two of us were talking about what we did yesterday, and another team member mentioned an old bug report that turned out to be relevant, quite possibly saving us a day of time. And if we hadn’t been broadcasting information, that wouldn’t have happened. Then again, some XP sources suggest that the need for a daily standup is a sign that your process isn’t working well enough, but then yet again that’s because they want information to radiate even more (by everybody working in a big, open room, constantly interacting). But that only works with teams below a certain size; eventually, you have to cut off universal chatter to save your sanity.

When I balance all that, I tend to think that I like broadcasting information within a group of a certain size. And I bet Toyota does, too, I just need to find the right section of the book. I could be wrong, though; it’s certainly something to think about. That’s the problem with reading books like this: I’m missing so much of the context, so many details, so much of the gestalt.

Moving along, we see that individuals are apparently able to pull the information they need at will. Which is certainly a problem that we have: we have various “data collectors” (especially our wiki), but they’re not well-organized, consistently maintained, and up to date.

One interesting thing about Toyota product development is apparently that, on the one hand, they’re quite good at consistently storing information, e.g. about results of experiments (whether positive or negative). But they also manage to do this in a terse, accessible form: I’m pretty sure they save the lab notebooks, but they also put their results on a standardized single piece of paper, the “A3 form”. I haven’t yet gotten to the section of the book where they talk about that in detail, but it sounds like it could be a really useful idea, and if done right a very useful antidote to wiki chaos.

Using my mad queue-management skillz, I’ve finally gotten caught up on my podcast listening: for every podcast that I regularly listen to, I now have either no or less than one week’s worth of episodes of any podcast, and have been in that blessed state for about a month by now. It doesn’t hurt that, since changing to the Menlo Park office, my new commute is a bit longer, especially going home – grr 101 grr – but I was heading in that direction even before the office move. In fact, some days, I don’t have any podcasts stored up to listen to, which leaves me at a bit of a loss. Especially with the holidays coming up, when some podcasters have the nerve to take a bit of time off.

Which means that I should find more podcasts to listen to, right? No! (This is how you can tell that I have mad queue-management skillz.) You see, I know about the virtues of maintaining a bit of slack in the schedule. I’ve reached a level where I can predictably listen to all of my podcasts almost every week – some episodes might stick around for two weeks, if a bunch of podcasts that publish on irregular averaging-to-about-once-a-month schedules all happen to arrive at about the same time, but it doesn’t get any worse than that. But I’m not too far away from my listening capacity; and, once I edge up to that capacity, my response time will go through the roof. I’m at that state on my magazine subscriptions, and it isn’t pretty – who knows when I’ll get around to reading those saved up NYRSF issues? And bye bye Granta subscription. So I don’t want that to happen with podcasts, and for all I know the next podcast could turn out to be one too many.

To be completely honest, that wouldn’t be the end of the world – I could occasionally delete an episode without listening to it, if my queue got bad. But that’s just not the way my psychology works: I’m a completist at heart. Also, I like the podcasts I’m listening to now, and I don’t want to delete any episodes of them. And they’re a nice mix: a little more than half music, of various sorts, but also several interesting non-music ones. At the very least, now that I’ve gotten my queue well under control, it’s time to re-evaluate the situation, and see if adding more podcasts is the best thing to do with the gaps that are opening up in my listening schedule.

And I’m pretty sure it’s not. If we think of this in terms of competing queues, then I’ve gotten the highest-priority queue in this area under control. But doing so makes me aware of two other queues that I can now consider dealing with in the same area. Namely:

• Listening to music that isn’t from podcasts. The Naxos classical music podcast is an incredibly effective advertising tool: something like a third of the episodes make me want to go out and buy the album in question. I’ve gotten other interesting music suggestions from other podcasts, too, and I wouldn’t mind going back and listening to some of my CD collection again.
• JapanesePod101. You see, I lied above when I said that I’d caught up on all of my podcasts: I’ve caught up on all but one, but on that one I’m 9 months behind. Which wouldn’t be too bad if they published once a month, but since they publish every day, I have my work cut out for me. I am consistently managing to not fall further behind on it now, but I would like to eat into the backlog. And it’s a really good podcast, so worth spending some effort on. I am a bit worried about burning out, but I have enough experience with (effectively) force-feeding myself knowledge in the past that I think I’ll be able to catch the warning signs before things get too bad.

So: one queue under control, two other queues revealed. All good fun, no? In fact, I think I’ll attack one of them right now by ordering a CD to listen to. (Update: no, I’ll break my lean vow and order two CDs. But one’s an EP, and they’re not available from Amazon, so it’s a bit easier for me to batch the order.)

If only queues at work were getting under control so well. Actually, several are, but there are two that are bedeviling me right now. I hope that we’ll make some progress on those during January…

Now, some of this is a classification issue: maybe a bug was incorrectly marked as high priority, and there are a lot of bugs open that shouldn’t be there in the first place. But a lot of the bugs that are open really do need to get fixed, and, as the product gets deployed more, there will be times in the future when we’ll run into bugs that really need to get fixed quickly. We’re capable of doing that now (we showed that during a recent trial, for example), but, even so, shouldn’t we spend more time practicing responding quickly, to make sure those skills don’t atrophy?

So, I think, we should rethink our bug prioritization system: we should make sure that high priority really means high priority (i.e. somebody starts work on it immediately), and we should also make sure that there’s a meaningful definition of medium priority (maybe it doesn’t get fixed this week, but it should get fixed this month). That would be a good first start.

Once we’ve gotten that under control, though, and are disciplined enough that high priority bugs are a rare event, quickly solved, we should try to react quickly to a larger class of bugs. After all, from a value stream perspective, the time spent waiting before we start fixing a bug is pure waste. If we’re not sure whether or not it’s valuable to fix a bug, then fine: we should wait until we have more clarity. But if we are going to fix a bug, what are we gaining by waiting? Why not just fix it immediately? We’re not saving work overall by delaying the fix: all we’re doing by waiting is building more debt that we’ll either have to pay off before the next release (if we fix it in time) or after the next release (if it makes it out into the wild). Neither of those is productive in general.

Of course, there are more levels to this problem: in particular, we shouldn’t be inserting defects into our code in the first place. (We’re getting better at that, fortunately.) And we don’t want to use Bugzilla as a substitute for our product backlog: there should be some control over how features get scheduled for implementation. And it can be hard to maintain a steady implementation pace if you’re getting constantly interrupted by bug work. There are solutions to all these problems, however. (E.g. for the latter, a two-part strategy of not writing defects in the first place and allocating slack in your schedule in the second place.) For now, from my point of view, our most urgent issues are (first) reducing the defect backlog and (second) improving response time.

Fortunately, other people agree. Some of my team members have been nagging me for months (years?) to make sure we don’t work on features at the expense of bug fixing, and my boss is more concerned right now with making sure we don’t have problems in the wild than with getting more features added to the product. And metrics have improved over the last week: if we really devote effort to fixing bugs, they do go away. But we have needed to devote the effort: maintaining a constant low-level hum wasn’t good enough, we needed a medium- to high-level hum.

The real test will be once we’ve worked bugs down to an acceptable level. We’ve built up technical debt; once we’ve paid that off, will we switch to a high level of productivity without building up new debt, or will we backslide? I’m optimistic that we’ll do the former: we’re getting pretty sensitized to bugs, and we’re aware of the problems that bugs cause to our normal development activities. On a basic level, they mean we have to waste time each morning investigating red bars on acceptance tests; on a slightly less basic level, the presence of those nondeterministic red bars means that, if you’ve implemented a new feature, it’s hard to be confident that you haven’t made mistakes, because you don’t have a completely clear good/bad signal from the tests.

And once we get bugs down to an acceptable level (zero, say), we can try to still leave some (not all, just some) of the former bug-fixing time in our schedule as slack. Now that I’m convinced (or at least strongly suspect) that slack is a good idea, I want to give it a try, but not without some external signal to let us know when we should stop slacking off! And the presence of bugs sounds like a great external signal to me.

Fun stuff, all this.