In contrast to the relatively plain and straightforward Zen literature of the Tang dynasty, Song dynasty Zen literature is convoluted and artful. This is not regarded, in Zen terms, as a development in Zen, but as a response to a more complex and pressured society and individual. The Zen adepts of Song times did not regard the reality of Zen as any different in its essence from that of classical times, but considered the function of Zen to have become complicated by the complexity of the contemporary mind and the rampant spread of artificial Zen based on imitations of a few Zen practices.

The proliferation of false Zen was stimulated by the enormous impact of real Zen on Asian civilization. After the Tang dynasty, there is hardly anywhere one can turn in Chinese culture without seeing the influence of the Zen charisma.

The ill effects of the resulting influx of insincere followers into public Zen institutions are already noted in the works of great masters of the latter Tang dynasty, and these Zen Lessons contain top-level notices of an even greater decline in quality of Zen institutions and followers in the Song dynasty, in spite of Zen’s unparalleled prestige in cultural terms.

There is even reason to believe that the creation of new Confucian and Taoist schools using Zen methods was especially encouraged by Zen adepts because of their awareness that the original Zen Buddhist order had become seriously enervated through the attachment of worldly feelings to its forms and personalities.

From the point of Buddhist historiography, this sort of involvement is predictable: a period of true teaching is eventually obscured by imitations, and even these break down into remnants with time. The Mahāparinirvānasūtra, or “Scripture of the Great Decrease,” among the classical scriptures traditionally most studied by Zen adepts, outlines these phenomena very clearly.

The false ideas about Zen and Buddhism that scandals at Zen centers have both arisen from and in turn re-created in many minds within and without these centers are also predictable and have existed ever since “Zen” became consciously articulated. Almost the entire literature of Zen, in all of its astonishing variety of forms, deals with nothing but misconceptions about the reality of Zen, which is said to be extremely simple in essence though complex in function or manifestation. The apparent complexity of Zen teaching and function is due to the complexity of the human mentality, as Zen perforce acted in more intricate ways to unify the threads of the contemporary mind.

Replace Zen with your favorite learning that you feel is widely misinterpreted; I’ll go with ‘agile’.

But I also have an opposing desire, one for following agreed upon procedures. I’m actually not sure where in my psyche this comes from—maybe the same part of me that wanted to be a good student and do well in contests? Tracing back through my post-academia history, though, it’s possible that it first came when I became fascinated by the rules of TDD and by the specifics of refactorings, and more broadly by XP’s prescriptiveness. It’s certainly explicit in the agile tradition more broadly in the notion of a retrospective, and lean puts it front and center.

The result is that my anti-hierarchical bent comes out in a desire for consensus as much as in a desire for autonomy. And consensus in a specific way: explicit consensus as a foundation for experimentation, so we can all get on the same page for something to do now, see how it works, and then accept/change/reject it.

At least that’s the theory; I haven’t yet been particularly successful at accomplishing that. Which is fine: right now, I’m at the stage where my brain is reminding me that, yes, this is somewhat important to me, but that I need to brush off / improve my consensus-building skills. (And my communication skills more broadly.) In the short term, it’s leading to somewhat odd interactions, where I see something that I interpret as a plan that we’ve agreed on, and then I see signs that we’re not acting as if we agree on the plan. Right now, I’m behaving in a not particularly productive way when I see that, instead acting a little grumpier than I’d like; now that I’m aware of my own behavior, though, hopefully I’ll be able to stop that and redirect it elsewhere.

The first step to that end is communicating where I’m coming from; this blog post is an effort to get that straight in my own head! In particular, I think I should be more explicit that I’m not pushing plans of record in to have my point of view win, and in fact I’m not wedded to always having a plan of record at all. So I’m happy for us to agree to follow a plan in some area, but I’m also happy for us to be explicit that we don’t yet have a plan of record in that area, or for us to agree that we had one but we’re modifying it, or that we’re following it in general but making an exception in this instance, or that we had one but are rejecting it. Much of the time, all of those are fine outcomes for me, I’d just like to know which one we think we’re doing in a given context!

Also, the other thing that I should make clear is that I’m not interested in plans as rules, I’m interested in plans as experiments. (I think Taiichi Ohno once said, when confronted with a standardized work document that was old enough that the paper started changing color, that any group who hadn’t changed their standardized work over the last thirty days was stealing from the company.)

With that understanding in place, the next question is: to the extent that I do want to poke at this issue, how to poke? And, in particular, how do I want to position myself in that effort? For example, one relevant role that I enjoy is acting the part of the fool: adopting an explicit pose of lack of power, noting that I’m listening to multiple people in positions of greater power and that I’m hearing them say different things on a topic, and naively wondering which one of them I should follow.

That role of fool works well to tease apart latent disagreements between others in more power; it doesn’t work so well to set up plans in areas where we don’t have pre-existing discussions, and probably doesn’t work as well when dealing with disagreements between people who are my peers. In fact, working with peers may, in an odd way, actually make building a consensus harder rather than easier: the existence of hierarchy gives planning discussions a place to start, and you can use hierarchy as a lever no matter which end you’re on. Maybe the real lesson there is that there’s no point in worrying about plans if you don’t have some sort of lever that you can push or pull; hierarchy is one such lever, but it’s not the only one, so for example the existence of an agreed-upon problem is another lever that you can work with.

And, stepping back: the general goal is to promote an experimental mindset. Having me pushing towards coming up with plans in areas that I care about is one tactic to that end; but if I can help other people push towards plans in areas that they happen to care about more than I do, then that will be even more effective. I’ve seen some interactions recently where somebody has identified a real problem and proposed a solution which hasn’t gotten traction; I should be more alert to those situations and see if I can help people in such situations drive the team towards some sort of consensus, by stepping back a bit from the details and focusing on what we agree on (the general problem space), what we don’t agree on (the details of a solution), and figuring out what’s going on with that disagreement and where to talk next.

Which is something that I thought about a fair amount when I was at Sun, and not so much since then. Probably time to revisit the relevant bits of the lean literature: in particular, I should take another look at the A3 report book. And I’m also tempted to use this as an excuse to revisit the Theory of Constraints thinking processes.

Though I’m not completely sure that the latter are directly relevant for this: I can’t remember how much they’re directed at groups coming to consensus versus individuals finding unexpected solutions to problems? Probably a bit of both; and to the extent that they’re focused on the latter, I should be able to get some personal benefit out of them right now even if they won’t suggest strategies for dealing with this specific issue.

I’m trying to make these descriptions neutral, to treat “agile” as a label rather than a value judgment. I won’t claim to be successful in that, because the truth is that I prefer the agile approach in everything that I list here, and I’d be surprised if that isn’t coloring the list at some level. But it’s also true that I’ve happily worked on projects where any one of these is missing. So this response is, I hope, mostly coming from the mathematician side of my brain instead of the pro-agile side: definitions are important, and it’s more important to use a definition correctly than to use a word that you like to apply to a situation that you like.

With that aside, here are some attributes that I see agile processes as having:

• Iterative and incremental development. You release software no less frequently than once a month or so; more frequent than that is better, with releasing multiple times a day not being a crazy idea.
• Business / Engineering split. The business side decides what features are the highest priority and what it means for them to be done; the engineering side decides how many of those top priority features they can do in any interval of time. There’s a preference for stack ranking instead of priority buckets, and for the stack to be as small as possible.
• Team focus. Decisions (e.g. about team processes, about estimates) are made by the team instead of individuals. Code is owned by the team instead of individuals.
• Focus on quality. The team tries to write good code, code that won’t be a burden on them in the future. They have a fairly stringent definition of what it means for a task to be done. They care about testing and refactoring.

I’m sure there are times in my life when I would have come up with a much more philosophical approach: you can map the above to the manifesto if you squint hard enough, but the manifesto is much more abstract. (And I’m not sure what to make about the fact that, if you do that mapping, my “Team focus” maps to “Individuals and interactions over processes and tools”: “team” and “individuals” are rather different words!) I think what’s going on there is that recently I’ve been reacting to descriptions of how teams work rather than reacting to philosophical treatises: so when people say “agile” in one sentence and talk about individual ownership in another sentence, my brain does a sort of pattern matching, hits upon a mismatch, and responds “why are you using those words together”?

And, of course, I could go more concrete as well: test-driven development, pair programming, etc. It’s possible that the main reason why I’m not doing that is that I don’t actually like pair programming (though I love TDD!); I hope that’s not the reason why I haven’t picked that level of specificity, though. Certainly I don’t want a definition of agile that nothing but eXtreme Programming can match; I guess, though, that I am happy enough with a definition of agile that Scrum alone doesn’t do a particularly good job of matching.

(Hmm, how does Scrum compare against this list? Certainly Scrum has the first item on the list; it should have the second, but my guess is that teams labeling their activities as Scrum are less likely to be good about the second item, though that’s probably a sign that they’re not doing Scrum well. And still less likely to be good about the third, and my vague memory is that Scrum is essentially silent about the fourth item. It’s been way too long since I’ve read the Scrum primary literature, though, so I could easily be wrong. Which makes me wonder if the reverse ever happens: what would a process be like that cared a lot about the fourth item but less and less as you went up the list?)

First: bug trackers are information absorbers. Because, while bug trackers sound like they could be useful as information radiators, encouraging people to put their thoughts and knowledge on a topic in some place publicly accessible, in practice they seem to me to have the opposite effect. In most of my recent jobs, I’ve been part of a team that’s working in close proximity with ample opportunity to discuss topics face-to-face; but we’re too introverted to do that. That’s okay, we have mailing lists and chat rooms where we could talk about problems we’ve noticed and brainstorm solutions; but we don’t do that either! Instead, we file bugs, bugs that almost nobody else notices.

Of course, it would be possible for the bug tracker to send updates to everybody, turning them into mailing lists with some useful auxiliary tools. But they don’t do that by default, and defaults matter. And, in fact, JIRA doesn’t (or at least didn’t the last time I checked) provide a mechanism where people can subscribe to all bugs by themselves, doing that requires administrative intervention. (Of course, JIRA’s information absorption possibilities are exceptional: I have no idea what JIRA’s authors were thinking when they designed its search box.)

Second: bug trackers encourage individual ownership. I’m sure it’s possible to set up a bug tracker without adding multiple categories, but I’ve never seen that done in practice: instead, the administrator picks categories corresponding to components of the system, and picks owners for those components. So, when somebody files a bug, that person picks a category that seems relevant, and the bug gets assigned to that category’s owner. (And, because of the information absorption properties mentioned above, quite possibly nobody other than the person who filed the bug and the component owner will ever learn about it!)

Third: bug trackers work against prioritization. Which is ironic, given how bug trackers love priority-related fields: priority, severity, target release, etc. But, in the agile world, prioritization means that the business side picks a very small number of tasks that are most important to work on right now, the development team works actively on those tasks, and the development team sets all other stories aside. And I’ve never seen prioritization systems in bug trackers that map to this workflow, or indeed that don’t actively work against this workflow.

Fourth: bug trackers work against clean code. Because bug trackers make it feel like you’re doing something useful if you see a problem with code (if you add a problem to code!) and report it. But you haven’t actually helped the code base by doing that; on the contrary, you’re giving yourself permission to leave the code base a little worse than you found it instead of a little better than you found it.

I won’t say that it’s impossible to get bug trackers to work in an agile fashion. And, in fact, the above points at what you should do if you want to pursue that course of action: have everybody receive updates for all bugs, either ignore the owner field or have the owner field always default to somebody on the business side, have as few priorities as possible (probably just two, namely “right now” and “later”), and keep an eagle eye out for clean code. But: defaults matter, the very existence of knobs and bells and whistles matter, and they all point in a rather different direction.

Despite all of that, there’s something that I like about bug trackers. It’s not something from agile, it’s something from GTD, and it’s something that I currently see as a point of tension between those philosophies. Namely:

Fifth: bug trackers encourage inventory. Bugs pile up, with every bug tracker I’ve worked with containing thousands of unresolved bugs in a shockingly short amount of time. And that inventory has all the problems that agile and lean teach us to expect.

Except: GTD teaches us that, if something comes across your mind, write it down. That certainly applies to problems in your code; and, if you’re going to write problems down, you should write them down somewhere that’s publicly accessible and where your notes won’t disappear. So private notes, mailing lists, and chat rooms actually aren’t a great place for this sort of thing: you instead want an area that has at least some of the attributes of bug trackers.

So how do we do this in a way that acknowledges the problems of inventory? GTD has one suggestion, namely the weekly review. So somebody needs to look at all open bugs every week.

Which is completely unworkable with the volumes of open bugs that I’ve seen on projects in the past. This is useful backpressure: if you have too many bugs to review them on a weekly basis, then stop writing so many bugs! Keep your code clean instead. Or, for “bugs” that are nice ideas but just end up not being pressing for week after week after week: just close them once your brain has come to terms with the fact that they’re not something that you think will ever be a good idea unless the context changes significantly. (But don’t close them until your brain has come to terms with that: GTD is all about listening to what your brain has come to terms with.)

Despite that last backtracing, I’m still quite dubious about bug trackers: I’d much rather focus on broad conversations, physical prioritization tools, and clean communal code, and see where that leads us. But I hope it leads us in a direction that isn’t averse to GTD’s insistence on trusted repositories outside of your head.

It took me quite some time to get around to writing the post, however; and, when I started typing, I realized why. To dig into Apple’s place in my psyche, I had to explain my history with Apple products, and indeed with computers in general. And, as it turns out, that takes a while. The result is a post where the tail is rather wagging the dog; interesting to me, at least, but one that could most charitably be described as ungainly. (Feel free to skip ahead to the Apple bits.)

At any rate: the computers I have owned, and why I am fascinated with Apple.

Prehistory

My parents bought us an Apple ][+ in May 1982; I was in fifth grade at the time. That was the only computer we had at home through at least 1989, when I went off to college (my brother got a computer when he went to college a few years earlier); hard to imagine these days. I'm not sure when my parents got a second computer, and I know they continued using the Apple ][+ for several years after I left home, at the very least to run a program they wrote to help manage their finances.

I programmed some on that Apple ][+ (the high point being a text adventure that I wrote), but my memory is that I didn't program particularly seriously on it. I used it to write papers (and for some other writing projects, I went through a phase when I wrote short stories and a novella). And I played quite a few games on it, high points being various Infocom games and the first four Ultima games, but I also think fondly of Robot Odyssey, Le Prisonnier, Lode Runner, and Wizardry.

In 1987 (my junior year of high school) I started hanging out more at Oberlin College, and I spent quite a bit of time in the various computer clusters in the school library. I got to be a rather fluent VAX/VMS user, and (presumably through some of the math courses I was taking?) started hanging out with some computer science majors. They got me interested in learning to program in C and Scheme, and in the 1988–1989 school year I started using Unix more. I also remember helping one of them install GNU Emacs on that VMS cluster. (At the time, the computer science’s Unix cluster actually had Gosmacs installed instead of (or at least in preference to?) GNU Emacs.)

Oberlin College could send e-mail to other institutions via Bitnet, and had a DECnet connection with a half-dozen or so other colleges. (DECnet was pretty cool.) It also had Usenet feeds. It was not yet on any of the TCP/IP-based networks that became the internet.

College

When I went off to college in the fall of 1989, my parents brought me a Macintosh SE/30; I used it to write papers in non-technical subjects, play games, and do some amount of programming. (I wrote my papers on technical subjects in LaTeX; I’m honestly not sure whether I mostly typed those on my Mac or on one of the clusters mentioned below.) Continuing my habits from the last two years of high school, however, I spent much much more time on the various computer clusters around the college. I begged an account on the math department’s Sun workstation cluster, though the sysadmin and I had an iffy enough relationship that I didn’t spend very much time there. I begged an account on the computer science department’s Sun workstation cluster as well, where I spent more time. (There were probably Ultrix machines in that cluster, too?) And I got a part time sysadmin helper job on the general school cluster. (Mostly Ultrix machines, initially with dumb terminals but X terminals showed up fairly soon.)

I probably spent most of my time on the general school cluster: programming, playing around, and doing system administration work. Coming out of that, I was much more comfortable on Unix than in any other computing environment, and had installed various bits of free software (mostly GNU tools of various sorts) over and over again. I also had a friend from Oberlin who was then working at the Free Software Foundation, so I was getting a strong free software philosophical dose from him as well.

I took a couple of computer science courses (an intro theory course, a compilers course), but not many: mostly because I could learn how to program computers just fine on my own, partly because I had enough other interests competing for my course time. Also, at that time Harvard’s computer science department didn’t have the buzz that I’d gotten from Oberlin. (Though there were students and faculty members that I learned a lot from, don’t get me wrong.) I was into programming languages and compilers at the time: I did some sort of undergrad research project on compilers, I was a course assistant for a few courses on programming languages and compilers, and I spent three out of my four summers during that period doing programming-related work. (One summer at MITRE, one at DEC, one being a course assistant at Boston College; the fourth summer was spent at a math research program whose main benefit was that I became a not-hopelessly-incompetent cook.)

During this period, I had access to TCP/IP-based networks: ARPAnet had evolved into NSFnet, with the internet coming. The web poked its head out right at the end of this period, but it certainly wasn’t clear to me that it was anything more than a peer to the various other network protocol that were floating around at the time.

Life as a Mathematician

Then, after a year’s interlude, I went to math grad school in 1994. I still had my old Mac, Jordan bought a new Mac (that I played Marathon on), Liesl bought a 486 machine running Windows 3.1 (I played Myst, System Shock, and Dark Forces on that), and at some point I was given an X terminal that I could use at home. Most of my computer time was spent on the math department machines, though; and I essentially wasn’t programming at all during this time period. Also, a friend of mine gave me an NES, which started me on a spiral of depravity that I still haven’t emerged from. (One of the first things I did after getting my postdoc acceptance letter was to get a Nintendo 64; good thing my thesis was almost completely written by then…) Actually, though, my dominant leisure activity during that time period was reading books, I averaged more than a book a day over the course of grad school.

I can’t remember if I moved my old (9 years old at the time!) Mac with me when we went to Stanford in 1998; we moved Liesl’s computer, but I’m not sure if we ever turned it on. In general, I did my computing on the machine in my office at the math department; I can’t remember its specs (though I believe it had 4 GB of hard drive space?), but it was running an early Red Hat Linux version. I still wasn’t programming significant amounts: I was busy being a mathematician and a parent (Miranda was born in 1999), trying to figure out how to teach well, and playing video games, doing the latter almost exclusively on consoles instead of computers.

Returning to the Apple theme that triggered this post: during this period, my interest in Apple was quite low. I had a Mac, but barely used it; I certainly wasn’t going to use Windows machines, but really my focus was on Unix. (So, in terms of recent computing deaths, Dennis Ritchie’s is a lot more relevant.) I was at least partly anti-Apple at the time: the Free Software Foundation and the League for Programming Freedom had boycotted Apple because of their use of user interface patents, and that had an effect on me.

Transitioning

In 2002, academia and I came to a mutual decision that we weren’t as good a fit as I had thought. Fortunately, the Stanford math department was willing to let me hang around for another year; so I spent half my time that year teaching calculus and half my time brushing up my programming skills. I learned C++ and Java (object-oriented programming was far from dominant when I was an undergraduate), and contributed a fair number of patches to GDB.

It also became clear that I wouldn’t be able to depend on my employer to provide my computing resources; so I bought domains to use for my various internet presences, and, for the first time since 1989 (13 years!), acquired a new computer. It was a Dell Inspiron 8200 laptop, a behemoth that was barely portable (and that, fortunately, I rarely needed to carry anywhere); we set it up to dual-boot Windows and Linux, and I spent the vast majority of the time on the Linux side.

Also, befitting my academic nature, I started reading books and going to talks. A lot of the books that I read were C++-specific (and I learned a lot from them, C++ is an extremely interesting language); in terms of non-language-specific books, the refactoring book had a big impact. The talk that had the most impact on me was one that a couple of researchers in a local corporate think-tank (?) gave about their experiences with something called “eXtreme Programming”; that was my first exposure to Agile software development.

The GDB work led to consulting work at a startup called Kealia, and I started working there full-time when I left academia in the summer of 2003. We got acquired by Sun a year later; soon after the acquisition, I became a manager, albeit a manager who spent a lot of time programming.

Agile

I spent a lot of time trying to understand Agile software development over the next five or seven years. At first, I was just trying to do this on a personal level, practicing refactoring and trying out test-driven development. Kealia’s legacy code provided some interesting challenges on the former front; the company also already had a bit of a testing culture when I showed up, and we experimented with going farther in that direction. And becoming a manager got me interested in other aspects of Agile: the more explicitly people-focused aspects, the planning aspects. And, as part of planning, the idea that programmers don’t make all of the design decisions (which was quite a change from working on GDB!): other people have a better idea of what the end users really value, what will work well in their context.

As an academic, I’d been quite ivory tower (at least aside from my interest in teaching); that changed. I was working at a startup which got acquired by a larger company that had suffered a lot over the last few years; part of startup life is trying to figure out how to make your business work, and Sun was trying to figure that out at a larger scale. Sun also put enough resources behind StreamStar (Kealia’s video server project) that we had quite a lot of room to experiment with different business strategies, trying to find one that would stick. (Far too much room: the fact that Sun didn’t cancel StreamStar years before I eventually left was a sign of Sun’s own management problems.)

My boss was a big fan of Clayton Christensen’s disruption theories, and I got to see both sides of the difficulties of disruption first-hand. Sun was a large company that was already far along the path of being disrupted by commodity hardware running Linux, and was trying to figure out how to deal with that; StreamStar was trying to disrupt the existing broadcast television infrastructure, replacing it with IP-based solutions. In neither case did we navigate the difficulties well, but I have quite a bit of sympathy for both sets of difficulties: surviving being disrupted is extremely difficult, and when it comes to broadcast television, you have to deal not only with the existing technological infrastructure but with the existing broadcasters and existing content providers. So it’s not surprising that we failed to disrupt broadcast television delivery, whereas Youtube was much more successful with its end run around the last two issues.

During this time, I won an iPod (one of the hard-drive based models), and a couple of years later, an iPod Nano at company raffles. I wouldn’t have bought the first iPod on my own, but its presence made my jogging a lot more presence; I probably wouldn’t have bought the iPod Nano on my own, but I was quite surprised how much more I liked its small size, the lack of skipping, and the general elegance of its design.

Our Dell laptop died in 2006, and had been showing its age enough by then that I was already planning to replace it. For my own Linux use, we got a Sun Ultra 20; to have a computer that Liesl could use and that I could run iTunes on, I got a MacBook Pro. This was the first model after the Intel transition; I felt more comfortable going back to the Mac instead of having a Windows machine around, and the fact that there was now Unix underneath MacOS was a real bonus. (Incidentally, back in 2003 I’d turned down a job offer working on GDB for Apple: I like Unix and the GNU toolchain, but I wasn’t really interested in specializing in the latter.)

At some point while I was at Sun (probably in 2008), I got an iPod Touch. That was really a revelation to me: it was wonderful having a little computer in my pocket, one that was already fairly versatile and was becoming more so every year; I had Wi-Fi access most of the places I spent time (there was even spotty Wi-Fi available from Google when wandering around Mountain View), but I could tell that having a phone network provide almost constant network access would be so much better.

But more than that: Tweetie made me sit up and take notice. That was the Twitter client that eventually became the first-party Twitter client; and despite running on this quite small device, I far preferred using it to any Twitter interface I had available on computers that didn’t fit in my pocket. That didn’t make much sense to me; clearly there was something going on with design that I didn’t understand and that could make a real difference.

At this time, I was also getting more and more tired with having Unix on my desktop. I love Emacs, but it’s stuck in the stone age in so many ways: what really drove that home was once when I fired it up on a machine where I didn’t have my standard .emacs file and realized that, by default, Emacs put the scroll bars on the left. That may have been a perfectly reasonable decision when it was first made, but it wasn’t any more and hadn’t been for at least a decade; did I really want to be working with tools that were so willfully ignorant about design conventions? GNOME had helped civilize X Windows, but it had only brought the experience up to a minimally acceptable level, and even so there were too many non-GNOME applications around.

Reaching the Present

So, when I started work at Playdom, I asked for a Mac for my work machine: that way I could have a Unix command line and tools combined with a GUI that accepted the idea that design was a virtue. Which the IT department was oddly hostile to: you’d think that a company with a large contingent of graphics artists that deploys software to Unix servers would be a natural fit for Macs, but Playdom had its quirks, and its IT department was definitely one of those quirks.

At around this time we got a second Mac laptop at home, and I got an iPhone. (My first cell phone; I am a luddite at times.) The Ultra 20 died; I decided that I wanted to continue to run a Linux server (e.g. to host this blog), but that I would prefer to interact with it through an ssh connection, so I got a virtual machine at Rackspace. Also, I was getting older, and carrying around a laptop during GDC 2010 put a surprising strain on my back; the iPad had been announced, so I decided I’d get one the next time I went to a conference. Which happened sooner than I expected, since I decided to go to GLS later that spring.

My back thanked me for the iPad purchase; but my psyche thanked me as well, to a surprising extent. I found that I preferred reading e-mail on the iPad to reading e-mail in a web browser, and that I far far preferred reading blogs in Reeder than through Google Reader’s web interface, whether I used the latter to go to the blogs’ web pages or stuck with the RSS feed. In both cases, the iPad acted like a wonderfully adaptable piece of paper: the words I wanted were right there, with enough style to be pleasant (unlike the Google Reader web interface) but without any surrounding crap (unlike blogs’ web pages). Having a screen that was much smaller than computer monitors that I was used to, and that was in portrait mode instead of landscape mode, turned out to be excellent for letting me focus on what I was reading. (As it turned out, I even slightly prefer reading blogs through Reeder on my iPhone over reading them through a web interface on a standard computer, despite the rather-too-small size of the former’s screen.)

In early 2011, one of our laptops died; rather than replace it with another laptop, we got an iMac and a second iPad. Our current technology roster is an iMac and a MacBook (one of the white plastic ones); two iPads (one from each generation); three iPhones (one from each of the last three generations, though the oldest one is being used by Miranda as an iPod Touch instead of as a phone); a virtual machine located elsewhere running Linux; and half a dozen game consoles. (My rate of technology purchases has increased enormously since 1998.) Also in 2011, I started working at Sumo Logic; as is typical in startups around here (at least judging from the ones I’ve interviewed at), it’s largely a Mac shop for development (with deployment happening on Linux virtual machines), and my coworkers generally prefer various Apple products for personal use, though there’s more variation on the personal side.

So: that’s the computers and other technology that I’ve used over the course of my life. Apple played a large role when I was young and more recently, but in the middle there was a long phase where my norm was Unix + GNU toolchain, with a strong free software ethos. Why did I shift out of that, what’s behind my recent fascination with Apple’s products and, increasingly, Apple as a company?

Habitable Software

The first is the concept of “habitable software”. I talked about this last year: the idea is that there is software that my brain shies away from using, and there’s software that I actively look forward to using, where the thought of using it relaxes me or brings a smile to my face.

I actually think that console gaming gave me my first nudge in this direction. You stick the cartridge into the machine, you pick up a controller with a relatively constrained set of inputs, you turn on the machine, and it just works. Note too that a console controller, unlike a mouse and a keyboard, is explicitly designed for the task at hand: yes, gamepads may have a few too many or too few buttons and sticks for a given game, but at least it’s focused on the domain of playing games. (Hmm, maybe the controller/game match is why I think back on text adventures with so much fondness?) I keep on installing Windows on machines with the thought that I’ll finally play the many important PC games that are missing from my background; and I keep on deciding that no, I really don’t want to put up with the crap that PC gaming makes you deal with.

But shifting from X Windows back to the Mac also gave me a huge shove towards being sensitive to habitable software; and going from the Mac to iPhone/iPad software like Tweetie and Reeder was, in its own way, just as large a leap. Every time I use X, I find something that feels wrong; a Mac feels neutral, but I don’t generally look forward to turning it on; Tweetie and Reeder make me actively happy. It’s not just software that I’m learning from, either: I was surprised how much happier I was with the iPod Nano because of its small size, light weight, pleasant screen, and lack of skipping.

The Unix command line also makes me actively happy. It’s wonderfully coherent; for certain tasks related to writing and, especially, deploying software, it’s just what I want, I love the interface that it presents to me. So it’s no coincidence that I do my programming on machines where a Unix terminal window is one key combination away, and that I use virtual machines running Linux to deploy software on: I feel completely at home in those contexts when working on those tasks.

Designing Software

Habitability is how I like to express the importance of design in software to me as a user. But I’m a programmer as well, so I see design from that side as well.

When I was younger, I spent much of my programming time concerned with tools for programmers: thinking about programming languages and compilers, working on GDB. In those contexts, I didn’t have to think too hard about design: I was an acceptable proxy for the end user for the software, so if something felt good for me, then that was good enough.

That’s a relatively unusual subset of software, however; as I started to work about other kinds of products, I realized that my design instincts wouldn’t do a very good job. And, at the same time, I got interested in Agile: and one of Agile’s main tenets is that design concerns (personified as the “Customer”) are paramount when deciding what to work on. Not that the technical details aren’t important as well—you get great benefits from keeping your code flexible and well-architected—but ultimately it’s not programmers’ jobs to decide what’s important to present to the users.

Even though it carves out a space where design can happen, Agile isn’t actually very good at giving you advice at how to design well: specific recommendations are much more focused on the programming side of things (e.g. refactoring, test-driven development) or the programming/design interface (estimating, iterating) than on the design side of things. Also, my talents and instincts are much stronger on programming than on design: I still have a lot of room for improvement, but I’ve got some understanding of what’s involved in writing code that’s clean and functional from a technical point of view, whereas I have much less understanding of what’s involved in developing a product that people are actively happy to use.

And, to produce really great products, I’m not convinced by Agile’s engineering/customer representative split. The Lean concept of a Chief Engineer who’s immersed in both worlds seems much more powerful to me, and I see around me wonderful pieces of software written by single individuals, or startups (including Sumo Logic!) run by people with both a vision for what they want to produce and the technical chops to help bring that into existence.

Apple can probably be argued as providing evidence on either side of the argument about that split, but there are clearly individuals who made a huge difference in its products. Apple also points out how ludicrous it is to label the designer as the “Customer” if you really want to produce something new and great, and at the limits of the analytics-focused mindset that I saw so much of at Playdom; in general, Apple’s approach to iteration seems interestingly different from yet related to Agile norms. And their systems approach gives Apple many more design knobs to turn than they would if they were exclusively a software company. (Or exclusively a hardware company, of course.)

Business Success

Back in my academic days, I didn’t care about practical applications of my research. When I started working for startups, though, that changed: if you don’t have your eyes on how you’re going to make money out of your startup, you’re doing the wrong thing. (Not that startups don’t have a heavy dose of ego satisfaction in them, of scratching your own itch.)

Once I started paying more attention to making money, it turns out to be totally fascinating: if you like complex systems, capitalism is full of them. Just figuring out cash flow: where money is coming in, where money is going out, the difference between those two in quantity and in in time. So many possibilities there!

Apple’s business success over the last decade is staggering, of course. But they are fascinating far beyond their simple profit figures: the consequences of their systems approach to design, their use of their savings to buy vast quantities of parts from their component vendors (and even to allow those vendors to purchase tooling!), the role of their physical stores, the list goes on and on. There’s still a stereotype of Apple as making overpriced products, but their competitors are finding it very difficult to build products with the hardware quality of the iPad or MacBook Air while maintaining any sort of profit margin at all.

Of course, lots of startups aren’t focused on being profitable: Silicon Valley is full of company that are trying to get eyeballs, hoping that profitability will come somehow, and perfectly happy to sell the company to somebody else who can worry about that problem. We see echoes of this in the Android / iPhone fight, and these days I’m generally more interested in making money than having users without a good business model; but the iPod shows that you don’t always have to compromise, that you can win on both fronts.

Disruption

I mentioned Clayton Christensen’s disruption theory above: living in Silicon Valley, there’s no end of startups trying to remake an industry, no end of once dominant companies that stumbled, got bought, died.

Apple looked like it was following that latter trajectory; it pulled out of its decline like no other company. And did so in a very interesting way: not only did it disrupt other industries, it also disrupted itself, with the iPhone cannibalizing iPod sales and with the iPad cannibalizing laptop sales. This is extremely difficult to do: existing successes almost always lead to institutional antibodies that attack new products, leaving that success to newcomers.

Over the last decade, we’ve all become aware of disruption; the companies that can figure out how to repeatedly harness the powers of disruption will be the ones that flourish (the ones that survive at all!) over the next few decades. They will have to learn from Apple. And if I’m going to continue to build a career working at exciting companies, I’m going to want to learn from Apple, too, to help me figure out what sorts of qualities to look for the next time I’m on the job market, to pick employers that will disrupt successfully!

Repeatable Creativity

Disruption aside, though, there’s something amazing about Apple’s run of products over the last decade: one interestingly new product after another. I wish I knew how they did that.

It’s easy to ascribe this to a solo genius theory; but, while I don’t want to minimize Steve Jobs’s contributions, I don’t think that’s all that’s going on here. Pixar is another relevant datum: they’ve also managed to be consistently creative, and they continued to do that after Jobs sold the company to Disney. Perhaps because of the domain, people don’t credit Jobs with the same influence on Pixar’s repeated creative success as they do with Jobs; but, to me, the two companies suggest that Jobs has learned something about helping groups to innovate repeatably in a way that goes well beyond his personal contributions.

Over the last couple of years, stories have come out about some sort of Apple University, which seems to be trying to systematize those ideas. This reminds me of Toyota’s conscious efforts to improve themselves as a learning company; Apple is, sadly, much more secretive than Toyota, but I hope more of Apple’s methods will become public over the next decade. And, of course, I hope that Apple will be able to continue to innovate over the next decade, that their innovation really is due in part to a systematizable process.

Bad Apple

During the mid-90′s, I was down on Apple. I hoped that had gone away with the new decade, however: their user interface patents had gone away, and they were active open source contributors, though that clearly wasn’t the company’s main focus.

Unfortunately, those problems have come back in spades. By far the one that I find most distasteful is their aggressive use of patents: I think software patents are bad for the industry, bad for the world, and while I’m more and more bored by other companies that seem to largely be trying to produce knockoffs of Apple’s products, I very much support allowing those companies to do so.

Apple’s recent systems are also much more closed than computing platforms I’d used before then. I would expect that to bother me; for whatever reason, though, it actually doesn’t particularly. Certainly it would if I didn’t have ample access to other computing platforms, or if the tools to develop for iOS platforms weren’t so readily available; and while Apple teeters on the edge of behaving in a manner I find unacceptable in their application approval process, for whatever reason I generally think they’re okay. (I’m actually more worried about Amazon’s behavior in that regard.)

I’m being ungenerous in saying this, but: these days, when I read Richard Stallman complaining about Apple’s closed systems, part of my brain interprets that as RMS wanting it not to be his fault if other people don’t have software they want to use: RMS has made an open system, it’s other people’s fault if they don’t take advantage of that. These open systems are, in all serious, a great good: but actually having good software on your computer is also worthy, and having software that’s a joy to use is a great good. It’s fine if having well-crafted software for the non-programming public isn’t RMS’s concern, there’s no reason why it should be; but I see him as a single-issue voter whose issue is no longer dominant to me, and who is willfully blind to other issues that are important to me.

To those of you who have read this far: I salute you. And to those of you who don’t like Apple’s products, who don’t care about what Apple has done as a company: that’s great, there’s no reason why others’ interests should be my own. And there’s no question that company has flaws, does things I really don’t like. But I’m fascinated for many reasons by what Apple has done over the last decade, and I fully expect to be trying to sort out the implications for much of the next decade.

Some Jobs-related posts that I particularly enjoyed:

• John Shook asking Was Steve Lean?
• Another lean-focused post, this time from Evolving Excellence
• Horace Dediu on what Steve Jobs didn’t do.
• A podcast reminiscence from John Siracusa.

One of my favorite blogs of the last year or so (is it two years old now? Wow) is Asymco. It’s written by Horace Dediu, a former Nokia analyst who is very interested in disruption (in the Clayton Christensen sense) in the mobile space. If you read that blog, you’ll see posts week after week slicing and dicing the data in different, interesting ways, all leading to the same story: the incumbent phone makers’ share of the profits in the industry has plummeted by a shocking extent, their revenue share is also in a dramatic decline, and their unit sales aren’t nearly as hot as it once was. Here’s a sample post (albeit with a somewhat less intelligible graph than he normally has): Apple is capturing most of the profits in the mobile phone space, and more than half the revenue.

Of course, the iPhone isn’t the only story there, in two different ways. One is that the iPhone isn’t the only source of disruption in the phone market: Android is also helping eat traditional phone vendors’ lunch. And the other is that it’s not the only way in which Apple’s business success is remarkable: going back to traditional spaces, Apple is capturing a proportion of the profit in the traditional computer space that’s completely out of line with their proportion of sales, and the iPad has singlehandedly created a new tablet market, a market with no other serious contenders and one that is eating into traditional computers’ market shares.

Which is pretty amazing: a single instance of disruption like that would be enough to make people sit up and take notice, but two back to back is really something. And it certainly makes people wonder what’s going on there: in particular, how much is Apple’s integrated hardware/software approach making a difference? (There’s also their increasing design for manufacturing expertise and their sales channels; I won’t talk about them much other than to note that the old story of Apple products being too expensive is quite out of date, with Apple maintaining healthy profit margins on the iPad while other competitors are unable to undercut them on price while maintaining any sort of decent manufacturing quality.)

In the phone operating system space (not yet in the tablet space, though maybe that will change), Google is of course their main competitor. And it’s a bloody, bloody battle, especially around patents. Not just between Apple and Google—Microsoft is making a quite credible amount of money off of patent licensing agreements with Android device manufacturers—but between Apple and Google it’s particularly bloody, because Apple has no apparent desire to license their patents: they’re trying to prevent distribution of competitors’ devices entirely.

This is, of course, a lousy thing: it’s bad for consumers, it’s bad for programmers, it’s bad for innovation. It is, however, fascinating to watch, and here my favorite source of information is the FOSS Patents blog. Its author, Florian Mueller, does a wonderful job of keeping track of all the various lawsuits in this area, both digging up primary documents and providing well-informed commentary. (And it’s not just about the Apple-Google fight, e.g. he covers the Lodsys patent battles better than anybody, and Google is also under heavy heavy attack from Oracle.)

Patents seem to clearly be an area where Apple is much better prepared to work than Google is: Google, to me, seems to be giving off an aura of not, at its core, believing that anybody would take software patents seriously. I’ve been shocked at how vulnerable they seem to be to Oracle’s attack, and when your Chief Legal Officer is made to look stupid on Twitter by Microsoft’s general counsel, you really are not doing a very good job fighting your battles.

Both of these stories led to Google’s announcement a week and a half ago of their plan to purchase Motorola Mobility. The big question there is: how does this fit into the above narratives? Is it a patent play, is it an integration play, is it both, does Google even have a clear plan for the acquisition? (In general, acquisitions do badly for the acquiring company, so the default assumption should be that this one won’t turn out well…)

The early coverage assumed that this was all about the patents: that with Motorola’s patents, finally Google would be able to defend themselves against Apple. The problem with that is that Apple was already going on the offensive against Motorola: so why should we assume that those patents will succeed in defending Android against Apple more broadly?

But if it’s an integration play, then that directly attacks Google’s main selling point of being a level playing field that all vendors can pick up and use on equal terms. There were already chinks in that the idealistic version of that story—Google didn’t release the Honeycomb source code, and the Skyhook fight makes it clear that Google will use their muscle to shut out third-party replacement of their components. But if Google turns into a major Android phone manufacturer, that story will be in tatters: maybe Samsung and HTC will stay strong Google partners, but my guess is that they’ll become a lot happier about adopting Windows Phone 7 and/or that we’ll see long-lived Android forks that evolve away from interoperability with the trunk.

I hope somebody at Google has a good answer to this, because I want there to be as many strong competitors in the mobile operating system space as possible—I certainly don’t trust Apple to behave themselves without frequent pushback!—but at least at first the Motorola deal seemed very rushed, especially for a 12.5 billion dollar acquisition. (With a 2.5 billion dollar kill fee, an amount that is shockingly high and suggests that Google didn’t have the upper hand in negotiations at all.)

So that’s the first half of the week; the second half of the week, however, brought the news that HP was not only stopping WebOS development (which was one of the few other plausible contenders, and probably the most plausible integrated stack play), but they were exiting the consumer PC business entirely! And in their announcement, they admitted that competition from tablets was a big part of the latter decision: basically, they were saying that they don’t know how to build a tablet that’s competitive in price and quality with the iPad, and that they (the largest computer manufacturer in the world!) also don’t know how to keep their profit margins on non-tablet PCs high enough for it to be worthwhile to stay in that business.

This is a classic disruption story: a new entrant has come in and redefined the bottom of the market in such a way that incumbents don’t have a good response; those incumbents, in turn, flee upmarket, which will help in the short term but who knows for how long. (With the extremely unusual twist that the new entrant isn’t new at all.) I was surprised when IBM sold off their laptop business to Lenovo a few years ago, but it looks like a great move in retrospect, getting out of the business while they could still sell it for a reasonable value while reinventing themselves as a service business. HP’s move seems a lot more desperate, though, and neither their Palm nor (I assume) their Compaq acquisitions have turned out at all well. Which puts quite a spin on the ten billion dollar acquisition of Autonomy that they announced the very same day; maybe that acquisition will do better, but personally I would not bet on that. (It does seem, however, that selling printer ink for thousands of dollars a gallon is still a good business to be in.)

But it’s one thing to read about classic disruption stories; it’s another thing to see them play out right in front of you in real time, involving three huge companies each of which is located a few miles away from your house. Definitely time to bring out the popcorn.

And, if WebOS is no longer a contender, Android is unable to produce competitive tablets (and is in for somewhat stormier weather in general), and Microsoft is insisting that desktop OSes are suitable for tablets, where is the iPad competition going to come from? My best guess right now is Amazon, but the way they’re treating developers in their app store (making Apple look good!) has me a lot less excited about that prospect than I once was. I really hope somebody succeeds, though, the idea of a monoculture doesn’t make me happy at all.

And, as it turns out, the news in this area didn’t end with HP’s dramatic shift out of the PC industry, which was when I’d started thinking about this post: the next week brought the sad announcement that Steve Jobs was retiring from Apple for health reasons. What I am most curious about in that regard is to what extent Apple has managed to successfully formalize their thought patterns. They’ve apparently been trying to do that over the last few years; I’m a big Toyota fan, and one of the ways in which Toyota impresses me the most is the extent to which they’ve built up a long-lasting company culture.

If Apple has been self-aware enough to do the same thing, then maybe the iPad isn’t the last great disruption to come out of there; and if others can learn about their culture the way others have learned about Toyota’s, then maybe that can transform design for the industry as a whole. Though the Toyota analogy suggests an alternate outcome: even if outsiders do learn about their culture, the vast majority of outsiders are unlikely to be able to learn enough from those lessons to really make a difference…

• When I was thinking about getting an iPad, I wondered what format I should buy books in. I was thinking the contenders were Amazon’s proprietary format versus ePub books (sadly largely with encryption in both cases); but when I actually got the iPad, I discovered that it’s a really great PDF reader. (Don’t get me wrong, I’d love a retina screen on it, but it works quite well as is.) And, as it happened, some of the early books that I bought were from the Pragmatic Programmers, which lets you get books in PDF and ePub (and Amazon’s format, but I don’t have a Kindle yet, so no reason to choose that if I’m not buying from Amazon). And, for now, I’m liking PDF books a lot more than ePub. I just hope that the book industry doesn’t take as long as the music industry to start embracing non-encrypted formats, so I can get PDF books from other sources.
• Having said that, non-page-based formats do have their uses. A couple of weeks ago, I was reading Nicola Griffith’s Always on the Kindle app on my iPad. And then I found myself out of the house with some time to kill, so I pulled out my phone and switched over to reading the book on that. (I didn’t have my iPad with me.) And that worked great, much better than reading a PDF on my phone would have or sitting around being bored would have.
• Another unexpected electronic book benefit: our dog Zippy is getting rather old, and wakes me up squeaking a couple of times a night on average. (For better or for worse, I’m a much lighter sleeper than Liesl is.) Sometimes he needs to go out, but sometimes he’s achy and just needs cuddling for a while. And I like being able to read while cuddling with him without having to turn on a light.
• Speaking of Nicola Griffith, I’d forgotten just how amazing an author she is. Or rather, I’d been somewhat reminded of that when I read her memoir, and I like her blog as well, so I’d been meaning to dig back into her fiction, but I hadn’t gotten around to it until the last month. I don’t think I’d reread Ammonite since it came out, but it’s quite good; better still is Slow River, and rereading The Blue Place was eye-opening. I’d never read Stay or Always, but I’m quite happy to have remedied that omission.
• Speaking of omissions, I’d somehow stopped reading Madeline L’Engle’s Crosswicks Journal after the first two books. No idea why I stopped then; I went back and reread them just now, and they’re rather wonderful. Though so far I’m not enjoying the third one as much; maybe it will grow on me (it took a while for me to appreciate the first one, I seem to recall), or maybe it’s just more targeted at Christians?
• I’m very glad to have been reading a lot of fiction these days. I’d been weighting my reading rather heavily towards technical books over much of the last year; partly for good reasons, but partly because I’d been swayed by sales of electronic books at a couple of publishers. And while electronic books don’t raise exactly the same inventory concerns as physical books, they’re still inventory, and the fact that I own them still unduly influences me to read them. I’ll have to be more vigilant about that in the future.
• Sad that Borders is going out of business. I like independent bookstores, but to me it’s much much more important to have a large selection of books available for purchase, and Borders did a great job of that as a chain; I visited the local Borders about as frequently over the last few years as any other physical bookstore. Their time has passed, but I salute them and will miss them.

Which means that people sometimes get woken up in the middle of the night. Initially, it was the same couple of people, but that’s obviously completely unfair (and goes against the Whole Team philosophy that I prefer), so now we’re rotating that job among all the developers. I had the pleasure of being on call a couple of weeks ago, and it was a lot more interesting experience than I expected.

And yes, part of that interesting experience involved staying up later in a couple of evenings than I would have liked, and being woken up a couple of times. The problems there happened to involve a piece of code that I was relatively familiar with; in fact, during that week I’d already been experimenting in test deployments to see where and how it would fall over, I just didn’t manage to get far enough ahead of production scaling to give us quite enough breathing room! But at least it meant that I had some idea of what to look for and what to do, which certainly helped. (And don’t get me wrong, even when I was the primary person on call, a lot of other people were ready and eager to help as necessary, and somebody else implemented the initial stability fix.)

It turned out, though, that being the primary person on call when issues like that come up has an interesting affect on my brain. In the past, when people had talked to me about issues that had come up when they were on call, I was happy to help fix things, but my brain was treating the problem somewhat abstractly and as a distraction to what I’d been planning to do that day. Whereas running into problems when I’m on call rather quickly puts my brain into problem solving mode, with the result that when I showed up at work the next day I knew exactly what I wanted to be working on, what steps I wanted to take, and where I needed to advocate for further work. So the upshot was that I felt a lot more focused for the rest of the week (well, when I wasn’t feeling sleep-deprived!), and it carried over to the next couple of weeks: I can program pretty effectively when I’m trying to answer the question of “what can I do to prevent other people from being woken up for similar reasons?”.

I am, of course, hoping not to always have to keep that motivation in the front of my mind when I’m programming. (And fully expecting that to be the case: we’re doing a very good job of leaping on small problems now and trying to take a systemic approach to solving them instead of a patchwork approach.) But it’s a helpful reminder of the virtues of Whole Team: exposing everybody to problems means that everybody has a chance to really feel their effects and get their brains directly interested in solving those problems.

And, as a side effect, it means that non-bug work can go a lot faster, too: in particular, it’s been great this week to have much faster startup speed for the component involved in the aforementioned issue, because that made it so much easier to run experiments. Goodness all around.

A month or two ago, we unlocked a Formula 1-style car; it’s faster than any other car we’ve unlocked, it accelerates much faster than the others, and controls surprisingly well given those qualities. So we’ve all switched over to using it: at first, we crashed enough more while driving it that other cars would frequently beat it, but once we put in the time to get used to its handling, it was clearly the way to go.

Its behavior on straightaways (or shallow bends) didn’t require much adjustment. You have to be on your toes a little more, and your ability to avoid cross traffic is reduced, but in general you pick your path and follow it. On curves, though, the car felt rather different: I almost always went wide while turning in the new car, and on right turns in particular I’d end up in dangerous situations.

Slowing down would, of course, be an option, but it’s rarely the recommended option in Burnout: instead, skidding is the best way to deal with high-speed turns. So I decided to work on my skidding skills.

It’s a little odd working on a new technical skill in a game for a few minutes at a time over the course of months. It takes a while for the skill to become ingrained: in fact, I still have to consciously think when approaching curves about how I want to approach them. Otherwise, I would forget to skid at all, going wide instead; and when I started skidding, I would often hold down the skid button too long, with the result that I would turn too far, typically crashing into the near wall.

But when I’m alert, I have an approach that works pretty well. I start skidding, but release the skid button quicker than my instincts would have me do. I’m out of control at this point, so I keep on turning; eventually, though, I regain the ability to steer, and after a bit of fishtailing, I’ll generally end up pointed up where I want to go, maintaining my speed in the process.

I’m not used to intentionally losing control in a video game like this. I’m used to games where I don’t know what good strategy or good tactics is; and I’m used to games where I can’t parse the action quickly enough to respond well. This is different, though: I’m happy with my strategy, and if I’m alert then my ability to parse oncoming traffic is generally acceptable. I’m just consciously making a choice that involves stepping back from detailed control for a bit in the belief that, once I regain control, the range of likely outcomes is one that I’ll be happy with.

And it’s a tactic that cries out for analogization. Take software development, for example. Normally, when developing software, I try to maintain rather tight control of how it’s going: test driven development can be thought of as reifying my constant feeling of control over my software. (And letting me know as soon as possible when that feeling is ill-founded, so I can re-establish my control!) And, of course, when playing Burnout, I normally try to maintain control: if I’m going mostly straight, then I try to plot my course as tightly as possible, I don’t just remove my hands from the controls and let the car meander as it will.

But, in software development as in racing games, there are times when you want to make a sharp turn. And I suspect that, in software development as in racing games, if you try to maintain tight control while doing so, you’ll end up not turning enough, instead going wide and (at best) slowing yourself down or (at worst) being obliterated by oncoming traffic.

Which doesn’t mean that I should relinquish control willy-nilly, even when I’m at a turn. In Burnout, it’s not a good strategy to stay skidding, hoping that you’ll end up in the right place. Instead, I get the best outcomes if I start skidding and then release the skid button as soon as I’ve lost control. If I do that, I’ll start drifting across a range of possible angles to take the turn at; the early angles are too shallow, but my tires wouldn’t stick if I tried to accelerate then anyways. So I sit and watch my car turn more and more; by the time it ends up pointed in a useful direction, it’s willing to respond again (after a bit of fishtailing!) if I try to reassert control.

And the same happens with experiments in software development. If you need to make a sharp turn, then throw yourself into it, and accept that you’ll lose control while doing so. But that doesn’t mean that you have to give up control for a long period of time: instead, make a short, focused experiment, and get ready to zoom off in the correct direction (also with a bit of fishtailing!) once you’ve turned enough to be able to see something promising.

Software development has its straightaways, and it has its curves. Know when you’re in the one, know when you’re in the other, switch your strategy to acknowledge those two situations. But, in either case, leave yourself able to respond to feedback as quickly as possible.

Much of that is due to how new my work setup is: my Playdom Next Action list got to be significantly longer over my time there than my Sumo Logic one currently is. And part of it is the nature of the tasks: at work, I generally have one large project that I’m focusing on at any given time (which I break up into multiple tasks, of course), with only a few side tasks, while at home, there are a bunch of different areas that I want to be working on.

Still, I like the feel of my Next Action list at work much much more than the feel of my home list. And I’m clearly misusing my home Someday/Maybe list: the age of several of my Next Action items strongly suggests that I’m not treating them as next actions, that I’m in fact treating them as someday items, just someday items that I wish I could wave a magic wand at and have them be done. Also, I’m pretty sure that the rarity with which I move something from my Someday/Maybe list onto my Next Action list is another sign that I’m doing things wrong.

Another piece of the puzzle is Kanban. (In its software development form, as developed by David Anderson, not its manufacturing form.) I’ve been following the kanbandev mailing list for the last year or so and read the book earlier this year, and while I need to get more hands-on experience with the methodology (which I hope will happen at work soon), it makes enough sense to me that it’s my default way of thinking about organizing software production. And, viewed in a Kanban light, I’m clearly managing my personal tasks wrong: I don’t have a pretense of Work in Progress limits, and I have no control over my cycle time.

And it’s not like there aren’t new things that I’d like to do but that haven’t made it onto my Next Action list, either: in fact, right now, my brain seems to be particularly good at thinking of new programming projects to undertake! But there’s no point in doing a half-hearted stab at a bunch of projects: that won’t make me feel any better.

Which means I need to get things under control. Part of that means looking at my Next Action list, and moving some of the stuff there to Someday/Maybe. And part of that means recognizing that a lot of the stuff there is things that is genuinely important to do but that I don’t enjoy doing, and I just have to suck it up and do it. For example, I haven’t done a weeding of financial records for more than a two years; the drawers are getting full, that stuff isn’t going to go away if I don’t spend time on it.

So I’m trying to spend more time driving the list down. Once I’ve done that, I’ll consider putting a work in progress limit in place, but right now I’m just trying to remove items more quickly than I add them for a bit. And it’s not that hard to do so once I put my mind to it: for example, I can easily knock off an hour-long item every evening if I just get it out of the way before reading blogs instead of reading blogs and noting that it’s 9:45 and I’ll be getting ready to go to bed in half an hour or so, and putzing away the remaining time.

I would warn that all this means that I may well not end up blogging as much here for the next couple of months. Honestly, though, that seems unlikely: doubtless going through those backlog items will turn into blog posts, too (and, indeed, several of the current backlog items are to write posts on various topics). And I’ve written ten posts here in the last two weeks even though I’ve been nibbling away at my Next Action list, which is noticeably higher than my average. The contents might change somewhat, though: in particular, I’m not planning to start any new video games for a little while. (But I’ll keep on playing Minecraft and Rock Band 3, and both of those will certainly lead to posts on my other blog and probably here as well.)

I don’t blog here nearly as often about organization stuff as I used to, but that’s not a sign that I don’t think it’s a good idea: it’s more a sign that I’ve internalized a lot of the ideas. For the record, then:

• Agile: still awesome.
• Lean: still awesome.
• GTD: still awesome.
• Inbox Zero: still awesome.
• Checklists / Standard Work: still awesome.
• Kanban: looks awesome.
• Pomodoro Technique: Has a few good ideas that I’ve brought into my practice, and I occasionally turn on my pomodoro timer when I need extra help focusing, but I don’t follow it in general.

But my GTD practice is definitely a bit chipped and tarnished (and should be better informed by Kanban); time to sharpen it up a bit.

After I wrote that post on precedence, map, and function composition in Scala, I started to wonder: I’ve been thinking that I should experiment more with applying Compose Method. That refactoring recommends that, if I start with the original version of my code,

data.foreach(s => writer.addDocument(createDocument s))

then I should extract the body to a method. Which, I guess, would lead me to something like this?

data.foreach(addStringAsDocument(_, writer))

Except that that’s actually not what Compose Method really recommends: that’s merely one standard way of applying it to languages that are somewhat lacking in expressive possibilities. If all you have are manual looping constructs, and if you want to “keep all of the operations in a method at the same level of abstraction” (Smalltalk Best Practice Patterns, p. 22), then yeah, you’ll pull out your loop bodies to methods, but there are other ways to reach that end.

So, looking at the code that I actually ended up with that post (with the kind help of my readers),

data.map(createDocument).foreach(writer.addDocument)

is everything there at the same level of abstraction? That’s not entirely clear to me: if I wanted to, I could certainly extract a couple of methods out of that, and end up with something like this:

addDocumentsToWriter(createDocuments(data), writer)

Examining Alternatives

We’ve seen four examples of how that code could look; let’s replace the first of those with one that raichoo suggested on the previous post, giving us the following list:

1. data.foreach(createDocument _ andThen writer.addDocument)
2. data.foreach(addStringAsDocument(_, writer))
3. data.map(createDocument).foreach(writer.addDocument)
4. addDocumentsToWriter(createDocuments(data), writer)

Anybody want to argue for any of these being noticeably better or worse than all of the others? I’ll have to say: while they all seem fine to me, I can’t get too worked up over the need for using Compose Method in this case. Though, as I’ve been typing them up, I’ve wanted to add “fromString” in various places, which suggests that the method name createDocument is perhaps not as well chosen as it could be: maybe I should have called it something like stringToDocument instead?

Hard to say, I’m still happy enough with the third option. It says fairly directly that I’m starting with a bunch of data, turning it into a bunch of documents, and adding them to the writer: fine by me. (The first option seems approximately similarly expressive to me, as well.) There are, of course, situations where one or the other composed method would be preferable (as I said at the end of that earlier post, I ran into one an hour after I ran into the above!), but this doesn’t seem like one.

Recomposing and Natural Transformations

Maybe it’s the category theorist in me, but this also raises one other question: consider the two composed methods, 2 and 4 in the above list. Say that you reflexively went with option 2, but then decided that it didn’t seem quite right. You could (probably would) inline the function, and then distribute and regroup to end up with the fourth version; that seems like a pretty standard sort of thing to want to do. (It wouldn’t be too much of an abuse of language to call it a natural transformation!)

So: if we’re going to make a taxonomy of micro refactorings, might we not only also want to list ways of composing them (as, indeed, Refactoring itself suggests; see also Refactoring to Patterns or the hierarchies of patterns in A Pattern Language), but also ways of undoing them and composing them differently, along the lines of associativity laws?

Part of me feels like I should be bothered by the man of great faith who got a long-distance cure for his household member; but I’m really not. “Let the dead bury their dead” (Matthew 8:22) seems a bit harsh, but only a bit.

Matthew 9 repeats much of the same themes, but there’s some new stuff, too. It starts off by saying that Jesus cures blasphemers, too, confirming my feeling that I shouldn’t be bothered by the man of great faith by the previous chapter. In fact, he turns this into a lesson on forgiveness, which I rather like: “For whether is easier, to say, Thy sins be forgiven thee; or to say, Arise, and walk?” (Matthew 9:5)

And then he goes to eat with “publicans and sinners” (Matthew 9:11). Which is a word I had to look up, and Lattimore translates that it as “tax collectors and sinners” instead. At any rate, an interesting pairing, especially in today’s political climate that pairs an absolutist anti-tax line with a refusal to examine the benefits that those taxes are bringing us.

Take, for example, health care. Yes, it would be great if we could have the son of God wandering around healing us whenever we get sick. But even two millennia ago that didn’t come close to scaling to meet the actual need, and these days our shortage of healing deities is even more sorely lacking. Fortunately, our health care has improved enormously over the intervening millennia, though (pace The Rational Optimist) I won’t credit government with much of the improvement there. But I will credit government with some of the improvement, and we need a health care system of last resort, one that will look after those who aren’t financially able to pay for their own care; hence, taxes.

Anyways, continuing on: not sure how I feel about the whole bit about Jesus eating instead of fasting, justified with “Can the children of the bridechamber mourn, as long as the bridegroom is with them?” (Matthew 9:15): I’m all for moderation in appropriate contexts, but I don’t think that’s what’s going on here. Still, I certainly won’t blame Jesus for wanting to have some food in his stomach, though, especially given what’s going to happen to him.

I also can’t say that I understand the “No man putteth a piece of new cloth unto an old garment” bit that follows next (Matthew 9:16)—I get the feeling that I should be learning something there, but it doesn’t quite fit into the narrative flow to me. Or maybe I should read those verses as an affront against the virtues of refactoring, or as a caution that it can be taken too far?

Things I’ve liked:

• Great teammates: I’ve been on two teams here, and quite enjoyed being a part of both groups.
• Having what I do make a difference in somebody else’s life in a matter of days or even hours; a huge change from my last job.
• Being able to talk to people on the business side about what I’m doing, showing them the current state and asking for (and responding to!) feedback.
• Learning technologies I wasn’t previously familiar with: JavaScript, a bit of ActionScript, a bit of Hadoop. And getting a chance to put a CSS Zen approach to concrete use, too.
• Tools supporting agile programming: getting to use an IDE with refactoring support; using a mocking framework.
• Brushing up on Java isn’t a bad idea, though not inherently exciting; at least it helps reduce my risk of being pigeonholed as a C++ guy.
• Doing front-end work; I love getting to work with what our artists create.
• Being a programmer instead of a manager.
• Working with games; getting back to playing board games regularly.
• Deepening my appreciation of testing by looking at business metrics.
• Getting experience with growing a successful product.

I could probably write one or more blog posts about each of those; lots of great experiences, in retrospect it’s a bit hard to believe all that happened over one year! But that’s last year, what do I want out of next year?

More exposure to new technologies would be nice, though I imagine the rate there will slow down. Still, there are some possibilities: I imagine I’ll get more familiar with ActionScript, I might get exposed to Lua. (And I might get more exposed to PHP, though I can’t say that I’m excited about that.) Might get to learn more about data scaling ideas, too.

Helping add significant new features to an existing game has been rather interesting, and I’m glad I did it. On the flip side, though, I’d like to work on a new game at some point, to get a better feel for the tradeoffs that are relevant there.

And then there are teams and roles. I’d been a manager on a team where we had some agile theory but had a hard time putting it into practice, especially on the business side. It’s been nice being a non-manager, and having a much tighter turnaround for the customers and customer proxies.

Ideally, I’d like to be on a team merging some of my previous experience with the virtues of my recent experience. Adding a bit more agile theory (and lean theory) to what we’re doing at Playdom would help, I think, but not at the cost of losing the virtues of our current approach. And the more the team (broadly conceived, of course, not just the programmers) works as empowered individuals evolving a common creative vision, the better. (I don’t want to be a manager, but I don’t want to be told what to do by a manager, either!)

Good times past; looking forward to good times future.

Which I found it really refreshing to read. It is, to me, undeniable that the world has gotten hugely better over the last few centuries; and I like seeing somebody say “why assume that that progress will stop now?”, backing it up with plausible reasons why that might be the case. As he says, there’s a lot of pessimism out there; if there’s reason for that, fine, but is there?

His focus on specializing was a bit of a wakeup. Over the last several years, I’ve spent time in intellectual climates that warn of the dangers of specializing; certainly I prefer the role of generalizing specialist myself. Having said that, there’s no particular reason why what I prefer should be good for the progress of society as a whole! He points out Ricardo’s law of comparative advantage: even if person A is better at person B at producing both X and Y, if it happens to be the case that A can produce X faster than Y while B can produce Y faster than X, it’s still best for everybody if A specializes in producing X and B specializes in producing Y, because that leads to the most production for the least effort, despite Y being produced less efficiently than it could be. (That sentence is pretty abstract, so follow the above link for some numeric examples.)

Given this, why not specialize wherever possible? Lean and ToC give one answer, namely the dangers of suboptimizing. And Ridley gives another answer: new ideas appear through the mixture of other ideas. (Among other means.) So I don’t have to feel bad about my penchant for sticking my nose into random areas of thought; but the flip side is that what I do professionally (and, to a large extent, in my free time) is solidly grounded in the details of programming, and that’s healthy. (And, looking back over the course of human history, my profession certainly qualifies me as highly specialized.)

He also had some interesting negative things to say about movements that I’m ideologically sympathetic to, namely the organic food movement and the movement against global warming. I don’t entirely agree with his criticisms of both of those movements—in particular, I think he both overestimates how seriously people took some previous disaster scenarios (e.g. acid rain) and underestimates how serious some other previous disaster scenarios actually were (nuclear war; actually strike the word “were” there and replace it by “are”), fitting them to a bit of a narrative Procrustean bed. And, to some extent, the reason why some of these didn’t turn out badly seems to me to be because of the sort of government intervention that Ridley dislikes; in general, he doesn’t seem to me to spend enough time on Tragedy of the Commons problems.

But there’s a lot in his criticisms of those movements to think about, and I agree whole-heartedly with one part of his criticism and approach to criticism, namely that you should get your metrics right. Metrics aren’t everything, but they’re important, and you shouldn’t cherry pick them to lead in the direction where you’d like the answer to be. (And his other point that there’s a lot of unwarranted pessimism going around is probably true, too.) I hope that the environmental movement isn’t as bad in this regard as Ridley claims (I certainly hear a lot more bad said than good about corn-based ethanol, for example), and the Azimuth Project (including blog posts like this one) makes me optimistic that good use of metrics will be (hopefully, already has been) a strong part of the discussion. But it’s an important enough point to deserve to be hammered home.

Certainly a book I’m glad to have read. (And it’s a fast read, too.) A hat tip to Jesse Schell for the recommendation.

So, once I’d made it through the accumulated post-conference tasks, I looked at the last set of unit tests that I’d written. And, actually, it looked like they’d be even easier to tame than I thought: I’d already put in a legacy code barrier by writing the new functionality there as a static method, so I didn’t even have to instantiate an instance of the class that I was allegedly testing, just instances of the arguments to the method in question.

Looking more closely, it turned out that there was really only one argument that I needed to worry about. It was an instance of a concrete class that was a pain to instantiate without running through all of our Spring machinery: it created a static Memcache object, it had another static instance variable that was looking up a Spring-initialized bean (that eventually depended on grabbing some data configuration from a static data file), etc. I certainly know techniques for delaying static initialization, and thought about going down that path for a minute, but then I stopped myself and said: I want to start using jMock for the heavy lifting of instantiation, so I shouldn’t go that route at all!

That means that I need an interface instead of a concrete class; is that okay here? I looked at the test and the code under test, and they weren’t about that messy concrete class, I was just testing something that was using it in a not-very-deep manner. Given that, I didn’t see any reason why instantiating an interface instead would weaken my test. I didn’t have an interface handy, so I created one for the concrete class to implement; I thought for a couple of minutes about what it should look like before realizing that I didn’t actually care right then, what it needed to look like was (at first) enough to get the test in question to compile! That bit of test and product code called a grand total of two methods of the original class; I added those two methods to the interface, and looked up the jMock boilerplate for instantiating an instance of that interface; and poof, my tests were compiling, without any database-dependent code involved!

Of course, they weren’t passing yet: assertions were failing, and jMock was also informing me that mocked out methods were called. I looked a little more closely, and realized that one of the two methods in question was a setter that (in the context of running these tests) was only being called to set up the object so that the other method would return what I wanted. Which makes sense in a non-mock context, but in a mock context, you can just directly tell the method to return what you want. So I deleted that setter from the interface and from the test code, told the test’s Mockery to return some appropriate data when the other method was called, and ran the tests.

And they passed! Indeed, they passed so quickly that, at first, I assumed things had gone wrong: the compilation still took a few seconds, but once the tests started running, IntelliJ didn’t even have time to show me the listing of all of the tests that it was running before they all finished. But I did some poking around, adding assertion failures or commenting out lines of product code, and the tests were indeed all running, they were just running approximately 250 times as fast as they had been before.

Which was a huge success! And not only were the tests fast to run, they were also fast to write. I’d estimated that this would take a day or so; in fact, it took somewhere between half an hour and an hour for me to get this working, despite my complete prior lack of jMock experience. Admittedly, I’d lucked out by picking a clump of tests that was particularly easy to convert, but still: my tests were running hundreds of times faster after less than an hour of work! Amazing.

Flush from that success, I used jMock for the next feature that I was working on. That feature involved bringing some (actually pretty well written) legacy code under test, doing some replumbing to make it more extensible, and then extending it slightly. There were several more function arguments to deal with here, but all but one of them was already an interface. The one exception was the class that I’d just started to convert to an interface; so my first step when bringing each chunk of code under test was to try to replace the class with the interface when it showed up as an argument in that chunk of code, see what the compiler complained about, pull those methods up to the new interface, and repeat. (And if it got too thorny and if I was in a code branch that I wasn’t going to reach with my test, I’d just throw in a cast to silence the compiler.)

Then, once I’d gotten it to compile, I’d create a unit test that instantiated mock objects and called the method. This would immediately point out a calls that I needed to tell the Mockery to expect; that’s fine, I could iterate on that every 30 seconds, so even in the most complicated case, I had the test running in under five minutes, and passing for the right reason a minute after that. The upshot was that, within a few hours, I had all of the relevant code running with tests that were good enough to support the refactoring I wanted; a few hours later, the first refactoring was done, and I was in a good TDD flow for the first time since I’ve started my Java work on that project. A couple of days later I had the new functionality in place, running correctly in the first end-to-end try, with the total number of unit tests on the project increased by 50 percent or so, and with a structure in place so that will enable us to knock off a bunch of upcoming feature requests in an hour or two each.

I am, obviously, very pleased with the results, and very impressed with jMock: it’s amazing how much work it’s been saving me. So I’m a total convert to using it to tame legacy code; what I’m interested in next is how to use it, as Growing Object-Oriented Software suggests, to guide the development of new interfaces going forward. (It pulled out an interface for me in the code I was working with last week, but it was an ugly interface, serving more to point out warts in the existing structure than anything else.) Lots of fun, I look forward to the next step in that journey.

And it’s funny in retrospect how blase I’ve gotten about open space conferences: of course we managed to self-organize and come up with loads of interesting and useful sessions, isn’t that the way it always works? Amazing when I first saw it, but now I’ve seen it happen four years in a row, so it’s clearly a quite repeatable process.

8:30am

I didn’t end up attending any of the early morning sessions today, what with dog- and daughter-related interruptions overnight; I guess that counts as my session to take a nap in today?

10:00am: Coding Dojo

This ended up rather interesting in a completely unexpected way. I was looking forward to having a technical hands-on session instead of a conceptual touchy-feely session, and the plan seemed promising: we picked a problem to TDD (the Bowling Kata), and there were few enough of us (five plus the convener) that we decided to just rotate pairs through a single computer every five minutes.

But things went wrong in a bunch of ways right from the start: the convener inserted himself way too often (in my opinion), participants ignored each others’ suggestions and blithely threw away their code, they occasionally openly insulted each others’ technical competence, at times a participant had a hard time picking up on what was going on quickly enough to make any progress during the 5-minute coding slot. (And that’s just what I saw in the behavior of the other five people there; extrapolating from that, it seems quite likely that I was behaving badly in a way that I was blind to as well.)

I was thinking of giving up and leaving, but the thing is: these are dysfunctions we see all the time on real teams, they were just magnified because of the quick pace of the format. So I decided to stick around to see if we could resolve them; and, indeed, we brought up some of the worst of the issues and started dealing with them; half an hour later, we were going much more smoothly, and were writing actual workable code.

So: good stuff, though I still kind of wish we’d been able to spend more time on code issues than people issues…

11:30am: Enterprise Agile Transformation

Pleasant enough chatting, but I don’t have a lot to report here.

1:00pm: TDDing JavaScript

This was one of my sessions; I was happy enough for the conversation to go wherever people wanted, but ultimately it came out my dissatisfactions with TDDing JavaScript at work. The problem that I have is that my TDD instincts aren’t working well: the first code that I’m tempted to write (if I start from the user point of view) is so closely tied to the layout details that turning into tests would be tedious and lead to low-value, brittle tests; I’m also not entirely sure about tests that I’d get starting from the back end communication point of view, because there I’d be testing XML parsing that is both relatively simple and could silently go stale if the format changes. So the upshot is that I start off not writing tests for my code, and I stay in that mode for far too long.

For the code adjacent to the back end, I really want integration testing; and, indeed, that would be useful at both ends, I should definitely do that more. So we spent some time talking about possibilities there (basically, me getting more of a Selenium education); that was useful, but I don’t want to go down that path too deeply, because I’ll end up with slow tests that have a hard time getting at non-surface problems.

Fortunately, Mike Wright managed to eventually break through some of my problems: after encouraging me to explain why I didn’t like the tests I would normally think about writing, he took a BDD-point of view to asking me how to express tests that I might like to write. And by taking that approach he managed to come up with tests that get close to the layout to while not being excessively brittle. E.g. if I’m testing a feature that involves voting for one of two avatars in a walkoff, I can test that the layout contains exactly two divs with a “walkoffAvatar” class: that’s expressive enough to force me to be doing something meaningful with the data when generating the layout, but it’s generic enough that it doesn’t tie me to the specifics of the layout, specifics that are only useful to the extent that they produce something pleasing to a human eye. Best of all, it means that I can start happily unit testing right from the start, instead of trying to pick a moment when the code is getting interesting enough to need testing.

Really useful, I’m hoping that this will encourage me to TDD my JavaScript significantly more often than I have been these last few months.

3:00pm: Agile for Introverts

I’m an introvert, as was the convener, but a lot of the participants weren’t; can’t I escape for a moment from those extroverts? (Joke.) Honestly, I was surprised how many extroverts attended, but they were really very well behaved, I probably spent as much time talking as most of them did.

The notes list many of the topics we talked about, but two concrete techniques seemed particularly useful. One is to maximize the chance that introverts will speak up in a meeting: when you’re thinking of moving on to a topic, ask if anybody has questions, then pause for a second, and then count down from five to zero, starting with an open hand and folding down one finger at each number. (A variant of a technique I found very useful when teaching: when you want to elicit questions, wait for noticeably longer than is necessary for people to start feel uncomfortable with the silence.)

And the other is some gestures that you can introduce: wave your hands with fingers spread out to express agreement, hold up a hand with two raised, crossed fingers to express that you’d like to speak fairly soon; and one other gesture whose details I can’t remember (maybe a sort of hitchhiker’s thumb?) to express that you’d like to make a followup comment to the last statement. So normally those last people get priority, but if somebody’s been waiting with fingers raised for a while, you give them priority.

I’m seeing a lot more familiar faces at Agile Open Northern California 2010 than I had been in previous years: before, I’d always been surprised at the low percentage of repeats, but this year I’m seeing people I know everywhere I look. So: glad others also find this valuable enough to keep coming back! (Or maybe they, like I, are suckers for the setting, by far my favorite conference venue.)

10:30am: Why Agile?; User Documentation and Agile

I stayed for about half of the first one session listed here. We spent most of that time talking about how people problems are important but hard. Which is true; it’s something I struggled with at Kealia/Sun, it’s something I’m struggling with at Playdom.

After that, I wandered around a bit, ultimately ending up in a doc writing session. Interesting questions there, e.g. If you implement a single button (of multiple planned ones) on a page, when do you document it? I have a first instinct for what the answer is, but I suspect the question could shed some new light on problems with feature flow. And if tech writers are a shared resource (which seems fairly common), that throws another wrench into the works.

(Though of course I have an answer to that from a technical lens: avoid specialization when possible. Is it possible there? E.g. How well do tech writer and QA skill sets overlap?)

12:00pm: So You Have Slow Tests, Now What?

We started off taking a poll of what we meant by slow tests, and got a huge range of responses, from 5 minutes (or even 25 seconds) to multiple hours (or even more than a day, but they parallelized that one to only be a few clock hours). Which would make me happier about the length that the tests in my current group take, were it not for the fact that we have so few! (In particular, no automated integration/end-to-end tests that the team maintains, though I’m actually working on improving that right now.)

I mainly went because our unit tests are way too slow: they almost all depend on doing database access, and while we try to avoid constantly setting that up, it’s still the case that running even one “unit” test can take a good 25 seconds, which is a real friction point for doing TDD. And the answer to that is pretty obvious: by hook or by crook, I need to mock out that. Which I’ve been putting off, but talking about it in public makes me confront the question of how hard it can be? And, when I think about it, my best guess is one day to get something in, plus or minus a factor of two; I can definitely find time to do that soon.

1:30pm: The Christopher Alexander Bench

After looking at the bench last year, I decided that I had to hold a session there unless it was pouring: its presence is too much of a coincidence to pass up, and the setting here is so gorgeous that I’ve always felt it was a shame that we never have sessions outside. So I scheduled one, with absolutely no plans for topics to discuss: if the setting inspires us to discuss something, that’s great, otherwise we’ll just look at Alcatraz and Angel Island for a bit and then wander off to other sessions.

And it was great! A dozen people showed up, Tom Looy began with an appreciation of Alexander’s notion that variation within a pattern leads to beauty, and we were off and running. For whatever reason, we talked a lot about planning meetings, starting with the benefits of physical cards over Jira tickets and ending up with a heated discussion about estimation; no idea why we ended up there, but people were actively participating, which is all I need to declare it a success.

We did touch on the bench occasionally, too: I passed around the book, and we were inspired by the paragraph at the beginning of the relevant section:

In the kind of dynamic process I have in mind, where each act that is taken, is related, in a structure-preserving way to the whole, and takes its place in a long time-line through a long sequence of events, each part, then, is carefully shaped and placed into the whole, making the whole more than it was before.

(Incidentally, I think the bench in its setting gives a great example of that last bit.) I talked a bit about those of the fifteen properties that I could remember; Lev Ayzner pointed out an example of Alternating Repetition that I’d missed, namely between more abstract and more naturalistic patterns on the front tiles.

Good stuff; I’ll probably do the same session again next year.

3:00pm: Kids / Education; Code Excellence; nap

I seem to be unable to attend a conference without napping through one session per day; this was today’s choice.

4:30pm: Key Unlearnings for Agile

Very good session, about bits of the pre-agile mindset that we not only needed to unlearn in the agile transition, but still need to unlearn! E.g. that I have to have all the answers, or an us versus them mentality between different groups (workers and managers, developers and product people, developers and testers). We generated lots and lots of examples, which almost all rang true to me; I recommend going to the notes on the wiki and the mindmap.

There’s a certain strain in the criticism of social game development that has always sounded bizarre to me, and I think I’ve finally understood why. It’s really a two-fold strain: on the one hand, it complains about the metrics-driven approach to game development that social game studios typically take, and on the other hand, it complains about the focus that social game developers typically have on how exactly they’re going to make money. With these two hands frequently united in an allegation that all that social game makers care about is whether a feature increases or decreases revenue.

These complaints were so foreign to me that I ignored their existence for quite some time, until some smart people that I have quite a lot of respect for caused me to realize that yes, many people really do have that point of view, and it was one that I should take seriously. And, once I started taking it seriously, I realized that it’s actually a completely natural point of view! I like making analogies on this blog; and if I replace video games by, say, books or music, then the idea of a metrics-based approach and a business focus would sound pretty weird/unpleasant to me, too. So what I’d discovered was that I had a pretty big blind spot in that area myself; where did it come from?

The answer that I’ve come to is that it’s because I’ve spent the last seven or so years immersed in agile software development; and a metrics-based approach and a business focus both fit into agile quite naturally. One of agile’s key tenets is that you should have a precise notion of what ‘done’ means: agile recommends this in your tight coding loops, with TDD; it recommends this at longer-term levels, with acceptance criteria for stories. And a metrics-based approach fits into that point of view very well, helping to extend the importance of ‘done’ from the programming side to the product owner side: it encourages the product owner to also have green bars of their own that their feature prioritization process is trying to attain, it encourages them to come up with formal criteria that they’re trying to satisfy.

Furthermore, looking at metrics is a way of paying attention to what your customers (and potential customers) are actually doing. And this, too, is firmly established within agile: XP has a “Customer” role, with a strong suggestion that that person be an actual customer of the product. This, of course, doesn’t make sense in all circumstances, but even when it doesn’t, similar ideas appear: e.g. Toyota famously sent its designers of their first minivans and first Lexus models to live with their target customer base.

So, the upshot is: paying close attention to customer-focused metrics makes a lot of sense from an agile viewpoint. Indeed, not doing so when such metrics are easily available (as is the case for web-based software) would be rather odd.

That’s the metrics side of this puzzlement; what about the business focus? A business focus is, honestly, something that I couldn’t have cared less about when I was younger—I was all set out to be an ivory-tower academic—but here too, agile was an awakening for me. Take the Customer concept I mentioned above: that person may not be an actual customer, but she’s definitely somebody with the business goals firmly in mind. This business/engineering collaboration is firmly established within agile: the business side sets the priorities, the engineering side takes charge of getting those priorities done. And both work together to proceed in an iterative approach, to deliver business value as quickly as possible and to discover actual (not hypothetical) business value as quickly as possible, allowing them to sculpt the product’s goals to meet business needs.

And this, it turns out, is pretty fascinating. It’s especially fascinating for me because of my personal history: I worked on the same product for several years before joining Playdom, and it was never a business success. Some of that is the fault of engineering execution (for which I shoulder as much blame as anybody), but a fair chunk of that is that we never managed to find a way to engage the market that allowed our engineering strategy to work. And that wasn’t because people thinking about the business side of the product were stupid, or made obviously bad choices: it’s because it’s a really hard problem to solve. So, from that point of view, working at Playdom is a huge relief: I can frequently point to something that I worked on last week that is making money for us this week, and that feels great.

So: agile has trained me to like metrics, agile has trained me to take a business focus. But where does that leave art, where does that leave craft?

Here, too, we run into a blind spot of mine: I wouldn’t normally think of those (especially metrics) as being in any way antithetical to art or craft. The reason for that again, of course, being agile: it has strong statements in favor of the craft of programming, paired with quite a lot of explicit guidance to that end. My favorite agile practice is TDD, and craftsmanship is right there on an equal footing with measuring your results: yes, you have to come up with a metric (“red”), but you also have to make your code look good (“refactor”). And lean also has an extremely strong craft focus.

It is, of course, the case that working to satisfy metrics and business concerns puts a constraint on your choices. But this isn’t inherently antithetical to art or craft: in general, I think that constraints are as likely to help art as they are to hurt it.

So, that’s where my blind spots in this area come from: my recent background means that certain questions that are natural to others just aren’t questions that I’d be likely to think of. At least if I’m primed to be in an agile mindset by being in a programming context; as I said above, I’d be quite likely to raise those complaints in, say, a literary context. Which raises the question: which is the best model to consider here?

And, to that, my best answer is “it depends”. If you’re driven to create games that come out of your experiences and vision, if you couldn’t care less what most people think or whether or not you make money, then I will in all sincerity say that that’s wonderful. (On which note, everybody please chip in some money for Jordan Magnuson’s Game Trekking project!) Even when doing so, you may find the occasional metric useful—see this report of how dramatically differently users may approach our games from what we expect for an example of why—but you may not.

But, for better or for worse, most video games take rather more work to produce than most books. So I personally would not be comfortable working on a project that multiple people depended on for their livelihood without a hard look at business considerations. (Heck, forget “multiple people”: I wouldn’t be comfortable working on a project that I depended on for my livelihood without a hard look at business considerations.) I don’t care about business considerations at all when writing this blog; I care about them a lot when working.

Our art form will become better the more tools there are in our bag. Being aware of tensions is great; treating tensions as creating separate worlds, not so much.

But there are a couple of lurking issues behind this change, too. Link roundup posts make it harder for me to save one particular link for later perusal; or, from a philosophical point of view, I don’t have a URI that I can use to refer to the link recommendation. (This is the main thing that bothers me with Twitter’s new retweet feature, too.) Also, I don’t want the link roundup posts to overwhelm this blog, but that has led to batching, which leads to a wealth of problems, so I’d rather eliminate that bit of Work in Progress.

So a separate linkblog seemed like the way to go. And Tumblr seemed easy to use, and I like the way it encourages building off of others’ recommendations while preserving recommendation history. (In other words, it nicely avoids the URI problem mentioned above.) And I’m certainly happy with it so far; I wish it had a native iPad client, and I wish that Reeder added Tumblr to its list of link forwarding locations, but neither of those is a serious problem.

I’m working on a legacy code base, with all that entails, but that hasn’t stopped me in the past. So why am I not writing as many tests this time? For that matter, is it possible that it’s correct for me to not be writing so many tests?

There is, I think, a good reason for some of this. A lot of what I’m doing is arranging graphical elements so they look good on screen. And tests wouldn’t help with that: they’re great for getting logic right, but with presentation, all they’d do is pin down unimportant details. So if I later decide that a margin should be seven pixels wide instead of five, having a test would be pure overhead. (Incidentally, placing images properly is another thing that I’m surprised how much I enjoy—I really like participating in a process that leads to a much better-looking page, even though the art team is doing most of the hard work.)

But some of my lack of testing stems from unfamiliarity with unit testing in JavaScript. I’d done my due diligence, and decided that JsTestDriver is a pretty good unit testing framework: it lets me kick off unit tests in multiple browsers simultaneously from the command line, so I can make sure my code runs in production environments while not having to leave my IDE.

So I wrote a few JsTestDriver tests as a proof of concept. At which point I ran into my first roadblock: without trying to do anything tricky, the first four tests that I wrote all failed in Internet Explorer! Which was useful information—I learned that querying CSS properties is not likely to give the results I expect in IE—but the functionality in question actually worked fine in IE, I was doing the querying to check results in tests, not as part of the product code.

Since then, I’ve written a few more unit tests, when I didn’t understand why a piece of JavaScript logic wasn’t doing what I expected, and it has saved me time on those occasions. But I haven’t even begun to get into a good TDD rhythm at work.

So I decided to experiment at home to understand how much of this had to do with JavaScript, how much of it had to do with my ignorance, how much of it had to do with the kind of layout-heavy work that I’d been doing, how much had to do with legacy code, how much had to do with Internet Explorer. (Which I’m certainly not worrying about at home!) Specifically, I wrote an animation of moving boxes; there’s enough layout in that code for it to not be too unrepresentative of front-end work, but it’s also not about getting images positioned in the most attractive locations. (In fact, there aren’t any images at all, as will be obvious if you take a look; none of this new HTML5 stuff, either, the animation is all done by manipulating CSS positions and dimensions on a timer.)

The good news: I could, indeed, get into a TDD rhythm. Looking at the code right now, I have 374 lines of product code and 294 lines of test code; that amount of test code looks a little small (or, actually, given what the animation does, that amount of product code looks a little high…), but it’s not a ridiculously low proportion of test code.

More good news: a couple of nice generic abstractions popped out of the code as I was writing it (including one which got better as I added more functionality), and there are a few more abstractions that are specific to the animation that are waiting to be teased out of it. So it’s helped me improve my feel for good JavaScript code. (Though I certainly have a ways to go there…)

Indifferent news: I’m still not sure that all the tests are really getting at the core of what I’m doing. A fair amount of the thought in the programming exercise involved fiddling with CSS classes and creating HTML elements to give the illusion that you have a single box that is stacked up, then moving, then entering the stack again, while behind the scenes there are actually two HTML elements involved. (One positioned in the normal document flow, one positioned absolutely.) And the tests for that part of the code have too little to do with the desired visual effect and too much to do with implementation details. (Though not my all unit tests involving animations were bad: e.g. the unit tests for the movement of the element once I’d lifted it out of the normal document flow were perfectly reasonable.)

And then the bad news: I’m used to unit tests that run in isolation and that I can feel confident are testing something. (Though, being a fallible programmer, they may not always be testing what I intend!) And, it turns out, JsTestDriver fails on both counts. (I don’t actually think that’s JsTestDriver’s fault, I think it has more to do with the way JavaScript implementations work.)

I typically was running my tests in both Safari and Firefox. And, on more than one occasion, a test would just fail repeatedly in one of the browsers, for no apparent reason. The first couple of times this happened, I would curse the lack of cross-browser compatibility, check that the product code behaved as I expected, and reluctantly proceed. But then I noticed that, if I actually killed the browser where the tests were passing and restarted it, then the tests would start working again! So it seems to be the case that it’s rather difficult for JsTestDriver to completely wipe out its state from test run to test run. (I’m not sure if I tested just closing the window where JsTestDriver was running instead of exiting the browser completely; so maybe unit test frameworks where you have to manually reload a page in every browser each run wouldn’t suffer from this problem.)

Worse, there were times where I would make a syntax error and a whole suite of tests would silently stop running in one (or both?) of the browsers. That’s really disturbing: I expect my code to either try to run or give me some indication that it might not be running, but that didn’t always happen. I wasn’t in the habit of having Firebug open in the web browser to see if I get errors there; next time this happens, I’ll do that, but I shouldn’t have to, given that many JavaScript errors did make it out to the command line where I was invoking JsTestDriver. I’ve never had something like this happen to me before: I could chalk it up to dynamic language weirdness, but Ruby is pretty permissive and I’ve never seen anything like that in my Ruby tests.

An interesting experience; I definitely want to spend more time playing around with JavaScript at home. (Anybody have JavaScript code they want me to write?) I’m less sure how I’ll apply this at work; though certainly, when I look for smaller classes to extract, I should make them testable and actually add tests for them. But I’m having enough fun (and being useful enough) working on presentation issues that I’m not going to stress if I don’t spend a lot of time doing that.

James Shore has also blogged recently about his experiences with test-driving JavaScript. And, though he was using a different unit testing framework (QUnit instead of JsTestDriver), he ran into the same problem that I did of tests silently failing to run. (His reaction: “The mind boggles.”)