malvasia bianca

I’m very glad that Michael suggested Grim Fandango as the introductory game for the Vintage Game Club, because adventure games and I go way back. I can’t remember the exact sequence of events, but I’m fairly sure that I was aware of the Colossal Cave adventure before we even owned a computer: I think my brother got an account (or used our dad’s account?) on the Oberlin College systems, they had a version of it there (a 550-point version, not the standard 350-point version), and he told me about it. In fact, he was once willing to let me “play” it via a typewriter, with him simulating the computer - isn’t he a sweetie?

We didn’t have any game consoles when I was growing up (I didn’t own any until grad school), but we did get an Apple ][+ at some point. (The summer after fifth grade sounds about right, but I can't remember for sure.) I played quite a few games on that system; probably the ones that stuck with me the most were the Infocom adventure games. (Though the various Ultima games are a close second.) Most people think of Zork when they think of Infocom, but we didn't have a copy of that (or its two sequels); I've played it since, but I'm not sure it added all that much to the Colossal Cave formula. We did, however, have the tangentially related games Enchanter and Sorcerer, and I very much enjoyed those: they present a much more coherent environment than their predecessors, they add a well-done magic system as a gameplay element. And the games were actually solvable without hints, which mattered a lot more in that pre-gamefaqs time. Though I do remember having to bang my head against the stairs in Enchanter a lot - did I eventually figure that out myself or did I get a hint from a friend somehow?

I believe (but I'm not entirely sure, I might have only played it later) that we got the Hitchiker's Guide game; not one of my favorites, they chose jokes over gameplay and the jokes were a lot better in the books. (At that time, the fact that works in other media rarely survive the journey to video games wasn't yet burned into my brain.) A game with experimental gameplay that worked better was Suspended, where you had to switch back and forth between six robots with different senses and abilities to explore the world.

Which brings me to the packaging. Those were the days when (the best) games came with real, well-written manuals. Actually, 'manual' isn't the right word, since that suggests something focused on instructions for playing the game: these, instead, were generally artifacts from the world of the game, e.g. a newspaper issue from the game, a one-zorkmid coin, etc. And Infocom was one of the best at this. (Though I also fondly remember the cloth map and ankh that came with Ultima II, if I'm remembering correctly.) Some of you may have seen the grey box editions of their games, but the earlier packaging of the games was even more impressive: I had the edition of Suspended that came with the huge inset plastic death mask and a map with tokens for the robots to move around, and I'm still kicking myself for not having saved that.

My favorite of their games, however, was Planetfall. Great gameplay, well thought-out puzzles; but the reason why the game sticks in my mind a quarter-century is Floyd. I'm probably missing some good examples, but I'm still not sure I've run into another video game NPC that mattered as much to me as Floyd did: he was this wonderful childish robot who tagged along, helping and entertaining you, and the scene where you have to send him to his doom (for which he bravely volunteers, if I'm remembering) combined with this later reappearance still choke me up as I'm typing this.

Sigh.

I didn't manage to play their later games at the time: Spellbreaker needed too much memory to run on the Apple ][+ (perhaps a blessing in disguise, I played it later and it wasn't nearly as good as Enchanter and Sorcerer), and I also missed out on Trinity and A Mind Forever Voyaging. I did play a bit of the former eventually, but I still haven’t either finished the former or gotten around to the latter; both would be excellent choices for later Vintage Game Club installments. (As would almost any of the games I’ve mentioned here!)

One other Infocom side note: their games were written in a language that compiled to a virtual machine, the Z-Machine. It has since been reverse engineered, and I wrote an interpreter for it in order to learn C++ when I was transitioning out of academia; if any of you are thinking of learning a new programming language and are looking for a decent-sized project to implement, you could do a lot worse than that.

I don’t recall playing many (any?) adventure games during my undergrad years. During grad school I rounded out my Infocom experience a bit more, and played Myst; I thought that game was excellent, but for whatever reason it didn’t inspire me to seek out more graphical adventures. Maybe I was jaded; maybe (despite Myst’s phenomenal popularity), adventure games just weren’t near the center of the video game world the way they were in the early 80’s; maybe I was spending too much time learning math, doing research, reading books, and other unwholesome activities. Which was probably the correct choice at the time, but I’m very glad to have this opportunity to fill in a bit of my missing cultural heritage.

The club launches today; please join in! Just play Grim Fandango, hop on over to the discussion forum, and you’re all set.

I’m rereading The Phenomenon of Life, by Christopher Alexander, in preparation for reading the other books in the series. And, again, I’m blown away by it: if the book contained nothing but the pictures in it, it would be worth it.

But, of course, there’s a lot more to the book than pretty (beautiful, profound) pictures: it’s a theory about the nature of life. (He’s not one to hide the ambition of his goals: the subtitle to the series is “An Essay on the Art of Building and The Nature of the Universe”.) While, of course, my first reaction to such sweeping claims is to roll my eyes at them, I just can’t do that here: he asks too many uncomfortable questions for me to simply ignore him.

Over lunch today (Cafe Brioche, yum), I finished the section talking about how the fifteen fundamental properties appeared in living objects. Which got me thinking: where else do they appear? I have go on the brain these days, and if any game is going to show signs of life, surely that’s the one, so let’s test them.

To my readers who are not go players, I apologize for the lack of context for the following. (And I have suggestions for how you can fix that!) I looked for good go pictures, but had a hard time finding ones I really liked; here’s one famous example, but that’s an abstract picture of a position, and of course in Alexander’s context I really shouldn’t be ignoring the actual physical objects involved. (Unfortunately, people who post pictures of go boards on flickr seem to like to take them from odd angles.) Anyways, let’s go through the properties:

• Levels of Scale Individual stones, adjacent stones, eyes, living groups, walls, territories. I can’t really imagine cramming in more levels of scale, given that we only have a 19×19 grid to work with! (And, if we talk about the physical objects involved, there are the lines on the board, the grain of the wood, the grain of the white stones, the room you’re playing in.)
• Strong Centers Thick positions, stones casting influence, the ear-reddening move, the pon-nuki that the proverb tells us is worth thirty points, areas of white or black territory.
• Boundaries The borders between black and white territories, that can be as proportionately thick and contested as any of Alexander’s examples.
• Alternating Repetition I’m not convinced that go games do a particularly good job of exhibiting this property. (The go board itself does, a little too rigidly perhaps.)
• Positive Space Territories expanding against each other. On a conceptual level, the space of two eyes giving life to the surrounding stones.
• Good Shape I don’t think I have to comment on the importance of this to anybody who has played any go at all.
• Local Symmetries The go board and stones exhibit this, of course; I’m not sure that positions generally do in a meaningful way. Though I suppose there are some conceptual manifestations of this idea, e.g. the notion of miai.
• Deep Interlock and Ambiguity White and black positions butting up against each other, a group of one color on the run between two groups of the other color and then, suddenly, turning the tables so the attacker becomes the attacked. (In fact, Alexander has a picture of a go board in his discussion of this property.)
• Contrast Black and White. Life and Death. Thickness and Weakness.
• Gradients I’m not sure the game does a great job of manifesting this.
• Roughness Boundaries between positions are never straight lines. Leaving a position slightly unfinished to move on to other areas of the game. The fact that people don’t place the stones exactly on the intersections: this stone is a little up, that one is a little to the right.
• Echoes I’ll have to think about this one a bit more; I think there’s something to it in the go context, but I’m not sure yet.
• The Void The board at the start of a game. Moyos. Large territories. The fact that (in Japanese rules, at least), you win by enclosing more empty space than your opponent.
• Simplicity and Inner Calm The game’s made out of a board, a grid, and black and white stones, nothing else.
• Not-Separateness The effects that stones have on adjacent stones, that groups have on adjacent groups, that (in the context of a ladder) a stone on one side of the board can determine tactical success or failure on the other side of the board.

Works for me; maybe this Alexander chap is on to something? Makes me wonder if I could improve my go game by concentrating more on expressing his properties.

What about a video game example? (I don’t expect them to do nearly as well as go.) I just finished Half-Life 2, so let’s use it as an example.

And, immediately, I run into a problem. The examples in his book are physical objects; in my go example, the physical positions of stones gave enough grist for my analytic mill that I didn’t have to go beyond that. But just talking about the physical layout of (the abstract space in) a video game leaves out so much of what makes them important! Not sure what to do about that; I’ll follow my nose and see where I end up.

• Levels of Scale Small objects, large objects and characters, vehicles, rooms, buildings, areas, levels, the game as a whole? Shooting a weapon, fighting an enemy, fighting a group of enemies? And perhaps some of my earlier complaints about excessive repetition could be ways in which the game misjudged this?
• Strong Centers The strikingly different character of some of the levels? The clear distinctions between types of weapons? Alyx? Large battle set pieces (boss battles, effectively) punctuating levels? Different places to take cover (with different virtues) in those battles? Maybe the levels could have used more of this in their physical layout, actually: if I’m going through section after section that feels the same, that’s a sign that I wanted more strong centers.
• Boundaries At first, I thought the game did a bad job of the sort of thick boundaries that Alexander is talking about. And there really isn’t that much transition from section to section. Then again, maybe the mini-levels (typically involving hooking up with the resistance) that punctuate the longer levels are an example of this? Or the approach to the prision (with bugs!) is a boundary between the travel level and the prison itself? So now I think there are some, but that the game could use more.
• Alternating Repetition Battle, quiet, battle, quiet. Building, outside, building, outside.
• Positive Space The buildings, the roads and the plazas between the buildings? That works to the extent that, for example, you can enter the areas on either side of a road; to the extent that you can’t, I don’t think roads feel like positive space. So they probably weren’t in the rural levels; the urban levels may or may not have had some positive space, I’d need to have a better global feel for the map.
• Good Shape Hmm, not sure I felt this one too strongly. At least on a larger scale, maybe the individual objects did a better job of manifesting this.
• Local Symmetries Maybe present in the objects/buildings; not sure. Not seeing it on a more conceptal level.
• Deep Interlock and Ambiguity I’m having a hard time finding good examples of this.
• Contrast The “Alternating Repetition” examples? The different feels of different levels? The fight of good versus evil? Not sure.
• Gradients The progression of enemy strength, of the strength of weapons, of the number of options you have available.
• Roughness I think the game does a great job of this in its physical design, the way buildings are lived-in, run down without descending into ruins.
• Echoes Again, I’ll have to think about this one; this may be the property that I understand the least.
• The Void Almost completely lacking (unfortunately the case for most video games). Maybe that’s why the “carried on a track through the Citadel” scene made such a big impact on me?
• Simplicity and Inner Calm Again, pretty much lacking. Though the game did a decent job of sticking to a not-too-large set of gameplay elements. (E.g. the limited set of weapons.)
• Not-Separateness At first, I was going to vote against this one: you couldn’t make choices that had significant ramifications elsewhere, the plot was going to do what it wanted whether you liked it or not. Then I went and read the description of the property, and now I’m not so sure: you are presented in the context of a larger world with signs of its own history. So maybe it’s not as lacking here as I thought.

Hmm. It seems like an interesting enough set of analytical categories, at least. And I suspect game designers could use the list to improve the design of their games. And I’d be very curious to see games that did a better job of bringing out The Void without having it dominate the games. Hyrule Field in Ocarina?

Of course, I came to Alexander through the programming community, specifically through groups influenced by his thoughts on patterns. (Which haven’t really lived up to their potential: Kent Beck seems to be the only person getting much mileage out of building a pattern language at multiple scales.) Can I use these ideas in my programming? Can I tell live code apart from dead code by how well it expresses these properties?

Something to think about. But later; this post is already quite long enough, and I’ll need some time to get my thoughts straight in that area anyways. Time to start looking at code through different lenses, though.

Warning: it is entirely possible that none of my regular blog readers will care about this in the slightest. I’m only writing it in case some other random person out there is dealing with this problem and googles for a solution. In particular, if the next paragraph makes your eyes glaze over, just move on, there’s nothing to see here.

Make has a feature called ‘vpath’. You can use this to tell make to look for your source files in some other directory or directories. We use this at work because, in a few circumstances, source files for one program/library/whatever can be taken from multiple other directories; that may sound like a bad idea to you, and I won’t argue with you, but I also don’t have the energy to eliminate all situations where we do that. And vpath deals with that situation just fine.

Except when it doesn’t. Sometimes, bad things happen for obvious reasons (e.g. identically named files in two different source directories); that’s easy enough to fix. And sometimes, as happened last week, bad things happen for really screwy reasons: like you move a source file, but a mention of the old location occurs in an automatically generated dependency file; that in of itself wouldn’t be a big deal, I know how to write dependency files in such a way as to be able to handle that (yay, gcc -MD -MP), but occasionally make’s vpath machinery can get really fixated on the old location.

And the screwy thing is, when we specify the source files, we specify the directories that they’re located in; it’s just hard to write wildcard rules to handle that without throwing away information and using vpath!

Well, it’s actually not that hard, it turns out. Here’s a sample Makefile using vpath:


all: foo

clean:
    -rm -f *.y foo


Try it out yourself: just create directories a and b, and files a.x and b.x in the corresponding directories. Notice how all the information is there in SOURCES; but then we strip it away to get the %.y: %.x rule to work. Which is screwy.

Here’s how you can avoid it:


all: foo

clean:
    -rm -f *.y foo


(Your web browser is probably wrapping the foreach bit over two or three lines, don’t be confused.)

Which is quite simple, as it turns out. (At least once you realize recent GNU Make versions have eval.)

Some warnings:

• The special variables from pattern rules (@$, @?, @*, etc.) don’t work in the eval version of the rule, so make sure you write the rule directly in terms of its arguments
• GNU Make 3.80 has bugs involving eval, so if you’re using that version (and it isn’t patched by your OS distro), you may want to upgrade to 3.81.

If you’re still reading by now, I, um, salute you? Glad to meet another member of the club of people-who-think-about-make-more-than-they-should…

The SuperMemo ideas don’t seem to be leaving my head, and I’ve finally gotten my todo backlog under control, so I think I’ll take a shot at implementing them. Some notes:

What algorithm should I use to schedule the reminders? I’ll work under the theory that each item that I want to remember is best reviewed at exponentially increasing intervals; see, for example, Wozniak’s Algorithm SM-4. (Though I should note that Algorithm SM-11, his most recent version, seems to drop this assumption.) And, of course, different items are best reviewed at different rates, depending on how easily my brain seems to remember them; let’s call the multiplication factor that we use in the exponential curve for an item its “difficulty factor”. (Where smaller DF = more frequent repetitions = more difficult.)

In SM-4, he suggests that a natural goal is to try to remember items correctly 90% of the time, and that a difficulty factor of 2.5 makes sense as a first guess. In earlier versions of the algorithm, he has what seem to me somewhat complicated ways of tweaking the difficulty factor, but, in later versions, it gets simpler: if you get it wrong, start over reviewing the item from scratch, and increase the difficulty.

Given that, assume you’ve gotten an item wrong W times, and have S winning streaks of 10 correct answers. (So a 20-win streak would count two towards S.) (Though, with the power of exponential growth, a 20-win streak would take decades at all but the hardest difficulty factors.) Then I’ll set the difficulty factor DF for that item equal to 2.5 - .1(W - S). The point here is that ten correct answers balances one mistake, which fits with getting it right 90% (or 91%, I suppose) of the time. And I pulled the .1 more or less out of my hat: I don’t want the difficulty factor to get too close to 1 most of the time, and subtracting .1 each time I get it wrong should fit in well with that goal, based on my experience over the last year with Japanese vocabulary.

With the above definitions, assume that your current winning streak for an item has length n. Then I’ll say that you review the item immediately if n=0, and review it in DF^(n-1) days if n > 0. Actually, I think I’ll change that last to (DF^(n-1)) - .5 days, since I only plan to do the review once a day, and I want to avoid weirdness that might occur if I do it at 8pm one day and at 6pm the next day, or even 11am.

This looks pretty simple: for any given item, I just have to store the next review time, W, S, and n to have enough information to schedule future reviews. (And the fact that one of the items to store is the next review time is very convenient, because that means I can figure out what to review by just doing a simple database query on that column.) I could even store W - S instead of storing both W and S, but I think I’ll keep them separate. One reason for this is that one of the ideas in the original is that different items can be considered more important to learn or less important to learn; if I have something that’s really important to learn, e.g. kana characters, then I could have each mistake count as -.2 instead of -.1, while each 10-time winning streak would still only count as +.1.

That’s the algorithm; what do I use to program the application? One of my motivations is to make it easy to continue my studies when I’m on vacation, or for that matter during my lunch break at work, so I’d like to make this a web app. It looks like a good fit for Rails, and I’d been meaning to get practice with Rails anyways, so that’s what I’ll use. Hmm, what are the best ways to learn about Rails? It’s alleged to be simple enough that I should probably just jump right in and program instead of spending hours reading about it.

The program will have to present questions and answers; each one will be a simple Unicode text field. (And I’ll want to do automatic line wrapping but to preserve manual line breaks.) It won’t affect the programming, but I’ll have to learn a bit more about multilingual input in the various operating systems I use; I’ve done kanji input on the Mac, hopefully it won’t be too hard in Linux, either, and hopefully it will be easy to switch between keyboard layouts on the fly. I can start off with buttons for “show me the answer” and “correct/incorrect”, but eventually I’ll want keystroke inputs for those, to minimize typing. And that’s all there is for the “test yourself” part of the interface; I’ll also need an interface for adding questions and for editing them. (Where the latter will require search functionality.)

If this works well for me, I may well want to let Miranda use it to help with her German; so I’ll have to think about account management, authorization, security. (And if any of my readers are curious, I’d be more than happy to open it up to y’all, too.) OpenID is the trendy up-and-coming authorization mechanism these days; maybe I’ll give that a shot? I would assume that there’s probably some Rails plugin that makes OpenID easy to use.

That should be enough to get me going; in particular, I have a clear next step, to get my hands dirty with Rails, and I have a clear first goal, to get the basics of adding and reviewing items working locally. And then followup goals of searching/editing and of doing this remotely in a not-hopelessly-insecure fashion; I’m going on vacation for a week in August, so I’ll try to get to that stage by then. The multiuser stuff can wait until after that.

Should be a pleasant way to spend some of my free time over the next few months. Andd it will directly address some of the issues I’m having with my current Japanese review: I’m currently managing to review vocabulary items both too frequently and not frequently enough! So I’d really like an algorithm that asked me questions at just the right time.

• Untangle, a flash game about representing graphs as planar with straight edges. (Hmm, are there planar graphs that can’t be drawn with straight edges? Not clear to me.) I linked to another game like this before, but this one’s better.
• TypeRacer, if you’re curious how fast you type. (I’m usually in the 95-100 wpm range, it seems.) (Via Game | Life.)
• Japanese anatomy scrolls from 1819.
• The Piet programming language. (Via Notional Slurry.)

• An amazing video:

  (You’ll have to bear with it a bit, it doesn’t really get going until about 40 second in.) (Via Penny Arcade)

• Japanese highway interchanges. (Via Pink Tentacle.)

• How to crush an egg:

  (Via Vacuum.)

• Pretty pictures. (Via Drawn!.)
• Virtual methods in the CLR and JVM. (Perhaps only interesting to me because I’m so ignorant about the CLR…)
• Samuel R. Delany gets laid more than I do. (Click on “The Polymath”; not even direct-linkable, alas, let alone embeddable.)
• As a manager is explaining that photography is allowed in Union Station, a security guard comes over telling them to turn off the camera…

(Mostly an e-mail to the leandevelopment group, but I figured I might as well stick it here, too.)

I just finished reading The Toyota Way, by Jeffrey Liker. Which I highly recommend: it may actually now be my favorite (non-software-specific) lean book. A clear presentation of a good set of principles; I saw a lot that was new to me, still more that would have been new to me half a year ago (or a year ago or two years ago), I’m sure that I’ll get a lot out of it the next time I read it.

One thing that struck me in the software development context is its principle 13: “Make Decisions Slowly by Consensus, Thoroughly Considering All Options; Implement Rapidly”. It seems to me that there’s a pretty big tension between a naive application of that rule and agile: it sounds to me a lot more like BDUF than traditional agile.

Now, I don’t really believe that that’s true, partly for ideological reasons and partly because I’m sure that most people doing BDUF would do it in a way that is far away from the spirit of that principle. But maybe there’s something to be learned (in both directions?) from Toyota’s slowly building a consensus before deciding as compared to Agile’s quick iterations on development?

Part of the bridge seems to be set-based development. (Which is one concept in the Poppendiecks’ books that I haven’t seen in many (any?) other agile books.) Are there other Toyota techniques in this area that we should take to heart? The chapter talks about A3 reports; I’d been thinking that they might be a good fit to some gaps at work; does anybody have experience using them in software development? (Hmm, the Poppendiecks talk about those, too, don’t they? My copies of their books are at work, so I can’t check.)

One thing about agile that I don’t want to give up is that we’ve got some good techniques for evolving well-designed software. (E.g. Kent Beck’s four rules. Which I wish I could find a good URL for; this is the best I’ve come up with.) Are they in tension with the Toyota principle? Or are they each applicable in different domains?

How do I tell on a software project when we should be spending more time thinking before making a decision and when we should be getting our hands dirty and checking in production code? (With, of course, the expectation that the code will change as our understanding and the requirements change.)

Here’s the book’s description of the principle (emphasis in the original):

The Principle: Thorough Consideration in Decision Making

Many employees outside Japan who have joined Toyota after working for another company have had to face the challenge of learning the Toyota approach to problem solving and decision making. Because Toyota’s process of consensus decision making deviates so dramatically from the way most other firms operate, it is a major reeducation process. New employees wonder how an efficient company like Toyota can use such a detailed, slow, cumbersome, and time-consuming decision-making process. But all the people I have met who have worked for or with Toyota for a few years are believers in the process and have been greatly enriched by it—even in their personal lives.

For Toyota, how you arrive at the decision is just as important as the quality of the decision. Taking the time and effort to do it right is mandatory. In fact, management will forgive a decision that does not work out as expected, if the process used was the right one. A decision that by chance works out well, but was based on a shortcut process, is more likely to lead to a reprimand from the boss. As Warren explained in this chapter’s opening quote, Toyota’s secret to smooth and often flawless implementation of new initiatives is careful, upfront planning. Underlying the entire process of planning, problem solving, and decision making is careful attention to every detail. This behavior is associated with many of the best Japanese firms and Toyota is a master at it. No stone is left unturned. In fact, every stone is inspected under a microscope.

…

Thorough consideration in decision making includes five major elements:

• Finding out what is really going on, including genchi gembutsu.
• Understanding underlying causes that explain surface appearances—asking “Why?” five times.
• Broadly considering alternative solutions and developing a detailed rationale for the preferred solution.
• Building consensus within the team, including Toyota employees and outside partners.
• Using very efficient communication vehicles to do one through four, preferably one side of one sheet of paper.

(Liker, The Toyota Way, pp. 238–9)

I finished reading The Art of Agile Development, by James Shore and Shane Warden a few weeks ago. It’s a quite good book: if you’re looking for a well-written, prescriptive guide for how to do XP, this is what I would recommend.

Though I won’t go into the book in general any more than that. But, a few thoughts that it triggered:

One thing I liked about the book’s discussion of “once and only once” (pp. 316–317) is that it spent some time on the first “once”. I’m used to thinking of the sentence in terms of eliminating duplication, which is great. But the first part of that sentence is important, too: as they say, “don’t just eliminate duplication; make sure that every important concept has an explicit representation in your design”. That way, you have a place to store behavior related to that concept: e.g. having a Dollar class may not seem like a big win over, say, just representing dollar values as integers, but it gives you a place to stash code for formatted output, or parsing, or conversions to related types.

I also really liked the discussion of levels, rhythms, and frequencies of refactoring in the “Incremental Design and Architecture” section. (pp. 321–330.) They distinguish between standard incremental improvements and “breakthrough” changes: the latter are “idea[s] for a new design approach, which will require a series of refactorings to support it. … Breakthroughs happen at all levels of the design from methods to architectures.”

After which they talk about how frequently you do various sorts of refactorings, and how to make time for them. At the method level, “Method refactorings happen every few minutes. Breakthroughs may happen several times per hour and can take 10 minutes or more to complete.” At the class level, “Class-level refactorings happen several times per day. Depending on your design, breakthroughs may happen a few times per week and can take several hours to complete.”

The latter raises the question: several hours is enough to make a noticeable blip in your team’s work for the week, so where do you find the time, if you see something that looks valuable but isn’t on the shortest path to completing a feature? Their answer: “Use your iteration slack to complete breakthrough refactorings.”

And then there are architectural breakthroughs; I’ll quote them at length on the subject (p. 325):

In my experience, breakthroughs in architecture happen every few months. (This estimate will vary widely depending on your team members and code quality.) Refactoring to support the breakthrough can take several weeks or longer because of the amount of duplication involved. Although changes to your architecture may be tedious, they usually aren’t difficult once you’ve identified the new architectural pattern. Start by trying out the new pattern in just one part of your design. Let it sit for a while—a week or two— to make sure the change works well in practice. Once you’re sure it does, bring the rest of the system into compliance with the new structure. Refactor each class you touch as you perform your everyday work, and use some of your slack in each iteration to fix other classes.

Keep delivering stories while you refactor. Although you could take a break from new development to refactor, that would disenfranchise your customers. Balance technical excellence with delivering value. Neither can take precedence over the other. This may lead to inconsistencies within the code during the changeover, but fortunately, that’s mostly an aesthetic problem—more annoying than problematic.

Definitely something to think about: over the last few years, my team has made some progress in terms of lessening the technical debt that we’re adding to the system, but we still haven’t done nearly as well on that as I’d like, and we’ve done a bad job in terms of actually eating away at the technical debt. (As you might suspect from my using the term “last few years”.) We’re just not good at maintaining a good refactoring rhythm at the various scales that are necessary; maybe the authors’ advice on the subject will help, by giving an idea of what our behavior would look like if we were successful.

Note also the importance of slack in carrying this out successfully.

Warning: While I don’t intend for anything here to be a spoiler for the “primes and fractions” post I wrote before it (the post following this if you’re looking at the front page), it’s possible that you’d have more fun working out that brainteaser if you didn’t read this post first.

At the time when I wrote my last post, I’d gotten as far as figuring how the fractions in question turned into a computational machine of a much more traditional form than the form in which it’s stated there. After writing the post (and eating lunch), I wrote down that machine on paper, and went through a series of simplifications. (Refactorings, basically, with the extra benefit that I could increase the power of my computational machine, because I didn’t have to restrict my actions to those that could be carried out by the original computational model.)

After an hour or so of that, I had a much simpler model; I couldn’t yet completely see how the prime numbers were going to appear, but I’d gotten a lot closer, and in particular the concept of divisibility had appeared. Hoping to get a bit more insight, I decided to run a few dozen iterations of my new, simpler model.

And, of course, I started getting bad answers; if I’m remembering correctly, I ran into an endless loop or something, and at any rate I certainly wasn’t computing prime numbers. Oops.

Which was frustrating. I mean, I’d been pretty careful, breaking the work up into explicit steps, double-checking my work after each one (and frequently finding mistakes, which in retrospect should have raised warning signs). But I was pretty obsessed by then, so I wasn’t going to let a little problem like having a mistake somewhere within the last hour or two of tedious work stop me!

To recap my progress so far: I started with a computation, I transformed it to simplify it and to improve my understanding, and I made mistakes. Which is something I have experience with when programming, and I already know the answer in that context: I should be doing formal refactorings backed up by running tests after every step. I’d been doing an okay good job of the former, but the latter was too laborious to carry out frequently by hand. (For that matter, it’s just as easy to make mistakes when running through the algorithm by hand to test it as it is when transforming it by refactoring!)

So I brought my work over to the computer: I wrote a Ruby version of the initial state machine, with a driver program that ran through as many iterations of the steps as I wanted. And then I refactored the state machine, reloading it into irb after every change, and saving it into a new file when my little changes added up to a more significant one (e.g. eliminating a state), just so I could look back on what I was doing. I didn’t have formal unit tests: I just ran the program for a few thousand steps after every change and checked that the primes were appearing, on the theory that I was unlikely to make a mistake that let the algorithm still work. (In retrospect, maybe that was a mistake: it would have been easy enough to turn that manual test into an automated test.)

And it went a lot faster. I didn’t have to spend as much time double-checking my work: I could just make changes and run the test. I could work in smaller steps, so where on paper I would have composed two transformations in my head (involving changes to multiple variables at once), I could just write them out as a program one after the other and then merge changes to variables one at a time. If I wanted to eliminate a state, I’d just paste the code for that state into every place where it was used, and then do transformations after that. If I wanted to reorganize the control flow by moving a block into all the branches of a conditional, I’d just paste it into all the branches, then simplify the branches one at a time.

I still made mistakes, but even without the tests I made fewer mistakes than I’d been making on paper: it’s really a lot easier to avoid mistakes if you just cut and paste whenever you’re moving operations around, and then break up your subsequent simplifications into smaller steps. (At some point I cross-referenced my work with my paper copies; I’d made two important mistakes pretty early on, and in retrospect I’m surprised that so much of the mathematics managed to survive in the paper version even with those mistakes.)

So now I have a proof; yay. But I’m actually thinking that there are some lessons here that the mathematics community could learn from: I worked faster when I started using the same techniques that I use to write correct, readable code as quickly as possible, and I made a lot fewer results at the same time. And who wouldn’t want that? In particular, I think an automated engine that lets you correctly carry out refactorings on steps of your proof could help a lot, done right: it could let you do the relatively mindless work faster, so you could spend more time thinking at a higher level, and you could spend more time quickly exploring different paths in that mindless work to see if any of it leads somewhere unexpected.

And you would make fewer mistakes. My impression is that most math papers contain mistakes in their proofs; this doesn’t necessarily mean that the results are wrong, but sometimes it does, and even if they aren’t, it’s still good to get the details right. It would also make it easier to be honest with yourself about when you’re handwaving and when you’re being precise; I certainly wouldn’t say that handwaving is inappropriate in all circumstances, but you want to take care when doing it. (What I’d really like is for papers to be published in a format where the printed version contained the steps that are most useful for expository purposes but where there’s a more complete computer-verifiable version of much or all the proof.)

This isn’t a new idea, of course; I worked on a project called “VLISP” the summer after my freshman year of college that, among other things, had a theorem verification engine, and the idea wasn’t new then, either. (The proof of the four color theorem was almost a decade and a half before that.) So why hasn’t it taken off? I’m not sure, but here are some ideas I have:

• It’s hard to translate even relatively formal human reasoning into something that a computer can work with.
• Partly because of that, and partly because mathematicians are frequently reductionists at heart, people have been tempted to concentrate on the oldest, most well-trodden parts of mathematics, which means that their engines aren’t going to be of use to the vast majority of working professional mathematicians.
• People have conflated the concepts of “automated refactoring” with “a complete computer-verified proof (from first principles)” and (worse yet) “computer-generated proof”. I don’t think the latter is any more likely to be useful (right now) than having a computer automatically simplify the former for me; as for the former, a computer-verifiable proof starting from first principles is a nice jugendtraum, but you’d be a lot more likely to make an impact if you could produce a tool that could automate certain parts of proofs in ways that helped actual working mathematicians, even if it couldn’t automate anywhere near all of it.
• Automated refactoring tools only showed up in programming settings a decade and a half ago, and still aren’t widely used. At least I think they aren’t; I’ve never used one, and while most IDEs these days contain a few refactorings, I don’t get the impression that most programmers use them a lot. (Which is one reason why I lean so much on automated tests; in mathematics, I think the automated tests would be a good deal harder to carry out, but fortunately I think automated refactorings would be easier.) So it’s not surprising that they haven’t made an impact in other fields yet.
• Writing a tool like this that is easier to use than paper and pencil would be a lot of work, and require very good judgment.

For all I know, people are making a lot of progress in this area, and I’m just not aware of it. But I think there’s something there: I think our understanding of these problems in the programming realm has advanced enough that we probable have something to teach mathematicians. One of these decades, somebody will push the tools far enough to where using them starts to become almost as fast as working by hand, at which point people will get a lot more interested.

In the spirit of “every long e-mail I send somewhere should be shamelessly recycled on my blog”, I present some random thoughts on testing.

Why do we release products with defects that we weren’t aware of? This is a sign of flaws in our testing; two possible causes are:

1. We don’t know what to test for.
2. We do know what to test for, but we’re not able to do enough testing before release.

For 1, how can we figure out where our blind spots are? Some tactics:

• Defect clusters.

If we can figure out in what areas we’ve historically had a large number of post-release bugs, then we can increase our testing in that area in future products. So if people can come up with useful suggestions for analyzing post-release data, that would be very useful.

• Different classes of tests.

One of the most interesting testing ideas I’ve seen over the last couple of years is the idea that you can analyze tests along two dimensions: are they business-facing or technology-facing, and are they intended to support engineering or to critique the product? (The idea comes from Brian Marick, I blather on about it elsewhere, and there’s also a section on it in Implementing Lean Software Development.)

This gives four quadrants. Technology facing tests designed to support engineering are unit tests, tests that narrowly focus on a specific internal interface. Business facing tests designed to support engineering are tests that are focused on a certain aspect of customer-visible behavior. Technology facing tests designed to critique the product are property testing, tests for “nonfunctional requirements”: load testing, security testing, combinatorial testing. And business facing tests designed to critique the product are various sorts of manual poking around: usability testing, exploratory testing, etc.

I know that, in the past, I’ve had huge blind spots in these quadrants. And we can gather data to figure out which quadrants we might be missing: if we’re either not implementing known basic requirements or taking too long for the product to stabilize its functionality in those basic requirements, then we might be missing tests in the two “support engineering” quandrants. If we’re running into lots of corner case bugs or stress bugs, we’re missing property testing. And if we’re producing products that behave according to spec, but isn’t what the customer wants, then we’re missing tests that are business facing and designed to critique the product.

The above assumes that we don’t know what to test for; what if we do know what to test for, we’re just not doing a good enough job? Here, testing is a bottleneck, and we want to speed it up. At least, it might be a bottleneck: it may also be the case that something else is a bottleneck, creating schedule pressure that isn’t caused by testing, and testing gets unfairly shrunk because it comes at the end of the development cycle. But, for now, let’s assume testing is a bottleneck.

There are certain obvious knobs we can turn here (hire more testers, build more machines to test on), and that may be what we have to do, but those knobs cost money. So we should also look at the testing value stream with lean eyes figure out where we can find waste, and eliminate as much as possible.

To that end, some questions:

• Are there manual tests that can be turned into automated tests?

Doing this would have three benefits:

1. If availability of human testers is a bottleneck, this helps alleviate that bottleneck.
2. Automated tests are generally faster than manual tests.
3. Engineers developing the product can run the tests more easily, which means that they can find defects sooner after introducing them, which has no end of benefits.

• Are there tests that can be sped up?

One technique that works really well on the software side is to directly test the smallest software interface relevant to the issue in question, instead of starting up the whole system: this can turn a 5 minute test into a 5 millisecond test. For example, every time I check in software, I first run a suite of 5000 or so automated tests; if I had to actually run the whole StreamStar system for each test, that would take weeks, but as it is it takes 15 minutes to run all 5000 tests. (And I wish they were faster than that!)

To be clear, we do have other tests that run the whole system. But, to return to the four quadrants above, try to move as many tests as possible to the “support engineering” side (by turning them into tests of clear functional requirements), and try to move as many of those as possible to the “technology-facing” quadrant (by shrinking the interfaces they test). You still need all four quadrants, but that’s the quadrant where you get the most bang for your time.

• Is the test analysis taking too long?

Maybe the problem isn’t with running the tests, it’s with making sense of the results of the tests. Do the tests give a clear pass/fail result? Failing tests take more time to analyze than passing tests (among many other problems, e.g. one bug can mask another); do we have too many failing tests? Do the tests not generate enough information in the failure case to make analysis easy (e.g. so you can tell different known bugs apart, or known bugs apart from unknown bugs)?

• Is the test writing taking too long?

If so, we should invest more time in test frameworks.

• Are people or machines idle inappropriately?

This is a dangerous issue to approach, because you don’t want to do makework for the sake of makework: for best utilization of a system, you should work your bottlenecks at as close to 100% as possible but explicitly allow slack in all other components. Having said that, sometimes waiting is just plain waste. For example, if you’re low on test machines, you want to separate running tests from analyzing tests as much as possible, so you can keep the machine busy running the next test while you’re still analyzing the previous one. (But if you’re not low on test machines, then if you can speed up the test writing/analyzing process by hogging the machine for a while longer, that’s a better choice. And still better is to make the writing and analyzing as easy as possible, so you don’t have to make that choice!)

• Do people have time to think about what they’re doing?

Overworked people make mistakes; even if they don’t make mistakes, it’s hard to devise a method to cut testing time in half in some area if your boss is harping on you to get dozens of things done today.

• Are good ideas spreading through the group?

We need a way to identify our best ideas and to get them adopted broadly.

• I still haven’t figured out Cursor*10, but it has some neat ideas. (Via GayGamer.)
• An ActiveRecord tutorial; a nice little introduction to the concepts, I liked the way they get their hands dirty with showing you the SQL.
• Yeah, that’s kind of the way I feel about Obama…

• For no particular reason, a random bit of Sesame Street:

  (Via Cosmic Variance.)

• Who am I? I’m Andrew F***ing Jackson! (Via The Edge of the American West.)

• Crazy things with soccer balls:

  (Via Tim Ferriss.)

• Quite a case mod.
• Tim Bray on people involved in the XML standard, ten years after its ratification.
• Some charming, subdued flash games. (The web site isn’t working as I type this, but it was as of yesterday…) (Via Game|Life.)

We’re writing an IDL compiler at work, to generate C++ code from a file containing a list of attributes. We’re doing this in TCL, which turns out to be surprisingly easy and fun: just define some procedures that correspond to the syntax of your language, eval the contents of the file, and you’re off and running!

The syntax is pretty simple, looking like this:

  attribute int foo
  attribute string bar

(Well, that’s what it started out looking like; there have been some changes, as we extended the language to handle more cases.) Loading this invokes a procedure attribute that sticks the names and types of the attributes in some lists/arrays. And then we call some output code that does stuff like this:

  proc add_members {} {
    global attributes types

    foreach attribute $attributes {
      set type $types($attribute)
      puts "  $type ${attribute}_;"
    }
  }

(I’m not cutting and pasting that, I may have some syntax details wrong. So don’t treat this post as a TCL tutorial!)

But after sticking in “foreach attribute $attributes { ... }” in a few places, it looked like a refactoring opportunity: I should define a procedure each_attribute that iterates over the attributes and binds the attribute and type to a local variable.

Here’s what the code currently looks like:

  proc each_attribute { block } {
      global attributes types

      foreach attribute $attributes {
        uplevel [list set attribute $attribute]
        uplevel [list set type $types($attribute)]
        uplevel $block
      }
  } 

  proc add_members {} {
     each_attribute {
        l "    $type ${attribute}_;"
    }
  }

Certainly the new definition of add_members is a lot shorter!

Some thoughts:

• I like using uplevel! For people who don’t know TCL, curly braces are a string quoting construct, so the block after each_attribute in the body of add_members is actually a string. And uplevel evaluates that string in the context of the calling function, letting us define new control structures.
• At first, I went back and forth about whether each_attribute should take another argument, namely the names of the variables that we want to bind the attribute name and type to. Ultimately, though, that would have let to writing each_attribute { attribute type } { ... } over and over again; why force people to constantly repeat those variable names? (Especially since a call to each_attribute within the block of another such call doesn’t make sense.)
• I do, however, feel guilty about injecting those variables directly into the calling scope: what I really want to do is create a new contour within the calling scope and have the new variables only defined there. Anbody know offhand how to do that in TCL? Can’t be too hard, I just need to look it up

• Pretty sand pictures:

  (Via Backreaction, which has links to more.)

• Those are some big crystals.
• Van Jacobson on networking. More than an hour long, which is why it took me more than a year to get around to watching it, but there’s a lot of interesting stuff here.
• You’re probably already all aware of Why’s (Poignant) Guide to Ruby, but if not, you should be.
• I’d never heard of High Dynamic Range photography before seeing these gorgeous photographs of various scenes in Japan.

• Fighting giraffes:

  (Via Mad Prime.)

• A different take on Zack & Wiki.

I’m hiring again. If you live in the S.F. Bay Area, are a good programmer, and want to be the first kid on your block to stream out 320Gbps of video data, please let me know. (You can also submit a resume via the above link.)

I’ve now written my first AJAX code: if you go to a random web page in my book/game database, you should be presented with a list of blog posts that refer to that item. At least assuming that I haven’t accidentally used functionality that your favorite browser doesn’t support, which I hear is easy to do with JavaScript; fortunately, Internet Explorer seems to be the most likely candidate, and my CSS is already broken there, so I should be safe enough. (I’ve only tested under Firefox and Safari 3.)

It was fun and not too hard, all things considered. I didn’t have any prior JavaScript experience, but I figured that googling would quickly turn up instructions for how to do what I wanted. Which didn’t seem to be the case in the first 15 minutes or so of searching, but I remembered getting the idea from an example in the REST book, so I looked that up, googled some of the more specific language constructs I had questions with, and had something working in another hour or so.

Aside from stupid mistakes, the main initial difficulty that I ran into was in the guards against cross-site scripting vulnerabilities: my blog has its own domain, while my database is in bactrian.org. (And I was doing my initial prototyping on my home computer, which is a different physical machine.) The easiest way to get around that seemed to be to set up a proxy (or rather, two of them, one on my home computer and one on bactrian.org); a bit of mod_proxy configuration later and my prototype worked on my home computer.

I copied the prototype over to bactrian.org, and updated paths; it stopped working, again giving me a frustrating error message related to cross-site scripting. I couldn’t figure out what was going on, and spent a quite frustrating hour or so alternating between googling for help and trying to install and run a JavaScript debugger. (For some reason, the Venkman package in Ubuntu didn’t work for me.) Eventually, though, I remembered one more path that I would need to translate when moving the prototype; I changed it, and the prototype worked in the deployment environment.

After which it was a simple matter of programming to make the change and update the tests. Most of the work was in the latter: I have the web page skeleton abstracted fairly well out of the tests, but even so I have to modify a few different places if I change the output in a way that affects all pages. And I ran into one more road bump along the way, where inserting some white space in the HTML turned out to require me to change my JavaScript, but I figured that out pretty quickly.

So here we are. I’m really pleased with the results: there’s a big difference between being invited to click on a link to search for related blog posts (which may not even exist) and having a list of posts appear in front of you. This is the last change that I plan to make to the database for the time being (well, I might do a bit of trivial tweaking); a good change to go out on.

Random JavaScript-related thoughts:

• Based on my limited experience, JavaScript is pleasant enough. I wasn’t impressed with its collections, but other than that the language behaved in the way I wanted, and it was quite easy to search through XML and pop some data into view.
• I’m still mostly at sea if something goes wrong with my JavaScript. The console gives some basic help, but there were a couple of instances where I ran into a more serious problem and either wished that I could get more information out of the error or poke around data structures or something. Probably the debugger would have helped, if I’d gotten farther with using it.
• I don’t have any acceptance tests for this, which makes me sad. It’s little enough JavaScript code that I’m happy to skip unit tests for it, but I really would like to be able to push a button and have reasonable confidence that I haven’t broken anything. (Especially since the functionality depends both on my JavaScript code and on WordPress’s behavior, so I’m going to have to manually test this every time I upgrade WordPress.) Some people on the XP list suggested some tools (Selenium in particular); maybe I’ll give that a try at some point.

From 37 Signals’ Getting Real:

People often spend too much time up front trying to solve problems they don’t even have yet. Don’t. Heck, we launched Basecamp without the ability to bill customers! Since the product billed in monthly cycles, we knew we had a 30-day gap to figure it out. We used that time to solve more urgent problems and then, after launch, we tackled billing. It worked out fine (and it forced us into a simple solution without unnecessary bells and whistles).

Not sure I would have thought of that strategy myself. Or had the courage to follow through if I did…

From Kent Beck’s Implementation Patterns (p. 20):

Once a set of implementation patterns has become habitual, I program faster and with fewer distracting thoughts. When I began writing my first set of implementation patterns (The Smalltalk Best Practice Patterns, Prentice Hall 1996) I thought I was a proficient programmer. To encourage myself to focus on patterns, I refused to type a character of code unless I had first written down the pattern I was following. It was frustrating, like I was coding with my fingers glued together. For the first week every minute of coding was preceded by an hour of writing. The second week I found I had most of the basic patterns in place and most of the time I was following existing patterns. By the third week I was coding much faster than I had before, because I had carefully looked at my own style and I wasn’t nagged by doubts.

I would seem to be more confused than normal these days. Which, in the past, has frequently been a good sign; maybe my brain is figuring something out? Or maybe I’m just clueless. Anyways, I present to you a random collection of thoughts, which may or may not be related to each other in some way.

• At work, I think we’re doing a reasonable job of adding new features. Though I’m sure there’s room for improvement.
• I also think we’re doing a reasonable job of fixing bugs: acceptance test failures are way down, and we’re even successfully attacking sporadic bugs and old, thorny bugs.
• Not so sure about code maintainability: there’s even some evidence to suggest that our code maintainability has, in some areas, gotten worse recently.
• Code maintainability is harder to measure than features and bugs. And there’s less external pressure to get it right, so not surprising that it’s fallen by the wayside. Because of our successes in other areas, and because we’re doing a better job of planning this release than the last one, though, we have some time to attack it.
• I wish I were better at helping various teams that I’m part of improve our processes.
• One-on-ones are a good idea, even (especially?) if you don’t know what you’ll get out of them. And the more frequently you have them, the lower the pressure, which is a big help to everybody.
• The book I’m currently reading at work is Matthew May’s The Elegant Solution. Which is reminding me of some aspects of lean that I hadn’t been focusing on, especially on the “respect for people” side.
• Having the team all focused on the same, small-granularity tasks is wonderful in terms of making concrete progress in ways where our work reinforces each other and matches business value. Not necessarily so great in terms of letting people, say, focus on what they do best or define their own job.
• One thing that May talks about is the power of opposing goals (make a car faster and get better mileage and lighter and cheaper by these specific amounts), and the evils of satisficing. Simultaneous satisfying all your goals sounds wonderful if you can do it; I wish I knew how. I suspect that Toyota has some very useful techniques to this end.
• Alexia Bowers gave a good examples of meetin opposing goals, if I’m remembering the podcast correctly.
• The book before last that I read was a guide to the ToC thinking tools. (See also It’s Not Luck.) Do these actually work? My brain is strangely resistant to even giving them a try.
• I think I’m getting better at not talking in meetings, about chiming in and then letting other people argue for a while. Gratifying that, not infrequently, other people make the arguments that I would have made were I talking.
• I stayed home on Friday, because Miranda was sick, and called into two meetings. Both of which were very frustrating. I think part of it was that I missed some of the cues of the flow of the meeting, and part of it was that I wasn’t very good at explaining, or even seeing, how we were talking past each other. (I did think of an evaporating cloud to explain one of the conflicts after the fact, for better or for worse.)
• I wish I spent more time talking to people in other parts of Sun.
• What do I want to do when I grow up?
• After taking a break for a few years, I’ve gone to one conference each of the last two years, and gotten a lot out of each of them. I should continue this going forward. (And possibly even ramp it up a bit, since if there are further Agile Open Californias, I’m not going to stop going to them.) Where should I go next year?
• What communities do I want to be part of? What does it mean to be part of those communities?
• What teams am I currently part of? Do those teams behave how I think a team should behave? If not, how should I behave?

I could probably ramble on in this vein for quite some time; time to go to bed. Happy Armistice Day, all.

• xkcd:
• One person, one vote in the Texas legislature: (Via Lawrence Lessig.)
• Starlings flocking; it starts to get good at about 2:00, though if that’s not good enough for you, go to 3:20. (Via Cosmic Variance.)
• Weird Hand-Drawn Game certainly qualifies on the first two counts… (Via Game|Life.)
• I don’t have much to say about The Wealth of Networks, other than to second Lawrence Lessig’s recommendation and to say that it’s useful to be reminded that, from the point of view of mainstream economy taking a snapshot of one isolated state the market, copyright makes no sense. (The book is available for free, by the way.)
• Synesthesia meets color blindness.

Also, some notes to myself: these are links that have stuck around as saved items in my blog reader where I can’t imagine what will either trigger me to act on the information therein yet where I want to keep them around somewhere. So I’m moving them here.

• Ergonomic keyboard recommendations.
• Martin Fowler on board games.
• Zelda reorchestrated. (Via Opposable Thumbs.)
• OpenID notes.

I’ve finally finished converting dbcdb from Java to Ruby. I’ve been using the Ruby version of the tool to write the database for about four months, but I’d still been using the Java version to write the web pages.

Nothing too deep going on here; I was actually done with everything but the indexes as of the middle of September, but I hadn’t gotten around to generating the indexes until this weekend. (Or do people prefer that I spell it ‘indices’?) We’ve been busy with some extra event every single weekend for about the last two months; combining that with wanting to learn Japanese, working through Metroid and Picross, and occasionally working on the game with Miranda means that, unless I’m feeling extraordinarily disciplined, dbcdb falls by the wayside. But we had nothing planned this weekend, so I seized the opportunity.

The new code is a little more than half as long; the acceptance tests also run almost twice as fast. (All that JVM startup takes time, I guess? I don’t think there are significant algorithmic performance variations in the two versions.) Go Ruby, though I’m sure it would be very easy to find situations where the performance goes the other way. Both generate the exact same output, as manifested by running the same acceptance tests on both versions and on, ultimately, doing a diff -r on both outputs from the current live database contents.

What next? There are some cosmetic tweaks I may or may not get around to making; I’m not feeling any urgency on that score right now. I had planned to next convert this from generating static web pages offline to generating them dynamically via mod_ruby; now I’m feeling distinctly less interested in that idea. (Partly because the REST book reminded me of some of the benefits of static web pages, ironically.) I still want to experiment with that at some point, but now I’m thinking I’ll just do that by coming up with a Rails project instead of doing everything from scratch.

So it looks like it might be time to declare this a success and move on. And it has been a success, no question: I’ve brushed up on my Java a bit, dabbled with SQL, learned Ruby, and basically enjoyed myself. So, from a purely didactic standpoint, I’m quite happy.

There is one thing that I’m not happy with, though. I’d originally envisioned the generated web pages as actually being useful in that they’d provide an index into my blog posts: they would give an easy way for people to find all the web pages where I write about a given game, say. And they do provide an index, but it’s not as easy as I’d like: people have to click on the link to the database and then click from there to a search link, and that’s expecting quite a bit from my readers. (Especially since there’s honestly nothing of particular interest on the database web page itself.)

So I’d like to remove one of those links, to compress it down to one level. In this AJAX-aware world, the mechanisms for doing that are pretty well-trodden: write some JavaScript to do the query in the background, and then stick the results in the database web page. And, in fact, it turns out that WordPress can generate an RSS feed of query results, so I don’t have to worry about page scraping and having details change as I upgrade my WordPress installation. (Which I should really do one of these days - I’m still on 2.0…)

One last task, then. Which is made a bit harder than it would otherwise be by the fact that I don’t know how to write JavaScript, I’m not familiar with the DOM model (if indeed that’s the right term to use), and I don’t know how to acceptance test AJAX. But I’m not particularly worried about either of these: like I said, this is a well-trodden path, so it shouldn’t be very hard to find examples that do pretty much exactly what I want to do.

And now to some actual content from Agile Open California. As I mentioned before, I hosted a session called I Don’t Like Pair Programming, since the topic had been on my mind after our team meeting the previous week.

The title isn’t really accurate: I usually enjoy pair programming when I’m doing it, but my brain nonetheless persists in not looking forward to it. Which is weird—I would like to think that my brain is generally more sensible than that—so I thought I should ask for help understanding myself.

My guess is that it was the single most sparsely attended session at the conference: there were about five of us there. But I really enjoyed it, and am quite grateful to the others who showed up. The main piece of psychological advice that I picked up was contributed by Rob Myers (who learned it from the AYE Conference): it’s not that introverts don’t like people, it’s that we need time alone to recharge. Which suggests that, perhaps, if I had, say, one or two hour-and-a-half pairing sessions a day, I’d be more likely to enjoy (or less likely to feel trepidation towards) pair programming than I do in my team now, where a pair typically sticks together until either the day ends or the card ends. This would require other changes in the way my team operates, but I suspect they’re all to the good: we hog cards too much as is.

Steve Bockman also gave us the following pairing mechanics suggestion: pass the keyboard every time you write a test, get a test to pass, or refactor. Which sounds like fun to try; I don’t think that I’m bothered by keyboard hogging, but other people probably are, and I’m certainly curious what the experiment would feel like.

The other thing I wanted to talk about from the conference: for whatever reason, some discussion that we had during a session that Rob hosted on “Maintaining a Collegial Environment in a Competitive World” got me thinking about teams transitioning to agile. (No idea how the topic led to this thought; which is a sign of the strengths of the conference.) Some people (e.g. Ron Jeffries) strongly suggest that, if you’re intrigued by XP but aren’t sure it’s all a good idea, you should just do it by the book for a while, and see how it works. (As opposed to starting with the parts that make sense to you, and then maybe adding in more stuff and maybe not.)

But we can juxtapose this against two other ideas from the agile and lean worlds:

• When dealing with legacy code, there’s a temptation to just rip it out wholesale. But that’s a good idea much less often than you’d think: it’s often much better to accept the code and gradually refactor and add tests as you need to touch it.
• Taiichi Ohno has suggested (I believe, I’m too lazy to dig up a reference) that you shouldn’t start with a standard work specification that’s too good: it’s better to start quickly with something mediocre, among other reasons that it will be easy to find ways to improve it!

I’m not going to take either of these as suggesting that, in general, Ron is wrong: for one thing, sometimes the right thing to do with legacy code is to throw it out, and I’ve read stories from the lean world about Japanese experts coming in, making drastic changes, and getting immediate results. But what they do suggest is that, rather than focusing on getting the details of the situation right immediately, better to focus on how to improve the situation in a disciplined matter. In other words, refactoring supported by testing in the code world, continuous improvement via formal experiments in the process world.