malvasia bianca

I’ve been really curious about lean manufacturing (which basically means the way Toyota does things) for a couple of months now. I was aware that people had made some analogies between it and agile software development, but my interest got more concrete when I started reading Silk and Spinach: that’s a blog that spends a lot of time talking about that analogy, using various Japanese terms to talk about high quality, continuous improvement, and other things that sound productive to me when developing software. From there, I learned about Evolving Excellence, a blog about lean manufacturing itself, which I also quite like.

I just finished reading Toyota Production System, the key original source book on the subject. Which wasn’t at all what I expected: I didn’t see any references to the specific Japanese words that I was curious about, and in general there wasn’t the explicit laser focus on high quality. (Though I can see how it developed: more below.) Instead, there was the term kanban, which are cards used for resource/production management whose use I’m only starting to understand, and a laser focus on eliminating waste.

Which they think about in quite interesting, and I think quite productive ways. As far as I can tell, Toyota’s ideal would be that, when a customer walks into a car dealership and buys a car (or maybe takes it for a test drive), a car would magically be created instantly on the spot, while nothing would be invested on building the car until that very moment. This is referred to as a “pull model” of production, as opposed to a “push model”, where forecasts drive production. And anything that doesn’t help with this goal is considered waste.

This is, obviously, impossible in real life, but it does give you a rather different way of looking at waste. We can all recognize that people standing around doing nothing is wasteful; what is less obvious is that focusing on having people work full time can also cover up waste. This is the waste of overproduction: the easiest way to have people work full-time is for them to produce products that aren’t immediately needed. And there are second-order effects, too: to enable those people to work full-time, the easist thing to do is to make sure that they always have an abundant supply of their necessary raw materials (e.g. the components they’re assembling) at hand, which leads to overproduction in the earlier stages, products that aren’t immediately needed.

And this is bad for (at least) the following reasons:

• That excess inventory needs to be stored somewhere, incurring warehouse costs.
• If demand for the excess inventory never materializes, the effort spent producing it was wasted: the people creating the excess inventory could potentially have been working on something valuable, or not been hired in the first place.

Analogies have problems, and in particular it’s not clear to me exactly what lesson the first point has for software development, but certainly the other point is relevant. So now I’m wandering around work trying to figure out where I’m committing the sin of overproduction, where there’s waste that I hadn’t been seeing.

To get back to our story, the lean solution to this problem is to manufacture everything “just in time”. They still want to keep their pipeline flowing as smoothly as possible: they just try to use buffering as little as possible while accomplishing that. This is the (or at least one) source of the lean focus on quality: if you have a defective part, you don’t have extra parts that you can use to replace it, so it stalls your entire pipeline. So you’d better work hard to not have defective parts in the first place!

Next in the reading list is Lean Software Development. Its authors have experience both in manufacturing and software development, and are quite aware that the two are very different things. Their thesis is that software development is more like product development than manufacturing; fortunately, Toyota has something to say here, too. But I am planning to read more books about lean manufacturing itself; it seems quite interesting, totally aside from its potential applications to software development.

Toyota has a few videos on the subject available from their web site. I enjoyed them, though I’m not sure I would have gotten much out of them if I hadn’t already been reading about the topic.

I’m too cheap to get a fixed IP address, and at some point recently, my router’s external IP address (i.e. the one assigned by my cable provider) changed. So I’d like to write a cron job that runs every hour and dumps my external IP address on a computer somewhere else, so I’ll always know how to connect to it.

I’m blanking on a good way to figure this out from within a cron job. If I just ssh ‘lastlog’ or ‘who’ to the remote computer, I only get my last actual login. If I try to mimic sshing in for real in a naive way (e.g. ssh $REMOTE_HOST << EOF) then I get complaints because stdin isn't a terminal. I'm pretty sure I could go down that path a little longer and fix it, but I get the feeling that I must be missing some much simpler solution to this problem: the information I'm trying to get isn't exactly a big secret, after all. And no, I don't want to log into the router and ask it, if for no other reason than that I've set up the router's administrative interface to only be accessible via a wired connection, which isn't possible from the computer where I want to run the cron job.

And now Jane Jacobs is dead. Sigh. Go out and read The Death and Life of Great American Cities, everybody.

I really like this picture of her.

One thing I forgot to mention in my last post: the reason why my boss was so concerned about integration testing was because of customer feedback. Which is a perfect example of why customer representatives should be involved in writing tests (instead of, say, just handing off requirements and leaving it up to the engineering team to provide the tests): if we use the analysis from that post, then customer representatives have insights into how it’s important for a test to be realistic that the engineering team doesn’t have. And, of course, the engineering team has its own insights into that as well (combined with an improved ability to make tests precise, fast, and easy to write), so they need to be involved, too.

We had a (very useful) meeting at work today which, at one point, turned to the extent to which our end-to-end tests should extend beyond the software that we are writing and actually invoke our software via our partners’ software. (As opposed to driving our external interfaces through test clients that we’ve written ourselves.)

My boss was strongly in favor of integration tests (i.e. ones that used our partners’ software); I was against doing that as the default (though I certainly agree that some basic integration tests are useful), but admittedly some of that was an emotional reflex instead of a well-considered response. So here’s a try at the latter.

What makes a good test? One answer: the test should be (in no particular order):

• Precise
• Fast
• Easy to write (and maintain)
• Realistic

Precise: a test is intended to verify certain behavior, and an ideal test will fail every time if your code sin’t as intended, pass every time if the code is as intended. Fast: the more (useful) feedback the better, and the quicker you can get back to programming the better. Easy to write: you want to encourage people to write as many tests as is useful, and to keep the tests up to date as the code changes. And realistic is good, too: what matters is how your product will behave when released into the wild.

Software is built out of subsystems, each having different interfaces. And the first three desiderata all argue for the same behavior: you should write your test at the interface of the subsystem that’s in charge of the behaviour you’re trying to test. Any further in is impossible; but the further out you go, the less likely it is that your test is going to be precise, fast, maintainable. Whereas the fourth desideratum suggests that you should write your tests at the largest-scale interface that is practical.

In the past, I’ve had a bit of a blind spot in this regard – I appreciated unit tests faster than I appreciated acceptance tests, for example. But acceptance tests have saved me several times over the last year, so I really do believe in them now. (Note, however, that acceptance tests don’t have to be end-to-end.) My unit tests aren’t perfect; also (as Phlip periodically points out on the XP newsgroup), if your tests are a bit broader than absolutely necessary, then they can also serve as backup tests for the systems other than the one that you’re focusing on. (This is why he warns against overuse of mocks.)

Anyways, clearly some level of end-to-end testing is required: even if you want to test through the interface of the subsystem controlling the behavior under test, that still suggests some amount of end-to-end testing, because sometimes the behavior under test is nothing less than the behavior of the entire system! But not always – usually the behavior you’re interested is more specific, and usually you have a choice of interfaces at which you could test the behavior under question.

The criteria above give one answer as to what interface to chose: chose the highest-level interface for which it’s possible to write the test without seriously sacrificing precision, speed, or ease of creation. Most of the time, this will be a narrow interface; sometimes, though, it will be a broader interface. And it’s worth investing some amount of effort into building up tools to make it easier to test via that broader interface: there may not be much you can do about the precision and speed metrics, but judicious tool-writing can be an enormous help with the ease of use metric.

At work right now, we have thousands of unit tests; you can run them all in ten minutes or so, and if you’re working on one particular piece of code, you can run the relevant ones in much less time than that. Which is necessary to maintain a good workflow: you can run some of them constantly to catch your mistakes quickly, you can run all of them before you check in your code, just to make sure you didn’t guess wrong about which tests are relevant.

We have hundreds of tests that test our system more broadly, typically running our entire system (but not our partners’ systems). We run these every night. This is good, too: if something slips through the unit tests, you’ll find out about it soon. I suspect we could get some mileage out of speeding up those tests, and they’re lacking in precision at times, but I don’t have anything useful to say on those fronts, and I don’t think that’s a pain point.

We have something like ten tests which check integration with our partners; those also run every night, except when they don’t because something has changed the behavior of our partners’ systems in a way that we need help to resolve. (As has been the case for the last couple of weeks.)

And my boss is right in saying that this isn’t enough tests at the integration layer. (And we all agree, of course, that their reliability has to improve.) On the other hand, I’m right in being worried about focusing excessively on that level: tests should be precise, fast, and easy to write, and currently our integration tests satisfy none of those criteria.

This suggests the following courses of action:

• We should examine our non-integration end-to-end tests, and try to judiciously pick some to move over to the integration layer, in a way that gives us the most bang for our buck.
• We should invest time in making our integration tests more precise, faster, and easier to write, so that we can move more tests to that level in the future.

To be sure, I have no idea how to balance that latter investment against other forces competing for our time. We certainly have other high-priority tasks (my current favorite of which is getting our Customer ducks in order), but it’s important enough that, even now, it’s worth spending effort on. And it should be an ongoing task for the indefinite future.

I’m in the middle of reading some Christopher Alexander, which of course gets me thinking about how our house works. And my conclusion is that I quite like it, but that Miranda can take almost sole credit for that.

The downstairs has one largish L-shaped room, a small kitchen, and a small den. The upstairs has a largish master bedroom, Miranda’s bedroom, and the library / guest room. There’s a bit of a garden in the back, a porch off of the master bedroom.

And we have stuff all over the place: books line most of the walls, and there’s a pleasant amount of artwork and such on the remaining walls. But Liesl and I don’t actually spend much time in most of the rooms. Miranda’s room doesn’t really count for this purpose, but as for the others: we spend lots of time in the kitchen, we spend lots of time in the den (both the TV and the cable modem connection are there), we spend meals at a table that is in one part of the large downstairs room. But, honestly, we don’t spend much time in the rest of the large downstairs room, or in the library / guestroom, and the bedroom basically gets used for sleeping and other related tasks. To be sure, those rooms aren’t completely dead – if nothing else, I regularly gaze at the bookshelves therein – but using them isn’t our first instinct. And we never go out on the balcony and basically never go into the backyard.

Or at least that’s the way we used to do things. Miranda, however, takes a different tack. She spends some amount of time in her room, of course, and some amount of time watching movies in the den. But she likes to draw in the large downstairs room, and she also likes to play in the library, either doing legos or working at a desk that we just constructed here. (Following a design from a quite nice activity book that she got for Christmas, if I’m recalling correctly – thanks, Uncle Brian!) And she even spends some amount of time sitting at the desk in our bedroom, drawing or playing with the little sculptures/etc. that she and I have put there. (She likes to keep me company while I’m stretching, for example.) And she likes the back yard, though she doesn’t spend a whole lot of time there. (Yosha has also decided that the back yard has its advantages, too.)

I’m glad that she’s getting so much use out of the rest of the house, but it also makes the rest of the house seem much more alive to me. I’ve find myself spending more time in the living room and library recently, for example (admittedly, sticking a second computer in the library doesn’t help), and hanging out in the non-den room downstairs. But I still have more progress to make.

Not that she can take all the credit – like I said, there’s a fair amount of our stuff around the house, stuff that’s not for show but that we care about and use. And I like the way the piano placement has worked, making the large room downstairs more of a defined room instead of an open area with a vague boundary.

Still a ways to go. I have to train myself to use more of the space. Now that it’s getting warmer, we should probably try to lessen the boundary between the den and the back yard: maybe spending time out there, maybe just leaving the door open (but with or without the screen? I would say with, but the dogs might destroy it), maybe leaving it closed but leaving the blinds completely open so we can get more of a view.

And our bedroom, despite its size, is woefully underused: if you just want to sit and read a book, there’s no good place to do that, the bed and desk both being possible but both having drawbacks. One of Alexander’s design patterns that I’d really like to use is a window nook; I’m not up for doing actual construction, but maybe we can stop leaving the windows there covered by default, put a slightly more comfortable chair there, and rearrange the desk and dresser somehow so there’s a more defined space near the window and balcony, lessening the inside/outside barrier.

We could even think about what would make the balcony a more pleasant place to spend time; I’m a little dubious about that, but on the other hand I do like the view of the courtyard of the townhouse complex. (Nice trees.)

Probably other things to do first, though. We just got a leak in the skylight fixed last month; Liesl’s in the middle of arranging for follow-up plaster work. Next on the priority list is to arrange for somebody to go down into the crawl space and tell us if we have termites, structural problems, or an overactive imagination; then we’ll have to deal with the consequences. And we have to get the backyard fence fixed. Miranda is actively lobbying for us to take control of the backyard and plant a garden in that; I’m tentatively in favor of the idea, but I doubt I’ll have the energy for that this year, so maybe I’ll try to stall her until next year. We just got some prints that need to be hung up (that will be easy, we can do that next weekend); and Miranda is running out of bookshelf space (good girl!), as are we, so we need to buy three more bookshelves at some point, maybe next month. But perhaps at some point over the summer we can think about rearranging stuff in our bedroom, possibly buying another chair in addition. I shall raise the issue with Liesl.

Today’s game: Mario & Luigi: Partners in Time, for the DS. It’s the sequel to Mario & Luigi: Superstar Saga, for the GBA, and is not entirely dissimilar to the Paper Mario series.

The gameplay is quite similar to its predecessor; in particular, the DS’s special features are almost entirely unused. (It could easily have been released as a GBA game; I don’t really care either way.) You’re controlling four people instead of two (baby and grownup Mario and Luigi), but it doesn’t make much of a difference – in fact, like the earlier game, it hardly matters that you’re controlling multiple people. The characters are much, much less distinct than those of a traditional RPG; I guess this is what happens when you start with a characterless hero and add more people who are essentially clones of him with only minor modifications.

The core gameplay is pleasant enough. In particular, I like the way combat works (both here and in Paper Mario): it’s turn-based but with button pressing controlling how you attack and defend. So when you see a new kind of monster, it takes a bit of experimentation to figure out how best to defend against it (and attack it, sometimes), but even once you’ve learned that, you’re reasonably involved in the battles without feeling oppressed.

Well, not too oppressed. There’s not a lot of variation to the combat in the battles – Paper Mario, with its badges and variety of partners (with actual different attacks) does better in that regard. Honestly, there were rather too many times when I spent an hour or so making my way across some bit of terrain, and getting absolutely nothing of interest out of it – sure, there are boxes to hit, and enemies to battle, but nothing that I haven’t done a zillion times before. So a bit more novelty would definitely be appreciated.

Another way in which this series falls down in comparison to Paper Mario is how it handles leveling up – it’s all quite generic here, whereas in that series you have an assortment of badges to chose from, giving you an interesting range of options. Also the production values are lower here, but that’s expected for a handheld game in comparison to a console game.

Ultimately, I don’t have a lot to say about the game. It’s pleasant enough, but quite lacking in ambition: coasting on earlier successes and gameplay advances without really adding anything to the party. I’m not sorry I played it, and I might even play the next game in the series (assuming there is one) if it comes out when I have a lull in my handheld game schedule. But if not, I won’t be missing anything.

Currently, I’m performing SQL queries that look something like this:

select id from books where author_id in
(select * from
((select 106 id) union all
(select id from compound_authors where author_id = 106))
as extended_authors);

I have a hard time believing that the select * from bit is really necessary, but I can’t find a way to get rid of it without introducing a syntax error. The lesson, I suppose, is that I should learn SQL syntax, but does anybody know the right way to write this?

I’m pretty sure this is from The Top One Hundred Pasta Sauces, by Diane Reed, but that book was Jordan’s, so I can’t say for sure. The original recipe recommends spaghetti (or even vermicelli, which really doesn’t sound like a fit to me), but I prefer the fusilli that another recipe recommends: that way, the capers can nestle in the spirals.

The eggplant is the chief determiner of the quality of the dish. I’m too lazy to do the whole “purge eggplant by salting it” deal, but I put in a lot of olive oil (not quite deep-fry level, but close – actually, deep frying would probably work great!) when cooking the eggplant, and cook it for a while. (And then pour out some of the olive oil after cooking the eggplant.)

Quite a substantial sauce, but everything in it is good.

Pasta Siracusani, from The Top One Hundred Pasta Sauces

1 lb fusilli
1 eggplant
olive oil
3 cloves garlic
4 anchovy fillets
14 oz can tomatoes
12 oz jar roasted red or yellow peppers
2 Tbsp capers
12 pitted black olives
a bit of minced basil, or dried basil
1/2 cup parmesan cheese

Cube the eggplant, and fry until soft in the olive oil. Mince garlic and anchovies, chop tomatoes, and add them to the sauce. Cook for about ten minutes. Slice peppers into strips add peppers, capers, olives, and basil, and cook for another ten or fifteen minutes. Meanwhile, cook pasta; mix pasta, sauce, and cheese.

• Antibubbles.
• More IP insanity. I wish I could get more info about this, but I haven’t yet been able to find German sources.
• Links to pretty bug pictures
• I could watch this for hours.
• How to tell if your methods are too long.
• Another Octavia Butler tribute.

I’ve just upgraded this blog to WordPress 2.0.2. (From 2.0.1; largely a security fix.) Pity they don’t make patch files available. Let me know if you see anything weird.

I saw four kids playing patty cake on the playground at school last week. I hadn’t realized that you could do that with more than two people: basically, whenever you would clap the other person’s left hand, you instead clap the left hand of the person to your right, and whenever you would clap the other person’s right hand, you instead clap the right hand of the person to your left. (Unless hands are crossed, in which case things get a little harder for me to describe.)

The other interesting thing about this particular grouping was that two of the four kids were boys. (All around fourth grade, I think, plus or minus one year.) And their hands were flying just as fast as the girls’ hands: clearly they’d done this before. The particular chant they were doing, lemonade, has a “freeze” bit at the end, and they were more into the competitive aspect of that than the girls were, but not so much as to harm the game for anybody else.

I was really glad to see all the pro-immigrant protests last week. Nice to see a bunch of students willing to cut school for that, which I totally wasn’t expecting – I wouldn’t have though a priori that immigration would get that sort of response. Shows how much I know, I guess. (Of course, I like to see students cutting school in general, but that’s a separate blog post.)

I’m somewhat unsure of the correct response to immigration. My moral standards suggest that there is almost no reason to restrict immigration at all. It is a fact of life that people have dramatically different rights and access to resources based on factors of their life over which they have no control. In many instances in the past, we’ve come to the conclusion that such discrimination is morally abhorrent; why is it so much more acceptable to treat people differently based on where they were born than it is to, say, treat people differently based on their gender?

But I suspect that there are some reasonable answers to that question that I haven’t seen yet – just because nativist bigots get most of the press, along with my governor equating illegal immigration to anarchy (hmm, actually I’m sympathetic to anarchists too, come to think of it), doesn’t mean that there aren’t good arguments to be made. And the pragmatics of the situation deserve a good look, which I haven’t seen. (I would have a hard time evaluating pragmatic arguments, admittedly; I should spend more time learning about economics, I think.)

How the World Works had a good post on the subject. Good blog, that; makes it worth clicking through the Salon ads every day.

I haven’t been all that excited about the upcoming baseball season so far, but here it is. Kicking off with an Indians / White Sox game; I can support that.

I picked up the new Baseball Prospectus yesterday. Some nice tweaks to PECOTA (their player forecasting system), and the articles on the Indians and A’s were both interesting and cause for hope.

And then, for Jordan’s sake, I read the article on the O’s. A couple of quotes:

The 1978 Oakland A’s. One of the most disorganized teams of all time, worse even than the 2005 Orioles. Winners of three straight World Series just four years earlier, owner Charley Finley was unable and unwilling to adjust to the free agent revolution. He dumped everybody in an effort to save money, including his entire front office, eventually running the team himself with the help of a teenage kid who would latter become MC Hammer.

In case you’re curious, this is in the context of teams who were 14 games over .500 at some point in the season and later fell to 14 games under .500. (As the O’s did last year, while last year’s A’s and Astros managed the reverse.) Another “interesting” bit:

The best aggregate DH performance by a Baltimore team this millennium is the weak .250/.329/.398 posted by the 2001 squad. Since then, they’ve actually declined every year.

Ouch. Last year, in case you’re curious, their DH’s produced a BA/OBP/SLG line of .210/.277/.362.

The mac is pretty well set up now. I’ve finished transferring all my music to it; the long part was just getting it on the USB drive from the old computer. (And I’ve even ripped a few more CD’s: I was really running out of room on the last computer’s hard drive.) The only thing that didn’t get transferred was my subscriptions to podcasts for which I didn’t have any episodes currently saved; I can live with that.

I made an account for Liesl. Safari hung the first time she tried to use it, which is pretty weird; hopefully that won’t repeat itself.

I’ve installed X. Still some futzing to do there; xemacs comes up from panini (the new Ultra 20, for those of you who aren’t closely keeping score) with ugly fonts, so I guess I’ll have to give my .Xresources file a look. xterm is clearly inferior to Terminal.app (or whatever the appropriately mac-savvy way is to name the beast); fortunately, if I set my DISPLAY in Terminal and log in, everything works just fine, but I wish X11 didn’t insist on bringing up an xterm when it starts. Does the mac have some hidden preferences somewhere that I can tweak behind the scenes?

I still need to figure out how to get Safari to behave the way I want, or for that matter how much I care. Or maybe I could switch to Camino and try to get it to behave the way I want; I’m not yet motivated to do so, however. And, in general, I should probably read through documentation to see what I’m missing about things. (The last time I got a new mac was 1989; the last time I regularly used one was 1998. I hadn’t quite realized that it’s almost 8 years since I left grad school…) Which is a big change from Windows – there, I felt dirty touching the thing, and had no desire to learn anything about it. (To be sure, that has at least as much to do with me as with Windows.)

The wireless reception in the front of my house would seem to be a bit spotty; unfortunate. Weird, too – we’re talking about 20-30 feet here. Maybe I’ll get another access point or something.

The nano is nice. I wish it had a bit more storage, but that will get fixed in a year or two. And I hadn’t realize that I had a full 2GB of podcasts stored; I suppose it would help if I didn’t have all sixty or so episodes of JapanesePod101 saved up. (Most of them are around 10 minutes each, but it still adds up.) Don’t ask me when I’m going to get around to listening to them; I have not the slightest idea. I was a little worried about how managing the collection would work, given that the nano doesn’t have enough space to store everything I’ve got, but it turns out that iTunes is happy to sync the podcasts automatically and leave music syncing up to me to perform manually. Which should work just fine.

Nice screen on the nano. My first reaction was that the form factor was a little small, and that I wished they’d kept the original iPod’s dimensions (other than the thickness – thin is good), but it seems to work fine. So maybe I was worried about nothing. I was also surprised to see that it didn’t come with a charger – does it use little enough power that plugging it in weekly for my podcast updates will keep it charged? If not, no big deal – after all, I’ll be using the mac frequently, unlike the situation with XP. (And even if I only plug it in once a week, I’ll be happy to leave it plugged in longer.) Will the other iPod’s charger (which worked through the firewire cable, which isn’t provided with the nano) work, or will it damage the nano somehow? I guess I’ll google. Also, I restored the old iPod to its factory settings while I was playing around with things, so now it uses the mac filesystem instead of FAT32. It looks like iTunes wouldn’t have complained too loudly if I’d left it in FAT32, but on the other hand it was insisting on deleting all the music stored there, so restoring it didn’t add significantly to the time to get the old iPod working with the new computer.

Hmm: this editing page doesn’t look right on Safari – the buttons to control formatting, add links, etc. aren’t there. That could be an annoyance; I’ll investigate that, but I could imagine that pushing me to use Camino.

I still need a name for the new computer; any suggestions?

I’m typing this on my new Mac. Which is adorable, and seems to work well enough, though there’s still a bit of work to do. I don’t have my music transferred yet – I tried sharing my music folder from my old laptop, but for whatever reason I can’t get the mac to see it. I have no idea where the problem is, but I don’t really want to become a samba expert, so I’m just copying all the data to a USB hard drive. (Those things are coming in surprisingly handy.) Which is taking a couple of hours, though I’m not quite sure why – is USB 1.0 (which is all the old computer has) really that slow? I guess so. And I need to get X working, but that shouldn’t be too hard. And figure out what my workflow will be, which will be a little harder, but I’ll manage.

Each of my dbcdb web pages corresponds to an instance of a class called Entity. And each entity has a key, which is the number used in the web page. The class Collection represents the collection of all the entities.

Currently, each entity knows its key. As part of the change that I’m about to make, I’ll have to have each entity know the collection that it’s part of, too: right now, each series, for example, keeps a list of the volumes that are part of it, but I want that information to be found by querying the collection instead. After a bit of thought, I decided that the right way to handle this is to just stick the collection in the key (which is, fortunately, already a class instead of a plain int), rather than pass the collection in as another constructor argument.

Last night, though, I realized that my thoughts on this matter are somewhat inconsistent: in the past, I’ve held a strong conviction that members of a collection shouldn’t store any sort of indexing information, shouldn’t know that they’re part of a collection at all. So what are these keys doing there, and why am I making matters worse by having the members be able to get at the entire collection?

My reasons for that in the past were that it’s a violation of a proper separation of concerns, that it leads to potential inconsistency, and it makes collections inappropriately non-generic. Which are all good reasons – for example, I wouldn’t claim that it makes much sense for elements of an array to have to know their indices. (Nor would anybody else, but when it gets to lists, there’s less of a consensus.)

The collection in question doesn’t claim to be generic, so the third reason isn’t relevant. The second reason is, I think, not at all likely to be a problem in practice in this case, though I suppose I’ll make some member variables final just for the sake of documentation. The first reason is potentially a show-stopper, though. So: why did I make the key accessible from the entity in the first place?

It’s really only used for one reason: each entity has a method called pageLink that prints a representation of an HTML link to that entity. And the href refers to the key, so that method has to have access to the key somehow. But it could get it from an argument, perhaps. After all, I was considering passing in the collection when generating the list of volumes contained in a series, instead of having the series extract the collection from a key; maybe this whole argument is a sign that I should go down that route instead.

For the pageLink case, though, that won’t work: an entity never calls pageLink on itself, it only calls pageLink on other entities that it knows about. For example, a book has a data member storing its author, and it calls author_->pageLink() when generating the Author: line on its HTML page. But the book doesn’t know what the author’s key is, it just knows about the author entity. So either it has to ask the author for its key, store the author’s key instead of (or in addition to) the author entity, or ask the collection what the author’s key is.

The second suggestion just sounds weird to me, valorizing the keys over the entities. The third suggestion might be sensible in some contexts, but in this case I can’t see any concrete benefit to it. So I’ll stick with the first suggestion.

So: what’s the lesson? One, if you have to do a back mapping from elements of the collection to indices, then storing that back mapping in the element might make sense in a special-case collection. Two, don’t be dogmatic. Three, don’t feel too guilty about being dogmatic: strongly held instincts that can be justified and whose justifications can be compared against particular situations are okay.

Every time I read My Name is Asher Lev, it makes me want to spend hours hanging out in art museums. Or even start drawing or painting, but I always come to the conclusion that I don’t want to take that up only to drop it before I get even vaguely competent, and I never have the time to do that.

But what are the good art museums around here? San Francisco isn’t exactly famous for them (though I did enjoy the Asian Art Museum last month), and San Jose is, well, San Jose. Stanford is closer than both of them, and its art museum has a quite nice Rodin sculpture garden outside; I’ve never been inside, and I can’t imagine that what’s inside is nearly as good as what’s outside, but it must be worth a look. So I guess I’ll give that a try.

This train of thought always leads to the conclusion that I should become rich, retire, and move to Paris. Aside from the practical difficulties with the former, I probably don’t really want to do (most of) that, but frequent visits to Paris are good. Which Miranda would support, I think. Of course, there are other places in the world that I’d like to travel to, as well. Hmm. Vacation plans are set for this year (relatives plus Canadian theater); maybe next spring?

I planned to say more about First, Break All the Rules, but I seeem to have gotten excessively sidetracked on computer and programming geekery, and worse yet geekery of relatively limited interest. For which I apologize: I’ll try not to let it happen again to quite that extent. Anyways, one last thing that I liked about the book: it has a very refreshing approach to how career paths should ideally work.

Recall that the book’s focus is on having employees spend as much time as possible doing what they’re most talented at. Unfortunately, many traditional career paths involve changes requiring quite different skill sets: a programmer might traditionally be promoted to a manager role, for example, or if a programmer resists that but still wants to move up, then he or she might turn into a software architect.

I happen to like managing and I happen to like designing software, and I wouldn’t claim that my programming skills are irrelevant to those ends, but ultimately the skills required for the jobs are different ones. And I’m not sure that I enjoy either managing or software architecture more than simply programming. So it’s refreshing to see somebody say “we shouldn’t present those job transitions as the only goal for our best programmers”. And the authors back up those words with, for example, recommendations that employees and their managers should have overlapping salary ranges, with the employee potentially earning more than the manager.

But, now that I write that, I realize that my take on the matter is rather different from the authors’. I don’t want to give up programming; but that doesn’t mean that I don’t want to, for example, have a say on the architecture of the software that I’m working on. The truth of the matter is that I want to exercise a range of my own skills, so I’d rather not have the pigeonholing that is present either in a traditional programmer-to-software architect career path or in this book’s presentation of those as distinct roles with less of a hierarchy between them. And it’s not just my skills: I’d be happiest if everybody around me was doing a mix of getting their hands dirty while improving the software architecture. (Part of the reason why I’m attracted to XP, doubtless.) Of course, this might change if I’d seen a situation where having separate software architects worked really well; in my brief programming career, I haven’t had that experience.

Some more random comments that I don’t have time to string into a coherent essay:

• I’m probably excessively biased against the notion that programmers’ managers should be ex-strong programmers. The reasons why I don’t like it are that the justifications that I’ve seen for that claim strike me as after-the-fact, and that I learn more every day (or at least every month) about the different skills managers need. But those aren’t the most solid of reasons, and I do accept the fact that, for example, managing a programmer is different from managing an accountant, and that some domain knowledge is probably useful to that end.
• A counterpoint to the notion that managers and employees should have overlapping pay ranges: one very good point that Andy Grove makes is that an effective manager’s actions will have much higher leverage than a regular employee’s actions, so have more value to the company. Then again, equating value with appropriate salary is pretty naive economics: there is this notion of supply and demand, for example.
• The book that I’m currently trying to integrate into my thought talks about how job transitions are increasingly horizontal. Which fits in well with the notion of focusing on what you do best.

I’m now generating my HTML pages based on the data in the SQL database. I’m still first assembling all the data in memory using Java method calls, and writing that out to an SQL database, but then I throw away the in-memory copy and regenerate it from the database.

I can’t really say that that step is done, though. There are a couple of things wrong:

1. I’m generating the various indices (e.g. the list of books by a given author, or the list of recently read books) in a very bad way: basically, I read all the data from the database into memory and generate my own lists out of Java collections, instead of using SQL queries. Given that I’m generating all the web pages statically at once, this is fine from an efficiency point of view, but it’s lousy from a “learn about SQL” point of view.
2. The separation of responsibilities isn’t nearly as clean as I’d like: in particular, I’m using proxy objects which get lazy-initialized from the database (which is fine, though I’m thinking of getting rid of that eventually), but the code gets confused at various points between the proxy and the object that the proxy creates underneath it.

Don’t get me wrong: it works, which is good. And to make sure it works, I even ran the one acceptance test that really matters, namely generating all 344 web pages corresponding to the current list of books/games/authors/etc., and checking that none of them changed betweend the old and new ways of generating them. I was pretty sure that my unit and acceptance tests were good enough that there wouldn’t be any bugs remaining; I was wrong. To be specific, I was generating links for volumes in a series incorrectly in index pages (point 1 above), where the bug was masked in the tests by point 2 above. Sigh, but at least it means that I have reasonable instincts for bad spots in the code.

Fortunately, attacking the first item will greatly ease the second, and I have a pretty good idea how some of this will work. The one problematic aspect is exactly how I’ll generate the indices, in a way that is nice SQL but still easily testable without finding an in-memory SQL database. Take, for example, the list of books by an author: if it were as simple as select * from books where author_id = AUTHOR; then that would be easy to test. But that doesn’t work for two reasons:

• If a book is by Fred and George, then I want it to be mentioned in Fred’s page, George’s page, and Fred and George’s page. Which means I have to do that query for various values of AUTHOR.
• If a book is a volume in a series, then its author_id field is NULL: the value is taken from the series’s author_id field.

So the right SQL query to use is reasonably complicated; and I certainly don’t want my testable SQL-like interface to handle that level of complexity. I think I’ll probably add some sort of ‘indexing hook’ that can generate a more complicated SQL query in the SQL version and evaluate custom Java code in the test version; not my favorite solution, but I can’t think of any other ideas offhand.

I’ll do that next. (Which will take a while, I think.) And then I’ll write a CLI tool to add stuff to the SQL database and modify it. At that point, the SQL transition will be more or less complete, and I’ll be able to get back to making changes that actually affect the HTML I’m generating.

Which I devoutly want to do, but this time hasn’t been wasted. For one thing, I’m learning about SQL, which was after all the goal of the whole exercise. For another thing, it’s very interesting watching the changes that the code undergoes as a result of this annealing, and trying to guess how the code will want to change next. I get the feeling that, when I’m done with the SQL transition, I’ll have the same number of lines of code as when I started, but 75% of them will be different and the structure will be somewhat improved. We’ll see.

Incidentally, while I was at it, I finally got around to installing viewvc (a pity there isn’t a released version with Subversion support, but the daily snapshot I grabbed seems to work fine), and tweaking my acceptance test framework to, among other things, print the time that each test takes.  By far the longest test is the one that connects to Amazon web sites in the US, UK, and France, to verify that my links work.