April 2004

« March 2004 | Main Index | Archives | May 2004 »

29
Apr

In passing

  • 11:09 PM

Lung cancer? I wanted heart disease. – man in the street complaining about the warning label on his cigarette packet.

And my mother, on how Microsoft may yet help her with her overly complicated life:

I need a Powerpoint life.
I need to condense my life into a series of dot-points.
I need to dream in dot-points.

...

  • Lonita is going to see the Pixies.
  • jwz is going to see the Pixies.
  • Even Wil Anderson is getting to fly over to the States to watch a gig.
  • You are ALL BASTARDS!

    ...

    I always thought “Palm Pilot” made a better euphemism than a product name:

    Well, he's OK, but he's a bit of a palm pilot...

    I was ego-surfing for Confluence references in Google this evening when I came across the following rather surprising sight:

    The search results actually contained a reference to... the results page for a completely different Google search.

    Is Google finally devouring itself?

    Random Shorts

    • 12:44 PM

    It’s interesting the little security decisions you make. Like waiting until after I had returned to mention on my weblog that I was spending the weekend in Perth.

    Niftiness from shimmer on development tips

    “Make your temporary programmer art as ugly as possible, so that the art department notices while there is still enough time left to replace it. If you steal something from the web for a prototype, ‘sign’ it with an ungodly hideous swath of colour for the same reason, so that it can’t possibly sneak into a shipped product.” I use fuschia. It seems to carry an implied message that perhaps this monstrosity was the Girl Programmer’s fault, and that she is the person to speak with about how the game uses it in order to make a working replacement. Also, I hate fuschia enough that sometimes it’ll force me to bother the artists directly and repeatedly just so it goes away.

    and on clothes shopping

    All they cost is twice what you want to pay for them and your pride.

    2004 Pixies live performance on mp3. Run, don’t walk.

    Various posts on the “Blog as Article” trend.

    I regularly write thousand-word blog posts that could, potentially, become real articles. Once or twice I’ve been approached to submit them to magazines. My opinion, though, is that we put up with a lot of bad writing on blogs that in other circumstances would render them unreadable. The immediacy of the blog medium is compensation for the bad writing. Take them out of that medium, they’d (with a few rare exceptions) just be badly-written articles.

    All my posts are strictly first-draft material, and I’d be pretty embarrassed to see them published anywhere. Even my best posts would require a great deal of work before I’d consider them suitable for publication anywhere but this personal space, where I’m not expected to be any good.

    Linking to blog-posts in lieu of paid content would satisfy bloggers, who are used to the blogging style, but would leave people used to quality in a publication out in the cold.

    CSS continues to truly shit me. It’s a wonderful idea, but the execution leaves a lot to be desired. The contortions you have to go through to make something as universal as columns, and the brittleness of the layout that results (what happens when one column is optional?) makes me run back to table-based layouts every single time. I think the only reason I stick with CSS-based layout on my blog is sheer bloody-mindedness.

    Everyone should know at least one scripting language. Even if you're paid to program in some other language, they're just such useful tools to have lying around on the off-chance you need to throw something together quickly.

    require 'rexml/document'
    doc = REXML::Document.new(File.new("project.xml"))
    doc.elements.each("//dependency") do |dep|
        name = dep.elements["id"].text
        version = dep.elements["version"].text if 
            dep.elements["version"]
        puts "%s %s" % [name, version]
    end

    I swear the light is a different colour in Perth. It's brighter, fresher almost. It's cleaner, although maybe that's just an indication of how much pollution I've become used to in three years living in Sydney. Or maybe it's just the characteristic wide blue sky: a sky that makes you want to lie back and take a picture of just how damn big and blue it is, until you realise how boring a solid blue photograph would be. You could just save your time and make a nice blue square in Paint.

    Except you'd never match that shade of blue. It's almost as if Perth has its very own sun.

    I summed up Perth pretty well looking out at it from my hotel window. It's wide, flat, and surprisingly green. The skyline, an almost embarrassed huddling of five or so skyscrapers amongst a clutter of squat tower blocks, is the only thing that tries to push its way out of the landscape: between the Darling Escarpment to the East and the Indian Ocean to the West, all you have is one big, flat plain.

    This might be why the sky is so big. There's nothing to compete with it.

    Or maybe the light is different there because everything's slower. Perth moves at a different pace to Sydney, and perhaps that even stretches to the speed of light? Even the weather in Perth moves at a glacial pace. I remarked to someone while I was over there that in Melbourne, if you don't like the weather you can wait an hour and it will have passed. In Sydney, weather changes from day to day. In Perth, even the weather takes a week to change: the clear blue, cloudless sky you have today is likely to stay with you for days to come.

    I've been back to Perth three times since moving to Sydney. For the first, I'd been away only half a year, so it felt as if I'd never really left. I just slotted into the Christmas routine at my mother's house, and that was it. The second time, though, was just weird. I was teaching a course, and IBM put me up in a hotel in the city. Staying in a hotel in the town I grew up in, after such a short absence, just felt strange. Or maybe it was that I was unused to feeling like the stranger.

    This time was different. After four years away, Perth has somehow lost that essence called "home". I was still a trifle freaked out when I landed in the airport, but I think it could be more that I expected to be freaked out, and was freaked out that I wasn't freaked out. Or something.

    This time around, it didn't feel like home. It felt like this place I happened to know my way around. Maybe this is natural: four years is enough time to lay down roots in a new home, be planted somewhere else. Or maybe it's just that I live so much of my life within myself already that I'm not so easily attached. I am the cat who walks by himself, and all places are alike to me.

    The feeling of home still lingered a little in Fremantle, although it took me three tries to find the street I lived in. And it permeated my mother's house, even with the furniture gone, and the real-estate agent explaining why the fence needed fixing. Standing in the laundry, I almost expected to be able to open the door and see the cat run in for her food.

    As an aside, I'm sorry anyone in Perth who may be reading this, and who I did not contact while I was over last weekend. I'm a lazy git, and what with having to go to my brother's wedding and all, the extra effort required to organise anything else just overwhelmed my poor little brain. Special apologies to Amy, although she doesn't read this, who I've not got in touch with for eighteen months.

    I should go back again some time soon, for more than just a weekend, and without the (pleasant) distraction of a family event. It would have to be Summer, though. I could lie on a beach and let Perth's very own sun shine down on me, and maybe recapture that glimpse of home.

    Over the weekend, my father was rather embarrassedly relating a story from the late 80's. At the time, there was a school of management that believed zero-defect software was possible, so long as we made the effort, and paid attention to detail.

    The story was from a workshop for programmers. To cut a long story short, the workshop was a classic tale of slippery assimilation, trying to find that ridiculous cut-off point where a program went from being short enough to be bug-free, to long enough to be inevitably buggy. I vaguely remember the answer being 137 lines of code.

    This, of course, is the promise of structured programming, of functions, of objects. If we can write 137 lines of code without a bug, then we can structure our programming style so that we're always writing units of fewer than 137 lines. We can build those units into components, and voila! No more bugs.

    This is also the promise of unit testing. We divide our programs into small pieces, and test the buggery out of each piece individually. This is quite an effective approach (and a much better time/effort trade-off than a formal proof), but the failure conditions of unit testing are interesting:

    1. It is rarely possible (or at least practical) to unit-test exhaustively. The art of effective unit-testing lies in being able to predict where your code is likely to break, and testing those edge-cases. If you fail to predict where an edge-case might occur, you won't write a unit test for it, and it's a potential source of error.
    2. More dangerously, code is frequently rewritten in ways that do not change its behaviour. A better algorithm is chosen, or the code is modified to fix a bug. Accepted wisdom is that the continued running of the existing unit-tests is sufficient proof that the code continues to work as intended. However, changing the code is likely to move the important edge-cases in ways that render existing tests pointless, and additional tests necessary.

    Ironically, at exactly the same time my father was flying around the Asia-Pacific giving these workshops, he recommended his son read Complexity by Mitchell Waldrop: a great little populist science book about the study of complex systems and emergent behaviours. The book described the fascinating and inherently unpredictable behaviour that arises when many small systems with clearly defined rules interact. I think you know where I'm going here.

    Computer programs are complex. We manage this complexity by dividing programs into manageable pieces that can be isolated and scrutinised closely enough to find the errors of logic that will inevitably sneak in. We then create complexity by having each of these systems interact with each other. The more points at which these pieces can interact, and the more variations in that interaction, the more chance for some chaotic, unwanted behaviour.

    Encapsulation in objects; presenting façades for components; design by contract: ways in which we attempt to keep some control over the scope of these interactions. Part of programming is fighting the battle against unnecessary complexity, because there's already more than enough necessary complexity waiting to trip us up.

    Automated or scripted functional tests don't really help us here. Unlike our 137-line units, in which the edge-cases can be determined by inspection, the brittle edges of our complex systems are much harder to spot. Scripted functional tests serve a single purpose: they verify that some single path through the maze works, and continues to work. Rarely does anyone write a new functional test without being sure before-hand whether it will succeed or not.

    Generally, functional tests end up working as regression tests. They demonstrate a particular base-line of stability where execution paths that must obviously work, or paths that have caused enough trouble in the past to warrant a test, continue to work.

    A certain number of "complexity bugs" can be found through programmer vigilance. Get to know your code. Get to know how the pieces work and how they talk to each other. The more broad a view you have of the system being programmed, the more likely you will catch those pieces of the puzzle that don't quite fit together, or spot the place a method on some object is being called for some purpose it might not be fully suited.

    Bruce Schneier makes a good case for the absolute need to examine code for security flaws from this position of knowledge. When considering security-related bugs, we have to ensure the system is proof against someone who knows how it works, and is deliberately trying to break it, again an order of magnitude harder to protect against than a user who might only occasionally stumble across the "wrong way to do things".

    For the rest, you need QA. You need QA who know told what they are trying to accomplish with the software and how that is being accomplished, but who want to find a way to break it, and who don’t fall in the trap where their tests unconsciously follow the path of least resistance through the code.

    The inevitablility of bugs also means we have to pay attention to the failure modes of the software we write. If an ACID database encounters a bug, it's vitally important that data integrity be maintained above all else. If a NullPointerException occurs in a consumer application, the user might even be happier not knowing anything went wrong. (I've lost count of the number of times I've been presented with a stack-trace in IDEA and just clicked "ignore" because there's absolutely nothing I could possibly do about the error. In that case, am I any better off having seen the dialog in the first place?)

    Support is another vital way to manage failure. If a bug is discovered in production, how is the problem communicated back to the developers, and how quickly can the fix be deployed or patch be released?

    Alternatively, of course, you could just make sure you never write any program longer than 137 lines.

    Two months ago, I thought reaching 1.0 was one of the toughest things a software project could do. On your left is a pile of bug reports, on your right a pile of feature requests, and in the middle there's a calendar screaming "Release, already!"

    You have to release software. Spending your entire life working towards the mythically perfect 1.0 is a conceit only the more naïve open source projects can sustain, and even then it's annoying as hell. So in the end you're brutal. You throw all the remaining feature requests into the next version, then go through each bug and ask two questions:

    • How serious is this, really?
    • How hard is this going to be to fix?

    Measuring how serious a bug is isn't as clear as you'd think. A minor niggling problem can be more serious than a crashing bug if the minor problem annoys everyone, while the crashing bug only hits one or two people. That said, I generally order bugs from 1 to 4, as follows:

    1. Crashes (in web-app terms, a 500 error)
    2. Behaves incorrectly (put good data in, get bad data back)
    3. Behaves sub-optimally
    4. Cosmetic problems

    (Update: Data-loss bugs are probably even more important than crashing bugs, and security flaws shouldn't even stay un-fixed long enough to make it onto the list.)

    You want to fix everything in all four categories. You can't. Even if you had exactly enough time to fix all your existing bugs, you'll run into the problem that with every bug you fix, you increase the likelihood of introducing a new bug, possibly one higher up the list than you started.

    The biggest risks are bugs in category 3 or 4. Fixes that affect performance or workflow are dangerous because of the significant likelihood of introducing incorrect behaviour in the process. Rigorous testing helps, but not as much as you'd think because often components will have been written to be used in a particular way. Tests are naturally biased towards the way the developer was thinking when they wrote the code. Change the workflow or introduce a cache, and suddenly you're using the code in a way the original designer didn't think of (or they'd have done it that way in the first place), and thus didn't test as thoroughly for.

    Even a simple thing like a cosmetic fix can be a risk: foo needs to link to bar, but the developer adding the link is rushed and does it the old way, which you'd stopped doing because it failed under certain circumstances.

    So obviously, on the road to 1.0, your risk/benefit analysis is weighted highly against fixing sub-optimal behaviour or cosmetic problems.

    This is why armchair user-interface criticism annoys me.

    Certain forms of UI criticism are valuable. While I don't wholly agree with it, John Siracusa's critique of the OS X Finder is a considered review that tackles the philosophy of the application's design from the ground up, questioning the basic assumptions the application made about how it was going to be used. Similarly, it's usually obvious when an application received no interface attention whatsoever, and was just a bunch of form fields slapped one after the other by a blind programmer1.

    However, random pundits looking through an application or suite of apps and pointing out all the places that the UI guidelines weren't strictly adhered to, or all the places that the buttons could perhaps be better arranged, is just the practice of finding all those category 3 and 4 bugs that didn't make the cut. By all means, file bug reports. Bug reports on UI issues are incredibly useful. But don't slap them all together in a web page, however entertaining, as if they somehow prove that the interface as a whole is slipshod2.

    Classic Mac users moving to OS X are a particularly annoying example. The original Mac OS was tiny: the Finder was about the same size as the ls binary on my Linux box. A marvel of engineering, indeed, but when you're writing something to be that small, you can't afford inconsistency, because inconsistency means more code, and suddenly you can't fit in ROM any more. Subsequent versions of the Classic OS were layered on top of this core, piece by piece, over a period of fifteen years.

    Of course an OS or application designed this way will end up more internally consistent and more polished than an OS that had to spring from its father's head fully-born and monolithic. It should come as no surprise that each version they polish things a bit more, but they don't get all the problems, because there's a simple commercial reality that says you have to release something that's imperfect in order to pay for perfecting it. Which isn't saying you should release something that's crap: you shouldn't. It's just to say that you shouldn't expect there not to be significant (but lessening with each subsequent version) room for improvement.

    Anyway, as I was saying in the first paragraph, I used to think that the hardest thing a software project had to do was make the painful cut of features and bugs for 1.0. It turns out I was wrong. The hardest thing to do is, in fact, to make the cut for 1.1. The moment you release 1.0, you start getting these incredible things called users, who find all those bugs you never turned up during development, and who make really cool suggestions for things you could add.

    And you've got those piles of issues in front of you again, and the calendar is looking threatening.

    1 My design skills suck3. I understand the principles of good UI design quite well, but I have always deferred to someone else to do the laying out of stuff because I know in advance that their eye is better than mine.
    2 I'm not talking about criticism of any product I've been involved with here: this is a general gripe, not a personal one.
    3 Although I'm quite happy how this weblog turned out.

    Like all plans, it was simple and effective. I even wrote it up on the whiteboard in clear steps:

    Javablogs
    1. Build dependant libraries
    2. Build Javablogs in parallel to existing site
    3. Migrate data
    4. Run first update
    5. Test
    6. Switch over
    7. Post announcement

    Pretty simple, huh?

    Well, except I screwed it up during step two. I carefully checked the new version of Javablogs out of CVS in the user's home directory instead of the deployment directory, so the old version could continue running in parallel until we were satisfied it was stable. The only problem was... a stray symbolic link caused the new copy to be checked out directly over the running version. Byebye website.

    The smart thing to do would have been to immediately back up to the old version and rebuild what we knew worked.

    I think this is final proof that I am not a smart person.

    A few hours frantic bug-fixing later, I get most of the site working live. And then, of course, there were the things I had to fix later from home because I didn't notice before leaving the office.

    The site still isn't entirely stable, but hopefully it'll make it through the night.

    Update. It didn't last the night. It didn't last even the hour after I went to bed. Luckily, though, sleeping on a problem often brings you more enlightenment than stressing over it. So as a short memo to myself, and anyone else who might read this cautionary tale:

    Never deploy a new version of a large public website with the maximum database connection pool size set to four. It's unlikely to do much more than stagger along for a few minutes and then expire messily.

    Update 2 Yes, that seems to have done it.

    SOAP on a BOAT

    • 1:49 PM

    I was pointed towards Web Services at Sea from Geek Cruises. As the person who pointed me to the site (who wishes to remain nameless) added:

    That Geek Cruise summarises everything I love about Web Services:

    • As big as a ship
    • Overpriced
    • Lots of people telling you what to think
    • Lots of pretty things to look at that aren't actually useful
    • Takes you on a long journey and lands you back where you started
    • Sponsored by IBM

    Just thought I'd share :)

    I was reading a few Microsofty blogs the other day, and their take on the Sun settlement was decidedly different than you see in Java-land. I think the general tone was something like: "$1.6 BILLION? What can Sun possibly have that's worth $1.6 billion to Microsoft?"

    Now call me a cynic, but I can't help wondering if the next year will see a less aggressive, more financially responsible Sun cutting back on projects that are unprofitable and that only exist as a weapon against a company they are no longer in pitched battle with. Projects like, say, OpenOffice.org.

    OpenOffice and StarOffice have both been gaining steady momentum in the past year: being rolled out in several big companies as a Microsoft Office replacement. Even in companies that haven't moved to OO.o, its existence has been used as a bargaining chip to push back against Microsoft's more "interesting" licensing plans.

    There wouldn't have to be anything as sordid as a conspiracy. It's just a logical move, now that the lawsuit is settled, for projects to be moved to the back burner that existed solely as a weapon against Microsoft in markets Sun has no reason to compete. Sun know in their hearts that they're a server company, and they're not really comfortable with any software that isn't part of Solaris.

    OO.o's front page news ticker links to this article in which Sun reiterates its support for Open Source despite the Microsoft settlement... except somehow they do so without once mentioning the project they support that most directly competes with Redmond's own.

    What is the current extent of Sun's contribution to OO.o? I can't tell, but given how much of OO.o's news-ticker is taken up with articles about the Sun/Microsoft settlement, I'd imagine it's still pretty significant. Could OO.o continute to be a viable project without Sun's support? I honestly don't know. I would hope so.

    Dear Microsoft

    • 8:21 AM

    Hi there Bill, Steve and the guys. I hope you're having a nice day. I don't know how it really is, but I have this mental image of the upper echelons of Microsoft living in a long wooden hall, quaffing mead, and singing loud songs about burning villages and making off in longboats with the gold and women.

    Anyway, it's nice to hear you've finally sorted things out with Sun. It was just lovely to see how friendly you and Scott are now. It must be difficult finding friends when you're the biggest software company in the world: you get yourself a bit of a reputation, and suddenly everyone is on the defensive and you can't get to know anyone any more.

    So it's nice to see you've found a new friend, but I think you're still probably a bit lonely, so I thought I'd make an offer.

    It only took one and a half billion dollars to turn Scott around. I just thought I'd let you know that I want to be your friend too, and it won't cost you nearly as much! One million US dollars, and I'm your guy. It's a bargain! A million dollars? Bill probably loses that much behind the couch in your average week.

    For that, I'll happily tell the world how excited I am to be in partnership with you, and how much I respect you as a company. I'll come to your barbecues and laugh at the funny slogan on your apron, even if nobody else does. I'll tell Steve Ballmer that he really isn't as bad a dancer as everyone says he is.

    Throw in a copy of Visual Studio, and I'll even tell everyone how cool .NET is!

    Well, so long as I can get it running on the big-ass G5 I'd buy with the money.

    When possible, cut with the grain. The grain tells you which direction the wood wants to be cut. If you cut against the grain, you're just making more work for yourself, and making it more likely you'll spoil the cut.

    I was working on a Java web application the other day, looking to move functionality out of base classes (where they were painfully tying unrelated actions into a rigid class hierarchy), and into the filter/interceptor chain that delivered data to the actions.

    It's dependency injection again, but the inversion that gets its cue from the contents of the incoming request. So, for example, one thing the Interceptor could do is see that since the request has a personId parameter and the action has a setPerson() method, it should retrieve the appropriatePerson on the action's behalf.

    This is actually pretty neat: it avoids cumbersome implementation inheritance, encourages you to use consistent parameter naming across the app, and makes actions a lot easier to test since now instead of having to mock out the retrieval of a person, you can just pass one in.

    Java put one little obstacle in the way. Java has no simple way of asking "Does this class respond to this method?" I could instead try Class.getMethod(), but that quite annoyingly throws an exception if the method isn't found, rather than just returning null. Or I could search the array returned from Class.getMethods() to see if there is a setPerson() contained within.

    We're only talking about a few lines of helper method here, of course. Apache Commons already has one ready-made and waiting for me. All this would be doing, though, is moving the ugly code into a ghetto so I wouldn't have to see it. It would still be there.

    I was cutting against the grain of the language. Java is statically typed. Java doesn't want me to use the existence of a method to determine if I should call it. Philosophically, the language considers the existence of a method with the right name to be a coincidence, unless that name is backed up by a specific type. (Witness the objections to dynamic typing that start: "What if I had two methods called shoot, one on a Camera object, the other on a Gun?)

    For Java, the correct way to do what I was trying to do was to avoid introspection entirely, and instead define a PersonAware interface that I could check for with instanceof and then cast to directly.

    It's a pattern I follow quite often. Over the last few years I've come up with a series of very clever, rather complicated introspection-based solutions to problems. Invariably, a cooler head has later either deleted them, or convinced me to delete them because while the Java way might be slightly less powerful and not as conceptually clean, by moving the code back to cut with the grain of the language, it became more readable and more maintainable.

    On the other hand, Ruby is very much a dynamic language. When I'm writing code in Ruby, I expect duck typing to be the norm. If it looks like a duck and quacks like a duck, we treat it like a duck.

    Which is why I was annoyed the other day. Some Ruby library told me it needed a Hash passed in, and I made the quite justifiable assumption that this meant it wanted some object that implemented certain methods of the Hash class. As far as I could see, it only really needed a [] method, which my MultilayeredStuffHolder already implemented. When I passed my object into the library, however, I got an error back because the library explicitly checked the class of my StuffHolder, and complained that it was not a Hash.

    Against the grain, again.

    On the other hand, when carving meat you carve against the grain. The carving knife is stronger and sharper than your teeth: carving the meat against the grain leaves your teeth a much easier job, and the meat feels more tender as a result.

    A lot of very useful Java libraries carve directly against the grain of the language. Hibernate. Groovy. Spring. Pretty much anything that makes use of cglib. Sometimes you have to carve against the grain to cut a large problem up into easily chewed pieces.

    At this point, a Lisp programmer might attempt to convince me that the best path is a language with no grain at all. Such malleability can be hard to get one's teeth into, though.

    One thing you see a lot in Java programmers is the use of Javabean-style properties: classes with a pile of "get" and "set" methods that allow you to manipulate their state. Now the arguments as to whether this is acceptable practice or Considered Harmful are legion, and I'm not going into them today. Suffice to say I spend most of my time writing Java, and some habits get carried around with you.

    One thing I wanted to do in Ruby, though, was to find some way to lessen the pain caused by having to change a bunch of properties at once. I did so by adding a method to the Object class that would allow me to pass in a hash of name->value pairs, and have them automatically mapped to object property "setter" methods. (In Ruby, the convention for a setter is to have a method called "foo=", which through a little operator-overloading magic allows you to write "myobject.foo = bar" in your code)

    class Object
      def set_properties(props)
        props.each { |key, value| self.send("#{key}=", value) }
      end
    end

    This makes the following fragments of code equivalent:

    1. user.name = "Charles"
      user.email="cmiller@pastiche.org"
      user.website="http://fishbowl.pastiche.org"
    2. user.set_properties ({ :name => "Charles", 
        :email => "cmiller@pastiche.org", 
        :website => "http://fishbowl.pastiche.org" })

    Of course, one of the most important things one should make sure of when writing a shorter way to do something, is that the short way isn't, say, 30% longer than the one you started with.

    "Alright, we're here. Let us never speak of the shortcut again.".

    Found on AccordionGuy's blog, this Penny Arcade strip pretty much sums up the Internet:

    Four things I have learned about Sybase Adaptive Server Enterprise in the last two weeks:

    1. It (or its JDBC driver) really doesn't like you putting 'null' in a CLOB column.
    2. It (or its JDBC driver or it's Hibernate dialect) is far less forgiving than many other databases of you using java.util.Date in queries in place of the java.sql.* equivalent
    3. It is far less forgiving than many other databases of you referring to a column in an order-by clause in lower-case when you used upper-case in the select part
    4. It seems to be designed to keep certified Sybase DBAs in full employment, and not run at all without one around to hold its hand. For example, rather than defaulting to grabbing disk space when it needs it like most sane databases, Sybase requires that you learn how to manually assign static chunks of disk to the database before you can create a database bigger than 2Mb.