November 2002

« October 2002 | Main Index | Archives | December 2002 »



  • 12:24 PM

When looking through the comments on Slashdot stories like this, and seeing the kind of ignorant garbage that gets moderated up, it really helps to keep repeating to myself: “They're only kids. They've probably never developed anything longer than a 100 line Perl-script in their lives.” But it's so disheartening to see so much ignorance concentrated in such a small place, labeled as the “voice of nerd culture”.

I was looking for something I wrote a while back about C++, and came across these two quotes in my blog archives:

“C++ is an octopus made by nailing extra legs onto a dog.” —unknown

“I quickly found out that C++ programmers are the smartest programmers in the world, as each one I met would quickly point out to me.” —Don Box

The basic unit of work in eXtreme Programming is the User Story. User stories describe some functionality that is to be added to the system under development. From the story, it should be possible to create a set of automated acceptance tests that, by succeeding, show that the story has been correctly implemented. Ideally (but sometimes not in practice) User Stories give the developers complete freedom to decide what the system should do internally in order to satisfy the user's needs.

The idea is, in a well-factored (OO?) system, you can treat bits of functionality independently, estimate them independently, and shuffle them around in the development schedule at the whim of the customer. Most of the time, in my experience, this works pretty well in practice as well as in theory. Often you discover there are far fewer inter-story dependencies than you originally thought.

Time after time on projects, though, a set of requirements end up written on cards that are the round peg to the User Story's square hole. They can't be treated as stories, they are something else. These requirements are: security, usability, performance, consistent architecture, and logging (especially audit logs). These are generally shoved in the catch-all category of Non-Functional Requirements. They get written on a card, pinned up onto the board with the rest of the cards, and worried about for the entire project because we're just not sure what to do with them.

One problem is that they violate the idea that each User Story is an independent piece of work that can be estimated in isolation. Each of these requirements places a burden on every story that gets developed, and often can only be estimated with all the other stories in mind. The classic XP response to this is that these are obviously not general requirements, but should be cut up and subsumed into each story that gets developed. I'll get back to this approach later, it's an important issue.

The other big problem is that worst way to develop some of these requirements is the piecemeal, story-by-story approach. The worst kind of UI is the one that accretes over time, with functions added here and there as they get developed. The worst kind of security is the one that treats each of the components of an application in isolation, and doesn't pay enough attention to how they fit together as a whole.

Security is particularly tough, because it goes against the grain of engineering practices. As Bruce Schneier wrote to comp.risks in November 1999:

Security engineering is different from any other type of engineering. Most products, such as word processors or cellular phones, are useful for what they do. Security products, or security features within products, are useful precisely because of what they don't allow to be done. Most engineering involves making things work. Think of the original definition of a hacker: someone who figured things out and made something cool happen. Security engineering involves making things not happen. It involves figuring out how things fail, and then preventing those failures.

Breaking these requirements up across stories also means that there's no single person with whom the buck stops. When you have a requirement, but no individual responsible for seeing it happen, you'll either end up with a piecemeal, incomplete solution, or one heroic programmer will unofficially take it upon themselves to make sure it happens (I'm not theorising here, but speaking from experience on a number of projects with varying degrees of XP-ness and agility). Trusting in the appearance of a hero is depressingly common in IT projects, but it's always bad planning, and it's one of the things that XP is supposed to protect us from.

I was looking into Aspect-Oriented Programming a few weeks ago, when it grabbed me. These aren't non-functional requirements at all. They're the requirements equivalent of cross-cuts. I internally labeled them in my head as cross-cutting concerns, and the term fit so well that my brain's been using it ever since.

Aspect Oriented programming is the natural extension to Object Oriented programming because it addresses something that was not handled very well by objects: concerns that are not easy to modularise because they cut across multiple modules or class heirarchies. Cross-cuts.

In OO, the solution to cross-cutting inevitably involves duplicating code across objects, refactoring mercilessly to minimise the extent of that duplication. The responsibility for each cross-cut gets placed in every object it effects. In XP, the solution to cross-cutting concerns inevitably involves duplicating these concerns across stories, refactoring so that stories can share code and techniques to minimise the duplicated effort. The responsibility for each cross-cut gets placed in every story it effects.

Just like AOP improved OO by recognising the existance of these cross-cuts and creating a new facility to deal with them explicitly and efficiently, XP could be improved by recognising the existance of cross-cutting concerns, and devoting explicit resources to them across User Stories. This means giving somebody responsibility for overseeing the system-wide architecture, someone to do the overall UI design, someone responsibile for overall security, even someone responsible for audit trails and logging. Who those someones are could change from iteration to iteration, to keep viewpoints fresh and open up the exchange of ideas, but they should be assigned.

This is destined to be an unpopular notion, because assigning someone this role explicitly will increase the load factor, and reduce the team's throughput of user stories. If someone spends the iteration thinking of, and testing the application's security, that's time they're not spending adding noticeable features to the program. That said, it will increase the quality of the finished product, and quality is that one lever in the XP planning game that must always stay at 100%, right?

(note: This story will probably be edited a few times over the next day or so. It kinda escaped before it was ready. Since the changes may be extensive, I'm not going to mark the edits as I normally would.)

My brother, in his capacity as a journalist (and Buffy fan), went to a “Buffy Symposium”, at which academics seriously discussed the ‘text’ that is Buffy the Vampire Slayer. He emailed me this example abstract:


This paper brings the Lacan-informed critical theory of Slavoj Zizek to bear upon BUFFY THE VAMPIRE SLAYER. The paper looks at both the internal structure of the diegetic universe of BUFFY, and considers the T.V. as a 'symptomatic' product of the current later capitalist culturo-political conjuncture. In particular, as the subtitle indicates, the paper uses a consideration of BUFFY to raise questions concerning where we might stand today vis-a-vis the enlightenment project of opposing all things spectral and vampyric in the name of rendering social and political life transparent to reason.

The paper has three parts. Part 1 gives a detailed Lacanian analysis of BUFFY drawing especially on Lacan's theorisation of paranoia in Seminar 3. Part 2 then raises questions concerning the vampyre mytheme in its historicity. Following Zizek, the suggestion is that the vampyre should not be read as a premodern mytheme that troubles modernity from the outsider, so much as its own internal excess. Part 3 then raises the question of BUFFY and later, or 'post' modernity, considering the ironic mediation or distantiation that it brings to its refiguring of the vampyre theme. The concern of this third part is Zizek's identification of the essential cynicism entrenched in later modern social reproduction. Contemporary power can always laugh at itself, Zizek argues. How this positions BUFFY's light-hearted self-referentiality is my concern.

I think I speak for everyone when I say... What the...?

Marcus J. Ranum comes up with a nice summary of Gartner on the firewall-wizards mailing-list.

You've got to understand that most of the input into Gartner is from briefings arranged by the marketing departments of companies that are paying them to listen to their briefings. Basically, Garter sits at the apex of the hype food-chain; they consume pure hype and produce little sh&t-pellets of hype that is[sic] as dense as neutronium.

Mark Pilgrim has upgraded his site to valid XHTML 1.1, and in the process provides several examples of why that's really quite a bad idea. I shall stick with my grossly invalid approximation of HTML4.

In addition, he starts the push to the second stage of reducing weblog bandwidth costs, mod_gzip. I considered mentioning this back during the Get thing, and it was suggested to me several times that I should have, but one axiom I've learned is: “Make one suggestion at a time, or you'll confuse people.”


John Robb takes a look at DSpace, and responds: “Earth to MIT, give everyone a Radio weblog.“ OK, so Robb has a product to sell, he's perfectly entitled to bang the drum a bit. But that doesn't stop him being wrong.

Weblogs are fun, and interesting, and a very neat way of exchanging timely information in a decentralised environment. But they have some significant weaknesses that make them unsuitable for the task of providing a repository of academic research:

  • Very time-constrained. Weblogs are organised chronologically. Posts fade from the blogosphere rapidly. DSpace acts more as a directory, where writing is organised into collections and communities.
  • Limited metadata. Generally, the post is all there is.
  • Limited workflow. The emphasis is on getting the post out there as quickly as possible, and then moving on, which is anathema to the process of producing a scholarly paper.
  • Ad-hoc quality control. A weblog post becomes prominent through what is essentially a big linking popularity contest. Introducing a Google search appliance (as Robb suggests) makes that even more an issue.

For example, I'd really like to harness the l33t sk1llz of the java.blogs community to create a centralised repository of our collective knowledge. All the information is out there on various weblogs, but its diffuse and mixed together with stories about our cats. A central site would give a newcomer the distilled core. I went as far as downloading and installing DSpace, to see if it could be useful. Perhaps after I've finished my manic week of studying I'll have more time to mess around with it.

It's not an either-or situation. Encouraging academics to use weblogs is a Good Thing, because weblogs are a great tool for freeform information-sharing. On the other hand, an organised, centralised repository can very effectively complement the informal distributed cloud. Weblogs are not the solution to every information architecture problem, any more than a hammer is the solution to every carpentry problem.

Here's another example: a military base protected by a fence and motion sensors. The attackers take a rabbit and throw it over the fence; then they leave. They motion sensors go off. The guards respond, find nothing, and return to their posts. The attackers do this again, and the guards respond again. After a few nights of this, the guards turn the motion sensors off. And the attackers drive a jeep right through the fence. This kind of thing was done repeatedly against the Russian military bases in Afghanistan, and in tests against several U.S. military bases. It's surprisingly successful. —Bruce Schneier, Secrets & Lies ch. 3

This quote sprung to mind after seeing yet another warning of imminent terrorist attack on the front cover of the newspapers. The warnings are always the same. “Al Qaeda are planning to attack in the US, or Europe, or Australia. They may use a train. Or a boat. Or a dirty bomb. Or poison gas. They are quite likely to hit a populated area. Or docks. Or a nuclear facility.” This is about as useful as telling us that there's a good chance the sun will rise tomorrow.

Our intelligence services are getting exactly the same amount of useful information as they always have. But because we blamed them for not warning us before the attacks on New York (and more locally, Kuta), they've just lowered their clipping level. That, unfortunately, doesn't make the information any more useful. We should consider these warnings to be a failure condition, proof that our alarm is still unable to discriminate between a rumoured attack and a real plan.

Maybe there is no way to make that discrimination? If so, we need to change the way we work, because the paranoia created by these constant warnings is unnatural, and can not last. Unless we find something other than paranoia to protect us, we're eventually going to turn the alarms off and be more vulnerable than ever.

To Mars!

  • 3:36 PM

I'm studying for my CISSP exam on Saturday. It's really bloody hard going, but I think I might make it. Anyway, the textbook study-guide I'm reading from contains the line:

DCOM uses a globally unique identifier (GUID) and DCE usees a universal unique identifier (UUID)

My first reaction was to wonder if this means that DCE is the only standard safe to use on other planets?


  • 4:32 PM

iPulse is pretty cool. It's a system monitor for OS X that can display CPU, disk, memory and network usage (and the time) all in a compact radial layout. (Found on Hack the Planet)

When I go to the ‘Official Site’ for a computer game, there's a good chance that I already have the game, and am looking to see if there are any patches available. Put that information, or a link to it, on the front page. Even if there are no patches, link to a page that says there are no patches.


  • 10:34 PM

I would just like to praise the incredible resiliance of the Linux operating system.

I overwrote the first megabyte of my primary hard drive. That's the boot sector, the partition table, and a large chunk of the start of my root filesystem. I have now recovered to the point where I believe (believe, but have not tested) I can reboot the system and have it come up in one piece. I've recovered my partition layout and built a new table. I've fscked the root filesystem and restored the obvious damage. I was even able to backup /home to another drive, just in case the recovery hasn't been as successful as I thought.

Some might tell you that I should never have been given the power to do such immense damage by mistake. That it's the fault of the OS. Sure, it probably should have warned me that I was about to overwrite something important, but for seven years I have been taking advantage of the way Linux does what you tell it without question. For seven years I've marvelled in the power and flexibility that gives me.

Ask any Unix system administrator, and they'll tell you about the one time they stopped respecting the OS. Unix respects you, it trusts you, because the only way to gain trust and respect is to offer it yourself.

And tonight Linux showed me how much it deserved my respect. I damaged its root filesystem beyond recognition and it didn't crash. It still hasn't crashed. I looked in directories with corrupt descriptors, files that made absolutely no sense, and Linux didn't panic on me, it just reported what it could see to the best of its ability. “Mate, that file looks totally screwed. I'm going to remount the filesystem read-only until you tell me you've fixed it. OK?” It let me rebuild the partitions. It let me fsck the broken partition and rewrite the boot block.


Oh God....

  • 7:52 PM

I meant to type: dd if=rescue.bin of=/dev/fd0. I typed dd if=rescue.bin of=/dev/hda. I have utterly destroyed the partition table on my primary hard drive. I have no backups. I am so completely fucked.

Don't drink and su.

Egress Filtering

  • 11:25 AM

Darren Hobbes points out egress filtering as a way to combat DDOS (in reply to my Seven Things Wrong With the Internet)

Egress filtering is a simple idea. When a packet crosses over between your network and another, check its source to make sure it says that it comes from inside your network. If it says it comes from anywhere else, it's forged and you should drop it on the floor. (There's also ingress filtering, where you do the reverse to protect yourself from packets that come from outside, but are attempting to imitate your own servers)

Egress filtering is a good idea, and every router should implement it as a matter of course. Sadly, a lot of them don't, and it's one of those things where even if you have egress filtering working fine across your border, you're not safe unless everyone else is doing it too. And sadly even though it's been a security best-practice for years, a lot of people don't put it in just for this reason—there's no immediate benefit to your network to have it in place, so it's not thought of.

A lawsuit or two from victims of spoofed attacks would fix this, but can you imagine trying to prove in court where spoofed IP packets came from?

On top of this, most DDOS attacks these days don't bother with spoofing, which makes egress filtering ineffective as a defense. Hell, most of them come from trojanned Windows 9x boxes that are incapable of packet spoofing. They just work because there's an infinite number of monkeys with typewriters out there, willing to install the latest trojan floodbot.

The Internet is in serious need of an upgrade in a number of areas. Some of these problems already have fixes, they're just not being rolled out because it's considered too hard, or there are large vested interests in maintaining the status quo.

This was originally going to be a top ten, but I ran out of writing time during my lunch hour. I may amend the list later.

  1. Security

    Occasionally, we read about the big DDOS attacks on players like Yahoo, or the DNS root servers. We look, see that Yahoo is still up and our domain names still resolve, shrug and go on with life.

    Someone as big as Yahoo or a root server can withstand a DDOS. The rest of us aren't that lucky. The current state of the Internet is that anyone with a grudge and sufficient spare time can do serious damage. holding to ransom peoples livelihoods. And you don't even have to do anything wrong. Bob the Warez Kiddie says the wrong thing on IRC, and suddenly his whole ISP is out of action for a month. These things happen all the time.

    The Internet is in desperate need of re-architecting so that you aren't under constant threat of having your business or leisure held to ransom by untraceable fourteen-year olds

  2. The Domain Name System

    Before the Internet, there was a system of reserving names called ‘the Trademark’. Perhaps you're familiar with some of them? When the trademark system was put together, it was recognised that due to the severe limitations of language, the scope of trademarks would be pretty narrow. You couldn't trademark common words or phrases, and your trademark only extended as far as your sphere of business.

    On the Internet, you can pick any name, and have it held globally by a single entity. There are geographic namespaces, sure, but that's still trying to squeeze too much out of a small language. In the trademark system even holding on to a single name across one country, let alone globally, is generally reserved for only the biggest businesses, who can demonstrate some need to have that word associated with them, and only them.

    DNS was a great tool when the Internet was a cooperative system. Now it has grown up into a competitive system, and the domain name system is no longer the right tool. We need to dispose of it, or at least make it largely invisible, and replace it with a more mature directory that doesn't have the shortcomings of its predecessor.

  3. IP Address Scarcity

    They're numbers for Christ's sake. Do you realise how ludicrous it is that the power and flexibility of the Internet is limited because we're running out of numbers? That you have to pay your ISP twice as much for your own number?

  4. The fixed-price, unlimited access myth

    Internet providers are continuing to propagate the myth that flat-rate Internet service is fair. It isn't. Your cost to your provider is directly related to the resources you consume. The 80/20 rule says that the top 20% of users use 80% of resources. (In my network admin days, I also noticed there's a 50/10 rule) This means that the bottom 80% of users are unfairly subsidising the top 20%. It also means that the providers are doing everything they can to limit the top 20%'s ability to do what they want.

    On many broadband services you're not allowed to run a server, and your IP address is artificially cycled. Why? What possible difference does it make to an Internet Service Provider if a user is running a webserver or not? Simple. Because they're supporting the fixed price, unlimited access myth, the provider must find ways to cut down on the things users may do that consume bandwidth. Surely, instead of arbitrarily refusing certain legitimate uses of TCP/IP, it would be more logical to make people pay for what they use?

  5. Drifting away from interoperability

    The strengths of the Internet are interoperability, and decentralisation. Look at the biggest thing to hit the Internet since the World Wide Web: Instant Messaging. The four main services, AIM, MSN Messenger, ICQ and Yahoo Messenger, are all incompatible, and remain so despite the occasional press release saying they might work together some time in the distant future. Where the Internet was built on the idea of creating a network to join together all the disparate closed networks with simple, interoperable protocols, there seems now a move towards trying to corner markets by isolating your users from your competitors.

  6. Drifting towards centralisation

    It's far easier to write a centralised service than architect a distributed one. So that's what everyone is doing. I'll pick on Instant Messaging again, the top four all rely on a central directory controlled by a single company. The trend seems to be away from writing services that get installed at each ISP and join together to form the Internet, towards services that exist on one server out there somewhere, that we hope stays up.

    Similarly, there was the promise a few years ago of a distributed caching architecture (via caches like Squid) that would reduce the bandwidth consumed by the web, and thus the cost to webservers. All the current kerfuffle about the inefficiency of RSS would be academic if we had a proper network of caches between us and the end servers. Sure, there were problems with the system, but instead of fixing those problems we seem to have abandoned the idea entirely. Instead, the closest things we get are either the incredibly specialised (and server-pays) Akamai-style edge servers, or when a site is slashdotted, the (once again centralised) Google cache.

  7. Amero-centricity

    Too much of the Internet's infrastructure and organisation is based in the USA. This makes the global Internet too beholden to the political interests of a single country. Sometimes you see the US government getting this “Hey, it was our defence department that started this thing, it's ours!” look, which generally means its time to run and hide.

A cow orker showed me this one. The game is called Porrasturvat, a word the author vaguely translates to English as ‘Stair Dismount’. It's a physics simulation, the physics in question being just how much damage you can do to somebody when you push them down a flight of stairs. It's more than a little scary how much enjoyment one can get out of the simplest cruelties.

(This was on slashdot a few days ago, but I missed it)

This really shits me. (link found on RockNerd) According to EMI Germany:

There are 250 Million blank CDRs and tapes bought and used this year for copying music in comparison to 213 Million prerecorded audio media. This means the owners are only being paid for 46 per cent of the musical content.

Bullshit. At every company I've worked for, we've gone through CD-R's quicker than we go through coffee. Backups, software distribution, file-transfers, coasters, you name it. When I was in Perth, the ISP I worked for couldn't find an economical CD pressing deal, so whoever was on the front desk would spend each day making sure the burner was cutting copies of our setup software. In contrast, most music copying goes on purely electronically, from hard drive to hard drive. (Then again, I haven't personally used a p2p file-sharing network in a very long time. I deleted Napster long before they crashed into nothingness because I used it so rarely. I was far more likely to get interesting things from my friends than an impersonal search-engine)

The number of blank CDs sold is absolutely no indication of the amount of music that is being copied. It's a red herring.

Copy-protecting CDs is worthless. Because the music is digital from start to finish, it just takes one copy to leak out unprotected and it's game-over—that one copy can multiply endlessly. Look at how the best quality pirated video CDs are transferred straight from film “borrowed” from the official distribution system. Look at how many popular CDs make it to the Internet before the music is officially released to the public. Do the maths.

All CD copy-protection does is inconvenience the consumer. It limits the devices we can play our discs on. It prevents us from playing them on our portable mp3 players. And yes, it prevents us from casually ripping them and sending them to our friends. (Although, as I said above, those songs will still end up on your p2p network of choice)

I have a friend called Susan, and we share a similar obsession with music. Our tastes are quite different, but that's more the influence of what was around us. The problem was, back in 1996 she lived in Idaho, so if I wanted to play her a new song I'd heard that I thought was really cool, I had to rip it and send her the mp3. I played her stuff she didn't usually get exposed to: trip-hop, brit-pop, Australian alternative music, even some obscure American stuff she'd never heard of.

A year or two later, she visited me in Perth (well, actually she lived with me for three months, but that's an entirely different story). She brought her CD collection with her in a big black case (this was before the advent of the portable mp3 player, and us music obsessives won't go anywhere without several hours of music). In the case I found a great many CDs that she had bought on the strength of the mp3s I had sent over the years, money to the record companies and the artists. Free money, because I paid the promotional costs.

Good music spreads. Music is something essential to the human spirit. There is no other art-form that grabs us, that speaks so directly to our emotions. Look at cinema. You can have all the great acting and cinematography you want, but if they really want to get a strong emotional reaction from the audience, they turn up the music. When we hear a good song, we want to play it to our friends, it's just that when before we could just invite them round and play the CD, now our reach is global.

Markets are conversations. By stifling the casual conversations in a way that does nothing to disrupt the large-scale, impersonal copying, the record companies are stifling their market. The just don't notice, because it's a new market. It's money on top of what they were making before, and they're killing it with their ignorant knee-jerking.

I will never buy a copy-protected CD. It's impractical because these days, a CD I can't transfer to my iPod is a CD I'll barely end up listening to. The day a CD comes out that I want, but that only exists in a copy-protected form, is the day I return to the p2p networks. I bet you any money I'll find the whole album there. If an industry believes that its oligopoly is an excuse to treat customers like criminals, then I will quite happily satisfy their presumption.