November 2003

« October 2003 | Main Index | Archives | December 2003 »

30
Nov

Misc

  • 8:43 PM

1.

While having my leaving-work drinks last Thursday, I had another encounter with the "Three Minute Angels".

It's a pretty clever business-plan. Recruit a small army of young, attractive women. Send them around Sydney's pubs in the evenings after work, offering neck and shoulders massages to the patrons (most of whom are cubicle-dwellers like myself). Technically, they seem to be covered by the same laws as buskers, which means they can't set prices, just ask for a 'voluntary donation' after the massage is over.

Of course, nobody out in the pub after work will want to look stingy in front of their mates.

The massage was great. I rarely exercise, spend most of my day sitting at a computer, and have disgustingly bad posture. A relaxed neck and shoulders is just what I need.

2.

While trying (unsuccessfully) to find a website for the afore-mentioned angels, I came across this old article from the Sydney Morning Herald describing the way a PR company is attempting to fake word-of-mouth advertising by sending out employees to give people "personal recommendations".

I'm pretty sure I've seen this in action.

I got on the train once, I think it was on the way back to Sydney from Alan's place. Two people got onto the train in the middle of a rather loud discussion about mobile phone plans. Something about it just struck me as being a little... artificial. It's amazing how cynical you have to be these days. All the advertisers desperately want to fake 'genuine', and those efforts are slowly making a fiction of everything.

3.

My cousin Kate and her husband Nick are visiting Sydney this weekend, so I've been doing my bit to show them around. Yesterday, we climbed up to the top of the Harbour Bridge. They don't let you take your own camera (or in fact, anything you could potentially drop) up with you, but they do take a few official photographs, which they then sell back to you for an exorbitant sum.

It was really cool. We managed to pick the clearest day in weeks--no clouds, low pollution--so you could see what seemed like forever in every direction. And standing on the cross-platform right at the top of the bridge between the flags, looking down through the mesh floor at eight lanes of traffic a hundred metres below, was pretty trippy for a guy who suffers from mild vertigo.

I didn't notice the vertigo for the whole climb: I enjoyed the whole thing and felt really relaxed about it. Then, about halfway down the arch on the way home, I noticed that my legs were shaking. Some little gland in the corner of my otherwise calm brain must have been quietly screaming what the HELL are you doing up HERE?, and pumping adrenalin through my body without me even noticing it.

While the climb was cool, though, I can't really say it was $175 worth of cool. Certainly not something I'll be doing again unless they drop the price, or some gorgeous female tourist begsasks me to accompany her.

4.

Today my mother and I took the cousins to Bondi. English tourists have to go to Bondi, it's just one of those necessary, inevitable facts of life.

We had breakfast (well, for some value of breakfast, since it was well after midday before we ate anything) at a cafe by the beach. A few doors down, some very picturesque women wearing very little were giving away candy-floss. Sadly, I didn't get any photographs, nor did I ever find out quite what they were selling.

There seemed to be pairs of them dotted around the foreshore. I started wondering whether they were being secretly cloned in vats.

Also, while wandering down by the beach, my mother remarked to the tourist cousins that you don't really get any topless bathing in Australia, which is blatantly untrue, and makes me wonder if female eyes work somehow differently to male eyes.

Ethereal Memory

  • 2:03 PM

On Monday morning, I came into work to discover two-thirds of the management team kneeling beside a dismantled Linux box which had until some time during the weekend served as the office router and firewall. The power-supply had given up the ghost, and until a new one (AT, not ATX!) had been procured, the office would have no Internet connection.

It's amazing how used one becomes to being always-connected. Unless I'm out in the middle of nowhere, I expect that when I sit down at a computer, it will have an Internet connection, and I will have access to that great global store of information. Without it, half my brain seems to freeze.

This morning, I was reading the Umberto Eco essay linked off Slashdot: Vegetal and mineral memory: The future of books (and being amused at the slashbots demanding that Eco be more "user-friendly"):

...Hermes, or Theut, the alleged inventor of writing, presented his invention to the Pharaoh Thamus, the Pharaoh praised such an unheard of technique supposed to allow human beings to remember what they would otherwise forget. But Thamus was not completely happy. "My skillful Theut," he said, "memory is a great gift that ought to be kept alive by continuous training. With your invention people will no longer be obliged to train their memory. They will remember things not because of an internal effort, but by mere virtue of an external device."

To some extent, the Internet is now an extension of my memory.

Modern computer hardware and operating systems support the concept of "Virtual Memory". Instead of programs having to speak directly to the RAM hardware to store data, the operating system assigns the program a big block of memory that doesn't really exist. The Operating System can then map between that virtual memory, and the real bits and bytes on the hardware. If the Operating System assigns more memory to programs than it has available, it can move some of the data out to "swap space" on the hard drive. So long as it remembers where it put the data, the OS can pull it back later when the program decides it needs it again.

I am increasingly finding that the Internet is my swap-space. I've stopped remembering "things", and started keeping a catalogue of references to things. If I find I need the information, I have enough recollection of where to find it that I can access the Internet and swap it back into main memory. Luckily, I have a really good associative memory for context, which helps me come up with the Google search I need to track down data I've otherwise swapped out.

Sure, this is a slower process than just keeping it all in core, but most of the time it's fast enough.

Thamus would have it that this is lazy, and that I need to train my memory more. I disagree. One of the biggest problems these days is information overload: there's just so much information out there that needs to be digested and kept on hand. Using the Internet as my swap-space, retaining only pointers to information instead of the information itself, leaves much more room in my head for important, immediate things.

Although I still can never quite remember where I put my apartment keys.

This one is going to annoy pretty much all of my dotnetblogger readers. You may want to skip it for now. :)

Hiawatha Bray, As Google grows, critics emerge, The Boston Globe:

Do you hate Google yet? At first glance, the question seems absurd. What's to hate about an Internet site where you can go and find out about practically anything, free of charge?

Less than a decade ago, you could have said the same of Microsoft Corp. It was once viewed as an heroic American institution, an upstart software company founded by a Harvard dropout who became a billionaire by outsmarting IBM Corp., the world's biggest computer firm...

Robert Scoble, in a comment in his weblog

...partly cause of American culture itself. We all love to see the powerful fall. Look at why all the TV stations covered Michael Jackson this week. Or, why many people who don't live in NY cheer against the Yankees. Our mythology is full of David vs. Goliath.

In Australia, we call this the "Tall Poppy Syndrome". When the poppy grows too tall, we find great delight in lopping off its head. It's something deeply ingrained into the psyche that when you become successful, people will want to bring you down.

Defenders of Microsoft would like to have us believe that it is a victim of its success. That up to the early- to mid-90's, everybody loved it, but then the enormous success of Windows 95 made them a target, and the whole Netscape thing became a hook by which the jealous tried to bring the company down.

This is pure revisionism. The truth is far from the conspiracy theory. Microsoft has always been a pretty nasty piece of work, as far as companies go. The only thing that has changed in the last decade has been the public visibility of the company, brought forward by the rise of the Internet, and the final movement of the personal computer into the realm of a must-own appliance. What's changed has not been Microsoft's behaviour, but its ability to do harm.

Post hoc ergo proptor hoc is such a convenient fallacy to fall into. As Microsoft grew in power and visibility, the negativity it began to attract grew. It's easy to blame the latter directly on the former. The truth is that the criticism is justified. Microsoft have seen their power grow, but never developed the corporate ethics to prevent the abuse of that power. As James Gleick wrote in Making Microsoft Safe for Capitalism

Microsoft is hardly alone, of course; plenty of its competitors would play as rough, if they only could. Others in the industry suggest that Microsoft's small-company scrappiness has kept it from facing the issue of corporate ethics: behavior that people will forgive, or at least understand, in a start-up looks considerably less attractive when David grows into Goliath.

Read Gleick's article. It discusses an antitrust investigation into Microsoft that began in 1990, an investigation that ended with no findings of guilt, but a consent decree. Gleick again:

[Anne Bingaman, assistant attorney general in charge of the Antitrust Division] also said: "I hope consumers, within a short period of time, will have more choice of operating systems."

It has not happened. The practices Microsoft agreed to forgo had already served their purpose. Gates was right when he summed up the effect of the consent decree in one word: "Nothing."

Sound familiar? Microsoft have since weathered another Antitrust investigation. This time they had to contend with a damning and comprehensive factual judgement. As a result they were forced to agree to a consent decree that asked them to allow competition in markets they had already burned to the ground. Sure, we are now seeing a minor resurgance of alternative web browsers, but that has nothing to do with any consent decree, and everything to do with the fact Microsoft have simply stopped bothering to improve Internet Explorer in the areas that people care about.

And meanwhile, Microsoft are pushing their tendrils into the lucrative and incredibly scary area of Digital Rights Management: the market that will decide over the next decade how we will be able to watch movies and listen to music -- and who will be allowed to build and market the devices that will allow us to do so. Eventually, we may have another antitrust investigation into this new leverage of the Microsoft monopoly, but given the pace of justice, that market will be dead too by the time it's finished.

And all the talk from Scoble about how Microsoft employees all need to have a values statement now doesn't wash with me.

Bill Gates' original goal in forming Microsoft was famously to have (emphasis mine) "A computer on every desk and in every home, running Microsoft software". You'll not find the last three words of that sentence in any official Microsoft history (or at least I couldn't, and I searched hard). They've been carefully un-happened: the dream of a nascent monopolist truncated into a facade of altruism.

The words still hover over every Microsoft strategy. A friend who worked in an executive position for Microsoft in the early 90's has (between hinting at horror stories of strong-arming OEMs), shared tales of the Bill Gates Cult of Personality that permeated everything the company did. Every decision was coloured with the question "What Would Bill Do?" By looking at Microsoft's actions rather than its words, looking at where the company is going and what it's still doing to get there, Bill clearly still believes that what is best for us is what is best for him: everyone running his software.

Which, belatedly, brings us to Google.

Google are reaching a point now where they've become synonymous with web searching in the same way that Windows is synonymous with the Operating System for most computer users. They're also reaching a point where they are growing as a business, trying to become more professional, trying to offer services that lead away from their core competence.

Similarly, they are also discovering their ability to do harm. They are discovering that being in such a position makes them a lumbering elephant, and it's too easy for those big feet to step on the smaller animals. When they make a mistake (as they do sometimes with the AdSense terms of service, or the maintenance of their search index), the repercussions are so much greater than if they were just Joe's Web Shack. When they move into a new area of business, their new competitors feel much more threatened.

The difference, I hope, is one of corporate philosophy. Just as famous as Gates' vision for Microsoft is Sergey Brin's vision for Google: “Don't be evil.” We can hope that the problems Google are encountering right now are growing pains, are honest mistakes, and that fears of them being malicious are unfounded.

Of course, this optimism only extends so far. For some people they've already crossed the line. I'm still giving them the benefit of the doubt. They've done a few stupid things, and a few questionable things, but I'm still waiting on 'evil'.

The subject line of this spam reads: ‘Seeking Perfect Male Body’

It's not spam, it's got to be targetted! They knew exactly where to find me!

(Hey, after a billion emails telling me I need a larger penis, its nice to get some encouragement now and then)

Jar File Hell

  • 8:09 AM

It seems it's impossible, these days, to write a decent-sized Java application without becoming reliant on fifty different library Jar files, culled from various open-source projects.

On one hand, this is great. It's great that Java has such a vibrant culture, and is producing such a large volume of helpful libraries. It's great that if you think "Hey, I'd like to do foo", there's probably a halfway decent free implementation of foo4j waiting for you to download it. Which brings us closer to that whole 'reuse' thing that everyone's been shouting about for years.

It also makes for frustratingly fragile applications.

You've got fifty libraries. Each may depend on, or plug in to some combination of the other libraries. Each has its own bugs (quickly fixed of course), and thus its own release schedule. New releases may change dependencies on other jars. We end up building tools to deal with this complexity, but then the tool just becomes another dependency.

And, of course, each library has its own configuration file. Configuration and properties files swarm over J2EE applications like kudzu, each with its own format, and each resisting to the end any attempt at centralisation or rationalisation. Want to have one big central switch that changes your app over from development to production mode? Well you'll have to script it externally, because the necessary changes are spread across two properties– and three XML files.

(And any attempt to provide a standard programmable configurability or configuration discovery mechanism would lead to: seven or eight competing implementations, arguments on the merits of which version of IOC is best for the task, flame-wars about the JCP/Jakarta/the EFnet #java ops, and yet another damn dependency!)

And you (OK, I) end up in situations where B is throwing a NullPointerException where it's being called from A (via the helpfully provided A->B plugin). Is it a bug in A? Is it a bug in B? Is it really a bug in C, the invisible piece of infrastructure that introduced A to B? Is it some missing or malformed piece of vital configuration that neither A, B nor C have warned me is wrong? Hello, several hours of fruitlessly debugging unfamiliar fragments of Other Peoples Code.

It's enough to do your head in.

Hacknot's Mr Ed, Agile Methodologies critic and otherwise interesting blogger who has sadly turned into a one-trick pony has linked to an ACM Queue article on stand-up meetings by Phillip Laplante. Laplante has long experience of stand-up meetings, and he's not afraid to say that they suck.

I know it's an Extreme Programming cliché to say "Well, you weren't doing it right!" in response to criticism. In this case, though, it seems to be the only appropriate response. I've had my own bad experience with stand-up meetings, but I've also had good ones. I think, however, that I've been able to observe what makes the meetings good or bad. So here's a couple of stand-up meeting antipatterns that I have observed, and refactored solutions that I have seen work.

I owe a debt of gratitude to Keith, who ran the stand-up meetings on my last project, and is thus responsible for quite a few of the good practices.

Magic Bullet

Problem: Stand-up meetings are seen as a substitute for full-length meetings, or in fact for any kind of developer interaction.

Discussion: The main problem with stand-up meetings comes from either not understanding what they are supposed to be, or not applying that understanding. A stand-up is a daily checkpoint between team members. It's a way to keep track from day-to-day of what other people are working on, what they are accomplishing and what broad problems they are encountering.

Deciding that all other team interaction is unnecessary just because the team talks to each other for ten minutes every morning is ludicrous. It's not the stand-up format that's to blame here, it's poor management. Believing that a stand-up obviates the need for more formal meetings, or for team members to talk between themselves or with team leaders outside the meeting is just very, very wrong.

Refactored Solution: Keep Talking

On the last page of his article, Laplante describes "management by walking around" as an alternative to the stand-up. When I read it, I couldn't help thinking "That serves a completely different purpose to the stand-up. Why can't you do that as well?"

Don't treat the stand-up as the only time the team can get together, formally or informally. Continue to have informal hallway meetings. Continue to have formal boardroom meetings with scope to do things that are not possible in the stand-up format. You will even find that the necessity for further discussion outside the stand-up will arise from things discussed within it.

Sermon On the Mount

Problem: The manager adopts the stand-up meeting as a way to deliver the day's rant. After the rant, the meeting is over.

Discussion: Laplante describes this as a problem with stand-up meetings, but it can happen in pretty much any environment. I've not quite had this problem with stand-up meetings, but I have had it with one past job: where any meeting with our nominal project manager was doomed to be unproductive and frustrating. In such situations it's even more important that the developers on the team have these daily checkpoints.

Refactored Solution: Meetings by Stealth

Do everything you can to have the official meetings cancelled, or reduced in frequency. Then, conduct your own meetings without inviting the disruptive element. You may have to disguise them as something else: the morning cup of coffee is a popular ruse.

This said, if you ever find yourself in a situation where you have to have meetings that specifically exclude someone, you've got problems that are much greater than just the mechanics of a meeting.

Aching Legs

Problem: The meetings drag on too long.

Discussion: Discussions between programmers tend to get out of hand. One person has had a particularly interesting day, or a particularly intractable problem, and the meeting gets hijacked by an urge to pursue a single issue to its conclusion. Anyone not involved with the discussion stops listening and just shifts from one foot to the other, wishing they could go back to their workstation and do something productive with their precious morning.

Laplante suggests sitting down, but having an egg-timer to keep the meetings short. To me, this is still a stand-up meeting. Despite what the name suggests, a stand-up meeting can be held sitting down. The name is just taken from one mechanism that is used to ensure that the meeting stays short. If you find some other mechanism that works for you better than standing up, so be it. For me, an egg-timer would be too rigid, too formal for the largely informal morning meetings.

I also wouldn't discount standing up as a great way to make sure people stay to the point. When rambling on is keeping you away from your chair, you care about that much more than the sand in an egg-timer.

Refactored Solution: The Convener

Appoint one person to be the convener of the meeting. You're probably best picking someone who is not in a management position: that way the role doesn't get confused with being one of real authority. Pick someone who is able to be polite but firm. The job of the convener is to cut off long conversations, and suggest they be completed in another forum.

Attendees at a stand-up should be told that their task is to summarise what they did yesterday, say what they're planning to do today, and raise any problems they may be encountering. If the problem can be solved on the spot, that's great. If the search for a solution starts spiralling off into free-form discussion, that's time for the convener to step in and suggest that perhaps those involved in the discussion get together after the stand-up to hash out a solution.

Chaos

Problem: Meetings are disorganised.

Discussion: Laplante worries about the total lack of structure to a stand-up meeting. There is no agenda, nobody knows who should speak first, and it's too easy for them to be dominated by the louder personalities to the detriment of the quieter ones.

Refactored Solution: The Token Ring

This solution pretty much started as a joke. We have this foam football that gets thrown around the office, and one day we were in the stand-up meeting, and it got jokingly turned into a "talking token". Then, the next day and the next meeting, we looked for the token again...

The meeting is convened by having the team stand in a circle. This generally isn't a problem, because that's how people tend to arrange when they're talking in a group. The convener tosses the talking token to someone completely at random. From then on, there's a general understanding that when someone's holding the token, they've got the floor. Either they're talking, or you're talking about them. When the person is done, they pass the token to their left.

Sensitivity Issues

Problem: Stand-up meetings disadvantage the shy or disabled.

Discussion: I can't help thinking Laplante was reaching with this one, but I'll address it anyway.

Stand-up meetings probably aren't suitable for physical reasons. For example, those with an inability to stand or those with speech, hearing, or language disabilities must find these meetings excruciating--if not humiliating--so isn't this practice discriminatory?

Those who are shy and have difficulty speaking or being in public must really dread these encounters. It must be very difficult for someone who just doesn't have the personality for a highly interactive meeting to be forced to endure one every day.

Of course, if you have someone on staff for whom the stand-up meeting is a particular problem, you'll have to make allowances in your process for that. Just as you would also have to make allowances in regular meetings. There's nothing special about a stand-up that is going to make it any more difficult for someone with speech or hearing problems than any other kind of meeting.

I will, however, grant that for the less gregarious, standing up in the middle of a group and talking can be daunting. Because stand-ups happen every morning, this can cause some additional anxiety. You don't want people to dread the beginning of the day, because that sets the tone for what's going to happen for the rest of the day.

Refactored Solution: Just Us Folks

Encourage the stand-up meeting to have the same atmosphere as, say, a chat around a cup of coffee. Have people bring their cups of coffee to the meeting. Nobody is making any formal presentation, it's just a few words about what you're doing. I'm yet to meet a programmer who doesn't like to talk about what he or she is doing informally.

If this atmosphere is not something you are able to muster during a morning gathering, then the project has far bigger problems than you can address by shifting meetings around.

Beware, however, of combining the coffee, the informality, and sitting down comfortably. Such events tend to stop being meetings, and start being chats.

Non-issue: Accountability

One last question Laplante asks: "Is there any accountability in these meetings? For example, who said what and when? How are disagreements resolved? What if someone is absent? Since there is no record of the meetings, how does one keep up if they are out for an extended periods of time? Are members sanctioned if they miss too many meetings?"

This is a big non-issue. Stand-up meetings are, as I said, checkpoints. They let you know what's happening from day to day. If the same issues start coming up repeatedly during the meetings, then that's time to do something formal about them. They're places for sharing information that is relevant to yesterday, today, and maybe tomorrow. Thus, records are unnecessary and attendance is only required if you're present on the day. If someone goes on holiday for two weeks, they can go to the stand-up meeting the morning they come back. They'll get a good idea of what everyone's up to, and when it comes to their turn to speak, they'll say "Well, I don't know what I'm doing today, and I need to be filled in with what happened while I was gone", and the convener will say "OK, why don't you and Bob talk about that after the meeting?"

Reading this article on Swing menus reminded me of how much I love programming in Apple's Cocoa framework

Cocoa is one of those frameworks that makes the simple things simple, and the difficult things possible. The standard way for handling menu item enabling and disabling echoes this approach. There are more complex techniques for more complex requirements, but the 80% solution is very simple indeed.

First: draw the menu. You do this graphically in Interface Builder, Apple's GUI interface layout tool. Except unlike other GUI layout tools, it doesn't suck. You arrange the objects, set up their properties, and tell it how they would talk to your controller objects if there were controller objects around. Then you write the objects themselves in XCode

You then connect the menu item to one of the "actions" in the FirstResponder object. For example, I would connect the "Print" menu item to the FirstResponder action called "print:"

The FirstResponder is a placeholder object. Whichever interface widget currently has the focus is the FirstResponder. Behind it is the "responder chain" of enclosing objects, all the way up to the main application object itself. (Interface elements can also be assigned "delegates" so you don't have to subclass the widgets themselves. These are also part of the chain by proxy).

Actions assigned in IB correspond to method selectors. Since the "Print" menu item is bound to the "print:" action, the menu will will search up the responder chain for an object that implements a print method. If it finds one, that method becomes the target of the menu item, and the menu item is enabled. If there are no objects in the responder chain with a print method, the menu item disables itself.

Like I said, there are other ways of achieving more complex menu requirements. This simple way just makes it so much easier to do all the simple things, and gives you an enormous amount of standard menu functionality for free. Aside from Print (OS X uses Display PDF, so pretty much everything knows how to print itself), there's stuff like this:

At the bottom of the Edit menu, there are Find, Spelling and Speech submenus...

This is the standard bottom section of the Edit menu in OS X. The standard Cocoa text area (NSTextView) knows how to hook into the spell-check service, or how to speak its contents, or how to pop up a search window. And because of the way the menus work, the menu items just hook up to whatever text view is in focus automatically.

The practical result is that OS X applications are naturally rich. Every app with even the most trivial text-editing function has a spell-checker and a text-search feature, because you essentially get them for free, fully hooked up to the standard menu UI and keyboard shortcuts the moment you drag the NSTextView onto your window in IB.

<sillygwailo> I love that how sometimes we see stuff and we think "oh man that reminds me of a weblog post from a year ago!" -- Richard

Like discussions on IRC seem to inevitably do, the subject of conversation had turned to nipples, and the merits thereof. This reminded me of an old post by Warren Ellis about how Batman should just rip the Joker's nipples off, which is especially memorable for containing the immortal line: "Choke on my fuck, Commissioner Gordon."

Which, of course, led to the quote at the top of this post.

This is one of the cool things about having read blogs for a while. There's a hell of a lot of them, and they cover topics from why PHP sucks to fish with OCD. After a while you get this cloud of references in your head, which gives you the ability to page-slap something pithy or informative on a wide variety of subjects.

It's the page slap that does it. Before the web, if you'd read something interesting, the most you could do was say "Hey, I read something interesting about that. I should dig it up and lend it to you", and then forget, because you're too busy doing 10,000 other things. Now you can have the original at hand in a few moments. Blogs are semi-permanent records, so it's very easy to dredge up a reference from just a vague memory of who said it, and what they might have said. All you need is a good search engine and you're away.

Which justifies the description of blogs as outboard brains -- we can store vague metadata in our heads, and trust the blogosphere to fill in the details on demand. But they can also be a collective consciousness, this cloud of memes.

And we take it all for granted because at the same time, it's really just a bunch of self-important nerds shouting into the void.

You really can't get much more Aussie than this story about Australian cricketer, Dean Jones:

...[Jones] went out to bat during the famous tied Test of 1986 at Chidambaran Stadium, Madras, despite suffering from a combination of dysentery, vomiting and feeling faint. The temperature was simmering around the 40 degree C mark, the humidity in excess of 90 per cent, and at one point Jones thought he was going to collapse.

“Mate,” Jones had told his batting partner, Australian captain Allan Border, “I'm going to have to go off, I just can't go on, I'm too crook.”

“Sure mate,” [Border] said in reply, his voice dripping with sarcasm, “and when you go back in, can you ask them to send an Australian back, because that's what we need out here.”

Jones stayed and went on to register his highest Test score of 210 runs.

You see, that's what being an Australian is about. When the chips are down; when it comes to the crunch, and everything is against you, what really matters is...

Sport.

Preferably, winning at sport.

Which is why, even though I find the game about as interesting as a kick in the head, I'll end up watching Australia get their arses kicked by New Zealand in rugby this weekend, and I'll honestly believe, despite all evidence to the contrary, that Australia are in with a chance to win it.

Updated 2003-11-15. Final score: Australia 22, New Zealand 10. Yay. Although I only ended up watching the last fifteen minutes.

Part 2 of A Handful of Vulnerabilities

Category: Input Validation

Short Description: A user is able to craft some input to the program that modifies an SQL query into doing something that it shouldn't.

Alarm-bell Code: sqlStatement.execute("UPDATE users SET firstName = '" + userInputFirstname + "' where userid = " + userId);

Discussion

In the above alarm-bell code, what happens if the user supplies a name as follows is left as an exercise for the reader:

Charles',admin='y

This is a really, really common problem in web applications. Parameter quoting is often forgotten, or hacked on the side of the SQL library, and the more complicated the application, the more likely something vital is missed. Just the week before writing this article, two new SQL injection vulnerabilities were announced for Bugzilla (one, two), a Perl application that has had its source open to public scrutiny for over five years.

The good news is that in Java, 95% of this problem is fixed by using PreparedStatement instead of Statement. Most JDBC drivers will prepare a statement with the server, and then supply the parameters separately.

The bad news is that none of this behaviour is mandated by the JDBC specification (as some primitive SQL servers or drivers don't support preparing statements this way). If the driver doesn't prepare statements properly, you have to trust that it knows how to quote parameters. While this is still more likely to be reliable than a home-cooked quoting solution, it means that as part of your system testing, you should test for the possibility of SQL injection against whichever driver you've chosen.

Try entering data with combinations of single quotes, double quotes, the same escaped by various levels of backslashes, and even NUL (\0) characters. If you get an SQLException, you have a problem.

SQL injection is not a particularly big problem in Java because PreparedStatement is mostly safe, and most of us know by now to use PreparedStatement. We are taught to do this for reasons of efficiency, but security is just as important a reason.

Because we're only taught to use PreparedStatement for efficiency, when you're building up a long dynamic statement it's sometimes tempting to hand-hack it together with string concatenation. After all, you're less likely to get the benefits of statement caching if you're building the query dynamically, and dealing with parameter numbering issues can be a pain. If you're querying on user-supplied data, resist that temptation.

Languages that don't have prepared statements should come with standard functions for quoting SQL parameters. Use them. However, you should mistrust any function that assumes generality across multiple databases: it's likely to have missed something. Use the one that was written specifically for the database driver you are accessing.

Related:

Java programs rarely exec() other apps, because it goes against that whole cross-platform thing. It is, however, a common way to get things done in a lot of other languages. You should therefore keep in mind that it is possible to manipulate quotes to supply bogus parameters to an application in Runtime.exec() in the same way that it is possible to inject code into an SQL query.

Your first line of defence is using the form exec() that takes an array of arguments instead of a single string. This should prevent you from having to worry about people escaping from quotes within a single argument, but when testing this theory on JDK1.3 on my Debian box, I was able to craft a command line using triple-backslashed double-quotes that crashed the JVM. So be very, very careful when it comes to exec().

Between broadcaster Channel 10, producers FremantleMedia/Grundy, record label BMG Australia and concept licensors 19, there must be a lot of people wringing their hands with glee. There's a lot of money to be made here, and with any luck, a bare minimum is going to be wasted on the “talent”

Firstly, there's all the advertising revenue. Idol is rating its arse off, and you can bet that Ten are charging an arm and a leg for any spots during the shows. Meanwhile, Fremantle Media are crowing over all the product-licensing revenue they're pulling in.

Then, there's the phone-in revenue. This is what's so amusing about Idol. The viewers have to put their hands in their pockets on top of what 10 are raking in from the advertisers to pay for their own “free-to-air” television. And boy are they encouraged to pay. “Vote often!” they're told. “Keep voting again and again!” Keep that money pouring in.

You could almost excuse this behaviour on Big Brother. The production cost of that show: keeping the house running 24/7, producing four or five hours of TV a week from the footage, and then handing $500,000 over to the winner (well, half to the winner, half to the ATO) would at least have made the show an expensive earner. Do you really think that Idol's weekly shows would be costing as much to produce as, say, the average episode of “Hey Hey It's Saturday”?

And then there's the prize. What exactly is the winner getting? Cash? A new car? Well, no. They're getting a recording contract. They're getting... the chance to work for a living. And we know from Popstars before it, that the career of an Australian TV talent-quest winner is short and unrewarding. The potential winner of Australian Idol would be better off walking out now and getting a job as a temp typist. They'll probably have more to show out of that in a year's time.

They're getting the wonderful opportunity for some short-lived fame, and the chance to make even more money for somebody else.

The single's going to enter the charts at number one. We know that. But having heard the single tonight, and having heard both finalists sing it, it's obvious that it's going to tank very quickly after that debut. It's behind the times. It's a foot-dragging ballad, that fits about as well into the upbeat Rap and R&B-laden Top 40 as it would if the winner were to record “Tie a Yellow Ribbon”.

Any album will do worse. Neither of the finalists have the charisma, passion or even the talent to hold up an entire album. They'll vanish into obscurity the same way as all the Popstars vanished before them.

Which isn't to say that BMG won't make money from the single or the album. It's impossible not to make money from a sure-fire number one. They just won't make money for the artist.

You see, when you make a CD, the artist has to pay back any advance (which includes the cost of the CD's production), and half the cost of making the video out of their royalties. Making sufficient royalties from the record to pay back the production costs is called ‘recouping’. It's only after you've recouped that you start to see any money from the record yourself.

Many people confuse recouping with the record making a profit. The record companies like to encourage this myth. Actually, the artist's royalties (especially the fraction of royalties given to artist who did not write the song) are a small percentage of the CD's profits. By the time the artist recoups, the record company has made their investment back many times over.

And then there's the merchandise. The Australian Idol will be property. They're the product of a world-wide brand, and that brand's not owned by them. It's owned by 19, the Idol show creators, and FremantleMedia, the licensee/producer. It's all built on top of exploiting the alleged talents of the contestants, but none of them, or the winner, will see any of it.

The system is rigged. Channel 10 make money. FremantleMedia/Grundy make money. 19 make money. BMG make money. The “talent”, the people the audience actually tune in every week to watch, have little chance of winning more than the joy of a few months fame, a few more months of shopping centre appearances, and then a realisation that they're yesterday's news, and the bills aren't getting paid.

Part 1 of A Handful of Vulnerabilities

Category: Input Validation

Short Description: A user supplies a specially crafted filename to a program (usually a server) that allows them to access files in areas of the filesystem that should be unavailable.

Alarm-bell Code: File file = new File(safeBaseDirectory, userInputFilename);

Discussion

Many applications create a direct mapping between user input, and files on the filesystem. This can happen explicitly, for example the way a webserver maps URLs to the filesystem, or it could be harder to spot: for example if issuing the "HELP FOO" command causes the underlying program to display the resource "help/foo.txt".

The most common directory traversal attack comes from a user making a request for '../../foo': using the '..' construct to escape to the directory above that in which the files should be found. This is, however, not the only unsafe pattern: several exploits have used '.|.' instead.

This is a very common vulnerability. It is most prevalent in P2P software, or in applications that "grow" some kind of fileserver bolted on the side: since fileserving isn't the core area of expertise for the developer, the dangers are often overlooked. For example, when ICQ was first shipped with a personal webserver, it was vulnerable to the simplest of directory traversal attacks.

One possible solution is to validate the input filename, filtering dangerous-looking characters. (Or more safely, only allowing through characters that are known to be safe). The advantage of this method is that you know exactly how it works. The disadvantage is that it is limited by your knowledge of what is, or is not a safe character. This is especially a problem with applications that may be deployed onto a platform you are unfamiliar with.

For example: The IIS Unicode exploit, one of the vectors for the Code Red virus, took advantage of the fact that IIS's input validation wasn't filtering the Unicode representation of the '/' character. As such, the string '..%c0%af../winnt/system32/cmd.exe' was being translated to '../../winnt/system32/cmd.exe', after the input validation had taken place.

There is another solution built into the Java standard library: the <a href="http://java.sun.com/j2se/1.4.1/docs/api/java/io/File.html#getCanonicalPath()">File.getCanonicalPath()</a> method. This method returns the direct path to a file with all symbolic links and path-traversal symbols already resolved. Thus, the following code will make sure that the file you're reading is really inside the directory you want it to be:

File file = new File(safeBaseDir, untrustedFileName); if (!file.getCanonicalPath().startsWith( &nbsp;&nbsp;safeBaseDir.getCanonicalPath())) { &nbsp;&nbsp;&nbsp;&nbsp;throw new InvalidFileRequestException(fileName); }

Of course, there may be bugs in getCanonicalPath. This is inevitable, but hopefully there are less likely to be bugs there than in your home-cooked solution. In addition, if a bug is found in the library method, you're more likely to find out about it and someone else will write the fix for you.

Application security is important.

Sure, Charles. Water is wet, too. Maybe you also want to point out that cutting off your finger might hurt a bit? Way to state the obvious. We live in an age of ubiquitous networking, where any security flaw can open a user up to untold damage from legions of script kiddies. Of course security is important.

So why do so few application developers pay attention to it?

If you're a professional programmer, how many books or articles on architecture or development practices have you read? How many books or articles on application security have you read?

At the technical book-shop down the road from where I work, there are shelves and shelves of architecture and "best practices" books aimed at programmers. Right beside them there are shelves of security books aimed somewhere between network security professionals, and wannabe hackers. The overlap, security books aimed at programmers, seems to consist of about three books that spend most of their time telling C programmers to stop producing those damn buffer overruns.

And we wonder why we keep producing vulnerable software.

"A Handful of Vulnerabilities" is my attempt to fill a gap in the education of modern developers. For us to produce secure software, we need our minds to be tuned to what programming pitfalls are out there, and how we can avoid them. We need to know what exploits are commonly found in software, and how they come about. That way, we have this important alarm bell that will go off inside our heads when we recognise a possibly dangerous pattern: "You're about to make a mistake. Go back now". This is how we make software safer.

Examples will be given in Java, because it's the most likely language to be familiar to the largest portion of the audience.

When I was writing the notes for one of the chapters of this series, those alarm-bells started ringing. I downloaded the source of a popular open-source Java program. I'd seen it in action here or there. I'd never even looked at the source before, but the pattern was enough for me to know it was going to be a problem. Sure enough, this hole was there: gaping and just waiting for someone to exploit it. If I had been less ethical, I could have made the lives of quite a few people distinctly less comfortable. As it was, I fired off an email to the maintainers, and they're busy fixing it now. But who knows, chances are there's an anti-Charles out there who thought of this long before I did...

So here's "A Handful of Vulnerabilities". A series of articles detailing common security flaws found in software: stuff that gets reported to Bugtraq every day. Mistakes that are made over and over again. If you're a developer, it's your job to know what these are, and how to avoid them. Because that's the only way we can stop them happening.

Contents so far:

  1. Directory Traversal
  2. SQL Injection

James Robertson on the recent security flaws announced in Jaguar:

Then you need to read this article (via Scoble). Yet another buffer overflow problem. This comes from the continued usage of C and C++. Yes, there are ways of finding those problems. And no, most people won't use those ways most of the time. The answer? Use a managed language - Smalltalk, Lisp, heck, even Java or one of the .NET languages. Or, keep producing security flaws....

The security flaws were announced on Bugtraq by @stake just after the release of Panther. Every OS has security problems (and these were rather low-impact bugs) so this would usually have been unremarkable. The controversy came from the fact that the only solution to the problems suggested in the security advisories was "upgrade to Panther", a $129 proposition in an industry where accepted practice is to fix security flaws for free.

Apple later announced their intention to fix the problems in Jaguar as well.

The bugs in question were:

  1. Mac OS X Long argv[] buffer overflow (CAN-2003-0895)
  2. Mac OS X Systemic Insecure File Permissions (CAN-2003-0876)
  3. Mac OS X Arbitrary File Overwrite via Core Files (CAN-2003-0877)

The first is a buffer overrun, certainly. The second is a file-permissions issue when copying an executable out of a disk image. The third is unsafe file-creation for core dumps that could lead to arbitrary files being overwritten, and core files being stolen.

Buffer overflows are dangerous and prevalent, and avoiding them is a good thing. One way to do this is to use a bounds-checking language/runtime, but there are certain things (such as large parts of an Operating System) that are still so performance-critical as to make such management an unwanted overhead. It is possible to write C programs securely (the $500 security bug bounty on qmail is yet to be collected), but it takes a lot of attention to fine detail, and avoiding the standard C library.

Don't let bounds-checking lull you into a false sense of security. Two of the three OS X bugs above weren't buffer overruns. Many IE exploits come from fooling the browser into thinking some content is trusted when it shouldn't be. A lot of the worst IIS exploits came from poor input verification, or Unicode hacks to get around it. Almost every webserver in existence has fallen victim to some form of directory-traversal attack. Applications like Bugzilla, PHPNuke and Postnuke have terrible security histories, despite being written entirely in managed languages.

Insufficient input validation and SQL/code injection attacks. Porous trust boundaries. Directory traversal. Cross-site scripting. Weak authentication policies. Insecure default configurations. Unsafe file creation. Race conditions. Information leakage. Resource starvation.

The enemies of security are complexity and inattention to detail. Using a "managed language" is one way to reduce complexity, but more than enough remains to make our lives very difficult. We just have to keep paying very close attention.

I was looking at my iTunes playlist this morning, and realised that I've started getting bored with my music. The problem is that I've got a few hundred songs that I tend to gravitate towards at the expense of the several thousand others.

So, in the interest of novelty, here's the deal:

  • I will play through the 3362 songs on my iPod in alphabetical (well, ASCII) order by song-name, starting with Regurgitator's ! (The Song Formerly Known As), and finishing with Faith No More's Zombie Eaters.
  • I may only use rewind to hear again the song I just listened to, if I happen to really like it.
  • I may only fast-forward to skip songs that are duplicates (I just discovered I have three copies of Filter and The Crystal Method doing (Can't You) Trip Like I Do), or songs I actively dislike.
  • I will resist as best I can the urge to skip songs that I am merely indifferent to, or that don't fit my mood.
  • No other navigation of the playlist is permitted, except to find where I last left off.
  • Exceptions may be made for newly purchased albums with songs that fall in the "already played" part of the playlist.

I should be done in a couple of months. :)

Starting off, it's surprising how many songs start with punctuation: particularly parentheses. I also have four songs starting with double-quotes, and they're all from Philip Glass movie soundtracks (One from Dracula, three from The Hours).

Now Playing 16/3362: (Exchange) by Massive Attack, off Mezzanine.

Aside: I just Googled for Philip Glass, and found this:

The IBM glass engine enables deep navigation of the music of Philip Glass. Personal interests, associations, and impulses guide the listener through an expanding selection of over sixty Glass works.

Funky. Must have a look at it some time.

Alan Smithee

  • 9:56 PM

Alan Smithee is a director famous for his bad movies. Almost all of the movies he has made have been really, really bad. He's famous for it. And yet, now and then, he still gets work in Hollywood.

This is because he doesn't really exist. The Directors Guild don't allow a movie to be released without naming its director: to do so would undermine the work they've done in building the role of the director as the primary creative force behind a movie. Sometimes, though, a director can go to the Guild and complain that the movie was taken out of their control. The studio pressured them to stray from their vision. It was cut against their wishes.

Then, the Guild will have the movie credited to Alan Smithee, and he'll take the blame.

Sometimes, when I've had to write code very quickly to reach a deadline, or I've cut corners, or I've had to hide something necessary but ugly in a Ghetto, I'm sorely tempted to leave this Javadoc behind:

/** @author Alan Smithee &lt;alan.smithee@example.com> */

Raymond Chen's story of the Windows “Shell Folders” key is an interesting one because it can be told equally persuasively from two perspectives.

There's the perspective Raymond tells it from: backwards compatibility is so overridingly important that Microsoft deprecated a registry key instead of removing it, so as not to break a few applications that were written for the Windows 95 Beta. Unfortunately, people started to use the deprecated registry key without realising there was a better, more reliable way.

Then there's the alternate perspective: Microsoft left a crufty, broken feature in Windows 95 that would cause programs written against it to break in unexpected and hard-to-diagnose ways, so as not to break four applications that had only been around since the beta anyway.

I think one thing we forget is how much easier software distribution has become in the last few years. Apple can get away with breaking a bunch of applications between Jaguar and Panther. The apps are being maintained on the Internet, so the fix is just a website away. Even as little as five years ago, this wasn't the case for most users.

Tangentially related, Brent Simmons recently wrote about how he does not identify with the word “Shareware”;, despite being a distributor of software that would have fit the Shareware definition five years ago. Brent links to some nice related articles:

  • Slava from Unsanity: Shareware Is Dead (Unsanity write "Haxies" for OS X: programs that hook into private APIs to do things the OS doesn't approve of. As such, they're the most likely to have their code broken by OS upgrades. Which takes this post full-circle, I guess.)
  • Buzz Anderson's response, on how we now just have “independent software” in the same way we have independent music.

Inspired by Diego's excellent two articles on blogging, here's the slides from a presentation I did for work a little while back on the subject. The presentation focuses on tech-blogging for Java programmers.

Quicktime. Click on it to advance the slides.

Warning: gratuitous use of Keynote 3-D transitions. (And for some weird Quicktime reason, the first one skips around)

I was thinking of writing a quick program that would process my iTunes XML-exported playlists into HTML. After some thought on what would be the simplest way, I decided it might be time to overcome my distaste for XSLT, and give it another go.

So I pulled out my trusty O'Reilly XSLT book and tried to gauge just how much effort was going to be required. I came across the following description of the "recommended" way of grouping nodes after a sort so you don't have to repeat information (like album title) for each entry. This was a basic requirement: if I didn't want to group, I could just apply a CSS stylesheet directly to the XML document and be done with it.

In a procedural language, this would be most easily done by, after sorting the tracks, keeping track of the value of the album title from the last pass through the loop, and checking if it's changed. In XSLT, it's not quite so simple.

In the book's example, we are trying to group addresses by their ZIP code. Note, this is the second attempt to explain how it's done.

For each &lt;address>, we use the key() function to retrieve all &lt;address>es that have the same &lt;zip>. We then take the first node from that node-set. Finally we use the generate-id() function to generate an id for both nodes. If the two generated ids are identical, then the two nodes are the same.

Whew. Let me catch my breath.

If this &lt;address> matches the first node returned by the key() function, then we know we've found the first &lt;address> that matches this grouping value. Selecting all of the first values (remember, our previous predicate ends with [1]) gives us a node-set of some number of &lt;address> elements, each of which contains one of the unique grouping values we need.

The explanation continues for another page, and revolves around the XPath expression: //address[generate-id(.)=generate-id(key('zipcodes', zip)[1])]

At this point, I started thinking I'd been right about XSLT in the first place.

It's 1am and I just turned off Knights of the Old Republic. While it's shaping up to be a pretty good CRPG (and, the prequel movies aside, I'm a sucker for all things Star Wars), here's the things I have found annoying so far:

  1. I decided to go with the "Scoundrel" character class. Because it just sounded cooler than the other two. True to form, I later discover this is the Class You Should Not Play. I do this with every single game I play. Ho hum, I'm too stubborn to change now.
  2. Sure, it's cool that you're using the d20 rules, but it'd be nice if the computer insulated me from them a little more.
  3. You can wander around a populated city waving a sword without any reaction from guards or NPCs.
  4. Not only are there no repercussions for looting random apartments, it's practically encouraged. You can gain a mountain of "good side of The Force" points for refusing a reward for helping someone, and then you can turn around and rob them, with no "Dark Side" points, while the victim watches and does nothing.
  5. Combat is frighteningly simplistic. It reflects the tabletop roleplaying combat experience just that little bit too accurately: where people just take it in turns to roll the dice to see if they hit or not. The more sophisticated the graphics for CRPGs become, the more stupid this system starts to look as characters that should by all rights be diving for cover or hiding around corners end up standing up in plain sight, guns blazing.
  6. On a related note, given I deliberately chose a stealthy character over a strong one: when I sneak up behind J. Random Grunt Guard completely unnoticed and hit him with a sword, I damn well expect him to die. I mean, I could do that with a piece of string in Hitman. Having my victim lose about 10% of his hit points and then turn around and kill me really breaks the old suspension of disbelief.
  7. I have a sneaking suspicion that the plot, script and acting in the game will turn out to be better than those in the next Star Wars movie.