July 2007

« June 2007 | Main Index | Archives | August 2007 »

31
Jul

I watched precisely four episodes of Australian Big Brother this year, including tonight's finale.

The general plan for the finale of Big Brother is as follows: with two people left in the house, they run a long recap of the entire season, while host Gretel Killeen stands in front of a live audience and exhorts viewers to keep dialing those numbers, sending in those votes and donating some proportion of their phone bill to Southen Star Endemol.

After the recap they close the voting lines, throw to an advertisement break, play one more video package, throw to another ad break, then announce the final evictee (runner-up), leaving only the winner in the house.

The runner-up does half an hour or so of post-eviction interview, after which they pull out the winner into some big procession between the house and the stage, interview the winner, give out the cash prize, and usually finish the show about half an hour over-time.

This year it didn't go to plan.

The official line was that the final vote was so close (a few hundred votes out of hundreds of thousands of entries) that they had to take extra-long to make the final tally. Given that last year the show embarrassingly evicted the wrong person and had to send them back in, I can't see any reason to doubt they needed to take time to be sure. I certainly don't think it was deliberate.

It was too funny to be deliberate.

You know how the English version of The Office was funny, the way it made you cringe so much that it was stomach-churningly painful even while you were laughing your ass off? That's how it felt watching Gretel more and more desperately trying to find ways to vamp out that last hour, her already artificial smile getting thinner and thinner as she played for time interviewing whoever stood closest to her, as they successively showed all the video packages they had prepared for the night, while the show fell apart around her.

It was frustrating. It was comical. It was an exercise in just how wrong live TV can go if you're performing without a net. The least-watched series in the show's history was capped by an unwatchable finale. Bravo.

Next year, I say, take another hour. Take two. Have the DJ do a whole set. Interview all the housemates again, plus every past-season housemate you can get on a last-minute hospital pass satellite hook-up. Replay every single highlight you can find in the archives. Announce who won UK Big Brother, and see if anyone notices the difference. Make the audience form conga-lines to spell out their favourite housemates. Hold an impromptu game of charades. Interview everyone again but make the entire cast and crew do a tequila shot every time someone says it was "just a great experience", or that it "changed their life". Show a montage of Gretel's less fortunate wardrobe choices over Green Day's Time of your life. Bus strippers up from the Gold Coast. Dancing bears. Contortionists. Elephants.

Bring on the farce!

In an internal blogpost inside Atlassian, I described a certain problem as being ‘very hard’, to explain why our efforts were better spent elsewhere. Later, as I was walking across the bridge into work I had a moment to look at that statement through the eyes of a non-engineer. Atlassian prides itself on hiring really smart people. What use are they if they can't solve hard problems?

To that end, here's a quick lexicon of what computer programmers generally mean when they're talking about how hard some problem is, starting with the most extreme:

Impossible

The man most commonly regarded as the 'father' of computer science is the English mathematician, Alan Turing. Turing did a lot of work in World War II helping the Allies break the German military ciphers. To reward him, the British Government convicted him of gross indecency after the war (he was homosexual), took away his security clearance and put him on hormone therapy. His death not long after is generally accepted as suicide.

Anyway. Turing's most famous contribution to computer science is the Turing Machine. This is a theoretical device that is both capable of solving any computational problem that could be represented as an algorithm, and of emulating any other device that solves computational problems.

Anything that can be (deterministically) mechanically computed can be computed by a Turing machine. Anything that performs deterministic mechanical computations is really just a Turing machine. Engineers speak of computer hardware or programming languages that have this full range of computation as being "Turing complete".

In this framework, the word 'impossible' has a definite meaning. A problem is impossible if its solution can not be computed by a Turing machine.

Admittedly, this isn't a very useful distinction. On one hand, Turing machines are a mathematical theory. They're infinitely fast and have unlimited storage, and you have an unbounded amount of time to write your program for it. As such, many things that are theoretically possible are practically impossible (more on that later).

On the other hand (and this trips up engineers all the time), it doesn't ascribe any value to mostly solving a problem. No Turing machine can tell you if you're going to like a book or not, but Amazon makes a lot of money out of coming out with a close-enough guess.

Trivial

At the other end of the scale, we have problems that are trivial. This definition can be expressed in far fewer words:

I know how to solve this problem.

To a programmer, a problem is trivial if there is a clear solution, and the only thing that needs to be done is to implement it.

The only caveat is that triviality refers to how hard the problem is to solve, not how hard it is to implement the solution. So there is no necessary relation between a task being trivial, and how long it takes. To the programmer, once the plans for the bridge have been drawn up, the materials chosen properly and the model tested for how it would survive wind, traffic and earthquakes, actually building the bridge is trivial.

Unfeasible1

A problem is unfeasible if enough of the solution is known to determine that you don't have the resources to solve it. You might not have the free developer resources, or the available expertise, or it might just need more hardware than you can ever afford.

The field of cryptography offers us a perfect example of a problem that is both trivial and unfeasible. The algorithms that encrypt and decrypt data are well-known and published. Assuming the algorithm works ‘as advertised’, to decrypt some data you just need to write a program that implements that algorithm, and throw enough computing hardware at the problem to run that algorithm with every possible key.

Cryptography works by choosing keys in a way that ensures there's not enough computing hardware available, practically speaking, to run that trivial program to completion. Decrypting such data is unfeasible.

Non-Trivial

An ex-Google non-engineer described 'non-trivial' thus in the Xooglers blog:

It means impossible. Since no engineer is going to admit something is impossible, they use this word instead. When an engineer says something is "non-trivial," it's the equivalent of an airline pilot calmly telling you that you might encounter "just a bit of turbulence" as he flies you into a cat 5 hurricane.

This quote shows both the difference between the engineer and non-engineer's conception of non-triviality, and their different definitions of impossible (Flying through a hurricane isn't necessarily impossible, you just really wouldn't want to do it).

In clear opposition to the meaning of trivial, non-trivial means:

I do not know how to solve this problem completely.

Non-trivial contains dangerous unknowns. Some part of it is not yet understood, or lies outside the range of things the programmer has done before, or can quickly imagine a workable solution to. The more experienced the programmer who tells you a problem is non-trivial, the more concerned you should be.

Given time for research and experimentation, a non-trivial problem can be made trivial. Or it could be determined to be hard.

Hard, and Very Hard.

Hard problems are a class of non-trivial problem2 where some of the unknowns, some of the problems the engineer would have to find a way to overcome, are known to be difficult to find solutions to. These are problems the engineer has tried to solve in the past, or has seen the resources that other developers have had to throw at similar problems to solve them.

'Hard' can also be used to describe non-trivial problems with a big potential for 'unknown unknowns' - things the developer not only doesn't know the solution to, but doesn't even know what kind of problems he's likely to face.

Scaling out a web application is hard. There are any number of thorny problems to do with performance, distribution, data access, caching and so on that take quite a bit of effort to solve, and are often different enough from application to application for there to be no good, general solution. There are also inevitable thorny problems that you haven't even anticipated, lurking in the wings.

Very Hard is the extreme of hard problems. You'll often see both words capitalised for emphasis, even in the middle of a sentence. Indexing the entire World Wide Web and providing relevant search results in millisecond response times is a Very Hard problem. Breaking commercial-grade encryption within practical hardware and time limitations is a Very Hard problem. Peace in the Middle East is a Very Hard problem.

'Very Hard' is usually reserved for the class of problem that if you solved it, you could change the world. Or at least build a successful business on top of your solution.

1 The Economist weighs in on “unfeasible” vs “infeasible”, and declares both are interchangeably correct.
2 Gary Capell pointed out to me in email that, in classic engineering understatement, hard, or even Very Hard problems can sometimes be referred to as ‘distinctly non-trivial’.

As a participant in Usenet during the mid-90's, I had the old-school netiquette drilled into me for quoting messages to which I was replying. The process is succinctly described by John Gruber in his article, On Top.

This style of quoting could be expressed in the form of a pattern:


Name:

Usenet Quoting

Problem:

When responding to a message, the writer wants to be sure readers can follow the resulting conversation.

Forces:

The architecture of Usenet means that:

  • Disk space and message-transmission bandwidth are at a premium
  • A reply may arrive on a server before the message it is replying to, or the original message may simply never arrive
  • Old messages may be deleted (expired) at any time
  • Conversations will be read by people who have had no involvement in the prior conversation
  • Conversations may involve any number of people, with forking subjects

Solution:

Include just enough quoted information from the previous post(s) in the conversation to provide context for your reply, but no more.

The point that you are trying to make should be understandable even if your reader can not access any of the other messages in the thread. On the other hand things other people have said that are not relevant to your post's direct point should be discarded.

This will ensure the thread of conversation is maintained with the minimum waste of bandwidth or storage.

Interspersing your responses with the relevant part of the message you are responding to will make your response more readable, and make it easier for future responders to isolate each point you are making when they are choosing what to quote in their reply.

Notes:

This quoting style encourages a style of argument where the poster's points are extracted from the original article as pull-quotes and discussed one-by-one. In fact, when arguing with someone on Usenet, netiquette almost requires that you delete from your response everything your opponent says that you can not disagree with.

This can distort the original piece a great deal, especially if the original is no longer available to read.


Corporate email communication in the noughties, however, has a totally different set of constraints:

  • Bandwidth and storage are somebody else's problem
  • Multi-party email conversations are mostly performed via CC: lists
  • Mail clients may or may not have decent threading implementations
  • Occasionally, someone new will be added to the CC: list who has no access to the prior conversation

Top-posting over entire messages actually makes sense in this context. Having the entire previous conversation available "bottom-up" at the end of the message allows anyone to read the full history of the discussion, regardless of how badly their mail-client sucks, even if they have played no part in the prior conversation. Everyone who already knows what is being discussed can just stop reading after the signature of the most recent poster.

Also, by reading just the most recent message in a conversation first, down through the quoted material to the point where you recognise something you've read before, you can be sure you're up to date.


In traditional, folder-based mail clients; new messages appear in your inbox and are moved into some storage folder (or deleted) after you read them, while messages that you send go into a third, unrelated folder. This design almost seems deliberately tooled not to show you messages in the context of their conversation. Even personal, one-on-one emails still need enough quoting to ‘nudge’ the reader back to where the conversation left off.

On the other hand, these forces would be reasonably familiar to a GMail user:

  • Bandwidth and storage are effectively infinite
  • Messages are never deleted
  • Each mail is displayed in the context of the conversation in which it occurred, accompanied by excerpts of previous messages

Therefore… if you're in an email conversation with one other person and you're both using Gmail, don't bother quoting at all.

It's not a good time to be a professional photographer. Ten years ago, digital cameras were bulky novelties that saved your 640×480 pixel image onto a floppy disk. Today, most people can (and do) take better quality photos with their cell-phones.

Meanwhile, at the same time that every man and his dog has a decent quality digital camera and access to cheap tools to touch-up and share their photos, demand for quality is actually going down: the difference between professional and amateur is much less noticeable when compressed for Internet delivery and displayed on an LCD screen at 72dpi.

Sites like iStockPhoto are riding this growing trend, attracting masses of photographers to sell their shots at rates that would be insulting to professionals, but that are flattering to amateurs looking only to make enough to maybe buy a new lens, happy just to know that someone thinks their snap is worth paying for.

News agencies know on what side their bread is buttered. The online version of the Sydney Morning Herald regularly accompanies current-events stories with calls for readers to send in their photos for publication, and they're certainly not rushing to compensate the photographers for the rights.

A recent campaign from Virgin Mobile in Australia also skipped the annoying step of having to pay for photography, basing their billboards on appropriately Creative Commons-licensed photos found online.

I noticed these adverts in my local train station1 largely because the not-so-small print included a link back to the Flickr page from which the photograph was obtained. Obviously, this is a logical move on the part of the advertising firm. There's so much free stuff out there, why pay when you don't have to?

Needless to say, there was a certain amount of discussion about this over on Flickr.

Flickr's default copyright setting for new photos is “All Rights Reserved”, so I have little sympathy for anyone who chose to make their photographs available for commercial use, then suffered “sharer's remorse” when someone actually took advantage of that licence.

Similarly, while it can't be fun to be a commercial photographer right now, I can't really condemn any photographer for wanting to share their photographs, or any consumer taking advantage of what is freely available. The number of people who take photos is always going to dwarf the number willing to make the substantial effort to turn that into a business, and the combination of digital photography and the Internet makes it almost easier to share than not.

Technology has radically overturned the balance of supply and demand, flooding the market with an abundance of cheap product which, while a lower quality on average, still contains quite a few gems2. You can't put the genie back in the bottle.

One interesting issue did, however, come out of the discussion of the Virgin billboards. In many jurisdictions, even though it is legal to photograph people without their permission, the rules change when the photograph is used for commercial purposes3. Standard practice is for photographers to get a signed “model release” from anyone appearing in such photos. Very few amateur photographers carry around model releases, even those who later release their photographs under licenses that allow for commercial reproduction.

Version 1.0 of the Creative Commons licence was pretty clear where the responsibility lay. Section 5 (Representations, Warranties and Disclaimer) read in part:

By offering the Work for public release under this License, Licensor represents and warrants that, to the best of Licensor's knowledge after reasonable inquiry: Licensor has secured all rights in the Work necessary to grant the license rights hereunder and to permit the lawful exercise of the rights granted hereunder without You having any obligation to pay any royalties, compulsory license fees, residuals or any other payments;

In other words: if you make your work available for commercial use, it's your responsibility to make sure everything in the work has been cleared for that use. Given that the model release is standard practice in photography, nobody could argue that it isn't part of the “reasonable inquiry” expected of a photographer.

This term caused a certain amount of backlash from bloggers worried that it imposed too much of a financial risk on Creative Commons licensors4. As such, this clause is conspicuously absent from version 2.0 of the licence, the version that is applied to Creative Commons photographs by Flickr. Does this absolve the photographer of responsibility? I am not a lawyer, but my guess is that the answer would be the sort of ‘maybe’ that makes lawyers salivate. Removing the clause takes away the voluntary assumption of responsibility, but only replaces it with the sort of catch-all disclaimer that courts look for reasons to ignore.

Unlike sharer's remorse, where a licensor belatedly realises the obvious implication of the licence they chose, this is a legal minefield that an amateur photographer may not even know they're getting into, and that could expose them to quite substantial liability. Perhaps the commercially friendly Creative Commons licences need to come with a warning label?

Update: I had a chance to talk about this with a Real Lawyer who works for the Australian Copyright Agency. According to her, Virgin would only be in trouble for including people in photos if they were implying some kind of endorsement of their product from the person pictured (which isn't likely), or if the person pictured was under 18 (in one photo, true). So the risk isn't so great after all.

1 The image this billboard was derived from is under a share-alike licence, which I'm assuming makes it OK for me to reproduce it here
2 Of the nine random photos on my last visit to Flickr's interesting photos page, two were Creative Commons-licensed, with one of those free for commercial use.
3 To further confuse things, “commercial use” has several meanings. The Creative Commons defines commercial use as you'd expect: exchanging copyrighted works for money. The New South Wales law regarding the use of your likeness in photographs, on the other hand, defines it much more narrowly.
4 I vaguely remember Dave Winer writing something about it, but can't find the link