The Middleware Company are idiots.

They publish a J2ee vs .NET benchmark, which is quickly determined to be worthless, follow it up with a FAQ acknowledging all the criticism is warranted, and now want to re-do the tests more fairly. But they've already shot themselves in the foot. They're idiots.

There's no two ways about it. The moment they got involved in the J2EE vs .NET benchmark, they were holding a big shotgun pointed directly at their feet, and waiting patiently for somebody to pull the trigger. By all accounts they did this voluntarily. The gun was paid for by Microsoft, but TMC did the rest.

I'm not going to weigh in on the Java vs .NET debate. I don't know enough about .NET to comment. But it's obvious that the comparison presented by TMC in its report was meaningless, and proves nothing but TMC's own incompetence. We all know the benchmark was ludicrous. TMC admit it in their hilarious FAQ/apologia on the subject:

Does these results mean that J2EE is slower than .NET?

No. The results mean that .NET was higher performing than J2EE in this particular comparison. Other comparisons could be conceived where results would be different.

If you read the FAQ, it's a long list of excuses for the ways that they knew the Java implementation of Pet Store was sub-optimal, but they didn't have time to/didn't want to/couldn't fix it. (Meanwhile, the .NET version was built from the ground up for speed, just not at TMC's expense) So what they advertise as a benchmarking exercise, and continue to refer to as benchmarking throughout the FAQ, is an apples-to-oranges comparison with absolutely no practical value. The last question, though, is the clincher:

After reading this benchmark I don't trust TMC's or TSS's advice anymore.

TMC had a lot of heartache seeing the results of this benchmark. We internally debated about whether we should post this or not. In the end, we decided to go forward and publish this report.

The results of the benchmark aren't the problem. The process by which those results were achieved is. If TMC had conducted the fairest test they could come up with and .NET had won, they would still have caught some flames, but they would have been able to keep respect amongst programmers able to see past the stupid advocacy garbage, to the fact that they did the best job possible.

Read the FAQ page again. Clifton Begin cloned the original Pet Store in his spare time in a week, but we are supposed to believe that TMC could not fix the existing Pet Store given team of experts working full time. Pull the other one, please.

Surely, if the comparison were so unbalanced, and they didn't have time to produce a fair comparison, the project should have been abandoned, or further resources sought? TMC had in their hands a test that they should have known had no value as a benchmark, and that was a pure marketing exercise for .NET. And yet they went ahead with the test anyway, and called it a “benchmark”

These reports tend to become synonymous with the company that produced them (remember the Mindcraft Report?), and that's the position TMC are going to find themselves in. Which is a bit stupid if you make a living from selling Java services.

Stupid, stupid, stupid.