This was originally going to be a comment to Rickard's latest article about JBoss's AOP implementation. Unfortunately, Freeroller died just as I was trying to post it, so I'll just reproduce the comment here, for posterity.
The classic definition of FUD, as pioneered by IBM back when they owned the world, was this:
Your competitor puts out a product. In order to stifle its adoption, you release a series of vapourware announcements that describe how your product, which is available 'real soon now', has more features, performs better, and if people will just wait until it's released, they'll be really glad they didn't lock themselves in to the competitor's platform.
When your product is essentially vapour, you have all the advantages. Maybe, compared to your AOP framework, JBoss sucks. But to the world at large, your AOP framework does not exist. Any judgements on the relative merits must wait until both platforms are available to compete on level terms.
JBoss, on the other hand, have the significant disadvantage that they are doing all their development, and thus their learning, in the open. This is a disaster for people who are heavily into performance, because by necessity, optimisation doesn't occur until after the thing is working, and as soon as the thing is working, the developers want people to start hacking on it.
It's one of the problems with the Bazaar model. Software is released deliberately before it is "ready", and often that means that programs can be tarred with the bad performance brush long after they've optimised away the problems that made them notorious.
(I really hesitated to use the expression “FUD”, because it has been so tarred by the Slashdork crowd to mean “anyone saying anything we disagree with”. The term has been so damaged in recent years that it's almost become one of those ‘Godwin's Law’-style expressions that will make me immediately discount someone's opinion. I promise not to use it again.)
good point--different than the way I said this morning but still very good :)
Vapor? You just don't know what you are talking about. http://www.senselogic.se (hope you can read Swedish). You see, Rickard has shipped a product using AOP while JBoss has not. That is unless you can prove he is a liar. So what if you can't see the code. If the founder of Checkpoint were to talk about firewall software would discount him because the source code isn't open? Absolutely not. Rickard is about as close to an 'expert' as you can have in this nascent technology.
Firewall vendors have independent certifications like this to demonstrate that their products work as advertised: http://www.icsalabs.com/html/communities/firewalls/index.shtml If CheckPoint were to say "We have this kick-ass Firewall that is so much better than our competitors', but it's not available right now so you'll just have to trust us on that", I'd be just as skeptical.
Whether some product has been built on top of Rickard's AOP framework is irrelevant: the framework itself remains un-released as a product. As such, it is impossible to evaluate beyond the single application that has been built on top of it.
A better analogy would be if Checkpoint were to release a firewall appliance on top of their own custom Operating System. The fact that it worked well for the particular task of firewalling would come as no surprise, but I would not make any judgement as to the underlying OS's general-purpose suitability unless I had the chance to evaluate it independently.
I have no doubt that Rickard is a very smart guy, and has written a good AOP implementation. I would even be likely to take his word that it is better than JBoss's. But how much better, and how relevant the improvements are in the context of a real application can't be measured when the two frameworks aren't competing as frameworks.
Until the framework itself is released as a general-purpose product, whether that be open- or closed-source, it can not be fairly and independently compared to the other AOP frameworks (AspectJ, JBoss, Nanning, AspectWerkz) that are actually out there.
Charles, can you quote a snippet from me (in blogs or otherwise) where I am comparing JBoss AOP with my own framework? You're saying that I am comparing JBoss AOP with my own, and I didn't know I had ever done that. So please, enlighten me.
Rickard: I'm at work, so any Googling for quotes might have to wait a while. You have, however, on a number of occasions discussed JBoss AOP in the context of your own AOP framework, for example referring to having seen "mistakes" in the JBoss implementation that you encountered (and presumably avoided) when doing your own.
More generally, every performance evaluation _must_ be a comparison: performance does not exist in isolation. A program can be slow in an absolute sense - since users have absolute threshholds for response times. On the other hand, a framework or API can only be slow in comparison with something else. "Doing foo this way adds a 2 millisecond overhead to each method call" is only meaningful if you have some alternative way of doing foo to compare it with.
As an implementor of an AOP framework, you making criticisms of the performance of JBoss implies directly that your own fares better in that area (otherwise the criticism would be of AOP in general, rather than just the individual implementation).
I apologise if I seem to be putting words into your mouth, but you must admit that the first thing that was asked when you posted your performance comparison was "And how does your framework compare?", and you dutifully directed the questioner to find the benchmarks of your own creation that you posted earlier.
You are in the unlucky position of being unable to _not_ make comparisons, simply due to your position in the debate. If you don't like that position, I suggest you be a trifle more circumspect, and liberal with disclaimers. :)
> Rickard: I'm at work, so any Googling for quotes
> might have to wait a while. You have, however,
> on a number of occasions discussed JBoss AOP in
> the context of your own AOP framework, for
> example referring to having seen "mistakes" in
> the JBoss implementation that you encountered
> (and presumably avoided) when doing your own.
So, does that make me unable to make the comment, or does it make me the perfect person to make such a comment? If I've done the same mistakes, and can say what they lead to, is that a bad thing? I don't think so, but feel free to disagree (FWIW, I see your point).
>More generally, every performance evaluation
>_must_ be a comparison: performance does not
>exist in isolation. A program can be slow in an
>absolute sense - since users have absolute
>threshholds for response times. On the other
>hand, a framework or API can only be slow in
>comparison with something else. "Doing foo this
>way adds a 2 millisecond overhead to each method
>call" is only meaningful if you have some
>alternative way of doing foo to compare it with.
An excellent point. Which is precisely why I included a number of comparisons between different scenarios *within* the JBoss AOP framework. I compared different things, and in a decently designed framework you'd get certain results. As I found, these results did not appear. I even *started* the whole entry saying that timings are relative. Your comment makes me wonder whether you actually read the entry or not.
>As an implementor of an AOP framework, you making
> criticisms of the performance of JBoss implies
>directly that your own fares better in that area
>(otherwise the criticism would be of AOP in
>general, rather than just the individual
>implementation).
Any such implication is purely in your own mind. Which is why you calling my post FUD makes your post... what? And if you want a comparison, I just reran similar tests on my own framework and the performance is similar, except there's no excessive GC and there's no overhead for having interceptors that don't fire. The timings in my blog entry are (as you note) more or less irrelevant, and if you read my conclusion you would see that what is important are those two things.
> I apologise if I seem to be putting words into
> your mouth,
Not only that, but you then accused me of spreading FUD with those imaginary words. What does that make your blog entry?
>but you must admit that the first thing that was
>asked when you posted your performance comparison
> was "And how does your framework compare?",
By our very own blog-Zahid-look-alike Fred Grott, sure. It was a fair question, and the answer had already been given. Should I have not answered Fred at all?
> and
>you dutifully directed the questioner to find the
> benchmarks of your own creation that you posted
>earlier.
I said that they were available if he wanted to find them. What was I supposed to do? Lie, and say that they were not?
>You are in the unlucky position of being unable
>to _not_ make comparisons, simply due to your
>position in the debate. If you don't like that
>position, I suggest you be a trifle more
>circumspect, and liberal with disclaimers. :)
I'd say that there are few who are more able to comment on AOP frameworks, since I have been actively involved in writing one, and also having worked on EJB containers (which are the fore-runners of AOP). *IF* I had made loose accusations which were not backed up, then you might have a point, but I'm pretty sure that all my conclusions about the AOP framework (both from a design and performance perspective) are pretty well explained. If there's anything that is unclear about why I came to my conclusions, feel free to ask.
But I would appreciated if you stopped putting words into my mouth and then calling me a FUD-spreader. It's rude.
I apologise for the FUD thing: http://fishbowl.pastiche.org/archives/001361.html#001361
Testing a single product is still a test in isolation. For example, (as far as I have been able to gather from the debate), the fact that an interceptor costs a certain amount whether it is, or is not disabled in meta-data is the result of a flexibility/performance trade-off that you feel it is not worth making, but the JBoss developers believed was. (the decision is too dynamic to be able to cache)
Which essentially means the "poor performance" of those calls is very much like the "poor performance" of Java method calls. Java was designed to do runtime-binding, hence its method calls are slower than a compile-time-bound language. And while there are many tricks you can pull to optimise that out at runtime, you'll never practically have one run as fast as the other.
What impact that 0.005ms-per-advised-call would have on a real-world application, is a lot harder to judge, because it depends on the relative density of those badly-behaving calls. You're in a position to know that because you've actually written a significant application on top of an AOP framework, which pretty much all of the rest of us have not.
On the other hand, your application and framework were developed together, so they will be playing to each others strengths.
I guess time will tell.
jeez Charles, you got nuked!
on the usage point, I'd say performance of an underlying AOP framework is going to be pretty critical to AOP implemented application code, simply due to the pervasive nature of AOP - my POJO just grew wings and is now a fully transactional, logged, persistable remote object that sends out state change notifications ACK! it better bloody perform as well ;-)
basically, I reckon the usage argument is usually pretty important in evaluating general application frameworks, but kind of falls by the wayside a bit for an AOP framework, you kinda have to presume high usage.
Jed,
I don't think you _have_ to presume high usage in JBoss's case. It seems to me that JBoss AOP is mainly to make POJOs transactional and persistent for server applications. In processing a request, how many calls are going to be made to AOP aware objects? Typically one to the object that is transactionally aware (which starts and finishes the transaction) and then a few to persistent objects.
By Rickard's timings (0.005ms for a call with one interceptor) this equates to perhaps 0.050ms per request, which is thoroughly insignificant compared with, say, a simple database query - typically tens of millisconds.
From all I can see, if used sparingly, JBoss AOP will only be a problem for very high throughput applications. And for those applications, you are likely to have bigger problems anyway!
I grant that logging is probably a different matter.
The point really is "if used sparingly", and for AOP you want to be able to instrument a lot of stuff for all sorts of reasons. Logging is the classic case, if I can't use AOP to get rid of those annoying and expensive Log.debug statements from within code, then WTF point? ;-)
BTW Charles, how come you seem to be the only blogger who's worked out how to handle single apostrophes in comments properly ... I'm impressed ;-)
Alan, I suppose this will differ quite a lot between projects/products, but the main issue is as Jed says: you want to apply AOP everywhere, or else it's kind of missing the point. For example, in our product, which is a CMS, every persistent object is an AOP object. So, every call to them will incur the extra overhead, and since we almost exclusively use mixins the overhead would be 0.04ms/call if we used JBoss/AOP (if the hardware is like my machine that I used in the testing referenced in Charles blog entry). Let's say that, for example, one page rendering in our CMS makes 1000 calls to AOP objects. This isn't so far-fetched, especially if the page contains a hierarchical menu system with the site outline as well as text with lots of links (links are AOP objects as well). That page would then have 40ms of pure call overhead, i.e. besides the actual computations and output done. This is almost as much time as the actual code-that-does-stuff, so instead of generating the page in 40ms it's done in 80ms.
I don't know about you, but to me that's pretty significant.