What Happened to Head to Head Reviews?

With all that bloggers are doing, what is left for the traditional press? By their numbers, voraciousness and speed, bloggers seemed to have nailed event coverage. It wasn't even a contest. Ditto, tips and tricks -- the press just cannot compete with hard core users -- which most bloggers are. What about tutorials? Once the realm of CAD print magazines and CAD books, I'm seeing an increasing number of good or good-enough tutorials on blogs. In fact, a recent series of tutorials by Orhan Toker on sectioning of AutoCAD solid models may have saved a recent project!

Hey, what about product reviews? Sure, bloggers may post their impressions of a new release and though it may be technically classified a review, in most cases they fall short of  those 5,000 word, comprehensive reviews that were the pride and joy back in the glory days of the trade press. Most blogger 'reviews' end up being short, pithy vignettes, but still one could argue that the collective work of the blogging community reviewing a single product (as happens with AutoCAD releases) more than compensates for the massive reviews of old.

Though trade press may have lost ground in the above types of coverage, one way it could still shine would be to do comparative reviews: looking at rival products that help users make a buying decision. Comparative reviews, aka head-to-heads, are done for other areas, notably software, digital cameras, etc. The traditional press editors seem ideally situated to do this. Most magazine editors are showered with CAD software products and end up with shelves full of them. How about comparing them and helping out the befuddled user who is trying to graduate from 2D and needs to know the best 3D CAD product, for example?

Why are well done head-to-head product comparisons so rare in our industry? Only Cadalyst still does them. Still, I don't recall a comparative review between Pro/E or CATIA, Revit or ArchiCAD. Too big of a project? I don't think so-- there's typically only a handful of major players in any vertical market. Not enough interest? Hardly! MCAD and BIM are hot subjects. Too complicated to pick a winner in something so complicated as a CAD program? Well, I have heard that a lot. Yet, we (as a society) don't have any trouble picking out the best movie, best restaurant, best breed of a dog, Time magazine even picks the Person of the Year.

Care to guess what the real reason you don't see more head to head reviews?

Comments

[Kevin Quigley]

I think people are missing the point here. Nearly every review I read has a conclusions section detailing the comparisons as that reviewer sees them. The problem is that reviewers will not be practising professionals depending on the software to get the job done so they just cannot be expected to know each system well enough to make the review worthwhile.

I believe what many users want is head to head reviews for certain industry fields. I'm an industrial designer. I use SolidWorks, Ashlar-Vellum Cobalt and VX. My single most important need is shape modelling. What I am interested in is how to model any shape I want.

For others is might be configurations, or drawing efficiency, integration with CAM or FEA, large assembly modelling and control, rendering.

Most reviews on new releases are aimed at users of those systems - that makes sense. But to be quite honest most upgrade info is available via the bloggers and even official sites long before you get a review. Also most CAD companies now have Labs or beta programmes for subscription users so far more users are now exposed to the new systems and as soon as the NDA's come down the floodgates of data opens - look at SolidWorks 2008 for example.

I think the problem for traditional reviewers is that the mainstream stuff is just too well covered these days to make any review worthwhile - frankly the magazines would be better of printing links to bloggers who have posted reviews in extreme detail.

The value, I think of traditional reviews, is to highlight the lesser known systems and specialist systems, explaining how and why they are differentiated from the mainstream. I certainly came across some apps I use via MCAD in the UK (thanks Al) that to be honest I probably would not have heard about amid the SolidWorks/Inventor/SolidEdge/ProE/CATIA/UG mainstream forest had they not been published in those pages.

So head to head reviews - waste of time. Reviews of new and emerging technologies - vital.

[Al Dean]

Roopinder

You know the answer to this as well as I do.. You can't do them.. Well you can't but who's to say who's right - they all do much the same, SolidWorks, is Solid Edge, is Inventor, is Catia, is NX, is whatever. only when you delve into very specifics, in terms of modelling technology, in terms of process, industry specialisation, in terms of specific workflow, can you find the highlights. Essentially, you could spend all year do this, going in circles and the end result would pretty much be a draw. SolidWorks ask me to do this every time I see them - and the answer is always Why? - Solid Edge is better than SolidWorks at Sheet Metal and intelligent sub-assemblies (or at least it was in the last nano-second that I checked), SolidWorks is better than Inventor at other stuff and Inventor is better than the others at other stuff. By trying to be specific, you end up being vague to the point of.. well.. pointlessness..

The other alternative is a list of check boxes..

Fillets X
Fillets with tangency control X
Fillets with corner set-up options X

And that is even more tedious.. A good review, in my opinion for what its worth, engages the reader to think, to think about what they're using, what they'd doing - should they upgrade, are they missing out on a competitive edge - keeping people informed.. give them a head start in a confusing world.. is that right? you guys tell me.. and I'll change how I do it (and I mean readers/users - not analysts and press - its all about the reader and the end user - and the ones I speak to, like what we do)..

AEC tools? buggered if I know, its all about drawing bricks right?

[Josh]

head-to-heads take A LOT of time. It would be a full-time. I'd love to see something like cmsmatrix.org for cad.

[Stefan Boeykens]

It is hard to thoroughly compare those complex applications. I use ArchiCAD and have (basic) experience with Revit and ADT (and some VectorWorks). Yet to translate that into an objective and deep comparison study would take a long time, which is not compensated by any financial payment whatsoever.

In the past, I made an extensive comparison study of rendering software (using a single CAD model), but that was more user-oriented rather than fully edited. There was an interest for it, though.

I currently maintain an applications database, but that too is not fully edited for full comparisons.

[Ed Goldberg AIA, NCARB]

Robin is partly correct - there are few people who know several competing programs well enough to analyze them. Besides this, I am the only practicing professional who is an Industry analyst - and I will not write for CADALYST. I know Architectural Desktop, Revit Architecture and ArchiCAD.

[Orhan Toker]

Hello Roopinder,

I'm completely agree with you. All of our readers' main language is not English except Scott. Therefore it's very hard to write head to head product reviews for us.
Thank you very much for your critics.

[ralphg]

Another issue is lack of pay. Which magazine (or commercial blog) can afford to pay enough to make such a review worth the time?

Back in the good old days (early 1990s), I could make $2,500 writing a 10-page comparative review for Cadence or PC World.

Last year CADalyst asked me to write for them, but there was no mention of financial compensation. (The review fell apart anyhow, after the vendor insisted I first fly to their facility for several days of training.)

[Jimmy Bergmark - JTB World]

I've been through the process as CAD Manager to evaluate a multitude of products for a EPC company looking at most all products in one way or the other. I think it's valuable to have reviews like this but it will always only be a small help because when you decide to go for a solution at a company there are so many more factors to take into consideration that a review with a comparison of products will only help a little if anything at all. As Robin says you really have to do a project to start seeing what can be done and what cannot be done and how that fits into what you want to achieve.
Many years ago when there where not as much information available like today on Internet including newsgroups, blogs, etc it sure was harder to get help to find pros and cons about different products.

[Greg Milliken]

There's no longer enough difference to matter.

[Robin Capper]

Like many Bloggers I generally only post on stuff I use (or would like to use). While anyone can form an impression of an application in a few hours it takes months, even years, of use before you really know it. I often wonder how those "big reviews" were done. Just running through the canned examples, or tutorials, will usually only show what a product can do. Often only completing full projects shows what it can't...