First lets start by answering “Do I agree with the VMworld 2010 Awards winners?”. My answer to the previous question to the most part I agree with most of the winners, though I believe the judging process might need a bit of improvement. I have just came across Eric Siebert, which discuss the judging process for VMworld Award winners. You can find Eric article at:
Although the process is quite fair to every vendor, I still believe without installing and testing the product running we will allow vendors to promise the sky and deliver nothing close. I am sure the judges like Eric are Virtualization experts that are not easy to fool about the products functionality, but still there is a gray area that vendors can play around. I believe it would be more fair if the judges get to test these products in the lab & ensure it meets the promised functionality. As the installation of all the products to be tested can be time consuming for the judges, it might make sense to let these app vendors install the application and keep it ready for the judges test drive.
Further, I was wondering if the judges declare them selves to the competing vendors as judges when they go to evaluate and ask about the products. I believe it will be quite interesting to get the judges to act as they are just a normal attendees and detect how of a professional reaction the vendor at the booth offer.
One extra tip that I think it would be nice to let VMworld attendees votes count for a percentage of the judging process. As what is the most innovative, creative, & the most important to fill market gap will highly differ from a customer view , an engineer at VMware, or Vendor view. It will help to gather the full vote. Let’s the attendees vote count for something!!
As VMworld Award is the most prestige award in the virtualization market today, keeping it fair and raising the bar for it is quite critical. Many customers will factor the award in their buying decision and are not aware of the judging process & will assume the product perform as advertised being a VMworld Award winner. It’s important to test these products to death before declaring them a winner not to let down customers on the long run.