Reliable, Repeatable and Verifiable

“You don’t understand how we calculate price/performance,” NetApp says.

Over the last few months it has become quite apparent that the performance numbers provided by NetApp are not in any way verifiable. Jon Toigo’s request for verification of their performance claims have gone unanswered by NetApp and the slow trickle of comments we used to receive from NetApp users asking for reliable performance figures has become a daily request for some sort of performance verification criteria. I think Jon Toigo’s test lab is the best place to get this information generated, since his company has the facilities to produce the data.

In past discussions with Jon, I have stressed that the information that he produces should be able to be reproduced by any user in their own lab set up. NetApp has fudged the test criteria in their SPEC.ORG tests by changing all sorts of parameters to make their numbers match their claims. Customers have noticed and I am surprised that EMC and Hitachi have not started to point out the inconsistencies in NetApp’s claims. Perhaps it is the nature of the beast. But our clients need reliable performance numbers to make their enterprise storage decisions. If their vendors won’t supply reliable performance numbers who can they trust?