Zerowait Corporation

One of the comments we hear a lot from field engineers is the difficulty of getting a 4 post rack mount kit that is adaptable to almost any configuration of 19″ rack and doesn’t cost a small fortune. Our company has always focused on customer service, so we searched for a solution to this seemingly easy problem. We weren’t able to find one either, so last year we said “enough, we‘ll build it ourselves”, and our rack kit – easily adjusted and priced right – is now available from our store.

This is just the first of a series of affordable products Zerowait is creating for our customers’ storage and networking needs. Keep your eyes on Zerowait as we introduce new products and services designed to make Networking and Storage simple and affordable..

Jon Toigo and I have been chatting about tactical and strategic storage decision making for many years. He posted a thread of a recent conversation on his blog site and we decided to cross post his blog.

The Long Slog Forward – A Dialog Begins

My previous post about the long slog ahead for IT business prompted a communique from my friend and fellow blogger, Mike Linett, over at Zerowait. We have been chatting back and forth via email over the past few days and I thought I would post the emails here so that others can chime in if so inclined.

ZEROWAIT:

There is a fundamental shift that is occurring in the enterprise storage and networking business. Enterprise network and storage consumers have encountered an uncertainty tax based on the unpredictable government regulatory and tax policies as well shifting models on hardware supercession and maintenance costs from hardware vendors. When combined, there is no way to accurately forecast what operational costs and support costs will be for the 24 month to 36 month time frame. Based on an inability to forecast future costs, we are seeing enterprise customers reduce their time horizons for hardware upgrade and support ROI calculations.

I feel that the “uncertainty tax” is causing enterprise network and storage consumers to look at third party support vendors. Corporate finance is asking which costs can be reduced for FY 2010? I suspect that in FY2010 organizations will try to find a rational predictive model from which to model their network operational costs. The uncertain Financial and Regulatory environment is causing corporate financial types to keep a stronger cash position, therefore CAPEX spending is rationally being curbed based on caution.

Corporations are acting like rational consumers, individuals are saving more and shopping at Walmart and Costco to cut their weekly Operating Expenses to live within their paychecks. Companies are now living within their cash flow statements, and managing risk with the security of a stronger cash position, and shopping for competitive support pricing.

DRUNKENDATA:

I agree with your view of corporate evaluations becoming more rational. But, if that is the case, how is EMC showing $1.1B in revenues per IDC from equipment sales? That is double its nearest competitor’s number for what is arguably inferior products in many cases. Even if you are an EMC fan, the long tail cost of their gear comes in the form of a warranty and maintenance contract that rivals NetApp in terms of its price.

Some might argue, as one financial analyst recently did with me, that the key to enterprise sales is really predictability. The big league consumers want an enterprise storage rig from a vendor who will still be around for three years. The CAPEX spend is either a write-off, or they lease it, which is also a write-off. All things being equal, they are taking Gartner’s cue that the most important criteria in vendor selection is “ability to fulfill.”

Everywhere else but Wall Street, I see what you are describing: close attention to economical purchasing. I have heard repeatedly that companies are dropping maintenance from the OEM and shopping it out to third parties where (1) knowledgability is there (often because the aftermarket service company is staffed by techs who were laid off by the OEM), (2) maintenance can be performed remotely (so the company doesn’t need to increase its own staff headcount), and (3) the support service provider has been around for a few years and is likely to exist over the life of the maintenance agreement.

As for equipment itself, the other-than-Wall-Street enterprises seem more interested than ever before in Red Hat, Gluster, ZFS, and other Linux/Open Source wares to provide value-add services around storage. They are not shopping for integrated one-stop-shop rigs with lots of value-add features embedded on the array controller that they may or may not use but must pay for nonetheless. Plus, the value add stuff tends to break down and requires patching, in some cases, with the same frequency as Microsoft software.

A big issue in every shop I go to is the mismatch between existing storage infrastructure and server virtualization. The latter is a huge driver of server consolidation, to be sure, but hypervisors are also gumming up I/O performance and creating real opportunities for third party I/O monitoring and measurement wares (like Virtual Instruments), for off-box I/O routing (Xsigo comes to mind), and for simply-connected “network storage” rigs – and in many cases Direct Attached Storage. Also, give the failure potential of virtual server-based hosting, I am also seeing virtualization create a lot of mindshare for data protection software. Even the hardware independent virtualization software firms seem to be garnering revenue from the decisions of CIOs to embrace VMware and Hyper-V.

What seems to be missing in all of this is a unified architectural model, complete with open management standards, that will enable cheaper storage to be integrated with virtual server environments in a predictable and cost effective way. Initially, I think that companies will use open source operating and file system features to wed storage to virtual servers. Going forward, however, I am a huge advocate of a Web Services management framework leveraging REST.

What are your thoughts? Is there an opportunity to turn a frown upside down and to get consumers thinking about the current situation as strategic, rather than a temporary response to a temporary economic downturn?

ZEROWAIT:

As a consequence of the financial panic of 2008 and 2009 there has been a tectonic shift in customers’ perceptions of proprietary features and associated values. This shift will have deleterious effects on enterprise OEM’s pricing models. For example, Enterprise customers have recognized that open source tools such as Nagios and Cacti are excellent, reliable and affordable for monitoring enterprise systems, and Apache is the choice of many enterprises for their web servers. Due to tightened budgets the value proposition of Open Source when there is a reliable support organization backing up the OS, like Red Hat, has been recognized by Enterprises.

The new model for enterprise customers is value investing in network and storage equipment. Customers will buy as they need it, and do not want to pay outrageous licensing fees based on marketing limitations. The software management costs should not trend up by terabyte, but should trend down, this can be accomplished with Open Source, but not with proprietary Hardware and Software Products. Proprietary vendors sales models are still based in the unit sales of box and license sellers, and this is a dying branch of the evolutionary storage tree. Conversely, Zerowait’s business is based on the long term service and support business, which aligns perfectly with our customers long term ROI needs. The new reality of storage is that customers are not going to buy over provisioned arrays or pay 300% markups for hard drives that the Enterprise OEM’s charge Although the major storage OEM’s and Gartner may not realize it, customers are changing their data models based on usage. Primary storage will probably stay on high priced OEM solutions for years to come, but the other 80% of data will slowly migrate to Open Source solutions that have a reliable service and support organization behind them.

Proprietary vendors claims of interoperability often require a toxic mix of patches and software upgrades to continue to work together. Over time these, patches and band aids have gotten to the point that no two networks are running the same code and revisions and they are all unique. Interoperability was supposed to leverage the idea of interchangeable parts, but now very few components can be unplugged from one network and plugged right into another without some revisions to firmware or software. Network engineering has turned back into an art more than a science as complexity has grown. Sometimes it seems that many of our network engineering customers have more in common with alchemists than with computer theorists.

If interchangeable parts made the industrial revolution possible, why is it that computer hardware has become less interchangeable over time? Perhaps the proprietary hardware vendors have wanted to lock in customers to their high priced solutions? The financial panic has broken that model though, and the marketplace is embracing open source software and POSIX very quickly due to the cost and restrictions of proprietary hardware. You can’t fight the marketplace, you must embrace its lessons.

DRUNKENDATA:

Mike, I follow you on the bulk of what you are suggesting. My concerns are perhaps further down in the weeds, or in the clouds, depending on your perception.

First, while you may be right that simpler technology based on open source may trump purchases of overbuilt Enterprise OEM rigs in the short term, both for reasons of acquisition cost and cost of ownership, I am worried about the way that companies are/aren’t considering the bigger drivers of cost.

Storage growth is predicated upon data growth. Data growth is out of control because data itself is so poorly managed. If our assessments are correct, and literally 70% of the data being written to disk is more appropriately stored in a cheap removable-media archive (or in the trash bin), then shouldn’t firms be looking at discovery and archive tools in a much bigger way?

When firms do deploy disk, it is critical that it be managed in accordance with some sort of framework. That way, you become proactive in fault resolution and develop a knowledge of trends that can help you plan your acquisitions and utilize better the investments you make. Without effective management, you again see the hardware cost curve accelerate. Yet, again, I do not see a huge push to get a hardware management paradigm in place in the companies I visit.

Finally, I don’t see companies adequately addressing the long tail cost of storage acquisition – the cost of redundancy required to safeguard data. The redundant components of infrastructure, whether in the form of a second array, a tape library, etc., required to ensure the availability of data that is being stored on the new primary rig, are not being purchased. That RAID 5 continues to exist at all is testimony to the fact that companies are not thinking through the ineffectiveness of that scheme when applied to large capacity drives. That said, mirroring is considered by many to be a cost too dear in the current economy.

In short, even with the advent of cheaper storage and open source software, I find myself wondering whether this is ultimately strategic in the absence of effective hardware and data management…

ZEROWAIT:

You have pinpointed the problem. The proprietary vendors are driven to simplify proprietary management of their hardware, while adding complexity to management of a mixed hardware environment. For hardware vendors, who are being judged by VC’s and public stock prices, units sold is a number that makes sense to their investors who are looking for short term Return on Investment. However, customers want to get a quick ROI and also a long and affordable service duty cycle out of Capital equipment. Most Enterprise storage consumers that we work with yearn to be freed from high priced proprietary licensing schemes and single vendor lock in support programs. Organizations that embrace Open Source can reduce the lock in possibilities for proprietary hardware manufacturers. Once freed from vendor lock in schemes, consumers can and will judge hardware and software by their reliability as well as their acquisition and support costs. Open source hardware and software solutions will commoditize pricing and force vendors to compete on the true value and price of their service and support. Once freed of lock in a true competitive market can flourish, which inevitably will reduce price and improve products and services.

Since there is not an open market for storage decision making, Enterprise consumers have to interpolate their changeover risk with their cost of being locked in to a particular vendor. The lock in and cost of change is well covered by the Economist Hal Varian. Our customers recognize that there can be competition within an OEM’s market for legacy support, and often the OEM’s will reduce their prices once we have a service contract in place within a customer’s data center.

In summary, the financial panic of 2008 has opened the door to a tectonic shift in the way storage is looked at by organizations. The over hyped concept of cloud storage is being considered by many, because it is an easy first baby step toward a new storage pragmatism. Everyone that has looked into the cloud model sees dangerous cells in the towering cumulonimbus of cloud storage models. Issues of security, accessibility and speed being the first concerns people voice. I predict that 2010 will be the beginning of the end of the reign of the proprietary hardware and software vendor. 2010 should also signal the beginning of storage pragmatism and organizations will rationalize their storage costs with open source hardware and storage management solutions.

(MORE TO COME)

One OEM has been a big believer in the usage of Avatars for a few years now, as this story from 2007 highlights. The second link shows how the movie industry has embraced the idea, and may have created a product from it.

http://www.thehindubusinessline.com/ew/2007/06/25/stories/2007062550110800.htm
Network Appliance (NetApp) is a $2.8-billion storage networking firm up against the likes of EMC Corporation and Hewlett-Packard. Maintaining the culture of innovation amongst its engineers so products are creative and useful is a prime focus to reach up the market share ladder.
One way of developing better products is by tuning into clients’ data management woes, understanding requirements, and shaping products to reflect needs. However, the knowledge that after-sales and marketing folks get from the field does not always permeate to the engineers/developers.
And that is why the company created Mike Raja and Joe and four other personas. Joe could be a DBA (database administrator)/Chief Information Officer/any other user of NetApp’s data management products.
“This is a new approach that we are trying. Innovation is important and these personas for users of our products help continually innovate,” Louis H. Selincourt, Vice-President, SMAI & Offshore Operations, told eWorld over lunch at the company’s headquarters in Sunnyvale, California, recently.
The idea came from the book, “The Lunatics Are Running The Asylum” penned by Alan Cooper, whose company makes software more user-friendly.
“Personas create a consistent ID of our users, so we can discuss them across the company while brainstorming,” says Selincourt.

http://www.theregister.co.uk/2009/12/21/avatar_storage_effects/
Weta used NetApp kit to store the incoming data, then used a huge number of workstations and bladed servers – with 30,000 cores in total – to work on it. The NetApp filers were fitted with up to five 160GB DRAM cache accelerator cards in their controllers, the PAM (Performance Acceleration Modules) caches, to speed file access by the Weta creative people and the servers.

There is a certain balance to this when we recognize that the same OEM that has used Avatars for years has its equipment used in the production of a movie based on Avatars. At Zerowait, we still view our customers as individuals with specific needs. Probably there won’t ever be a movie based on our customer service and support models.

I have known Nathan Thomspon of Spectra logic – www.spectralogic.com – for many years, we share a common interest in aviation. Nathan’s company is celebrating its thirtieth anniversary. I had dinner with the Spectra folks recently at the Super Computer conference in Portland.

Here is a snippet on an article in his local paper on the story:

Data-storage company started in a dorm room 30 years ago
By Alicia Wallace
Daily Camera
Posted: 12/14/2009 01:00:00 AM MST

As the founder and chief executive of a high-tech company, Nathan Thompson is used to looking forward, not back.

But he’s been doing a lot of the latter recently as his Spectra Logic Corp. — a Boulder- based data-storage company whose roots trace back to Thompson’s dorm room at the University of Colorado — marks 30 years in business.

In 1979, Thompson was a sophomore engineering major at CU when he withdrew the $500 that was in his bank account to start a memory-board manufacturer called Western Automation. In 1987, Western Automation bought the assets of a company called Spectra Logic, and the company reincorporated as Spectra Logic in 1993 because it was a better- known name in data storage, Thompson said.

The 30-year milestone is significant, Thompson and others say, considering the dynamic nature of the storage industry.

“It is kind of amazing . . . the degree to which companies have started and died — (companies) that were competitors or the hot company around here,” he said. “We’ve just sort of steadily marched along.”

Congratulations Nathan and all of Spectra Logic.

People often ask me what differentiates the service and support that Zerowait provides from that of an OEM. I like to break it down to three component parts: Cost, Complexity and Compromises.

The cost of support services from Zerowait is typically 40% – 60% lower than the cost for support from the OEM on the same equipment. The primary reason is that the OEM typically assumes that there is no competition for their unique product and the customer will not try to use another support organization on their proprietary parts. OEM’s seem to believe they have cornered the market. However, many customers have figured out that their vendors are using Off-the-Shelf components which can be (and have been) successfully cross referenced and inventoried by other support organizations. Further the OEM often recycles used equipment to service RMAs, while not exactly publicizing the fact. This leaves customers wondering what’s the difference? Pretty much none, as it turns out. Therefore, a customer looking for a support organization for legacy equipment support or seeking a source for inexpensive system improvements often discovers that their OEM is more expensive for no other reason than “because I said so”, coupled with an insistence on upgrading the customer to a completely new system. The problem for the OEM is that, especially in tough economic times, top-to-bottom upgrades are often not needed and cannot be justified. Cost containment is a big reason that independent parts, service and support organizations are growing.

Often OEM’s will say that their proprietary systems are so complex that no independent service organization can possibly handle their equipment’s unique needs. Meanwhile, these same companies habitually outsource their phone and field service to third party companies for their own cost containment reasons. As a result, complexity and simplicity are mixed in marketing messages. The reality of service and support is that OEMs design their equipment for either efficient parts replacement and repair or for OEM-service only repairs. For systems no longer under OEM contract, third party support organizations can and do provide an affordable and simple option for taking the complexity out of support and providing an alternative.

Many OEM’s try to say that using third party support compromises your legacy equipment’s reliability. This is perverse logic since these same companies, when asked about legacy support, demur that their engineers are no longer proficient with EOL equipment and therefore a customer needs to upgrade to maintain the highest levels of support. This logic makes it seem as if the high availability equipment that you purchased last year is now low availability simply because the OEM has superseded the equipment and no longer wants its people to know how it works. Does that make sense?

Ultimately, customers are put into a compromising situation if they do not allow competition into their data center. Without competition proprietary OEM’s will not have any reason to improve their support quality or lower their support prices. That is why savvy customers are engaging third party support providers and driving support prices down and driving support quality up.

I have been traveling over the last few weeks visiting customers around the Northwest. The story is the same pretty much everywhere… Budgets are tight, and our customers are looking for value in their storage support services. The tough economy causes a lot of customers to try new and creative things to keep their costs down.

The supercomputer show in Portland was well attended and their were a lot of our customers there, I was able to catch up with many folks that I have not seen in a while.

2009 has been a tumultuous business year and many of our customers have cut back on new equipment purchases. Today in the Wall Street Journal there was an article with the following headline. “Business Bankruptcy Filings Increased 7% in October”

Later in the article there was this gloomy prediction.

“On a year-to-year basis, business bankruptcies shot up 24% in October compared with the same month in 2008. Mr. Williams called that increase “substantial” and said it is a bad omen for the final months of 2009 and the first quarter of 2010.

“Bankruptcy filings are a lagging economic indicator so it’s likely that we’ll see bankruptcy filings increase for the next several quarters,” he said.

Through the first 10 months of the year, 74,832 businesses filed for bankruptcy, a 16% increase from the same period last year.

The total number of October bankruptcies, including both personal and commercial filings, increased 20% from the same month last year. “

It is hard to be optimistic in this business environment, and yet our sales continue to grow as people embrace alternatives to save money.

Over the last several weeks we have been talking to customers about what they like and dislike about their storage providers. Listening to the stories of what they are told by their OEM salesman and engineers and comparing that to the reality of service and support after product delivery reveals a startling comparison of promises versus reality.

Dealing with the big Storage OEMS’s reminded one customer of The Star Trek scene of Fizzbin. I like that comparison.

Fizzbin – The Rules of the Game
Kirk: The name of the game is called… Fizzbin. Each player gets 6 cards, except for the player on the dealer’s right, who gets 7.
Thug: On the right.
Kirk: Yeah. The second card is turned up, except on Tuesdays.
Thug: Tuesday.
Kirk: Ohh! Look what you’ve got- 2 jacks. You got a half-fizzbin already.
Thug: I need another jack?
Kirk: No. If you got another jack, why you’d have a shralk.
Thug: A shralk?
Kirk: Yes, you’d be disqualified. No, what you need now is either a king and a deuce, except at night, of course, when you’d need a queen and a four.
Thug: Except at night.
Kirk: Right. Oh, look at that, you’ve got another jack! How lucky you are! How wonderful for you! Now if you didn’t get another jack, if you had gotten a king, why then you’d get another card except when it’s dark , when you’d have to give it back.
Thug: If it were dark on Tuesday.
Kirk: Yes, but what you’re after is a royal fizzbin, but the odds of getting a royal fizzbin are astron – Spock, what are the odds on getting a royal fizzbin?
Spock: I have never computed them, Captain.
Kirk: Well, they’re astronomical, believe me. Now, for the last card, we’ll call it a kronk. You got that?
Thug: What?

Compare Captain Kirk’s exchange with the response you get when you are trying to navigate the RMA process or find out where your 4 hour parts guy is.

Repeatedly we are told that Zerowait provides easy to understand service and support, and no hassles. Things don’t have to be confusing, so why do manufacturers make it that way?

A lot of the trade press is talking about cloud computing and even the Economist has gotten the bug. That surprised me because the cloud is a fairly technical concept.

“Granted, there are hundreds if not thousands of firms offering cloud services—web-based applications living in data centres, such as music sites or social networks. But Microsoft, Google and Apple play in a different league. Each has its own global network of data centres. They intend to offer not just one or two services, but whole suites of them, with services including e-mail, address books, storage, collaboration tools and business applications. They are also vying to dominate the periphery, either by developing software for smart-phones and other small devices or by making such devices themselves.”

I speak to customers about cloud services on a daily basis, depending on the definition of “cloud computing” we are all using some or maybe using a lot. But for enterprise storage security reasons most of the folks we work with are leery of the amorphous cloud as their depository for critical company data. At Zerowait we don’t consider remote access of data cloud computing, since most of the companies we deal with have remote data sources and applications.

If remote data centers, web enabled applications and Citrix type access are defined as part of cloud computing, then it is ubiquitous. It seems to depend on the definitions and the expectations of the users and whether they consider the cloud to be third party hosted applications and storage or not.

I think the question should revolve around who owns and runs your organization’s critical databases and applications. If the lines of ownership of the hardware assets and the data are unclear you may be a cloud user already.

Wikipedia says this… “A technical definition is “a computing capability that provides an abstraction between the computing resource and its underlying technical architecture (e.g., servers, storage, networks), enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction.” This definition states that clouds have five essential characteristics: on-demand self-service, broad network access, resource pooling, rapid elasticity, and measured service.“

Whatever your opinion is, I think everyone will agree that some stuff should not leave your organization’s control.

A lot of our customers and friends will be at the SC09 show in Portland this year. I have been invited to the show, and will be working with a few of our customers on how to mange and reduce their costs of storage on their 100TB- 1PB storage archives.

Over the last 20 years Zerowait has built a reputation for providing outstanding service and support to our customers. Over the last few months we have received a series of requests to help our customers with extraordinary data sets and tremendous volumes of data.

Outrageously large data sets and constrained budgets should create an atmosphere for some very interesting discussions.