Zerowait Corporation

Imagine if before you saved a PDF on your corporate storage a statement came up that said “This 500 meg file will cost you $1.00 to save on our corporate system.” Charging users for storage is something that is talked about often, but rarely put into place. Setting up the parameters for creating a system would not be that hard, budgets could be set by departments and per person. And there could be extra charges for saving files on based on different back up criteria. The program could also provide weekly, monthly and quarterly email to users suggesting which files had not been accessed since being saved and suggesting they might be ripe for deletion, but unless people are charged for their storage resource usage most would probably ignore the suggested deletions.

Implementing the program would be relatively trivial, but I imagine creating the charge back scheme would elicit howls from just about every department in an organization. Storage is viewed as a free resource by most corporate users, and like any free resource it is used inefficiently. However, as soon as users saw that they were running up against their hard budget end point, I’ll bet they would start deleting the unimportant jokes, presentations and copies of the garbage they are keeping on corporate spinning disks.

Until enterprise storage users are charged for their resource usage they will not have any economic reason to use storage efficiently. Although enterprise storage remains an expensive resource to purchase, manage and maintain by corporate IT staffs, it is viewed as free to end users. Since storage users are not using their storage resources economically or being charged for their usage, corporate IT storage will remain a cost center in the eyes of management.

In this environment, cost conscience corporations will be forced to impose budget constraints on the IT department’s storage acquisitions, service and support contracts. Since companies can’t go after the users of storage, they go after the suppliers. Today many companies are pushing for discounts from their primary storage suppliers, while others are looking for alternatives to the high cost of arrays and support that the major OEM’s are charging.

Zerowait does not have a toll based solution for charging corporate storage resource users, but we do have a variety of solutions for helping storage IT save money on hardware, service and support. In addition to our legacy hardware support business we are introducing a new storage solution which changes the dynamics of enterprise storage costs. Zerowait’s SimplStor product line was designed with input from our customers to help them reduce their storage expenditures. When viewed together, Zerowait’s legacy support and SimplStor are an economical solution for the acquisition and maintenance of our customers’ storage resources.

Last week I was in Texas visiting clients and working with them to solve their enterprise storage management problems. I really enjoyed working with my clients, many of them said that our products and pricing were right on target and that they would be pulling the trigger soon.

We had our Dallas customer appreciation dinner and had a great time with our customers and friends. Jon Toigo gave a presentation on the state of Enterprise Storage and it was very well received and prompted many discussions during dinner.

Last year was the first time we ever displayed at EMC World and it was a great success for us. We saw many of our customers and met many new folks who are trying to control their support costs. The folks at EMC put on a great show last year in Orlando, and we expect this year to be just as good in Boston.

We hope to see you there.

Jon Toigo initiated another dialogue with me which you can read below.

Conversations with Mike 2

Once again, I exchanged some emails with a guy who is in the trenches with customers as much as I am: Mike Linett, CEO of Zerowait. This is from a continuing email exchange that I thought was worth sharing.

Hi Mike,

Several inquiries have been coming my way to perform storage assessments as companies seek to drive cost out of infrastructure. One, in particular, is looking for a better management plan to reduce the costs of labor associated with storage administration, but the majority of the others are seeking tactical cost reductions through the use of less name brand-y gear and the replacement of service contracts on the gear they already have (sourced from their OEMs) with qualified third party maintenance and support. Many have moved older gear into a secondary storage role, which I take to mean that they are not taking it off the line, but they are using it to store less frequently accessed data.

Are you seeing similar trends? What, honestly, are the benefits and the potential foibles of such a strategy?

As I see it, going to off-brand storage arrays that do a yeoman’s job of primary storage but without the brand name OEM pricing is a tactic that is well worth a look in this economy. The problem is that some of my clients are already hurting for staff and seem to be buying into the claims of brand name vendors that wrap around service and support contracts are an offset to being short handed. True?

Second, band name or not, the real cost drivers in storage are mismanagement of data and lack of visibility into infrastructure that enables proactive redress of problems that are building up. There is also a huge disconnect between the storage folks (most of whom are server folks) and the infrastructure changes that they are making and whoever is doing disaster recovery planning. Data volumes are being added or moved and mirrors/backup schemes are being broken unnoticed. That’s scary.

Third, I have always been a proponent of qualified third party maintenance as an alternative to overpriced maintenance agreements from OEMs, as you know. I mean, why should you pay for maintenance from the OEM, which tends to increase in cost as the gear gets older, if you have migrated the gear itself into a secondary storage role?

Finally, some customers tell me that they are being wooed to outsource – I’m sorry, cloud-ify – their secondary storage at 20 cents per GB. As an aside, I wonder if anyone is doing the math on drive technology advances, like Toshiba’s 5TB per square inch drives, which are coming to market within the next 36 months? Assuming that drives are enterprise quality and follow the same cost pattern as previous units, it seems to me that we are looking at pennies per GB on massive drive capacities in the near future. So, why use a cloud for capacity?

Looking forward to your response.

Hi Jon:

Storage assessments and a reappraisal of the way organizations are structuring their data has become a large part of our engineering team’s daily discussions with customers. Even the folks in the executive offices understand that a Seagate drive should not cost 400% more when purchased through their storage OEM’s, than from NewEgg. The perceived value of enterprise storage vendor’s wares has declined as the effects of the financial panic of 2008 and 2009 have trickled down and curtailed the Enterprise storage acquisition budgets of most organizations.

It used to be that when we would offer a quote for support to customer they would ask us “How can you charge so little?” In 2010, savvy customers ask us “ Why are OEM’s charging so much for support for of their equipment?” The answer remains that for some organizations there is a perceived value to having a different vinyl sticker or bezel color that makes the high price seem rational.

Often, I go to see a customer and we walk through their Data Center and assess the equipment within it. Most customers recognize that their equipment is made up of component parts from the same manufacturers, and that there are vast differences in price based on the OEM’s sales and marketing staff. IT managers understand that they don’t have the staff or resources required to integrate and debug hardware and firmware, but they can certainly see that a Seagate drive or an LSI card is the same in their Dell server as in their high priced array. So, they have to question the pricing models that the manufacturers’ are using, and they do not see the offsetting value of proprietary software from vendor’s offsetting their engineers’ costs. In fact, just the opposite is occurring. Customers are quickly recognizing that their engineering generalists can manage more open source equipment than their specialists can manage of proprietary arrays. Therefore, the higher priced array’s specialization is costing more to manage than an open sourced simple storage solution.

This model for 80% of an IT infrastructure creates more value out of best of breed components then piecing together proprietary solutions from a mix of vendors, and paying support for equipment which is used with F150 Pickup requirements.

JT – Second, band name or not, the real cost drivers in storage are mismanagement of data and lack of visibility into infrastructure that enables proactive redress of problems that are building up. There is also a huge disconnect between the storage folks (most of whom are server folks) and the infrastructure changes that they are making and whoever is doing disaster recovery planning. Data volumes are being added or moved and mirrors/backup schemes are being broken unnoticed. That’s scary.

IT today is very similar to business in general and the 80/20 rule applies. 80 percent of most business comes from 20% of the customer list. In IT, 80% of most organizations’ IT assets handle 20% of traffic, while 20% handle 80% of traffic. What is great about managing open source solutions is that it is easy to create a POSIX like solution for your own architecture and environment. Once your organization determines a “good enough” server and storage solution, your favorite integrator can build all types of hardware with minor changes in cards, memory and processor that use homogenous components. Using open source software the maintenance and support costs are lower and since there is an easily tapped community of experts to answer questions, a good staff of IT generalists will be able to provide a larger portion of an Organization’s IT infrastructure without specialized training.
Don’t panic. Embrace the wisdom of the marketplace. Slowly migrate to a solution of best of breed commodity hardware, and a staff of IT engineering generalists who know how to find answers.

JT – Third, I have always been a proponent of qualified third party maintenance as an alternative to overpriced maintenance agreements from OEMs, as you know. I mean, why should you pay for maintenance from the OEM, which tends to increase in cost as the gear gets older, if you have migrated the gear itself into a secondary storage role?

I sort of covered this earlier in my response, essentially people and organizations have different perceived values on the prices that OEM’s charge. Hard times demand more attention be paid to these costs, and this inevitably drives more companies to embrace third party support, as well as other self support models. Essentially, a movement toward commoditization and rationalization of computer hardware is occurring because of the tightened IT budgets.

JT -Finally, some customers tell me that they are being wooed to outsource – I’m sorry, cloud-ify – their secondary storage at 20 cents per GB. As an aside, I wonder if anyone is doing the math on drive technology advances, like Toshiba’s 5TB per square inch drives, which are coming to market within the next 36 months? Assuming that drives are enterprise quality and follow the same cost pattern as previous units, it seems to me that we are looking at pennies per GB on massive drive capacities in the near future. So, why use a cloud for capacity?

I have been trying to understand the foggy logic of cloud computing for a while, and although the marketing and sales pitches are really nice, the dollars and sense logic has escaped me. I can understand why certain applications can make sense as an outsourced solution, but I don’t see the benefits in cloud computing for most organizations that need data security as well as uninterrupted connectivity. I think there will be some good technologies developed through the cloud computing efforts that are being marketed today. I don’t see cloud computing taking away the corporate responsibility of providing a reliable network of systems to deliver content and data to users for the enterprise to thrive.

Mike

Jon and I would like your comments .

One of our European customers has been getting the run around from their current vendor’s sales team on their equipment pricing, upgrade and licensing policies and has been working with our European sales team to try to get to the bottom of the issue. The root cause of the issue is that the OEM sales team is focused on new product sales through bonuses and incentives, and hence that is the direction that they take. Our business is a service business. Our focus is long term retention of customers by providing them with affordable service and support.

High availability storage equipment is made to last for the long term. Coincidentally, our aim is to provide reliable support for the long term. Our focus provides a long term low cost strategy for customers of their storage equipment and also provides a long term business mode for Zerowait.

This week I visited several clients in North Carolina. Although their budgets are tight they are still trying to manage their growing storage requirements. The companies I visited with are working with our company to help them manage more storage with less money. Most of the clients I was meeting were referred to our company through the years and have referred our company to their peers. Two of the clients I met with were customers at their old companies and as they moved to new positions they recommended us and eventually their new employers became our clients also.

That is the way it seems to work. Our company has earned a reputation for honesty and dependability, and we really care about providing our customers with the service and support they need. Times are hard in many of the IT departments I met with this week, and they really appreciate how Zerowait helps them stretch their storage networking budgets.

Earlier this week I was visiting some of our clients in Canada. Each of the clients had a story to tell about how they were stretching their IT budgets and finding new ways to redeploy or use their older equipment. Rather than buying new, they have recognized that their older equipment is still High Availability although it has been superseded by the manufacturer. We have entered an era of austerity and the virtues of conservation of capital will become evident to the CFO’s of many organizations over the next few years.

We are talking to customers who are redeploying older equipment in the USA and throughout the world as customers try to rationalize the features and values they are receiving from their software and hardware vendors. It makes sense to maintain equipment and stretch out the service duty cycle the same way many people are keeping their cars longer than they used to.

In my opinion, it is the uncertainty of the future of business taxes and regulations which are causing a wave of cautious conservatism in IT budgets. Uncertainty has costs which are hard to quantify, but when reviewed make sense.

I am optimistic about 2010, because our customers recognize that we can help them thrive in the era of austerity.

One of the comments we hear a lot from field engineers is the difficulty of getting a 4 post rack mount kit that is adaptable to almost any configuration of 19″ rack and doesn’t cost a small fortune. Our company has always focused on customer service, so we searched for a solution to this seemingly easy problem. We weren’t able to find one either, so last year we said “enough, we‘ll build it ourselves”, and our rack kit – easily adjusted and priced right – is now available from our store.

This is just the first of a series of affordable products Zerowait is creating for our customers’ storage and networking needs. Keep your eyes on Zerowait as we introduce new products and services designed to make Networking and Storage simple and affordable..

Jon Toigo and I have been chatting about tactical and strategic storage decision making for many years. He posted a thread of a recent conversation on his blog site and we decided to cross post his blog.

The Long Slog Forward – A Dialog Begins

My previous post about the long slog ahead for IT business prompted a communique from my friend and fellow blogger, Mike Linett, over at Zerowait. We have been chatting back and forth via email over the past few days and I thought I would post the emails here so that others can chime in if so inclined.

ZEROWAIT:

There is a fundamental shift that is occurring in the enterprise storage and networking business. Enterprise network and storage consumers have encountered an uncertainty tax based on the unpredictable government regulatory and tax policies as well shifting models on hardware supercession and maintenance costs from hardware vendors. When combined, there is no way to accurately forecast what operational costs and support costs will be for the 24 month to 36 month time frame. Based on an inability to forecast future costs, we are seeing enterprise customers reduce their time horizons for hardware upgrade and support ROI calculations.

I feel that the “uncertainty tax” is causing enterprise network and storage consumers to look at third party support vendors. Corporate finance is asking which costs can be reduced for FY 2010? I suspect that in FY2010 organizations will try to find a rational predictive model from which to model their network operational costs. The uncertain Financial and Regulatory environment is causing corporate financial types to keep a stronger cash position, therefore CAPEX spending is rationally being curbed based on caution.

Corporations are acting like rational consumers, individuals are saving more and shopping at Walmart and Costco to cut their weekly Operating Expenses to live within their paychecks. Companies are now living within their cash flow statements, and managing risk with the security of a stronger cash position, and shopping for competitive support pricing.

DRUNKENDATA:

I agree with your view of corporate evaluations becoming more rational. But, if that is the case, how is EMC showing $1.1B in revenues per IDC from equipment sales? That is double its nearest competitor’s number for what is arguably inferior products in many cases. Even if you are an EMC fan, the long tail cost of their gear comes in the form of a warranty and maintenance contract that rivals NetApp in terms of its price.

Some might argue, as one financial analyst recently did with me, that the key to enterprise sales is really predictability. The big league consumers want an enterprise storage rig from a vendor who will still be around for three years. The CAPEX spend is either a write-off, or they lease it, which is also a write-off. All things being equal, they are taking Gartner’s cue that the most important criteria in vendor selection is “ability to fulfill.”

Everywhere else but Wall Street, I see what you are describing: close attention to economical purchasing. I have heard repeatedly that companies are dropping maintenance from the OEM and shopping it out to third parties where (1) knowledgability is there (often because the aftermarket service company is staffed by techs who were laid off by the OEM), (2) maintenance can be performed remotely (so the company doesn’t need to increase its own staff headcount), and (3) the support service provider has been around for a few years and is likely to exist over the life of the maintenance agreement.

As for equipment itself, the other-than-Wall-Street enterprises seem more interested than ever before in Red Hat, Gluster, ZFS, and other Linux/Open Source wares to provide value-add services around storage. They are not shopping for integrated one-stop-shop rigs with lots of value-add features embedded on the array controller that they may or may not use but must pay for nonetheless. Plus, the value add stuff tends to break down and requires patching, in some cases, with the same frequency as Microsoft software.

A big issue in every shop I go to is the mismatch between existing storage infrastructure and server virtualization. The latter is a huge driver of server consolidation, to be sure, but hypervisors are also gumming up I/O performance and creating real opportunities for third party I/O monitoring and measurement wares (like Virtual Instruments), for off-box I/O routing (Xsigo comes to mind), and for simply-connected “network storage” rigs – and in many cases Direct Attached Storage. Also, give the failure potential of virtual server-based hosting, I am also seeing virtualization create a lot of mindshare for data protection software. Even the hardware independent virtualization software firms seem to be garnering revenue from the decisions of CIOs to embrace VMware and Hyper-V.

What seems to be missing in all of this is a unified architectural model, complete with open management standards, that will enable cheaper storage to be integrated with virtual server environments in a predictable and cost effective way. Initially, I think that companies will use open source operating and file system features to wed storage to virtual servers. Going forward, however, I am a huge advocate of a Web Services management framework leveraging REST.

What are your thoughts? Is there an opportunity to turn a frown upside down and to get consumers thinking about the current situation as strategic, rather than a temporary response to a temporary economic downturn?

ZEROWAIT:

As a consequence of the financial panic of 2008 and 2009 there has been a tectonic shift in customers’ perceptions of proprietary features and associated values. This shift will have deleterious effects on enterprise OEM’s pricing models. For example, Enterprise customers have recognized that open source tools such as Nagios and Cacti are excellent, reliable and affordable for monitoring enterprise systems, and Apache is the choice of many enterprises for their web servers. Due to tightened budgets the value proposition of Open Source when there is a reliable support organization backing up the OS, like Red Hat, has been recognized by Enterprises.

The new model for enterprise customers is value investing in network and storage equipment. Customers will buy as they need it, and do not want to pay outrageous licensing fees based on marketing limitations. The software management costs should not trend up by terabyte, but should trend down, this can be accomplished with Open Source, but not with proprietary Hardware and Software Products. Proprietary vendors sales models are still based in the unit sales of box and license sellers, and this is a dying branch of the evolutionary storage tree. Conversely, Zerowait’s business is based on the long term service and support business, which aligns perfectly with our customers long term ROI needs. The new reality of storage is that customers are not going to buy over provisioned arrays or pay 300% markups for hard drives that the Enterprise OEM’s charge Although the major storage OEM’s and Gartner may not realize it, customers are changing their data models based on usage. Primary storage will probably stay on high priced OEM solutions for years to come, but the other 80% of data will slowly migrate to Open Source solutions that have a reliable service and support organization behind them.

Proprietary vendors claims of interoperability often require a toxic mix of patches and software upgrades to continue to work together. Over time these, patches and band aids have gotten to the point that no two networks are running the same code and revisions and they are all unique. Interoperability was supposed to leverage the idea of interchangeable parts, but now very few components can be unplugged from one network and plugged right into another without some revisions to firmware or software. Network engineering has turned back into an art more than a science as complexity has grown. Sometimes it seems that many of our network engineering customers have more in common with alchemists than with computer theorists.

If interchangeable parts made the industrial revolution possible, why is it that computer hardware has become less interchangeable over time? Perhaps the proprietary hardware vendors have wanted to lock in customers to their high priced solutions? The financial panic has broken that model though, and the marketplace is embracing open source software and POSIX very quickly due to the cost and restrictions of proprietary hardware. You can’t fight the marketplace, you must embrace its lessons.

DRUNKENDATA:

Mike, I follow you on the bulk of what you are suggesting. My concerns are perhaps further down in the weeds, or in the clouds, depending on your perception.

First, while you may be right that simpler technology based on open source may trump purchases of overbuilt Enterprise OEM rigs in the short term, both for reasons of acquisition cost and cost of ownership, I am worried about the way that companies are/aren’t considering the bigger drivers of cost.

Storage growth is predicated upon data growth. Data growth is out of control because data itself is so poorly managed. If our assessments are correct, and literally 70% of the data being written to disk is more appropriately stored in a cheap removable-media archive (or in the trash bin), then shouldn’t firms be looking at discovery and archive tools in a much bigger way?

When firms do deploy disk, it is critical that it be managed in accordance with some sort of framework. That way, you become proactive in fault resolution and develop a knowledge of trends that can help you plan your acquisitions and utilize better the investments you make. Without effective management, you again see the hardware cost curve accelerate. Yet, again, I do not see a huge push to get a hardware management paradigm in place in the companies I visit.

Finally, I don’t see companies adequately addressing the long tail cost of storage acquisition – the cost of redundancy required to safeguard data. The redundant components of infrastructure, whether in the form of a second array, a tape library, etc., required to ensure the availability of data that is being stored on the new primary rig, are not being purchased. That RAID 5 continues to exist at all is testimony to the fact that companies are not thinking through the ineffectiveness of that scheme when applied to large capacity drives. That said, mirroring is considered by many to be a cost too dear in the current economy.

In short, even with the advent of cheaper storage and open source software, I find myself wondering whether this is ultimately strategic in the absence of effective hardware and data management…

ZEROWAIT:

You have pinpointed the problem. The proprietary vendors are driven to simplify proprietary management of their hardware, while adding complexity to management of a mixed hardware environment. For hardware vendors, who are being judged by VC’s and public stock prices, units sold is a number that makes sense to their investors who are looking for short term Return on Investment. However, customers want to get a quick ROI and also a long and affordable service duty cycle out of Capital equipment. Most Enterprise storage consumers that we work with yearn to be freed from high priced proprietary licensing schemes and single vendor lock in support programs. Organizations that embrace Open Source can reduce the lock in possibilities for proprietary hardware manufacturers. Once freed from vendor lock in schemes, consumers can and will judge hardware and software by their reliability as well as their acquisition and support costs. Open source hardware and software solutions will commoditize pricing and force vendors to compete on the true value and price of their service and support. Once freed of lock in a true competitive market can flourish, which inevitably will reduce price and improve products and services.

Since there is not an open market for storage decision making, Enterprise consumers have to interpolate their changeover risk with their cost of being locked in to a particular vendor. The lock in and cost of change is well covered by the Economist Hal Varian. Our customers recognize that there can be competition within an OEM’s market for legacy support, and often the OEM’s will reduce their prices once we have a service contract in place within a customer’s data center.

In summary, the financial panic of 2008 has opened the door to a tectonic shift in the way storage is looked at by organizations. The over hyped concept of cloud storage is being considered by many, because it is an easy first baby step toward a new storage pragmatism. Everyone that has looked into the cloud model sees dangerous cells in the towering cumulonimbus of cloud storage models. Issues of security, accessibility and speed being the first concerns people voice. I predict that 2010 will be the beginning of the end of the reign of the proprietary hardware and software vendor. 2010 should also signal the beginning of storage pragmatism and organizations will rationalize their storage costs with open source hardware and storage management solutions.

(MORE TO COME)

One OEM has been a big believer in the usage of Avatars for a few years now, as this story from 2007 highlights. The second link shows how the movie industry has embraced the idea, and may have created a product from it.

http://www.thehindubusinessline.com/ew/2007/06/25/stories/2007062550110800.htm
Network Appliance (NetApp) is a $2.8-billion storage networking firm up against the likes of EMC Corporation and Hewlett-Packard. Maintaining the culture of innovation amongst its engineers so products are creative and useful is a prime focus to reach up the market share ladder.
One way of developing better products is by tuning into clients’ data management woes, understanding requirements, and shaping products to reflect needs. However, the knowledge that after-sales and marketing folks get from the field does not always permeate to the engineers/developers.
And that is why the company created Mike Raja and Joe and four other personas. Joe could be a DBA (database administrator)/Chief Information Officer/any other user of NetApp’s data management products.
“This is a new approach that we are trying. Innovation is important and these personas for users of our products help continually innovate,” Louis H. Selincourt, Vice-President, SMAI & Offshore Operations, told eWorld over lunch at the company’s headquarters in Sunnyvale, California, recently.
The idea came from the book, “The Lunatics Are Running The Asylum” penned by Alan Cooper, whose company makes software more user-friendly.
“Personas create a consistent ID of our users, so we can discuss them across the company while brainstorming,” says Selincourt.

http://www.theregister.co.uk/2009/12/21/avatar_storage_effects/
Weta used NetApp kit to store the incoming data, then used a huge number of workstations and bladed servers – with 30,000 cores in total – to work on it. The NetApp filers were fitted with up to five 160GB DRAM cache accelerator cards in their controllers, the PAM (Performance Acceleration Modules) caches, to speed file access by the Weta creative people and the servers.

There is a certain balance to this when we recognize that the same OEM that has used Avatars for years has its equipment used in the production of a movie based on Avatars. At Zerowait, we still view our customers as individuals with specific needs. Probably there won’t ever be a movie based on our customer service and support models.