It would be nice, but cloud doesn’t have to be “interoperable”
Disclosure: If you’re coming straight here you may not know work for Rackspace Hosting and I’ve been involved with OpenStack since the inception of the project. The opinions on this blog are my personal ones, not those of my employer.
This post is an assessment, a thought. I don’t really explore the meaning or outcomes completely. I may do that in a future rambling… on to the thought…
For the first decade of networking or more we had many competing technologies that didn’t interoperate: SNA, IPX/SPX, TCP/IP, AppleTalk, DECnet, NetBEUI, and more. The lack of a consistent and unified standard didn’t stop networking from succeeding anymore than it will stop cloud computing. Cloud is a fundamental shift that dramatically increases productivity just like networking did — businesses love increases in productivity and will adopt anything that yields one – often the first one presented to them and they’ll run it for a refresh cycle before switching to the “interoperable” platform. Ok, so I’m a networking geek but this isn’t the only analogy that holds true…
We have many programming languages.. compiled, interpreted, functional, object oriented.. all with major differences.
We have many types of processors from low energy mobile chips to super fast server chips all with different instruction sets.
We have a variety of operating systems all with a loyal following and a vastly different set of capabilities.
I believe cloud could see wider and more rapid adoption if interoperability is figured out but looking back at history, and even history specifically in the technology world, we have many successful markets without true interoperability as a fundamental capability.
Over time most of these markets have achieved the guise interoperability through consolidation and it looks like cloud computing is headed the same way. Networking is predominately IP; programming is C/C++/C# for OS/infrastructure, Java for enterprise applications, and PHP for web applications; processors are x86 in desktops and servers, and ARM in mobile devices; Operating Systems are generally Windows for consumer and SMB/departmental large business IT, and Linux for web and larger business core IT.
With the pace of innovation and the foundation laid down by previous generational shifts the cloud market will grow and reach a critical mass market share much more rapidly as technology companies that are involved know the path to follow. Microprocessors, operating systems, and networking took many decades. Java swept through the enterprise software development market in a decade as did PHP across the web. The cloud market really started to emerge around the start of the decade and by the current look of things by middle we’ll have a clear picture of interoperability for clouds.
Three Predictions about Cloud Computing for 2011
With all the talk in 2010 about cloud computing you’d think the entire Internet was running on it. We’re at the point now with cloud computing as we were in the late ’80s through the mid-late ’90s with networking. Everyone can clearly see the benefits in cloud but the market is hyper fragmented as different pockets of users form a community around one of the available solutions. To ensure new readers are aware, while I am employed by Rackspace Hosting working on the OpenStack project, the opinions expressed on the blog are mine and I try to present a non-biased view of the market.
With that opening I’ll dive into the three items of importance to cloud computing for the coming year..
..with additional more minor predictions in italics.
1. Cloud computing needs will change as we move from early adopters to mainstream users
Thus far the primary users of cloud infrastructure as a service (IaaS) offerings have been early adopter technology savvy users. Those users may be founders of a Web 2.0 startup, consultants working for the R&D department of a system integrator, or forward thinking IT professionals on enterprise IT strategy teams. Based on the stats in the chart, we still have less than 2% of the top 500k websites hosted on IaaS. In 2011 this number will grow to 5-10% of the top 500k sites, more than doubling again like it did in 2010.
Source: http://www.jackofallclouds.com/ - Guy Rosen
Inside startup communities everyone will be using the cloud to start their business and in enterprises departmental innovation success stories will start to bubble up to corporate leadership. This doesn’t mean existing applications will be migrated — people will experiment with migration as disaster recovery innovation but it won’t be a major driver of cloud growth in 2011.
The major driver of growth will be new applications and much of this growth won’t be consumer Internet sites that are easy to track making the “who’s winning in cloud” leader board more difficult to display. This next wave of adopters will also require additional levels of support as they won’t have the same “DIY” mentality as early adopters. Cloud providers will need to raise their own service levels or spend significant effort building a system integrator and consulting ecosystem that can provide it for them.
The ecosystem of tools built on the IaaS cloud APIs will be a foundation to enable the higher service levels. They will be utilized by the practices for the SIs and consultants as well as the software development teams of many ISVs. For cloud providers that do not yet have an ecosystem built around their API this will be the year they move on to adopting one of the open APIs with market traction. Enough providers will use a group of 3-5 APIs that ISVs and startups will refuse to developer for others. API abstractions like jclouds, Libcloud, and Deltacloud will start to add depth rather than additional breadth.
2. Technology heavyweights with large developer communities will escalate their efforts to define and control PaaS
2006 brought us Rackspace Cloud Sites (originally branded as Mosso)…
2007 gave us Force.com…
2008 launched Microsoft Windows Azure and Google App Engine…
2009 delivered Heroku’s commercial release and moved VMware into the platform space with the acquisition of SpringSource…
2010 veterans like Red Hat and Oracle [PDF] start announcing platform strategies and making acquisitions such as the recent purchase of Makara (by Red Hat)…
Over the past few years many platforms and application frameworks simplified development by providing a foundation and abstracting away lower level details. This came with some drawbacks as most frameworks were not aware of their resource utilization nor did they have the ability to utilize the programmatic capabilities of IaaS to change their resource allocation based on load. In 2011 many platform solutions will become tightly integrated with IaaS APIs providing dynamic resource management — the auto-scaling cloud workload across public, community, and private cloud installations will become an early adopter reality.
Building a PaaS solution is possible for a startup if they make it compatible with widely accepted development languages and frameworks as Mosso did by selecting PHP/Python/.NET with support for common applications like WordPress, Drupal, Django, and more. Another example is what Heroku did with Ruby and PostgreSQL. The heavyweights are the only players with deep enough pockets and the required patience to push new programming dynamics. Microsoft will look at how things are evolving and in order to defend .NET/C# will embrace the Mono Project creating a real threat to Java in the enterprise.
Java won’t take things laying down. Despite the fallout in 2010 over Oracle vs. Google, over the Apache Foundation first joking with Oracle and then stepping down from the JCP Executive Committee, and the many other examples such as James Gosling, the creator of Java, coming out against its new steward Java still maintains #1 position on the TIOBE index. Oracle, IBM, and VMware all have deep pockets and big revenue streams tied to the continued success of the language. IBM, while late to the PaaS party, will tie together Tivoli, WAS, and other components to build a robust platform for their customer base.
The blog-o-sphere will erupt in debate about “what is a true platform as a service” as much as they went on and on in 2010 about infrastructure as a service APIs. Despite what the pundits believe the majority of the enterprise IT dollars will go towards “false platform” private cloud solutions. Making the leap for major projects from current development methodologies and procedures to one of the new platforms will be too much — IT organizations need evolution, not revolution.
3. Enterprises will begin to evolve their virtualization deployments into private clouds and they’ll expect networking and audit controls beyond the capabilities of many current systems
Cloud deployments in enterprises will go in two different directions depending on the expectations of the project sponsoring executive team. Departmental usage of cloud that flies under the official process RADAR is not what I’m talking about. Over the past year I’ve had conversations with numerous people on Fortune 500 IT strategy teams and cloud is being looked at a couple of different ways. One group looks at it only as a technology solution that will magically make their operations more efficient. Another group realizes that the automation cloud brings only benefits them if process changes happen in parallel with with the systems improvements. Enterprise cloud projects that are only technology focused will not provide any meaningful savings and enterprises going this route will become disenfranchised. Virtualization was about consolidation not automation and because of that it didn’t include business process changes that cloud requires. You can’t simply install a “cloud upgrade” to your virtualization system and instantly have a cloud.
The cloud projects are going to run into a second set of hurdles. Up to this point typically departmental non-audited, non-regulated applications have been deployed by enterprises on clouds. In 2011 projects will need to address the corporate risk management and IT audit requirements. Major public clouds such as Amazon Web Services and Rackspace have addressed and received various attestations such as SAS70 Type II (Rackspace example), ISO27001 (AWS example), and PCI DSS [PDF] (Visa Global List of Validated Service Providers) showing that it is possible to build cloud services that meet compliance requirements. For enterprise projects to be successful they need to involve risk and audit up front so the proper control mechanisms are in the deployment. Because cloud is about workflow automation having to insert a manual audit control late in the project in order to meet the launch plans will eliminate many, if not all, of the projected benefits.
Corporate risk and IT audit teams will invalidate a number of cloud software fabrics and those platforms will quickly try to re-engage on projects by announcing partnerships with security companies. Platforms that come from service provider and government backgrounds such as OpenStack have a head start along with platforms that have evolved from enterprise DNA such as VMware. As enterprises start to spend significant dollars on cloud the R&D investment in cloud platforms will dwarf what has been spent to date and any head start present currently can see it easily vanish in 2011.
Audit controls for cloud platforms need to include both host and network services. It is possible to architect a cloud and map the controls into existing systems though this won’t be instantly turn-key or easy in 2011. Network virtualization will make cloud systems more flexible at a cost of making compliance controls more complicated. Process wise this also introduces another department into the cloud deployment discussions. Most clouds deployed in 2011 will focus on server automation and networking will be addressed in subsequent phases of the technology transition in 2012+.
Conclusion
2011 will be another big year for the adoption of cloud technology but these fundamental shifts happen slowly — especially when they involve people learning new processes and not just transparent technology replacement. When done right, cloud will make IT vastly more efficient and when cost of services decline the demand for those services often skyrocket.
This post focused on the cloud computing. I’ll be making other posts in January about distributed storage platforms (aka. “cloud storage”) and why they’ll be important for enterprises to understand and have readily available to their users before the middle of the decade. It is a fundamentally different problem as many cloud storage systems can be installed transparently without end user process changes.
How to tell the difference between “cloud” and “virtualization”
Many people seem to think “cloud” is just off-premise “virtualization”. Cloud comes in a few flavors and I’ll argue that you can have “private cloud” either hosted off-premise in a provider’s facility or in your own. The fundamental difference between cloud and virtualization is the goal of cloud is to automate provisioning (this applies to IaaS, PaaS, and SaaS) and the goal of virtualization is resource utilization optimization. You can (and many providers do) use virtualization as the basis for building a cloud but it is not required.
If we take a look at the Reductive Labs presentation from OpsCamp slide 3 illustrates the primary benefit of cloud. Cloud helps companies even if their minimum unit of work is larger than a single host machine where virtualization just adds overhead in that case. The difference between “cloud” and “grid computing” or HPC is that grid/HPC process jobs in a batch manner rather than serve interactive applications. You can build a compute grid on top of a cloud but not vice versa.
Other folks are saying “private clouds can’t exist because you can’t have rapid elasticity and pay for what you use”. For a small company you may not be able to have a private cloud but for a large enterprise with many business units you certainly can. An IT infrastructure BU can provide other organizations in the company all of the requirements of a cloud.
For public cloud to succeed they need to provide all three
Depending on the current utilization across an enterprises infrastructure they may be able to defer spending for a number of years by moving to a fully cloud enabled business. Right now many departments cling to servers they don’t need because they’re afraid if they release it they’ll never get it back. With cloud removing that fear resource hoarding ends and many enterprises will have a significant increase in available computing power.
Over the long term if the public computing clouds continue to grow, increase their transparency, and optimize their delivery models it will no longer make financial sense for enterprises to build their own infrastructure. Public cloud providers will need to prove over the next decade they can deliver on all three corners of the “impossible triangle”.
Unexpected use cases, what a Service Provider never sees coming…
Service Providers launch offerings with a particular use case or set of use cases in mind. Flickr launched a photo/video sharing site, Joyent offers Accelerators to host websites, and Google/Yahoo/Hotmail/etc. offer free e-mail. All of these services have other ways they could and probably are being used….ways the product teams never expected.
Plenty of on-line backup services exist, Jungle Disk (disclosure, they’re owned by my employer), Carbonite, Mozy, AT&T Remote Vault, and more. These services are all priced around $50+/year after you have a resonable amount of data. This is where Flickr enters the picture in our unexpected use cases…by using steganography I can have 50GB of free backup and after that it only costs me $24.95/year for unlimited.
Why are free e-mail services interesting? They offer free storage and bandwidth again expecting you to use it for e-mail. Many of their terms of service don’t prohibit account sharing. If you are a software company looking for a low cost way to distribute your newest release get company@<freemail>.com and upload it there. Setup the auto-updater in your software to check for a new version in the e-mail and if it is there download and update.
For general web hosting providers offer different package offerings with CPU/Disk/Bandwidth under the expectation you’ll host your entire site with them. To differentiate some offer extreme amounts in one of the categories to attract customers. One example of this is Joyent Accelerators with 10TB of transfer for $45/month (or $199/year prepaid). They figure you can’t use 10TB of transfer with 5GB of storage. I can host all of my images for an advertising campaign at Joyent and store the DB and rest of the site elsewhere. Off AWS 10TB of download transfer would cost you $1,700/month but CPU/Disk are much less expensive than they are from Joyent (my employer is Rackspace where our cloud offerings are priced much more like AWS).
This extends beyond just technical services as well. The best example of this is the “all you can eat buffet”. They pick a price based on average consumption and charge everyone the same. This works better in the buffet world than it does in the information services world as people generally eat in groups of friends or family. Information services however we can consume individually thus allowing use case versus price arbitrage much more efficiently.
Service cost arbitrage exists all over, the reason we have it, and the reason more people don’t take advantage of it is many of them aren’t efficient to utilize. If I release a steganographic Flickr backup client they’d change their terms of service. If all software companies started using free e-mail services for distribution, they’d change the terms of service. Even the “all you can eat buffet” would change the terms of service. When I used to wrestle in high school our team of 40+ would go to buffets after our tournaments and we’d frequently see a near-realtime change in the terms of service — i.e. we’d get thrown out of the buffet after we ran them out of food.
When developing a service offering the best way to meet the expectations of all of your customers is to price each component in a fair manner. Consider running promotions or specials with limited terms if you’re trying ways to attract new customers rather than setting artificially low prices in a particular category believing people will use the expected use case.
