A History of Silicon Valley

Table of Contents | Timeline of Silicon Valley | A photographic tour
History pages | Editor | Correspondence
Purchase the book

These are excerpts from Piero Scaruffi's book
"A History of Silicon Valley"

(Copyright © 2016 Piero Scaruffi)

The Selfies (2011-16)

click here for the other sections of this chapter

Beyond the Cloud

The biggest outside threat to Google and Apple was probably Amazon. On the surface the Seattle-based rival was simply a huge retailer (in 2014 carrying more than 200 million items sold by over two million third-party vendors), and mainly famous (or infamous) for destroying the business of bookstores, but deeper down Amazon was an extremely sophisticated technology company that pioneered online customer reviews and ratings, that in 2006 introduced pay-as-you-go cloud computing with S3 (later used by the likes of Instagram, Pinterest and Spotify), that in 2007 turned ebook readers into commodities, that in the 2010s turned its colossal PHX6 fulfilment center at Phoenix (Arizona) into an automated, robot-intensive facility (while making the whole supply chain available to other merchants as well), and that was rapidly moving online business towards same-day delivery and mobile shopping.

Introduced in 2008 and based on open-source Xen technology, Amazon's Elastic Compute Cloud (EC2) provided a self-service portal for developers to rent their own virtual private servers on which to run their own applications.

Not many dared to compete with Amazon S3, the dominant on-demand cloud storage. However, Minio, founded in 2014 in Palo Alto, unveiled an open-source cloud storage for storing unstructured data such as photos and videos.

Google launched its rather simple Cloud Storage service in 2010, and then proceeded to beef it up with a series of acquisitions: in 2014 it acquired Boston-based Stackdriver, launched in 2012 and already the monitoring platform for Amazon Web Services, as well as San Francisco-based Firebase, founded in 2011 by Andrew Lee and James Tamplin; in 2015 it acquired Bebop, founded in 2012 in San Francisco by Diane Green (of VMware fame); in 2016 it acquired Apigee, founded in 2004 in Santa Clara as Sonoa Systems by Raj Singh and Ravi Chandra; and in 2019 it acquired Looker, co-founded in 2012 in Santa Cruz by Lloyd Tabb (a Netscape and Borland veteran), a provider of business intelligence, as well as the Israeli-based Elastifile, a provider of enterprise file storage for the cloud.

New methods for software development made it easier for startups to create distributed applications.

Containers were a legacy of the Unix operating system (SUN's Solaris edition of 2004, an evolution of Bill Joy's "chroot" command incorporated into Berkeley Unix in 1982). An application run within a container is isolated from the hardware platform. If it sounds like virtualization, that's because they are closely related. Containers are a faster way to implementat virtual machines and a better way to optimize hardware resources. The net result for the software developer is the same: application portability across platforms. In 2008 Solomon Hykes and Sebastien Pahl founded DotCloud (later Decker) in San Francisco to provide an open-source platform based on containers that quickly built an entire community of developers. Google then legitimized the field when it introduced Kubernetes to manage clusters of containers.

The Nebula project, started in 2008 by Ray O'Brien at NASA Ames in Mountain View, evolved in 2010 into an open-source cloud platform, OpenStack, technically a joint venture with Texas-based cloud service Rackspace (a 1998 offshoot of San Antonio's Trinity University). Rackspace donated the software behind OpenStack's Swift storage service, and in 2011 acquired Anso Labs, the San Francisco-based company that maintained Nasa's Nebula cloud (and that developed the Nova computer). In 2012 some of Anso Labs' original team (including former NASA Ames scientists Chris Kemp and Devin Carlen) left RackSpace and founded Nebula in Mountain View. By 2015 more than 500 companies had joined OpenStack, basically an open-source alternative to Amazon's and VMware's cloud services.

The new generation of middleware was represented by platforms like Anypoint, introduced in 2013 by MuleSoft, a company started in 2006 in San Francisco by Ross Mason and Dave Rosenberg. It connected and integrated software as a service (SaaS) on the cloud with legacy databases and applications running on personal devices: "any application, any data source, any device, any API".

In 2015 Cisco acquired cloud-security provider OpenDNS, founded in 2006 by David Ulevitch in San Francisco (who in 2001 in Saint Louis had launched the free DNS management service EveryDNS).

Jyoti Bansal founded AppDynamics (in 2008 in San Francisco) to improve performance of cloud-based applications, which were typically heavily distributed (the startup would become a unicorn in 2015).

HashiCorp, founded in San Francisco in 2012 by Mitchell Hashimoto and Armon Dadgar, offered a cloud infrastructure for datacenters.

Neo Technologies, founded in 2007 in Malmo by Emil Eifrem and Johan Svensson but relocated to San Mateo, commercialized a database originally developed for the Swedish military, Neo4j. First released in 2010, within five years it became the most popular graph database in the world.

Storage had evolved from the era of mainframe (that owned all the storage) to the network of servers with direct-attached storage, and then to the sharable pool of storage, especially after virtualization enabled multiple operating systems to run as virtual machines on a single machine. This had enabled cloud computing. In the 2010s the big IT providers started moving towards "converged infrastructure" in which IT infrastructure was pooled and shared, thus simplifying hardware and software management. VMware, the pioneer of data center virtualization, now part of Dell, had elected vSphere (2009) as its platform for cloud computing, and in 2014 built vSAN (virtual storage area network) on top of it for converged infrastructure. Acadia, the joint venture of EMC, Cisco, and VMware formed in 2009 (later renamed VCE for "virtual computing environment"), introduced vBlocks, i.e. basically VMware's vSphere software running on Cisco's datacenter servers UCS attached to EMC's storage.

Then came the idea of the software-centric datacenter, or the "hyper-converged infrastructure", originally promoted by Nutanix with its Complete Cluster (2011, later renamed Virtual Computing Platform). Here the software-defined storage comprises all the local storage across the cluster and presents it as a single storage pool. Founded in 2009 in San Jose by Dheeraj Pandey, Mohit Aron and Ajeet Singh to provide enterprise cloud computing, Nutanix by 2013 was already a unicorn. SimpliVity (acquired by Hewlett Packard in 2017) spread the concept to Boston with its OmniCube (2013). Maxta, founded in 2009 in Sunnyvale by Yoram Novick, introduced MaxDeploy in 2014. This one supported both VMware's vSAN and Red Hat Virtualization, the latter introduced in 2010 by Red Hat and based on Linux's Kernel-based Virtual Machine or KVM (developed in 2007 by the Israeli startup Qumranet, acquired by Red Hat in 2008). Then came VMware's EVO (2014), Nimboxx (2014) in Texas, and EMC's Vspex (2015, based on EVO). Confusingly, in 2016 Dell (that owned both EMC and VMware) introduced a turnkey solution based on VMware's vSAN called EMC VxRail.

A new trend appeared in the mid 2010s towards "server-less" cloud computing ("functions as a service"), in which the cloud automatically figures out the server setup needed by an application. Serverless, founded in 2015 in San Francisco by Austen Collins, was one of the market pioneers.

In 2019 Bay Area firms, trying to compete with the likes of Microsoft's Power BI and Amazon' QuickSight, launched multi-billion dollar deals in data analytics: Salesforce acquired Seattle-based Tableau Software, and Google acquired Looker (Google's biggest acquisition since Nest), which had been founded in 2012 in Santa Cruz by Netflix's cofounder Marc Randolph and by Lloyd Tabb (whose claim to fame was to have written in 1994 a pioneering Web application server at Commerce Tools as well as Netscape's first web-page editor, Composer, a few years later).

By 2019 Amazon was making two-thirds of its profit from its cloud-computing platform Amazon Web Services. In 2018 Microsoft spent a fortune ($7.5 billion) to acquire GitHub and then in 2019 it spent $1 billion to acquire OpenAI. That money bought Microsoft a top-rated AI software that was mostly nonprofit but ready for commercialization on its Azure platform.

Notice that peer-to-peer, revitalized by the blockchain mechanism, was competing with cloud computing to introduce collectivization in cyberspace. The two models were wildly different, but both were removing the data from the individual's home computer and relocating the data somewhere else: either on a distributed network of participants or on a distributed network of corporate computers. Peer-to-peer computing had generated a lot of philosophical, sociological and political discussion by the independents who had created it and were using it. Cloud computing was less naturally analyzed and discussed, because it mostly belonged to governments and corporations, its "architects" mostly unknown.

click here for the other sections of the chapter "The Selfies (2011-16)"
(Copyright © 2016 Piero Scaruffi)

Table of Contents | Timeline of Silicon Valley | A photographic tour | History pages | Editor | Correspondence