Council Post: Cloud Scale Takeaways: The Art Of Possibility
Digital luminaries from around the world descend on Davos, Switzerland, each January for the World Economic Forum annual meeting. For the past four years, it seems one of the primary purposes has been to unify around the revelation that the fourth industrial revolution has begun. Akin to the transformational value of the steam engine or Ford’s assembly line, advancements in digital technology from augmented reality, artificial intelligence and cloud are changing the world around us.
But how? Have we seen augmented reality (AR) adoption change the way we think? Has artificial intelligence (AI) changed the way we work each day? Popular culture, such as episodes of Black Mirror, helps us begin to visualize possible outcomes on how these new technologies may change our lives in the future. However, I believe cloud is unlocking new possibilities today by enabling massive-scale data analysis.
For years, I’ve heard technology experts highlight the impact of public cloud: lower costs, streamlining engineering, elastic scaling, more rapid development and so on. But I believe these almost menial benefits are far from the true value of cloud, and perhaps we are just now realizing the power of cloud scale.
Every Company Is Becoming A Data Company
As noted in the Economist, the world’s most valuable resource is data. Every company is becoming a data company, and with the scale at which we are collecting data growing exponentially, competitive advantage now lies in the ability to uncover novel ways to use that data to enhance experiences. From Netflix’s just-right movie suggestions to Uber’s algorithms that ensure enough drivers are in the right locations at the right times, we are surrounded with examples of this advantage.
Data Superpowers For The Masses
In the past, computer scientists contemplated how massive raw compute power would be used for such esoteric tasks as computing 22 trillion numbers in pi or brute-force attack encryption algorithms. But cloud superpowers are much more than raw compute. For my analytics startup, our differentiation lies in finding insights using the power of Google BigQuery, a platform for massive-scale data analysis once only available to the largest of enterprises.
Renting Massive Scale
Imagine needing to fly from NYC to Tokyo. It would be impractical for an individual to purchase a 787 for a single trip. Airlines have created the infrastructure for us to rent a single seat on that 787 for a fraction of the cost. This same hurdle presents itself in massive-scale data analysis. Imagine needing to search through a mountain of data for an answer.
At my company, our workload often requires us to search through at least a petabyte of data. To put this in perspective, a single petabyte is 100 times the entire printed collection of the U.S. Library of Congress. A massive infrastructure is required to answer any question quickly, but we typically only ask a few in-depth questions at a time. The workload is sporadic, and similar to the Tokyo flight, it’s often impractical to build the infrastructure for a single question.
What makes Google’s BigQuery transformational is that its architecture is designed to take advantage of massive shared infrastructure. It’s secret? Transparently providing instant and metered access to hundreds of thousands of virtual CPU workloads, with each workload designed to tackle a small part of the data puzzle. BigQuery can analyzepetabytes of data in seconds, and by leasing the massive-scale compute power for pennies, our way of thinking about how to solve problems is changing.
Cloud Scale Takeaways: The Art Of Possibility
The power of cloud scale is clearly effective, but beyond its sheer impressive force, it is also about the art of possibility. With this engine in our grasp, my company considered new ways to approach data and the possible insights. While there are many books and courses on big data, I don’t believe you can prescriptively take a class or course on how to think differently about data, so it will be in your hands to explore. However, here are some takeaways that we have considered with cloud in mind:
• Find data insights your team has uncovered that are meaningful but time intensive.
• Evaluate the human decision process that led to these insights.
• Identify simple data queries that can mimic these decisions, layering in complexity as you hit edge cases. The resulting analysis may involve hundreds of queries.
• Highlight data points that will influence priority in your organization. For example, for digital commerce, revenue loss is a strong motivator for teams to act on insights.
• Connect real experiences with analytics. Show examples that reinforce the data insights with actual users to gain buy-in.