5 Non-Obvious Things to Consider When Building Your Data Platform

Here are 5 important questions to answer when migrating to the self-service data platform of your dreams.

搭建数据平台—i.e., a central repository for all your company’s data, which enables the acquisition, 存储, 交付, and governance of that data while maintaining security across the data lifecycle—has become a rite of passage for today’s data teams. Data platforms are critical in a modern organization because they help optimize operations by allowing leaders to glean actionable insights from data more easily. 

In this brave new data-driven world, 虽然, ensuring you’re getting started on the right foot as you build your data platform can be challenging. Some considerations as you build are obvious: the tools you’ll need, 例如, 你要服务的用户, the data sources you’ll leverage, and the ultimate uses of the platform. But there are several organizational and cultural considerations that even the best-intended data teams might overlook in their eagerness to build. 

We sat down with Noah Abramson and Angie Delatorre from Toast, a leading point of sale provider for restaurants and a recent unicorn, to learn about their approach to building a high-functioning, 现代数据平台. 

推荐一个正规滚球网站的谈话, we learned that there are five important considerations when building your data platform. 就让推荐一个正规滚球网站一探究竟吧.

Consideration 1: How will you gain stakeholder buy-in?

A data platform is only helpful if its users—i.e., stakeholders across the business—are open to and familiar with it. Before creating a data platform, it’s critical to get all the teams that might take advantage of the platform on board. 

At Toast, this was blessedly simple.

“Toast is a super data-driven company,”诺亚说. “I think that’s been really beneficial for our [data] team, to be honest. Our data engineering team director has done a really nice job positioning us to be as valuable as possible for the rest of the company, and enabling people with those types of data insights to then have support to make decisions, then measure those decisions and outcomes.”

Employees in every division 跨组织的 should understand how the data platform will ultimately provide value to 他们. That’s the initial job of the data team: to explain and showcase that value, and to establish a method of measuring success even as the company scales. The Toast data team began by understanding the business problems affecting their colleagues, then positioned the data team as purveyors of potential solutions.

“Our director gets our team involved early into these problems and helps us understand how we’re going to solve it, how we’re going to measure that we’re solving it correctly, and how we’re going to start to track that data—not just now but also in the future,”诺亚说. 

随着时间的推移, Toast developed a system that removed bottlenecks by building a self-serve analytics model to service the broader company and remove bottlenecks. 

“[Our process was originally] super centralized, and we owned the entire stack,” explains Noah. “As the company started to grow, it got overwhelming. We pivoted to a self-service model, and we became the team that you would consult with as you were building these dashboards and owning the data.”

Consideration 2: Who owns what in the data stack?

To be used most effectively, data should be viewed as a shared resource 跨组织的. Various teams take ownership of the company’s data at various points in its lifecycle: the data engineering team may own the raw data, 例如, before they hand it off to the 分析工程 team for analysis and insights, which can then be parsed and applied by the business intelligence team. 

The end-to-end data stack comprises multiple tools and technologies that support each of these teams. Toast derives data from sources including Salesforce, NetSuite, Workday, and Toast itself. 这些数据流入S3, 团队的数据湖, which is then copied into Snowflake, 云数据仓库. The team uses Looker as its front-end tool, and all jobs are orchestrated via Airflow.

烤面包的数据堆栈. 图片由Toast提供.

At Toast, the data platform team owns the company’s external-facing data insights and analytics. 

“One of our big value ads [as an organization] is giving business insights to our customers: restaurants,”诺亚说. “How did they do over time? How much were their sales yesterday? Who is their top customer? It’s the data platform team’s job to engage with our restaurant customers.”

Noah’s team, in contrast, is largely internal. 

“We say our customers are all Toast employees,” he says. “We try to enable all of 他们 with as much data as possible. Our team services all internal data requests from product to go-to-market to customer support to hardware operations.” It’s thus Noah’s team’s job to build out data flows into overarching systems and help stakeholders 跨组织的 derive insights from tools including Snowflake and Looker.

Consideration 3: How will you measure success?

If you can’t measure, you can’t manage—and that truism applies when it comes to assessing the impact of data on the business. When building a data platform, it’s important both to measure how stakeholders can leverage data to support business needs and to ascertain the quality and efficiency of the data team’s performance.

“We really listen to what the business needs,”诺亚说 regarding how his team thinks about measuring data-related KPIs. “At the top level, they come up with a couple different objectives to hit on: e.g., growing customers, growing revenue, cutting costs in some spend area.” 

Noah and his team then take these high-level 业务目标 and use 他们 to build out Objectives and Key Results (OKRs).

“We’re able to do that in a few ways,”诺亚说. “If you think about growing the customer base, 例如, 推荐一个正规滚球网站问, ‘how do we enable people with data to make more decisions?’ If somebody has a new product idea, how do we play with that and let 他们 put it out there and then measure it?”

Additionally, the team focuses on measuring the scalability of its processes. “Not only do we listen to the business needs and obviously support 他们, but we also look internally and address scalability,”安吉说. “If a job used to take one hour, and now it takes three hours, we always need to go back and look at those instances, so that shapes our OKRs as well.”

Consideration 4: Will you centralize or decentralize your org 结构?

Every data team is different, and each team’s needs will change over time. Should your company pursue a centralized organizational 结构 for your data team? Will centralization impose too many bottlenecks? Will a decentralized approach lead to duplication and complexity? Understanding what each option looks like—and choosing the model that’s best for your business at a given point in time—is an important consideration as you build your data platform. 

Over the years, Toast’s own 结构 has run the gamut: from centralized to decentralized to hybrid. 最初, all requests flowed through the data team. “And that worked,”诺亚说. “But as the company started to grow, it got overwhelming.”

Toast pivoted to a decentralized model that emphasized self-service for analysts and positioned the data team in a more consultative role. This change arose due to several vestiges of rapid growth, including a huge volume of incoming data, an increased number of people relying on that data, and the limited resources of the growing data engineering team.

As Toast grew, the company knew it would need to leverage data in new ways to support growth. 

“We needed to think about how to enable all of these new business lines that were being spun up to have the same level of insights that our go-to-market team had,”诺亚说. “We have this high expectation of data usage, and we wanted to enable all the people [跨组织的] to have that same access that we’ve built out for go-to-market and sales.” 

That led to questions of prioritization for a stretched-thin data engineering team with limited bandwidth. “Our most current evolution of our team really addresses those types of [business] needs,”诺亚说. “在推荐一个正规滚球网站当前的设置中, a super new paradigm that we’re living in is data engineering, 分析工程, and then visualizations on top of it. We are now owning efficiently getting data into Snowflake and with all of our many different sources.” 

反过来, the 分析工程 team is focused on creating a data model that can service the various ways different Toast constituents think about data and the business. 

“我认为是新的 分析工程 team—and we’re really excited about it—is going to zoom out a little bit and understand all the problems and see how we can build a data model to service that,”诺亚说. Toast’s current hybrid organizational 结构 works best for its current needs—and the team is always willing to reevaluate and pivot should its needs and circumstances change.

Consideration 5: How will you tackle data reliability and trust? 

As volumes of data continue to increase in lockstep with the willingness of various business units to leverage 他们, 数据的可靠性, an organization’s ability to deliver high data availability and health throughout the entire data lifecycle—becomes increasingly important. Whether you choose to build your own data reliability tool or buy one, it will become an essential part of a functional data platform.

As the Toast team began working with ever-growing volumes of data, ensuring data reliability became mission-critical. “There’s a lot of moving parts,”诺亚说 of the data stack. “There’s a lot of logic in the staging areas and lots of things that happen. So that kind of begs the question, how do we observe all of this data? And how do we make sure when data gets to production and Looker that it is what we want it to be, 这是准确的, 它是及时的, and all of those fun things that we actually care about?”

最初, Noah and two other engineers spent a day building a data freshness tool they called Breadbox. The tool could conduct basic data observability tasks including storing raw counts, 存储null百分比, ensuring data would land in the data lake when expected, 和更多的.

 “That was really cool,” notes Noah, “but as our data grew, we didn’t keep up. As all of these new sources came in and demanded a different type of observation, we were spending time building the integration into the tool and not as much time in building out the new test for that tool.”

Once the team reached that pivotal level of growth, it was time to consider purchasing a data observability platform rather than pouring time and resources into perfecting its own. 

“With 蒙特卡罗, we got the thing up and running within a few hours and then let it go,”诺亚说. “We’ve been comparing our custom tool to 蒙特卡罗 in this implementation process. We didn’t write any code. We didn’t do anything except click a few buttons. And it’s giving us insights that we spent time writing and building  or maintaining.” 

最终, it was a no-brainer for Toast to purchase the tool it needed while allowing the data engineering team to focus on adding value to the business. 

When it comes to building or buying, Angie says, the right choice will differ for any given company. 

“I think that our build worked very well for what it was,” she says. “But when you talk about scalability, we’re growing so rapidly and onboarding people, and it’s going to be tough to teach 他们 how to use this tool. And if something breaks, it’s tough to [fix].”

In the long run, she says, “I want our team focusing on enriching data and enabling Toast. And if there is a best-in-class tools that does this, we are very willing to pay for that and enable our expertise to do what we do best and enable business users to do their thing with the data.”

Since implementing 蒙特卡罗, Toast’s data engineering team can do just that. 更重要的是, 蒙特卡罗 offers additional value to the company beyond what a bespoke tool could provide.

 “There are a few things [about 蒙特卡罗] that I think are extremely valuable,”安吉说. “Machine learning, being able to go in and say ‘this is expected,’ or ‘this is fixed.’ Another thing is the record of the incidents and how they happened. I think it’s extremely valuable to be able to go back and have that record.” 

蒙特卡罗’s simplicity also helps the team avoid alert fatigue, she suggests. 

“It just cuts down on time,” she says. “You’re directed exactly toward what the problem could be, and from there, you can expand.” 

The best data platform for your team

Each data platform at each company will look a little different—and it should. When it comes to creating the best data platform for your team and organization, it’s important to ask yourself nuanced questions about your company’s culture, 业务目标, 结构, 和更多的. 

Interested in learning more about how to build reliable data platforms? Reach out to Alex and the 可以玩滚球的正规app队!