如何利用端到端数据可观察性改进数据工程工作流

随着企业越来越多地利用数据驱动的见解来推动创新和保持竞争优势, it’s important that their data is accurate and trustworthy. With data observability, data teams can now identify and prevent inaccurate, missing, or erroneous data from breaking your analytics dashboards, delivering more reliable insights.

Does this situation ring a bell?

您的市场分析团队使用Looker将纽约市时代广场的新广告牌每天产生的合格销售线索可视化. 在与CEO进行第四季度计划会议前几分钟,营销副总裁提醒你:

“The data is all wrong… what happened?!”

You open Looker and realize the numbers, which are normally updated every 15 minutes, haven’t been touched in 24 hours!

而强大的解决方案可以在现有数据的基础上提供数据分析, 许多数据团队都将数据质量和完整性视为关键问题, costing them millions of dollars in wasted revenue and up to 50 percent of their team’s time. 而不是做那些对公司有帮助的项目, 数据专业人员被迫调试数据管道和被破坏的仪表盘.

To address this all-too-common reality, data analysts, engineers, 科学家们需要一种简单而合作的方式来监测和提醒他们数据中的异常, from ingestion to analytics. 同样重要的是,他们有必要的工具来映射他们的数据谱系, 密切关注数据管道中的变化如何影响下游源, such as analytics and business intelligence reports.

Image for post
Data Observability仪表板监视数据管道和业务智能仪表板中的数据异常. Image courtesy of Barr Moses.

How data workflows break

Bad data spares no one, and can crop up in a variety of ways, causing sleepless nights, wasted resources, and erosion of data trust.

在过去的12个月里,我与数百个数据工程团队进行了交谈, 我注意到,好数据变坏有三个主要原因:1)单一数据生态系统中越来越多的数据源, 2) the increasing complexity of data pipelines, and 3) bigger, more specialized data teams.

More and more data sources

Nowadays, 公司使用数十到数百个内部和外部数据源来生成分析和ML模型. 这些来源中的任何一个都可能以意想不到的方式发生变化,并且不需要通知, compromising the data the company uses to make decisions. 

For example, 工程团队可能会对公司的网站进行更改, 因此,修改数据集的输出是市场分析的关键. As a result, key marketing metrics may be wrong, 导致公司在广告宣传方面做出糟糕的决定, sales targets, and other important, revenue-driving projects.

Increasingly complex data pipelines

数据管道越来越复杂,有多个处理阶段和各种数据资产之间的重要依赖关系. With little visibility into these dependencies, 对一个数据集进行的任何更改都可能产生意想不到的后果,影响相关数据资产的正确性. 

一个系统中简单的单位改变可能会严重影响另一个系统的正确性, as in the case of the Mars Climate Orbiter. A NASA space probe, 火星气候轨道飞行器坠毁的原因是数据输入错误,产生的输出是非单位制单位与单位制单位, bringing it too close to the planet. Like spacecraft, 在过程的任何阶段,分析管道都很容易受到最无害的变化的影响.

Bigger, more specialized data teams

随着企业越来越依赖数据来推动明智的决策, they are hiring more and more data analysts, scientists, and engineers to build and maintain the data pipelines, analytics, and ML models that power their services and products, as well as their business operations.

Miscommunication or insufficient coordination is inevitable,并会导致这些复杂的系统在做出改变时崩溃. For example, 一个团队在数据表中添加的新字段可能会导致另一个团队的管道失败, resulting in missing or partial data. Downstream, this bad data can lead to millions of dollars in lost revenue, erosion of customer trust, and even compliance risk.

The solution to broken data workflows? Data observability.

Ensuring reliable insights with data observability

数据团队需要一种方法来无缝地监控并提醒仪表板上的数据问题, 让他们全面了解其数据资产的健康和可靠性.

To tackle this, 数据可观察性自动监视数据生态系统的关键特性, including data freshness, distribution, volume, schema, and lineage. 数据可观察性无需手动设置阈值,可回答如下问题:

  • When was my table last updated?
  • Is my data within an accepted range?
  • Is my data complete? Did 2,000 rows suddenly turn into 50?
  • 谁有权使用推荐一个正规滚球网站的营销表格并对其进行修改?
  • Where did my data break? Which tables or reports were affected?

With the right approach to data observability, 数据团队可以跨整个数据工作流跟踪字段级沿袭, 促进对其数据的健康状况和这些管道提供的见解的更大的可见性. Such functionality allows data engineers, analysts, 科学家们要找出为什么他们的仪表盘不能为利益相关者提供最新的数据.e., is there a missing data set? A null value? Did someone use the CSV file type instead of XLS?).

How Compass prevents broken workflows with Data Observability

数据可观察性在理论上是很好的,但是在实践中它是如何工作的呢?

As a digital-first real estate platform, Compass 利用数据驱动的技术,为全球代理商和房地产买家提供智能和无缝的搜索和销售体验. Suvayan Roy, Senior Product Manager at Compass, 监督数据团队的工作流程,负责建立和维护整个13个分析管道,000-person organization. Keeping tabs on the upstream and downstream dependencies for their Looker dashboards is top-of-mind for Suvayan; if data breaks, his team needs to be the first to know and solve.

“我向Slack提供的数据可观察性监控反馈让我感到安慰,推荐一个正规滚球网站的数据是健康的,一切都按照设计进行. And on days where something goes wrong, 我知道我的团队将是第一个知道的,推荐一个正规滚球网站将控制局势,” Suvayan said.

With automated data observability, 罗伊和他的团队可以在晚上睡个好觉,继续专注于开发产品, meeting customer needs, 在不需要担心数据和分析的情况下彻底革新房地产. 

Compass’ story is just the beginning. 就我个人而言,我对这个类别的未来感到兴奋.

Reliable workflows, here we come!

Want to learn more about how data observability can help your data team? Reach out to Monte Carlo and check out the Data Downtime Blog.