解密数据可观测性

As companies ingest more 和 more data 和 data pipelines become increasingly complex, 出错的机会越来越大, manifesting in everything from a broken dashboard to a null value.

11月12日,星期四,12点.m. 美国东部时间/ 9.m. PST, we’ll be discussing how to solve this problem during 数据可观察性网络研讨会 with the Data Engineering 和 Business Intelligence Leads at Yotpo,一个电子商务营销平台.

在推荐一个正规滚球网站的 谈话, here are 3 tell-tale signs your data engineering team could benefit from 数据可观测性是现代数据栈的最新层.

A customer recently asked me: “how do I know if I can trust my data?”

When I was the VP of Customer Success at Gainsight, this question came up a lot. 我工作过的每一个数据组织都是不同的, 服务水平协议(sla), 安全需求, 和 kpi “准确、高质量的数据”是什么样子的. 在所有这些数据团队之间, 然而, a common theme emerged: the need for a better approach to monitoring the reliability of data 和 eliminating 数据停机时间.

Data downtime refers to periods of time when data is missing, 不准确的, 或其他错误, 和 it can range from a few missing values in a data table to a misstep in the data transformation process. As companies increasingly leverage more 和 more data sources 和 build increasingly complex data pipelines, 数据停机的可能性只会增加.

我记得我对自己说: Wouldn’t it be great if there was a way for data teams to monitor 和 alert for these incongruities the same way that software engineers can track application downtime through observability solutions like New Relic 和 Data Dog?

五年前, there wasn’t a vocabulary — or an approach — holistic enough to address this evolving need. 2020年,推荐一个正规滚球网站终于实现了:世界,相遇 数据可观测性.

In 前面的文章, I’ve discussed why 数据可观测性 is foundational to trusting your data (in other words, 数据的可靠性), 但是在引擎盖下面是什么样子的呢?

Here are 3 tell-tale signs your data team should invest in 数据可观测性:

1. Someone changes a field upstream, resulting in missing or partial data downstream

An end-to-end Data Reliability Platform allows teams to explore 和 underst和 their data lineage, automatically mapping upstream 和 downstream dependencies, 以及这些资产的健康状况. 图片由巴尔摩西提供.
情况

满足斯蒂芬妮. 她是一名数据科学家. Stephanie is responsible for modeling a data set about the success of her company’s dem和-gen marketing campaigns. 有一天,, 肯, 广告团队的一员, 更改该数据集中的字段, 导致她最近的A/B测试产生了不稳定的结果. 不幸的是, Stephanie has no way of knowing why her results were off, 和 instead of proudly presenting the results of her experiment at her company’s next Data Science All-H和s, 她放弃了这个实验,重新开始.

解决方案

Data observability fixes this lack of end-to-end visibility into your data pipeline’s upstream 和 downstream dependencies so you can identify where data fire drills occurred 和 resolve them quickly. 即使管道破裂或A/B测试出错, you can identify the root cause of the error 和 update your experiment accordingly, 增加对数据的信任并降低计算成本.

#2: Your Looker dashboard hasn’t been updated in 24 hours

A data catalog brings all metadata about a data asset into a single pane of glass, 所以你可以看到你可以看到血统, 模式, 历史变化, 新鲜, 体积, 用户, 查询, 在单个视图中还有更多. 图片由巴尔摩西提供.
情况

在斯蒂芬妮的公司, the Marketing Analytics team uses Looker to visualize how many sales qualified leads are generated per day as a result of a new billboard in NYC’s Times Square. 4季度计划会议前几分钟他们的CEO, the VP of Marketing pings Stephanie on Slack: “The data is all wrong… what happened?!”

她打开Looker,意识到数字, 通常每15分钟更新一次, 24小时没被碰过了!

解决方案

A good approach to 数据可观测性 provides ML-based detection of 新鲜 issues 和 monitors your data for abnormalities, 实时提醒您. 一个好的数据可观察性策略可以做到这一切, 并提供单一的, 窗格式视图到您的数据运行状况, cataloging your metadata across your various data sources. 此外, automated detection reduces manual toil 和 frees up time to work on projects that will actually move the needle for your company.

3 .数据分析师之间的沟通不畅, 数据科学家, 数据工程师负责数据消防演习

Data observability facilitates greater collaboration within data teams by making it easy to identify 和 resolve issues as they arise, 不是几个小时后. 图片由巴尔摩西提供.
情况

Stephanie’s team is building a new data model to better underst和 what type of prospective customer is most interested in her company’s product, 按地理区域和行业部门过滤. 这是数据分析师之间的跨团队协作, 数据科学家, 和数据工程师, 结果决定了很多事情.

当Stephanie部署模型时,什么也没有发生. 她又试. 一次又一次. 不过,不行. 不知道她, a data scientist in Chicago has made a 模式 change to a data set that’s forever altered the model as she knows it. There’s no way to tell what data was updated 和 where the break happened, let alone how to fix it!

解决方案

A strong approach to 数据可观测性 will enable teams to collaboratively triage 和 troubleshoot 数据停机时间 incidents, allowing you to identify the root cause of the issue 和 fix it fast. 或许最重要的是, such a solution prevents 数据停机时间 incidents from happening in the first place by exposing rich information about your data assets so that changes 和 modifications can be made responsibly 和 proactively.

While 数据停机时间 varies from company to company (和 team-to-team), 数据可观测性 can help. Maybe you even saw yourself in Stephanie 和 have some of your own good 坏数据的故事 分享,!

我洗耳恭听.

有兴趣了解更多? 接触 巴尔摩西 以及可以玩滚球的正规app团队的其他成员!