《TDWI：2024數據流實施流程和最佳實踐報告：數據監控、管理與可觀測性（英文版）（28頁）.pdf》由會員分享，可在線閱讀，更多相關《TDWI：2024數據流實施流程和最佳實踐報告：數據監控、管理與可觀測性（英文版）（28頁）.pdf（28頁珍藏版）》請在三個皮匠報告上搜索。

1、Q3 2024Data Monitoring,Management,and ObservabilityBy Fern Halper,Ph.D.BEST PRACTICES REPORTCo-sponsored by:tdwi.orgTDWI RESEARCH1BEST PRACTICES REPORTQ3 2024Data Monitoring,Management,and Observability By Fern Halper,Ph.D.Table of ContentsExecutive Summary .5The Move Forward:The Need for Robust Mon

2、itoring and Observability Practices .5What Is Data Monitoring and Observability?.6Deriving Value from Data .7Factors Contributing to Data Value .9Challenges Organizations Face Deriving Value from Data .11State of Monitoring and Observability .11Factors Impacting the Health of Company Data .12Tools f

3、or Data Health .13Software Models .15Practices for Monitoring and Observability .15Automated Versus Manual Processes .18Challenges for Monitoring and Observability .18Developing Metrics .19Challenges with Alerting .21Characteristics of Successful Organizations .22Those Who Obtain Value from Data .22

4、Recommendations .24Research Co-sponsor:Snowflake .26 2024 by TDWI,a division of 1105 Media,Inc.All rights reserved.Reproductions in whole or in part are prohibited except by written permission.Email requests or feedback to infotdwi.org.Product and company names mentioned herein may be trademarks and

5、/or registered trademarks of their respective companies.Inclusion of a vendor,product,or service in TDWI research does not constitute an endorsement by TDWI or its management.Sponsorship of a publication should not be construed as an endorsement of the sponsor organization or validation of its claim

6、s.This report is based on independent research and represents TDWIs findings;reader experience may differ.The information contained in this report was obtained from sources believed to be reliable at the time of publication.Features and specifications can and do change frequently;readers are encoura

7、ged to visit vendor websites for updated information.TDWI shall not be liable for any omissions or errors in the information in this report.Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH2About the AuthorFERN HALPER,Ph.D.,is vice president and senior dir

8、ector of TDWI Research for advanced analytics.She is well known in the analytics community,having been published hundreds of times on data mining and information technology over the past 20 years.Halper is also coauthor of several Dummies books on cloud computing and big data.She focuses on advanced

9、 analytics,including predictive analytics,machine learning,AI,cognitive computing,and big data analytics approaches.She has been a partner at industry analyst firm Hurwitz&Associates and a lead data analyst for Bell Labs.She has taught at both Colgate University and Bentley University.Her Ph.D.is fr

10、om Texas A&M University.You can reach her by email(fhalpertdwi.org)and on LinkedIn( TDWI ResearchTDWI Research provides industry-leading research and advice for data and analytics professionals worldwide.TDWI Research focuses on modern data management,analytics,and data science approaches and teams

11、up with industry thought leaders and practitioners to deliver both broad and deep understanding of business and technical challenges surrounding the deployment and use of data and analytics.TDWI Research offers in-depth research reports,commentary,assessments,inquiry services,and topical conferences

12、 as well as strategic planning services to user and vendor organizations.About the TDWI Best Practices Reports SeriesThis series is designed to educate technical and business professionals about new business intelligence technologies,concepts,or approaches that address a significant problem or issue

13、.Research for the reports is conducted via interviews with industry experts and leading-edge user companies and is supplemented by surveys of business intelligence professionals.To support the program,TDWI seeks vendors that collectively wish to evangelize a new approach to solving business intellig

14、ence problems or an emerging technology discipline.By banding together,sponsors can validate a new market niche and educate organizations about alternative solutions to critical business intelligence issues.To suggest a topic that meets these requirements,please contact TDWI Senior Research Director

15、s James Kobielus(jkobielustdwi.org)or Fern Halper(fhalpertdwi.org).AcknowledgmentsTDWI would like to thank many people who contributed to this report.First,we appreciate the many users who responded to our survey,especially those who agreed to our requests for phone interviews.Second,our report spon

16、sors,who diligently reviewed outlines,survey questions,and report drafts.Finally,we would like to recognize TDWIs production team:Lindsay Stares and Rod Gosser.SponsorsMonte Carlo,SAP,and Snowflake sponsored the research and writing of this report.Data Monitoring,Management,and Observability:Best Pr

17、actices for Success Survey methodology.In June 2024,TDWI sent an invitation via email to the analytics and data professionals in our database,asking them to complete an online survey.Over 200 respondents answered the survey;183 met quality standards for completeness and validity.This group is primar

18、ily used for analysis.Research methods.In addition to the survey,TDWI conducted interviews with technical users,business sponsors,and data experts.TDWI also received briefings from vendors that offer products and services related to these technologies.Survey demographics.Respondents act in a variety

19、 of roles and came from a range of industries.Most survey respondents reside in the U.S.(74%).Respondents come from enterprises of all sizes.Research Methodology and Demographics Report purpose.To harness the full potential of your companys data assets,the assets need to be trustedthey need to maint

20、ain their integrity and be used in accordance with business policies and objectives.Yet,many organizations dont trust the quality of their data and other data assets.This report examines the best practices organizations have implemented to maintain the health and performance of data ecosystems,ensur

21、ing that data sources and outputs are trusted.It examines the extent to which organizations have and are implementing a holistic approach and how they are moving forward with data monitoring and observability.RolesC-suite12%Director/VP IT16%Data scientists and analysts15%Data engineers and data role

22、s22%Architects12%Consultants7%Others16%tdwi.orgTDWI RESEARCH4Based on 183 respondents.GeographyUnited States 74%Mexico/Central America/South America 4%Canada 6%Asia 3%Australia/New Zealand 2%Europe 6%Africa 3%Middle East 2%Company Size by RevenueLess than$100 million28%$100 million to$499 million16%

23、$500 million to$999 million12%$1 billion to$4.9 billion21%$5 billion to$9.9 billion8%$10 billion or higher15%Industry(“Other”consists of multiple industries,each represented by less than 4%of respondents.)Information Technology23%Public Sector15%Financial services12%Retail/CPG9%Healthcare/Lifescienc

24、es 8%Manufacturing(non-computer)8%Construction/Engineering6%Insurance5%Other14%Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH5Executive SummaryTo harness the full potential of data,organizations must ensure the integrity,availability,and efficiency of t

25、heir data processes.This report examines the processes and best practices organizations are putting in place to track data flows,detect and respond to issues in real time,and maintain overall data health.The picture that emerges from the survey results is one where organizations are using a range of

26、 different tools and tool types to perform different aspects of data monitoring and observability.These include core data management tools as well as newer automated and observability tools.Additionally,many respondents still use manual checks and homegrown tools.Despite challenges with budget,data

27、silos,and skills,adoption of data monitoring and observability solutions is growing,with many organizations planning to implement these tools within the next year.A unique feature of this report is its examination of the characteristics of companies that self-identify as deriving business value from

28、 their data.The majority of organizations surveyed report getting some value from their data.The report categorizes respondents into“Achievers”(60%)and“Seekers”(40%),with Achievers obtaining significantly higher value from their data across various business functions.Key findings include:Achievers a

29、re more likely to utilize advanced tools such as modern automated data quality tools,specific data quality tools,and data observability tools.These tools facilitate better data management by providing comprehensive visibility,automated data profiling,and real-time monitoring capabilities.Achievers a

30、re also more likely to have systematic processes in place for dealing with data health.Seekers are less likely to have incident management protocols in place than Achievers.Foundational practices such as regular data audits,secure databases,strict access controls,the role of the data steward,and Dat

31、aOps are generally recognized and adopted across all organizations.It is still relatively early days for cross-platform visibility using modern tools to ensure data health.The Move Forward:The Need for Robust Monitoring and Observability Practices In todays rapidly evolving digital landscape,organiz

32、ations are increasingly navigating through complex data and analytics environments characterized by a diverse array of data types,multifaceted platforms,and innovative analytics methodologies.Previous TDWI research has illustrated that many organizations are collecting vast amounts of diverse data.T

33、his includes both structured data and unstructured data such as text data,image data,and video data.A primary priority for collecting and utilizing this data is to support more advanced analytics such as AI and AI applications.Many enterprises are moving their data assets to cloud data platforms,Dat

34、a Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH6ensure data integrity has cost their company.This may be time spent fixing poor quality data,fixing pipelines,or rebuilding AI models.In this survey,across all respondents,a rough estimate of the median amount

35、 of time lost to data integrity issues is hundreds of hours per person per year.This TDWI Best Practices Report dives into the strategic importance of implementing robust practices to maintain the health and performance of data ecosystems,ensuring that data is reliable,that organizations know when t

36、here are issues with their data,and outcomes are impactful and trustworthy.A hypothesis is that organizations with healthy data derive more value from their data.We will see that this is true,which then raises the question:what practices and tools are those who are deriving value utilizing?Although

37、the healthy data topic is obviously connected to data governance,which helps establish some of the framework and policies for data health,the focus of this report is specifically on data health and the processes and tools for it.This is sometimes referred to as data monitoring and observability.What

38、 Is Data Monitoring and Observability?Maintaining the health of data ecosystems means that organizations need to keep track of what is happening to their data across their data operations and across different platforms and pipelines.This will involve holistic monitoring.Data monitoring is the proact

39、ive process of reviewing and evaluating data and its quality using software that measures and tracks performance through dashboards,alerts,and reports to ensure the data is fit for purpose.Healthy data also requires visibility.Data and most organizations today have a hybrid environment that consists

40、 of multiple platforms both on-premises and in the cloud.This has resulted in a complex data ecosystem.To harness the power of their data,organizations need to ensure the integrity,availability,and efficiency of their data and data environments.They need to be able to track and observe what is flowi

41、ng through these data environments to rely on the integrity of their data and data processes.When problems occur,they need to be alerted in real time to be able to address them.Ideally,organizations should be able to anticipate when problems might occur.This is a multifaceted issue;on the one hand,t

42、o get the most value from their data for analytics,AI,and applications,organizations must trust the integrity of their data.The adage“garbage in,garbage out”certainly applies here.Yet,in TDWI research,we see that many organizations are not satisfied with the quality of their data.For instance,in a r

43、ecent TDWI survey,slightly less than 50%were satisfied with the quality of their data;the rest were either dissatisfied or on the fence.Likewise,slightly less than 50%of respondents trusted their data;the rest were either dissatisfied or neutral.1 Additionally,data integrity and health is about more

44、 than simple data quality;it is about maintaining the health of data ecosystems across the data life cycle,which includes ensuring that data sources are reliable,data pipelines are running smoothly,and there is transparency across systems.It is about ensuring the health of data operations.Poor data

45、quality can even be costly.For instance,in the survey for this report,we asked respondents how much time they believed not being able to 1 Unpublished 2024 TDWI Data and Analytics Survey.Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH7observability ensur

46、es comprehensive and holistic visibility into the health of data systems using advanced analytics such as machine learning to detect anomalies,analyze root causes,and track data lineage.In some respects,data monitoring is a subset of data observability.However,no matter what you call it,the point is

47、 to be able to detect and respond to problems with data in platforms,in pipelines,and other parts of the data ecosystem,across the data life cycle.Data quality is a core component of data health and deriving data value.Quality has always been important,even if it was handled manually(e.g.,think abou

48、t footing and cross-footing ledgers).It obviously became even more important in the 1990s as databases and data warehouses were developed.Data quality tools have been in the market since then,focused on ensuring the stan-dardization,quality,and consistency of company data.These tools helped organiza

49、tions determine that the data in their data warehouse matched source data,identified duplicate data,looked for missing data,and performed other quality checks.What has changed is that the volume and complexity of data has increased.It is no longer possible to perform data quality checks using only m

50、anual methods.Today,tools are available that can help determine and track data quality and integrity.Some tools enable organizations to set up custom rules for data quality.Some tools incorporate artificial intelligence and machine learning to automate data quality tasks.Tools have evolved to monito

51、r data and data quality metrics.Some tools provide visibility into data pipelines,including data lineage,quality checks,and anomaly detection.These can detect when pipelines change,they can compare data quality metrics against defined thresholds,and they can provide alerts when problems occur.Modern

52、 observability platforms often provide a holistic approach to analyze the data flowing in and across systems by monitoring data quality and ensuring data is reliable.This is important to ensure that data continues to provide value.If data changes,loses its integrity,or gets lost in pipelines that co

53、nnect infrastructure,its value diminishes.Deriving Value from DataHealthy data should help drive business value.To understand whether data is driving value and the practices used to ensure integrity and value,we asked respondents how they would rate the business value their company is obtaining from

54、 its data and data-derived assets.Respondents rated the value as either very low,low,neither high nor low(i.e.,on the fence),high,or very high.In this report,the group rating the value from data as high or very high are referred to as the Achievers.Those who rated the value as low,very low,or on the

55、 fence are referred to as the Seekers.Sixty percent of respondents rated the value they are obtaining from data as high or very high.In the survey for this report,approximately 60%of respondents are Achievers and 40%are Seekers.The Achievers tend to be more advanced in data and analytics practices t

56、han the Seekers.For instance,53%of the Achievers are already driving value from being proactive with predictive analytics;only 35%of the Seekers are doing this.The Achievers are also more likely to be building internal data products than the Seekers(73%versus 53%,not shown).This is significantly dif

57、ferent(p-value of 0.05).Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH8The Achievers group is deriving high value across many business functions(Figure 1).For instance,when we asked,“For each area where your company uses data,please rate the value from

58、it as low,medium,or high,”there were numerous areas where the majority of Achievers ranked the value as high.These included customer insights,operational insights,and financial insights.They rated innovation and collaborative data sharing as medium to high.This is contrasted with the Seekers group t

59、hat is deriving less value in different business functions.There was only one area where more than a quarter of Seekers rated the value of data as high.These differences were statistically significant;the achievers were more likely than the seekers to say they are deriving value in all these functio

60、nal areas.We have seen this in other TDWI research,as well.Those who are utilizing advanced analytics are more likely to achieve a top or bottom-line impact than those who arent making use of the technologies.2 Figure 1How would you rate the value you are deriving from data in the following areas?(O

61、nly“high value”responses shown.)Achievers rate the value of data across business functions much higher than Seekers.Based on 115 respondents for Achievers and 68 responses for Seekers.2 See 2023 TDWI Best Practices Report:Achieving Success with Modern Analytics,online at tdwi.org/bpreports.Acheivers

62、SeekersOperational insights73%24%Financial insights62%28%Customer insights59%22%Collaborative data sharing40%9%Innovation40%9%Business partner network35%12%HR insights25%7%Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH9Factors Contributing to Data Value

63、There are numerous organizational and technology factors that contribute to getting value from data.When asked what factors contribute to getting value or not getting value from your data and data assets,survey respondents highlighted many issues(Table 1).Many Achievers who are deriving value from t

64、heir data stated that their data is high quality;it is accurate,timely,complete,and reliable.Others talked about having data management in place;they have the right platform.They noted they have access to data and they understand their data.Some talked about the ability to tie data collection into b

65、usiness objectives.On the flip side,the Seekers acknowledged that they were dealing with silos and issues with processes for data quality,data literacy,and governance.They lack the funding for tools.They also noted some organizational issues such as,“lack of understanding of the art of the possible,

66、”no“sense of urgency,”and a“lack of a central focus on business use cases that can drive value.”Achievers(getting high value from data)Seekers(not getting high value from data)“Flexibility of platform:being able to move a data set from one system to another with ease.Visibility:getting the right out

67、puts at the right time.Automation:being able to extract data from various sources without a lot of manual effort.”“Lack of governance and data quality programs;too many silos.”“Data quality:Accurate,timely,reliable data.”“Unfortunately,a sense of urgency.We are not proactive,but reactive and then,so

68、metimes,it is too late.”“Access to and knowledge of the data available;a data strategy is in place.”“Lack of data literacy,data experts,data skill sets,buy-in from stakeholders.”“Committed business owner with a strong use case and ability to advocate for resources to support good data;executive supp

69、ort.”“While the operational efficiencies are very visible to leadership teams,the lack of unified and centralized focus for driving use cases and investments results in low value from the data and data assets.”“Alignment with business processes.Broad awareness of how the data can be accessed and use

70、d.Integration of data and consistency in ways to access and analyze data.”“Our organizational data culturewe value metrics,but not insights or elements that drive strategy.”“Establishing clear policies for data management and usage.”“Lack of understanding the art of the possible,lack of focus on dat

71、a quality by our business unit,lack of data skill sets.”“Accessibility and data literacy.”“Lack of staff and budget for data efforts,legacy business processes,low adoption of new cloud data platform,lack of tools.”Table 1.Responses from Achievers and Seekers to the question“what factors contribute t

72、o getting value or not getting value from your data and data assets?”Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH10Figure 2What specific challenges does your company face in terms of deriving value from data and analytics?Select all that apply.Based o

73、n 183 completed responses.Lack of data ownership 43%Data is too siloed to use it to drive value 40%No or too few data product managers/owners to build data products 40%Lack of integrity in our data sources 37%No or too few developers to build applications 33%Inadequate data literacywe dont know how

74、to analyze it,even in a simple way 32%No or too few digital resources to perform advanced analytics 32%No visibility into data processes 27%Pipelines are too complicated 25%We cant manage the data we want to capture and analyze 23%Data is not understandable 22%Data changes too fast 19%Data is not ac

75、cessible 15%Pipelines are not managed properly and break 15%Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH11Challenges Organizations Face Deriving Value from DataWe also asked about specific challenges organizations face deriving value from data and ana

76、lytics(Figure 2).The results reflect the statements made in Table 1.As illustrated in the figure,challenges are both organizational and technological in nature.For instance,lack of data ownership(43%)is a problem which makes it harder to ensure trustworthy data.Likewise,as organizations move to deve

77、lop data products(i.e.,derived assets from data),respondents also cite lack of data product owners as a challenge(40%).There are also issues associated with data being too siloed to use to drive value(40%).Another top challenge is lack of integrity in data sources(37%).There are also issues related

78、to data integrity such as visibility into processes,data changing too fast,and data not being understandable.State of Monitoring and ObservabilityAlthough numerous factors play into whether a company obtains value from its data,one thing is clear.If a company doesnt pay attention to issues surroundi

79、ng the integrity of its data,across its systems,it will not gain value.This was cited by numerous respondents to this survey.To understand the current state of monitoring and observability for ensuring data health,we first asked respondents how they would rate the overall health and integrity of the

80、ir data(Figure 3).Overall,the majority of respondents(57%)rated their data as mostly trustworthy.Not surprisingly,the Achievers group was more likely to rate the health of their data as high or very high versus the Seekers.This was significantly different.Later in the report we will discuss what thi

81、s group is doing to help maintain the health of their data.Figure 3How would you rate the overall health and integrity of your companys data?Based on 183 completed responses.Very low 3%Low 16%Mostly trustworthy,although some problems persist 57%High 18%Very high 6%Data Monitoring,Management,and Obse

82、rvability:Best Practices for Success tdwi.orgTDWI RESEARCH12Factors Impacting the Health of Company DataAs mentioned,data health takes a broader view than data quality by also ensuring that the data is secure,available,and integrated properly within the data ecosystem.It looks across data systems as

83、 well.Ensuring data health involves ongoing processes that include regular monitoring,validation,and updating of data.Numerous factors can affect the health of company data.Many also impact getting value from data,but data health issues can go beyond these.For example,in this survey,we asked respond

84、ents what factors contribute to the health of their data(Table 2).Responses fell into four buckets:human factors and knowledge;data governance and quality initiatives;tools,technologies,and infrastructure;and processes and procedures.Respondents highlighted the need for executive and management buy-

85、in,which includes commitment,business participation in data governance and quality initiatives,good understanding of data,continuous monitoring tools,and consistency and automation.CategoryExample responsesHuman Factors and Knowledge“Management buy-in is probably one of our most important reasons th

86、at the overall health of our data is so good.Our upper and senior management knows the importance of this data to our customers and the entire organization.”“Committed employees,technology,and continuous learning.”“Knowledge and skills.”Data Governance and Quality Initiatives“We take its accuracy an

87、d reliability seriously and make large financial investments based on its predictions.”“Data governance and data quality initiatives with strong business ownership.”“Transparency:when data is wrong,it gets noticed and then changes happen to get the root cause fixed.”“Good understanding of data and d

88、ata-related definitions.”Tools,Technology,and Infrastructure“Continuous pipeline monitoring,data quality conformance(opera-tions),and service-level visibility”“Good tools for tracking and evaluating data.”Processes and Procedures“Implementing automated data quality checks has greatly improved our da

89、ta health.”“Consistent data validation processes are key.”“Ensuring data accuracy through regular audits has been beneficial.”“Established checks and controls owned by business stakeholders.”Table 2.Representative responses about what contributes to data health from 183 respondents.Data Monitoring,M

90、anagement,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH13Tools for Data HealthTools for maintaining the health of data can be found in many products and platforms across the data and analytics life cycle(Figure 4).There can be overlap between the kinds of tools found in differen

91、t solutions.For instance,some cloud platforms provide data quality tools.Data catalogs can provide data quality tools.Cloud platforms can provide data catalogs.In other words,tools to support data health can be found in a number of different solution and platform types.In this survey,we asked about

92、the technology solutions that organizations use for improving the overall health and integrity of data and data assets.The results indicate that respondents are using multiple tool types.Data catalog use is on the rise.Catalogs help users identify and understand what data exists and is available for

93、 analysis.They can be valuable for ensuring the health of data for a number of reasons.Data catalogs provide a centralized repository where all data assets are cataloged,making it easier to manage and access data across the organization.Some data catalogs include data profiling tools that automatica

94、lly scan and profile data assets to assess data quality,such as checking for completeness,accuracy,and consistency.Traditional data catalogs may have included metadata about data type,data source,date of creation,and other data properties,but newer catalogs are often AI-enabled.In other words,AI tec

95、hnologies(such as machine learning or NLP)are infused into the software to automate the processing of data or the interaction with it.Some Figure 4Which of the following technology solutions do you currently use for improving the overall health and integrity of data and data assets?Which are you pla

96、nning to use?Based on 183 completed responses.Data transformation tools,including ETL/ELT65%15%11%9%Cloud data management platform58%21%15%6%Data preparation tools54%28%12%6%Data quality,profiling,and/or validation tools46%35%13%6%Data catalog or metadata management44%31%17%8%Modern,more automated d

97、ata management tools41%40%14%5%Data observability tools34%33%24%9%Semantic layer27%33%25%15%Currently useNo plansPlan to use in the next yearDont knowData Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH14tools parse and deduce credible metadata.Other tools sc

98、an each new data set for sensitive data and tag that data appropriately.In this survey,44%of respondents were using some kind of data catalog to help improve the health of their data.Cloud data management tools are also used.Cloud platforms often include data catalogs as well as other tools for mana

99、ging data quality or integrity.Some provide data lineage.Cloud platform providers either offer these as features on their platforms or via partnerships.For instance,they include data profiling tools to ensure the accuracy,completeness,and timeliness of data.Some platforms include tools that use AI t

100、o automatically tag and classify data,improving data organization and searchability.Some platforms offer the ability to specify metrics,write rules,and alert users via email or other notification platforms when thresholds are violated.They can provide anomaly detection tools or tools that provide re

101、al-time visibility into data pipelines and workflows to detect and resolve issues.In this survey,58%of respondents used cloud data management tools that presumably provide some software for data quality and health.ETL,pipeline,and transformation tools provide some features.Modern ETL and data pipeli

102、ne tools offer a variety of features to help manage data health.These features ensure data quality,integrity,security,and reliability throughout the data life cycle.For instance,some pipeline tools provide data profiling and cleansing capabilities.Others provide security features such as encryption

103、and masking.Some provide data lineage tools.They monitor data as it moves through the pipeline for quality and other metrics.In this survey,65%of respondents were using these tools.In this survey,34%of respondents were using some kind of data observability platform,whether homegrown or commercial.Da

104、ta observability platforms are a modern alternative.Given the importance of healthy data,vendors have recently started to offer data observability platforms.Data observability tools are designed to provide holistic visibility into the health and performance of data systems.They include features and

105、capabilities that help organizations monitor,understand,and maintain the quality and reliability of their data.For example,some provide automated data lineage,incident management,and impact analysis.Others provide data quality checks or data quality metric monitoring.As mentioned,some of the same fe

106、atures and/or functionality can be found on other platforms,such as cloud platforms,but not necessarily labeled as observability.Data observability platforms often consolidate various data health metrics,such as freshness,accuracy,completeness,and consistency,into a unified dashboard,offering a holi

107、stic view of data health.They provide customizable KPIs,real-time alerts,and root cause analysis,and they can help assess the impact of data quality issues on downstream applications,reports,and business processes.In this survey,only 34%of respondents were currently making use of these platforms,but

108、 this is set to double in the next year if users stick to their plans.Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH15Software ModelsTools for monitoring and observability within data and analytics pipelines include open source,commercial,or tools devel

109、oped in house.Organizations may use more than one kind of model.We asked respondents what kinds of tools they are using.Forty-nine percent utilize tools developed in house for monitoring and observability;48%are using third-party commercial tools;and 45%are using cloud-based services.Thirty percent

110、are using solutions from their data management providers.Only 13%said that they werent using any kind of tool(all not shown).Commercial tools often include solutions from integration vendors and cloud providers,as mentioned earlier.Some are using specific data observability tools.Many organizations

111、build their own monitoring environment with SQL and BI reporting.Yet only 20%of respondents said that their monitoring and observability tools work across all of their environments(not shown).If monitoring and observability tools dont work across data environments,it creates a fragmented view of dat

112、a operations,leading to several issues.Without a unified monitoring solution,organizations can struggle to detect problems across their data ecosystem,resulting in blind spots that can compromise data quality,security,and performance.This fragmentation hampers the ability to track data lineage and d

113、ependencies,making it difficult to ensure data integrity and compliance with regulatory standards.Additionally,the lack of visibility increases the complexity and time required for troubleshoot-ing and resolving data issues,which can impact decision-making and operational efficiency.So,while many re

114、spondents are utilizing various tool types to help manage their data health,they may not have a complete view across platforms,and they may be using redundant tools or tools that dont necessarily solve all their issues.Practices for Monitoring and ObservabilityIn addition to asking about the tools u

115、sed for monitoring and observability,we also asked about practices.In this survey,we asked all respondents,“What practices and tools has your organization currently implemented to monitor,assess,and remediate the trustworthiness of your data?”We were particularly interested in the use of manual vers

116、us automated processes.Figure 5 illustrates that many respondents are still using manual methods to ensure the trustworthiness of their data.Sixty percent of respondents were using manual validation checks.Manual processes lead the way.Manual validation checks can be helpful for ensuring the trustwo

117、rthiness and accuracy of data,especially in scenarios where automated tools may not be available,the team requires a custom check,or the rule is complex and not part of a package.These checks may involve human intervention to review and verify data against predefined criteria or standards.These chec

118、ks may include comparing data entries across multiple sources or systems to ensure consistency or manually inspecting data to ensure that values fall within expected ranges and adhere to specific formats.People may look for null value records Data Monitoring,Management,and Observability:Best Practic

119、es for Success tdwi.orgTDWI RESEARCH16Figure 5What practices and tools has your organization currently implemented to monitor,assess,and remediate the trustworthiness of your data?Select all that apply.Based on 183 completed responses.Manual validation checks 60%Secure databases to protect against u

120、nauthorized access or corruption 55%Strict access controls 52%Periodic review and maintenance 51%Regular data audits for accuracy and consistency 44%Monitoring tools to track data 43%Data stewards 41%Automated validation checks 41%Data integrity tools 34%Automated data quality tools 33%Comprehensive

121、 data governance processes 32%DataOps team 28%Metadata tools to capture metadata about our data 26%Data observability tools for continuous health checks of our data 26%Data lineage tools 25%Not currently ensuring data quality systematically 13%Data Monitoring,Management,and Observability:Best Practi

122、ces for Success tdwi.orgTDWI RESEARCH17or perform spot checks on data for accuracy or documenting data.As mentioned above,in some cases these tools are created internally,using SQL and analytics reports.Some organizations use Excel;others are building Python scripts.Organizations that perform manual

123、 checks may also use automated tools.In this survey,60%of respondents performed at least some manual validation checks.Fifty-one percent performed periodic reviews and maintenance.Securing databases was also a top response.Securing databases to protect against unauthorized access or corruption was a

124、lso at the top of the list,with 55%of respondents selecting this option.Organizations need to protect against unauthorized access that can result in discrepancies,removing data from a database,or manipulating the data as well as data breaches.Fifty-two percent cited strict access controls as a way t

125、hey maintained(at least some of)the health of their data.Regular audits are also important.Regular audits,both internal and external,are important for maintaining trusted data.They can help enhance data security and ensure data integrity.They help with ensuring compliance,as well.Importantly,they ca

126、n also help mitigate risks.Audits can also help build stakeholder trust in the data.In this survey,44%of respondents noted that they performed audits for accuracy and consistency.Data stewards are key for ensuring integrity.Data stewards play an important role in ensuring data integrity because they

127、 are responsible for overseeing and managing the quality,consistency,and accuracy of data within an organization.They help establish and enforce data governance policies,ensuring that data is handled according to defined standards and best practices.Additionally,they monitor data usage and quality,p

128、roactively identifying and addressing any issues that could compromise data integrity,thereby maintaining the trustworthiness of the data.In other words,they take action on problems.In this survey,41%of respondents had data stewards in place to help support the health of their data.Others(28%)are re

129、lying on their DataOps teams to help.These are most likely data engineers,who may be using a different set of tools than the data stewards.Respondents are using different automated techniques.As mentioned previously,organizations are facing a complex data environment.Although many organizations use

130、manual processes to help ensure data health,others are turning to automated and augmented processes and tools.These augmented tools are often infused with advanced AI technologies to help detect and surface issues.For instance,a tool might read your data to understand what“normal”looks like and then

131、 alert the user when the pattern is anomalous.In this survey,43%of respondents claimed to be using monitoring tools to track data.Forty-one percent were using automated data validation tools.Thirty-three percent were using automated data quality tools.In addition to finding these tools in observabil

132、ity platforms,many of these tools can be part of data catalogs or cloud data platforms,as mentioned earlier.In this survey,26%of respondents were using data observability tools for continuous health checks of their data.In a separate question we asked about the maturity of observability practices.Fo

133、rty-two percent stated they were in early development(not shown).So,while some organizations have implemented data observability,they may not be that far along in utilizing the tools or processes to their fullest.We will see that this is the case,even in the Achievers group.Data Monitoring,Managemen

134、t,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH18Automated Versus Manual ProcessesTo drill further into how organizations are using automation versus manual processes for data health,we asked about whether and how organizations monitor certain data-related processes(Figure 6).As

135、 illustrated in the figure,much of this is currently manual.In fact,the only area where automated processes are significantly more widespread than manual ones is data pipelines.Although we are starting to see organizations adopt tools for automated data quality,in this survey these processes are mon

136、itored mostly manually.Team members(most likely data stewards)play a large role,even when tools are used.This isnt that surprising.People and processes are going to be an important part of the monitoring and observability capability to provide the necessary structure,context,and oversight to ensure

137、these tools are used effectively and that systems are managed proactively and responsively.For instance,people can provide contextual understanding that automated tools can miss.Established processes can provide consistent incident response.Challenges for Monitoring and ObservabilityOrganizations th

138、at implement monitoring and observability practices and tools still face challenges with maintaining the health of their data.We asked respondents to the survey about the challenges they have encountered in implementing observability in their data operations(Figure 7).Figure 6What aspects of the dat

139、a and analytics life cycle does your company currently monitor?Please select all that apply and whether this is automated or done manually.Based on 183 completed responses.Data governance65%22%13%Data quality54%39%7%Model inputs44%31%25%Data usage(who is accessing what data and when)43%50%7%Data lin

140、eage42%37%21%Model outputs37%38%25%Data pipelines34%55%11%N/AManualAutomaticData Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH19Not surprisingly,obtaining the budget to make observability happen can be difficult.Forty-six percent of respondents cited this a

141、s an issue.Sometimes,it may be difficult to make the case for holistic monitoring,and 40%of respondents cited siloed monitoring as a challenge.That means that different tools are being used to monitor different platforms,which is not efficient and can result in errors.Likewise,training on tools was

142、cited as an issue by 46%of respondents.This includes not only training on the tool itself but understanding the principles behind holistic monitoring as well as developing metrics.Thirty-four percent of respondents cited that meaningful metrics and alerts are hard to develop.Yet to be able to monito

143、r the health of data,it will be critical to develop metrics.Developing MetricsIn a separate question we asked about the key metrics respondents use to monitor and assess the health of their data(Figure 8).Creating and monitoring metrics for data quality,operational efficiency,supporting compliance,a

144、nd optimizing resources can all contribute to better business Figure 7What challenges have you encountered in implementing observability in your data operations?Select all that apply.Based on 183 completed responses.Insufficient budget or resources 46%Lack of expertise or training 46%Monitoring is s

145、iloed;we have inadequate observability across data systems and cloud platforms 40%Integration with existing systems is complex 38%Overwhelming volume of data to monitor 34%Difficulty in defining meaningful metrics and alerts 34%Overwhelming types of data to monitor 31%No significant problems encount

146、ered 8%Not applicable we dont utilize observability 7%Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH20outcomes.Some organizations create these metrics separately from any kind of monitoring or observability system;others use tools to help them track the

147、se metrics.Metrics such as system uptime/availability(51%),data processing times(48%),error rates in data processing(43%),query performance metrics(43%),data quality indicators(40%),resource utilization(40%),and user engagement metrics(28%)are important for assessing various aspects of data health.T

148、hese metrics ensure that the data infrastructure is robust,performs well,and meets user expectations.For instance,high system uptime and fast data processing times ensure that data is available and accessible when needed,while low error rates and high query performance indicate reliable and efficien

149、t data processing.Data quality indicators ensure the accuracy,completeness,and consistency of data,while resource utilization metrics help in optimizing the performance and cost-efficiency of the data infrastructure.User engagement metrics provide insights into how frequently and how effectively use

150、rs are interacting with data,which is important for understanding its impact on business processes and whether suspicious activity is occurring.However,these metrics alone are not sufficient for a comprehensive measurement of data health.Although they provide valuable information Figure 8Please iden

151、tify the key metrics that you monitor regularly to assess the health and performance of your data and analytics systems.Select all that apply.Based on 183 completed responses.System uptime/availability 51%Data processing times 48%Error rates in data processing 43%Query performance metrics 43%Data qu

152、ality indicators 40%Resource utilization(CPU,memory,storage)40%User engagement metrics 28%N/A 9%Other 2%Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH21about the operational aspects and immediate performance of data systems,they do not fully capture how

153、 long it takes companies to realize there is a problem,respond to incidents,or fix the problem.They dont capture how long a table is unavailable for use.Metrics around incident response time,for example,are important for understanding how quickly and effectively an organization can respond to and re

154、solve data-related issues.The low mention(only 2%of respondents)of other critical metrics suggests that many organizations may not be fully considering the broader aspects of data health.Challenges with AlertingIn fact,when we asked,“How do you handle and respond to real-time incidents detected by y

155、our data monitoring and observability systems?”the most popular method(39%)was manual intervention by the IT team(not shown).Having an incident response process in place for data integrity is crucial because it enables rapid identification and resolution of issues,thereby minimizing the impact on bu

156、siness operations and decision-making.It also helps facilitate continuous improvement by identifying root causes and enhancing data management practices,ultimately maintaining smooth business operations and reliable data availability.Most respondents also did not have observability processes in plac

157、e for anomaly detection and predictive alerts in their data and analytics workflows(Figure 9).For instance,49%stated that they relied on manual monitoring for anomaly detection and 42%use rule-based thresholds.However,over 60%of those using some sort of observability tool are developing rule-based t

158、hresholds(most likely for specific metrics or Figure 9What observability processes do you have in place for anomaly detection and predictive alerting in your data and analytics workflows?Select all that apply.Based on 183 completed responses.We rely on manual monitoring for anomaly detection 49%Rule

159、-based threshold alerts 42%No processes in place for anomaly detection or predictive alerting 25%Predictive analytics for trend forecasting 24%Advanced machine learning models for anomaly detection 12%Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH22comp

160、lex rules,mentioned earlier),although many(56%)also rely on manual monitoring for anomaly detection(not shown).Characteristics of Successful OrganizationsTo explore best practices further,we wanted to understand what those in the Achievers group were doing differently from the Seekers that might be

161、helpful for data health.The specific focus here was on tools and processes for healthy data rather than other important factors such as data literacy,skills,culture,organizational models,or executive support.Those Who Obtain Value from DataWe compared the groups across several dimensions,including t

162、he kinds of tools they are using,how they are responding to incidents,and how they are putting practices in place(Figure 10).As mentioned previously,Achievers are more advanced in general,and some of the findings may be a result of this,but the analysis is telling nonetheless,and can help guide thos

163、e who want to become more successful in maintaining the integrity of their data.Achievers are more likely to make use of certain tools.Achievers are more likely to use tools for managing the health of their data.Overall use of these tools was illustrated in Figure 4.An analysis indicates:Achievers(5

164、3%)are significantly more likely than Seekers(30%)to currently use data catalog or metadata management tools(p=0.001).Seekers are more likely to have no plans for data catalogs or metadata management tools.Achievers(46%)are significantly more likely than Seekers(27%)to currently use more automated d

165、ata management tools(p=0.006).Achievers(36%)are also significantly more likely than Seekers(23%)to plan to use these tools within the next year(p=0.042,not shown).These tools often include data quality and profiling capabilities.Note that this was a borderline significant difference.Achievers(55%)ar

166、e significantly more likely than Seekers(32%)to currently use data quality,profiling,and/or validation tools,in general(p=0.001).Achievers(45%)are significantly more likely than Seekers(23%)to currently use data observability tools(p=0.001).While overall adoption of these tools is low,it is the Achi

167、evers who are making more use of them than the Seekers.There was no significant difference between the Achievers and the Seekers in using cloud data platforms.In other words,Achievers are more likely to currently use modern automated data management tools,data quality/profiling/validation tools,and

168、data observability tools compared to Seekers.Both Seekers and Achievers are using cloud data management platforms and data catalogs.This suggests that although most organizations are implementing core technologies for data health such as those found in cloud data platforms and data catalogs,the Achi

169、evers appear to be more proactive in implementing newer and more specific technologies for data health.They are most likely Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH23more mature in their data management and analytics efforts.Performing a similar a

170、nalysis on the practices and tools listed in Figure 5 indicates that Achievers and Seekers demonstrate similar implementation levels for some of the core data trustworthiness practices,indicating that some foundational practices are generally recognized and adopted across organizations regardless of

171、 the value derived from data.These include manual validation checks and strict access controls.There was a significant difference between the two groups for other practices such as periodic review and maintenance and automated validation checks.Interestingly,again,Achievers(55%)are more likely than

172、Seekers(31%)to use data observability tools for continuous checks of data health,with a statistically significant difference(p=0.0005).This indicates that organizations deriving higher value from their data are more proactive in continuously monitoring the health of their data.The Achievers were als

173、o more likely to state that these tools work across multiple environments.Ensure data quality systematically.The Seekers(25%)are more likely than the Achievers(9%)to not ensure data quality systematically(p=0.005).This significant difference highlights that Seekers are more likely to not have a syst

174、ematic approach to ensuring data quality compared to Achievers.Incident response.Although both Achievers and Seekers are equally as likely to use manual intervention by the IT team,Seekers are more likely to lack a protocol(25%)than Achievers(11%).In other words,Achievers are more likely Figure 10Co

175、mparisons between groups.AcheiversSeekersPROCESSES AND PRACTICESUse data observability tools for continuous health checks of our data55%31%Lack protocols for incident response11%25%Dont ensure data quality systematically9%25%TOOL USEData catalog or metadata management tools53%30%Automated data manag

176、ement46%27%Data quality/profiling55%32%Data observability45%23%Software model used not using a tool to monitor or improve data health4%11%Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH24to have some protocol in place for dealing with incidents.Type of s

177、oftware model used.There was no significant difference between the kinds of models(commercial,open source,or in-house)used for ensuring data health and obtaining value.Not surprisingly,Seekers(11%)are more likely to not be using any tools compared to Achievers(4%)(p=0.005672).Demographic factors.Com

178、pany size was not determined to be a significant factor in determining whether respondents get value from their data.RecommendationsAchieving value from data is a complex undertaking that involves both organizational and technology factors.This report has examined the state of and best practices for

179、 data monitoring and observability related to achieving value.In closing,we explain the top best practices that can guide your organization to be successful with obtaining value from data.Foster a culture of data value recognition.Organizations face challenges when employees are not aware of the imp

180、ortance of data health.To achieve value,it will be important for everyone to buy into a culture of data.You must educate your staff on the importance of data health and share metrics so they know their data is healthy.This will also involve engaging stakeholders in terms of education as well as noti

181、fication and following up when data incidents occur.Put plans in place for ensuring data quality and health.In this report,weve seen that organizations utilize a mix of tools and technologies for ensuring data health and value.The key is to put a plan in place to systematically monitor data quality.

182、Start somewhere.Organizations that systematically ensure data quality tend to derive more value from their data.Implement comprehensive data management tools and practices.Organizations that dont already have tools such as data catalogs and cloud data management platforms in place should consider th

183、em for enhancing their data health capabilities.To ensure robust data quality and integrity,utilize automated data quality frameworks,real-time monitoring,and periodic reviews.However,organizations should also explore modern,more automated tools for data quality/profiling/validation to improve data

184、quality and management efficiency.Look at vendors that have a holistic perspective.Have an incident response plan.You can improve your incident response protocols by establishing clear,documented procedures and creating a dedicated incident response team with defined roles.Implementing automated det

185、ection and response tools,regular training and simulations,and effective stakeholder communication are crucial.Continuous improvement through post-incident reviews and feedback mechanisms,leveraging advanced incident response platforms,and adhering to industry best practices such as the NIST inciden

186、t response life cycle will enhance your capabilities.Ensure visibility across platforms.Organizations are often dealing with complex and hybrid environments.Deploy tools that work against multiple environments.Tools that work across all environments significantly correlate with higher perceived data

187、 value.Make sure automated tools are on your road map.As data and analytics environments become larger Data Monitoring,Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH25and more complex,automation will be critical.Automation reduces the likelihood of human error and incr

188、eases efficiency.Whether you use that automation to automate pipelines,identify sensitive data,perform real-time monitoring,or address data quality,these tools can help your organization become more productive.Benchmark against high-performing organizations.Although companies are implementing metric

189、s,they need to measure the right thing.System uptime is an important metric,but it will also be important to put data health metrics in place.Think about your KPIs tied to business goals.Develop metrics from these KPIs.This will help you know whether your team is meeting its goals and help you illus

190、trate success.That success can then build on itself.Additionally,benchmark against other organizations,and regularly measure and track data management performance metrics to identify areas for improvement.The use of scorecards to get a snapshot view of data health is also recommended.Invest in data

191、governance and compliance.Although this report did not talk about data governance directly,we touched on many of the processes,roles,and technologies for data governance.Regular audits,secure databases,strict access controls,and data integrity tools are essential for high data value.Data Monitoring,

192、Management,and Observability:Best Practices for Success tdwi.orgTDWI RESEARCH26Research Co-sponsorSnowflake enables every organization to mobilize their data with Snowflakes Data Cloud.Customers use the Data Cloud to unite siloed data,discover and securely share data,and execute diverse analytics wo

193、rkloads.Wherever data or users live,Snowflake delivers a single data experience that spans multiple clouds and geographies.Thousands of customers across many industries,including 639 of the 2023 Forbes Global 2000(G2K)as of July 31,2023,use the Snowflake Data Cloud to power their businesses.Learn mo

194、re at .TDWI Research provides research and advice for data professionals worldwide.TDWI Research focuses exclusively on data management and analytics issues and teams up with industry thought leaders and practitioners to deliver both broad and deep understanding of the business and technical challen

195、ges surrounding the deployment and use of data management and analytics solutions.TDWI Research offers in-depth research reports,commentary,inquiry services,and topical conferences as well as strategic planning services to user and vendor organizations.A Division of 1105 Media 6300 Canoga Avenue,Suite 1150 Woodland Hills,CA 91367E infotdwi.orgtdwi.org