Lightup Blog on Lightup

Introducing Lightup Recommendations for Automated Data Quality Optimization

Wed, 27 Aug 2025 09:00:00 -0700

You’ve enabled no-code Auto Metrics, monitors, and even some business-specific custom SQL rules in Lightup — but how do you know what’s missing or what to fine-tune next?

That’s where Lightup Recommendations come in.

We’re excited to announce Lightup Recommendations, a new beta feature coming soon to Lightup that provides intelligent, actionable suggestions to optimize your Data Quality coverage. Think of it as your Data Quality advisor — always scanning your environment for missing checks, monitors, and other Data Quality best practices. Lightup automatically analyzes your configurations and usage, always learning and always recommending and automating the next best action to take.

Introducing Lightup Data Quality for Unstructured Data

Thu, 31 Jul 2025 09:00:00 -0700

The AI Era Needs Data Quality for Unstructured Data, Starting With Documents

More than 80% of enterprise data is unstructured — but traditional Data Quality tools are designed to run checks or queries on structured data in databases and data warehouses.

That means many enterprises aren’t leveraging unstructured data in critical documents that power operations, analytics, compliance, and customer experience, such as:

Financial reports with updated numbers.
Product documentation scattered across folders or document repositories.
Customer support knowledge bases that need constant updating.

Even small changes in these documents can introduce inaccuracy, inconsistency, or incompleteness — and in some cases, even PII contamination. Unfortunately, monitoring the quality of document data isn’t possible with traditional Data Quality tools and has been challenging to operationalize at enterprise scale.

Feature Roundup Webinar: What's New in Lightup?

Wed, 30 Jul 2025 09:00:00 -0700

As organizations scale their data operations, one truth becomes clear: Data Quality and Observability are no longer “nice to have” — they are mission-critical.

At Lightup, we’ve taken a deep dive into the architecture of our platform and discovered more opportunities than we originally imagined. Each layer — from ingestion to visualization — presents unique challenges at scale, and we’ve been systematically upgrading every one of them.

The result? A new wave of features that radically improve both automation and control — enabling faster time-to-value and safer enterprise-wide collaboration.

How to Build Data Quality Dashboards with Lightup Metrics and Your Favorite BI Tools

Fri, 02 May 2025 09:00:00 -0700

If you care about Data Quality reporting for broader transparency at your organization, you need custom dashboards. Whether you’re tracking data freshness, schema changes, or incidents, creating Data Quality dashboards is the best way to report usage, trends, and visually summarize insights by product or department for stakeholders and executive leadership.

Lightup provides built-in Data Quality dashboards for at-a-glance insights that matter most to your team.

For organizations that prefer to use their enterprise BI tool, Lightup enables teams to build executive reports and custom dashboards with PowerBI, Tableau, Apache Superset, and more.

Introducing Lightup for Data Lineage: Enabling Data Quality Assurance with Enhanced Traceability and Faster Root Cause Analysis

Tue, 01 Apr 2025 09:00:00 -0700

As organizations continue to modernize their data management and race to implement more AI-driven data products, the need for reliable, accurate, and auditable data is now more critical than ever. Large enterprises managing massive amounts of data, running complex pipelines, and working with AI and ML applications rely heavily on the integrity and accuracy of data.

So, how can enterprises maintain data reliability and traceability throughout the data life cycle?

Data Quality and Observability for Streaming Data: Introducing Lightup’s New ksqlDB Connector

Sun, 16 Mar 2025 09:00:00 -0700

Data Quality Monitoring for Streaming Data

Monitoring Data Quality for structured data at rest is challenging enough in modern data environments without adding another layer of complexity to it, like real-time streaming data. With data in motion, Data Quality issues are amplified since hidden bad data can cascade quickly, proliferating to more places downstream — likely, undetected.

For example, if incorrect data and invalid schemas go undetected in event-driven streaming platforms like Kafka, downstream applications and supported services in that ecosystem can fail or run at suboptimal performance. Since many organizations use Kafka to run real-time applications for high-volume and high-speed transactions — such as restaurant sales, e-commerce, Internet of Things (IoT), telco systems, streaming video services, or even tracking customer loyalty points — it’s critical to ensure every system is running on high-quality data.

4 Out of 4 LLMs Tested Got It Wrong: Examining the Impact of Data Inconsistency in Unstructured Data

Tue, 25 Feb 2025 09:00:00 -0700

What Is Data Inconsistency?

Data inconsistency is data that isn’t standardized or uniform across various data sources, systems, or formats. This often occurs when data is pulled from outdated or incorrect data sources containing conflicting, invalid, or partial information for the same attribute.

Inconsistent data can arise from differences in formats, currencies, or units between sources, typically caused by mistakes like typos, improper data formats, or invalid data entry due to human error or lack of knowledge.

Lightup Fireside Chat with Guest Malcolm Hawker: Challenging the Core Foundations of Data to Enable GenAI

Tue, 04 Feb 2025 09:00:00 -0700

Introduction

In our latest Fireside Chat, Malcolm Hawker, CDO of Profisee, joins Lightup CEO and Founder Manu Bansal to unpack a question posed in Malcolm’s LinkedIn post: “Why should data teams be skeptical if they hear a ‘focus on foundations’ is needed to enable GenAI?”

Fireside Chat

For this “poke-the-bear-worthy” fireside chat, challenging some of the core foundations of data, we exceeded 5 minutes. But, it was well worth it to hear Malcolm and Manu engage in a lively conversation with probing questions and surprising answers, covering:

Lightup Feature Release: Zero-Config for Auto Metrics

Thu, 30 Jan 2025 09:00:00 -0700

By design, metrics aren’t automatically enabled by default in Lightup. Instead, metrics must be turned on or enabled, offering full control of metric configurations while mitigating unexpected query computation costs.

With advancements in AI and Data Quality automation, we’re excited to announce that Lightup now supports plug-and-play Auto Metrics for metadata with our latest new feature: Zero-Config.

Zero-Config eliminates tedious clicks and repetitive configuration tasks for Auto Metrics, helping you increase efficiency and productivity.

Revisiting Data Quality and Data Observability: What's (E)merging and Why That Matters

Fri, 20 Dec 2024 09:00:00 -0700

In 2023, I wrote a blog highlighting the similarities and differences between Data Quality and Data Observability technology platforms. Back then, there was still a fair amount of confusion about the terminology and categorization of products in the Data Quality realm. “Data Quality” has been the accepted category, which was disrupted by new vendors that carved out a new space — “Data Observability.”

But, as anticipated, as more and more companies learned about modern Data Quality Monitoring and new Data Observability platforms, we started seeing customers’ perceptions and expectations converge — requiring features and capabilities of both technologies in one solution.

Announcing the Beta Preview of Genie, Lightup’s New GenAI Copilot Assistant

Tue, 17 Dec 2024 09:00:00 -0700

You asked for more AI, and we’re on it! We’re excited to announce a new Lightup feature, available in beta preview.

Get ready to meet Genie , Lightup’s copilot assistant, powered by GenAI.

Chat with Genie anytime by clicking the “Ask Genie” button in the lower right corner of Lightup.

Your Wish Is Genie’s Command

Genie is designed to help automate tasks, making it even easier and faster to take actions, such as enabling and disabling Data Quality Checks in Lightup. And just like any handy GenAI copilot assistant, Genie answers questions and provides guidance for Lightup configurations and processes — all through a simple, natural language chat experience.

5 Years of Lessons in Data Quality & Observability: My Retrospective & Future Vision for Lightup

Tue, 26 Nov 2024 09:00:00 -0700

Taking time to reflect on the past five years of running Lightup has re-energized me as I remember our humble beginnings and all the lessons we’ve learned along the way — some hard, some easy, some good … and some completely unexpected.

I’m proud to share that we’ve reached a huge five-year milestone, supporting Data Quality (DQ) at an unprecedented scale, where our biggest enterprise customers are now able to:

Lightup Recognized as Enterprise Data Quality Monitoring Solution Company of the Year 2024 by CIOReview

Thu, 05 Sep 2024 09:00:00 -0700

We’re honored to announce that Lightup has been recognized as the Enterprise Data Quality Monitoring Solution Company of the Year 2024 by CIOReview! This award is a testament to our unwavering commitment to innovation, excellence, and customer satisfaction in the realm of Data Quality.

See why Lightup was recognized, check out CIOReview Magazine’s cover story, Proactively Detect Bad Data with Faster Data Quality Checks, featuring our CEO and Co-founder Manu Bansal.

CIOReview Magazine featuring CEO and Co-founder of Lightup Manu Bansal

A Beginner’s Walkthrough of Databricks Lakehouse Monitoring and Lightup Data Quality Monitoring

Fri, 02 Aug 2024 09:00:00 -0700

With the launch of Databricks Lakehouse Monitoring in public preview late last year, customers have asked us where it fits in their data stack, especially in relation to Data Quality Monitoring platforms like Lightup.

We’ll walk through an overview of Lakehouse Monitoring and discuss where Lightup fits in for monitoring Data Quality and ensuring trusted data in Databricks and beyond.

After reading this blog, you’ll be able to answer these questions and more:

Automating Data Remediation with Lightup

Mon, 22 Jul 2024 09:00:00 -0700

What is data remediation and how does Lightup help simplify data remediation processes through automation?

Data remediation is the systematic process of identifying and rectifying Data Quality issues that arise within a dataset. Lightup helps automate data remediation processes by offering an easy way for enterprise data teams to identify, analyze, and fix data issues at scale. These issues may include anything that compromises the integrity and usability of the data, such as:

Ensuring Reliable Data for Self-Service Analytics: How Lightup and Dremio Work Better Together

Tue, 16 Jul 2024 09:00:00 -0700

Data lakes, data warehouses, databases, data lakehouses — data is vast, varied, and scattered everywhere in modern enterprise data stacks. Traditional methods of running analytics in these complex environments present ongoing challenges for enterprises due to:

Costly and time-consuming data management processes
Scarce and expensive data engineering resources
Poor-quality data

That’s where Lightup and Dremio come in, providing easy access to no-code Data Quality checks for self-service analytics.

While there are a variety of vendors that provide Data Quality solutions based on specific data sources, there are very few that connect directly to Dremio for Data Quality Monitoring. Connect Lightup directly to Dremio in minutes with a prebuilt connector.

5-Minute Fireside Chat with Lightup:The Importance of Fine-grained Data Quality ChecksGuest speaker: Olga Maydanchik, Data Management Strategist

Wed, 15 May 2024 09:00:00 -0700

We’re excited to kick off a new series, 5-Minute Fireside Chat with Lightup , where we engage with data leaders and experts in the data management field to discuss data trends, technology evolution, and insider insights about the world of Data Quality.

In our first quick and cozy 5-minute fireside chat, CEO and Co-founder of Lightup Manu Bansal and Olga Maydanchik, a data management strategist, discuss the importance of fine-grained Data Quality Checks.

Expanding Our Databricks Partnership: Lightup Now Supports Databricks Unity Catalog

Thu, 18 Apr 2024 09:00:00 -0700

We’re excited to announce that our partnership with Databricks has expanded with full support for Unity Catalog through Databricks Partner Connect.

While Unity Catalog helps simplify data discovery in Databricks, that requires ongoing certification from a Data Quality Monitoring tool. Without that, Databricks users won’t know if they can trust the data they pull from Unity Catalog.

That alone limits their access to real self-service data in Databricks, because they’ll constantly depend on data engineers for help certifying the data, causing time-consuming bottlenecks. Even worse, they might blindly use bad data, causing hidden bugs in reports, inaccurate machine learning models, and low-quality products and services.

Fact or Fiction: Queries Are Either Fast or Quickly Built, Not Both

Tue, 09 Apr 2024 09:00:00 -0700

In a sea of LinkedIn memes, this one begs a discussion. Not just the meme itself, but how it’s framed: “You can either have a fast query or a quickly built query, you can’t have both.”

Source: LinkedIn

We beg to differ…

At Lightup, we think you can have both: A Data Quality query that’s both fast and quickly built.

Lightup Named to the 2024 CB Insights AI 100 List:Lightup Recognized for Achievements in Data Quality Monitoring with AI-powered Anomaly Detection

Tue, 02 Apr 2024 09:00:00 -0700

NEW YORK, April 2, 2024 — CB Insights today named Lightup to its eighth-annual AI 100, showcasing the 100 most promising private AI companies of 2024.

“AI is taking off at lightning speed, and it’s not just big tech companies at the forefront of it,” said Deepashri Varadharajan, director of AI research at CB Insights. “Our AI 100 winners – many of them early stage startups, some with very small teams — are pushing the boundaries of AI in everything from game development and battery design to agentic AI systems.”

‘Shifting Left’ with an Industry-First from Lightup: Data Quality Checks on Object Storage

Mon, 25 Mar 2024 09:00:00 -0700

In today’s data-driven world, enterprises rely on data more than ever for everything from operations, customer experience, Artificial Intelligence (AI)/Machine Learning (ML) applications, and Business Intelligence (BI) for informed decision-making. However, the quality of this data is critical, especially when it comes to fueling BI and AI/ML systems.

Applying the “shift left” DevOps approach to Data Quality, forward-thinking enterprise organizations want to deploy Data Quality Checks as early as possible, so they can understand the quality of their raw data at the beginning of the data life cycle in their Object Storage.

Lighting up Data Discrepancies with Lightup’s Data Reconciliation

Fri, 22 Mar 2024 09:00:00 -0700

In today’s modern enterprise where every question is a data question, the precision and reliability of data is non-negotiable. Inaccurate data leads to poor decision-making, decreased productivity, compliance risks, and more…

That’s where Data Reconciliation comes in.

As a critical process in Data Quality Management, reconciling data involves comparing data from disparate sources to ensure accuracy, consistency, and completeness, safeguarding against costly errors and skewed insights.

By comparing data at a row-by-row level, Data Reconciliation identifies inconsistencies between datasets.

Think Small, Scale Big: The New Lean Approach to Enterprise Data Quality Monitoring — Without the Headache

Wed, 06 Mar 2024 09:00:00 -0700

Abandoned shelfware. It happens to the best of us…

You know the drill: In the world of enterprise Data Management, integrating a legacy Data Quality platform is generally perceived as a daunting initiative. Often seen as a complex, resource-intensive process, enterprises invest a lot of time, money, and planning to implement traditional Data Quality solutions.

We’ve heard this across industries: They attempt to implement legacy Data Quality tools or they try to build their own homegrown solution. Ultimately, both end up being way too cumbersome to actually adopt and their Data Quality initiative falls flat.

Democratizing Data Quality from Data Engineers to Executives: Empowering Enterprise Organizations with Accessible, Trusted Data

Tue, 13 Feb 2024 09:00:00 -0700

Today’s data-driven enterprise organizations rely heavily on data to make informed decisions, drive innovation, and gain a competitive edge. However, the value of data is intrinsically linked to its quality. Poor quality data can lead to inaccurate insights, misguided decision-making, and wasted resources.

Given the pivotal role of Data Quality within organizations, there’s an increasing demand to establish comprehensive and early Data Quality measures throughout the data life cycle.

Faced with enormous data volumes, leading enterprises are moving away from centralizing Data Quality within the IT organization. Instead, they’re decentralizing Data Quality, democratizing access to Data Quality tools and inviting broader participation from business analysts, leadership teams, data consumers, and data domain experts in the Data Quality life cycle.

Navigating the Data Quality Monitoring and Data Observability Landscape: Understanding Architectural Nuances

Mon, 29 Jan 2024 09:00:00 -0700

In the ever-evolving realm of Data Quality Monitoring and Data Observability, we’ve witnessed a significant technology transformation over the past few years. Our earlier blog post explored the diverse product philosophies within these products, highlighting the differences and similarities between Data Quality Monitoring and Data Observability tools.

Now, let’s dive into the underlying architectures of these technologies, explaining their intricacies and implications for business users and vendors.

Our goal is to help you understand the fundamental architectures of Data Quality Monitoring and Data Observability platforms, so you can make the right long-term strategic investment in the right tool.

10 Must-Have Features for Digging Deeper into Data Quality Analysis

Fri, 26 Jan 2024 09:00:00 -0700

Embarking on a Data Quality journey is a crucial endeavor for data-driven organizations. The initial fundamental steps often include confirming data frequency and volume through metadata analysis. This preliminary phase lays the groundwork for what lies ahead.

So, what happens next?

How can you ensure that the data being processed is not only the right volume and frequency, but also reliable, accurate, and trustworthy?

This is where having the right features and capabilities in your Data Quality Monitoring platform becomes paramount.

10 Essential Questions for Choosing a Data Quality Monitoring Tool

Tue, 12 Dec 2023 09:00:00 -0700

Selecting the best Data Quality Monitoring tool for your organization is a critical decision. With countless options available in the market, it can be daunting to make the right choice.

To help fast-track your Data Quality journey, we’ve compiled a list of 10 essential questions to ask when choosing a Data Quality Monitoring tool.

Whether you’re a data analyst, data engineer, data scientist, IT professional, or a business leader, these questions will help guide you through the initial process of evaluating and picking a Data Quality Monitoring platform to meet your organization’s specific requirements.

Unlocking a New Era of Data Democratization: Introducing Lightup’s Industry-First Integration with Collibra for Trusted Data Assets

Tue, 05 Dec 2023 09:00:00 -0700

Introducing Lightup’s Integration with Collibra

We’re excited to announce that Lightup now integrates with Collibra’s Data Governance and Catalog platform. For this industry-first integration with Collibra, we partnered with leading Systems Integrator XponentL Data to surface Lightup Data Quality Indicators (DQI) inside Collibra’s Data Governance Center.

Lightup advances Collibra’s mission to democratize Data Discovery for data citizens by enhancing the platform with no-code Data Quality Checks — bridging the skills gap between technical data teams and non-technical business users that need to define, update, and deploy Data Quality Checks for Collibra data assets.

Myth-Busting Straight Talk About Data Quality Monitoring and Data Observability

Thu, 14 Sep 2023 09:00:00 -0700

What is Data Observability vs. Data Quality Monitoring? What are they used for?

Are they interchangeable? Do they overlap?

Whether posed by customers, investors, or reporters, these questions inevitably come up.

Blurred messaging. Inflated positioning. Broad claims. Overstated capabilities. No wonder there’s market confusion over the differences between Data Observability and Data Quality Monitoring.

Is there a straightforward way to classify and understand these products and capabilities? We’re here to set the record straight and explain the key differences and use cases for Data Observability and Data Quality Monitoring.

Announcing Our Featured Customer Speaker at the Databricks Data+AI Summit 2023

Thu, 08 Jun 2023 09:00:00 -0700

Meet Matt Sandler, Sr. Director of Data and Analytics, McDonald’s

We’re excited to announce that Matt Sandler, Senior Director of Data and Analytics at McDonald’s, will be our featured customer speaker at the Databricks Data+AI Summit, hosted at the Moscone Center in San Francisco, CA, from June 26 - 29, 2023.

Join our featured customer session on June 29th at 12:30 PM, where you’ll hear Matt Sandler share how they use the Lightup Deep Data Quality platform to deploy pushdown data quality checks in minutes, not months — without developer support. From reactive to proactive, the McDonald’s data team leverages Lightup to scale their data quality checks across petabytes of data, ensuring high-quality data and reliable analytics for their products and services.

Which do you choose — data quality or data quantity? At McDonald’s, we choose both

Mon, 17 Apr 2023 09:00:00 -0700

Original Article Published on Medium

“Managing data quality is critical to enabling informed business decision making, but McDonald’s scale means massive amounts of data — that’s where our data-quality monitoring tool is helping us deliver consistent and accurate insights.” Keep reading Medium article here.

Data Quality Coverage and Readiness Part 2

Mon, 23 May 2022 09:00:00 -0700

Image by Juliana Castro , CC BY-SA 4.0, via Wikimedia Commons

In the first part of this series, we took a look at the different factors that impact your data quality and symptoms of “bad data”. We also learned that the actual percentages of all data an organization collects that is considered business-critical are actually a subset of the total amount of data the organization collects. Because this data is essential for effective decision-making, implementing a system of quality checks, which are applied throughout your data pipelines, should be a priority for any data-driven organization in order to ensure and maintain data quality.

Data Quality Coverage and Readiness Part 1

Tue, 17 May 2022 09:00:00 -0700

Photo by Mike Cohen on Flickr

For growth-minded organizations, the ability to effectively respond to market conditions, competitive pressures, and customer expectations is dependent on one key asset — Data. But having just massive troves of data isn’t enough. The key to being truly data-driven is having access to accurate, complete, and reliable data. In fact, Gartner recently found that organizations believe poor data quality to be responsible for an average of $15 million per year in losses — a figure that can cripple most companies. Unfortunately, ensuring — and maintaining — data quality can be incredibly difficult. This is being exacerbated by the data architecture choices of an organization. Legacy architectures often lack the ability to scale to support the ever-increasing volumes of real-time data and cause data silos that slow the necessary democratization of data needed for an entire organization to benefit from.

Data Observability: What's Changed and Why Now is the Time to Solve It

Sun, 01 Aug 2021 09:00:00 -0700

Photo by Markus Spiske from Pexels

Introduction

Why has data observability suddenly become such a hot topic?

What’s changed in the data stack that’s made observability a new, different, and solvable problem — and why can’t legacy tools seem to deal with it?

I recently answered these questions, and many more, in a far-ranging chat with TFiR Media’s CEO and host Swapnil Bhartiya.

To watch the full video interview, click here.

Or, read on for a full transcript that has been lightly edited for readability.