AI Anomaly Detection
Prebuilt AI Models That Detect Data Quality Issues at Scale.
Lightup provides prebuilt AI Anomaly Detection models, specifically designed to detect Data Quality issues at scale. Developed using advanced statistical processing and machine learning techniques, Lightup's AI Anomaly Detection has been field-tested and optimized based on real data from enterprise customers.
Ensure Data Quality Expectations
Using prebuilt Anomaly Detection models, Lightup pinpoints data outliers, reveals hidden trends, and identifies business-specific seasonality for real-time insights when data fails to meet Data Quality expectations.
Lightup’s Anomaly Detection includes:
- Automated Data Quality expectations
- Absolute or dynamic threshold setting
- Customizable settings to meet your desired outcomes
3 Advanced Anomaly Detection Algorithms
Proven effective and accurate by our enterprise customers, Lightup’s AI Anomaly Detection is supported by three advanced algorithms:
- Values Outside Expectations
- Sharp Change
- Slow Burn Trend Change
Lightup automatically selects the most suitable algorithm for each Data Quality Indicator (DQI) defined in the system, with options to customize the Anomaly Detection configurations to meet your organization’s desired outcome.
Algorithm 1: Values Outside Expectations
Algorithm 2: Sharp Change
Algorithm 3: Slow Burn Trend Change
This algorithm detects changes in long-term metric trends — very useful for early detection of trend changes that are usually caught too late because of their slow burn nature.
For example, if the number of users on the platform that used to grow at 1% week over week starts decaying 1% week over week, trend change would be the best algorithm to quickly spot this and take action.
Training Anomaly Detection Models
Lightup’s AI Anomaly Detection continuously evaluates the behavior of defined metrics, flagging outliers or abnormal patterns as “incidents.”
Unlike other Data Quality and Observability solutions that need to run for weeks or months to train Anomaly Detection models, Lightup uses historical data with correct Data Quality expectations to train models. To further optimize performance, Lightup includes backtesting and feedback loops, for flexible fine-tuning as you go.
For complex data elements, simply specify as many unique historical date ranges as needed to enhance seasonal awareness and accurate trend analysis.
Enhancing AI Anomaly Detection
Enabling Lightup’s AI Anomaly Detection for metrics is easy for technical and non-technical users — no statistical expertise or coding required.
The best part? Lightup is designed to make fine-tuning and supervising Anomaly Detection models simple, even for beginners.
Rule Preview
Lightup provides an intuitive backtesting and preview workflow that lets users quickly configure detection criteria without requiring statistical expertise.
Each rule can be backtested on historical data to assess performance before going live. Fine-tune results using simple settings like rule aggressiveness — more aggressive to catch more incidents, less to catch fewer.
Online Feedback
Users can pass online feedback to the Lightup system by rejecting Data Quality incidents the system incorrectly cites.
This feedback is used to:
Add supervised training to the Anomaly Detection model for improved accuracy over time.
Ensure ongoing detection accuracy with minimal maintenance for end users.
Help the system continuously adapt to your data’s evolving patterns.
Resources
Find hidden bad data, before stakeholders report it.
