Anomaly Detection Techniques for Real World Applications

Author

Reads 5.1K

Photo Of An Industrial Factory Emitting Smoke
Credit: pexels.com, Photo Of An Industrial Factory Emitting Smoke

Anomaly detection is a powerful tool that can help us identify unusual patterns in data. It's used in various real-world applications, including credit card transactions and medical diagnosis.

One common technique used in anomaly detection is statistical analysis. This involves using algorithms to identify data points that are significantly different from the norm. For instance, a credit card company might use statistical analysis to detect suspicious transactions that are outside the usual spending patterns of a customer.

Machine learning algorithms are also widely used in anomaly detection. These algorithms can learn from large datasets and identify patterns that are not immediately apparent to humans. In the medical field, machine learning algorithms can be used to detect unusual symptoms that may indicate a serious underlying condition.

In addition to statistical analysis and machine learning, other techniques such as rule-based systems and visualization are also used in anomaly detection. These techniques can help identify anomalies by applying predefined rules or visualizing data in a way that highlights unusual patterns.

Anomaly Detection Basics

Credit: youtube.com, Anomaly detection 101

Anomaly detection is a powerful tool for identifying unusual patterns in data. The process can be broken down into several key steps.

To start, you'll need to create an anomaly detection model using the CREATE SNOWFLAKE.ML.ANOMALY_DETECTION command. This command trains the model on the dataset you provide.

The inference step, which involves processing the input data, takes approximately 1 second to process 100 rows, regardless of the warehouse size. This makes it a relatively efficient process.

To use the anomaly detection model, you'll need to create a view or design a query that returns the data for analysis. For example, you can create a view named view_with_data_to_analyze that contains the date and sales information.

Once you have the data in a view, you can use the anomaly detection object to detect anomalies. This involves calling the DETECT_ANOMALIES method of the object, as shown in the example: CALL basic_model!DETECT_ANOMALIES(INPUT_DATA=>TABLE(view_with_data_to_analyze), TIMESTAMP_COLNAME=>'date', TARGET_COLNAME=>'sales');.

The DETECT_ANOMALIES method returns a table that includes rows for the data currently in the view, along with the prediction of the detector.

Data Preparation

Credit: youtube.com, Anomaly Detector v1.0 Best Practices

Data Preparation is a crucial step in anomaly detection, and it's where the magic happens. The quality of your data can make or break the accuracy of your model.

To prepare your data, you need to consider the input requirements for the ML.DETECT_ANOMALIES function, which depend on the input model type. This is where things can get tricky, so it's essential to get it right.

When dealing with missing data, you have several options to choose from. You can ignore missing data altogether, fill it with custom values, replace it with zeros, or use the previous values to maintain continuity. The choice ultimately depends on the nature of your data and the type of anomalies you're trying to detect.

Here are the options for handling missing data:

Remember, poor data quality leads to poor model performance, so it's essential to get your data in order before training your model.

On a similar theme: Ecod Outlier Detection

Data Preprocessing Methods

Autoencoder, k-means, or PCA models require input data with matching column names and compatible data types according to BigQuery implicit coercion rules.

Credit: youtube.com, How is data prepared for machine learning?

It's essential to ensure your data meets these requirements to get accurate results from these models. I've seen it happen where a small mistake in data formatting can lead to hours of wasted time trying to troubleshoot the issue.

If you used the TRANSFORM clause in the CREATE MODEL statement, only the input columns present in the TRANSFORM clause can appear in the query statement.

You can also set an imputation option to manage missing data in your streams. This is particularly useful when dealing with time-series data.

The imputation options include Ignore Missing Data, Fill with Custom Values, Fill with Zeros, and Use Previous Values. Each option has its own use case, such as detecting significant events like drops in event counts.

Here are the imputation options in a concise table:

Be cautious when imputing extensively missing data, as excessive gaps can compromise model accuracy.

On a similar theme: Automl Paper

Configuring Based on Aggregation

You can configure an anomaly detection model based on an aggregation method. This involves selecting the desired aggregation method for each feature, such as average(), count(), sum(), min(), or max(). The aggregation method determines what constitutes an anomaly.

Credit: youtube.com, Prepare the data: Data Aggregation

For example, if you choose min(), the detector focuses on finding anomalies based on the minimum values of your feature. If you choose average(), the detector finds anomalies based on the average values of your feature.

To configure an anomaly detection model based on an aggregation method, you can follow these steps:

  • Select the desired detector from the list on the Detectors page.
  • Select the Actions button to activate the dropdown menu and then select Edit model configuration.
  • Select the Add another feature button.
  • Enter a name in the Feature name field and select the Enable feature checkbox.
  • Select Field value from the dropdown menu under Find anomalies based on.
  • Select the desired aggregation from the dropdown menu under Aggregation method.
  • Select the desired field from the options listed in the dropdown menu under Field.
  • Select the Save changes button.

The maximum number of features for a detector is 5 by default. You can adjust this limit using the plugins.anomaly_detection.max_anomaly_features setting.

Setting Up Examples

To set up examples for data preparation, you'll need to create a table named new_sales_data that contains the sample dataset. This dataset includes daily sales for items in different stores along with daily weather data, such as humidity and temperature.

The data also contains a column that indicates whether the day is a holiday. To create this table, you can execute the following statements.

Here's a step-by-step guide to creating the table reference:

  1. Use the TABLE keyword to create a reference to the table.
  2. Pass along the caller's privileges by using the SYSTEM$REFERENCE or SYSTEM$QUERY_REFERENCE function.

Remember to use the table name, view name, or query to create the reference. This will ensure that you have access to the data you need for your analysis.

Setting Categorical Fields for High Cardinality

Credit: youtube.com, Featuring Engineering- Handle Categorical Features Many Categories(Count/Frequency Encoding)

Setting categorical fields for high cardinality can be a game-changer when it comes to anomaly detection. You can categorize anomalies based on a keyword or IP field type.

To set a category field, you need to enable the Categorical fields option and select a field. Once you've done this, you can't change the category fields after you create the detector.

A good rule of thumb is to use the following equation to calculate the recommended total number of entities supported in a cluster: actual total number of unique entities / (1 + number of category fields). If the actual total number of unique entities is higher than the number you calculate, the anomaly detector will attempt to model the extra entities.

The detector prioritizes both entities that occur more often and are more recent. This means that if you have a large number of unique entities, the detector will focus on the ones that are most relevant to your analysis.

Credit: youtube.com, Fletcher Riehl: Using Embedding Layers to Manage High Cardinality Categorical Data | PyData LA 2019

To give you a better idea, let's consider an example. Suppose you have a dataset with 2,429 unique IP addresses. If you try to use all of these IP addresses as category fields, the detector will attempt to model the extra entities. This can help you identify anomalies that might have been missed otherwise.

Here's a quick rundown of the key points to keep in mind:

Previewing Sample

Previewing sample anomalies is a great way to get a sense of what's going on in your data. This feature allows you to select a small number of data samples, for example, one data point every 30 minutes.

To use this feature, you can choose "Preview sample anomalies" and then select the "Next" button.

The Anomaly Detection plugin then uses interpolation to estimate the remaining data points, giving you an approximate view of the actual feature data. This helps you adjust the feature settings as needed.

You can load the sample dataset into the detector, which will then generate a preview of the anomalies.

Algorithms and Techniques

Credit: youtube.com, Anomaly Detection Algorithms and Techniques

The anomaly detection algorithm used by Snowflake is powered by a gradient boosting machine (GBM), which uses a differencing transformation to model data with a non-stationary trend. This algorithm can model data with a non-stationary trend and uses auto-regressive lags of the historical target data as model variables.

There are several techniques used in anomaly detection, including distance-based techniques, clustering-based techniques, density-based techniques, and classification-based techniques. These techniques can help identify anomalies in data by analyzing the relationships between data points.

Some common distance metrics used in distance-based techniques include Manhattan or Euclidian distance, but Euclidian distance can be problematic in high-dimensional data sets. To address this issue, a fast distributed anomaly detection strategy can be used, which is extremely scalable and takes the sparseness of the data set into consideration.

Here is a list of some dimensionality reduction methods:

  • PCA (Principal Component Analysis)
  • MDS (Multidimensional Scaling)
  • Karhunen–Loeve Transform
  • Local Linear Embedding
  • Laplacian Eigenmaps
  • Diffusion Maps

These methods can be used to reduce the dimensionality of high-dimensional data sets, making it easier to identify anomalies.

Classification-Based Techniques

Credit: youtube.com, All Learning Algorithms Explained in 14 Minutes

Classification-Based Techniques are a type of anomaly detection method that rely on machine learning algorithms to identify patterns in data and classify them as normal or anomalous.

You can use Snowflake's Anomaly Detection functions within Snowflake Tasks or Alerts to automate anomaly detection, both for retraining the model and for monitoring your data for anomalies. This allows you to create an automated anomaly detection pipeline.

The ARIMA_PLUS model is an example of a classification-based technique that uses a custom anomaly_prob_threshold value to detect anomalies. For example, if you set the anomaly_prob_threshold value to 0.8, the model will detect anomalies that are more likely to occur.

Classification-based techniques can be more accurate than other methods, especially when dealing with complex data. However, they require a large amount of training data to learn the patterns and relationships in the data.

You can specify the prediction interval for anomaly detection by creating an OBJECT that contains configuration settings for the model. This allows you to control the sensitivity of the model and detect anomalies at different levels.

Credit: youtube.com, Classification and Regression in Machine Learning

Here is an example of how to specify the prediction interval:

Classification-based techniques can be used in conjunction with other methods, such as threshold-based rules, to improve the accuracy of anomaly detection. For example, you can use a classification-based technique to identify potential anomalies, and then use threshold-based rules to filter out false positives.

Clustering-Based Techniques

Clustering is an important technique of data analysis that can handle multidimensionality and changing densities.

The shared nearest neighbor clustering algorithm, introduced by Ertoz et al., automatically calculates the number of clusters and recognizes core points to construct clusters around them.

This algorithm redefines the similarity among pairs of data instances in terms of the number of nearest neighbors they share, which diminishes issues with differing densities and increasing dimensions.

The shared nearest neighbor definition of similarity allows the method to process large data sets efficiently.

The algorithm initially recognizes the nearest neighbors of each data instance, and then uses the similarity to construct clusters around core points.

This technique is particularly useful for identifying clusters of fluctuating shapes, sizes, and densities, along with anomalies.

Density-Based Techniques

Credit: youtube.com, Clustering with DBSCAN, Clearly Explained!!!

Density-based techniques are particularly useful for identifying dense localities in data spaces. They're great for finding clusters and anomalies in data.

These techniques focus on regions of lower object density, which can be a challenge when dealing with high-dimensional data. The curse of dimensionality makes it harder to find relevant data points as data points are scattered through a large volume.

Chen et al. introduced a density estimator to bypass the curse of dimensionality by assuming the probability of data μ lies around a low-dimensional subset embedded in a high-dimensional space. Their estimator is based on a geometric multiscale decomposition of the given data.

This approach controls the overall model complexity and has strong finite sample performance bounds. Chen et al. proved these bounds for various models and target probability measures.

The techniques implementing this construction are fast and parallelizable, making them efficient for large data sets.

Understanding Feature Importance

Feature importance scores are an approximate ranking of the features in your trained model, calculated by counting the number of times the model's trees used each feature to make a decision.

Credit: youtube.com, How to find Feature Importance in your model

These scores are then normalized to values between 0 and 1 so that their sum is 1.

Features that are close in score have similar importance. For example, if one feature is quantity of items sold and another is quantity of items in inventory, the values may be correlated because you can't sell more than you have.

If multiple features are very similar to each other, using them may result in reduced importance scores for those features.

Feature importance also reports lag features, which are the value of the target variable x time units ago. For example, if your data is inferred to be hourly, lag24 represents your target variable 24 hours ago.

All other transformations of your target variable (such as rolling averages) are summarized as aggregated_endogenous_features in the results table.

You cannot choose the technique used to calculate feature importance, so you'll have to work with what you get.

Feature importance scores can be helpful for gaining intuition about which features are important to your model's accuracy, but the actual values should be considered estimates.

Ensemble Strategies

Credit: youtube.com, Tutorial 42 - Ensemble: What is Bagging (Bootstrap Aggregation)?

Ensemble strategies are used to improve the efficiency of anomaly detection techniques.

Arning et al. proposed a linear algorithm using a dissimilarity function to capture the similarity rate of a data point to solve the sequential exception problem in large databases.

More and Hall proposed an algorithm to group large-scale datasets without clustering entire data in a single instance.

Erfani et al. introduced an unsupervised technique for high-dimensional large-scale unlabeled data sets to detect anomalies that are a combination of a deep belief network and one-class support vector machines.

One-class SVMs are used for detecting outliers through unsupervised learning and aim to model the underlying distribution of data while not considering irrelevant attributes or outliers in the training records.

Features derived from training samples are taken as input to train 1SVMs.

A DBN is a multiclass semi-supervised approach and dimensionality reduction tool that uses multilayer generative models to learn one layer of features at a time from unlabeled data.

Credit: youtube.com, Ensemble Techniques for High Performance Machine Learning - Gilberto Titericz (Ople)

Camacho et al. presented an interesting outline for anomaly detection, identifying and interpreting unusual events in a forensics system network by addressing the 4Vs of big data—volume, variety, veracity, and velocity.

Their framework is based on input from several heterogeneous data sources and follows an exponentially weighted moving average method and uses a PCA model.

The two popular ensemble learning techniques are Bagging and Boosting.

Bagging is used for variance reduction and was first introduced by Breiman.

Boosting was first proposed by Schapire to strengthen the ability of weak learners to achieve arbitrarily high accuracy.

Oza and Russell proposed two online variations of bagging and boosting for data streams.

They showed how the training data could be simulated from the data stream perspective by replicating the sampling bootstrap process.

Bifet et al. developed a model for examining concept drift in data streams with two new adaptations of bagging: ADWIN bagging and adaptive-size Hoeffding tree bagging.

The Hoeffding tree is a technique that can learn from data streams that does not change over time and is an incremental algorithm based on decision tree induction.

High Dimensionality Problem

Credit: youtube.com, Anomaly detection for multivariate high-dimensional data by Mia Hubert

High dimensionality refers to data sets with a large number of independent variables, components, features, or attributes, making data analysis complex and requiring more sophisticated methods.

Data sets are growing in terms of sample size, but even more so in terms of dimension, with thousands of dimensions being common in the era of big data. This can lead to the curse of dimensionality, where data size increases proportionally with dimensions, resulting in data sparsity and difficulty in analysis.

The curse of dimensionality affects anomaly detection techniques, obscuring or concealing abnormal data instances due to unnecessary attributes, and making it challenging to find global optima in high-dimensional spaces.

PCA

High dimensionality can be a challenge in data analysis, but there are strategies to tackle it. One of the main sources of high-dimensional data sets are organizations with huge transactions and associated databases.

Data sets with thousands of dimensions are common, especially in the era of big data. A data set with n samples and m dimensions can be referred to as m-dimensional, and it's considered high-dimensional when the number of dimensions m causes the effect of the 'curse of dimensionality'.

Credit: youtube.com, StatQuest: PCA main ideas in only 5 minutes!!!

High-dimensional data is often referred to as multi-dimensional or multi-aspect or multi-modal data throughout the literature. The complexity of the data analysis increases with respect to the number of dimensions, requiring more sophisticated methods to process the data.

To address the problem of high dimensionality, one way is to reduce the dimensionality by projecting the whole data set into a lower dimensional space. Many dimensionality reduction methods have been proposed, including PCA, MDS, Karhunen–Loeve Transform, local linear embedding, Laplacian Eigenmaps, and diffusion maps.

The intrinsic dimensionality of data representation is often smaller than the available relevant dimensions, and many techniques have been proposed to measure it. This is particularly useful in application areas where data representations consist of dimensions that are dependent on each other.

High Dimensionality Problem

High dimensionality refers to data sets with a large number of independent variables, components, features, or attributes, which increases the complexity of data analysis.

Credit: youtube.com, Solving High Dimensional Problems Using Monte Carlo

Data sets are growing in terms of sample size, but even more so in terms of dimension, with thousands of dimensions being common in big data. This can make data points scattered and isolated, making it elusive to find the global optima of the data set.

The curse of dimensionality affects anomaly detection techniques, obscuring or concealing the level of abnormal nature with respect to increasing dimensions. Outliers are defined as data instances in sparse regions, and an inadequate discriminative region is observed in almost equally sparse locations of high-dimensional space.

High-dimensional data requires significant computational memory and brings a huge computational burden. Recognizing useful insights or patterns becomes complex and challenging, making it essential to understand the subtle difference between intrinsic and embedding dimensionality.

The simplest approach to handle high dimensionality is to minimize features, which can be better understood by studying either the intrinsic or embedding dimensionality of data sets.

Implementation and Management

Credit: youtube.com, Complete Anomaly Detection Tutorials Machine Learning And Its Types With Implementation | Krish Naik

You can manage your anomaly detection detectors by going to the Detectors page, where you can choose the detector name and select Start real-time detectors, Stop real-time detectors, or Delete detectors.

To start or stop a detector, simply choose Actions and select the desired option. This will allow you to control the flow of data and ensure your detectors are running smoothly.

Here's a quick rundown of the steps to manage your detectors:

  1. Choose the detector name.
  2. Choose Actions and select Start real-time detectors, Stop real-time detectors, or Delete detectors.

For more advanced monitoring, you can use Snowflake Tasks to monitor your data at a given frequency. This involves creating a table to hold the results of anomaly detection and setting up a task to store the results of a recurring anomaly detection operation in the table.

Manage Your Detectors

Managing your detectors is a crucial part of implementing and managing anomaly detection. You can start, stop, or delete a detector by going to the Detectors page.

To start a detector, choose the detector name and select Start real-time detectors. This will begin the detector's real-time analysis. You can also stop a detector by choosing Stop real-time detectors, which will pause the analysis. If you want to delete a detector, choose Delete detectors.

Credit: youtube.com, Alarm Management 101: Everything You Want to Know, But Are Afraid to Ask

You can manage multiple detectors simultaneously, with each analyzing data from different sources. This allows you to customize your anomaly detection pipeline to suit your specific needs. By managing your detectors effectively, you can ensure that your anomaly detection system is running smoothly and accurately detecting anomalies in your data.

To start a detector to find anomalies in your data in near real time, select Start real-time detector automatically (recommended). Alternatively, if you want to perform historical analysis and find patterns in longer historical data windows, select the Run historical analysis detection box and select a date range of at least 128 detection intervals.

Here's a summary of the detector management options:

By following these steps and understanding the detector management options, you can effectively manage your detectors and ensure that your anomaly detection system is running smoothly and accurately detecting anomalies in your data.

Arima Plus Default Settings

The ARIMA_PLUS model can be used to detect anomalies in a time series without specifying certain settings.

Intricate dashboard featuring digital displays and wiring, suggesting a high-tech or futuristic vehicle interior.
Credit: pexels.com, Intricate dashboard featuring digital displays and wiring, suggesting a high-tech or futuristic vehicle interior.

If you don't specify the anomaly_prob_threshold argument, the model will still work, but you'll get results similar to the ones you'd get if you had set the DECOMPOSE_TIME_SERIES training option to TRUE.

The default value of DECOMPOSE_TIME_SERIES is TRUE, which means the model will automatically decompose the time series.

This can be useful if you're not sure what settings to use or if you want to see how the model performs with its default settings.

Database Management

Database Management is a crucial aspect of any implementation, and Snowflake's anomaly detection feature makes it easy to manage your models. You can view a list of your models by using the SHOW SNOWFLAKE.ML.ANOMALY_DETECTION command.

To remove a model, use the DROP SNOWFLAKE.ML.ANOMALY_DETECTION command, which is a straightforward process that doesn't require any complex steps. Deleting a model is a simple way to manage your database and keep it organized.

Models are immutable in Snowflake, which means they cannot be updated in place. To update a model, you'll need to delete it and train a new one, which might seem like a hassle, but it's actually a good practice to keep your models fresh and up-to-date.

Here are some common database management tasks related to anomaly detection:

  • View models: SHOW SNOWFLAKE.ML.ANOMALY_DETECTION
  • Remove models: DROP SNOWFLAKE.ML.ANOMALY_DETECTION
  • Update models: Delete and train a new one

Granting Object Creation Privileges

Illuminated pedestrian and bicycle exception sign at night in an urban setting.
Credit: pexels.com, Illuminated pedestrian and bicycle exception sign at night in an urban setting.

Granting Object Creation Privileges is a crucial step in implementing and managing anomaly detection models in Snowflake.

To create anomaly detection models, you need a role with the CREATE SNOWFLAKE.ML.ANOMALY_DETECTION privilege on the schema where the model is created.

You can create a specific role, like the analyst role, for users who need to detect anomalies.

The admin role can own the schema and grant the analyst role the necessary privileges.

If the analyst role has CREATE SCHEMA privileges in a database, it can create a new schema and create anomaly detection models in that schema.

To revoke a role's model creation privilege on the schema, use the REVOKE command.

For complete details about the SNOWFLAKE.ML.ANOMALY_DETECTION constructor, see the ANOMALY_DETECTION (SNOWFLAKE.ML) documentation.

Curious to learn more? Check out: Unsupervised Anomaly Detection

Frequently Asked Questions

What are the three types of anomaly detection?

There are three main types of anomaly detection techniques: unsupervised, semi-supervised, and supervised. Each type uses different approaches to identify unusual patterns in data.

What are the three 3 basic approaches to anomaly detection?

There are three primary approaches to anomaly detection: Unsupervised Clustering, Supervised Classification, and Semi-supervised Detection, each with its unique strengths and applications. Understanding these approaches is crucial for identifying and addressing anomalies in various data sets.

What is an example of anomaly detection system?

Anomaly detection systems identify unusual patterns, such as large purchases in a new location, to detect potential issues like credit card fraud. This is achieved through real-time algorithms that alert banks to review suspicious activity.

Carrie Chambers

Senior Writer

Carrie Chambers is a seasoned blogger with years of experience in writing about a variety of topics. She is passionate about sharing her knowledge and insights with others, and her writing style is engaging, informative and thought-provoking. Carrie's blog covers a wide range of subjects, from travel and lifestyle to health and wellness.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.