
Expectation Propagation is a powerful tool for machine learning, particularly in handling complex probabilistic models. It's a method that's been around since the early 2000s.
The core idea behind Expectation Propagation is to update the parameters of a probabilistic model by iteratively refining the model's parameters. This is done by propagating the expectations of the model's variables through the model's structure.
The Expectation Propagation algorithm is based on the concept of message passing, where each node in the model sends messages to its neighbors. This allows the model to iteratively refine its parameters until convergence.
Expectation Propagation has been widely used in various machine learning applications, including Bayesian inference and probabilistic graphical models.
Check this out: Right to Explanation
Expectation Propagation Basics
Expectation propagation is a method that's often used in machine learning workshops, like the one on Hulu's internal decks. It's a way to approximate complex distributions.
To get started with expectation propagation, you need to choose some natural parameters for the prior distribution. This involves selecting values for \(\boldsymbol{\mathbf{r}}_0\) and \(\boldsymbol{\mathbf{Q}}_0\).
Here are the steps to initialize expectation propagation:
- Choose natural parameters for the prior distribution \(\boldsymbol{\mathbf{r}}_0\) and \(\boldsymbol{\mathbf{Q}}_0\).
- For every site \(k=1, \ldots, K\) choose the parameters \(\boldsymbol{\mathbf{r}}_k\) and \(\boldsymbol{\mathbf{Q}}_k\).
- Compute the natural parameters of the global approximation \(\boldsymbol{\mathbf{r}}\leftarrow \sum_{k=0}^K \boldsymbol{\mathbf{r}}_k \qquad \text{and} \qquad \boldsymbol{\mathbf{Q}}\leftarrow \sum_{k=0}^K \boldsymbol{\mathbf{Q}}_k\).
Initialization
To initialize the Expectation Propagation algorithm, you need to choose natural parameters for the prior distribution. This involves selecting values for the parameters r0 and Q0.
First, choose natural parameters for the prior distribution r0 and Q0. These values will serve as the foundation for the rest of the algorithm.
Next, for every site k=1, ..., K, choose the parameters r_k and Q_k. This step is crucial in defining the local approximations that will be used in the Expectation Propagation algorithm.
You can think of the local approximations as building blocks that will be combined to form the global approximation. To do this, compute the natural parameters of the global approximation by summing the local parameters: r ← ∑k=0^K r_k and Q ← ∑k=0^K Q_k.
Here's a summary of the initialization steps:
- Choose natural parameters for the prior distribution r0 and Q0.
- For every site k=1, ..., K, choose the parameters r_k and Q_k.
- Compute the natural parameters of the global approximation by summing the local parameters.
Expectation Propagation
Expectation propagation is a method that has been used in internal machine learning workshops, such as one held by Hulu.
The expectation propagation method is introduced in a specific deck, which covers its background, theory, and application.
This method is used to introduce the background, theory, and application of expectation propagation, indicating its importance in machine learning.
Expectation propagation can be found in a specific deck used for a Hulu internal machine learning workshop.
Related Concepts
Expectation propagation is closely related to other concepts in probabilistic modeling and inference.
Loopy Belief Propagation is a type of message passing algorithm that's similar to expectation propagation, but it doesn't require a tree-like structure.
Expectation propagation can be seen as a generalization of the Laplace approximation, which is a method for approximating the posterior distribution of a model.
The mean field approximation is another technique that's related to expectation propagation, as it also involves approximating the posterior distribution by a simpler distribution.
Loopy Belief Propagation and expectation propagation can be used to perform inference in models with loops, which are common in many real-world applications.
Mathematical Framework
Expectation propagation is a powerful algorithm for approximating intractable posterior distributions. It works by iteratively updating a set of approximate distributions, called factors, to match the true posterior.
In the context of expectation propagation, we're often dealing with a time series of observations, and we want to update our approximate prior with each new measurement. This is where the concept of assumed density filtering (ADF) comes in. ADF approximates the posteriors by matching the moments of the individual terms in a product approximation.
The key idea behind ADF is to update the approximate prior with the new measurement, and then seek an approximation by computing the next-step predictive distribution. This is done by minimizing the Kullback-Leibler (KL) divergence between the true posterior and the approximate distribution.
The KL divergence is a measure of how different two distributions are. It's defined as the integral of the product of the true posterior and the logarithm of the ratio of the true posterior to the approximate distribution, plus the integral of the difference between the true posterior and the approximate distribution.
In the case of Gaussians, this minimization is equivalent to moment matching, also known as weak marginalization. This is a common case in expectation propagation.
Here's a summary of the ADF update process:
- Initialize the approximate prior
- Repeat until convergence:
+ Update the approximate prior with the new measurement
+ Compute the next-step predictive distribution
+ Minimize the KL divergence between the true posterior and the approximate distribution
By iteratively updating the approximate prior with each new measurement, we can obtain a good approximation of the true posterior distribution.
Updates and Messages
EP can be viewed as a message passing algorithm, where cavity q^{\backslash i} is a message carrying information about \theta to the factor i. This message is denoted by m_{\theta\rightarrow f_{i}}(\theta).
In the KL objective, one can replace f_{i}(X_{0}|\theta) with \int dX\, m_{X\rightarrow f_{i}}(X)f_{i}(X|\theta), where m_{X\rightarrow f_{i}}(X) is a message from X carrying information about X to the factor f_{i}. This message can indicate that the distribution of X is a point mass at X_{0}.
A factor update can be rewritten as m_{f_{i}\rightarrow\theta}(\theta)=\frac{\text{proj}\left[\int dX\, f(X|\theta)m_{X\rightarrow f_{i}}(X)m_{\theta\rightarrow f_{i}}(\theta)\right]}{m_{\theta\rightarrow f_{i}}(\theta)}.
Gaussian Updates
Gaussian Updates are a type of update that can be used to improve the accuracy of a model. They work by adjusting the mean and standard deviation of the data.
In a Gaussian Update, the new mean is calculated as the weighted average of the old mean and the new observation. The weight given to the old mean is determined by the variance of the old mean.
The variance of the old mean is calculated as the inverse of the learning rate. A higher learning rate means a lower variance, which can lead to faster convergence but also more oscillations.
In practice, Gaussian Updates can be used to adapt to changing data distributions. For example, if the data distribution shifts over time, a Gaussian Update can help the model to adjust to the new distribution.
Message View
In the message view of EP, we can think of the cavity as a message carrying information about the variable to the factor. This message is denoted by \(m_{\theta\rightarrow f_{i}}(\theta)\).

The cavity can be treated as a message from the variable to the factor, denoted by \(m_{f_{i}\rightarrow\theta}(\theta)\). This is the message that is used to update the factor.
In the KL objective, we can replace the factor with an integral over the messages from the variables to the factor. This gives us a new way of thinking about the factor update.
The factor update can be rewritten as \(m_{f_{i}\rightarrow\theta}(\theta)=\frac{\text{proj}\left[\int dX\, f(X|\theta)m_{X\rightarrow f_{i}}(X)m_{\theta\rightarrow f_{i}}(\theta)\right]}{m_{\theta\rightarrow f_{i}}(\theta)}\).
Here are the key differences between EP and BP:
In EP, we need to gather a message from the variable to the factor to compute the outgoing message. This gives context to the global posterior and allows us to divide it out after the projection.
EP Update Sequence
In the EP update sequence, we start with a prior distribution q(θ) = N(θ|m0, v0). The key takeaway here is that the prior is a normal distribution with mean m0 and variance v0.
We then have a set of functions fi that we're trying to approximate. The expectation propagation algorithm updates the distribution q(θ) iteratively until convergence. This involves making several passes over the functions fi.
Here are the key steps involved in the EP update sequence:
- Start with a prior distribution q(θ) = N(θ|m0, v0)
- Repeat EP iterations until convergence (make several passes over 1, ..., m)
The algorithm updates the distribution q(θ) at each iteration, and we can use the updated distribution to make predictions or perform inference.
6 Incoming
Expectation Propagation (EP) is a popular algorithm for approximating posterior distributions, and it's known to give good approximations at a low computational cost.
EP works particularly well for hierarchical models, where it focuses on the shared parameters (hyperparameters) of the model.
The algorithm partitions big datasets into many parts and analyzes each part in parallel to perform inference of shared parameters.
EP iteratively approximates the moments of the tilted distributions and incorporates those approximations into a global posterior approximation.
This process can be made parallel with occasional exchanging of information between processes through the global posterior approximation.

Moments of multivariate tilted distributions can be approximated in various ways, including MCMC, Laplace approximations, and importance sampling.
EP can be used to divide the computation for large models into manageable sizes, making it a scalable solution.
The central idea of EP is to work at each step with a "tilted distribution" that combines the likelihood for a part of the data with the "cavity distribution", which is the approximate model for the prior and all other parts of the data.
EP is exact in the large data limit, as shown by Guillaume Dehaene, and this has implications for its practical use and implementation.
EP Update Sequence
The EP update sequence is a crucial step in the EP algorithm, and it's essential to understand how it works. It starts with a prior distribution \(q(\theta) = \mathcal{N}(\theta|m_{0},v_{0})\), which is a normal distribution with mean \(m_{0}\) and variance \(v_{0}\).
The first step in the EP update sequence is to set the message from the factor to the latent variable \(\theta\) to 1, denoted as \(m_{f_{i}\rightarrow\theta}(\theta) = \tilde{f}_{i}(\theta) = 1\) for all \(i\). This is a key assumption that simplifies the update process.

To perform the EP update sequence, you need to repeat the iterations until convergence, which means making several passes over the factors \(1, \ldots, m\). This process helps to refine the posterior distribution of the latent variable \(\theta\).
Here's a summary of the EP update sequence:
Featured Images: pexels.com





