Gaussian Mixture Models (GMM)

Suppose we want to classify coffee beans by flavor. We could divide them into a few types: beans with pronounced acidity, beans with a balance of sweetness and body, and beans with a rich, heavy flavor.

But not every bean fits neatly into a single type. Some beans have strong acidity and strong sweetness, showing the features of two types at once.

Methods like K-Means, which assign each observation to one cluster, struggle to capture such in-between characteristics. A Gaussian mixture model (GMM) represents each cluster as a probability distribution and estimates the relative probability that an observation belongs to each cluster. That makes it useful for data in which the boundaries between clusters aren’t sharp.

In this lesson we’ll look at the basic structure of GMM, membership probabilities and the EM algorithm, covariance settings, choosing the number of clusters, interpreting an actual results table, how it differs from K-Means and what to watch out for in the analysis.

Each normal distribution represents one latent cluster, and each cluster can have its own center, spread and relationships between variables.

For example, if we plot beans on two axes, acidity and body, one cluster may be spread around high acidity and low body and another around low acidity and high body. A bean in the region where the two distributions overlap could belong to either cluster.

A GMM estimates each cluster’s distribution and then computes membership probabilities for each bean. You can designate the cluster with the highest probability as the bean’s main cluster, but a key feature is that the possibility of belonging to other clusters is kept as well.

A GMM, in contrast, computes the likelihood of membership using each cluster’s probability distribution.

Note, though, that if you pick only the cluster with the highest membership probability in a GMM, you can end up with a single cluster label after all. And a GMM doesn’t find every irregularly shaped cluster well. For long, curved shapes or clusters connected in complex ways, density-based methods may be more appropriate.

Trained tasters rated four flavor attributes on a 0–10 scale using the same criteria. The team wants to find clusters that summarize the beans’ characteristics and see how similar a new bean is to the existing types.

We assume every bean has values for all four variables and check the variables’ distributions and correlations before the analysis. In reality a bean’s flavor is influenced by its variety, origin, roasting method and so on, but here we explore types using the flavor scores alone. So we don’t interpret the clusters as a definitive classification by origin or variety.

Here K is the number of mixture components, π_k is the mixing proportion of component k, μ_k is its mean vector, and Σ_k is its covariance matrix. The mixing proportions are non-negative and sum to 1.

For example, if the mixing proportions of three components are 0.40, 0.35 and 0.25, the overall distribution is explained as a mixture of three normal distributions. The mixing proportions themselves aren’t necessarily equal to the shares of observations finally assigned in the actual sample.

In particular, if you assign each observation to its most probable cluster, the count per component can differ from the expected count calculated from the mixing proportions.

This is called the responsibility, or membership probability. For example, suppose bean A’s membership probabilities came out as follows.

Bean A is most likely to belong to cluster 1, but it can be seen as sharing some of cluster 2’s characteristics too. Even if its main cluster is shown as cluster 1, giving the membership probability of 0.68 alongside lets you see how clear-cut the classification is.

But this probability is a model-based probability, conditional on the chosen GMM and its estimated parameters being appropriate. It shouldn’t be taken to mean that we are 68% sure of the bean’s true type.

EM alternates between estimating the latent component memberships of the observations and updating the parameters of each component.

First, initial means, covariances and mixing proportions are set. In the E-step (Expectation), the current parameters are used to compute every observation’s membership probability for each component. In the M-step (Maximization), those probabilities are used as weights to re-estimate each component’s mean, covariance and mixing proportion.

These two steps are repeated until a convergence criterion is met, such as the increase in log-likelihood becoming small enough. However, because EM can converge to different local optima depending on the starting values, it’s good to run it from several initial values and check that the results are stable.

For example, if the relationship between acidity and sweetness differs from cluster to cluster, the full setting can represent this more flexibly. But it has many more parameters to estimate, so it can be unstable when the sample is small.

So rather than always choosing full, consider the size of the data and number of variables, the stability of the covariance matrices, information criteria and interpretability together.

And with other data, the units of measurement themselves may differ, for example acidity on a 0–10 scale and an aroma-related measure on a 0–100 scale.

Because the estimated means, covariances and membership probabilities of a GMM can change with the scales of the variables, check the scales and distributions before the analysis. If needed, you can standardize the variables to mean 0 and standard deviation 1.

Standardizing isn’t always the right answer, though. If a particular variable genuinely carries more important information, or the unit of measurement itself is meaningful, it may be better to keep that structure. So decide whether to standardize based on the purpose of the analysis, and report it.

Commonly, candidate models from 1 up to several components are fitted and information criteria such as AIC or BIC are compared. Information criteria take both the model’s fit and its complexity into account. With the same data and comparable likelihood definitions, lower values are generally preferred.

In these results the three-component model has the lowest BIC, so it is a strong candidate. But the lowest BIC alone doesn’t prove that three real groups exist. You also need to review stability across initializations, the size and interpretability of each component, the prediction goal and so on.

Assigned counts are based on labels set by the maximum membership probability.

Cluster 1 can be interpreted as a type with high acidity and weak body and roast aroma, and cluster 2 as a type with high sweetness and fairly balanced characteristics. Cluster 3 is a type with strong body and roast aroma.

These names aren’t objective diagnoses the model assigned automatically. They are interpretive labels the analyst gave after reading the variables’ characteristics. Also, the means in this table are calculated from the beans finally assigned to each cluster, so they may not match the model’s component mean vectors exactly.

For example, one bean may have a clear highest membership probability of 0.95, while another shows similar probabilities for two components, such as 0.43 and 0.40.

Bean B’s main cluster is 1, but its chance of belonging to cluster 2 is about as high. In cases like this, presenting only a single label loses the information that it shares the features of two types.

So in a results report or an actual classification system, it’s good to provide the maximum membership probability, or all membership probabilities, along with the main cluster. However, you shouldn’t conclude that an observation is a data error or an outlier just because its membership probability is low.

Because a GMM assumes a mixture of normal distributions, it can be inappropriate for complex curved structures or highly non-normal clusters. Review each cluster’s distribution, residual-type goodness-of-fit diagnostics and visualizations together.

Watch out for initial values and local optima.

The EM algorithm can converge to different results depending on the starting values. It’s good to estimate from several initial values and check how stable the log-likelihood and cluster composition are.

Don’t assume the number of components equals the number of real groups.

Sometimes a single complex group is approximated by several normal components. So mixture components are units of statistical description and don’t always correspond to clearly defined real groups.

Check the stability of the covariance matrix estimates.

If very few observations belong to a component or there are too many variables, covariance estimation can become unstable. You can consider regularization options, reducing the number of variables, adjusting the number of components and so on.

Don’t interpret clustering results as causal relationships or official quality grades.

Forming clusters by flavor doesn’t prove that a particular roasting method caused that flavor. And cluster labels are interpretive names given according to the research goal.

In actual research you should also present the covariance type used, the number of initializations and whether the model converged, the mixing proportions, the parameters of each component and the results of stability checks.

Its distinctive feature is that, beyond assigning an observation to one cluster, it lets you see how far the observation straddles two or more clusters.

In a GMM, the EM algorithm estimates the mixing proportions, means and covariances, and the model is chosen considering AIC and BIC, stability across initializations, interpretability of each cluster and so on. Because the results depend on the normality assumption, the number of components and the covariance setting, the analysis settings and limitations should be reported too.