Discriminant Analysis (LDA/QDA)

If you know a wine's acidity and sweetness, can you guess where it was produced? Would comparing alcohol content as well separate the regions better? And when wines from two regions have similar characteristics, what criterion should you use to classify them?

For questions like these, the statistical classification method that decides which group an observation belongs to based on the values of several variables is discriminant analysis. The two best-known forms are linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA).

LDA assumes the groups share the same covariance structure and separates them with lines or planes, while QDA allows a different covariance structure for each group and produces curved decision boundaries. Like logistic regression, it is used to classify categories, but it makes different assumptions about the distribution of the predictors within each group.

Discriminant analysis is used in quality control, biological classification, identifying customer types, supporting medical diagnosis and more. In this article, using an example that classifies a wine's region of production from its chemical characteristics, we look at how discriminant analysis works, the difference between LDA and QDA, how to interpret the results table and decision boundaries, performance evaluation, and cautions for the analysis.

Discriminant analysis starts from training data in which each observation's group label is known. For example, from samples whose region of production has already been confirmed, it learns how acidity, sweetness and alcohol content are distributed in each region. Then, when you enter the measurements of a new wine, it classifies which region the wine most likely belongs to.

LDA and QDA are both generative classification models that assume the numeric predictors in each group follow a multivariate normal distribution. The key difference is whether the groups share one covariance matrix (LDA) or each has its own estimated covariance matrix (QDA).

Comparing acidity alone, the distributions of Regions A and B may overlap, and sweetness alone may not separate them easily either. Observed together with alcohol content, however, the differences between groups can become clearer. Discriminant analysis combines this multivariate information to build a classification rule.

Explaining differences between groups and correctly predicting new observations are different goals. Even if the groups are well separated in the training data, accuracy on new wines can be low, so classification performance must always be checked on separate test data or with cross-validation.

We use 300 hypothetical bottles of wine collected from three production regions (A, B, C). We obtained 100 bottles from each region, and the region of production is already confirmed. The data and performance figures below are hypothetical examples for explaining the concept, not results of actually running a model.

Assume the wines were split into 240 bottles of training data and 60 bottles of final test data. In the example evaluation below, the split was stratified so that each region contributes 20 bottles. In real research, you should also report the sample composition after splitting and the evaluation procedure.

LDA can be more stable than QDA with small samples, but you cannot decide in advance which method is better. Consider the number of features and the amount of data per group, and if needed consider covariance regularization or shrinkage estimation.

① Define the target groups and predictors

Set production regions A, B and C as the target and use total acidity, residual sugar and alcohol content. Exclude leakage variables such as IDs that hint at the region.

② Split training and test data

Of the 300 bottles, use 240 for training and 60 for the final evaluation. Model selection is done by cross-validation within the training data.

③ Fit LDA and QDA separately

LDA estimates a shared covariance matrix and QDA estimates a covariance matrix for each group. If needed, choose a regularization parameter within the training data.

④ Compare classification results and error patterns

Check not only accuracy but also recall and precision by region and the confusion matrix, then evaluate the chosen model once on the final test data.

Suppose a new wine measures 6.5 g/L total acidity, 2.8 g/L residual sugar and 12.9% alcohol. The model computes the relative likelihood of the three regions. The values below are examples.

This wine is classified as Region B. 0.73 is an estimated probability under the chosen model and prior-probability assumptions; it does not guarantee that the probability the wine actually comes from B is always 73%. If needed, check the probability calibration separately.

Suppose we obtained the following hypothetical confusion matrix for the 60 test bottles. Rows are the actual production region and columns are the predicted region.

Dividing the 50 correctly classified bottles on the diagonal by all 60 bottles gives an accuracy of 83.3%. Region B's recall is 16/20 = 80%, and its precision is also 16/20 = 80%. With other data, a model can keep missing a particular region even when overall accuracy is high, so check the per-region metrics too.

In the hypothetical results, QDA is slightly higher in cross-validation within the training data, but LDA is higher on the final test data. Such differences alone do not let you conclude that either method is inherently better, and with small samples you should also consider the uncertainty in the score difference. The confusion matrix in Section 8 is the LDA final evaluation.

The decision boundary mockup is meant to explain the principle; it is not a 2D projection that exactly reproduces a model trained on the three actual variables. When data have three or more dimensions, some information can be lost in a 2D visualization.

Is the assumption of a normal distribution within each group appropriate?

Strong skewness or extreme values affect estimates of means and covariances. Look at the distributions and consider comparing with other classification models.

Is the sample large enough to use QDA?

If there are few observations per group relative to the number of features, covariance matrices are hard to estimate stably. Consider reducing the number of variables or using regularization.

Are you using the test data for model selection?

If you repeatedly check the final test data and pick the model with the higher score, that data is no longer an independent test set. Select models by cross-validation within the training data.

Do the regional sample proportions match reality?

When the training data have equal sample sizes per region but real production shares differ in the field, prior probabilities and classification performance can change. Check the group proportions in the actual deployment setting.

Are you interpreting classification probabilities as definite facts?

Estimated probabilities depend on the assumptions and the training data. Check how reliable the classification probabilities are and whether performance holds up when the data change.

Linear discriminant analysis is a classification model that also provides discriminant axes for dimension reduction. Unlike PCA, however, it does not look for axes with large total variance but for directions that separate the groups well.

In the report, present the target groups and predictors, sample size per group, preprocessing and how training and test data were split, the criteria for comparing LDA and QDA, classification accuracy, precision and recall by region, the confusion matrix and the limitations. Because the size of discriminant axis coefficients depends on whether variables were standardized, do not draw firm conclusions about variable importance.

In real research, it is good practice to set the final selection criteria in advance and to describe both the uncertainty in the performance estimates and how representative the data-collection setting is.

In fields such as medicine or quality inspection, where misclassification costs differ, you need to define separately what false positives and false negatives mean and set decision criteria, beyond overall accuracy.

When separating wines by region of production, considering several chemical characteristics together lets you examine group differences in more detail than checking acidity or sweetness alone. LDA builds relatively simple straight boundaries under the assumption that the regions have similar covariance structures, while QDA allows a different covariance for each region and builds more flexible curved boundaries. Which method is appropriate therefore depends on the distribution of the data and the sample size.

When interpreting classification results, how accurately the model separates wines it has never seen matters more than how well it fits the training data. Do not judge by accuracy alone: check the confusion matrix and per-region recall too, and also make sure no information from the test data leaked in during preprocessing. The goal of discriminant analysis is not to attach a correct answer to every observation, but to present reasonable criteria for separating groups under the given data and assumptions, along with their limitations.