Missing Data
Missing data are values that should have been observed but are absent, and the results of an analysis can change depending on how they are handled.
When you run a survey or collect data, some values often end up empty. For example, suppose you ran a satisfaction survey of 500 customers, and some respondents did not answer the question about income or skipped certain items.
Values that should have been observed in the data but are absent like this are called missing data. Missing data are not just a matter of blank cells. Depending on why the values are missing and how they are handled, the sample size used in the analysis changes, and the results of the statistical analysis can be affected.
In this article, we cover missing data from the basic concept to the types of missingness, common handling methods, a real analysis example, and points to watch when interpreting results.
What are missing data?
Missing data are values that should be in the data but were not recorded or not observed.
For example, suppose we surveyed 100 students on study time, test score, and learning satisfaction. Most students answered every item, but some may not have entered their study time or may not have answered the satisfaction item. The empty values here are missing data.
Missing data can occur for many reasons.
- A survey respondent did not answer a particular question
- A participant stopped or dropped out during the study
- A value was not recorded because of a measuring equipment error
- Some values were lost during data entry or merging
- The question itself did not apply because a condition was not met
Values left empty because the question did not apply, as in the last case, should be distinguished from ordinary nonresponse. For example, if people with no purchase experience were not asked about purchase satisfaction, the value is not simply missing; it may be a not-applicable value that arises from the structure of the survey.
So before handling missing data, it is important to first check why the values are empty.
Why are missing data a problem?
If you analyze data with missing values as they are, depending on the statistical method used, some observations may be excluded automatically, or the analysis may not run at all.
Also, depending on how the missing data arose, the results of the analysis may be biased. For example, suppose that in a customer satisfaction survey, the more dissatisfied a customer is, the less likely they are to answer the satisfaction item. If you then exclude all customers with missing responses, a relatively large share of the remaining responses may come from highly satisfied customers.
As a result, the average satisfaction may appear higher than it really is. There are three main reasons missing data are a problem.
First, the sample size used in the analysis can shrink.
When the sample size decreases, the precision of estimates goes down and statistical power can also drop.
Second, the results of the analysis can be biased.
If missing values are concentrated among people or observations with certain characteristics, the remaining data may not adequately represent the original population being studied.
Third, the relationships between variables can change.
Means, variances, correlation coefficients, regression coefficients, and so on can change depending on how missing values are handled. So before deciding whether to simply delete missing values or replace them with other values, you should check the pattern of missingness.
What types of missing data are there?
In statistics, the mechanisms by which missing data arise are divided into three main types.
They are MCAR, MAR, and MNAR. This classification is an important basis for deciding how to handle missing data.
| Missingness type | Meaning | Example |
|---|---|---|
| MCAR | Missingness occurs unrelated to both observed and unobserved values | Random equipment error |
| MAR | Given the observed variables, missingness is unrelated to the unobserved values | Income response rate differs by age |
| MNAR | Even given the observed variables, missingness is related to the missing value itself | People with high incomes do not disclose their income |
Missing completely at random (MCAR)
MCAR is when missingness occurs completely at random, unrelated to either the observed or the missing values in the data. For example, suppose measuring equipment broke down at random, so some participants' measurements were not recorded. If the equipment failure occurred regardless of the participants' characteristics or actual measured values, it may be MCAR.
Under MCAR, analyzing only the complete observations can estimate parameters without bias under certain conditions, but the smaller sample size can make the analysis less efficient.
Missing at random (MAR)
MAR is when whether a value is missing is related to other observed variables, but, once those variables are taken into account, it is assumed to have no additional relationship with the missing value itself. For example, suppose the response rate to the income question differs by age group.
If age information has been collected, and after accounting for age, actual income has no additional effect on whether people respond, the data can be considered MAR. MAR is an important assumption in missing data methods such as multiple imputation and maximum likelihood. However, it is hard to prove from the actual data alone that the MAR assumption definitely holds.
Missing not at random (MNAR)
MNAR is when, even after accounting for the observed variables, the missing value itself is related to whether it is missing. For example, if people with higher incomes are more likely not to disclose their income, and the other observed variables cannot fully explain this tendency, it may be MNAR.
In this case, ordinary deletion or imputation methods alone may not be able to fix the bias. So you may need to model the missingness process separately or run sensitivity analyses under various assumptions.
Real example: what if a satisfaction survey has missing data?
Suppose a company ran a service satisfaction survey of 200 customers.
The survey items are service quality, price satisfaction, and intent to return. The missing data in the collected data look like this.
| Variable | Total respondents | Missing count | Missing rate |
|---|---|---|---|
| Service quality | 200 | 4 | 2.0% |
| Price satisfaction | 200 | 20 | 10.0% |
| Intent to return | 200 | 10 | 5.0% |
For price satisfaction, 10% of all respondents did not answer, a higher missing rate than for the other variables. Having seen this result, you should first look into why so many responses were missing on the price satisfaction item. For example, you can check whether the customers who did not answer price satisfaction are concentrated in a particular age group or pricing plan.
Also, even if each of the three variables has missing values, the number of people actually excluded from the analysis depends on how the missing values overlap across respondents. If a total of 28 respondents are missing a value on at least one of the three variables, an analysis using only complete responses will be left with data from 172 people.
In this way, you should check not only the missing rate for each variable but also the number of complete cases that have all the variables needed for the analysis.
How are missing data handled?
There are several ways to handle missing data.
The main approaches are to exclude observations with missing values or to estimate and fill in the missing values.
| Method | Characteristics |
|---|---|
| Listwise deletion | Exclude observations missing any of the required variables |
| Pairwise deletion | Use the available observations for each pair of variables |
| Mean/median imputation | Replace missing values with a typical value |
| Regression imputation | Predict values using relationships with other variables |
| Multiple imputation | Create several imputed datasets to reflect uncertainty |
| Maximum likelihood methods | Estimate with a probability model using the observed information |
Listwise deletion
Listwise deletion excludes any observation that has a missing value on even one of the variables needed for the analysis. For example, in an analysis using three variables (study time, test score, and learning satisfaction), if one student's learning satisfaction value is empty, all of that student's data are excluded from that analysis.
It has the advantages of being simple and having a clear procedure. But if there are many missing values, the sample size available for analysis can drop sharply. And if appropriate conditions such as MCAR are not met, the results can be biased.
Pairwise deletion
Pairwise deletion uses only the observations available for each pair of variables. For example, when calculating the correlation between study time and test score, you use only students who have values for both variables, and when calculating the correlation between study time and satisfaction, you may use a different set of observations.
It has the advantage of using more data than listwise deletion, but the sample size can differ from one result to another. Also, a correlation matrix calculated from different sets of observations can end up mathematically inconsistent.
Mean/median imputation
Mean or median imputation fills missing values with the variable's mean or median. For example, if the mean customer satisfaction score is 3.8 points, missing values can be replaced with 3.8. It has the advantages of being easy to implement and keeping every row.
But inserting the same value repeatedly can make the variability of the data smaller than it really is. It can also distort the relationships between variables, so it is limited as a default solution in studies where statistical inference matters. In particular, if you run an ordinary analysis after single imputation, the uncertainty of the imputation process may not be adequately reflected.
Regression imputation
Regression imputation predicts missing values using their relationships with other variables. For example, if income information is missing, you can predict income using other variables such as age, occupation, and education level.
It can use the relationships between variables better than mean imputation, but filling in a single predicted value may not adequately reflect the uncertainty of the imputed values. So simple regression-prediction imputation should be distinguished from multiple imputation, which reflects uncertainty.
What is multiple imputation?
Instead of filling each missing value with a single fixed number, multiple imputation creates several completed datasets that reflect the uncertainty about the possible values.
For example, suppose you have original data that contain missing values. With multiple imputation, you can create several datasets in which the missing values are filled in slightly differently. You then run the same statistical analysis on each dataset and combine the results.
The general process is as follows.
① Check missing patterns
Check which variables have missing values and what form the missingness takes.
② Create several imputed datasets
Impute the missing values in several ways using the relationships among observed variables and a statistical model.
③ Run the same analysis on each dataset
For each dataset created, run the analysis that fits the study's purpose, such as a regression or a comparison of means.
④ Pool the results
Combine the estimates and variances from each analysis to produce the final result. Rubin's rules are typically used for this. Unlike simple mean imputation, multiple imputation has the advantage of reflecting the uncertainty in estimating the missing values.
However, using multiple imputation does not solve every missing data problem.
The imputation model must be appropriate, and assumptions about the missingness mechanism, typically MAR, must also be considered.
If the missing rate is high, do you always have to impute?
A high missing rate does not mean you must always impute values.
Conversely, a low missing rate does not mean you can always just delete. For example, even with a missing rate of 2%, if the missing values occur only in a particular group, the results can be biased. And even if the missing rate is relatively high, if you understand how the missingness arose and can apply an appropriate analysis model, you can make use of some of the information.
When deciding how to handle missing data, consider the following together.
- How high the missing rate is
- Which variables have missing values
- Whether missing values are concentrated in a particular group
- What caused the missing values
- What statistical analysis you plan to run
- How much the sample size shrinks if missing values are excluded
So rather than applying the same missing-rate threshold to all data, you should judge according to the purpose of the study and the characteristics of the data.
How do you report how missing data were handled?
In research papers and analysis reports, it is important to explain clearly how missing data were handled.
In particular, if the sample size differs before and after handling missing data, readers should be able to see the difference. For example, suppose data were collected from a total of 200 people in a customer satisfaction survey, but 28 people with missing values on variables needed for the analysis were excluded. In this case, you could report it as follows.
Example report
Of the 200 total responses, 28 respondents with missing values on variables needed for the analysis were excluded, and data from the final 172 respondents were used in the analysis. Missing data were handled by listwise deletion.
If you used multiple imputation, it is good to also explain the imputation method, the number of datasets created, how the analysis results were pooled, and so on. A sensitivity analysis checking how much the main results change depending on how missing data are handled can also be useful.
Where does handling missing data matter?
Handling missing data matters not only in surveys but in almost every kind of data analysis.
Surveys and social science research
Respondents may skip some items or not answer sensitive questions.
Education research
Some test scores or learning activity records may be missing, or students may stop participating partway through the study.
Health and medical research
During follow-up, some patients' measurements may be missing or they may stop participating in the study.
Business and customer data analysis
Purchase records, customer information, service usage records, and so on may not have been collected or may be lost when merging data across systems.
Machine learning and predictive analytics
If input variables have missing values, some algorithms may not be able to handle them directly, and predictive performance can vary with how missing values are handled. In this way, handling missing data is a data preparation step that must be reviewed to ensure the accuracy and reliability of the results.
With missing data, understanding comes before filling in.
Missing data are values that are absent from the data, and they occur in many situations, such as surveys, experimental studies, and business data analysis.
Methods for handling missing data include listwise deletion, pairwise deletion, mean/median imputation, regression imputation, multiple imputation, and maximum likelihood methods. But no one method is always the best.
You need to look not only at how high the missing rate is, but also at why the values are missing, which variables they are missing from, and how they might affect the results.
The goal of handling missing data is not simply to get rid of the blanks in the data, but to reduce the chance that missing information distorts the results and to reach trustworthy conclusions.