Mini widget·

Transforms & Log Transformation

Averaging in logs and converting back came out 24% low

Four hundred incomes have a mean of 484.8. The distribution is heavily right-skewed, so you take logs, average in log space, and exponentiate back to the original units. Out comes 369.8.

No arithmetic went wrong. What you converted back is not the mean — it is the geometric mean. On a skewed distribution that always sits below the arithmetic mean, and on this sample it is 23.7% lower. Report it as "average income" and you will be wrong in the same direction every single year.

How the same 400 incomes were summarised	Value	What that value actually is
Add them up and divide by 400	484.8	the thing people normally mean by "average"
Log, average, exponentiate back	369.8	not the mean but the geometric mean — 23.7% lower

Going out through a transform and back in quietly changed which quantity you are reporting. Both rows come from the same data, and both get called "the average".

Transformation is the cheapest tool in statistics and the most casually misapplied. Knowing what you are trying to fix, and what quantity you end up reporting, removes most of the traps at once.

The key question

What does this transform straighten, and whose meaning does it change in exchange?

Concept

One rung on the ladder of powers

Three things a transform fixes

The purpose is not "to make the data normal". In practice a transform is always aimed at one of three quite separate problems.

Straighten the shape

skew → symmetry

A long right tail becomes a bell.

Straighten the relation

curve → line

An exponential trend becomes linear.

Even out the spread

fan → band

Residuals stop widening with the fitted value.

Change the meaning

difference → ratio

A gap in logs is a percentage change.

•
The shape of a distribution — a long right tail becomes roughly symmetric.
•
The relation between variables — a curve straightens, so a linear model becomes usable.
•
The spread of the residuals — errors that widen with the fitted value settle into a band.
•
The units of interpretation — a bonus, and an important one. A difference in logs is a percentage change.

Fixing one often improves the others, but not always. Deciding which one you are aiming at is what determines which transform to use and how strong to go.

The ladder of powers

The transforms in common use are not a grab bag — they sit on a single continuous ladder. You are raising x to a power p, and the further down you go, the stronger the correction.

\ldots,\; x^2,\; x,\; \sqrt{x},\; \log x,\; -\tfrac{1}{\sqrt{x}},\; -\tfrac{1}{x},\; \ldots

$\log x$ occupies the rung at power 0. Not because $x^0$ is useful — it is just 1 — but because the limit of $(x^p - 1)/p$ as p → 0 is exactly $\log x$ . The log is not a separate idea; it is the value that fills the gap in the ladder.

▼ Table 1 Skewness of the same 400 incomes at each rung

Transform	Power	Skewness	Strength
x — untouched	1	3.24	none
√x — square root	0.5	1.05	mild
log x — logarithm	0	-0.07	moderate
−1/√x — reciprocal root	-0.5	-0.93	strong
−1/x — reciprocal	-1	-1.79	strong

Raw skewness of 3.24 falls to 1.05 under the square root and reaches −0.07 at the log. Keep going and it becomes −0.93, then −1.79 — the tail has flipped to the other side. Descending the ladder does not monotonically help; there is a rung to stop on.

Why It Matters

Are you fixing the shape, the relation, or the spread?

Why the log works quite so often

Incomes, share prices, city populations, company revenues, reaction times — take logs of any of them and they come out near-symmetric. That is not luck. All of them are built by multiplying rather than by adding.

\log(a \times b) = \log a + \log b

A salary rising 3–7% a year is, after a decade, a starting salary multiplied by ten ratios. Products come out right-skewed. Take logs and the product becomes a sum — and sums drift toward normal by the central limit theorem.

That is what lognormal means

A distribution that becomes normal under logs is called lognormal; the name is the recipe. Whenever a quantity varies in percentage terms, lognormal is the first thing worth suspecting.

Interpretation often gets easier, not harder

Transforms have a reputation for making results hard to explain. The log frequently does the opposite: in a regression it turns coefficients into percentages, which is often the more natural unit to begin with.

Model	Coefficient reads as	For a small β
y ~ x	x up 1 unit → y up β units	—
log y ~ x	x up 1 unit → y up 100·β percent	Exact: (e^β − 1)·100
y ~ log x	x up 1 percent → y up β/100 units	Doubling x → y up 0.693·β
log y ~ log x	x up 1 percent → y up β percent	β is an elasticity

The last row is the useful one. Log on both sides and the coefficient is an elasticity. "A 1% price rise cuts demand by 1.4%" is readable directly off a single number.

The trouble arrives on the way back

Average in transformed space, convert back, and what you get is not the mean in the original units. This is Jensen's inequality, and it applies to every convex or concave transform.

\exp\!\left(\frac{1}{n}\sum \log x_i\right) \;=\; \sqrt[n]{\prod x_i} \;\le\; \frac{1}{n}\sum x_i

On the same 400 incomes

Arithmetic mean: 484.8

Back-transformed log mean: 369.8 (the geometric mean)

Sample median: 386.1 — the geometric mean is close to it

What you recovered is not the mean but something near the median. On a lognormal distribution the geometric mean and the median coincide exactly.

To get the mean you need a smearing correction: multiply by $\exp(s^2/2)$ , where s is the standard deviation in log space. On this sample that factor is 1.322, and 369.8 × 1.322 = 488.8, close to the true 484.8.

How It Works

Deciding which way down the ladder, and how far

1. Picking a transform

The data looks like	Try	Watch out for
Right-skewed, all positive	log x	Zeros break it — use log(x+1) or a shift
Right-skewed counts	√x	Stabilises Poisson variance; gentler than log
Proportions between 0 and 1	logit	Exact 0 and 1 have to be handled first
Left-skewed	x², x³	Go up the ladder, not down
Both signs, heavy tails	asinh x	Behaves like log for large \|x\|, fine at 0
Nothing obvious	Box–Cox / Yeo–Johnson	It fits λ for you — but only on the training rows

2. Dealing with zeros and negatives

This is the log's hard constraint. $\log 0$ is undefined and so is the log of a negative. The reflex is $\log(x+1)$ , but that is not free.

Approach	When it is fine	The catch
log(x + 1)	x runs in the tens to thousands	If x is between 0 and 1, the +1 dominates
log(x + c), c estimated	Zeros are a small minority	The answer moves with your choice of c
Treat zero as missing	Zero means "not measured"	Loses information if zero is real
asinh(x)	Negative values are present	Nearly identical to x when x is small
Yeo–Johnson	Mixed signs and no clear call	Requires estimating lambda

A lot of zeros may not be a transform problem

If 40% of the values are zero, that is not a skewed distribution — it is two processes stacked in one column. "Did they buy?" and "how much did they spend?" are different questions, and a two-part model is usually the right answer rather than any transform.

3. Letting Box–Cox choose

When the right rung is not obvious, the data can pick. Box–Cox estimates by maximum likelihood the λ that makes the result most nearly normal.

x^{(\lambda)} = \begin{cases} \dfrac{x^\lambda - 1}{\lambda} & \lambda \neq 0 \\[6pt] \log x & \lambda = 0 \end{cases}

•
Round λ to a recognisable rung — 0.03 means use the log, 0.48 means use the square root. An interpretable transform is worth far more than the fit you give up.
•
It needs strictly positive data — with negatives, use Yeo–Johnson instead.
•
Estimate λ on the training rows only — fitting it on everything lets the test set influence the transform.
•
It optimises normality alone — if your goal is straightening a relationship, Box–Cox may not be aiming at it.

4. When not to transform at all

Skew is not automatically a reason to transform. There are cases where leaving the data alone is clearly better.

Situation	Call	Why
Using a tree-based model	Unnecessary	Splits depend on order alone
You only care about prediction	Usually unnecessary	Normality is not a prediction requirement
Large sample, comparing means	Unnecessary	The CLT is already doing the work
Results must be in original units	Proceed carefully	Converting back changes what you report
The tail is the subject of interest	Do not	The transform compresses what you came to study

Example

Back-transformed means are not means

In practice: ad spend against revenue

A scatter of ad spend against revenue curves upward. A straight-line regression gives an acceptable $R^2$ , but the residuals fan out.

Model	R²	Residuals	What the coefficient says
revenue ~ spend	0.71	Fan out	$2.30 of revenue per $1 of spend
log(revenue) ~ spend	0.83	Nearly even	4.1% more revenue per $1,000 of spend
revenue ~ log(spend)	0.79	Wide on the left	$1,800 more revenue per 1% of spend
log(revenue) ~ log(spend)	0.89	Even	0.62% more revenue per 1% of spend

The fourth fits best and is also the easiest to state. A coefficient of 0.62 is an advertising elasticity, and because it is below 1 you can read diminishing returns straight off the same number.

The lesson here

A transform is not a trick for forcing a model to fit. If advertising genuinely works in proportional terms, then the log-log model is simply using the structure that is already there. It fits well because the model and the phenomenon agree.

In practice: the shortfall on the way back

A claims-severity model was trained in log space. Exponentiating the predictions and summing them produced an annual total that was persistently short.

Summed over 10,000 claims

Actual annual payout: $4.848M

Naive back-transform, summed: $3.698M

After the smearing correction: $4.888M

Before correcting, $1.150M was missing. Per claim it looks like rounding; across the book it moves the whole reserve calculation.

The gap does not close as the sample grows, because it is bias, not noise. Whenever a log-space model is used to predict a total in original units, the correction is mandatory.

Common misunderstandings

Misconception 1

❌ Skewed data always gets a log.

The ladder has several rungs and the log is one of them. With a skewness around 1 the square root is the better fit, and the log overshoots into the other tail.

Misconception 2

❌ The point of transforming is to make the data normal.

What regression asks to be normal is the residuals, not the variables. And when the goal is prediction, normality is not a requirement at all.

Misconception 3

❌ log(x+1) is the standard fix for zeros.

If x lives between 0 and 1, the +1 is larger than the data. The constant has to match the scale of the variable — and often asinh or a two-part model is a better answer than any constant.

Misconception 4

❌ Predict in log space, exponentiate, done.

Exponentiating gives you a prediction of the median. If you need a mean or a total, apply the smearing correction — without it you under-predict in the same direction every time.

Misconception 5

❌ Use whatever λ Box–Cox returns.

λ = 0.037 is uninterpretable to every reader you have. Round it to 0 and use the log — the loss in fit is negligible and the gain in explicability is not.

Interactive

Walk the ladder with one slider

The same 400 incomes under a single power. Moving left strengthens the correction — watch where the skewness crosses zero.

400 incomes, transformed by a single power. The log is not a special case — it is the rung at power 0.

Power

Skewness

3.24

Verdict

right tail

Rung on the ladder of powers1 (raw)

Untouched, the sample has a skewness of 3.24: a dense clump at the low end and a long thin tail to the right. Any mean-based method is going to struggle here.

Pick the rung that straightens the data — not the strongest one available.

What to look for

•
What is the raw skewness, and what shape does the histogram have?
•
How far does the square root pull it down? Far enough?
•
Which rung brings the skewness closest to zero?
•
Go below the log — what happens to the histogram?
•
Notice that the x-axis units change completely from rung to rung.

Learning points

Going further down the ladder does not keep helping — you can overshoot.

The log is not a special technique but the rung sitting at power 0.

After transforming, the horizontal axis is in new units — reporting means converting back.

Key takeaways

What it fixes

Shape, curvature, spreading residuals

• Skew becomes symmetry
• Curves become lines
• Fanning residuals become a band

Why the log

It turns products into sums

• Multiplicative data becomes additive
• Differences become percentages
• Zeros and negatives are undefined

What it costs

You now report a different quantity

• Back-transforming a mean gives the median
• Coefficients change meaning
• Fit λ on the training rows only

The question is not "which transform should I use?"

It is "what quantity am I reporting once I have transformed?" Average in log space, convert back, and what you have is closer to a median than to a mean.