8.3 Transformations for Skewed Data

Nearly all of the data fall into the left-most bin, and the extreme skew obscures many of the potentially interesting details in the data.

There are some standard transformations that may be useful for strongly right-skewed data where much of the data is positive but clustered near zero. A transformation is a rescaling of the data using a function. For instance, a plot of the logarithm (base 10) of county populations results in the new histogram in Figure 8.22(b). This data is symmetric, and any potential outliers appear much less extreme than in the original data set. By reining in the outliers and extreme skew, transformations like this often make it easier to build statistical models against the data.

Transformations can also be applied to one or both variables in a scatterplot. A scatterplot of the population change from 2010 to 2017 against the population in 2010 is shown in Figure 8.23(a). In this first scatterplot, it's hard to decipher any interesting patterns because the population variable is so strongly skewed. However, if we apply a \(\log_{10}\) transformation to the population variable, as shown in Figure 8.23(b), a positive association between the variables is revealed. While fitting a line to predict population change (2010 to 2017) from population (in 2010) does not seem reasonable, fitting a line to predict population change from \(\log_{10}(\text{population})\) does seem reasonable.

Transformations other than the logarithm can be useful, too. For instance, the square root (\(\sqrt{\text{original observation}}\)) and inverse (\(\frac{1}{\text{original observation}}\)) are commonly used by data scientists. Common goals in transforming data are to see the data structure differently, reduce skew, assist in modeling, or straighten a nonlinear relationship in a scatterplot.

(a)

(b)

Figure 8.22: (a) A histogram of the populations of all US counties. (b) A histogram of \(\log_{10}\)-transformed county populations. For this plot, the x-value corresponds to the power of 10, e.g. "4" on the x-axis corresponds to \(10^{4} = 10{,}000\).

(a)

(b)

Figure 8.23: (a) Scatterplot of population change against the population before the change. (b) A scatterplot of the same data but where the population size has been log-transformed.

(a) Strongly right-skewed: most values are small (between 5 and 40) and there is a long tail of large values (200, 900, 5000). The mean is pulled far to the right of the median by the big values.

(b) Taking \(\log_{10}\) of each (populations in thousands):

Raw \(x\)	\(\log_{10}(x)\)
5	0.70
8	0.90
12	1.08
15	1.18
20	1.30
40	1.60
80	1.90
200	2.30
900	2.95
5000	3.70

(c) Much more symmetric. The raw range is 5 to 5000 (three orders of magnitude); the transformed range is 0.70 to 3.70 (three units on the \(\log_{10}\) scale). The values now spread out evenly instead of piling up at the small end, so the distribution looks roughly bell-shaped rather than severely right-skewed.

The regression equation is

$$ \hat{y} = -52.3564 + 2.7842 x $$

Predictor	Coef	SE Coef	T	P
Constant	-52.3564	7.2757	-7.196	3e-08
x	2.7842	0.1768	15.752	\(< 2\)e-16

\(S = 13.76\), \(R\)-Sq \(= 88.26\%\), \(R\)-Sq(adj) \(= 87.91\%\).

We can note the \(R^{2}\) value is fairly large. However, this alone does not mean that the model is good. Another model might be much better. When assessing the appropriateness of a linear model, we should look at the residual plot. The \(\cup\)-pattern in the residual plot tells us the original data is curved. If we inspect the two plots, we can see that for small and large values of \(x\) we systematically underestimate \(y\), whereas for middle values of \(x\), we systematically overestimate \(y\). The curved trend can also be seen in the original scatterplot. Because of this, the linear model is not appropriate, and it would not be appropriate to perform a \(t\)-test for the slope because the conditions for inference are not met. However, we might be able to use a transformation to linearize the data.

Regression analysis is easier to perform on linear data. When data are nonlinear, we sometimes transform the data in a way that makes the resulting relationship linear. The most common transformation is log of the \(y\)-values. Sometimes we also apply a transformation to the \(x\)-values. We generally use the residuals as a way to evaluate whether the transformed data are more linear. If so, we can say that a better model has been found.

The regression equation is

$$ \widehat{\log(y)} = 1.722540 + 0.052985 x $$

Predictor	Coef	SE Coef	T	P
Constant	1.722540	0.056731	30.36	\(< 2\)e-16
x	0.052985	0.001378	38.45	\(< 2\)e-16

$$ S = 0.1073, \quad R\text{-Sq} = 97.82\%, \quad R\text{-Sq(adj)} = 97.75\% $$

The linear regression equation can be written as: \(\widehat{\log(y)} = 1.723 + 0.053 x\).

Figure 8.25: A plot of \(\log(y)\) against \(x\). The residuals don't show any evident patterns, which suggests the transformed data is well-fit by a linear model.

The point estimate is \(-0.0431\) and the standard error is \(SE = 0.0108\). When constructing a confidence interval for a model coefficient, we generally use a \(t\)-distribution. The degrees of freedom for the distribution are noted in the regression output, \(df = 48\), allowing us to identify \(t_{48}^{\star} = 2.01\) for use in the confidence interval.

We can now construct the confidence interval in the usual way:

$$ \text{point estimate} \pm t_{48}^{\star} \times SE \quad \rightarrow \quad -0.0431 \pm 2.01 \times 0.0108 \quad \rightarrow \quad (-0.0648,\ -0.0214) $$

We are \(95\%\) confident that with each dollar increase in family income, the university's gift aid is predicted to decrease on average by \$0.0214 to \$0.0648.

True statements: (b) and (d).

(a) False. The original scatterplot shows clear curvature, and the residual plot in Figure 8.24 shows a \(\cup\)-pattern — strong evidence that \(x\) and \(y\) are not linearly related.

(b) True. After applying the log transformation to \(y\), the residuals in Figure 8.25 show no pattern, which is the diagnostic for a good linear fit. The scatterplot of \(\log(y)\) versus \(x\) also looks linear.

(c) False. Regression I has \(R^{2} = 88.26\%\), but the curved residual pattern invalidates it — the high \(R^{2}\) is misleading because the model is fundamentally the wrong shape.

(d) True. Regression II has \(R^{2} = 97.82\%\) and a clean residual plot. Both indicators agree: the log-transformed model is a much better fit.

The pattern in residuals always trumps a single summary statistic like \(R^{2}\). A model with smaller \(R^{2}\) but clean residuals is almost always better than one with larger \(R^{2}\) but systematic residual patterns.

(a) Yes — \(R^{2} = 0.92\) means the linear model explains \(92\%\) of the variability in \(y\). Taken alone, this looks like a strong fit.

(b) No. The \(\cap\)-pattern in the residual plot means the model is systematically wrong: it overestimates \(y\) at the extremes and underestimates \(y\) in the middle (or vice versa, depending on sign). Systematic error is exactly what a linear model is not supposed to have. The \(R^{2}\) is misleading because the model is the wrong shape, not because it is noisy.

(c) Try a transformation. A \(\cap\)-shape often suggests taking \(\log(y)\) (or \(\sqrt{y}\)) to flatten the curve, or transforming \(x\) (e.g., \(x \mapsto x^{2}\) or \(x \mapsto \log(x)\)) to straighten the underlying relationship. After the transformation, make a new residual plot: if it looks like random static, the transformed model is a better fit. Only report the transformed model if the residuals support it.