Chapter 1 Simple Linear Regression

1.1 Deterministic Models

Regression analysis establishes a functional relationship between X and Y. The simplest type of relationship between X and Y is a deterministic relationship [latex]Y=f(X)[/latex]. In this rare case, the value of Y can be determined without error once the value of X is known, so Y is not a random variable when the relationship between X and Y is deterministic. The deterministic model is described by [latex]Y=f(X)[/latex]. Deterministic relationships are uncommon in real-world applications because there is typically uncertainty in the dependent variable. If data pairs (X₁, Y₁), (X₂, Y₂), …, (X_n, Y_n) are collected and the deterministic relationship [latex]Y=f(X)[/latex] establishes the correct functional relationship between X and Y, then all of the data pairs will fall on the graph of the function [latex]Y=f(X)[/latex].

Figure 1.1: A deterministic linear relationship between X and Y.

Determining the relationship between the number of sales per week X and the commissions paid per week Y did not require the collection of any data to determine the function [latex]Y=f(X)[/latex]. That linear relationship was implicit in the problem statement. Other cases can arise, such as (a) the relationship is deterministic but requires data to determine its functional form, or (b) the relationship is deterministic, but unlike the relationship in the previous example, it is not linear. The following example illustrates a nonlinear deterministic relationship between the independent variable X and the dependent variable Y.

In most applications, the relationship between the independent variable X and the dependent variable Y is not deterministic because Y is typically a random variable. The next section introduces some of the thinking behind the development of a statistical model that describes the relationship between X and Y.

1.2 Statistical Models

The goal in constructing a statistical model is to write a formula that adequately captures the governing probabilistic relationship between an independent variable X and a dependent variable Y. This formula might be used subsequently for prediction or some other form of statistical inference. In this section, we assume that the dependent variable Y is a continuous random variable that can assume a range of values associated with a particular setting of the independent variable X. The relationship

$$\begin{array}{l}Y=f(X)\end{array}$$

that was used in the previous section is no longer adequate because X is assumed to be observed without error, and this formula results in a value of Y which is deterministic rather than random. One way of overcoming this problem is to replace the left-hand side of this equation by the expected value of Y, which is a constant, resulting in

$$\begin{array}{l}E[Y]=f(X).\end{array}$$

To be a little more careful about what is meant by this statistical relationship, the left-hand side is actually a conditional expectation, namely

$$\begin{array}{l}E[Y|X=x]=f(x).\end{array}$$

In words, given that the independent variable X assumes the value x, the transformation [latex]f(X)[/latex] gives the conditional expected value of the dependent variable Y. Notice that this statistical model does not specify the distribution of the random variable Y for a particular value of X; it only tells us the expected value of Y for a particular value of X. This statistical regression model defines a hypothesized relationship between the observed value of X on the right-hand side of the model and the conditional expected value of Y on the left-hand side of the model. The hypothesized relationship might be adequate for modeling or it might need some refining. There is typically no model that perfectly captures the relationship between X and Y. This was recognized by George Box, who wrote:

All models are wrong; some models are useful.

In a statistical model that involves parameters, the estimation of the model parameters will be followed by assessments to determine whether the model holds in an empirical sense. If the model needs refining, the new set of parameters are estimated and new assessments are made to see if the refined model is an improvement over the previous model. Regression modeling is an iterative process.

There is a second way to write a statistical model that is equivalent to the statistical model described in the previous paragraph. The model can be written as

$$\begin{array}{l}Y=f(X)+ϵ,\end{array}$$

where the error term ϵ (also known as the “noise” or “disturbance” term) is a random variable that accounts for the fact that the independent variable cannot predict the dependent variable with certainty. This term makes the relationship between X and Y a random (or statistical or stochastic) relationship rather than a deterministic relationship. If the probability distribution of the error term is specified, then not only is the expected value of Y conditioned on the value of X determined, but also the entire probability distribution of Y conditioned on the value of X is specified. It is common practice to assume that the expected value of ϵ is zero. The probability distribution of ϵ establishes the nature and magnitude of the scatter of the data values about the regression function. When the population variance of ϵ is small, the values of Y are tightly clustered about the regression function [latex]f(X)[/latex]; when the population variance of ϵ is large, the values of Y stray further from the regression function [latex]f(X)[/latex].

Regression modeling involves determining the functional form of [latex]f(X)[/latex] from a data set of n data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)[/latex]. The statistical model for X and Y in a general sense also applies to each of the data points, so

$$\begin{array}{l}{Y}_i=f(X_i)+{ϵ}_i\end{array}$$

for i = 1, 2, . . . , n. The sign of ϵ_i indicates whether the observed data pair (X_i, Y_i) falls above [latex](\epsilon_i > 0)[/latex] or below [latex](\epsilon_i < 0)[/latex] the conditional expected value of Y_i, for i = 1, 2, . . . , n.

The function [latex]f(X)[/latex] is called the regression function, and was first referred to in print as such by Sir Francis Galton (1822–1911), a British anthropologist and meteorologist, in his 1885 paper titled “Regression Toward Mediocrity in Hereditary Stature” published in the Journal of the Anthropological Institute. He established a regression function relating the adult height of an offspring, Y, as a function of an average of the parent’s heights, X, which had been adjusted for gender.

The regression function [latex]Y=f(X)[/latex] can be either linear or nonlinear. The next section focuses on the easier case, a linear regression function. In this case, the model is typically referred to as a simple linear regression model, which is often abbreviated as an SLR model. The model is simple because there is only one independent variable X that is used to predict the dependent variable Y. The model is linear because the regression function [latex]f(X)=\beta_{0}+\beta_{1}X[/latex] is assumed to be linear in the parameters β₀ and β₁. The more complicated cases of multiple linear regression, which involve more than one independent variable, and nonlinear regression, in which [latex]f(X)[/latex] is not a linear function, will be introduced later.

1.3 Simple Linear Regression Model

A simple linear regression model assumes a linear relationship between an independent variable X and a dependent variable Y. In this section, the more general regression model

$$\begin{array}{l}Y=f(X)+ϵ\end{array}$$

is reduced to the simple linear regression model given in the definition below.

Stating the simple linear regression model in this fashion will not seem natural from probability theory. As a non-regression illustration from probability theory, [latex]W \sim N(\mu, \sigma^{2})[/latex] indicates that W has a normal distribution with population mean μ and population variance σ². Although much less compact, the probability distribution of W can also be written as [latex]W = \mu + \epsilon[/latex], where [latex]\epsilon \sim N(0, \sigma^{2})[/latex]. This illustration reflects the essence behind writing the simple linear regression model in the form [latex]Y=\beta_{0}+\beta_{1}X + \epsilon[/latex] in Definition 1.1.

The formulation of the simple linear model from Definition 1.1 involves a random variable ϵ on the right-hand side of the model. In some settings, this model might be viewed as a transformation of a random variable, but this is not the correct interpretation of the model in this setting. The simple linear regression model defines a hypothesized relationship between the random variable on the left-hand side of the model and terms on the right-hand side of the model. This probability model is hypothesized to govern the relationship between X and Y. The goal in constructing a simple linear regression model is to determine if it adequately captures the probabilistic relationship between X and Y. Estimation of the model parameters will be followed by assessment to see if the model holds in an empirical sense.

The assumption that the random variable ϵ has population mean zero and population variance σ² in the most basic simple linear regression model in Definition 1.1 allows for mathematically tractable statistical inference. In models that allow for confidence intervals and hypothesis testing concerning the estimated slope and intercept, the error term is assumed to have a specific distribution, which is typically the normal distribution. The error term models all sources of variation, both known and unknown, other than the variation in Y associated with the particular level of X. Notice that σ² is constant over all values of X.

The assumption that the independent variable X is not subject to random variability is not always satisfied in practice. The fitting procedure becomes more complicated when X is considered to be a random variable. For this reason, we assume that the observed value of X is either exact or that the variation of X is small enough so that its observed value can be assumed to be exact.

The assumption of a linear relationship between X and Y might also be flawed. In some cases it might not be a perfectly linear relationship, but a linear relationship provides a close enough approximation between X and Y to be useful for associated statistical inference. In other cases, a linear relationship might be appropriate for some range of values of X, known as the scope of the model, but not others. One important step in establishing a simple linear regression model is to specify the values of X for which the simple linear regression model is valid.

The procedure for establishing a simple linear regression model that relates the dependent variable Y to the independent variable X is given below.

1. Collect the data pairs. The data pairs are denoted by [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)[/latex]. In some settings, it is possible to exert some control over the X_i values. As will be seen later, there are advantages to having the X_i values spread out as much as possible in terms of the precision of the fitted regression model.
2. Make a scatterplot of data pairs. A scatterplot is just a plot of the points [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)[/latex] on a set of axes. The purpose of the scatterplot is to see if the linear relationship between X and Y is appropriate and to visually assess the spread of the data values about the regression function. With modern statistical software, scatterplots are easy to generate.
3. Inspect the scatterplot. Although this step is subjective, it is important to visually assess
   (a) whether the relationship between X and Y appears to be linear or nonlinear, (b) whether the spread of the data pairs about the regression function is small or large, and (c) whether the variability of the data pairs about the regression function remains constant over the range of X values that have been collected.
4. State the regression model. In this chapter, the regression model is assumed to be the simple linear regression model [latex]Y=\beta_{0}+\beta_{1}X + \epsilon[/latex]. Nonlinear regression models, such as the quadratic model [latex]Y=\beta_{0}+\beta_{1}X +\beta_{2}X^{2} + \epsilon[/latex], and multiple regression models with more than one independent variable, such as [latex]Y=\beta_{0}+\beta_{1}X_{1} +\beta_{2}X_{2} + \epsilon[/latex], will be considered later.
5. Fit the regression model to the data pairs. The method of least squares, which will be described in the next section, is commonly used to estimate the parameters in the regression model. The least squares criterion is to choose the regression model that minimizes the sum of the squares of the vertical differences between data points and the fitted regression model.
6. Assess the adequacy of the fitted regression model. Visual assessment techniques for assessing the fitted regression model include superimposing the fitted regression model onto the scatterplot of the data pairs and examining a plot of the residuals. A residual is the signed vertical distance between a data pair and its associated value on the regression function. In addition, there are statistical methods that can be applied to the fitted regression model to see if it adequately describes the relationship between X and Y.
7. Perform statistical inference. Once the fitted regression model is deemed an acceptable approximation to the relationship between X and Y, it can be used for statistical inference. One simple example of statistical inference that occurs often in practice is the prediction of a future value of Y for a particular level of X.

The seven steps for establishing a regression model are not necessarily performed in the order given here. Many times the fitted regression model is rejected in Step 6, and it is necessary to return to Step 4 in order to formulate an alternative model. Steps 4 through 6 might need to be repeated several times before arriving at an acceptable model for statistical inference.

The simple linear regression model given in Definition 1.1 implies that all of the [latex](X_{i}, Y_{i})[/latex] pairs also follow the simple linear regression model:

$$\begin{array}{l}{Y}_i={\beta }_0+{\beta }_1{X}_i+{ϵ}_i\end{array}$$

for [latex]i = 1, 2,\ldots, n[/latex], where

• (X_i, Y_i) are the data pairs, for [latex]i = 1, 2,\ldots, n[/latex],
• X_i is the value of the independent variable for observation i, which is observed without error, for [latex]i = 1, 2,\ldots, n[/latex],
• Y_i is the value of the dependent variable for observation i, which is a continuous random variable, for [latex]i = 1, 2,\ldots, n[/latex],
• β₀ is the population intercept of the regression line,
• β₁ is the population slope of the regression line, and
• ϵ_i is the random error term for observation i which satisfies
  • [latex]E[\epsilon_{i}]=0[/latex] for [latex]i = 1, 2,\ldots, n[/latex],
  • [latex]V[\epsilon_{i}]=\sigma^{2}[/latex] for [latex]i = 1, 2,\ldots, n[/latex],
  • the random ϵ_i values are mutually independent random variables, which implies that their variance–covariance matrix is diagonal.

When the simple linear regression model is stated in this fashion, four properties become apparent. First, Y_i is a random variable that can be broken into two components: a deterministic component [latex]\beta_{0}+\beta_{1}X_{i}[/latex], and a random component ϵ_i, for [latex]i = 1, 2,\ldots, n[/latex]. Second, Y_i has population mean

$$\begin{array}{l}E[Y_i]=E[{\beta }_0+{\beta }_1{X}_i+{ϵ}_i]={\beta }_0+{\beta }_1{X}_i\end{array}$$

for [latex]i = 1, 2,\ldots, n[/latex] and population variance

$$\begin{array}{l}V[Y_i]=V[{\beta }_0+{\beta }_1{X}_i+{ϵ}_i]=V\left[{ϵ}_i\right]={\sigma }^2\end{array}$$

for [latex]i = 1, 2,\ldots, n[/latex]. Using slightly different notation, it would be reasonable to write the population mean and variance as the conditional expectations

$$\begin{array}{l}E[Y_i|X_i]={\beta }_0+{\beta }_1{X}_i\text{and}V[Y_i|X_i]={\sigma }^2\end{array}$$

for [latex]i = 1, 2,\ldots, n[/latex]. The property that the variance does not change with X_i is known as homoscedasticity. Temporarily dropping the subscripts, the line

$$\begin{array}{l}E\left[Y\right]={\beta }_0+{\beta }_{1}X,\end{array}$$

with β₀ and β₁ replaced by the associated estimated values [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex], is oftentimes superimposed onto the scatterplot to visualize the fitted regression model. Third, each data pair (X_i, X_i) has a Y_i value that misses the regression function by the error term ϵ_i, for [latex]i = 1, 2,\ldots, n[/latex]. Fourth, the values of the observed dependent variables Y₁, Y₂, …, Y_n must be mutually independent random variables because the error terms [latex]\epsilon_{1}, \epsilon_{2},\dots, \epsilon_{n}[/latex] are mutually independent random variables.

1.4 Least Squares Estimators

We now turn to the question of estimating the intercept β₀ and the slope β₁ by the method of least squares. German mathematician Carl Friedrich Gauss (1777–1855) invented the least squares method and French mathematician Adrien–Marie Legendre (1752–1833) first published the method in 1805. The least squares method determines the values of β₀ and β₁ that minimize the sum of the squares of the errors, where the error is the vertical distance between the Y_i value and the fitted regression line. The term estimator will be used here to refer to a generic formula for [latex]\hat{\beta}_0[/latex] or [latex]\hat{\beta}_1[/latex]; the term estimate will be used to refer to a specific numeric value for [latex]\hat{\beta}_0[/latex] or [latex]\hat{\beta}_1[/latex].

One bit of notation that will make the expressions of the point estimators more compact is

$$\begin{array}{ll}{S}_{XY}& =\sum _{i=1}^n(X_i-\overline{X})(Y_i-\overline{Y})\\ & =\sum _{i=1}^n(X_i{Y}_i-X_i\overline{Y}-\overline{X}{Y}_i+\overline{X}\overline{Y})\\ & =\sum _{i=1}^n{X}_i{Y}_i-n\overline{X}\overline{Y}-n\overline{X}\overline{Y}+n\overline{X}\overline{Y}\\ & =\sum _{i=1}^n{X}_i{Y}_i-n\overline{X}\overline{Y}.\end{array}$$

Similarly,

$$\begin{array}{l}{S}_{XX}=\sum _{i=1}^n(X_i-\overline{X}{)}^2=\sum _{i=1}^n{X}_i^2-n{\overline{X}}^2\end{array}$$

and

$$\begin{array}{l}{S}_{YY}=\sum _{i=1}^n(Y_i-\overline{Y}{)}^2=\sum _{i=1}^n{Y}_i^2-n{\overline{Y}}^2.\end{array}$$

This new notation allow us to express [latex]nS_{XY}[/latex], [latex]nS_{XX}[/latex], and [latex]nS_{YY}[/latex] as

$$\begin{array}{l}n{S}_{XY}=n\sum _{i=1}^n{X}_i{Y}_i-\sum _{i=1}^n{X}_i\sum _{i=1}^n{Y}_i,\end{array}$$

$$\begin{array}{l}n{S}_{XX}=n\sum _{i=1}^n{X}_i^2-{\left(\sum _{i=1}^n{X}_i\right)}^2,\end{array}$$

and

$$\begin{array}{l}n{S}_{YY}=n\sum _{i=1}^n{Y}_i^2-{\left(\sum _{i=1}^n{Y}_i\right)}^2.\end{array}$$

Using this notation, the least squares estimators for the slope and intercept of the model, denoted by [latex]\hat{\beta}_1[/latex] and [latex]\hat{\beta}_0[/latex], are given in the following theorem. Notice that the term normal equations in the theorem is not related to the normal distribution.

The requirement that there are at least two distinct X_i values in Theorem 1.1 is consistent with intuition. Figure 1.3 shows [latex]n=5[/latex] data pairs in which the independent variable assumes the same value for each pair: [latex]X_{1}=X_{2}=X_{3}=X_{4}=X_{5}=3[/latex]. It is not possible to estimate the slope of the regression line in this particular setting. This is the geometric reason for the requirement that there are at least two distinct X_i values. In addition, the denominator in [latex]\hat{\beta}_1 = S_{XY} / S_{XX}[/latex] is zero when all X_i values are equal, which gives the associated algebraic reason for the requirement. From this point forward, whenever the simple linear regression model is used, it is assumed that the associated data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)[/latex] have at least two distinct X_i values.

Figure 1.4 shows the geometric interpretation associated with the estimated intercept [latex]\hat{\beta}_0[/latex] and estimated slope [latex]\hat{\beta}_1[/latex]. The [latex]n=9[/latex] data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_9, Y_9)[/latex] are plotted as points, along with the associated estimated regression line [latex]Y=\hat{\beta}_{0}+\hat{\beta}_{1}X[/latex]. The y-intercept of the graph [latex]\hat{\beta}_0[/latex] is the height of the estimated regression line at [latex]X = 0[/latex]. The “rise over run” interpretation of the slope is illustrated by the right triangle with legs consisting of dotted lines.

The next example illustrates the mechanics associated with calculating the least squares estimates [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex]. In order to focus on the calculations performed by hand, a small sample size of [latex]n = 3[/latex] data pairs is used. The numbers have been handpicked in order to make the resulting parameter estimates come out to whole numbers. A sample size of [latex]n = 2[/latex] is too simplistic in that two points determine a line, and the estimated regression line will always pass through those two points.

1.5 Properties of Least Squares Estimators

The least squares estimators of β₀ and β₁ possess several properties which are important for statistical inference. The four properties established in this section are:

• the least squares estimators [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] are unbiased estimators of β₀ and β₁,
• the least squares estimators [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] can be written as linear combinations of the dependent variables [latex]Y_{1}, Y_{2}, \dots, Y_{n}[/latex],
• the variance–covariance matrix of [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] can be written in closed form, and
• the least squares estimators [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] have the smallest population variance among all unbiased estimators that can be expressed as linear combinations of the dependent variables.

Proofs of the associated results are included in each of the following subsections.

1.5.4 Gauss–Markov Theorem

Recall from Theorem 1.3 that the least squares estimators for the slope and intercept of the regression line were expressed as linear combinations of the dependent variables:

$$\begin{array}{l}{\hat{\beta }}_1=a_1{Y}_1+a_2{Y}_2+\cdots +a_n{Y}_n\end{array}$$

and

$$\begin{array}{l}{\hat{\beta }}_0=c_1{Y}_1+c_2{Y}_2+\cdots +c_n{Y}_n.\end{array}$$

But are these linear combinations the best possible linear combinations for estimating β₁ and β₀? The Gauss–Markov theorem is used to show that these estimators have the minimum variance of all possible unbiased estimators which are linear combinations of the dependent variables. These estimators are known as Best Linear Unbiased Estimators, typically abbreviated with the colorful acronym BLUE. The Venn diagram in Figure 1.13 might be helpful in categorizing the various types of estimators. The set L consists of all point estimators for the regression parameters β₀ and β₁ which can be expressed as linear combinations of the dependent variables [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex]. The set U consists of all point estimators for the regression parameters β₀ and β₁ which are unbiased estimators of β₀ and β₁. The shaded intersection of L and U (that is, [latex]L \cap U[/latex]) is all estimators which are both linear combinations of [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] and unbiased. An example of an estimator of β₁ which is neither in L nor in U is [latex]Y^{2}_{i}[/latex]. The Gauss–Markov theorem states that the least squares estimators have the smallest possible variance among all estimators in [latex]L \cap U[/latex].

The Gauss–Markov theorem indicates that the least squares estimators for β₀ and β₁ have minimal variance among all linear estimators. It does not indicate whether the least squares estimators for β₀ and β₁ have minimal variance among all estimators. The Gauss–Markov theorem extends to the case of multiple linear regression in which there are several independent variables. The least squares estimators are also the best linear unbiased estimators in this case.

To review the results that have been introduced so far, the simple linear regression model

$$\begin{array}{l}Y={\beta }_0+{\beta }_{1}X+ϵ\end{array}$$

defines a linear statistical relationship between an independent variable X, observed without error, and a random dependent variable Y as given in Definition 1.1. The point estimators for β₁ and β₀ from n data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)[/latex] using the least squares criterion are

$$\begin{array}{l}{\hat{\beta }}_1=\frac{S_{XY}}{S_{XX}}\mathrm{a}\mathrm{n}\mathrm{d}{\hat{\beta }}_0=\overline{Y}-{\hat{\beta }}_1\overline{X}\end{array}$$

as given in Theorem 1.1. The least squares estimators are unbiased estimators of their associated parameters because

$$\begin{array}{l}E[{\hat{\beta }}_1]={\beta }_1\mathrm{a}\mathrm{n}\mathrm{d}E[{\hat{\beta }}_0]={\beta }_0\end{array}$$

as given in Theorem 1.2. The least squares estimators of β₀ and β₁ can be expressed as linear combinations of [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] as

$$\begin{array}{l}{\hat{\beta }}_0=c_1{Y}_1+c_2{Y}_2+\cdots +c_n{Y}_n\mathrm{a}\mathrm{n}\mathrm{d}{\hat{\beta }}_1=a_1{Y}_1+a_2{Y}_2+\cdots +a_n{Y}_n,\end{array}$$

with coefficients [latex]c_{1}, c_{2}, \dots , c_{n}[/latex] and [latex]a_{1}, a_{2}, \dots , a_{n}[/latex] given in Theorem 1.3. The variance–covariance matrix of [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] is

$$\begin{array}{l}\left[\begin{array}{cc}V[{\hat{\beta }}_0]& \mathrm{C}\mathrm{o}\mathrm{v}({\hat{\beta }}_0,{\hat{\beta }}_1)\\ \mathrm{C}\mathrm{o}\mathrm{v}({\hat{\beta }}_1,{\hat{\beta }}_0)& V[{\hat{\beta }}_1]\end{array}\right]=\left[\begin{array}{cc}\sum _{i=1}^n{X}_i^2/\left(n{S}_{XX}\right)& -\overline{X}/S_{XX}\\ -\overline{X}/S_{XX}& 1/S_{XX}\end{array}\right]{\sigma }^2\end{array}$$

as given in Theorem 1.4. Finally, the Gauss–Markov theorem given in Theorem 1.5 states that the least squares estimators of β₀ and β₁ have the smallest population variance among all unbiased estimators that can be expressed as a linear combination of [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex].

The next section defines fitted values and residuals. Fitted values are the heights of the regression line associated with the observed values of the independent variable [latex]X_{1}, X_{2}, \dots , X_{n}[/latex]. The residuals are the vertical signed distances between the observed values of the dependent variable [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] and the associated fitted values that fall on the regression line. Residuals play an analogous role to the error terms in the simple linear regression model.

1.6 Fitted Values and Residuals

The simple linear regression model

$$\begin{array}{l}Y={\beta }_0+{\beta }_{1}X+ϵ\end{array}$$

was introduced in the previous section as a linear statistical model for describing the relationship between an independent variable X and a dependent variable Y. Taking the expected value of both sides of this equation yields

$$\begin{array}{l}E[Y]={\beta }_0+{\beta }_{1}X\end{array}$$

because [latex]E[\varepsilon] = 0[/latex] and X is a fixed value assumed to be observed without error, which are two key assumptions in Definition 1.1. When the population intercept β₀ and the population slope β₁ are replaced by their associated least squares point estimators [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] the resulting estimated regression line is

$$\begin{array}{l}\hat{Y}={\hat{\beta }}_0+{\hat{\beta }}_{1}X.\end{array}$$

This estimated regression line is typically plotted on a scatterplot that contains the data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)[/latex]. Seeing the data pairs and the least squares regression line on the same plot often makes the visual assessment of linearity easier. For any value X in which the simple linear regression model is valid, [latex]\hat{Y}[/latex] is the point estimator for the value of the dependent variable based on the data pairs and associated estimated regression line. This equation can be rewritten for the particular values of the independent variable collected as

$$\begin{array}{l}{\hat{Y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{X}_i\end{array}$$

for [latex]i = 1, 2,\ldots, n[/latex]. The value [latex]\hat{Y}_{i}[/latex] is known as the fitted value associated with data pair i, for [latex]null[/latex] [latex]i = 1, 2,\ldots, n[/latex]. When [latex]\hat{Y}_i \ne Y_{i}[/latex], which is almost always the case in applications, the fitted value does not fall on the estimated regression line; when [latex]\hat{Y}_i = Y_{i}[/latex], the fitted value falls on the estimated regression line. The next example illustrates the notion of fitted values for the sales data set.

The spread of the data pair [latex](X_{i}, Y_{i})[/latex] from the fitted regression line [latex]\hat{Y} = \hat{\beta}_0 + \hat{\beta}_{1}X[/latex] is reflected in the vertical signed distance between the data pair [latex](X_{i}, Y_{i})[/latex] and the associated fitted value [latex](X_{i}, \hat{Y}_{i})[/latex], These signed distances are known as the residuals, and are defined by

$$\begin{array}{l}{e}_i=Y_i-{\hat{Y}}_i\end{array}$$

for [latex]i = 1, 2,\ldots, n[/latex]. Data pairs that fall above the regression line correspond to positive residuals; data pairs that fall below the regression line correspond to negative residuals. The least squares approach used so far in estimating the intercept and slope of the regression line is a matter of finding the values of [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] which minimize the sum of the squares of the residuals. In other words, minimize

$$\begin{array}{l}S=\sum _{i=1}^n{e}_i^2.\end{array}$$

The fitted values and residuals are formally defined next.

Choosing to use the vertical distance between the observed value of the dependent variable and the regression line in the definition of the residual was based on the fact that the values of the independent variable [latex]X_{1}, X_{2}, \dots , X_{n}[/latex] are assumed to be observed without error in Definition 1.1. The mathematics associated with simple linear regression changes substantially if both X and Y are considered to be random variables.

A subtle but important distinction should be drawn between the model error term ϵ_i for data pair i and the residual e_i for data pair i. The model error terms are defined by

$$\begin{array}{l}{ϵ}_i=Y_i-({\beta }_0+{\beta }_1{X}_i)\end{array}$$

for [latex]i = 1, 2,\ldots, n[/latex], and represent the vertical distances between the observed dependent variable Y_i and the true (population) regression line [latex]Y=\beta_{0}+\beta_{1}X[/latex]. The simple linear regression model assumes that [latex]\varepsilon_{1}, \varepsilon_{2},\dots, \varepsilon_{n}[/latex] are mutually independent random variables. In nearly all applications, however, β₀ and β₁ are unknown. This means that for a particular data set, these model error terms are also unknown. On the other hand, the residuals are defined by

$$\begin{array}{l}{e}_i=Y_i-{\hat{Y}}_i=Y_i-({\hat{\beta }}_0+{\hat{\beta }}_1{X}_i)\end{array}$$

for [latex]i = 1, 2,\ldots, n[/latex], and represent the error for data pair i when compared to the estimated regression line [latex]\hat{Y}=\hat{\beta}_{0}+\hat{\beta}_{1}X[/latex], which is calculated from the n data pairs. Thus, [latex]\hat{\varepsilon}_{i} = e_{i}[/latex], for [latex]i = 1, 2,\ldots, n[/latex]. The [latex]e_{1}, e_{2}, \ldots, e_{n}[/latex] values are not mutually independent random variables because they must sum to zero. (This will be proven subsequently in Theorem 1.6.) For a particular data set, these residuals are known. The residuals are calculated for the sales data next.

A close inspection of the entries in Table 1.2 reveals that there are some curious outcomes that occur, such as

$$\begin{array}{l}\sum _{i=1}^n{e}_i=0\mathrm{a}\mathrm{n}\mathrm{d}\sum _{i=1}^n{Y}_i=\sum _{i=1}^n{\hat{Y}}_i.\end{array}$$

In other words, (a) the sum of the residuals is zero, and (b) the sum of the observed values of the dependent variable equals the sum of the fitted values. These were not just a matter of coincidence. The following theorem confirms that these relationships, along with a few other relationships, are true in general.

These five results from Theorem 1.6 will be illustrated for the sales data in the example that follows.

1.7 Estimating the Variance of the Error Terms

The emphasis so far has been focused on the estimation of the intercept and slope of the regression line. While [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] are the most critical parameters in most applications of a simple linear regression model, there is another parameter, the population variance of the error terms σ², which should also be estimated from the data pairs.

To establish a foundation for the estimation of σ², assume for this paragraph only that there is a univariate, rather than a bivariate, sample of values denoted by [latex]X_{1}, X_{2}, \dots , X_{n}[/latex]. These will not be fixed values observed without error as they were in regression modeling. It is assumed that these values constitute a random sample from a population that has finite population mean μ and finite population variance σ². The goal in this paragraph is to estimate σ² as a function of the data values. If the population mean μ is known (which is rare in practice), then an unbiased estimator of σ² is

$$\begin{array}{l}\frac{1}{n}\sum _{i=1}^n{(X_i-\mu )}^2.\end{array}$$

If the first [latex]n - 1[/latex] deviations between the sample values and the population mean [latex]X_1 - \mu, X_2 - \mu, \ldots, X_{n-1} - \mu[/latex] were known, the final deviation, [latex]X_{n} - \mu[/latex], would be free to take on any value. It is in this sense that the sum of squares

$$\begin{array}{l}\sum _{i=1}^n{(X_i-\mu )}^2\end{array}$$

is said to have n “degrees of freedom.” It is common practice in statistics to divide a sum of squares by its degrees of freedom to arrive at a point estimator. In this particular instance, dividing by n makes the point estimator an unbiased estimator of σ². The problem that arises more often in practice is to estimate σ² when μ is unknown. An unbiased estimator of σ² in this case is the sample variance

$$\begin{array}{l}\frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X}{)}^2,\end{array}$$

which is typically denoted by S² by statisticians. There are three reasons why the term outside of the summation has [latex]n - 1[/latex] in the denominator. The first reason is that this is the appropriate term so that this estimator is an unbiased estimator of σ². This can be stated as [latex]E[S^2] = \sigma^2[/latex]. The second reason is that one can’t estimate the dispersion of a distribution from a single data value, so the sample variance is undefined when [latex]n = 1[/latex]. The third reason is that the sum of squares has [latex]n - 1[/latex] degrees of freedom. One degree of freedom is lost because the sample mean [latex]\bar{X}[/latex] is used to estimate the population mean μ. If the first [latex]n - 1[/latex] deviations between the sample values and the sample mean [latex]X_{1} - \bar{X}, X_{2} - \bar{X}, \dots, X_{n-1} - \bar{X}[/latex] were known, the final deviation, [latex]X_{n} - \bar{X}[/latex], could be calculated from the other [latex]n - 1[/latex] values because

$$\begin{array}{l}\sum _{i=1}^n(X_i-\overline{X})=\sum _{i=1}^n{X}_i-n\overline{X}=0.\end{array}$$

It is in this sense that the sum of squares

$$\begin{array}{l}\sum _{i=1}^n(X_i-\overline{X}{)}^2\end{array}$$

is said to have [latex]n - 1[/latex] degrees of freedom. This ends the discussion of degrees of freedom for a univariate data set.

We now return to the problem of estimating σ² in simple linear regression. The independent variables [latex]X_{1}, X_{2}, \dots , X_{n}[/latex] are once again assumed to be fixed values observed without error as they have been throughout this chapter. Based on the fact that the error terms [latex]\varepsilon_{1}, \varepsilon_{2},\dots, \varepsilon_{n}[/latex] in the simple linear regression model are assumed to be mutually independent and identically distributed random variables, each with population mean 0 and finite population variance σ², the population variance of the error terms can be estimated with the unbiased estimator

$$\begin{array}{l}\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2=\frac{1}{n}\sum _{i=1}^n{(Y_i-{\beta }_0-{\beta }_1{X}_i)}^2\end{array}$$

if β₀ and β₁ were known. But in practice, the two parameters β₀ and β₁ are estimated from the data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)[/latex], so two degrees of freedom are lost and an appropriate point estimator for the population variance σ² is given by

$$\begin{array}{l}{\hat{\sigma }}^2=\frac{1}{n-2}\sum _{i=1}^n{e}_i^2=\frac{1}{n-2}\sum _{i=1}^n(Y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{X}_i{)}^2.\end{array}$$

It is important that the population variance of the error terms σ² remain constant over the range of X values in which the simple linear regression model is appropriate. One tool for visually assessing this assumption is a scatterplot of the data pairs with the estimated regression line superimposed.

The point estimator for σ² when β₀ and β₁ are estimated from the data pairs involves the sum of squares of the residuals, and this is often abbreviated as SSE, for sum of squares for error:

$$\begin{array}{l}SSE=\sum _{i=1}^n{e}_i^2,\end{array}$$

which is also known as the error sum of squares, residual sum of squares, and sum of squares due to error. When this quantity is divided by its degrees of freedom, it is known as the mean square error, which is abbreviated by MSE:

$$\begin{array}{l}{\hat{\sigma }}^2=\mathit{\text{MSE}}=\frac{SSE}{n-2}=\frac{1}{n-2}\sum _{i=1}^n{e}_i^2.\end{array}$$

Some good news is provided by the next result, which states that [latex]MSE = \hat{\sigma}^2[/latex] is an unbiased estimator of σ².

To summarize, there are three parameters in a simple linear regression model: the population intercept β₀, the population slope β₁, and the population variance of the error terms σ². These parameters can be estimated from n data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)[/latex] by the least squares method. Theorem 1.2 indicates that the least squares point estimator [latex]\hat{\beta}_0[/latex] is an unbiased estimator of β₀ and the least squares point estimator [latex]\hat{\beta}_1[/latex] is an unbiased estimator of β₁. Theorem 1.7 indicates that the MSE is an unbiased estimator of σ². All three parameter estimators are on target on average. The next three examples illustrate the estimation of σ².

The magnitude of the point estimate of σ² is a reflection of whether the data points are tightly clustered about the estimated regression line (for small values of [latex]\hat{\sigma}^{2}[/latex]) or whether the data points stray significantly from the estimated regression line (for large values of [latex]\hat{\sigma}^{2}[/latex]). In the previous example involving the sales data pairs, there is significant vertical deviation between the data points and the associated fitted values, as seen in Figure 1.15. The next example illustrates the case in which the data pairs are tightly clustered about the regression line.

In the two previous examples, point estimates of the population variance of the error terms σ² were calculated. In the sales data example, the estimated error term variance [latex]\hat{\sigma}^2 = 14[/latex] indicated that the data pairs strayed a large distance from the estimated regression line, as illustrated in Figure 1.6. In the Forbes data set, the estimated error term variance [latex]\hat{\sigma}^2 = 0.05421[/latex] reflects data pairs that cluster closely to the estimated regression line, as illustrated in Figure 1.18. But these two examples involving individual data sets do not indicate anything about the distribution of [latex]\hat{\sigma}^2[/latex]. The next example addresses this topic by extending the Monte Carlo simulation experiment from Example 1.4.

Before leaving the topic of the estimation of σ² behind, consider the case of collecting just [latex]n = 2[/latex] data pairs [latex](X_1, Y_1)[/latex] and [latex](X_2, Y_2)[/latex], as illustrated in Figure 1.22. One of the assumptions associated with the observations in a simple linear regression model is that there are at least two distinct values of the independent variable observed. So when [latex]n = 2[/latex], it must be the case that [latex]X_1 \ne X_2[/latex] In this case, the least squares regression line will pass through the points [latex](X_1, Y_1)[/latex] and [latex](X_2, Y_2)[/latex]. This means that the fitted values are identical to the data pairs, and hence, both residuals are zero. So the sum of squares for error is [latex]SSE = e^{2}_{1} + e^{2}_{2} = 0[/latex]. But is an SSE of zero an appropriate estimate for the population variance of the spread of the values about the regression line? Can one conclude that this is really a deterministic relationship and any additional data pairs collected will fall on the fitted regression line? Certainly not, because it is not possible to draw that conclusion based on just two data pairs. A third data pair might fall on the regression line or fall significantly off of the regression line, as was the case with the sales data from Example 1.3. The unbiased estimator of σ² is undefined because of the [latex]n - 2[/latex] in the denominator of the formula for [latex]\hat{\sigma}^2[/latex], as it should be. Two data pairs are adequate for estimating the population slope and population intercept of the regression line, but they are not adequate for estimating σ². The mathematics and intuition are consistent in this setting.

1.8 Sums of Squares

Certain sums of squares play a key role in simple linear regression. This section considers three topics related to these sums of squares: (a) partitioning the total sum of squares, (b) defining and interpreting the coefficient of determination and the coefficient of correlation, and (c) displaying the sums of squares in an ANOVA table.

1.8.1 Partitioning the Total Sum of Squares

A topic that is closely related to fitted values and residuals is the partitioning of the total sum of squares. Figure 1.23 provides the geometric framework for the mathematical derivation provided next. There are only three points plotted in Figure 1.23. The first point plotted is [latex](X_i, Y_i)[/latex], which is a generic data pair. The other [latex]n - 1[/latex] data pairs are not plotted in order to keep the figure uncluttered. The estimated regression line associated with the n data pairs, which happens to have a negative slope, is also plotted. The second point plotted is the fitted value [latex](X_i, \hat{Y}_i)[/latex] associated with the ith data pair, which is located directly below data pair i and falls on the estimated regression line. The third point plotted is [latex](\bar{X}, \bar{Y})[/latex], which, by Theorem 1.6, will always fall on the regression line.

Figure 1.23 provides a geometric proof of the relationship

$$\begin{array}{l}{Y}_i-\overline{Y}={\hat{Y}}_i-\overline{Y}+Y_i-{\hat{Y}}_i\end{array}$$

for [latex]i = 1, 2,\ldots, n[/latex]. The relationship can also be established algebraically by recognizing that the right-hand side of this equation can be determined by just adding and subtracting [latex]\hat{Y}_i[/latex] to the left-hand side of the equation. As will be stated and proved subsequently, squaring both sides of this equation and summing results in

$$\begin{array}{l}\sum _{i=1}^n(Y_i-\overline{Y}{)}^2=\sum _{i=1}^n({\hat{Y}}_i-\overline{Y}{)}^2+\sum _{i=1}^n(Y_i-{\hat{Y}}_i{)}^2.\end{array}$$

Long Description for Figure 1.23

“Two collinear data pairs (X i, Y i), (X i, Y cap i), and a data pair (X bar, Y bar) are plotted in the quadrant. A diagonal line with a negative slope passing through the points (X i, Y cap i) and (X bar, Y bar) is the regression line. The Y intercept of (X i, Y i) is indicated by a horizontal dotted line. Similarly, the X and Y intercepts of (X bar, Y bar) are indicated by dotted lines. The distance between the data pairs (X i, Y i) and (X bar, Y bar) is indicated as Y i minus Y bar. The distance between the data pairs (X i, Y i) and (X i, Y cap i) is indicated as Y i minus Y cap i. The distance between the data pairs (X i, Y cap i) and (X bar, Y bar) is indicated as Y cap I minus Y bar.”

This equation involves three sums of squares that occur so often in regression analysis that they are given the abbreviations

$$\begin{array}{l}SST=SSR+SSE,\end{array}$$

where SST stands for total sum of squares, SSR stands for sum of squares for regression, and SSE stands for sum of squares for error. (The sum of squares for error has already been encountered in Theorem 1.7.) This equation expresses the total variation of the observed values of the dependent variable [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] about their sample mean [latex]\bar{Y}[/latex] in SST as the sum of two sums of squares. The first term on the right-hand side, SSR, reflects the variation of the fitted values [latex]\hat{Y}_{1}, \hat{Y}_{2}, \dots , \hat{Y}_{n}[/latex] about the sample mean [latex]\bar{Y}[/latex]. The second term on the right-hand side, SSE, reflects the variation of the observed values [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] about their associated fitted values [latex]\hat{Y}_{1}, \hat{Y}_{2}, \dots , \hat{Y}_{n}[/latex]. Since all three terms in this equation are sums of squares, all three terms are nonnegative. Notice that [latex]SST / (n-1)[/latex] is the sample variance of [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex].

The equation

$$\begin{array}{l}SST=SSR+SSE\end{array}$$

partitions SST into two pieces: SSR, which accounts for the total variability in [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] that is accounted for by the regression line (that is, the linear relationship between X and Y), and SSE, which accounts for the remaining variability that is not associated with the regression line. This is why SSR measures the total variability in [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] “explained” by the relationship between X and Y, whereas SSE measures the total variability in [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] left “unexplained” by the relationship between X and Y. It is reasonable to think of SSR as measuring the “signal” associated with the linear relationship and SSE as measuring the “noise” associated with the linear relationship. The result is stated formally and proven next.

1.8.2 Coefficients of Determination and Correlation

There are two measures that are helpful in assessing the degree of the linear relationship between X and Y in a simple linear regression model. The coefficient of determination and the coefficient of correlation are defined next. The thinking behind the way that the coefficient of determination [latex]R^{2} = SSR / SST[/latex] is defined is as follows. The value of SST reflects the variability in [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] when the values of the associated independent variables [latex]X_{1}, X_{2}, \dots , X_{n}[/latex] are ignored. The value of SSE reflects the variability in [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] when a fitted regression model uses [latex]X_{1}, X_{2}, \dots , X_{n}[/latex] as predictors. Their difference, [latex]SSR = SST - SSE[/latex], reflects the reduction in variability associated with using the regression model. The ratio [latex]SSR / SST[/latex] captures the fraction of that reduction in variability.

The coefficient of determination R² is the fraction of the variation in [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] about [latex]\bar{Y}[/latex] that is accounted for by the linear relationship between X and Y. Based on the result from Theorem 1.8, [latex]SST = SSR + SSE[/latex], the coefficient of determination must satisfy [latex]0 \leq R^{2} \leq 1[/latex]. Likewise, the coefficient of correlation must satisfy [latex]-1 \leq r \leq 1[/latex], which is true for all population and sample correlations.

Values of R² that are near 1 indicate that nearly all of the variation in [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] about [latex]\bar{Y}[/latex] can be explained by the linear relationship between X and Y. This in turn implies that X is a useful predictor for Y. On the other hand, values of R² that are near 0 indicate that very little of the variation in [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] about [latex]\bar{Y}[/latex] can be explained by the linear relationship between X and Y. This in turn implies that X is not a useful predictor for Y. It is in this sense that R² is a measure of the strength of the linear relationship between X and Y.

There are some important limitations associated with R² and r. First, it is important to remember that the linear relationship between X and Y might only be appropriate on a limited range of X values. Second, even a relatively large value of R² might not provide the precision necessary for a particular application. Third, regardless of the value of R², the scatterplot of the data pairs must always be inspected to see if a simple linear regression model is warranted. Both high and low values of R² can be associated with a strong nonlinear relationship between X and Y. Fourth, in the case in which the experimenter can control the values of [latex]X_{1}, X_{2}, \dots , X_{n}[/latex], the magnitude of R² depends on the choices of the independent variables, which clouds its interpretation. Fifth, the usual interpretation of the coefficient of correlation r as an estimator of [latex]\rho = {\text{Cov}(X,Y)} / {\sigma_X \sigma_Y}[/latex] is only appropriate when X and Y are random variables, which is not the case in simple linear regression because X is assumed to be observed without error.

It is a useful thought experiment to consider the scatterplots associated with the values of SST, SSR, and SSE at their extremes. These three extreme cases will be described in the next three paragraphs.

The first of these extreme cases is illustrated for [latex]n = 7[/latex] in Figure 1.24 in which

$$\begin{array}{l}SSE=\sum _{i=1}^n(Y_i-{\hat{Y}}_i{)}^2=\sum _{i=1}^n{e}_i^2=0.\end{array}$$

The only way to achieve a sum of squares for error of zero is to have the data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)[/latex] all fall on a line, which is the regression line. Using the result from Theorem 1.8 that [latex]SST = SSR + SSE[/latex], in this case [latex]SST = SSR[/latex], which implies that [latex]R^{2} = 1[/latex]. Therefore, all of the variation in [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] is explained by the linear relationship between X and Y. In addition, [latex]r = -1[/latex] if the slope of the regression line is negative and [latex]r = 1[/latex] if the slope of the regression line is positive.

The second of these extreme cases is illustrated for [latex]n = 7[/latex] in Figure 1.25 in which

$$\begin{array}{l}SSR=\sum _{i=1}^n({\hat{Y}}_i-\overline{Y}{)}^2=0.\end{array}$$

The only way to achieve a sum of squares for regression of zero is to have an estimated regression line with slope zero. Using the result from Theorem 1.8 that [latex]SST = SSR + SSE[/latex], in this case [latex]SST = SSE[/latex], which implies that [latex]R^{2} = 0[/latex]. This means that none of the variation in [latex]Y_{1}, Y_{2}, \dots , Y_{n}[/latex] is explained by the linear relationship between X and Y. In addition, [latex]r = 0[/latex].

The third of these extreme cases is illustrated for [latex]n = 7[/latex] in Figure 1.26 in which

$$\begin{array}{l}SST=\sum _{i=1}^n(Y_i-\overline{Y}{)}^2=0.\end{array}$$

The only way to achieve a total sum of squares of zero is to have an estimated regression line with slope zero and all points lying on the estimated regression line. Using the result from Theorem 1.8 that [latex]SST = SSR + SSE[/latex], in this case [latex]SSR = SSE = 0[/latex], and the coefficient of determination and coefficient of correlation are undefined because the denominator is zero.

Each of the sums of squares has an associated degrees of freedom. The total sum of squares

$$\begin{array}{l}SST=\sum _{i=1}^n(Y_i-\overline{Y}{)}^2\end{array}$$

has [latex]n - 1[/latex] degrees of freedom for either of two reasons: (1) one degree of freedom is lost because [latex]\bar{Y}[/latex] is used to estimate the population mean, and (2) the terms in the summation above are subject to the one constraint—they must sum to zero. The sum of squares for regression

$$\begin{array}{l}SSR=\sum _{i=1}^n({\hat{Y}}_i-\overline{Y}{)}^2\end{array}$$

has 1 degree of freedom because each of the [latex]\hat{Y}_{i}[/latex] values is calculated from the same regression line which has two degrees of freedom, but is subject to the additional constraint [latex]\sum^{n}_{i=1} (\hat{Y}_{i} - \bar{Y})=0[/latex] by Theorem 1.6. The sum of squares for error

$$\begin{array}{l}SSE=\sum _{i=1}^n(Y_i-{\hat{Y}}_i{)}^2\end{array}$$

has [latex]n - 2[/latex] degrees of freedom for the reasons outlined just before Theorem 1.7.

An alternative definition for computing the coefficient of correlation r can save on computation time, as given in the following theorem.

1.9 Exercises

• 1.1 Establish a linear deterministic relationship between the independent variable X, the temperature in degrees Fahrenheit, and the dependent variable Y, the associated temperature in degrees Celsius.

• 1.2 Establish a nonlinear deterministic relationship between the independent variable X, the distance between two objects with fixed masses m₁ and m₂, and the dependent variable Y, the gravitational force acting between the two objects, using Newton’s Law of Universal Gravitation.

• 1.3 For the following interpretations of the independent and dependent variables, predict whether the estimated slope [latex]\hat{\beta}_1[/latex] in a simple linear regression model will be positive or negative.

  1. The independent variable X is a car’s speed and the dependent variable Y is the car’s stopping distance.
  2. The independent variable X is a car’s weight and the dependent variable Y is the car’s fuel efficiency measured in miles per gallon.
  3. The independent variable X is a husband’s height and the dependent variable Y is the wife’s height for a married couple.
  4. The independent variable X is the average annual unemployment rate and the dependent variable Y is the annual GDP for a particular country.

• 1.4 For the simple linear regression model, show that solving the [latex]2 \times 2[/latex] set of linear normal equations

  $$\begin{array}{ll}\hfill n{\hat{\beta }}_0+{\hat{\beta }}_1\sum _{i=1}^n{X}_i& ={\displaystyle \sum }_{i=1}^n{Y}_i\hfill \\ {\hat{\beta }}_0{\displaystyle \sum }_{i=1}^n{X}_i+{\widehat{\beta }}_1{\displaystyle \sum }_{i=1}^n{X}_i^2\hfill & ={\displaystyle \sum }_{i=1}^n{X}_i{Y}_i.\hfill \end{array}$$

  for [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] gives the expressions for [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] given in Theorem 1.1.

• 1.5 Consider the simple linear regression model

  $$\begin{array}{l}Y={\beta }_0+{\beta }_{1}X+ϵ,\end{array}$$

  where

  • the population intercept is [latex]{\beta}_0 = 1[/latex],
  • the population slope is [latex]{\beta}_1 = 1/2[/latex], and
  • the error term [latex]\varepsilon[/latex] has a [latex]U(-1,1)[/latex] distribution.

  Assume that [latex]n = 10[/latex] data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_{10}, Y_{10})[/latex] are collected. The values of the independent variable X are equally likely to be one of the integers [latex]0, 1, 2,\ldots, 9[/latex], What are the minimum and maximum values that the estimated parameters [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] can assume?

• 1.6 For the values of the independent variables [latex]X_{1}, X_{2}, \dots , X_{n}[/latex], show that

  $$\begin{array}{l}\sum _{i=1}^n(X_i-\overline{X})=0.\end{array}$$

• 1.7 Write R commands to plot contours of the sum of squares for the sales data pairs

  $$\begin{array}{l}(X_1,Y_1)=(6,2),(X_2,Y_2)=(8,9),(X_3,Y_3)=(2,2)\end{array}$$

  in the [latex](\beta_0, \beta_1)[/latex] plane.

• 1.8 The least squares criterion applied to a simple linear regression model minimizes

  $$\begin{array}{l}S=\sum _{i=1}^n(Y_i-{\beta }_0-{\beta }_1{X}_i{)}^2.\end{array}$$

  If instead the least absolute deviation criterion (also known as the minimum absolute deviation or MAD criterion) were applied to a simple linear regression model to minimize

  $$\begin{array}{l}S=\sum _{i=1}^n|Y_i-{\beta }_0-{\beta }_1{X}_i|,\end{array}$$

  what are the values of [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] for the sales data pairs

  $$\begin{array}{l}(X_1,Y_1)=(6,2)(X_2,Y_2)=(8,9)(X_3,Y_3)=(2,2)?\end{array}$$

• 1.9 Write a Monte Carlo simulation experiment that uses the same parameters as those in Example 1.4 (that is, [latex]\beta_0 = 1[/latex], [latex]\beta_1 = 1 / 2[/latex], [latex]\varepsilon \sim U(-1,1)[/latex], [latex]n = 10[/latex]) for 5000 replications, but this time selects the independent variable values to be equally likely integers from [latex]-5[/latex] and 5. Produce analogous figures to those of Figure 1.11 and Figure 1.12. Comment on your figures and how they relate to the variance–covariance matrix from Theorem 1.4.

• 1.10 For a simple linear regression model with [latex]X_{1} = 1, X_{2} = 2, \dots , X_{n} = n[/latex] and [latex]\sigma^{2} = 1[/latex] find the variance–covariance matrix of [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex].

• 1.11 Use Theorems 1.2 and 1.4 to show that the least squares estimator of the intercept of the regression line β₀ in the simple linear regression model is a consistent estimator of β₀.

• 1.12 Example 1.6 calculates the variance–covariance matrix for a single replication of a Monte Carlo simulation experiment. Conduct this experiment for 5000 replications and report the average of the values in the variance–covariance matrix.

• 1.13 Let L be the set of all linear estimators of the slope β₁ in a simple linear regression model. Let U be the set of all unbiased estimators of the slope β₁ in a simple linear regression model. Give an example of an estimator of β₁ in [latex]L \cap U'[/latex].

• 1.14 Show that the fitted simple linear regression model

  $$\begin{array}{l}{\hat{Y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{X}_i\end{array}$$

  for [latex]i = 1, 2,\ldots, n[/latex] can be written as

  $$\begin{array}{l}{\hat{Y}}_i-\overline{Y}={\hat{\beta }}_1(X_i-\overline{X}),\end{array}$$

  where [latex]\hat{\beta}_0[/latex] and [latex]\hat{\beta}_1[/latex] are the least squares estimators of β₀ and β₁ and [latex]\bar{X}[/latex] and [latex]\bar{Y}[/latex] are the sample means of the observed values of the independent and dependent variables.

• 1.15 Write a paragraph that argues why a fitted least squares regression line cannot pass through all data pairs except for one of the data pairs.

• 1.16 One of the most common error distributions used in simple linear regression is the normal distribution with population mean 0 and finite population variance σ², which has probability density function

  $$\begin{array}{l}f(x)=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/\left(2{\sigma }^2\right)}-\mathrm{\infty }<x<\mathrm{\infty }.\end{array}$$

  An alternative error distribution is the Laplace distribution with probability density function

  $$\begin{array}{l}f(x)=\frac{1}{\sqrt{2}\sigma }{e}^{-\sqrt{2}|x-\mu |/\sigma }-\mathrm{\infty }<x<\mathrm{\infty }.\end{array}$$

  Since the error distribution must have expected value zero by assumption, this reduces to

  $$\begin{array}{l}f(x)=\frac{1}{\sqrt{2}\sigma }{e}^{-\sqrt{2}\left|x\right|/\sigma }-\mathrm{\infty }<x<\mathrm{\infty }.\end{array}$$

  As parameterized here, the Laplace distribution has population variance σ². Both of these distributions are symmetric and centered about zero.

  1. Plot the normal and Laplace error probability density functions on [latex]-3 < x < 3[/latex] and comment on any differences between the two error distributions. Use [latex]\sigma = 1[/latex] for the plots.
  2. Plot the normal and Laplace error probability density functions on [latex]4 < x < 5[/latex] and comment on any differences between the tails of the two error distributions.
  3. Fit both of these error distributions (that is, find [latex]\hat{\sigma}^{2}[/latex] for each distribution) for the forbes data set from the MASS package in R using the simple linear regression model.

• 1.17 Let the independent variable X be a car’s speed and the dependent variable Y be the car’s stopping distance, which are going to be modeled with a simple linear regression model. In which of the following scenarios do you expect to have a larger population variance of the error term?

  1. The data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_{20}, Y_{20})[/latex] are [latex]n = 20[/latex] new cars that are all of the same make and model.
  2. The data pairs [latex](X_1, Y_1), (X_2, Y_2), \ldots, (X_{20}, Y_{20})[/latex] are [latex]n = 20[/latex] new cars from [latex]n = 20[/latex] different car manufacturers.

• 1.18 Show that the sum of squares for regression in a simple linear regression model can be written as

  $$\begin{array}{l}SSR={\hat{\beta }}_1{S}_{XY}.\end{array}$$

• 1.19 Show that the sum of squares for regression in a simple linear regression model can be written as

  $$\begin{array}{l}SSR={\hat{\beta }}_1^2{S}_{XX}.\end{array}$$

• 1.20 Consider the data pairs in the Formaldehyde data set built into the base R language. Use the help function in R to determine the interpretation of the independent and dependent variables. Fit a simple linear regression model to the data pairs and interpret the meaning of [latex]\hat{\beta}_0[/latex], [latex]\hat{\beta}_1[/latex], and [latex]\hat{\sigma}^{2}[/latex]. Also, calculate SST, SSR, and SSE for this data set.

• 1.21 Consider the data pairs collected by James Forbes that are given in the data frame forbes contained in the MASS package in R. The independent variable is the boiling point (in degrees Fahrenheit) and the dependent variable is the barometric pressure (in inches of mercury). For a simple linear regression model, calculate

  • the fitted values,
  • the residuals,
  • the sum of squares for error, and
  • the mean square error

  without using the lm function. Then use the lm function to check the correctness of the values that you calculate.

• 1.22 This exercise investigates the effect of controllable values of [latex]X_{1}, X_{2}, \dots , X_{n}[/latex] on the coefficient of determination R² in simple linear regression. Consider the simple linear regression model

  $$\begin{array}{l}Y={\beta }_0+{\beta }_{1}X+ϵ,\end{array}$$

  where

  • the population intercept is [latex]\beta_{0} = 1[/latex],
  • the population slope is [latex]\beta_{1} = 1 / 2[/latex], and
  • the error term [latex]\varepsilon[/latex] has a [latex]N(0,1)[/latex] distribution.

  Conduct a Monte Carlo simulation with 40,000 replications that estimates the expected coefficient of determination for [latex]n = 10[/latex] data pairs under the following two ways of setting the values of [latex]X_{1}, X_{2}, \dots , X_{10}[/latex].

  1. Let [latex]X_{i} = i[/latex] for [latex]1, 2, \dots, 10[/latex].
  2. Let [latex]X_{1} = X_{2} = \cdots = X_{5} = 5[/latex] and [latex]X_{6} = X_{7} = \cdots = X_{10} = 6[/latex].

• 1.23 Let S_X and S_Y be the sample standard deviations of the independent and dependent variables, respectively. Show that the following four definitions of the coefficient of correlation are equivalent.

  1. ${\displaystyle r=\frac{1}{n-1}\sum _{i=1}^n\left(\frac{X_i-\overline{X}}{S_X}\right)\left(\frac{Y_i-\overline{Y}}{S_Y}\right)}$
  2. ${\displaystyle r=\pm \sqrt{\frac{SSR}{SSE}}}$
  3. ${\displaystyle r=\frac{S_{XY}}{\sqrt{S_{XX}{S}_{YY}}}}$
  4. ${\displaystyle r={\hat{\beta }}_1\sqrt{\frac{S_{XX}}{S_{YY}}}}$