Confidence interval and the Student's t-test 

By Joy Ying Zhang, joy+@cs.cmu.edu

• At a glance
• Main ideas
• Which t-test to use
• t-test for one variable: calculating confidence interval for mean
• Three types of t-test to compare two variables

• t-test: Two-Sample Assuming Equal Variances: 
• t-test: Two-Sample Assuming Unequal Variances
• t-test: Paired Two Sample for Means analysis

• Studentized bootstrap test
• Critical Values
• Confidence and Precision
• References

Main Ideas

The story starts here: Let's take a look at the Standard Normal Distribution. 
The standard normal distribution is a normal distribution with a mean of 0 and a standard deviation of 1.

Or we can view this in another way:

We can interpret the table above as: for 68.26% of times, z will fall into range (- 1,+ 1), for 95.44% z will be in the range (-2,+2) and for 99.74% z will have value in (-3,+3). Remember this, and we will be back. 

===========================

Let's switch the topic to confidence interval for a moment:

If we want to find out the mean of a population, but

• the population is very large
• the population is not accessible

We can only take samples from the population. Assume this population has a true mean μ and true standard deviation σ, but of course we don't know their values. Suppose each sample is of size n. For each sample, we can calculate the sample mean X  . The mean of all the sample means: X₁, X₂,X₃... is μ_X, and the standard deviation of the sampling distribution of the sample mean is σ_X , also called the standard error of X

BTW, There appear to be two different definitions of the standard error. 1) The standard error of a sample of sample size n is the sample's standard deviation divided by . It therefore estimates the standard deviation of the sample mean based on the population mean (Press et al. 1992, p. 465). Note that while this definition makes no reference to a normal distribution, many uses of this quantity implicitly assume such a distribution. 2) The standard error of an estimate may also be defined as the square root of the estimated error variance of the quantity,.

We have:

• μ_X = μ, meaning that the sampling distribution of  X is centered around the true population mean. And because μ_X = μ, the sample mean is called an unbiased estimator of μ
•  σ_X is the typical amount of error that will be incurred by estimating the population mean with the sample mean. Note that  σ_X=σ/ gets smaller as n gets larger. This means that X more accurately estimates the population mean for larger samples than smaller ones.  

In conclusion, suppose a sample of size n is taken from a normal population with mean μ and standard deviation σ, the sampling distribution of X is also a normal distribution with mean  μ_X = μ and standard deviation σ_X=σ/ . The sampling distribution is normal if the original population is normal.

Now, the standardized version of X is:

~ has a standard normal distribution

This means, whatever μ is, we have: 

(see table)

Or, in other words,

Or, that the interval contains the population mean (μ) with 99.54% confidence. This is a 99.54% confidence interval for μ.

In conclusion: 

We can measure the confidence intervals for the "real" mean μ if:

• Population is normal, or if the sample size is large
• σ is known.
• 100*(1-α)% confidence interval for the population mean is:
  

Here are some critical Z values. Z-values can be calculated and demonstrated here

=========================

In the above section, we assume that we know the standard deviation (σ) of the population and transferred the X into Z which is standard normal distribution and use the z-value to estimate the confidence intervals for the population mean μ

~ standard normal distribution 

Yet, in the cases  when σ is unknown, we can only estimate it with the sample standard deviation S and transfer the X into T which does not have a standard normal distribution. T follows what is called Student's t-distribution.

~ t-distribution

=========================

"Student" (real name: W. S. Gossett [1876-1937]) developed statistical methods to solve problems stemming from his employment in a brewery. 

The t-distribution has one parameter called the degree of freedom (df), DF=n-1

The t-distribution is similar to the normal distribution:

• Symmetric about 0
• Bell shaped

The main differences between the t-distribution and the normal distribution is in tails (Play around with DF and see the difference of the tails):

• T-distribution has larger tails than the normal
• Larger DF means smaller tails, the larger the DF, the closer to the normal distribution
• Small DF means larger tails

T-test for one variable: calculating confidence interval for mean μ, σ unknown

• Suppose a sample of size n is taken from a population with mean μ and standard deviation σ
  Assumptions:
  • Population is normal, or the sample is large
  • σ is unknown
• A 100*(1-α)% confidence interval for μ is:

        

Which T-test to use  

(for more information of how to choose a statistical test)

Goal and Data	Type of T-test	Assumption	Comments	T	DF
Compare one group to a hypothetical value	one-sample t test	Subjects are randomly drawn from a population and the distribution of the mean being tested is normal	Usually used to compare the mean of a sample to a know number (often 0)		n-1
Compare two unpaired groups	unpaired t test	Two-sample assuming  equal variance  (homoscedastic t-test)	Two samples are referred to as independent if the observations in one sample are not in any way related to the observations in the other. This is also used in cases where one randomly assign subjects to two groups, give first group treatment A and the second group treatment B and compare the two groups		n1+n2-2
		Two-sample assuming  unequal variance (heteroscedastic t-test)	The variance in the two groups are extremely different. e.g. the two samples are of very different sizes
Compare two paired groups	paired t test	The observed data are from the same subject or from a matched subject and are drawn from a population with a normal distribution does not assume that the variance of both populations are equal	used to compare means on the same or related subject over time or in differing circumstances; subjects are often tested in a before-after situation		n-1

Data set

t-Test: Two-Sample Assuming Equal Variances

To compute the two-sample t-test two major computations are needed before computing the t-test. First, you need to estimate the pooled standard deviation of the two samples. The pooled standard deviation gives an weighted average of the standard deviations of the two samples. The pooled standard deviation is going to be between the two standard deviations, with greater weight given to the standard deviation from a larger sample. The equation for the pooled standard deviation is:

In all work with two-sample t-test the degrees of freedom or df is:

The formula for the two sample t-test is:

For example, for this data set

t-Test: Two-Sample Assuming Unequal Variances

Assumption:

1. The samples (n₁ and n₂) from two normal populations are independent
2. One or both sample sizes are less than 30
3. The appropriate sampling distribution of the test statistic is the t distribution
4. The unknown variances of the two populations are not equal

Note in this case the Degree of Freedom is measured by 

and round up to integer.

For example, for this data set

Paired Student's t-test

For each pair of data, think of creating a new sequence of data: differences.

Hypothesis: Difference = μ, usually, if we just want to test if two systems are different μ=0

Apply the one-sample t-test on the difference sequence

Or,

Given two paired sets and of n measured values, the paired t-test determines if they differ from each other in a significant way. Let

with degree of freedom = n-1

For example, for the following data set:

Critical Values

You can find the t test critical values online

Or, you can use the perl library to calculate the values:

Statistics::Distributions - Perl module for calculating critical values and upper probabilities of common statistical distributions (download the package)

e.g.

$tprob=Statistics::Distributions::tprob (3,6.251); print "upper probability of the t distribution (3 degrees of " ."freedom, t = 6.251): Q = 1-G = $tprob\n";

Be Careful Here:

The returned p value stands for the proportion of the area under the curve between t and ∞, if one wants to measure the confidence C,

C=1-2p

Confidence and Precision

The confidence level of a confidence interval is an assessment of how confident we are that the true population mean is within the interval.
The precision of the interval is given by its width (the difference between the upper and lower endpoint).  Wide intervals do not provide us with very precise information about the location of the true population mean. Short intervals provide us with very precise information about the location of the population mean.

If the sample size n remains the same:

• Increasing the confidence level of an interval decreases precision
• Decreasing the confidence level of an interval increases its precision

Generally confidence levels are chosen to be between about 90% and 99%. These confidence levels usually provide reasonable precision and confidence.

References:

• Student's t-Tests
• Student's t-test on wikipedia
• The Insignificance of Statistical Significance Testing: What is Statistical Hypothesis Testing?
• Resampling techniques for statistical modeling
• Resampling and the Bootstrap
• Confidence Intervals for One Population Mean, by Dr. Alan M. Polansky
• Z-value
• Standard deviation and variance
• Difference of Two-Means Test
• Statistical inference