The other consideration in the choice of statistical procedures is power, defined as the sensitivity to detect group differences when in fact they exist. Among other things affecting power is the size of the sample and the choice of the statistical method. Often in evaluation research, the tester must deal with small samples. The t-test is most useful for comparing groups when the samples are small—30 or fewer people. However, if you have fewer than 10 or 15 people per group, the power diminishes to the point that it may be better to abandon the entire evaluation.
To measure the impact of its sales-training course, Chicago-based R.R. Connelley & Sons used the t-test. The t-test is used to test whether two means or averages, such as the average productivity measures of the individuals within a trained group and an untrained group, are statistically significantly different.
Computational procedures result in a t value and associated probability level (p-value). A probability level of .05 or less indicates that the two means are indifferent. More specifically, it means that the testers can be 95% confident that the differences and not simply a result of how the samples were drawn from the population.
If you hypothesize a direction of the difference between two means (such as, mean A is larger than mean B), then a one-tailed t-test is used. This test is more sensitive and powerful for detecting differences than a two-tailed t-test. This is because a p-value of .05 (95% confidence level) is divided among the two tails of the distribution of scores around the mean. As a result, the mean of the comparison group must fall within either .025 region (above or below the mean) of the distribution rather than a larger .05 area when a one-tailed test is used.
Computational procedures for the t-test are available in any introductory statistics text and on commercially available software programs, such as Microsoft Excel and Statistical Package for the Social Sciences.
Personnel Journal, February 1994, Vol.73, No. 2, p. 87.