Lecture Notes

Chapter 9 Hypothesis Testing and Effect Size: Two Sample Tests (Modified: 2003-08-05)

• Experiments

  Experiments provide the only method in which you can specify cause and effect. Experiments are set up with minimum of two groups. One group, called the control group, is not given the experimental treatment by the experimenter. The other group, called the experimental group, is given some procedure, treatment, or substance by the experimenter. Everything else about the two groups is as similar as possible. More than one procedure, treatment, or substance can be tested at the same time by adding additional experimental groups.

  Bad Experiments

  A good way to learn about experiments is to look at bad ones. Examine the following bad experiments. Suppose you have created a new drug, a sleeping pill, call it Nox-Out©. Now, you are interested in marketing it by saying that it is better at putting people to sleep than other sleeping pills. So, you decide to conduct an experiment.

  First, you need some terminology. You already know that the control group is the group that does not get the procedure, treatment, or substance. In the Nox-Out© case the dose of Nox-Out© is the substance, here, the independent variable. So, now you can say that the control group does not get the independent variable. For contrast, the experimental group does get the independent variable. Finally, the dependent variable is how you measure the effects of the independent variable. Both groups will be measured in terms of the dependent variable. In this experiment, a good dependent variable might be hours of sleep after taking Nox-Out©.

  Now comes the first bad experiment. Suppose the subjects in the control group are all between 60 and 80 years of age, and the subjects in the experimental group are all between 20 and 30 years of age. When you measure both groups using the dependent variable of hours of sleep, you may find that the young group sleeps longer. But, did that difference in the dependent variable come from the independent variable or from the age difference? Here, you cannot tell. When two or more factors contribute to a dependent-variable difference, it is called a confound. Confounds are always a problem in experimentation, and they are the main reason that the study of experimental design is important.

  Here is another bad experiment. Now let's house the control group in the new Hilton Hotel, and let's house the experimental group in the El Sleazo Motel, the one with the neon light that flashes all night. Now our dependent variable difference may show that Nox-Out© does not work very well. Again, the problem is one of experimental design. Subjects just find it more difficult to sleep at the El Sleazo, even when they have taken Nox-Out©. This example, too, is a confound.

  A final bad experiment could be one in which you give Nox-Out© to the experimental group at 10:00 a.m. and then put them to bed; you put the control group to bed at 11:00 p.m. Again, the dependent-variable differences will lead us to believe that Nox-Out© does not work very effectively.

  The message here is pretty obvious. Good experiments should differentiate the experimental and control groups only by giving or not giving the independent variable. So, to properly test Nox-Out©, both groups should be of the same age, sex, and degree of sleepiness. Also, they should be tested in the same environment (e.g.,. the Hilton).

  Summary

  When all of these details are taken care of, and when a difference in the dependent variable exists, then the experimenter can say it was the independent variable that caused the difference. The ability to make such statements is the main benefit of the experimental method. However, just because a study is an experiment does not give it that power. There have been plenty of bad experiments like those described above. So, a sophisticated student will look beyond the dependent variable to the design and the conduct of the experiment in order to determine the validity of the conclusions drawn from the data.

• Analyzing Experiments
  • Null hypothesis: Treatment does not have an effect.
    • or: H₀: m_treatment - m_{no treatment} = 0; or H₀: m_treatment = m_{no treatment}
  • Alternative hypotheses: Treatment does have an effect
    • Two-tailed--the means are different, but no direction is specified
      • H₁: m_treatment <> m_{no treatment}
    • One-tailed--the means are different, and a direction is specified
      • H₁: m_treatment > m_{no treatment}
      • H₁: m_treatment < m_{no treatment}
  • The logic of experiments with more than two groups will be explained later
• Two t tests: Independent Samples (between Ss) and Correlated Samples (within Ss)
  • How to tell which design to use?
    • Ask yourself, what is happening to the subjects? If they are being paired, then use correlated sample design. If not, use the independent samples design.
  • Correlated Designs
    • Natural Pairs
      • these are pairs which already exist, the researcher did not create them (i.e., from same family)
    • Matched Pairs
      • these pairs are created by the researcher (i.e., split litters, yoked controls)
    • Repeated Measures
      • these pairs are when the same subject is measured twice or more, the pair is the set of scores from the same subject (i.e., before and after measures)
  • Independent Designs
    • subjects are randomly assigned (singly) to treatment or no treatment groups
  • Both Designs
    • have one independent variable with two levels (treatment or no treatment)
    • have one dependent variable and every subject has a score on that dependent variable
    • can be tested via two-tailed or one-tailed approach
    • can have random assignment (random assignment of pairs in correlated designs)
• Effect Size
  • Calculate the effect size (d) for two sample t-tests using the following formulas. Note that the formulas are different depending on whether you have equal numbers of participants in your two groups:

•  
• For correlated t-tests use:

 

• Confidence Intervals for Independent Samples

• Confidence Intervals for Correlated Samples

• In either case above, use t from Table D, not the t calculated!

• Reaching Correct Conclusions
  • This is a difficult topic and one that requires both statistical and experimental sophistication
  • Nature of Populations from Which Samples were Drawn
    • Both should be:
      • normally distributed
      • have equal variance
    • In practice, however, these assumptions are often violated by researchers
  • Control of Extraneous Variables
    • Control of extraneous variables is dealt with in Research Methods I
• Rejecting the Null Hypothesis: The Topic of Power
  • Power = 1 - b
  • "The more powerful the experiment, the more likely you are to detect a false null hypothesis." (p. 204)
  • Factors related to power:
    • Effect size
    • Standard Error of the Difference (smaller is better)
      • Sample size (bigger is better)
      • Sample variability (less is better)
    • a (bigger is better)
  • Power Analysis is a topic in intermediate statistics
• Computing t tests using FASTAT (we will do this one in class using Statistica)

• Paired Samples t-test on FAVORABL vs UNFAVORA with 11 cases.
• Mean difference = -4.727
• SD difference = 3.259
• DF = 10
• T = 4.812
• Prob = .001

Total observations:   11
 
                 FAVORABL    UNFAVORA
 
  N of cases               11          11
  Minimum               8.000      18.000
  Maximum              24.000      30.000
  Range                16.000      12.000
  Mean                 18.182      22.909
  Variance             26.764      14.691
  Standard dev          5.173       3.833
  Std. error            1.560       1.156
  Sum                 200.000     252.000
 

• Computing t tests using Statistica
  • In the computer lab
    • Click on "Start"
    • Click on "Programs"
    • Click on "Statistica"
  • Enter your data
    • for two sample t-tests you will use two columns
  • In this example, Select Stats, then select T-Test Correlated Samples
  • Choose on variable from each column (i.e., Favorabl and Unfavora)
  • Then click Ok
  • See example below: