Tutorial 5

Advanced ANOVA, Power & Effect Sizes

Assumed knowledge
Data files
General steps
Visual ANOVA
Francis
Design your own ANOVAs
Interactions
Follow-up tests
Power
Effects sizes
Confidence intervals
Error bar graphs

Assumed knowledge

Properties of the normal distribution, independent, paired and single-sample t-tests, one-way ANOVA, MLR

Data files

General steps

Establish hypothesis/hypotheses
Identify IV/IVs (categorical), DV/DVs (at least interval), and CV/CVs (dichotomous or at least interval)
Examine assumptions
- The data in each cell is normally distributed
- Homogeneity of variance (the variance in each cell is similar)
Create descriptive statistics and graphs
Conduct significance test for main effects and interactions
Interpret significance and direction of effects
Interpret effect sizes (later in this tutorial)
- eta-square (omnibus)
- standardised mean effect size (difference b/w two means)

Visual ANOVA Exercise

Understanding ANOVA Visually
Under what conditions would F be the smallest?
Under what conditions would F be the largest?
Now explore the same ideas with this more advanced Visualisation Tool for One-way and Two-way ANOVA Applet

Francis

3.3 (Analysing Differences)
5.2 (ANCOVA)
5.3 (MANOVA)
(indepth knowledge of MANOVA is not examinable, but understanding when you would use MANOVA is important)

Design your own ANOVAs

Conduct and interpret your own:
(including descriptives, graphs, F tests, and follow-up tests as appropriate)
- one-sample t-test
- independent samples t-test
- paired samples t-test
- one-way ANOVA
- repeated measures ANOVA
- factorial ANOVA
- split-plot ANOVA (SPANOVA or mixed ANOVA)
- ANCOVA
- MANOVA

Understanding Interactions

Notes on understanding interaction
Create your own data set for conducting some mock 2-way ANOVA analyses
Adjust the data and / or ANOVA analyses in order to create the following (include ANOVA results and appropriate graphs):
1. No effects
2. Main effect A, no main effect B, no interaction
3. Main effect A, no main effect B, interaction
4. No main effect A, main effect B, no interaction
5. No main effect A, main effect B, interaction
6. Main effect A, main effect B, no interaction
7. Main effect A, main effect B, interaction
8. Interaction, no main effects
If unsure, you can download and try out interactions.sav (dataset), interactions.sps (syntax file), and interactions.spo (output file).

Follow-up Tests

The first steps here I can suggest are to consult:

Howell (Fundamental Statistics): Section 16.5: Multiple comparison procedures (375-383)
Howell (Statistical Methods): Chapter 12: Multiple comparison among treatment means (343-389)
Francis: Section 3.3.6.1: Post hoc tests and planned contrasts (61-63)
Francis: Section 3.3.8.4: Planned contrasts for within subjects ANOVA (71-71)

For a factorial ANOVA, if you get a significant F for an IV which has more then 2 groups and you had made no hypotheses, then your main options are to followup with post-hoc tests, choosing among:

Fisher's Least Significant Difference (LSD) (or protected t test))
Bonferroni
Tukey's test (or Tukey's Honestly Significant Difference (HSD))
Scheffe

In order to get these analyses:
SPSS > Analyze > General Linear Model > Univariate > Insert DV and Fixed Factors (IV) > Post-hoc > Insert Factors for post-hoc analysis > Tick the boxes for the post-hoc tests you want ---> OK

You only need to report one set of post-hoc analyses.

Once you get the results, interpretation is pretty straightforward, because you will have a series of comparison tests between each pair of means, showing either significant or non-significant differences.

Power

Power.xls
If I am using a confidence interval of 95% and I want a power level .8, what would be the minimum sample size to detect a standardised mean difference of .3?
If I am using a confidence interval of 99% and I want a power level of .7, what would be the minimum sample size to detect an effect size of .7?
If I am using a confidence interval of 90% and I want a power level of .9, with a sample size of 200, what effect size will I be able to detect?
If I am using a confidence interval of 95% and I will have a sample size of 50, and I want a power level of .8, what effect size will I be able to detect?

Effect Sizes (Cohen's d)

Between groups

Cohensd.xls
Cohen's d is the recommend effect size for expressing the difference between two means. The following samples involve computing the effect size between two independent means which you can work out using the downloadable calculator.
What is the effect size between two means (and how large is it?), where M1 = 10, SD1 = 3, M2 = 12.5, SD2 = 4
- Using a 99% confidence interval, N = 20 in each group, is this a significant difference?
- Using a 95% confidence interval, N = 20 in each group, is this a significant difference?
- Using a 90% confidence interval, N = 20 in each group, is this a significant difference?
- What happens if we increase the N in each group to 100?
What is the effect size between two means (and how large is it?), where M1 = 8, SD1 = 2, M2 = 3.5, SD2 = 2
- Using a 99% confidence interval, N = 40 in each group, is this a significant difference?
- Using a 95% confidence interval, N = 40 in each group, is this a significant difference?
- Using a 90% confidence interval, N = 40 in each group, is this a significant difference?
- What happens if we decrease the N in each group to 10?
What is the effect size between two means (and how large is it?), where M1 = .5, SD 1= .1, M2 = .6, SD2 = .07
- Using a 99% confidence interval, N = 200 in each group, is this a significant difference?
- Using a 95% confidence interval, N = 200 in each group, is this a significant difference?
- Using a 90% confidence interval, N = 200 in each group, is this a significant difference?
What is the effect size between two means, N = 15, where M1 = 6.5, SD1 = 1, M2 = 7.5, SD2 = 1.1
- What is the confidence interval? Using only, the confidence interval, is this difference significant?
- How large would N need to be to get a just significant result?
- Using N = 15, how large would M2 need to be to get a just significant result?

Repeated measures

There is controversy about computing effect sizes for repeated measures. See Effect Size Measures for Two Dependent Groups:

"The ES computed using the paired t-test value will always be larger than the ES computed using a between groups t-test value, or the original standard deviations of the scores...However, Dunlop, et al. convincingly argue that the original standard deviations (or the between group t-test value) should be used to compute ES for correlated designs. They argue that if the pooled standard deviation is corrected for the amount of correlation between the measures, then the ES estimate will be an overestimate of the actual ES....In summary, when you have correlated designs you should use the original standard deviations to compute the ES rather than the paired t-test value or the within subject's F value."

If you would like to calculate the repeated measures Cohen's d, based on the pooled standard deviation corrected for the correlation, use this calculator (Note: You need to input the correlation.)
Cohensdrepeatedmeasures.xls

Confidence Intervals

Confidence Intervals for Cohen's d effect sizes can be calculated using the above calculators.
When N increases, the CI decreases - see this java applet. When you hit Run, it will start sampling and show the means and 95 % CIs for each. Stop the apple, change the desired N, and start again. Compare the obtained distributions for samples with different Ns.

Error Bar Graphs

Use any dataset
Conduct a one-way ANOVA and graphically present the means and confidence intervals using an Error Bar Graph - is this error bar chart consistent with the statistical results?
Conduct a two-way ANOVA and graphically present the means and confidence intervals using an Error Bar Graph - is this error bar chart consistent with the statistical results?