1


2

 To explain
 how effect sizes and confidence intervals
 can be employed
 in primary studies
 as indicators of amount of psychological change.

3

 efficacy of the intervention
 validity, sensitivity and relevance of the DV
 appropriateness of the analyses
 interpretation of the results
 > understanding and controlling of causative processes

4

 The Significance Testing Controversy
 What is Metaanalysis?
 Effect Sizes
 Interpretation of Effect sizes
 Confidence Intervals
 Graphical Displays
 Benchmarking & Comparisons
 Future directions

5

 Statistical significance testing was developed by Fisher to determine
whether some agricultural techniques were superior to other techniques

6

 Statistical significance in a study with:
 N=10?
 N=100?

7

 Statistical significant testing has been utilised with little adaptation
in psychological research, even though quite different questions are
often being asked
 This has undermined the value of much psychological research

8

 Calls for a shift away from significance testing have been largely
unheeded for approx. 30 years

9


10

 Power ~.60 in social science research
i.e. on average, 40% chance of Type II error
 Under reporting of power
 Under reporting of effect sizes

11

 “Despite numerous efforts to change selfconcept there appears to be no
consistent answer as to whether it is possible”
 Janet Hattie (1992, p.221)

12

 Ways of Measuring Psychological Change
Clinical Observation/Opinion
 Difference Scores
 T Scores
 Significance Testing
 Effect Sizes & Confidence Intervals

13

 Ways of Reviewing Research on Psychological Change
Traditional Literature Review
 Vote Counting
 Secondary Analysis
 Metaanalysis
 Megaanalysis

14

 Psychotherapy Debate
 To counter what appeared to
be
selectivity of studies
included in a review of
psychotherapy effects by Eysenck,
Glass introduced a
procedure
he termed metaanalysis.
[1976,1977]

15

 Equivalent to traditional (qualitative) review paper
 Enters summary quantitative data from each study into a new database,
with IV codings
 Overall effects are summarised and variance predicted
 Used in medicine, psychology and education
 Outcome measure of interest is the ‘effect size’

16

 A standardised measure of
 ‘how much change’ OR
 ‘how much shared variation’

17

 Cohen’s d
 Hedge’s g
 Pearson’s r
 ANOVA  etasquare, omegasquared
 Regression  R squared
 Categorical  Phi & Cramer’s V

18

  norms
  control group
  pooled

19

 A measure of
the difference between two means
in standard deviation
units.
d is equivalent to the differences between two z scores

20

 ve = negative change
 0 = no change
 +ve = positive change

21

 Cohen (1977): .2 = small

.5 = moderate

.8 = large
 Wolf (1986): .25 = educationally
significant

.50 = practically
signficant
(therapeutic)
 ESs are proportional, e.g.,
.40 is twice as much change as .20

22

 No agreed standards
 Interpretation is subjective
 Best approach
 compare with previous findings

23

 Adult psychotherapy outcomes
: .68
(Smith, Glass & Miller, 1980)
 Children psychotherapy outcomes
: .71
(Casey & Berman, 1980)
 Classroom intervention  Achievement
: .40
(cited in Hattie, Marsh, Neill, & Richards, 1997)
 Classroom intervention  Affective : .28
(cited in Hattie, Marsh, Neill, & Richards, 1997)
 Selfconcept intervention programs
: .37
(Hattie, J.A., 1992)

24


25

 Adolescent OE programs (43 studies) : .31
(Cason & Gillis, 1994)
 All OE research (96 studies) : .34
(cited in Hattie, Marsh, Neil, & Richards, 1997)
 Adventure Therapy  LOC : .38
(Hans, 1997, 2000)
 USA summer camps with selffocus
: .41
(cited in Hattie, Marsh, Neil, & Richards, 1997)

26


27


28

 Psychotherapy
30% improvement for average client
 Classroombased affective programs
11% improvement for average students
 Outdoor education
13% improvement for average participant
 65% of OE participants are better off
than people who don’t do an OE program
(35% are not better off!)

29


30


31

 efficacy of the intervention
 validity, sensitivity and relevance of the DV
 appropriateness of the analyses
 interpretation of the results
 > understanding and controlling causative processes

32

 Benchmarking for program evaluation and quality assurance
 Increasing opportunities for cumulative, primary data research
 MA may become common expectation for literature reviewing

33

 Use MAs and ESs in your literature reviews
 Report ESs and CIs for your primary data
 Discuss relevant ES comparisons
 Suggest benchmarks
 When reporting significance, report power

34

 Abbott, C. (1987). Does outdoor education really work? Perks, research and reality. Journal of Adventure Education and
Outdoor Leadership, 4(2), 2225.
 Caldarella, P., & Merrell,K.W. (1997). Common dimensions of social
skills of children and adolescents: A taxonomy of positive
behaviors. School Psychology
Review, 26, 264278.
 Cason D., & Gillis, H.L. (1994). A metaanalysis of outdoor
adventure programming with adolescents. Journal of Experiential
Education, 17(1), 4047.
 Cohen, J. (1977). Statistical power analysis for behavioral sciences (revised
ed.). New York: Academic Press.
 Hans, T. (1997). A metaanalysis of the effects of adventure programming
on locus of control. Unpublished Master of Science thesis, Psychology
Graduate Faculty, Georgia College, Milledgeville, GA.
 Hans, T. (2000). A metaanalysis
of the effects of adventure programming on locus of control. Journal of Contemporary Psychotherapy,
30(1),3360.

35

 Hattie, J.A. (1992). Enhancing selfconcept. In J.M. Hattie. Selfconcept (pp.221240).
New York: Lawrence Erlbaum.
 Hattie, J. (1992). Measuring the effects of schooling. Australian Journal of Education, 36(1),
513.
 Hattie, J. (1992). Selfconcept. New York:: Lawrence Erlbaum.
 Hattie, J., Marsh, H.W., Neill, J.T. & Richards, G.E. (1997).
Adventure Education and Outward Bound: Outofclass experiences that
have a lasting effect. Review of Educational Research, 67, 4387.
 Lawson, M. (1997, November 24). Wilderness training yet to prove its
worth.. The Australian Financial Review (p.7).
 Neill, J.T., & Richards, G.E. (1998). Does outdoor education really
work? A summary of recent
metaanalyses. Australian Journal
of Outdoor Education, 3(1), 29.
 Marsh, P.E. (2000).*
 Smith, M.L., Glass, G.V., & Miller, T.I. (1980). The benefits of
psychotherapy. Baltimore: Johns Hopkins University Press.
 Wolf, F.M. (1986). Metaanalysis: Quantitative Methods for Research
Synthesis. Beverly Hills, CA: Sage.
