LEQ/YAR-PET
From Wilderdom
Assessing the Effects of Adventure-based Programs
on Outcomes for Youth-at-Risk Participants
Downloads
- YARPET package (.zip)
- instrumentation
- self-report
- observer
- administration guidelines
- briefing staff
- briefing students
- data entry
- instructions
- excel templates
- data analysis
- notes
- sample SPSS syntax
- instrumentation
Abstract
The rationale, development, and items for 17 typical youth development objectives of youth-at-risk adventure-based or experiential interventional programs are presented. The underlying intention of the instrument design was to provide an administratively efficient and psychologically valid method for assessing typically targeted youth development objectives for adventure-based youth at risk programs. The measured objectives emerged from consultation with numerous youth-at-risk programs. Sixty-five self-report measurement items are proposed to to assess the 17 youth development objectives (approximately 3 to 5 items per scale) in Personal, Social and Environmental domains. In addition, observer items have been developed. The style of the items and the measurement scale derives from the Life Effectiveness Questionnaire ( Neill, Marsh, & Richards, 1997), an instrument which has been widely used to measure the effectiveness of life skill education programs, particularly in outdoor education. It is recommended that potential users select from the 17 measured objectives in order to reduce the overall number of items and to customise the instrument so that its relevant to their program goals. Potential users should also consider that approximately half the scales have not been previously psychometrically tested, although their face validity appears reasonable. It is also requested that potential users read the conditions of use.
General description of the YAR-PET
The Youth at Risk Program Evaluation Tool (YAR-PET) consists of a set of 17 proposed factors and 65 measurement items designed to assess life skill and personal, social, community, and environmental development constructs typically targeted by intervention programs with youth 'at risk'. The measured objectives emerged from consulting with youth-at-risk programs about commonly sought outcomes and build upon previous constructs from the Life Effectiveness Questionnaire and the Review of Personal Effectiveness. Self-report and observer assessments can be gathered at least before an after an intervention program. Empirical analysis can then be used to estimate the degree to which intervention program participants may have experienced change with regard to the desired outcomes. Users are encouraged to select from amongst the factors and to consider developing new factors in order to customise the measurement to match specific program objectives. Limitations of the instrument include a lack of reported psychometric properties, thus reliability and validity is yet to be demonstrated.
Introduction: Evaluating Youth Development Outcomes
Measuring youth development outcomes in outdoor education
Good instrumentation is vital for good quality empirical research (Neill, 2003). Designing valid measurement of psycho-social constructs, particularly amongst youth at risk, has attracted considerable attention, but relatively little methodology and measurement development hasbeen directly applied to experiential and outdoor education settings. In other words, there are some useful research tools (e.g., Dahl & Reed, 1999), but none that are ideally suited to immediately measure all the objectives identified by any particular youth-at-risk adventure-based intervention program.
For example, Russell (2002) used the Youth Outcomes Questionnaire (Y-OQ) in his longitudinal assessment of seven outdoor behavioral health programs (which ran wilderness-based programs of between 21 and 56 days in length). Advantages of the Y-OQ include that self-report and parent observer versions exist, the psychometric development has been solid, and that clinically comparable norms are available. Downsides include the cost (it is a commercial instrument), its length, and that it lacks customizability for adventure-based youth-at-risk interventions.
Of course, no instrument is perfect, particularly when it is applied to a variety of circumstances. The current instrument is designed to provide a low-cost, adventure-oriented, user-friendly instrument for program evaluation and research into youth-at-risk programs which target typical youth development outcomes. Some other possible instruments and places to search are listed on the Tools, Instruments, and Questionnaires page.
Holistic range of youth development objectives
Experiential intervention programs for youth at risk typically target a holistic range of objectives. The proposed items are designed for use in longitudinal research and evaluation designs which aim to determine the degree to which a range of typical youth development objectives are achieved by outdoor and adventure intervention programs for youth at risk.
Guiding principles for this evaluation tool and system
The evaluation tool derives from a program evaluation system that has emerged from the Life Effectiveness Questionnaire project.
This new tool aims to provide for a rigorous and reasonably comprehensive evaluation of the extent to which program objectives are achieved in a cost and time efficient manner.
Most youth-at-risk programs are conducted by leaders and administrators who do not have a background in research, but who are skilled in report writing and knowledge about their own program. Thus, as long as program operators are supported in selecting an appropriate evaluation methodology, and with good quality data analysis, they can construct good quality reports to funders or evaluators, or for internal program development.
This instrumentation is also designed to be user-friendly for participants, program instructors, and administrators.
The self-report tool can be administered pre- (Time 1) and post-program (Time 2) to get a measure of the short-term program impact. It is recommended, however, that baseline and followup assessments are obtained, as well as observer measures (e.g., by leaders, parents, teachers, and/or peers).
This evaluation tool is also designed to be customizable. Program evaluators too often use full instruments, when only part of the instrument is relevant to program objectives, for fearing of undermining the overall validity of the questionnaire. With this instrument, the goal is to get the best of both worlds - psychometric validity and flexibility. Evaluators can select from the pool of 17 scales, to develop a set of items for evaluating the effects of a particular program.
Developing the list of objectives and the item pool
An initial youth development objectives was developed through consulting with a range of youth-at-risk programs and outdoor education organizations. In addition, current instruments (e.g., the Life Effectiveness Questionnaire) and lists of youth development objectives (e.g., the Search Institute's 40 Developmental Assets) often referred to by adventure-based youth-at-risk programs were consulted. Objectives most commonly cited by youth at risk programs were emphasized in selecting the objectives for measurement.
Electronic database searches were then conducted for existing instrumentation which could be adapted. Where suitable instrumentation was not identified, new items were constructed. Note that where there was existing instrumentation, three items per scale are recommended. For new items, five items per scale are used with the intention of refining this to three items per scale when sufficient data is available for psychometric analysis. Potential users should view the proposed tool as a promising pilot tool which is ready for use, but which requires establishment of reliability and validity through analysis of initial data.
Youth Development Objectives
Personal objectives include self-esteem and self-confidence, managing personal risks, managing health and well-being, setting and achieving appropriate goals, making decisions effectively, and physical and outdoor skill development.
Social objectives include being able to work in a team, taking on leadership opportunities, conflict resolution skills, listening skills, and engagement with community.
Environmental awareness and skills include knowledge and interest in local environment.
Institutional outcomes include recidivism and completing of intervention programs. The current measurement tool does not assess institutional outcomes since these are usually derived from existing records.
Table 1. Seventeen youth development objectives which are typical of adventure-based youth-at-risk programs
|
Youth development objectives (17) |
Description |
|
Personal (12) |
' |
|
Sense of personal worth and value | |
|
General confidence in one's capacity to manifest one's goals in life | |
|
Sense of control over one’s own life | |
|
The extent to which a person takes effective action to solve problems | |
|
Ability to set/achieve goals | |
|
Effective use of journaling to reflect on self, others, and experiences | |
|
Finds creative avenues for expressing thoughts and feelings such as through art, drama, or music | |
|
Know difference between healthy risks (outdoor adventure) and unhealthy risks | |
|
Physical & outdoor skill development |
Physical competence, e.g., physical fitness, competency in kayaking |
| Social (6) | |
| Respect/understand personal boundaries |
Understands issues of personal space, touch, appropriate conduct |
|
Ability to avoid and resolve interpersonal and group conflicts | |
|
Effectively communicates with other people in interpersonal and group settings | |
|
Work and cooperative with others to achieve group tasks | |
|
Ability to take on role of leader when necessary | |
|
Meaningful engages with community | |
|
Environmental (2) | |
|
Sense of responsibility for quality of surrounding environment and conservation | |
|
Understanding of issues affecting local ecosystem |
Searching for Measurement Methods
Generating the items
Initially, searches were conducted for existing instrumentation to measure each of the objectives. Relevant scales and items were consulted and adopted or adapted as appropriated. For approximately half the objectives, items from existing measurement tools were used or adapted; for the other objectives, new items were generated. More detailed academic background behind each of the proposed items will be provided in a separate report.
Improving reliability of the measurement
In order to reliably assess the “fuzzy” constructs, several general strategies were adopted / are suggested:
- Where appropriate, intact items were used directly or slightly adapted from other questionnaires. In such cases, the best three items per scale were selected. Three items per scale (outcome) is generally considered the minimum necessary to create a psychometrically reliable measure. For scales where major adaptations or item generation was performed, a minimum of five items were selected.
- Collect multiple sources of data (i.e., items were constructed so that they could be completed as both self-report and observer-report)
- The psychometrics and practical utility of those who trial evaluations using these scales should be systematically examined to improve the reliability of future evaluation efforts.
The measurement scale
The recommended measurement scale on which each item would be rated is based on that used for the “Life Effectiveness Questionnaire” (LEQ), a well-known and widely used tool for assessing personal development outcomes from outdoor education programs. An 8-point Likert rating scale is used, as follows:
| False Not Like Me | True Like Me | ||||||
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| This statement doesn't describe me at all; it isn't like me at all | More false than true | More true than false | This statement describes me very well; it is very much like me | ||||
The measurement items
The proposed measurement items are in draft format. In a program’s final questionnaire, the items would be intermixed. For review, however, they are presented within their target categories.
For each outcome, self-report items and one or two observer items are indicated. Observer items would be completed by the instructors and/or youth workers who know the participants. Bold items indicate the most prototypical item in each scale – these items are recommended for use in a short version for use with participant groups who may have difficulty completing the longer version.
Important criteria to be considered in review the utility of the items:
- Do the items target reflect the program’s intended objectives?
- Will typical clients be able to provide an appropriate response?
Important Decisions to be Made
The total item pool of 65 items may be too onerous for participants. Completion of the 65 self-report items is predicted to involve approximately 30 minutes, but for slower participants could conceivably take up to 45 minutes.
Possible strategies for reducing the number of items:
- Reduce the number of objectives being measured: Perhaps most important is that the selection of scales to include should reflect program objectives e.g., rank the 17 outcomes from most to least important and select those which really matter. It is generally better to measure the most important constructs and make good use of the outcome data than to simply measure lots of outcomes.
- Reduce the number of items per factor: The proposed items include the minimum per scale that is recommended for reliable self-assessment of these constructs. However, the best single item has been indicated for each scale, to allow for a short (1 item per scale) instrument. Alternatively, a 2-item per scale instrument could be constructed, consisting of 34 items, which would take approximately 15 minutes to complete. This would provide broad coverage in a less demanding format for participants. The data would be useful for program evaluation purposes, but as on ongoing tool, the instrument may not be ideal for future reporting which would be scrutinized for its academic rigor.
A further consideration is whether you would like to also include direct measure of participants' mental health and psychological well-being. If so, there are many possibilities, but consider a short version of the Mental Health Inventory which measures psychological well-being (10 items) and psychological distress (10 items) in adolescents (Heubeck & Neill, 2000).
Table 2. Summary of the youth development objectives and their proposed measures
|
Program Outcome |
Proposed Scale |
|
Personal (9)
Social (6)
Environmental (2)
| |
|
References
Dahl, & Reed (1999). The challenges of measuring the impact of recreation programs on youth resiliency: Designing culturally sensitive and age-appropriate evaluation instruments. California Parks and Recreation magazine, Summer.
Frydenberg, E., & Lewis, R. (1993). Adolescent Coping Scale: Administrator's manual. The Australian Council for Educational Research, Hawthorn, Australia.
Heubeck, B. G., & Neill, J. T. (2000). Internal validity and reliability of the 30 item Mental Health Inventory for Australian Adolescents. Psychological Reports, 87, 431-440.
Marsh, H. W. (1990). Self-Description Questionnaire - II: Manual. Sydney: University of Western Sydney.
Neill, J. T. (2004). How to Choose Tools, Instruments, & Questionnaires for Intervention Research & Evaluation. Outdoor Education Research & Evaluation Center.
Neill, J. T., Marsh, H. W., & Richards, G. E. (1997). Development and psychometrics of the Life Effectiveness Questionnaire. Sydney: University of Western Sydney.
Richards, G. E , Ellis, L. A., Neill, J. T. (2002). The ROPELOC: Review of Personal Effectiveness and Locus of Control: A comprehensive instrument for reviewing life effectiveness</u>. Paper presented at Self-Concept Research: Driving International Research Agendas, 6-8 August, Sydney.
Russell, K. (2002). Longitudinal assessment of treatment outcomes in outdoor behavioral healthcare. Technical Report 28. Wilderness Research Center, University of Idaho, Moscow, ID. [.16 MB, .pdf]
