Five Guiding Principles for Instrument Development (LEQ)
16 Nov 2005
Five guiding principles were used in the development of the LEQ:
Length and Complexity
On outdoor and experience-based programs, instruments are often administered in field settings (e.g. in the bush, on board a boat, in various weather conditions), on multiple occasions (e.g. pre-program, first day of program, last day of program and post-program followup) and to a wide range of participants (e.g. people with learning disabilities, people without English as their first language, school children, corporate managers). Hence, the shorter and simpler an instrument (reliability and validity aside), the greater the instrumentís potential applicability. The aim was to develop an instrument which would provide a maximum amount and type of unique information in as short a time as possible (i.e., a maximum of about 10 minutes). The instrumentís instructions and layout also needed to be straight-forward to allow people without research experience, such as group leaders or teachers, to administer the instrument consistently across different groups.
Relevance to Program Aims
Generally, a major aim of many outdoor experiential programs is to facilitate individualsí personal development in a broad range of life skills (e.g. self-confidence, initiative, communication skills, etc.), although different programs may have more specific aims such as the development of teamwork and leadership skills. Ideally, the instrument would encompass a wide range of life proficiency domains relevant to general and specific program aims, so as to allow for within and between program comparison of different program outcomes.
Assessment of Competence
The range of life skills assessed by the instrument should be necessary, or at least beneficial, to effective and successful living and working. The greater the specificity of a measure, the more likely it is to predict actual performance (Blau, 1993). Hence, the aim was for the instrument items to focus on self-assessment of competence at practical skills which are relevant to a broad range of personal and professional situations.
Sensitivity to Change
The response scale is an important aspect of an instrumentís sensitivity to change. A dichotomous (yes/no) scoring does not provide much sensitivity, whereas a large range can reduce the instrumentís reliability. A balance needs to be reached between sensitivity to change and reliability. Despite being a critical issue, a search of the literature revealed little research, for example, on the effects of Likert-type scales with different numbers of responses for measuring change.
Two further issues to be considered in developing an instrument for measuring change are ceiling/floor effects and test-retest correlations. It is desirable that the wording of the items and the response scale produce responses from participants that leave room for detecting shifts in self-perceptions up or down -- hence the means should be examined during the instrument development. In addition, test-retest correlations give an indication of the stability of items and scales. If these correlations are low, then participantsí responses to the item or scale may change for reasons other than those that can be attributed to an intervention experience.
The methods used to facilitate personal change during outdoor experiential programs include providing opportunities for self-assessment, goal-setting, and feedback on personal progress. An instrument which can serve not only as a program evaluation tool, but which can also facilitate the processes of self-examination, goal-setting and feedback, would have added value.