Home

Back to
Tools & Instruments

How to Choose
Tools, Instruments, & Questionnaires for
Intervention Research & Evaluation

James Neill
Last updated:
25 Jul 2004

Choosing appropriate instrumentation (surveys, questionnaires, etc.)
is a vital part of conducting good quality empirical research and evaluation. 

Too often researchers fall vulnerable to 'availability bias' and simply select whatever they can get their hands on, or they default to using instruments that have commonly been used  in the past.  Poor instrument selection adds noise and error to your research. 

A thorough search and evaluation of all possible measures is recommended.  Time spend critical reviewing possible instruments is time well spend.  Key factors to consider are:

(i) Length & complexity: Is the instrument appropriate for the participants?
(ii) Match between program objectives and the instrument
(iii) Sensitivity: Has the instrument been built with a view to being used for assessing what you want to measure, e.g., change?
(iv) Specificity: The greater the specificity of a measure, the more likely it is to predict actual performance (Blau, 1993).
(v) Reliability and Validity: Has the reliability and validity of the instrument been well established via peer-reviewed publication?  Do you understand the strengths and limitations of the instrument?
(vi) Ethical/Educational Issues: If possible, can the instrument be used not only for the interests of the researcher, but also in the education/development of participants?  For example, a self-assessment tool could be used not only for research purposes but also to lead onto a goal-setting and feedback session with participants.

Try to think independently about what type of research tool you might use.  Is using an empirical tool even the most appropriate method for examining your research question?  You may wish to consider a qualitative or mixed mode approach.  In the past, outdoor education research has, arguably, over-emphasized measurement of self-esteem and self-concept outcomes and under-emphasized potentially interesting outcomes such as quality of relationships and personal skills.

Length and Complexity

On outdoor and experience-based programs, instruments are often administered in field settings (e.g. in the bush, on board a boat, in various weather conditions), on multiple occasions (e.g. pre-program, first day of program, last day of program and post-program followup) and to a wide range of participants (e.g. people with learning disabilities, people without English as their first language, school children, corporate managers). Hence, the shorter and simpler an instrument (reliability and validity aside), the greater the instrumentís potential applicability.  The aim was to develop an instrument which would provide a maximum amount and type of unique information in as short a time as possible (i.e. a maximum of about ten minutes).  The instrumentís instructions and layout also needed to be straight-forward to allow people without research experience, such as group leaders or teachers, to administer the instrument consistently across different groups.

Relevance to Program Aims

Generally, a major aim of many outdoor experiential programs is to facilitate individualsí personal development in a broad range of life skills (e.g. self-confidence, initiative, communication skills, etc.), although different programs may have more specific aims such as the development of teamwork and leadership skills.  Ideally, the instrument would  encompass a wide range of life proficiency domains relevant to general and specific program aims, so as to allow for within and between program comparison of different program outcomes.

Sensitivity to Change

The scoring system is an important aspect of an instrumentís sensitivity to change.  A dichotomous (yes/no) scoring does not provide much sensitivity, whereas a large range can reduce the instrumentís reliability.  A balance needs to be reached between sensitivity to change and reliability.  Despite being a critical issue, a search of the literature revealed little research, for example, on the relative efficacy of likert-type scales with different numbers of responses for measuring change. 

Two further issues to be considered in developing an instrument for measuring change are ceiling/floor effects and test-retest correlations.  It is desirable that the wording of the items and the response scale tends to produce responses from participants that leave room for detecting shifts in self-perceptions either up or down -- hence the means need to be examined during the instrument development.  In addition, test-retest correlations give an indication of the stability of items and scales.  If these correlations are low, then participantsí responses to the item or scale may change for reasons other than those that can be attributed to an intervention experience. 

Educational Exercise

The methods used to facilitate personal change during outdoor experiential programs include providing opportunities for self-assessment, goal-setting, and feedback on personal progress.  An instrument which can be used to facilitate the processes of self-examination, goal-setting and feedback would give it added value.


Also see: Essentials of a Good Psychological Test