NN39, October 2007
Best Practice in Education and Training: Hype or Hope?
Best Practices for Equity and Efficiency
By Noel McGinn, Former President of NORRAG
In this note we discuss three broad strategies to identify policies that promise to raise both equity and efficiency. All have been implemented in some system of instruction, and all have been subjected to rigorous research. None actually reduces total spending on instruction; they require additional effort and improved technologies, but in turn they promise that the returns to increased investments will be higher than those obtained with current methods of finance. As you will see, the strategies are linked and overlapping.The three strategies are: 1) maximal use of "best practices" to increase learning outcomes; 2) increased use of relevant information by shifting decisions about allocation and utilization of resources to local instructional units, specifically schools; and 3) linkage of disbursement (payment) of funds to improved performance.
Best Practices
A "best practice" is defined as a methodology or technique that is believed to be more effective than others for a particular objective. This term enjoys great popularity and as a consequence there is no single, short list of best practices for instruction. The term "practices" includes every aspect of the organization and operation of a system of instruction that can be affected by policy, including structural characteristics (size, arrangement of grades); teacher recruitment and selection criteria; content and methods of teacher training; teacher assignment to grades and subjects; curriculum content and organization; instructional methods; classroom management; opportunities to learn; evaluation; instructional materials; rewards and punishments; and others. Lists of "best practices" are available for every level of education, from early childhood training to graduate education, and for types of education (for example, literacy training, vocational training, mathematics education, etc.).
Most of the lists of "best practices" are based on limited experience and not systematic, comparative research. We introduce the concept not to promote the efficacy of particular methods or techniques but rather to emphasize importance of careful analysis to identify which practices are in fact most effective.
Production Function Studies
A major source of lists of best practices is so-called production function research. The phrase "production function" is derived from the factory metaphor in which production results from the combination of resources or inputs. Instruction is taken as a process of combining inputs, including students, to produce learning (knowledge, skills and values). As in industry, this perspective makes it possible to answer the question, What is the most cost-effective, or efficient, way to produce a unit of learning? Statistical analysis relates measures of variable inputs (for example, school characteristics, teacher characteristics, instructional materials, student and household characteristics) to measures of learning outcomes, for a sample of, say, schools. The results of the analysis specify how much of each input variable to use to maximize the score on the measure of learning (usually an achievement test), taking into account student and household characteristics. Then, using information about the cost of each input, one can determine the least expensive way to produce the most learning.
Production function studies have two characteristics that limit the generalizability of their findings. First, there has been little consistency in the input variables included. Second, the studies rely on measures of association rather than causality and therefore cannot identify causal mechanisms. It is possible, however, to extract general conclusions about "best practices". This has been done in two ways. The first approach treats all studies as equivalent in statistical value, and "sums up" the results, adding across the studies the number of times a given input variable is identified as having a significant impact on learning. A second approach uses statistical procedures to weight the results of the different studies so that they can be combined or displayed as a distribution. Illustrative of the inputs that often appear in the various summaries of production function studies are school size, class size, teacher education, teacher experience, student/teacher ratio, and teacher salary. Of these, teacher ability, student/teacher ratio (smaller is better) and teacher salary (higher is better) appear as important for learning more often than the others. Even for these variables, however, there are studies that report either no or opposing impacts. A "best practice" is therefore a "good bet" but not a guarantee that a given policy will be effective.
Should this research be used as a source of policy recommendations? There are several reasons why not. The method assumes that schools have the same objectives, that they compete with each other attempting to maximize learning outcomes, and that they have equal information about and access to the various inputs. We know, however, that schools (and students and their parents) vary in the importance they give to the cognitive learning measured on achievement tests. Some schools and parents value high test scores, others do not. Second, the method assumes that the "inputs" have a direct impact on learning outcomes (as they would in a factory setting). If in fact their impact is indirect, mediated by how teachers and students use the input (for example, does a teacher with few students use a different method of instruction than with many students?), then failure to include the mediating variable produces misleading results. Very few production function studies include measures of what actually takes place in classrooms. It is difficult and expensive to collect reliable information on teacher and student behaviour, and even harder to estimate variations in costs.
Case Studies
An alternative method for developing lists of "best practices" relies on close up study of the teaching and learning processes in systems and schools that have achieved outstanding learning outcomes. An early version of this approach used statistical analysis to identify schools with levels of achievement much higher than would be expected given student backgrounds known to be correlated with low achievement. These effective schools were then visited to identify practices that explained the high achievement levels. The procedure has been broadened to include examples from all levels of education, including financial management. The method complements the production function approach by providing detailed recommendations for the process of implementation of improvements.
Expert Judgment
A third approach uses the so-called Delphi technique to develop a collective best judgment about the effectiveness and cost of various best practices. A panel of highly experienced researchers and practitioners is asked individually to estimate the likely impact and probable cost of a number of practices. Respondents are asked to justify their estimations when they differ from the modal judgment. Only those practices remain on the list for which there is eventual agreement. Because this method combines cost with effectiveness, the results can be more useful to policy makers.
It should be noted, however, that to date few studies on best practices have been replicated. There is no assurance, therefore, that they will be effective in other contexts. Like the practices derived from production function studies these lists do suggest actions that should be considered for improving the performance of a system or school, but they are not guaranteed solutions.
Back to full contents of NORRAG NEWS 39.
Download the full issue of NORRAG NEWS 39 in pdf.