NN43, February 2010
A World of Reports? A Critical Review of Global Development Reports with an Angle on Education and Training
Problems of Comparability, Political Interest, and Perspective in ‘a World of Reports’ on Educational Progress and Development
By Mark Mason, Hong Kong Institute of Education
Email: mmason@ied.edu.hk
Keywords: comparability; policy transfer; political interests; ‘ways of seeing’; EFA GMR; IEA; PISA
Summary: This article considers problems of comparability among countries and territories participating in the various reports discussed in this issue of NORRAG News, and related questions of policy transfer across contexts, of whose interests these reports serve, and of the ‘ways of seeing’ implicitly contained within particular reports.
------------
Most education policy makers would probably agree with the claim that, in the best of all worlds, the plethora of reports under consideration in this issue of NORRAG News makes better policy development possible through more informed comparison among alternatives. An understanding of what Ghana is doing to improve literacy rates among adults, and how it is doing it, might, after all, help Niger’s education policy makers in their efforts to do likewise, whether this would mean wholesale copying, selective adaptation or complete avoidance of Ghanaian policy. Such policy ‘borrowing’, or transfer across contexts, is of course not without risk. Robin Alexander (2000 p. 30) warns against it implicitly in his claim that “Culture, in comparative analysis and understanding, and certainly in national systems of education, is all”. Crossley and Watson (2003 p. 6) remind us that “context matters” in their caution to education policy makers that “major problems lie in any simplistic transfer of educational policy and practice from one socio-cultural context to another”. Such policy transfer tends sometimes to the absurd: if I might be forgiven the anecdote, I was once approached, working in Hong Kong, by an ‘education policy consultant’ from a small Caribbean island state for assistance in transferring “Asian education policy” (which, in his view, was evidently most successful) to his country. Never mind trying to transplant it – even attempting to compare education policy in China (what China – Shanghai, Gansu Province, the Hong Kong Special Administrative Region?) with that of the Turks and Caicos Islands (not the state I alluded to, I should add) beggars belief. And yet this seemingly endless ‘world of reports’ appears to encourage just that. That, of course, is the problem I’m driving at here.
In many cases, admittedly, countries and territories considered in particular reports are indeed comparable in key dimensions. The annual EFA Global Monitoring Report is a case in point, given that the countries highlighted in the GMR generally share some core, if very broad, characteristics: they are developing countries, they face substantial challenges related to poverty, and the like. But even then, how to draw meaningful comparative inferences from a report on progress in the achievement of UPE in Brazil and in Burkina Faso?
Possible policy comparisons can be even more strained in reports on countries participating in studies conducted under the auspices of the IEA (International Association for the Evaluation of Educational Achievement) or the OECD’s Programme for International Student Assessment (PISA), which rank countries by dimensions of educational achievement. Finland’s educational achievements have famously topped the PISA rankings recently, which has sent policy makers from across the globe rushing to Scandinavia to find out how they do it. (Interestingly, Finnish educationalists tend to respond to questions about excellence with answers that have more to do with equity, but that is not our purpose here.) PISA describes itself as “an internationally standardised assessment jointly developed by participating economies and administered to 15-year-olds in schools”. It is conducted under the auspices of the OECD, whose countries may be comparable in some core respects (the OECD describes itself as “bringing together the governments of countries committed to democracy and the market economy from around the world, … helping governments tackle the economic, social and governance challenges of a globalised economy”), but can countries like Indonesia and the Kyrgyz Republic, both participants in PISA 2009, really draw much of specific, directed and practical value (beyond, in other words, inferences of a highly general or abstract nature) from the experience and policies of the likes of Finland and countries and territories characterized by Confucian Heritage cultures (such as Japan, Singapore, Korea and Hong Kong, which also tend to sweep the boards in PISA)? It would take a skilled comparative education policy researcher indeed to do so.
Current IEA studies include, among others, TIMSS 2011 (the Trends in International Mathematics and Science Study), PIRLS 2011 (which assesses “trends in children’s reading literacy achievement and policy and practices related to literacy”), ICILS 2013 (the International Computer and Information Literacy Study, which will examine “the outcomes of student computer and information literacy education across countries”), and ICCS 2009 (the International Civic and Citizenship Education Study, which “investigates the ways in which young people are prepared to undertake their roles as citizens”). All well and good, but can the Palestinian National Authority, conducting education under conditions of, or at least akin to, occupation, really compare its trends in Mathematics and Science education (TIMSS 2007) to those of Sweden?
This has got to raise the question of whose interests these reports serve? Whose values and purposes do they reflect? Why would education policy makers in the occupied territories of the West Bank and Gaza seek to participate, at some cost, in studies such as these? Perhaps to show the rest of the world some of the consequences of occupation for the education of young Palestinians. So participation could be for reasons as much political as anything else. Or, countries might take part in these assessments for reasons as much aspirational as empirical. The answers are not simple, and too complex to be considered in the space allowed here. But any consideration of this plethora of reports would need to start with questions such as these.
A related and equally important question has to do with whether the categories within which these reports assess and describe the world become the categories within which we see the world. What is then pushed to the margins, or even lost? Does each really measure what it claims to measure? Have designers and researchers been sufficiently skilled to identify where some variables are perhaps proxies for something else? What is not in these reports? How long, for example, did it take before it was agreed to publish a skills-based GMR? It is only in 2012 that we will see a GMR conducted through the lens of skills. Had nobody thought until now about the importance of looking at and evaluating EFA in those terms? Is it because we don’t have much consensus on a definition of skills, on what skills we might want to assess (fitting and turning, entrepreneurship, or life skills?); or is it because we just had higher priorities (the quality imperative [2005], for example)? Again, the answers are not simple, and too complex for a piece of this nature, but any meaningful engagement with these reports, comparative or otherwise, would need to ask sophisticated questions about the ways in which any report leads us to see. These need not be insidious, of course – it’s just that they’re pretty much always implicit and not, well, easily seen. This report on the reports, in asking such questions, might thus help to better inform our reading of them.
References
Alexander, R. J. (2000) Culture and Pedagogy: International Comparisons in Primary Education. Oxford: Blackwell Publishers.
Crossley, M., & Watson, K. (Eds.) (2003). Comparative and international research in education: globalisation, context and differences. London: Routledge/Falmer.
Follow-up resources / references
EFA GMR: www.efareport.unesco.org
IEA: www.iea.nl
PISA: www.pisa.oecd.org
For the difficulties of comparing and drawing inferences across cultural contexts, see:
Mason, M. (2007) Comparing Cultures. In M. Bray, B. Adamson & M. Mason (Eds.) Comparative Education Research: Approaches and Methods (165-196). Hong Kong and Dordrecht: CERC & Springer.