Title: The Impact of Large-scale Datasets on Evidenceā€based Educational Policymaking and Reform in the Gulf States

Workshop Directors:

Dr. Alexander W. Wiseman
Associate Professor Comparative & International Education Lehigh University
Dr. Naif H. Alromi
Deputy Minister Educational Development & Planning Ministry of Education
Dr. Saleh A. Alshumrani
Assistant Professor Measurement, Evaluation, & Statistics King Saud University


Given the importance of evidence-based educational policymaking and reform in the GCC, and the contribution that large-scale datasets make to evidence-based decisions in the Gulf, this workshop’s goal is to definitively describe, synthesize and forecast how large-scale datasets impact evidence-based policymaking in the Gulf.

The main question this workshop addresses is how and why large-scale educational datasets impact evidence-based educational policymaking in the Arabian Gulf countries. As the stakes for education in the Arabian Gulf States have risen, so has the call for more and improved use of scientific evidence as a basis for educational policymaking (Luke, 2003; Slavin, 2002). In fact, evidence-based decision-making has become a staple of educational reform and funding requirements worldwide. For example, significant multinational organizations such as the Organization for Economic Co-operation and Development (OECD) and the World Bank have made evidence-based policymaking a priority both in their own work as influential research and policy organizations as well as for their member or client nations (OECD, 2007).

Educational policymaking around the world has been permeated by a tendency to validate and legitimize educational processes and products as “evidence-based” (Oakley, 2002). There is, therefore, a growing prevalence and clear importance of large-scale datasets for evidence-based educational development, policymaking and reform in the Gulf. Evidence from averaged scores on international assessments of math and science achievement, in particular, have become important indicators of national political and economic strength (LeTendre, Baker, Akiba, & Wiseman, 2001; Wiseman & Baker, 2005). There are many kinds of large-scale datasets that are relevant to evidence-based educational policymaking in the Gulf. Several of the internationally-comparative datasets include the Trends in International Mathematics and Science Study (TIMSS), Progress in International Reading and Literacy Study (PIRLS), and the Programme for International Student Assessment (PISA). Of these large-scale assessments, the TIMSS currently includes representatives from all of the Gulf Cooperation Council (GCC) countries as well as many from the Arab community worldwide. For example, in 2011, the number of countries participating in TIMSS grew to 64 countries and 4 benchmarking communities. Of these 68 countries and communities, 15 (or 22 percent) were Arab and represented all six of the GCC countries.

A key factor in the approach to evidence-based policymaking is the nature of the evidence being used. It is increasingly taken-for-granted that policymakers will make decisions that are evidence-based rather than based on intuition or tradition, and evidence-based educational decision-making and policy implementation often depend on assessments of “what works.” These types of approaches to educational policymaking typically prescribe the type of educational research done to inform policymaking –namely using empirical, quantitatively-based methodologies (Chatterji, 2004).

Quantitative research evidence is often the most legitimized among GulfState politicians and policymakers because it is perceived to be more accurate and trustworthy. In fact, the phrase “scientific” for international or national educational agencies often means “based in empirical research” – typically quantitative (Center for Education, 2004; Towne & Shavelson, 2001). Useful qualitative data is not often considered “scientific” enough for national or international educational funding in the Gulf or elsewhere (e.g., Ravitch, 2005). The measurability of achievement scores from large-scale datasets, for example, is perceived by policymakers to be clearer and more direct than that of learning potential or the transferability of ideas that may be measured with less quantitative instruments or methods. However, these rich data are not used to their full potential by policymakers because of the predominant focus on student achievement and rankings systems. While student achievement data can offer great insight into educational systems, the unique country-level background data available through large international datasets provide opportunities for scholars and policymakers to develop greater insight into the social and cultural factors that influence education systems in the Gulf and worldwide.

Unfortunately, very little research exists on the application of evidence from large-scale educational datasets and studies on educational policymaking in the Gulf. This workshop will address this gap in the literature.


Scope and Contribution of Workshop

The workshop will primarily focus on internationally-comparative, large-scale datasets, such as TIMSS, PIRLS, and PISA. It is also expected that regional and national datasets, which includes the Saudi National Assessment of Educational Progress (SNAEP), would be within the scope of this workshop. And, although the workshop directors do not know of any current large-scale qualitative data efforts in the Gulf countries to examine education or provide evidence for educational decision-making, papers that discuss the potential of large-scale qualitative data to impact evidence-based educational policymaking in the Gulf would be welcome. Finally, it is expected that the scope of the workshop be particularly focused on comparative studies of the use of large-scale datasets for educational policymaking that uses evidence from within and between Gulf States as well as specific Gulf countries’ national educational systems.

The workshop will provide a forum for scholars and policymakers to identify how large-scale data studies can be used to inform policymaking at all levels of education, and how these data can be used to better understand specific country- and regional-level challenges in the Gulf. Since no other published or documented work has done so before, one contribution of this workshop will be to build a foundation for large-scale data studies and their systematic impact on evidence-based policy development and educational reform in each Gulf country and across the Gulf as a whole. It is also expected that this workshop will contribute to Gulf studies and to the implementation of actual evidence-based policymaking in the Gulf by developing a draft plan for ways in which Gulf countries participate in large-scale educational studies and assessments, and particularly for ways in which Gulf policymakers and educators use evidence from these large-scale datasets to make policy and implement educational change. As a result, the workshop not only will contribute to Gulf studies, but it will also make a significant contribution to the real-world infrastructure, capacity-building and organizational-sustainability of evidence-based educational policymaking in the Gulf.


Workshop Objectives

1.      Identify how large-scale data is being used for evidence-based policymaking in the Gulf.

This objective will be met by papers that describe the participation of Gulf educational systems in large-scale educational studies and assessments and link this achievement to policies and educational change taking place in those educational systems.

2.      Identify areas of educational policy interest in order to synthesize large-scale data and apply it to educational problems in the Gulf.

Papers that compare educational policy development and application to shared educational problems and challenges either across or between Gulf States will meet this objective.

3.      Develop a draft plan for the further analysis of both large-scale educational data on the Gulf and the evaluation and investigation of its use for evidence-based policymaking among Gulf leaders at every level of the educational system.

The larger outcome of the proposed workshop is not just a scholarly discussion of the impact of large-scale datasets on educational policymaking in the Gulf, but also to develop a scaffold upon which Gulf leaders and decision-makers can build a sustainable and contextualized infrastructure and human capacity for evidence-based decision-making that productively uses the evidence from these large-scale datasets.


Anticipated Paper Submissions

The workshop directors invite papers that address any of the following:

a)      Historical overviews of the development of large-scale dataset participation, administration, analysis and use for policymaking in the Gulf States either individually or comparatively;

b)      Evaluations and analyses of specific changes in large-scale dataset participation, administration or use in the Gulf States;

c)      Research on educational borrowing and adaptation of non-Gulf States’ educational reforms as a result of participation in large-scale international or regional tests and how these borrowed reforms were adapted to the unique cultural and economic contexts in the Gulf;

d)     Implications of the participation in large-scale data collection, analysis and application for evidence-based policymaking and educational reform in the Gulf States; or

e)      Any aspect of large-scale datasets and their impact on evidence-based educational policymaking and reform in the Gulf States, especially those related to their formation, mission and purpose, workforce development impact, lifelong learning impact, and role in the overall development of national educational systems in the Gulf.

The intent is to have a combination of stakeholder reviews, theory-driven syntheses of current scholarship, reports of new empirical research, and critical discussions of major topics around the workshop’s theme. The goal on the final day of the workshop is to synthesize the areas and papers presented in order to develop a draft plan for the further analysis and productive use of large-scale datasets in the Gulf States.


Workshop Director Profiles

Dr. Naif H. Alromi is Deputy Minister of Educational Development and Planning in the Ministry of Education in Saudi Arabia. Dr. Alromi has been a school counselor, assistant head of the Riyadh school district, and a key official in the Ministry of Education for more than 22 years. His accomplishments include leadership of several multi-billion Riyal projects for the Ministry such as the King Abdullah bin Abdulaziz Public Education Development Project (Tatweer) as well as the Trends in International Mathematics and Science Study (TIMSS) in Saudi Arabia. Dr. Alromi’s research interests include school-to-work transition and the role of vocational education in national development, which has led to publications in many languages, including The Employability Imperative: Schooling for Work as a National Project (Nova Publishing, 2005).

Dr. Alexander W. Wiseman has more than 17 years of professional experience working with government education departments, university-based teacher education programs, community-based professional development for teachers, and as a classroom teacher in both the US and East Asia. Dr. Wiseman is current Associate Professor of Comparative and International Education in the College of Education at Lehigh University, USA. He conducts educational research using large international datasets on math and science education, information and communication technology (ICT), teacher preparation, professional development and curriculum as well as school principal’s instructional leadership activity, and is the author of many research-to-practice articles and books, including The Impact of International Achievement Studies on National Education Policymaking (Emerald UK, 2010).

Dr. Saleh A. Alshumrani is Assistant Professor of Measurement, Evaluation and Statistics at King Saud University in Saudi Arabia. In addition to his responsibilities at King Saud University, Dr. Alshumrani has 20 years of experience including positions as a secondary science teacher, Director of Teacher Assessment for the King Abdullah Public Education Development Project (Tatweer), Director of Evaluation in the Ministry of Education, and as a consultant for independent research groups as well as for the Excellence Center for Mathematics & Science Education at King Saud University. His publications include several articles on educational testing and assessment, including the TIMSS 2007 Report on the Performance of Students in the Kingdom (2009).


Selected Bibliography

Center for Education. Advancing Scientific Research in Education. Washington, D.C.: National Academies Press, 2004.

Chatterji, M. Evidence on "What Works": An Argument for Extended-Term Mixed Method (ETMM) Evaluation Designs. Educational Researcher, 33, no. 9 (2004): 3-13.

Chen, S.-S., andM.C. Luoh Are Mathematics and Science Test Scores Good Indicators of Labor-Force Quality? Social Indicators Research, 96, no. 1 (2010): 133-143.

Kamens, D. H., and C.L. McNeely. Globalization and the Growth of International Educational Testing and National Assessment. Comparative Education Review 54, no 1(2010): 5-26.

LeTendre, G. K., D.P. Baker, M. Akiba, andA.W. Wiseman. The Policy Trap: National Educational Policy and the Third International Math and Science Study. International Journal of Educational Policy, Research and Practice, 2, no. 1 (2001): 45-64.

Luke, A. After the Marketplace: Evidence, Social Science and Educational Research. The Australian Educational Researcher, 30, no. 2 (2003), 87-107.

Martens, K., A. Rusconi, andK. Leuze. (eds.). New Arenas of Education Governance: The Impact of International Organizations and Markets on Educational Policy Making. New York: Palgrave Macmillan, 2007.

Oakley, A. Social Science and Evidence-Based Everything: The Case of Education. Educational Review, 54, no. 3 (2002): 277-286.

OECD. Evidence in Education: Linking Research and Policy. Paris: 2007.

Ramirez, F. O., X. Luo, E. Schofer, andJ.W. Meyer. Student Achievement and National Economic Growth. American Journal of Education, 113, no. 1 (2006): 1-30.

Ravitch, D. (ed.). Brookings Papers on Education Policy. Washington, DC: Brookings Institution, 2005.

Slavin, R. E. Evidence-Based Education Policies: Transforming Educational Practice and Research. Educational Researcher, 31, no. 7 (2002): 15-21.

Smith, M. C. Drawing Inferences for National Policy from Large-Scale Cross-National Education Surveys. In Methodological Advances in Cross-National Surveys of Educational Achievementeds. A. C. Porter & A. Gamoran, 295-317. Washington, D.C.: National Academy Press, 2002.

Smith, T. M., and D.P. Baker Worldwide Growth and Institutionalization of Statistical Indicators for Education Policy-Making. Peabody Journal of Education 76, no. 3/4 (2001): 141-152.

Towne, L., and R.J. Shavelson, Scientific Research in Education. Washington, DC: National Academies Press, 2001.

UNDP. Achievements of Arab Countries That Participated in the Trends in International Mathematics and Science Study (TIMSS 2003). UNDP/Arab TIMSS Regional Office, 2005.

UNDP. Arab Human Development Report. New York: United Nations Development Programme, 2009.

Wiseman, A. W. (ed.). The Impact of International Achievement Studies on National Education Policymaking (Vol. 13). Bingley, UK: Emerald Group Publishing, Ltd., 2010.

Wiseman, A. W. The Uses of Evidence for Educational Policymaking: Global Contexts and International Trends. In Review of Research in Education 34 (2010): 1-24. eds. Luke, G. J. Kelly & J. Green. Washington, D.C.: American Educational Research Association.

Wiseman, A. W. andN.H. Alromi. The Intersection of Traditional and Modern Institutions in Gulf States: A Contextual Analysis of Educational Opportunities and Outcomes in Iran and Kuwait. Compare, 33, no. 2 (2003): 207-234.

Wiseman, A. W., and D.P. Baker. The Worldwide Explosion of Internationalized Education Policy. In Global Trends in Educational Policy6 (2005): 1-21, eds. D. P. Baker & A. W. Wiseman. London: Elsevier Science, Ltd.