Evaluation as a Management Tool

This post is also available in: Spanish

Ariel Fiszbein interviews Silvia Montoya, the new Director of the UNESCO Institute of Statistics, about her experience as Director in charge of evaluation in the Ministry of Education in the City of Buenos Aires, the lessons she learned about learning assessments and education statistics, and her expectations about her new job.

AF: What were the most important initiatives that pushed you to improve the quality of education during your time in the Ministry of Education?

SM: They can be divided into three fronts. The first was to work on data collection processes by improving technology, systematizing the consistency checks and shortening the periods for reporting the information. For that, one of the actions was the creation of an online system for school enrollment. This software not only speeds up the process but also offers a system of automatic validation. This improves the quality of information and enables management teams to correct information as loading occurs, and having a sworn statement issued at the end, can check the final data.

The second point was to define an integral agenda of evaluation that would contemplate the different dimensions that make quality education. This involved the participation of the city in international tests in which even the federal government did not participate. In this way, it raised the sample size of the PISA test so as to have representation as an individual jurisdiction and was responsible for the national coordination of the tests of Civic Values and Citizens (ICCS), digital literacy (ICILS), science and mathematics (TIMSS), reading comprehension (PIRLS), social and emotional skills (ESP), adult skills (LAMP/UNESCO and PIAAC/OECD), among other studies. This had two benefits: first, the interaction with the most prestigious organizations of quality assessment such as IEA, OECD, and ACER strengthened the team and forced it to work with international quality standards that were then passed on to other processes, such as local evaluations. Furthermore, this gave visibility to the evaluation actions of the city.

The third initiative was the dissemination of information defining a system of education indicators and also a feedback policy of information specific to each of the stakeholders in the education field. . At the beginning of my term, the flow of information was from the schools to the Ministry. The Ministry prepared documents with information in large aggregates, generally in the form of Statistical Yearbooks. With my team we produced school-specific newsletters. This broke with the misconception that what was prescribed by the National Education Law Article 97 implied that information should not be returned to stakeholders. For the first time all school supervisors and directors could have school indicators under their responsibility. First it was on paper. Then we developed a system online. In Argentina school-specific data cannot be published. However, we developed a site in which the parents can access aggregated information (by commune, by neighborhood) and also request, under the existing legal framework, certain school data. We also published a system of indicators at the commune, district, and city level that include not only data on the educational system but also economic, social, and investment data.

AF: In your experience, what are the main barriers faced by governments in establishing sound systems of educational assessment?

SM: A first key point is the strong and sustained political decision to make the evaluation of and access to indicators a vital part of the educational process.

I think the Argentine statistical system has already taken a giant step forward in recognizing that evaluation is a key input for improving education. However, it has not been so easy to see how that path is built. At this point, I think it is essential to establish channels of communication between the areas of quality assessment of Ministries and supervisors, directors and teachers so that all stakeholders have adequate information. For example, when I took office there was a lot of fear in the education system that we would publish school rankings or even that teachers would lose their jobs due to the survey on pedagogical practice. We created an environment of trust and in no case were any of these results made public at the individual level.

Another challenge is the independence of the assessment bodies. Brazil, Chile, Colombia, Ecuador, Peru, and Uruguay are some examples of countries in the region that have institutes of evaluation. In August of 2013 the city of Buenos Aires was added to this list. Of course, there are differences between them. What I want to point out is that there has been an effort to institutionalize evaluation and separate it from political power in order to gain independence and objectivity. Personally I have had to go through that process in Argentina and as it happened I am the only professional with the great fortune of having participated and presented the project of the creation of the Institute for the Development of Quality Education (IDECE) in the federal government in 2001 as a decentralized body and then, over a decade later, having presented and obtained approval for the Unit of Comprehensive Assessment of Educational Quality (UEICEE) in the city of Buenos Aires, this time as an autonomous body within the ministry. I strongly believe in the independence and technical professionalism of the assessment bodies.

AF: Do you think it is possible to generate interest in information on educational outcomes from parents and citizens in general?

SM: I think the interest is there. What parent does not care how his or her child does in school? What parent does not want to know if the school his or her child attends gets good results? What needs to be generated, on one hand, are mechanisms that allow parents and citizens to access information on results, devoid of expert jargon, and accompanied by solutions and  concrete alternatives for action for everyone in his or her role.

Also, I think it is essential to bring the areas of quality measurement and statistics to society in general. In the city of Buenos Aires, we developed workshops for officials. A further step would be to develop workshops for parents. Understanding the indicators, understanding why the large-scale evaluations differ from classroom assessments, for example. Learning to read the indicators and statistics helps to not  trivialize the issue. It is not a question of the standings in a World Cup, but about our children and the future generations of a country.

And PISA, perhaps the most well-known assessment, need not be used in political discourse as the only symptom of the educational problem nor as a demon. It has to be used as a symptom of a reality that we know as complex and frustrating but one that forces us to ask ourselves if we are depriving our children and grandchildren of a better life if we do not react.

Moreover, the PISA results show a failure of society and we do not deal with it as such. It is a test that is not based on curriculum design, but measures what 15-year old kids in the educational system have learned in all spheres in which they move. If we are concerned, for example, about  the result of Argentina or Buenos Aires, it is not the fault of one government or another but the “laurels that we were able to achieve” as the Argentine anthem says. It is the quality of education they receive in our homes, in the media, our schools and through spaces of social interaction, clubs, cinemas, etc.

AF: There is a growing recognition of the importance of socio-emotional skills as factors that influence not only educational outcomes but also success in working life. Do you think that governments should begin to measure these skills on a regular basis?

SM: No doubt, many studies show that perseverance, sociability and self-esteem have an influence on better health, better subjective well-being and that these skills are important both in and outside of school. Most important from the point of view of the statistical system is that they are malleable throughout people’s lives, especially during childhood and adolescence.

This is an extraordinary fact for the policy maker, for management teams, teachers, and parents who know that they can generate favorable environments for the promotion of “positive” skills. And that is where measurement enters since we all know that they matter but we do not know which ones, how, or how much. Ultimately what to do and what not to do, providing information on making a choice and supporting positive aspects while minimizing what is not good.

In this sense, from the city of Buenos Aires we joined the longitudinal study of skills for social progress carried out by the Centre for Educational Research and Innovation (CERI) of the Organisation for Economic Co-operation and Development (OECD). The aim is to measure non-cognitive skills in a sample of students and then “follow it” over the years to compare this information with socioeconomic conditions, academic performance, and job placement.

AF: What will be your strategic priorities in your new UIS management position?

SM: I am taking a new position during an important phase for the international discussion, concerning the post-2015 development agenda, the objectives and which indicators are most relevant for monitoring the compliance of the goals that are set. Setting realistic goals and determining how to measure them while also obtaining information will prove a significant challenge. The UIS has three giant assets thanks to the efforts of a group of high-quality professionals and to the remarkable leadership of its former directors, Denise Lievesley and Hendrik van der Pol. These three assets are independence, transparency, and quality of technical work. That undoubtedly simplifies my job.

There will be three priorities. . On the one hand, it is essential to strengthen the presence of UIS in areas in which perhaps have not had much involvement until now, despite having been recognized as priorities in previous years, for instance the metrics of learning in key areas and stages in the lives of people such as preschool education.

A second area is to build capacity not only at the level of the central ministries but also to build the capacity of social actors to understand the data,  so that the information is transformed into something useful for people to impel political action in their own right, since they are the ones who transform the reality of our children. The data collection systems (except in some regions with higher difficulty) have achieved legitimacy and acceptance among schools but have not been as effective in providing information to improve classroom and school practices. This entails enormous efforts of decentralization to build capacity to understand how to load, interpret and use the data.

Related to diffusion, another priority will be to work on simplifying the number and amount of indicators and improve their relevance so that they respond more accurately to questions that we are asking today, and the visualization of the data. The great challenge is to work on the legitimacy in implementing the policy through transparency in the use and dissemination of information in a manner that favors the voice, participation, and social control, and establish practices in the use of information and evaluation that are coherent and complementary to the improvement in quality with equity.

AF: What lessons from your experience in Buenos Aires do you think will be most useful in your new position?

SM: That’s an excellent question. Although the scale was much smaller I think a first important point is to make the processes for collecting and disseminating information more friendly. Those who collect the data and report the information to us are management teams, teachers and supervisors whose responsibility is educational management, or that which is the same, that our children learn more effectively. Our role is to help with data and tracking processes so they can do their job better. We are a service that has the responsibility and enormous power to transform, to the saints of Julio Cortázar in that speech to the graduates of the Normal School of Chivilcoy in 1939.

We cannot transform the statistical and measurement system into a monster with its own life. It only makes sense if it serves for improvement. The collection and use of data should be a habit and not a source of regret, a nuisance. The data and the information are vital for schools to work on a daily basis. It is to provide information to an expert that extracts data for analysis that leads to identifying where there are problems and to outline actions. Nothing different from drawing blood or taking an electrocardiogram (EKG).  Therefore it is vital to make it less difficult and share with the “patient” the diagnosis and provide the tools so that the next time these results are improved.

The city of Buenos Aires has made significant advances in the computerization of data collection and with friendlier programs. The differentiated diffusion for different types of actors is another important point that showed us that when stakeholders understand the meaning of information, why and how it connects with their realities (i.e. when capacity is built), and the data stops being only the domain of experts and becomes something for all, then the quality and the usefulness of information increases.

The experience has been very enriching because previously I had worked two years at the federal government level, coordinating data collection. Now I have been given the opportunity to be in direct contact with the schools and with the system, which is quite different. The task in the UIS brings me to a more supranational role, having been in almost all stages of the lifecycle of data and at all levels of government. I greatly appreciate having had all of these roles.