The Annual Status of the Education Report (ASER) survey is now an indispensable reference point for education policy making in India. The ASER reports have been regularly cited in Parliamentary proceedings, the National Economic Surveys and the erstwhile Planning Commission’s reports. Several states have started remedial education programs based on ASER findings. Most importantly, ASER through its work has been able to highlight the fundamental importance of learning outcomes and has made education policy look beyond enrollment, in India as well as other countries including Pakistan, Uganda, Kenya, Tanzania, Mali and Mexico.
To better understand ASER’s data-driven impact, we spoke to Mr Ranajit Bhattacharyya, who is the General Manager of the ASER Centre.
Having introduced ASER’s work and their community driven data collection model in Part I, we now shift our attention to the ASER tools and data quality frameworks.
Understanding ASER’s Tools
One of the key reasons behind ASER’s scalability across rural India has been the simplicity of its tools to measure learning outcomes. The reading and math assessment tools are designed to be simple, quick and cost effective ‘floor’ tests’ – not grade level tests – to maintain a standard, objective approach of testing across linguistic and state curriculum variations. Many empirical studies have confirmed the reliability and validity of ASER tools.
The Reading Tools:
- Test the foundational reading skills like letter identification, word decoding and reading passages
- Aligned to Grade 1 and Grade 2 level state textbooks
- Available in 20 languages including English. The objective is to assess the most basic foundation skills for literacy acquisition without drawing any comparison across languages
The Math Tools:
- Test the basic math abilities like number recognition, subtraction and division
- Math test is aligned to Grade 1, 2, 3, and 4 level state textbooks
- Available in 12 languages including English
Given the sheer scale of the survey, data quality and standardisation at each level is of prime importance. ASER’s data quality framework involves three phases – training, monitoring and rechecks.
- National level training of 6 days for state teams across the country
- State level training of 5-6 days for master trainers
- District level training of 2-3 days for volunteers
- Field Monitoring: Master trainers shortlist villages to monitor based on the quiz based evaluation of the surveyors. Since 2012, in most districts they have conducted the survey over two weekends, which allowed the master trainers to personally monitor the survey in 3-4 villages in their respective districts. This resulted in field monitoring of more than 10% of the total sample.
- Phone Monitoring: Master Trainers keep track of the surveyors whom they are not monitoring in the field via phone. Moreover, since 2011 they have set a call centre in most of the states to help the state teams monitor the master trainers and keep track of the progress of the survey. This added a quicker feedback loop between the district teams and the state teams.
- Desk Rechecks: Post data collection, master trainers review the survey booklets to check for incomplete or problematic data and then verify this with surveyors.
- Phone Rechecks: Master trainers make phone calls to respondent households to check if the survey was conducted properly
- Field Rechecks: After the desk and phone recheck, problematic villages are selected for field recheck by the master trainers. Since 2011, they have introduced cross-state rechecks to ensure same standards were being followed across the country.
- SMS Rechecks: Also since 2011, they have introduced a SMS recheck system which compiled district level data via SMS-es to a central server enabling real time reviews.
After these elaborate quality control processes, the data entry staff receive training in software tools. One of the remarkable aspects of ASER has been that they have been using an in-house information management software since their inception, and thus all their data is in standard formats.
The ASER story captures the power of simple tools and standardisation in driving decision making in a country as diverse as India. As we enter the data revolution, ASER’s data-driven impact for grassroots level policy making should serve as a primer for context-sensitive problem solving.
This is a part of our ‘Data Ecosystem’ series, an effort to highlight organizations & non-profits leading the curve of evolution towards data-driven decision making. Refer to Part 1 of this post to get an introduction into ASER’s work & their community driven data collection model, here.
Catch the full series here:
- Akshaya Patra: How the world’s largest school lunch program is managing data