External Evaluation

For all AgResults projects, the External Evaluator determines on a scientific basis if each Pay-for-Results (PfR) prize competition achieves its objectives. Abt Associates and IDinsight, which jointly serve in this role, apply the following approaches in their work:

  • Use a mix of quantitative and qualitative methods
  • Assess whether Challenge Projects produce private sector behaviors or social outcomes that are better or different than without PfR incentives
  • When possible, compare outcomes of prize competitions against outcomes of traditional "push" approaches
  • Analyze cost effectiveness of Challenge Projects and sustainability of results

Abt Associates and IDinsight split evaluation responsibilities for the different AgResults projects: While Abt Associates leads evaluation for the Brucellosis and Tanzania Dairy Challenge Projects, IDinsight leads evaluation for the FMD Vaccine and Senegal Crop Storage Finance Challenge Projects. Abt Associates also served as the External Evaluator for the now-closed projects in Kenya, Nigeria, Uganda, Vietnam, and Zambia.

Evaluator Questions

What has been the impact of the AgResults Challenge project on private sector involvement in the development and uptake of agricultural innovations?
What has been the impact of the AgResults Challenge project on smallholder farmers’ uptake of innovative technologies?
What has been the impact of the AgResults Challenge project on smallholder farmers' incomes?
What evidence exists that the AgResults prize competitions are scalable and that their effects will be sustainable in the medium to long term?
What has been the impact of the AgResults Challenge project on poor customers’ demand for derivative food products?
What is the evidence on the scale of any effect on private sector investment and uptake, and on the cost effectiveness of AgResults as an approach?
What lessons can be learned about the best practices in the design and implementation of agricultural prize competitions?

AgResults Theory of Change

AgResults uses incentives to overcome market failures for specific agricultural technologies. The prize competition model, which is a sub-class of Pay-for-Results approaches, incentivizes the private sector to work towards pre-defined results through rewards that are paid out only if the results are achieved. AgResults projects use prize competition incentives to encourage key private sector actors’ involvement in the market for a technology, either by developing the technology or encouraging its adoption by poor households.

The failure of a market to develop can be caused by low supply, low demand, a weak policy environment, or a combination of these. This diagram shows some examples:

Constraints Limiting Market for Agriculture Technology

Evaluation Framework

The Evaluator is charged with assessing whether the prize competition (what was previously called pull mechanism) incentives address the key underlying market failures and engage the private sector in the market while leading to smallholder farmer impact as implied in the evaluation questions.

The Evaluator employs a common evaluation across all AgResults prize competitions, applying economic theory to examine how each incentivizes the private sector to overcome the market failure and improve smallholder farmers' welfare. It also identifies behaviors of market actors that might impede or aid development impact on smallholders farmers.

The common evaluation framework allows for comparison across AgResults' prize competitions and generalization of results and lessons learned. In all AgResults project evaluations, the Evaluator applies rigorous qualitative and quantitative data gathering and analytic methods to test economic theory-driven hypotheses. The diagram below displays how the Evaluator answers all evaluation questions in its charge.

Evaluation Stages

1. Initial Qualitative Assessment

To inform each Challenge project-specific evaluation design, conduct an initial qualitative assessment to assess the project context, form a hypothesis of expected impact, examine the expected heterogeneity of impacts, identify external factors that might inhibit or boost impact, and meet with key stakeholders to secure their cooperation and buy-in.

2. Baseline

During the baseline stage, gather information on market for the technology, its adoption, initial engagement of potential competitors, as well as smallholder farmers' livelihoods and agricultural practices.

3. Endline

During the endline stage, gather information on competitor activities, development of the technology market, and smallholder farmer adoption and income gains.

4. Cost-Effectiveness Analysis/Synthesis

At the end of AgResults, synthesize the findings across Challenge projects to draw lessons on the design and implementation of prize competitions. Assess AgResults’ cost effectiveness in achieving key outcomes across projects and also as compared to traditional approaches.

5. Sustainability Analysis

For key projects, evaluate the sustainability of AgResults’ impact by assessing private sector engagement in technology provision and the continuation of favorable development outcomes two years after the project ends.

Find out more about the Learning Library.