Overview

Impact, and how to measure it, is an essential component of an initiative like Africa CAN. To understand whether a project is making a positive difference, you first need to establish what success looks like in practice. What factors or metrics will be measured, and what type of results will indicate a notable impact for good? The initiative aimed to train 200 data practitioners in three cohorts over three years, equipping them with the necessary skills to address climate and health data. It also planned to deploy 20–30 paid data Fellows to collaborate with public-sector and social impact organizations, focusing on real-world problems, building sustainable data science capacity, and developing scalable tools for data-driven decision-making.

These goals shaped the initiative’s comprehensive measurement framework, keeping the program on track, assessing and measuring progress along the way, monitoring the effectiveness of the training program and fellowships, and making adjustments to the methodology or approach as needed. 

Tools for Data Collection

Survey tools 

• Pre- and post-training surveys with learners to assess skill improvements and practical applications
• Post-fellowship surveys with government and SIO officials to gather feedback on skill enhancements
• Post-event surveys for convenings and webinars to measure satisfaction and intent to apply knowledge

Key informant interviews

• Interviews with learners from government and SIOs involved in hosting Fellows
• Periodic interviews with Fellows and host organizations to monitor placements and progress

Qualitative research

• Review of fellowship project proposals/action plans to track institutional engagement
• In-depth qualitative research with government institutions and SIO representatives to assess data-driven practices and institutional impact
• Co-development (and assessments) of case studies between the Global Partnership, Fellows, and government/SIO partner institutions illustrating adopted data-driven practices and their impacts

Event data collection

• Attendee tracking and analysis for convenings, webinars, and peer exchanges
• Event participation metrics and satisfaction ratings

Database tracking systems

• Africa CAN Learner Database for tracking participant completion rates and demographics
• Global Partnership Learning Materials Tracker for monitoring content development
• Global Partnership Fellowship Overview for placement tracking
• Partner databases for routine data collection

Data Maturity Assessment (DMA)

• DMA is being used to conduct an in-depth readiness assessment of participating SIOs to be able to measure their improved change in data capacity and practices. This tool is to be administered before and after an intervention with SIOs. 

Partner Data Collection Forms 

• TSIC has developed data collection forms and documentation forms with the list of key indicators that partners will contribute to. Partners will fill out this form, attach relevant means of verification, and share with data.org on an agreed frequency. 

Feedback forms

• Developed as needed to capture indicators that measured satisfaction of participants and planned use of resources, training, webinars, meetings, conferences, etc.

Designing multi-level impact indicators

The key to effective monitoring, evaluation, accountability, and learning (MEAL) in this context lies in understanding that capacity-building is a multi-level process consisting of: 

• Systemic improvements
• Institutional change 
• Individual learning

Consequently, the initiative required the development of multi-level impact indicators to help determine whether it was on course to meet its goals and achieve the desired impact. For each indicator, a target was set to help determine whether the initiative was working effectively.

At the highest level, Africa CAN measures system transformation. The ultimate goal of the program—improved data practices within government and SIOs—was captured by this impact indicator: “Number of government and social impact organizations that can demonstrate they are data-driven and how that has impacted their planning, implementation, and results.” The formula for this indicator—“Total institutions showing application of skills gained from the virtual training six months later (through a Global Partnership-led verification process) / Total institutions who successfully completed the virtual training component”—ensures that measurement focuses on sustained behavior change rather than mere training completion, which is often an incomplete definition for actual capacity development.

At the institutional level, the framework tracked both organizational readiness and individual capacity development. These indicators helped to determine whether the training translated into practical application and whether organizations were ready to make productive use of embedded Fellows. The framework also demonstrated how to effectively integrate diverse contributions of our partners, through its incorporation of data.org’s Data Maturity Assessment within the broader MEAL framework, providing baseline information about organizational readiness while also delivering value to participants through increased self-awareness about their data practices and capacity gaps.

Outcome 1: Country partner organizations 

1. Percentage of government and social impact organizations producing successful Fellowship project proposals or action plans.
2. Percentage of case studies demonstrating improved use of data for decision-making (post-Fellowship institutions) in government and SIOs.

Output 1.1: Country partner organizations 

1. Number of government and SIOs that have participated in the data.org training on data management and taken the data maturity assessment.

For individual learners, the framework measured not only participants of the training but additional staff members within host institutions who demonstrated improved skills and practical application resulting from the Fellowship. This approach recognizes that effective capacity-building creates a ripple effect, with knowledge and skills spreading beyond the immediate participants to their colleagues and broader institutional networks.

Outcome 2: Learners 

1. Number of learners who completed the virtual training that have increased skills.
2. Number of additional learners (government, SIO, or public-sector officials/employees) showing improved skills and practical application resulting from the fellowship, demonstrated via specific use cases.

Output 2.21: Learning and knowledge sharing

1. Number of learning materials developed as part of the training and made available to learners, disaggregated by audience (internal vs public).
2. Number of Fellows successfully placed with government and social impact organizations or public-sector organizations.
3. Number of knowledge products published and available to the public.
4. Number of attendees at convenings held to share knowledge, experiences, and best practices.
5. Number of convenings held to share knowledge, experiences, and best practices. 
6. Number of providers engaged in and delivering capacity as part of the Africa CAN program. 

Ensuring equity and inclusion

The initiative paid careful attention to the principles of equity and inclusion, both in program design and measurement approaches. This was addressed through comprehensive disaggregation of all major indicators by gender, disability status, language, and geography, with specific targets for female participation that reflect a commitment to addressing historical inequities in technical fields. The framework not only tracked participation rates, but also completion rates and impact indicators across these demographic categories, enabling staff to identify and address differential outcomes. This approach, which included setting parameters for completion based on attendance and engagement metrics (and rigorously tracking the same), as well as regular reflection points to collect feedback from all stakeholders involved, ensured that measurement activities supported, rather than obscured, efforts to promote equity and inclusion.

Sustaining impact beyond training completion

Perhaps the most challenging aspect of measuring capacity-building impact is determining whether changes will persist beyond the program period. Africa CAN addressed this challenge through several mechanisms, including creating structured opportunities for peer learning, to providing microgrants to support promising Fellowship projects that required longer timeframes than the standard six-month period, and ongoing tracking of institutional practices and case study development well beyond the program’s formal conclusion. 

The framework also recognizes the value of external validation and coverage from partners and independent observers, thus tracking mentions and features in partner communications and coverage by development sector media outlets, among other touchpoints. This type of external recognition signals that the program has achieved sufficient impact and quality to warrant attention from respected voices in the field, which in turn can attract additional resources and replication opportunities.  

Further, Africa CAN leveraged the broader awareness and visibility that a successful capacity development initiative can generate by showcasing its impact through various mediums, including complementing case studies with multimedia content, from organizing or participating in convenings and knowledge-sharing events discussed earlier in this document, to producing videos and blogs documenting Fellow experiences and testimonials from host institution staff describing institutional transformations.

By focusing on all these dimensions of sustainability, the MEAL framework helps ensure that capacity-building investments generate lasting returns.