Tag Archive for: monitoring and evaluation

Ethiopia is at a pivotal moment in its efforts to improve the health status of its people and move the country into a new phase of social and economic development. The country’s massive Health Extension Program (HEP) program has placed over 34,000 community health workers in 14,000 health posts in less than 8 years. Now, health authorities are exploring ways to improve the program with mobile solutions.

Sponsored by The Bill & Melinda Gates Foundation, Vital Wave Consulting authored the “mHealth in Ethiopia: Strategies for a new Framework” report for the Ethiopian Ministry of Health. The report offers a framework for addressing specific information, communication and inventory management issues with mHealth interventions.

Download the report by clicking the link below – and let us know what you think in the comments!

MSF doctor kneels next to young girl with cast on her leg

Photo: MSF

By Médecins Sans Frontières

As a project manager for MSF (Médecins Sans Frontières/Doctors Without Borders), a medical emergency humanitarian agency, I attended this year’s South by Southwest (SXSW) festival in Austin, Texas, in the company of a friend and collaborator from Google who is involved in crisis mapping. We gave a presentation on some mapping work we had done together, and inevitably we discussed the differences and similarities in our geek (high-technology) and primitive fieldworker (humanitarian) cultures.

The two cultures are, on the face of it, rather distinct.

Geeks live in highly connected environments, usually urban, surrounded by technological infrastructure straight from the most outrageous science fiction of decades past. They inhabit an intellectual world of abstractions: math, code, logic, creativity, and business; while their physical existence is supported by a seamless layer of luxury and connectivity. The food, and the coffee, is good.

Humanitarian fieldworkers live a variety of environments from luxury hotels or palatial mansions to mud huts or tents, with variable connectivity, but almost always with constraints. Even in the most modern of African or South Asian capitals the technological infrastructure is often wobbly; there may be high-speed internet but it usually features random periods of downtime. In the mud hut scenario, internet and telephone use generally involves a satellite rig, great expense, and substantial cursing at the fiddly configurations and on-again-off-again unreliability. Our lives are never far abstracted from the constant effort required to enable working conditions, and the intensely pragmatic work of solving immediate problems. That patient is in danger of dying NOW, that refugee camp needs clean water NOW, and if we don’t get the car fixed in the next few hours we’ll be spending the night by the riverbank eating dry emergency ration biscuits. Most of us have at one time or another shared a single broken-down laptop with several people, a leaky thatched roof over a mud floor, and a plate of sandy, oily rice that is as good as we’re going to get for dinner.

Despite these disparate viewpoints (abstraction vs. immediate physical pragmatism) and differing ecological niches (urban café vs. mud hut) there is an unexpected similarity; primitive fieldworkers and geeks share a culture of problem-solving. Nothing makes a geek happier than a tough problem that gets all the neurons firing, and nothing makes a fieldworker happier than a serious emergency that gets the adrenaline flowing (if that sounds callous, in our defence it’s not the suffering that we enjoy, but the chance to make a real difference with our work).

At SXSW we had the chance to share that cultural crossover with a broad audience of geeks, fieldworkers, and an assortment of others, all of whom shared an interest in the intersection of humanitarian work and technology.

My main take-home message was: we are not alone.

We humanitarians tend to take pride in our ability to deal with problems by stretching our ingenuity and using only what is available in the field. My friend from Google was astounded at what we do with spreadsheets, saying “I didn’t think that this could be done without software coding capacity”. We use spreadsheets as databases, stock management systems, maps, payroll systems, and sketchpads. This is, perhaps a strength but I am beginning to realize that we take it too far. There is an enormous community out there in the world, with increasingly robust electronic links to even the most remote field locations, who can help us. Open source code can be written to address problems that we would normally tackle with tortuous repurposing of spreadsheets; there are incredibly talented programmers delighted to donate their efforts, especially to interesting problems. Informal slums can be mapped by volunteers, either people who once lived in the country or even by people who have never been there simply hand-tracing satellite imagery. The astounding success of the Ushahidi project (follow the link to read about an open-source crisis mapping project that started in Kenya to assist people during electoral violence and has since been used around the world, including Washington DC, to map emergencies) shows the power of crowdsourcing or distributed voluntarism to assist people in crisis.

Humanitarians need tools and information, particularly during crises. The tech world is bursting with possibilities to provide just that, often free of charge and with an astonishing level of professionalism. I hope that this meeting of cultures continues to deepen and that the early promise of these innovations translates to real benefit to the populations in crisis that we serve.

Ivan Gayton

Ivan Gayton is a project manager with Médecins Sans Frontières, currently working in Nigeria. During the Haiti cholera outbreak in 2010, he worked together with a team from Google to develop tools to map the outbreak using freely available software (Google Earth). He and Google’s Pablo Mayrgundter continue to work on an open-source epidemiological mapping tool in their scant spare time. Other than an interest in seeing further cultural cross-pollination between humanitarians and techies, Ivan has no conflict of interest, and no financial interests whatsoever in the matter.

This and many other MSF blogs are available at PLoS – Speaking of Medicine http://blogs.plos.org/speakingofmedicine/category/msf-2/

Photo Credit: Daniel Katz

Access to timely and accurate data on farmers, their households and farm activities is key for policy, decision-making and quality control for development organizations, national governments, funding agencies, project implementers, field workers, researchers and farmers themselves. Demographic data (past and present) on farm households such as land sizes, assets owned, types of soil, weather conditions, gender distribution, literacy levels, types of commodities being produced, diseases and pest, facilities for storage, among others are critical.

Unfortunately, the current status of data on developing nations’ agriculture at both local and global levels is far from reaching the stage at which policy makers can confidently draw upon for intervention due to the complexities with collection and analysis. The result is inefficient flow of resources into these communities due to under or over investments. The challenge is both socio-technical – human skills to design the necessary protocols for capturing these data as well as technological tools to facilitate the management (capturing, analyzing, sharing, etc.) of the data.

For far too long, exploring the role of ICT solutions to support value chain actors in this area have been ignored even though viable and potent ICT tools are in the market. ICT solutions identified in this component could be used in building and generating electronic forms for data gathering, help in timely access to data, facilitate easy and accurate data analysis, ensure monitoring of field activities, help in tracing of goods from farm gate to consumers, and assist in certifying commodities for quality assurance.

Photo Credit: Uganda App Lab

Potential ICT Solutions to Facilitate Agricultural Policy & Decision-Making

These are ICT solutions that facilitate accurate data capturing, analysis and sharing on farmers, their farm sizes, assets, commodities and other key identifications for enhancing policy decisions making by field staffs, governments, investors, donors and feedback into research and development. Examples of apps identified in this category includes iFormBuilder, a mobile platform for building robust forms, offline data capturing and managing data and users from any browser with the iPhone, iPod touch, or iPad with image and audio recording, GPS and mapping functionality, etc; Mobenzi Researcher that uses simple feature phones to high-end handsets to provide a tried and tested solution to enhance field research and data collection; and PoiMapper, a mobile point of interest data collection and sharing solution for affordable GPS-enabled feature phones that can make agricultural fieldwork more efficient and reliable through planning and monitoring of field activities.

Potential ICT Solutions for Traceability and Quality Assurance

These are ICT solutions to facilitate data gathering on farmers, their fields and specific information on their commodities for traceability and quality assurance. Examples include SourceTrace, a suite of ICT applications including traceability module that records delivery and transaction of data both entered manually into mobile device as well as from GPS, RFID and bar code readers, certification module for internal agricultural monitoring processes of agricultural commodity firms such as Fair Trade, and processing module that automates the capture of valuable information on the various light industrial processes of any agricultural commodity; Reliable Information Tracking System (RITS), a new coffee traceability program that is helping coffee growers become more efficient, reliable, and quality-focused by tracking deliveries of coffee from each member down to the details of what coffee varietals and quality score each lot of coffee receives; and, Integrating ICT for Quality Assurance and Marketing, a project that helps to build an internal control system for inspectors of Organic Producers and Processors Association of Zambia (OPPAZ) for quality assurance and thereby improve the value of the products for increased income.

In summary, ICTs have great potential for data management within the agricultural value chain for increased agricultural growth. Improved data used will influence how research is conducted and subsequently the kind of policy decisions that need to be made for funding and investment. For detailed information on ICT solutions for monitoring, evaluation and quality assurance visit ICT4Ag Database by GBI for an interactive experience and feedback.

Last week’s mHealth working group meeting laid out the opportunities, challenges, and potential of monitoring and evaluation (M&E) within the mHealth sector.

JhPiego circumcision promotion through texting service

Poster in Swahili promoting male circumcision through SMS service, part of Jhpiego program

 

Several experts in the field presented their experiences of monitoring and evaluating mHealth projects, emphasizing the considerable potential that mobile projects offer in generating robust and accurate data. Kelly L’Engle, a behavior scientist at FHI 360, discussed the need for M&E in order to gauge the impact of mHealth. She claimed that mHealth technology is not being fully leveraged and that the current mHealth research “doesn’t provide evidence on actual impacts…or answers to critical research questions…”

James BonTempo from Jhpiego presented on evaluating behavior change evidence from a text message project that promotes male circumcision (MC) for HIV prevention in Tanzania. He referred to this evidence extraction as “mining the data exhaust” – that is, the data generated as a byproduct of ICTs, the “trail of clicks” that ICT users leave behind.

The MC program offered a toll-free text-messaging/SMS service. With the service, individuals could request to receive information on the benefits of MC (using key word: TOHARA), where to find MC services (WAPI), as well as receive follow-up care information after undergoing the circumcision (BAADA). While the SMS platform was intended to capture requests to the three keywords and generate access logs for system performance monitoring, Jhpiego has analyzed this data in order to see if there was a relationship between requests for the MC data and actual utilization of MC services.

Jhpiego male circumcision project in Tanzania- behavior change evaluation

Potential link to behavior change in Jhpiego male circumcision project

The data set included 12,056 keyword requests sent by 4,954 users. After performing a Chi Square analysis on the data, the project team found that requests for information on the benefits of MC (TOHARA) was not strongly associated with going to receive the circumcision. However, it was found that those who requested where to find the data (WAPI) did have a statistically significant association with undergoing the circumcision. These results are consistent with pathway models of behavior change, indicating that those who simply wanted to know more about MC were not quite prepared to undergo the procedure, but those looking for service availability were ready to use the services.

The associations found in this particular project imply that providing text or voice messages that tell people where to get a particular service could be more effective in encouraging clients to utilize that service. M&E that finds results such as these could help project leaders design SMS services that generate more useful data.

Like most forms of M&E, there are limitations on findings that use mobile data. There is a lot of information that mobile devices do not provide, such as which phones belong to whom and who actually sends the text messages; this makes it difficult to link messages to specific individuals. In the MC case, data analysts found an association, but did not necessarily know the nature of the association. It can also be challenging to find the time or manpower to rake through the massive amount of data that is produced by mobile devices.

While M&E in mHealth has its limitations, it is difficult to find data that can be collected and analyzed as quickly, cheaply, or easily using other means. Paper, radio, and television simply do not offer the same kind of easily-mined data exhaust that mobile does.

 

 

This summer I have wrote a lot about good governance programs to fight corruption, improve government effectiveness and accountability, and how they they are crucial to developing countries economic development, overall prosperity, and empowerment of civil society. One issue, however, can be the monitoring and evaluation of democracy and governance projects, which can sometimes be difficult–public opinion surveys as a form of measurement can be fraudulent, or uneven, and systems can be disorderly. Although ICTs are not a panacea for a development, they can help to streamline democratic and good governance strategies, and embolden civil society to play a participatory role. Some of the ways ICTs can be employed in democracy and governance projects, such as e-government strategies, election monitoring systems and enabling citizen media, can drastically improve the efficiency of these initiatives. Based on what I have learned so far, below are suggestions for monitoring and evaluation for an e-governance strategy, how to implement an election monitoring system from the beginning til the end, and how best to measure the effectiveness of citizen media:

1. E-government and Participation

  • Benefits: Transparency can be enhanced through the free sharing of government data based on open standards. Citizens are empowered to question the actions of regulators and bring up issues. The ability of e-government to handle speed and complexity can also underpin regulatory reform.  E-government can add agility to public service delivery to help governments respond to an expanded set of demands even as revenues fall short.

First, on the project level, question if the inputs used for implementation and direct deliverables were actually produced. The government’s progression or regression should not rely solely on this because there are other outside variables. For the overall implementation, ask if the resources requested in place, and were the benchmarks that were set reached? Featured below is a timeline on how to implement a good e-government strategy.

Phases of e-government

Source: ITU

 

2. Strengthen Rule of Law with Crowdsource Election monitoring:

  • Benefits: Support for election monitoring may be provided prior to and/or during national or local elections and can encourage citizens to share reports from their community about voting crimes, ballot stuffing and map these crimes using Ushahidi. By documenting election crimes, it can provide evidence of corrupt practices by election officials, and empower citizens to become more engaged.
  • Drawbacks: Publicizing information to the  broad public means without checking the information’s validity these systems can be abused in favor of one political party or the other, and elections can be highly contested.
Photo Credit: movement.org

Photo Credit: movement.org

 

Below are systematic instructions on how to implement the “all other stuff” needed for a election monitoring system, like Ushahidi:

Step 1. Create a timeline that includes goals you have accomplished by different marker points leading up to the election, and reaching target audiences

Step 2. The more information reports the better for the platform, but consider a primary goal and focus on filtering information about that goal to the platform, put it in the About section.

Step 3. Target your audience and know how they can be reached for example

  • Community partners
  • Crowd
  • Volunteers

Step 4. Figure out who your allies are—NGOs and civil society organizations that will want to support, and provide resources for more free and fair elections in your country. Figure out what groups would be best for voter education, voter registration drives, civic engagement or anti-corruption. Building a new strategy on top of the already existing ones will help to promote the campaign and making it more sustainable overtime.

Step 5. Reach out and meet with the groups you have targeted—and make sure to identify people from that country living abroad, reach out to the diaspora. Ask yourself the following questions when the program is implemented: should all reports be part of the same platform? Should reports come in before voting begins or just offenses taking place during elections? What about outreach after the election takes place for follow-up M&E?

Step 6. Get the word out to as many citizens as possible using flyers, local media, and target online influencers, such as those on Twitter or Facebook. Attract volunteers to assist in the overall outreach and publicity plan—a volunteer coordinator, technical advisor and, if possible, a verification team or local representatives, to relay and confirm what monitoring the electoral processes is all about.

Step 7. Information sources:

  • Mobiles: Frontline SMS can work as reception software for submissions via text.
  • Email/Twitter/Facebook: Consider creating a web form to link people to on social networks which asks for everything you need, including, detailed location information, category and multimedia.
  • Media Reports and Journalists: Have volunteers look in the news for relevant information to be included in the reports
  • Verification team: Either a local organization or journalist works best—on site that is able to receive alerts from the platform on events happening around their polling stations to be able to verify what is going on. Cuidemos el Voto modeled Ushahidi slightly for incoming reports from whitelisted people to show up automatically, for example non-governmental election monitoring organizations.

Step 9. Monitoring and Evaluation

  • Closing the loop of information: How will you show citizens who provided information on electoral fraud that you received it? Have a system in place to tell community representatives that the information was received and it will be acted upon.
  • How will you act on that information in the country’s courtrooms, though? Make sure to preserve the documentation of election fraud that your platform has received so that it can serve to hold the perpetrators accountable in court.

3. Citizen Media

Citizen media allows content to be produced by private citizens outside of large media conglomerates and state run media outlets to tell their stories and provide bottom up information. Also known as citizen journalism, participatory media, and democratic media, citizen media is burgeoning with all of the technological tools and systems available that simplify the production and distribution of media

  1. Benefits: In addition to the above-mentioned benefits, citizen media also allows a sense of community where up-to date news covers a variety of angles, stories, and topics found in hard to reach places.
  2. Drawbacks: It can be risky for the citizens journalists and their supporters. They can be identified and targeted by members of the oppression, where they will be put in jail or tortured. There is no gatekeeping, verifying, or regulating the information—this is not a problem when it comes to video or photos, but definitely with information. Also, connectivity issues may not allow citizens to upload the information.
  3. Helpful Resources: This journalist’s toolkit is a training site for multimedia and online journalists.
  4. Monitoring and Evaluation for citizen media projects: Governments have foreign policy and economic agendas that guide their choices on how they fund projects, therefore, it’s important that the grantees and activists understand and share the same objectives. This is also beneficial to learn from projects over time to avoid redundancy and enhance efficiency of implementation.
  5. Measurement approaches—Some corporate funding agencies like the Gates Foundation, Skoll Foundation, and Omidyar Network insist on measuring citizen media projects, while other funding agencies like the Knight Foundation insist less on measurement. It’s important to measure both quantitative and qualitative outcomes and give constructive feedback to the contributors so that they can become more effective.
  • Quantitative—Objectives may sometimes change in response to your context, but keep the end goal in mind, continue to measure yourself against the objectives. This can be done through web analytics or web metrics—website performance monitoring service to understand and optimize website usage
  • Qualitative—Primarily anecdotal and used to shift policy objectives. In the end, however, it’s about visualizing the change you are trying to bring in the world, and making it happen.

 

 

 

 

 

 

 

 

Copyright © 2020 Integra Government Services International LLC