mHealth is by no means the magic bullet (I believe there is no magic in international development; every situation is so different and complex for one solution). But it could help to improve the public health workforce or women’s health in developing countries. In order to determine this, there needs to be further field research. And it needs to focus around the true impact of the intervention. While the social sciences are imperfect and it is very difficult to know an intervention’s true impact (meaning if the mobile device is removed, how different would the outcome be), there is a best practice to assess the true impact – Randomized Controlled Trails.

There are multiple examples of projects providing “evidence,” but they tend to only show who has been reached (reproductive health information was received by 20,000 mothers). We need further evidence to show that mobile phones and/or their content are creating the sought behavior change. mHealth could be like microfinance where there has been a lack of evidence showing the impact of it on families. But, knowing this, microfinance is still essential for the bottom of the pyramid to access necessary formal financial services (savings/insurance/credit). In the same light, mhealth tools help fill in gaps (ie sending information via text message instead of walking it in paper form). But does it really make a community health worker (CHW) more effective and efficient at their jobs? It will give women and families reproductive health information.  But does it improve ART intake or change a women’s behavior? These are the questions that must be answered in order to know the true impact of mhealth applications.

To the benefit of the sector, RCTs have and are being conducted. At the ICTD 2012 Conference in Atlanta a few weeks ago, Brian DeRenzi, who completed his Ph D at University of Washington in the department of Computer Science and Engineering, presented his paper entitled “Improving Community Health Worker Performance Through Automated SMS.” As the title indicates, the focus of his research was testing the impact of reminders via SMS to CHW’s follow up visits to their patients. The study was conducted in Tanzania and in collaboration with D-Tree International, Pathfinder International, and Dimagi. It included one pilot project and two larger studies. In the end, the reminders reduced the average number of days between follow up visits to patients by CHWs. This is a clear benefit to helping improve the care provided to patients. But the study also provided further knowledge into issues with patient reminders to CHWs. This included the benefit of having the supervisor of the CHWs receive a SMS reminder if their employee did not conduct a follow up visit. When the supervisor was taken out of the equation, the performance of CHWs decreased considerably. This is an important design aspect to the program that could have been missed without a RCT. By testing the removal of the supervisor, it showed that the mobile phones helped but combining it with the supervisor’s real time knowledge of their work provided greater incentive to the CHWs to follow up with greater regularity with their patients.

Another example of an RCT currently in progress is in Ethiopia. In a very similar study, Kate Otto, from the World Bank, is working in collaboration with Addis Ababa University to test the impact of mhealth interventions on the care provided by Health Extension Workers (HEWs) in rural areas to women and child (In Ethiopia, the HEWs are equivalent to CHWs in Tanzania).  The research question is “does the use of a mobile phone-based tool enabling patient registration, appointment reminders, and inventory management – in the hands of Health Extension Workers – result in improved maternal and child health outcomes in a rural Ethiopian setting?” The goal is to find evidence supporting the use of mhealth interventions with HEWs.

Since these RCTs seem very similar, it would be easy to say that the World Bank and Addis Ababa University should have simply copied the intervention and implementation of the best practices learned in Tanzania. But this is the power of RCTs. Since each region is different with varying degrees of complexity, solutions that work in one area will not produce the same outcomes in another. Just because a program and intervention worked one place does not mean the same model or theory will work in another setting (see m-Pesa). The benefit of RCTs is that it will assist in finding interventions that will provide a solution for the region/country. This is especially necessary in the mhealth sector as the complexity of each situation dictates the need for specific program requires. But we are not sure what those are. The benefit of RCTs is that it can remove these variables and focus directly on the how and why interventions work or not, especially in behavior change. And as shown in Tanzania, it helps us understand what is and provides insight into areas of improvements.

Funding is always going to be an issue with RCTs. But the knowledge gained makes up for the investment by further understanding the true impact (if any) of the device. In the end, we need to find answers to what changes behavior, especially the role of mhealth in the change. There has been a greater increase in RCTs in mHealth projects to test how mobile phones and their applications are improving health. But there needs to be more done.

For those interested in learning more about M+E in global health programs, there will be a panel through GHDonline.org at the beginning of April. Please find further information about the panel here.