09 June 2011

Thoughts on RCTs from Microfinance USA

I should have known better than to have entered the Microfinance USA conference with assumptions about how ideas would be received. In my pre-post, I expressed interest in observing whether the gap between practitioners and researchers had reduced by any measureable amount. “Surely the publication of More than Good Intentions and Poor Economics would have cleared up any of the existing tensions,” I thought to myself while drinking tea at the conference blogger’s station.
The end of my fantasy came quickly, as if a dream itself, with the opening remarks by Jonathan Lewis, CEO of Opportunity Collaboration:
The latest storm cloud from international microfinance is the social measurement and evaluation fad. Inventing social metrics has become the popular pet project of academics, foundations and consultants… 
Micro-enterprise is a social venture with ripple impacts in many directions and for many years. Its value extends beyond – way beyond – the lifespan of a randomized controlled trial. Let’s avoid delegating to the measurement mafia -- even when they are motivated by good intent -- a kind of hegemony over social change.
If anything, it appears that RCTs have grown in popularity and opposition at a relatively equal rate.  The gap was as present as ever and did not appear to be getting smaller.  In fact, the two sites could be growing apart.  It was my hope that panels with researchers and practitioners during the day would quell my growing concerns.

The ensuing panel, with RCT implementer Dean Karlan, did bring up how RCTs are created, but stayed on the topic of discussing savings as the future of microfinance.  However, it was an afternoon panel “Promise and Peril of Microfinance Impact Evaluations” that brought Karlan back on the stage with David Roodman of the Center for Global Development, Jonathan Morduch of NYU and Christopher Dunford of Freedom from Hunger to debate and discuss the issue of impact evaluations.

A version of this panel appeared at the Microfinance Innovation and Impact Conference this past fall, with Dunford serving as the antagonist to the RCT researchers. In the first instance Dunford was constructive in a reserved way; this time he offered a strong bite with no venom. Speaking last, Dunford praised the efforts of RCTs, but expressed concerns with the interpretation of data by non-academics and felt that, “RCT research has the risk of being oversold, much like microfinance has been accused of doing. Problems occur when results do not match rhetoric.” To Dunford, the greatest reason for concern is RCTs becoming gate keepers for program implementation due to problems with external validity.

Karlan followed up the raised issue of external validity by saying it can be overcome via theory, but did not share an example of how it can be accomplished.  The members of the panel sparred a bit as they knew it came from a place of concern being that Dunford’s Freedom From Hunger has utilized RCTs to measure its work on numerous occasions.

Prior to Dunford, David Roodman gave what I think to be the best illustration of how research findings can have a big impact for better or for worse.  Although he did not call out the researchers, Roodman has been engaged with a bit of a debate, alongside Morduch, with Mark Pitt and Shahid Khandker over their 1998 microcredit paper which cemented the idea of microcredit reducing poverty.

In attempting to replicate the original research, Roodman and Morduch were unable to come up with the same results as Pitt and Khandker.  The announcement of the inability to replicate the trial sparked a bit of a spirited exchange between the four.

Whether the findings by Pitt and Khandker should or should not be considered valid any longer is not as much of a concern as the problems with replication.  In telling the story, Roodman helped to illustrate some of Dunford’s reservations.  Roodman summed up what was a cautionary panel well saying, “We need to understand the dynamics of the microfinance industry and figure out how to make those as healthy as possible.”  

The gap remains, as evidenced by the remarks by Lewis, but more pragmatic discussions, like the one held between Roodman, Dunford, Karlan and Morduch will be the most effective way of shrinking the distance between researchers and practitioners.  Morduch’s quick thoughts after the conference make me think that it could shrink soon saying, “Dunford’s right: we need to have a bigger conversation on the ways that evaluations get used and positioned – and when they’re appropriate and when they’re not. I just don’t want that conversation to lead to excuses to pull back on doing evaluations (and neither does Dunford): we already have too many excuses not to hold ourselves accountable.” Also, IPA is looking for research partners for domestic microfinance programs, maybe some at the conference have been convinced to give it a shot.