The following post is by Howard White, Executive Director of the International Initiative for Impact Evaluation and originally appears on the 3ie blog.
Recently the Chris Evans breakfast show on UK’s Radio 2 picked up a news story on a Danish study reporting that half an hour’s exercise a day is better for you than one hour. Like me, the radio presenters were puzzled by this finding and wanted to know more.
Middle aged men of reasonable fitness were randomly assigned to two groups, one doing half an hour a day and the other a full hour. After three months the group exercising less had lost more weight. The half hour exercise group lost eight pounds compared to six pounds for the one hour group; about one kilo more weight loss.
The study was widely covered in the press; partly because it has a message most people would like to hear, but also for the novelty value of counterintuitive results. Indeed, the researcher himself is at a loss to explain why this should be so.
Perhaps, he says, the half hour group weren’t tired after half an hour. Hence they exercised some more and so they didn’t actually exercise less! Or perhaps they exercised much harder in that short time. But some simple calculations of plausible differences in exercise intensity show it’s not possible to burn more calories in half an hour than what would be burned in one hour. Maybe, he added, those exercising more felt the need to eat more, and over-compensated the calories loss. This last explanation is plausible: a sports drink and chocolate bar wipes the calorie loss from about half an hour of exercise. And a listener to the programme pointed out that those exercising more may have built more muscle, which weighs more than fat.
To an outsider the real puzzle here is that the study wasn’t set up to be able to explain its findings.
The strength of randomized control trials (RCTS), like this study, is their ability to establish causal relationships between the intervention and the outcome. But we need factual analysis of what happened, to help complement the counterfactual analysis of causality. In the case of this study, participants should have been asked to keep food and exercise diaries. (Though doing so raises another often ignored problem in research that ‘the measurement is a treatment’. Keeping diaries often reduces food intake and increases exercise volume).
Similarly many randomized control trials in international development fail to pay adequate attention to collecting data around the intervention’s theory of change. And so the authors resort to non-evidence based speculation to explain their findings. But without such explanation the appropriate policy response to study findings is often unclear.
The Danish researcher plans to go on to study exercise and commuting. But as one of the radio presenters exclaimed ‘he can’t do that, he hasn’t finished this research yet’. The presenter is right. RCTs which just link the intervention to outcomes are unfinished research projects. Incredible findings won’t become credible without a plausible, evidence-based explanation.
The UK Department for International Development recently released a new paper on the use of theories of change in international development. 3ie’s paper on theory-based impact evaluation also offers guidelines for mapping out the causal chain of an intervention. Developing a theory of change helps identify the evaluation questions and can very often increase a study’s policy relevance.
Recently the Chris Evans breakfast show on UK’s Radio 2 picked up a news story on a Danish study reporting that half an hour’s exercise a day is better for you than one hour. Like me, the radio presenters were puzzled by this finding and wanted to know more.
Middle aged men of reasonable fitness were randomly assigned to two groups, one doing half an hour a day and the other a full hour. After three months the group exercising less had lost more weight. The half hour exercise group lost eight pounds compared to six pounds for the one hour group; about one kilo more weight loss.
The study was widely covered in the press; partly because it has a message most people would like to hear, but also for the novelty value of counterintuitive results. Indeed, the researcher himself is at a loss to explain why this should be so.
Perhaps, he says, the half hour group weren’t tired after half an hour. Hence they exercised some more and so they didn’t actually exercise less! Or perhaps they exercised much harder in that short time. But some simple calculations of plausible differences in exercise intensity show it’s not possible to burn more calories in half an hour than what would be burned in one hour. Maybe, he added, those exercising more felt the need to eat more, and over-compensated the calories loss. This last explanation is plausible: a sports drink and chocolate bar wipes the calorie loss from about half an hour of exercise. And a listener to the programme pointed out that those exercising more may have built more muscle, which weighs more than fat.
To an outsider the real puzzle here is that the study wasn’t set up to be able to explain its findings.
The strength of randomized control trials (RCTS), like this study, is their ability to establish causal relationships between the intervention and the outcome. But we need factual analysis of what happened, to help complement the counterfactual analysis of causality. In the case of this study, participants should have been asked to keep food and exercise diaries. (Though doing so raises another often ignored problem in research that ‘the measurement is a treatment’. Keeping diaries often reduces food intake and increases exercise volume).
Similarly many randomized control trials in international development fail to pay adequate attention to collecting data around the intervention’s theory of change. And so the authors resort to non-evidence based speculation to explain their findings. But without such explanation the appropriate policy response to study findings is often unclear.
The Danish researcher plans to go on to study exercise and commuting. But as one of the radio presenters exclaimed ‘he can’t do that, he hasn’t finished this research yet’. The presenter is right. RCTs which just link the intervention to outcomes are unfinished research projects. Incredible findings won’t become credible without a plausible, evidence-based explanation.
The UK Department for International Development recently released a new paper on the use of theories of change in international development. 3ie’s paper on theory-based impact evaluation also offers guidelines for mapping out the causal chain of an intervention. Developing a theory of change helps identify the evaluation questions and can very often increase a study’s policy relevance.
4 comments:
Hi Tom. That's partly what I was trying to say a while back, when everyone went nuts on Jeff Sachs:
http://humanosphere.kplu.org/2012/06/metrics-mania-afflicts-the-fight-against-poverty/
I suspect Sachs critics will respond that, in fact, Sachs' MVP suffers for lack of a theory of change and clear causality.
Okay, but I think this essay implies is that trying too hard to assign simple causality can be a mistake. In development, I think it can have a fairly adverse effect -- that of pushing funding and donors toward the simplest of unilateral interventions rather than working on the more complex root causes that drive and sustain poverty.
I don't have a solution for this problem (and I don't have to, since I'm just a journalist and my job is to prod the experts to solve it). Reductionist thinking can be powerful if used within its limits. But it can also quite clearly create distortion and send people off in the wrong direction. See drunk under lamp post ....
chocolate
chocolate, candy, cookie and candy gifts, and candy gourmet gift holders delivered national. Send a candy bouquet today.
"...those exercising more may have built more muscle, which weighs more than fat." This is why trainers usually advise to weight loss candidates that only "a little" exercise would help. Their purpose is to burn the fat and not to build muscles - maybe that's why the group who exercised for a full hour only lost a little weight. Most would rather prefer toning instead of muscle building, because they would feel better with less weight than bigger muscles.
organic grocery
I Interesting post. I have been wondering about this issue,so thanks for posting.
Post a Comment