evaluation https://blogs.worldbank.org/publicsphere/taxonomy/term/2622/all en From method to market: Some thoughts on the responses to "Tomayto tomahto" https://blogs.worldbank.org/publicsphere/method-market-some-thoughts-responses-tomayto-tomahto <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><h4> In this final post, Deval Desai and Rebecca Tapscott respond to comments by <a href="https://blogs.worldbank.org/publicsphere/turning-gaze-ourselves-acknowledging-political-economy-development-research" target="_blank" rel="nofollow">Lisa Denney and Pilar Domingo</a>, <a href="https://blogs.worldbank.org/publicsphere/beyond-quest-policy-implications-alternative-options-applied-development-researchers" target="_blank" rel="nofollow">Michael Woolcock</a>, <a href="https://blogs.worldbank.org/publicsphere/evidence-based-policy-or-policy-based-evidence-supply-and-demand-data-donor-dominant-world" target="_blank" rel="nofollow">Morten Jerven</a>, <a href="https://blogs.worldbank.org/publicsphere/policy-research-policy-difficult-places" target="_blank" rel="nofollow">Alex de Waal</a>, and <a href="https://blogs.worldbank.org/publicsphere/alchemy-relationships-and-production-evidence" target="_blank" rel="nofollow">Holly Porter</a>.</h4> <p> <img alt="Paktika Youth Shura " height="186" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/5595096029_0408cedf37_z.jpg" style="float:right" title="" width="280" />Our paper, <a href="https://blogs.worldbank.org/publicsphere/tomayto-tomahto-research-supply-chain-and-ethics-knowledge-production" target="_blank" rel="nofollow">Tomayto Tomahto</a>, is in essence an exhortation and an ethical question. The exhortation: treat and unpack <span>fragility research (</span>for we limit our observations to research conducted for policy-making about fragile and conflict-affected places) as an<em> institution of global governance</em>, a set of complex social processes and knowledge practices that produce evidence as part of policy-making. The ethical question: all institutions contain struggles over the language and rules by which they allocate responsibility between individual actors (ethics) and structural factors (politics) for their effects—this might be law, democratic process, religious dictate. In light of the trends of saturation and professionalization that we identify (and as Jerven astutely points out in his response, a profound intensification of research), is it still sufficient to allocate responsibility for the effects of fragility research using the language and rules of method?</p> <p> The five responses to our piece enthusiastically take up the exhortation. A series of positions are represented: the anthropologist (Porter), the applied development researcher (Denney and Domingo), the anthropologist/practitioner (DeWaal), the practitioner/sociologist (Woolcock), and the economist (Jerven). They unpack the profoundly socio-political nature of the relationship between research and policy from a number of different perspectives: Porter’s intimate view from the field, Jerven’s sympathetic ear in the statistics office, Woolcock and Denney and Domingo’s feel for the alchemic moments when research turns into policy at the global level, and de Waal’s distaste for the global laboratories in which those moments occur, preferring the local re-embedding of research. These all, of course, spatialize the research-policy nexus, just as we do; however, all then ask us to privilege one space over the others.</p> </div></div></div> Tue, 19 Jan 2016 20:18:00 +0000 Humanity Journal 7281 at https://blogs.worldbank.org/publicsphere Avoiding perversions of evidence-informed decision-making https://blogs.worldbank.org/publicsphere/avoiding-perversions-evidence-informed-decision-making <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p> <img alt="Emanuel Migo giving a presentation in Garantung village, Palangkaraya, Central Kalimantan, Indonesia." height="386" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/8206721435_1cd0bfe08d_z.jpg" style="float:left" title="" width="280" />How to avoid “We saw the evidence and made a decision…and that decision was: since the evidence didn’t confirm our priors, to try to downplay the evidence”</p> <p> Before we dig into that statement (based-on-a-true-story-involving-people-like-us), we start with a simpler, obvious one: many people are involved in evaluations. We use the word ‘involved’ rather broadly. Our central focus for this post is people who may block the honest presentation of evaluation results.</p> <p> In any given evaluation, there are several groups of organizations and people with stake in an evaluation of a program or policy. Most obviously, there are researchers and implementers. There are also participants. And, for much of the global development ecosystem, there are funders of the program, who may be separate from the funders of the evaluation. Both of these may work through sub-contractors and consultants, bringing yet others on board.</p> <p> Our contention is that not all of these actors are currently, explicitly acknowledged in the current transparency movement in social science evaluation, with implications for the later acceptance and use of the results. The current focus is often on a contract between researchers and evidence consumers as a sign that, in Ben Olken’s terms, researchers are not nefarious and power (statistically speaking) -hungry (<a href="https://www.aeaweb.org/articles.php?doi=10.1257/jep.29.3.61" target="_blank" rel="nofollow">2015</a>). To achieve its objectives, the transparency movement requires more than committing to a core set of analyses ex ante (through pre-analysis or commitment to analysis plans) and study registration.</p> <p> To make sure that research is conducted openly at all phases, transparency must include engaging all stakeholders — perhaps particularly those that can block the honest sharing of results. This is in line with, for example,<a href="https://egap.org/egap-statement-of-principles" target="_blank" rel="nofollow"> EGAP’s third research principle</a> on rights to review and publish results. We return to some ideas of how to encourage this at the end of the blog.</p> </div></div></div> Thu, 14 Jan 2016 19:47:00 +0000 Suvojit Chattopadhyay 7279 at https://blogs.worldbank.org/publicsphere Beyond the quest for "policy implications": Alternative options for applied development researchers https://blogs.worldbank.org/publicsphere/beyond-quest-policy-implications-alternative-options-applied-development-researchers <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><h4> This post, written by <a href="https://blogs.worldbank.org/team/michael-woolcock" target="_blank" rel="nofollow">Michael Woolcock</a>, is a contribution to an online symposium on the changing nature of knowledge production in fragile states. Be sure to read other entries by <a href="https://blogs.worldbank.org/publicsphere/tomayto-tomahto-research-supply-chain-and-ethics-knowledge-production" target="_blank" rel="nofollow">Deval Desai and Rebecca Tapscott</a> and <a href="https://blogs.worldbank.org/publicsphere/turning-gaze-ourselves-acknowledging-political-economy-development-research" target="_blank" rel="nofollow">Lisa Denney and Pilar Domingo</a>.</h4> <p> <img alt="Indonesia fills out form on rice" height="186" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/16246146098_fb21fd4e1c_z.jpg" style="float:left" title="" width="280" />My nomination for development’s ‘Most Insightful, Least Cited’ paper is Ariel Heryanto’s “The development of ‘development.'”<a href="https://humanityjournal.org/blog/beyond-the-quest-for-policy-implications-alternative-options-for-applied-development-researchers/#_ftn1" name="_ftnref1" rel="nofollow" id="_ftnref1">[1]</a> Originally written in Indonesian in the mid-1980s, Heryanto’s gem has been cited a mere 79 times (according to Google Scholar), even in its carefully-translated English incarnation. For me, this paper is so wonderful because it makes, in clear and clever ways, two key points that bear endless repetition, especially to today’s junior scholars. The first point is that inference from evidence is never self-evident: significance must always be interpreted through theory. Consider the seemingly obvious fact that the sun rises in the east every morning, he writes. What could be more universally and unambiguously true? The problem, of course, is that the sun does not rise in the east; instead, despite every piece of sensory evidence to the contrary, the earth rotates counterclockwise on its axis and revolves around a stationary sun, making it appear as ifthe sun rises in the east. But we only know this – or, more accurately, claim to know this – because today we happen to have a theory, itself based on more complex forms of observation and theory, that helps us interpret the prevailing evidence, to reconcile it with evidence from analyses of other cosmic phenomena, and thus draw broadly coherent conclusions and inferences.</p> <p> Heryanto’s second key point is that we are all captives of language, of the limits of any given tongue to convey the subtleties of complex issues. From this premise he proceeds to unpack the clumsy, alluring yet powerful word that in English we call ‘development’, noting that in Indonesian there are at least two very different interpretations of its meaning, and with this, two very different words – perkembangan and pembangunan – connoting two very different teleologies and policy agendas: the former a natural, ‘organic’ process akin to flowers blooming (“software”); the latter to an overt, intentional and ‘constructed’ political project of nation building (“hardware”). When translated into English, however, both perkembangan and pembangunan are typically rendered simply as “development,” thereby collapsing into a singular popular conception what in Indonesian discourse is a distinctly pluralist one. In the opening week of my class at the Kennedy School, which typically has 50 students who between them speak around 30 languages, we begin with a lively discussion of what “development” means in Arabic, Hindi, French, Turkish, Spanish, Swahili, Swedish… It turns out to mean all sorts of things.<a href="https://humanityjournal.org/blog/beyond-the-quest-for-policy-implications-alternative-options-for-applied-development-researchers/#_ftn2" name="_ftnref2" rel="nofollow" id="_ftnref2">[2]</a></p> <p> I open this way because I think the next article we need in this “genre” – though hopefully one that quickly transcends it because it is both highly insightful and highly cited! – is something akin to what Desai and Tapscott have begun with their ‘Tomayto Tomahto’ paper. In short, echoing Heryanto, we need more development research on development research. Such scholarship, however, would go beyond providing a mere chronology of changing professional styles, methodological emphases and funding characteristics (scale, sources, time horizons, expectations) to explanations of how and why such changes have occurred. Such explanations would be grounded in analyses of the shifting historical experiences and geo-political imperatives different generations of researchers have sought to accommodate, the particular ideas these experiences and imperatives rendered normative, and the concomitant gains and losses these changes have entailed for those finding themselves managing the “trade-offs” (such as they are) between scholarly independence and public utility.</p> </div></div></div> Tue, 22 Dec 2015 19:51:00 +0000 Humanity Journal 7255 at https://blogs.worldbank.org/publicsphere Turning the gaze on ourselves: Acknowledging the political economy of development research https://blogs.worldbank.org/publicsphere/turning-gaze-ourselves-acknowledging-political-economy-development-research <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><h4> This post by <a href="https://humanityjournal.org/author/lisa-denney/" target="_blank" title="Posts by Lisa Denney" rel="nofollow">Lisa Denney</a> and <a href="https://humanityjournal.org/author/pilar-domingo/" target="_blank" title="Posts by Pilar Domingo" rel="nofollow">Pilar Domingo</a> is a contribution to an online symposium from <em>Humanity Journal </em>on the changing nature of knowledge production in fragile states. Be sure to read other entries, beginning with <a href="https://blogs.worldbank.org/publicsphere/tomayto-tomahto-research-supply-chain-and-ethics-knowledge-production" target="_blank" rel="nofollow">Deval Desai and Rebecca Tapscott</a>'s piece.</h4> <p> <img alt="IBM Research - Africa Scientists at Riara School, Nairobi" height="190" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/19175369269_e19a8d2609_z.jpg" style="float:left" title="" width="280" />While researchers (ourselves included) now consistently underline the importance of understanding the political economy of developing countries and donors that support them in order to achieve better aid outcomes, the research industry remains largely ambivalent about questions of our own political economy. Desai and Tapscott’s paper is therefore a refreshing attempt to start unpacking this and the ways in which ‘evidence’ is produced within the development industry.</p> <p> Here, we offer reflections on three stages of this process: building evidence, translating evidence and dislodging evidence. But a word of caution is also merited upfront. The fact that we are talking about “evidence,” rather than research, is itself telling and underscores a shift in the development industry in the last ten years. Speaking about ‘evidence’ rather than about “research” suggests something much more concrete and indisputable. Evidence is taken as proof. But surely research is also debate. While there are, of course, things for which largely indisputable evidence can be found (the effects of vaccines on disease, for instance), the use of this terminology, particularly in the social sciences where little is concrete or universal, suggests that final answers are discoverable. It can, thus, be used to close down debate, as much as to encourage it. Research, on the other hand, recognizes that most findings are contributions to knowledge that helpfully allow to move us towards deeper understanding and greater awareness but do not claim to be the final word on a given topic.<br />  </p> </div></div></div> Wed, 16 Dec 2015 20:04:00 +0000 Humanity Journal 7251 at https://blogs.worldbank.org/publicsphere Tomayto tomahto: The research supply chain and the ethics of knowledge production https://blogs.worldbank.org/publicsphere/tomayto-tomahto-research-supply-chain-and-ethics-knowledge-production <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><h4> <img alt="Pre-test of Rural Household Survey, Pakistan" height="210" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/8339662450_d187df14b0_z.jpg" style="float:right" title="" width="280" />This post is the first in a symposium from <em>Humanity Journal</em> on the changing nature of knowledge production in fragile states. It was written by <a href="https://www.odi.org/staff/deval-desai" target="_blank" rel="nofollow">Deval Desai</a>, a Research Associate at ODI, and <a href="https://tufts.academia.edu/RebeccaTapscott" target="_blank" rel="nofollow">Rebecca Tapscott</a>, a PhD Candidate at the Fletcher School at Tufts University.</h4> <p> Aid in the 21st century is increasingly evidence-driven. Between 2000 and 2006, the World Bank spent a total of <a href="https://siteresources.worldbank.org/EXTECOSECWOR/Resources/chap2.pdf" target="_blank" rel="nofollow">$630 million on research</a>. By 2011 the World Bank was spending <a href="https://ieg.worldbankgroup.org/Data/reports/chapters/matrix_chap3.pdf" target="_blank" rel="nofollow">$606 million per year</a>, or about a quarter of its country budgets. In September of this year, by signing up to the Sustainable Development Goals, the global community enshrined a commitment to <a href="https://www.un.org/sustainabledevelopment/globalpartnerships/" target="_blank" rel="nofollow">“increase significantly”</a> a range of <a href="https://sdgfunders.org/data-sharing-to-achieve-the-post-2015-development-agenda/" target="_blank" rel="nofollow">high-quality data</a> over the next 15 years, to facilitate <a href="https://www.un.org/sustainabledevelopment/blog/2015/09/african-leaders-at-un-cite-remarkable-progress-on-mdgs-and-urge-commitment-to-post-2015-agenda/" target="_blank" rel="nofollow">qualitative as well as quantitative understandings of growth and progress</a>.</p> <p> As the international community seeks to tackle the “hard problems” of development—fragility, conflict, endemic poverty—qualitative research is ever-more important. These problems are not amenable to best-practice solutions but must be tackled through deep contextual understanding of their drivers. Or so the policy story goes.<a href="https://humanityjournal.org/blog/tomayto-tomahto-the-research-supply-chain-and-the-ethics-of-knowledge-production/#_ftn1" name="_ftnref1" rel="nofollow" id="_ftnref1">[1]</a> As a result, conducting qualitative research today is different from the days when <a href="https://itu.dk/~miguel/ddp/Deep%20play%20Notes%20on%20the%20Balinese%20cockfight.pdf" target="_blank" rel="nofollow">Geertz set out for Bali</a>. Gone are the intrepid individuals setting off to explore and explain an untouched environment, unaware of the demands of policymakers.<a href="https://humanityjournal.org/blog/tomayto-tomahto-the-research-supply-chain-and-the-ethics-of-knowledge-production/#_ftn2" name="_ftnref2" rel="nofollow" id="_ftnref2">[2]</a></p> <p> We argue that while practice has changed, the ideology of qualitative research has not. Qualitative research is generally understood as the<em> individual</em> exercise of research methods to produce knowledge about the world, knowledge that can then be taken up by governance actors of all stripes. By contrast, we believe that today we must understand research as <em>asystemic</em> intervention, within the broader context of globalization and international development. Therefore, we should start with the political economy of contemporary research—an iterative, professionalized and increasingly saturated practice—to rethink the political and ethical implications of the research that we do.</p> <p> As a first step to this end, we contrast two stylized frameworks for understanding qualitative research in fragile contexts: The “fragility research” framework, which we argue dominates the current debate; and the “research supply chain” framework, which we offer as a new framework and a provocation to discussion. We discuss each in turn, first considering how fragility research frames knowledge production in fragile or conflicted-affected states, identifying some assumptions the fragility research framework rests on, and critiquing some of its key conclusions. We then discuss the research supply chain as an alternative framework to explore the relationship between knowledge generation and policy. Finally, we raise some questions based on the new framework’s implications.</p> </div></div></div> Mon, 14 Dec 2015 18:45:00 +0000 Humanity Journal 7249 at https://blogs.worldbank.org/publicsphere Rethinking research: Systemic approaches to the ethics and politics of knowledge production in fragile states https://blogs.worldbank.org/publicsphere/rethinking-research-systemic-approaches-ethics-and-politics-knowledge-production-fragile-states <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p> <img alt="Classroom in Mali" height="186" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/2766432164_e6d8b5623d_z.jpg" style="float:left" title="" width="280" />Recently, <a href="https://humanityjournal.org/" target="_blank" rel="nofollow">Humanity</a>, a peer-reviewed academic journal from the University of Pennsylvania, has been hosting an online symposium on the changing nature of knowledge production in fragile states. In light of the intensification of evidence-based policymaking and the “data revolution” in development, the symposium asked what the ethical and political implications are for qualitative research as a tool of governance.<br /><br /> We are presenting their articles in the coming days to share the authors' thoughts with the <em>People, Spaces, Deliberation </em>community and generate further discussion.<br /><br /> The symposium will begin tomorrow with a short paper from <a href="https://www.odi.org/staff/deval-desai" target="_blank" rel="nofollow">Deval Desai</a> and <a href="https://tufts.academia.edu/RebeccaTapscott" target="_blank" rel="nofollow">Rebecca Tapscott</a>, followed by responses during the coming weeks from <a href="https://www.odi.org/experts/848-lisa-denney" target="_blank" rel="nofollow">Lisa Denney</a> and <a href="https://www.odi.org/experts/510-pilar-domingo" target="_blank" rel="nofollow">Pilar Domingo (</a>ODI); <a href="https://www.worldbank.org/en/about/people/michael-woolcock" target="_blank" rel="nofollow">Michael Woolcock</a> (World Bank); <a href="https://mortenjerven.com/" target="_blank" rel="nofollow">Morten Jerven </a>(Norwegian University of Life Sciences and Simon Fraser University); <a href="https://fletcher.tufts.edu/Fletcher_Directory/Directory/Faculty%20Profile?personkey=6D6C9778-0DD9-4125-963C-62E636338014" target="_blank" rel="nofollow">Alex de Waal</a> (World Peace Foundation); and <a href="https://www.lse.ac.uk/internationalDevelopment/people/PorterH.aspx" target="_blank" rel="nofollow">Holly Porter</a> (LSE). We hope that you enjoy the symposium and participate in the debate!</p> </div></div></div> Thu, 10 Dec 2015 16:45:00 +0000 Humanity Journal 7242 at https://blogs.worldbank.org/publicsphere Humanitarian broadcasting in emergencies https://blogs.worldbank.org/publicsphere/humanitarian-broadcasting-emergencies <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p> <img alt="A recording of BBC Media Action’s ‘Milijuli Nepali’ (Together Nepal)" height="210" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/bbcma-nepal.jpg" style="float:left" title="" width="280" />It is several days after the earthquake in Nepal. A small group of Nepali women sit on the side of the road in a village in Dhading district, 26 kilometres from Kathmandu. In this village, many people lost their homes and several died in the earthquake.<br /><br /> The women are listening attentively to a radio programme, <em>Milijuli Nepali </em>meaning ‘Together Nepal’. After it finishes, one of the women starts asking the others questions<em>: What did they think to the programme? Did they learn anything? What else would they like to hear to help them cope in the aftermath of the earthquake? </em>The women start discussing some of the issues raised around shelter and hygiene, they like the creative ideas suggestions, particularly as they comes from a source they like and trust - the BBC.  They give the researcher their ideas for future programmes and she writes them down.</p> <div class="asset-wrapper asset aid-214 asset-video"> <strong > BBC Media Action’s ‘Milijuli Nepali’ (Together Nepal) </strong> <div class="content"> <div class="field field-name-field-asset-video-file field-type-emvideo field-label-hidden"><div class="field-items"><div class="field-item even"><iframe width="854" height="480" src="//www.youtube.com/embed/3lFGvrka-CE?wmode=transparent&wmode=opaque" frameborder="0" allowfullscreen></iframe></div></div></div><div class="field field-name-field-asset-video-desc field-type-text-long field-label-hidden"><div class="field-items"><div class="field-item even"></div></div></div></div> </div><br /></div></div></div> Mon, 07 Dec 2015 18:38:00 +0000 Theo Hannides 7237 at https://blogs.worldbank.org/publicsphere Hawthorne effects: Past and future https://blogs.worldbank.org/publicsphere/hawthorne-effects-past-and-future <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p> <img alt="Maseru Shining Centuary Textiles" height="187" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/14093613796_a964c8ccb7_z.jpg" style="float:left" title="" width="280" />I have two main points in this blog. The first is a public service announcement in the guise of history. Not so long ago, I heard someone credit the Hawthorne effect to an elusive, eponymous Dr. Hawthorne, of which, in this case, there is not one directly tied to these studies. The second is a call to expand our conception of Hawthorne effects – or really, observer or evaluator effects – in the practice of social science monitoring and evaluation.<br />  <br /><strong>Hawthorne history</strong><br /><br /> The <em>Hawthorne effect</em> earned its name from the factory in which the study was sited: the Western Electric Company’s Hawthorne plant, near Chicago. These mid-1920s studies, carried out by MIT, Harvard, and the US National Research Council researchers were predicated on in-vogue ideas related to <a href="https://en.wikipedia.org/wiki/The_Principles_of_Scientific_Management" target="_blank" rel="nofollow">scientific management</a>. Specifically, the researchers examined the effect of artificial illumination on worker productivity, raising and lowering the artificial light available to the women assembling electric relays (winding coils of wire) in a factory until the artificial light available was equivalent to moonlight.<br />  <br /><img alt="" height="88" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/hawthorne_illinois_works_of_the_western_electric_company_1925.jpg" style="float:right" title="" width="240" />The finding that made social science history (first in the nascent fields of industrial and organizational psychology and slowly trickling out from there) was that worker productivity increased when the amount of light was changed, and productivity decreased when the study ended. It was then suggested that the workers’ productivity increased because of the attention paid to them via the study, not because the light was altered.<br /><br /> Thus, the “Hawthorne effect” was named and acknowledged: the change in an outcome that can be attributed to behavioral responses among subjects/respondents/beneficiaries simply by virtue of being observed as part of an experiment or evaluation.<br /></div></div></div> Thu, 05 Nov 2015 19:00:00 +0000 Heather Lanthorn 7212 at https://blogs.worldbank.org/publicsphere What do we know about the long-term legacy of aid programmes? Very little, so why not go and find out? https://blogs.worldbank.org/publicsphere/what-do-we-know-about-long-term-legacy-aid-programmes-very-little-so-why-not-go-and-find-out <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><h4> We talk a lot in the aid biz about wanting to achieve long-term impact, but most of the time, aid organizations work in a time bubble set by the duration of a project. We seldom go back a decade later and see what happened after we left. Why not?</h4> <p> <img alt="Orphaned and homeless children being given a non-formal education at a school in India" height="187" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/2347596279_18ffa58f80_z.jpg" style="float:left" title="" width="280" />Everyone has their favourite story of the project that turned into a spectacular social movement (<a href="https://www.sewa.org/" target="_blank" rel="nofollow">SEWA</a>) or produced a technological innovation (<a href="https://en.wikipedia.org/wiki/M-Pesa" target="_blank" rel="nofollow">M-PESA</a>) or spun off a flourishing new organization (<a href="https://newint.org/about/" target="_blank" rel="nofollow">New Internationalist</a>, <a href="https://www.fairtrade.org.uk/" target="_blank" rel="nofollow">Fairtrade Foundation</a>), but this is all cherry-picking.  What about something more rigorous:  how would you design a piece of research to look at the long term impacts across all of our work? Some initial thoughts, but I would welcome your suggestions:</p> <p> One option would be to do something like our own <a href="https://policy-practice.oxfam.org.uk/our-work/methods-approaches/effectiveness-reviews" target="_blank" rel="nofollow">Effectiveness Reviews</a>,  but backdated – take a random sample of 20 projects from our portfolio in, say, 2005, and then design the most rigorous possible research to assess their impact.</p> <p> There will be some serious methodological challenges to doing that, of course. The further back in time you go, the more confounding events and players will have appeared in the interim, diluting attribution like water running into sand. If farming practices are more productive in this village than a neighbour, who’s to say it was down to that particular project you did a decade ago? And anyway, if practices have been successful, other communities will probably have noticed – how do you allow for positive spillovers and ripple effects? And those ripple effects could have spread much wider – to government policy, or changes in attitudes and beliefs.<br />  </p> </div></div></div> Thu, 28 May 2015 16:32:00 +0000 Duncan Green 7063 at https://blogs.worldbank.org/publicsphere Where have we got to on Theories of Change? Passing fad or paradigm shift? https://blogs.worldbank.org/publicsphere/where-have-we-got-theories-change-passing-fad-or-paradigm-shift <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><h4> <img alt="Gum Arabic farmers at Hilat Ismaiel, North Kordofan, Sudan" height="185" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/8803372218_d5b396d207_o.jpg" style="float:left" title="" width="280" />Theories of change (ToCs) – will the idea stick around and shape future thinking on development, or slide back into the bubbling morass of aid jargon, forgotten and unlamented? Last week some leading ToC-istas at ODI, LSE and The Asia Foundation and a bunch of other organisations spent a whole day taking stock, and the discussion highlighted strengths, weaknesses and some looming decisions.</h4> <p> (Summary, agenda + presentations <a href="https://www.odi.org/events/4194-theories-change-development" target="_blank" rel="nofollow">here</a>)</p> <p> According to an excellent 2011 overview by <a href="https://www.theoryofchange.org/pdf/James_ToC.pdf" target="_blank" rel="nofollow">Comic Relief, ToCs are</a> an "on-going process of reflection to explore change and how it happens – and what that means for the part we play". They locate a programme or project within a wider analysis of how change comes about, draw on external learning about development, articulate our understanding of change and acknowledge wider systems and actors that influence change.</p> <p> But the concept remains very fuzzy, partly because (according to a useful <a href="https://www.google.co.uk/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=0CCQQFjAA&amp;url=http%3A%2F%2Fwww.theoryofchange.org%2Fpdf%2FDFID_ToC_Review_VogelV7.pdf&amp;ei=0vwkVZaSEYvXas6mgdgI&amp;usg=AFQjCNF8DB4HVgavUge7RPUt1VUkKZh6mQ&amp;sig2=0V9vxFeMoP0XHA8Bs" target="_blank" rel="nofollow">survey by Isobel Vogel</a>) ToCs originated from two very different kinds of thought: evaluation (trying to clarify the links between inputs and outcomes) and social action, especially participatory and consciously <a href="https://en.wikipedia.org/wiki/Reflexivity_%28social_theory%29" target="_blank" rel="nofollow">reflexive</a> approaches.</p> <p> At the risk of gross generalization, the first group tends to treat ToCs as ‘logframes on steroids’, a useful tool to develop more complete and accurate chains of cause and effect. The second group tend to see the world in terms of complex adaptive systems, and believe the more linear approaches (if we do X then we will achieve Y) are a wild goose chase. These groups (well, actually they’re more of a spectrum) co-exist <em>within</em> organisations, and even between different individuals in country offices.</p> </div></div></div> Tue, 21 Apr 2015 19:22:00 +0000 Duncan Green 7028 at https://blogs.worldbank.org/publicsphere Social Marketing Master Class: The Importance of Evaluation https://blogs.worldbank.org/publicsphere/social-marketing-master-class-importance-evaluation <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p> <a href="https://www.i-socialmarketing.org/social-marketing-definition#.VFvQSDJdV_A" target="_blank" rel="nofollow">Social marketing</a> emerged from the realization that marketing principles can be used not just to sell products but also to "sell" ideas, attitudes and behaviors.  The purpose of any social marketing program, therefore, is to change the attitudes or behaviors of a target population-- for the greater social good.<br /><br /> In evaluating social marketing programs, the true test of effectiveness is not the number of flyers distributed or public service announcements aired but how the program impacted the lives of people.<br /><br /><a href="https://www.psi.org/rebecca-firestone" target="_blank" rel="nofollow">Rebecca Firestone</a>, a social epidemiologist at <a href="https://www.psi.org/" target="_blank" rel="nofollow">PSI</a> with area specialties in sexual and reproductive health and non-communicable diseases, walks us through some best practices of social marketing and offers suggestions for improvement in the future.  Chief among her suggestions is the need for more and better evaluation of social marketing programs.</p> <br /> <div class="asset-wrapper asset aid-116 asset-video"> <strong > Social Marketing Master Class: The Importance of Evaluation </strong> <div class="content"> <div class="field field-name-field-asset-video-file field-type-emvideo field-label-hidden"><div class="field-items"><div class="field-item even"><iframe width="854" height="510" src="//www.youtube.com/embed/J5RIJaLjIbw" frameborder="0" allowfullscreen></iframe></div></div></div><div class="field field-name-field-asset-video-desc field-type-text-long field-label-hidden"><div class="field-items"><div class="field-item even"></div></div></div></div> </div></div></div></div> Thu, 06 Nov 2014 20:27:00 +0000 Roxanne Bauer 6869 at https://blogs.worldbank.org/publicsphere ‘Relevant Reasons’ in Decision-Making (3 of 3) https://blogs.worldbank.org/publicsphere/relevant-reasons-decision-making-3-3 <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p> <img alt="" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/7165383694_c6c41490de_o.jpg" style="float:left; height:186px; width:280px" />This is the third in our series of posts on evidence and decision-making; also posted on <a href="https://hlanthorn.com/2014/03/17/have-evidence-will-um-erm-3-of-6/" target="_blank" rel="nofollow">Heather’s blog</a>. Here are <a href="https://blogs.worldbank.org/publicsphere/have-evidence-will-um-erm" target="_blank" rel="nofollow">Part 1</a> and <a href="https://blogs.worldbank.org/publicsphere/have-evidence-will-um-erm-2-2" target="_blank" rel="nofollow">Part 2</a><br /> ***<br /> In our <a href="https://blogs.worldbank.org/publicsphere/have-evidence-will-um-erm-2-2" target="_blank" rel="nofollow">last post</a>, we wrote about factors – evidence and otherwise – influencing decision-making about development programmes. To do so, we have considered the premise of an agency deciding whether to continue or scale a given programme after piloting it and including an accompanying evaluation commissioned explicitly to inform that decision. This is a potential ‘ideal case’ of evidence-informed decision-making. Yet, the role of evidence in informing decisions is often unclear in practice.</p> <p> What is clear is that transparent parameters for making decisions about how to allocate resources following a pilot may improve the legitimacy of those decisions. We have started, and continue in this post, to explore whether decision-making deliberations can be shaped ex ante so that, regardless of the outcome, stakeholders feel it was arrived at fairly. Such pre-commitment to the process of deliberation could carve out a specific role for evidence in decision-making. Clarifying the role of evidence would inform what types of questions decision-makers need answered and with what kinds of data, as we discussed <a href="https://blogs.worldbank.org/publicsphere/have-evidence-will-um-erm" target="_blank" rel="nofollow">here</a>.</p> </div></div></div> Thu, 20 Mar 2014 16:30:00 +0000 Heather Lanthorn 6644 at https://blogs.worldbank.org/publicsphere Have Evidence, Will… Um, Erm (2 of 2) https://blogs.worldbank.org/publicsphere/have-evidence-will-um-erm-2-2 <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p> T<img alt="" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/2367330134_c6e0191752_b.jpg" style="float:left; height:224px; width:280px" />his is the second in a series of posts with <a href="https://suvojitchattopadhyay.wordpress.com/2014/02/07/have-evidence-will-um-erm/" target="_blank" rel="nofollow">suvojit</a>, initially planned as a series of two but growing to six…</p> <p> <strong><em>Reminder: The Scenario</em></strong><strong> </strong><br /> In our last post, we set up a scenario that we* have both seen several times: a donor or large implementing agency (our focus, though we think our arguments apply to governmental ministries) commissions an evaluation, with explicit (or implicit) commitments to ‘use’ the evidence generated to drive their own decisions about continuing/scaling/modifying/scrapping a policy/program/project.</p> <p> And yet. the role of evidence in decision-making of this kind is unclear.</p> <p> In response, we argued for something akin to Patton’s <a href="https://tinyurl.com/m9z8f4x" target="_blank" rel="nofollow">utilisation-focused evaluation</a>. Such an approach assesses the “quality” or “rigor” of evidence by considering how well it addresses the questions and purposes needed for decision-making with the most appropriate tools and timings to facilitate decision-making in particular political-economic moment, including the capacity of decision-makers to act on evidence.</p> </div></div></div> Wed, 12 Mar 2014 16:21:00 +0000 Heather Lanthorn 6634 at https://blogs.worldbank.org/publicsphere Have Evidence, Will… Um, Erm? https://blogs.worldbank.org/publicsphere/have-evidence-will-um-erm <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p> <strong><em><img alt="" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/7165365958_5c7c64728e_b.jpg" style="float:left; height:270px; width:180px" />Commissioning Evidence</em></strong></p> <p> Among those who talk about development &amp; welfare policy/programs/projects, it is <a href="https://www.inasp.info/uploads/filer_public/2013/04/22/what_is_the_evidence_on_eipm.pdf" target="_blank" rel="nofollow"><em>tres chic</em></a> to talk about evidence-informed decision-making (including the evidence on evidence-informed decision-making and the evidence on the evidence on…[<em>insert infinite recursion</em>]).</p> <p> This concept — formerly best-known as evidence-<em>based</em> policy-making — is contrasted with faith-based or we-thought-really-<em>really</em>-hard-about-this-and-mean-well-based decision-making. It is also contrasted with the (sneaky) strategy of policy-based evidence-making. Using these approaches may lead to not-optimal decision-making, adoption of not-optimal policies and subsequent not-optimal outcomes.</p> <p> In contrast, proponents of the evidence-informed decision-making approach believe that through their approach, decision-makers are able to make more sound judgments between those policies that will provide the best way forward, those that may not and/or those that should maybe be repealed or revised. This may lead them to make decisions on policies according to these judgments, which, if properly implemented or rolled-back may, in turn, improve development and welfare outcomes. It is also important to bear in mind, however, that it is not evidence alone that drives policymaking. We discuss this idea in more detail in our next post.</p> </div></div></div> Tue, 11 Mar 2014 15:56:00 +0000 Heather Lanthorn 6632 at https://blogs.worldbank.org/publicsphere #4 from 2013: Numbers Are Never Enough (especially when dealing with Big Data) https://blogs.worldbank.org/publicsphere/4-2013-numbers-are-never-enough-especially-when-dealing-big-data <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p> <img alt="" src="https://blogs.worldbank.org/publicsphere/files/publicsphere/Johanna/123962091_16ab427137.jpeg" style="float:left; height:210px; width:280px" /><em><strong>Our Top Ten Blog Posts by readership in 2013</strong><br /> This post was originally published on January 8, 2013</em><br /> <br /> The newest trend in Big Data is the personal touch.&nbsp; When both the <em>New York Times</em> and <em>Fast Company</em> have headlines that trumpet: “Sure, Big Data Is Great. But So Is Intuition.” (<em><a href="https://www.nytimes.com/2012/12/30/technology/big-data-is-great-but-dont-forget-intuition.html" target="_blank">The Times</a></em>) and “Without Human Insight, Big Data Is Just A Bunch Of Numbers.” (<a href="https://www.fastcompany.com/3004000/without-human-insight-big-data-just-bunch-numbers" target="_blank"><em>Fast Company</em></a>) you know that a major trend is afoot.</p> <p> So what’s up?</p> <p> The claims for what Big Data can do have been extraordinary, witness Andrew McAfee and Erik Brynjolfsson’s seminal article in October in the <em><a href="https://hbr.org/2012/10/big-data-the-management-revolution/ar/1" target="_blank">Harvard Business Review</a></em>: “Big Data: The Management Revolution,” which began with the showstopper:&nbsp; “‘<em>You can’t manage what you don’t measure</em>.’”&nbsp; It’s hard not to feel that Big Data will provide the solutions to everything after that statement.&nbsp; As the HBR article noted:&nbsp; “…the recent explosion of digital data is so important. Simply put, because of big data, managers can measure, and hence know, radically more about their businesses, and directly translate that knowledge into improved decision making and performance.”</div></div></div> Mon, 06 Jan 2014 20:30:00 +0000 Susan Moeller 6199 at https://blogs.worldbank.org/publicsphere