EVIDENCE PUZZLES
The history and politics of RCTs
In a 2016 working paper, JUDITH GUERON recounts and evaluates the history of randomized controlled trials (RCTs) in the US, through her own experience in the development of welfare experiments through the MDRC and the HHS:
“To varying degrees, the proponents of welfare experiments at MDRC and HHS shared three mutually reinforcing goals. The first was to obtain reliable and—given the long and heated controversy about welfare reform—defensible evidence of what worked and, just as importantly, what did not. Over a pivotal ten years from 1975 to 1985, these individuals became convinced that high-quality RCTs were uniquely able to produce such evidence and that there was simply no adequate alternative. Thus, their first challenge was to demonstrate feasibility: that it was ethical, legal, and possible to implement this untried—and at first blush to some people immoral—approach in diverse conditions. The other two goals sprang from their reasons for seeking rigorous evidence. They were not motivated by an abstract interest in methodology or theory; they wanted to inform policy and make government more effective and efficient. As a result, they sought to make the body of studies useful, by assuring that it addressed the most significant questions about policy and practice, and to structure the research and communicate the findings in ways that would increase the potential that they might actually be used.”
Link popup: yes to full paper. ht Will
- In a 2000 paper titled “The Politics of Random Assignment,” Gueron delves into further historically-informed questions about experimental methodology: “Pointing to the conservative effect on policymakers of disputes among experts, [Henry Aaron] asked: ‘What is an ordinary member of the public to do when the witch doctors (the scientists and scholars) disagree?’ He went further, arguing that such conflict not only paralyzes policy but also undercuts the ‘simple faiths’ that often make action possible. Random assignment, because of its unique methodological strengths, can help avoid this kind of conflict—what Aaron called ‘self-cancelling research.” Link popup: yes.
- An article from last summer by leading anti-poverty researchers Mary Ann Bates and Rachel Glennerster takes up the “generalizability puzzle” in the global development context, tackling four “common, but misguided” approaches to evidence-based policy, and presents their work—in the form of a “generalizability framework”—as a model for resolving these issues. Link popup: yes.
- A 2014 entry in the RCT wars from noted skeptic Lant Pritchett (whose work we’ve linked to previously popup: yes) makes the case against trials, specifically discussing Glennerster’s work at the Abdul Latif Jameel Poverty Action Lab: “An Homage to the Randomistas.” Link popup: yes. (And here’s the link popup: yes to a video of Pritchett’s February presentation at NYU’s Development Research Institute, previously noted in this space.)
- And link, again, to an excellent 2017 paper by Angus Deaton and Nancy Cartwright that threads the needle between the opposing sides of the RCT debate. Link popup: yes.
VERTICAL DESPOT
“How the connectedness of a social network affects the coevolution of despotism in leaders and tolerance to despotism in followers”
CEDRIC PERRET of EDINBURGH NAPIER UNIVERSITY et. al examine how the levels of clique influence, centralization, and connectedness among members of a social network relate to increased or decreased despotism.
From the conclusion:
“Social systems organized in hierarchy tend to develop into despotism with inequality created by and for the leaders. It has been proposed that the sole asymmetrical distribution of power is enough to lead to such transition. In particular, the leader and its influential clique could bias the opinion of weakly connected followers, ultimately crippling the followers’ capacity to control the leader’s decision. Yet, this scenario was missing of a model integrating and testing explicitly these mechanisms… Our results have demonstrated that the centralization of a social network would lead to higher despotism and inequality. It predicts that a transition from equality to despotism will happen in presence of (i) highly influential individuals with a preferential access of resources; and (ii) lowly connected followers. In addition, our model demonstrates how a low-level process such as opinion formation can strongly drive the evolution of a higher property, here the group organization.”
Link popup: yes to the preprint on Arxiv.
- For more on social networks generally, a 2015 paper by Stefano Tasselli et al provides an overview of “the microfoundations of organizational social networks,” emphasizing “dynamic process of reciprocal influence” between the structure of the network and the individuals within it. Link popup: yes.
- Work from 2012 by Cedric Sueur et al looks at social networks among primates as well: “We … showed a link between the type of social network and the resulting consensus. By comparing our theoretical data with data on five groups of primates, we confirmed that this relationship between social network and consensus also appears to exist in animal societies.” Link popup: yes.
+++
- From Stanford’s Computational Policy Lab, specific recommendations for making pretrial risk assessments more fair in the context of the California Bail Reform Act. Link popup: yes.
- In the New Yorker, Nathan Heller reviews a variety of books on Universal Basic Income. Link popup: yes.
- A novel paper on SNAP examines the geographic variation in the purchasing power of food stamps across states, and looks at the effects of those disparities on child health. The findings: “Lower SNAP purchasing power leads to lower utilization of preventive health care and more days of school missed due to illness.” Link popup: yes.
- “My analyses are unequivocal that racial resentment is reliably the largest and most precise predictor of attitudes toward immigration.” Link popup: yes.
- A compelling essay at Aeon about visual attention studies: “Are humans truly blind to the obvious? Recent research suggests otherwise. It suggests that this claim—so important to much of the cognitive sciences, behavioural economics, and now AI—is wrong. So, how could such an influential claim get it so wrong?” Link popup: yes. Relevant to the above RCT links, Dani Rodrik comments: “Theory comes first before evidence can count as evidence. The article applies this conclusion to AI’s limitations. I’d also add RCTs, which are often assumed to ‘let the evidence speak for itself.’ They don’t. Not without an interpretive frame.” Link popup: yes.
- A large group of digital ethics researchers and advocates signed an open letter petitioning the White House Office of Science and Technology Policy to solicit public comments on AI policy. Link popup: yes.
- Andrew Gelman posts about a flawed algorithm used by Illinois’s Department of Children and Family Services–and how numerical probabilities revealed the issue. Link popup: yes.
- For the monopsony file: a new IMF paper by Federico Diez, Daniel Leigh, and Suchanan Tambunlertchai finds an average increase of 39 percent in markups in advanced economies since 1980, with those higher markups being correlated to decreasing investment, innovation, and labor share. Link popup: yes. (And link popup: yes to an accompanying chart of the week post.)
- A characteristically excellent, detailed, and barely prompted thread from Pseudoerasmus on the British enclosures. Link popup: yes.
- Channeled attention and stable errors: “While many costly errors are prone to persist, in some situations a person will recognize her mistakes via “incidental learning”: when the data she values given her mistaken theory happen to also tell her how unlikely her theory is.” Link popup: yes.
- A 2016 paper by Christopher Beauchamp discusses the “forgotten history of the first patent litigation explosion” in light of our own present one. Link popup: yes.
- “Can stronger political competition improve rural livelihoods in developing countries? We explore this question in rural Pakistan, showing that greater political competition predicts significantly better access to publicly-provided infrastructure and amenities …It also predicts higher land values, greater land wealth, and lower land wealth inequality. Further, political competition increases land rental, possibly indicating improved functionality of land markets.” Link popup: yes.
- From Lukas Schlogl and Andy Sumner at the Center for Global Development, a paper on automation and the future of work in developing countries: “The Rise of the Robot Reserve Army.” Link popup: yes. ht Sidhya
- More future of work thoughts from Shane Greenstein at Digitopoly: Adjusting to Autonomous Trucking. Link popup: yes.
- “In an effort to woo the faithful, competing confessions advertised their superior ability to protect citizens against worldly manifestations of Satan’s evil by prosecuting suspected witches.” A post at EHS’s The Long Run blog on new research examining post-Reformation witch hunts. Link popup: yes.
Filed Under
Each week we highlight research from a graduate student, postdoc, or early-career professor. Send us recommendations: editorial@jainfamilyinstitute.org