As part of SDI's "Foundations of Democracy" series, we are asking academics and thinkers from around the world what conditions they believe we need to have in place as a society to enable a meaningful and equal democracy.
Dr. Roslyn Fuller speaks with Professor Anthony Fowler about his research, his views on the role of expertise in democracy and why shark attacks don't spell trouble for democracy.
RF: Thanks for taking the time to do this interview today. I’ve been looking forward to speaking to you for a long time. I came across your research several years ago, while writing a book defending the idea of democracy. As part of that process I decided to engage with some of the best (or at least most current) arguments I could find against democracy. And one that I came across again and again was the theory put forward by Achen and Bartels in their book Democracy for Realists and other papers that people change their votes because of shark attacks and therefore democracy, as we know it, cannot function properly.
That theory sounded pretty bizarre to me, but it seemed to be something that was very widely accepted and their book won many awards and substantial praise. When I looked into it about the only things that I came across that disputed their findings were your excellent paper that you wrote in collaboration with Andrew Hall and one guy in the Washington Post. So, could you tell us a bit about the research you did there?
AF: Yes, the claim in the original Achen and Bartels study is that shark attacks influenced Woodrow Wilson’s vote share in the 1916 Presidential election. There were these four fatalities in the summer of 1916 in New Jersey. It’s extremely unusual to have four shark fatalities in one summer in one place and it was a bit of a crisis in that area with the tourism industry taking a big hit.
It was a big deal, and Achen and Bartels went back and collected data. They did a couple of different analyses. They did an analysis at the town level, they did one at the county level and their claim was that in the affected counties and towns Woodrow Wilson’s vote share declined [compared to the 1912 election]. And the argument was, ‘Well, of course it wasn’t Woodrow Wilson’s fault that there were these shark attacks, and so it was irrational and incompetent and irresponsible of voters to have penalized him like that [by withholding their votes].’
One thing that is actually wrong about that argument, even before we get into the evidence, is that just because the shark attacks weren’t Woodrow Wilson’s fault, that doesn’t mean that it is irrational for voters to be affected by them. It could be the case that the shark attacks changed the information the voters had about Woodrow Wilson. It could be that they expected the federal government would come in and provide more support and more aid and they didn’t, and because of that the voters were disappointed by Woodrow Wilson and they ended up voting against him for that reason. So just on theoretical grounds, I think it’s not a very compelling argument to say that if shark attacks affected elections in this one instance, therefore democracy is broken.
But regardless, we could just ask: is the evidence even compelling? Is it true that shark attacks actually affected elections in this case? And so we do a number of different things in the paper you referred to. One thing we do is say: OK – if this is right, we should see it not just in this one instance, but we should see it in a bunch of other instances of shark attacks as well.
There have been about a hundred fatal shark attacks in US history, so we collected data on every single one of them. We pulled the data together and we got a pretty precise zero estimate. There is not a meaningful effect of shark attacks on Presidential vote shares. So that’s one thing we did in the paper. We looked at other cases and it looks like there is nothing going on on average.
The other thing we can do is we can go back to their town and county level analyses and ask, how robust are those findings? We’re looking at relatively small sample sizes, lots of researcher degrees of freedom, let’s just see: how reliable do those results look if we look at other specifications and use other degrees of freedom to see what results we would get?
So we actually have these plots in the paper where we look at thousands of different regressions to see what we would have gotten if we had used the same degrees of freedom that they used in their paper. And you can see that you could have gotten an estimate that made it look like shark attacks really hurt Woodrow Wilson’s vote share, or you could have made it look like shark attacks really helped Woodrow Wilson’s vote share, but on average the estimates are pretty much around zero. So again, there’s really not a whole lot going on there.
One other little thing we do in the paper is we show that the particular inferential strategy that they use is prone to false positives. And so that is a technical, methodological point, but one way that we demonstrate that is that we do a bunch of placebo tests.
We say, let’s take a bunch of states that are coastal states that didn’t have any shark attacks in a particular Presidential election cycle, let’s pretend they had a shark attack and then let’s re-do Achen and Bartels’ analysis and see how often would it look like shark attacks affected Presidential elections. And it turns out we get what look like significant results a third of the time, when you should only be getting it 5% of the time, so that’s another kind of technical methodological thing. So this particular approach is giving you standard errors that are too small and it is just a methodologically unreliable way to detect these effects.
There’s some other technical things in the paper. One of the things that we show is that a lot of the results appear to come from the unusualness of the 1912 Presidential election rather than whatever happened with shark attacks in 1916. 1912 was one of the most unusual Presidential elections in the sense that we actually had three Presidents on the ballot and the incumbent President came in third place. We show it was really 1912 that was a historical anomaly, not 1916.
So there’s lots of details in the paper, but essentially we go through all of those kinds of analyses, we show you all the different estimates you could plausibly get, we show you some of the methodological problems with what they have done. Overall, our conclusion is there is not much compelling evidence that shark attacks affect elections and this was probably just a false positive. Not that they did anything explicitly fraudulent. I think they believed that they had obtained a genuine result, but overall when you look at all the evidence it’s probably a false positive and probably nothing’s going on.
RF: So, what led you to conduct your research into this shark attack theory? And how did a theory based on not entirely rigorous research become so widespread?
AF: We looked into it because it is a really important finding in our field. Achen and Bartels are prominent scholars, they are very well respected, the finding itself was very well-cited and very well-regarded, and it was an important finding in the sense that people were drawing very significant conclusions from it and saying things like, ‘Because of the shark attack finding in New Jersey from 1916, democracy is broken’. They were making these very strong claims and we thought, well, maybe it is worth looking into the evidence a little more closely.
Even when you read their original paper, their evidence didn’t seem especially convincing to us. There are not a lot of data points and lots of opportunities for things you might call researcher degrees of freedom – things you can kind of play around with that make a result look more or less compelling. So that was the reason we looked into it.
Before I wrote this paper with Andy Hall, I had also written a paper on a college football study with Pablo Montagnes. That was a similar case where if it were true that voters were changing who they voted for based on the results of who won college football games that would seem to suggest that voters are somewhat incompetent and not doing a very good job of evaluating the actual quality and ability of their politicians, and they’re not setting good incentives for those politicians and so forth. So this was another one where we thought that this would be a pretty shocking finding for democracy if it were true, so let’s look into it in a little more detail.
In both cases we thought this is such a striking finding and people were drawing such strong conclusions that it warranted re-investigation just to see if these findings are genuine.
RF: I have heard that no one wants to try to replicate research any more (exactly what you and your colleague did in this case) as there is no 'glory' to be had in such activities, and thus claims are going unchecked to a much wider degree than they did in the past. Is this true, and if so, what are the consequences?
AF: Unfortunately, I think that is true. That’s not the way science should operate. Science should be a cumulative process with lots of back and forth. We should say, ‘Here’s an interesting finding and let’s re-assess it and let’s do another study and collect more data’. That kind of thing often doesn’t happen as much as you would like, because the incentives are such that you often don’t get lots of career rewards for being a really good scientist and figuring out what was right or wrong – you get them for being associated with new ideas and new theories.
So, if you’re just the person who goes around reassessing other people’s findings, I think a lot of people might view that as a good service to the field, but they won’t regard you as a really serious scholar. So, I think that is right and a lot of the top journals will even come out and say, ‘we only publish new ideas and new theories, we don’t publish replications and re-assessments’. So, I think that’s true to some extent, and I think it’s really unfortunate that that is true, because we should care a lot about whether we are getting things right or not.
At the same time, my paper with Andy was published in the Journal of Politics, which is regarded as one of the top three journals in the field of political science, he and I are both tenured at elite institutions and have had very successful careers. Obviously, we’ve been publishing some new research, but also doing a lot of what we think of as good science, figuring out what’s right and answering important questions and reassessing old findings. So, if I was giving advice to graduate students, I wouldn’t tell them to focus their entire career on replicating other people’s studies, but it is certainly something you can do and should do as part of a successful career in political science and lots of other scientific fields as well.
RF: I used to lecture at a university myself, and I have spoken to many people still at universities about the 'publish or perish' culture, preferably on things that can achieve 'impact'. I've also observed that there are a lot of self-promoting behaviours that used to be considered rather déclassé for an academic to engage in that are now de rigueur.
Obviously, one could argue that it is important for academics to engage in disseminating public knowledge and not lock themselves into an Ivory Tower, and also that one doesn't want to end up with tenured professors who effectively stop working.
Nonetheless, something seems to have gone terribly awry in this quest to achieve academic production and impact (studies getting twisted by media, people publishing unoriginal content, people feeling pressured to push their conclusions well beyond what their findings seem to support). Could you shed some light on this?
And also could you perhaps give us some thoughts on the role of academics in knowledge production and disseminating that knowledge in a democracy? What could an ideal look like in your view and how could it be achieved?
AF: I think the scientific incentives and the career incentives of scientists are really important to think about. And you can see these problems most clearly in some of the more sensational fields, like social psychology where every study you’ve ever seen discussed in the New York Times is a silly finding that probably isn’t even true. And so then your incentives as a social psychologist are, ‘let’s come up with a crazy theory or crazy hypothesis that probably isn’t right, but if I could find evidence of it then I can get in a top journal, I can get it covered in the New York Times and I can get all kinds of praise and accolades and so forth’.
So, I think those incentives are there and I think it is a very troubling thing for science and I think we have to try very hard to resist that amongst ourselves and police ourselves to make sure what we are putting out there is reliable. And that manifests itself in different ways.
Sometimes there is an incentive to publish sensational results. There’s the famous example of a Cornell psychologist publishing that people have ESP [extra-sensory perception] and that is obviously wrong and it has been debunked and replicated many times, but that’s the kind of finding where a journal is going to say, ‘Wow, that’s really exciting. What if people do have ESP?’ So there is that kind of extremely implausible finding.
Related to that – I think there is a politicization of science, which is really unfortunate. There’s certain kinds of results that are much more likely to make their way into top journals than other kinds of results just because the results or findings are more politically desirable. There’s lots of examples of that.
And then there is, related to what we are talking about today, the sensationalization of results in the sense that if I publish a paper that says, ‘You know what – democracy is not perfect, but it is more or less working the way it is supposed to and voters are doing a reasonably good job,’ then that is probably not going to make its way into a really prominent journal. It’s not going to be a famous, splashy paper. But if I say, ‘Oh my goodness, here’s this horrifying thing that is going on and democracy is broken’, that’s going to get everyone’s attention. So, I’m not saying that specific scholars intentionally misinterpret their evidence just to get more attention, but the incentives are there and it is easy to subconsciously fall into that trap and end up writing a splashy paper that may not be reliable.
RF: How can academics interact with the general public in a way that minimizes distortions? We seem to get these journalistic interpretations of science that are often contradictory. For example, you’ll see one article exclaiming that tomatoes give you cancer and another headline that you have to eat tomatoes to prevent cancer with this kind of utter yet contradictory certainty and when you look at the underlying data it’s usually not all it’s cracked up to be either way. The same could be said of many articles relating to political science or social science as well.
AF: There’s a lot to say about this whole topic. I think it is a good thing for academics and scientific experts to be involved in public debates, and to be involved in public policy making, because if we don’t have scientists and we don’t have evidence, what else are we using to make these decisions? You’d rather have the evidence available than not have it. When it comes to making policies, as flawed as the scientific process can be and as many limitations as we have, I still think it is a good idea to have scientific experts and academics weighing in on all those policy debates, as opposed to just practitioners with their gut feelings.
So, I think it is a good thing to have science and evidence and logic and reason and compelling theory involved in these policy debates. But at the same time, of course, we want to set the incentives such that academics are not prone to sensationalizing their results and instead they are weighing evidence in reasonable ways.
One possibility is that scientists are better than individual studies. If scientists are really well-trained to evaluate things and think about evidence, it might be the case that we shouldn’t hang our hat on one study, but we should be able to call upon experts who have been working in these fields for a long time and say ‘can you weigh all the evidence for us and help us navigate these things and consider all of these costs and benefits and trade-offs and so forth?’
I think there are productive ways for scientists and academics to be involved in important decisions, but we shouldn’t expect any one study to be the end-all, be-all answer to these questions.
Another thing that I think is very important is for experts to be very careful not to impose their own normative views on everyone else. And I think there is a way that that can happen. I can come out and say, ‘Look, we’ve done the best studies we can on this question. We’ve figured out that there are the following costs and the following benefits and the following effects and my own personal philosophical and moral view, whatever that is, would weigh these things in a certain way, but maybe the public weighs them differently, maybe the politicians weigh them differently’.
I think it is a mistake when academic experts, who should be really good at evaluating the evidence and quantifying things and theorizing about things, impose their own normative values on everyone else. That happens a lot more often than you would like and it happens often without people even realizing that it is happening. Anytime an economist tells you, ‘I did a cost-benefit calculation and I figured out that this is the right policy,’ they adopted a particular normative framework that may not be your normative framework. And I think that’s a mistake on the part of the scholar.
This is relevant for the kind of sensational work where scholars write that ‘democracy is broken’. When Jason Brennan says, ‘democracy is broken we should abandon it all together’ he’s adopting a very particular normative framework and imposing it on everybody else. He should be very clear about which parts of his argument come from evidence, which parts of his argument come from his own moral views, and be careful not to assume that everyone should have his moral views. He can try to persuade them, but he should be clear about what conclusions are coming from his normative views and what conclusions are coming from the evidence.
RF: It seems to be that there has been an increasing tendency to view democracy as an exercise in divining ‘the one right way’ to do things via expertise, rather than a more traditional view of party democracy as being composed of ‘party interests’. Traditionally political parties have often been somewhat aligned with various interests, for example, union workers, with it being understood that their interests weren’t necessarily identical to the interests of say, freelance professionals or large landowners. Now it seems like there is this expectation in some circles that there is a ‘right answer’ to things like international trade, unemployment or a topic like ‘race in America’ that can be divined by experts.
AF: I think it would be a huge mistake for us to fall into that language and say that, ‘science tells us there is a right policy on trade’ or ‘science tells us there is a right policy on climate’, or pick whatever controversial policy you want to think about. I think the job of scientists should be to say, ‘Look, we’ve done the best studies we can and we’ve figured out that if we implement the following trade policy, it is going to have these different effects. It’s going to hurt these people and it is going to help these people and it’s going to have these short-term effects and these long-term effects, etc.’ And then from there, members of the public, policy-makers, politicians can decide how they want to trade off those costs and benefits.
I think it is a mistake to say, ‘oh, we figured out the right policy is this, so you should just shut up and deal with it’, because there is no such thing as the right policy. There’s lots of different things to trade off and depending on your own views you might trade those things off differently.
RF: How do people in democracies today interact with experts and how do they decide which experts to trust?
AF: Most members of the public are not interacting very regularly with experts, but they are being presented with expert views when they pick up the newspaper. And I think it is very common for members of the public to figure out ‘what should I trust and what expert should I trust?’ and I do think it is very difficult for them to navigate this.
I think unless you are an expert in one of these fields it is almost impossible to wade through all of the arguments and even have the time, the energy and expertise to be able to trade these things off.
So, I think another reason why it is a mistake for science to be so bold in proclaiming that they have figured out the right answers is that it can actually erode their own credibility over time. I understand why members of the public might just say, ‘You know, I don’t know who to trust anymore, so I’m just going to go with my gut.’ I think that is not a good thing, because we should be able to do better than that.
RF: So maybe encourage people to present their findings or advice in terms of probability? Like, ‘this is probably the right answer’ or ‘I’ve done these five experiments and it all seems to point this way, so that is where we are at’?
AF: Yes, I think that is a big thing, conveying uncertainty. Every study comes with uncertainty, so we should do a better job of conveying the amount of uncertainty that is associated with certain results and conclusions and recommendations. And I think also conveying which part of scientific recommendations are coming from the evidence and which parts are coming from our different moral views and how we are trading things off. All of those things are important to discuss.
RF: Is there anything else you would like to discuss?
AF: Yes, we talked about Achen and Bartels, but there are actually a lot of books now that are pushing against democracy and saying democracy is broken. I mentioned Jason Brennan’s book, there’s Ilya Somin’s book, and Bryan Caplan. There are these four books and probably more prominent arguments as well, but all coming up and saying democracy is broken in some way and maybe we should do away with it or reform it dramatically.
They end up in different places with different recommendations, but they all start with the premise that things like the shark attack result and other empirical findings show that voters are irrational, voters are incompetent, voters are overly partisan, etc. So that’s kind of the starting point for all of these books and I think there are a bunch of mistakes that all these books are making, at least four mistakes in fact.
I think Mistake Number One is what we’ve already talked about – that the evidence itself is just not very compelling. These things like shark attacks and college football affecting elections is just not true. The claims that voters are overly partisan is I think also not very compelling – I have some other work related to that, so I think point one is that the starting evidence is just not very compelling. Maybe the voters are not as incompetent as people think.
The second mistake they’re making is that even if the voters are somewhat incompetent or irrational that doesn’t necessarily mean that aggregate election results are unreasonable. It might be the case that you got 10% of voters that are just crazy partisans and 20% of voters are affected by random things like football games, and nevertheless the median voter is still responding to the quality and the performance of the incumbent in a reasonable way such that the incentives are good and selection is good. I have a mostly theoretical paper with Scott Ashworth on that point that shows that you can have lots of irrational, incompetent voters and nevertheless have aggregate election results that behave more or less the way you would hope. So that’s a second reason this argument is not very compelling.
A third thing is that even if aggregate election results are somewhat irrational, that is not necessarily bad for political outcomes either, because there are actually some cases where what might look like irrational behaviour on the part of voters could actually be better for setting incentives for politicians to make them work harder. Imagine in an extreme case, an incumbent in a very partisan district who matches the party of their voters. They might have no incentive to work hard, but if the voters are a little bit irrational and every once in a while vote against them because of the football game that might motivate them to actually go out and work harder to make sure they can please the voters. My colleagues Scott Ashworth and Ethan Bueno de Mesquita have a paper on that topic.
And then the fourth thing that I think is wrong with these arguments is that even if democracy is bad in some sense, you still have to make the argument that it is worse than the alternatives. Of course, voters do not live up to our ideal view of what a perfect democracy would look like, but nobody ever thought that they could. The question is, is there some better alternative? And often the alternatives are equally flawed such as ‘people like me should get to be dictators’.
So, I think those are arguments that should be out there, we should be pushing against that. Just because we have these studies that say, voters are kind of irresponsible sometimes doesn’t mean that democracy is broken.
RF: You are definitely the most optimistic American academic I have spoken to in a long time. After Trump and Brexit, people seemed extremely pessimistic about democracy all of a sudden. What is the mood in the US and what is the mood among your colleagues? What is your personal view on democracy and the future of democracy in the US?
AF: To put my cards on the table, I’m not a big fan of Donald Trump and I was not a big fan of Brexit. There are certainly democratic results that I would prefer had gone the other way. But that’s the way democracy works. No one person gets to have their way all the time in a democracy.
There is a lot of elitism coming from academia that says things like, ‘Oh my goodness, look at these idiots who voted for Donald Trump, look at these idiots who voted for Brexit, they must just not understand, or they must be racist and have horrible preferences’.
And maybe there’s some of that. Maybe there is some extent to which had voters been better informed and had voters had better preferences maybe the outcomes of these elections would have been different. But I also think it is elitist to say that, in the sense that there are lots of people with very different circumstances than us privileged academics or privileged journalists and so forth. And I think that it is a mistake for us to just look down on these people and say, ‘Oh, they must be idiots because they voted differently than the way I would have voted’.
It might just be the case that they are thinking about different trade-offs than we are. In the case of Donald Trump, there are a bunch of working class people in formerly industrial places that lost their jobs and they are being affected by trade and immigration and so forth. They obviously place a lot of value on those issues and they had a politician come along for the first time and say, ‘hey, I really care about you and I’m going to try to get you your jobs back’.
Even if that turned out to be wrong, I understand why somebody in that position might be tempted to vote for Donald Trump, so I think it’s a mistake for us to sit in our Ivory Towers and say, ‘oh look at those idiots who voted for Donald Trump’. I think we should understand that people have different preferences, that people face different challenges than we do and it is not a lot of comfort for them to have the economists on television telling them ‘on average trade is good for the country’ when it might be hurting their family and their household directly. So that’s just one example of the elitism that plays a role in these academic conversations.
What do I think about the health of democracy overall?
Again, democracy is not perfect, there’s no sense in which the voters are behaving in some kind of idealized view of democracy where everything works perfectly, we always select the high-quality candidate, we always set the right incentives for them and so forth. That’s certainly not right.
But on the whole I still have a lot of faith in democracy in the sense that the voters appear to be behaving reasonably, the voters care about policy, the voters don’t act so irrationally and in such an incompetent way and the incentives are such that politicians do have to do a reasonably good job. They have to moderate their positions, they have to work hard if they want to stay in office. It’s not great. It’s not perfect. There’s lots of problems. We have lots of polarization among our elected officials in the United States and it seems like neither party is doing a very good job of catering to the preferences of the median voter. There’s corruption, there’s all kinds of problems, but I haven’t seen any convincing evidence that there’s some other system other than democracy that’s going to produce better outcomes.
RF: That’s interesting that you referred to polarization of officials, rather than polarization in general. I interviewed Prof. Morris P. Fiorina a few years ago and his work largely shows that American voters are not actually very polarized (despite what we hear to the contrary) but that higher-profile politicians and activists are. What is your take on that?
AF: I agree completely. American voters are not especially polarized. I have a working paper called ‘Moderates’ with a bunch of co-authors where we have taken a bunch of fresh data and applied some new methods to that question to try to figure out how many Americans really are somewhere in between the two parties and the answer is: the vast majority of them. So, I think that is right, I think the vast majority of Americans are ideologically moderate.
We’ve asked a bunch of questions directly. They don’t want a $15 minimum wage which is what Nancy Pelosi and the Democrats want, but they don’t want the status quo which is what Mitch McConnell and the Republicans want. They want an $11, $12, $13 minimum wage. And that is true even of people who identify as Democrats or Republicans. The median Republican is around $11 and the median Democrat is around $13. So, they are not nearly as extreme as their elected officials, and that’s just one example.
It is also true that moderate candidates do better in elections. When voters are presented with a moderate choice, they are much more likely to pick that moderate choice, it’s just that they almost never get a moderate candidate to even consider. That’s one of the big questions in American politics right now: Why are the elected officials so extreme and so polarized when the voters are relatively moderate?
RF: Well, thanks so much for taking the time to talk to me. I feel like we could talk about these issues all day, but it has been great to have this conversation and hear more about your work, and hopefully we can have you back again, sometime.
Anthony Fowler is a Professor in the Harris School of Public Policy at the University of Chicago. His research applies econometric methods for causal inference to questions in political science, with particular emphasis on elections and political representation. Specific interests include the causes and consequences of unequal voter turnout, explanations for incumbent success, the politics of policymaking in legislatures, and the credibility of empirical research.