Flying blind, or why I don't take surveys anymore

Commentary on the attempted mind-reading complex

Two interesting articles in the Wall Street Journal didn’t appear next to each other on my app. But they were close enough that I wondered why they didn’t.

The first was about how polling didn’t work in 2020, with fears that it won’t work in 2024:

The second was about how fundamental economic theories are no longer explaining how the economy works:

I’ll give you the TL;DR on these, because I think the appearance of both in the same morning says some interesting things about how we understand what people are thinking (polling), who they’re likely to vote for (polysci) and what they’re up to when it comes to money (economics).

Maybe it’s bad form to bash social sciences at this point, but go ahead and get out your handkerchief, this might get gory.

After the most inaccurate set of predictions in the last “40 years” in 2020, pollsters and academics studying polling are saying that 2024’s polling might be insufficient for an accurate prediction of the election’s progress and outcome (I have to wonder what they thought of 2016, which seemed even worse). The upshot of the WSJ article is everyone is confused: of the two main polling methods, online and phone, neither has emerged to be more effective than the other. We are losing touch with what people actually think about elections. People aren’t being sampled correctly, aren’t being asked the right questions, aren’t answering truthfully. The list of possible explanations goes on and on (after the expected ‘blame covid’ is put forth).

But I also wonder if something else is at play. It has to do with a personal conviction I have, one that I’m adamant about in my middle age: t few years ago, I pledged to not do any surveys - for anyone - unless I was being paid for it.

Unless you are like the farmer across the field from me, you know why I feel this way. Everything has a survey attached to it: post-takeout restaurant food surveys, surveys at the cash register, and don’t forget surveys popping on websites, on apps, and at the doctors office.

Data is now emerging (from more surveys) that customers hate long surveys so much that they’ll stop using your product. Similarly, I’m no longer providing my opinion for free. I doubt I’m alone. Maybe I should come up with a name and a URL for the pledge. Something like, “The Do Not Bother Me Pledge Against Frivolous Surveys”.

Extended to broader polling, I just can’t see how the survey exhaustion on a national scale doesn’t bleed into polling data. The old joke asks whether you actually know anyone who was polled about their opinion. Taking that a step further, are you sure they’re actually being called, actually answering, or actually giving their opinion? How do we know it’s not AI’s at this point (Dead Internet theory)? And is this a growing problem, where the quality of surveys goes down as the collective effort Americans expend on surveys gets spread across countless industries and interests?

Maybe there’s no problem with this. Elections always look sketchy. But when people start throwing around the idea of election fraud, polling comes under scrutiny.

Hold onto the idea of shaky ground when it comes to surveys as I talk about that other article about the failure of economics to understand the current economy. Author James Mackintosh distills the subject pretty well: “Government spending matters, monetary policy matters and there is a natural link between growth, unemployment and inflation. None of the three proved reliable in the past four years, yet each seems self-evidently true, baked into the very structure of the economy. It is, in a word, confusing.” The most business-friendly paper in America just put out an editorial saying that fundamental economic ideas don’t apply today.

Janet Yellen spoke about this in 2016, in one of my favorite Fed speeches of all time (everyone should have a favorite Fed speech). As Chair of the Federal Reserve, she gave an arcane talk for a conference titled “"The Elusive 'Great' Recovery: Causes and Implications for Future Business Cycle Dynamics", which was the subject of the 60th annual economic conference sponsored by the Federal Reserve Bank of Boston.

In it, Yellen said that “Extreme economic events have often challenged existing views of how the economy works and exposed shortcomings in the collective knowledge of economists.” In English, she is saying that extreme economic problems tend to make existing economic theory look like steaming hot garbage. In 2016, she was talking about the 2008 financial crisis as an extreme economic event. Between the time of her speech and today, we had another economic shock in Covid and subsequent inflation. So take her conclusions and double down on the conclusions if you think she’s onto something.

In the speech, she addresses several problems she thinks have become unglued from previous explanations since the financial crisis. I’ll simplify:

• Does aggregate supply have anything to do with aggregate demand?

• How do human differences (everything from household composition to demographics) influence the economy?

• How does the financial sector interact with the economy?

• What causes inflation?

If those sound like some really basic economic concepts, like the kind that you’d cover in Econ 101, I’m with you. Again, that was in 2016. Now we hear complaints in WSJ that the basic questions don’t explain current events.

The data underpinning all of economic research is starting to look cloudy, and what’s interesting is that the road to better economic research has started to look rugged. Not only because we have to revisit the basics, but also because survey fatigue seems to be striking economic research too. Last year, the Bureau of Labor Statistics, which puts together information on the labor force, issued a warning in the most anodyne language possible. As they put it, “Response rates to most federal surveys have been declining for many years.” They’re scrambling to figure out what to do, because a declining response rate means the data can’t portray reality. We rely on government data for a lot of different economic dynamics. And if the government data has started to smell bad, the other powerhouses of economic research like ADP must be having the same problem.

I wouldn’t respond to an economics survey either, when my gas pump already queried me about having a good time.

What this says to me is that at the same time that economic theory has started to get choppy, maybe because of the proliferation of crises, the quality data is in decline. Maybe crises like Covid and the Great Recession broken the basic tools of research, or shattered the basic assumptions.

It seems that we have arrived at the edge of a vast fog of war in basic social sciences.

And let’s say that the tools of economics, political science, and polling are broken. Is this a problem? Is flying blind so bad? It could be, especially when the stakes are so high, namely the material wellbeing and political climate for millions of Americans (and indirectly billions of people living under Pax Americana).

Then I have to wonder if it’s always been this way, and we don’t know much more than we did, say, 50 years ago. There’s nothing new under the sun. Did economists, academics, government officials and policymakers, journalists and the common man, ever really know what people were thinking? Was there a golden age of the survey, where we had a good balance of surveys that someone encountered in a day?

I’m not sure. We may have just done a better job as a society of stringing together a common narrative. Or, that narrative may have only been laid on top of the historical record after the fact as we tried to understand the past. Maybe things have always been this messy.

If you were looking for a definitive explanation of where we go from here, I don’t have one. But I do have some questions to ask you:

• Did you like this post?

• Did you really like this post?

• On a scale of 1-10 with 3 being great and 1 and 10 being opposite kinds of good, and 5 being good enough, please tell me how much you liked this post.