As I write this column, our nation’s economy is looking pretty shaky. In early January, the labor market report showed a significant uptick in the unemployment rate. This led worried investors to sell stocks, and the Federal Reserve lowered the interest rate dramatically in response to the sudden fall in the stock market. Or at least that’s the version of events we’re being fed by most of the press.
Do investors overreact to job numbers? There is a lot of debate about the accuracy of job and unemployment statistics. Clearly they are valuable, but there are all kinds of problems with the way the labor surveys are conducted and the resulting data analyzed.
There may not be a better way to collect the data, so despite their flaws surveys may provide the best information on the labor market that we can get.
The real problem arises when investors—sort of like you and me but a lot richer—look at these data and fail to keep in mind all its strengths and weaknesses. There is a risk people will focus on a single number and overestimate its precision. This has been called the “salience bias.” Writing in The New Yorker, James Surowiecki says this salience bias can lead to “a hard-to-break feedback loop: The fact that traders act as if the jobs report were definitive makes it so. A little information can be a dangerous thing.”1
I discuss hospitalist survey data with people all the time. I’m struck by how often they seem misled by salience bias, among other things. With SHM’s release this month of its latest biannual survey of hospitalist productivity and compensation, now seems like a good time to discuss the strengths and weaknesses in the data—and cautions when interpreting it.
Understand the strengths and limitations of the survey. SHM’s “Bi-Annual Survey on the State of the Hospital Medicine Movement” is a self-reporting survey in which each practice leader (or his/her designee) completes the survey. The responses aren’t verified or audited, so some respondents might submit shoddy data. Perhaps a busy group leader might complete the survey from memory and estimate things like each doctor’s production of work-only relative value units (wRVUs). When I’ve looked at the raw data, I’ve wondered if some respondents are trying to “spin” their numbers higher or lower for a variety of reasons (e.g., to look unusually good or show how hard their doctors can work). And there may be a response bias: Those who think their practice is atypical might not respond to the survey.
This year, SHM worked to “scrub” the data. Outlier metrics were established for each question, and SHM staff followed up with the respondent to ensure he/she understood the question and provided accurate data. In fact, I completed the survey for the group I’m part of and got a call from a survey staffer questioning the productivity I reported for some members of the group (our nocturnists have lower wRVU productivity than others in the group—that is one reason they’re willing to work at night).
Remember that data are historical and should be “aged” to the time period you’re using it. The data in the 2008 SHM survey were collected from October through December.
Review the original questions asked in the survey. To make sense of the survey responses, you will need to clearly understand the question asked. Review the survey instrument and form your own conclusions about ways the questions were posed that might influence the responses. And don’t assume you understand what a particular term means—verify it by looking at the survey instrument. For example, I encounter a wide variety of opinions regarding what constitutes base salary, incentive pay, productivity compensation, bonus, and total compensation. The survey instrument spells these things out clearly.