In the game-theoretic world, the gunner never shoots: the other side looks at the options down the game tree, realizes that one action will lead to his or her getting shot, and doesn’t take that action. In Game Theory textbooks, cases never go to court: both sides calculate the risk-adjusted expected payoff from trial, and if it is positive for one hyperrational side, then it is negative for the other hyperrational side, and a settlement can be calculated based on that. In both cases, knowledge that an event could occur largely has the same effect as the event itself. Continue reading
Science, a peer reviewed journal, recently published an article lambasting the quality of peer review in many journals that are not Science. The article described itself as “the first global snapshot of peer review across the open-access scientific enterprise”, and found peer review in that context to be lacking.
As one who leans toward the theoretical and the methodological, I naturally wonder what is the model underlying the claim that “peer review across the open-access scientific enterprise” would be of low quality. My understanding is that “open-access” is defined to include any journal that does not charge subscription fees, but allows readers free access via the Web. So we need some sort of model that explains why the lack of reader fees would lead to a consistently lower quality of referee effort.
Generally speaking, the discussion about scientific peer review tends to be…lacking in scientific rigor. Those who have written on the matter, including some involved in open access journals, all seem to agree that a claim that open access would induce lower referee effort makes little sense. It’s basically impossible to write down into a real model.
So in this and the next column, I attempt to fill the gap and provide a theoretical framework for describing a one-paper peer review process. I get halfway: I stop short of the general-equilibrium model covering the entire publication market. I also don’t specify the cost functions that one would need to complete the model, because they wouldn’t make sense in a partial equilibrium model (i.e., there’s no point in a specific functional form for the cost function without other goods and a budget constraint).
Nonetheless, we already run into problems with this simple model. The central enigma is this: what incentive does a referee have to exert real effort in reviewing a paper?
After the break, I will give many more details of the game, but here are the headline implications of the partial model so far, which don’t yet address the central enigma:
- The top-tier journal does not necessarily have the best papers. This is because the lower-tier journals have papers that have gone through more extensive review.
- More reviews can raise reader confidence that a paper is good. However, the paper is published after only a handful of reviews. Stepping out of the game, situations where dozens or hundreds read the paper before publication would do much to diminish both false positives and false negatives in the publication decision.
- Readers are more likely to read journals that maintain a high standard.
- Readers are also more likely to read journals where the referee exerted a high level of effort in reviewing the papers, and can also read those papers with less effort. The problem of trusting a false paper is mitigated, because careful reviews produce fewer false positives. However, referee effort is not observable.
All of this is still under the assumption that referees have an incentive to put real effort into the review process, an assumption I’ll discuss further next time.
After the jump, the rest of this entry will go into more precise detail (~3,000 words) about the game and some of its implications. Continue reading
A genuine “big sky” shout-out to grad school friends Dave Parker and Erika Franklin Fowler, both PROMINENTLY featured in this week’s fascinating PBS Frontline “Big Sky, Big Money” examining “dark money” in Montana politics and in campaigns around the country in the wake of Citizens United (2010). Parker, a coauthor on research looking at congressional investigations, is a dedicated student of American politics. He’s driven countless hundreds of miles this year collecting information on campaign advertisements from local television stations, filling a vital gap in available information about what’s going on in American politics. And he is rewarded with a spot on PBS Frontline, the coolest show on TV’s nerdiest channel. Not bad!
For the past thirty years, students of American government have leaned hard on a metaphor contrasting “police patrol” and “fire alarm” oversight. It’s an interesting and useful idea, but basically unsupported by careful empirical work. My esteemed colleague David C.W. Parker (who blogs about Montana politics here) and I have looked at the partisan dimensions of congressional oversight in a couple academic articles – a 2009 article here published in Legislative Studies Quarterly and a forthcoming article in Political Research Quarterly. This summer we published a short essay, “Oversight: Overlooked or Unhinged?” in Extension of Remarks, the newsletter of the Legislative Studies Section of the American Political Science Association. It’s basically an effort to work through the critique of the “fire alarm” metaphor with an eye on current events. Did you miss it? Here it is again.
Harper’s runs a feature every so often that the bureauphile will seek to emulate in the coming months. Until then, please enjoy the most recent installment from Harper’s chronicling President Reagan’s relationship with the FBI. Continue reading
A lot of people use the terms analyst, investigator, or auditor interchangeably, however, each job is different. Continue reading
Apparently, Louis Brandeis, Associate Justice on the Supreme Court of the United States from 1916 to 1939 coined the phrase, “sunlight is said to be the best of disinfectants.” A 1913 Harper’s Weekly article, entitled “What Publicity Can Do,” is where you can find the original article. (Hat tip to the Sunlight Foundation.)
So is it true? No.
While sunshine is a very good disinfectant, but it takes longer to disinfect:
And it might depend on what you are disinfecting:
For close observers of American government, Congressional Research Service (CRS) reports are indispensable, but by longstanding policy CRS does not release its reports to the public. CRS’s client is Congress, the reasoning goes, and in order to offer independent and unbiased advice CRS speaks (more or less) only to Congress, a position it claims is rooted in the Constitution’s speech or debate clause. This argument might hold water if other Congressional agencies like GAO, which are also presumably hamstrung by this same clause, did not publicly release their reports. Continue reading
On the Media, an NPR show about, big shocker, the media, re-ran an episode about data last week. One article was about The Texas Tribune, a non-profit and non-partisan media organization, which compiles data from Texas. The data is posted on their site both in raw form and with analysis from The Texas Tribune. Lots of interesting information there.
For example, the annual salaries for Texas government workers can be found there now. Posting government employees salaries, as I ponder it more, however, seems akin to the Sweedish custom of posting everyone’s tax returns online. The practical effects of this policy are reviewed by an admittedly biased writer for The Telegraph in this article.
As they say though, sunshine makes the best disinfectant, so let there be light. (Is that even true? I will run that down next…)
Here is a link to the newly released National Security Letter templates used by the FBI. The ACLU used FOIA to get access to the template. The Wall Street Journal does an excellent job explaining these letters and why this template is of import.