A model of peer review

Science, a peer reviewed journal, recently published an article lambasting the quality of peer review in many journals that are not Science. The article described itself as “the first global snapshot of peer review across the open-access scientific enterprise”, and found peer review in that context to be lacking.

As one who leans toward the theoretical and the methodological, I naturally wonder what is the model underlying the claim that “peer review across the open-access scientific enterprise” would be of low quality. My understanding is that “open-access” is defined to include any journal that does not charge subscription fees, but allows readers free access via the Web. So we need some sort of model that explains why the lack of reader fees would lead to a consistently lower quality of referee effort.

Generally speaking, the discussion about scientific peer review tends to be…lacking in scientific rigor. Those who have written on the matter, including some involved in open access journals, all seem to agree that a claim that open access would induce lower referee effort makes little sense. It’s basically impossible to write down into a real model.

So in this and the next column, I attempt to fill the gap and provide a theoretical framework for describing a one-paper peer review process. I get halfway: I stop short of the general-equilibrium model covering the entire publication market. I also don’t specify the cost functions that one would need to complete the model, because they wouldn’t make sense in a partial equilibrium model (i.e., there’s no point in a specific functional form for the cost function without other goods and a budget constraint).

Nonetheless, we already run into problems with this simple model. The central enigma is this: what incentive does a referee have to exert real effort in reviewing a paper?

After the break, I will give many more details of the game, but here are the headline implications of the partial model so far, which don’t yet address the central enigma:

  • The top-tier journal does not necessarily have the best papers. This is because the lower-tier journals have papers that have gone through more extensive review.
  • More reviews can raise reader confidence that a paper is good. However, the paper is published after only a handful of reviews. Stepping out of the game, situations where dozens or hundreds read the paper before publication would do much to diminish both false positives and false negatives in the publication decision.
  • Readers are more likely to read journals that maintain a high standard.
  • Readers are also more likely to read journals where the referee exerted a high level of effort in reviewing the papers, and can also read those papers with less effort. The problem of trusting a false paper is mitigated, because careful reviews produce fewer false positives. However, referee effort is not observable.

All of this is still under the assumption that referees have an incentive to put real effort into the review process, an assumption I’ll discuss further next time.

After the jump, the rest of this entry will go into more precise detail (~3,000 words) about the game and some of its implications. Continue reading

Kung fu legislation

The kung fu master teaches the student to use the smallest possible movement that achieves the desired result. Flailing around looks dramatic but is a waste of energy. The kung fu master rarely speaks, but each statement follows the same principle, using as few words as possible.

There are multiple ways to apply the teachings of the master to legislation. To give a concrete example, the SHIELD Act (Saving High-Tech Innovators from Egregious Legal Disputes Act of 2013) is an attempt to make as small a change as possible to existing law.

Here’s the problem it hopes to surgically eliminate: a company with no engineers and a lot of lawyers obtains a patent (typically for software), finds somebody who (typically by simple coincidence) uses the method described in the patent, then sends a letter demanding a licensing fee. A full trial can cost millions just to defend against, and until the suit is resolved it must be disclosed as a risk and potential liability for the company, so a licensing fee of $10,000 to $100,000 to make the pain go away is worth it for most companies. Few people think this sort of behavior promotes the progress of science and useful arts.

This blog summary of research by Colleen Chien is a good round-up of numbers showing that this isn’t a trivial problem. Depending on how you define patent-assertion entities (PAEs, or patent trolls for short), they are 62% of U.S. patent litigation, and growing.

How can we change the law to solve this problem? One read of the kung fu principle is that the change should be carefully designed to affect trolls and only trolls: we like individual inventors who really came up with something original but who don’t have the capital to put out a full implementation, and want them to keep on patenting things.

More on the problem, from the above-linked blog roundup by Professor Chien:

Though the PAE share may surprise some, patented technologies like software are the building blocks of modern commerce. “Low-tech” industries like funeral homes, advertising agencies, and retailers like JC Penny […] are all taking steps to protect themselves from troll demands. Though historically a “tech” problem, in 2012 PAEs sued more non-tech companies than tech companies […].

Those of us who were paying attention weren’t at all surprised: Math You Can’t Use, a book I wrote in 2006, has a full chapter on how allowing the patenting of software, which courts in the mid-00s were increasingly willing to do, will create liability on all companies, not just those in the narrowly-defined software industry. I bring up my own writing not to say I told you so (because, frankly, it’s not a good feeling), but to point out that there’s a larger problem. What is patentable has grown far beyond expectations, to the point that every company with a web site has patent liability. How will a carefully-circumscribed law about trolls change this? IBM doesn’t fit the definition of PAE the SHIELD Act is written around, but it has patents on web-related elements and will still has the ability to send licensing demands to funeral homes and retailers if it so desires. Practicing the typical web-based software patent requires only an appropriately-configured web site, so can a PAE set up its own web site for a few hundred bucks and then claim to be a practicing entity? [Conversely, if you’re going to argue that setting up a web site is not a “substantial” investment, then how can any holder of a web-related patent ever claim to have made “substantial investment” and thus avoid the additional barriers put up by the SHIELD Act?]

When applied to modifying law, the principle of least action and the principle of fewest words actually contradict: the smallest change to law requires voluminous verbiage to pick out only those parties and activities that are disfavored, while a brief, simply worded change would have far-reaching implications.

A law clarifying that software and information processing without serious real-world effect is not patentable would be a short paragraph, but would have real effects on the sort of companies that have a lobbying presence in DC; the SHIELD Act is multiple pages of little details about how to define a non-practicing entity and what those entities are required to do when entering into litigation. Nobody is happy with how practicing entities are defined in the law, and you’ll find more than enough people (myself included) who think no definition exists that commits neither Type I nor Type II errors. Further, nobody said patent trolls are all idiots, and the smaller the range of prohibited action, the greater latitude they have to circumvent the law by inventing strategies that are still trolling but just outside the definitions. Perhaps we might see subsequent legislation two years from now that modifies the law just a little bit more to chase the trolls out of their new methods. Such kung fu surgery is why the tax code is an incoherent mess.

The Electronic Frontier Foundation is famously pro-software patent, but knows that patents are a thorn in the side of its constituents, who are very likely to oppose softpatents. They are thus always on the lookout for ways to propose small changes to patent law that still allow softpatents to flourish, such as being on record opposing software patents that fail non-subject-matter requirements like being obvious or badly specified. The SHIELD Act is thus exactly up their alley, and they support the SHIELD Act.

Summary paragraph: we like laws of few words and few loopholes, but the incentives consistently push for the production of verbose laws of minimal effect. The SHIELD Act is a fine example of such a law, and it has low odds of achieving even its stated effect.

open data day versus legal codes

Yesterday was International Open Data Day, wherein people from around the world volunteer their time to help make data more accessible. My big lesson from the event was that we still have a long way to go to making the legal code accessible by modern standards.


I was mostly self-interested and used it as an excuse to get my feet wet with GIS software. I worked with Andrew Salzberg on his proposal to encode and map the DC zoning laws into something useful. By the end of the day, we had a first draft of zoning-allowed floor-to-area ratio (FAR) and metro stops. It’s missing zoning overlays (like the one that limits density around Dupont Circle Metro), but it’s already pretty obvious that there is no real relationship between what sort of density (measured by FAR) the zoning code allows and where the Metro stops are.

The big lesson in putting this together is what a lousy job the authors of legal code do in producing anything with much logic to it. Here’s the original DC zoning summary that we had to parse down to make the map. Some zone designations, like C-1, give a FAR directly; others, like the HE-1 through HE-4 zone designations, list lot occupancy and the maximum number of floors. In short, it’s a document that evolved, with no serious concern for consistency or legibility. Continue reading

The US Code in Git format

Last time, I alluded to how one could, conceivably, think of the US Code as a series of revisions, in the style of a CompSci revision control systems like git.

A few days after posting that, I met up with Tom Lee, the Director of Sunlight Labs (for entirely random reasons unrelated to the subject at hand). He pointed me to the many projects that they have on GitHub in coauthorship with authors from govtrack.us and others.

So, here it is: the US Code as a git repository.

Why is this interesting to a non-geek? Have a look at a given section of a random passage, like 26 USC §106. Each revision shows the change from the previous revision, including deletions, additions, and modifications. Looking at the difference (the diff) between one revision and the previous one clearly shows when dates are pushed forward, when wording is tweaked, and where wholly new chunks of law are introduced and then modified over later revisions.

For fans of patent law (and a perhaps simpler example), here’s the effect of the 2011 patent reform. For example, in 123.txt, we see the invention of a new micro-entity category. In 145.txt, we see that people dissatisfied with a decision of the Patent Office’s internal review board (BPAI) used to be able to take civil action in the US District Court for DC; now they can take it up with the District Court for the Eastern District of Virginia.

For a nice visual aid and an example of this in use by a real live government, here’s a blog post on how the DC government handles changes in its Code.

Revision control: A fiefdom and a federation

This is a comparison of two web sites that are experiments in self-governance: Wikipedia and GitHub. As producers of content, both are immense successes, but the governance models (i.e., the rules and process by which content is written and kept) have grown to be very different. To a great extent, we can trace most of the differences in governance to a key difference in setup: one allows forking and the other doesn’t. Continue reading