Bryan Caplan links to a paper by Tim Groseclose and Jeff Milyo, "A Measure of Media Bias," that proposes to measure media bias by comparing how often media stories cite ostensibly "left wing" or "right wing" think tanks with how often those same sources are cited by members of Congress.
Leaving aside some of the survey's bizarre choices in methodology -- such as excluding from consideration any citation that labels a given sources as "conservative" or "liberal" -- its base assumption is just ridiculous. Whatever one thinks about the merits or practicality of the journalistic goal of objectivity, the bottom line is that this goal is most emphatically NOT seeking to replicate the same amount of bias that would be held by some "median" member of Congress, or even a median member of the reading public. The goal is to be "balanced."
Since Congress is not balanced (both houses are dominated by Republicans) then a media corps that met its own goal of perfect objectivity should be to the left of Congress. That would be what one ought expect. Similarly, a media displaying perfect objectivity in, say, Venezuela ought to be to the right of the Chavez government.
Nonetheless, as a measure of general media bias, even this deeply flawed experiment is preferable to the one Caplan would opt for instead:
While I love this paper, I want to propose a much more straightforward test of media bias: Simply by reading the title of the article, can you tell what the reader is supposed to think about the story? This is amusingly easy.
Amusingly easy? Yes. Indicative of a tendency worth pondering? Certainly. Relevant to the broader subject at hand? I think not.
What I think Caplan's test displays -- and it's a valid concern that I've long-shared -- is that headlines are more likely to display bias than the stories to which they are attached. There are a few notable reasons for this tendency, and they're all related:
1. Headlines aren't typically written by an article's author. They usually come from the copy desk, who may or may not consult with the writer on the headline's appropriateness. In my day, I've seen some pretty "out-there" heads attached to pieces I've written, to the extent that I've been left in some cases wondering if the editor even read the piece.
2. Headlines are constrained by space. Granted, so are stories, but in headline writing, the space is so small that conveying nuance becomes a virtual impossibility. Of course, even if it weren't, that would still leave the problem that...
3. Headlines are supposed to be punchy. They have to draw the reader into a piece. Perfect balance is a laudable goal, but if it's a choice between punchy but biased, or balanced and dull, the headline writer is going to opt for the latter every time.
Mind you, none of this excuses biased headlines, but it wouldn't generally be appropriate to make broad generalizations about a piece's slant just from reading the head. Although one potential experiment to judge the bias of various copy desks -- if that's something of tremendous importance to you -- could be to compare how the heads they give to syndicated and wire pieces differ from publication to publication.