Version classiqueVersion mobile

Read/Write Book

 | 
Marin Dacos

Édition électronique scientifique

The Saga of the Journal Comment

Janet Stemwedel

Texte intégral

  • 1 Steinn Sigurðsson, « How to Publish a Scientific Comment in 1 2 3 Easy Steps », Dynamics of Cats, 1 (...)
  • 2 Rick Trebino, How to Publish a Scientific Comment in 123 Easy Steps, [en ligne], consulté le 10 jan (...)

1Recently, Steinn brought our attention to some of the difficulties involved in getting a scientific journal to publish a “Comment” on a article1. He drew on a document (PDF) by Prof. Rick Trebino of the Georgia Institute of Technology School of Physics detailing (in 123 numbered steps) his own difficulties in advancing what is supposed to be an ongoing conversation between practicing scientists in the peer reviewed scientific literature2. Indeed, I think this chronology of exasperation raises some questions about just what interests journal editors are actually working towards, and about how as a result journals may be failing to play the role that the scientific community has expected them to play.

2If the journals aren’t playing this role, the scientific community may well need to find another way to get the job done.

3But I’m getting ahead of myself. First, let’s look at some key stretches of Trebino’s timeline:

41. Read a paper in the most prestigious journal in your field that “proves” that your entire life’s work is wrong.

52. Realize that the paper is completely wrong, its conclusions based entirely on several misconceptions. It also claims that an approach you showed to be fundamentally impossible is preferable to one that you pioneered in its place and that actually works. And among other errors, it also includes a serious miscalculation – a number wrong by a factor of about 1000 – a fact that’s obvious from a glance at the paper’s main figure.

63. Decide to write a Comment to correct these mistakes – the option conveniently provided by scientific journals precisely for such situations.

7Now, of course we might guess that the authors of the original paper are just as sure of their correctness as Trebino is certain that they have erred. And, as obvious as it is to Trebino that they have made significant errors in their paper, Trebino could be the one that’s mistaken. However, in the scientific discourse, there is supposed to be an ongoing back-and-forth between scientists, raising objections to assumptions, calculations, and interpretations, and responding to the objections. Publishing a paper is not supposed to bring that exchange to an end, but rather to bring it to a larger slice of the scientific community with something relevant to add to the exchange. In other words, if you read a published paper in your field and are convinced that there are significant problems with it, you are “supposedto” communicate those problems to the rest of the scientific community – including the authors of the paper you think has problems. Committed scientists are “supposed to” want to know if they have messed up their calculations or drawn their conclusions on the basis of bad assumptions. This kind of post-publication critique is an important factor in making sure the body of knowledge that a scientific community is working to build is well-tested and reliable – important quality control if the community of science is planning on using that knowledge or building further research upon it.

8Noting that, at least in theory, the authors of the original paper would want to know about the flaws in their paper (and perhaps recognizing that his own judgment that the paper was flawed might change in the light of additional details from the authors that did not appear in the paper), Trebino wrote to the authors privately to request some of the details that were not provided in the paper. He received no response.

9Is it surprising that the authors did not respond to a fellow scientist’s request for further information? Scientists are “supposed to” share information with each other, on the theory that they are engaged in the shared enterprise of building a body of reliable knowledge to be shared by the scientific community. Since what gives findings the status of knowledge is that they survive the scrutiny of other scientists, they are also “supposed to” provide the details that are needed for scientists to perform such scrutiny. And while there is surely a competitive element in the manner science is practiced, presumably scientists understand themselves to be involved in a quest for truth about the bits of the universe they study – meaning that a failure to help provide details that would be useful in pursuing a particular piece of truth may come across to others as obstruction.

10Surely, scientists who opt out of participating in the follow-up discussion privately don’t have firm ground from which to object if the follow-up discussion is then conducted publicly.

11Of course, one of the most effective ways to conduct that discussion in public is to harness the power of the journal that published the paper at the center of the discussion.

1215. Write a Comment, politely explaining the authors’ misconceptions and correcting their miscalculation, including illustrative figures, important equations, and simple explanations of perhaps how they got it wrong, so others won’t make the same mistake in the future.

1316. Submit your Comment.

1417. Wait two weeks.

1518. Receive a response from the journal, stating that your Comment is 2.39 pages long. Unfortunately, Comments can be no more than 1.00 pages long, so your Comment cannot be considered until it is shortened to less than 1.00 pages long.

1619. Take a look at the journal again, and note that the title, author list, author addresses, submission date, database codes, abstract, references, and other administrative text occupy about half a page, leaving only half a page for actual commenting in your Comment.

1720. Remove all unnecessary quantities such as figures, equations, and explanations. Also remove mention of some of the authors’ numerous errors, for which there is now no room in your Comment; the archival literature would simply have to be content with a few uncorrected falsehoods. Note that your Comment is now 0.90 pages.

1821. Resubmit your Comment.

1922. Wait two weeks.

2023. Receive a response from the journal, stating that your Comment is 1.07 pages long. Unfortunately, Comments can be no more than 1.00 pages long, so your Comment cannot be considered until it is shortened to less than 1.00 pages long.

21The journal’s insistence on this immovable length limit is galling in light of the fact that, in the self-same journal, Trebino found a number of published Comments that ran to three pages. It’s almost as if the official rules on Comment length were either not official rules, or were bendable if one could only come up with a magic word.

22After a wait of a few months:

2331. Receive the reviews of your Comment.

2432. Notice that Reviewer #3 likes your Comment, considers it important that the incorrect paper’s errors be corrected and recommends publication of your Comment as is.

2533. Notice that Reviewer #2 hates your Comment for taking issue with such a phenomenal paper, which finally debunked such terrible work as yours, and insists that your Comment not be published under any circumstances.

2634. Notice that Reviewer #1 doesn’t like it either, but considers that its short length may have prevented him from understanding it.

2735. Also receive the topical editor’s response, pointing out that no decision can be made at this time, but also kindly suggesting that you consider expanding your Comment to three pages and resubmitting it along with your responses to the reviews.

28Perhaps, then, allowing authors (whether of Comments or articles) “enough” space to lay out their assumptions, methods, data, and conclusions, and to explain what they’re saying in sufficient detail that someone else could actually “understand” what they are saying might be a good idea all around. Communication is hard enough without arbitrarily strict page limits.

29Might Trebino have felt a renewed sense of optimism as he revised his Comment and prepared his responses to the reviews? I’m guessing nothing that another few months of waiting couldn’t deplete:

3041. Receive the second set of reviews of your Comment.

3142. Notice that Reviewer #3 continues to like your Comment and continues to recommend its publication.

3243. Notice that Reviewer #2 continues to hate it for taking issue with such a phenomenal paper, which finally debunked such terrible work as yours, and again insists that your worthless Comment not be published.

3344. Note further that Reviewer #2 now adds that your Comment should under no circumstances be published until you obtain the important details from the authors that you confessed in your response to the reviewers you were not able to obtain and are not ever going to.

3445. Realize that Reviewer #2’s final criticism inevitably dooms your Comment to oblivion until such time as the authors provide you with the important details, your best estimate for which is never.

35Here, notice that the unwillingness of authors to participate in a private discussion of their published work is being used as a reason to obstruct a public discussion of their work. Seemingly, this gives authors the option of halting “any” productive discussion of their work once it has been published. This does not fit easily at all with the picture of the community of science as engaged in ongoing discussion and scrutiny, checking results and interpretations even after they have been published and sharing important information with the community. How is science supposed to be self-correcting if recalcitrant authors can take discussion of their results off the table at their whim?

36But, the plot thickens:

3747. And, in an absolutely stunning turn of events, note also that Reviewer #1 writes further that he has also somehow secretly obtained from the authors the important details they neglected to provide in their paper and refused to send to you. Even better, using them, he has actually checked the relevant calculation. And he finds that the authors are wrong, and you are correct.

3848. Realize that it is now no longer necessary to respond to the impossible criticism of Reviewer #2, as Reviewer #1 has kindly done this for you.

3949. Add a sentence to your Comment thanking Reviewer #1 for his heroic efforts in obtaining the authors’ important details and for confirming your calculations.

4050. Receive the editor’s decision that your Comment could perhaps now be published. Unfortunately, Comments can be no more than 1.00 pages long, so your Comment cannot be considered further until it is shortened to less than 1.00 pages long.

4151. Point out to the editor that most Comments in his journal are two to three pages long. Furthermore, it was the editor himself who suggested lengthening it to three pages in the first place. And Reviewer #1 strongly recommended leaving it that long.

42I don’t know about you, but at this point I need to pause for some deep, cleansing breaths.

43More editing for length ensues. More time passes.

4480. Receive a response from the senior editor that, while your Comment is now short enough and properly formatted, over the many modifications and shortenings that have occurred, its tone has become somewhat harsh. For example, a sentence that originally read, “The authors appear to have perhaps accidentally utilized an array size that was somewhat disproportionate for the corresponding and relevant waveform complexity,” has evolved into: “The authors are wrong.”

45Can you imagine that this continuing exchange with the editors might have an effect on the tone of a Comment?

4683. Wonder whether your Comment has finally been sent to the authors for their Reply, or instead was lost, trashed, or sent back to the reviewers for further review and possible rejection. ...

4790. Realize that you had stopped carefully reading the journal, and, as a result, had missed the “Erratum” published by the authors on the paper in question six months earlier, shortly after you submitted your short-lived three-page version of the Comment.

4891. Note that, in this “Erratum”, the authors actually admitted no errors and instead reported new – similarly incorrect – numbers, which they concluded “do not change any conclusions” in their original paper.

4992. Feel old, as you can remember the days when Errata involved correcting old errors and not introducing new ones.

5093. Note also that, in their “Erratum,” the authors have actually responded to some highly specific criticisms of their errors you mentioned in the three-page version of your Comment – criticisms that you had removed when shortening it to meet the journal’s strict 1.00-page limit. Criticisms the authors couldn’t possibly have known about in view of the journal’s strict confidentiality rules for submitted papers, unless this version of your Comment was somehow leaked to them...

51It would seem, then, that the original authors “did” want to be involved in the follow-up conversation about their published findings, sort of. They just preferred not to provide particular details requested by other scientists, or to answer the specific questions raised by those scientists. And someone involved on the editorial end of the journal – an editor, a reviewer, or someone else with access to the submitted but not-yet-published Comment – decided that it was OK to help the authors dodge participating in an actual discussion with give and take, one over which the authors did not have most of the control.

52More deep, cleansing breaths.

53108. Receive a phone call from the senior editor, who takes advantage of this opportunity. He has suddenly remembered that your Comment’s tone is a bit harsh. He is concerned that the authors, who appear to be highly motivated and quite crafty, will complain loudly and aggressively about the obviously preferential treatment your Comment is clearly receiving from the journal and make his life miserable. He objects to nearly every sentence in your Comment, in each case, insisting on a considerably longer sentence. He insists that you not say that the authors are “wrong” and suggests instead “perhaps mistaken.” He also insists on replacing the word “so” with the unforgivably long “therefore.”

54Some thoughts for the put-upon senior editor here: Loud and aggressive complaining happens, and working out ways to deal with it might be a useful life skill. Publishing the Comment and inviting the authors submit their own response to it would certainly diffuse worries about preferential treatment. And throttling serious communication between scientists seems like the opposite of what a scientific journal ought to be doing.

55119. Encounter a journal representative at a conference, who kindly mentions that the one-page version of your Comment was, in fact, sent to the authors for their Reply. And, after a series of delays, they have submitted it. But, unfortunately, it is extremely contentious and will be rejected unless toned down significantly. It’s as if, for some reason, they want it to be rejected.

56120. In preparation for the final phase of the Comment process, write to the editor asking if you will be able to see the Reply to your Comment and make minor modifications in view of it, as allowed by most journals.

57121. For once, obtain a quick response: “No”

58122. Finally receive notice from the editor that the authors’ official Reply to your Comment has been reviewed and processed. Unfortunately, it was not found suitable for publication and so was rejected. And because, for maximum reader enjoyment, it is the policy of this journal that a Comment cannot be published without a Reply, your Comment cannot be published. This decision is final.

59I’m going to need something stronger than deep, cleansing breaths here. While the authors of the original paper seem not to be behaving well here at all, what makes me pig-biting mad about this chronology is the journal policies which seem designed to give too much power to the authors of a paper to shut down follow-up discussion of their work if they so choose, whether by refusing to reply to a submitted Comment, or by submitting a reply which they have reason to believe will be judged “unpublishable”. Journal policies like these do not advance the ongoing scientific discussions by which science’s “self-correcting” mechanisms work. I’m inclined to think that if a journal has policies that get in the way of science working as it should, scientists would be well-advised to dump that journal in favor of others with less stupid policies.

60Indeed, Trebino expresses the central concern pretty calmly. Nearly everyone I’ve encountered who has written a Comment has found the system to be heavily biased against well intentioned correcting of errors – often serious ones – in the archival literature. I find this quite disturbing.

61The idea that the journal here seems to be missing is that they have a duty to their readers, not just to the authors whose papers they publish. That duty includes transmitting the (peer reviewed) concerns communicated to them about the papers they have published – whether or not the authors of those papers respond to these concerns in a civil manner, or at all. Indeed, if the authors’ response to a Comment on their paper were essentially. “You are a big poopyhead to question our work!” I think there might be a certain value in publishing that Reply. It would, at least, let the scientific community know about the authors’ best responses to the objections other scientists have raised.

62The scientific community, of course, will draw on multiple sources of data to make its own judgments about who the real poopyheads are.

Notes

Version originale disponible sur Adventures in Ethics and Science.
http://fr.readwriteweb.com/2009/06/29/a-la-une/contenus-tags-donnes-metadonnes-transfert-valeur/

1 Steinn Sigurðsson, « How to Publish a Scientific Comment in 1 2 3 Easy Steps », Dynamics of Cats, 17/08/09, http://scienceblogs.com/catdynamics/2009/08/how_to_publish_a_scientific_co.php, consulté le 10 janvier 2010.

2 Rick Trebino, How to Publish a Scientific Comment in 123 Easy Steps, [en ligne], consulté le 10 janvier 2010.

Auteur

Maître de conférences en philosophie à l’Université d’État de San José, Janet Stemwedel est rédactrice en chef du blog Adventures in Ethics and Science qui traite de la déontologie de la recherche scientifique, de l’interaction entre les scientifiques et le grand public et de la façon dont éthique et épistémologie sont liées.

CC-BY-NC-ND-4.0

Le texte seul est utilisable sous licence CC BY-NC-ND 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search