Tuesday 21 August 2012

Peer-Review Works (or Doesn't) Both Ways

Below is the abstract (in English and Chinese) for an article that I have previously mentioned on this blog ("Canon and Reason…").  I was reassured that it would be published several months ago --but after repeated delays, I lost faith in the journal that would have (supposedly) already published it by now.  That's right: I completed the laborious peer-review process, and then I myself ended up cancelling the article anyway.

This isn't the first time that this has happened to me.  It is the second time.  I hope it will be the last.

A handful of people might have noticed that I had an article listed as "forthcoming" for a period of about one year, because it had already been approved and accepted by the editors at the Journal of Contemporary Buddhism.

The eventual cancellation had nothing to do with the quality of the articles: in both cases the articles in question passed peer-review, and were accepted by the editors.  If you asked me when you'll be able to read either one of these articles in print, the answer is that I don't know.

Publishing specialized non-fiction of this kind isn't about money: there's no way that any of these publications could remunerate me for the number of hours of work that go into research of this kind.  Even if they could, they don't.  There's no money in the game, so what an author is hoping for is professionalism, some degree of mutual respect, and (hopefully) access to an audience of other specialists.

However, at some point (after inexplicable delays of some number of months, and some number of incoherent e-mails) the author may feel that it's better to walk away from the publication than to continue to pursue it (not knowing when the thing will appear, or if any of the promises made on the editorial side will be kept).

In the case of my article that was accepted by Contemporary Buddhism in the U.K., so much time had passed since the editors assured me that it was accepted for publication, that I started asking questions.  The answers that I received were incoherent at best (and insulting at worst).

In that case, I had received detailed comments from one of the two editors, showing that he had really read it in detail (and considered the subsequent revisions, etc.) and, for that reason, I had taken the approval of the article seriously.  It became evident (many, many months later) that the other editor had not read the article (and perhaps never did).  In the antics that ensued (in which, basically, my one question was, "All the revisions were approved, so when's the article going to press?") I eventually cancelled the article.  Again: I cancelled it myself, not the editors.

Several of the deadlines the editors had declared for themselves elapsed (with no explanation) and, as alluded to, the answers to my simple questions were raising more questions.

The process was painfully unprofessional: I would receive an isolated e-mail after they broke their own deadline saying, "we'll have another meeting in a week, and get back to you then", but then I would, in fact, hear nothing.  Months and months passed during this silent drama, and when I finally cancelled the article (to enter it into peer-review elsewhere) I never heard any reply to confirm the cancellation from the editors' side, either.

"Canon and Reason" was supposed to be published months ago, but now I have no idea when it will be published.  Likewise, "Mind and the Material Elements" should have been published in Contemporary Buddhism more than one year ago by now, and instead it has had to re-start the peer-review process (in the latter case, repeatedly).

I have worked as a professional editor, of both academic non-fiction (in book form) and of newsprint with daily deadlines (but lower standards of accuracy, etc.).  I have also been a peer-reviewer (i.e., reviewing some other author's article for some journal: you may be thinking this is impossible because I don't have a PhD, but it isn't).  I know what the editorial process is, and I know what the peer-review process is, and there is a huge difference between the two.

The ritual of peer-review is not nearly as thorough as a professional editor who is paid to conduct fact-checking.  In the humanities, peer-review isn't a mode of fact-checking at all.  It has an interesting history, originating as a uniquely British system of decentralized state censorship (read more about the history here) [alternative link added in 2024] and evolving through its interaction with the "hard sciences".  However well (or poorly) it may work for physics or chemistry, there's an entirely separate question of its effectiveness in the quagmire of cultural issues where religious, political and linguistic domains overlap.

…Mario Biagioli argues in “From Book Censorship to Academic Peer Review” that a deeper excavation of the genealogy of peer review suggests that its origins may, significantly, lie in seventeenth-century academic book publishing, and that peer review of journal articles formed a much later stage in the process’s development. Biagioli ties the establishment of editorial peer review to the royal license that was required for the legal sale of printed texts; this mode of state censorship, employed to prevent sedition or heresy, was delegated to the royal academies through the imprimatur granted them at the time of their founding. The Royal Society of London, for instance, took on that imprimatur by passing a resolution in December 1663, one year after its founding, which stated that “No book be printed by order of the council, which hath not been perused and considered by two of the council, who shall report, that such book contains nothing but what is suitable to the design and work of the society” (qtd. in Biagioli 21). The purpose of such review, as Biagioli emphasizes, is more related to censorship than to quality control… [op. cit. supra]

Peer-review didn't originate in Asia, and it certainly didn't originate as an instrument that was useful for (e.g.) cultural anthropology, comparative philology, or the study of contemporary politics and religion.  There are many mismatched expectations applied to the process of peer-review and the results are… not what Francis Bacon would have hoped for.

From direct experience, I can report that some some reviewers (in Buddhist Studies) think that their role is to prevent anything offensive to Buddhism being published (which is a fairly fundamental conflict of interest), and some think that their role is to prevent anything being said to diminish the memory of deceased European scholars (again, a conflict of interest, and an obstruction to ever admitting or addressing the errors of past scholarship… one of the favorite subjects of Francis Bacon, it so happens).  I've now received more than one peer-review stating that I should not be allowed to point out errors in a 100 year old Pali-English dictionary; ask yourself if a scholar of Ancient Greek would receive a similar criticism, for reporting an error in a 100 year old dictionary for translating Greek to English, or if a scholar of Christianity would be discouraged from discussing a similar problem of the definition of terms in Aramaic, Hebrew and so on.  "Buddhist Studies" exists at a very problematic intersection of religious attitudes, and may be especially ill-suited to the process of peer-review.

This is part of the scientism of the social sciences: we pretend that a reviewer can verify my critique of Cambodian Anthropology in the same way that a reviewer of chemistry can check the equations in a lab report.  It is difficult to pretend that any reviewer has the time or ability to check even the most dry, factual, and philological claims in my work --because, in fact, the more dry they are, the more time-consuming they would be to verify, and the more demanding of the special expertise of the reviewer.  At some level everyone knows that peer-review isn't equally effective for all of the sciences and humanities; if it were, we'd use the same method to gamble on racehorses.

This is all very much a digression for the purposes of this short article: all that I have to say here is that (to my own surprise) I have ordered one of my own articles to be cancelled, and for the second time.