The Institute for American Values’ recently released report, My Daddy’s Name is Donor: A New Study of Young Adults Conceived Through Sperm Donation, has generated quite a heated discussion, especially in the blogosphere, about the subject of “peer review.” Many of the study’s critics, while all but ignoring the study’s findings, have insisted with remarkable self-righteousness that, since the research was not “peer reviewed,” the study as a whole is illegitimate and therefore unworthy of serious consideration. Â
Two questions may help to shed some light. The first is particular. Was My Daddy’s Name is Donor peer reviewed? The second is more philosophical. What is peer review, and what if anything is the relationship between peer review and scholarly excellence?  I’d like briefly to try to answer these questions.
Here is “Dr.” Jack Drescher, who didn’t think much of the newspaper column he read about My Daddy’s Name is Donor, in a letter to the editor of the New York Times:
But advocacy-group reports like this one are rarely subject to blind peer review, a minimum requirement for scientific objectivity. Without critical feedback from scientific peers, such reports usually support the pre-existing prejudices and assumptions of the authors or the organization financing the work. These “studies” offer little scientific understanding of the complex issues involved.
And here is Rachel Gurevich, similarly troubled by the media reports she was reading about My Daddy’s Name is Donor, writing in About.com:
This organization is certainly biased against assisted reproductive technologies, and their study has not been peer reviewed in an established professional journal.
Many other critics made basically the same point. But before we simply watch as these incantatory phrases about “peer review” congeal into unquestioned truisms, let’s re-wind the tape and try to think afresh about this subject for moment.
 One. Sometimes scholars create and seek to disseminate works of scholarship. There are diverse formats for such endeavors. Sometimes the format is an “article,” sometimes  a “book,” and other times a “report.” (There are other possibilities, such as “research brief.”)Â
 Now, there is simply no way for a would-be critic to know, based only on knowing that a particular work is called, say, a “report,” whether or not that work has been peer reviewed. Whether a work has been adequately peer reviewed depends only on whether and in what way it has in fact been reviewed by scholarly peers. Nor can the quality or even presence of peer review be accurately deduced simply on the basis of knowing whether the publishing entity is a think tank (e.g., the Brookings Institution), a journal (e.g. the Journal of Marriage and Family), or some other entity.
 So, if a would-be critic learns from the media that a “think tank” has issued a “report,” and on that basis alone starts shouting that that the work is bogus because it has not appeared in a “peer reviewed journal” – well, that person might as well be shouting “Biscuits!” or “Lug wrenches!” for all the light being shed on either the integrity of the report in question or the general topic of peer review.    Â
 Two. In many of the social sciences today, the tendency toward hyper-specialization  – or what the late University of Chicago president Robert M. Hutchins called “learning more and more about less and less” – means that growing numbers of scholars today have almost nothing to say to anyone outside of their increasingly narrow fields. Relatedly, many social scientists today lack even a rudimentary knowledge of the broad intellectual currents which have shaped their society and their disciplines. In particular, and notwithstanding frequently impressive displays of quantitative methodological sophistication, growing numbers of today’s social scientists seem to have reflected little if any on the epistemology and philosophy of the social sciences – types of knowledge which are essential for understanding how the disciplines can elevate and enrich one another.
 Three. The “peer review process” as it operates in university presses and social science journals today tends to be unserious and intellectually deadening.  For starters, unless you are already a certified member of the guild in question, your work is highly unlikely even to be considered for publication, no matter how good it is. Second, since almost all of the peer review occurring in these publications is intra-disciplinary – sociologists reviewing the work of sociologists, economists reviewing the work of economists, etc. – any genuinely broadly-based review and collaboration is, almost by definition, out the question from the start. Â
 Third, usually the process itself is both agonizingly drawn-out and remarkably anemic. From what I have observed, here’s how it usually works. You write and submit, say, an article. The editor, who usually enjoys a great measure of control, either rejects it or sends it out to three reviewers, chosen by the editor. Months go by. The editor then gets back all of several (in my experience between about two and four) written paragraphs from each of the reviewers. The editor shows you at least some of the substance of the reviews, and, if the editor invites you, you then make any revisions that you want or are willing to make, based on the reviews. This process can continue indefinitely. More months go by. Then the editor decides whether and when to publish the piece. Usually, that’s about it.Â
 Moreover, one of the core functions of this entire process is preventing the publication of material that the editors view as politically or philosophically unwanted. What’s largely being “peer reviewed,” at least in the fields I know about, and on the topics I work on, is political content. In most instances, this fact is blatant and overriding. It’s the elephant in the room, and nearly everyone I know who has ever been involved in this process knows it. It is a great tragedy, not least because it frequently turns the entire process into something very close to a farce. Â
 Four. The Institute for American Values began its work in 1989, led by a number of distinguished scholars and writers, including Don S. Browning of the University of Chicago; Jean Bethke Elshtain (then) of Vanderbilt University; Mary Ann Glendon of Harvard Law School; Norval D. Glenn of the University of Texas; David Popenoe of Rutgers University; Dr. Lee Salk; Dr. Judith S. Wallerstein; Dr. Barbara Dafoe Whitehead, and others.Â
 One of our main goals in founding the Institute was – and remains – to develop a new and richer model of scholarly collaboration. We use the terms “collaborative research” and “sustained interdisciplinary deliberation” to describe this model. From the start, we have taken the challenge of working together as a collaborative very seriously, including when it comes to meeting the highest standards of scholarly integrity and conducting what is typically (by others) called “peer review.”
 Our reports are almost always collaboratively produced and they typically go through many rounds of careful review by scholarly peers. In the case of My Daddy’s Name is Donor, for example, there are three co-investigators, one of whom is Professor Norval Glenn of the University Texas, one of the nation’s most respected family scholars. Twenty-three other persons, most of them tenured faculty at U.S. universities, gave of their time to read drafts, provide critical reactions and helpful advice, and in other ways provide a supportive and challenging interdisciplinary community of “peer review” for this piece of work. Are you wondering who they are? Their names are listed up front, on page ii, of the report itself – no one has to guess or take our word for it when it comes to who exactly are the “peers” that “review” our reports. These groups of collaborators are typically distinguished, rigorous thinkers from across the human sciences.Â
 This is the way that we have worked together since 1989. We didn’t just stumble into these procedures, or put them together as an after-thought, or as a kind of window-dressing, and we certainly weren’t trying to imitate what we saw around us.  It’s an honest, careful, good way to work together – a better way than one would find almost anywhere else, in my view, and also in the view of numerous others who have helped to develop and carry out this model. Â
 Five.  Finally, of course, the proof is in the pudding, not the container that the pudding comes in. One can shout “peer reviewed!” or “not peer reviewed!” until Judgment Day, but what really matters is, first, whether the work is any good and, second, whether the work is relevant to anything important. On both of these criteria, I am more than happy to let our record of accomplishment speak for itself.
Categories: General
It’s an honest, careful, good way to work together – a better way than one would find almost anywhere else, in my view, and also in the view of numerous others who have helped to develop and carry out this model.
But, like the Academy, it can also result in “group think” and a lack of critical analysis. The criticisms you make of the academy and peer review are quite fair, but they aren’t ameliorated by bringing together scholars who all essentially agree on a specific policy agenda and then “critique” a study conducted by people with whom they already agree.
While it’s a true their critiques may highlight questions about the research, but how much real disagreement is there among your handpicked groups of scholars?
That’s the problem with interest group-conducted research, as well as academic research. I agree that saying the research isn’t peer reviewed is not the most potent argument, although it is a characteristic of reliable research. But saying your research was vetted by people who agree with you–and may even receive remuneration from you–doesn’t really serve as a a gold-standard approach to research either.
The concern that two of the investigators were already out-spoken advocates for a policy positions–and have almost no experience as researchers–still raises red-flags about the research. Add to that the research exactly matched the argument they had been making before the research was even conducted. Toss in the fact that third investigator is the “resident’ academic who puts his name on all IAV research and it does raise some eyebrows. A little “peer review” could have answered some concerns and I’m not sure handpicked scholars already associated with the organization paying for the research can provide the cleansing appearance peer review would have.
I would add that the purpose of peer reviewing has historically varied a great deal by field. In the Humanities, and to a lesser extent the Social Sciences, the purpose of peer review tends to be geared toward saying no. There were, until the explosion of cheap publishing methods, relatively few distinguished journals or presses and a many scholars who wanted to publish in them. So the peer reviewing process was geared to winnowing out all but those works deemed very significant.
In contrast, scientists were disposed toward accepting papers that were at least plausible because they thought the real danger was overlooking something important. It would be deeply embarrassing to turn down a paper that might later lead to a Nobel Prize, so the inclination was to accept papers and subject them to verification after publication. Because the sciences subsidized publication (either through university funds or through funds included in research grants), the publication of scientific papers was “easier” in the sense that journals did not have to pay the cost of publishing them (as did Humanities journals, which generally did not require subvention, and often paid modest honoraria).
These differences have generally smoothed out in the last two decades as there has been a multiplication of Humanities journals (and a decline in the number of publishing humanities scholars), but historically it is true that in the Humanities the tendency of peer reviewers is to look for a reason to say no, while in the sciences the tendency of peer reviewers is to look for a reason to say yes. I am not sure what the inclination is in the social sciences, though it probably varies according to field.
Shorter. I’d be wary of research conducted by–and paid for by– a drug company that was reviewed only by drug company “advisors” where two of the researchers were from the marketing department and where the drug company refused to let anyone else review their research until the drug was already on the market.
Thank you David, this discussion was needed.
Peter, you are correct in that “it can also result in “group think””, my emphasis added. David’s item #3 almost assuredly does result in groupthink, as we’ve seen time and time and time again.
Trust the experts they tell us… they know what’s best for us.
Yep, and almost without fail, 15 years later all the experts are proven to be wrong. The days of blind trust are over.
Topic for another thread, but did anyone else see the article today that showed that 2/3rds of adult lesbians had actually changed sexual orientations over 10 years? Things that make you go hmmm…
It is simply not true that our work is reviewed only by people who already agree.
Also, the charge of group think only works when you can show specific cases or patterns of group think. Simply making the charge, whenever you don’t like a result, is not sufficient. We are very attentive to instances of bias — but only when a specific, legitimate, real example of bias can be cited, not simply when people hold their nose and say, over and over, “peer review,” as if that settles the whole matter.
During the Nixon era, the phrase “non-denial denial” became popular. People would say, “Nixon did the following bad thing.” And Nixon would reply, “How dare my enemies make those baseless false charges.” That’s a non-denial denial, because, while purporting to deny wrong-doing, Nixon was in fact not denying any particular thing. And that was the whole point; that’s what kept the circus going. I feel like that’s what’s going on with most of this criticism. If bias is your worry, show me a non-trivial, substantive example of what you mean, and let’s see if it’s real, or not.
Peter:
1- I’ve been co-investigator of two significant studies prior to this one. I am not akin to a hack in the “marketing department.”
2- Norval Glenn does not “put his name” on studies which he investigates or co-investigates. He works. Hard.
3 – There is a heck of a lot of disagreement among that Commission, and among all our scholars. By far the hardest part of my job is hammering out consensus among the very smart people we bring together.
I’ve been co-investigator of two significant studies prior to this one. I am not akin to a hack in the “marketing department.”
Both sponsored by your employer? What is your academic training in conducting and evaluating social science research? What is the academic training in conducting and evaluating social science research of Karen Clark?
There are the questions people are bound to ask when assessing the research. I’m sorry that it is insulting, but if you aren’t going to have research reviewed outside the confines of your organization, you need to be willing to withstand some questioning about the legitimacy of the research and those involved in the research.
Is your research biased? I don’t know. But one of the ways we evaluate the bias of research is looking at (a) who pays for it (b) who conducts it and (c) what kind of safeguards exist to avoid bias or flawed research. All we have is our word that the experts associated with your organization have chimed in and critiqued it. That’s not terribly reassuring.
Is it a Catch 22 and unfair? Absolutely. But that’s the price of running an ideological think tank where you take money to conduct research without traditional safeguards about the efficacy of the research. It’s why David was subjected to a Daubert challenge as an expert on social science research. People have questions and concerns.
It is simply not true that our work is reviewed only by people who already agree.
How many of the reviewers are not participants in your organization or linked to your organization? Have any of the reviewers ever published research that run counter to or contradict your organization’s policy interests?
David and Elizabeth (I’d probably send this privately if I had your current email addresses), many of us do really appreciate the hard work that you do to bring important facts to light, even when it seems to some of us that you ignore the forest for the trees. But a forest is nothing if not trees…
I for one may be too much like some of my favorite adversaries here, and more focused on the grand Principle Of The Thing, rather than the individual cases and their idiosyncrasies… for instance I can certainly see the value of giving an abandoned or neglected child a mother and father of his own, or even sympathize with the husband and wife whose dreams of a family are shattered when one of them is medically diagnosed with a fertility problem… or with both of their children, who grow up asking WHY? and WHO?
While on the other hand we have people who insist that Marriage isn’t about creating a family… that procreation has nothing to do with family at all… that old fashioned ideas about “Motherhood” or “Fatherhood” pale in comparison to ultra-modern ideas about “sexual orientation”. Forgive me if I have no sympathy for such ideas… it’s the Principle Of The Thing: We are all the result of the union of exactly one man and one woman. No more, no less, and regardless of whether there was any “love” involved at the time.
But yes, in addressing the health of the forest, somebody actually has to go in and do the mundane job of counting trees. Thank you for doing the work that you do.
All the points that Peter has made are right on target. You seem to want academic respectability but not to have to do the hard work and institute the safeguards necessary to achieve that respectability. I find it strange that you think critics should simply accept your work without acknowledging (or noting) your biases or the fact that your funding comes from particular sources, etc.
If the work is excellent, that will ultimately surface. But I don’t see how you can fault others from a certain level of suspicion. One reason one publishes in peer-reviewed journals or in scholarly presses is that certain assumptions come with those kind of publications.
If you choose not to publish in those venues, then you must accept the suspicions attendant upon that choice.
We now live in a very divided society, with sharp ideological differences. It seems to me if one participation in the culture wars, one has to resign oneself to the consequences of that participation.
I should add, of course, that I have not prejudged your work on donor conception. I am quite prepared to think that it is the kind of excellent research that it seems to be. And if it is, I suspect that it will be acknowledged as such even by those suspicious of your biases.
If you published a report that purported to say that same-sex marriages are somehow inferior, I would simply say, “consider the source” and not take it seriously because I know that that would be the kind of self-fulfilling “research” one might expect given the public positions you have taken on that question.
Peter: My work, including publications and funding, here:
http://familyscholars.org/assets/CV-Elizabeth-Marquardt.pdf
On the note of peer reviewed studies, it should be known that I’m all for them.
But where a reasonable person accepts the rigor taken as a good faith measure, I feel Jay is making too much an item in and of itself. Like the arguments against Faraday by the “experts” of his day, if taken as an end in and of themselves can entirely mislead a similarly dimmed set of naive or entrenched intellectuals.
How to solve the quandry? Instead of complaining about peer review, better to be the peer reviewer.
One should not mistake the call for rigor, easily made by any brute in mimicry of studious understanding, for real intellect. Until such provide real qualms to consider from these studies, I will think their words indistinguishable from such noise.
Peer reviews generally raise more questions than they answer and are always subject to the biases of the reviewer. I do think that releasing the report has the same effect as a peer review in that many more peers are taking a long look at it. If lack of peer review is the best the readers can come up with in terms of criticism, I’d be pretty happy if I authored the report. This report is really a descriptive survey of how donor conceived people feel about their experience of being donor conceived. How does one refute another’s testimony?
I also agree that politics plays an enormous role in how peer reviewers critique reports/studies. Trying separate the “wheat” from the “chaff” is still pretty much in the eyes of whoever reads it.
Jay, the point is not that some people are suspicious of non-peer-reviewed work. The point is that these critics themselves have pre-existing biases and use the fact of non-peer-review to dismiss this study out of hand and condemn it to the public. That itself is inappropriate behaviour.
If these critics have allegations of substance to make, let’s hear them and address them. I suspect any such allegations will apply equally to peer-reviewed works as they do to this.
Tom, I grant that everyone is biased about something, but as Peter has pointed out, academia has done a pretty good job of building in checks and balances, while ideological think-tanks have no incentive to do so: indeed, they usually construct litmus tests that prevent the participation of people with competing ideologies.
Still, I think that the notion that scholarship is somehow “disinterested” is naive. Most scholars, after immersing themselves in research, reach conclusions that they want to present and often advocate.
The problem with ideologically-driven research is that the conclusions are reached before the research, and the purpose of the research is simply to buttress the conclusions.
Still, I agree that one should not prejudge a work simply because it is produced by ideologically-driven research. The proof, as David Blankenhorn observed, is in the pudding. At the same time, however, I hope that critics who do not share the Institute’s ideology will subject the research to rigorous testing and evaluation. Such testing might verify the conclusions and thereby earn the book wider respect.
Jay I entirely agree with you. My point is supplementary: it is that those who are crying “peer review” are *themselves* ideologically motivated.
18 comments in and counting, and a LOT of high-falutin words under the bridge, and I am still waiting for someone to point out to me a substantive, non-trivial example of the bias that is being so loudly alleged. On Lawn, above, got it right: If you are so worried about no peer review, try doing some peer review! Otherwise, it’s mainly just self-indulgent bloviation — and bloviation that is itself an almost sure sign of bias, since it suggests that no actual evidence is needed to justify making an accusation of bias.
Tom, I would also add that some academic journals and presses are themselves ideologically driven. Journals that publish only postmodern criticism, for example, or that are devoted to a particular brand of philosophy. Many journals and university presses strive to create a “brand name,” usually by virtue of the subjects that they are interested in, but sometimes also by virtue of their approaches. If potential contributors depart from too far from the “party line,” their work is not likely to be accepted for publication.
I would certainly read a book published by Brigham Young University Press or Baylor University Press or University of Notre Dame Press quite differently from a book published by, say, the University of Chicago Press or Yale University Press (widely considered the best academic presses). I would assume that the books published by Chicago or Yale are more rigorous academically, and less constrained by ideological restrictions than the books published by BYU, Baylor, or Notre Dame even if all five presses practice peer review.
Re David’s comment, a reminder for those just joining the discussion, the full 140 page report, “My Daddy’s Name is Donor,” is available as a free PDF at FamilyScholars.org. Table 1 reports all the questions we asked and the responses. (Hmm, when was the last time anybody saw a researcher do *that*?) Ample tables and figures follow. A lengthy discussion of methodology and limitations is found at the end.
Review it.
Jay, I’ve not experienced the different approaches of different publishers myself, but I understand what you are saying.
This case is indeed even simpler however, because it doesn’t claim to be a deep academic study with lots of complicated technical terminology and background requirements. It’s simply asking some people some questions, and finding out that a lot of donor-conceived people are hurting. What’s hard to understand about that?
Peter asks: “Have any of the reviewers ever published research that run counter to or contradict your organization’s policy interests?”
The answer is, yes. Here’s an example: Not long ago we published a literature review on African American marriage. One of the findings was that marriage for Black U.S. women seems to be correlated to higher rates of depression and mental distress.
Now, we are widely viewed, generally accurately, as being ideologically “pro-marriage.” And it would have been the easiest thing in the world to omit or downplay, in our report, that particular finding about Black women and depression, since it not certainly not a “pro-marriage” finding. Things like that are done every day in the academy, not only despite but usually because of so-called “peer review”; I have personally witnessed this more times than I care to recall. In addition, we KNEW that if we published this particular finding, our critics would jump all over it, shouting with glee and doing war-victory dances in the media.
Which is exactly what happened. An article in the Washington Post, a piece in the Atlantic Monthly, etc. — all saying that new data show that marriage is bad for Black women. (Which was NOT the study’s main or overall finding, but never mind.)
But the point is, we published the research, untampered with. The reason why is that we try, in our frail way, to have a little integrity, and we take our work seriously.
So, Peter, there is your answer.
P.S. If we had wanted to massage the data from our survey of donor-conceived persons to suit our (alleged) ideological agenda, it would have been easy enough to do. For example: Omit or downplay the finding that a significant number of donor-conceived persons are themselves donors. For example: Omit, or try to explain away, the findings suggesting that d/c persons have a pretty strong libertarian streak when it comes to this cluster of issues. This would have been very easy to do — as I say, it’s done ALL THE TIME — especially if we had chosen not to include, in the report itself, every single piece of data from the survey, so that readers could see for themselves the raw evidence, as it were, and then judge for themselves whether the interpretative portions of the report are reasonable or not reasonable.
Oh, but pardon me — now I’m getting into the actual substance of the report.
it’s mainly just self-indulgent bloviation — and bloviation that is itself an almost sure sign of bias, since it suggests that no actual evidence is needed to justify making an accusation of bias.
Talk about bloviation. All the handwaving and “how dare you” and “I’m insulted” doesn’t change the perception of bias. And it’s the perception of bias that underscores the need for safeguards. Ethics is about perception as much as reality. If people perceive there is an ethical problem with research, that’s as important as there actually being biased.
The fact that the researchers were outspoken advocates/activists and have no serious academic training in conducting and evaluating social science research taints the research, regardless of whether actual bias exists in the data which can be cooked in a variety of ways.
Imagine the shoe was on the other foot and a IVF organization paid for and conducted research that was done by outspoken activists for increased IVF and 2/3 of the researchers had no serious academic background in research. Wouldn’t you view that research as tainted?
Peter:
I answered your question in some detail, and your response was … entirely non-responsive. It’s like I never said it.
Norval Glenn is not an untrained person, nor is he an advocate. The fact that you would say that suggests to me that you have not even read the report you are attacking.
I am still waiting for you to raise an actual, non-trivial, substantive concern about the research you are attacking. Seems like I’ll be waiting a long time!
I am still waiting for you to raise an actual, non-trivial, substantive concern about the research you are attacking.
I’ve raised the concern and you are going LALALALALALALALA. There’s a perception of bias that taints the research. Norval Glenn is your go-to academic who conducts most of your research. He’s like the paid researcher for Schering Plough. No one is questioning his credentials, beyond the fact that he is on the payroll, so to speak.
The concern is that there are three co-investigators, two of whom appear to have no training in conducting or evaluating social science research and were both activists and advocates on the narrow topic they were researching.
I encourage more review and I, like you, would like to see some non-IAV connected researchers to do similar work or try to replicate the research conducted by IAV. Hopefully those researchers will have put in place safeguards which eliminate the perception of bias that you seem so unconcerned about and are trying to wave away by saying, “I’m offended.”
And to be “responsive,” it is reassuring that in the anecdote you describe that you released a piece of research that ran counter to the IAV ideology. That speaks well of IAV.
Norval Glenn is not our our payroll. Nor does he conduct most of our research. This kind of exchange is pointless.
Peter: click “publications” at the top of this page to see 20 years worth of publications from the Institute on marriage and family topics (be sure to click, marriage, mothers, fathers, civil society, and the future of parenthood). Norval Glenn is one among many, many dozens of scholars, many from leading institutions, who are investigators, authors, or signatories to these books, reports, statements, briefs, and fact sheets.
It’s revealing though, that what they mean by “peers” is someone vetted by their academic and professional boards, which have already forced out anyone isn’t on board with destroying patriarchal man-necessary marriage and society. If anything, “peer-reviewed” simply means “biased” now, I wouldn’t trust any study that was “peer-reviewed” by biased self-selected discriminatory agenda-driven judges.