Thursday, February 04, 2010

Plaintiffs’ Experts And Peer Review Don’t Mix

Just the other day we brought you news that the British medical journal Lancet issued a full retraction of an purported scientific article by a plaintiffs’ expert in autism litigation. The author, who was at the time also serving as a plaintiff-side expert, described parts of his research in a manner that “have been proven to be false.”

Something similar seems to be happening in the Accutane litigation. There’s a new opinion, Palazzolo v. Hoffman-La Roche Inc., No. A-3789-07T3, slip op. (N.J. Super. App. Div. Feb. 3, 2010), in which another plaintiff’s expert has gotten called out for essentially the same thing – publishing an article in a medical journal that failed to accurately describe what was actually done.

Be ready to hold your nose, here's what went down, and it ain't pretty:

In Palazzolo the plaintiff’s expert, James Bremner, was hired to offer an opinion that Accutane caused depression and suicide. The sticky situation began when, in the words of the Court, “Plaintiffs paid Bremner to undertake a further study. There is no dispute that the study was commissioned specifically for use in this litigation.” Slip op. at 3. That study – bought and paid for by plaintiffs’ counsel – was nevertheless published at J. Douglas Bremner, M.D., et. al., “Functional Brain Imaging Alterations in Acne Patients Treated With Isotretinoin,” 162 Am. J. Psychiatry 983 (May 2005). Slip op. at 4.  You can read the whole thing on line here.

According to the opinion, the study protocol (we use the term advisedly) injected some patients with “radioactive glucose” and then “sliced” their brains with something called a “PET scan” – “PET” being short for "positron emission tomography.” Slip op. at 3. However, there didn't seem to be much dispute that PET scans can’t diagnose depression, therefore, the court held a hearing. Id. at 4 (“the judge questioned how the PET scan study, which all parties agreed did not diagnose depression, nonetheless allowed Bremner to reach the conclusion that Accutane caused depression”).

That hearing was beginning of the end for Dr. Bremner’s study, because the court ordered him deposed, and cross-examination brought out what four judges (the trial judge and the unanimous three-judge panel) all agree happened:
  • “Bremner did not actually use the methodology he claimed to have used. Although his PET scan article was peer-reviewed, he admitted that he did not in fact follow the steps described in the article.” Slip op. at 10.
  • “[C]ontrary to representations made in the article, he did not get before-and-after. . .questionnaires from many of the subjects.” Id. at 10-11.
  • "Bremner also could not document much of the data on which his published results were based."  Id. at 11.
  • “[H]e admitted that some of the statistical analysis was inaccurate. For example. . .Bremner admitted that, for each study participant, comparing the activity in the orbital frontal cortex with the activity in the whole brain revealed no difference between the subjects who took Accutane and those who took antibiotics.” Id.
  • “[H]e testified. . .that the ‘absolute metabolic rates’ for the two groups was significantly different, and contended that was the key finding of the PET study. However, Bremner. . .could not produce the source data for that analysis because [it] was on an optical computer drive that could not be opened.” Id.
  • “[H]e admitted that some of the [data] he used in his calculations were inaccurate, [but] could not check the accuracy of the remaining numbers because the original data could not be retrieved.” Id. at 12.
Thus the court affirmed throwing the study out even under New Jersey’s relatively liberal standard for admissibility of expert evidence. “An expert's scientific peers cannot fairly judge the expert’s written work. . .if his article does not accurately represent either the underlying data or what the author did to produce his results.” Slip op. at 12. Four judges unanimously agree that “Bremner’s study was not soundly and reliably generated.” Id.
But just when we're getting ready to cheer the panel for calling out Dr. Bremner's faked article, we nearly swallow our tongues.  The trial court's order excluded Bremner altogether, probably (we haven't seen it) reasoning along the lines of the old legal maxim, "falsus in uno, falsus in omnibus."  State v. Wesler, 59 A.2d 834, 837 (N.J. 1948); see, e.g., Howard v. Fortuna, 2009 WL 2031060, at *7 (N.J. Super. A.D. July 15, 2009) (unpublished, affirming trial judge's decision on basis of "false in one, false in all" doctrine). 
The appellate court reversed the total exclusion and – despite all the findings about how the article had been faked – remanded at plaintiff's insistence to determine if, "without reference to the PET study," Dr. Bremner could nonetheless give a causation opinion.  Slip op. at 20.  Our own words fail us, so we rely upon Joseph Welch's retort to Sen. Joseph McCarthy:  "Have you no sense of decency, sir, at long last? Have you left no sense of decency?" 

Thus, even after holding that Dr. Bremner's main basis for his testimony was an "article [that] does not accurately represent either the underlying data or what the author did," the court was nevertheless willing to allow him to offer an opinion in the same matter.  Amazing.
On a more general level, we are, frankly, sick and tired of hearing the academic types and the popular press moan and wring their hands about the fact that our clients fund a lot of published studies in the medical literature. If they didn’t, there wouldn’t be very much literature, and the journals would have to print more junk science along the lines of the Lancet and the American Journal of Psychiatry articles.

Compared to that kind of litigation-inspired trash, the safeguards built into FDA-regulated clinical trials are ... well, there's no comparison, really.  A drug manufacturer’s studies have to meet rigorous FDA standards. These studies are subject to written, detailed protocols, those protocols have to be followed or there’s Hell to pay (and a great deal of money down the drain), every step of the way has to be documented, and after that, everything is submitted for review by independent FDA scientists who can require submission of any additional data that they want.

Those are a lot of reasonable guarantees of genuineness – to steal a phrase from hearsay evidence cases. They’re not perfect, of course, but FDA-regulated clinical trials are already subject to pretty stringent standards.

What’s really appalling is what happened twice this week – exposure of the kind of shenanigans that turn up all too often when studies are actually paid for by litigants for litigation purposes. Where's the outrage (other than here)?  We think that if academic journals don’t already ask authors whether they’ve been retained as experts in litigation concerning any drug/device/vaccine they are studying, they damn well should.

And mere disclosure's not enough.  The ordinary peer review process doesn't seem to be robust enough to ferret out litigation-driven academic fraud.  So we think that if the answer to the disclosure question is “yes,” more needs to be done.  If there’s a litigation expert seeking to publish about anything litigation related, we think academic journals – to safeguard their own integrity – should include at least one opposing-side litigation expert in the peer review process.

Note that we’re not limiting our proposal just to plaintiff-side experts, although that seems to be from where most of the “smoke” is now rising. That's because:  (1) we trust our experts more than we trust the other side’s, so we’re comfortable with increased scrutiny across the board, and (2) if we’ve got a bad apple with a rotten study, we’d like to know about that sooner rather than later.

If special scrutiny is not paid to studies being pushed by litigation experts, there are only going to be more embarrassments on the order of what we’ve seen this week. Junk science doesn’t stop being junk science just because some academic journal is bamboozled into publishing it.

8 comments:

Ted Heise said...

Good points about clinical studies supporting FDA approval processes. I would add that a subset of the data is usually inspected for many studies.

http://www.fda.gov/ICECI/EnforcementActions/DisqualifiedRestrictedAssuranceList/ucm132596.htm

Bill Childs said...

I address some of these same issues, in particular the issue of disclosure of compensation, in a piece a couple of years ago:

The Overlapping Magisteria of Law and Science: When Litigation and Science Collide, 85 NEB. L. REV. 643 (2007).

Alas, I don't think it's available online in any free service, but I'm happy to share reprints for those who send me an e-mail -- wchilds@law.wnec.edu

Evidence for Defense said...

we have been able to intensely research clinical studies, pool those studies and provide a meta-analysis in real time based on targeted litigation issues. Effective use of such data would have revealed such abuse early on in the litigation and challenged plaintiff's expert.

I am happy to discuss this litigation strategy.

rherschenfeld@doctorevidence.com

bmartinmd said...

Thanks for the information. A bit more is provided over at the Pathophilia blog.

Specifically the integrity of the peer-review process has been sustained merely by a small-type "correction" in a 2008 issue of the American Journal of Psychiatry.

Also it's not clear how much money Emory/Bremner received from an Accutane plaintiff for the PET study, but a USA Today article suggests a substantial chunk of $1 million.

hg said...

Was it "junk science" when Merck attempted to explain the results of VIGOR as reflecting the "cardioprotective" effects of naproxen?

Carelli said...

what about all the drug company studies that leave out the people who got worse or even died? drug companies falsify their research everyday to get their drugs through the FDA (hopelessly easy) and onto the market as fast as possible where millions of unsuspecting Americans can pay for the privilege of being “lab rats” in stage 4 post marketing studies.
when the body count gets so high that even the FDA begins to suspect cause and effect, then the drug might be withdrawn from the market but only after a replacement one has been introduced or the patent protection period is over.
more people were killed by VIOXX than died in Vietnam and yet i don’t recall seeing mass protests against the war being waged on Americans by drug companies.
of course Dr. Bremner should testify and his studies should be allowed as evidence as well!

Amy Philo said...

Oh my goodness well y'all are mighty sure of yourselves aren't you. After all everyone knows the world can't live without clear skin and it's worth it if Johnny kills himself because at least you can bury him without acne.

Dr. Bremner responded to all of this over on Pharmalot, here. http://www.pharmalot.com/2010/02/accutane-depression-and-expert-testimony/

shilpa said...

Effective use of such data would have revealed such abuse early on in the litigation and challenged plaintiff's expert.
I completely happy to share link this