Misleading medical tests

Topics with information and discussion about published studies related to Lyme disease and other tick-borne diseases.
Lorima
Posts: 914
Joined: Mon 29 Oct 2007 20:47

Misleading medical tests

Post by Lorima » Sun 6 Jan 2013 21:24

Looking at the history of the two-tier test, one can't help wondering if there are other diagnostic tests that have as poor justification in evidence, and are hurting as many patients. 

I did some searches in the medical literature, and was a bit surprised to find that this sort of thing is a known, and fairly big, problem. The literature trail starts back at the beginning of PubMed, where the articles are scanned in from paper copies, so I have to post images to give you text, and keeps right on going up to the present, despite all the attempts to improve things in the interim. 

Image 
CMAJ. 1986 March 15; 134(6): 587–594.
PMCID: PMC1490902
A framework for clinical evaluation of diagnostic technologies.

G H Guyatt, P X Tugwell, D H Feeny, R B Haynes, and M Drummond

Most new diagnostic technologies have not been adequately assessed to determine whether their application improves health. Comprehensive evaluation of diagnostic technologies includes establishing technologic capability and determining the range of possible uses, diagnostic accuracy, impact on the health care provider, therapeutic impact and impact on patient outcome. Guidelines to determine whether each of these criteria have been met adequately are presented. Diagnostic technologies should be disseminated only if they are less expensive, produce fewer untoward effects and are at least as accurate as existing methods, if they eliminate the need for other investigations without loss of accuracy, or if they lead to institution of effective therapy. Establishing patient benefit often requires a randomized controlled trial in which patients receive the new test or an alternative diagnostic strategy. Other study designs are logistically less difficult but may not provide accurate assessment of benefit. Rigorous assessment of diagnostic technologies is needed for efficient use of health care resources.
Here are some highly cited papers, which indicates that the problem is well-known, and a serious source of concern among mainstream academics. (note that these citation counts are just for PMC articles, which is a minority subset of all PubMed articles.). It is not just a fringe group of idealistic reformers who are concerned, though I suspect it is a rather small minority, among physicians at large. 
Jaeschke R, Guyatt G, Sackett DL. Users’ guide to the medical literature. III. How to use an article about a diagnostic test. B. What are the results and will they help me in caring for my patients? JAMA. 1994;271:703–707. doi: 10.1001/jama.1994.03510330081039. [PubMed] [Cross Ref]
Cited by over 100 PubMed Central articles

 R, Guyatt G, Sackett DL. Users’ guides to the medical literature. III. How to use an article about a diagnostic test A. Are the results of the study valid? JAMA. 1994;271:389–391. doi: 10.1001/jama.1994.03510290071040. [PubMed] [Cross Ref]
Cited by 86 PubMed Central articles

Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, Bossuyt PM. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282:1061–1066. doi: 10.1001/jama.282.11.1061. [PubMed] [Cross Ref]
Cited by over 100 PubMed Central articles
The STARD Statement for Reporting Studies of Diagnostic Accuracy: Explanation and Elaboration
Patrick M. Bossuyt1,a, Johannes B. Reitsma1, David E. Bruns2,3, Constantine A. Gatsonis4, Paul P. Glasziou5, Les M. Irwig6, David Moher7, Drummond Rennie8,9, Henrica C.W. de Vet10 and Jeroen G. Lijmer1

- Author Affiliations
1Department of Clinical Epidemiology and Biostatistics, Academic Medical Center—University of Amsterdam, 1100 DE Amsterdam, The Netherlands.
2Department of Pathology, University of Virginia, Charlottesville, VA 22903.
3Clinical Chemistry, Washington, DC 20037.
4Centre for Statistical Sciences, Brown University, Providence, RI 02912.
5Centre for General Practice, University of Queensland, Herston QLD 4006, Australia.
6Department of Public Health & Community Medicine, University of Sydney, Sydney NSW 2006, Australia.
7Chalmers Research Group, Ottowa, Ontario, K1N 6M4 Canada.
8Institute for Health Policy Studies, University of California, San Francisco, San Francisco, CA 94118.
9Journal of the American Medical Association, Chicago, IL 60610.
10Institute for Research in Extramural Medicine, Free University, 1081 BT Amsterdam, The Netherlands.
↵aAddress correspondence to this author at: Department of Clinical Epidemiology and Biostatistics, Academic Medical Center—University of Amsterdam, PO Box 22700, 1100 DE Amsterdam, The Netherlands. Fax 31-20-6912683; e-mail p.m.bossuyt@amc.uva.nl.
 
Abstract

The quality of reporting of studies of diagnostic accuracy is less than optimal. Complete and accurate reporting is necessary to enable readers to assess the potential for bias in the study and to evaluate the generalisability of the results. A group of scientists and editors has developed the STARD (Standards for Reporting of Diagnostic Accuracy) statement to improve the reporting the quality of reporting of studies of diagnostic accuracy. The statement consists of a checklist of 25 items and flow diagram that authors can use to ensure that all relevant information is present. This explanatory document aims to facilitate the use, understanding and dissemination of the checklist. The document contains a clarification of the meaning, rationale and optimal use of each item on the checklist, as well as a short summary of the available evidence on bias and applicability. The STARD statement, checklist, flowchart and this explanation and elaboration document should be useful resources to improve reporting of diagnostic accuracy studies. Complete and informative reporting can only lead to better decisions in healthcare.

Introduction

In studies of diagnostic accuracy, results from one or more tests are compared with the results obtained with the reference standard on the same subjects. Such accuracy studies are a vital step in the evaluation of new and existing diagnostic technologies (1)(2).

Several factors threaten the internal and external validity of a study of diagnostic accuracy (3)(4)(5)(6)(7)(8). Some of these factors have to do with the design of such studies, others with the selection of patients, the execution of the tests or the analysis of the data. In a study involving several metaanalyses a number of design deficiencies were shown to be related to overly optimistic estimates of diagnostic accuracy (9).

Exaggerated results from poorly designed studies can trigger premature adoption of diagnostic tests and can mislead physicians to incorrect decisions about the care for individual patients.
Reviewers and other readers of diagnostic studies must therefore be aware of the potential for bias and a possible lack of applicability.

A survey of studies of diagnostic accuracy published in four major medical journals between 1978 and 1993 revealed that the methodological quality was mediocre at best (8). Furthermore, this review showed that information on key elements of design, conduct and analysis of diagnostic studies was often not reported.
snip
"I have to understand the world, you see."
Richard Feynman

Joanne60
Posts: 110
Joined: Mon 13 Feb 2012 15:49
Location: Guildford Surrey UK

Re: Misleading medical tests

Post by Joanne60 » Mon 7 Jan 2013 11:13

I find the arguments about testing rather strange we are talking antibody tests - there should be no argument it is commonly recognised in medicine that antibody tests are not always reliable. At least that was how my 91 year old Dad was taught during his pharmaceutical training and when chatting about my case to his cousin who was involved with Lyme in the early 1980's ( he was in charge of the blood banks at that time in US) the first thing the cousin said was well you know they are antibody tests so you would know they are not reliable.

I accept that improvements can be made to antibody tests but until we get more direct testing such as maybe proteomics available for use with patients instead of just research, the nay sayers haven't a leg to stand on and need to be ignored and the rest of us will move on with a clinical diagnosis and treatment that helps in each individual case.

Interestingly the WHO published a report a couple of years ago that showed the TB tests missed about 50% of cases but I didn't keep a reference to that research.

Henry
Posts: 1108
Joined: Thu 10 Nov 2011 18:49

Re: Misleading medical tests

Post by Henry » Mon 7 Jan 2013 16:15

I wish to call attention to this recent reference as well:

http://www.ncbi.nlm.nih.gov/pubmed/1994 ... dinalpos=2.

Note that the very low percentage (30%) of positives in individuals with EM is due to the fact that it is too early to detect serum antibodies at that time. It is not simply a matter of sensitivity or specificity. Also, there are many problems with the IgM Western blot -- as this and other recent publications indicate. Remember-- even the best and most sensitive diagnostic test one can imagine is not going to detect Lyme disease in someone who doesn't have it.

nnecker
Posts: 215
Joined: Wed 19 Dec 2012 22:57

Re: Misleading medical tests

Post by nnecker » Tue 8 Jan 2013 16:49

Clinical diagnosis has it's problems too.Here is one example:

http://flash.lymenet.org/ubb/ultimatebb ... 0744#00000

User avatar
LHCTom
Posts: 341
Joined: Mon 22 Oct 2012 4:18

Re: Misleading medical tests

Post by LHCTom » Tue 8 Jan 2013 19:39

Note that the very low percentage (30%) of positives in individuals with EM is due to the fact that it is too early to detect serum antibodies at that time. It is not simply a matter of sensitivity or specificity. Also, there are many problems with the IgM Western blot -- as this and other recent publications indicate. Remember-- even the best and most sensitive diagnostic test one can imagine is not going to detect Lyme disease in someone who doesn't have it.
If you believe someone "doesn't have it", you will not detect Lyme either. Dogma is the larger problem. The validity or accuracy or sensitivity of any test requires a thoroughness that is rarely achieved due to the dogmas. Selection bias and the lack of using a "good" reference standard are among the most common problems. Studies such as the one cited, select their populations based on either an EM or positive 2T serology. Not all Borrelia infections cause an EM. This is a risky assumption. Using a test to validate itself is meaningless. A proper study requires knowing which patients are "truly" positive for Borrelia and they strain diversity known. This can and has been done by combining sensitive direct molecular DNA amplification techniques as in http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3348129/ , xenodiagnosis and culture.

By combining multiple direct detection techniques, "true infections" can be determined as an accurate reference for the validation of a test. Patient populations must not be restricted geographically and strain/species diversity of infections guaranteed. Limiting the population studied to those with EM or CDC positive serology almost certainly leaves off the cases causing the inaccuracy. Fear and dogma seems to limit the population in most studies. Populations should include early, disseminated and very late disseminated cases. Any selection bias will lead to inaccuracy. This diverse population of suspected Borrelia infected must then be tested by the multiple direct testing to find true positives. Any assumptions or limitations in geography, strain/species, stage of infection, etc.. will result in a flawed validation.

Once a highly diverse population has been tested with accurate reference testing plus the diversity of the positive group is assured, then a specific implementation of an antibody test can be validated. One cannot assume a lab has a "good test" unless its validated against the properly verified and diverse infected population. Any good scientist knows a validation with assumptions or selection bias is worse than useless. This is the case with the existing CDC 2T test.

Until investigators become willing to explore the many cases that are written off as "someone who doesn't have it" due to dogma, this will remain a psuedo-science due to bias and bad assumptions fed by dogma. Wormser and the boys could clear this up easily but they fear the results. So they just keep making assumptions and carefully biasing their studies and not surprisingly keep getting the same flawed results.
The greater the ignorance, the greater the dogmatism.

Attributed to William Osler, 1902

Henry
Posts: 1108
Joined: Thu 10 Nov 2011 18:49

Re: Misleading medical tests

Post by Henry » Wed 9 Jan 2013 15:35

LHCTom: The major flaw in the paper that you cite is assuming that the detection of DNA by PCR indicates active infection, in the absence of other evidence (culture positivity) to support such an assumption. Although PCR is extremely sensitive, it detects only DNA -- not viable bacteria that are rarely detected (by culture) in the blood of patients with an EM. Borrelia often can be detected in patients with an EM; however, only from skin biopsies from the edge of an EM lesion. DNA specific for Borrelia has been detected by PCR long after an infection has been cured by antibiotics -- even in centuries old museum specimens, including the 5,000 year old Tyrolean iceman. So, arguments based on the results of PCR tests must be view with these reservations in mind.

Although there were problems related to specificity and sensitivity during the early years (the 1990s) when the two-tiered test was first developed and used, many improvements and refinements have been made since those early days. These have been documented in MANY published studies on the application of two-tiered testing to the diagnosis of Lyme disease. It is not just "a feeling" , but a fact that detectable amounts of antibody are simply not present during the early EM stages of infection, which is the main reason why only about 30% of two-tiered tests are positive at that time. However, one should not be surprised to detect Borrelia DNA at that time. It is very clear that, from the standpoint of sensitivity and specificity, two-tiered testing performs very well and is a reliable and reproducible diagnostic test procedure at later stages of infection when the antibody response is largely IgG. That is not too late to treat at that time since antibiotic therapy has been shown to be quite effective.

duncan
Posts: 1370
Joined: Wed 5 Sep 2012 18:48

Re: Misleading medical tests

Post by duncan » Wed 9 Jan 2013 16:02

Henry: "It is very clear that, from the standpoint of sensitivity and specificity, two-tier testing performs very well and is a reliable and reproducible diagnostic test procedure at later stages of infection when the antibody response is largely IgG."

Well, except, no, it is not very clear all the time. And "performs very well" doesn't cut it; too vague, you see.

Henry: "That is not too late to treat at that time since antibiotic therapy has been shown to be quite effective."

Hmmm...I fear this may seem somewhat disingenuous as even the IDSA admits to exceptions. The trick is qualifying the amount of exceptions, wouldn't you agree? For some, using the term "exceptions" may even prove egregious, as the numbers may prove large in absolute terms (as opposed to percentages).

Henry
Posts: 1108
Joined: Thu 10 Nov 2011 18:49

Re: Misleading medical tests

Post by Henry » Wed 9 Jan 2013 17:53

Believe what you want.....................

User avatar
LHCTom
Posts: 341
Joined: Mon 22 Oct 2012 4:18

Re: Misleading medical tests

Post by LHCTom » Thu 10 Jan 2013 6:48

Believe what you want.....................
:lol: :lol: :lol:

My belief is in science with integrity. Any ethical person trained in the sciences understands scientific knowledge is constantly evolving and blind defense of any idea like the quality of the CDC 2T testing is nothing more than dogma - not science. Integrity is based on always challenging any theory trying to evolve as faulty assumptions or bias is uncovered. Real integrity is not defending an idea like the CDC 2T test but reaching beyond belief and testing it to improve it. Its not possible to explain integrity to dogmatics as they accept their beliefs without challenging. They often don't understand integrity. There may be plenty of dogmatics within the Lyme world but those who stop reaching for better theories are the worst . I would hope everyone would agree in the integrity of real science. The validation CDC 2T test has been based on assumptions and a variety of biases which leave it open to improvement. Failing to accept that is pure dogma. :woohoo: :woohoo: :woohoo:

This is in the same category as Richard Feynman's Cargo-Cult-Science. Its about those scientists who believe they have found the ultimate theory - never to be overturned. Einstein overturned Newton and I'm sure Einstein's General Theory of Relativity will eventually be found to be incomplete. Defending the CDC 2T test on its face is absurd. It shows a lack of understanding of the most basic aspects of the scientific method.

Even Wormser and the boys have been forced to admit the test reliability is significantly effected by strain and species. Its well known the US CDC 2T test failed in Europe for this reason. The test is based on one lab strain. As Lyme research has moved from the US Northeast to all parts of the US and Canada, its become more and more clear that strain and species diversity in North America is much broader than originally assumed. There is a long list of potential flaws that have come to light since 1994 that must be considered. Any person who accepts science understands these assumptions undermines the CDC test validity. It will undoubtedly be found to be flawed as "good scientists" enter the arena. This is already happening. If this and the other biases are not understood, then its dogma - not science on which this blind defense is made...

believe what you like.... enjoy

:bonk: :bonk: :bonk:
The greater the ignorance, the greater the dogmatism.

Attributed to William Osler, 1902

Henry
Posts: 1108
Joined: Thu 10 Nov 2011 18:49

Re: Misleading medical tests

Post by Henry » Thu 10 Jan 2013 14:48

LHCTom: My, my. You deliver such a biased response and then have the nerve to speak about integrity ? Dream on...........

Post Reply