Guidelines for Assessing Professional Information

Updated December 07, 2006

Introduction and Background:

 

This is several sets of questions to assist the clinician in determining whether or not to incorporate the information contained in a paper, in a conference presentation or on the Internet into their professional knowledge base for application in the user’s clinical practice. This list is in addition to a general background knowledge about clinical studies such as the forms of bias, the types of study designs, and their inherent strength of evidence when properly executed. As most information sources contain flaws of varying importance and as this assessment process is subjective, the user must make their own decision about the strength of empirical evidence that the source contains. First some definitions.

Primary Source: A primary scientific paper is defined as the first publication of original research results that is in a form with sufficient detail whereby the author’s peers can critically evaluate the research process and could repeat the study to test the conclusions. The strongest form of primary publication is a widely available, peer-reviewed scientific journal. By convention, most primary scientific papers have six distinct sections: abstract, introduction, materials and methods, results, discussion and conclusions, and references. Although not all primary sources are structured in this fashion, their presence is the hallmark of a primary scientific paper, the only exception being systematic reviews or meta-analyses.

Each section has a distinct purpose. The abstract concisely defines what is in the paper by stating in 250 to 400 words the study’s principle objective and the methodology used, summarizing the results, and stating the principle conclusions. The introduction includes a brief review of pertinent literature that provides a rationale for the study and ends by clearly stating the problem being investigated. The materials and methods section provides either sufficient detail or citations about the research methods so that a competent investigator can evaluate and could repeat the work. The results are just that. The discussion contains the author’s interpretation of the generalization and significance of the study including how the results are consistent or inconsistent with previously published work.

Secondary Source: A secondary source is an information source that does not have as a major component the description of formal observations or experiments but rather is synthesized from some combination of primary sources, experience, or authoritative belief (dogma). The primary literature used may have been selected in a biased or incomplete fashion and may have been used without comprehensive critical appraisal to establish the relative strength of evidence in each source. Examples of secondary sources are review articles, journals specializing in practitioner-oriented reviews, most practitioner-oriented conference proceedings, trade publications, most e-mail conversations, and authorities presenting information without supporting evidence in whatever format (lectures, CE meetings, e-mail forums).

Tertiary Source: A tertiary source is a compilation of information for application across a broad spectrum, typically represented by class notes and textbooks intended for use in core courses. The information is often presented in a dogmatic, authoritative fashion as a sequence of facts and interpretations of their meaning that the reader is expected to believe without reservation or evaluation. The strength of the underlying evidence is not indicated, any current controversy between researchers in the area is not addressed and areas where further scientific evidence is needed are not indicated. The bibliography is usually predominately of secondary literature and is usually intended to provide the interested reader with entry points to the underlying primary literature. Much of the evidence-based information contained in textbooks is filtered sufficiently that it is accepted by most all of the experts in the field, much of it is unlikely to change in the future, and most of the changes will be minor. However, depending on the field, textbooks contain a varying amount of dogma and interpretations of facts that will change with the progress of research in the area, sometimes significantly. Between the publication of significant new research results in the primary literature to their integrated into tertiary sources is often a considerable time lag. Because clinical experience is often not examined critically, clinical textbooks tend to contain a larger proportion of dogma than do basic science textbooks. Class notes usually contain less information than do textbooks and do not undergo the auditing process as part of publication as textbooks do.

[Return to Contents List]


 

10 Quick Questions to Apply to Primary Papers:

  • Does the introduction section clearly state the problem being investigated and provide a conceptual justification for the study supported by a review of the relevant literature? What is the specific question that the study was intended to answer? Is it clinically relevant?
  • Does the material and methods section clearly state how the problem was studied in sufficient detail that a person familiar with the area could repeat the study? Were the study design and methods current and appropriate to answer the question?
  • Were the subjects selected, the research variables measured, and the outcomes established appropriate for the question? Are they clearly relevant to your clinical situation?
  • Was symmetry maintained at every opportunity (e.g., randomization, blinding, . . .) in the design, execution and interpretation of the study?
  • Does the results section clearly state the findings?
  • How likely are the results due to the presence of systematic bias (selection, measurement, confounding bias) rather than the effect investigated?
  • Are the effects of chance on the study assessed by presenting p-values or confidence intervals?
  • Are the effects sufficiently large to be biologically or clinically significant? If the effects are not statistically significant (a negative study), is the power of the study to detect a minimum difference given and is this difference a reasonable minimum for biologically or clinically significance?
  • Does the discussion section clearly state what the findings mean in the context of the study and discuss their clinical relevance? Are areas needing further study indicated?
  • Are the citations current, relevant, and mostly primary papers in refereed scientific journals?
  • [Return to Contents List]


     

    10 Quick Questions to Apply to Reviews

    (modified from CMAJ 138:697-703 (1988) and J Clin Epidemiol 44:91-98 (1991))

      1. What are the academic (M.S., PhD degrees) and clinical (specialty boards) credentials of the authors in the area being reviewed? Do they likely have the expertise to find and to critically evaluate the relevant literature?
      2. Was the purpose of the review and the specific question(s) being addressed clearly stated?
      3. Were the sources and methods of locating relevant studies stated?
      4. Were explicit guidelines and inclusion and exclusion criteria used to determine which to include in the review stated? Or does the bibliography reflect primarily secondary and tertiary sources, implying the authors relied on others for critical evaluation?
      5. Was selection bias minimized (e.g., not selecting only studies with positive findings)?
      6. Was the methodological validity of primary studies assessed and were the validity criteria reported?
      7. Was the assessment of validity reproducible and was it done by more than one person and in a blinded fashion with minimal opportunity for bias?
      8. Was the information systematically integrated with explanation of data limitations and inconsistencies?
      9. Were clinically important areas lacking support in the primary literature identified or were specific needs for further research identified?
      10. Are you aware of important and relevant primary papers that should have been included but were not?

    [Return to Contents List]


     

    Guidelines for Attending Professional Meetings:

    Pointers for the Evidence Wary
    • Beware of the insidious, ever present "Dr. Fox effect".
    • "Dr. Fox" was an actor who bamboozled an audience of professionals after being given some buzzwords from the profession to string together for his presentation.
      • Study of the effect in a medical school class: Ware, JE, RG Williams (1975). The Dr. Fox effect: A study of lecturer effectiveness and ratings of instruction. J Med Educ 50:149-156.
    • Lesson: Evaluate the evidence in the message and not the graphics and the delivery (for the travel weary, flashy and polished we hope but don’t be seduced). The message, not the messenger, has the evidence!
    • Remember that numerous studies have shown that only 1/2 of new information presented at scientific meetings will eventually be published in scientific, peer-reviewed journals.

      Of the new findings being presented, many are preliminary. Further work often does not bear them out or the study is too flawed for acceptance by a scientific refereed journal.

    • Evaluate presentations for content and strength of primary evidence.

    If the presentation is of new primary evidence, evaluate it in the same fashion as a primary evidence paper in a scientific journal.

    • Inherent strength of study type? (Case series vs. RBCT) Were the key elements of design stated and executed correctly?
      • John’s Law - If they don’t state it, they likely didn’t know they should do it (or know how to do it right) so only by blind luck will they have done it right.
      • If they don’t state it, stand up and ask.
    • Was the study of sufficient size to detect clinically important differences?
      • Were the results evaluated statistically (error bars on plots, p-values)? If not, could the outcome have been due to chance?
      • Remember that approximately 350 animals per group are required to have a reasonable chance of detecting a 10% difference in a yes / no outcome between groups.
      • Remember the "3 / N Rule" – Even if all N were done with great success, this is still consistent with a failure rate of 3 / N.
    • Integration of relevant primary literature, particularly the conflicting? Does this person keep in touch with the current scientific evidence and do they know how to evaluate it properly? What is the conceptual and theoretical basis of this study?
    • Presentations that are review, summary or integrative oral presentations (e.g.: How to diagnose and treat "X") are difficult to evaluate and often do not contain citations.

    • If not, are citations to the relevant primary literature appearing in scientific journals given in the proceedings paper or is this presentation likely strictly authority (dogma) based?
    • What proportion of citations are primary rather than secondary? Is the presenter relying on the information synthesis of others rather than doing the critical evaluation himself or herself?
    • What is the indication that the presenter keeps up with the current scientific evidence and knows how to evaluate it?
    • Rub shoulders with lots of people, listen with an open but sufficiently skeptical mind, take good notes, and ask good questions to fill in the gaps.

      Often, much useful information is exchanged between colleagues meeting in the halls and over meals outside of the formal meetings.

    • Have a Great meeting, an excellent means of recharging your batteries for clinical practice.

    [Return to Contents List]

    Guidelines for Evaluating Internet Materials:

    (Original set from Bibliografia sulla qualità delle informazioni mediche)

    [Return to Contents List]