Determining Extent Adherence to the IOM Standards in AHRQ's NGC
The following information describes the Institute of Medicine (IOM) standards for trustworthy guidelines and updates the National Guideline Clearinghouse's (NGC's) activities related to the standards.
The IOM Standards and Why They Were Issued
In 2008, the U.S. Congress requested that the Secretary of the Department of Health and Human Services (DHHS) contract with the IOM to identify "the best methods used in developing clinical practice guidelines in order to ensure that organizations developing such guidelines have information on approaches that are objective, scientifically valid, and consistent." (1) The resulting report, Clinical Practice Guidelines We Can Trust (2), was published in March 2011. It provided a revised definition of an evidence-based clinical practice guideline and eight standards for clinical practice guidelines to guide the process of developing trustworthy guidelines. In summary, the IOM recommended that guidelines should:
- Be based on a systematic review of the existing evidence
- Be developed by a knowledgeable, multidisciplinary panel of experts and representatives from key affected groups
- Consider important patient subgroups and patient preferences, as appropriate
- Be based on an explicit and transparent process that minimizes distortions, biases, and conflicts of interest
- Provide a clear explanation of the logical relationships between alternative care options and health outcomes, and provide ratings of both the quality of evidence and the strength of recommendations
- Be reconsidered and revised as appropriate when important new evidence warrants modification of recommendations. (2)
NGC used the IOM's revised definition of a guideline to modify its inclusion criteria. More is available on the Inclusion Criteria page.
IOM Recommendation for the NGC: Indicating Extent Adherence
The 2011 IOM report on trustworthy guidelines included the following recommendation to help identify guidelines that used trustworthy processes when being developed:
"The Agency for Healthcare Research and Quality should…require the National Guideline Clearinghouse (NGC) to provide a clear indication of the extent to which clinical practice guidelines (CPGs) submitted adhere to the standards for trustworthiness." (2)
Following the release of the IOM report, the Agency for Healthcare Research and Quality (AHRQ) explored options to determine how best to meet the above recommendation, if possible. Importantly, the AHRQ and the NGC reached out through meetings, phone calls, conference presentations, and webinars to gather input from a variety of stakeholders, including the NGC/NQMC Editorial Board, guideline developers participating in the NGC, the AGREE Research Trust, and the NGC users.
That exploration is now complete. The NGC is finalizing the AHRQ-approved process of assessing CPGs against select standards/substandards. Following this, NGC will begin assessing guidelines in our work queue and will then add assessment results to the Web display. We anticipate that the process of determining and posting the extent adherence of guidelines against the IOM standards may begin by the summer of 2017.
Assessing CPGs against select standards/substandards involves the use of the National Guideline Clearinghouse Extent Adherence to Trustworthy Standards (NEATS) Instrument. An AHRQ/NGC team developed this instrument and pilot tested it to ensure reliability among NGC staff trained to use it. NGC assessors will use the NEATS Instrument to determine the extent to which CPGs adhere to elements of these IOM standards:
- 1.1 (Establishing transparency)
- 2.2 (Management of conflict of interest)
- 3.1 and 3.2 (Guideline development group composition)
- 4.1 (Clinical practice guideline–systematic review intersection)
- 5.1 (Establishing evidence foundations for and rating strength of recommendations)
- 6.1 (Articulation of recommendations)
- 7.1 (External review)
- 8.1 (Updating)
Assessments developed from the NEATS instrument will be sent to the guideline developers for verification prior to being posted on NGC.
The final step of implementation is developing a meaningful display of the assessments on the NGC Web site. The goal is to present this information visually so that users can access it quickly and understand it easily. The NGC technology team is currently working on this display. This approach incorporates many of the suggestions received from the guideline community and other sources, such as the AHRQ leadership.
As the NGC moves forward with this effort, we will update this information. Please send your comments and helpful suggestions on the approach to determining the extent adherence to the IOM Standards for Trustworthy Guidelines to firstname.lastname@example.org. NGC will acknowledge comments but will not publish them. We look forward to hearing from you.
When will NGC implement the NEATS guideline assessments?
Determining and posting assessments of the extent to which guidelines accepted for inclusion in NGC adhere to the IOM standards may begin sometime in 2017.
Will NGC assess all the guidelines represented on the site?
No, only guidelines accepted for inclusion after implementation of NEATS will be assessed.
Can guideline developers give input on NGC's assessment of their guidelines?
Yes, guideline developers can review and respond to the NEATS assessments of their guidelines prior to publication on the NGC Web site.
Can a guideline developer choose not to have their guideline assessed?
No, following implementation, any guideline accepted for NGC inclusion will be assessed.
Can users see which guidelines have assessments?
Yes, search results will display a NEATS Assessment icon next to assessed NGC summaries to help users identify them. In addition, a specialized search filter will identify all NGC summaries in the database that have assessments.
What do the NEATS assessments mean?
The NEATS assessments provide insight into whether a guideline was developed in a transparent and rigorous fashion, as deemed by the IOM standards. Please note, NGC does not develop, produce, approve, or endorse the guidelines represented on this site. Refer to the Disclaimer page for more information.
Will a lower score on any of the NEATS items mean that the guideline will be withdrawn from NGC?
No, once a guideline is accepted into NGC, it will remain represented on the web site until it no longer meets NGC's inclusion criteria.
Is there a single overall assessment for each guideline assessed?
No, at this time, there is no overall assessment. Each item stands alone. Determining how best to weight the individual items in order to make an overall assessment requires more thought.
- Medicare Improvements for Patients and Providers Act, Public Law 110-275, 110th Cong. (July 15, 2008).
- Institute of Medicine. Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E, editor(s). Clinical practice guidelines we can trust. Washington (DC): National Academies Press; 2011. 2p. Also available: http://nationalacademies.org/hmd/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust.aspx .
Page Last Reviewed or Updated: May 26, 2017