Methodological Details

Environmental Scan

Main Indicator Sources

The following are the main indicator sources for the measures included in the IBHC Measures Atlas:

  1. Literature search. A systematic literature search was conducted for measures relevant to integrated behavioral health care. The search strategy is outlined below.
  2. AHRQ-commissioned studies by evidence-based practice centers (EPCs). The following studies were consulted:
    (a)   Integration of Mental Health/Substance Abuse and Primary Care (2009), available at: http://www.ahrq.gov/research/findings/evidence-based-reports/mhsapctp.html (accessed on January 23, 2013);
    (b)  Care Coordination (2007), available at: http://www.ahrq.gov/clinic/tp/caregaptp.htm (accessed on January 23, 2013), which is volume 7 of the series Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies; and
    (c)    Future Research Needs for the Integration of Mental Health/Substance Abuse and Primary Care (2010) available at: http://effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=524 (accessed on January 23, 2013).

    All three sources contained relevant background research as well as measures on integration and related health care approaches.

  3. Literature reviews conducted by the National Integration Advisory Council’s (NIAC) Academy for Integrating Behavioral Health and Primary Care. These reviews continue the work of the prior AHRQ evidence-based practices studies through a comprehensive literature search of integration articles published since 2010. The Academy is also identifying literature published since 2000 on research gaps found in the literature, including those identified in the prioritized list of gaps in the Future Research Needs report.
  4. Major sources on quality measures related to integration. They included the National Quality Measures Clearinghouse, available at: http://www.qualitymeasures.ahrq.gov/ and the National Quality Forum, available at: http://www.qualityforum.org/Home.aspx (link is external).
  5. The Care Coordination Measures Atlas (2011). This atlas, available at: http://www.ahrq.gov/qual/careatlas/, on a related topic, contained a number of relevant measures.
  6. Expert panel calls and in-person meetings. Electronic communications and an in-person meeting were also used to solicit suggestions from NIAC members and an expert panel on integration-related measures used in the field or those identified in published or grey literature.

Literature Search Strategy

The overall search strategy was guided by the definition and framework of primary and behavioral health care integration. Integrated behavioral health care was defined as:

The care a patient experiences as a result of a team of primary care and behavioral health clinicians, working together with patients and families, using a systematic and cost-effective approach to provide patient-centered care for a defined population.

This care may address mental health and substance abuse conditions, health behaviors (including their contribution to chronic medical illnesses), life stressors and crises, stress-related physical symptoms, and ineffective patterns of health care utilization.

The Integration Framework contains the following functional domains, which encompass the structure, process, and outcomes of integrated behavioral health care:

1. Care Team Expertise: The team is tailored to the needs of particular patients and populations—with a suitable range of expertise and roles.

2. Clinical Workflow: The team uses shared operations, workflows, and protocols to facilitate collaboration.

3. Patient Identification: The team employs systematic methods to identify and prioritize individuals in need of integrated care.

4. Patient and Family Engagement: The team engages patients and family (as appropriate) as active members in the integrated care team and in shared care plans.

5. Treatment Monitoring: The team systematically measures patient outcomes over time and adjusts treatment as needed.

6. Leadership Alignment: The team is supported by leadership and administrative alignment with goals of integrated behavioral health care.

7. Operational Reliability: The team is supported by reliable and robust office processes.

8. Business Model Sustainability: The team is supported by a sustainable business model.

9. Data Collection and Use: The team is supported by the collection and use of practice-level data to achieve high quality, high value care.

10. Patient Experience: Patient experience (provider and system experience, clinical and financial outcomes to be included in subsequent editions of this IBHC Measures Atlas).

The definition and framework were then used to develop a list of keywords that were used to search the sources listed below. Measures were included in the IBHC Measures Atlas if they were publically available and were published or included in publications since 2001. Copyrighted or otherwise protected measures are included if they are available to the public at no cost. The search was limited to English language publications. The final search was conducted from December 2011 to February 2012.

National Quality Forum and National Quality Measure Clearinghouse searches. To effectively search the measurement collections of the National Quality Forum (NQF) and National Quality Measures Clearinghouse (NQMC), a list of keywords related to integration and framework measurement constructs was compiled in consultation with experts associated with the project. See List of Keywords Used to Search the National Quality Forum and National Quality Measures Clearinghouse Collections. Our search of the two collections using these keywords resulted in a combined total of 262 unduplicated measures that primarily focused on patient behavioral care such as patient behavioral health care screening, needs assessment, medication therapy, other therapies, or health outcomes. Most of these care measures were relevant but not specific to integrated care settings. See Results of Searching the National Quality Measures Clearinghouse, and Results of Searching the National Quality Forum Measurement Collection. To ensure adequate attention in the IBHC Measures Atlas to measures on the structure and process of integration, both examples of applicable patient care measures in the IBHC Measures Atlas and suggestions on how to access the fuller set of patient care measures in the NQF and NQMC collections were selected.

Published literature search. The terms for the published literature search were based on the list of key words developed by project staff and subject experts, the search strategy of the Care Coordination report (2007), the Care Coordination Measures Atlas (2011) search strategy, and the knowledge and expertise of the librarian, who is also conducting searches for the Academy for Integrating Behavioral Health and Primary Care. In particular, the following major sources of publications on measures and databases pertaining to primary and behavioral health care were searched:

See Results of Searching Major Sources of Publications on Measures and Databases on Integrated Care.

Grey literature search. To guide the grey literature search, research staff consulted references to measures in publications identified in the National Integration Advisory Council’s (NIAC) Academy for Integrating Behavioral Health and Primary Care, which continues the work of those prior AHRQ studies through a comprehensive literature search of integration articles published since 2010. The Academy is also identifying literature published since 2000 on research gaps found in the literature, including those identified in the prioritized list of gaps in the Future Research Needs report. The Academy has selected key/foundational literature on the topic of integration to provide a framework for the modern literature. In addition, the Academy literature enhances prior collections of integration resources through the inclusion of grey literature references.

Measure Selection

Measures were included in the IBHC Measures Atlas based on a series of expert reviews.

(1)  A team member reviewed the titles and abstracts of all search results and eliminated sources that did not have clear relevance to the definition of integrated behavioral health care or health care quality measurement.

(2)  A panel of experts then reviewed the full text of each measure to assess relevance to the functional domain of the Integration Framework. Measures initially considered relevant were passed along for mapping to the Framework; measures that were not relevant were eliminated.

A team of reviewers engaged in an iterative process of mapping the remaining measures to the measurement constructs of the Integration Framework. Reviewers worked consecutively and had access to prior reviewers’ mapping decisions. Discrepancies were resolved through discussions among the reviewers. When reviewers disagreed with the expert panel regarding inclusion of a measure in the IBHC Measures Atlas, the measure was re-presented to the expert panel for a final decision.

Following the mapping of the measures, reviewers conducted a final usability and usefulness check in which they assessed how well each measure aligned with the measurement constructs, the breadth of constructs addressed by a given measure, and the burden to complete the measure. Results of the usability and usefulness check were used to divide the final 17 measures into two categories. See Overview of Measures included in the IBHC Measures Atlas, earlier in this document, for the full list of measures.

List of Keywords Used to Search the National Quality Forum and National Quality Measures Clearinghouse Collections

Research staff consulted with project experts and compiled a range of terms that captured the major domains and measurement constructs of the performance framework.

Keywords included the following:

  • Anxiety
  • Behavioral
  • Behavioral health disorder
  • Behavioral medicine
  • Bipolar disorder
  • Bundled care
  • Business model
  • Capitated care
  • Capitation
  • Care coordination
  • Clinical monitoring
  • Co-located care
  • Collaborative care
  • Coordination/coordinated care
  • Delivery of health care
  • Depression
  • Family-centered medical home
  • Formal stepped-up care
  • Health behaviors (e.g., wellness, smoking cessation, exercise, healthy eating, obesity, achieving health behavioral change)
  • Fee-for-service
  • Health care home
  • Integrated care
  • Integrated administration/organization
  • Integrated delivery system
  • Integration
  • Managed care
  • Meaningful use
  • Measures
  • Medical home
  • Medical care home
  • Medication
  • Medication adherence
  • Mental health
  • Mental health services
  • Models
  • Navigation
  • Organizational
  • Outcomes assessment methods
  • Patient-centered care
  • Patient-centered care home
  • Patient-centered medical home
  • Patient education/self-management
  • Patient engagement
  • Physical activity
  • Primary health care/Primary care
  • Primary care home
  • Promotoras/Promotores
  • Process evaluation/assessment methods
  • Psychotherapy
  • Rating/rated
  • Reproducibility of results
  • Scales/subscales
  • Screening
  • Self-management
  • Standardized followup
  • Stepped-up care
  • Substance/substance-related disorders (alcohol, drug)
  • Supervision
  • Team-based care

Results of Searching the National Quality Measures Clearinghouse (NQMC)

The following table presents the search terms and number of measures identified in a search of the National Quality Measures Clearinghouse conducted in October 2011. Measurement setting was limited to Ambulatory/Office-Based Care, Community Health Care, Managed Care Plans, and Rural Health Care.

KeywordsNumber of Measures
“primary care home” OR “medical home” or “medical care home” OR “patient centered care home”11
“mental health” OR anxiety OR depression OR “bipolar disorder”272
“capitated care”0
“co-located care”0
“fee for service”26
“bundled care” OR “capitated care” OR capitation OR “business model”>0
“care coordination” OR “coordinated care”3
“team-based care” OR “collaborative care” OR “integrated care” OR integration OR “patient centered care”5
“primary care”365
psychotherapy9
“medication adherence”5
“patient education” OR “self management”31
“patient engagement”0
navigation OR navigators OR promotores OR promotoras0
(smoking OR tobacco) AND (cessation OR cease OR quit)33
“wellness” OR “exercise” OR “physical activity” OR “obesity”82
“clinical monitoring”3
“stepped-up care”0
“meaningful use”33
“standardized follow-up”0
“standardized” AND “follow-up”13
psychotherapy9
“medication adherence”5
“patient education” OR “self-management”31
“patient engagement”0
navigation OR navigators OR promotores OR promotoras0
 (smoking OR tobacco) AND (cessation OR cease OR quit)33
wellness OR exercise OR “physical activity” OR obesity82
“clinical monitoring”3
“stepped-up care”0
“meaningful use”33
“standardized follow-up”0
standardized AND “follow-up”13
 

Results of Searching the National Quality Forum (NQF) Measurement Collection

The following table presents the search terms and number of measures identified in the search of the National Quality Forum measurement collection conducted in October 2011. Measurement setting was limited to Ambulatory/Office-Based Care, Community Health Care, Managed Care Plans, and Rural Health Care. All terms that were searched at the National Quality Measures Clearinghouse site were searched at the National Quality Forum Measurement Collection site. Many terms yielded no results. Only those terms that produced results appear in the search table.

KeywordsNumber of Measures
“Depression”4
“Depression”1
“Care Coordination”4
“Mental Health”2
“Bipolar Disorder”2
“Smoking Cessation”3
“Exercise”1
“Medical Home”2
“Integrated delivery system”2
“Managed Care”1
“Medication”5
“Mental Health”1
“Self Management”1
“Patient Engagement”5

Results of Searching Major Sources of Publications on Measures and Databases Pertaining to Measures of Primary and Behavioral Health Care

DatabaseKeywordsNumber of Measures
PubMed(“Outcome Assessment (Health Care)/methods”[Mesh] OR “process Assessment (Health Care)/methods”[Mesh] ) AND (delivery of health care, integrated[mh] OR primary health care[mh]) AND (mental health services[mh] OR behavioral medicine[mh] OR substance related disorders[mh]) Limits: English, published in the last 10 years1
(bundled OR coordinated OR coordination OR colocated OR collaborative OR “medical home” OR integration OR integrated OR “health care home” OR “medical care home” OR primary health care[mh] OR team OR delivery of health care, integrated[mh]) AND (“Outcome Assessment (Health Care)/methods”[Major] OR program evaluation/methods[Major] OR “process Assessment (Health Care)/methods”[Major] ) AND (mental health services[mh] OR behavioral medicine[mh] OR substance related disorders[mh]) Limits: English, published in the last 10 years30
(bundled OR coordinated OR coordination OR co-located OR collaborative OR “medical home” OR integration OR integrated OR “health care home” OR “medical care home” OR primary health care[mh] OR team OR delivery of health care, integrated[mh]) AND (mental health services[mh] OR behavioral medicine[mh] OR substance related disorders[mh]) Limits: English, published in the last 10 years8345
reproducibility of results[mh] AND (bundled OR coordinated OR coordination OR co-located OR collaborative OR “medical home” OR integration OR integrated OR “health care home” OR “medical care home” OR primary health care[mh] OR team OR delivery of health care, integrated[mh]) AND (mental health services[mh] OR behavioral medicine[mh] OR substance related disorders[mh]) Limits: English, published in the last 10 years108
integrated[title] AND (measure[title] OR measures[title] OR measurement[title]) Limits: English, published in the last 10 years113
(behavioral OR mental OR psychiatr* OR substance OR drug+ OR alcohol*) AND integrated[title] AND (measure[title] OR measures[title] OR measurement[title]) Limits: English, published in the last 10 years21
"Delivery of Health Care, Integrated/organization and administration"[Mesh] Limits: English, published in the last 10 years  2166
(mental health services[mh] OR substance related disorders[mh] OR behavioral medicine[mh]) AND “Delivery of Health Care, Integrated/organization and administration”[Mesh] Limits: English, published in the last 10 years  247
(program evaluation/methods[mh] OR “process assessment(Health Care)”/methods[mh] OR “outcome assessment(health care)”/methods[mh] OR reproducibility of results[mh] OR measure OR measures OR models, organizational[mh] OR scale OR subscale) AND (mental health services[mh] OR substance related disorders[mh] OR behavioral medicine[mh]) AND “Delivery of Health Care, Integrated/organization and administration”[Mesh] Limits: English, published in the last 10 years  48
Medline(((kw: bundled w care OR kw: coordinated w care OR kw: coordination w1 care OR kw: co-located OR kw: collaborative w care OR kw: medical w home OR kw: integrated w care OR kw: health w care w home OR kw: medical w care w home)) and kw: primary) and (kw: mental or kw: behavioral or kw: substance) and yr: 2001-2011 and ln= “english”337
#1 AND (kw: process w measures OR kw: outcome w measures OR kw: structural w measures)  24
#1 AND (kw: process w measures OR kw: outcome w measures OR kw: structural w measures) NOT (mh=”treatment outcome”)  13
((kw: bundled w care OR kw: coordinated w care OR kw: coordination w1 care OR kw: co- w located OR kw: collaborative w care OR kw: medical w home OR kw: integrated w care OR kw: health w care w home OR kw: medical w care w home and yr: 2001-2011 and ln= “english”)) AND (kw: mental OR kw: behavioral OR kw: substance) AND ((kw: rated OR kw: rating+ OR kw: indicator+ OR kw: measure+ OR kw: valid* OR kw: reliab* OR kw: model+ OR kw: scale+ OR kw: subscale+ OR kw: questionnaire+) AND de: methods)  71
PsycINFO    DE=(“integrated services” or “interdisciplinary treatment approach” or collaboration or cooperation) AND DE=(“primary health care”) AND DE=(program evaluation) Date Range:     2001 to 2012 Limited to: English Only15
DE=(“integrated services” or “interdisciplinary treatment approach” or collaboration or cooperation) AND DE=(“primary health care”) AND DE=(rating scales or test reliability or test construction) Date Range:     2001 to 2012 Limited to: English Only0
DE=(“integrated services” or “interdisciplinary treatment approach” or collaboration or cooperation) AND DE=(“primary health care”) AND DE=(measurement or evaluation or "statistical analysis” or methodology) Date Range:     2001 to 2012 Limited to: English Only36
DE=(“mental health services”) AND DE=(“primary health care”) AND DE=(measurement or evaluation or “statistical analysis” or methodology) Date Range:     2001 to 2012 Limited to: English Only28
ASSIA (Applied Social Science Index & Abstracts)DE=(“mental health”) AND DE = “primary health care” AND DE=(measurement or "process evaluation" or analysis or assessment or models or scales or methodology)9
HAPI Health & Psychosoc-ial Instru-ments)“integrated care”.mp. [mp=title, acronym, descriptors, measure descriptors, sample descriptors, abstract, source]12
find similar to Barriers to Integrated Mental and Physical Health Care Interview  7
primary AND (mental or behavioral or substance or drugs) AND (health or care).mp. [mp=title, acronym, descriptors, measure descriptors, sample descriptors, abstract, source] (english language and yr = “2001 - 2011”)104

Clinical Methods

The purpose of the literature search was to find peer reviewed articles and/or reports that focused on the quality of integration between behavioral healthcare and physical healthcare settings (e.g.- Integrated Behavioral Health Care (IBHC)). There are multiple dimensions to consider when assessing health care quality. For this literature search project, we are using clinical health outcomes as the primary markers of quality. The literature search was limited to prevalent mental and physical health conditions that are commonly comorbid. The physical health conditions included: diabetes, cardiovascular disease, high blood pressure, osteoarthritis, and rheumatoid arthritis. The mental health conditions were depression, anxiety disorders, and substance abuse.

A three phase search strategy was used. First, staff from Westat’s IR Center conducted an extensive search for clinical outcome measures that assessed the quality of integration in healthcare settings. Initial searches were conducted in the following resources:

  • National Quality Measures Clearinghouse. Covers evidence-based quality measures and measure sets, including health care delivery measures and population health measures.
  • Applied Social Sciences Index and Abstracts (ASSIA). Covers health, social services, psychology, sociology, economics, politics, race relations, and education literature.
  • Health and Psychosocial Instruments (HAPI). Provides information on measurement instruments (i.e., questionnaires, rating scales, tests, index measures, coding schemes, checklists) in the fields of public health and medicine, psychosocial sciences, communication, organizational behavior, and others. HAPI records do not contain the actual instruments, but often direct the researcher to sources of measures needed for research, assessment, and evaluation.
  • PsycINFO. Covers international literature in the psychological, psychiatric, social, behavioral, and health sciences.
  • PUBMED. Covers all aspects of health and medical literature.

Results were limited to English language and to articles or reports published between 2012 and 2013.  The general search was an update of a search conducted in 2012, so the choice of publication dates was designed with an overlap of 1 year, in order to be certain of coverage. 

After the general literature search that updated the 2012 work, two additional searches were conducted covering financial outcomes and provider experience with integrated behavioral health care.  These two literature searches included documents published between 2001 and 2013. 

The Financial Outcomes search focused on the costs of planning and delivering integrated behavioral health care from the perspective of health care provider organizations, the cost of receiving integrated behavioral health care from the patients’ perspective, and health care utilization costs for patients receiving integrated behavioral health care from the payers’ perspective (public and private insurers).

The Providers search focused on provider experience with the structures and processes involved in implementation of integrated behavioral health care to patients; the perceived changes in culture and professional identity, experience working in an integrated setting; and the quality of care that results from implementation of integrated behavioral health care strategies. Provider was broadly defined as including: physician, nurse, nurse practitioner, physician assistant, psychologist, behavioral therapist, nutritionist, counselor, and therapist.

The terms used in the searches appear in the table below.  Terms were combined with Boolean operators (i.e., AND, OR connectors). Various combinations of terms were searched and citation titles and abstracts were scanned to determine if the content related to measures of integrated behavioral health care in the focus areas noted above. The results from this search identified several articles.

  

National Quality Measures Clearinghouse

Ambulatory/Office-based care
Behavioral health care
Health care
Patient-centered medical homes
Primary care

Applied Social Sciences Index and Abstracts (ASSIA)

Analysis
Assessment
Measurement
Methodology
Models
Process evaluation
Scales

Collaboration
Collaborative approach
Integrated management
Integrated services
Interdisciplinary approach

Costs

Health care
Mental health
Mental health services
Primary health care
Professionals

Health and Psychosocial Instruments (HAPI)

Assess*
Measure*

Behavioral health
Mental disorders 
Mental health

Health care delivery
Integrat*
Integrated services
Physicians
Primary care
Provider

PsycINFO

Behavioral health
Integrated behavioral health
Health care
Mental health services
Primary health care

Collaboration
Collaborative approach
Integrated services
Interdisciplinary treatment approach

Cost containment
Costs
Health care costs
Health care expenditures

Evaluation
Measurement
Methodology
Statistical analysis

Professionals

PUBMED

Assessment
Methods
Outcome and process assessment

Behavior
Behavioral
Mental disorders
Mental health
Mental health services

Attitude
Cooperative behavior
Integrated
Integrated behavioral health
Integrated delivery of health care
Delivery of health care
Health personnel
Primary health care
Providers

Costs and cost analysis
Economics

 

* truncated term, used to capture various suffixes.

During the second phase of our search strategy, we performed an initial review of the articles from phase one. The results indicated no specific clinical outcome measures for evaluating quality of integrated care existed. Measures in the articles either focused on an individual behavioral health condition or the physical health outcome alone. Several measures focused on the identification of specific health conditions (e.g.-screening), tracking symptoms, patient experience, and processes of care. Measuring the impact of integrated care on clinical outcomes requires an innovative approach where we use a collection of measures to evaluate different dimensions of the clinical care experience.  The team completed a thorough review of the individual measures and the following selection criteria were used: administrative ease for the PCP/ staff, publicly available, commonly used in research, tool to track improvement over time, application to multiple framework domains, and cultural sensitivity/used in diverse populations. The number of promising measures was limited.

The third phase of the literature search required the identification of additional articles on the psychometric properties of the outcomes measures that are based on survey instruments. The IR Center identified the primary source document for each of the proposed measures. The IR Center librarians used HAPI (Health and Psychosocial Instruments), PsycInfo, and PubMed databases to search for the measure names or acronyms.

We evaluated the quality of the measures based on the following factors: sensitivity, specificity, validity, and reliability. Given the limited information on some of these criteria in the articles, we assessed the psychometric properties as available. In order to be selected for the final list of clinical outcome measures, the measure was required to have at least 2 good psychometric properties. From the literature search results, many of the commonly used clinical outcome measures for physical health conditions are not based on brief survey instrument data like the ones for depression, anxiety, or health-related quality of life. Nonetheless, conventional clinical outcomes like controlled blood glucose for Type II diabetes are equally important for assessing quality of integrated care for patients.

Key definitions:
Sensitivity is the ability of a test to correctly classify an individual as ‘with health condition’.
Specificity is the ability of a test to correctly classify an individual as ‘health condition-free’.
Reliability is the extent to which a measure produces consistent results under various conditions.
Content validity refers to the extent to which a measure represents all dimensions of the construct (e.g. personality, ability) that is being measured.