U.S. flag

An official website of the Department of Health & Human Services

AHRQ: Agency for Healthcare Research and Quality

Potential Usefulness of Apps and Other Digital Technologies for Improving Access to Behavioral Health in Primary Care

Purpose

The role of digital technologies in behavioral healthcare, including telehealth and smart phone or computer-based apps, is growing. Persistent workforce shortages in behavioral healthcare — particularly in rural areas — have helped drive the trend toward using technologies to expand the availability and the quality of care. "One in five individuals over the age of 18 experienced a mental health condition in 2021,[1] and one in six individuals over the age of 12 experienced a substance use disorder.[2] Fewer than half of these individuals received the behavioral healthcare they needed.[3] The ability of apps and other digital technologies to connect people and information across time and location makes them a potentially valuable support to efforts to integrate behavioral health and primary care." These developments have been spurred by restrictions during the COVID-19 pandemic, and by the accumulation of evidence for the acceptability and effectiveness of selected technology-assisted approaches to treatment and recovery support.

At the same time, apps and other digital technologies can be challenging to implement and do pose some level of risk to patients with behavioral health conditions. There is no single governmental or private organization that assesses these digital tools and certifies them as to privacy, safety, or effectiveness.[4] Thousands of behavioral health apps are currently available in commercial app stores, but it's been estimated that only 3 to 4 percent are evidence based.[5],[6] While some of these apps have been extensively studied and shown to be efficacious, the vast majority (an estimated 98 percent) have not been systematically evaluated.[7]

This brief provides an overview of the state of the evidence for behavioral health apps and key considerations and concerns related to finding, evaluating, and selecting smartphone and Internet-based behavioral health apps to recommend or prescribe in primary care and integrated care settings. Additionally, this brief outlines how healthcare professionals, policymakers, and researchers can improve and expand the safe and effective use of behavioral health apps in primary care and integrated care settings. While the field of digital healthcare is too young for definitive guidance, this brief aims to provide some basic information on what behavioral apps are, the benefits and risks of recommending apps for your patients, questions to think about when selecting an app, and further sources of information.

Marsch LA. Digital health data-driven approaches to understand human behavior. Neuropsychopharmacology. 2021 Jan;46(1):191-196. https://doi.org/10.1038%2Fs41386-020-0761-5. Accessed May 11, 2023. [adapted]

Behavioral health apps are programs or software applications designed to run on computers and mobile devices, such as phones, tablets, or watches, to support management, treatment, diagnosis, and recovery for people with mental health and substance use disorders, life stressors and crises, and stress-related physical symptoms.

Behavioral health apps are an example of digital medicine. They can:

  • Capture, store, and transmit health data;
  • Support diagnosis, patient monitoring, and self-management;
  • Connect users to behavioral health professionals and peer support;
  • Deliver behavioral health interventions; and
  • Be used with the support of a healthcare professional (guided) or used without human support (unguided and fully automated).

Behavioral health apps that deliver evidence based behavioral health interventions and therapies are called digital therapeutics. A very small subset of digital therapeutics has been evaluated and approved by the FDA for use with a prescription and are called prescription digital therapeutics.[8] Digital therapeutics manufacturers are required to register and list their products with the FDA. However, most digital therapeutic apps are classified as presenting a low to moderate risk to patients, and FDA clearance is not required before marketing those apps in the United States.[9],[10]

Behavioral health apps are designed to target several behavioral health conditions, including:[11]

  • Anxiety disorders;
  • Attention-deficit/Hyperactivity disorder
  • Bipolar disorders;
  • Depressive disorders;
  • Feeding and eating disorders;
  • Obsessive-compulsive and related disorders;
  • Personality disorders;
  • Schizophrenia spectrum and psychotic disorders;
  • Self-harm;
  • Sleep-wake disorders;
  • Substance-related and addictive disorders; and
  • Trauma- and stressor-related disorders.

While most commercially available behavioral health apps are not digital therapeutics or prescription digital therapeutics, they do provide features that can be beneficial, such as psychoeducation, symptom and other health data tracking, goal and habit setting, mindfulness meditation and other coping techniques to patients with behavioral health conditions.[12]

There is no agency responsible for monitoring and evaluating the approximately 20,000 behavioral health apps on the market. However, only an estimated 2 percent of behavioral health apps included in the commercial markets are supported by original peer-reviewed research publications on efficacy or feasibility.[13] Efficacy is used here to refer to findings that the app has been shown to achieve desired outcomes in controlled clinical research settings. Effectiveness is used here to indicate that desired outcomes have been achieved in real-world clinical settings.

The nature of behavioral health apps presents several challenges in research design for controlled and real-world clinical settings, such as establishing appropriate control groups or maintaining blind conditions in randomized control trials (RCTs), that contribute to the limited clinical evidence of efficacy and effectiveness.[14] The existing RCTs, meta-analyses, systematic reviews, and pilot and feasibility studies focus mainly on the small number of behavioral health apps that are considered digital therapeutics and prescription digital therapeutics.[15],[16] Both prescription and nonprescription digital therapeutics have demonstrated modest efficacy for addressing common mental health and substance use disorders, but these findings are impacted by limitations in methodology, design, and sampling.[17]-[57]While behavioral health apps show promising potential, the evidence of their effectiveness and safety is emerging and not definitive.

Below is a sample of available studies on behavioral health apps.

One in five individuals over the age of 18 experienced a mental health condition in 2021,[58] and one in six individuals over the age of 12 experienced a substance use disorder.[59] Fewer than half of these individuals received the behavioral health care they needed.[60] Apps have the potential to improve availability and accessibility of behavioral healthcare services and increase the capacity of integrated behavioral health efforts in primary care.

Behavioral health apps can mitigate common barriers to accessing behavioral health care.

  • Behavioral health apps can expand the capacity of the existing behavioral health workforce. Many behavioral health apps are designed to complement and extend the services provided by the care team, with features such as automated behavioral therapy and learning modules. For example, the U.S. Department of Veterans Affairs (VA) National Center for PTSD offers several apps that, "provide self-help, education and support following trauma."[61] There are also several behavioral health apps that maintain a salaried team of peer support specialists who can monitor and reach out to patients routinely or in times of difficulty, and alert clinicians if needed.[62],[63] These apps allow access to peer and recovery supports that supplement clinical care without the need to hire additional staff in the clinic. This expanded capacity can be especially beneficial in rural areas or at any primary care setting affected by workforce shortages. Some clinicians report providing apps to patients who are on waiting lists or having difficulty accessing healthcare services in their community.[64],[65],[66],[67]
    • Behavioral health apps can serve as an effective adjunct to other ongoing behavioral health treatment. Many apps are intended for use as adjuncts to more traditional in-person or telehealth treatment interventions, and have been found to be effective when used this way.[68] For example, the VA National Center for PTSD offers several "coaching apps" that are intended to serve as "treatment companion apps, for use with a health care provider, to make treatment easier."[69]
    • Some behavioral health apps can be effective when used on a standalone basis. There is some evidence that some behavioral health apps can also be effective when used in place of conventional face-to-face or telehealth treatment.[70],[71] Electronically delivered cognitive behavioral therapy (CBT), for example, was found to be more effective than face-to-face CBT in a 2020 systematic review and meta-analysis.[72]
  • Behavioral health apps can remove implementation barriers for evidence based treatments.
    • Apps can make it possible to virtually deliver several well-established, evidence based behavioral health psychotherapies, such as cognitive behavioral therapy (CBT), motivational interviewing (MI), contingency management (CM), community reinforcement approach (CRA), and acceptance and commitment therapy (ACT). CM, for example, is one of the few effective treatments for stimulant use disorder and other substance use, but can be complex to set up and operate.[73],[74] Apps can provide a ready-made structure for CM implementation, allowing CM to be more easily offered in primary care.[75],[76],[77],[78],[79] Virtual delivery of these interventions can help to address barriers to treatment, like transportation and staffing shortages.[80],[81] However, it is also important to consider limitations related to access to the technology, cost, and cultural and linguistic appropriateness when considering a role for apps.
  • Behavioral health apps can remove engagement barriers and increase the convenience of engaging in behavioral health treatments.
    • Many behavioral health apps allow patients to communicate with behavioral health providers and peer support via text, audio, and video. During the COVID-19 pandemic, individual and group therapy sessions and peer support groups were conducted using apps and phone-connected devices.[82],[83],[84]
    • Some behavioral health apps allow patients to participate in remote drug testing. Some apps and phone-connected devices are able to use self-administered, remote, video-monitored breath and saliva testing for substance use or to confirm medication adherence, reducing the need to travel for treatment and drug testing.[85],[86],[87]
    • Behavioral health apps have high acceptability among patients. Contrary to some initial concerns, most patients find apps acceptable for behavioral health treatment, including patients with little or no prior smartphone experience and older and digitally naïve patient groups.[88],[89],[90],[91],[92],[93]

Multiple organizations have proposed review criteria for behavioral health apps, but no single organization provides definitive information about the thousands of apps available.[94] Until a regulatory body and standardized review system and framework is established and widely adopted, patients and providers will have to make their own assessments.[95],[96] There are several governmental and independent organizations that provide ratings and reviews of some behavioral health apps. Links to some of these app databases and listings are provided in the next section.

Following a review of several notable frameworks for assessing health apps, the Agency for Healthcare Research and Quality has developed the Framework to Assist Stakeholders in Technology Evaluation for Recovery (FASTER) to Mental Health and Wellness. This systematic review has identified several common criteria for assessing behavioral health apps, including clinical evidence of efficacy, data privacy and security, usability and accessibility, functions for remote patient monitoring, interoperability and data sharing, and app use and alignment. The FASTER Framework also provides questionnaires that primary care providers and care team members can use to assess behavioral health apps using these criteria. It should be noted that there is as yet no single organization that is applying the FASTER criteria to the large number of behavioral health apps.

Clinical Evidence of Efficacy

Most of the approximately 20,000 behavioral health apps available in commercial app stores are not evidence based and have not been rigorously tested for efficacy. Efficacy is used here to refer to findings that an app has been shown to achieve desired outcomes in controlled clinical research settings, as implementation in real-world clinical settings presents many challenges. It is important to verify app claims, because false claims are common.[97] Behavioral health apps may claim to be evidence based; but only an estimated 3 to 4 percent actually are.[98] Moreover, apps that claim to use an evidence based theoretical framework may lack published evidence of their efficacy.[99] A 2020 systematic review concluded that, "Only 2.08% (21/1009) of publicly available psychosocial wellness and stress management mobile apps discoverable to self-help seekers have published, peer-reviewed evidence of feasibility and/or efficacy."[100]

  • Is the app based on evidence based behavioral health frameworks, interventions, and approaches? An evidence based app will have fidelity to core elements of the evidence based framework, intervention, or approach it is using. While being evidence based is generally better than not, it is also important to review the published evidence of an app's efficacy.
  • Has the app been shown to be efficacious? Randomized clinical trials (RCTs) and meta-analyses or systematic reviews can provide that information.
    • Check to see that the app(s) included in a study (PDF - 1.03MB) were used in a relevant setting with participants who represent the target population of interest.
    • Take note of research methodology and whether studies were conducted by independent researchers (preferred) or paid for by app developers or other entities with a vested interest in the success of the app.

Data Privacy and Security

Behavioral health apps deal with very sensitive personal information and health data. It is important to understand an app's policies, security, ownership, and funding sources. A systematic assessment of popular behavioral health apps found 44 percent of the studied apps shared personal health information with third parties.[101] Apps may be collecting user data without obtaining consent, or requesting permission to access elements of users' mobile devices without having a privacy policy.[102],[103] Apps may have a privacy policy in place, but may not be in compliance with that policy.[104],[105] Apps may also be sharing data with third parties that are then sharing data with additional entities.[106]

  • Does the app have a transparent privacy policy that is clear and accessible before use?
  • Does the privacy policy disclose the collection, storage, use, and/or transmission of sensitive data? If yes, check to see if the app claims to do so securely and in compliance with industry standards.
  • Are there any potential conflicts of interest that could jeopardize data privacy and security? For example, if an app is free of cost, could it be using a business model that relies on selling patient information and health data?

The resource below provides reviews and ratings for a small number of apps specifically on their policies and practices related to protecting the confidentiality of personal data.

Usability and Accessibility

Ideally, developers involve real patients in the app development process to ensure that the app's features are easy to understand and use. Poor usability and lack of relevant and customizable content are common reasons why users abandon behavioral health apps.[107],[108],[109] Systematic assessments of popular behavioral health apps have found that nearly half or more of the apps assessed did not meet criteria for ease of use, engagement, and customizability; did not have accessibility features; and were only available in English.[110],[111] Only 39 percent of behavioral health apps were completely free.[112] It is important to consider the usability and accessibility of an app before deciding to recommend or prescribe it.

  • Were patients involved in designing and testing the app to make sure it is easy to understand and use? Involving patients and other potential users, including clinicians, in this way is referred to as user-centered design. An app with a user-centered design will be easy for patients and providers to use, and responsive to the goals and needs of users.
  • Does the app have accessibility features? Accessibility features provide the ability to enlarge text size, enable text-to-speech or speech-to-text, access closed captioning, and adjust color schemes.
  • Does the app require access to the internet? Inadequate or unaffordable broadband and data plans and insufficient financial support for those who are un- or underinsured result in disparities in the uptake of digital health technologies.[113],[114],[115]
  • Does the app have offline capabilities? If so, the app will store data when there is no internet connection or phone carrier access, and then transmit the data when a connection is available. This can be an especially important consideration in rural areas with limited broadband.
  • Is the app culturally and linguistically appropriate? Culturally and linguistically appropriate apps allow users to access content that is in the language they are most comfortable with and that reflects their culture.
  • Will the app cost patients and if so, how much? Many behavioral health apps are direct-to-consumer and may require the end user (the patient) to pay out-of-pocket for in-app purchases or paid subscriptions in order to access full functionality.[116],[117]
  • Is there external financial support for app adoption and use? Some state Medicaid programs, managed care organizations (MCOs), and private insurance companies are providing funding to test or support the implementation of particular behavioral health apps.[118],[119]

Functions for Remote Patient Monitoring

Many behavioral health apps use patient-reported outcome measures (PROM) questionnaires and other methods to collect patient data and monitor patient status and outcomes outside of the clinical setting.[120],[121] A systematic assessment of popular behavioral health apps found that mood-tracking, journaling, and symptom-tracking are among the most common patient data inputs, and that 15 percent of the apps assessed were collecting passive data (e.g., biodata or geolocation) from sensors.[122] Data from PROM questionnaires, wearables, and other surveys and sensors can be used to support in-time clinical decision-making; provide real-time feedback and recommendations; personalize behavioral health support; and increase patient engagement.[123],[124],[125],[126]

  • Does the app enable PROM questionnaires? PROM questionnaires, such as the instruments listed in the NIH PROMIS system, PHQ-9 for depression, GAD-7 for anxiety, and Brief Addictions Monitor (BAM) for substance use disorders, can be used in clinical settings to screen for indicators that a patient may have a particular behavioral health condition.
  • Does the app support self-monitoring? Encouraging patients to use PROM measures on a periodic or regular basis via apps can support self-monitoring and assess progress toward recovery.
  • Does the app offer geo-tracking? With appropriate privacy protections and patient permissions, it is possible to use the geo-tracking capability of smartphones to determine if a patient was present for a treatment session or a recovery support group. Some apps have also used geo-tracking to alert the patient if they are approaching self-identified dangerous areas, such as neighborhoods with street drug trafficking, liquor stores, or bars.
  • Does the app send notifications to patients and caregivers? Push notifications and text messages can prompt patients to take medications, and appointment reminders can help to reduce no-shows.
  • Does the app include behavioral health crisis features? Some apps include a help button that can be used at a time of crisis to reach out to a clinician or trusted peer.[127],[128]

Interoperability and Data Sharing

Some behavioral health apps offer users the option to export or share their data with a third party (e.g., a clinician). A systematic assessment of popular behavioral health apps found that 30 percent of the apps assessed allowed users to email or export their data, and 2 percent offered integration with an electronic health record (EHR).[129] Providing users the ability to share data in an accurate and clinically meaningful way may contribute to clinician and patient engagement with the app and the accomplishment of the therapeutic goals.

  • Does the app facilitate data sharing between patients and clinicians? If the PROM or other patient monitoring results are shared with clinicians, the information garnered from the PROMs can also help to improve the responsiveness of treatment. Data sharing provides providers with the clinical information needed to tailor behavioral health treatment session content to issues of immediate relevance to the patient.
  • Does the app aggregate data? Can the PROM data be aggregated across patients at the level of the clinic or the individual clinician to provide indicators of clinical and program effectiveness? Data aggregation is essential for providing measurement-based care, a proven approach to quality improvement (QI).[130] Using apps to collect PROMs data and health indicator data from wearables makes it possible to access PROMs at any time or place and aggregate the data for QI.
  • Does the app have the capacity to integrate with an EHR (PDF - 6.1 MB) or does it have a clinician dashboard that allows behavioral health providers to monitor patient-shared data? Features that minimize the disruption of clinician routine increase the likelihood that clinicians will take advantage of apps.

App Use and Alignment

Behavioral health apps are typically introduced as part of a comprehensive treatment plan that complements other services and supports, rather than used as a standalone intervention. [131]

The following questions are important to consider when planning to recommend or prescribe behavioral health apps to a patient.

  • Is the patient able to use an app and interested in using an app?
  • Is there an evidence based app that can provide treatment for the patient's condition?
    • Identifying the "active ingredient" in the app (the evidence based framework, approach, or intervention it is based on) can provide some perspective on how useful the app may be.
  • Does the app support the patient's therapeutic goals?
    • For example, apps that are abstinence oriented for alcohol use disorders are not going to be a good fit for the patient interested in a harm reduction approach.
  • Is the patient willing to follow the treatment protocol provided in the app?
    • Assessing the patient's interest and confidence in following the treatment plan of the app will be useful in determining the utility of the app as well as help solve any treatment adherence problems.
  • How will the app fit into the patient's overall comprehensive treatment plan?
    • Is the app a central feature to support behavioral change in between visits, or is it primarily educational? Determining the app's place will help providers prioritize how closely they should follow a patient's use of the app.
  • Is the patient already using an app for their behavioral health goals?
    • Screening patients about behavioral health app use and documenting that use provides more comprehensive clinical information and allows for informed decisions during treatment planning. Non-prescription apps are comparable to dietary supplements. While patients self-direct their use, engaging with apps as they feel is appropriate for their needs, it is important to be proactive and aware of what they are using. Going through some of the questions above will be helpful in discussing the utility of these apps with patients.

The resource below provides key considerations for appropriate app utilization.

There are over 20,000 publicly available apps that target behavioral health, with an estimated 900 targeting alcohol and illicit substance use, specifically.[132],[133] Most of these apps are direct-to-consumer and publicly available to download and use from commercial app marketplaces (e.g., the Apple App Store or Google Play Store). Some apps, however, are limited to specific groups (e.g., health insurance members or health system patients) or may require a prescription (i.e., prescription digital therapeutics).

Inclusion in commercial marketplaces does not guarantee that an app has been evaluated for effectiveness or privacy protections, and consumer ratings do not correlate with professional ratings or with evidence of effectiveness.[134],[135],[136] To assist clinicians and consumers with finding and selecting behavioral health apps, several government agencies, professional associations, healthcare systems, and industry trade groups maintain behavioral health specific app listings and databases. Each listing and database use a different methodology and set of evaluation criteria to review and assess behavioral health apps.

The table below includes a sample of the available behavioral health app databases and listings. Be sure to understand the ownership, funding sources, and evaluation methods and criteria of any behavioral health app listing or database before using it to select apps to recommend or prescribe to patients.

Behavioral Health App Databases and Listings

Database/Listing

Organization Name

Organization Type

Evaluation Criteria

mHealth Index & Navigation Database (MIND)

Division of Digital Psychiatry at the Beth Israel Deaconess Medical Center

Collaborative research group in an academic medical center

Based on American Psychiatric Association's App Evaluation Model

One Mind PsyberGuide

One Mind

Non-profit organization

Based on app credibility, user experience, and transparency of privacy practices

Technology Assisted Care for Substance Use Disorders

National Institute on Drug Abuse (NIDA) and the Substance Abuse and Mental Health Services Administration (SAMHSA) Blending Initiative

Federal agency

Based on research findings from the National Drug Abuse Treatment Clinical Trials Network (CTN) trials and other NIDA-funded treatment studies

Department of Veteran Affairs (VA) Mobile Apps for PTSD

VA National Center for PTSD

Federal agency

N/A - Features apps developed by the VA National Center for PTSD

Digital Therapeutics (DTx) Product Library

Digital Therapeutics Alliance (DTA)

Non-profit trade association of companies that manufacture, evaluate, and distribute digital therapeutics

Based on product availability and alignment with the industry's definition and core principles

New Zealand (NZ) Health App Library

Health Navigator Charitable Trust

Non-profit organization

Based on features, functionality, information quality, relevance to NZ audiences, clinical value, and usability

Self-Care App Listing

Kaiser Permanente (KP)

Healthcare provider and non-profit health plan

Based on usability, effectiveness, safety, and confidentiality

Implementing any kind of change in a clinical practice always presents challenges. While there are currently no comprehensive guidance documents on how to use behavioral health apps in primary care settings, common implementation steps emerge from the peer-reviewed literature:[137],[138],[139]

  • Assessing practice, staff, and patient readiness.
  • Identifying champions to build awareness and garner patient and provider support.
  • Allocating and securing sufficient funds for integrating behavioral health apps.
  • Determining how behavioral health apps fit into clinical workflows.
  • Determining if/how to hire or expand staff roles to support integration.
  • Establishing onboarding processes to provide education and training for staff and patients.
  • Establishing practice guidelines for integrating behavioral health apps.
  • Establishing plans for monitoring and addressing safety and privacy.

The following resources can also be adapted for using behavioral health apps in primary care and integrated care settings:

Implementation Guides

Readiness Assessments

Staff Trainings

Several larger healthcare systems have used behavioral health apps following some of these implementation steps.

  • Kaiser Permanente (KP) has taken a two-pronged strategy for implementing apps. They carefully vetted apps and provided training to clinicians on how to use and refer apps to patients (members). They then offered three apps directly to all KP members via the KP website. KP:
    • Drew on education and training materials from app companies to develop organization-specific materials;
    • Created a dedicated internal website for the clinician training program to house all materials; and
    • Created a patient-facing version of the website that houses three free basic mental health and wellness apps available to all KP members without referral.
  • The Digital Psychiatry Division of the Department of Psychiatry at The Beth Israel Deaconess Medical Center uses a model involving the development of special purpose software to share information and connect individual apps. The model builds from a single flexible app that allows symptom assessment, patient input into care planning, and support for measurement-based care. The model also includes the development of educational components for both members of the clinical team and for patients, and the creation of a new care team staff position, "digital navigator".
  • Northwestern University's Center for Behavioral Intervention Technologies has implemented a platform approach that bundles mental health apps into a suite. Each app included in the platform is brief and honed to a single clinical target. While the individual apps are available in commercial app marketplaces, when used in the platform, patients can receive recommendations about which apps may be the best fit for them.
  • Behavioral health apps are programs or software applications designed to run on computers and mobile devices to support management, treatment, diagnosis, and recovery for people with mental health and substance use disorders, life stressors and crises, and stress-related physical symptoms.
  • Integrating behavioral health apps into primary care can increase much-needed access to effective behavioral health services and improve the efficiency of those services.
  • Most behavioral health apps that are available for patient and clinician use have not been systematically evaluated in clinical trials.
  • Behavioral health apps that are considered digital therapeutics or prescription digital therapeutics have an emerging evidence base that suggests efficacy, but there are challenges with implementation and limitations in methodology, design, and sampling of research studies.
  • Before recommending or prescribing a behavioral health app, there are several considerations that need to be made regarding clinical evidence of efficacy, data security and privacy, usability and accessibility, functions for remote patient monitoring, interoperability and data sharing, and app use and alignment within existing treatment goals.
  • There is no single organization that assesses behavioral health apps. Using curated databases and listings when trying to find and evaluate behavioral health apps can narrow the options, versus searching commercial app marketplaces.
  • While there is no established process or comprehensive guidance for using behavioral health apps, several common steps have emerged from the literature and been used to implement behavioral health apps into large health systems.
  • Integrated care team members should familiarize themselves with the basics of behavioral health apps as outlined in this document, because patients are already using them and the potential to increase team capacity is present. Key knowledge base elements include:
    • Evidence base
    • What they can and can't do regarding treatment
    • Risks and benefits
    • Trusted websites and organizations that evaluate behavioral health apps
    • Accessibility
  • Determine the goals of using behavioral health apps, especially those that are digital therapeutics, for both patients and providers, and the targeted patient population.
  • Review how digital therapeutics and other behavioral health apps can be incorporated into your current team's efforts and how these apps will be supported.
  • Identify a clinical champion to advocate for and support this clinical effort.
  • Reach out to existing practices using these tools and learn from their experience.

For Policymakers

  • Develop standardized guidelines for data privacy and security for behavioral health apps.
  • Develop standardized evaluation and review criteria for behavioral health apps.
  • Establish an entity to oversee behavioral health apps.
  • Establish clear funding mechanisms to support research on the implementation of behavioral health apps in primary care.

For Researchers

  • Conduct more clinical research to determine if behavioral health apps in primary care provide improved health outcomes.
  • Conduct more financial analyses to determine if integrating behavioral health apps in primary care is financially sustainable.
  • Conduct more implementation research to determine ways to reduce the burden of incorporating behavioral health apps in primary care.

Research and Policy

References

[1] Results from the 2021 National Survey on Drug Use and Health: Graphics from the Key Findings Report. Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration; 2022. https://www.samhsa.gov/data/sites/default/files/reports/rpt39443/2021_NNR_figure_slides.pdf (PDF - 3.7 MB). Accessed May 15, 2023.

[2] Key Substance Use and Mental Health Indicators in the United States: Results from the 2021 National Survey on Drug Use and Health. HHS Publication No. PEP22-07-01-005, NSDUH Series H-57. Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration; 2022. https://www.samhsa.gov/data/report/2021-nsduh-annual-national-report. Accessed May 15, 2023.

[3] Key Substance Use and Mental Health Indicators in the United States: Results from the 2021 National Survey on Drug Use and Health. HHS Publication No. PEP22-07-01-005, NSDUH Series H-57. Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration; 2022. https://www.samhsa.gov/data/report/2021-nsduh-annual-national-report. Accessed May 15, 2023.

[4] Insel T. Digital mental health care: Five lessons from Act 1 and a preview of Acts 2-5. NPJ Digit Med. 2023 Jan 26;6(1):9. https://doi.org/10.1038/s41746-023-00760-8. Accessed May 15, 2023.

[5] Agarwal S, Jalan M, Wilcox HC, Sharma R, Hill R, Pantalone E, Thrul J, Rainey JC, Robinson KA. Evaluation of Mental Health Mobile Applications. Technical Brief 41. (Prepared by the Johns Hopkins University Evidence-based Practice Center under Contract No. 75Q80120D00003.) AHRQ Publication No. 22-EHC016. Rockville, MD: Agency for Healthcare Research and Quality; May 2022. https://doi.org/10.23970/AHRQEPCTB41. Accessed May 15, 2023.

[6] Lecomte T, Potvin S, Corbière M, Guay S, Samson C, Cloutier B, Francoeur A, Pennou A, Khazaal Y. Mobile apps for mental health issues: Meta-review of meta-analyses. JMIR Mhealth Uhealth. 2020 May 29;8(5):e17458. https://doi.org/10.2196/17458. Accessed May 15, 2023.

[7] Lau N, O'Daffer A, Colt S, Yi-Frazier JP, Palermo TM, McCauley E, Rosenberg AR. Android and iPhone mobile apps for psychosocial wellness and stress management: Systematic search in app stores and literature review. JMIR Mhealth Uhealth. 2020 May 22;8(5):e17798. https://doi.org/10.2196/17798. Accessed June 6, 2023.

[8] Watson A, Chapman R, Shafai G, Maricich YA. FDA regulations and prescription digital therapeutics: Evolving with the technologies they regulate. Front Digit Health. 2023 Apr 17;5:1086219. https://doi.org/10.3389/fdgth.2023.1086219.

[9] Brezing CA, Brixner DI. The rise of prescription digital therapeutics in behavioral health. Adv Ther. 2022 Dec;39(12):5301-5306. https://doi.org/10.1007/s12325-022-02320-0. Accessed June 6, 2023.

[10] Wang C, Lee C, Shin H. Digital therapeutics from bench to bedside. npj Digit Med. 2023;6(38). https://doi.org/10.1038/s41746-023-00777-z. Accessed June 6, 2023.

[11] Agarwal S, Jalan M, Wilcox HC, Sharma R, Hill R, Pantalone E, Thrul J, Rainey JC, Robinson KA. Evaluation of Mental Health Mobile Applications. Technical Brief 41. (Prepared by the Johns Hopkins University Evidence-based Practice Center under Contract No. 75Q80120D00003.) AHRQ Publication No. 22-EHC016. Rockville, MD: Agency for Healthcare Research and Quality; May 2022. https://doi.org/10.23970/AHRQEPCTB41. Accessed May 15, 2023.

[12] Camacho E, Cohen A, Torous J. Assessment of mental health services available through smartphone apps. JAMA Netw Open. 2022 Dec 1;5(12):e2248784. https://doi.org/10.1001/jamanetworkopen.2022.48784. Accessed May 15, 2023.

[13] Lau N, O'Daffer A, Colt S, Yi-Frazier JP, Palermo TM, McCauley E, Rosenberg AR. Android and iPhone mobile apps for psychosocial wellness and stress management: Systematic search in app stores and literature review. JMIR Mhealth Uhealth. 2020 May 22;8(5):e17798. https://doi.org/10.2196/17798. Accessed June 6, 2023.

[14] Wang C, Lee C, Shin H. Digital therapeutics from bench to bedside. npj Digit Med. 2023;6(38). https://doi.org/10.1038/s41746-023-00777-z. Accessed June 6, 2023.

[15] Huh KY, Oh J, Lee S, Yu KS. Clinical evaluation of digital therapeutics: Present and future. Healthc Inform Res. 2022 Jul;28(3):188-197. https://doi.org/10.4258/hir.2022.28.3.188. Accessed May 15, 2023.

[16] Digital Therapeutics: Reducing Rural Health Inequalities. Arlington, VA: Digital Therapeutics Alliance; 2020. https://dtxalliance.org/wp-content/uploads/2021/01/DTA_Rural-Health_r13_110220.pdf (PDF - 1,532 KB). Accessed May 15, 2023.

[17] Goldberg SB, Lam SU, Simonsson O, Torous J, Sun S. Mobile phone-based interventions for mental health: A systematic meta-review of 14 meta-analyses of randomized controlled trials. PLOS Digit Health. 2022;1(1):e0000002. https://doi.org/10.1371/journal.pdig.0000002. Accessed May 15, 2023.

[18] Boumparis N, Schulte MHJ, Riper H. Digital mental health for alcohol and substance use disorders. Curr Treat Options Psych. 2019;6: 352-366 https://doi.org/10.1007/s40501-019-00190-y. Accessed May 15, 2023.

[19] Mohr DC, Azocar F, Bertagnolli A, Choudhury T, Chrisp P, Frank R, Harbin H, Histon T, Kaysen D, Nebeker C, Richards D, Schueller SM, Titov N, Torous J, Areán PA; Banbury Forum on Digital Mental Health. Banbury Forum consensus statement on the path forward for digital mental health treatment. Psychiatr Serv. 2021 Jun;72(6):677-683. https://doi.org/10.1176/appi.ps.202000561. Accessed May 15, 2023.

[20] Eisenstadt M, Liverpool S, Infanti E, Ciuvat RM, Carlsson C. Mobile apps that promote emotion regulation, positive mental health, and well-being in the general population: Systematic review and meta-analysis. JMIR Ment Health. 2021 Nov 8;8(11):e31170. https://doi.org/10.2196/31170. Accessed May 15, 2023.

[21] Comtois KA, Mata-Greve F, Johnson M, Pullmann MD, Mosser B, Arean P. Effectiveness of mental health apps for distress during COVID-19 in US unemployed and essential workers: Remote pragmatic randomized clinical trial. JMIR Mhealth Uhealth. 2022 Nov 7;10(11):e41689. https://doi.org/10.2196/41689. Accessed May 11, 2023.

[22] Lattie EG, Adkins EC, Winquist N, Stiles-Shields C, Wafford QE, Graham AK. Digital mental health interventions for depression, anxiety, and enhancement of psychological well-being among college students: Systematic review. J Med Internet Res. 2019 Jul 22;21(7):e12869. https://doi.org/10.2196/12869. Accessed May 11, 2023.

[23] Leech T, Dorstyn D, Taylor A, Li W. Mental health apps for adolescents and young adults: A systematic review of randomised controlled trials. Child Youth Serv Rev. 2021;127:106073. https://doi.org/10.1016/j.childyouth.2021.106073. Accessed May 11, 2023.

[24] Linardon J, Cuijpers P, Carlbring P, Messer M, Fuller-Tyszkiewicz M. The efficacy of app-supported smartphone interventions for mental health problems: A meta-analysis of randomized controlled trials. World Psychiatry. 2019 Oct;18(3):325-336. https://doi.org/10.1002%2Fwps.20673. Accessed May 11, 2023.

[25] Moberg C, Niles A, Beermann D. Guided self-help works: Randomized waitlist controlled trial of pacifica, a mobile app integrating cognitive behavioral therapy and mindfulness for stress, anxiety, and depression. J Med Internet Res. 2019 Jun 8;21(6):e12556. https://doi.org/10.2196%2F12556. Accessed May 11, 2023.

[26] Firth J, Torous J, Nicholas J, Carney R, Pratap A, Rosenbaum S, Sarris J. The efficacy of smartphone-based mental health interventions for depressive symptoms: A meta-analysis of randomized controlled trials. World Psychiatry. 2017 Oct;16(3):287-298. https://doi.org/10.1002/wps.20472. Accessed May 11, 2023.

[27] Firth J, Torous J, Nicholas J, Carney R, Rosenbaum S, Sarris J. Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials. J Affect Disord. 2017 Aug 15;218:15-22. https://doi.org/10.1016/j.jad.2017.04.046. Accessed May 11, 2023.

[28] Garrido S, Millington C, Cheers D, Boydell K, Schubert E, Meade T, Nguyen QV. What works and what doesn't work? A systematic review of digital mental health interventions for depression and anxiety in young people. Front Psychiatry. 2019;10:759. https://doi.org/10.3389/fpsyt.2019.00759. Accessed May 11, 2023.

[29] Gu J, Miller CB, Henry AL, Espie CA, Davis ML, Stott R, Emsley R, Smits JAJ, Craske M, Saunders KEA, Goodwin G, Carl JR. Efficacy of digital cognitive behavioural therapy for symptoms of generalised anxiety disorder: A study protocol for a randomised controlled trial. Trials. 2020 Apr 23;21(1):357. https://doi.org/10.1186%2Fs13063-020-4230-6. Accessed May 11, 2023.

[30] Olthuis JV, Watt MC, Bailey K, Hayden JA, Stewart SH. Therapist-supported Internet cognitive behavioural therapy for anxiety disorders in adults. Cochrane Database Syst Rev. 2016 Mar 12;3(3):CD011565. https://doi.org/10.1002/14651858.cd011565.pub2. Accessed May 11, 2023.

[31] Richards D, Enrique A, Eilert N, Franklin M, Palacios J, Duffy D, Earley C, Chapman J, Jell G, Sollesse S, Timulak L. A pragmatic randomized waitlist-controlled effectiveness and cost-effectiveness trial of digital interventions for depression and anxiety. npj Digit Med. 2020;3(85). https://doi.org/10.1038/s41746-020-0293-8. Accessed May 11, 2023.

[32] Thabrew H, Stasiak K, Hetrick SE, Wong S, Huss JH, Merry SN. E-Health interventions for anxiety and depression in children and adolescents with long-term physical conditions. Cochrane Database Syst Rev. 2018; 8: CD012489. https://doi.org/10.1002/14651858.CD012489.pub2. Accessed May 11, 2023.

[33] Wang K, Varma DS, Prosperi M. A systematic review of the effectiveness of mobile apps for monitoring and management of mental health symptoms or disorders. J Psychiatr Res. 2018 Dec;107:73-78. https://doi.org/10.1016/j.jpsychires.2018.10.006. Accessed May 11, 2023.

[34] Lewis C, Roberts NP, Bethell A, Robertson L, Bisson JI. Internet-based cognitive and behavioural therapies for post-traumatic stress disorder (PTSD) in adults. Cochrane Database Syst Rev. 2018;12: CD011710. https://doi.org/10.1002/14651858.CD011710.pub2. Accessed May 11, 2023.

[35] Simblett S, Birch J, Matcham F, Yaguez L, Morris R. A systematic review and meta-analysis of e-mental health interventions to treat symptoms of posttraumatic stress. JMIR Ment Health. 2017 May 17;4(2):e14. https://doi.org/10.2196/mental.5558. Accessed May 11, 2023.

[36] Espie CA, Kyle SD, Williams C, Ong JC, Douglas NJ, Hames P, Brown JS. A randomized, placebo-controlled trial of online cognitive behavioral therapy for chronic insomnia disorder delivered via an automated media-rich web application. Sleep. 2012 Jun 1;35(6):769-81. https://doi.org/10.5665/sleep.1872. Accessed May 11, 2023.

[37] Forma F, Pratiwadi R, El-Moustaid F, Smith N, Thorndike F, Velez F. Network meta-analysis comparing the effectiveness of a prescription digital therapeutic for chronic insomnia to medications and face-to-face cognitive behavioral therapy in adults. Curr Med Res Opin. 2022 Oct;38(10):1727-1738. https://doi.org/10.1080/03007995.2022.2108616. Accessed May 11, 2023.

[38] Luik AI, Kyle SD, Espie CA. Digital cognitive behavioral therapy (DCBT) for insomnia: A state-of-the-science review. Curr Sleep Med Rep. 2017;3(2):48-56. https://doi.org/10.1007/s40675-017-0065-4. Accessed May 11, 2023.

[39] Ritterband LM, Thorndike FP, Ingersoll KS, Lord HR, Gonder-Frederick L, Frederick C, Quigg MS, Cohn WF, Morin CM. Effect of a web-based cognitive behavior therapy for insomnia intervention with 1-year follow-up: A randomized clinical trial. JAMA Psychiatry. 2017 Jan 1;74(1):68-75. https://doi.org/10.1001/jamapsychiatry.2016.3249. Accessed May 11, 2023.

[40] Campbell AN, Nunes EV, Matthews AG, Stitzer M, Miele GM, Polsky D, Turrigiano E, Walters S, McClure EA, Kyle TL, Wahle A, Van Veldhuisen P, Goldman B, Babcock D, Stabile PQ, Winhusen T, Ghitza UE. Internet-delivered treatment for substance abuse: A multisite randomized controlled trial. Am J Psychiatry. 2014 Jun;171(6):683-90. Erratum in: Am J Psychiatry. 2014 Dec 1;171(12):1338. https://doi.org/10.1176/appi.ajp.2014.13081055. Accessed May 11, 2023.

[41] Maricich Y, Nunes EV, Campbell ANC, Botbyl JD, Luderer HF. Safety and efficacy of a digital therapeutic for substance use disorder: Secondary analysis of data from a NIDA clinical trials network study. Substance Abuse. 2022;43(1):937-42. https://doi.org/10.1080/08897077.2022.2060425. Accessed May 11, 2023.

[42] Scott CK, Dennis ML, Gustafson DH. Using smartphones to decrease substance use via self-monitoring and recovery support: Study protocol for a randomized control trial. Trials. 2017;18(374). https://doi.org/10.1186/s13063-017-2096-z. Accessed May 11, 2023.

[43] Shah N, Velez FF, Colman S, Kauffman L, Ruetsch C, Anastassopoulos K, Maricich Y. Real-world reductions in healthcare resource utilization over 6 months in patients with substance use disorders treated with a prescription digital therapeutic. Adv Ther. 2022 Sep;39(9):4146-4156. https://doi.org/10.1007/s12325-022-02215-0. Accessed May 11, 2023.

[44] Regmi K, Kassim N, Ahmad N, Tuah NA. Effectiveness of mobile apps for smoking cessation: A review. Tob Prev Cessat. 2017 Apr 12;3:12. https://doi.org/10.18332/tpc/70088. Accessed May 11, 2023.

[45] Whittaker R, McRobbie H, Bullen C, Rodgers A, Gu Y, Dobson R. Mobile phone text messaging and app-based interventions for smoking cessation. Cochrane Database Syst Rev 2019;10: CD006611. https://doi.org/10.1002/14651858.CD006611.pub5. Accessed May 11, 2023.

[46] Carreiro S, Newcomb M, Leach R, Ostrowski S, Boudreaux ED, Amante D. Current reporting of usability and impact of mHealth interventions for substance use disorder: A systematic review. Drug Alcohol Depend. 2020 Oct 1;215:108201. https://doi.org/10.1016%2Fj.drugalcdep.2020.108201. Accessed May 11, 2023.

[47] Hammond AS, Sweeney MM, Chikosi TU, Stitzer ML. Digital delivery of a contingency management intervention for substance use disorder: A feasibility study with DynamiCare Health. J Subst Abuse Treat. 2021 Jul;126:108425. https://doi.org/10.1016/j.jsat.2021.108425. Accessed May 11, 2023.

[48] Miguel AQ, Smith CL, Rodin NM, Johnson RK, McDonell MG, McPherson SM. Automated Reinforcement Management System: Feasibility study findings of an app-based contingency management treatment for alcohol use disorder. Drug Alcohol Depend Rep. 2023 Feb 1;6:100140. https://doi.org/10.1016%2Fj.dadr.2023.100140. Accessed May 11, 2023.

[49] Zill JM, Christalle E, Meyer B, Härter M, Dirmaier J. The effectiveness of an internet intervention aimed at reducing alcohol consumption in adults. Dtsch Arztebl Int. 2019 Feb 22;116(8):127-133. https://doi.org/10.3238/arztebl.2019.0127. Accessed May 11, 2023.

[50] Zhang L, Li N, Li Y, Zhang T, Li D, Liu Y, Liu X, Hao W. Preliminary efficacy of a digital therapeutics smartphone application for methamphetamine use disorder: An experimental study. Front Psychiatry. 2022 Oct 19;13:1027695. https://doi.org/10.3389/fpsyt.2022.1027695. Accessed May 11, 2023.

[51] Cavazos-Rehg PA, Krauss MJ, Costello SJ, Ramsey AT, Petkas D, Gunderson S, Bierut LJ, Marsch LA. Delivering information about medication assisted treatment to individuals who misuse opioids through a mobile app: A pilot study. J Public Health (Oxf). 2020 Feb 28;42(1):149-154. https://doi.org/10.1093/pubmed/fdy207. Accessed May 11, 2023.

[52] DeFulio A, Rzeszutek MJ, Furgeson J, Ryan S, Rezania S. A smartphone-smartcard platform for contingency management in an inner-city substance use disorder outpatient program. J Subst Abuse Treat. 2021 Jan;120:108188. https://doi.org/10.1016/j.jsat.2020.108188. Accessed May 11, 2023.

[53] Maricich YA, Gerwien R, Kuo A, Malone DC, Velez FF. Real-world use and clinical outcomes after 24 weeks of treatment with a prescription digital therapeutic for opioid use disorder. Hosp Pract (1995). 2021 Dec;49(5):348-355. https://doi.org/10.1080/21548331.2021.1974243. Accessed May 11, 2023.

[54] Maricich YA, Xiong X, Gerwien R, Kuo A, Velez F, Imbert B, Boyer K, Luderer HF, Braun S, Williams K. Real-world evidence for a prescription digital therapeutic to treat opioid use disorder. Curr Med Res Opin. 2021 Feb;37(2):175-183. https://doi.org/10.1080/03007995.2020.1846023. Accessed May 11, 2023.

[55] Proctor SL, Rigg KK, Tien AY. Acceptability and usability of a reward-based mobile app for opioid treatment settings: Mixed methods pilot study. JMIR Form Res. 2022 Oct 5;6(10):e37474. https://doi.org/10.2196%2F37474. Accessed May 11, 2023.

[56] Velez FF, Anastassopoulos KP, Colman S, Shah N, Kauffman L, Murphy SM, Ruetsch C, Maricich YA. Reduced healthcare resource utilization in patients with opioid use disorder in the 12 months after initiation of a prescription digital therapeutic. Adv Ther. 2022 Sep;39(9):4131-4145. https://doi.org/10.1007/s12325-022-02217-y. Accessed May 11, 2023.

[57] Voelker R. App aids treatment retention for opioid use disorder. JAMA. 2019 Feb 5;321(5):444. https://doi.org/10.1001/jama.2018.21932. Accessed May 11, 2023.

[58] Results from the 2021 National Survey on Drug Use and Health: Graphics from the Key Findings Report. Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration; 2022. https://www.samhsa.gov/data/sites/default/files/reports/rpt39443/2021_NNR_figure_slides.pdf (PDF - 3.7 MB). Accessed May 15, 2023.

[59] Key Substance Use and Mental Health Indicators in the United States: Results from the 2021 National Survey on Drug Use and Health. HHS Publication No. PEP22-07-01-005, NSDUH Series H-57. Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration; 2022. https://www.samhsa.gov/data/report/2021-nsduh-annual-national-report. Accessed May 15, 2023.

[60] Key Substance Use and Mental Health Indicators in the United States: Results from the 2021 National Survey on Drug Use and Health. HHS Publication No. PEP22-07-01-005, NSDUH Series H-57. Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration; 2022. https://www.samhsa.gov/data/report/2021-nsduh-annual-national-report. Accessed May 15, 2023.

[61] National Center for PTSD, U.S. Department of Veterans Affairs. VA Mobile Apps. 2022. https://www.ptsd.va.gov/appvid/mobile/index.asp. Accessed May 15, 2023.

[62] Ashford RD, Bergman BG, Kelly JF, Curtis B. Systematic review: Digital recovery support services used to support substance use disorder recovery. Hum Behav Emerg Technol. 2020;2(1):18-32. https://doi.org/10.1002/hbe2.148. Accessed May 15, 2023.

[63] Fortuna K, Naslund J, LaCroix J, Bianco C, Brooks J, Zisman-Ilani Y, Muralidharan A, Deegan P. Digital peer support mental health interventions for people with a lived experience of a serious mental illness: Systematic review. JMIR Ment Health 2020;7(4):e16460. https://doi.org/10.2196/16460. Accessed May 15, 2023.

[64] Philippe TJ, Sikder N, Jackson A, Koblanski ME, Liow E, Pilarinos A, Vasarhelyi K. Digital health interventions for delivery of mental health care: Systematic and comprehensive meta-review. JMIR Ment Health. 2022 May 12;9(5):e35159. https://doi.org/10.2196/35159. Accessed May 15, 2023.

[65] Koh J, Tng GYQ, Hartanto A. Potential and Pitfalls of Mobile Mental Health Apps in Traditional Treatment: An Umbrella Review. J Pers Med. 2022 Aug 25;12(9):1376. https://doi.org/10.3390/jpm12091376. Accessed May 15, 2023.

[66] Digital Therapeutics: Reducing Rural Health Inequalities. Arlington, VA: Digital Therapeutics Alliance; 2020. https://dtxalliance.org/wp-content/uploads/2021/01/DTA_Rural-Health_r13_110220.pdf (PDF - 1,532 KB). Accessed May 15, 2023.

[67] Ly KH, Topooco N, Cederlund H, Wallin A, Bergström J, Molander O, Carlbring P, Andersson G. Smartphone-Supported versus Full Behavioural Activation for Depression: A Randomised Controlled Trial. PLoS One. 2015 May 26;10(5):e0126559. https://doi.org/10.1371%2Fjournal.pone.0126559. Accessed May 15, 2023.

[68] Weisel KK, Fuhrmann LM, Berking M, Baumeister H, Cuijpers P, Ebert DD. Standalone smartphone apps for mental health-A systematic review and meta-analysis. NPJ Digit Med. 2019 Dec 2;2:118. https://doi.org/10.1038/s41746-019-0188-8. Accessed May 15, 2023.

[69] National Center for PTSD, U.S. Department of Veterans Affairs. VA Mobile Apps. 2022. https://www.ptsd.va.gov/appvid/mobile/index.asp. Accessed May 15, 2023.

[70] Lecomte T, Potvin S, Corbière M, Guay S, Samson C, Cloutier B, Francoeur A, Pennou A, Khazaal Y. Mobile apps for mental health issues: Meta-review of meta-analyses. JMIR Mhealth Uhealth. 2020 May 29;8(5):e17458. https://doi.org/10.2196/17458. Accessed May 15, 2023.

[71] Johnston DC, Mathews WD, Maus A, Gustafson DH. Using smartphones to improve treatment retention among impoverished substance-using Appalachian women: A naturalistic study. Subst Abuse. 2019 Jul 8;13:1178221819861377. https://doi.org/10.1177/1178221819861377. Accessed May 15, 2023.

[72] Luo C, Sanger N, Singhal N, Pattrick K, Shams I, Shahid H, Hoang P, Schmidt J, Lee J, Haber S, Puckering M, Buchanan N, Lee P, Ng K, Sun S, Kheyson S, Chung DC, Sanger S, Thabane L, Samaan Z. A comparison of electronically-delivered and face to face cognitive behavioural therapies in depressive disorders: A systematic review and meta-analysis. EClinicalMedicine. 2020 Jun 27;24:100442. https://doi.org/10.1016%2Fj.eclinm.2020.100442. Accessed May 15, 2023.

[73] Petry NM. Contingency management treatments: Controversies and challenges. Addiction. 2010 Sep;105(9):1507-9. https://dx.doi.org/10.1111%2Fj.1360-0443.2009.02879.x. Accessed May 15, 2023.

[74] Glass JE, Nunes EV, Bradley KA. Contingency management: A highly effective treatment for substance use disorders and the legal barriers that stand in its way. https://doi.org/10.1377/forefront.20200305.965186. ​​Accessed May 15, 2023.

[75] DeFulio A, Rzeszutek MJ, Furgeson J, Ryan S, Rezania S. A smartphone-smartcard platform for contingency management in an inner-city substance use disorder outpatient program. J Subst Abuse Treat. 2021 Jan;120:108188. https://doi.org/10.1016/j.jsat.2020.108188. Accessed May 15, 2023.

[76] Hammond AS, Sweeney MM, Chikosi TU, Stitzer ML. Digital delivery of a contingency management intervention for substance use disorder: A feasibility study with DynamiCare Health. J Subst Abuse Treat. 2021 Jul;126:108425. https://doi.org/10.1016/j.jsat.2021.108425. Accessed May 15, 2023.

[77] Proctor S, Rigg K, Tien A. Acceptability and usability of a reward-based mobile app for opioid treatment settings: Mixed methods pilot study. JMIR Form Res. 2022;6(10):e37474. https://doi.org/10.2196/37474. Accessed May 15, 2023.

[78] Miguel AQ, Smith CL, Rodin NM, Johnson RK, McDonell MG, McPherson SM. Automated reinforcement management system: Feasibility study findings of an app-based contingency management treatment for alcohol use disorder. Drug and Alcohol Dependence Reports. 2023 Mar 1;6:100140. https://doi.org/10.1016/j.dadr.2023.100140. Accessed May 15, 2023.

[79] Green B, Parent S, Ware J, Hasson AL, McDonell M, Nauts T, Collins M, Kim F, Rawson R. Expanding access to treatment for stimulant use disorder in a frontier state: A qualitative study of contingency management and TRUST program implementation in Montana. J Subst Use Addict Treat. 2023 Apr 13:209032. https://doi.org/10.1016/j.josat.2023.209032. Accessed May 15, 2023.

[80] Brezing CA, Brixner DI. The rise of prescription digital therapeutics in behavioral health. Adv Ther. 2022;39:5301-5306. https://doi.org/10.1007/s12325-022-02320-0. Accessed May 15, 2023.

[81] Substance Abuse and Mental Health Services Administration. Rural behavioral health: Telehealth challenges and opportunities. In Brief. 2022;9(2). https://store.samhsa.gov/sites/default/files/d7/priv/sma16-4989.pdf (PDF - 593 KB). Accessed May 15, 2023.

[82] Academy for Integrating Behavioral Health and Primary Care. COVID-19. 2021. https://integrationacademy.ahrq.gov/products/topic-briefs/telehealth. Accessed May 15, 2023.

[83] Whaibeh E, Mahmoud H, Naal H. Telemental health in the context of a pandemic: the COVID-19 experience. Curr Treat Options Psychiatry. 2020;7(2):198-202. https://doi.org/10.1007/s40501-020-00210-2. Accessed May 15, 2023.

[84] Academy for Integrating Behavioral Health and Primary Care. Telehealth and Behavioral Health Integration. 2021. https://integrationacademy.ahrq.gov/products/topic-briefs/telehealth. Accessed May 15, 2023.

[85] Academy for Integrating Behavioral Health and Primary Care. COVID-19. 2021. https://integrationacademy.ahrq.gov/products/topic-briefs/telehealth. Accessed May 15, 2023.

[86] Whaibeh E, Mahmoud H, Naal H. Telemental health in the context of a pandemic: the COVID-19 experience. Curr Treat Options Psychiatry. 2020;7(2):198-202. https://doi.org/10.1007/s40501-020-00210-2. Accessed May 15, 2023.

[87] Academy for Integrating Behavioral Health and Primary Care. Telehealth and Behavioral Health Integration. 2021. https://integrationacademy.ahrq.gov/products/topic-briefs/telehealth. Accessed May 15, 2023.

[88] Carreiro S, Newcomb M, Leach R, Ostrowski S, Boudreaux ED, Amante D. Current reporting of usability and impact of mHealth interventions for substance use disorder: A systematic review. Drug Alcohol Depend. 2020 Oct 1;215:108201. https://doi.org/10.1016/j.drugalcdep.2020.108201. Accessed May 15, 2023.

[89] Oliveira C, Pereira A, Vagos P, Nóbrega C, Gonçalves J, Afonso B. Effectiveness of mobile app-based psychological interventions for college students: A systematic review of the literature. Front Psychol. 2021 May 11;12:647606. https://doi.org/10.3389/fpsyg.2021.647606. Accessed May 15, 2023.

[90] Patterson Silver Wolf DA, Ramsey AT, Epstein J, Beeler-Stinn S, Deer AAB. Bridges to Sobriety: Testing the Feasibility and acceptability of a mobile app designed to supplement an adolescent substance use disorder treatment program. Clin Soc Work J. 2022 Sep;50(3):308-315. https://doi.org/10.1007/s10615-020-00765-w. Accessed May 15, 2023.

[91] Nicholas J, Fogarty AS, Boydell K, Christensen H. The reviews are in: A qualitative content analysis of consumer perspectives on apps for bipolar disorder. J Med Internet Res. 2017 Apr 7;19(4):e105. https://doi.org/10.2196%2Fjmir.7273. Accessed May 15, 2023.

[92] Chan AHY, Honey MLL. User perceptions of mobile digital apps for mental health: Acceptability and usability - An integrative review. J Psychiatr Ment Health Nurs. 2022 Feb;29(1):147-168. https://doi.org/10.1111/jpm.12744. Accessed May 15, 2023.

[93] Hoffman L, Benedetto E, Huang H, Grossman E, Kaluma D, Mann Z, Torous J. Augmenting mental health in primary care: A 1-Year study of deploying smartphone apps in a multi-site primary care/behavioral health integration program. Front Psychiatry. 2019 Feb 28;10:94. https://doi.org/10.3389/fpsyt.2019.00094. Accessed May 15, 2023.

[94] Agarwal S, Jalan M, Wilcox HC, Sharma R, Hill R, Pantalone E, Thrul J, Rainey JC, Robinson KA. Evaluation of Mental Health Mobile Applications. Technical Brief 41. (Prepared by the Johns Hopkins University Evidence-based Practice Center under Contract No. 75Q80120D00003.) AHRQ Publication No. 22-EHC016. Rockville, MD: Agency for Healthcare Research and Quality; May 2022. https://doi.org/10.23970/AHRQEPCTB41. Accessed May 15, 2023.

[95] Mohr DC, Azocar F, Bertagnolli A, Choudhury T, Chrisp P, Frank R, Harbin H, Histon T, Kaysen D, Nebeker C, Richards D, Schueller SM, Titov N, Torous J, Areán PA; Banbury Forum on Digital Mental Health. Banbury Forum consensus statement on the path forward for digital mental health treatment. Psychiatr Serv. 2021 Jun;72(6):677-683. https://doi.org/10.1176/appi.ps.202000561. Accessed May 15, 2023.

[96] Rickard NS, Kurt P, Meade T. Systematic assessment of the quality and integrity of popular mental health smartphone apps using the American Psychiatric Association's app evaluation model. Front Digit Health. 2022 Sep 29;4:1003181. https://doi.org/10.3389/fdgth.2022.1003181. Accessed May 15, 2023.

[97] Larsen ME, Huckvale K, Nicholas J, Torous J, Birrell L, Li E, Reda B. Using science to sell apps: Evaluation of mental health app store quality claims. NPJ Digit Med. 2019 Mar 22;2:18. https://doi.org/10.1038/s41746-019-0093-1. Accessed May 15, 2023.

[98] Lecomte T, Potvin S, Corbière M, Guay S, Samson C, Cloutier B, Francoeur A, Pennou A, Khazaal Y. Mobile apps for mental health issues: Meta-review of meta-analyses. JMIR Mhealth Uhealth. 2020 May 29;8(5):e17458. https://doi.org/10.2196/17458. Accessed May 15, 2023.

[99] Marshall JM, Dunstan DA, Bartik W. Apps with maps-anxiety and depression mobile apps with evidence-based frameworks: Systematic search of major app stores. JMIR Ment Health. 2020 Jun 24;7(6):e16525. https://doi.org/10.2196/16525. Accessed May 15, 2023.

[100] Lau N, O'Daffer A, Colt S, Yi-Frazier JP, Palermo TM, McCauley E, Rosenberg AR. Android and iPhone mobile apps for psychosocial wellness and stress management: Systematic search in app stores and literature review. JMIR Mhealth Uhealth. 2020 May 22;8(5):e17798. https://doi.org/10.2196/17798. Accessed June 6, 2023.

[101] Camacho E, Cohen A, Torous J. Assessment of mental health services available through smartphone apps. JAMA Netw Open. 2022 Dec 1;5(12):e2248784. https://doi.org/10.1001/jamanetworkopen.2022.48784. Accessed May 15, 2023.

[102] Alfawzan N, Christen M, Spitale G, Biller-Andorno N. Privacy, data sharing, and data security policies of women's mhealth apps: Scoping review and content analysis. JMIR Mhealth Uhealth. 2022 May 6;10(5):e33735. https://doi.org/10.2196%2F33735. Accessed May 15, 2023.

[103] Parker L, Halter V, Karliychuk T, Grundy Q. How private is your mental health app data? An empirical study of mental health app privacy policies and practices. Int J Law Psychiatry. 2019 May-Jun;64:198-204. https://doi.org/10.1016/j.ijlp.2019.04.002. Accessed May 15, 2023.

[104] Tangari G, Ikram M, Ijaz K, Kaafar MA, Berkovsky S. Mobile health and privacy: cross sectional study. BMJ. 2021 Jun 16;373:n1248. https://doi.org/10.1136/bmj.n1248. Accessed May 15, 2023.

[105] Huckvale K, Torous J, Larsen ME. Assessment of the data sharing and privacy practices of smartphone apps for depression and smoking cessation. JAMA Netw Open. 2019 Apr 5;2(4):e192542. https://doi.org/10.1001/jamanetworkopen.2019.2542. Accessed May 15, 2023.

[106] Grundy Q, Chiu K, Held F, Continella A, Bero L, Holz R. Data sharing practices of medicines related apps and the mobile ecosystem: Traffic, content, and network analysis. BMJ. 2019 Mar 20;364:l920. https://doi.org/10.1136/bmj.l920. Accessed May 15, 2023.

[107] Mendiola MF, Kalnicki M, Lindenauer S. Valuable features in mobile health apps for patients and consumers: Content analysis of apps and user ratings. JMIR Mhealth Uhealth. 2015 May 13;3(2):e40. https://doi.org/10.2196/mhealth.4283. Accessed May 15, 2023.

[108] Alqahtani F, Orji R. Insights from user reviews to improve mental health apps. Health Informatics J. 2020 Sep;26(3):2042-2066. https://doi.org/10.1177/1460458219896492. Accessed May 15, 2023.

[109] Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, Stadnick N, Zheng K, Mukamel D, Sorkin DH. Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review. J Med Internet Res. 2021 Mar 24;23(3):e24387. https://doi.org/10.2196/24387. Accessed May 15, 2023.

[110] Camacho E, Cohen A, Torous J. Assessment of mental health services available through smartphone apps. JAMA Netw Open. 2022 Dec 1;5(12):e2248784. https://doi.org/10.1001/jamanetworkopen.2022.48784. Accessed May 15, 2023.

[111] Rickard NS, Kurt P, Meade T. Systematic assessment of the quality and integrity of popular mental health smartphone apps using the American Psychiatric Association's app evaluation model. Front Digit Health. 2022 Sep 29;4:1003181. https://doi.org/10.3389/fdgth.2022.1003181. Accessed May 15, 2023.

[112] Camacho E, Cohen A, Torous J. Assessment of mental health services available through smartphone apps. JAMA Netw Open. 2022 Dec 1;5(12):e2248784. https://doi.org/10.1001/jamanetworkopen.2022.48784. Accessed May 15, 2023.

[113] Roberts ET, Mehrotra A. Assessment of disparities in digital access among medicare beneficiaries and implications for telemedicine. JAMA Intern Med. 2020 Oct 1;180(10):1386-1389. https://doi.org/10.1001/jamainternmed.2020.2666. Accessed May 15, 2023.

[114] Kaihlanen AM, Virtanen L, Buchert U, Safarov N, Valkonen P, Hietapakka L, Hörhammer I, Kujala S, Kouvonen A, Heponiemi T. Towards digital health equity - A qualitative study of the challenges experienced by vulnerable groups in using digital health services in the COVID-19 era. BMC Health Serv Res. 2022 Feb 12;22(1):188. https://doi.org/10.1186%2Fs12913-022-07584-4. Accessed May 15, 2023.

[115] Pew Research Center. Mobile Fact Sheet. 2021 Apr. https://www.pewresearch.org/internet/fact-sheet/mobile/. Accessed May 15, 2023.

[116] Powell AC, Torous JB, Firth J, Kaufman KR. Generating value with mental health apps. BJPsych Open. 2020 Feb 5;6(2):e16. https://doi.org/10.1192/bjo.2019.98. Accessed May 15, 2023.

[117] Camacho E, Cohen A, Torous J. Assessment of mental health services available through smartphone apps. JAMA Netw Open. 2022 Dec 1;5(12):e2248784. https://doi.org/10.1001/jamanetworkopen.2022.48784. Accessed May 15, 2023.

[118] State of California. Recovery Incentives Program: California's Contingency Management Benefit. 2023 Apr. https://www.dhcs.ca.gov/Pages/DMC-ODS-Contingency-Management.aspx. Accessed May 15, 2023.

[119] AHIP. Health Insurance Providers Actions Concerning Mental Health. 2023 May. https://www.ahip.org/news/articles/health-insurance-providers-actions-concerning-mental-health. Accessed May 15, 2023.

[120] Currey D, Torous J. Digital phenotyping data to predict symptom improvement and app personalization: Protocol for a prospective study. JMIR Res Protoc. 2022 Nov 29;11(11):e37954. https://doi.org/10.2196/37954. Accessed May 15, 2023.

[121] Rauseo-Ricupero N, Henson P, Agate-Mays M, Torous J. Case studies from the digital clinic: integrating digital phenotyping and clinical practice into today's world. Int Rev Psychiatry. 2021 Jun;33(4):394-403. https://doi.org/10.1080/09540261.2020.1859465. Accessed May 15, 2023.

[122] Camacho E, Cohen A, Torous J. Assessment of mental health services available through smartphone apps. JAMA Netw Open. 2022 Dec 1;5(12):e2248784. https://doi.org/10.1001/jamanetworkopen.2022.48784. Accessed May 15, 2023.

[123] Currey D, Torous J. Digital phenotyping data to predict symptom improvement and app personalization: Protocol for a prospective study. JMIR Res Protoc. 2022 Nov 29;11(11):e37954. https://doi.org/10.2196/37954. Accessed May 15, 2023.

[124] Rauseo-Ricupero N, Henson P, Agate-Mays M, Torous J. Case studies from the digital clinic: integrating digital phenotyping and clinical practice into today's world. Int Rev Psychiatry. 2021 Jun;33(4):394-403. https://doi.org/10.1080/09540261.2020.1859465. Accessed May 15, 2023.

[125] Teepe GW, Da Fonseca A, Kleim B, Jacobson NC, Salamanca Sanabria A, Tudor Car L, Fleisch E, Kowatsch T. Just-in-time adaptive mechanisms of popular mobile apps for individuals with depression: Systematic app search and literature review. J Med Internet Res. 2021 Sep 28;23(9):e29412. https://doi.org/10.2196/29412. Accessed May 15, 2023.

[126] Dao KP, De Cocker K, Tong HL, Kocaballi AB, Chow C, Laranjo L. Smartphone-delivered ecological momentary interventions based on ecological momentary assessments to promote health behaviors: Systematic review and adapted checklist for reporting ecological momentary assessment and intervention studies. JMIR Mhealth Uhealth. 2021 Nov 19;9(11):e22890. https://doi.org/10.2196/22890. Accessed May 15, 2023.

[127] Teepe GW, Da Fonseca A, Kleim B, Jacobson NC, Salamanca Sanabria A, Tudor Car L, Fleisch E, Kowatsch T. Just-in-time adaptive mechanisms of popular mobile apps for individuals with depression: Systematic app search and literature review. J Med Internet Res. 2021 Sep 28;23(9):e29412. https://doi.org/10.2196/29412. Accessed May 15, 2023.

[128] Dao KP, De Cocker K, Tong HL, Kocaballi AB, Chow C, Laranjo L. Smartphone-delivered ecological momentary interventions based on ecological momentary assessments to promote health behaviors: Systematic review and adapted checklist for reporting ecological momentary assessment and intervention studies. JMIR Mhealth Uhealth. 2021 Nov 19;9(11):e22890. https://doi.org/10.2196/22890. Accessed May 15, 2023.

[129] Camacho E, Cohen A, Torous J. Assessment of mental health services available through smartphone apps. JAMA Netw Open. 2022 Dec 1;5(12):e2248784. https://doi.org/10.1001/jamanetworkopen.2022.48784. Accessed May 15, 2023.

[130] Fortney JC, Unützer J, Wrenn G, Pyne JM, Smith GR, Schoenbaum M, Harbin HT. A Tipping Point for Measurement-Based Care. Psychiatr Serv. 2017 Feb 1;68(2):179-188. https://doi.org/10.1176/appi.ps.201500439. Accessed May 15, 2023.

[131] Weisel KK, Fuhrmann LM, Berking M, Baumeister H, Cuijpers P, Ebert DD. Standalone smartphone apps for mental health-A systematic review and meta-analysis. NPJ Digit Med. 2019 Dec 2;2:118. https://doi.org/10.1038/s41746-019-0188-8. Accessed May 15, 2023.

[132] Agarwal S, Jalan M, Wilcox HC, Sharma R, Hill R, Pantalone E, Thrul J, Rainey JC, Robinson KA. Evaluation of Mental Health Mobile Applications. Technical Brief 41. (Prepared by the Johns Hopkins University Evidence-based Practice Center under Contract No. 75Q80120D00003.) AHRQ Publication No. 22-EHC016. Rockville, MD: Agency for Healthcare Research and Quality; May 2022. https://doi.org/10.23970/AHRQEPCTB41. Accessed May 15, 2023.

[133] Tofighi B, Chemi C, Ruiz-Valcarcel J, Hein P, Hu L. Smartphone apps targeting alcohol and illicit substance use: Systematic search in commercial app stores and critical content analysis. JMIR Mhealth Uhealth. 2019 Apr 22;7(4):e11831. https://doi.org/10.2196/11831. Accessed May 15, 2023.

[134] Hudson G, Negbenose E, Neary M, Jansli SM, Schueller SM, Wykes T, Jilka S. Comparing professional and consumer ratings of mental health apps: Mixed methods study. JMIR Form Res. 2022 Sep 23;6(9):e39813. https://doi.org/10.2196/39813. Accessed May 15, 2023.

[135] de Chantal PL, Chagnon A, Cardinal M, Faieta J, Guertin A. Evidence of user-expert gaps in health app ratings and implications for practice. Front Digit Health. 2022 Feb 17;4:765993. https://doi.org/10.3389/fdgth.2022.765993. Accessed May 15, 2023.

[136] Singh K, Drouin K, Newmark LP, Lee J, Faxvaag A, Rozenblum R, Pabo EA, Landman A, Klinger E, Bates DW. Many mobile health apps target high-need, high-cost populations, but gaps remain. Health Aff (Millwood). 2016 Dec 1;35(12):2310-2318. https://doi.org/10.1377/hlthaff.2016.0578. Accessed May 15, 2023.

[137] Schueller SM, Torous J. Scaling evidence-based treatments through digital mental health. Am Psychol. 2020 Nov;75(8):1093-1104. https://doi.org/10.1037/amp0000654. Accessed May 15, 2023.

[138] Graham AK, Lattie EG, Powell BJ, Lyon AR, Smith JD, Schueller SM, Stadnick NA, Brown CH, Mohr DC. Implementation strategies for digital mental health interventions in health care settings. Am Psychol. 2020 Nov;75(8):1080-1092. https://doi.org/10.1037/amp0000686. Accessed May 15, 2023.

[139] Gordon WJ, Landman A, Zhang H, Bates DW. Beyond validation: Getting health apps into clinical practice. NPJ Digit Med. 2020 Feb 3;3:14. https://doi.org/10.1038/s41746-019-0212-z. Accessed May 15, 2023.