Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Jan 1.
Published in final edited form as: Adm Policy Ment Health. 2016 Jan;43(1):79–92. doi: 10.1007/s10488-014-0621-x

Can Learning Collaboratives Support Implementation By Rewiring Professional Networks?

Alicia C Bunger 1,✉, Rochelle F Hanson 2, Nathan J Doogan 3, Byron J Powell 4, Yiwen Cao 5, Jerry Dunn 6
PMCID: PMC4485606  NIHMSID: NIHMS651916  PMID: 25542237

Abstract

This study examined how a learning collaborative focusing on Trauma-Focused CBT (TF-CBT) impacted advice-seeking patterns between clinicians and three key learning sources: (1) training experts who share technical knowledge about TF-CBT, (2) peers from other participating organizations who share their implementation experiences, and (3) colleagues from their own agency who provide social and professional support. Based on surveys administered to 132 clinicians from 32 agencies, participants’ professional networks changed slightly over time by forming new advice-seeking relationships with training experts. While small, these changes at the clinician-level yielded substantial changes in the structure of the regional advice network.

Keywords: Learning Collaborative, Implementation, Social Networks, Implementation Strategies, Mental Health

Collaborative learning models are strategies that can be used to implement empirically supported treatments (ESTs) within multiple organizations or multiple levels of a healthcare system (Nadeem et al 2013, Ebert et al, 2011). Based on the Institute for Healthcare Improvement’s Breakthrough Series Collaborative, the learning collaborative (LC) model was originally intended as a method for improving care quality within traditional health and medical settings by emphasizing shared learning within and across agency quality improvement teams (Institute for Healthcare Improvement, 2003; Kilo, 1998). A number of core LC components have been identified, such as in-person learning sessions, phone meetings, data reporting, feedback, and the use of process improvement methods (see Nadeem, Olin, Hill, Hoagwood, & Horwitz, 2013). However, LCs remain a “black box,” as the key change mechanisms targeted by these components are unknown (Cretin, Shortell, & Keeler, 2004; Dückers, Spreeuwenberg, Wagner, & Groenewegen, 2009; Mittman, 2004).

Given the emphasis on participant interactions, multi-organizational interventions like LCs may facilitate implementation by altering social networks among participants to promote the transmission of new ideas and social support (Clarke, 2005; Garcia, 2007; Tenkasi & Chesmore, 2003). Social support shared via interactions with external and internal agency colleagues may encourage and communicate expectations to clinicians related to their use of a new treatment (Langley, Nadeem, Kataoka, Stein, & Jaycox, 2010). Technical information transmitted through contact with treatment experts leading the LC may directly influence team members’ knowledge about the treatment and implementation strategies (Nembhard, 2009).

In other studies, opportunities for participants to interact and learn from expert faculty and other participants were perceived as the most helpful LC features (Carter, Ozieranski, McNicol, Power, & Dixon-Woods, 2014; Ebert, Amaya-Jackson, Markiewicz, Kisiel, & Fairbank, 2011; Nembhard, 2009), and the inter-organizational contacts, specifically, were significantly associated with improved team performance (Nembhard, 2011). However, the overall evidence of LCs’ effectiveness in improving care quality and implementing treatment innovations is mixed (Schouten, Hulscher, van Everdingen, Huijsman, & Grol, 2008), and little is known about whether LCs “rewire” social networks in a way that supports implementation. Understanding how LCs influence interpersonal networks and EST implementation processes will allow LCs to be refined to emphasize relationship building components and optimize outcomes, which is important considering collaborative learning models have been criticized as resource intensive (Ovretveit, 2002). Accordingly, this study assessed change in advice-seeking patterns among clinicians who participated in a LC to implement an EST—Trauma-Focused Cognitive Behavioral Therapy (TF-CBT)—for youth (Cohen, Mannarino, & Deblinger, 2006).

Learning Collaboratives as Network Building Implementation Strategies

Generally, clinicians seek advice from a small number of peers with perceived expertise, and often from within the same agency, unit, or work division (Yousefi-Nooraie, Dobbins, & Marin, 2014). Learning collaboratives provide opportunities for participants to access new sources of external expertise (treatment experts, and colleagues at other agencies) while providing opportunities to strengthen relationships with colleagues at their home agency. Collaboratives bring together multiple agency teams typically comprised of a senior leader (e.g., the executive director), clinical supervisor(s), and practitioners. Once selected, teams complete initial pre-work activities and demonstrate readiness to implement to provide a foundation in the EST and enhance the in-person training (i.e., learning sessions), which focuses on teaching mental health practitioners how to deliver the treatment model. Learning sessions are led by faculty experts, and typically last 2–3 days depending on the specific EST. These sessions are interactive and emphasize adult learning principles. They include opportunities for skill practice and behavioral rehearsal, case vignettes and problem-based learning, sharing experiences and expertise, and training in quality improvement strategies, such as Plan-Do-Study-Act (PDSA) cycles (Taylor et al., 2014). In-person learning sessions are interspersed with ‘action periods’ in which practitioners take on training cases and receive ongoing consultation with a treatment expert (usually by telephone on a monthly or bi-monthly basis). Agency teams also conduct small, iterative tests of change, focusing on removing barriers and creating a supportive climate for implementation using PDSA strategies. Consultation calls with faculty experts are intended to promote successful delivery of the EST by providing access to clinical expertise and addressing barriers as practitioners work through treatment cases. Senior leaders also participate in consultation calls, usually on a monthly basis, to discuss ways to strengthen agency infrastructure and support sustained implementation (IHI, 2003; Kilo, 1998).

Changes in the Composition of Clinicians’ Professional Advice Networks (Ego-Network Level)

LC participants have the opportunity to build and strengthen advice-seeking relationships via learning and action activities, such as interactive learning sessions across multiple organizations, collaborative problem-solving at the home agency, and consultation calls with faculty experts. Well-developed relationships are critical in promoting EST implementation. For example, Palinkas and colleagues (2011) note several studies and meta-analyses demonstrating that having trustworthy others in one’s personal network and access to external information increases the rate at which innovative practices are adopted (e.g. Valente, Chou, & Pentz, 2007; Valente & Davis, 1999). Over time, the composition of clinicians’ advice or ‘ego’ networks may change, reflecting greater exposure to individuals who can provide technical knowledge and social support to facilitate implementation. Changes in clinicians’ advice networks may contribute to large-scale changes in the advice sharing network.

LCs may support implementation by altering the composition of clinicians’ professional advice networks in three ways. First, LCs facilitate interactions directly between participants and faculty content experts during the in-person learning sessions, conference calls, and team consultations (Carter et al., 2014). Faculty content experts are a direct source of technical information about new practices and implementation strategies and thus, are critical for promoting successful implementation. In fact, expert coaching was demonstrated to be the most critical LC element in a recent cluster randomized controlled trial (Gustafson et al., 2013).

Second, during the learning sessions, small tests of change using the PDSA cycles at the home agency, and regular case consultations among agency team members, LCs provide a structure for staff within organizations to communicate frequently and directly about an EST. Therefore, over the duration of the learning collaborative, participants may build stronger advice-seeking relationships with colleagues from their home agency. Dense intra-agency communication patterns among clinicians, supervisors, and leaders facilitate innovation implementation (Meyers, Sivakumar, & Nakata, 1999) by disseminating adoption decisions and information about preparations made by other personnel and organizational units for a new practice (West, Barron, Dowsett, & Newton, 1999).

Third, learning sessions and ongoing conference calls provide a forum for clinicians to interact with peers across organizational boundaries. Opportunities to connect with peers outside of the home agency create buy-in, support and energy for the EST, and facilitate opportunities for collaborative problem solving, training, and implementation planning that can enhance full implementation and the routine delivery of new practices (Carter et al., 2014; Glisson & Schoenwald, 2005; Hanf & O’Toole, 1992; Meyers et al., 1999). Therefore agency staff members’ professional advice networks beyond their home agencies also shape the dissemination and interpretation of information that could influence clinicians’ use of TF-CBT.

Changes in the Structure of the Whole Network (Network-Level)

The larger network is derived from relationships between individuals; therefore as clinicians interact with one another and build new relationships, several properties of the larger network may change. The network density or overall connectedness could increase as clinicians form advice relationships with other participants. The advice network may become more centralized especially around faculty experts as clinicians build new relationships, and seek their advice. In highly centralized networks, interactions focus around key players who serve critical roles in the diffusion of information by virtue of their position. Centralization around faculty experts could potentially speed and support implementation by providing knowledge and advice for using and integrating new treatments into practice (Valente, 2010).

Over time, advice-seeking relationships may strengthen, producing small subgroups of well-connected clinicians. As clinicians interact intensively with one another during the LC, they may be more likely to reciprocate advice-seeking and sharing patterns, and connect their colleagues to other sources of advice yielding densely connected subgroups, or clusters (Wasserman & Faust, 1994). In a highly clustered network, some groups may be disconnected from key sources of advice and knowledge, thereby slowing information dissemination across the network (Valente, 2010). However, implementation may be supported by frequent and dense connections within the subgroups especially if they are connected to key sources of expertise. Dense interactions within subgroups may generate consensus, local knowledge, and shared social support for implementing new treatments (Meyers et al., 1999).

Finally, general tendencies towards agency homophily, or seeking advice from a clinician from the same agency, may also decrease as clinicians build relationships with professionals from other agencies participating in the LC. These inter-agency ties may be important external sources of advice for overcoming implementation challenges (Palinkas et al., 2011, 2013).

Study Purpose

During LCs, participants learn about new practices and their implementation through their interactions with experts and colleagues at their own and other agencies. In addition, relationships with peers who are engaged in similar implementation efforts provide social support (Nembhard, 2009). By providing interaction opportunities, LCs may strengthen existing relationships and/or facilitate further relationship development among professional colleagues, essentially changing the composition of a clinician’s professional network and the structure of the larger advice-sharing network. However, the extent to which LCs effectively “rewire” social networks in a way that supports EST implementation has yet to be investigated. This study addresses this gap through two primary aims:

  1. To assess change in the composition of clinicians’ professional advice networks over the duration of a learning collaborative.

  2. To examine how changes in clinician advice-seeking patterns alter the structure of the system-wide network.

Methods

Study Context

This study took place within the context of a naturally-occurring regional initiative to expand trauma-informed evidence-based treatment to children and youth, funded by the county government in a large Midwestern city. A local university-based mental health treatment center was contracted by the county to design and lead a regional LC. The LC leaders chose to focus on implementing TF-CBT given the strong empirical evidence supporting its effectiveness for treating youth with trauma-related mental health problems (Carey & McMillen, 2012).

Behavioral health agencies in the region funded by the county were invited to send implementation teams (3–10 employees comprised of a senior leader, supervisor, and clinicians) to a year-long LC focused on implementing TF-CBT. The LC was free of charge to participants. Agencies voluntarily completed an application that detailed implementation team member qualifications and their agency’s commitment to implementing TF-CBT. To accommodate the 32 agencies that volunteered to participate, four staggered LCs were conducted beginning in April 2011 and ending in September 2012. A team of five therapists who work for the university-based center led the LCs. All five faculty experts were trained by the TF-CBT treatment developers, and three of the trainers completed the national “Train the Trainer” program conducted by the treatment developers.

The LC design was based on the National Center for Child Traumatic Stress (NCCTS) Learning Collaborative model, which incorporates clinical training and coaching with the core features of the IHI’s Breakthrough Series Collaborative learning model focused on quality improvement processes (IHI, 2003; Ebert, Amaya-Jackson, Markiewicz, & Fairbank, 2012). Before the LC began, participants received a “change package” comprised of written TF-CBT and implementation resources. As the LCs commenced, LC faculty experts engaged participants in active learning during three in-person learning sessions, coached them through TF-CBT test cases, and suggested strategies for overcoming implementation barriers using PDSA cycles at their home agencies in between learning sessions. Ongoing support for implementation was also provided via monthly LC conference calls for clinicians, supervisors and senior leaders across organizations, fidelity monitoring (clinicians’ self-reported fidelity metrics), and listservs for information sharing. In addition, LCs incorporated several enhanced features to support TF-CBT implementation beyond the NCCTS and IHI models: local trainers conducted on-site case consultations at agencies, provided additional personal coaching over the phone for clinicians who requested it, and established a roster of clinicians who completed TF-CBT with at least three clients. Together, these components were intended to promote the adoption, spread (penetration) and sustainment of TF-CBT within participating agencies by targeting implementation team members’ knowledge and skills, access to clinical expertise within and across agencies, and organizational support for implementation (Proctor et al, 2011). Consistent with implementation strategy specification guidelines (Proctor, Powell, & McMillen, 2013), each of these components is further described and justified in Table 1. Institutional Review Boards at the first author’s former and current institutions approved and oversaw this research.

Table 1.

Specification of the TF-CBT learning collaboratives (LCs)

Goal: Expand regional capacity to meet the mental health service needs of youth who have experienced trauma by scaling up TF-CBT among behavioral health agencies funded by the county.
Description: The LCs focused on providing clinical training and consultation for clinicians, supervisors, and senior leaders from participating agencies. The LCs also provided training on quality improvement techniques for senior leaders.
Actors:
  • -

    Faculty experts from a local university-based treatment center designed and conducted the LCs, and trained and supported clinicians from other agencies to implement TF-CBT.

  • -

    Agency Implementation Teams (comprised of senior leaders, supervisors, and clinicians) were tasked with implementing TF-CBT.
Specification of LC Components
Actions Target Temporality Dose Outcome Justification*
Preparatory Work
  Prepare Change Package Faculty experts prepare resources on TF-CBT, and implementation strategies Agency implementation team members’ knowledge Before learning sessions Once Adoption, fidelity, penetration, and sustainment of TF-CBT Theoretical: Knowledge (CFIR & TDF); planning (CFIR)
Empirical: Farmer et al. (2011)
  Commitment Implementation team members described their commitment to, and resources allocated for implementing TF-CBT Agency implementation team members’ awareness of their readiness to implement Before learning sessions; before TF-CBT implementation Once Adoption, fidelity, penetration, and sustainment of TF-CBT Theoretical: Leadership engagement; planning (CFIR); intentions; environmental context and resources (TDF)
Active Learning
  Learning Sessions
  • -

    Present information about trauma and TF-CBT practice components

  • -

    Skill practice and behavioral rehearsal

  • -

    Case vignettes and problem-based learning

  • -

    Share experiences, expertise, and lessons learned

 Agency implementation team members’ knowledge, skills, and access to expertise within and outside of their home agency 3 sessions over 12 months (approx. Month 1, Month 3–4, Month 9) Three 2-day sessions Adoption, fidelity penetration, and sustainment of TF-CBT Theoretical: Knowledge (CFIR & TDF); self-efficacy (CFIR); skills; beliefs about capabilities (TDF)
Empirical: Herschell et al. (2010)
  PDSA Cycles Use TF-CBT with test cases, identify barriers, plan strategies to remove barriers, study and refine strategy; support learning within teams; support team members. Agency implementation team members’ knowledge, skills, access to clinical expertise at their home agency; Removes barriers; Promotes supportive organizational climate for TF-CBT. 3 action periods in between learning sessions 12 months total Adoption, fidelity, penetration, and sustainment of TF-CBT Theoretical: Planning; executing; reflecting & evaluating (CFIR); environmental context and resources (TDF)
Empirical: Taylor et al. (2014)
Supports
  Conference Calls Faculty experts organize senior leader, supervisor, and clinician conference calls for participants to share strategies across sites and speed implementation Agency implementation team members’ knowledge, skills, access to clinical and implementation expertise outside of their home agency Monthly (for clinicians and supervisors); every other month (senior leaders) One hour per call Adoption, fidelity, penetration, and sustainment of TF-CBT Theoretical: Comopolitanism; peer pressure, reflecting & evaluating (CFIR); knowledge (CFIR & TDF); social influences (TDF)
Empirical: Palinkas et al. (2013); Palinkas et al. (2014)
  Web Support (Resource library, discussion board, listserv) Share TF-CBT resources across teams Agency implementation team members’ access to resources, collaboration among participants Ongoing As nedded Adoption, fidelity, penetration, and sustainment of TF-CBT Theoretical: Cosmopolitanism; peer pressure, reflecting & evaluating; available resources; access to knowledge and information (CFIR); knowledge (CFIR & TDF); social influences (TDF)
Empirical: Palinkas et al. (2013); Palinkas et al. (2014)
  Quality Improvement (QI) Technique training Faculty experts present information on QI techniques (PDSA cycles) in implementation Agency implementation team members’ ability to lead and manage implementation in their agency; promotes access to implementation expertise Once, during the first learning session, and during on-site visits 1 hour at initial learning session, and as needed during site visits Adoption, fidelity, penetration, and sustainment of TF-CBT Theoretical: Planning; executing; reflecting & evaluating (CFIR); environmental context and resources (TDF)
Empirical: Taylor et al. (2014)
  Metrics Reporting Implementation team members report on test cases including youths’ trauma exposure clinicians’ fidelity to TF-CBT model, client outcomes, and supervision received in test cases Clinicians’ ability to monitor fidelity to the treatment model, and track client outcomes Monthly 3 test cases for clinicians, and 2 test cases for supervisor Adoption, fidelity, penetration and sustainment of TF-CBT Theoretical: Self-efficacy; reflecting & evaluating (CFIR); skills; beliefs about capabilities (TDF)
Empirical: Aarons et al. (2009); Allen et al. (2012); Durlak & DuPre (2008)
Enhanced Features
  Coaching Calls Faculty experts provide clinical consultation and support over the phone to individual clinicians Clinician’s knowledge, skills, and access to clinical expertise Up to twice monthly 1 hour Adoption, fidelity, penetration, and sustainment of TF-CBT Theoretical: Knowledge (CFIR & TDF); self-efficacy (CFIR); skills; beliefs about capabilities (TDF)
Empirical: Beidas et al. (2012); Gustafson et al. (2013); Herschell et al. (2010)
  On-site visits Faculty experts visit participating agency implementation teams Agency implementation team members’ knowledge, skills, access to expertise Twice monthly 1 hour Adoption, fidelity, penetration, and sustainment of TF-CBT Theoretical: Knowledge (CFIR & TDF); self-efficacy; access to knowledge and information (CFIR); skills; beliefs about capabilities (TDF)
Empirical: Beidas et al. (2012); Gustafson et al. (2013)
  Rostering Provides incentive for delivering TF-CBT with fidelity Clinicians’ motivation to deliver TF-CBT By the end of the LC Ongoing Adoption, fidelity, penetration, and sustainment of TF-CBT Theoretical: Peer pressure; organizational incentives & rewards (CFIR) social/professional role and identity; reinforcement; social influences (TDF)
Open in a new tab
*

Note. We highlight the theoretical justification for the different components of the LCs through two widely used theoretical frameworks: the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009) and the Theoretical Domains Framework (TDF; Cane, O’Connor, & Michine,2012). These frameworks highlight many important constructs that may need to be addressed in implementation efforts. We list the primary theoretical constructs addressed by the corresponding components of the LCs. Where possible, we provide references of primary studies or reviews that provide empirical evidence for the LC components.

Participants and Data Collection

Of the 145 clinical and supervisory staff from 32 agencies in the region who completed the TF-CBT LCs, 132 (91%) responded to two surveys administered in person during the first and third learning sessions of each of the four LCs. Most participants were clinicians (67%), followed by supervisors (22%), and senior leaders (10%). Nearly all (98%) held a master’s degree or higher in social work (53%), counseling (28%), psychology (12%), or other helping profession (7%). Participants had substantial professional experience with 65% reporting more than five years in the field, although 43% reported being in their current position for less than one year when they began participating in the learning collaborative (Table 2).

Table 2.

Participant Characteristics (n=132)

n %
Job role
  Senior Leader 13 9.85
  Supervisor 29 21.97
  Clinician 90 67.42
Education
  Bachelor's degree 2 1.52
  Master's degree 121 91.67
  Doctoral degree or equivalent 9 6.82
Discipline
  Social Work 70 53.03
  Psychology 16 12.12
  Education 2 1.52
  Medicine 4 3.03
  Administration/Management 8 6.06
  Addictions Counseling 7 5.30
  Counseling 25 18.94
  Other Mental Health 12 9.09
Experience in the Field
  0–6 months 1 0.76
  6–11 months 6 4.55
  1–3 years 19 14.39
  3–5 years 19 14.39
  more than 5 years 87 65.91
Experience in current job
  0–6 months 28 21.21
  6–11 months 29 21.97
  1–3 years 22 16.67
  3–5 years 17 12.88
  more than 5 years 36 27.27
Open in a new tab

Measures and Analysis

Data on professional advice networks were collected using an ego-network approach. At both survey time points, participants (ego) nominated up to five individuals (alters) from whom they sought professional advice about serving youth with trauma histories. For each alter nominated, participants reported organizational affiliation and rated frequency of communication over the past six months across a seven point Likert scale ranging from “not once” (1) to “many times daily” (7). These procedures identified 422 unique alters who shared practice advice.

We compared the size of each participant’s ego-network, or the number of individuals nominated at the first and third learning sessions using paired sample t-tests. To examine composition, alters nominated by each respondent were categorized as a (1) TF-CBT faculty expert, (2) colleague from their home agency, or (3) peer participating in the LC from another organization. However, respondents nominated a variety of other sources of professional advice including colleagues affiliated with non-participating agencies, private practitioners, other clinical trainers or former professors, and researchers. These alters were categorized as either (4) private practitioner or (5) other. Clinicians’ exposure to each of the five alter-types (the proportion of each participants’ ego-network; Valente, 2010) at the first and third learning session was calculated, and compared using paired-samples t-tests in Stata 13 (StataCorp, 2013).

Next, changes in the structure of the advice-seeking network were examined using a collection of relevant descriptive network statistics. Density captures the proportion of possible ties among individuals that exist, representing the overall degree to which actors are connected. Three isolation measures were calculated: (1) in-isolates, the number of individuals receiving zero nominations, (2) out-isolates, the number of individuals producing zero nominations, and (3) full-isolates, the number of individuals neither receiving nor producing nominations. Two measures of reciprocity were calculated. Reciprocity captures the proportion of all nominations that are reciprocated (i.e., A nominates B, and B nominates A). Weighted reciprocity accounts for the frequency of communication, by capturing the proportion of the communication frequency of all existing ties that is reciprocated. Thus, if A nominates B with a value of 2, and B nominates A with a value of 1, A’s nomination of B is only partially reciprocated. Indegree centralization measures how centralized the network is, and ranges from zero (all nodes receive the same number of nominations), to one (one node is nominated by everyone and no one else is nominated. Clustering (bound between 0 and 1) describes the degree to which the network is composed of relatively small and dense sub-networks, which may be disconnected from or weakly connected with one another. Agency homophily is an odds ratio representing the tendency of nominating a colleague from the home agency. For further details on each of these metrics, see Wasserman and Faust (1994). All of statistics are implemented in either the R “sna” package (Butts, 2014), or in the R “igraph” package (Csardi & Nepusz, 2006). We also plotted the networks to visualize network change over time in the R sna package plotting functions.

Results

Change in Clinicians’ Professional Advice Networks

Table 3 presents results of paired samples t-tests comparing participants’ average network size and exposure to the five professional advice sources at the first and third learning sessions. At the beginning of the LCs, participants nominated an average of 3.9 (SD=1.4) alters as sources of professional advice. Peers from participants’ home agencies were the primary sources of advice, comprising an average of 72% (M=.72, SD=.11) of their ego-networks. “Other” colleagues comprised an average of 15% (SD=23) of ego-networks, followed by private practitioners (M=.06, SD=.14), faculty experts (M=.05, SD=.16), and peers from other agencies participating in the LCs (M=.03, SD=.11).

Table 3.

Ego-Network Size and Exposure to Professional Advice Sources At Learning Sessions 1 and 3: Means (SD) and Results of Paired Samples T-Tests (n=132)

LS 1 LS 3 t
Network Size 3.9 (1.4) 3.6 (1.4) −2.06 +
Composition
Experts 0.05 (.16) 0.20 (.29) 6.60 **
Peers-Home Agency 0.72 (.32) 0.66 (.11) −1.82
Peers-Other LC Participants 0.03 (.11) 0.05 (.11) 1.53
Private Practitioners 0.06 (.14) 0.03 (.08) −3.24 **
Other 0.15 (.23) 0.07 (.18) −3.41 **
Open in a new tab
**

p<.001,

*

p<.01,

+

p<.05

By the end of the LCs, the size of participants’ professional advice networks decreased slightly to 3.6 (SD=1.4); although this change is small, it was significant[t(131)=−2.06, p<.05]. The composition of participants’ advice networks also changed. Exposure to experts increased significantly to an average of 20% [(SD=29), t(131)=6.60, p<.001], while exposure to private practitioners [M=.03, SD=.08, t(131) =−3.24, p<.001] and “others”[M=.07, SD=.18, t(131)= −3.41, p<.001] significantly declined. However, there were no significant changes in exposure to peers at the home agency or to peers from other agencies participating in the LC.

In post-hoc analyses we examined whether changes in ego-network size and exposure to the five advice sources varied by several participant characteristics. There were no significant differences in ego-network change by participant experience in the field, experience at the agency, or prior training. However, results from a one-way analysis of variance (predicting change scores) suggest that change in network size varied by role in the implementation team [F(2, 129) = 3.41, p=.04]. A Tukey test indicates that the size of senior leaders’ networks decreased significantly more than clinicians (p<.01).

Change in Regional Network Structure

Figures 1 and 2 illustrate the professional advice sharing networks, while Table 4 details network diagnostics at the beginning and end of the LCs based on the 422 unique individuals identified during data collection. Over the duration of the LCs, the overall density of the professional advice sharing network decreased slightly (from 0.14 to 0.13) and the number of isolates, or number of individuals with no nominations, increased suggesting that the network was slightly sparser by the end of the LCs than at the beginning. Although this change seems small, a closer inspection of the pair-wise ties among the respondents suggests substantial turnover among advice-relationships. At the beginning of the LCs, there were 525 reported ties among the 422 individuals. By the end of the LC, 287 of those initial ties (55%) dissolved by the while an additional 220 new ties were formed, resulting in a net loss of 67 ties.

Figure 1.

Figure 1

Open in a new tab

Professional Advice Sharing Network at the Beginning of the LCs (LS1) (N=422)

*Note: Each individual is represented as a node (circle), colored nodes are LC participants; diamond shaped nodes are faculty trainers. Lines between nodes represent an advice-seeking relationship. Participants from the same agency have similar colors.

Figure 2.

Figure 2

Open in a new tab

Professional Advice Sharing Network at the End of the LCs (LS3) (N=422)

*Note: Each individual is represented as a node (circle), colored nodes are LC participants; diamond shaped nodes are faculty trainers. Lines between nodes represent an advice-seeking relationship. Participants from the same agency have similar colors.

Table 4.

Descriptive Network Metrics at Learning Sessions 1 and 3

LS1 LS3
n 422 422
Isolates
  Full isolates 74 177
  In-isolates 115 212
  Out-isolates 285 298
Density 0.014 0.013
Centralization (In-Degree) 0.097 0.182
Reciprocal Relationships
  Reciprocity 0.164 0.227
  Weighted Reciprocity 0.188 0.246
Weighted Clustering 0.293 0.346
Agency Homophily 89.077 77.717
Open in a new tab

Figures 1 and 2 illustrate how advice sharing became more concentrated around a central core of actors, especially faculty experts, by the third learning session. Centralization, or the degree to which professional advice sharing is focused around a core group of actors, increased from .097 to .182. Weighted clustering also increased from .293 to .356 suggesting that advice sharing became more clustered into densely connected subgroups.

Both reciprocity measures increased suggesting that advice-seeking relationships strengthened over time. For example, the percentage of reciprocated advice sharing relationships increased from 16% to 23% by Learning Session 3. In addition, weighted reciprocity increased from .188 to .246 suggesting that participants reciprocally seek advice from one another with similar frequency. Agency homophily, the tendency to connect with a peer at the home agency, decreased slightly from 89.08 to 77.72 indicating that participants were slightly more likely to turn to individuals outside of their agency for advice at the end of the LCs than at the beginning.

Discussion

The present study explored whether collaborative learning models alter advice-seeking networks among LC participants focused on implementing TF-CBT. Our findings demonstrate that participants predominantly rely on colleagues from their home agency for professional advice; however, over the duration of the LCs, participants increasingly turned to LC faculty experts and relied less on private practitioners or other sources of professional advice who may not support or have expertise in TF-CBT. As a result, the regional advice network became more centralized around the LC faculty. Advice-seeking among participants also became more reciprocal over time indicating strong patterns of advice sharing. These findings suggest that LCs alter professional advice networks by strengthening and centralizing the flow of technical information and support for implementing an EST from faculty experts to participants, and their colleagues at their home agency.

At both time points, LC participants sought advice predominantly from colleagues at their home agency. This preference for local sources of information and expertise is consistent with other studies demonstrating how clinicians maintain a small, close network of peers (Yousefi-Nooraie et al., 2014). Access to expertise figures prominently in whom workers turn to for information in work contexts (Borgatti & Cross, 2003) and colleagues from the home agency were probably the most accessible sources for case-specific advice at each stage of implementation. By virtue of working in the same office, receiving or seeking regular supervision and consultation, and participating in team meetings, participants have natural opportunities to access their colleagues’ expertise. The accessibility of expertise at home was likely enhanced during the LC as implementation team members interacted more intensively during learning session activities, and PDSA cycles during the action periods. Also, in this study, clinicians’ reliance on colleagues at their home agency may be high since many (42%) were employed at their agency for less than a year. As newer employees, clinicians may have been focused on establishing themselves and identifying appropriate internal consultation sources.

Although colleagues from their home agency were primary sources of advice, participants’ within-agency advice seeking decreased slightly (agency homophily) by the end of the LC, likely due to the formation of new relationships with LC faculty experts. Exposure to faculty experts increased significantly from the first to the third learning session; by the end of the LCs, faculty experts accounted for an average of 20% of clinicians’ ego-networks, suggesting that LCs facilitate access to clinical expertise. Standard LC components including the learning sessions and conference calls provide ample opportunity for participants to seek advice from faculty experts. However, in this study, the faculty experts also provided intensive personal coaching and on-site consultation to support participants’ use of TF-CBT. These LC enhancements may account, in part, for the significant increase in new relationships between participants and local faculty experts.

Clinicians’ access to faculty experts with extensive TF-CBT knowledge can be critical for supporting its implementation. Direct training and ongoing consultation from those with specific expertise in the targeted EST contribute to quality implementation and EST fidelity (Beidas, Edmunds, Marcus, & Kendall, 2012; Herschell, Kolko, Baumann, & Davis, 2010; Nadeem, Gleacher, & Beidas, 2013). Thus, increased advice-seeking from faculty experts may also be associated with treatment fidelity. As participants increasingly relied on faculty experts to provide clinical advice, they relied less on private practitioners or other external individuals who did not participate in the LC. This shift in advice sources suggests that participants decreased their exposure to individuals who may not support or have expertise in TF-CBT. External sources of advice often introduce information about innovations and spur their adoption (Palinkas et al., 2011, 2013). However, in the context of a regional EST scale-up initiative, these external advice sources may not support or have expertise in the chosen EST. Thus, LCs may streamline the flow of technical information and support from key sources (faculty experts) that facilitate implementation, while reducing reliance on external advice sources that may not support or even interfere with implementation efforts.

Although inter-organizational learning is a hallmark of LCs, participants did not appear to form new advice-sharing relationships with colleagues from other agencies. Advice-seeking across agencies may be limited if participants are disengaged, competitive, and reluctant to share information, as in one LC focused on improving stroke care quality (Carter et al., 2014). However, participants in the TF-CBT LCs engaged willingly in the LC components that emphasized inter-agency interactions such as the group activities and discussions during learning sessions. Especially during regular conference calls, senior leaders, supervisors, and clinicians interacted with their counterparts at other agencies, shared experiences with implementation and test TF-CBT cases, and shared advice. These LC components integrated opportunities to learn about one another’s expertise, and form new relationships that could have preceded inter-agency advice-seeking. Yet, our findings suggest that these LC components focused on inter-agency learning do not necessarily lead to clinical advice seeking across agencies.

LC participants might not seek advice from colleagues at other agencies if they have adequate and appropriate access to clinical expertise from supervisors and colleagues at their home agency, and LC faculty experts. While LCs may not generate cross-agency clinical advice seeking, LC components that emphasize inter-agency interactions might still be important for shared learning and implementation especially in community systems. In other LC studies, these components were perceived as helpful (Ebert et al., 2011; Nembhard, 2009) and associated with team performance (Nembhard, 2011). Perhaps LCs facilitate general information-sharing, or other relationships across agencies, but not the development of case-specific advice-sharing ties.

Even though the composition of participants’ professional advice networks changed slightly over the duration of the LCs, the descriptive network metrics demonstrate how these subtle shifts gave rise to notable structural changes in the regional advice network. Over time, the advice network became more centralized especially around the local LC faculty. These findings reinforce the clinician-level results and illustrate how local faculty experts were elevated to central roles in the local advice network by the end of the LCs.

Implications for Administration and Policy

Our results have implications for LC facilitators, local policy makers, administrators, and organizational leaders interested in EST implementation. Those interested in promoting clinical skill development, might select implementation strategies focused on participant-expert interactions including coaching, and enhanced consultation, which could be more cost-effective than the full LC model (Gustafson et al, 2013). Although, it bears mentioning that Gustafson and colleagues focused on strategies for improving patient wait times and retention; future studies are needed to determine whether coaching and consultation are similarly cost-effective for clinical skill acquisition and other EST implementation outcomes.

On the other hand, implementers concerned with supporting broader organizational and system level changes in addition to clinical skills development might modify LCs to include additional components designed to foster inter-agency advice-sharing and collaboration such as inter-agency team meetings, or team building activities. For example, the Community-based Learning Collaborative (CBLC) was developed as part of a South Carolina statewide TF-CBT implementation initiative (Saunders & Hanson, 2014). The CBLC extends the LC model by including broker professionals (i.e., professionals responsible for identification, assessment, treatment referral, and ongoing monitoring of children and families) as well as agency implementation teams within a targeted community. Together, participating clinical and broker professionals develop a community change team and engage in joint in-person training sessions, interactive small group activities, and small tests of change outside of in-person training. Thus, the CBLC is intended to build local capacity to deliver TF-CBT, and facilitate treatment access by building ties between brokers and providers.

Local administrators interested in EST scale up also may consider selecting local experts to lead implementation efforts along with TF-CBT trainers. In this study, and in others (Nadeem, Gleacher, et al., 2013; Schoenwald, Sheidow, & Letourneau, 2004), faculty experts and consultants were an important source of advice for participants as they learned and implemented new treatment models. Accordingly, the regional advice network became centralized around faculty experts. Although centralized networks diffuse information, support and other resources efficiently, highly centralized systems can be destabilized and fragmented easily with the removal of a key player from the network (Gesell, Barkin, & Valente, 2013). In this study, the faculty experts were local therapists/TF-CBT trainers familiar with the community and service delivery system, increasing the chance that they will continue practicing and providing advice to other clinicians in the community even after the LCs (even if intensive expert consultation is time-limited). Fragmentation may be less likely in this advice network compared to networks that form around faculty experts from outside of the community. Therefore, LCs staffed by local experts may produce more sustained network reconfiguration than those led exclusively by external experts, although this warrants further investigation.

Limitations and Future Research Directions

The findings from this study should be interpreted in light of a few limitations. First, the study focused on children’s mental health organizations in one region of the U.S., thereby limiting generalizability to other organizations, systems, and regions. Second, the lack of a control or comparison group limits inferences about the impact of the LCs on networks, or the specific LC components responsible for driving network change. Since this LC included enhanced coaching and consultation in addition to traditional LC components, it cannot be determined if comparable results related to increasing exposure to LC faculty experts may be observed in other LCs. Third, to enhance the feasibility of this type of research, the results are based on clinician self-report rather than objective or confirmed observations of advice-seeking. Fourth, results highlight change in one type of professional relationship (professional advice about serving youth with trauma histories). Given that the majority of participants were clinicians, the survey question focused only on professional clinical advice-seeking. Yet LCs could influence other types of advice-seeking relationships also relevant to implementation that our study could have missed. Future studies should include additional questions related to information sharing, referrals, or implementation-specific advice sharing. Finally, although the response rate is high (91%), missing data may have inflated the degree to which the larger advice sharing network became centralized over the duration of the LCs, or underestimated the findings related to reciprocity (Kossinets, 2006).

Despite these limitations, study findings highlight how social networks among participants may be an important target of LC components and raise new research questions related to the evolution of social networks and their role in EST implementation. Although participants’ professional networks changed over time, it is not clear why or how. Certain LC components (or combinations of components) could be more effective for building social networks than others. Future studies similar to Gustafson et al. (2013) are needed that test and compare the costs and impact of different LC components on participants’ relationships with team members, faculty experts, and other sources of expertise and support. Formation of advice-seeking relationships may also be shaped by participants’ enhanced proximity and access to expertise within and outside of their agency (Borgatti & Cross, 2003). Longitudinal models of social networks can help to identify individual and network characteristics related to the formation and maintenance of key advice seeking ties. Finally, although findings suggest that professional networks evolved during the collaborative, it is not clear whether these changes support implementation success, highlighting the need for research examining the impact of social networks on other implementation and clinical outcomes. With a better understanding of how collaborative learning models build interpersonal networks, how network change influences implementation outcomes, and the types of agencies and clinicians most likely to benefit, LCs can be adapted to emphasize relationship building components most relevant for agencies’ implementation success.

Conclusion

Over the duration of a regional learning collaborative focused in implementing TF-CBT, slight changes in the composition of clinicians’ professional advice networks yielded substantial changes in the structure of the regional advice network supporting the notion that multi-organizational interventions like LCs alter social networks. Collaborative learning models may be particularly useful for building ties between clinicians and faculty experts who play critical roles in transferring specialized practice knowledge and supporting EST implementation. However, to realize the goal of shared learning across organizational boundaries, collaborative learning models may need to incorporate additional strategies that directly target professional relationship formation among clinicians across agencies to increase diversity in knowledge sharing and to increase resiliency of the knowledge sharing network.

Footnotes

**

A version of this paper was presented at the 2014 National Institute of Mental Health Conference on Mental Health Services Research in Bethesda, MD.

Contributor Information

Alicia C. Bunger, College of Social Work, Ohio State University, 1947 College Road, Columbus, Ohio 43210, Bunger.5@osu.edu, Phone: 614-688-8366; Fax: 614-292-7522.

Rochelle F. Hanson, National Crime Victims Research and Treatment Center, Medical University of South Carolina.

Nathan J. Doogan, College of Public Health, Ohio State University.

Byron J. Powell, Center for Mental Health Policy and Services Research, Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania.

Yiwen Cao, College of Social Work, Ohio State University.

Jerry Dunn, Children’s Advocacy Services of Greater St. Louis, University of Missouri-St. Louis.

References

  1. Aarons GA, Sommerfeld D, Hecht DB, Silovsky JF, Chaffin MJ. The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology. 2009;77(2):270–280. doi: 10.1037/a0013223. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Allen IJD, Linnan LA, Emmons KM. Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: Translating science to practice. New York: Oxford University Press; 2012. p. 281. [Google Scholar]
  3. Beidas RS, Edmunds JM, Marcus SC, Kendall PC. Training and consultation to promote implementation of an empirically supported treatment: a randomized trial. Psychiatric Services. 2012;63(7):660–665. doi: 10.1176/appi.ps.201100401. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Borgatti SP, Cross R. A relational view of information seeking and learning in social networks. Management Science. 2003;49(4):432–445. [Google Scholar]
  5. Butts CT. sna: Tools for Social Network Analysis. R package version 2.3-2. 2014 Retrieved from http://cran.r-project.org/package=sna. [Google Scholar]
  6. Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implementation Science. 2012;7(37):1–17. doi: 10.1186/1748-5908-7-37. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Cary CE, McMillen JC. The data behind the dissemination: A systematic review of trauma-focused cognitive behavioral therapy for use with children and youth. Children and Youth Services Review. 2012;34(4):748–757. [Google Scholar]
  8. Carter P, Ozieranski P, McNicol S, Power M, Dixon-Woods M. How collaborative are quality improvement collaboratives: a qualitative study in stroke care. Implementation Science. 2014;9(1):32. doi: 10.1186/1748-5908-9-32. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Clarke N. Transorganization Development for Network Building. The Journal of Applied Behavioral Science. 2005;41(1):30–46. [Google Scholar]
  10. Cohen JA, Mannarino AP, Deblinger E. Treating Trauma and Traumatic Grief in Children and Adolescents. Guilford Press; 2006. p. 256. [Google Scholar]
  11. Cretin S, Shortell SM, Keeler EB. An evaluation of collaborative interventions to improve chronic illness care. Framework and study design. Evaluation Review. 2004;28(1):28–51. doi: 10.1177/0193841X03256298. [DOI] [PubMed] [Google Scholar]
  12. Csardi G, Nepusz T. The igraph software package for complex network research. InterJournal. Complex Systems. 2006 Retrieved from http://igraph.org. [Google Scholar]
  13. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science. 2009;4(50):1–15. doi: 10.1186/1748-5908-4-50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Dückers M, Spreeuwenberg P, Wagner C, Groenewegen PP. Exploring the black box of quality improvement collaboratives: modelling relations between conditions, applied changes and outcomes. Implementation Science. 2009;4(1):74. doi: 10.1186/1748-5908-4-74. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Durlak JA, DuPre EP. Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology. 2008;41:327–350. doi: 10.1007/s10464-008-9165-0. [DOI] [PubMed] [Google Scholar]
  16. Ebert L, Amaya-Jackson L, Markiewicz JM, Fairbank JA. Development and application of the NCCTS learning collaborative model for the implementation of evidence-based child trauma treatment. In: McHugh RK, Barlow DH, editors. Dissemination and Implementation of Evidence-Based Psychological Interventions. New York: Oxford; 2012. [Google Scholar]
  17. Ebert L, Amaya-Jackson L, Markiewicz JM, Kisiel C, Fairbank JA. Use of the Breakthrough Series Collaborative to support broad and sustained use of evidence-based trauma treatment for children in community practice settings. Administration and Policy in Mental Health. 2011:1–13. doi: 10.1007/s10488-011-0347-y. [DOI] [PubMed] [Google Scholar]
  18. Farmer AP, Légaré F, Turcot L, Grimshaw JM, Harvey E, McGowan J, Wolf FM. Printed educational materials: Effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews. 2011;3 doi: 10.1002/14651858.CD004398.pub2. [DOI] [PubMed] [Google Scholar]
  19. Garcia SK. Developing social network propositions to explain large-group intervention theory and practice. Advances in Developing Human Resources. 2007;9(3):341–358. [Google Scholar]
  20. Gesell SB, Barkin SL, Valente TW. Social network diagnostics: a tool for monitoring group interventions. Implementation Science. 2013;8(1):116. doi: 10.1186/1748-5908-8-116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Glisson C, Schoenwald SK. The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research. 2005;7(4):243–259. doi: 10.1007/s11020-005-7456-1. [DOI] [PubMed] [Google Scholar]
  22. Gustafson DH, Quanbeck AR, Robinson JM, Ford JH, Pulvermacher A, French MT, McCarty D. Which elements of improvement collaboratives are most effective? A cluster-randomized trial. Addiction. 2013;108(6):1145–1157. doi: 10.1111/add.12117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Hanf K, O’Toole LJ. Revisiting old friends: networks, implementation structures and the management of inter-organizational relations. European Journal of Political Research. 1992;21(1–2):163–180. [Google Scholar]
  24. Herschell AD, Kolko DJ, Baumann BL, Davis AC. The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. Clinical Psychology Review. 2010;30(4):448–466. doi: 10.1016/j.cpr.2010.02.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. IHI. The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Improvement. Cambridge, MA: 2003. Retrieved from http://www.ihi.org/IHI/Results/WhitePapers/ [Google Scholar]
  26. Kilo CM. A framework for collaborative improvement: Lessons from the Institute for Healthcare Improvement’s breakthrough series. Quality Management in Health Care. 1998;8(4):1–13. doi: 10.1097/00019514-199806040-00001. [DOI] [PubMed] [Google Scholar]
  27. Kossinets G. Effects of missing data in social networks. Social Networks. 2006;28(3):247–268. [Google Scholar]
  28. Langley AK, Nadeem E, Kataoka SH, Stein BD, Jaycox LH. Evidence-Based Mental Health Programs in Schools: Barriers and Facilitators of Successful Implementation. School Mental Health. 2010;2(3):105–113. doi: 10.1007/s12310-010-9038-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Meyers PW, Sivakumar K, Nakata C. Implementation of Industrial Process Innovations: Factors, Effects, and Marketing Implications. Journal of Product Innovation Management. 1999;16(3):295–311. [Google Scholar]
  30. Mittman BS. Creating the Evidence Base for Quality Improvement Collaboratives. Ann Intern Med. 2004;140(11):897–901. doi: 10.7326/0003-4819-140-11-200406010-00011. [DOI] [PubMed] [Google Scholar]
  31. Nadeem E, Gleacher A, Beidas RS. Consultation as an implementation strategy for evidence-based practices across multiple contexts: unpacking the black box. Administration and Policy in Mental Health. 2013;40(6):439–450. doi: 10.1007/s10488-013-0502-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Nadeem E, Olin SS, Hill LC, Hoagwood KE, Horwitz SM. Understanding the components of quality improvement collaboratives: a systematic literature review. The Milbank Quarterly. 2013;91(2):354–394. doi: 10.1111/milq.12016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Nembhard IM. Learning and improving in quality improvement collaboratives: which collaborative features do participants value most? Health Services Research. 2009;44(2 Pt 1):359–378. doi: 10.1111/j.1475-6773.2008.00923.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Nembhard IM. All teach, all learn, all improve? The role of interorganizational learning in quality improvement collaboratives. Health Care Management Review. 2011 doi: 10.1097/HMR.0b013e31822af831. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Ovretveit J. Quality collaboratives: lessons from research. Quality and Safety in Health Care. 2002;11(4):345–351. doi: 10.1136/qhc.11.4.345. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Palinkas LA, Holloway IW, Rice E, Fuentes D, Wu Q, Chamberlain PC. Social networks and implementation of evidence-based practices in public youth-serving systems: a mixed-methods study. Implement Science. 2011;6 doi: 10.1186/1748-5908-6-113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Palinkas LA, Fuentes D, Finno M, Garcia AR, Holloway IW, Chamberlain P. Inter-organizational collaboration in the implementation of evidence-based practices among public agencies serving abused and neglected youth. Administration and Policy in Mental Health and Mental Health Services Research. 2014;41:74–85. doi: 10.1007/s10488-012-0437-5. [DOI] [PubMed] [Google Scholar]
  38. Palinkas LA, Holloway IW, Rice E, Brown CH, Valente TW, Chamberlain P. Influence network linkages across implementation strategy conditions in a randomized controlled trial of two strategies for scaling up evidence-based practices in public youth-serving systems. Implementation Science. 2013;8(133):1–11. doi: 10.1186/1748-5908-8-133. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: Recommendations for specifying and reporting. Implementation Science. 2013;8:139. doi: 10.1186/1748-5908-8-139. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health. 2011;38(2):65–76. doi: 10.1007/s10488-010-0319-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Saunders BE, Hanson RF. Innovative Methods for Implementing Evidence-Supported Interventions for Mental Health Treatment of Child and Adolescent Victims of Violence. In: Reece R, Hanson RF, Sargent J, editors. Treatment of Child Abuse: Common Ground for Mental Health, Medical, and Legal Practitioners. 2nd ed. Baltimore, MD: Johns Hopkins University Press; 2014. [Google Scholar]
  42. Schoenwald SK, Sheidow AJ, Letourneau EJ. Toward effective quality assurance in evidence-based practice: links between expert consultation, therapist fidelity, and child outcomes. Journal of Clinical Child and Adolescent Psychology. 2004;33(1):94–104. doi: 10.1207/S15374424JCCP3301_10. [DOI] [PubMed] [Google Scholar]
  43. Schouten L, Hulscher M, van Everdingen J, Huijsman R, Grol R. Evidence for the impact of quality improvement collaboratives: systematic review. BMJ (Clinical Research Ed.) 2008;336(7659):1491–1494. doi: 10.1136/bmj.39570.749884.BE. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. StataCorp. College Station, TX: StataCorp LP; 2013. Stata Statistical Software: Release 13. [Google Scholar]
  45. Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Quality & Safety. 2014;23(4):290–298. doi: 10.1136/bmjqs-2013-001862. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Tenkasi RV, Chesmore MC. Social networks and planned organizational change: The impact of strong network ties on effective change implementation and use. Journal of Applied Behavioral Science. 2003;39(3):281–300. [Google Scholar]
  47. Valente TW. Social networks and health: models, methods, and applications. Oxford University Press; 2010. [Google Scholar]
  48. Valente TW, Chou CP, Pentz MA. Community coalitions as a system: effects of network change on adoption of evidence-based substance abuse prevention. American Journal of Public Health. 2007;97(5):880–886. doi: 10.2105/AJPH.2005.063644. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Valente TW, Davis RL. Accelerating the Diffusion of Innovations Using Opinion Leaders. The ANNALS of the American Academy of Political and Social Science. 1999;566(1):55–67. [Google Scholar]
  50. Wasserman S, Faust K. Social Network Analysis: Methods and Applications. New York: Cambridge University Press; 1994. [Google Scholar]
  51. West E, Barron D, Dowsett J, Newton J. Hierarchies and cliques in the social networks of health care professionals: implications for the design of dissemination strategies. Social Science & Medicine. 1999;48(5):633–646. doi: 10.1016/s0277-9536(98)00361-x. [DOI] [PubMed] [Google Scholar]
  52. Yousefi-Nooraie R, Dobbins M, Marin A. Social and organizational factors affecting implementation of evidence-informed practice in a public health department in Ontario: a network modelling approach. Implementation Science. 2014;9(1):29. doi: 10.1186/1748-5908-9-29. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES