Melissa Tremblay, Lola Baydala, Natasha Rabbit, Jennilee Louis, and Kisikaw Ksay-yin
The aim of the current paper is to discuss the use of Outcome Mapping as a tool for evaluating community and stakeholder changes that occurred when a prevention program was culturally adapted and delivered through a community-university partnership. To the authors’ knowledge, this paper represents the first account of using Outcome Mapping as an evaluation tool in a Canadian Indigenous context. Changes in the behavior, actions, activities, and relationships of five boundary partners were retrospectively documented using the tool. Data demonstrated positive impact on Elders and students; growing community investment in and support for the Maskwacis Life Skills Training program’s cultural components; progressive increases in community ownership of the program; and growth in the community-university partnership. Overall, Outcome Mapping provided a systematic method for understanding peripheral changes that are often overlooked in conventional research and evaluation, but that nonetheless indicate progress toward community changes and long-term impact.
Growing evidence for the value of community-based participatory research (CBPR) (Israel, Schulz, Parker, & Becker, 1998; Minkler & Wallerstein, 2003) has resulted in a proliferation of studies that utilize this approach (Jagosh, Macaulay, Pluye, Salsberg, Bush, Henderson, Sirett, Wong, Cargo, Herbert, Seifer, Green, Greenhalgh, 2012). With an emphasis on translating research findings for use in communities, bidirectional learning and capacity building, equitable involvement, and honoring multiple forms of knowledge, CBPR does away with the conventional hierarchy between researchers and those being researched. In this way, CBPR is particularly suitable for work with Indigenous communities (Castleden, Morgan, & Lamb, 2012; Cross, Friesen, Jivanjee, Gowen, Bandurraga, Matthew, & Maher, 2011; Gauld, Smith, & Kendall, 2011). Indeed, the option of community participation in research affecting Indigenous peoples is recognized as an ethical imperative (Canadian Institutes of Health Research, 2007). When community control and agency are emphasized and Indigenous community members contribute in meaningful ways to the research process, local capacity is strengthened and community-level changes can occur (Kelly, Saggers, Taylor, Pearce, Massey, Bull, Odo, Thomas, Billycan, Judd, Reilly, & Ahboo, 2012; Salimi, Shahandeh, Malekafzali, Loori, Kheiltash, Jamshidi, Frouzan, & Majdzadeh, 2012). The task of evaluating and measuring community-level change is not simple or straightforward. Although the randomized controlled trial has long been recognized as the gold standard for academic research, experimental approaches are often inappropriate for evaluating programs that take place in community settings (Judge & Bauld, 2001; Victora, Habicht, & Bryce, 2004). Kelly (2010) notes that communities are complex and constantly evolving; as such, evaluations of community-based initiatives cannot adhere to a single theory or model, and cannot test direct cause-and-effect relationships as in a closed system. In response to concerns with utilizing experimental designs to evaluate complex community initiatives, innovative evaluation approaches are beginning to emerge (Dart & Davies, 2003). Outcome Mapping (Earl, Carden, & Smutylo, 2001) represents one such approach. The aim of the current paper is to discuss the use of the tool for evaluating community and stakeholder changes that occurred when a substance abuse and violence prevention program was culturally adapted and delivered through a community-university partnership. To the authors’ knowledge, this paper represents the first study that uses Outcome Mapping as an evaluation tool in a Canadian Indigenous context.
The Community and the Project
The Maskwacis community, located in central Alberta, Canada, is made up of four Cree Nations: Louis Bull, Montana, Samson, and Ermineskin. The four nations have a combined population of approximately 15,000. These neighboring nations function independently, with separate chief and councils, education directors, and schools. Approximately 53% of the Maskwacis population is under the age of 18 (Grekul & Sanderson, 2011). As a result of colonization and its ongoing impact on Indigenous peoples, high rates of substance abuse and acts of violence pose a challenge for many First Nations communities in Canada. In response to these challenges, Maskwacis community members invited University of Alberta researchers to partner with them to culturally, adapt, deliver, and evaluate a substance abuse and violence prevention program to children and youth in Maskwacis schools. In order to realize this goal, a community-university partnership was formed in the spirit of CBPR. The first stage of the project involved culturally adapting the evidence-based Life Skills Training (Botvin, Baker, Renick, Filazzola, & Botvin, 1984; Botvin & Griffin, 2014) program to incorporate Cree culture, language, and values. Elders and community staff led the cultural adaptation process (Baydala, Fletcher, Tremblay, Rabbit, Louis, Ksay-Yin, & Sinclair, 2016). The adapted Maskwacis Life Skills Training (MLST) program was subsequently delivered for three years in Maskwacis schools by community program facilitators.
Evaluation of the MLST program was carried out over three years of program implementation. The evaluation consisted of (1) focus groups conducted at the end of each program year with school personnel, Elders, program facilitators, parents, and students to obtain feedback on the program, discuss suggestions for improvement, and understand the successes and challenges of the program; and (2) pre- and post- program questionnaires distributed to student participants (National Health Promotion Associates, 2011a, 2011b). By the third year of implementation, it was clear that MLST program effects were extending into the wider community. Partners were aware of a web of individual and community-level outcomes that could not be comprehensively captured with questionnaires or focus groups alone. Further, program facilitators often shared meaningful stories about program impact, but did not feel that evaluation processes allowed for documenting this informal data. Accordingly, Outcome Mapping (Earl et al., 2001) was utilized to supplement the evaluation of the MLST program. This paper will share project findings documented through Outcome Mapping, describe the use of Outcome Mapping as a retrospective evaluation tool, and discuss the suitability of this method for use in a CBPR partnership as well as with Indigenous communities.
Outcome Mapping was developed in Canada by the International Development Research Centre, and was released to researchers and practitioners in 2001. Since that time, Outcome Mapping has mainly been used in Africa, Latin America/ Caribbean, and Asia, with only 2% of reported Outcome Mapping use in North America (Smith, Mauremootoo, & Rassmann, 2012). Although the tool was created for use in an international development context, research has documented its potential applicability to more diverse settings, including economically developed countries (Smith et al., 2012).
Outcome Mapping is a participatory approach to planning, monitoring, and evaluation. It represents a shift from defining results in terms of long-term impact to defining results in terms of observable changes in partners’ behaviors, actions, activities, and relationships. Although long-term impact is the ultimate goal of community-based projects, Outcome Mapping recognizes the importance of also tracking smaller-scale, incremental changes that indicate meaningful progress. An exclusive focus on impact can preclude a focus on these incremental changes, “severely limiting…potential for understanding how and why impact occurs” (Earl et al., 2001, p. 6). Further, the complex nature of community initiatives causes difficulty in linking large-scale impacts to discrete projects or initiatives. For this reason, Outcome Mapping acknowledges that the contributions of multiple projects, programs, organizations, and individuals are necessary to achieve large-scale impact. The work of a single initiative in isolation is not sufficient to achieve social change. Outcome Mapping is therefore useful in focusing on a project’s contributions to outcomes and impacts rather than attempting to attribute outcomes and impacts to one particular project.
The Outcome Mapping manual (Earl et al., 2001) describes 12 steps divided into three stages. The first stage is Intentional Design. This is a planning stage during which a project defines the changes it will aim to bring about as well as the strategies that will be used to contribute to the change process. The second stage is Outcome and Performance Monitoring. During this stage, monitoring priorities are identified, and methods are provided for monitoring progress toward desired outcomes. The third and final stage is Evaluation Planning. Appendix A (Page 67) lists the three stages and the steps required. Moving through the stages requires a series of workshops, with workshop implementation instructions provided in the manual.
Our Approach to Outcome Mapping
Recent research indicates that the procedure is especially relevant when adapted to meet the needs of an individual project (Smith et al., 2012). In practice, many researchers have employed the tool as part of a retrospective evaluation (Nyangaga, Smutylo, Romney, & Kristjanson, 2010). In this case, only the first stage, Intentional Design, is applicable. Stages 2 (Outcome and Performance Monitoring) and 3 (Evaluation Planning) are applicable when Outcome Mapping is used to prospectively plan, monitor, and evaluate a project.
For the current project, Outcome Mapping was used as a retrospective evaluation tool. From January to December 2013, approximately 150 hours were dedicated to workshops designed to move through the required steps described in stage one. These workshops were facilitated by the team’s program evaluator and always took place in the community setting. Workshop participants consisted of four academic team members, seven community team members, and two community Elders; this team is referred to as the working group committee (WGC).
When the WGC began the process, the project’s mission and vision statements had already been established (see Appendix B). Subsequent steps were followed in order, with boundary partners, outcome challenges, progress markers, and strategies developed by the WGC. After generating retrospective progress markers, the evaluator examined a number of data sources to collect evidence for progress. In particular, the evaluator examined minutes from four years of weekly WGC meetings; bi-annual reports to the project’s funder; transcripts from post-program focus groups conducted over three years; facilitator reports completed after each MLST class session over three years; and pre and post program questionnaires completed by MLST students over three years. These data were systematically extracted, coded, and classified according to the identified progress markers. After data were extracted and compiled, the WGC engaged in a series of meetings to discuss findings. These discussions allowed the WGC to reflect on areas where progress had advanced considerably as well as areas where less progress had occurred.
After progress markers were identified, workshops focused on identifying the strategies used to achieve the outcome challenges. In consultation with a practitioner, the WGC decided not to apply the strategy categorization process described in the Outcome Mapping manual. Instead, the evaluator separated identified strategies into those that the WGC had already successfully employed and those that would be useful in moving forward. In this way, the technique was not only useful as a retrospective evaluation tool, but was informative for ongoing program planning and improvement. The WGC decided not to employ Step Seven: Organizational Practices, as it was determined that the first six steps provided sufficient information and detail for the retrospective evaluation.
Changes in the behavior, actions, activities, and relationships of five boundary partners including Elders, leadership/education directors, schools, community and university partners were documented using the first stage. Changes in the student boundary partner were also documented, but were included in a separate paper (Baydala et al., 2016). These changes indicated meaningful progress toward desired outcomes. Consistent with the program’s vision and mission statements, outcome challenges and examples of progress markers are presented below for each of the five boundary partners. Examples of progress markers are presented in table form along with a summary of changes to support each progress marker. In keeping with guidelines outlined in the manual, progress markers were divided according to changes that the WGC expected to see (indicating an early response to project activities), liked to see (indicating more active engagement of boundary partners), and loved to see (indicating profound change in boundary partners). Finally, strategies for achieving progress markers are summarized for each boundary partner.
As in other Cree communities, Maskwacis Elders are held in the highest regard and act as community educators, historians, and storytellers. Because Cree culture and language were at the heart of the MLST program, Elders were critically important to the program’s planning and implementation. Following from the program’s vision toward healthy First Nation communities, the outcome challenge identified for Elders was to strengthen relationships between Elders and youth. Table 1 lists progress markers and a summary of changes for this outcome challenge.
Outcome Mapping allowed the WGC to evaluate a bi-directional learning relationship between the MLST program and community Elders. In particular, not only was the MLST program founded on the knowledge and direction of Elders; the WGC was also promoting behavioral changes in Elders by encouraging the formation of Elder-youth relationships in the community.
The primary strategy toward building relationships between Elders and youth was to invite Elders into Maskwacis classrooms as part of the MLST program. When Elders could not be physically present during MLST classes, digital stories, narrated in Cree by community Elders, were used to reinforce cultural teachings. Also, facilitators deliberately emphasized the importance of respecting community Elders and created opportunities for Elders and youth to interact in positive ways both in the classroom and through extracurricular events including a hide tanning cultural camp. In addition to allowing the WGC to identify existing strategies for promoting Elder-youth relationships, Outcome Mapping prompted the WGC to generate ideas for additional relationship-building activities. For example, the WGC planned to implement a tea and bannock day where students could invite their grandparents to engage in storytelling at their schools.
The Maskwacis four Nations are governed by separate leaders (i.e., chiefs and council members), and have separate education directors. In Maskwacis as in other First Nation communities, formal approval from community leaders during the project’s initiation was required; this approval was granted in the form of band council resolutions from each Nation. The outcome challenge here was to reawaken the spirit of our leaders’ and education directors’ Nehiyaw mamitonecikan (i.e., Cree spirit). Progress markers are listed in Table 2.
Toward the outcome challenge of reawakening their Cree spirit, leaders and education directors were strongly supportive of the MLST program’s goal to promote culture in Maskwacis schools.
Although communicating with leaders was difficult due to their busy schedules and demanding jobs, the WGC did establish and maintain ongoing relationships with a number of prominent community members, resulting in one chief taking on an advocacy role in support of the program, and in all leaders and education directors signing letters in support of the program.
In order to facilitate progressive changes in leaders and education directors, community staff employed a number of relationship building strategies. In particular, leaders and education directors were invited to all MLST events, were given MLST newsletters, and were provided with a comprehensive business plan. Community staff also regularly attempted to schedule meetings to provide updates. Regarding prospective strategies, the WGC committed to provide leaders and education directors with monthly updates, and to invite leaders from all four Nations to meet as a collective.
Evaluating leaders and education directors as boundary partners allowed the WGC to recognize how important they were to program sustainability and to facilitating positive community change. As part of the process, the WGC discussed that it was important to maintain relationships with leaders and education directors beyond obtaining band council resolutions and formal written approval. In order for the program to thrive and have a community-wide impact, it was necessary for the WGC to provide leaders and education directors with multiple opportunities to become familiar with and involved in the MLST program.
While the support of community leaders and education directors was necessary to allow the MLST project in the Maskwacis community, the support of school personnel was necessary to allow for the project to experience success in Maskwacis schools. The outcome challenge for schools was to support and promote the program in the school environment, particularly the cultural aspects of the program. Progress markers and a summary of changes are listed in Table 3.
The above summary highlights that school personnel became increasingly welcoming, supportive, and interested in the MLST program and the program’s cultural teachings. During the first year of the program, teachers were hesitant to accept the program, rarely supported facilitators with classroom management, and reported a number of issues with facilitator reliability. By the third year, teachers provided expressly positive feedback about facilitators and the program, allowed extra MLST class time, began to advocate for the program, and incorporated cultural teachings into core subjects. Outcome Mapping allowed for the WGC to identify these progressive changes in school personnel.
To promote these changes, a number of strategies were utilized. An important strategy at the outset of the project was to invite schools to participate in program training delivered to facilitators. This allowed for schools to gain an understanding of the program before it began. Additionally, schools were given the opportunity to participate in focus groups at the end of each project year in order to provide feedback. Each year, this feedback led to additional strategies being employed for schools. In particular, the second and third years of the program saw more emphasis placed on facilitators being punctual and reliable, facilitators communicating with teachers, and facilitators being present in schools outside of MLST class time. Evaluation updates and promotional material were also provided to schools as a means of keeping school personnel informed about the program. Through Outcome Mapping, the WGC was able to identify and reflect on each of these strategies and the extent to which they were successful with schools.
Using the technique, a number of prospective strategies were also identified for schools. Primarily, because all three years of the program saw low attendance from school personnel at MLST events, the WGC determined that they needed to employ more targeted efforts to involve schools in MLST events. Additionally, it was determined that facilitators should spend time in their classrooms before the MLST program began in order to develop rapport with students and teachers and to establish mutual expectations. The WGC also committed to sharing a facilitator code of conduct with teachers, providing more regular updates to school principals, and annually revisiting the program’s memorandum of understanding with schools. Finally, unlike schoolteachers, facilitators were not University educated; rather, they were well-informed and knowledgeable regarding their culture and language. It was important for teachers to understand and respect the cultural qualifications of facilitators from the outset of the project. Accordingly, it would be important for the MSLT program to more clearly communicate with school personnel regarding the qualifications of facilitators at the beginning of the school year. Overall, Outcome Mapping provided a means for the WGC to generate prospective strategies in a systematic way that allowed for targeting areas where additional progress could be made.
Community partners (Table 4) included both MLST facilitators and program administrative staff. The outcome challenge for community staff was to practice, promote, and support a Nehiyaw (i.e., Cree) worldview through the program.
In identifying and evaluating progress markers for community partners, the WGC began to recognize the complexity of community partners’ roles, which extended beyond delivering the MLST program to students. These included developing their leadership capacity, as well as building relationships with Elders, students, schools, other MLST staff, the wider community, and university partners.
It was important to clarify strategies that facilitated the success of community partners in their complex roles. Strategies to this end were focused on promoting culture within the workplace. In particular, the job descriptions of community staff emphasized a significant cultural component, and it was clearly communicated to community staff that they were expected to spend time each day engaging with and learning from Elders. Additionally, community partners were permitted to participate in ceremonies and other cultural events during work hours, and took part in multiple culturally relevant professional development opportunities. To the extent that community partners were able to strengthen their cultural knowledge and connection with Elders, relationship building and accountability were similarly strengthened.
Moving forward, the WGC determined that it would be important to regularly revisit the staff code of conduct and handbook to ensure consistent staff expectations and standards. Another prospective strategy was to focus more on enhancing Cree language skills, and relatedly, to institute a Cree naming ceremony for all community staff.
University partners (Table 5) included the project’s principal investigator, research coordinators, research assistants, and a program evaluator. For university partners, the outcome challenge was to practice authentic CBPR.
The mapping technique highlighted that University partners experienced significant growth in their capacity to practice authentic CBPR. This was made possible by community partners’ willingness to educate university partners, which in turn facilitated a strong and trusting partnership.
A number of strategies were relevant to university partners. Perhaps most importantly, university partners were physically present in the community as often as possible. The WGC held weekly meetings in the community; accordingly, university partners were present in the community at least once per week. Having regular meetings between partners was essential for relationship building. University partners also traveled to the community to work with community partners on program adaptations, conference abstracts, presentations, and to complete day-to-day project tasks together. Community partners shared that it was also imperative for university partners to attend community events and ceremonies, and to create opportunities for partners to informally socialize. While these strategies were employed throughout the project, the WGC had not evaluated why and how these strategies were effective until they began to use the technique. In this way, the process allowed for partners to discuss and clarify their assumptions regarding CBPR and the strategies necessary for establishing a strong and equitable community-university partnership.
Through these discussions, the WGC identified a number of additional strategies for community-university partnerships. Primarily, at the outset of a project, partners should establish a mutual understanding of expectations in order to avoid later confusion and misunderstanding. In this vein, the WGC recommended that community partners be involved in budget discussions from the beginning of the project in order to understand how and why budget decisions are made; this would help community partners to develop a more solid understanding of budget management when projects later transition into sustainable, community-led programs. Additionally, both partners should be aware of the power imbalance that is inherent to grant funds being held at the university rather than in the community. Most, if not all, community-university partnerships must deal with the reality that large funding agencies award research grants solely to university partners, who are then accountable for managing funds. Although in the current project, a portion of the overall grant was allotted to community partners to manage themselves, funds were still administratively filtered through the university, which was reportedly frustrating for community partners. In order to deal with these frustrations and the power imbalance that accompanies this situation, partners must be open and honest with one another, and willing to discuss the uncomfortable circumstances that such a funding arrangement can result in. Again, the overarching strategy for working through these challenges was to develop strong and trusting relationships between partners throughout the project.
To supplement the MLST program evaluation, Outcome Mapping was employed as a retrospective evaluation tool by community and academic research partners. Through the process, a number of boundary partners were identified, including Elders, community leaders and education directors, schools, as well as community and university partners. Outcome challenges, progress markers, and strategies were identified for each boundary partner through a series of team workshops, and evidence to support progress was systematically collected from meeting minutes, focus groups, reports to funders, and program facilitators’ daily reports.
Overall, partners were in strong agreement that Outcome Mapping provided a meaningful method for evaluating and telling the MLST program’s story of progress. In this way, the process was congruent with Indigenous worldviews. In particular, Indigenous research methods often emphasize narrative and relationality. Through workshops, team members told stories of their experiences with the program, informing the creation of outcome challenges and progress markers. This also allowed for the project’s milestones to be structured as more of a narrative than conventional evaluation methods, resulting in a coherent story of progress. Similarly, because the WGC created progress markers, the project was able to capture outcomes that were significant to community partners.
Importantly, the process was extremely beneficial for the current project. Of particular importance was the opportunity that Outcome Mapping offered for relationship building, both among community staff members and between community and academic partners. Bringing team members together to collaboratively define evaluation outcomes served as a reminder that partners were working toward a common goal. Participating in interactive workshops also allowed for partners to more fully understand multiple perspectives, and to consider ideas that were often innovative and novel. In this way, rich opportunities were provided for partners to learn from one another, and provided a catalyst for community partners to articulate and communicate expectations of university partners. Further, it provided a means for all WGC members to feel a sense of ownership over and investment in the evaluation process. The MLST evaluation was perceived as less of an academic endeavor, and more of a learning opportunity for both community and academic partners. Disentangling the complex web of progress also served to enhance staff morale. The WGC described feeling a sense of immense accomplishment at the evidence for progress elucidated by the process.
Similarly, Outcome Mapping prompted the WGC to recognize outcomes that might otherwise be overlooked by more conventional evaluation tools. Although there is obvious importance in remaining focused on long-term impacts such as reduced substance abuse and healthy First Nation communities, the process allowed for the WGC to take pride in smaller accomplishments indicating progress toward these long-term goals. A systematic examination of smaller-scale changes and strategies also allowed for the WGC to examine how changes occurred, and why less progress occurred in certain areas. Although the WGC had used tools such as phase diagrams in the past in order to illustrate progress, Outcome Mapping demonstrated far more complexity as well as peripheral and unanticipated changes. Likewise, it allowed for the WGC to examine outcomes and strategies that reached beyond the end of the evaluation, and to consider strategies and markers of progress moving forward into a phase of program sustainability.
Although the program was highly beneficial for the current project, a number of challenges must be highlighted. Primarily, it required a large time commitment from both partners. Because Outcome Mapping is a participatory process, it is essential for all team members to be present at workshops. Additionally, one person must be assigned to facilitate workshops and to organize and track data, which in itself represents a significant time commitment. Further, because Outcome Mapping is a relatively new method, it can be difficult to secure a facilitator who has an in-depth understanding and experience with the process. Investing time and energy into it also requires a flexible project funder. For the current project, funders did not impose particular evaluation methods, and trusted team members to employ appropriate evaluation tools. This flexibility is essential for the success of Outcome Mapping. Indeed, framing knowledge with its use could be seen as a risky undertaking in the world of academia, where randomized controlled trials and experimental methods are the standard. However, in complex community settings, the technique is highly valuable and arguably necessary for evaluating and understanding community progress and change.
This project represents the first account of Outcome Mapping in a Canadian Indigenous community. Given the need for evaluation methods that align with the non-linear, multi-faceted processes of community change and development, this project makes an important contribution to the evaluation literature. Our project not only demonstrates how Outcome Mapping can accurately and comprehensively capture the change process in community projects; it also highlights how the method can be used to bring together and engage community and university partners in the evaluation process.
Finally, this project details how Outcome Mapping can be effectively adapted by a community-university partnership.
While conventional research and evaluation methods can provide valuable information regarding the effectiveness of a program, CBPR projects require the additional use of innovative methods that can account for the complex nature of community change. For the current project, Outcome Mapping provided a systematic method for understanding peripheral changes that are often overlooked in conventional research and evaluation, but that nonetheless indicate progress toward community changes and long-term impact. Among other findings, data demonstrated positive impact on Elders and students; growing community investment in and support for the MLST program’s cultural components; progressive increases in community ownership of the program; and growth in the community-university partnership throughout the project. The method is a highly valuable tool for CBPR projects. It supports growth in community and academic capacity and relationship building between community and academic partners. Outcome Mapping can enhance research and evaluation quality and contribute to project sustainability by offering a framework for capturing outcomes that are meaningful for community partners.
Baydala, L., Fletcher, F., Tremblay, M., Rabbit, N., Louis, J., Ksay-yin, K., & Sinclair, C. (2016). A community-university approach to substance abuse prevention. Journal of Community Engagement and Scholarship, 9(1), 67–75.
Botvin, G.J., Baker, E., Renick, N.L., Filazzola, A.D., & Botvin, E.M. (1984). A cognitive-behavioral approach to substance abuse prevention. Addictive Behaviors, 9(2), 137–147.
Botvin, G.J., & Griffin, K.W. (2014). Life skills training: Preventing substance misuse by enhancing individual and social competence. New Directions for Youth Development, 2014(141), 57–65.
Canadian Institutes of Health Research (CIHR). 2007. CIHR guidelines for health research involving Aboriginal people. Ottawa, ON: CIHR.
Castleden, H., Morgan, V.S., & Lamb, C. (2012). “I spent the first year drinking tea”: Exploring Canadian university researchers’ perspectives on community-based participatory research involving Indigenous peoples. The Canadian Geographer, 56(2), 160–179.
Cross, T.L., Friesen, B.J., Jivanjee, P., Gowen, L., Bandurraga, A., Matthew, C., & Maher, C. (2011). Defining youth success using culturally appropriate community-based participatory research methods. Best Practices in Mental Health, 7(1), 94–114.
Dart, J., & Davies, R. (2003). A dialogical, story-based evaluation tool: The most significant change technique. American Journal of Evaluation, 24(2), 137-155.
Earl, S., Carden, F., & Smutylo, T. (2001). Outcome Mapping: Building learning and reflection into development programs. Ottawa, ON: International Development Research Centre.
Gauld, S., Smith, S., & Kendall, M.B. (2011). Using participatory action research in community-based rehabilitation for people with acquired brain injury: From service provision to partnership with Aboriginal communities. Disability and Rehabilitation, 33(19-20), 1,901–1,911.
Grekul, J., & Sanderson, K. (2011). ‘I thought people would be mean and shout.’ Introducing the Hobbema Community Cadet Corps: A response to youth gang involvement? Journal of Youth Studies, 14(1), 41–57.
Israel, B.A., Schulz, A.J., Parker, E.A., & Becker, A.B. (1998). Community-based research: A partnership approach to improve public health. Annual Review of Public Health, 19(), 173–202.
Jagosh, J., Macaulay, A.C., Pluye, P., Salsberg, J., Bush, P.L., Henderson, J., Sirett, E., Wong, G., Cargo, M., Herbert, C.P., Seifer, S.D., Green, L.W., & Greenhalgh, T. (2012). Uncovering the benefits of participatory research: Implications of a realist review for health research and practice. Milbank Quarterly, 90(2), 311–346.
Judge, K., & Bauld, L. (2001). Strong theory, flexible methods: Evaluating complex community-based initiatives. Critical Public Health, 11(1), 19–38.
Kelly, T. (2010). Five simple rules for evaluating complex community initiatives. Community Investment 22(1), 19–23.
Kelly, J., Saggers, S., Taylor, K., Pearce, G., Massey, P., Bull, J., Odo, T., Thomas, J. Billycan, R. Judd, J., Reilly, S., & Ahboo, S. 2012). “Makes you proud to be black eh?”: Reflections on meaningful Indigenous research participation. International Journal for Equity in Health, 11(40), 1–8.
Minkler, M., & Wallerstein, N. (Eds.). 2003. Community-based participatory research for health. San Francisco, CA: Jossey-Bass.
National Health Promotion Associates, Inc. (2011a). Life skills training questionnaire: Elementary school version. Retrieved from www.lifeskillstaining.com.
National Health Promotion Associates, Inc. (2011b). Life skills training questionnaire: Middle school version. Retrieved from www.lifeskillstaining.com
Nyangaga, J., Smutylo, T., Romney, D., & Kristjanson, P. (2010). Research that matters: Outcome Mapping for linking knowledge to poverty-reduction actions. Development in Practice, 20(8), 972–984.
Salimi, Y., Shahandeh, K., Malekafzali, H., Loori, N., Kheiltash, A., Jamshidi, E., Frouzan, A. S., & Majdzadeh, R. (2012). Is community-based participatory research (CBPR) useful? A systematic review on papers in a decade. International Journal of Preventive Medicine, 3(6), 386–393.
Smith, R. Mauremootoo, J., & Rassmann, K. (2012). Ten years of Outcome Mapping adaptations and support. Ottawa, ON: Outcome Mapping Learning Community.
Victora, C.G., Habicht, J., & Bryce, J. (2004). Evidence-based public health: Moving beyond randomized trials. American Journal of Public Health, 94(3), 400–405.
About the Authors
Melissa Tremblay is a program evaluator in the Department of Pediatrics at the University of Alberta. Lola Baydala is an associate professor in the Department of Pediatrics at the University of Alberta. Natasha Rabbit is the executive director of the Nehiyaw Kakeskewina Learning Society in Maskwacis, Alberta. Jennilee Louis is a research assistant for the Nehiyaw Kakeskewina Learning Society in Maskwacis, Alberta. Kisikaw Ksay-yin is an Elder in the Nehiyaw Kakeskewina Learning Society, Maskwacis, Alberta.