Evaluation Report 2015-16,
Pilot Sites, 0-3 Supplement
R. Greenberg, Ph.D. & E. Cohen, Ph.D.
Celebrating Families!TM is an evidence based program developed to work with every family member from birth to adulthood to strengthen family recovery from alcohol and/or other drugs, break the cycle of addiction and increase successful family reunification. CF! fosters the development of safe, healthy, fulfilled, and addiction-free individuals and families by increasing protective factors and decreasing risk factors while incorporating addiction recovery concepts with healthy family living skills. It works to build healthy family environments so children can be safe and healthy (physically, mentally, and emotionally) by enhancing parent- child relationships and attachment.
This study was implemented to obtain information on the efficacy of Celebrating Families!TM to meet the goals and objectives of the program. The three program objectives, which relate to this evaluation, are:
To improve parents’ and children’s mental health including decreased stress and adults’ use of alcohol and other drugs by increasing their knowledge and use of healthy living skills including communication, expression of feelings, anger management, problem solving, stress management, and access to support.
To improve relationships between children and parents including increased interactions, communication, and “positive parenting” (affirming, holding, teaching, playing, and reading).
To reduce risk factors and adverse childhood experiences (ACEs): familial violence, extreme, inconsistent discipline, and parental substance use.
Findings in this report come from two instruments: 1) The CF! Pre- and Post Parent Assessment developed for use with families with children ages 4 – 17 and the newer supplement for families with children ages birth through three (0-3), available for use by any agency implementing CF! The instrument has ten questions using a five point Likert scale. In addition, the post Parent assessment has five opened ended questions. 2) The second instrument is the Protective Factors Survey (PFS), a 20-item scale, with established reliability and validity from four national field tests. The PFS identifies five key protective factors that have been identified in the research literature. They are: family functioning/resilience; social support; concrete support; child development/knowledge of parenting and nurturing/ attachment.
Study Sites and Demographic Information
This study obtained data from two sites in California: 1). A residential substance abuse treatment program for women and their young children, Parisi House on the Hill (PHOTH). Women in residence are primarily referred by Dept. of Social Services (DSS). 2). A community- based organization, Uplift Family Services (previously EMQ FamiliesFirst), the largest non-profit behavioral health and foster care agency in California. Families served struggle with mental health and substance use issues, primarily because of debilitating trauma histories and poverty. Participants in these groups were referred by judges from family drug courts.
Table 1: Demographics
This section provides findings from Celebrating Families!TM 0-3 Pre-and Post Parent Evaluation Instrument for questions 1 – 10 from the two sites. Table 2 shows the sample sizes and completed pre- and post-test data used in these analyses.
Table 2. Sample Size and Completed Pre- Post- Pairs
Comparability of Groups
Uplift and PHOTH did not differ on baseline (pre-test) scores for the 0-3 assessment. However, the two groups did differ on baseline (pre-test) scores for the PFS. The average PFS total score for Uplift was 82.36, while for PHOTH it was 73.31, indicating that the PHOTH group may have been more challenged initially in the areas of family functioning parent skill protective factors (p = .01).
The 0-3 Assessment asks nine questions about family functioning, parenting behavior, and self- care. Responses are scaled from 1 (“Never”) to 5 (“Always”). The average post-test total score (38.24) for the nine 0-3 quantitative items was higher than the pre-test total score (32.86) for Uplift (p = 0.014), showing overall improvement. Although a statistically significant improvement is a positive sign, this does not tell us how much of an effect the program may have had. The effect size is a statistic that quantifies how much change occurred, relative to other outcome evaluations of therapeutic programs. The effect size for the change from pre- to post-test was 0.49, indicating a moderate effect size. There were no differences in the overall total pre- and post-test scores for PHOTH (nor were there any differences in either direction for any of the individual items).
Table 3 shows the Uplift results for each item. While all items showed an increase in score from pre- to post-test, items 1, 3, 7, 8 and 9 showed statistically significant improvement.
Table 3. Uplift Family Services 0-3 Assessment Results
Questions 10-13 of the 0-3 Assessment ask participants about their family’s safety plan and test their knowledge about three aspects of alcohol/drug harm. These questions elicit a “yes” or “no” answer. The survey questions and frequencies of response are shown in Table 4.
Table 4. Items 10-13 – Response Frequencies
All the items showed an increase in the percentage of respondents who answered “Yes” over time. Item 10 approached statistical significance (though not at the .05 threshold). Item 13 did reach the statistical significance threshold, showing that participants did increase their knowledge about drug tolerance as a sign of addiction.
Protective Factors Survey
The Protective Factors Survey (PFS) is a reliable and valid self-response measure of various parent and family factors that are known to influence child abuse and neglect (Counts, Buffington, Chang-Rios, Rasmussen, & Preacher, 2010). None of the subscale scores of the PFS (Family Functioning, Social Support, Concrete Support, or Nurturing and Attachment), nor the PFS total score, showed any statistically significant change for either Uplift Family Services or PHOTH. For both agencies, one dimension – Social Support – showed slightly lower (worse) scores in the post-test. All others showed very modest gains.
PFS authors do not recommend adding up a total score for items related to the fifth dimension, Child Development/Knowledge of Parenting, since the items were never expected to correlate with each other. The individual items may still be used to track progress over time. While modest improvements were made in some of these items, there were no statistically significant changes in any items for either agency.
One reason why PFS scores did not change over time is that many of individual items scored in the high range for the pre-test. High pre-test scores will limit how far scores can improve for the post-test.
Assessment and Survey
An overall caveat to these analyses for both the 0-3 Assessment and PFS is that the low response rate (60% for the 0-3 Assessment, and 36% for the PFS) did not provide a fair test of the intervention using these measures. A higher response rate may show stronger positive results.
Qualitative findings from the 0-3 Post Parent Assessment
The “0-3 Pre- Post Parents Assessment” instrument contains five open-ended questions administered in the post-test assessment, after the quantitative component of the instrument. These questions are:
Describe one new way you have learned to express your anger safely.
List two things you have learned about being a parent. A and B.
List two new ways you have learned that help you communicate. A and B.
What is one important thing you have learned to do as a result of attending CF!
What suggestions do you have to make CF! stronger?
The questions were designed to elicit concrete examples of the impact of CF! They
represent some of the key learning objectives of the curriculum – the parent’s management of anger, learning skills to be a better parent, and improved communication skills with children and others. The questions also elicit the parent’s opinions about improving CF! The raw data for each item are appended to this report Tables 7 and 8.
The items, along with the number responding and response rate for each item (out of a total number of respondents) are shown in Table 6. The response rate mainly reflects missing instruments, not missing data (with some small variation among items). Out of the sample of 47, 29 instruments were available for analysis.
Table 6. Open-ended Questions and Response Rate, 0-3 Parent Assessment
Question 1: To manage anger, parents reported learning a combination of internal emotional regulation exercises (such as meditation, breathing techniques, and other self- regulation activities), along with behaviors to express feelings and/or take measures to avoid situations that trigger strong or difficult emotions. “Meditating and calming down” was a typical response as a self-regulation activity. Several respondents mentioned controlling breathing or using breathing as a meditative technique, as a way they learned to regain internal control. Other respondents mentioned “taking a walk” or walking away from a stimulus (“Walk away and return to the person later”), as a way to avoid a stimulus which would likely result in a maladaptive response if no action were taken. Yet another four respondents learned how to talk about their feelings as an alternative to behavior: “How to talk about anger instead of blowing up, how to find what actually made me angry”. “Setting boundaries” was mentioned by one respondent as one way to deal with angry emotions—i.e. dealing with a stimulus (probably the child’s behavior) by establishing external control with an appropriate parental response.
Question 2: There were three main dimensions to what respondents learned about being a parent. They can be categorized as: a) the self-image dimension, b) the behavioral dimension, and c) the relational dimension.
a) The self-image dimension has themes related to changes in how the parent views his/her identity, roles and responsibilities as a parent. “My children learn from what I do” and “practice what I preach” were mentioned as examples – the parents were expressing the importance of being a role model, which means that the parent had to learn something about how he/she comes across to others and that his/her own behavior will have an impact on the child. “To put effort towards your kids and be a good parent” was how another respondent framed his/her shift in identity. In general, parents had to learn “the importance of my role in family” to change their behavior and parent more appropriately, i.e. “how my new change can save lots of pain in my family's future.”
b) The behavioral dimension has themes related to specific things parents will do as a result of what they learned. Key words such as “talk,” “listen,” “spending time,” indicate the importance placed on practicing behaviors that will result in positive relationships with their children. Better ways to speak to children and the importance of listening are prominent in parents’ responses. They also value spending quality time with children in addition to paying more attention to the children’s concrete needs for vaccinations, safety, eating as a family, and “asking them what’s wrong instead of yelling...” Responding to the child’s behavior with “consistency” and “structure” is mentioned, probably as learned skills about using parental authority appropriately.
c) The relational dimension (which overlaps with both the self-image and behavioral dimensions) covers lessons learned about parents’ relationship with their children. “Showing affection” (one parent put it like this: “love each child separately”) and putting effort into the relationship are important relational characteristics of good parenting. Statements about the importance of establishing and maintaining a strong family unit would fit under this category, as well. A better understanding of child development provides a foundation for parents’ view about their relationship to the child, i.e. “...know [children] have feelings they can’t communicate,” which allows the parent to step back and interpret the child’s behavior in a more developmentally appropriate and productive way. Similarly, an understanding how important it is to “respect boundaries of children” also shows the parent’s ability to think about his/her relationship to the child in developmental terms.
Question 3: As an important aspect of the curriculum, learning effective communication skills warranted a specific question to participants. “Listen more, talk less” sums up many respondents’ thoughts about communication, which is related to “take turns” when conversing, “thinking before speaking,” “pay attention and focus,” and using “logic and reason not emotions.” Using “I statements” directly reflects the language of the curriculum, which is related to “[not assuming] things before talking...” and generally respecting others’ viewpoints, even if they trigger emotional responses, yet to “be honest not matter what” and always
“express how I feel.” These responses reflect the nature of open communication – listening to the other person while honestly expressing oneself.
Question 4: The “most important thing you have learned to do as a result of attending CF!” is a summary question reflecting responses to the previous open-ended questions, i.e., all of the responses can be found in one of the previous categories. There were two notable exceptions. Respondents took this opportunity to
Recognize the new knowledge about substance use and its harmful effects (a curricular content area rather than a “to do” action item). This shows how important the topic was for participants – they found a way to insert it in questions that did not directly elicit the curricular content strengths.
The idea of “how to celebrate my family,” indicating the importance of the curriculum of the concept that a family is worth celebrating, despite the hardships and low points.
Question 5: How to make CF! stronger – there were some specific suggestions directly related to the curriculum and programming: fewer videos, more organized family activity at end, include family or relatives who are fostering, longer meditation time, one class “with the kids”, and more engaging activities.
Conclusions and Implications for Future Evaluations
Responses to the open-ended questions on the “0-3 Pre-Post Parents Assessment” showed that parents were able to relate the curriculum to their own lives and families. The integration of unique program language (such as “I statements”) can be seen in their responses. Asking open-ended questions to complement closed-ended questions is helpful in that it has the dual purpose of allowing parents to think about how the program helped them and providing a way for parents to report individualized responses to questions about program impact.
The existing post-test, open-ended questions did not seem redundant with the closed- ended questions, since they allow parents to express themselves individually about the impact of the program content areas. To target parents’ thoughts about the curriculum, an additional question might be helpful: What area of the classes did you find most helpful or was especially important to you?” (While this can also be structured as a closed-ended question, listing choices for parents to check, an open-ended question is better since it allows the parents to phrase the answer in their own words.) To gauge how successful parents are in putting the curriculum into practice at home, as well as the challenges, additional open-ended questions might include “Give an example of how you put what you learned into practice” and “Give an example of how difficult it is to put something you learned into practice at home.” Responses to these questions can serve as illustrations of the practical implications of the curriculum in program reports.
It would be helpful to ask similar open-ended questions prior to the start of the program. The questions would focus on parents’ goals for themselves, i.e. “What would you like to learn about how to express anger?” and “What would you like to learn about being a better parent?” Asking goal-directed questions at the beginning of a program would help facilitators to better tailor the curriculum to individual needs and provide a way for parents to quickly relate the subsequent classes to their own context. In terms of impact analysis, the pre- and post- responses can be compared. Their responses to the pre-test questions can be presented to parents upon program completion and parents can then rate the program’s success in helping them accomplish their objectives or to comment on how their objectives changed over time as they went through the curriculum.
In terms of implications for the development of other focused quantitative measures based on these open-ended questions, the dimensions of themes would indicate the priority areas for development of further evaluation:
Anger expression and anger management
Dimensions of learning to be a better parent: self-image, behavioral, and relational
Communication skills – changes in attitudes and behaviors.
1. To increase data collection, better systems need to be developed structuring dissemination and collection of the evaluation instruments.
2. The two sites, Uplift and Parisi House on the Hill, provided different program structures. Uplift has a closed program, clients are enrolled at the beginning and then no new clients are enrolled. PHOTH has an open enrollment structure, clients can join the group at any time, whenever they enter treatment. These structural differences may account for the greater difficulty in obtaining PFS and completed pre and post instruments for PHOTH and supports developing data collection systems fitting the program structure.
3. The Evaluation Committee needs to review the pre-test to determine if the addition of open-ended questions for comparison to post-test questions is desirable.
4. As indicated earlier, with regards to the findings of the Protective Factor Survey, an increased number of survey results could result in a more robust statistical outcome.
5. The Evaluation Committee and CF! Administrators need to review the participants’ recommendations and discuss with site leaders. (See Question 5, page 9.)
The findings from this study indicate that CF! 0-3 supplement is meeting its objectives and supporting the development of healthy resiliency factors in families and parenting, as well as assisting in interrupting risk factors.
Changes in substance abuse treatment approaches are increasingly moving from deficit to strength/resilience based orientation, including reframing from breaking the intergenerational cycle of addiction to supporting intergenerational family treatment approaches (Klostermann & O’Farrell, 2013; Warner, Young, Dennis, & Amatetti, 2007). There is also increasing attention directed to parenting/family interventions in substance abuse treatment that jointly address the two generation needs of families, children and parents (Arria et al., 2013; Niccols et al., 2012; Usher, McShane, & Dwyer, 2015). CF! provides concrete programming addressing intergenerational needs in one comprehensive program. The findings from this study show positive results supporting this shift to supporting intergenerational family treatment approaches. Therefore, further study with an increased sample size is warranted.
Arria, A. M., Mericle, A. A., Rallo, D., Moe, J., White, W. L., Winters, K. C., and O’Connor, G. (2013). Integration of parenting skills education and interventions in addiction treatment. Journal of Addiction Medicine 7(1), 1-7.
Counts, J. M., Buffington, E. S., Chang-Rios, K., Rasmussen, H. N., & Preacher, K. J. (2010). The development and validation of the protective factors survey: A self-report measure of protective factors against child maltreatment. Child Abuse and Neglect, 34(10), 762–772.
Klostermann, K. and O’Farrell, T. J. (20130. Treating substance abuse: Partner and family approaches. Social Work in Public Health, 28(3-4), 234-247.
Niccols, A., Milligan, K., Sword, W., Thabane, L., Henderson, J. and Smith, A. (2012). Integrated programs for mothers with substance abuse issues: A systematic review of studies reporting on parenting outcomes. Harm Reduction Journal, 9(14). DOI: 10.1 186/1477- 7517-9-14
Usher, A. M., McShane, K. E. and Dwyer, C. (2015). A realistic review of family-based interventions for children of substance abusing parents. Systematic Reviews, 4, 177-189. Doi:10.1186/s13643-015-0158-4
Warner, D., Young, N. K., Dennis, K., & Amatetti, S. (2007). Family-centered treatment for women with substance use disorders: Key elements and challenges. Department of Health and Human Services. Substance Abuse and Mental Health Administration.
©2017 Greenberg & Tisch