Part I. Community Guide Update and Rationale for Intervention
Research
| Author & Year | Intervention Setting, Description, and Comparison Group(s) | Study Population Description and Sample Size | Effect Measure (Variables) | Results including Test Statistics and Significance | Follow-up Time |
| Cutucache, Luhr, Nelson, Grandgenett, & Tapprich, 2016 | The NE STEM (The Nebraska Science, Technology, Engineering, and Mathematics 4U) program was led by faculty members and run by college students; the program involved facilitating problem based learning activities that reinforced and built upon concepts learned during the school day in the STEM disciplines for economically disadvantaged students in grades K-8. The program with the possibility of benefiting both the college mentors and the K-8 students took place twice each week during the academic year in the after school hours at schools in the Omaha, Nebraska area. Students were engaged in hands on activities to increase their critical thinking skills while building their proficiency in the STEM fields. STEM 4U K-8 student participant knowledge growth was assessed via pre- and post-program assessments; the control group did not participate in any STEM 4U programming. | The students in grades K-8 who participated in the STEM 4U program were all eligible for free and reduced lunch; they were the underperforming, minority, and underprivileged population at each participating school. The schools where the STEM 4U program was implemented were those with the lowest standardized test scores in both science and mathematics. The initial population size for participating students was n= 297. | The variables being measured were two-fold; first measured were the pre and post-test assessment scores of those students who participated in the program. The second variable measured was the long term retention rate of the STEM based content by the program participants and the control group population not participating in the program. | Students who participated in the program at least once per week saw an increase in STEM knowledge (1.7-2.5) in comparison to their non-participating counterparts. Participants also illustrated they had longer retention rates of the content than their non-participating peers by scoring better on long term retention STEM assessments, 45% for participants compared to 30% for non-participants (p=0.0026). | At the completion of the program, students were given a post-assessment to measure the growth of their knowledge in the STEM disciplines. The program has continued since the first year of implementation. |
| Paluta, Lower, Anderson-Butcher, Gibson, & Iachini, 2016 | The researchers studied the quality indicators that were evident (or not evident) to stakeholders associated with the 21st Century Community Learning Centers (CCLCs), a type of out of school time program, and then they looked at the perceived quality of the programs in relation to favorable student outcomes. The CCLC programs consist of character education, academic enhancement, and literacy support programs. For this particular study, the CCLCs from which data were collected were all in their first or third year of their funding cycles and located in a midwestern state and served elementary, middle, and high school aged youth from local schools, YMCA’s, or other educational service centers. Programs took place after school, on weekends, and during the summers. Stakeholders in 405 various CCLC programs throughout the state were asked to complete surveys either online or with paper/pencil. Researchers were to establish the outcomes the program stakeholders thought were most affected by their out of school programs. | 3928 surveys were taken by stakeholders (teachers, school administrators, community partners) with 332 deemed to be incomplete, n=3388. Demographics of the stakeholders varied across the board in terms of gender, ethnicity, age, and level of education. Survey respondents also encompassed various ranges of involvement in the CCLCs. | The survey distributed to stakeholders consisted of response items following the Likert scale ranging from 1-5, with 1=strongly disagree and 5=strongly agree. The items included questions pertaining to such topics as youth development, academic learning, family engagement, diversity, and leadership. | All outcome means were above the midpoint and deemed to be reliable. The quality indicator deemed most favorable was overall youth development (mean 4.49) and least favorable indicator was the professional development of the staff (mean 3.79). Academic outcomes (mean 4.09) being not as favorable as overall youth development, left room for improvement. Overall, perceptions across all domains measured were positive indicating to the researchers that the programs were achieving their purpose of increasing learning opportunities and support for youths. | After initial survey distribution and collection, no follow up has been conducted. |
| Almus & Dogan, 2016 | The intervention, an intensive five- week long summer school program, took place in Kansas City, Missouri in the summer of 2013 at two urban public charter schools with a free and reduced lunch population of 94%. The academic subject areas being targeted included science, mathematics, and reading. During the summer school program for each course, the curriculum was created in order to reinforce those topics students did not “get” during the school year. Intertwined with the program were also opportunities for students to participate in extracurricular activities such as PE classes and incentives for program attendance, engagement, and participation; incentives included extra PE time, free breakfast and lunch, free bus transportation, free t-shirts, and credits for the next school year. In addition to qualitative surveys conducted by the researchers, student achievement was measured based on a change in STAR test results prior to completion of the program compared to results on the same assessment after completion of the program. | Students who participated in the program attended one of two high poverty urban charter schools and were in grades K-8. Of the 534 students who participated in the program, 80 were new to the school and voluntarily attended for orientation purposes, 24 were mandated to participate due to low academic performance during the school year, and the remaining participants did so voluntarily because of low standardized test scores or were otherwise targeted by administration for participation. | The variables being measured were the standardized assessment STAR’s results for the group of students before and after participation in the program. Statistical analyses were run on the data to determine significance of the changes in standardized test scores. Qualitative surveys were also distributed to students for them to assess their feelings about the program and how they feel they progressed during the summer and also to determine if students in different grade levels felt differently towards the program. Teachers and parents also completed Likert style surveys regarding the program. | For a handful of reasons, only 386 of the 534 students completed the posttest and therefore results are based on that number of participants. As a result of participation in the program, it was shown that the change in pre vs. post test scores is statistically significant for reading achievement as measured by the STAR test (t=2.371, p=0.018). As a result of the surveys, most students believed that their standardized test scores and achievement in other subjects in the future will be better because of their participation in the summer program. There was no difference found amongst students in various grade levels. | At the end of the summer program, participating students were given a post test (the same standardized STAR assessment given prior to the program) to measure their academic growth during the summer program. The summer school program has continued. |
| Allen, 2015 | An after school tutoring program was modified in hopes of increasing the academic levels of students participating in the subjects of reading, English language arts, and mathematics. The program was implemented in an elementary school in a southern school district and operated once a week for ninety minutes for twenty-seven weeks; the school was described as having 500 students with 99% being African American and 98% being of low socioeconomic status. The effectiveness of the modified program was measured by the comparison of student grades before and after participation in the program; the comparison group consisted of non-program participants. For the modified after school tutoring program, lessons were created around the weak areas as illustrated by the previous year’s standardized test scores. | The students targeted for participation were in grades 3 and 4 and had scored lowest on the previous year’s standardized test used to measure academic achievement in reading, English language arts, and mathematics. For the test group, n=100, and for the control group n=57. | The effectiveness of the program was based on the change in standardized test scores for those students participating in the program compared to the change in test scores of those not participating in the program. Statistical analyses were performed on the various groups of data. | In comparing the change in test scores, program participants were found to show more growth than the control group in all three subject areas being targeted.
Reading grades: (t=-3.345; p<0.001) English language arts grades: (t=-3.43; p<0.001) Mathematics grades: (t=-7.32; p<0.001) |
In depth follow up interviews have been conducted with several teachers who implemented the program. The focus of each interview was on the qualitative aspects of an after school tutoring program. |
Updated Recommendations and Justification
After completing reviews of a meta-analysis published in 2006 and twenty-five additional studies, the Task Force concluded that out of school time academic focused programs lead to achievement gains by at risk and low performing youths in grades K-12. The Task Force recommends programs that focus on reading, math, and general academic content out-of- school time academic programs because of their effectiveness in decreasing the achievement gaps for at risk students. With the implementation of these programs and subsequent rise in academic achievement, health equity is likely to be improved because academic achievement is linked to an individual’s long term health. The programs target groups likely to fall into cycles of poverty: racial and ethnic minorities and those with low income. The findings and recommendations of the Task Force were published in 2015 (Guide to Community Preventive Services).
As outlined in the table above, with the review and analysis of three studies conducted since the findings of the Task Force were published, it is also my belief that there is evidence to support that academic focused out-of-school programs be recommended for implementation in schools and community centers serving the at risk student populations. The Task Force based their recommendations on improvements made on standardized test scores by those students participating in out-of-school time academic programs; reading scores increased by 0.31 standard deviations, math scores by 0.12 standard deviations; in general content programs, reading improved by 0.09 standard deviations and math by 0.06 standard deviations (Guide to Community Preventive Services).
The study conducted by Cutucache, Luhr, Nelson, Grandgenett, & Tapprich (2016) shows similar gains in academic achievement with the implementation of the NE STEM 4U program. The out-of-school time academic program geared towards problem solving and critical thinking in the STEM disciplines for underperforming and at risk youths in low income areas of Nebraska illustrated the academic achievement gains made by the students participating in the program versus those who did not participate in the program. The gains were measured by comparing pre and post program assessment scores and were shown to be significant for the test group of students who participated in the STEM 4U programming; the control group showed fewer gains and less long term retention of the content knowledge.
In another research study conducted by Allen (2015), academic gains were also made by at risk elementary aged students participating in an out-of-school academic focused program. Third and fourth grade students who performed poorly on the previous year’s high stakes standardized tests, participated in a twenty-seven week long after school tutoring program in an attempt to improve their achievement on the standardized test. The subject areas being targeted by the tutoring program were reading, English language arts, and mathematics. In comparison to students not participating in the after school tutoring program, the participating students saw more substantial gains in achievement in the targeted academic areas; reading scores for participants increased by 6.1% for participants and only 2.6% for non-participants; English language arts scores by 4.2% and 1.4%, respectively; mathematics scores saw the most dramatic change for participants with an increase of 8.0% and only 1.1% for non-participants.
Another study by Paluta, Lower, Anderson-Butcher, Gibson, & Iachini (2016) focused on the perceptions of the quality of out-of-school academic programs. The Task Force did note that the differences in the quality and type of programs available can affect the outcomes. The study found that overall youth development is a positively perceived outcome of out-of-school programs and thus that the out-of-school programs are doing their job in providing learning opportunities and support for the at risk youth populations where implemented.
The studies show sufficient evidence to recommend the implementation of out-of-school academic programs to improve at risk student achievement. The long term health of an individual is directly linked to his or her academic achievement. It has often been found that minority and low income groups do not attain academic achievement levels as high as those of the higher income and majority groups. As a consequence of this, the at risk students tend to become adults with lower income and thus poorer health. If the programs promoting academic growth and achievement of the minority and low income groups can be implemented effectively in areas of need, then a step towards future health equity is being taken (Guide to Community Preventive Services, 2015).
Part II. Theoretical Framework/Model
The Theory & its Constructs
If implemented effectively, the intervention strategy of out-of-school time programs focused on academics for at risk youth in grades K-12 improves not only academic achievement but in turn long term individual health as outlined by the Community Guide. Students falling into at risk socioeconomic and minority groups must be encouraged to participate in such programs if and when they are offered to them.
The Theory of Planned Behavior (TPB) states that intention to perform a behavior is the most important factor in an individual deciding to or not to engage in a particular behavior. The TPB examines an individual’s relationship between behavior and beliefs, behavior and attitudes, and behavior and intentions regarding the behavior. An individual’s perception of what others close to him/her think about the behavior and the individual’s attitude towards carrying out the behavior shape a person’s intention regarding the behavior. The constructs of the Theory of Planned Behavior are as follows: behavioral intention, attitude, subjective norm, and perceived behavioral control. In order to improve participation in out-of-school academic programs, school officials should evaluate their eligible population via the TPB as the constructs influence behavior (National Cancer Institute, 2005).
The perception that an individual has of his or her chance of performing a certain behavior is defined as behavioral intention. This can be measured via a question or discussion with an individual regarding whether he or she will be likely or not likely to engage in a behavior. When planning out-of-school programs for at risk youths, it would be important for program planners to talk to potential program participants about their intent to participate in such a program as behavioral intent in this case could lead to participation or non-participation in the school program; school and program officials can only do so much as to encourage participation. In this case for the intervention of out-of-school time programs, the intent to participate is being defined as the student’s and/or parent of that student’s likelihood to have the student participate in the program.
Attitude impacts intent and is defined as an individual’s evaluation of the behavior, in this case his or her attitude towards attending an out-of-school program. The individual must assess his or her belief in regards to whether the behavior (attending out-of-school program) would have a positive effect, negative effect, or no effect on him/her. Throughout discussions regarding the program, school officials need to ensure that the program is marketed appropriately towards the intended audience (at risk students) so that attitudes towards the program are positive and attendance is high.
The beliefs that a potential program participant has about how others in his/her life feel regarding participation in the behavior is defined as subjective norm; subjective norm also influences an individual’s intent to behavior. So, do the people in the individual’s sphere of influence support participation in the behavior? A student who might consider participating in an out-of-school program would be motivated by the thoughts of those who have influence over them such as their parents, teachers, and peers. It becomes important that these other stakeholders (parents, teachers, peers) in a student’s life know the potential impact and benefit of participation in out-of-school programs so that they can positively influence the student considering participation.
The final construct, perceived behavioral control, pertains to whether an individual has control in performing a particular behavior. In the case of attending out-of-school programs this may pertain to a student feeling that he or she has a choice in attending or not attending the program. Ultimately the decision to intend to or not intend to participate in the program (behavior) lies with the student, and the student needs to know that while also being guided to make the decision that would benefit him/her the most.
The Model (revised)
Part III. Logic Model, Causal and Intervention Hypotheses, and Intervention Strategies
Target Population & Program Setting
The out-of-school time academic focused program intervention will target the underperforming at risk students in minority and low socioeconomic status groups at the targeted schools and community centers. The schools targeted will be those in areas with standardized test averages lower than that of a state’s overall average and thus underperforming for at least two consecutive years; these may be rural or urban schools in a district. The community centers targeted will be those that serve students at the targeted schools located in the surrounding areas of the identified schools. As previously mentioned, academic achievement is linked to long term health. If the academic achievement of the underperforming at risk groups of students can be improved, then the chance of long term health is increased. The intervention will take place during the after school hours at the participating schools and community centers up to twice per week. The number of teachers who will conduct the program will be dependent upon the number of student participants at each site.
Intervention Methods & Strategies
| Intervention Method | Alignment with Theory | Intervention Strategy |
| Increase awareness of out-of-school programs and their short and long term benefits (persuasive communication) | Aligned with the Theory of Planned behavior at the perceived behavioral control construct, using persuasive communication can lead prospective program participants (students) to feel they are able to make the decision to participate (or not) in the program | *distribute brochures/flyers to prospective students
*teacher led class discussions during the school day referencing the purpose and benefits of the programs *targeted small group meetings with school counselors/administrators, giving students more opportunities to realize the benefits and to ask questions in a smaller setting *parent nights in which students and parents meet in the evening hours with program leaders to learn more about the benefits of the program |
| Offering of incentives to program participants who reach certain benchmarks while participating in the program | Before committing to program participation, students will know that there is an incentive plan in place for the duration of their participation, thus positively influencing the attitude of the prospective students, increasing the likelihood of their participation (intent). | *signed student contracts before program participation begins outlining the incentive plan; incentives for reaching benchmarks may include the opportunity for extracurricular activity while in the program (social time, gym time, computer time, phone time etc.)
*if students close achievement gap via new test scores or other program measure, participation in the program may cease if student chooses |
| Grant access to prospective student participants and their parents to students and parents of students who have participated and benefited from various out-of-school programs in the past (modeling) | The subjective norm construct may be influenced with this particular intervention. If prospective participants and those close to them see that participation in the program had a positive influence on past participants, then prospective students will be more likely to commit to model themselves after successful participants. Participants will likely be motivated by the reactions of their parents and/or peers who also have participated in this intervention. | *set up one on one mentoring sessions with a past participant and a prospective participant
*set up sessions between student/parent and past participant *once program begins, allow participating student to confide in and continue to communicate with past participant *past participants write notes of encouragement to participating students on regular basis |
| Allow students to choose academic area in which they would like to further develop their skills | Giving prospective students choice in their program focus will lead to perceived control and thus more positive intent to participate | *when targeted for the program, allow students to choose the subject area for their studies if they write a convincing reason/story as to why the chosen subject should be their focus |
Logic Model
| Inputs/Resources | Activities | Outputs | Short-term Outcomes | Intermediate Outcomes | Long-term Outcomes |
| *funding to train and pay educators; 1 educator per 15 students
*funding to keep the school buildings/rooms opened after hours; 1 room per 15 students (heat/AC, lights, janitorial staff) *program curricula; 1 per subject area offered via the program |
*distribute brochures/flyers to prospective students
*teacher led class discussions during the school day referencing the purpose and benefits of the programs *targeted small group meetings with school counselors/administrators, giving students more opportunities to realize the benefits and to ask questions in a smaller setting *parent nights in which students and parents meet in the evening hours with program leaders to learn more about the benefits of the program *signed student contracts before program participation begins outlining the incentive plan; incentives for reaching benchmarks may include the opportunity for extracurricular activity while in the program (social time, gym time, computer time, phone time etc.) *if student closes achievement gap via new test scores or other program measure, participation in the program may cease if student chooses *set up one on one mentoring sessions with a past participant and a prospective participant *set up sessions between student/parent and past participant *once program begins, allow participating student to confide in and continue to communicate with past participant *past participants write notes of encouragement to participating students on regular basis *when targeted for the program, allow students to choose the subject area for their studies if they write a convincing reason/story as to why the chosen subject should be their focus |
*ALL students in the school receive flyer outlining the purpose and design of the program
*school chooses one day to have all core subject educators to present the information to all of their classes at all grade levels *teachers recommend (based on test scores) students to participate in the small group meetings with counselors to further discuss details of the program *two parent meetings on different nights during the same week to introduce program to parents (parents attend one) *design and distribute program contract with incentives outlined to all students who have attended small group or parent meeting or who have otherwise shown interest *initial meeting (make it fun) between past and prospective participants, keep ratio low, no more than two participants per one past participant *on a monthly basis, past participant will write notes of encouragement to current participants |
*student and parent awareness of the program will increase, therefore the realization of the benefits of the program will increase (attitude & subjective norm)
*students will decide to participate in and commit to the program (intent) *students will actually attend the weekly program sessions (perceived control) |
*student standardized test scores will increase
*student classroom grades will increase *more students will begin to realize the value of and enjoy their time in school |
*the achievement gap between groups of students will begin to close
*the long-term health of the minority groups of students participating will be better than it would have been without participation in the program |
Intervention Hypotheses
*By distributing brochures/flyers to prospective students and parents, the awareness of the benefits of the program will increase.
*If teachers lead class discussions during the school day referencing the purpose and benefits of the programs, student awareness of the benefits of the program will increase along with number of students deciding to participate in the program.
*In targeted small group meetings with school counselors/administrators, the intent of the student to participate in the program will increase, thus increasing actual participation in the program.
*At parent nights in which students and parents meet in the evening hours with program leaders, the realization of the benefits of the program will increase amongst both parents and students potentially increasing participation.
*If signed student contracts are secured before program participation begins outlining the incentive plan, then student participation in the weekly sessions will increase.
*If a student closes achievement gap via new test scores or other program measure and student has the choice to end participation in the program, then initial commitment along with student attendance in the program will increase.
*By allowing communication, interaction, and relationship building between prospective/current student participants with past participants, increases in program participation and attendance will be realized.
*When students are able to choose the subject area for their studies during the program, an increase in student attendance will be observed.
Causal Hypotheses
*An increase in student and parent awareness of the benefits of the program will lead to increases in the enjoyment of school by program participants.
*An increase in participation and commitment to the program by students will lead to increases in standardized test scores and classroom grades thus closing the achievement gap between minority groups of students and their higher achieving peers.
*By increasing student attendance at the weekly program sessions and the closing of the achievement gap, an increase in long-term health of the minority groups of students participating in the program will be realized.
SMART Outcome Objectives
Goal 1: Increase awareness of the benefits of the out-of-school time program. Objective 1: Within two weeks of brochure distribution, classroom sessions, and parent meetings, awareness of the program and its benefits will increase at least 30%. Objective 2: Awareness of the program will increase by at least 15% in each subsequent year of implementation.
Goal 2: Student commitment to the program will increase. Objective 1: After awareness of the program benefits has increased, initial student commitment to the program will increase at least 30% within one month. Objective 2: Before the program begins because of increased awareness of the program, the commitment rate to the program by prospective students will increase 20% each year.
Goal 3: Attendance at the weekly program sessions will increase. Objective 1: At least 85% of the students committing to the program will actually show up for at least 90% of the program sessions. Objective 2: Actual attendance of students who committed to participation in the program will increase 20% with each implementation of the program.
Goal 4: Standardized test scores will increase. Objective 1: Pre and post-test scores of program participants will increase at least 15%. Objective 2: Within two months, participation in standardized assessments will increase 15% as a result of participation in the program (ex: fewer students will be absent on test days).
Goal 5: During the regular school day/year, classroom grades of student program participants will increase. Objective 1: By the end of the grading period (or semester), the classroom grade in the content area in which each student participated in during the program will increase by at least 10% (a letter grade). Objective 2: Because engagement and grades of the student will increase, within six weeks, teacher reported discipline problems will decrease by at least 20%.
Goal 6: Student realization of the value of school and the enjoyment of school will increase. Objective 1: In six weeks time, positive student perceptions of school will increase by at least 20% because of participation in the program.
Goal 7: The achievement gap between minority groups of students and their higher performing peers will begin to close. Objective 1: As measured by classroom grades and standardized test scores, the achievement gap will begin to close by 1-2% each school year when the out-of-school program is being implemented.
Goal 8: The long-term health of the groups of students participating in the out-of-school time program will be better than it would have been without participation in the program. Objective 1: In comparing minority groups of students each year out of the program compared to minority students who did not participate in the out-of-school time program, better health will be self-reported by the students who participated in the program at each year increment. Objective 2: Over the long-term with subsequent implementations of the program and increased minority student participation, gains in long-term health of the minority groups will be realized.
Part IV. Evaluation Design and Measures
Stakeholders
| Stakeholder | Role in Intervention | Evaluation Questions from Stakeholder | Effect on Stakeholder of a Successful Program | Effect on Stakeholder of an Unsuccessful Program |
| Students | Participants in the program | If I participate in the program, how will I benefit? What incentives are there for me to participate in the program? | Classroom grades will improve; standardized test scores will improve; enjoyment of school will increase; better long-term health outcomes | Grades and test scores will remain stagnant or even decrease; attendance may decrease, drop out may occur; poor long-term health outcomes likely |
| Parents | Support students throughout duration of program | How do I know this is the right thing for my student to be doing? How will it help him/her and me? How will I know the program is working? | Child will become a better, more engaged student, grades and test scores will increase; behavior may improve, long-term health outcomes of child will be better | Grades and test scores of child in program will not improve and may even decrease; no improvement of long-term health outcomes |
| Educators/Peer Mentors | Create and deliver curricula to program participants; monitor academic achievement and growth; accurate record keeping; provide motivation and encouragement to participants | Is it an expectation that I contribute to this program or is there compensation associated with the extra effort? How will I know what I am doing is working and that the program is successful? How much time am I expected to devote to the program on a weekly basis? | Educator/mentor will want to continue working for the program from year to year; will encourage others to participate; positive relationships established | Educator/mentor will not continue with the program from year to year; will dissuade others from participating by sharing their experiences; meaningful relationships won’t be established and carried forward; wasted time to develop curricula for program |
| School Administrators & Program Directors | Oversee development and implementation of entire program | What are the expectations of the program? How much will it cost to implement the program; where will the funds come from? How will we know if the program is meeting its goals? | Will continue working with the program from year to year; will want to work to make program improvements each year | Will not want to continue working with the program from year to year; waste of time and resources in implementation; decreased support for future implementation of program |
Evaluation Design
To evaluate the effectiveness of the out-of-school time academic program intervention strategy, a randomized controlled trial design will be used. After similar schools are recruited and then elect to participate in the study, they will be randomly assigned to either receive the intervention or to continue moving forward with whatever teacher/individual classroom tutoring programs already in place (no intervention program). Because the study will take place over the full school year, surveys and pre-assessments will be completed prior to the program beginning by participating students in both the intervention and control groups; assessments will then be given in bi-monthly intervals throughout the school year to program participants to measure academic growth.
R O1 X O2 O3 O4 O5 (intervention)
R O1 O2 O3 O4 O5 (no intervention)
Standardized assessments will also be given to all students via the state mandated testing schedule; data from these exams may also be employed to gauge the effectiveness of this intervention. Surveys will be given to participating students prior to the beginning of the intervention/program and at the conclusion of the program/school year; surveys will be used to see where students stand on various aspects of the intervention/program.
Threats to Validity
By using a randomized evaluation design, most threats to internal validity are eliminated and internal validity should be maximized. With that being said, some threats may still exist. Bias in the instrumentation being used, specifically the pre and post program surveys, will exist if the instrument was not tested first with focus groups during the design of the intervention prior to its implementation. Even with the randomization of the study, attrition bias may exist; students may drop out of their school’s program, whether a part of the intervention group or not, students may drop out of school, students may move schools, etc. Because the study will not be running for longer than one school year, some attrition bias may be eliminated. Maturation may also be a threat to this study; it might have been found that academic growth would have occurred with the targeted groups of students even without the intervention. By having a control group in this study, maturation bias will be eliminated if the intervention group shows more academic growth than the control group.
Variables to be Measured and How
| Short-term or Intermediate Outcome Variable | Scale, Questionnaire, Assessment Method | Brief Description of Instrument | Example item (for surveys, scales, or questionnaires) | Reliability and/or Validity Description |
| Awareness of the program (if awareness of the program and its benefits are high/positive, then attitudes of prospective students towards the program will increase along with their evaluation of subjective norm) | Young and Rubicam’s Brand Asset Valuator; the instrument will be revised to reflect relevance, esteem, knowledge, and differentiation of the out-of-school time program (Zaichkowsky, Parlee, & Hill, 2008) | The measure is designed to measure four dimensions of brand awareness: relevance, esteem, knowledge, and differentiation. It is administered in the form of a Likert style questionnaire, 1=strongly disagree, 5=strongly agree. | Questions would be designed to compare the four parameters being measured between the out-of-school academic programs and other after school and/or summer activities taking place in a school setting (not necessarily academic).
*The out-of-school time academic program is the best option to help me (my student) meet my (his/her) academic potential. *The out-of-school academic program will meet my (my student’s) needs. *I have other options to help me meet my academic goals. (My student has other options to help him/her meet his/her academic goals.) |
Reliability (Cronbach’s alpha):*relevance 0.68
*esteem 0.72 *knowledge 0.30 *differentiation 0.44 |
| Commitment to the program (intent) | TCM Employee Commitment Survey- revised to reflect student commitment; printed/administered in both English and Spanish to better reach target participants(Meyer & Allen, 2004) | The survey addresses three types of commitment: normative, affective, and continuance. Survey items are answered using a 7 point Likert Scale with 1 equal to strongly disagree and 7 equal to strongly agree. | Normative Commitment: I was taught to believe in the value of remaining loyal to one’s organization.Affective Commitment: I enjoy discussing my organization with people outside of it.
Continuance Commitment: One of the major reasons I continue to work for this organization is that leaving would require considerable personal sacrifice- another organization may not match the overall benefits I have here. *will be revised as follows organization: out-of-school time program (intervention)work: attend, participate in |
Reliability (Cronbach’s alpha) 0.858 in a pilot study & 0.912 in the main studyThe face and content validity of the scale was confirmed by subject area experts.
(Sarwar & Ashrafi, 2014) |
Part V. Process Evaluation and Data Collection Forms
Recruitment and Enrollment
To begin the recruitment process, school districts will be contacted first for approval for program planners to contact schools within the district in regards to the proposed intervention. Once a district approves of the intervention, then the program planners will work with the district officials to identify schools willing to participate. Program planners will draft and send letters to school principals/counselors and then follow up via phone call within two weeks of mailing the letter. The following form will be used to track schools contacted and their response:
Once schools have elected to participate, participants will then be identified. With information from the individual schools, the program planners will write and distribute letters to prospective students and parents. Letters with a parent consent form to be returned will first be distributed at school to prospective students outlining the benefits, parameters, and details regarding the program. If consent forms are not received within one week of distribution, program planners will follow-up with parents via phone call. The following form will be used to track students contacted for participation in the program:
Attrition
Detailed contact information for each student and parent will be collected via the consent form during the recruitment and enrollment period. To ensure that each student/parent pair has received all relevant information before consenting to participate in the program, the following form will be used to track receipt of information, attendance at meetings, etc.
Once a commitment has been made to the program and the program is implemented, the following form will be used to track student participation in scheduled program observations.
Fidelity
To capture the fidelity of the program, the following form will be distributed at each participating school site; the administrator overseeing the program at each location will be asked to complete the form. Forms from all locations will then be compared to get an idea of what worked, what did not, and what might need to be changed for future implementations.
Classroom teachers conducting initial information sessions during class time, the counselors conducting small group meetings, the individuals running the evening parent meetings, and past participants (mentors) will each be asked to complete minutes/a survey regarding their part in the process. This data will then be compiled at the school level on the following form:
References
Allen, B. (2015). After school tutoring increases academic performance. Journal of Teaching, Learning, and Scholarship, 2(3), article 1.
Almus, K., & Dogan, B. (2016). A study of summer school enrichment program in high-poverty urban public charter school. Reading Improvement, 53(1), 1-16.
Cutucache, C., Luhr, J., Nelson, K., Grandgenett, N., & Tapprich, W. (2016). NE STEM 4U: an out-of-school time academic program to improve achievement of socioeconomically disadvantaged youth in STEM areas. International Journal of STEM Education, 3(1), 1-7.
Guide to Community Preventive Services. Promoting health equity through education programs and policies: out-of-school time academic programs. http://www.thecommunityguide.org/healthequity/education/outofschooltime.html. Last updated: 05/18/2016.
Meyer, J., & Allen, N. (2004). TCM Employee Commitment Survey: Academic Users Guide 2004. Ontario, Canada: University of Western Ontario.
National Cancer Institute. (2005). Theory at a glance: a guide for health promotion and practice (NIH Pub. No. 05-3896). Washington, DC: National Institutes of Health.
Paluta, L., Lower, L., Anderson-Butcher, D., Gibson, A., & Iachini, A. (2016). Examining the quality of 21st century community learning center after-school programs: current practices and their relationship to outcomes. Children and Schools, 38(1), 49-56.
Sarwar, M., & Ashrafi, G. (2014). Students’ commitment, engagement and locus of control as predictor of academic achievement at higher education level. Current Issues in Education, 17(3). Retrieved from http://cie.asu.edu/ojs/index.php/cieatasu/article/view/1314.
Zaichkowsky, J., Parlee, M., & Hill, J. (2010). Managing industrial brand equity: developing tangible benefits for intangible assets. Industrial Marketing Management, 39(5), 776-783.