Theory-Driven_Evaluation

Theory-Driven Evaluation: Conceptual Framework, Methodology,and ApplicationE-mail: hbc2@necessarily represent the views of CDC.•evaluation•Black -Box Evaluation: Assessing the relationship desirable outcomes?driven evaluation: Using a research method (e.g. RCT, survey, etc. ) as a basis to guide an evaluation design Intervention OutcomesProvide little information for program improvement transferabilityComic Book Changing smoking belief,attitude,and behaviorProgram theory: stakeholders’implicit and explicitconceptual framework for evaluating effectivenessPROGRAM THEORY Intervention Determinants OutcomesChange ModelWhy question : Why does the intervention affectthe outcomes? (change model)How question : How are the contextual factors and1.2.Assure an evaluation evaluates the right thing 3.4.5.6.bottom -up evaluation approachprogram “failed ”or “succeeded ”in reaching its goal, to(Ocabbot , Muchoborro , Philip Horrid, etc.) to save theComic BookChanging smoking belief, attitude, and behaviorand characterChange Model Underlying an Anti-Smoking ProgramFamily Planning Reduce fertility rateFamily planning programcouples’knowledge, skills, and favorable values on birthcontrolReduce fertilityrateinterventionPolice chief believed the policy was successful.New policyEnhancing performance as shown in the mileage of odometersIncreasing patrols in neighborhoodsReducing crime ratesNew policyEnhancing performance as shown in the mileage of odometersRaising officers’concerns on performance rating and payCruising highways around thecityAssures same salaries or increasing salariesNew policyEnhancing performance as shown in the mileage of odometersRaising concerns on performance rating and payImplementers : recruit, train, and maintain both: identify, recruit, screen, servePolicy makers are worrying that AFDC is creatingAction Model Underlying Learnfare Component Plan Actual implementationTarget populationletter regardingabsencesfrom school toImplementing organization Welfare agencies As plannedAssociateabsences and Ecological context Support fromdecision makersand the public As planned, but lacked understanding ofscope of programhighly successful.External ValiditypartnersConcept of unintended effects: Negative vs. positive2). Assessing the implementation of the action modelTutoring andrecreation activities Improving homeworkEngaging insports andoutdoor activities Reducing juvenile delinquencyUnintended effects? Enhancingschool performanceIntegrative validity model•EBIs : Interventions proven efficacious by rigorous •2. Effectiveness evaluation: Providing evidence ofits generalizability (external validity)3. Dissemination•••Many funding agencies, researchers and healthpromotion/social betterment evaluators are attracted to them.Limitations:1. 2. 3.4.EBIs can not be implemented by stakeholders with high ••National Cooperative Inner -City Asthma Study•food/refreshments during counseling,--frequent contacts with participants,--counseling sessions were held at regular hours,etc.world: Many adaptations and changes.•Only 25% of the children completed the interventionWhat is a good intervention according to researchers EBIWhat is a good intervention according to stakeholders ’PROGRAM THEORY Intervention Determinants OutcomesChange Model•••••* The model is an expansion of the distinction of internal and external validity by Campbell and Stanley ’s (1963)Components:eCampbell: Internal validity (effectual validity): External validity (transferable validity)–Stakeholders: ?)••Methodology: Mixed methods (e.g., pretest -–Start with addressing viable validity (viable evaluation), then maximize effectual validity (efficacy evaluation)capable of transferability are worthy of efficacy evaluationEfficacy Evaluation Effectiveness EvaluationDisseminationEffectiveness EvaluationEfficacyEvaluationDissemination Top-DownApproachExample:Intervention: provide free condoms and educationalTransmission•Netherlands in 1984.••conducting effectiveness evaluations.•are used to evaluate the program andconfirmed its efficacy.••Researchers ’innovative interventions •(fidelity)? upon the implementation standardize it?assessment)Recognizing it as preliminaryevidence, indicating thatan intervention is on the righttrack。