Quasi-Experimental Designs

Fearon J D1991Counterfactuals and hypothesis testing in political science.World Politics43:169–95Goldberger A1983Abnormal selection bias.In:Karlin S, Amemyia T,Goodman L A(eds.)Studies in Econometrics, Time Series,and Multi ariate Statistics.Academic Press,New YorkGoldstein H1987Multile el Models in Educational and Social Research.Oxford University Press,New YorkHeckman J J1988The microeconomic evaluation of social programs and economic institutions.In:Chung-Hua Series of Lectures by In ited Eminent Economists no.14.The Institute of Economics,Academia Sinica,Teipei,ChinaHsiao C1986Analysis of Panel Data.Cambridge University Press,Cambridge,UKIsaac L W,Griffin L J1989A historicism in time-series analyses of historical process.American Sociological Re iew54: 873–90Jackman R1986The politics of economic growth in industrial democracies,1974–1980:Leftist strength or North Sea oil? Journal of Politics49:242–56Janoski T,Hicks A(eds.)1994The Comparati e Political Economy of the Welfare State.Cambridge University Press, Cambridge,UKLange P,Jarrett G1985The politics of growth.Journal of Politics47:792–827Maddala G S1983Limited-dependent and Qualitati e Variables in Econometrics.Cambridge University Press,Cambridge, UKNickell S1997Unemployment and labor market rigidities: Europe versus North America.Journal of Economic Per-specti es11(3):55–74Nieuwbeerta P,Ultee W1999Class voting in Western industrial-ized countries,1945–1990:systematizing and testing explan-ations.European Journal of Political Research35:123–60 Przeworski A,Alvarez M R,Cheibub J A,Limongi F in press Democracy and De elopment:Political Regimes and Material Welfare in the World,1950–1990.Cambridge Uni-versity Press,New YorkRagin C1987The Comparati e Method.University of California Press,Berkeley,CARagin C1994A qualitative comparative analysis of pension systems.In:Janoski T,Hicks A M(eds.)The Comparati e Political Economy of the Welfare State.Cambridge University Press,Cambridge,UK,pp.320–45Stimson J A1985Regression in time and space:A statistical essay.American Journal of Political Science29:914–47 Western B1998a Causal heterogeneity in comparative research: A Bayesian hierarchical modeling approach.American Journal of Political Science42:1233–59Western B1998b Between Class and Market:Postwar Unioniza-tion in the Capitalist Democracies.Princeton University Press, Princeton,NJG.Esping-Andersen and A.PrzeworskiQuasi-Experimental DesignsThe theory of quasi-experimentation addresses one of the great unanswered questions in the social and behavioral sciences:how can researchers make a valid causal inference when units cannot be randomly assigned to conditions?Like all experiments,quasi-experiments deliberately manipulate presumed causes to discover their effects.However,quasi-experiments are also defined by what they do not do:the researcher does not assign units to conditions randomly.Instead, quasi-experiments use a combination of design features,practical logic,and statistical analysis to show that the presumed cause is likely to be responsible for the observed effect,and other causes are not.The term nonrandomized experiment is synonymous with quasi-experiment;and the terms observational study and nonexperimental design often include quasi-experiments as a subset.This article discusses the need for quasi-experimentation,describes the kinds of designs that fall into this class of methods,reviews the intellectual and practical history of these designs,and notes important current developments.1.The Need for Quasi-experimentationGiven the desirable properties of randomized experi-ments,one might question why quasi-experiments are needed.When properly implemented,randomized experiments yield unbiased estimates of treatment effects,accompanied by known probabilities of error in identifying effect size.Quasi-experimental designs do not have these properties.Yet quasi-experiments are necessary in the arsenal of science because it is not always possible to randomize.Ethical constraints may preclude withholding treatment from needy people based on chance,those who administer treatment may refuse to honor randomization,or questions about program effects may arise after a treatment was already implemented so that randomization is im-possible.Consequently,the use of quasi-experimental designs is frequent and inevitable in practice.2.Kinds of Quasi-experimental DesignsThe range of quasi-experimental designs is large, including but not limited to:(a)Nonequivalent control group designs in which the outcomes of two or more treatment or comparison conditions are studied but the experimenter does not control assignment to conditions;(b)Interrupted time series designs in which many consecutive observations over time(proto-typically100)are available on an outcome,and treatment is introduced in the midst of those observa-tions to demonstrate its impact on the outcome through a discontinuity in the time series after treat-ment;(c)Regression discontinuity designs in which the experimenter uses a cutoffscore on a measured variable to determine eligibility for treatment,and12655Quasi-Experimental Designsan effect is observed if the regression line(of the assignment variable on outcome)for the treatment group is discontinuous from that of the comparison group at the cutoffscore;(d)Single-case designs in which one participant is repeatedly observed over time (usually on fewer occasions than in time series)while the scheduling and dose of treatment are manipulated to demonstrate that treatment controls outcome.In the preceding designs,treatment is manipulated, and outcome is then observed.Two other classes of designs are sometimes included as quasi-experiments, even though the presumed cause is not manipulated (and often not even manipulable)prior to observing the outcome.In(e)case–control designs,a group with an outcome of interest is compared to a group without that outcome to see if they differ retrospectively in exposure to possible causes in the past;and in(f) correlational designs,observations on possible treat-ments and outcomes are observed simultaneously, often with a survey,to see if they are related.Because these designs do not ensure that cause precedes effect, as it must logically do,they usually yield more equivocal causal inferences.3.The History of Quasi-experimental Designs Quasi-experimental designs have an even longer his-tory than randomized experiments.For example, around1,850epidemiologists used case–control meth-ods to identify contaminated water supplies as the cause of cholera in London(Schlesselman1982),and in1898,Triplett used a nonequivalent control group design to show that the presence of audience and competitors improved the performance of bicyclists. In fact,nearly all experiments conducted prior to Fisher’s work were quasi-experiments.However,it was not until1963that the term quasi-experiment was coined by Campbell and Stanley (1963)to describe this class of designs.Campbell and his colleagues(Cook and Campbell1979,Shadish et al.in press)extended the theory and practice of these designs in three ways.First,they described a large number of these designs,including variations of the designs described above.For example,some quasi-experimental designs are inherently longitudinal(e.g., time series,single case designs),observing participants over time,but other designs can be made longitudinal by adding more observations before or after treatment. Similarly,more than one treatment or control group can be used,and the designs can be combined,as when adding a nonequivalent control group to a time series. Second,Campbell developed a method to evaluate the quality of causal inferences resulting from quasi-experimental designs—a validity typology that was elaborated in Cook and Campbell(1979).The ty-pology includes four validity types and threats to validity for each type.Threats are common reasons why researchers may be wrong about the causal inferences they draw.Statistical conclusion validity concerns inferences about whether and how much presumed cause and effect co-vary;examples of threats to statistical conclusion validity include low statistical power,violated assumptions of statistical tests,and inaccurate effect size estimates.Internal validity con-cerns inferences that observed co-variation is due to the presumed treatment causing the presumed out-come;examples include history(extraneous events that could also cause the effect),maturation(natural growth processes that could cause an observed change),and selection(differences between groups before treatment that may cause differences after treatment).Construct validity concerns inferences about higher-order constructs that research operations represent;threats include experimenter expectancy effects whereby participants react to what they believe the experimenter wants to observe rather than to the intended treatment,and mono-operation bias in which researchers use only one measure that reflects a construct imperfectly or incorrectly.External validity concerns inferences about generalizing a causal relationship over variations in units,treatments, observations,settings,and times;threats include interactions of the treatment with other features of the design that produce unique effects that would not otherwise be observed.Third,Campbell’s theory emphasized addressing threats to validity using design features—things that a researcher can manipulate to prevent a threat from occurring or to diagnose its presence and potential impact on study results(see Table1).For example, suppose maturation(normal development)is an an-ticipated threat to validity because it could cause a pretest–post-test change like that attributed to the treatment.The inclusion of several consecutive pre-tests before treatment can indicate whether the rate of maturation before treatment is similar to the rate of change from during and after treatment.If it is similar, maturation is a threat.All quasi-experiments are combinations of these design features,thoughtfully chosen to diagnose or rule out threats to validity in a particular context.Conversely,Campbell was skep-tical about the more difficult task of trying to adjust threats statistically after they have already occurred. The reason is that statistical adjustments require making assumptions,the validity of which are usually impossible to test,and some of which are dubious (e.g.,that the selection model is known fully,or that the functional form of errors is known).Other scholars during this time were also interested in causal inferences in quasi-experiments,such as Cochran(1965)in statistics,Heckman(1979)in economics,and Hill(1953)in epidemiology.However, Campbell’s work was unique for its extensive emphasis on design rather than statistical analysis,for its theory of how to evaluate causal inferences,and for its sustained development of quasi-experimental theory12656Quasi-Experimental DesignsTable1Design elements used in constructing quasi-experimentsAssignment(Control of assignment strategies to increase group comparability)Cutoff-based assignment.Controlled assignment to conditions based solely on one or more fully measured covariates.This can yield an unbiased effect estimateOther nonrandom assignment.Various forms of‘haphazard’assignment that sometimes approximate randomization(e.g.,alternating assignment in a two condition quasi-experiment whereby every other unit is assigned to one condition,etc.)Matching and stratifying.Efforts to create groups equivalent on observed covariates in ways that are stable,do not lead to regression artifacts,and are correlated with the outcome.Preference is for pretreatment measures of the outcome itselfMeasurement(Use of measures to learn whether threats to causal inference actually operate)Post-test observationsNonequi alent dependent ariables.Measures that are not sensitive to the causal forces of the treatment,but are sensitive to all or most of the confounding causal forces that might lead to false conclusions about treatment effects(if such measures show no effect,but the outcome measures do show an effect,the causal inference is bolstered because it is less likely due to the confounds)Multiple substanti e ed to assess whether the treatment affects a complex pattern of theoretically predicted outcomesPretest observationsSingle pretest.A pretreatment measure on the outcome variable,useful to help diagnose selection biasRetrospecti e pretest.Reconstructed pretests when actual pretests are not feasible—by itself,a very weak design feature,but sometimes better than nothingProxy pretest.When a true pretest is not feasible,a pretest on a variable correlated with the outcome—also often weak by itselfMultiple pretest time points on the outcome.Helps reveal pretreatment trends or regression artifacts that might complicate causal inferencePretests on independent samples.When a pretest is not feasible on the treated sample,one is obtained from a randomly equivalent sampleComplex predictions such as predicted interactions.Successfully predicted interactions lend support to causal inference because alternative explanations become less plausibleMeasurement of threats to internal alidity.Help diagnose the presence of specific threats to the inference that A caused B such as whether units actively sought out additional treatments outside the experimentComparison groups(Selecting comparisons that are‘less nonequivalent’or that bracket the treatment group at the pretest(s))Single nonequi alent pared to studies without control groups,using a nonequivalent control group helps identify many plausible threats to validityMultiple nonequi alent groups.Serve several functions.For instance,groups are selected that are as similar as possible to the treated group but at least one outperforms it initially and at least one underperforms it,thus bracketing the treated groupparison groups chosen from the same institution in a different cycle(e.g.,sibling controls in families or last year’s students in schools)Internal( s.external)controls.Plausibly chosen from within the same population(e.g.,within the same school rather than from a different school)Treatment(Manipulations of the treatment to demonstrate that treatment variability affects outcome variability) Remo ed treatments.Showing an effect diminishes if treatment is removedRepeated treatments.Reintroduces treatments after they have been removed from some group—common in laboratory sciences or where treatments have short-term effectsSwitching replications.Reverses treatment and control group roles so that one group is the control while the other receives treatment,but the controls receive treatment later while the original treatment group receives no further treatment or has treatment removedRe ersed treatments.Provides a conceptually similar treatment that reverses an effect—e.g.,reducing access to a computer for some students but increasing access for othersDosage ariation.Demonstrates that outcome responds systematically to different levels of treatmentand method over four decades.Both the theory and the methods he outlined were widely adopted in practice during the last half of the twentieth century,and his terms like internal and external validity became so much a part of the scientific lexicon that today they are often used without reference to Campbell.12657Quasi-Experimental Designs4.Contemporary Research aboutQuasi-experimental Design4.1Statistics and Quasi-experimental Design Although work on the statistical analysis of quasi-experimental designs deserves separate treatment, several contemporary developments deserve mention here.One is the work of statisticians such as Paul Holland,Paul Rosenbaum,and Donald Rubin on statistical models for quasi-experimental designs(e.g., Rosenbaum1995).They emphasize the need to measure what would have happened to treatment participants without treatment(the counterfactual), and focus on statistics that can improve estimates of the counterfactual without randomization.A central method uses propensity scores,a predicted probability of group membership obtained from logistic regres-sion of actual group membership on predictors of outcome or of how participants got into treatment. Matching,stratifying,or co-varying on the propensity score can balance nonequivalent groups on those predictors,but they cannot balance groups for un-observed variables,so hidden bias may remain.Hence these statisticians have developed sensitivity analyses to measure how much hidden bias would be necessary to change an effect in important ways.Both propensity scores and sensitivity analysis are promising develop-ments warranting wider exploration in quasi-experi-mental designs.A second statistical development has been pursued mostly by economists,especially James Heckman and his colleagues,called selection bias modeling(Winship and Mare1992).The aim is to remove hidden bias in effect estimates from quasi-experiments by modeling the selection process.In principle the statistical models are exciting,but in practice they have been less successful.A series of studies in the1980s and1990s found that effect estimates from selection bias models did not match results from randomized experiments. Economists responded with various adjustments to these models,and proposed tests for their appropriate application,but so far results remain discouraging. Most recently,some economists have improved results by combining selection bias models with propensity scores.Although selection bias models cannot yet be recommended for widespread adoption,this topic continues to develop rapidly and serious scholars must attend to it.For example,other economists have developed useful econometrically based sensitivity analyses.Along with the incorporation of propensity scores,this may promise a future convergence in statistical and econometric literatures.A third development is the use of structural equation modeling(SEM)to study causal relationships in quasi-experiments,but this effort has also been only partly successful(Bollen1989).The capacity of SEM to model latent variables can sometimes reduce problems of bias caused by unreliability of measurement,but its capacity to generate unbiased effect estimates is hamstrung by the same lack of knowledge of selection that thwarts selection bias models.4.2The Empirical Program of Quasi-experimental DesignMany features of quasi-experimentation pertain to matters of empirical fact that cannot be resolved by statistical theory or logical analysis.For these features, a theory of quasi-experimental design benefits from empirical research about these facts.Shadish(2000) has presented an extended discussion of what such an empirical program of quasi-experimentation might look like,including studies of questions like the following:(a)Can quasi-experiments yield accurate effect estimates,and if so,under what conditions?(b)Which threats to validity actually occur in practice(e.g.,pretest sensitization,experimenter ex-pectancy effects),and if so,under what conditions?(c)Do the design features in Table1improve causal inference when applied to quasi-experimental design, and if so,under what conditions?Some of this empirical research has already been conducted(see Shadish2000,for examples).The methodologies used to investigate such questions are eclectic,including case studies,surveys,literature reviews(quantitative and qualitative),and experi-ments themselves.Until very recently,however,these studies have generally not been systematically used to critique and improve quasi-experimental theory.5.ConclusionThree important factors have converged at the end of the twentieth century to create the conditions under which the development of better quasi-experimen-tation may be possible.First,over30years of practical experience with quasi-experimental designs have pro-vided a database from which we can conduct empirical studies of the theory.Second,after decades of focus on randomized designs,statisticians and economists have turned their attention to improving quasi-experimen-tal designs.Third,the computer revolution provided both theorists and practitioners with increased ca-pacity to invent and use more sophisticated and computationally intense methods for improving quasi-experiments.Each in their own way,these three factors have taken us several steps closer to answering that great unanswered question with which this article began.See also:Campbell,Donald Thomas(1916–96); Comparative Studies:Method and Design;Exper-imental Design:Overview;Experimental Design:12658Quasi-Experimental DesignsRandomization and Social Experiments;Experi-mentation in Psychology,History of;Panel Surveys: Uses and ApplicationsBibliographyBollen K A1989Structural Equations with Latent Variables. Wiley,New YorkCampbell D T,Stanley J C C1963\1966Experimental and Quasi-experimental Designs for Research.Rand-McNally, ChicagoCochran W G1965The planning of observational studies in human populations.Journal of the Royal Statistical Society, Series A General128:234–66Cook T D,Campbell D T1979Quasi-experimentation:Design and Analysis Issues for Field Settings.Rand-McNally,Chicago Heckman J J1979Sample selection bias as a specification error. Econometrica47:153–61Hill A B1953Observation and experiment.The New England Journal of Medicine248:995–1001Rosenbaum P R1995Obser ational Studies.Springer-Verlag, New YorkSchlesselman J J1982Case–Control Studies:Design,Conduct, Analysis.Oxford University Press,New YorkShadish W R2000The empirical program of quasi-experimentation.In:Bickman L(ed.)Validity and Social Experimentation:Donald Campbell’s Legacy.Sage Publica-tions,Thousand Oaks,CAShadish W R,Cook T D,Campbell D T in press Experi-mental and Quasi-experimental Designs for Generalized Causal Inference.Houghton-Mifflin,BostonTriplett N1898The dynamogenic factors in pacemaking and competition.American Journal of Psychology9:507–33 Winship C,Mare R D1992Models for sample selection bias. Annual Re iew of Sociology18:327–50W.R.ShadishQueer Theory‘Queer theory’is a notoriously unstable phrase,and one much in contention.As a new theoretical move-ment with equally new political counterparts,it is in constantflux and development,and is charac-terized more by what it challenges and contests than by what it offers in the shape of a unified social theory. Drawing on the work of theorists such as Eve Sedgwick and Judith Butler,queer theory‘describes those gestures or analytical models which dramatise incoherencies in the allegedly stable relations between chromosomal sex,gender and sexual desire(Jagose 1996).In this sense,queer theory is a challenge to the ‘obvious categories(man,woman,latina,jew,butch, femme),oppositions(man vs.woman,heterosexual vs. homosexual),or equations(gender l sex)upon which conventional notions of sexuality and identity rely’(Hennessy1993).Queer theory argues instead that sexual desire and sexual practices are not reducible or explicable solely in terms of identity categories,such as gender,race,class,or sexual orientation.It is radically anti-essentialist,in that it challenges a notion of homosexuality as intrinsic,fixed,innate,and univer-sally present across time and space.Queer theorists reject any mode of thought that relies on a conception of identity as unified and self-evident(e.g.,I have sex with people of the opposite sex,therefore I must be heterosexual),and instead demonstrate that desires,sexual practices,and gen-dered identities are performances and enactments, rather than expressions of‘true’subjectivity.Hetero-sexuality is therefore challenged by queer theory not simply as a‘hegemonic’mode of identity,but as a false claim to unity and coherence that is constantly undermined by the incoherencies of sex and gender, incoherencies that the queer analytic hopes to expose and celebrate.1.Intellectual OriginsIn the broadest sense,queer theory emerged in what might be called the postmodern moment,when in-tellectual unease with unitary and cohesive frame-works of knowing reached a fever pitch.While impossible to summarize here,queer theory’s alle-giance to postmodern and\or poststructural modes of thought can be traced in its challenge to the notion of unitary identity(as in‘gay’or‘straight’),its refusal to understand sexuality through a singular and unified lens(homosexual desire,feminist theory,gender),a rejection of binary models(gay\straight,man\ woman,biological\social,real\constructed),and a more generic critique of identity-based theories and politics that,according to poststructuralist accounts, invariably reproduce the very conditions of repression they desire to challenge.For example,the term‘gay’or ‘homosexual’might be critiqued as a(fictional)cat-egory that shores up the binary opposition between ‘gay’and‘straight’that is itself part of the repressive logic of identity.To claim‘gayness’is therefore not simply or solely an act of self-revelation but is also a way of corralling sexuality within the framework of a category that only appears coherent but that,when opened up,reveals its instability(e.g.,Am I still gay if I sleep with a person of the opposite sex?Or if I sleep with those of the same sex but only in certain conditions and in certain ways?Or if my self-under-standing is of myself as‘straight?’).Queer theory,in that sense,has developed within and through the deconstructive impulse of poststructuralism,chal-lenging assertions of unitary identity and necessary linkages(between,say,sexual desire and gender orientation)and arguing instead for a more pro-visional,contingent,andfluid conception of the‘queer’in contemporary culture.While queer theory emerges as coterminous with postmodern impulses,it also traces its intellectual origins in lesbian\gay studies and feminist theory even12659Queer TheoryCopyright#2001Elsevier Science Ltd.All rights reserved.International Encyclopedia of the Social&Behavioral Sciences ISBN:0-08-043076-7。