教育外文文献综述

Educational Research Review 5(2010)25–49Contents lists available at ScienceDirectEducational ResearchReviewj o u r n a l h o m e p a g e :w w w.e l s e v i e r.c o m /l o c a t e /E D U R EVThe impact of instructional development in higher education:The state-of-the-art of the researchAnn Stes a ,∗,Mariska Min-Leliveld b ,David Gijbels a ,Peter Van Petegem aaUniversity of Antwerp,Institute for Education and Information Sciences,Venusstraat 35,2000Antwerpen,Belgium b Leiden University,ICLON,P.O.Box 905,2300AX Leiden,The Netherlandsa r t i c l e i n f o Article history:Received 6November 2008Received in revised form 29June 2009Accepted 9July 2009Keywords:Instructional development Higher education Impacta b s t r a c tIn this article we give a systematic review of the nature and design of earlier research into theimpact of instructional development in higher education.Studies are clustered on the basisof the level of outcome that was measured,meaning that another synthesis technique isused than in prior reviews related to the same topic.In addition,we address some questionsrelated to the differential impact of initiatives with varied duration,format,or target group,because these questions were left unanswered in earlier reviews.The results of our reviewprovide a guide to improve studies of instructional development in order to get more insightinto the real impact at different levels (teachers’learning,teachers’behavior,the institution,and the students).Some evidence is found of the influence of the duration and nature ofinstructional development on its impact.©2009Elsevier Ltd.All rights reserved.1.IntroductionInstructional development in higher education has become an important topic in recent years.In spite of its acknowledgedimportance,evaluations have generally been limited to measures of participants’satisfaction:little is known about the impacton daily teaching practice (Eison &Stevens,1995;Norton,Richardson,Hartley,Newstead,&Mayes,2005;Wilson &Berne,1999).What teachers learn from instructional development remains unclear (Fishman,Marx,Best,&Tal,2003).In this article,we will summarize the findings of prior reviews into the effects of instructional development in higher education.We willaddress some questions left unanswered in previous reviews as well as describe the state-of-the-art of the nature (in termsof levels of outcome)and design of existing research.In the past,terminology regarding instructional development was usedinconsistently (Freeth,Hammick,Koppel,Reeves,&Barr,2003;Taylor &Rege Colet,2009).Taylor and Rege Colet (2009)defined the different related terms very clearly.We will begin by referring to their study in order to clarify what we mean inthis article by instructional development .Instructional development can be described as any initiative specifically planned to enhance course design so that studentlearning is supported (Taylor &Rege Colet,2009).This definition excludes:(a)curriculum development,which focuses onthe development and improvement of programs of study as a whole (Cook,2001);and (b)organizational development,which focuses on creating institutional policies and structures that foster an effective learning and teaching environment(Taylor &Rege Colet,2009).Professional development,faculty development and academic development are terms related∗Corresponding author at:University of Antwerp,Centre of Excellence in Higher Education (ECHO),Venusstraat 35,2000Antwerpen,Belgium.Tel.:+3232654353;fax:+3232654501.E-mail addresses:ann.stes@ua.ac.be (A.Stes),mjleliveld@iclon.leidenuniv.nl (M.Min-Leliveld),david.gijbels@ua.ac.be (D.Gijbels),peter.vanpetegem@ua.ac.be (P.Van Petegem).1747-938X/$–see front matter ©2009Elsevier Ltd.All rights reserved.doi:10.1016/rev.2009.07.00126 A.Stes et al./Educational Research Review5(2010)25–49to instructional development as well;however,each concept has its own specific focus.Whereas instructional development explicitly aims to develop faculty in their role as a teacher,professional development concerns the entire career development of a faculty member and is not limited to teaching,but also considers research and social services(Centra,1989).The terms academic development and faculty development have the same focus as the concept of professional development,but they also include the aspect of organizational development as described above.While the term academic development is used in Australasian and British contexts,the term faculty development is common in North America(Taylor&Rege Colet, 2009).The concept of educational development is used by Taylor and Rege Colet(2009)to indicate the whole range of development activities as described above:instructional,curriculum,organizational,professional,academic,and faculty development.2.Past syntheses of instructional development researchFive prior reviews related to the effects of instructional development in higher education(Levinson-Rose&Menges,1981; McAlpine,2003;Prebble et al.,2004;Steinert et al.,2006;Weimer&Lenze,1998)informed the current synthesis.Levinson-Rose and Menges(1981)reported on a synthesis of71studies(from the mid-sixties to1980)of interventions to improve college teaching using narrative review.Studies were clustered on the basis of the kind of intervention they described,using the followingfive categories:(1)grants for faculty projects;(2)workshops and seminars;(3)feedback from student ratings;(4)practice with feedback and(5)concept-based training on the basis offilms or videotapes illustrating educationally relevant concepts.The effect of the intervention as reported in the study was labeled as:(a)self-reported change in teacher attitude;(b)tested or observed change in teacher knowledge;(c)observed change in teacher skill;(d) self-reported change in student attitude;or(e)tested or observed change in student learning.Studies reporting only on effects belonging to one or both of thefirst two categories were excluded from the review.The results indicated positive effects for78%of the interventions studied.However,this percentage diminished to62%when taking into account only those studies where high confidence could be placed in the results.Levinson-Rose and Menges(1981)concluded that quite a lot of the research into the impact of instructional development up to1980was of low quality.Individual differences between participants in a development initiative and/or between students were seldom taken into account.Studies were not comparable with regard to the way dependent variables were operationalized or measured.Qualitative data to reveal deeper levels of experience as well as collaborative research were missing.Weimer and Lenze(1998)updated the review of Levinson-Rose and Menges(1981)by focusing on literature published in the eighties.Their review consideredfive kinds of interventions:(1)workshops,seminars and programs;(2)consultation;(3)instructional grants;(4)resource materials such as newsletters,manuals or sourcebooks;and(5)colleagues helping colleagues.While two of these categories(workshops/seminars and grants)were identified by Levinson-Rose and Menges (1981),the others were not.This was explained by the fact that the character of instructional development had undergone some changes during the eighties(Weimer&Lenze,1998).Weimer and Lenze replaced two of the categories identified by Levinson-Rose and Menges(1981)(namely practice with feedback and concept-based training)by two new ones(resource materials and collegial help),while one,feedback from student ratings,was adapted(i.e.,Weimer and Lenze took into account only feedback in combination with a form of consultation).The studies reviewed were not just clustered on the basis of the kind of intervention they described:if a study described the effect of an intervention meant for a specific target group,this was also taken into account.A distinction was made between interventions targeting new faculty and those targeting teaching assistants.Reported effects were categorized using the samefive categories used by Levinson-Rose and Menges(1981).Whereas Levinson-Rose and Menges(1981)excluded studies reporting only effects concerning change in self-reported teacher attitude and/or in tested or observed teacher knowledge,Weimer and Lenze(1998)did not.Overall, the results of their review were inconclusive about the effects of instructional development in higher education.Weimer and Lenze(1998),as well as Levinson-Rose and Menges(1981)17years before,concluded that more research and especially research with greater sophistication in empirical design should be conducted.Types of inquiry other than quantitative surveys should be used.The effect of instructional interventions on specific faculty groups should be studied.Relatedfields of knowledge(e.g.,regarding adult learning,diffusion of innovation or motivation)should be taken into account.McAlpine(2003)addressed the question of how instructional development initiatives in higher education can best be evaluated.Seven studies(published between1983and2002)reporting on the impact of an instructional workshop at the level of students and/or the institution(not only at the level of the participants)were reviewed.The studies were compared on the basis of characteristics of the workshop described(i.e.,aim,extent of generic and voluntary character,number of participants,duration,and kind of activities)as well as on the basis of methodology(i.e.,design,focus of evaluation,and instruments used).The categorization described by Levinson-Rose and Menges(1981)and Weimer and Lenze(1998)was used to compare the studies regarding their focus of evaluation.In addition,an extra category described by Kreber and Brook(2001)concerning effects on the culture of the organization was included.The description of the studies made clear that in some cases other dependent variables were also examined,namely the learning experiences of students,their study approaches,and the teaching conceptions of teachers(i.e.,the way teachers think about teaching).All studies reviewed paid careful attention to rigorous research design;in each case comparison was made to a control group and three of the seven studies used a pretest/posttest design.It is possible that her consideration only of studies examining an impact beyond the level of the individual participants is why McAlpine found studies with higher-quality design than those found by Levinson-Rose and Menges(1981)and Weimer and Lenze(1998).McAlpine(2003)concluded that measuring the impactA.Stes et al./Educational Research Review5(2010)25–4927 of instructional development initiatives,especially impact that goes beyond the level of the individual participants,is not easy.Future research should concentrate on the development of instruments to measure the impact at the level of students’learning and/or the institution.On request of the Ministry of Education of New Zealand,Prebble et al.(2004)synthesized the research about the impact of student support services and instructional development programs on student outcomes in higher education.Part of their report gave an overview of the research evidence for the effects of instructional development,referring to the earlier reviews by Levinson-Rose and Menges(1981)and Weimer and Lenze(1998).The main focus of the report was studies published between1990and2004.In order to cluster the studies on the basis of the kind of intervention they described,the categories identified by Levinson-Rose and Menges(1981)and Weimer and Lenze(1998)were adapted to take into account new developments in thefield(Prebble et al.,2004).The following categories were used:(1)short training courses;(2)in situ training,whereby an activity is built to meet the objectives of a specific and entire academic group;(3)consulting,peer assessment and mentoring;(4)student assessment of teaching;and(5)intensive staff development.For each category, reported effects were labeled using the categories as determined by Levinson-Rose and Menges(1981).As in the review by McAlpine(2003),an extra category concerning effects at the level of the institution was included.On the basis of the studies reviewed,Prebble et al.(2004)came to the conclusion that short training courses tend to have only limited impact on actual teaching practice and should be reserved for disseminating institutional policy information or for training in concrete techniques.Other forms of instructional development were viewed more favorably:in situ training,(peer)consulting,student assessments,and intensive programs were described as potentially leading to significant improvements in the quality of teaching and student learning.Although Prebble et al.(2004)pointed out limitations and shortcomings of the published research,as did Levinson-Rose and Menges(1981)and Weimer and Lenze(1998),Prebble et al.(2004)paid less attention to these in reporting their synthesis of studies.A discipline-specific review was conducted by Steinert et al.(2006).They synthesized the existing evidence regarding the effects of instructional development interventions in medical education,covering the period of1980–2002.They did not use the categories described by Levinson-Rose and Menges(1981),Weimer and Lenze(1998),or Prebble et al.(2004) to cluster the53studies included in their review on the basis of the kind of intervention.Instead they used the following categorization:(1)workshops;(2)short courses;(3)seminar series;and(4)longitudinal programs and fellowships.Here, the collective,course-like types of instructional interventions were considered,and were distinguished on the basis of their duration.Studies describing the effects of instructional interventions such as grants,student feedback,consultation,or in situ training were categorized as other and were given less consideration.A slightly adapted version of Kirkpatrick’s(1994) model of educational outcomes was used to label the outcomes of the interventions.The following levels of outcome were taken into account:(1)reaction;(2)learning,further subdivided into(2a)change in attitudes and(2b)change in knowledge or skills;(3)behavior;(4)results,further subdivided into(4a)change in the organization;and(4b)change among students, residents or colleagues.We recognized these categories as those introduced by Levinson-Rose and Menges(1981);however they are supplemented with the levels reaction,behavior,organization and residents and colleagues.The level of organization was also considered by McAlpine(2003)as well as by Prebble et al.(2004).Steinert et al.(2006)concluded that the literature on medical education suggested high satisfaction of teachers with instructional development initiatives and positive changes in teachers’attitudes,knowledge,skills and behavior following participation in an instructional development activity.Like McAlpine(2003),they suggested that future research should concentrate on measuring the impact at the level of students’learning and/or the institution,because the impact at these levels was not as frequently investigated.Steinert et al.(2006)also addressed the issue of the methodological quality of the reported studies and concluded that many of the studies employed weak designs.More rigorous research studies should be conducted with randomized controlled trials,comparison groups, qualitative methods,or mixed designs.Evaluation methods other than self-assessments and survey questionnaires should be developed and tested for validity and parable measures should be used across studies.In addition,the durability of change and the interaction between different factors should be given increased attention.In short,Steinert et al.(2006)suggested that new methodologies to assess impact should be developed by collaborating systematically across programs and institutions.3.The need for a new synthesis of the instructional development literatureThefirst reason for conducting a new synthesis of the instructional development literature is to apply a new synthesis technique.The earlier reviews summarized above(Levinson-Rose&Menges,1981;McAlpine,2003;Prebble et al.,2004; Steinert et al.,2006;Weimer&Lenze,1998)made a plea for more collaboration across programs and institutions in order to enhance the knowledge of thefield.However,in these reviews studies were clustered on the basis of the type of intervention they described.Research in other domains has indicated that taking into account the level of measured outcome and the research designs used in the original studies can result in morefine-grainedfindings(e.g.,Dochy,Segers,Van den Bossche, &Gijbels,2003;Gijbels,Dochy,Van den Bossche,&Segers,2005).Our synthesis will cluster studies on the basis of the level of outcome that is measured and will attend explicitly to the research design used.The state-of-the-art of the nature(in terms of levels of outcome)and design of existing research will be described.This will provide a guide for researchers to use similar methods across studies and a comparable measurement of dependent variables,so that studies can build upon one another and collaboration across programs and institutions can be more easily achieved.28 A.Stes et al./Educational Research Review5(2010)25–49Ourfirst research question is:What is the nature(in terms of levels of outcome)and design of instructional development research in higher education?To answer this question we will look at the literature from a similar angle as did McAlpine (2003).She compared studies on the basis of research methodology.However,she only reviewed studies reporting on the impact of an instructional workshop and only considered the impact at the level of students and/or the institution.Studies examining the impact solely at the level of the participants were not taken into account.Moreover,McAlpine(2003)did not report a systematic literature search.Our synthesis will focus on all types of instructional development and will also consider studies solely examining the impact on participants.A systematic literature search will also be reported.Earlier reviews (Levinson-Rose&Menges,1981;McAlpine,2003;Prebble et al.,2004;Weimer&Lenze,1998)pointed out the importance of rigorous research designs.As far back as1981,Levinson-Rose and Menges made a plea for impact research drawing not only on participant self-reports,but also measuring actual changes in performance.Steinert et al.(2006)synthesized the existing evidence regarding the effects of instructional development in medical education,covering the period1980–2002. They predicted an increase in well-designed studies researching behavioral and systems outcomes in thefirstfive years of the twenty-first century.Our synthesis aims to examine whether the suggestions of earlier reviews were put into practice.A second reason for conducting a new synthesis of the instructional development literature is to address some questions related to the differential impact of initiatives with varied duration,format,or target group,because these questions were left unanswered in the earlier reviews summarized above.An important aim of these reviews was to give an overview of the effects of different kinds of instructional development.Earlier evidence and conclusions on such effects from Levinson-Rose and Menges(1981)and Weimer and Lenze(1998)were updated by Prebble et al.(2004)and,with regard to medical education by Steinert et al.(2006).However their reviews still left unanswered some questions that are relevant to investigate,because they have practical implications regarding the implementation of instructional development in higher education.So,a second reason for conducting a new synthesis of the instructional development literature is to determine whether the following four questions can now be answered.1.Do instructional development initiatives extended over time have more positive outcomes than one-time events?Although Weimer and Lenze(1998)analyzed the research on workshops along the dimension of varying length,no conclusions about the effect of length on thefinal impact of the instructional development initiative were drawn.The reviews by McAlpine(2003)and Steinert et al.(2006)suggested that longer programs,extended over time,tended to produce more positive outcomes than one-time interventions.However,it was concluded that further investigation was needed to test out the hypothesis that longer interventions may have more outcomes.2.Do collective,course-like instructional development initiatives have outcomes comparable to those of initiatives that arealternative in nature?The reviews by Levinson-Rose and Menges(1981),Weimer and Lenze(1998),Prebble et al.(2004)and Steinert et al.(2006)all clustered studies on the basis of the kind of intervention they described.Each review used a different categorization in order to take into account new developments in thefield.However,collective,course-like interventions appeared in every categorization;they seem to be a constant in thefield of instructional development.This led us to wonder if their impact is comparable to that of the alternative initiatives whose specific character seems to change over time(e.g.,research grants regarding teaching,peer learning).3.Do instructional development initiatives targeting teaching assistants or new faculty have more positive outcomes thaninitiatives with another or no specific target group?Weimer and Lenze(1998)remarked that targeting teaching assistants and new faculty stems from two assumptions, namely that these target groups have:(a)little or no teaching experience resulting in lower teaching quality;and(b)no tenure yet so that it is easier to encourage them to engage in instructional development.However,Weimer and Lenze (1998)stated that these assumptions were not tested and that it remained unclear if the beginning of the teaching career is indeed the best time to intervene.4.Do instructional development initiatives targeting a discipline-specific group have outcomes comparable to discipline-general initiatives?Because initiatives are sometimes organized as discipline-specific and at other times given a generic character,we would like to examine whether targeting a discipline-specific group makes any difference with regard to the impact.In answering these four questions the results for ourfirst research question regarding the nature and design of instruc-tional development research in higher education will be taken into account.4.Levels of outcome of instructional developmentBecause our synthesis will cluster studies on the basis of the level of outcome measured,it was important to take into account as broad as possible a range of measures of effects.The categorization of levels of outcome used by Steinert et al. (2006),which was an adaptation of Kirkpatrick’s(1994)model of educational outcomes,seemed the most appropriate.It encompassed all levels as described in the reviews by Levinson-Rose and Menges(1981),Weimer and Lenze(1998),McAlpine (2003),and Prebble et al.(2004),by broadening them with the levels reaction,behavior,and residents and colleagues.Holton (1996)questions whether participants’reactions(i.e.,their views on the instructional development learning experience, its organization,presentation,content,teaching methods,materials and quality of instruction;Steinert et al.,2006)can be considered as a measure of impact.He comes to the conclusion that“participants’reactions should be removed fromA.Stes et al./Educational Research Review5(2010)25–4929Table1Kirkpatrick’s model for evaluating outcomes of instructional development.Level DescriptionChange within teachersLearningChange in attitudes Change in attitudes towards teaching and learningChange in conceptions Change in conceptions of(i.e.,in ways of thinking about)teaching and learningChange in knowledge Acquisition of concepts,procedures and principlesChange in skills Acquisition of thinking/problem-solving,psychomotor and social skillsBehavior Transfer of learning to the workplaceInstitutional impact Wider changes in the organization,attributable to the instructional development interventionChange within studentsChange in perceptions Change in students’perceptions of the teaching and learning environmentChange in study approaches Change in students’approaches to studyingChange in learning outcomes Improvement in students’performance as a direct result of the instructional development Note:Kirkpatrick’s(1994)model was modified by Steinert et al.(2006).It was further adapted for this review.evaluation models as a primary outcome of training”(Holton,1996,p.10),they can at a maximum be considered as an intervening variable.Also Guskey(2000)describes participants’reactions as‘happiness indicators’indicating how well participants liked the intervention without addressing the issue of attaining change.Participants’reactions to instructional development do not contribute to a clear picture of its real impact(Weimer&Lenze,1998).We decided to exclude this level in our review.Because colleagues are part of teachers’institutional context,we further adapted the model as described by Steinert et al.(2006)by considering change among colleagues as an example of institutional impact.Taking into account the categorization introduced by Levinson-Rose and Menges(1981),as well as the extra dependent variables noted in McAlpine’s review(2003),we subdivided the category change among students into the following subcategories:students’perceptions, students’study approaches,and students’learning outcomes;and we added the subcategory change in teachers’conceptions to the level of learning.We excluded change among residents since it is a level of outcome typical for medical education,not applicable to other disciplines.In order to structure the categorization somewhat further,the category change among teachers was added,with the levels of learning and behavior belonging to this category.Table1presents ourfinal list of outcomes of interest for this synthesis.5.Methods for research synthesis5.1.Literature search proceduresBecause of the inconsistent use of terminology in the past(Freeth et al.,2003;Taylor&Rege Colet,2009),the literature search for studies into the impact of instructional development in higher education was based on a variety of terms that can refer to instructional development.Based on the conceptualization in a recent study by Taylor and Rege Colet(2009) and ourfirst readings of studies,we composed a list of the following keywords instructional development,instructional training,academic development,faculty development,faculty training,professional development,educational development, educational training,staff development,pedagogical training,and university teacher.In February2008,we conducted searches of the electronic database ERIC.Each time one of the above mentioned terms was indicated as word that had to appear in the title,in combination with the term‘teaching’as word that had to appear in the abstract.The last in order to get the focus we wanted and to exclude for example the large number of studies regarding the professional development of doctors or researchers.We did not limit the search in time or in source of publication.Our search strategy was meant to uncover both published and unpublished research to prevent publication bias.The search resulted in1833citations.We read the abstracts of these citations,except when the titles indicated that the studies would be excluded from the synthesis(e.g.,studies of instructional development in elementary or secondary education).After examining abstracts for relevance to the synthesis based on the criteria described in the next section,101 articles were selected to be screened.Of those,we were able tofind80studies;as we did not limit the search to time or source,21studies were not traceable despite several attempts to contact the authors.5.2.Inclusion criteriaFor a study to be included in the research synthesis,the following criteria had to be met:1.Studies had to concern an instructional development initiative for teachers in higher education.An instructional devel-opment initiative was defined as an initiative specifically planned to enhance course design so that student learning is supported(Taylor&Rege Colet,2009).。