statistics

College Learning in the Spotlight (U.S. Policy Context)
“As other nations rapidly improve their higher education systems, we are disturbed by evidence that the quality of student learning at U.S. colleges and universities is inadequate, and in some cases, declining.”
Distinguishing characteristics Direct measures (as opposed to student reports) NOT multiple choice Holistic assessment based on open-ended prompts representing “real-world” scenarios
A Test of Leadership U.S. Secretary of Education‟s Commission on the Future of Higher Education (2006)
Measurement of Learning in U.S. Higher Education
Dearth of direct measures of higher education
student learning that are comparable across institutions and/or states
Measuring Up 2008 – Assigned a grade of
Collegiate Learning Assessment (CLA)
Components Performance task Make an argument Break an argument
Performance Task (example)
You are the assistant to Pat Williams, the president of DynaTech, a company that makes precision electronic instruments and navigational equipment. Sally Evans, a member of DynaTech‟s sales force, recommended that DynaTech buy a small private plane (a SwiftAir 235) that she and other members of the sales force could use to visit customers. Pat was about to approve the purchase when there was an accident involving a SwiftAir 235.
Determinants of College Learning Dataset
Longitudinal Design Fall 2005 and Spring 2007 (beginning of freshman and end of sophomore years) Large Scale 24 diverse four-year institutions; 2,341 students Breath of Information Family background and high school information, college experiences and contexts, college transcripts Collegiate Learning Assessment (CLA)
Measurement Challenges
Curriculum varies widely across fields of study
and institutions – little consensus on what is to be learned
Practitioner resistance to “reductionist”
institutional factors account for variation across disadvantaged groups?
Overview of the Conceptual Model Employed in the Study
Measures of Disadvantage: Race/Ethnicity Parental Education Parental Occupation Non-English Language Racially Segregated High School (70+ % minority) Control Variables: 2005 Test Score Sibling Number Gender Urbanicity Two Parent Household Geographic Region 2007 Test Score High School Academic Preparation: GPA Number of AP Courses Taken
ห้องสมุดไป่ตู้
approaches
Students are sorted by ability and other factors
into different institutions
Collegiate Learning Assessment (CLA)
Dimensions of learning assessed critical thinking, analytical reasoning, and written communication
College Experiences: Hours Spent Studying Alone Hours Spent in a Fraternity/Sorority Faculty Expectations College Fixed Effects Hours Spent Studying with Peers Hours Worked On/Off Campus Field of Study
are associated with learning in higher education?
How do disadvantaged groups of students fare in
college in terms of measured learning?
To what extent do individual, social and
“These shortcomings have real-world consequences. Employers report repeatedly that many new graduates they hire are not prepared to work, lacking the critical thinking, writing and problem-solving skills needed in today‟s workplaces.”
A Test of Leadership U.S. Secretary of Education‟s Commission on the Future of Higher Education (2006)
College Learning in the Spotlight (U.S. Policy Context)
Models for Future Comparative Measurement of Higher Education Learning:
Lessons from the Collegiate Learning Assessment Longitudinal Study in the U.S.*
Incomplete to all states in the area of measuring learning: “All states receive an „incomplete‟ in learning because there are not sufficient data to allow meaningful state-by-state comparisons.”
Sample Characteristics: Who are These Students?
CLA Analysis Sample Demographics Male White African-American Hispanic Asian 0.37 0.59 0.19 0.05 0.11 1052.83 1212.83 22.05 26.29 0.46 0.61 0.14 0.08 0.10 995.15 1219.02 20.86 25.77 0.45 0.59 0.13 0.13 0.06 993.14 1219.23 20.33 25.31 IPEDS – CLA Schools IPEDS – All Schools
Performance Task (example, cont.)
Students are provided with a set of materials (e.g. newspaper articles, Federal Accident Report, e-mail exchanges, description and performance characteristics of AirSwift 235 and another model, etc.) and asked to prepare a memo that addresses several questions, including what data support or refute the claim that the type of wing on the SwiftAir 235 leads to more in-flight breakups, what other factors may have contributed to the accident and should be taken into account, and their overall recommendation about whether or not DynaTech should purchase the plane.
Test Scores
SAT, 25th percentile SAT, 75th percentile ACT, 25th percentile ACT, 75th percentile
Research Questions
What individual, social and institutional factors
Richard Arum New York University and Social Science Research Council
* Josipa Roksa (University of Virginia) and Melissa Velez (NYU) collaborated on research findings presented here. We thank Ford and Lumina Foundations for their generous financial support and the Council for Aid to Education for assistance with data collection.