Evaluating OER Effectiveness and Impact

Browse Topics

Human Subject Research and IRB

It is important to consider the extent to which OER related assessments are done as program improvements or qualitative improvement and are not “research” requiring an IRB exemption or approval. Even if the assessment activity is “human subject research”, when data is collected anonymously, the research may fall within the exempt categories 45 CRF 46.101(b)(1) (established educational settings) and (b)(2) (anonymous data collection).

Literature and Systematic Review

CUNY OER and related studies

CUNY OER and related assessments on CUNY Academic Works include:

Questions to consider in reviewing the literature

  1. What is the problem statement? How is the problem, question, or issue framed?
  2. Are the students sampled in the studies similar to those at LGACC (Overall, in 2018, 22% Asian, 17% Black, 48% Hispanic, 11% White and 2% Other/Unknown); 55% full-time; 46% part-time; 59% foreign born; 59% received financial aid)? See 2019 Institutional Profile – LaGuardia Community College
  3. Is the study at the community college level?
  4. What is the sample size/response rate?
  5. What is the theory of change? What are the hypotheses? What are the assumptions?
  6. What types of OER are included in the study?
  7. What disciplines are included?
  8. What is the research design? 
  9. What/who are the comparison groups, if any?  Is there a controlled group? What comparisons are being made?
  10. Has there been a change in learning or teaching? What is/are the change(s)?  Positive, negative, neutral/none?
  11. Are the data and instruments available for review?
Heppich Galaxy 4 Painting
Eppich, Helmut. Galaxy 4. ©199? LaGuardia Community College. Photo by Diana Pizzarello.©2020.

Program and Project evaluation

Research Design

  • OER Research Guidebook (PDF, Word, InDesign), COUP Framework
  • Do open educational resources improve student learning? Implications of the access hypothesis. (Online, PDF)
    The article recommends surveying students before the intervention to see who did not get the textbook and after to see if their access changed. The study should have an explicit hypothesis about why using OER would result in improved learning outcomes.  Under the authors’ access hypothesis, access — getting the textbook — is the primary mechanism for why there might be improved learning. This assumes that if students get the textbook, they will use it, with the caveat that different types of usage can results in different learning outcomes.

Data Collection and Analysis

Datasets

  • First cross-campus survey of CUNY student opinions on Zero Textbook Cost Data (2018 survey data download)
  • OER Hub survey of 7000 OER users and their perceptions of OER impact on teaching and learning (Data CVS , xlsx)

Funder Grantmaking and Reports

William and Flora Hewlett Foundation