Lesson Plan 6 - Guest Lecture by Dr. Lauren Herckis

What students had as an assignment: Assignment 5

Today was a guest lecture on Pedagogy research in postsecondary education by Dr. Lauren Herckis, Carnegie Mellon University

[10 mins] Part 1: Introduction and Goal-Setting

  • Starts by introducing her lab and their primary research directions which include:
    • Intergrations of technologies into classrooms
  • Introductions and goal-setting. What do you want to know? Some of the answers from students:
    • What her research tells us about adoption of technoogies in classrooms
    • How to improve representation of students in CS especially at higher leveels
    • How to identify research from different communities that could be helpful
    • What are potential successful features of technologies that were incorporated into the classroom
    • How different learning styles can be accounted for in post-secondary education and any research on that
    • How do you ethically evaluate new strategies in classrooms
    • How to cope with evolution of hybrid classrooms
    • How to evaluate efficacy of technologies that are incorporated into classrooms in practice and evaluating whether they are actually a good fit

[50 mins] Part 2: Discussion: Community Sourced, Data-Driven Improvements Reseach Project

  • Community Sourced, Data-Driven Improvements - Tells class about this project that demonstrates the type of work Lauren’s lab does
  • Introducting adaptive learning technologies and OLI
    • Does anybody examples of about adaptive-learning technologies?
    • What makes adaptive-learning technologies different from non-adaptive learning technologies?
      • It is reactive to feedback from the student interactions (e.g., a smart text book with checkpoints throughout that adapts the content shown to the student based on how they perform)
    • How many of you have used or designed for OLI?
    • OLI enables building of course tools and incorporates adapting learning technologies into the materials
    • OLI can provide real-time enagement data on student performance that instructors can use to adapt themselves so they can drive more impactful learning experiences
  • Research Project Goal: There is literature giving evidence that adaptive learning technologies can be better for students and teachers can use data to improve materials, but can we community source some of those improvements? (what if the learners themselves can contribute to improve the materials
  • Thrusts of this research project:
    • Effectiveness research - how can faculty customization and data-driven improvement of courseware
    • Barriers research - why and how do and don’t instructors engage with data-driven improvements of courseware. What are barriers and facilitators to adoption (this is what the focus of this talk will be)
    • Investigated why adopt this courseware, and identified instructional benefits and other benefits
  • What are the barriers of effective use and adopting innovative edtech? Understanding this can allow for the improvements to the technology
  • Findings from this study include:
    • Innovation is viewed as valuable in principle and in retrospect. But most who express intent and desire fail to implement
    • Adoption of innovative technologies is difficult and nontrivial for faculty and students
    • Its hard to know if an innovative tool is a good fit and integration takes an unpredictable amount of time
    • Pandemic stress impedes
    • Transitions are opportunities because they are times of change necessity. Transitions can facilitate and impede
    • Unfamiliar tools take time to learn. It can be challenging or impossible to adjust nomenclature. Outdated content such as examples/datasets may be poor fit
  • Identified Barriers to use for these technologies:
    • Increasing complexity implies increasing time required to plan and execute course plans
    • Modifications take significant effort
    • Deep knowledge of structure and content enale deeper integration with existing materials
    • Interpretation of results requires time and training
    • Translating student outcomes for tactical decision-making requires novel skills and practices
  • Diffusions of innovations framework used to understand intersection of science and technology in society
  • Discussing Diffusion of innovations
    • Innovations- Increasing complexity of innovation represents aded challenge to diffusion
    • Communication Channels- are pathways of diffusion of innovations
    • Time- was a factor in most barriers
  • What do we do this this information? -> This is what implementation science is used for
  • Theres lots of research and educational technological advancements
  • How can we get this technology into the classrooms of instructors who are not researchers, how can we overcome the barriers faced
  • Some opportunities that have demonstrated effect:
    • Novel contexts- of development facilitate and speed up adoption and implementation and support iterative improvement
    • Local champions- ease adoption and implementation by creating robust communication channels
    • Incremental Implementation- change little things slowly

[15 mins] BREAK

[30 mins] Part 3: Discussion: Conducting Education Research

  • Once you design an instructional plan, why do you think it will work and how do you know if it works?
  • How do you know if your class is effective? What are some approaches we might take to evaluate that?
    • Ask students what they thought about the course to evaluate student experience
    • Use tests to evaluate learning gains
    • Look at performance in follow up courses to evaluate transfer of learning
    • Doing an assessment at the beginning of the class and again at the end can allow for comparative assessment of learning growth during the class. Assessments at the beginning of courses are also inmprtant to for getting a sense of student preexisting knowledge
  • Assess student learning by giving them a task to perform that matches the type of tasks you want students to be able to do
  • Why do you think it’ll work. You’ve taken good classes before. What makes them so great?
    • Relevance - connecting learning to something meaningful to students
    • Active learning engagement
  • Mental models of good teaching are sticky
    • Instructors often determine what “good teaching” is based on their individual experiences
    • Formative learning experiences shape teaching practice
    • Mental Models of good teaching
      • Emotion connection and relationships
      • Content delivery
      • Measurable outcomes
      • Active experiences
    • How instructors prioritize these components will influence teaching strategies
  • Faculty are particularly skeptical of evidence-based instructional practices when they do not align with mental models of “good teaching”
  • Research to evaluate educational implementation
    • How can I document implementation fidelity, particularly of the key aspects of the intervention so you know what happened that could potentially help or hinder learning?
  • Research to evaluate educational impact
    • How can I test the impact of key aspects experimentally, by contrasting two instructional approaches, documenting both before and after
    • Active ingredients of the design are important to consider and evaluate
  • Think about the independent variables are that are being manipulated
  • Dependent variables are what the researchers are measuring
  • Co-variates - demographics and other Individual Differences that might be correlated with impact
  • Research Designs
    • Split-Class Design (everyone experiences all as
    • Now & Later Design (everyone in the class gets the treatment but at differnt times)
    • Note: with both these designs, it is important to consider equity among students
    • Stratified Random Assignmnet
    • Counter balanced assessments
  • In Sumary:
    • Education research can measure impact
    • Or can tell us something about implementation
    • Create research-informed designs
    • Evaluate designs and make data-driven improvements

[30 mins] Part 4: Discussion: Readings From Education Resarch Articles

  • Are there pedagogical journals that you didn’t know existed before you went searching for articles? How would you use them in the future?
  • What kinds of articles or topics did you find?