Lesson Plan 9

What students had as an assignment: Assignment 8

[15 mins] Part 1: Motivation for Discussing CS Education Research and Situating the Field

  • Motivation:
    • If you, as a lecturer, learn something and want to share, you need to be aware of the field. It can also be helpful to be able to evaluate research so you can determine what is useful
    • Here at CMU, we have the Eberly Center who will parse research and share what is interesting and innovative, but not all universities have that
  • Situating the Field:
    • Q: In your own words (based on the Amy Co article), what is the difference between teaching and research?
    • Q: What is the difference between CS and CER, according to Amy Ko/Guzdial?
      • CS (more narrow, focused on specific topics) vs computing (more general, spanning a variety of issues)
    • CS/CER vs other DBR
      • CS/CER is part of discipline-based education research (like Math, Bio, and Physics) as well as part of ed & learning sciences research more in general.
      • Q: How is CS/CER different from other DBR? (based on Guzdial)
        • Young field (less knowledge of how to teach and how status best learn)
        • Not have the same mature and independent status of other content area education fields. (Lishinski et al.)
        • Fewer practitioners and researchers
        • Less research-informed based practice
        • Enormous demand for CS/computing

[45 mins] Part 2: Research in CS/CER vs “True” Education Research

  • A general lack of rigor in CS Ed research, supported by Lishinski et al., my own experience reading publications of CS/CER.
  • Let’s start with: How were the papers from Xhakaj and Holstein different in your view? Feel free to criticize my paper, I picked it on purpose! Areas to focus discussion
    • Publication venue
    • Research Question
    • Theoretical foundation
    • Methodology
      • Tool designed with the user in mind vs not
      • Proper experimental design vs 1 condition
      • Statistical significance and power analysis vs not
      • Evaluation metrics

General discussion of the issues of CS/CER research vs education research.

Issue 1: Research paper vs experience reports

  • We would hope there is a clear line, but there isn’t always.
  • Amy Ko says: “Unfortunately, the community hasn’t developed much clarity about the differences between these. The result is that many papers published in the SIGCSE experience report track look like research papers, and many of the papers published in the SIGCSE research track look like experience reports.”
  • Q: Why is this the case?
    • Based on Lishinski et al. “ Computer science education research is largely done by practitioners, computer scientists who teach and computer science teachers, rather than dedicated education researchers with specialized training in social science research methods and theory [17]”.
    • Aka, this is a largely practitioner-driven field.
  • Q: Why does this matter? Issues with rigor and innovation

Issue 2: Innovation/usefulness/importance of CS/CER

  • Based on Guzdial’s reading, what he was told:
    • Claim: “The things I’m working on have already been done in education research.” A lot of work does not further the understanding of education, but rather just computing specifically.
  • Q: Counterclaim: More interpreting and applying (and some innovating) is necessary for CS/CER (one could argue education more in general). As Lauren Herckis says, applying research in practice is hard (and maybe the issue is with the research, which is not generalizable enough!)
  • Part of the reason, a lot of researchers see teaching as “just teaching.”
  • Part of the issue is that we do not have research for every aspect of teaching & learning.
  • Fran’s take: I see a lot of (especially older) CS/CER research as almost in Bain territory sometimes, while more traditional ED research as HLW.
  • Side note: The only “real” research is CS/Bio/Physics! Ed is also looked down upon. Similar to HCI as well.

Issue 3: Theoretical and Methodological Rigor

  • Q: From Lishinski et al.:
  • Methodologies in CS/CER:
    • Q: What is the state of the art based on Lishinski et al.?
      • Being clarified over the years. A bit more solid
      • CSEdResearch.org
      • Alex Lishinski et al. study, “Overall, our analysis shows a significant increase in the proportion of articles drawing on theory from outside CS education, compared to earlier literature reviews, whereas indicators of methodological quality show no such change.”
    • Q: What are the reasons for this state-of-the-art?
      • People with no research training (practitioners, people with PhDs in CS)
      • Very few people like myself to bridge that gap (and I don’t do research funny enough)
  • Methodologies in Ed:
    • Strong theoretical foundations
    • Methodology: Much more rigorous, show Trochim book.
    • Reason? Older field with a strong connection to Psych
    • Q: From your reading what was interesting to you about Ed Research?
      • (First thing HCI teaches and Fran did.. ) Hypothesis. Experiment design (qual, aunt, mixed, etc.), evaluation instruments, self report vs validated measures.
      • Fran notes hypothesis registering in psych: https://www.psychologicalscience.org/observer/research-preregistration-101
      • “Preregistering a research project involves creating a permanent record of your study plans before you look at the data. The plan is stored in a date-stamped, uneditable file in a secure online archive. You can give others (e.g., reviewers) access to the preregistered plan, and you can do so while maintaining your and the reviewers’ anonymity. The main purpose of preregistration is to make clear which hypotheses and analyses were specified a priori and which were more exploratory and driven by the data.”

[15 mins] Part 3: How to do Proper Research in CER/CS Education

  • Q: From the reading, what do you think you need to conduct CS/CER research:
    • Some computing
    • Familiarize yourself with prior work in the field
    • Familiarize yourself with theoretical foundations in the field
    • Familiarize yourself with research methods in the field
    • Or find a collaborator!
  • PhD in CS is different from Ed or learning science. Your training is different, you are an expert in some skills Like Amy Ko says

    “For example, if you go to a CS Ph.D. program, you’re going to learn about the latest research in various areas of CS, be surrounded by people interested in computing, but possibly not many interested in computing education; you’ll have resources for getting faculty jobs in CS departments, but not really Colleges of Education. In contrast, if you go to a College or School of Education Ph.D. program, you’re going to learn about the latest knowledge in education and learning sciences, and be surrounded by people passionate about learning, equity, and justice, but possibly not many people interested in computing. And if you go to a place like an information school, you’ll gain new perspectives about data and computing, be surrounded by a radical diversity of people with interests that span many disciplines, but possibly one of only a few people interested in computing education.”

  • Q: What are some venues that could be of interest to you if you want to publish a paper about something you did when teaching a class?
    • CS Specific: ICER
    • Ed specific: ICLS, JLE, CSCL, L@S, AIED, CHI, AERA (practitioner-based)
    • CS practice: SIGCE (ACM Special Interest Groups (SIGs)), ITICSE

[15 mins] BREAK

[60 mins] Part 4: In-Class Activity: Presenting Learning Objectives

  • Each student takes a turn presenting their final learning objectives from the assignment to the rest of the class