Learning from data: assessing outcomes and measuring success

panel at Coursera Conference, April 6, 2013, UPenn

homepage

How long do you think it takes a student to watch video material that corresponds to 2 hours lecture? Range from 1.5h to 5h for different MOOCs - there is not “ONE MOOC”, cannot say “MOOCs are good”.

The need for reasonable over-generalisations

Key questions:

• monitoring: what's happening in our MOOCs → adaptation/design
• What else do we expect from Coursera?
  • survey tools
  • items analysis

This isn't new, but we need to replicate findings to be able to generalize.

Data is not only about data from Coursera. We organized study groups, for people on campus, come to a lab (teams of five), watch MOOCs together. Individual or shared player. They love it, first time we finish the lecture, and we understood it - because we paused, replayed, explained to peers, etc. In a lecture, you can't ask the professor “can you be silent for five minutes so I can explain for my friend”. Students say they miss pause-button when they go back to live lectures.

On week 3, it became interactive. Why, what happened? Logfile won't tell us, but we can combine with video analysis.

Eye-tracking experiment on MOOC video. Sharma/Jermann/Dillenbourg. Overlap gaze of four people on the video

homepage

What sort of MOOC-style innovation will be applicable to our offering in Illinois? Starting to gather data across all of our courses, building profile of who are taking courses, motivations, perceptions.

Two surveys. Very international audience. China surprisingly poorly represented.

Mean years of schooling 16, 40% of students have MAs.

Got very high ratings, 88% says “course was great” - but what does that mean? We don't have comparison right now. Need to collaborate with other institutions, are these results typical, better than average, etc?

A/B test, pedagogical designs. Results not ready. Do more testing as we re-offer already designed courses.

Challenges:

• difficult to manage large amounts of data from many courses
  • learning curve on making meaning from the data
• combining data from Coursera, Illinois (online) and Illinois (F2F)

homepage

General interest highest, can't afford formal education very low. Varies across courses/topics. Pretty sustained that about 30% in courses say “We're just curious”. Don't feel hurt that they don't finish the course.

Hard to specify enrolment if you leave it open - at what point in time?

How many students are still watching lectures at the midpoint is a partially good predictor.

homepage

Data we have now, and data we hope to have.

Data structure document

Three components (+1)

• forum data (public, can be browsed by anyone) - don't need to worry about privacy since it's already out there
• general course data - course grades, quizzes etc
• written assignments - peer-graded/programming assignments (tricky in terms of privacy, might have people who sign their name, allude to personal information etc)
• click-stream logs, click by click view of where students go, video playing, pausing/seeking etc.

Putting all the info together in an export raises privacy concerns. Separate these three components out.

Getting the data - institutions have data coordinators

5% of students who enrol make it to the end. (4-6% 25-75% percentile).

Red is a million students, color is number of students. Aggregate for lots of courses.

Populations of students are not homogenous, even with just watching lectures alone, some are motivated and make it to the end, others drop off along the way.

was also shown during keynote on day 1

And how can we capture the value for students who don't intend to complete, but still want something out of the course?

• Lytics Lab at Stanford - doing research on MOOCs
• MOOCship - workshop on MOOC-research
• How can Coursera enable sharing of research among community?
  • not many partners have asked about data export so far
  • need to develop systems that centralize ability to share - we will have better systems
  • Dillenbourg: we can share data, but you need to know very detailed information about what was going on in the MOOC, conceptualized research to avoid “silly conclusions”
  • need taxonomy of courses - categories etc - reasonable generalizations
  • researcher who wants to analyze the data needs to take the MOOC first
• ethics and human subject research, how will that change?
  • Tom: data brokering happens between course ops and university data coordinator - big responsibility, they are figuring out ERB issues, when a study needs approval, etc. Need for more guidance on ERB issues, many data coordinators trying to solve this problem in parallel. Common practices for ERBs (MOOC on MOOC forums).
  • Jim: lot of work to do ERBs, Coursera is new. Coursera Terms of Use - you agree that your data will be shared with partner-institutions. Huge experiment. But can't make access to a course contingent on participating in a research study.
  • Yvonne: is it internal evaluation or research, is it secondary data or not? Need conversations with ERB chairs. Templates from Coursera will be very useful once we start doing inter-university collaboration.
• Why do we need to go through so many contorsions to collect data that Google collects every day?
  • Prior experience in 23andme, we had to go through ERB experience (to be able to publish), at Coursera we don't do ERB for internal research that we share at conferences.
• We should make data available for analysis when we publish. Not a traditional in educational science, but maybe we should learn from physics, etc.
• Use of external survey tools vs built-in quiz tool. Support in the platform for external survey tools in the future?
  • Recognize deficiencies of current survey tool - people want to be able to do A/B testing with email surveys, etc - will roll out next week a new emailing infrastructure, very specifically attach user-IDs (anonmyized) as a macro in the e-mail, can send out customized survey-link. Can also differentially send different surveys to different populations. Stop-gap feature.