George Siemens, Carolyn Rose, Dragan Gasevic, Annika Wolff, Zdenek Zdrahal

Differentiating analytics in education, and learning analytics, the latter is benefitting the learner. Centrality of learner, analytics process to improve instructional process. Leading to actionable insights.

• bricolage of different fields
• citation analysis (Garfield 1995, Page et al 1999). scientometrics? backrubber - PageRank algorithm. Looking at how nodes were linked, and assigning value to links.
• Social Network Analysis - being rediscovered by physics? How we understand social systems
  • Milgram
  • Granovetter
  • Wellman
  • Haythornthwaite
• User modeling
  • Rich 1979
  • Fischer 2001
• cognitive models (how concepts are formed)
  • Ausubel
  • Anderson et al, 1995
• Tutors
  • Burns 1989
• Knowledge discovery in databases
  • Fayyad, 1996
• Adaptive hypermedia
  • Brusilovsky, 2001
• growth of online learning, MOOCs, digital trails – treating data as a mediating agent/process, we can infer states of user competence, motivation, etc.
• related
  • business intelligence
  • academic analytics

• Baker and Yacef (2009): five primary areas of analysis
  • prediction
  • clustering
  • relationship mining
  • distillation of data for human judgment
  • discovery with models
• Bienkowski etc (2012)
  • modeling user knowledge, behaviour and experience
  • profiles of users
  • modeling knowledge domains
  • trend analysis
  • personalization, adaptation

• neuroscience
  • lot's of potential, not ready for broad adoption
• machine learning
  • highest potential to help LA to advance
• also wearable computing, capturing more data (not analytic technique)

Literature analysis

Method

• text of papers → concept graph extraction (ontoCMaps)
• clustering
• manual content analysis
• research methods, author background, categories of papers
• interpretation

Clusters

• learning behaviour and feedback
  • PLE
  • providing feedback and enhancing reflective learning
  • visualization
  • both comp sci and education researchers, not clear research methods, mostly proposing solutions
• learning success and retention (trending down)
  • identify students at risk
  • adaptive learning systems
  • foster metacognition through peer-to-peer interaction and feedback
• data mining and learning enhancement
  • analytics of teaching and education system
  • different pedagogical models using SNA
  • (mainly) opinion papers on EDM and LA
• recommender systems (mainly in 2011)
  • attention metadata for visualization and recommendation
  • performance of recommendation algorithms
  • repositories for storing user data
  • detecting semantic relationships between learning resources
  • text mining for usage data analysis and text chat data
  • cultural influences in social behaviour
  • mainly computer science, more empirical/mature methodological approach.
• SNA (mainly first LAK)
  • sociocultural discourse analyis and argumentation theory
  • conceptual and social network patterns as indicators of meaningful learning
  • generating graphs from log data in order to reveal interactions between participants
  • predicting learning outcome and persistence using demographics, grades, learning artefacts
  • context aware interactions between learners
• social learning (growing trend)
  • association between egocentric network properties and performance
  • bimodal network from learner generated artifacts
  • SNA for reflective learning support
  • forum analysis to see different patterns of contribution
  • social blogging and knowledge development
  • visualization for reflective learning
  • self-reflection on activities and comparison with peers
  • twice as many CS papers as education papers

I really appreciate the meta-awareness of LAK as a field, thinking about how it is developing, and how it can actively "onboard" graduate students and academics - letting the community be "interpretable", providing "maps" to newcomers, seeing how contributions from different fields fit together. Very nice approach which other fields can learn from

Workshop at ML conference to work on models to predict drop-out. Want good invited speakers to make ML experts understand the educational perspective - going beyond “the data”.

Shorthand for structural equation modelling. Wants to help us relate to machine learning work with methodologies that we are more comfortable with.

Conversational role - connection with what you talk about, and how you reflect your stance within conversation, in relation to other people. People have a way that they tend to act in groups (internal scripts?).

Based on Bayes theorem. Modelling tendencies, as opposed to making predictions.

Books

• Latent class and latent transition analysis
• Multilevel and longitudinal modeling using Stata

Growth models consisting of

• personal trajectory (intercept and slope)
• subcommunity
• community

Hierarchical model

• first calculate general trend
• then adjust for each user's intercept… and slope

You can find the average community by mean, but you can also cluster groups of people that show similar patterns over time. Average within clusters, and show trends.

How to find the clusters? is this like the k-means, knn algorithms?

• each cluster has an associated distribution of attribute values for each attribute
  • based on the extent to which instances are in the cluster
• each instance has a certain probability of being in each cluster
  • based on how close its attirbute values are to typical attribute values for the cluster

Expectation maximization – iterative way.

Latent dirichlet allocation - doing this with text, a bit more complex.

• Daphne Koller MOOC on Probabilistic graphic models
• LightSide Labs - text mining tools

We can predict student drop out by the mid term, but students drop out in the first few weeks…

Paper

From QA

• Analyze 1000s of courses, cluster learning approaches (STEM learners vs others, etc).
• How did we choose features?
• Time-zone issue in MOOC data
• Becomes time-intensive to generate features - interested in scaling up features in a tractable way
• Showing models to students, what's actionable? Latent analysis.
  • Gender - if gender is a predictor in a STEM-environment… what do you do? Do you make that available to the student? Hard one.
  • Interaction variables - just because you clicked a lot, doesn't mean you consumed a lot of knowledge… Not always clear what a pattern means.
  • How do we get the right information out in week 1, week 2, etc. Best prediction is previous grade, want to do something better.
• What happens when data goes against long-established theory/ideology? How can data inform theory / be understood in the light of theory?

Approaches

• randomized trials
  • expensive
  • policy barriers (FERPA/IRB)
• data mining
  • difficult to draw conclusions post-hoc
• blending of RCT and data mining
  • Aist & Mostow
  • Pardos & Heffernan

Sequence of resource access - infer what was useful, based on sequence of responses.

• could look at most common resource accessed just before answering correctly
• optimize: score all resources, probabilistic graphic model
• Pardos EDM 2013 (Bayesian knowledge tracing in MOOC/EdX)

When will it work?

• use simulation to test out method
• 50,000 experiments, run 20 times each with different parameters
• use model to infer efficacy, sort inferred list, sort true list, look at correlation (compare using Pearson correlation)

Doesn't the simulated experiment embody our assumptions about how latent variables are connected to measured variables? So what exactly does the simulation verify?

Applications

• recommendations to students
• instructor feedback on efficacy re objectives

QA

• maybe it was the cumulation of all the steps she went through, rather than the final resource she used

Automate content analysis of discussion transcripts. Want to analyze cognitive presence, one of three main components of community of inquiry framework.

Provide feedback to instructors and learners (how they are standing compared to other learners)

Asynchronous online discussions have been researched heavily, but usually only after the course has finished – has no impact on the practice.

Content analysis techniques take a lot of time.

Community of inquiry

• social presence
• teaching presence
• cognitive presence
  • participants able to construct meaning through sustained communication
  • phases
    • triggering event
      • issue, problem, dilemma identified
    • exploration
      • move between private world of reflection and shared world of social knowledge construction
    • integration
      • filter irrelevant information, synthesize new knowledge
    • resolution
      • analyze applicability, test hypotheses, start new cycle
• all three together → educational presence

Coding scheme:

• whole message as unit of analysis
• look for particular indicators of sociocognitive processes
• requires expertise with coding instrument and domain knowledge

Text mining

• supervised learning classification problem
• cognitive presence is a latent construct
  • similar examples
    • opining mining
    • gender style difference detection

Manually coding, high IRR. Software engineering course.

Features

• unigrams, bigrams, trigrams
• part-of-speech bigrams/trigrams
• backoff bigrams/trigrams
  • backoff bigram: john is, becomes <noun> is, john <verb>
• dependency triplets (rel, head, modifier)
• backoff dependency triplets
• n of named entities in messages
• is the message the first in the discussion (“triggering event”)
• is the message a reply to the first message in the discussion

Classifier

• Linear SVM
• only features with support of 10 or more
• accuracy using 10 fold cross-evaluation
• comparison of models using McNemar's test

Tech

• Java
• Stanford CoreNLP
• Weka
• LibSVM

Problem/future work

• models cannot give us probabilities, and not easy to understand for humans
• type of features that look at previous messages
• very surface-based features

Goals

• intervention at-risk students
• advise students of best learning steps
• understand factors of behavior that determine success/failure

Data

• VLE activity
• student assessments

Challenge

• Need to understand course, not all structured the same way

Modelling

• GUHA
  • discovery of new hypothesis from the data
  • association rules
  • antecedent → consequent
  • example: week1(noclick)→week2(noclick)→submit(no) - support 0.1, confidence 0.95
  • uses LISP miner
  • hard to interpret
• Markov-chain analysis
  • stochastic process
  • no memory - next state based on current state
  • graphical model representation
  • better interpretation for users
  • 2 types:
    • activity intensity in week
    • content type in week

Activities:

• forum activity
• resource
• view of test assignment
• web subpage
• tag each activity per week (binary - use or no use)

Several presentations I missed. Quick thoughts:

• need for common data formats across LMSes, MOOCdb should include LMSes as well (what about distributed learning environments?)
  • could also be done for specific types of data, for example discussion forums
• need to share predictive models (PMML?) and make these objects of inquiry as well. need new academic incentives?
• sharing data - and even better, having human-tagged data that we can test different algorithms on
• having well-designed algorithm can help learning researchers do the next step - use it for analysis (connecting with other data sources), using it for interventions and measuring which kinds of interventions are useful etc. If everyone are using all their time on feature engineering and data transformation, we'll never get to where the data is actually useful
  • also good way of letting computer scientists collaborate with learning scientists
• look into Tincan API and X-API
• danger of using humanely tagged data and IRR as “gold standard” - often IRR is quite low, grading/coding is can be subjective